<!-- Source: https://docs.squirro.com/en/latest/api/squirro.lib.nlp.steps.tokenizers.HtmlTokenizer.html -->
# HtmlTokenizer

**`class HtmlTokenizer(config)`**

Bases: [`Tokenizer`](squirro.lib.nlp.steps.tokenizers.Tokenizer.md#squirro.lib.nlp.steps.tokenizers.Tokenizer)

The HTML [`Tokenizer`](squirro.lib.nlp.steps.tokenizers.Tokenizer.md#squirro.lib.nlp.steps.tokenizers.Tokenizer) splits the input fields by HTML tags into sentences.

**Input** - all input fields need to be of type [`str`](https://docs.python.org/3.11/library/stdtypes.html#str).

**Output** - all output fields are filled with data of type [`list`](https://docs.python.org/3.11/library/stdtypes.html#list) [ [`str`](https://docs.python.org/3.11/library/stdtypes.html#str) ].

Parameters

`type` ([`str`](https://docs.python.org/3.11/library/stdtypes.html#str)) – html

**Example**

```json
{
    "name": "html",
    "step": "tokenizer",
    "type": "html",
    "input_fields": ["body"],
    "output_fields": ["tokenized_body"]
}
```

Methods SummaryMethods Documentation

**`process_doc(doc)`**

Process a document

Parameters

`doc` ([`Document`](../technical/libnlp/base.md#squirro.lib.nlp.document.Document)) – Document

Returns

Processed document

Return type

[Document](../technical/libnlp/base.md#squirro.lib.nlp.document.Document)