<!-- Source: https://docs.squirro.com/en/latest/api/squirro.lib.nlp.steps.tokenizers.SentencesNLTKTokenizer.html -->
# SentencesNLTKTokenizer

**`class SentencesNLTKTokenizer(config)`**

Bases: [`Tokenizer`](squirro.lib.nlp.steps.tokenizers.Tokenizer.md#squirro.lib.nlp.steps.tokenizers.Tokenizer)

Sentences [`Tokenizer`](squirro.lib.nlp.steps.tokenizers.Tokenizer.md#squirro.lib.nlp.steps.tokenizers.Tokenizer) splits the input fields by sentences with NLTK and additional custom rules.

**Input** - all input fields need to be of type [`str`](https://docs.python.org/3.11/library/stdtypes.html#str).

**Output** - all output fields are filled with data of type [`list`](https://docs.python.org/3.11/library/stdtypes.html#list) [ [`str`](https://docs.python.org/3.11/library/stdtypes.html#str) ].

Parameters

- `type` ([`str`](https://docs.python.org/3.11/library/stdtypes.html#str)) – sentences_nltk
- `rules` ([`list`](https://docs.python.org/3.11/library/stdtypes.html#list)`,``[``]`) – list of additional splitting rules. example: [‘-‘,’**’]
- `cleaning` ([`dict`](https://docs.python.org/3.11/library/stdtypes.html#dict)`,``{}`) – dict of additional cleaning rules. example: {‘U.N.’:’UN’}

**Example**

```json
{
    "step": "tokenizer",
    "type": "sentences_nltk",
    "input_fields": ["body"],
    "output_fields": ["sentences"],
    "rules": ["**", "...", "…",": "],
    "cleaning": {
        "\t": " ",
        "\n": "",
        "  ": " ",
        "approx.": "approx",
        "etc.": "etc",
        "i.e.": "ie"
    }
}
```

Methods SummaryMethods Documentation

**`process_doc(doc)`**

Process a document

Parameters

`doc` ([`Document`](../technical/libnlp/base.md#squirro.lib.nlp.document.Document)) – Document

Returns

Processed document

Return type

[Document](../technical/libnlp/base.md#squirro.lib.nlp.document.Document)