Tokenizer

class Tokenizer(config)

Bases: BatchedStep

The Tokenizer step takes specified fields and splits them into tokens to be used by a downstream step.

Parameters

• type (str) – Type of Tokenizer (sentences, word_ngrams, or spaces)

• fields (list, []) – List of fields to tokenize (in place)

• input_fields (list, None) – List of fields to tokenize from (defaults to fields)

• output_fields (list, None) – List of fields to tokenize to (defaults to fields)