Tokenizer
Tokenizer#
- class squirro.lib.nlp.steps.tokenizers.Tokenizer(config)#
Bases:
squirro.lib.nlp.steps.batched_step.BatchedStep
The
Tokenizer
step takes specified fields and splits them into tokens to be used by a downstream step.
Bases: squirro.lib.nlp.steps.batched_step.BatchedStep
The Tokenizer
step takes specified fields and splits them into tokens
to be used by a downstream step.