Tokenizer¶
-
class
squirro.lib.nlp.steps.tokenizers.
Tokenizer
(config)¶ Bases:
squirro.lib.nlp.steps.batched_step.BatchedStep
The
Tokenizer
step takes specified fields and splits them into tokens to be used by a downstream step.
squirro.lib.nlp.steps.tokenizers.
Tokenizer
(config)¶Bases: squirro.lib.nlp.steps.batched_step.BatchedStep
The Tokenizer
step takes specified fields and splits them into tokens
to be used by a downstream step.