make_tokenizer

squirro.lib.nlp.steps.tokenizers.make_tokenizer(config)

Tokenizer factory

Parameters

config (dict) – step config

Returns

created Tokenizer object

Return type

Tokenizer