make_tokenizer#

make_tokenizer(config)#

Tokenizer factory

Parameters

config (dict) – step config

Returns

created Tokenizer object

Return type

Tokenizer