make_tokenizer

make_tokenizer(config)

Tokenizer factory

Parameters:

config (dict) – step config

Returns:

created Tokenizer object

Return type:

Tokenizer