FileLoader

class squirro.lib.nlp.steps.loaders.FileLoader(config)

Bases: squirro.lib.nlp.steps.loaders.Loader

The FileLoader step looks in a specified directory_field directory for files to read in either line by line or as a whole.

Parameters
  • type (str) – file

  • directory_field (str) – Field to write directory into

  • encoding (str, 'utf-8') – File encoding

  • output_field (str) – Field to write data into

  • per_line (bool, False) – Whether or not each line represents a different document

Methods Summary

process(directory)

Process a directory and yield documents.

Methods Documentation

process(directory)

Process a directory and yield documents.

Parameters

directory (str) – path to directory with *.txt files

Returns

Generator of Documents read from files

Return type

generator(Document)