FileLoader#

class squirro.lib.nlp.steps.loaders.FileLoader(config)#

Bases: squirro.lib.nlp.steps.loaders.base.Loader

The file Loader looks in a specified directory_field directory for files and loads these files either line by line or as a whole and transforms them into Document items.

Parameters

  • type (str) – file

  • directory_field (str) – name of field to store directory path of a loaded file

  • encoding (str, 'utf-8') – given file’s encoding

  • output_field (str) – name of field to store line or whole file

  • per_line (bool, False) – indicates if each line represents a different item document

Example

{
    "step": "loader",
    "type": "file",
    "directory_field": "loaded_file_path",
    "output_field": "file_content"
}

Methods Summary

process(directory)

Process a directory and yield documents.

Methods Documentation

process(directory)#

Process a directory and yield documents.

Parameters

directory (str) – path to directory with *.txt files

Returns

Generator of Documents read from files

Return type

generator(Document)