FileLoader

FileLoader#

class FileLoader(config)#

Bases: Loader

The file Loader looks in a specified directory_field directory for files and loads these files either line by line or as a whole and transforms them into Document items.

Parameters:

  • type (str) – file

  • directory_field (str) – name of field to store directory path of a loaded file

  • encoding (str, 'utf-8') – given file’s encoding

  • output_field (str) – name of field to store line or whole file

  • per_line (bool, False) – indicates if each line represents a different item document

Example

{
    "step": "loader",
    "type": "file",
    "directory_field": "loaded_file_path",
    "output_field": "file_content"
}

Methods Summary

process(directory)

Process a directory and yield documents.

Methods Documentation

process(directory)#

Process a directory and yield documents.

Parameters:

directory (str) – path to directory with *.txt files

Returns:

Generator of Documents read from files

Return type:

generator(Document)

Contents