FileLoader¶

class squirro.lib.nlp.steps.loaders.FileLoader(config)¶

The FileLoader step looks in a specified directory_field directory for files to read in either line by line or as a whole.

Parameters

type (str) – file
directory_field (str) – Field to write directory into
encoding (str, 'utf-8') – File encoding
output_field (str) – Field to write data into
per_line (bool, False) – Whether or not each line represents a different document

Methods Summary

process(directory)

Process a directory and yield documents.

Methods Documentation

process(directory)¶

Process a directory and yield documents.