# Base Types
Introduces the fundament of libNLP.
## Document
Document class
**`class Document(doc_id, fields=None, skipped=False, copy_fields=True, fully_processed=True)`**
A Document is the internal representation of data inside libNLP.
Documents are streamed through a [`Pipeline`](#squirro.lib.nlp.pipeline.Pipeline) with each [`Step`](steps/standard.md#squirro.lib.nlp.steps.base.Step) acting on it
in-memory accordingly.
Parameters
- `doc_id` ([`str`](https://docs.python.org/3.11/library/stdtypes.html#str)` or `[`int`](https://docs.python.org/3.11/library/functions.html#int)) – Given [`Document`](#squirro.lib.nlp.document.Document) id
- `fields` ([`dict`](https://docs.python.org/3.11/library/stdtypes.html#dict)`, ``None`) – Dictionary of fields
- `skipped` ([`bool`](https://docs.python.org/3.11/library/functions.html#bool)`, ``False`) – Skip document during processing
- `copy_fields` ([`bool`](https://docs.python.org/3.11/library/functions.html#bool)`, ``True`) – Deep copy of fields
- `fully_processed` ([`bool`](https://docs.python.org/3.11/library/functions.html#bool)`, ``True`) – Indicates whether the document has been fully processed.
It allows marking that a step has not been prepared to process document.
Example:
```python
from squirro.lib.nlp.document import Document
document = Document(0, {"a_field": "this is a field"})
```
**`abort_processing()`**
Handle document state for steps that stopped execution pre-maturely.
Set metadata that is used outside, e.g. for pipeline-level caching.
Return type
[`Document`](#squirro.lib.nlp.document.Document)
## Pipeline
Pipeline class
**`class Pipeline(configs, path=None, cache_client=None, ml_workflow_id=None, project_id=None)`**
The [`Pipeline`](#squirro.lib.nlp.pipeline.Pipeline) class is defined by a sequential list of [`Step`](steps/standard.md#squirro.lib.nlp.steps.base.Step). It
handles [`Document`](#squirro.lib.nlp.document.Document) streaming through the each [`Step`](steps/standard.md#squirro.lib.nlp.steps.base.Step), as well as loading
and saving [`Step`](steps/standard.md#squirro.lib.nlp.steps.base.Step) configurations.
Parameters
- `configs` ([`list`](https://docs.python.org/3.11/library/stdtypes.html#list)) – sorted list of [`Step`](steps/standard.md#squirro.lib.nlp.steps.base.Step) configs
- `path` ([`str`](https://docs.python.org/3.11/library/stdtypes.html#str)`, ``'.'`) – Path to [`Step`](steps/standard.md#squirro.lib.nlp.steps.base.Step) storage
- `cache_client` ([`CacheWithExpiration`](../../api/squirro.lib.nlp.utils.cache.CacheWithExpiration.md#squirro.lib.nlp.utils.cache.CacheWithExpiration)`, ``None`) – Cache client
- `ml_workflow_id` ([`str`](https://docs.python.org/3.11/library/stdtypes.html#str)`, ``None`) – Machine learning workflow ID
- `project_id` ([`str`](https://docs.python.org/3.11/library/stdtypes.html#str)`, ``None`) – Project ID
Example:
```python
from squirro.lib.nlp.pipeline import Pipeline
from squirro.lib.nlp.document import Document
steps = [{
"step": "normalizer",
"type": "punctuation",
"fields": ["body"]
}]
pipeline = Pipeline(steps, path='.')
documents = [Document(0, {"body": "this is a field!"})]
documents = pipeline.process(documents) # returns a generator
document = list(documents) # run the pipeline
print(documents)
```
**`save()`**
Save the steps
Return type
[`None`](https://docs.python.org/3.11/library/constants.html#None)
**`load()`**
Load the steps
Return type
[`None`](https://docs.python.org/3.11/library/constants.html#None)
**`clean()`**
Clean up after steps
Return type
[`None`](https://docs.python.org/3.11/library/constants.html#None)
**`terminate()`**
Terminate any extra running processes in the steps
**`process(x)`**
Process the steps
Parameters
`x` – input for first Step of the steps
Returns
generator of Documents from the last Step of the steps
Return type
generator([Document](#squirro.lib.nlp.document.Document))
**`train(x)`**
Train the steps
Parameters
`x` – input for first Step of the steps
Returns
generator of Documents from the last Step of the steps
Return type
generator([Document](#squirro.lib.nlp.document.Document))
## Runner
Runner class
**`class Runner(config, ml_workflow_id=None, project_id=None)`**
The [`Runner`](#squirro.lib.nlp.runner.Runner) controls libNLP runs. It provides train, test, infer,
and clean functions.
Parameters
- `analyzer` ([`dict`](https://docs.python.org/3.11/library/stdtypes.html#dict)) –
Dictionary defining #Analyzer for the [`Pipeline`](#squirro.lib.nlp.pipeline.Pipeline)
Deprecated since version 3.6.3.
- `dataset` ([`dict`](https://docs.python.org/3.11/library/stdtypes.html#dict)) – Dictionary defining train, test, and infer datasets
- `path` ([`str`](https://docs.python.org/3.11/library/stdtypes.html#str)) – Path to model storage
- `pipeline` ([`dict`](https://docs.python.org/3.11/library/stdtypes.html#dict)) – Dictionary defining the [`Pipeline`](#squirro.lib.nlp.pipeline.Pipeline)
Example:
```python
from squirro.lib.nlp.runner import Runner
config = {
"dataset": {
"items": [{
"id": "0",
"label": ["fake"],
"body": "This is a fake Squirro Item. It is composed of a couple fake sentences.
"
},{
"id": "1",
"label": ["not fake"],
"body": "This is not a fake Squirro Item. It is composed of a couple not fake sentences.
"
},{
"id": "2",
"label": ["fake"],
"body": "This is a fake Squirro Item. It is composed of a couple fake sentences.
"
},{
"id": "3",
"label": ["not fake"],
"body": "This is not a fake Squirro Item. It is composed of a couple not fake sentences.
"
},{
"id": "4",
"label": ["fake"],
"body": "This is a fake Squirro Item. It is composed of a couple fake sentences.
"
}]
},
"pipeline": [
{
"fields": [
"body",
"label"
],
"step": "loader",
"type": "squirro_item"
},
{
"fields": [
"body"
],
"step": "filter",
"type": "empty"
},
{
"input_fields": [
"extract_sentences"
],
"output_fields": [
"normalized_extract"
],
"step": "normalizer",
"type": "html"
},
{
"fields": [
"normalized_extract"
],
"step": "normalizer",
"type": "punctuation"
},
{
"fields": [
"normalized_extract"
],
"mark_as_skipped": true,
"step": "filter",
"type": "regex",
"whitelist_regexes": [
"^.{20,}$"
]
},
{
"step": "embedder",
"type": "transformers",
"transformer": "huggingface",
"model_name": "https://www.kaggle.com/models/google/universal-sentence-encoder/tensorFlow2/universal-sentence-encoder/2",
"input_field": "body",
"output_field": "embedded_extract"
},
{
"step": "randomizer",
"type": "randomizer"
},
{
"input_field": "embedded_extract",
"label_field": "label",
"output_field": "prediction",
"step": "classifier",
"type": "cosine_similarity"
},
{
"step": "debugger",
"type": "log_fields",
"fields": [
"extract_sentences",
"prediction"
],
"log_level": "warning"
}
]
}
runner = Runner(config)
try:
for doc in runner.train():
print(doc)
for doc in runner.infer():
print(doc)
finally:
runner.clean()
```
**`property cache_client: CacheWithExpiration | None`**
Placeholder for the Runner’s cache client.
Subclasses can implement their own cache client using the property.
**`train()`**
Train pipeline
Returns
Generator of processed documents
Return type
generator([Document](#squirro.lib.nlp.document.Document))
**`test()`**
Validate trained pipeline
Returns
Test summary from analyzer
Return type
[dict](https://docs.python.org/3.11/library/stdtypes.html#dict)
**`infer()`**
Infer with pipeline
Returns
Generator of processed documents
Return type
generator([Document](#squirro.lib.nlp.document.Document))
**`clean()`**
Clean the pipeline
Return type
[`None`](https://docs.python.org/3.11/library/constants.html#None)
**`terminate()`**
Terminates the pipeline
Return type
[`None`](https://docs.python.org/3.11/library/constants.html#None)