# libNLP This page introduces libNLP and provides information on libNLP apps, base types, and troubleshooting. ## Introduction libNLP is a proprietary Squirro text processing and understanding library powered by machine learning (ML). It is the core of ML-powered applications at Squirro such as AI Studio, Query Processing, and Q&A. ## Installation To get started, follow the instrucitons below: 1. Download it from the [Downloads section](https://go.squirro.com/downloads). 2. Install it using pip: ```console pip install squirro.lib.nlp-SQUIRRO_VERSION-py2.py3-none-any.whl ``` ## Structure and Configuration libNLP is structured as a pipeline where a user can specify a sequence of steps to load and transform unstructured data. That data is then classified and saved either within Squirro or to disk (in CSV or JSON format). The pipeline configuration itself is specified in JSON format. See below for an example `config.json`. ```json { "dataset": { "items": [{ "id": "0", "label": ["fake"], "body": "

This is a fake Squirro Item. It is composed of a couple fake sentences.

" },{ "id": "1", "label": ["not fake"], "body": "

This is not a fake Squirro Item. It is composed of a couple not fake sentences.

" },{ "id": "2", "label": ["fake"], "body": "

This is a fake Squirro Item. It is composed of a couple fake sentences.

" },{ "id": "3", "label": ["not fake"], "body": "

This is not a fake Squirro Item. It is composed of a couple not fake sentences.

" },{ "id": "4", "label": ["fake"], "body": "

This is a fake Squirro Item. It is composed of a couple fake sentences.

" }] }, "pipeline": [ { "fields": [ "body", "label" ], "step": "loader", "type": "squirro_item" }, { "fields": [ "body" ], "step": "filter", "type": "empty" }, { "input_fields": [ "extract_sentences" ], "output_fields": [ "normalized_extract" ], "step": "normalizer", "type": "html" }, { "fields": [ "normalized_extract" ], "step": "normalizer", "type": "punctuation" }, { "fields": [ "normalized_extract" ], "mark_as_skipped": true, "step": "filter", "type": "regex", "whitelist_regexes": [ "^.{20,}$" ] }, { "step": "embedder", "type": "transformers", "transformer": "huggingface", "model_name": "https://www.kaggle.com/models/google/universal-sentence-encoder/tensorFlow2/universal-sentence-encoder/2", "input_field": "body", "output_field": "embedded_extract" }, { "step": "randomizer", "type": "randomizer" }, { "input_field": "embedded_extract", "label_field": "label", "output_field": "prediction", "step": "classifier", "type": "cosine_similarity" }, { "step": "debugger", "type": "log_fields", "fields": [ "extract_sentences", "prediction" ], "log_level": "warning" } ] } ``` ## Running Configuration You can run the [`Pipeline`](base.md#squirro.lib.nlp.pipeline.Pipeline) configuration by either of the following methods: - Publish a ML workflow via [SquirroClient (Python SDK)](../api/squirro_client/index.md#squirro-client) as described in [How To Publish ML Models Using the Squirro Client](../ml/models-squirro-client.md#ml-models-squirro-client). - Adjust the ML workflow published from AI Studio as described in [AI Studio](../ai-studio/index.md#ai-studio). Note that a ML workflow created from the frontend ML workflow page is run purely as-it-is. A published ML workflow used in the enrich pipeline is adjusted on-the-fly during the run phase. For example, when applying optimization for execution. See [ML Enrichments for Pipeline Workflows](../ai-studio/enrichments-workflows.md#ai-studio-enrichments-workflow) to learn more. ## Query-Processing App For information on Query Processing app classes within libNLP, see [Query Processing](apps-query.md#libnlp-apps-query). For step-by-step instructions on how a python engineer can build custom query processing steps, see [How to Create Custom Query-Processing Steps](how-custom-query.md#libnlp-how-custom-query). ## Base types The fundament of libNLP, base types include the following classes: - Document - Pipeline - Runner For detailed information on libNLP base types, see [Base Types](base.md#libnlp-base). ## Step Types These provide access to all predefined steps in libNLP, including the following: - [Analyzers Package](steps/analyzers.md#libnlp-steps-analyzers) - [Classifiers Package](steps/classifiers.md#libnlp-steps-classifiers) - [Debuggers Package](steps/debuggers.md#libnlp-steps-debuggers) - [Embedders Package](steps/embedders.md#libnlp-steps-embedders) - [External Package](steps/external.md#libnlp-steps-external) - [Filters Package](steps/filters.md#libnlp-steps-filters) - [Flow Package](steps/flow.md#libnlp-steps-flow) - [Loaders Package](steps/loaders.md#libnlp-steps-loaders) - [Normalizers Package](steps/normalizers.md#libnlp-steps-normalizers) - [Projectors Package](steps/projectors.md#libnlp-steps-projectors) - [Runtimes Package](steps/runtimes.md#libnlp-steps-runtimes) - [Savers Package](steps/savers.md#libnlp-steps-savers) - [Standard Types](steps/standard.md#libnlp-steps-standard) - [Tokenizers Package](steps/tokenizers.md#libnlp-steps-tokenizers) For detailed information on libNLP step types, see [Step Types](steps.md#libnlp-steps). ## Utils Utils defined in libNLP include the following: - [Standard Types](utils/standard.md#libnlp-utils-standard) - [Cache Package](utils/cache.md#libnlp-utils-cache) - [Model Management Package](utils/model_management.md#libnlp-utils-model-management) - [Transformers Package](utils/transformers.md#libnlp-utils-transformers) For more detailed information, see [Utils](utils.md#libnlp-utils). ## Troubleshooting For answers to common troubleshooting questions, including those listed below, see [Troubleshooting & FAQ](trouble.md#libnlp-trouble). - Something is not working properly. How can I debug it? - How can I test my ML Worklflow before uploading to the Squirro Platform? - How can I upload my ML Worklflow to the Squirro Platform? - How can I use a ML Workflow in the enrich pipeline? - I have developed my own machine learning approach and would like to know how to use it for data enrichment in Squirro.