Data Processing Pipeline#

This section outlines the various ways data is enriched, related, classified, indexed, and otherwise processed as part of the data processing pipeline.

Throughout these docs, you’ll see the data processing pipeline referred to simply as the pipeline.


For an overview of the data processing pipeline architecture, sections, and processing, see the Pipeline Overview.

Pipeline Steps#

The data processing pipeline is a collection of built-in steps, with the option to create custom steps.

For information on built-in steps, see Built-In Steps.

For information on custom steps, see Custom Steps.

Pipeline Editor#

The Pipeline Editor allows you to configure pipeline steps directly within the Squirro user interface.

To learn more, see Pipeline Editor.

Pipeline Reruns#

There are situations where the pipeline will need to be rerun. This includes rerunning pipeline workflows, individual pipeline steps, and specific enrichments.

See Pipeline Reruns for more information.

Pipeline Prioritization#

Squirro supports prioritization for certain data sources, which can be configured within the user interface.

To learn more, see Pipeline Prioritization.

Content Streamers#

In Squirro, a content streamer is a component that provides the functionality for managing the inputstream.

Content streamers are used by any Squirro service (e.g., api.provider, ingester) that needs to interact with the inputstream.

For more information on this topic, see Content Streamers.

Indexing Common Document Formats#

Squirro’s pipeline supports the processing of complex document types, such as common office formats like Word or Excel files or PDFs.

See Indexing Common Formats for instructions on setting up the pipeline to achieve this.

Trend Detection#

Trend detection is used to find anomalies in time-series data.

For reference information about trend detection, see Trend Detection.

For step-by-step instructions on using trend detection, see data-processing-trend-how.

Reset Project#

If you need to reset your project, you can do so from the user interface.

See Reset Project for more information.

Processing Errors#

Some steps in the pipeline may abort with an error. In those cases, the item is tagged with a Processing Error keyword.

For information on these errors, see Processing Errors.