Flow (Change Pipeline) Pipeline Step#
Flow includes steps that manipulate the direction of the items in the pipeline workflows.
Change Pipeline is the exposed Flow step.
Change Pipeline Overview#
The Change Pipeline step redirects the data that it receives into a different Pipeline Workflow by enabling the creation of pipeline stages: chaining multiple pipeline workflows, where each workflow indexes the data into Squirro, and then passes them to the next workflow for further processing.
For example, it can be used as the last step of a lightweight workflow to index the data as fast as possible, and then redirect them to a more heavyweight workflow that includes computationally expensive steps.
Additionally this step can change the priority of the redirected items. This can be done to ensure that the important pipeline steps such as initial data indexing happens more quickly and that the slower components do not block the ingestion of new data. See Pipeline Prioritization for further information.
The picture above depicts two workflows:
the Light Workflow, which includes a small set of steps that they are generally fast to execute, and as its last step, the Change Pipeline step which is configured to redirect the data into the Heavy Workflow.
the Heavy Workflow, which includes a fictional “Heavy Pipelet” step, which represents a pipelet that is computationally expensive and may take a lot of time to execute.
In this scenario, the data will undergo a basic enrichment and get indexed into Squirro by the Light Workflow, and then continue to the Heavy Workflow, which will further enrich the items and finally index them into Squirro again for the new enrichment to become available.
Configuration#
Field |
Description |
workflow_name |
The name of the Pipeline Workflow into which the data will be redirected. |
batch_priorioty |
The priority with which to queue the redirected items. Allowed values: Default value: |