The page provides an overview of pipeline reruns and instructions on how to perform the following types of reruns:
Rerun pipeline workflows
Rerun individual pipeline steps
Rerun Pipeline Workflows#
The Pipeline Workflows support processing data that have been already processed once.
Every workflow can be used by none, one, or more data sources. The rerun functionality of a workflow uses the data from data sources that have been already retrieved and ingested once.
For example, in the above screenshot, there is a workflow for processing Binary Documents and is configured to be used by four data sources. Therefore, the rerun functionality will use as input the processed data of these four sources.
For the Rerun from Index mode, you can enable the project configuration option
datasource.rerun.index.consider-all-project-items if you would like to use all the processed data in the project as input and not only the data from the configured data sources of the workflow.
This functionality can be proven useful when you want to modify your workflow, for example, by adding a new step to it or modifying the configuration of an existing step, and would like to execute it against the same set of data and retrieve a potentially different set of Squirro Items which may be more relevant to your needs.
To invoke the rerun of a workflow, click on its three dots menu, and there you’ll find the
When you click on the
Rerun option, a popup window is displayed where you can choose which rerun mode you would like to invoke.
There are two rerun modes available:
Rerun from Raw Data
Rerun from Index
Rerun from Raw Data#
The rerun from raw data utilizes the data as they were retrieved by a Squirro data loader plugin from the actual data source.
When it is available, it is recommended to be used as it ensures a clean rerun.
Rerun from Index#
The rerun from index utilizes the actual Squirro Items which are stored in the storage layer of your Squirro instance.
This mode is always available. However, it is not considered as stable as the other mode, and when the workflow includes a non-idempotent step the resulting Squirro Item will not be the same as the first time that it got ingested, even if the workflow has not changed.
As an example, consider a pipelet that modifies the title of an item by appending an exclamation mark. If we invoke the rerun functionality of a workflow on this item n times, the resulting Squirro Item will contain n exclamation marks in its title.
Rerun Individual Pipeline Steps#
The Pipeline Editor provides the functionality to rerun individual steps of a pipeline workflow on data for which the pipeline is configured.
Rerunning of an individual step is typically required when you have added or changed an enrichment in your pipeline workflow and want to (re-) apply the enrichment to already indexed data.
Example: Say you have ingested a set of PDF documents using a computationally expensive workflow for binary documents. You would now like to run the documents through a recently developed text classification model to enrich the data with labels resulting from the classification. To avoid rerunning the entire workflow again, you can choose to only run the classification step.
Navigate to the pipeline that contains the step you want to rerun. In the pipeline editor, hover over the step, click the three dots, then select Rerun from the dropdown (see screenshot below).
You can configure the following two options:
The query filters the set of item for which you want to rerun the step. Use the standard Squirro query syntax.
Providing a query is optional. If no query is provided, the step runs on all the items of the data sources that are configured to use this pipeline workflow.
Check to run any linked steps that are required for ingesting and enriching the data successfully.
Enrichments of the step you are rerunning are not persisted if you omit running the linked steps. Omitting linked steps is meant for development and testing purposes, to check that items successfully run through the enrichment step.
Please be aware that when you add a new step to a pipeline workflow, you are not able to access its Rerun option unless you save the workflow and then hit refresh.
In the above screenshot, we submit the query
source_type:ZIP. The query selects items that were indexed using a ZIP data source. The rerun will not affect any other items from data sources that are configured to use this workflow. By checking the Run linked steps option, the 5 steps linked to the Proximity Filter step will also run upon clicking the RERUN button.
The Studio plugin for rerunning enrichments has been deprecated since Squirro version 3.4.4. It’s recommended that you use the rerun functionalities provided in the Pipeline Editor.
Rerun Enrichments is used yo apply enrichments after ingestion.
Select Rerun Enrichments in the AI Studio tab.
The drop-down provides a list of the configured enrichments in the project. Select the enrichment, that should be executed.
Optionally enter a query on which the enrichment should be run. For search tagging enrichments the query is pre-filled automatically, but can also be changed. An empty query will cause the enrichment to run on all the items in the project.
Rerunning of enrichments has a few limitations at the moment. If any of these are an issue, then the best alternative is to do a fresh load of the data. Where that does not seem possible, contact Squirro support.
Type of Enrichments#
Rerunning is only supported for search tagging and pipelets. Built-In Steps and AI Studio models cannot be rerun.
Number of items: only pipelets that return exactly one item are supported. There is no support for pipelets that return
None(to remove an item from the index) or that
yieldmore than one item.
Type of updates: only
keywordscan be updated with a rerun. Any other updates, such as changing the title or body of an item, are silently ignored.