Troubleshooting and FAQ

How can I remove a facet from an item using a pipelet?

You can remove a facet within a pipelet by setting it to an empty list:

item["keywords"][facetName] = []

where facetName is the name of the facet you want to remove.

My pipelet is validated but upload fails.

The pipelet validate throws no error but when uploading the pipelet, I see the following error:

2021-12-06 12:43:16,023 ERROR: Could not upload pipelet: b'{"error": "{\\"error\\": \\"Unknown pipelet loading error\\"}"}'

Check the /var/log/squirro/plumber/plumber.log file for errors. Likely it will give you a hint on the error, for example, a failing import of a custom library that needs to be installed first.

My pipelet is not executed when run by a data loader.

Check the following components.

Configuration file:

Make sure to reference the correct pipelet configuration file from your load.sh script. For example add:

--pipelets-file 'pipelets.json' \

Item transformation:

If you see the following error:

INFO:squirro_data_load[23476]:--pipelets-file is ignored when item transformation takes place on the server. Please add the pipelet as a step of the workflow.

make sure to include the transform step in your load.sh script:

--transform-items \

Consume method:

In order to use pipelets with the data loader, the consume() method must be implemented. The consume_multiple() method is only executed when the pipelet is uploaded to the server and included in the pipeline workflow.