# Troubleshooting and FAQ Profile: Project Creator, Python Engineer This page contains some common issues and questions regarding pipelets. Project creators are typically responsible for resolving issues with pipelets, though python engineers may also be involved in the writing of pipelets. ## Pipelet Not Uploading If you are having trouble uploading a pipelet, you may receive an error message similar to the following: ``` ERROR: Could not upload pipelet: b’{“error”: “{\“error\”: \“Unknown pipelet loading error\”}”}’ ``` If this is the case, try following the steps below to identify and resolve the issue: 1. Check the outputs of the `pipelet validate` and `pipeline upload` commands for any errors as follows: ``` pipelet -vvvv validate .py ``` ``` pipelet -vvvv upload --token --cluster pipelet.py "Pipelet Name" ``` 1. Look through the output(s) and try to identify any errors. If you are unable to identify the error, proceed to the next step. 2. Check the `plumber.log` for errors. You can check the logs by SSHing into your instance: ``` $ ssh username@host ``` then checking the log files: ``` [user@host] $ less /var/log/squirro/plumber/plumber.log ``` > **Note** > > Alternatively, you may be able to catch it as it happens by attempting to upload again using `[user@host] $ squirro_tail_errors`. 1. At this point, the error should be identified and you can resolve it. If it is not, see potential _Module Errors_ troubleshooting steps below. ### Module Errors Oftentimes, errors found in the plumber logs relate to a module that is not installed on the server, or if modules are versioned, an incorrectly versioned module. To identify these issues, log on to the remote server as shown in the section above, then follow the steps below: 1. Check which modules are already installed on the server python environment by running the following: ``` [user@host] $ squirro_activate ``` 1. List packages using the following command: ``` [(virtualenv311) user@host] $ pip list | column ``` 1. Verify if you are importing any package NOT listed. If this is the case (i.e. if you are importing an unlisted package), visit the [Squirro Support website](https://go.squirro.com/support) and submit a technical support request to install the module. ## 500 Errors If you are receiving 500 errors, there is likely an NGINX issue. This is particularly true of servers running heavy loads where timeouts are more likely. If you believe this may be the case, you can try increasing the timeout value by following the steps below: 1. Modify `step_plumber_timeout_secs` in `/etc/squirro/ingester.ini` under the `[pipeline]` heading. 2. For NGINX (the Squirro webserver), open `/etc/nginx/conf.d/plumber-fastcgi.inc` and edit the value for `fastcgi_read_timeout`. Its default is 3600, so you can increase to between 1.5x (5400) and 2x (7200) as a potential starting point. Reference: To learn more, see the [Pipeline](../admin/configuration/config-files/common-ini.md#common-ini-pipeline) section of the `commin.ini` config documentation. If this doesn’t resolve the issue, visit the [Squirro Support website](https://go.squirro.com/support) and submit a technical support request. ## Questions How can I remove a facet from an item using a pipelet? You can remove a facet within a pipelet by setting it to an empty list: ``` item["keywords"][facetName] = [] ``` where _facetName_ is the name of the facet you want to remove. My pipelet is validated but upload fails. The `pipelet validate` throws no error but when uploading the pipelet, I see the following error: ```console 2021-12-06 12:43:16,023 ERROR: Could not upload pipelet: b'{"error": "{\\"error\\": \\"Unknown pipelet loading error\\"}"}' ``` Check the `/var/log/squirro/plumber/plumber.log` file for errors. Likely it will give you a hint on the error, for example, a failing import of a custom library that needs to be installed first. My pipelet is not executed when run by a data loader. Check the following components. _Configuration file:_ Make sure to reference the correct pipelet configuration file from your `load.sh` script. For example add: ```console --pipelets-file 'pipelets.json' \ ``` _Item transformation:_ If you see the following error: ```console INFO:squirro_data_load[23476]:--pipelets-file is ignored when item transformation takes place on the server. Please add the pipelet as a step of the workflow. ``` make sure to include the transform step in your `load.sh` script: ```console --transform-items \ ``` _Consume method:_ In order to use pipelets with the data loader, the [`consume()`](reference.md#squirro.sdk.PipeletV1.consume) method must be implemented. The [`consume_multiple()`](reference.md#squirro.sdk.PipeletV1.consume_multiple) method is only executed when the pipelet is uploaded to the server and included in the pipeline workflow.