# Writing Pipelets Python Engineer This page describes how to write pipelets for Squirro. This work is typically performed by python engineers, who create pipelets that project creators can use to customize their projects. ## Overview Pipelets are Python scripts that are executed on each item before it is added to Squirro’s search index. Pipelets are written in Python. They need to inherit from the [`squirro.sdk.PipeletV1`](reference.md#squirro.sdk.PipeletV1) class and implement the [`consume()`](reference.md#squirro.sdk.PipeletV1.consume) method. The simplest possible pipelet looks like this: ``` from squirro.sdk import PipeletV1 class NoopPipelet(PipeletV1): def consume(self, item): return item ``` As its name says it does nothing but return the `item` unchanged. ## Modifying Items Pipelets modify the `item` before it is returned. The `item` is represented as a Python dictionary. The available fields are documented in the [Item Format](../api/item-format.md#data-loading-item) reference. The following example illustrates modifying an `item`: ``` from squirro.sdk import PipeletV1 class ModifyTitlePipelet(PipeletV1): def consume(self, item): item['title'] = item.get('title', '') + ' - Hello, World!' return item ``` This pipelet will modify each `item` it processes, appending the string _“ - Hello, World!”_ to the title. ### Skipping Items To skip an `item` (i.e., don’t add it to Squirro’s search index) return `None` instead of the `item`. For example: ``` from squirro.sdk import PipeletV1 class SkipItemPipelet(PipeletV1): def consume(self, item): if not item.get('title', '').startswith('[IMPORTANT]') return None return item ``` In this example, we discard each `item` if the title does not start with the string _“[IMPORTANT]”_. ## Returning Multiple Items The pipelet is always called for each `item` individually. But in some use cases the pipelet should not just return one `item` but multiple ones. In those cases use the Python `yield` statement to return each individual `item`. For example: ``` from squirro.sdk import PipeletV1 class ExtendTitlePipelet(PipeletV1): def consume(self, item): for i in range(10): new_item = dict(item) new_item['title'] = '{0} ({1})'.format(item.get('title', ''), i) yield new_item ``` ## Dependencies Pipelets are limited in what you can do. For example the `print` statement is disallowed and you can not import any external libraries except `squirro.sdk`. If you do need access to external libraries, you need to use the [`require()`](reference.md#squirro.sdk.require) decorator. For example to log some output: ``` from squirro.sdk import PipeletV1, require @require('log') class LoggingPipelet(PipeletV1): def consume(self, item): self.log.debug('Processing item: %r', item['id']) return item ``` As seen from the example, the `@require` decorator takes a name of a dependency. That dependency is then available within the pipelet class. Use the `requests` dependency to execute HTTP requests. The following pipelet shows an example for sentiment detection: ``` from squirro.sdk import PipeletV1, require @require('requests') class SentimentPipelet(PipeletV1): def consume(self, item): text_content = ' '.join([item.get('title', ''), item.get('body', '')]) res = self.requests.post('http://example.com/detect', data={'text': text_content}, headers={'Accept': 'application/json'}) sentiment = res.json()['sentiment'] item.setdefault('keywords', {})['sentiment'] = [sentiment] return item ``` ## Available Dependencies The following dependencies are available: | Dependency | Description | | --- | --- | | `cache` | Non-persisted cache. | | `log` | Provides loggers (unstructured and structured) as class attributes: `log`, `slog`, `dlog`. | | `requests` | Provides access to the Python `requests` library to execute HTTP requests. | | `files` | Python component which provides access to data files on disk. See [Using Additional Files with Pipelets](workflow.md#pipelet-data-files). | ## Configuration ### Custom Configuration Pipelets can define a custom set of configuration properties by implementing the [`getArguments()`](reference.md#squirro.sdk.PipeletV1.getArguments) method. The configuration properties are exposed in the UI in a user friendly form (instead of a JSON input). The [`getArguments()`](reference.md#squirro.sdk.PipeletV1.getArguments) method is expected to return an array of objects defining each property. Inside each object, the fields `name`, `display_label` and `type` are required. The optional field `required` specifies whether the property is required to be filled by the user. A property can be placed in an _Advanced_ section of the configuration, by setting `advanced` to `True` on that property. The `type` property should be one of the following: `int`, `string`, `bool`, `password`, `code`. The below image shows how the different types of configuration options render in the UI. [](https://s3.amazonaws.com/download.squirro.net/docs/technical/pipelets/pipelet_config_types.png) The data structure of the configuration defined in [`getArguments()`](reference.md#squirro.sdk.PipeletV1.getArguments) is passed in to the pipelet constructor, where you can retrieve it. The example below illustrates how you can retrieve and set the configuration to `self.conig` to make it available to the pipelet instance. In this example we request the value of type `str` from the configuration property with name `suffix` and add it to the items title. ``` from squirro.sdk import PipeletV1 DEFAULT_SUFFIX = ' - Hello, World!' class ModifyTitlePipelet(PipeletV1): def __init__(self, config): self.config = config def consume(self, item): suffix = self.config.get('suffix', DEFAULT_SUFFIX) item['title'] = item.get('title', '') + suffix return item @staticmethod def getArguments(): return [ { 'name': 'suffix', 'display_label': 'Suffix', 'type': 'string', 'default': DEFAULT_SUFFIX, } ] ``` Note, that also a default value `DEFAULT_SUFFIX` is set for the suffix. This default value, as defined in the pipelet code in this example, is shown in the UI and is used as suffix if not changed. [](https://s3.amazonaws.com/download.squirro.net/docs/technical/pipelets/pipelet_configuration.png) ### Default Configuration > **Warning** > > The default configuration handling is deprecated. For new pipelets please add a [`getArguments()`](reference.md#squirro.sdk.PipeletV1.getArguments) method, as described in [Custom Configuration](#custom-configuration). It is recommended to always define [`getArguments()`](reference.md#squirro.sdk.PipeletV1.getArguments) in the pipelet. If it is not defined, by default a JSON editor is exposed. ## Documentation A pipelet class is documented using doc-strings. The first sentence (separated by period) serves as a summary in the user interface. All the remaining text is the pipelets description and is often used to document the expected configuration. The description is parsed as Markdown (using the [CommonMark](https://commonmark.org/) dialect). The [60-second overview](https://commonmark.org/help/) serves as a good reference. ``` from squirro.sdk import PipeletV1 DEFAULT_SUFFIX = ' - Hello, World!' class ModifyTitlePipelet(PipeletV1): """Modify item titles. This appends a suffix to the title of each item. When no suffix is provided, it appends the default suffix of "- Hello, World!". """ def __init__(self, config): self.config = config def consume(self, item): suffix = self.config.get('suffix', DEFAULT_SUFFIX) item['title'] = item.get('title', '') + suffix return item @staticmethod def getArguments(): return [ { 'name': 'suffix', 'display_label': 'Suffix', 'type': 'string', 'default': DEFAULT_SUFFIX, } ] ```