<!-- Source: https://docs.squirro.com/en/latest/technical/pipelets/workflow.html -->
# Development Workflow

Profile: Project Creator, Python Engineer

For developing pipelets, Squirro provides the `pipelet` command line tool as part of the [Squirro Toolbox](../api/toolbox/index.md#api-sdk-toolbox).

Project creators and python engineers working from the command line can use the `pipelet` tool to create, validate, test and deploy pipelets.

## Develop

The first step is to create the pipelet. In the following examples the pipelet code was written to a file called `pipelet.py` in the current directory.

**pipelet.py**

```
from squirro.sdk import PipeletV1

class ModifyTitlePipelet(PipeletV1):
    def consume(self, item):
        item['title'] = item.get('title', '') + ' - Hello, World!'
        return item
```

## Validate

On the command line execute the `pipelet validate` command to verify that there are no errors in the pipelet code. This will ensure that no modules are imported that are disallowed from pipelets. See the section on [Dependencies](write.md#pipelets-dependencies) for more information.

```
pipelet validate pipelet.py
```

## Test

Use the `pipelet consume` command to simulate pipelet running. For this purpose, test items should be present in JSON text files on the disk. In the following example, there is a file called `item.json` located in the current directory with the content:

**item.json**

```
{
    "title": "Sample",
    "id": "first_item"
}
```

To test the pipelet with this test file, use:

```
pipelet consume pipelet.py -i item.json
```

This command will output the items that have been returned by the pipelet:

```
Loading items...
Loading item.json ...
Loaded.
Consuming item first_item
yielded item
{u'id': u'first_item', u'title': u'Sample - Hello, World!'}
```

On top of these manual tests, you can implement automated tests using the usual Python tools such as [Nose](https://nose.readthedocs.io/en/latest/).

## Deploy

Once the pipelet is ready, a server administrator can upload it to the Squirro server using the following command:

```
pipelet upload --token <your_token> --cluster <cluster> pipelet.py "Hello World"
```

This will make the pipelet available with the name “Hello World”. In order to update the pipelet code on the server, you can re-executed this command at any time.

To use the pipelet in the pipeline of a project, you can use the [Pipeline Editor](../data-processing/pipeline-editor.md#data-processing-pipeline-editor) UI to add the pipelet as an enrichment step to the selected pipeline workflow.

## Using Additional Files with Pipelets

In many cases, additional files like libraries and pre-trained models must be uploaded and used by a pipelet when it is run.

Using the following steps, a server administrator can upload files together with the pipelet and access them in the pipelet code:

1. Specify the additional files in the pipelet upload command using the `--data-file` flag. For Example:

   > **Pipelet Upload Command**
   > 
   > 
   > 
   > ```
   > pipelet upload \
   >     --data-file 'resource.txt' \
   >     --cluster <cluster> \
   >     --token <your_token> \
   >     'pipelet.py' \
   >     'TestPipelet'
   > ```
   > 
   > 
   > 
   > In this example, we are uploading a file _“resource.txt”_ along with the pipelet.
2. Access the contents of the file(s) from within the pipelet

   > **Pipelet File**
   > 
   > 
   > 
   > ```
   > from squirro.sdk import require, PipeletV1
   > 
   > @require('files')
   > class TestPipelet(PipeletV1):
   >     def consume(self, item):
   >         with self.files.get_file('resource.txt') as f:
   >             data = f.read()
   > ```