<!-- Source: https://docs.squirro.com/en/latest/technical/data-processing/pipeline-steps/enrich/content-augmentation.html -->
# Content Augmentation

The content augmentation enrichment fetches additional content from third-party systems.

The data processing pipeline can be instructed to fetch third-party content. Examples include external websites accessible via the HTTP(S) protocol.

|  |  |
| --- | --- |
| Enrichment name | content-augmentation |
| Stage | content |

## Overview

The `content-augmentation` step is used to fetch third party content.

It has two steps during this process: first the content of all uploaded files is fetched. That content is also used to guess the MIME type for these files. After that, and only if configured to do so, the content augmentation enrichment downloads the content from the `link` attribute. That fetched content is used to set the `body` attribute.

When link fetching is enabled, this step will often be combined with the [Noise Removal](noise-removal.md#pipeline-steps-noise-removal) enrichment.

[![image1](https://s3.amazonaws.com/download.squirro.net/docs/migrated-attachments/2949443/2950046.png)](https://s3.amazonaws.com/download.squirro.net/docs/migrated-attachments/2949443/2950046.png)

## Configuration

|  |  |
| --- | --- |
| Field | Description |
| fetch_link_content | Boolean value indicating whether to fetch the content from the web site referenced with the `link` attribute. Default: `false`. |

## Examples

The following examples all use the [SquirroClient (Python SDK)](../../../api/squirro_client/index.md#squirro-client) to show how the content augmentation enrichment step can be used.

## Item Uploader

The following example details how to enable third-party content fetching.

```python
from squirro_client import ItemUploader

# processing config to fetch 3rd party content
processing_config = {
    'content-augmentation': {
        'enabled': True,
        'fetch_link_content': True,
    },
}

uploader = ItemUploader(..., processing_config=config)
# item with a link attribute
items = [
    {
        'link': 'http://www.example.com',
        'title': 'Item 01',
    },
]
uploader.upload(items)
```

In the example above the processing pipeline is instructed to fetch the content from the site [http://www.example.com](http://www.example.com) and use it as the item `body`.

## New Data Source

The following example details how to enable third-party content fetching for a new feed data source.

```python
from squirro_client import SquirroClient

client = SquirroClient(None, None, cluster='https://next.squirro.net/')
client.authenticate(refresh_token='293d…a13b')

# processing config to fetch 3rd party content and detect boilerplate with the
# news classifier
processing_config = {
    'content-augmentation': {
        'enabled': True,
        'fetch_link_content': True,
    },
}

# source configuration
config = {
    'url': 'http://newsfeed.zeit.de/index',
    'processing': processing_config
}

# create new source subscription
client.new_subscription(
    project_id='…', object_id='default', provider='feed', config=config)
```

## Existing Data Source

The following example details how to enable third-party content fetching for an existing source. Items that have already been processed are not updated.

```python
from squirro_client import SquirroClient

client = SquirroClient(None, None, cluster='https://next.squirro.net/')
client.authenticate(refresh_token='293d…a13b')

# Get existing source configuration (including processing configuration)
source = client.get_subscription(project_id='…', object_id='…', subscription_id='…')
config = source.get('config', {})
processing_config = config.get('processing_config', {})

# Modify processing configuration
processing_config['content-augmentation'] = {
    'enabled': True,
    'fetch_link_content': True,
}
config['processing'] = processing_config
client.modify_subscription(project_id='…', object_id='…', subscription_id='…', config=config)
```

In the example above the processing pipeline is instructed to fetch the content for every new incoming item (from the `link` attribute) and use it as the item `body`.