<!-- Source: https://docs.squirro.com/en/latest/technical/api/provider-api.html -->
# Provider API

The Provider API is a REST API for pushing data directly into Squirro. It is the push-based counterpart to the native data connectors, which pull data from external sources.

Use the Provider API when:

- A native Squirro connector does not exist for your data source.
- A third-party middleware or vendor collects data on your behalf and needs to forward it to Squirro.
- Your integration architecture requires the source system to initiate data transfers rather than relying on scheduled Squirro jobs.

When items are submitted through the Provider API, they go through the same ingestion pipeline as items from native connectors. That pipeline applies the same data processing steps, including transformation, enrichment, and indexing.

This page covers the three endpoints that a push-based integration uses: [Push Items](#push-items), [Push Rows](#push-rows), and [Mark Source as Processed](#mark-source-as-processed).

## Before You Start

To make a successful call to the Provider API, you need:

1. A cluster URL.
2. A tenant identifier.
3. A project identifier.
4. A source identifier.
5. An authentication token.

### Cluster URL

Your cluster URL is the address of your Squirro instance, for example `https://instance.example.com`. For step-by-step instructions, see the [Connecting to Squirro](../ui/connecting.md#ui-connecting) page.

### Tenant Identifier

Every endpoint path contains the tenant identifier. It does not necessarily match the subdomain of your cluster URL, so do not derive it from the URL. When you exchange a refresh token for an access token, the response returns the tenant identifier in the `tenant` field. For the request and response format, see the [Authentication](authentication.md#api-authentication) page.

The Squirro Python SDK reads the tenant identifier from that response and exposes it as `client.tenant` after authentication. The `ItemUploader` class uses it automatically.

### Project Identifier

Navigate to Setup → Settings → Properties in the Squirro UI to find your project identifier. For step-by-step instructions, see the [Locate Your Project Identifier](../ui/connecting.md#connecting-project-id) page.

To retrieve it programmatically, use the `get_projects()` method from the Squirro Python SDK:

```python
from squirro_client import SquirroClient

client = SquirroClient(None, None, cluster="https://<cluster>")
client.authenticate(refresh_token="<your_refresh_token>")

for project in client.get_projects():
    print(project["id"], project["title"])
```

### Source Identifier

Open the Spaces Menu in the top-right corner of the Squirro UI (for more information, see the [Squirro Spaces](../ui/spaces.md#ui-spaces) page), navigate to Server → Data → Data Sources, open the three-dot menu next to the source, and select Copy Data Source ID.

To retrieve it programmatically instead, use the `get_sources()` method from the Squirro Python SDK, with the `client` from the previous example:

```python
sources = client.get_sources("<project_id>")
for source in sources:
    print(source["id"], source["name"])
```

For the full reference, see the [APIs by Topic](squirro_client/sc-api-topics.md#sc-api-topics) page.

### Create a Source for Push Ingestion

The Provider API requires a source configured for push-based ingestion. To create one, navigate to Setup → Data in the Squirro UI, and select the Data Import category. Choose either JSON or CSV depending on the format of data your integration sends.

The source does not need to have a schedule configured. It only needs to exist so that the Provider API has a valid destination to push data to.

## Authentication

All Provider API endpoints require a valid Bearer access token in the `Authorization` header. The `Content-Type` header is required for all POST requests.

```text
Authorization: Bearer <access_token>
Content-Type: application/json
```

Always transmit tokens over HTTPS. Never log or print the `Authorization` header value.

For information about generating and using access tokens, see the [Authentication](authentication.md#api-authentication) page.

For automated integrations, use a service account rather than a personal user token. A service account provides a long-lived refresh token that does not depend on a specific user. For instructions on creating a service account and generating a token, see the [Create a Service Account](squirro_client/service-account.md#service-account) page.

By default, the access token derived from the refresh token expires after 10 minutes. The Squirro Python SDK handles this automatically. If you call the Provider API directly without the SDK, exchange the refresh token for a new access token by sending a `POST` request to `/api/user/oauth2/token` with `grant_type=refresh_token`. Repeat this exchange before each session or whenever a `401` response is returned. For the request and response format, see the [Authentication](authentication.md#api-authentication) page.

The token must have the `sources.write.*` permission on the target project. To limit the impact of a compromised token, restrict the service account to only the specific project it needs to write to, rather than granting tenant-wide access. When creating a service account grant, include this permission explicitly:

```python
grant = client.new_grant(
    user_id=user_id,
    type="user",
    project_permissions=["sources.write.*"],
)
```

For instructions on scoping a service account to specific projects and rotating a compromised token, see the [Create a Service Account](squirro_client/service-account.md#service-account) page.

### Network-Level Access Control

Bearer token authentication is the primary access control for the Provider API. For deployments that require an additional network-level layer, such as restricting inbound calls to known source IP ranges or enforcing firewall or VPN policies, contact your Squirro administrator to discuss the options available for your deployment.

The Provider API is served over HTTPS. Clients pushing data to the cluster must allow outbound HTTPS traffic on port 443 to the cluster URL.

## Base URL

All Provider API endpoints are served under the following base path:

```text
https://<cluster>/api/provider
```

For example, the full URL for the Push Items endpoint on a cluster at `https://instance.example.com` is:

```text
https://instance.example.com/api/provider/v1/{tenant}/projects/{project_id}/sources/{source_id}/push_items
```

## Integration Workflow

A typical push-based integration repeats the same cycle on every run:

1. **Authenticate.** Exchange the service account refresh token for an access token. The Squirro Python SDK does this automatically. For direct HTTP calls, see the [Authentication](authentication.md#api-authentication) page.
2. **Push data.** Send items to the [Push Items](#push-items) endpoint, or rows to the [Push Rows](#push-rows) endpoint. Split large datasets into batches and submit each batch separately. Because items are deduplicated by `id`, re-pushing an item with the same `id` updates it rather than creating a duplicate, which makes a failed batch safe to resend.
3. **Mark the source as processed.** Optionally, call the [Mark Source as Processed](#mark-source-as-processed) endpoint at the end of the run to set the `processed` flag of the source.

In asynchronous mode (the default), a `202` response confirms that items were accepted into the ingestion queue, not that they were indexed. To confirm indexing and surface any pipeline errors, check the source status under Server → Data → Data Sources in the Squirro UI, or retrieve it programmatically with the `get_sources()` method.

## Synchronous vs Asynchronous Processing

By default, the Provider API queues items asynchronously (`priority=0`). The API returns `202` as soon as the items are accepted, and processing happens in the background. That is the recommended mode for most integrations.

Setting `priority=1` switches to synchronous processing. In that mode, the batch is processed before the response returns. That does not mean the items already appear in search results when the response arrives. Use synchronous mode only when your integration depends on the batch being processed before its next step.

> **Notes**
>
> - Synchronous mode has a maximum batch size enforced by the server. If your batch exceeds this limit, the API returns a `400` error. Contact your Squirro administrator to check or adjust this limit.
> - For large-volume ingestion, asynchronous mode is more reliable. Split large datasets into batches of fewer than 100 items and submit each batch separately.

## Push Items

Submits a batch of Squirro items to an existing source. The items are validated, preprocessed, and placed in the ingestion queue.

`POST /v1/{tenant}/projects/{project_id}/sources/{source_id}/push_items`

Path parameters:

| Parameter | Description |
| --- | --- |
| `tenant` | Your Squirro tenant identifier. |
| `project_id` | The identifier of the project that contains the source. |
| `source_id` | The identifier of the source to push data to. |

Query parameters:

| Parameter | Description |
| --- | --- |
| `priority` | Processing priority. Set to `1` for synchronous (priority) processing. Defaults to `0` (asynchronous). See [Synchronous vs Asynchronous Processing](#synchronous-vs-asynchronous-processing) for details. |
| `pipeline_workflow_id` | Optional identifier of a specific pipeline workflow to use when processing the items. This parameter applies only in synchronous mode (`priority=1`). In asynchronous mode, the default workflow for the source is always used, and any value set here is ignored. For more information about pipeline workflows, see the [ML Enrichments for Pipeline Workflows](../ai-studio/enrichments-workflows.md#ai-studio-enrichments-workflow) page. |

Request body:

A JSON array of items. Each item must include at minimum one of `title`, `body`, or `files`. Use the `body` field for inline content and the `files` field to attach documents, either inline as Base64 or by URL. Squirro recommends also providing `id`, `link`, and `created_at` (in ISO 8601 format, for example `2025-04-01T08:00:00Z`) to ensure correct deduplication and display. For a complete description of all available fields, see the [Item Format](item-format.md#data-loading-item) page.

```json
[
    {
        "id": "unique-item-id-1",
        "title": "Quarterly Report Q1 2025",
        "body": "<p>Revenue increased by 12% in Q1 2025.</p>",
        "link": "https://intranet.example.com/reports/q1-2025",
        "created_at": "2025-04-01T08:00:00Z",
        "keywords": {
            "department": ["Finance"],
            "document_type": ["Report"]
        }
    }
]
```

> **Notes**
>
> - If you do not provide an `id`, one is generated from the file names of the item, from its `link` and `title`, or from its `body` when those are missing. Changing those fields later creates a new item instead of updating the existing one. Squirro recommends providing a stable `id` from your source system.
> - If you submit an item with the same `id` as an existing item in the source, the existing item is updated.
> - The Provider API creates and updates items only. It cannot delete items. To remove items that no longer exist in your source system, use the `delete_item()` method of the Squirro Python SDK. For the full reference, see the [APIs by Topic](squirro_client/sc-api-topics.md#sc-api-topics) page.
> - The processing order of items is not guaranteed, whether within a single batch or across separate requests. If two items share the same `id`, the last one processed determines the stored content.
> - The `body` field accepts HTML. Plain text values are also accepted.
> - A keyword with a reserved name, such as `source`, is dropped from the item without an error. The item and the rest of the batch are still indexed. For the list of reserved names, see the [Labels](../data-loading/labels.md#data-loading-labels) page.
> - Item content is indexed in plaintext and is visible in search results to any user with access to the project. Do not push credentials, secrets, or unredacted personal data unless your project access controls and data retention policies explicitly permit it.

Response:

| Status code | Description |
| --- | --- |
| `202` | Items accepted and queued for processing. |
| `400` | One or more items failed validation. Items that pass validation are still queued. The response body contains an `errors` list with details about each rejected item. |
| `401` | The access token is missing or expired. Exchange the refresh token for a new access token and retry. |
| `403` | The token does not have the `sources.write.*` permission on this project, or the tenant or project identifier is incorrect. |
| `404` | The specified source was not found. |
| `413` | The request body exceeds the maximum size allowed by the server. Reduce the batch size and resubmit. |

Example response (success):

```text
{
    "status": "queued"
}
```

Example response (validation error):

```text
{
    "status": "error",
    "errors": [
        "invalid item 0: one of `title`, `body`, or `files` is mandatory"
    ]
}
```

> **Note**
>
> A `400` response with an `errors` list does not mean the entire batch was rejected. Valid items in the same batch are still queued.

## Push Rows

Submits raw tabular data (rows) to an existing source. Unlike the Push Items endpoint, rows are not validated or preprocessed before queuing. Transformation and field mapping happen later in the pipeline.

Use this endpoint when your data is in a flat, tabular format. If your data is already structured as Squirro items, use [Push Items](#push-items) instead.

The field mappings of the source turn rows into items, so the source must have them. You set them when you create the source in the UI, on the _Map to Item Fields_ and _Map to Labels_ steps. For more information, see the [How to Load Data Using the UI](../data-loading/how-to/how-load-ui.md#data-loading-how-ui) page. Without mappings, the endpoint still returns `202` and the batch is processed, but no items are created.

`POST /v1/{tenant}/projects/{project_id}/sources/{source_id}/push_rows`

Path parameters:

| Parameter | Description |
| --- | --- |
| `tenant` | Your Squirro tenant identifier. |
| `project_id` | The identifier of the project that contains the source. |
| `source_id` | The identifier of the source to push data to. |

Query parameters:

| Parameter | Description |
| --- | --- |
| `priority` | Processing priority. Set to `1` for synchronous (priority) processing. Defaults to `0` (asynchronous). See [Synchronous vs Asynchronous Processing](#synchronous-vs-asynchronous-processing) for details. |
| `pipeline_workflow_id` | Optional identifier of a specific pipeline workflow to use when processing the rows. This parameter applies only in synchronous mode (`priority=1`). In asynchronous mode, the default workflow for the source is always used, and any value set here is ignored. For more information about pipeline workflows, see the [ML Enrichments for Pipeline Workflows](../ai-studio/enrichments-workflows.md#ai-studio-enrichments-workflow) page. |
| `base64_encoded_columns` | Comma-separated list of column names whose values are Base64-encoded. The pipeline decodes them before processing. |

Request body:

A JSON array of row objects. The field names in each row must match the field mappings of the source. There are no required fields at the API level.

```json
[
    {
        "id": "row-001",
        "title": "Support Ticket #1042",
        "status": "open",
        "created_at": "2025-03-15T14:30:00Z"
    }
]
```

Response:

| Status code | Description |
| --- | --- |
| `202` | Rows accepted and queued for processing. |
| `400` | Invalid request body. |
| `401` | The access token is missing or expired. Exchange the refresh token for a new access token and retry. |
| `403` | The token does not have the `sources.write.*` permission on this project, or the tenant or project identifier is incorrect. |
| `413` | The request body exceeds the maximum size allowed by the server. Reduce the batch size and resubmit. |
| `500` | An unexpected server error occurred. This endpoint also returns `500` when the source does not exist, so verify the source identifier before you retry with exponential backoff. |

Example response:

```text
{
    "status": "queued"
}
```

## Mark Source as Processed

Sets the `processed` flag of the source to `true`, the same flag that a scheduled data load sets when it finishes. The call does not change the source status or its last run time. You can call it at the end of a sync cycle, including a cycle that pushed no items.

`POST /v1/{tenant}/projects/{project_id}/sources/{source_id}/processed`

This endpoint requires no request body.

Path parameters:

| Parameter | Description |
| --- | --- |
| `tenant` | Your Squirro tenant identifier. |
| `project_id` | The identifier of the project that contains the source. |
| `source_id` | The identifier of the source to mark as processed. |

Response:

| Status code | Description |
| --- | --- |
| `202` | Source marked as processed. |
| `400` | Invalid input. |
| `401` | The access token is missing or expired. Exchange the refresh token for a new access token and retry. |
| `403` | The token does not have the `sources.write.*` permission on this project, or the tenant or project identifier is incorrect. |
| `404` | The specified source was not found. |
| `500` | An unexpected server error occurred. Retry the request with exponential backoff. |

## Error Handling

To avoid overwhelming the ingestion queue, follow the batch size guidance in [Synchronous vs Asynchronous Processing](#synchronous-vs-asynchronous-processing).

The following table summarizes how to handle each response code:

| Status code | Recommended action |
| --- | --- |
| `202` | Items accepted. No action needed. |
| `400` | Inspect the `errors` field in the response body. Correct the affected items and resubmit them. Valid items in the same batch have already been queued and do not need to be resubmitted. |
| `401` | The access token is missing or expired. Exchange the refresh token for a new access token and retry the request. For details, see the [Authentication](authentication.md#api-authentication) page. |
| `403` | The tenant or project identifier is incorrect, or the token does not have the `sources.write.*` permission on the project. An incorrect tenant identifier returns an error message that mentions `tenant mismatch`. Verify the identifiers, then update the grant or use a different token. |
| `404` | The source identifier is incorrect. Verify it and retry. |
| `413` | The request body is too large. Reduce the number of items per batch or the size of individual items, then resubmit. Contact your Squirro administrator to check the configured limit. |
| `500` | An unexpected server error occurred. Retry the request with exponential backoff. If the error persists, visit the [Squirro Support website](https://go.squirro.com/support) and submit a technical support request. |

> **Note**
>
> Retrying a failed request does not create duplicates. An item with an `id` updates the existing item with that `id`. An item without an `id` gets the same generated ID again as long as its content is unchanged. For more information, see [Push Items](#push-items).

## Python Example

### Using the Squirro Python SDK

The Squirro Python SDK provides the `ItemUploader` class, which is the recommended way to push data for most integrations. It handles batching, automatic batch-size calculation, token refresh, and retries. Use it instead of making raw HTTP requests when you need to push large volumes of data reliably.

```python
from squirro_client import ItemUploader

uploader = ItemUploader(
    project_id="<project_id>",
    source_id="<source_id>",
    cluster="https://<cluster>",
    token="<refresh_token>",
)

uploader.upload([
    {
        "id": "doc-001",
        "title": "Annual Review 2024",
        "body": "<p>Summary of the annual review for 2024.</p>",
        "link": "https://intranet.example.com/docs/annual-review-2024",
        "created_at": "2024-12-01T09:00:00Z",
    }
])
```

For information about the `ItemUploader` class and its configuration options, see the [ItemUploader Class](squirro_client/itemuploader.md#itemuploader) page.

### Using the Requests Library

The following examples show how to call each endpoint directly using the Python `requests` library. Use this approach when you cannot use the SDK or need fine-grained control over the request.

#### Pushing items

```python
import os
import requests

# Load credentials from environment variables. Never hard-code tokens in source code.
CLUSTER = os.environ["SQUIRRO_CLUSTER"]        # for example, https://instance.example.com
TENANT = os.environ["SQUIRRO_TENANT"]          # the "tenant" field of the token response
ACCESS_TOKEN = os.environ["SQUIRRO_ACCESS_TOKEN"]
PROJECT_ID = os.environ["SQUIRRO_PROJECT_ID"]
SOURCE_ID = os.environ["SQUIRRO_SOURCE_ID"]

url = (
    f"{CLUSTER}/api/provider/v1/{TENANT}"
    f"/projects/{PROJECT_ID}/sources/{SOURCE_ID}/push_items"
)

items = [
    {
        "id": "doc-001",
        "title": "Annual Review 2024",
        "body": "<p>Summary of the annual review for 2024.</p>",
        "link": "https://intranet.example.com/docs/annual-review-2024",
        "created_at": "2024-12-01T09:00:00Z",
    }
]

# The `verify` parameter defaults to True (TLS certificate validation on).
# Do not set verify=False in production. Use verify="/path/to/ca-bundle.pem"
# if your cluster uses an internal or self-signed certificate.
response = requests.post(
    url,
    json=items,
    headers={
        "Authorization": f"Bearer {ACCESS_TOKEN}",
        "Content-Type": "application/json",
    },
)
response.raise_for_status()

# Avoid logging or printing full response bodies in production.
# They may contain sensitive item content.
result = response.json()
print(result.get("status"))
```

#### Pushing rows

```python
url = (
    f"{CLUSTER}/api/provider/v1/{TENANT}"
    f"/projects/{PROJECT_ID}/sources/{SOURCE_ID}/push_rows"
)

rows = [
    {
        "id": "row-001",
        "title": "Support Ticket #1042",
        "status": "open",
        "created_at": "2025-03-15T14:30:00Z",
    }
]

response = requests.post(
    url,
    json=rows,
    headers={
        "Authorization": f"Bearer {ACCESS_TOKEN}",
        "Content-Type": "application/json",
    },
)
response.raise_for_status()
```

#### Marking a source as processed

```python
url = (
    f"{CLUSTER}/api/provider/v1/{TENANT}"
    f"/projects/{PROJECT_ID}/sources/{SOURCE_ID}/processed"
)

response = requests.post(
    url,
    headers={"Authorization": f"Bearer {ACCESS_TOKEN}"},
)
response.raise_for_status()
```