Provider API

Provider API#

The Provider API is a REST API for pushing data directly into Squirro. It is the push-based counterpart to the native data connectors, which pull data from external sources.

Use the Provider API when:

A native Squirro connector does not exist for your data source.
A third-party middleware or vendor collects data on your behalf and needs to forward it to Squirro.
Your integration architecture requires the source system to initiate data transfers rather than relying on scheduled Squirro jobs.

When items are submitted through the Provider API, they go through the same ingestion pipeline as items from native connectors. That pipeline applies the same data processing steps, including transformation, enrichment, and indexing.

This page covers the three endpoints that a push-based integration uses: Push Items, Push Rows, and Mark Source as Processed.

Before You Start#

To make a successful call to the Provider API, you need:

A cluster URL.
A tenant identifier.
A project identifier.
A source identifier.
An authentication token.

Cluster URL#

Your cluster URL is the address of your Squirro instance, for example https://instance.example.com. For step-by-step instructions, see the Connecting to Squirro page.

Tenant Identifier#

Every endpoint path contains the tenant identifier. It does not necessarily match the subdomain of your cluster URL, so do not derive it from the URL. When you exchange a refresh token for an access token, the response returns the tenant identifier in the tenant field. For the request and response format, see the Authentication page.

The Squirro Python SDK reads the tenant identifier from that response and exposes it as client.tenant after authentication. The ItemUploader class uses it automatically.

Project Identifier#

Navigate to Setup → Settings → Properties in the Squirro UI to find your project identifier. For step-by-step instructions, see the Locate Your Project Identifier page.

To retrieve it programmatically, use the get_projects() method from the Squirro Python SDK:

from squirro_client import SquirroClient

client = SquirroClient(None, None, cluster="https://<cluster>")
client.authenticate(refresh_token="<your_refresh_token>")

for project in client.get_projects():
    print(project["id"], project["title"])

Source Identifier#

Open the Spaces Menu in the top-right corner of the Squirro UI (for more information, see the Squirro Spaces page), navigate to Server → Data → Data Sources, open the three-dot menu next to the source, and select Copy Data Source ID.

To retrieve it programmatically instead, use the get_sources() method from the Squirro Python SDK, with the client from the previous example:

sources = client.get_sources("<project_id>")
for source in sources:
    print(source["id"], source["name"])

For the full reference, see the APIs by Topic page.

Create a Source for Push Ingestion#

The Provider API requires a source configured for push-based ingestion. To create one, navigate to Setup → Data in the Squirro UI, and select the Data Import category. Choose either JSON or CSV depending on the format of data your integration sends.

The source does not need to have a schedule configured. It only needs to exist so that the Provider API has a valid destination to push data to.

Authentication#

All Provider API endpoints require a valid Bearer access token in the Authorization header. The Content-Type header is required for all POST requests.

Authorization: Bearer <access_token>
Content-Type: application/json

Always transmit tokens over HTTPS. Never log or print the Authorization header value.

For information about generating and using access tokens, see the Authentication page.

For automated integrations, use a service account rather than a personal user token. A service account provides a long-lived refresh token that does not depend on a specific user. For instructions on creating a service account and generating a token, see the Create a Service Account page.

By default, the access token derived from the refresh token expires after 10 minutes. The Squirro Python SDK handles this automatically. If you call the Provider API directly without the SDK, exchange the refresh token for a new access token by sending a POST request to /api/user/oauth2/token with grant_type=refresh_token. Repeat this exchange before each session or whenever a 401 response is returned. For the request and response format, see the Authentication page.

The token must have the sources.write.* permission on the target project. To limit the impact of a compromised token, restrict the service account to only the specific project it needs to write to, rather than granting tenant-wide access. When creating a service account grant, include this permission explicitly:

grant = client.new_grant(
    user_id=user_id,
    type="user",
    project_permissions=["sources.write.*"],
)

For instructions on scoping a service account to specific projects and rotating a compromised token, see the Create a Service Account page.

Network-Level Access Control#

Bearer token authentication is the primary access control for the Provider API. For deployments that require an additional network-level layer, such as restricting inbound calls to known source IP ranges or enforcing firewall or VPN policies, contact your Squirro administrator to discuss the options available for your deployment.

The Provider API is served over HTTPS. Clients pushing data to the cluster must allow outbound HTTPS traffic on port 443 to the cluster URL.

Base URL#

All Provider API endpoints are served under the following base path:

https://<cluster>/api/provider

For example, the full URL for the Push Items endpoint on a cluster at https://instance.example.com is:

https://instance.example.com/api/provider/v1/{tenant}/projects/{project_id}/sources/{source_id}/push_items

Integration Workflow#

A typical push-based integration repeats the same cycle on every run:

Authenticate. Exchange the service account refresh token for an access token. The Squirro Python SDK does this automatically. For direct HTTP calls, see the Authentication page.
Push data. Send items to the Push Items endpoint, or rows to the Push Rows endpoint. Split large datasets into batches and submit each batch separately. Because items are deduplicated by id, re-pushing an item with the same id updates it rather than creating a duplicate, which makes a failed batch safe to resend.
Mark the source as processed. Optionally, call the Mark Source as Processed endpoint at the end of the run to set the processed flag of the source.

In asynchronous mode (the default), a 202 response confirms that items were accepted into the ingestion queue, not that they were indexed. To confirm indexing and surface any pipeline errors, check the source status under Server → Data → Data Sources in the Squirro UI, or retrieve it programmatically with the get_sources() method.

Synchronous vs Asynchronous Processing#

By default, the Provider API queues items asynchronously (priority=0). The API returns 202 as soon as the items are accepted, and processing happens in the background. That is the recommended mode for most integrations.

Setting priority=1 switches to synchronous processing. In that mode, the batch is processed before the response returns. That does not mean the items already appear in search results when the response arrives. Use synchronous mode only when your integration depends on the batch being processed before its next step.

Notes

Synchronous mode has a maximum batch size enforced by the server. If your batch exceeds this limit, the API returns a 400 error. Contact your Squirro administrator to check or adjust this limit.
For large-volume ingestion, asynchronous mode is more reliable. Split large datasets into batches of fewer than 100 items and submit each batch separately.

Push Items#

Submits a batch of Squirro items to an existing source. The items are validated, preprocessed, and placed in the ingestion queue.

POST /v1/{tenant}/projects/{project_id}/sources/{source_id}/push_items

Path parameters:

Parameter	Description
`tenant`	Your Squirro tenant identifier.
`project_id`	The identifier of the project that contains the source.
`source_id`	The identifier of the source to push data to.

Query parameters:

Parameter	Description
`priority`	Processing priority. Set to `1` for synchronous (priority) processing. Defaults to `0` (asynchronous). See Synchronous vs Asynchronous Processing for details.
`pipeline_workflow_id`	Optional identifier of a specific pipeline workflow to use when processing the items. This parameter applies only in synchronous mode (`priority=1`). In asynchronous mode, the default workflow for the source is always used, and any value set here is ignored. For more information about pipeline workflows, see the ML Enrichments for Pipeline Workflows page.

Request body:

A JSON array of items. Each item must include at minimum one of title, body, or files. Use the body field for inline content and the files field to attach documents, either inline as Base64 or by URL. Squirro recommends also providing id, link, and created_at (in ISO 8601 format, for example 2025-04-01T08:00:00Z) to ensure correct deduplication and display. For a complete description of all available fields, see the Item Format page.

[
    {
        "id": "unique-item-id-1",
        "title": "Quarterly Report Q1 2025",
        "body": "<p>Revenue increased by 12% in Q1 2025.</p>",
        "link": "https://intranet.example.com/reports/q1-2025",
        "created_at": "2025-04-01T08:00:00Z",
        "keywords": {
            "department": ["Finance"],
            "document_type": ["Report"]
        }
    }
]

Notes

If you do not provide an id, one is generated from the file names of the item, from its link and title, or from its body when those are missing. Changing those fields later creates a new item instead of updating the existing one. Squirro recommends providing a stable id from your source system.
If you submit an item with the same id as an existing item in the source, the existing item is updated.
The Provider API creates and updates items only. It cannot delete items. To remove items that no longer exist in your source system, use the delete_item() method of the Squirro Python SDK. For the full reference, see the APIs by Topic page.
The processing order of items is not guaranteed, whether within a single batch or across separate requests. If two items share the same id, the last one processed determines the stored content.
The body field accepts HTML. Plain text values are also accepted.
A keyword with a reserved name, such as source, is dropped from the item without an error. The item and the rest of the batch are still indexed. For the list of reserved names, see the Labels page.
Item content is indexed in plaintext and is visible in search results to any user with access to the project. Do not push credentials, secrets, or unredacted personal data unless your project access controls and data retention policies explicitly permit it.

Response:

Status code	Description
`202`	Items accepted and queued for processing.
`400`	One or more items failed validation. Items that pass validation are still queued. The response body contains an `errors` list with details about each rejected item.
`401`	The access token is missing or expired. Exchange the refresh token for a new access token and retry.
`403`	The token does not have the `sources.write.*` permission on this project, or the tenant or project identifier is incorrect.
`404`	The specified source was not found.
`413`	The request body exceeds the maximum size allowed by the server. Reduce the batch size and resubmit.

Example response (success):

{
    "status": "queued"
}

Example response (validation error):

{
    "status": "error",
    "errors": [
        "invalid item 0: one of `title`, `body`, or `files` is mandatory"
    ]
}

Note

A 400 response with an errors list does not mean the entire batch was rejected. Valid items in the same batch are still queued.

Push Rows#

Submits raw tabular data (rows) to an existing source. Unlike the Push Items endpoint, rows are not validated or preprocessed before queuing. Transformation and field mapping happen later in the pipeline.

Use this endpoint when your data is in a flat, tabular format. If your data is already structured as Squirro items, use Push Items instead.

The field mappings of the source turn rows into items, so the source must have them. You set them when you create the source in the UI, on the Map to Item Fields and Map to Labels steps. For more information, see the How to Load Data Using the UI page. Without mappings, the endpoint still returns 202 and the batch is processed, but no items are created.

POST /v1/{tenant}/projects/{project_id}/sources/{source_id}/push_rows

Path parameters:

Parameter	Description
`tenant`	Your Squirro tenant identifier.
`project_id`	The identifier of the project that contains the source.
`source_id`	The identifier of the source to push data to.

Query parameters:

Parameter	Description
`priority`	Processing priority. Set to `1` for synchronous (priority) processing. Defaults to `0` (asynchronous). See Synchronous vs Asynchronous Processing for details.
`pipeline_workflow_id`	Optional identifier of a specific pipeline workflow to use when processing the rows. This parameter applies only in synchronous mode (`priority=1`). In asynchronous mode, the default workflow for the source is always used, and any value set here is ignored. For more information about pipeline workflows, see the ML Enrichments for Pipeline Workflows page.
`base64_encoded_columns`	Comma-separated list of column names whose values are Base64-encoded. The pipeline decodes them before processing.

Request body:

A JSON array of row objects. The field names in each row must match the field mappings of the source. There are no required fields at the API level.

[
    {
        "id": "row-001",
        "title": "Support Ticket #1042",
        "status": "open",
        "created_at": "2025-03-15T14:30:00Z"
    }
]

Response:

Status code	Description
`202`	Rows accepted and queued for processing.
`400`	Invalid request body.
`401`	The access token is missing or expired. Exchange the refresh token for a new access token and retry.
`403`	The token does not have the `sources.write.*` permission on this project, or the tenant or project identifier is incorrect.
`413`	The request body exceeds the maximum size allowed by the server. Reduce the batch size and resubmit.
`500`	An unexpected server error occurred. This endpoint also returns `500` when the source does not exist, so verify the source identifier before you retry with exponential backoff.

Example response:

{
    "status": "queued"
}

Mark Source as Processed#

Sets the processed flag of the source to true, the same flag that a scheduled data load sets when it finishes. The call does not change the source status or its last run time. You can call it at the end of a sync cycle, including a cycle that pushed no items.

POST /v1/{tenant}/projects/{project_id}/sources/{source_id}/processed

This endpoint requires no request body.

Path parameters:

Parameter	Description
`tenant`	Your Squirro tenant identifier.
`project_id`	The identifier of the project that contains the source.
`source_id`	The identifier of the source to mark as processed.

Response:

Status code	Description
`202`	Source marked as processed.
`400`	Invalid input.
`401`	The access token is missing or expired. Exchange the refresh token for a new access token and retry.
`403`	The token does not have the `sources.write.*` permission on this project, or the tenant or project identifier is incorrect.
`404`	The specified source was not found.
`500`	An unexpected server error occurred. Retry the request with exponential backoff.

Error Handling#

To avoid overwhelming the ingestion queue, follow the batch size guidance in Synchronous vs Asynchronous Processing.

The following table summarizes how to handle each response code:

Status code	Recommended action
`202`	Items accepted. No action needed.
`400`	Inspect the `errors` field in the response body. Correct the affected items and resubmit them. Valid items in the same batch have already been queued and do not need to be resubmitted.
`401`	The access token is missing or expired. Exchange the refresh token for a new access token and retry the request. For details, see the Authentication page.
`403`	The tenant or project identifier is incorrect, or the token does not have the `sources.write.*` permission on the project. An incorrect tenant identifier returns an error message that mentions `tenant mismatch`. Verify the identifiers, then update the grant or use a different token.
`404`	The source identifier is incorrect. Verify it and retry.
`413`	The request body is too large. Reduce the number of items per batch or the size of individual items, then resubmit. Contact your Squirro administrator to check the configured limit.
`500`	An unexpected server error occurred. Retry the request with exponential backoff. If the error persists, visit the Squirro Support website and submit a technical support request.

Note

Retrying a failed request does not create duplicates. An item with an id updates the existing item with that id. An item without an id gets the same generated ID again as long as its content is unchanged. For more information, see Push Items.

Python Example#

Using the Squirro Python SDK#

The Squirro Python SDK provides the ItemUploader class, which is the recommended way to push data for most integrations. It handles batching, automatic batch-size calculation, token refresh, and retries. Use it instead of making raw HTTP requests when you need to push large volumes of data reliably.

from squirro_client import ItemUploader

uploader = ItemUploader(
    project_id="<project_id>",
    source_id="<source_id>",
    cluster="https://<cluster>",
    token="<refresh_token>",
)

uploader.upload([
    {
        "id": "doc-001",
        "title": "Annual Review 2024",
        "body": "<p>Summary of the annual review for 2024.</p>",
        "link": "https://intranet.example.com/docs/annual-review-2024",
        "created_at": "2024-12-01T09:00:00Z",
    }
])

For information about the ItemUploader class and its configuration options, see the ItemUploader Class page.

Using the Requests Library#

The following examples show how to call each endpoint directly using the Python requests library. Use this approach when you cannot use the SDK or need fine-grained control over the request.

Pushing items#

import os
import requests

# Load credentials from environment variables. Never hard-code tokens in source code.
CLUSTER = os.environ["SQUIRRO_CLUSTER"]        # for example, https://instance.example.com
TENANT = os.environ["SQUIRRO_TENANT"]          # the "tenant" field of the token response
ACCESS_TOKEN = os.environ["SQUIRRO_ACCESS_TOKEN"]
PROJECT_ID = os.environ["SQUIRRO_PROJECT_ID"]
SOURCE_ID = os.environ["SQUIRRO_SOURCE_ID"]

url = (
    f"{CLUSTER}/api/provider/v1/{TENANT}"
    f"/projects/{PROJECT_ID}/sources/{SOURCE_ID}/push_items"
)

items = [
    {
        "id": "doc-001",
        "title": "Annual Review 2024",
        "body": "<p>Summary of the annual review for 2024.</p>",
        "link": "https://intranet.example.com/docs/annual-review-2024",
        "created_at": "2024-12-01T09:00:00Z",
    }
]

# The `verify` parameter defaults to True (TLS certificate validation on).
# Do not set verify=False in production. Use verify="/path/to/ca-bundle.pem"
# if your cluster uses an internal or self-signed certificate.
response = requests.post(
    url,
    json=items,
    headers={
        "Authorization": f"Bearer {ACCESS_TOKEN}",
        "Content-Type": "application/json",
    },
)
response.raise_for_status()

# Avoid logging or printing full response bodies in production.
# They may contain sensitive item content.
result = response.json()
print(result.get("status"))

Pushing rows#

url = (
    f"{CLUSTER}/api/provider/v1/{TENANT}"
    f"/projects/{PROJECT_ID}/sources/{SOURCE_ID}/push_rows"
)

rows = [
    {
        "id": "row-001",
        "title": "Support Ticket #1042",
        "status": "open",
        "created_at": "2025-03-15T14:30:00Z",
    }
]

response = requests.post(
    url,
    json=rows,
    headers={
        "Authorization": f"Bearer {ACCESS_TOKEN}",
        "Content-Type": "application/json",
    },
)
response.raise_for_status()

Marking a source as processed#

url = (
    f"{CLUSTER}/api/provider/v1/{TENANT}"
    f"/projects/{PROJECT_ID}/sources/{SOURCE_ID}/processed"
)

response = requests.post(
    url,
    headers={"Authorization": f"Bearer {ACCESS_TOKEN}"},
)
response.raise_for_status()

Provider API

Contents

Provider API#

Before You Start#

Cluster URL#

Tenant Identifier#

Project Identifier#

Source Identifier#

Create a Source for Push Ingestion#

Authentication#

Network-Level Access Control#

Base URL#

Integration Workflow#

Synchronous vs Asynchronous Processing#

Push Items#

Push Rows#

Mark Source as Processed#

Error Handling#

Python Example#

Using the Squirro Python SDK#

Using the Requests Library#

Pushing items#

Pushing rows#

Marking a source as processed#