How to Connect Squirro Chat to a Third-Party LLM

Profile: System Administrator

This page describes how system or project administrators, or anyone with access to the Server or Setup spaces of a Squirro Chat project, can connect their Squirro Chat application to a third-party LLM. Squirro Chat requires a configured LLM to generate responses. Without one, the application does not function.

Connecting Squirro Chat to a third-party LLM involves two steps: obtaining the required information from your provider, then configuring your Squirro instance to use it.

Obtain Third-Party LLM Information

To set up Squirro Chat access to a specific third-party LLM, you need to set up the LLM externally and record certain information for use in the Squirro Chat configuration. Squirro Chat uses that information to authenticate your connection.

The information you need depends on the type of LLM you are connecting to. The table below outlines the required information for each type of supported LLM.

Model Type	Required Information
OpenAI	OpenAI API key
Azure	Deployment Name Base URL for API Azure API key API version (optional)
OpenAI-API compatible	API key (if required by the provider) Model name Base URL of the provider

Note

How you access this information depends on your specific third-party LLM provider.

The OpenAI-API compatible category covers any LLM that exposes an OpenAI-compatible endpoint, including open-source models hosted on third-party platforms and models running on your own infrastructure.

Configure Your Squirro Server or Project Settings

You have two options in terms of configuring usage of the third-party LLM:

• Configure your Squirro instance at the server level, meaning all existing and future Squirro Chat projects will use the third-party LLM you specify.

• Configure your Squirro Chat application at the project level, meaning only that particular project you are working within will use the third-party LLM you specify.

When both server-level and project-level configurations are present, the project-level configuration takes precedence for that specific project.

Note

The server-level and project-level configurations use different key names: genai.sqgpt.default-settings at the server level and genai.sqgpt.settings at the project level. That difference is intentional. The two keys serve different scopes and are not interchangeable.

Tip

Typically, Squirro recommends configuring your Squirro instance at the server level.

Configuring Your Squirro Instance at the Server Level

To configure your Squirro instance at the server level, follow the steps below:

1. Log in to your Squirro instance.

2. Click on Server in the spaces navigation bar.

3. Click on Configuration.

4. Search for the configuration named genai.sqgpt.default-settings.

5. Click on the edit icon to modify the configuration.

6. Adjust the default values following the LLM Configuration Values section.

You can also edit the configuration as a JSON dictionary by clicking the toggle:

Configuring Your Squirro Chat Application at the Project Level

To configure your Squirro Chat application at the project level, follow the steps below:

1. Log in to your Squirro instance.

2. Click on Setup in the spaces navigation bar.

3. Click on Settings.

4. Click on Project Configuration.

5. Search for the configuration named genai.sqgpt.settings.

6. Click on the edit icon to modify the configuration.

7. Adjust the default values following the LLM Configuration Values section.

Tip

To configure a certain setting to fall back to the server setting, select Default to Server Setting (or assign null to the setting in the JSON editor):

LLM Configuration Values

The following configuration values apply to both the server-level and project-level configuration steps above. Paste appropriate values for your third-party LLM into the configuration.

Configuration for OpenAI Models

{
    "llm": {
        "kind": "openai",                          # Indicates OpenAI model (required)
        "openai_api_key": "<your OpenAI API key>", # Your OpenAI API key (required)
        "model": "gpt-5.2"                         # Model name (required)
    }
}

Configuration for Azure-hosted OpenAI Models

{
    "llm": {
        "kind": "azure",                           # Indicates Azure-hosted model (required)
        "deployment_name": "<your deployment>",    # Your deployment name (required)
        "openai_api_base": "<your base URL>",      # Base URL for the API (required)
        "openai_api_key": "<your Azure API key>",  # Your Azure API key (required)
        "openai_api_version": "2023-05-15",        # API version (optional, default is "2023-05-15")
        "use_responses_api": false                 # Use the Responses API (optional, default is false)
    }
}

The default API version may not support reasoning_effort. If you set reasoning_effort, see its entry in Advanced configuration parameters.

Some Azure reasoning deployments, for example gpt-6.1-sol, accept function tools only through the Azure Responses API. On such a deployment, Squirro Chat requests fail with an error that function tools with reasoning_effort are not supported, or with an Unsupported value error for reasoning_effort. Set use_responses_api to true to send chat answers, summaries, starter questions, conversation titles, and source match detection in query processing through the Responses API instead of Chat Completions. The API version set with openai_api_version must support the Responses API. A configured reasoning_effort is then sent as the reasoning effort of the Responses API, and temperature, top_p, and seed are not sent.

On Azure deployments of gpt-5.4 and later models, Chat Completions rejects an explicit reasoning_effort of "low", "medium", or "high" together with function tools. Set use_responses_api to true. Some of these models, such as the gpt-5.6 family, also accept reasoning_effort "none" with function tools on Chat Completions.

For more information on connecting to a Microsoft Azure LLM, see the How to Connect Squirro Chat to a Microsoft Azure LLM page.

Configuration for OpenAI-API Compatible Models

Important

The LLM you connect must support tool calling. Squirro agents rely on tool calling to function. Models that do not support it cannot be used with Squirro Chat. For more information, see the Large Language Model Support page.

OpenAI-API compatible models are LLMs that expose an HTTP API following the same request and response format as the OpenAI API. That compatibility makes it possible to connect Squirro Chat to a wide range of models beyond standard OpenAI offerings, including open-source models hosted on third-party cloud platforms.

For organizations that already run LLMs on their own infrastructure, Squirro Chat can connect directly to that infrastructure using the same configuration, provided the deployment exposes an OpenAI-compatible endpoint. For compatibility requirements, see the Large Language Model Support page.

{
    "llm": {
        "kind": "openai",                          # Same value as for OpenAI. The base_url field determines the provider.
        "openai_api_key": "<your API key>",        # Your API key (required, or any string if no auth needed)
        "model": "<your model>",                   # Model name (required)
        "base_url": "<base URL of provider>"       # Base URL of the model provider (required)
    }
}

Advanced configuration parameters

All LLM configurations accept the following optional parameters in addition to the required fields shown above:

• temperature: Sampling temperature (float, default 0.0). Higher values produce more varied responses. On Azure, it is not sent when use_responses_api is true.

• top_p: Nucleus sampling threshold (float, default 1.0). On Azure, it is not sent when use_responses_api is true.

• seed: Optional integer seed for more reproducible outputs. Support varies by model. On Azure, it is not sent when use_responses_api is true.

• reasoning_effort: Controls how much reasoning the model applies to chat answers, summaries, and starter questions. Other requests do not use it. Accepted values are "none", "minimal", "low", "medium", and "high". A configured value is sent as is to every Azure deployment, whatever its model, and to OpenAI GPT-5 and o-series models. Other models and providers ignore it. On Azure, the reasoning effort needs an API version that supports it. Set that version with openai_api_version. Not every model accepts every value, for example "none" or "minimal", and a model may reject requests with a value it does not support. At the server level, leaving it unset sends no reasoning effort, so the model uses its own default. At the project level, leaving it unset uses the server value, unless the project LLM configuration sets every required field, in which case no reasoning effort is sent.

• request_timeout_s: HTTP timeout in seconds, as an object with connect and read keys (for example, {"connect": 5, "read": 120}).

• context_length: Override the model context window size (integer). Set to -1 to auto-detect.

• max_retries: Number of retry attempts on API failure (integer, default 3).

Verify Your Configuration

After saving the configuration, open a Squirro Chat project and send a test message. If the LLM responds correctly, the configuration is working.

If you see an error, check the following:

• The API key is correct and has not expired.

• The base URL is reachable from the Squirro server (not just from your local machine).

• The model name matches exactly what the provider expects.

• The LLM supports tool calling. For more information, see the Large Language Model Support page.

If the issue persists, visit the Squirro Support website and submit a technical support request.

Timeout Configuration for Reasoning Models

Some LLMs require significantly more time to generate responses than the default timeout allows. Common scenarios include reasoning models operating at high reasoning levels (for example, OpenAI GPT-5) and large models running on constrained hardware. When the timeout is exceeded, Squirro Chat returns an error before the LLM finishes generating.

To accommodate longer response times, increase the frontend.sqgpt.default-timeout project configuration. That setting controls the inactivity timeout for Squirro Chat functionality, including Batch Q&A, the Chat widget, Chat with Document, and Summarization. The value is in milliseconds, and the default is 60000 (one minute). Proxies and load balancers in front of your Squirro instance apply their own connection timeouts, which depend on the deployment, so a very high value does not guarantee that a request stays open.

1. Navigate to Setup → Settings → Project Configuration.

2. Search for frontend.sqgpt.default-timeout.

3. Edit the configuration and increase the value (in milliseconds).