Observability and Telemetry

Observability and Telemetry#

Telemetry is the operational data that a software product emits about its own behavior while it runs, such as how requests move through the system, how long operations take, and where errors occur. Collecting telemetry lets you understand the health and performance of the product, troubleshoot problems, and see how the product is used.

Squirro uses OpenTelemetry to produce telemetry about how the platform behaves at runtime. That data answers operational questions such as how long a request took, which service or query caused a slowdown, and where an error originated.

No telemetry data leaves the Squirro instance unless a Squirro administrator specifically configures an external destination.

This page explains how OpenTelemetry is integrated into Squirro, what data it produces, and how to turn it on. To send that data to a third-party observability platform such as Grafana or Langfuse, see the Connecting to an Observability Platform page. Those platforms are third-party software, not part of the Squirro platform, and Squirro does not provide or support them.

What OpenTelemetry Provides#

OpenTelemetry is an open-source observability framework. It defines a vendor-neutral way to generate telemetry data and ship it to any compatible backend.

OpenTelemetry produces three kinds of signals:

Traces

The end-to-end journey of a request as it moves through Squirro services. A trace is made up of spans, where each span represents one unit of work, such as an HTTP call, a database query, or a search operation.
Logs

Standard log messages, correlated with the trace they belong to so that you can move from a slow trace directly to the related log lines.
Metrics

Numeric measurements over time, such as counters and durations. The GenAI service defines metrics for language model usage and agent execution.

Squirro exports traces and logs over the OpenTelemetry Protocol (OTLP). Metric instruments exist inside the GenAI service for internal use, but metrics are not exported over OTLP. Plan dashboards and alerts around traces and logs. Language model token usage is recorded as attributes on the GenAI service spans, so it remains available to your platform through traces.

The GenAI service provides the AI features for both Squirro and Project Neo. Internally, these run on two agent runtimes. The V0 runtime powers the AI features in Squirro, and the newer V1 runtime powers Project Neo. Telemetry instrumentation, including sanitization, applies to the whole GenAI service and covers both runtimes. For observability details specific to Project Neo, see the Observability page.

By default, OpenTelemetry is turned off. No telemetry is produced or exported until an administrator turns it on, as described in the Turning on Telemetry section below.

What Squirro Captures#

Squirro instruments its services automatically, so that the most common operations are captured without any extra setup. When telemetry is turned on, Squirro records the following activities:

Activity	What is captured
Incoming requests	HTTP requests and responses handled by Squirro services.
Service-to-service calls	Calls between Squirro services and to external systems.
Database queries	Queries run against the relational database.
Search queries	Queries run against Elasticsearch, captured by the built-in OpenTelemetry instrumentation of the Elasticsearch client, when it is turned on.
Logs	Log messages, stamped with the trace and span identifiers.
AI agent execution	Agent runs, prompts, and language model usage in the GenAI service.

This coverage is built in, so you can connect Squirro to your observability platform and see meaningful data right away.

Correlation IDs#

A single user action can touch several services. To follow that action from end to end, Squirro supports correlation identifiers on requests.

A client can send the X-Squirro-Correlation-ID header with a request.
Each response returns the X-Trace-ID header, which holds the OpenTelemetry trace identifier.
The correlation identifier is attached to the trace as the squirro.correlation_id attribute, so that you can search for it in your observability platform.

Trace context is propagated across synchronous calls between services, so a single trace can span several services that handle the same request. Asynchronous work that passes through a queue, such as data ingestion, starts a separate trace, so traces do not span those boundaries.

Where Telemetry Data Is Sent#

Squirro does not send telemetry to Squirro or to any third party by default. The data is sent only to the collector endpoint that you configure, using the OpenTelemetry Protocol (OTLP). The default endpoint is on the local host, so nothing leaves the instance unless an administrator configures a different destination.

OTLP is supported over two transports:

gRPC, on port 4317 by default. This transport is preferred for efficiency.
HTTP, on port 4318 by default.

Squirro does not run an OpenTelemetry Collector by default, and the default endpoint expects a collector on the local host. Before any data flows, deploy a collector or point Squirro at an existing one. For more information, see the Connecting to an Observability Platform page.

Spans are batched before they are sent, to reduce overhead on the running services.

Security Considerations#

Telemetry can contain sensitive operational data, so the export path and the receiving platform are part of your security boundary. The following practices help you export telemetry safely.

Sensitive Data#

Telemetry can include content such as HTTP request bodies, database queries, indexed document content and search results, API keys and tokens passed in requests, and, in the GenAI service, prompt and response data. As a safeguard, the GenAI service scrubs known sensitive values, such as credentials and authorization headers, from its spans before export.

That safeguard is best-effort and applies only to the GenAI service, so use the OpenTelemetry Collector as your main control. Review what your instrumentation captures, and filter sensitive values at the collector, before you export telemetry outside your own network.

Encrypt Telemetry in Transit#

The gRPC exporter uses an unencrypted connection by default, set by SQ_OPENTELEMETRY_EXPORTER_INSECURE. An unencrypted connection is suitable for a collector on the local host. For any destination reached over a network, set SQ_OPENTELEMETRY_EXPORTER_INSECURE to false and use a TLS-protected endpoint, so that spans, logs, and any captured content stay encrypted in transit.

Protect Exporter Credentials#

A hosted platform authenticates with a token or key that you place in the collector configuration, the service configuration, or an environment variable. Treat those values as secrets. Restrict the file permissions on any configuration that contains them, keep them out of version control, and confirm that they are not written to logs. Where your deployment supports secret redaction, mark these values for redaction.

Restrict Access to the Observability Platform#

The receiving platform holds sensitive operational data, so protect it as you would the source. Require authentication and authorization on the platform, limit access to the people who need it, and prefer a platform inside the same trust boundary as Squirro.

Personal and Regulated Data#

Search queries and GenAI prompts and responses can contain personal or regulated data. Exporting that data to an external platform, including a hosted service, may carry data-processing and data-residency obligations. Involve your compliance team when you plan to export telemetry to an external service.

Validate the Correlation ID#

The X-Squirro-Correlation-ID request header is supplied by the client. Squirro records it on the trace and returns it unchanged in the response, without validating it. To keep unexpected or excessive values out of your observability platform, validate or bound the header at your edge proxy, or drop and replace it at the OpenTelemetry Collector.

Turning on Telemetry#

Telemetry is controlled through the [opentelemetry] section of the service configuration, or through the matching environment variables. The environment variables are the most common way to control telemetry in a containerized deployment.

On a Squirro server, the GenAI service runs as a Docker container managed by the sqgenaid service. Provide the environment variables through the docker-override.env file in the /etc/squirro/genai.d/ directory. For the step-by-step procedure, see the genai.ini page.

Across all Squirro services, only SQ_OPENTELEMETRY_TRACING is wired through the shared configuration. It is the master switch that instruments a service. The GenAI service exposes the full set of settings below, including the exporter endpoint, and is the primary source of language model and agent telemetry. To export telemetry from other services, add the exporter configuration to each of those services.

The GenAI service supports the following environment variables:

Environment variable	Default	Description
`SQ_OPENTELEMETRY_TRACING`	`false`	Master switch for OpenTelemetry instrumentation.
`SQ_OPENTELEMETRY_INSTRUMENT_LOGGING`	`false`	Correlate logs with traces and export logs over OTLP.
`SQ_OPENTELEMETRY_INSTRUMENT_REQUESTS`	`true`	Instrument the `requests` library for outbound calls.
`SQ_OPENTELEMETRY_EXPORTER_ENDPOINT`	`localhost:4317`	OTLP gRPC collector endpoint that receives the data.
`SQ_OPENTELEMETRY_EXPORTER_INSECURE`	`true`	Allow an insecure gRPC connection to the collector.

The equivalent configuration section looks as follows:

[opentelemetry]
tracing = true
instrument_logging = true
instrument_requests = true

[opentelemetry_span_exporter_grpc]
endpoint = localhost:4317
insecure = true

To export over HTTP instead of gRPC, configure the HTTP exporter section:

[opentelemetry_span_exporter_http]
endpoint = http://localhost:4318

A service reads these settings only at startup, so restart any service whose configuration you changed for the new settings to take effect. For example, restart the GenAI service through its sqgenaid service. For how to restart Squirro services, see the Services page.

Exporting telemetry from every Squirro service, rather than from the GenAI service alone, is not a self-service setup. For help with a platform-wide rollout, visit the Squirro Support website and submit a technical support request.

To trace Elasticsearch search queries, set the OTEL_PYTHON_INSTRUMENTATION_ELASTICSEARCH_ENABLED environment variable to true.

The standard environment variables of the OpenTelemetry instrumentation libraries are also honored for fine-grained control. To tell several Squirro deployments apart in one observability platform, add an identifying attribute, such as an environment name, at the OpenTelemetry Collector of each deployment.

Telemetry Volume#

Squirro exports every span when tracing is turned on. It does not sample traces. Turning on instrument_logging also exports every log message over OTLP. On a busy production deployment, size the collector accordingly, and reduce volume with sampling or filtering at the OpenTelemetry Collector rather than at the services.

For step-by-step instructions on connecting Squirro to a specific observability platform, see the Connecting to an Observability Platform page.