Squirro Retriever Tool

Squirro Retriever Tool#

Squirro Retriever tool is a powerful paragraph-level retrieval tool that allows AI agents to search and extract relevant information from documents indexed in Squirro projects. That tool combines keyword and semantic search methods to identify relevant paragraphs, groups results into positional clusters, and maintains contextual continuity through context expansion.

The Squirro Retriever tool operates through a multi-stage process:

  1. Intent Detection

    Analyzes user queries to extract key terms, entities, and temporal constraints.

  2. Retrieval

    Performs keyword, semantic, or hybrid search to find relevant paragraphs.

  3. Context Expansion

    Groups retrieved paragraphs into coherent sections and fetches all surrounding paragraphs within each group.

  4. Ranking

    Applies recency boosting and relevance scoring to prioritize results.

Configuration Options#

The Squirro Retriever tool offers comprehensive configuration through parameters that control retrieval behavior, precision, context expansion, and advanced search features.

Note

Each setting in the configuration panel appears as two controls: a dropdown and a field directly below it. The dropdown offers two options. When you select the first option, the field is ignored and the project-level default applies instead. When you select the second option, the value you enter in the field overrides the project-level default for this specific agent.

Filter Query#

The Filter Query setting allows you to restrict retrieval using the Squirro Query syntax. It allows filtering by metadata fields such as data source, document type, date ranges, or custom facets. The filter is applied before any retrieval operations, limiting the search scope to items matching the specified criteria.

Examples:

  • Limit search to specific data sources: datasource_name:reports

  • Filter by date range: $item_created_at>="2024-01-01T00:00:00" AND $item_created_at<"2025-01-01T00:00:00"

  • Combine multiple criteria: datasource_name:contracts AND document_type:legal

For more information about query syntax, see the Query Syntax page.

Enabled Intent Filters#

The Enabled Intent Filters parameter controls which types of intent filters are used to enhance search precision by narrowing subsequent searches to document subsets that match the detected intent. Intent filters help the agent understand and apply user intent from natural language queries.

  • main_entity_phrase

    Identifies and filters by the main entity or subject mentioned in the query. It focuses retrieval on documents specifically about the primary topic of interest.

  • key_term_phrases

    Extracts and applies key terms and phrases from the query to filter search results. It helps focus retrieval on documents mentioning specific concepts or terminology identified in the conversation.

  • date_range

    Detects and applies temporal constraints from the query (for example, “last quarter,” “in 2023,” “recent documents”). It filters results to documents matching the identified time period.

  • facet_filter

    Applies facet-based filtering using the Squirro Query syntax when the agent identifies specific facet values in the query. That allows precise filtering by metadata fields such as data source, author, or document type.

    The prompt used to instruct the LLM on how to construct the filter query is configurable at the project level via the genai.sqgpt.squirro-retriever-field-prompts project setting, using the squirro_query_syntax_filter key. By default, the prompt directs the LLM to apply facet filters only. You can customize the prompt to allow any Squirro Query syntax expression to be passed directly to the retriever.

    For example, a customized prompt can instruct the LLM to filter by paragraph offset range:

    $offset > 10 AND $offset < 100

    That allows the agent to target a specific section of a document based on paragraph position.

Retrieval with Intents#

The Retrieval with Intents setting determines whether inferred intent filters must match search results, controlling the balance between precision and recall.

  • without_intents

    Completely ignores intent filters and performs only a baseline search. That mode prioritizes recall over precision and is useful when broad coverage is more important than targeted results, or when intent detection might incorrectly narrow the search scope.

  • optional

    Performs both a baseline search without intent filters and a filtered search with intent filters, then combines the results. That approach maximizes recall while still benefiting from precision improvements when intent filters match. It provides the most comprehensive results across different query types.

    When combined with hybrid_rrf retrieval mode, that approach triggers four searches internally: keyword, semantic, keyword filtered, and semantic filtered. Results are merged via Reciprocal Rank Fusion. That combination delivers the broadest coverage but increases latency compared to other configurations.

  • only_with_intents

    Applies intent filters strictly, returning only results that match all detected intents. That mode maximizes precision by focusing exclusively on documents matching the interpreted user intent. It works best when queries are specific and intent detection is highly accurate.

Retrieval Mode#

The retrieval mode determines how the tool searches for relevant paragraphs within documents.

  • keyword

    Uses traditional keyword-based search on paragraphs only. That mode excels when searching for specific terms, technical jargon, or exact phrases. It provides fast, deterministic results based on term frequency and position.

  • semantic

    Uses embeddings-based semantic search on paragraphs only. That mode understands the meaning and context of queries, making it effective for conceptual searches where the exact terminology may vary. It can find relevant content even when different words express similar ideas.

  • hybrid_rrf

    Combines keyword and semantic search using the Reciprocal Rank Fusion algorithm. That approach leverages both exact term matching and semantic understanding to provide comprehensive results. It delivers the most balanced performance across different document types and query styles.

Fetch Top N Summarize#

The Fetch Top N Summarize parameter controls how many paragraphs are retrieved when the agent performs broader summarization tasks. That setting fetches more paragraphs than standard search operations to increase the overall context available for comprehensive summaries. That higher threshold ensures adequate coverage when users request document overviews or need information spanning multiple sections.

Enable Context Expansion#

The Enable Context Expansion setting controls whether the tool groups retrieved paragraphs into contiguous sections and returns surrounding context. When turned on, the tool identifies clusters of nearby paragraphs based on their position in the document, then expands left and right from each cluster center to produce coherent sections in reading order. Top-ranked paragraphs are always treated as independent cluster centers, ensuring the most relevant content is preserved throughout that process. When turned off, only the top-ranked individual paragraphs are returned. Turning off that setting improves response speed but may return isolated paragraphs without surrounding context.

Number of Clusters#

The Number of Clusters parameter determines how many positional clusters the tool identifies within a single document during context expansion. Each cluster represents a contiguous section of paragraphs expanded around a group of positionally nearby retrieved paragraphs. That setting only applies when Enable Context Expansion is turned on. Increasing the number produces more, narrower sections. Decreasing it produces fewer, broader ones.

Context Expansion on Top N Items#

The Context Expansion on Top N Items setting controls how many top-level documents receive context expansion analysis. Since the tool operates on paragraphs, retrieved paragraphs are first folded back to their parent top-level documents (for example, a PDF or other source item). Context expansion then runs on the paragraphs belonging to the highest-ranked parent documents only. That setting optimizes performance by limiting clustering operations to the most promising candidates. It only applies when Enable Context Expansion is turned on.

Image Reasoning#

Image Reasoning is a multimodal feature that extracts information from visual content in PDF documents. When turned on, the tool converts selected pages of a PDF to high-resolution images and sends them to a vision-capable large language model, which analyzes tables, charts, diagrams, and embedded images to produce structured answers.

The feature operates in two steps. First, the tool uses a large language model to identify the most relevant pages in each top-ranked document, based on the retrieved text and the user question. Second, only those selected pages are rendered and processed for image-based reasoning, which limits the number of images analyzed and reduces cost and latency.

For each document, image reasoning returns a concise answer, supporting facts extracted from the visual elements, and the page number where the information was found.

Image reasoning uses the large language model configured for the agent, which must support vision input, and requires PDF documents with embedded images, charts, or tables indexed in the Squirro project. That feature applies to PDF documents only, and it requires the data source to be configured to retain the original PDF file as an item attachment during ingestion. Processing images increases response latency and token costs compared to text-only retrieval.

Notes

  • Image reasoning only runs when Enable Context Expansion is turned on. If that setting is turned off, image reasoning is skipped regardless of the Image Reasoning setting.

  • The number of documents processed for image reasoning is controlled by the Context Expansion on Top N Items setting, as image reasoning runs on the same set of top-ranked documents as context expansion.

To turn on Image Reasoning for an agent, open Project Configuration and navigate to the Agents tab. Expand the agent you want to configure, then select Squirro Retriever in the Standard Tools panel to open its settings. In the Image Reasoning group, select the image_reasoning checkbox, then select Save Changes.

Image Reasoning Pages#

The Image Reasoning Pages parameter controls the maximum number of pages per top-ranked document to process for image-based reasoning. Before any images are rendered, the tool uses a large language model to rank all available pages by relevance to the user question, based on the retrieved text context. Only the highest-ranked pages, up to that maximum, are then converted to images and sent to the vision-capable large language model for analysis. That parameter only applies when Image Reasoning is turned on. Higher values provide broader visual coverage but increase processing time and token costs significantly, as each additional page is rendered as a full high-resolution image.

Semantic Profile Config#

The Semantic Profile Config parameter defines advanced configuration for semantic search behavior. That configuration string controls how the semantic search engine processes queries and retrieves results. That advanced setting is typically used for fine-tuning semantic search performance and should only be modified by specialists when specific retrieval behavior adjustments are required. Multiple profile configurations can be specified as a comma-separated list for different search stages. For assistance, contact Squirro Support and submit a technical support request.

Recency Profile Config#

The Recency Profile Config parameter controls how the retrieval system boosts recent documents in search results. That configuration allows time-based relevance adjustments, making newer documents more prominent in results when appropriate. The setting supports multiple profile configurations via a comma-separated list, allowing different recency treatments at various search stages. That advanced parameter is useful for use cases where document freshness significantly impacts relevance and should only be modified by specialists. For assistance, contact Squirro Support and submit a technical support request.

Contents