Squirro Retriever Tool

Squirro Retriever Tool#

Squirro Retriever tool is a powerful paragraph-level retrieval tool that allows AI agents to search and extract relevant information from documents indexed in Squirro projects. That tool combines keyword and semantic search methods to identify relevant paragraphs, groups results into positional clusters, and maintains contextual continuity through context expansion.

The Squirro Retriever tool operates through a multi-stage process:

  1. Intent Detection

    Analyzes user queries to extract key terms, entities, and temporal constraints.

  2. Retrieval

    Performs keyword, semantic, or hybrid search to find relevant paragraphs.

  3. Context Expansion

    Groups retrieved paragraphs into coherent sections and fetches all surrounding paragraphs within each group.

  4. Ranking

    Applies recency boosting and relevance scoring to prioritize results.

Configuration Options#

The Squirro Retriever tool offers comprehensive configuration through parameters that control retrieval behavior, precision, context expansion, and advanced search features.

Filter Query#

The Filter Query setting allows you to restrict retrieval using the Squirro Query syntax. It allows filtering by metadata fields such as data source, document type, date ranges, or custom facets. The filter is applied before any retrieval operations, limiting the search scope to items matching the specified criteria.

Examples:

  • Limit search to specific data sources: datasource_name:reports

  • Filter by date range: $item_created_at>="2024-01-01T00:00:00" AND $item_created_at<"2025-01-01T00:00:00"

  • Combine multiple criteria: datasource_name:contracts AND document_type:legal

For more information about query syntax, see the Query Syntax page.

Enabled Intent Filters#

The Enabled Intent Filters parameter controls which types of intent filters are used to enhance search precision by narrowing subsequent searches to document subsets that match the detected intent. Intent filters help the agent understand and apply user intent from natural language queries.

  • main_entity_phrase

    Identifies and filters by the main entity or subject mentioned in the query. It focuses retrieval on documents specifically about the primary topic of interest.

  • key_term_phrases

    Extracts and applies key terms and phrases from the query to filter search results. It helps focus retrieval on documents mentioning specific concepts or terminology identified in the conversation.

  • date_range

    Detects and applies temporal constraints from the query (for example, “last quarter,” “in 2023,” “recent documents”). It filters results to documents matching the identified time period.

  • facet_filter

    Applies facet-based filtering using the Squirro Query syntax when the agent identifies specific facet values in the query. That allows precise filtering by metadata fields such as data source, author, or document type.

    The prompt used to instruct the LLM on how to construct the filter query is configurable at the project level via the genai.sqgpt.squirro-retriever-field-prompts project setting, using the squirro_query_syntax_filter key. By default, the prompt directs the LLM to apply facet filters only. You can customize the prompt to allow any Squirro Query syntax expression to be passed directly to the retriever.

    For example, a customized prompt can instruct the LLM to filter by paragraph offset range:

    $offset > 10 AND $offset < 100

    That allows the agent to target a specific section of a document based on paragraph position.

Retrieval with Intents#

The Retrieval with Intents setting determines whether inferred intent filters must match search results, controlling the balance between precision and recall.

  • without_intents

    Completely ignores intent filters and performs only a baseline search. That mode prioritizes recall over precision and is useful when broad coverage is more important than targeted results, or when intent detection might incorrectly narrow the search scope.

  • optional

    Performs both a baseline search without intent filters and a filtered search with intent filters, then combines the results. That approach maximizes recall while still benefiting from precision improvements when intent filters match. It provides the most comprehensive results across different query types.

    When combined with hybrid_rrf retrieval mode, that approach triggers four searches internally: keyword, semantic, keyword filtered, and semantic filtered. Results are merged via Reciprocal Rank Fusion. That combination delivers the broadest coverage but increases latency compared to other configurations.

  • only_with_intents

    Applies intent filters strictly, returning only results that match all detected intents. That mode maximizes precision by focusing exclusively on documents matching the interpreted user intent. It works best when queries are specific and intent detection is highly accurate.

Retrieval Mode#

The retrieval mode determines how the tool searches for relevant paragraphs within documents.

  • keyword

    Uses traditional keyword-based search on paragraphs only. That mode excels when searching for specific terms, technical jargon, or exact phrases. It provides fast, deterministic results based on term frequency and position.

  • semantic

    Uses embeddings-based semantic search on paragraphs only. That mode understands the meaning and context of queries, making it effective for conceptual searches where the exact terminology may vary. It can find relevant content even when different words express similar ideas.

  • hybrid_rrf

    Combines keyword and semantic search using the Reciprocal Rank Fusion algorithm. That approach leverages both exact term matching and semantic understanding to provide comprehensive results. It delivers the most balanced performance across different document types and query styles.

Fetch Top N Summarize#

The Fetch Top N Summarize parameter controls how many paragraphs are retrieved when the agent performs broader summarization tasks. That setting fetches more paragraphs than standard search operations to increase the overall context available for comprehensive summaries. That higher threshold ensures adequate coverage when users request document overviews or need information spanning multiple sections.

Enable Context Expansion#

The Enable Context Expansion setting controls whether the tool groups retrieved paragraphs into contiguous sections and returns surrounding context. When turned on, the tool identifies clusters of nearby paragraphs based on their position in the document, then expands left and right from each cluster center to produce coherent sections in reading order. Top-ranked paragraphs are always treated as independent cluster centers, ensuring the most relevant content is preserved throughout that process. When turned off, only the top-ranked individual paragraphs are returned. Turning off that setting improves response speed but may return isolated paragraphs without surrounding context.

Number of Clusters#

The Number of Clusters parameter determines how many positional clusters the tool identifies within a single document during context expansion. Each cluster represents a contiguous section of paragraphs expanded around a group of positionally nearby retrieved paragraphs. That setting only applies when Enable Context Expansion is turned on. Increasing the number produces more, narrower sections. Decreasing it produces fewer, broader ones.

Context Expansion on Top N Items#

The Context Expansion on Top N Items setting controls how many top-level documents receive context expansion analysis. Since the tool operates on paragraphs, retrieved paragraphs are first folded back to their parent top-level documents (for example, a PDF or other source item). Context expansion then runs on the paragraphs belonging to the highest-ranked parent documents only. That setting optimizes performance by limiting clustering operations to the most promising candidates. It only applies when Enable Context Expansion is turned on.

Image Reasoning#

The Image Reasoning setting allows for multimodal reasoning in top-ranked documents. When activated, the tool processes visual content such as tables, charts, diagrams, and images using a multimodal large language model. That feature enhances the extraction of information from non-textual elements, though it introduces additional latency and higher token usage due to the processing of images.

Image Reasoning Pages#

The Image Reasoning Pages parameter controls the maximum number of relevant pages per top-ranked document to process for image-based reasoning. Processing is limited to the most relevant pages where visual content is most likely to contain the answer. That parameter only applies when Image Reasoning is activated. Higher values provide broader coverage but increase processing time and token costs significantly due to image encoding.

Semantic Profile Config#

The Semantic Profile Config parameter defines advanced configuration for semantic search behavior. That configuration string controls how the semantic search engine processes queries and retrieves results. That advanced setting is typically used for fine-tuning semantic search performance and should only be modified by specialists when specific retrieval behavior adjustments are required. Multiple profile configurations can be specified as a comma-separated list for different search stages. For assistance, contact Squirro Support and submit a technical support request.

Recency Profile Config#

The Recency Profile Config parameter controls how the retrieval system boosts recent documents in search results. That configuration allows time-based relevance adjustments, making newer documents more prominent in results when appropriate. The setting supports multiple profile configurations via a comma-separated list, allowing different recency treatments at various search stages. That advanced parameter is useful for use cases where document freshness significantly impacts relevance and should only be modified by specialists. For assistance, contact Squirro Support and submit a technical support request.

Contents