# 3.8.6 LTS Release Notes > **Warning** > > **End of Life Release** > > > > Squirro 3.8.6 LTS reached end of life in October 2025 and is no longer supported. This release no longer receives security updates or bug fixes. > > > > For the current supported LTS release and upgrade instructions, see [Release Notes](../index.md#getting-release-notes). Squirro 3.8.6 LTS was released October 16, 2023. > **Note** > > On November 03, 2023 a patch was released for Squirro 3.8.6 LTS. For details, see the [3.8.7 LTS Release Notes](3.8.7-release-notes.md#release-notes-3-8-7). Reference: Learn more about the [Squirro Release Process](../squirro-release-process.md#getting-squirro-process). > **Caution** > > This release includes several breaking changes. See the [Breaking Changes](#breaking) section at the end of this page to learn more. ## What’s New Squirro 3.8.6 LTS includes significant platform improvements, including the following: - A redesigned and improved AI Studio. - New search features, including new scoring profiles, query syntax features, and the introduction of semantic _Hybrid_ search. - Several Cognitive Search user interface improvements. - A new _Documents_ connector for uploading local files to Squirro, replacing the _ZIP_ connector. - Built-in forms functionality for Project Settings configuration options. - Many more general platform improvements! ### SquirroGPT Although not part of this specific release, Squirro also introduced several major new SquirroGPT features and improvements over the past few months. Reference: To learn more about Squirro’s generative AI offering, see [Squirro Chat](../../../products/squirrogpt.md#products-squirrogpt). ### Redesigned and Improved AI Studio AI Studio has been completely reworked, with a redesigned UI, new features, and improved performance. Creating no-code machine learning models has never been easier. You can now create, train, and deploy models in three steps. [](https://s3.amazonaws.com/download.squirro.net/docs/technical/ai-studio/ai-3.png) New or improved features include: - The introduction of AutoML templates (Fast, Balanced, and Accurate), which will train multiple models with automated hyperparameter tuning, model ensemble, and automatically deliver the best-performing model for the dataset. - AutoML-supported libraries, models, and algorithms include LightGBM, CatBoost, XGBoost, Fasttext, RandomForest, ExtraTrees, KNeighbour, NeuralNetTorch and FastText. - Proximity-rule-based bulk labeling for sentence-level classification models. - Query-based bulk labeling for document-level classification models. This enables importing training data into the ground truth using Squirro Item labels. Additionally, for more advanced users, this feature enables the creation of training data utilizing the full potential of Squirro’s Query Syntax. - Unpublish models from the publish screen, which will delete the published workflow from a Squirro project. - Added label selection during deploy of a model within AI Studio. - Added support for custom templates in AI Studio and improved template handling generally. - Deploy or undeploy a published model directly into/from a pipeline from the AI Studio, without having to open the Pipeline Editor. This will automatically place (or remove) the model as a pipelet within the classification step of a selected Squirro pipeline. - A completely redesigned and improved interface, as shown in the example screenshot below: [](https://s3.amazonaws.com/download.squirro.net/docs/technical/getting/new-ai-studio.png) Technical improvements to AI Studio in this release include the following: - Labeling focus mode has been reworked and now delivers faster results. - Added a help dropdown for AI Studio pages. - Added an option to limit bulk labeling items. - Implemented document-level Bulk Labeling. - Added a more informative error message when trying to delete a ground truth that is being used by a model. - Reworked the sentences highlighting logic when fetching all groundtruth items to be the same as for an individual item (this fixed discrepancies for sentences). - Query labeling templates now accept a `query_count` parameter to limit the number of items retrieved for bulk labeling. - Added relationships from groundtruth to model and from model to publish in the machinelearning database. `ground_truth_name` in model and publish will now automatically update with changes to the `ground_truth`, `model_name` will now automatically update with changes to the model. Also migrated all `machinelearning` service migrations to use the new Redis functionality to remember if it has been run before. - Added the `include_pipeline_workflows` parameter to `get` requests for published AI Studio models, which tells the ML service to retrieve the `datasource` pipeline workflows where the published models are being used. - `GET` requests to endpoints for getting data about published models now enrich each model with the pipeline workflows they are used in. - There is a new integration endpoint for query-based bulk labeling. - Disabled balancing dataset on AutoML templates to avoid downsampling. Reference: To learn more, see [AI Studio](../../../technical/ai-studio/index.md#ai-studio). ### A New Editable Explore Dashboard in the Setup Space The _Explore_ dashboard in the Squirro Setup space is now editable thanks to the creation of a new `Explore` dashboard designation included with all new projects. [](https://s3.amazonaws.com/download.squirro.net/docs/technical/getting/new-explore.png) As part of this update, general _Explore_ screen performance was improved, including the Items widget performance within the dashboard. Reference: To learn more about this new dashboard designation, see the [Explore Dashboard](../../../technical/dashboards/types/explore.md#dashboards-explore) documentation. ### New Search Scoring Profiles and Semantic Hybrid Search Squirro search has been improved and extended in this release, including the introduction of multiple new _Scoring Profiles_ and the introduction of the semantic _Hybrid_ search feature. The following screenshot demonstrates its improved performance with natural language queries and its ability to highlight text that matches the query: [](https://s3.amazonaws.com/download.squirro.net/docs/technical/getting/semantic-example.png) > **Note** > > Hybrid search is currently in beta. Its pipelet and semantic search components only work if a separate model is available, which is not the case for current Squirro out-of-the-box deployments. New or improved search features include: - Introduced the beta feature _Hybrid Search_, which retrieves results based on semantic/vector processes and keywords. This feature was created by adding the scoring profile `semantic`, allowing users to perform approximate vector searches on paragraph embeddings. - Added the scoring profile `extractive-qa`. This highlights the answer to a query within the top-ranked paragraphs. - Added the scoring profile `cross-encoder`. This allows for top-N paragraph reranking based on contextualized similarity of `query and paragraph`. - The new Query Syntax feature `rank_by:{}:` allows you to rank on optional scoring signals to boost matching documents without applying an overall filter. For example, `guest wifi rank_by:{ source:faq }` enforces matches on `guest wifi` but ranks documents higher that are tagged with `source:faq`. - The new Query Syntax feature `profile:{}:` allows you to inject Scoring Profiles dynamically within the query syntax. For example, to filter items that have been read by the user in the last 10 days, the syntax is `profile:{ plugin:last_read $last_days:10 }`. Another use case is to apply scoring profiles as additional ranking signals by combining the literals `rank_by` & `profile`, for example `genai how to rank_by:{ profile:{last_read} }`. This matches documents with `genai how to` and further boosts last read items. - Add new scoring-plugins for recommendations: on user-item-history (`recommend_on_user_items`) & item id’s (`recommend_on_item`). - Added _Scoring Profiles: Boost_, allowing you to boost documents that belong to the user’s subscribed communities. The plugin is usable within the query syntax via `profile:{plugin:subscribed_communities}`. Boosting was previously known as `rank_by`. - A new Scoring Profile plugin was created to recommend items based on users’ past searches. Use the profile within a query like profile:`{ plugin:recommend_on_searches $last_searches:5 }`; This will return concept search results on the last five unique user queries. Related matching terms used for the recommendations are marked in bold and red. - Search Scoring Profiles: Implemented `with_user_read` profile to tag `read` items before returning them via the `query` API. - Added new Squirro Syntax, which allows proximity search using [Span Near ES queries](https://www.elastic.co/guide/en/elasticsearch/reference/current/query-dsl-span-near-query.html). An example of such syntax is `"auction of gas"~(3)` which searches for documents where the terms inside quotes occur within 3 words in any order and `"auction of gas"[3]` which is similar, but only considers documents where the items occur in the order provided. - Contextualized PDF Inline Search: An additional widget query template can now evaluate the parsed query context (to activate semantic search vs keyword search). - Squirro query syntax now supports multiple sort clauses, allowing you to add additional sorting clauses when two or more items match their primary sort criteria. See [Multiple Sorting Criteria](../../../technical/search/features/query-syntax.md#qs-sort-multiple) to learn more. - Squirro Docs now includes auto-generated [Scoring Profile Plugin Reference](../../../technical/search/relevancy/scoring-plugins/index.md#scoring-plugins-reference) documentation. Reference: For more information on semantic and _Hybrid_ search, see [Semantic and Hybrid Search](../../../technical/search/features/semantic-search.md#search-semantic). ### Cognitive Search User Interface Improvements Cognitive search has undergone a number of UI and UX improvements in this release. This includes a fresh new look and feel when searching, as well as a number of new features and improvements, including the ability to choose document highlighting color. [](https://s3.amazonaws.com/download.squirro.net/docs/technical/getting/highlight-feature.png) Also, document labels can now be viewed in a popover by clicking the label icon in the _Items_ widget, allowing you to browse labels without leaving the search results page, as shown in the example image below: [](https://s3.amazonaws.com/download.squirro.net/docs/technical/getting/label-popover.png) Other new Cognitive Search features and improvements include: - Global search now breaks queries into individual chips, making it easier to edit queries after the initial search. Through `frontend.userapp.query-chips-enabled` in Project Settings this feature can be enabled or disabled. Learn more about [Search Bar Chips](../../../technical/search/features/chips.md#search-chips). - You can now configure the text highlight color via the Setup space App & Nav Bar Settings. - Upgrades to the Google Drive, SharePoint, and Dropbox 1-click connectors, including the input label, placeholder, and helper text. - Data sources now have an affiliated picture. They are initialized with the picture from the Dataloader plugin they are created from. - There is now support for loading images (TIFF, PNG, JPEG) into Squirro as PDFs. - Squirro now supports deploying a model to multiple pipelines at once. - Large PDFs are auto-scaled down to the viewer size. - Enabled typeahead for PDF inline search inside the _Item Detail_ modal. - PDF inline search, which appears within the item detail screen, now uses the same query processing as global search. This significantly improves search quality inside the item detail screen and allows for future improvements on the backend. - Squirro now supports _Speech to Text_ for the global search bar within Chrome browsers. - Added a Similar Items tab to the Items Detail widget. - The Items widget now has a configuration to specify which labels are shown when creating a new note using the Note Taking feature. - Added the Community Info Card widget to the platform. ### A New Documents Connector A new _Documents_ connector replaces the _ZIP_ connector. In addition to handling ZIP files, this connector supports uploading individual, non-zipped documents (a single PDF file, for example). The range of supported documents was also expanded, including ODT, ODP, ODS, CSV, and RTF file formats. [](https://s3.amazonaws.com/download.squirro.net/docs/technical/getting/documents-3-8.png) ### Built-In Forms for Configuration Options Under Project Settings, intuitive built-in forms are now available for editing some configuration settings instead of plain text JSON. See the example `topic.typeahead.content.configuration` settings screenshot below: [](https://s3.amazonaws.com/download.squirro.net/docs/technical/getting/json-form.png) ### Other New Platform Features and Major Improvements - A new sessions mode with two default timeouts, _session timeout_ and _maximum session duration_. Learn more at [Reducing Session Lifetime](../../../technical/admin/configuration/secure-config-guide.md#session-lifetime). - A new and improved Squirro Monitoring dashboard. To learn more about manually upgrading an existing project dashboard, see [Upgrading Dashboards Manually](../../../technical/dashboards/upgrade.md#dashboards-upgrade). (New Squirro releases update widgets automatically. Dashboards must be manually updated.) - Studio plugins using project-scoped endpoints now support token-based authentications. Token-based authentication is not supported for non-project-scoped endpoints for security reasons. - Added a new endpoint to expose SharePoint metadata in the SAML SSO Plugin. - Exposed priority levels in `squirro_data_load`. - Added a new libNLP step that truncates the fields of the given documents down to the first `X` words. - Storybook was updated to the latest version, add full widgets to storybook and community list widget tests - The `plumber` service can now emit structured logs. Structured logs generated during data ingestion (e.g. from the execution of a pipelet in a pipeline workflow) can now be included in the Data Ingestion Logs dashboard of the Squirro Monitoring project. Authors of pipelets can use the new attribute `self.slog` to emit structured logs from within their pipelets. - Added the `indexmanager` service. - Admins can now access query-processing results in dashboard query templates using the `queryContext` variable. - There is a new tag for highlighting answer context. - Introduced a new pipeline step called _Archive Extraction_ that extracts archive files. At the moment only ZIP files are supported. Each file in the archive becomes an individual item, and the archive itself is removed. - Created a new index manager service for Elasticsearch maintenance. - Introduced a new pipeline step called _MIME Type Detection_ that identifies the MIME type of file-based items. This step can serve as a central way to detect MIME types in the pipeline, instead of replicating this logic in various dataloader plugins in cases where the MIME type is either not provided or is not trusted because it may be incorrect. - Created the option to display **only** unique items within the _Timeline_ and _Metric_ widgets. - Included [XlsxWriter](https://pypi.org/project/XlsxWriter/) as an optional package on the Squirro mirror. - Added GlobalSearch query to EntitiesListWidget query. - Now users can use `searchbar.empty` and `searchbar.value` in dashboard query templates. - Added a new type of content streamer called QueueFileSystemStreamer. This streamer extends the existing FileSystemStreamer with the idea that each data source has its own distinct queue. It can be enabled by modifying the `/etc/squirro/common.ini` file, by setting the `stream_type` option of the content section to `filesystem-queue` from filesystem. - Added documentation for the content streamers of the data processing pipeline. See [Content Streamers](../../../technical/data-processing/content-streamers.md#data-processing-content-streamers). - Added an _autogluon_ classifier to the libNLP classification steps. - You can now store metadata information on sub-items (pages) in the same way you can for top-level items. This metadata information can be used for internal operations and is not exposed via the Facet Aggregation API, which only considers top-level items. ## Improvements > **Note** > > Bug fixes are included with individual biweekly non-LTS releases. If you are looking for bug fix information, see specific [Squirro 3.8.x Release Notes](squirro-3.8.x-release-notes/index.md#release-notes-3-8-x). The following general platform improvements are included with this LTS release: ### Dashboards and Widgets - The Items Table widget was migrated to React. - Show more than X number of results in Items widget when approximation is used on the backend. - General Items card improvements. - Items card in ItemsWidget now has a larger font and there is answer context highlighting in the abstract. - Global Search and Community 360 dashboards are no longer loaded on top of other dashboards. - Changed the default font size for the Chip component to 13px. - Changed yellow highlighting to bold font for highlighting inside item preview, item detail sidebar and HTML-based items. Yellow highlighting will still be used in the PDF highlighting. - Improved Tabs performance. - Updated default colors and themes based on the latest Cognitive Search design. - Users scroll to the first highlighted section within a document after clicking on a QA widget answer. - Added _translate_ and _compare_ tooltips to the _ItemsWidget_ card. ### Data Loading and Processing - The three cloud connectors (Dropbox, Google Drive, OneDrive) now support zip files. - Added support for zip files in the Microsoft Exchange plugin. - The data sources in the Setup space now have a tooltip on their names in case their display name is clipped. - Ensured all connections have a 20-second timeout, including the `/_internal/status` endpoint. - Enabled placeholders in `dataloader` inputs. - Added an integration test for the synchronous batch processing feature. - Improvements to the Data Ingestion Logs dashboard in the Squirro Monitoring project. Specifically, batch-level start/end logs have been added for the pipeline steps operating in parallel execution mode. All batch-level start/end logs are now logged at the `info` logging level, while all item-level start/end logs (which provide more detail, supported for steps with parallel execution mode) are logged at the `debug` logging level. Therefore, from now on, start/end logs will be displayed by default in the dashboard without the need to modify the logging level to `debug`. Finally, structured logs have been added in a few places to assist with understanding. - The `pdfconversion` service can now emit structured logs, which are linked to the rest of the data ingestion logs. Therefore, structured logs from the `pdfconversion` service will now be included in the _Data Ingestion Logs_ dashboard of the Squirro Monitoring project. - Increased and synchronized the timeout options related to pipelets involved in data ingestion. - The default pipeline workflows for Dropbox, Google Drive, and Microsoft OneDrive connectors now include the pdf-conversion and webshot steps. This means that MS Office documents found in these services will be converted to PDFs and a thumbnail will be created for them. - Improved the reliability of PDF conversions when multiple conversions take place in parallel. ### Communities - The CommunityItems widget was ported to React and a custom widget API was implemented. - Added feedback to the Communities List widget. - Community augmentation is now easier to implement by making all the returned fields optional. Name, photo, and properties are now used if present but don’t have to be returned. - Photos returned by community augmentation can now be absolute URLs as well. - Squirro now includes community types in the response after subscribing to a community. - Community subscriptions now reflect the state of applied filters and the search bar query. - Now the horizontal mode is available without relevant communities in the Communities List widget. - Added a widget query to the Communities List widget. - Added `query` to community subscriptions collection, allowing for community subscription filtering via query. This is currently available in the Items Table widget. ### Search - Simplified usage of scoring profiles. Plugins can now be more easily used and configured directly by their name as `profile:{ last_read count:10 }` (This example boosts the last 10 read items). - Within search, for better statistics on the indexed data, page offsets are now stored on PDF sub-items as numbers. - Added support to leverage multiple search scoring profiles at the execution stage (`rerank`, `render`). - Paragraphs are now stored in a separate Elasticsearch index. - Squirro now applies ACL queries as Elasticsearch filters. It allows Elasticsearch to cache the filters, thus users with the same ACLs will get results faster. - Added support for the `NOT` literal for search scoring profiles. - Exposed search tuning parameters to the project configuration. This allows users to config additional Elasticsearch query API performance parameters. - Squirro now uses Orjson for Elasticsearch serialization and deserialization. This improves performance as Orjson is much faster than the simple JSON used by the Elasticsearch client. - Now, Orjson is used for the SquirroClient. It improves the performance of serializing and deserializing data. It is also used in the WebOb package to improve the speed of parsing responses. - When searching, improved the retrieval of relevant documents and highlighting of matching tokens (including lemmas & stems) based on the detected language of the provided query. The language distribution across indexed documents is considered to achieve stable language detection for short keyword queries. - Search indexing now uses the recommended (more aggressive) stemmers for all supported languages. - Removed typeahead suggestions that don’t include meaningful content, a standalone asterisk symbol (*) for example. - Returned scoring profile items are now tagged with associated/relevant scoring-profiles (= profiles that matched on the item and boosted its relevancy score). See `item.matched_profiles`. - Added an option to specify `sampler` for search aggregations. It can be used within instances with a large number of documents to drastically improve performance. - Added an option to quantize embeddings. This allows for storing embeddings as bytes in the Elasticsearch index. - Now, when a phrase or proximity filter is applied to the query, the whole phrase is highlighted, not individual terms. - Squirro now returns the number of sampled documents by Elasticsearch. This allows the UI to display how many documents were sampled when using the random or aggregation sampler. - Within the typeahead API, suggestions may contain the new field `chips` if requested (`options.with_tokenized_suggestion:bool`). These additional `chip` tokens are used to render the chips in the global search bar. - Introduced a new highlighting tag for quick answers, with key answer terms bolded. - Phrases are now highlighted in a more robust way (don’t split tokens). - Enriched session with query-context metadata like user terms and filters, detected query language, noun chunks, and named entities. - Stopwords are no longer highlighted in abstracts. Highlights are stripped from Elasticsearch and the abstract is highlighted by Squirro based on “cleaned” terms. ### Other Improvements - Now, `squirro_status` works even if a service is not installed. - Suppressed the `Instantiate SkipTimeoutConnectionClass` message in `squirro_status` output. - Reduced noisy logging in `frontend.log` by the monitoring plugin. - Store read item IDs on the frontend to show the correct status immediately to the user. - Enabled `http2` for all resources. - Various extensions and improvements to the structured logging system. - File-based Squirro items (i.e. items that include the `files` attribute) returned by the API will now include an `original: true` flag to indicate the file from which the item is derived. - Updates to 1-Click Connector configuration forms. - Moved the `nlp service` config to the configuration service so that the config is displayed in a user-friendly way. - Added config to enable/disable balancing of dataset inside balancer step. - Activity Insights now has higher scheduling frequency and separate backoff time for long-lived sessions with 15min (dashboard & browsing session) and short-lived query session (3 minutes). - Added validation to the configuration service. Now, dictionary settings defined in the configuration service may include a schema definition that is then used for validation, e.g. provided text instead of number, value not within specified range etc. - Upgraded Filebeat to version 8.8.1. - Limited the number of `max_features` used on SKLearn transformer to avoid too much memory consumption when a dataset is too large. ## Breaking Changes See the following three categories for breaking changes included with Squirro 3.8.6 LTS: ### General - The Items widget was redesigned. While the API remains the same, verify that any customizations you’ve made within your projects still work correctly. - Result List and Cards widgets have been removed from the UI. Both are still supported if they already exist within a project, though they are slated for full depreciation during the next LTS release. - The public methods `enqueue` and `enqueue_data` of the `FileSystemStreamer` will now return the absolute path to the enqueued data file. - The plumber service can no longer be forked as forking was causing degraded service performance. - In the App and Nav Bar Settings, project colors now support only the HEX color schema. Admins should update project colors accordingly. ### AI Studio - Dropped the `ground_truth_name` column in the `models` table. - Deprecated the ability to change the `id` and `name` of the groundtruth in the `models` table. - Dropped the `gt_id`, `gt_name`, and `model_name` columns in the `publish` table. - Adding a document-level classification model to a pipeline now requires the `document_label` argument that defines the label category used for the model to output its predictions. - If you are using custom templates within AI Studio, they must be handled using the new structure. ### Search - The _Smart Filters_ feature is now fully deprecated. - The `perform_on_languages` option in the QA widget configuration no longer supports the `None` value. To find an answer regardless of the language of the question, set this option to `*`. - The `terminate_after` option in the content typeahead configuration no longer supports boolean values. To stop query execution from terminating early, set this option to `0`. - Changed the default value of `track_total_hits` from `true` to 500,000 to optimize query performance. This means that Elasticsearch will now stop counting documents when the value exceeds this threshold. To change this behavior, see the `track_total_hits` option in the `topic.search.search-settings` configuration service option. > **Note** > > For help with breaking changes, visit the [Squirro Support website](https://go.squirro.com/support) and submit a technical support request. ## Installation and Upgrade For new installations, find step-by-step instructions in [Install and Manage Squirro with Ansible](../../install/ansible/index.md#ansible) (recommended) or [Installing Squirro on Linux](../../install/linux.md#install-linux). To upgrade an existing installation, see [Upgrading Squirro](../../install/upgrade.md#install-upgrade).