3.7.5 LTS Release Notes
3.7.5 LTS Release Notes#
This release comes with a major Elasticsearch upgrade. Please read the Elasticsearch section of Breaking Changes before upgrading.
Squirro 3.7.5 long-term support (LTS) was released on May 02, 2023.
It highlights the progress since the previous LTS release 3.6.8. Most of these additions were already introduced in the intermediate releases of the 3.7.x series and have been documented in those release notes as well.
As a long-term support (LTS) release, Squirro 3.7.5 LTS will receive updates for security issues and important bug fixes for the next two years.
Reference: See Squirro Release Process for details on Squirro’s versioning.
Elasticsearch has been upgraded to Version 8.6.2.
Now, Elasticsearch has the following security features applied out of the box:
Secure internode communication
Secure API traffic
Reference: For step-by-step instructions on how to migrate indices, see Elasticsearch Migration.
Note Taking is a new feature that allows users to add private sentence-level notes to documents and associate those notes with project labels or communities. It also allows for bookmarking at the document level, which prioritizes bookmarked documents in search results.
Notes can be hidden in the UI if
frontend.notes.enabled project-level configuration is set to
Sentence-level notes are only available for PDF documents and documents that Squirro indexes as PDFs.
Reference: To learn more, see Note Taking.
AI Studio now includes a new Bulk Labeling feature that allows you to bulk label items based on proximity rules that you’ve previously defined within your project.
This allows you to create large training sets in relatively short amounts of time.
Reference: To learn more, see Bulk Labeling.
Now, all customizable React widgets have been added to the Create Custom Widget page.
Reference: To learn more, see How to Create a Squirro Custom Widget.
A Search Feedback mode toggle button was added to the Actions widget, allowing you to collect search feedback whenever the Global Search Dashboard is displayed.
Global search typeahead now shows two columns if there are community page suggestions.
Enhanced Storage Efficiency#
All new projects also have significantly improved storage efficiency through changes to the mapped summary field, which is now just an abstract and not the full body text.
You can manually set the maximum size of this abstract using the new server setting
Truncation can be disabled by setting the value to 0. When you load data with an existing summary field, you can automatically map that field. Otherwise, the first paragraph up to the max summary size is used. This also means that PDF sub-items do not populate the
sub_item.summary field anymore.
New Project Configuration Options#
New project-level configuration options include the following:
topic.item.allowed-reader-edit-keywordsconfiguration, which allows setting specific Item keywords that users with the Reader role are allowed to edit.
Added the configuration option
topic.typeahead.facetvalue.configurationto allow simpler performance tuning for facet-value typeahead. For more information, see Typeahead Suggestions.
Added the configuration option
datasource.rerun.index.consider-all-project-items, which enables Rerun from Index. This considers all the items within the project instead of restricting it to the items that belong to the configured data sources of the pipeline workflows.
Exposed Relevant Community Retrieval configuration. Configured semantic term expansion on detected concepts via
topic.search.communities.relevant.semantic-term-expansion; Aggregation of the most significant communities is configured via
topic.search.communities.relevant.term-aggregation. Those settings are useful to tune for speed vs precision.
Reference: For information on how to change project configuration settings, see Project Configuration.
Relevant Community search suggestions are now faster (improved caching, several bug fixes) and more relevant by leveraging concept search. Those changes now find the best matching community for the given query.
Extended Relevant Community API to exclude certain communities by their ID. This is useful when the Communities List widget (mode: relevant communities) is used from within a Community 360 page.
When the Communities List widget is in Show relevant, there’s a new All option. When selected, the widget will show the most relevant communities from all community types.
Data Processing Pipeline#
Now pipeline save error messages are shown to the user.
Added Info and Rerun actions to pipeline steps in the non-edit mode.
Rerunning pipelines can now be scoped to query.
Concept search performance has been improved, including better identification of semantically-similar terms for searches.
In search query-processing, added noun-chunks & near entity recognition (NER) to the response.
Global Search configuration was simplified to a boolean checkbox.
The width of the Global Search bar was increased.
After favoriting a query, the toast will show a link to see all favorites if the link is configured in server settings.
Query endpoint search performance was improved by reducing the amount of fetched payload by loading only a specific subset of interesting facets
client.query(fields=[keywords.facet_name, keywords.facet_name2]. To fetch all, provide
fields=["keywords"] or fields=["keywords.*"]
For searches utilizing advanced query syntax techniques, added support for nested, negated groups like
NOT ( (tag:foo OR tag:bar) AND hello ).
Added query parameter to template.
Sped up the query-processing workflow by delegating the loading model to the external NLP service if it’s available on the instance.
In the Dashboard Editor, the widget browser Close and Add Layer/Widget buttons were moved to the top.
Added a link to open Community 360 pages in a new tab from the Tabs widget.
Removed the Excel and Alert options from the dashboard More button. Note that Save to Excel option is available from the Action widget now.
Added dashboard filters to query when saving favorite searches.
Now, vertical scaling of the dashboard search widget is allowed.
Previously, users could not clear filters applied through persistent links (if they pasted a URL that contained filters, for example). Now, users can reset those filters via the Reset Filters widget.
Label names suggestions occupy the first row, the ones that are not fitting the first row are hidden.
Added a new default
main.jsfor React custom widgets.
Added Item detail Storybook files and documentation. Updated many other stories based on recent changes.
Added tooltips to Item Card widget actions.
Implemented the stacked bar chart entities mode.
The Navigation Chips and Table widgets were migrated to React.
Improved community list empty image styling to be consistent with other widgets.
Added a migration script for the Community Items widget.
CommunityCardoverrides to include
useWithoutCollectionReact hook to include props and config.
Now, the Reset Filters widget accordion shows the selection count.
Exposed facets aggregations collection to custom widgets via own factory.
Added support for adding custom widget context values inside of the
Added Community Context to CommunityCard for custom widgets.
Improved Monitoring Studio plugin to fully support secured Elasticsearch.
Improved Monitoring Studio plugin to support the
Added support for nested aggregations for merged entities setup.
For those with the preview service-nlp enabled, added the possibility to accelerate QA models via ONNX runtime. This can be done in the configuration file by setting
A wider range of languages is now supported in the NLP Keyphrase Tagger, including Chinese and Russian.
Improved nested entity highlighting and entity sorting for PDFs and non-PDFs.
Entities for PDFs are now stored with a
page_infoproperty, which includes information about the size (width and height) of the page that the entity was found on.
Created a conditional NLP step that executes depending on a specified condition (i.e. whether the service is healthy).
Upgraded to Elasticsearch 8.6.2, which can only read indices created in version 7.0 or above.
All indices created in Elasticsearch 6.x and earlier versions are not supported. You must reindex them with Elasticsearch 7.x before proceeding with the upgrade.
All custom Elasticsearch plugins will be removed as part of the update process and need to be re-installed.
Futher Upgrade Notes#
Elasticsearch templates can no longer be configured by the changing files in the filesystem. Instead, use the
topic.elasticsearch.index-templates-settingsoption in the server configuration.
Removed legacy Elasticsearch templates and now use new Index Templates (learn more at Index Templates). This means that all custom changes made to the templates have to be reapplied to the new templates.
common.inifile now points directly to the Elasticsearch instead of to the NGINX location.
Custom Widgets Hooks under the hooks object becomes
Hooks.widgetName, for example
The Result widget was changed to the Items widget in default dashboards.
Now, it is no longer possible to receive a Squirro token by providing a mapped Exchange user identity string, but the signed Exchange user identity token is needed, to protect against malicious actions.
get_token_from_msexchuidmethod has been replaced by the
Installation and Upgrade#
You will have to resolve at least the following config files when upgrading from Squirro 3.3.0
For new installations, please follow the Installing Squirro on Linux instructions.
To upgrade an existing installation, please consult Upgrading Squirro.