3.7.5 LTS Release Notes

Warning

This release comes with a major Elasticsearch upgrade. Please read the Elasticsearch section of Breaking Changes before upgrading.

Squirro 3.7.5 long-term support (LTS) was released on May 02, 2023.

It highlights the progress since the previous LTS release 3.6.8. Most of these additions were already introduced in the intermediate releases of the 3.7.x series and have been documented in those release notes as well.

As a long-term support (LTS) release, Squirro 3.7.5 LTS will receive updates for security issues and important bug fixes for the next two years.

Reference: See Squirro Release Process for details on Squirro’s versioning.

What’s New

Elasticsearch Upgrade

Elasticsearch has been upgraded to Version 8.6.2.

Now, Elasticsearch has the following security features applied out of the box:

• Basic authentication

• Secure internode communication

• Secure API traffic

Reference: For step-by-step instructions on how to migrate indices, see Elasticsearch Migration.

Note Taking

Note Taking is a new feature that allows users to add private sentence-level notes to documents and associate those notes with project labels or communities. It also allows for bookmarking at the document level, which prioritizes bookmarked documents in search results.

Notes can be hidden in the UI if frontend.notes.enabled project-level configuration is set to false.

Note

Sentence-level notes are only available for PDF documents and documents that Squirro indexes as PDFs.

Reference: To learn more, see Note Taking.

Bulk Labeling

AI Studio now includes a new Bulk Labeling feature that allows you to bulk label items based on proximity rules that you’ve previously defined within your project.

This allows you to create large training sets in relatively short amounts of time.

Reference: To learn more, see Bulk Labeling.

Widgets

Now, all customizable React widgets have been added to the Create Custom Widget page.

Reference: To learn more, see How to Create a Squirro Custom Widget.

A Search Feedback mode toggle button was added to the Actions widget, allowing you to collect search feedback whenever the Global Search Dashboard is displayed.

Search

Global search typeahead now shows two columns if there are community page suggestions.

Enhanced Storage Efficiency

All new projects also have significantly improved storage efficiency through changes to the mapped summary field, which is now just an abstract and not the full body text.

You can manually set the maximum size of this abstract using the new server setting lib.pipeline.content.max-summary-size.

Truncation can be disabled by setting the value to 0. When you load data with an existing summary field, you can automatically map that field. Otherwise, the first paragraph up to the max summary size is used. This also means that PDF sub-items do not populate the sub_item.summary field anymore.

New Project Configuration Options

New project-level configuration options include the following:

• Added the topic.item.allowed-reader-edit-keywords configuration, which allows setting specific Item keywords that users with the Reader role are allowed to edit.

• Added the configuration option topic.typeahead.facetvalue.configuration to allow simpler performance tuning for facet-value typeahead. For more information, see Typeahead Suggestions.

• Added the configuration option datasource.rerun.index.consider-all-project-items, which enables Rerun from Index. This considers all the items within the project instead of restricting it to the items that belong to the configured data sources of the pipeline workflows.

• Exposed Relevant Community Retrieval configuration. Configured semantic term expansion on detected concepts via topic.search.communities.relevant.semantic-term-expansion; Aggregation of the most significant communities is configured via topic.search.communities.relevant.term-aggregation. Those settings are useful to tune for speed vs precision.

Reference: For information on how to change project configuration settings, see Project Configuration.

Improvements

Communities

• Relevant Community search suggestions are now faster (improved caching, several bug fixes) and more relevant by leveraging concept search. Those changes now find the best matching community for the given query.

• Extended Relevant Community API to exclude certain communities by their ID. This is useful when the Communities List widget (mode: relevant communities) is used from within a Community 360 page.

• When the Communities List widget is in Show relevant, there’s a new All option. When selected, the widget will show the most relevant communities from all community types.

Data Processing Pipeline

• Now pipeline save error messages are shown to the user.

• Added Info and Rerun actions to pipeline steps in the non-edit mode.

• Rerunning pipelines can now be scoped to query.

Search

• Concept search performance has been improved, including better identification of semantically-similar terms for searches.

• In search query-processing, added noun-chunks & near entity recognition (NER) to the response.

• Global Search configuration was simplified to a boolean checkbox.

• The width of the Global Search bar was increased.

• After favoriting a query, the toast will show a link to see all favorites if the link is configured in server settings.

• Query endpoint search performance was improved by reducing the amount of fetched payload by loading only a specific subset of interesting facets client.query(fields=[keywords.facet_name, keywords.facet_name2]. To fetch all, provide fields=["keywords"] or fields=["keywords.*"]

• For searches utilizing advanced query syntax techniques, added support for nested, negated groups like NOT ( (tag:foo OR tag:bar) AND hello ).

• Added query parameter to template.

• Sped up the query-processing workflow by delegating the loading model to the external NLP service if it’s available on the instance.

UI

• In the Dashboard Editor, the widget browser Close and Add Layer/Widget buttons were moved to the top.

• Added a link to open Community 360 pages in a new tab from the Tabs widget.

• Removed the Excel and Alert options from the dashboard More button. Note that Save to Excel option is available from the Action widget now.

• Added dashboard filters to query when saving favorite searches.

• Now, vertical scaling of the dashboard search widget is allowed.

• Previously, users could not clear filters applied through persistent links (if they pasted a URL that contained filters, for example). Now, users can reset those filters via the Reset Filters widget.

• Label names suggestions occupy the first row, the ones that are not fitting the first row are hidden.

Widgets

• Added a new default main.js for React custom widgets.

• Added Item detail Storybook files and documentation. Updated many other stories based on recent changes.

• Added tooltips to Item Card widget actions.

• Implemented the stacked bar chart entities mode.

• The Navigation Chips and Table widgets were migrated to React.

• Improved community list empty image styling to be consistent with other widgets.

• Added a migration script for the Community Items widget.

• Updated CommunityCard overrides to include CommunityCardComponent.

• Improved the useWithoutCollection React hook to include props and config.

• Now, the Reset Filters widget accordion shows the selection count.

• Exposed facets aggregations collection to custom widgets via own factory.

• Added support for adding custom widget context values inside of the main.js file.

• Added Community Context to CommunityCard for custom widgets.

Other Improvements

• Improved Monitoring Studio plugin to fully support secured Elasticsearch.

• Improved Monitoring Studio plugin to support the notes and pdfconversion services.

• Added support for nested aggregations for merged entities setup.

• For those with the preview service-nlp enabled, added the possibility to accelerate QA models via ONNX runtime. This can be done in the configuration file by setting use_ort to True.

• A wider range of languages is now supported in the NLP Keyphrase Tagger, including Chinese and Russian.

• Improved nested entity highlighting and entity sorting for PDFs and non-PDFs.

• Entities for PDFs are now stored with a page_info property, which includes information about the size (width and height) of the page that the entity was found on.

• Created a conditional NLP step that executes depending on a specified condition (i.e. whether the service is healthy).

Breaking Changes

Elasticsearch

Upgraded to Elasticsearch 8.6.2, which can only read indices created in version 7.0 or above.

Required Actions

• All indices created in Elasticsearch 6.x and earlier versions are not supported. You must reindex them with Elasticsearch 7.x before proceeding with the upgrade.

• All custom Elasticsearch plugins will be removed as part of the update process and need to be re-installed.

Attention

See Elasticsearch Migration for migration instructions.

If you are upgrading a multi-node installation, please reach out to Squirro Support for assistance before upgrading, as the exact steps can vary widely depending on your existing setup.

Futher Upgrade Notes

• Elasticsearch templates can no longer be configured by the changing files in the filesystem. Instead, use the topic.elasticsearch.index-templates-settings option in the server configuration.

• Removed legacy Elasticsearch templates and now use new Index Templates (learn more at Index Templates). This means that all custom changes made to the templates have to be reapplied to the new templates.

• The es_index_servers in the common.ini file now points directly to the Elasticsearch instead of to the NGINX location.

Widgets

• Custom Widgets Hooks under the hooks object becomes Hooks.widgetName, for example useCollection becomes Hooks.useCollection.

• The Result widget was changed to the Items widget in default dashboards.

Microsoft Exchange

• Now, it is no longer possible to receive a Squirro token by providing a mapped Exchange user identity string, but the signed Exchange user identity token is needed, to protect against malicious actions.

• The get_token_from_msexchuid method has been replaced by the get_token_from_msexchuid_token method.

Installation and Upgrade

Note

For migrating Elasticsearch, see the previous notes in Breaking Changes and refer to Elasticsearch Migration for step-by-step migration instructions.

If you are upgrading a multi-node installation, please reach out to Squirro Support for assistance before upgrading, as the exact steps can vary widely depending on your existing setup.

You will have to resolve at least the following config files when upgrading from Squirro 3.3.0

• /etc/nginx/conf.d/ssl.inc

• /etc/squirro/common.ini

For new installations, please follow the Installing Squirro on Linux instructions.

To upgrade an existing installation, please consult Upgrading Squirro.