Activity Tracking

This page provides an overview of Activity Tracking, which is used to track data about user interactions on the platform.

Note

Activity Tracking is visualized as part of Squirro Monitoring. If you are looking to view project user activity, Squirro Monitoring is the recommended place to start.

Overview

You can configure Squirro to collect data on user interactions with Squirro.

The collected activity data provides insights on the overall usage of the system. For example, what dashboards were visited or what queries a user submitted.

Squirro uses the recorded queries to improve the overall user experience by suggesting popular queries in typeahead or recommending similar queries when a user types into the search bar.

Below you will find a complete list of the data collected by Squirro.

Configuration

Activity tracking is enabled by default on every new Squriro instance.

If you would rather not collect activity logs, go to Server→Configuration and uncheck the (frontend.userapp.log-activity) option.

The activity logs are by default stored under /var/log/squirro/frontend/$hostname/activity.$year-$month-$day.jsonl.

The path can be changed in the common.ini file by adjusting the path config in the activity section:

[activity]
path = /var/log/squirro/frontend/

Activity Tracking Project

Automatic Creation

Self-service instances are shipped with a preconfigured monitoring project space. On new deployments, Squirro creates the monitoring project named Squirro Monitoring upon first admin-user sign up (for each tenant).

You can configure the behaviour of the automatic monitoring project creation in the Server space under Configuration by setting the value of topic.monitoring.default-monitoring-project to true or false (default is true)

Monitoring spaces come with the following:

• Preconfigured sources and pipelines to load activity logs and compute activity insights.

• Dashboard to display user activity within the project.

Manual Creation

On existing deployments, you can manually create a monitoring project by using the Squirro Monitoring project template, as shown in the example screenshot below:

Activity Tracking Components

The Squirro Monitoring uses the following components to collect user activity data and extract knowledge about user behavior:

• Squirro Activity Log: A Data Loader plugin that continuously loads activity data from the activity logs.

• Activity Tracking: A pipelet that enriches activity data with additional user information.

• Squirro Activity Insights: A Data Loader plugin that generates enriched insights about user behavior.

Squirro Activity Log Data Loader Plugin

The Squirro Activity Log Data Loader plugin loads the activity logs saved in the filesystem and inserts them to the Elasticsearch index.

By default, the Data Loader plugin reads the path with activity files from the common.ini config.

If you would prefer to provide a custom path where your activity files are stored, you can specify the Activity path option in the advanced options.

Running Data Loader on Multi-Node Instances and Storing Files in the EFS

The Squirro Activity Log Data Loader plugin works both on multi-node instances and with activity files placed in the EFS. It takes care of loading proper data by resolving the hostname folder where the activity files are stored.

The plugin gathers the activity files from the root level of the activity path and then checks for files inside the folder with the corresponding hostname. If it finds a folder with the same hostname as the hostname that it is running on, it also searches for files inside this folder.

Example

Assuming the Data Loader runs on the instance with the hostname: hostname-1, it will gather the following files:

• /var/log/squirro/frontend/activity.2022-01-01.jsonl

• /var/log/squirro/frontend/hostname-1/activity.2022-01-02.jsonl

But won’t gather the file below:

• /var/log/squirro/frontend/hostname-2/activity.2022-01-03.jsonl

Activity Tracking Pipelet

The Activity Tracking pipelet is part of the Squirro Activity Log Data Loader plugin’s ingestion pipeline.

It enriches loading activity logs with additional user metadata like the user’s name, email, role, and groups that the user belongs to.

If you want to enrich activity logs with custom user-related information, you can configure the pipelet by going to the Pipeline Editor and clicking on the Activity Tracking step.

Squirro Activity Insights Data Loader Plugin

When the Squirro Activity Log Data Loader plugin provides information on user actions, the Squirro Activity Insights Data Loader plugin combines this information to give a bigger picture overview of user behavior.

The Data Loader plugin collects activity data from Elasticsearch and generates insights called sessions.

These sessions allow Squirro to understand how the user behaves on the platform.

For example, it can answer a question like, “How many times does the user type a query but then cannot find a relevant document and leave?”.

The Data Loader computes the following three types of sessions:

• Query sessions

• Dashboard sessions

• Browsing sessions

Query Session

A query session is defined as the activities that are logged between a user’s searches.

It answers the question, “What does a user do after performing a search?”

Dashboard Session

A dashboard session is defined as the activities that are logged on a particular dashboard before the user switches dashboards.

It answers the question: “What does a user do on a particular dashboard?”.

Browsing Session

A browsing session is defined as the activities that are logged between a user’s activity sessions (see below).

It answers the question, “How does a user interact with the Squirro product in general?”

Activity Session

An activity session is a top-level session that enables Squirro to detect when the user entered and left the platform.

It helps produce more detailed insights by creating boundaries between user visits.

The activity session ID is stored in the Redis database and attached to each activity event.

The session starts at the first visit to the platform and ends after an explicit logout or after some period of inactivity.

Each user’s activity extends the session time.

After no activity within the specified time, it is assumed to be the end of the session and further activities will generate a new one.

To configure the session expiration time, go to Server→Configuration, and change the following option:

Activity data collected by Squirro

Label	Description
Email	Email
Groups	Groups
Name	Name
Role	Role
action	Action
community_id	Community ID
community_type_id	Community Type ID
dashboard_id	Dashboard ID
dashboard_title	Dashboard
drop_to	Item Dropped To
edit	Edit
event_source	Source
feedback	Feedback Score
feedback_comment	Feedback Comment
hostname	Hostname
item_id	Item ID
item_link	Item Link
item_rank	Item Rank
items_result	Items Results
last_read_result	Last Read Items Results
layer_title	Layer
microseconds	Microseconds
pageview_id	Pageview
project_id	Project ID
project_title	Project
qa_result	QA Results
query	Query
query_id	Query ID
query_language	Query Language
query_named_entities	Query Named Entities
query_noun_chunks	Query Noun Chunks
query_result	Items Results (deprecated)
query_type	Query Type
query_user_filters	Query User Filters
query_user_terms	Query User Terms (separated tokens)
query_user_terms_count	Query Amount User Terms
query_user_terms_str	Query User Terms
result_start	Result List Offset
result_total	Results
session_duration	Session Duration
session_id	Session
simsearch_result	Similar Searches Results
source_type	Source Type
starred_result	Starred Items Results
type	Type
user_id	User ID
widget_type	Widget Type

Actions Data Collected

Activity Action	Description
item.read	This activity is logged when user opens item/document detail. For example, opening an item detail modal within the Items widget. This is not logged when scrolling over a list of items, only when the full item/document is shown or opened.
query	Every search query that is submitted within the Search widget or Global Search input.
query.result${widgetActionType}	${widgetActionType} is the name of the widget that triggered the activity log. This activity is logged whenever the items/documents query endpoint is fetched and contains information about the submitted query and results. This is used inside of all the widgets that fetch items.
feedback_items_widget	This activity is logged when user submits feedback on an item/document within the Items widget results. Items widget feedback can be enabled within Items widget configuration.
item_detail.query	This activity is logged when a user submits a search within a PDF item/document.
qa.result	This activity is logged when QA results are returned with the QA widget.
simsearch.result	This activity is logged when Similar Searches results are returned within the Similar Searches widget
item_detail.query.results (Depreciated?)	This deprecated activity query is used within the old results widget, currently not implemented for Items widget. It returns info about matching_sub_items.
qa.click	This activity is logged whenever a user clicks on the answer within the QA widget.
simsearch.click	This activity is logged whenever a user clicks on the suggestion/answer within the Similar Searches widget.
feedback_communities_widget	This activity is logged when the user submits feedback on an item/document within the Communities List widget results. Communities List feedback can be enabled within the Communities List widget configuration.
community.click	This activity is logged when a user clicks on a community card and gets redirected to the Community 360 view.
dashboard.view	This activity is logged whenever a user opens a dashboard, by clicking a dashboard tab, opening the dashboard URL or using the dashboard edit mode.
community_360.view	This activity is logged whenever a user opens a Community 360 dashboard.