UI Data Loader#
This page provides an overview of data loading using the Squirro UI.
It is intended for Project Creators with access to their project’s Setup space.
Squirro UI Data Loader#
Squirro recommends using the Squirro UI in your browser (as opposed to the Data Loader CLI Tool) to add data to your projects.
To access the Squirro UI Data Loader, follow the steps below:
Sign in to your Squirro project.
Click Setup in the Squirro Spaces Navigation menu in the top-right corner of the page.
Click the Data tab. (Result: The Data Sources page will load.)
Click the plus icon as shown in the screenshot below.
From here, you have a range of options spread across different categories, including the following:
Web, including RSS and Atom feeds, track website changes, and crawl a website.
Social Media such as Facebook, Twitter, Instagram, LinkedIn, YouTube, and more.
Data Import, allowing you to upload CSV, JSON, and Excel files and connect other Squirro data.
Enterprise Data, including Confluence, Dropbox, Gmail, Exchange, Salesforce Cloud, and more.
Business Intelligence connectors such as Power BI, Qlik, Tableau, SAP, Microstrategy, and Sisense.
CRMs, including Microsoft Dynamics CRM, Oracle Netsuite, Salesforce, and SugarCRM.
IT Support Management (ITSM) such as BMC Remedy, ServiceNow, Zendesk, and more.
Premium Data, ranging from Dow Jones Newswires to LexisNexis and many more.
Reference: See Built-In Data Connectors for more details.
Tip: Don’t see a way to connect to your data source? To build your own custom data loader plugin or connector, see How To Write a Custom Data Loader Plugin or How To Write a Custom 1-Click Connector respectively.
Troubleshooting#
If you are having issues with a data source, the first thing you can do is check the Logs for the data source in question.
To access a specific data source’s logs, follow the steps below:
Log in to your project in your browser.
Navigate to Setup → Data Sources.
Click the three dots that appear after hovering over the data source in question.
Click Logs, as shown in the screenshot below:
After clicking Logs, a window will open displaying logs that will help you troubleshoot issues with the data loader and that particular source.
Note
If you are looking to troubleshoot issues with the Squirro Ingester service, view your instance’s Squirro Monitoring.
Source Batch Size#
Squirro dataloaders contain batch size settings.
Source batch size refers to the number of items fetched from the source at once and is set to default at 1000.
Ingestion Batch Size#
You can also set an ingestion batch size by enabling Use custom ingestion batch size.
Ingestion batch size refers to the number of items ingested in Squirro at once after they are fetched from the source.
Ingestion Limit#
The maximum number of items ingested into Squirro can be set by enabling Use custom max no. of items to ingest.
Kill a Running Dataloader Job#
If the dataloader job is running, you can end the job using the Kill option shown below:
Logs for Individual Dataloader Job#
The logs for each dataloader job can be fetched by clicking Logs.
Shown below are the logs of an example dataloader job:
To improve the log visibility for the datasources, users can fetch the logs per source.
These logs are placed in /var/lib/squirro/datasource/source_logs. Given a project_id and a source_id, the log file is represented as project_<project_id>_source_<source_id>.log
Language Field Mapping#
While creating or editing a source, in addition to other Squirro item fields, Language can now be mapped with an incoming source field.
Entities Transfer Using Squirro Project:#
While using the Squirro project dataloader, entities can now be transferred from the source project. This can be set in the Configure Source step while importing data from Squirro project as shown below:
The entities corresponding to Squirro items are imported as shown below:
