Data Loader Plugins#
The data loader can be extended with data loader plugins. Those plugins are used to create custom data connectors to sources that are not yet included in Squirro.
Introduction#
Data loader plugins are created as Python classes. Once written, they can be uploaded to a Squirro instance.
From that point on a project creator can use the data loader plugins as a connector without needing any programming knowledge. Such plugins look and feel the same as any Squirro built-in data connector.
Process#
The task of a data loader plugin is to connect to the data source and output the records it retrieves from that source.
In the context of the data loader these records are referred to as rows. The project creator will then use the data loader’s various mapping capabilities and the pipeline to convert these rows into Squirro items.
Features#
To achieve this process, a data loader plugin has access to a number of features:
Most basically Squirro handles managing of the Data Loader Plugin Configuration.
The data loader plugin can also delegate some of the handling of Incremental Loading.
To assist the project creator with the mapping, the Data Loader Plugin Preview is useful.
The data loader plugin can read and write a cache and a persistent store. See API for Caching and Custom State Management.
1-Click Connectors#
Almost all data loader plugins will need to authenticate to the source system.
In modern systems this can be done using
OAuthwhich permits authenticating against other systems without having to persist a username and password combination.Squirro refers to data loader plugins that make use of this as 1-Click connectors.
data-loader-customer-connector covers how these can be created.
To learn more and browse available 1-Click Connectors, see 1-Click Enterprise Connectors.
Custom Plugins#
Squirro offers the flexibility to create custom Data Loader Plugins.
See the following resources: