Accessing Data

Accessing Data#

The primary job of Squirro dashboards is to display data that have been loaded into Squirro in an intuitive form. This guide explains how you can change which data is fetched in a custom widget and explores the recommended best practices.

Collections and Models#

To make data available to a custom widget, it first needs to be loaded from a web-based resource, such as the Squirro API, or a third-party API. To load such data, the AJAX technology is used, specifically the browser XMLHttpRequest object or fetch() function. A common way of executing such requests is jQuery method $.ajax() and its derivatives such as $.get(). For example, the following code uses the Squirro API to run a query and return its results:

const query = this.search.get('query');

const project_id = this.project.id;
const queryEncoded = encodeURIComponent(query);

const url = '/v0/items/query?project_id=' + project_id + '&query=' + queryEncoded;

$.get(url)
    .done((data) => {
        console.log('data fetched', data);
    })
    .fail((err) => {
        console.log('could not fetch data', err);
    });

The main difficulty for newer JavaScript developers in working with AJAX is the asynchronous nature of the requests. All handling of success and error conditions need to be handled through callbacks. This can quickly lead to a complicated callback structure. One simplification is using Promises, (see e.g. jQuery Deferred Object).

Manually fetching the data using AJAX calls presents additional challenges. This approach requires you to correctly handle error conditions, manually refresh the data on query change, construct the correct API URLs, and parse the result. To simplify all of this, Squirro custom widgets rely on collections and models. Collections and models are a component that is provided by Backbone.js, the framework on which custom widgets are based, to handle all of these elements.

For working with Squirro items and aggregations, Squirro provides factories that make it easy to instantiate these collections, and handle all the API-knowledge for you. For example to execute a specific Squirro query using a collection, the following code would work:

const factory = Factories.Collections.Items;
const collection = factory.create(this.options.search, this.options.project);
collection.on('reset', () => {
    console.log('data fetched', collection.toJSON());
});
return collection;

The collection then exposes additional methods to work with this data, and importantly, each individual element is represented as a Model:

collection.on('reset', () => {
    const first_item = collection.at(0);
    console.log('title:', first_item.get("title"));
});

Using Collections in Widgets#

Squirro custom widgets provide built-in support for working with collections. By returning a valid collection from the getCollection() method, the custom widget base class takes care of fetching the collection, and calls the rendering methods once that has successfully finished. The following is a full widget that renders the number of items in a collection. As you can see, there is no overhead for fetching the collection, nor for waiting for the data to come back.

export default class Widget extends Widgets.Base {
    afterInitialize() {
        this.customWidgetTemplate = _.template(
            'Have <%- JSON.stringify(items.length) %> items'
        );
    }

    getCollection() {
        const factory = Factories.Collections.Items;
        return factory.create(this.options.search, this.options.project);
    }

    getCustomTemplateParams() {
        return { items: this.collection.toJSON() };
    }
}

Accessing Custom Endpoints#

To access a custom endpoint, a custom collection can be created using the BaseFactory(). For this example, you want to access the user list, that is exposed by Squirro at /v0/users. Start by creating a collection, that accesses the specific URL:

getCollection() {
    const factory = Factories.Collections.Base;
    const collection = factory.create({
        url: '/v0/users',
    });
    return collection;
}

This can be used in a custom widget like any other collection. For example, start with this simple HTML file and store it as widget.html:

<% _.each(items, function (user) { %>
    <div class="user">
        <%- user.fullName %> - <%- user.email %>
    </div>
<% }) %>

The custom widget then needs to be extended with a getCustomTemplateParams() method:

export default class Widget extends Widgets.Base {
    getCollection() {
        const factory = Factories.Collections.Base;
        const collection = factory.create({
            url: '/v0/users',
        });
        return collection;
    }

    getCustomTemplateParams() {
        return {
            items: this.collection.toJSON(),
        };
    }
}

This wires the collection together with the template, and will output a list of all users.

Accessing Data from Remote Servers#

The custom collection approach can also be used for data that is located on other servers. However due to the security restrictions of JavaScript, additional work is required to make this work. By default, JavaScript does not allow any requests to other servers. With CORS, a secure mechanism has been developed, that allows a server to grant exceptions to this blanket rule.

When using a cross-domain API, the collection can be instantiate in the same way as the prior example:

getCollection() {
    const factory = Factories.Collections.Base;
    const collection = factory.create({
        url: 'https://myapi.example.com/demo_resource',
    });
    return collection;
}

But on that API side, a number of HTTP headers need to be set, so that the widget is allowed to access the resource:

Access-Control-Allow-Origin: https://example.squirro.cloud
Access-Control-Request-Method: GET

How those headers are set is outside of the scope of this article. Please refer to the Mozilla web docs reference for Cross-Origin Resource Sharing (CORS) and enable-cors.org for more information.

Manipulating Data#

Sometimes the API does not return data in a format that can be used directly in the collection or the widget. In these cases, a number of custom methods can be added to the collection, to tailor the parsing of the data.

For example, the following response could be returned by an API:

{
    "results": [
        { "title": "Lorem ipsum", "created_at": 1607682249.9941251 },
        { "title": "dolor sit amet", "created_at": 1607599702.4165745 }
    ],
    "eof": false
}

This response can not be used directly by a collection, because collections expect the response to be the list. This is done with a one-line code block in a custom parse() method on the collection:

getCollection() {
    const factory = Factories.Collections.Base;
    const collection = factory.create({
        url: 'https://myapi.example.com/demo_resource',
    });
    collection.parse = function (data) {
        return data.results
    };
    return collection;
}

While we are putting in custom logic, we might want to convert the created_at time stamp into a date object:

collection.parse = function (data) {
    var results = data.results;
    _.forEach(results, function (result) {
        result.created_at = new Date(result.created_at * 1000);
    });
    return results
};

Sort the Collection#

Not all server responses are sorted already. In these cases, the collection can be adjusted to sort the data on the client side. One option is to implement this in the parse() method, as seen in the previous section. However Backbone collections provide an easier mechanism to do this, by implementing a comparator. That comparator function can return the sort value on which the individual items should be sorted. It can also be implemented as a traditional comparator function which compares two values. In both cases, the objects passed in are models, so the model.get() method needs to be used to access the values. For details, please refer to the Backbone.js reference for Collection.comparator.

For example if the API response should be sorted by title, a comparator can be implemented as follows:

getCollection() {
    const factory = Factories.Collections.Base;
    const collection = factory.create({
        url: 'https://myapi.example.com/demo_resource',
    });
    collection.comparator = function(item) {
        return item.get("title");
    };
    return collection;
}
Contents