Crawler Connector

This page explains how to configure the Crawler connector.

Project creators use this connector to crawl websites or sitemaps and index their content within their Squirro project.

When to Use the Crawler Connector

The Crawler connector is designed for websites that you own or control, such as your own corporate website, documentation portal, or knowledge base. Crawling arbitrary third-party websites on the public internet is not a supported use case. Many websites block automated crawling, and crawling a website without permission can breach its terms of use. Before you configure a crawl, make sure that you own the target website or have explicit permission to crawl it.

Within that scope, the connector is best suited to websites that have many pages, content that changes over time, or both. It indexes each page as a separate item with its own source link, and a scheduled crawl keeps the indexed content current as the site changes.

For a single, static document, the Documents connector is usually the better choice: upload the file, such as a PDF, directly instead of crawling. Crawling adds value when the content spans many pages, so that each page becomes a separate and precisely searchable item, or when the content is updated regularly and you want those updates indexed automatically.

How to Crawl a Website

Before you start, make sure that you have a Squirro project and can access its Setup space.

1. Open your Squirro project.

2. Navigate to the Setup space.

3. Click the Data tab. By default, you land on the Data Sources page.

4. Click Configure A New Data Source in the top-right corner of the page.

5. In the dialog that opens, select the Crawler connector, then click Configure.

6. Complete the four configuration steps described in the sections below, then load the data into your project.

The configuration wizard guides you through four steps: Configure source, Map to item fields, Map to labels, and Schedule job.

Step 1 - Configure Source

On the Configure source step, set the website or sitemap to crawl and, optionally, the advanced crawl settings.

For most websites the default configuration works well: enter the website URL, keep every other setting at its default, and add the data source. Consider adjusting the configuration only in the following cases:

• For a large or frequently updated site, crawl a sitemap by entering a URL that contains .xml. A sitemap crawl is faster, more complete, and updates incrementally. See the Crawl Strategies section below.

• To limit the crawl to part of a site, use the include path and exclude path options instead of crawling the entire domain.

• If the crawl is blocked or returns errors, increase the delay between requests, or lower the number of concurrent requests, to reduce the load on the website server.

• To improve the quality of the indexed text, use the XPath to exclude content option to remove repeated elements such as navigation menus and footers.

Basic Options

• Title

  Enter the title of the data source as it appears in your project. This field is required. Choose a descriptive title, because it is how you identify the data source in the list of data sources. With only a few data sources, any title is easy to find, but as the list grows, a clear and specific title makes the right entry much quicker to spot.

• Description

  Enter a description for the data source. This field is required and is prefilled with a default description. Replace the default with a description that explains what the data source covers and how it is configured, for example which part of the site it crawls and on what schedule. A well-crafted description helps you and your colleagues understand the purpose and setup of the data source when you return to the list of data sources later, without having to open it and review each setting.

• Website or sitemap URL

  Enter the URL of the website or sitemap to crawl, starting with https://, for example, https://www.squirro.com. To crawl a sitemap, enter a URL that contains .xml. For more information about sitemap crawling, see the Crawl Strategies section below. If the URL points directly to a document file that ends in .pdf, the connector downloads and indexes that single file instead of crawling.

• Priority

  Select the priority that Squirro uses when it processes and indexes the data fetched by this source. Squirro processes higher-priority data first, so when several sources load at the same time, a source set to High is indexed ahead of sources set to Normal or Low. This setting affects the processing and indexing order, not the speed of the crawl itself. Choose Low, Normal, or High. The default is Normal. Use a higher priority for time-sensitive sources, and a lower priority for large background loads that should not delay other sources.

Advanced Options

Click Advanced Options to expand the following settings. All of these settings are optional.

• Keep source formatting

  Select this checkbox to retrieve the data in its source HTML format, preserving elements such as images and links. This option is turned off by default.

• Include path

  Enter part of a URL so that the crawler only follows and indexes pages whose URL contains that text, for example, /news/ or /blog/. The match is a substring match anywhere in the URL, so /news/ also includes nested pages such as /news/switzerland/, as well as any other URL that contains /news/. Click Add Include Path to add more than one path. A page qualifies if its URL matches any of the include paths.

• Exclude path

  Enter part of a URL so that the crawler skips pages whose URL contains that text, for example, /archive/. The match works the same way as the include path. Exclude paths take precedence: if a URL matches both an include path and an exclude path, the page is skipped. Click Add Exclude Path to add more than one path.

The include and exclude paths are applied while the crawler discovers links, before it follows them, so pages that do not qualify are never fetched and do not add to the crawl time. This makes the include and exclude paths an efficient way to scope a large crawl.

• XPath to exclude content

  Enter an XPath expression to remove specific parts of each page before its text is indexed. XPath is a query language for selecting elements in an HTML page, usually by tag name and attributes such as class or id. Every element that matches the expression, together with its content, is removed from the page, so it does not appear in the indexed text.

  Use this option to strip repeated or irrelevant elements that add noise to search results, such as cookie banners, promotional boxes, or sidebars. For example:

  • //div[@class='footer'] removes every <div> element whose class is footer.

  • //div[@id='cookie-banner'] removes the element whose id is cookie-banner.

  • //aside removes every <aside> element.

  The connector already removes common boilerplate automatically, such as navigation menus, headers, and footers, as described in the How Content Is Extracted section below. Use this option for site-specific elements that are not removed automatically. Click Add XPath to Exclude Content to add more than one expression.

• Number of concurrent requests

  Enter the number of concurrent requests sent to the site. To speed up a crawl of a large site, increase this number, though a higher number places additional load on the website server. The default is 4. Increase it with care: too many concurrent requests can overload the target server, cause the website to rate limit or block the crawler (for example, with HTTP 429 or HTTP 403 responses), and in some cases lead to the crawler IP address being banned. Aggressive crawling can also breach the terms of use of the site. Raise this number only for servers that can handle the additional load and that you are permitted to crawl. See the Crawling Responsibly section below.

• Delay between requests

  Enter the delay, in whole seconds, between successive requests to the same site. A delay spreads requests out over time, which lowers the request rate and places a fairer, lighter load on the website server. The default is 0. Increasing the delay is the main way to crawl politely: it reduces the chance that the website rate limits or blocks the crawler (for example, with HTTP 429 or HTTP 403 responses) or bans the crawler IP address, and it helps you stay within the terms of use of the site. Increase the delay for sensitive servers, or when a crawl is being throttled or blocked. The delay and the number of concurrent requests work together to control the overall load. See the Crawling Responsibly section below.

• Depth limit

  Enter the maximum number of link hops the crawler follows from the start URL. The start page is at depth 0, the pages it links to are at depth 1, the pages those pages link to are at depth 2, and so on. For example, a depth limit of 1 crawls the start page and the pages directly linked from it, but not pages further away. The default is 0, which means no limit: the crawler follows links as far as they lead and crawls the entire reachable site, within the same domain and any include and exclude path rules. Set a depth limit to keep a crawl shallow or to bound the size of a crawl on a very large site, and combine it with the include and exclude paths for finer control.

• User agent

  Enter the user agent that the crawler sends with each request. The user agent is a short text string that identifies the client making the request, and websites can use it to decide what content to return or whether to allow the request. The default value identifies the crawler as a standard web browser, so that sites which serve content normally to browsers respond as expected. Squirro recommends keeping the default. Change it only for advanced use cases, for example when the website serves different content for a specific user agent, or when the site owner has allowlisted a particular user agent for your crawler. Do not use the user agent to disguise the crawler in order to bypass access restrictions, as this can breach the terms of use of the site.

• HTTP user credential

  Enter the user name for HTTP authentication. Use this setting for sites that are protected by HTTP basic authentication.

• HTTP password credential

  Enter the password for HTTP authentication.

• Apply include and exclude rules to PDF files

  By default, the include path and exclude path rules apply only to web pages, not to the document files that the crawler finds linked on those pages, so every discovered file is downloaded and indexed. Take an include path of /reports/ as an example. A page under /reports/ might link to two files, /reports/annual-2026.pdf and /downloads/brochure.pdf. By default, both files are downloaded, because the include path is not applied to file URLs. When you select this checkbox, the include path and exclude path rules are applied to the file URLs as well, so only files whose URL matches the rules are downloaded. In the example, /reports/annual-2026.pdf is downloaded because its URL contains /reports/, while /downloads/brochure.pdf is skipped. Use this option when you want the file downloads to follow the same scope as the crawled pages. This option is turned off by default.

• Remove query parameters

  Select this checkbox to remove query parameters from URLs before the crawler follows them, so that variant URLs that differ only by query parameters are not crawled more than once. For example, https://example.com/articles?ref=newsletter&sort=desc is reduced to https://example.com/articles, so the same article list is not crawled several times under different tracking or sorting parameters. The pagination parameters page, limit, and offset are kept, so that paginated pages are still followed. For example, https://example.com/articles?page=2 is left unchanged. This option is turned on by default.

• Crawl content starting from a specific date or all content

  Set the starting point for a sitemap crawl. This setting applies only to the first run and only to sitemap URLs that contain .xml. Enter all to crawl all content, or enter a date in YYYY-MM-DD format to crawl only the content published on or after that date. The default is all. For more information, see the Crawl Strategies section below.

Crawl Strategies

The Crawler connector supports two crawl strategies, selected automatically based on the value you enter in the Website or sitemap URL field.

Standard Crawl Compared to Sitemap Crawl

• Standard crawl

  When the URL points to a web page, the connector starts from that page and discovers further pages by following the links it finds, up to the configured depth limit. Use a standard crawl when the site has no sitemap, or when you want to crawl only a part of the site by combining the depth limit with include path and exclude path rules.

• Sitemap crawl

  When the URL contains .xml, the connector treats the URL as a sitemap and crawls the URLs listed in that sitemap rather than following links. Use a sitemap crawl when the site publishes a sitemap, because it is usually faster and more complete: the connector retrieves the full list of URLs directly, instead of relying on links being reachable from the start page. A sitemap index file, which is a sitemap that lists other sitemaps, is also supported: the connector follows the nested sitemaps automatically. A sitemap crawl also supports incremental updates, so it keeps the index current efficiently on later runs.

Choosing a Crawl Strategy

To decide which strategy to use for a given website, follow these steps:

1. Check whether the site publishes a sitemap. Many sites expose one at /sitemap.xml. The most reliable way to find it is the site robots.txt file: by the robots exclusion standard, robots.txt is always located at the root of the domain, for example https://www.example.com/robots.txt. That file commonly lists the sitemap locations in one or more Sitemap: directives. Open it in a browser and look for Sitemap: lines.

2. If a sitemap exists, confirm that it covers the content you want to index and that its entries include last-modified dates. A sitemap that meets both conditions is the preferred option, because it is faster, more complete, and updates incrementally.

3. If the site has no sitemap, or the sitemap does not cover the content you need, or it omits last-modified dates, use a standard crawl instead. Scope the crawl to the relevant content with the include path and exclude path options.

What Happens on Subsequent Runs

After the first crawl, the connector behaves differently on each later run depending on the crawl strategy.

• Standard crawl

  The connector crawls the whole site again from the start URL on every run. Each page is fetched again, but pages whose content has not changed since the previous run are not indexed a second time. As a result, unchanged pages are re-fetched but not reprocessed, while new and changed pages are indexed.

• Sitemap crawl

  The connector crawls incrementally and does not re-fetch the whole site. It fetches only the URLs whose last-modified date is on or after the last successful run.

• Document files

  Linked document files, such as PDF and Word files, are identified by their URL. Once a file has been indexed, the connector does not download or index it again on later runs, even if the file content changes at the same URL. To re-index updated document files, reset the data source.

Note

Pages that are removed from the website after they have been indexed are not removed from Squirro automatically. Those pages remain in the project until you remove them manually or reset the data source. Resetting the data source clears the record of previously indexed pages and files, so the next run re-fetches and re-indexes all content from the start.

Crawling Responsibly

The connector includes several built-in measures to limit the load it places on a website and to reduce the chance of being rate limited or blocked: it limits concurrency to at most four requests at a time by default, supports a configurable delay between requests, and identifies itself with a standard browser user agent. You can tune these through the Number of concurrent requests, Delay between requests, and User agent options described in the Advanced Options section above.

If a crawl is rate limited or blocked, for example the logs show repeated HTTP 403 or HTTP 429 responses, increase the delay between requests and reduce the number of concurrent requests, then run the data source again.

Important

The connector does not automatically apply the rules in a website robots.txt file. Before you crawl a website, make sure that you are permitted to crawl it and that you comply with the terms of use of the site. Crawl only websites that you own or have permission to crawl.

How Content Is Extracted

For each HTML page, the connector automatically removes common boilerplate, such as navigation menus, headers, footers, and scripts, and extracts the main text content, so that the indexed item holds the relevant content. The alternative text of images is included as part of the text.

The automatic extraction works well for most pages. Keep the following points in mind:

• The automatic cleanup handles standard page structures. A site that builds its navigation, sidebar, or footer from non-standard elements may not be cleaned completely. Use the XPath to exclude content option to remove the remaining elements.

• On pages with a complex or fragmented layout, the extracted text can be incomplete. If the indexed text is missing content, inspect the page structure and adjust the XPath to exclude content option.

• A page is indexed only when both a title and body text are extracted. Pages with no extractable text are skipped.

To refine the result, use the following options:

• To remove specific repeated elements that are not caught automatically, such as a cookie banner, use the XPath to exclude content option.

• To keep the original HTML formatting instead of the extracted text, turn on the Keep source formatting option.

The connector also detects links to document files on the crawled pages and downloads and indexes those files. The supported file types are PDF, Word (.doc and .docx), Excel (.xls and .xlsx), and PowerPoint (.ppt and .pptx). Only files hosted on the same domain as the crawled page are downloaded. Files linked from another domain, such as a separate content delivery network, are skipped. Because these are binary documents, the data source must use a pipeline workflow that processes binary documents, such as the default Binary Documents workflow selected on the Schedule job step.

Considerations

Keep the following points in mind when you plan a crawl:

• JavaScript rendering

  The connector indexes the HTML returned by the server and does not execute JavaScript. Content that a page loads dynamically through JavaScript after the initial response is not captured. For a site that renders its content with JavaScript, crawl a sitemap that links directly to the content pages, or crawl a server-rendered version of the site.

• Authentication

  For sites protected by HTTP basic authentication, enter your credentials in the HTTP user credential and HTTP password credential options, and the connector authenticates each request automatically. For other authentication methods, such as form-based login, single sign-on, or a custom request header or API token, contact the Squirro Support website to discuss your requirements.

• Network access

  The crawler must be able to reach the target website over the network, and the connector configuration does not include a proxy option. To crawl through a corporate proxy, or from an environment with restricted outbound access, the proxy must be configured at the infrastructure level. For help with that setup, visit the Squirro Support website and submit a technical support request.

• Request timeout

  Each request has a timeout of 60 seconds. Pages that do not respond within that time are skipped.

• Number of pages

  The crawl options do not include a maximum-pages setting. To bound a large crawl, use the depth limit, include path, and exclude path options to keep it focused on the relevant content, and use the maximum number of items per run on the Schedule job step to cap how many items are ingested in a single run.

• Domain scope

  A standard crawl follows links only within the same domain as the start URL, including its subdomains. For some domains, particularly those with a multi-part suffix such as .gov.uk or .co.uk, the crawl scope can be broader than the individual site. In that case, use the include path option to scope the crawl to the content you want.

• Item identity

  Each crawled page becomes one item, identified by its URL. If the same URL is reached by more than one source, or by the same source over time, it maps to the same item rather than creating a duplicate.

• Access control

  Crawled content has no per-page access control. Every indexed page is visible to anyone with access to the project. Do not crawl content that requires access restrictions in Squirro.

Step 2 - Map to Item Fields

On the Map to item fields step, map the source fields produced by the crawler to Squirro item fields. The crawler produces the following source fields:

• url

  The page URL.

• title

  The page title, taken from the HTML title element.

• content

  The main text extracted from the page.

• created_at

  The publication date of the page. The connector detects the date only from structured page metadata, such as JSON-LD or meta tags. If a page does not expose a date in its metadata, this field is empty, even when a date is visible in the page text. Map this field to the item date so that the indexed items can be sorted and filtered by date. Mapping the date is particularly useful for news, press releases, and other time-sensitive content.

To continue, map at least the Title or Body field. For more information about mapping source fields, see the How to Load Data Using the UI page.

Note

The preview shown on this step displays the raw text of the start page and does not reflect the final indexed content. The indexed text has the boilerplate elements removed, as described in the How Content Is Extracted section above, so it differs from the preview.

Step 3 - Map to Labels

On the Map to labels step, optionally map source fields to labels to enrich the indexed items with structured metadata. For more information about labels, see the Labels page.

Step 4 - Schedule Job

On the Schedule job step, configure the ingestion batch size, the incremental import behavior, the schedule, and the pipeline workflow. Because the connector can index document files as binary documents, keep a pipeline workflow that processes binary documents, such as the default Binary Documents workflow. Then click Load Data to start the crawl.

After the crawl starts, monitor its progress on the Data Sources page, where each data source shows its status, the number of items fetched, and any errors from the most recent run.

For more information about scheduling and loading data, see the How to Load Data Using the UI page.

Configuring the Connector via the API

You can also create a Crawler source programmatically instead of using the UI, with the SquirroClient new_source method. For the method and its parameters, see the new_source() API reference. For a complete walkthrough of creating a source with a data loader plugin, see the Data Loader Templates page.

To select the Crawler connector, set "plugin_name": "crawler_plugin" in dataloader_options and pass the crawler arguments in dataloader_plugin_options, for example:

config = {
    "dataloader_options": {"plugin_name": "crawler_plugin"},
    "dataloader_plugin_options": {
        "website_url": "https://www.ecb.europa.eu/press/",
        "include_path": ["/press/"],
        "depth_limit": 0,
    },
}

The keys in dataloader_plugin_options are the internal argument names, which differ from the names used for the configuration options on the Configure source step. For a description of each option, see the Step 1 - Configure Source section above and match it by the configuration option name.

Argument	Configuration option
website_url	Website or sitemap URL
keep_source_formatting	Keep source formatting
include_path	Include path
exclude_path	Exclude path
exclude_content_xpath	XPath to exclude content
concurrent_requests	Number of concurrent requests
download_delay	Delay between requests
depth_limit	Depth limit
user_agent	User agent
http_user	HTTP user credential
http_pass	HTTP password credential
apply_include_exclude_rules_to_pdf_files	Apply include and exclude rules to PDF files
suppressing_query_parameters	Remove query parameters
sitemap_crawler_mode	Crawl content starting from a specific date or all content

For more information about the SquirroClient source methods, see the APIs by Topic page.

Troubleshooting

If a crawl does not return the content you expect, check the following common situations:

• The crawl returns no content or very few pages.

  The site probably renders its content with JavaScript, which the connector does not execute. Confirm that the pages serve their content as HTML, or crawl a sitemap that links directly to the content pages. See the Considerations section above.

• The indexed text is incomplete or contains unwanted elements.

  On pages with a complex or fragmented layout, or where a large unwanted element is not removed automatically, the extracted text can be incomplete or noisy. Use the XPath to exclude content option to remove the unwanted elements. See the How Content Is Extracted section above.

• A notification appears stating that the website might be blocking the crawler.

  When you configure the source and continue to the preview step, the connector checks that the start URL is reachable. If the website returns an error status, such as HTTP 429 or HTTP 403, a notification appears briefly stating that the website might be blocking the crawler, together with the status code returned. Check that the URL is correct and reachable, that any HTTP credentials are valid, and that you are permitted to crawl the website. If the website is rate limiting the crawler, increase the delay between requests and reduce the number of concurrent requests, then run the data source again.

• The crawl is blocked, or the logs show repeated HTTP 403 or HTTP 429 responses.

  The website is rate limiting or blocking the crawler. Increase the delay between requests and reduce the number of concurrent requests, then run the data source again. See the Crawling Responsibly section above.

• Only part of the site is crawled.

  The include path or exclude path rules may be too restrictive, the depth limit may be too low, or the maximum number of items per run may have been reached. Review these settings.

• Pages behind a login are not crawled.

  The connector supports HTTP basic authentication only. It cannot crawl content behind a form-based login, cookies, or single sign-on.

• Document files are indexed without their text.

  Make sure the data source uses a pipeline workflow that processes binary documents, such as the default Binary Documents workflow.

• A sitemap crawl indexes nothing on later runs.

  The sitemap entries may have no last-modified dates, so the connector cannot detect which pages changed. Crawl the site with a standard crawl instead.

For more information, see the Data Loading Troubleshooting page or visit the Squirro Support website and submit a technical support request.