convert.ini

convert.ini#

The convert.ini config file, located at /etc/squirro/convert.ini configures the content extraction (see Content Extraction).

Memory Limit#

Changed in version 3.17.4: This section applies to Squirro 3.17.4 and later, which ships a new version of the Tika server.

Content extraction aborts a document with the SQ-05105 processing error when the document exceeds one of two limits of the Tika server. See Processing Errors to learn more. The two limits are:

  • The memory available to parse a single document, 1 GB by default.

  • The maximum size of the content and metadata extracted from a single document, 256 MB by default.

On installations that run the tika service of the squirro-tika-server package, both limits are set in the /etc/tika-config.json file. To support larger documents, raise them there and restart the tika service:

  • For the parsing memory, raise the -Xmx value in pipes.forkedJvmArgs.

  • For the extraction size, raise pipes.maxIpcPayloadBytes, in bytes. The Tika server also holds each extraction result in memory, so raise its own -Xmx value as well. Run systemctl edit tika and, under [Service], add an empty ExecStart= line followed by the complete ExecStart line from systemctl cat tika, with the new -Xmx value. After each Squirro upgrade, compare the drop-in with the new ExecStart line, because the drop-in replaces it.

Added in version 3.6.9: By default, in Squirro 3.6.9 and later, Squirro uses an external dedicated Tika service. As such, auto-spawner settings do not apply.

If there is no Tika already running under the http://localhost:9998 address, the Squirro Ingester service auto-spawns the Tika server, with the default limits above. The vmargs option in the apache-tika section sets the memory of the auto-spawned Tika server, which must hold the largest extraction result. When you change vmargs, keep -XX:+ExitOnOutOfMemoryError, so that a Tika server that runs out of memory is replaced.

Key

Usage

Default

Example

vmargs

Java VM parameters. See Oracle’s Java HotSpot VM Options reference for the options.

-Xmx1g,-Xms64M,-XX:+ExitOnOutOfMemoryError

 
[apache-tika]
# comma-separated list of additional Java Virtual Machine command-line options
# to use
vmargs = -Xmx1g,-Xms64M,-XX:+ExitOnOutOfMemoryError

Using External Tika Service#

The Tika web service can be configured to run separately for more fine-grained control.

Added in version 3.6.9: Releases Squirro 3.6.9 and later use the dedicated Tika service by default.

Key

Usage

Default

Example

tika-url

Point to a Tika server, http://localhost:9998 is the default.

 
[apache-tika]
tika-url = http://localhost:9998

Changed in version 3.17.4: An external Tika server must run the Tika version that ships with your Squirro release. Earlier Tika versions do not provide the API that the Squirro Ingester uses.

The squirro-tika-server package provides the tika systemd service, which reads its configuration from the /etc/tika-config.json file. For all available options, see the Apache Tika documentation website.

Note that changing the default configuration is not recommended, as it must be compatible with the Squirro Ingester and required configurations may change in the future.

Contents
This page
View as Markdown