storage.ini

storage.ini#

The storage.ini file configures the file storage buckets that Squirro uses to hold uploaded and generated files, such as project and community pictures, newsletter assets, converted documents, and files produced by AI agents.

The file is located at /etc/squirro/storage.ini. Squirro ships a default set of bucket definitions there. Add or change only the sections your deployment needs.

A bucket is a named storage location with its own backend, directory, and access rules. Each bucket is defined by a [storage_<name>] section, where <name> is the bucket name. The storage_ prefix belongs to the section title only. When you reference a bucket from the API or the Python SDK, use the bare name, for example project_pictures.

Files in a bucket are addressed by a storage URL in the form storage://<bucket>/<path>.

For instructions on storing and retrieving files from these buckets with the Python SDK, see the Upload and Manage Storage Files page.

Warning

A bucket that uses the file container and sets url_prefix is served over HTTP without authentication. Anyone who has the address of a stored file can download it, so the address is the only thing keeping that file private. Set a unique hash_salt on every such bucket in a production deployment so that stored file paths cannot be guessed, and do not use a served bucket for content that requires access control. A bucket that uses the s3 container ignores url_prefix and builds a temporary address instead, described under Keys for the s3 container.

Example Configuration#

A standard installation ships with the following bucket definitions:

[storage]
default_bucket = localfile

[storage_localfile]
container = file
directory = /var/lib/squirro/storage/localfile
url_prefix = /storage/localfile

[storage_cluster]
container = file
directory = /mnt/gv0/storage
url_prefix = /storage/cluster

[storage_community_pictures]
container = file
directory = /var/lib/squirro/storage/community_pictures
url_prefix = /storage/community_pictures
enable_api = True

[storage_community_type_pictures]
container = file
directory = /var/lib/squirro/storage/community_type_pictures
url_prefix = /storage/community_type_pictures
enable_api = True

[storage_newsletter]
container = file
directory = /var/lib/squirro/storage/newsletter
url_prefix = /storage/newsletter
enable_api = True

[storage_project_pictures]
container = file
directory = /var/lib/squirro/storage/project_pictures
url_prefix = /storage/project_pictures
enable_api = True

[storage_datasource_pictures]
container = file
directory = /var/lib/squirro/storage/datasource_pictures
url_prefix = /storage/datasource_pictures
enable_api = True

[storage_pdf_conversion]
container = file
directory = /var/lib/squirro/storage/pdf_conversion
url_prefix = /storage/pdf_conversion

[storage_genai]
container = file
directory = /var/lib/squirro/storage/genai
url_prefix = /storage/genai
enable_api = True
preserve_filename = True

Note

Directories differ on a containerized installation, where the values come from environment variables. The pdf_conversion bucket also carries a hook_pre_access key, omitted above, which Squirro sets to an internal service address. Leave that key as shipped.

To add a bucket of your own, append a section for it. The following example adds a bucket named reports that accepts uploads through the API and keeps the file names supplied by the caller:

[storage_reports]
container = file
directory = /var/lib/squirro/storage/reports
url_prefix = /storage/reports
enable_api = True
preserve_filename = True

Configuration Reference#

The reference below documents the sections and keys you set in storage.ini, along with the default value of each key. The file contains a single [storage] section and one [storage_<name>] section per bucket.

Add only the keys your deployment needs. The exception is container, which every bucket requires, because it selects the storage backend and therefore determines which of the remaining keys apply. An empty entry in the Default column means the key has no default value.

[storage]#

Sets the storage defaults that apply across buckets.

Key	Description	Default
`default_bucket`	Bucket used when a caller stores a file without naming a bucket. This key is required. A service that starts without it fails to start at all, in the same way as a bucket that omits `container`.

The shipped configuration points this key at the localfile bucket:

[storage]
default_bucket = localfile

[storage_<name>]#

Defines a single bucket. Repeat this section once per bucket, replacing <name> with the bucket name.

The section title carries the storage_ prefix, but the bucket is addressed by its bare name. The following default section defines a bucket that the API and the SDK reference as project_pictures:

[storage_project_pictures]
container = file
directory = /var/lib/squirro/storage/project_pictures
url_prefix = /storage/project_pictures
enable_api = True

Common keys#

These keys apply to every bucket, whatever its backend.

Key	Description	Default
`container`	Storage backend for the bucket. Use `file` for a local or mounted directory, or `s3` for Amazon S3 compatible object storage. This key is required.
`enable_api`	When set to `True`, the bucket accepts file uploads and deletions through the topic API and the Python SDK. Set it to `False` to reject those requests cleanly, with a client error naming the bucket. A bucket that omits the key altogether, such as `localfile`, also refuses them, but fails with a server error instead, which gives the caller no indication that the bucket configuration is the cause.
`preserve_filename`	When set to `True`, a stored file keeps the file name supplied by the caller. Otherwise Squirro assigns a generated file name.	`False`
`hash_salt`	Salt applied when Squirro builds the path of a stored file, so that stored paths cannot be guessed. Because a served bucket is not protected by authentication, this value is the main safeguard for the files it holds. Set a unique value in every production deployment, ideally at install time, since changing it later only affects files stored from that point on.
`hook_pre_access`	Address of a service that Squirro calls before a file in the bucket is read. Squirro sets this key on the buckets that need it, such as `pdf_conversion`. Leave it as shipped, and do not add it to a bucket of your own.

The genai bucket is the one default that sets both flags. It accepts uploads through the API and keeps the file names it is given, so that files produced by AI agents stay recognizable:

[storage_genai]
container = file
directory = /var/lib/squirro/storage/genai
url_prefix = /storage/genai
enable_api = True
preserve_filename = True

Keys for the file container#

These keys apply when container is set to file.

Key	Description	Default
`directory`	Directory that holds the files of the bucket. If you omit this key, Squirro uses a temporary directory.
`url_prefix`	Web path under which the directory is served, used to build the external download URL of a file. A standard installation serves the whole of `/var/lib/squirro/storage/` under `/storage/`, so a value of `/storage/<name>` resolves only when `directory` is `/var/lib/squirro/storage/<name>`. A bucket that sets this key is served without authentication. Omit it for a bucket whose files must not be downloadable by address alone.
`overwrite_files`	When set to `True`, a file whose content and file name both match a file already in the bucket replaces that file instead of being stored separately. Use it only when files are not shared between items, because deleting one item then removes a file that another item may still reference. When left at the default, each upload is stored as a separate entry with its own storage URL, even when the content is identical.	`False`

Every bucket in the default configuration uses the file container. The localfile bucket shows the minimum a file bucket needs:

[storage_localfile]
container = file
directory = /var/lib/squirro/storage/localfile
url_prefix = /storage/localfile

Keys for the s3 container#

These keys apply when container is set to s3.

Key	Description	Default
`access_key`	Access key of the account used to reach the object storage. This key is required.
`secret_key`	Secret key of the account used to reach the object storage. This key is required.
`bucket`	Name of the S3 bucket that holds the files. This key is required.
`region_name`	Region of the S3 bucket.
`cache_directory`	Local directory used to cache files downloaded from the object storage.
`guestpass_ttl_secs`	How long a generated external download URL remains valid, in seconds.	`3600`

No bucket in the default configuration uses the s3 container, so there is no shipped extract to show. A bucket backed by object storage looks as follows:

[storage_archive]
container = s3
access_key = <access_key>
secret_key = <secret_key>
bucket = squirro-archive
region_name = eu-central-1
enable_api = True

Store the credentials securely rather than in plain text. For the available options, see the Securing Configuration Files page.

Note

The bucket key names the bucket in the object storage service. That name is independent of the Squirro bucket name in the section title, and the two do not have to match.

Default Buckets#

A standard Squirro installation defines the following buckets. The exact set depends on the deployment, so treat the list as a starting point and check /etc/squirro/storage.ini on the server for the buckets it actually defines.

Bucket	Purpose	Accepts API uploads
`localfile`	General file storage. This is the default bucket.	No
`cluster`	Shared file storage on a clustered installation.	No
`community_pictures`	Images used by communities.	Yes
`community_type_pictures`	Images used by community types.	Yes
`newsletter`	Assets used in newsletter templates.	Yes
`project_pictures`	Images used by projects.	Yes
`datasource_pictures`	Images used by data sources.	Yes
`pdf_conversion`	Documents produced by the PDF conversion step.	No
`genai`	Files produced by AI agents, such as generated charts and data files.	Yes

Important

The default bucket, localfile, does not accept uploads through the API. To store files with the API or the Python SDK, target a bucket configured with enable_api = True, or add a bucket of your own. For instructions on storing files from the Python SDK, see the Upload and Manage Storage Files page.

Add a Bucket#

To add a bucket:

Open /etc/squirro/storage.ini on the server.
Add a [storage_<name>] section, replacing <name> with the bucket name you want to use from the API.
Set container, and set directory for a file bucket or the account and bucket keys for an s3 bucket. To have the files of a file bucket downloadable by address, place directory under /var/lib/squirro/storage/ and set url_prefix to the matching /storage/<name> path. A directory elsewhere still stores files, but its url_prefix does not resolve.
Set enable_api = True if the bucket has to accept uploads and deletions through the API. Set it to False rather than omitting it if the bucket has to refuse them, so that a caller targeting it receives a clear rejection.
Set a unique hash_salt on a bucket that sets url_prefix, since the file path is the only thing keeping its files private.
Create the target directory for a file bucket, owned by sqprovid:squirro with mode 775, matching the directories Squirro ships. Several services write to a bucket under different accounts, including the topic API as sqtopic and the ingester as sqingest, and they reach it through the shared squirro group.
Restart the affected services, as described below.

Important

On a clustered installation, point directory at shared storage that every node can reach, in the same way as the cluster bucket. A bucket backed by a directory local to one node makes a file stored through that node unavailable from the others, which surfaces as files that are found only intermittently.

Shared storage generally sits outside /var/lib/squirro/storage/, so such a bucket cannot also be served by address through the standard /storage/ path. The cluster bucket is served from its own dedicated rule. Omit url_prefix on a bucket of your own that is backed by shared storage, or ask Squirro to add a rule for it.

For changes in /etc/squirro/storage.ini to take effect, restart the Squirro services:

squirro_restart

Several services read the file, not only the topic and provider APIs, and each one loads the bucket definitions once at startup. A service that is left running cannot resolve a bucket added afterwards, so restarting only some of them produces a bucket that works through the API while the ingestion pipeline still fails on the same storage URL. The squirro_restart command covers every Squirro service installed on the node, which avoids that. On a distributed installation, run it on every node. For more information, see the Services page.