# storage.ini The `storage.ini` file configures the file storage buckets that Squirro uses to hold uploaded and generated files, such as project and community pictures, newsletter assets, converted documents, and files produced by AI agents. The file is located at `/etc/squirro/storage.ini`. Squirro ships a default set of bucket definitions there. Add or change only the sections your deployment needs. A bucket is a named storage location with its own backend, directory, and access rules. Each bucket is defined by a `[storage_]` section, where `` is the bucket name. The `storage_` prefix belongs to the section title only. When you reference a bucket from the API or the Python SDK, use the bare name, for example `project_pictures`. Files in a bucket are addressed by a storage URL in the form `storage:///`. For instructions on storing and retrieving files from these buckets with the Python SDK, see the [Upload and Manage Storage Files](../../../api/squirro_client/storage-files.md#sc-storage-files) page. > **Warning** > > A bucket that uses the `file` container and sets `url_prefix` is served over HTTP without authentication. Anyone who has the address of a stored file can download it, so the address is the only thing keeping that file private. Set a unique `hash_salt` on every such bucket in a production deployment so that stored file paths cannot be guessed, and do not use a served bucket for content that requires access control. A bucket that uses the `s3` container ignores `url_prefix` and builds a temporary address instead, described under [Keys for the s3 container](#keys-for-the-s3-container). ## Example Configuration A standard installation ships with the following bucket definitions: ```ini [storage] default_bucket = localfile [storage_localfile] container = file directory = /var/lib/squirro/storage/localfile url_prefix = /storage/localfile [storage_cluster] container = file directory = /mnt/gv0/storage url_prefix = /storage/cluster [storage_community_pictures] container = file directory = /var/lib/squirro/storage/community_pictures url_prefix = /storage/community_pictures enable_api = True [storage_community_type_pictures] container = file directory = /var/lib/squirro/storage/community_type_pictures url_prefix = /storage/community_type_pictures enable_api = True [storage_newsletter] container = file directory = /var/lib/squirro/storage/newsletter url_prefix = /storage/newsletter enable_api = True [storage_project_pictures] container = file directory = /var/lib/squirro/storage/project_pictures url_prefix = /storage/project_pictures enable_api = True [storage_datasource_pictures] container = file directory = /var/lib/squirro/storage/datasource_pictures url_prefix = /storage/datasource_pictures enable_api = True [storage_pdf_conversion] container = file directory = /var/lib/squirro/storage/pdf_conversion url_prefix = /storage/pdf_conversion [storage_genai] container = file directory = /var/lib/squirro/storage/genai url_prefix = /storage/genai enable_api = True preserve_filename = True ``` > **Note** > > Directories differ on a containerized installation, where the values come from environment variables. The `pdf_conversion` bucket also carries a `hook_pre_access` key, omitted above, which Squirro sets to an internal service address. Leave that key as shipped. To add a bucket of your own, append a section for it. The following example adds a bucket named `reports` that accepts uploads through the API and keeps the file names supplied by the caller: ```ini [storage_reports] container = file directory = /var/lib/squirro/storage/reports url_prefix = /storage/reports enable_api = True preserve_filename = True ``` ## Configuration Reference The reference below documents the sections and keys you set in `storage.ini`, along with the default value of each key. The file contains a single `[storage]` section and one `[storage_]` section per bucket. Add only the keys your deployment needs. The exception is `container`, which every bucket requires, because it selects the storage backend and therefore determines which of the remaining keys apply. An empty entry in the Default column means the key has no default value. ### [storage] Sets the storage defaults that apply across buckets. | Key | Description | Default | | --- | --- | --- | | `default_bucket` | Bucket used when a caller stores a file without naming a bucket. This key is required. A service that starts without it fails to start at all, in the same way as a bucket that omits `container`. | | The shipped configuration points this key at the `localfile` bucket: ```ini [storage] default_bucket = localfile ``` ### [storage_] Defines a single bucket. Repeat this section once per bucket, replacing `` with the bucket name. The section title carries the `storage_` prefix, but the bucket is addressed by its bare name. The following default section defines a bucket that the API and the SDK reference as `project_pictures`: ```ini [storage_project_pictures] container = file directory = /var/lib/squirro/storage/project_pictures url_prefix = /storage/project_pictures enable_api = True ``` #### Common keys These keys apply to every bucket, whatever its backend. | Key | Description | Default | | --- | --- | --- | | `container` | Storage backend for the bucket. Use `file` for a local or mounted directory, or `s3` for Amazon S3 compatible object storage. This key is required. | | | `enable_api` | When set to `True`, the bucket accepts file uploads and deletions through the topic API and the Python SDK. Set it to `False` to reject those requests cleanly, with a client error naming the bucket. A bucket that omits the key altogether, such as `localfile`, also refuses them, but fails with a server error instead, which gives the caller no indication that the bucket configuration is the cause. | | | `preserve_filename` | When set to `True`, a stored file keeps the file name supplied by the caller. Otherwise Squirro assigns a generated file name. | `False` | | `hash_salt` | Salt applied when Squirro builds the path of a stored file, so that stored paths cannot be guessed. Because a served bucket is not protected by authentication, this value is the main safeguard for the files it holds. Set a unique value in every production deployment, ideally at install time, since changing it later only affects files stored from that point on. | | | `hook_pre_access` | Address of a service that Squirro calls before a file in the bucket is read. Squirro sets this key on the buckets that need it, such as `pdf_conversion`. Leave it as shipped, and do not add it to a bucket of your own. | | The `genai` bucket is the one default that sets both flags. It accepts uploads through the API and keeps the file names it is given, so that files produced by AI agents stay recognizable: ```ini [storage_genai] container = file directory = /var/lib/squirro/storage/genai url_prefix = /storage/genai enable_api = True preserve_filename = True ``` #### Keys for the file container These keys apply when `container` is set to `file`. | Key | Description | Default | | --- | --- | --- | | `directory` | Directory that holds the files of the bucket. If you omit this key, Squirro uses a temporary directory. | | | `url_prefix` | Web path under which the directory is served, used to build the external download URL of a file. A standard installation serves the whole of `/var/lib/squirro/storage/` under `/storage/`, so a value of `/storage/` resolves only when `directory` is `/var/lib/squirro/storage/`. A bucket that sets this key is served without authentication. Omit it for a bucket whose files must not be downloadable by address alone. | | | `overwrite_files` | When set to `True`, a file whose content and file name both match a file already in the bucket replaces that file instead of being stored separately. Use it only when files are not shared between items, because deleting one item then removes a file that another item may still reference. When left at the default, each upload is stored as a separate entry with its own storage URL, even when the content is identical. | `False` | Every bucket in the default configuration uses the file container. The `localfile` bucket shows the minimum a file bucket needs: ```ini [storage_localfile] container = file directory = /var/lib/squirro/storage/localfile url_prefix = /storage/localfile ``` #### Keys for the s3 container These keys apply when `container` is set to `s3`. | Key | Description | Default | | --- | --- | --- | | `access_key` | Access key of the account used to reach the object storage. This key is required. | | | `secret_key` | Secret key of the account used to reach the object storage. This key is required. | | | `bucket` | Name of the S3 bucket that holds the files. This key is required. | | | `region_name` | Region of the S3 bucket. | | | `cache_directory` | Local directory used to cache files downloaded from the object storage. | | | `guestpass_ttl_secs` | How long a generated external download URL remains valid, in seconds. | `3600` | No bucket in the default configuration uses the s3 container, so there is no shipped extract to show. A bucket backed by object storage looks as follows: ```ini [storage_archive] container = s3 access_key = secret_key = bucket = squirro-archive region_name = eu-central-1 enable_api = True ``` Store the credentials securely rather than in plain text. For the available options, see the [Securing Configuration Files](../securing-config-files.md#admin-securing-config-files) page. > **Note** > > The `bucket` key names the bucket in the object storage service. That name is independent of the Squirro bucket name in the section title, and the two do not have to match. ## Default Buckets A standard Squirro installation defines the following buckets. The exact set depends on the deployment, so treat the list as a starting point and check `/etc/squirro/storage.ini` on the server for the buckets it actually defines. | Bucket | Purpose | Accepts API uploads | | --- | --- | --- | | `localfile` | General file storage. This is the default bucket. | No | | `cluster` | Shared file storage on a clustered installation. | No | | `community_pictures` | Images used by communities. | Yes | | `community_type_pictures` | Images used by community types. | Yes | | `newsletter` | Assets used in newsletter templates. | Yes | | `project_pictures` | Images used by projects. | Yes | | `datasource_pictures` | Images used by data sources. | Yes | | `pdf_conversion` | Documents produced by the PDF conversion step. | No | | `genai` | Files produced by AI agents, such as generated charts and data files. | Yes | > **Important** > > The default bucket, `localfile`, does not accept uploads through the API. To store files with the API or the Python SDK, target a bucket configured with `enable_api = True`, or add a bucket of your own. For instructions on storing files from the Python SDK, see the [Upload and Manage Storage Files](../../../api/squirro_client/storage-files.md#sc-storage-files) page. ## Add a Bucket To add a bucket: 1. Open `/etc/squirro/storage.ini` on the server. 2. Add a `[storage_]` section, replacing `` with the bucket name you want to use from the API. 3. Set `container`, and set `directory` for a file bucket or the account and bucket keys for an s3 bucket. To have the files of a file bucket downloadable by address, place `directory` under `/var/lib/squirro/storage/` and set `url_prefix` to the matching `/storage/` path. A directory elsewhere still stores files, but its `url_prefix` does not resolve. 4. Set `enable_api = True` if the bucket has to accept uploads and deletions through the API. Set it to `False` rather than omitting it if the bucket has to refuse them, so that a caller targeting it receives a clear rejection. 5. Set a unique `hash_salt` on a bucket that sets `url_prefix`, since the file path is the only thing keeping its files private. 6. Create the target directory for a file bucket, owned by `sqprovid:squirro` with mode 775, matching the directories Squirro ships. Several services write to a bucket under different accounts, including the topic API as `sqtopic` and the ingester as `sqingest`, and they reach it through the shared `squirro` group. 7. Restart the affected services, as described below. > **Important** > > On a clustered installation, point `directory` at shared storage that every node can reach, in the same way as the `cluster` bucket. A bucket backed by a directory local to one node makes a file stored through that node unavailable from the others, which surfaces as files that are found only intermittently. > > > > Shared storage generally sits outside `/var/lib/squirro/storage/`, so such a bucket cannot also be served by address through the standard `/storage/` path. The `cluster` bucket is served from its own dedicated rule. Omit `url_prefix` on a bucket of your own that is backed by shared storage, or ask Squirro to add a rule for it. For changes in `/etc/squirro/storage.ini` to take effect, restart the Squirro services: ```bash squirro_restart ``` Several services read the file, not only the topic and provider APIs, and each one loads the bucket definitions once at startup. A service that is left running cannot resolve a bucket added afterwards, so restarting only some of them produces a bucket that works through the API while the ingestion pipeline still fails on the same storage URL. The `squirro_restart` command covers every Squirro service installed on the node, which avoids that. On a distributed installation, run it on every node. For more information, see the [Services](../../operations/services.md#admin-services) page.