Upload and Manage Storage Files

Upload and Manage Storage Files#

Squirro stores uploaded files, such as pictures, newsletter assets, and documents, in named storage locations called buckets. You can upload, download, and delete these files with the SquirroClient Python SDK.

Each stored file is identified by a storage URL in the form storage://<bucket>/<path>. Pass that URL to other Squirro APIs whenever they expect a stored file, and use it to download or delete the file later.

This page explains how to store and retrieve files with the SDK. For the bucket configuration itself, see the storage.ini page.

Prerequisites#

Before you start, confirm that you have:

  • Python installed and a SquirroClient environment set up. For installation instructions, see the SquirroClient Installation page.

  • A token for the target tenant, belonging to an account with the user or admin role. An account with the reader role, shown as Restricted in the Squirro UI, cannot store, read, or delete files through these methods. To generate a token, see the Connecting to Squirro page.

  • A bucket that accepts uploads through the API. See the next section.

Choose a Bucket#

Every upload targets a bucket, and the bucket has to be configured with enable_api = True before it accepts files through the API. Buckets without that setting reject uploads and deletions, and the request fails with an error.

Important

The default bucket, localfile, does not accept uploads through the API. Passing it as the bucket argument fails. Target a bucket configured with enable_api = True instead.

Adding enable_api = True to localfile is possible, but Squirro does not recommend it. That bucket is the general write target for internal processing, so opening it to the API allows any account with the user role to change or remove files that other parts of the platform rely on. Use a bucket dedicated to your own content instead.

Only a bucket configured with an explicit enable_api = False produces a clean rejection, raising ClientError. A bucket name that does not exist, and a bucket that carries no enable_api setting at all, both fail as a server error instead, raising UnknownError. A misspelled bucket name therefore looks like a platform failure rather than a bad argument, so check the name for a typo first and confirm the bucket configuration with a Squirro administrator.

A standard installation provides several buckets that accept API uploads, including project_pictures, community_pictures, community_type_pictures, datasource_pictures, and newsletter. Each one is intended for the content its name describes.

Warning

The newsletter bucket is not a general purpose bucket. Storing a file in it also writes the new storage_url into the squirro_newsletter email template, replacing the value already there, and deleting the file removes that value again. That write is not reported back: the call returns successfully whether the template was updated or not, so an upload that appears to have worked may have left the template unchanged. Use a different bucket for content that is not a newsletter asset.

If none of the available buckets fits your use case, ask a Squirro administrator to add one. For the full list of default buckets and instructions on adding a bucket, see the storage.ini page.

Upload a File#

Use the new_storage_file() method in your Python script to store the contents of a file. Provide the bucket name and the file content as bytes.

Note

An upload is limited to 96 MB in a standard installation. To raise that limit, see the File Upload Size Limits page. Only the client_max_body_size setting described there governs this path. The frontend.max-file-size setting on the same page applies to uploads made through the Squirro user interface, so raising it alone does not change what these methods accept.

Uploads are also not deduplicated: storing the same content a second time creates a second entry under a new storage_url, so retrying a call that timed out can leave two entries to clean up.

from squirro_client import SquirroClient

client = SquirroClient("<client_id>", "<client_secret>", cluster="<cluster_url>", tenant="<tenant>")
client.authenticate(refresh_token="<refresh_token>")

with open("project-logo.png", "rb") as handle:
    data = handle.read()

result = client.new_storage_file(bucket="project_pictures", data=data)

storage_url = result["storage_url"]
external_url = result["external_url"]

The call returns a dictionary:

{
  "storage_url": "storage://project_pictures/c4/36/a99c856defd26454/c436a99c856defd264541883519ad6a8eead80a6",
  "external_url": "/storage/project_pictures/c4/36/a99c856defd26454/c436a99c856defd264541883519ad6a8eead80a6"
}

The dictionary has two entries:

  • storage_url

    Reference to the stored file, in the form storage://project_pictures/<path>. Use that value with the other methods on this page, and wherever a Squirro API expects a stored file.

  • external_url

    Address where the file can be downloaded. The value depends on the container setting of the bucket, described on the storage.ini page. It is None for a bucket that does not serve its files externally.

    For a bucket held in a local or mounted directory, the value is usually a path relative to the cluster host rather than a complete URL, so prepend the host when you need an absolute address. That address is not protected by authentication. Anyone who has it can download the file, so treat it as confidential and do not use a served bucket for content that requires access control.

    For a bucket held in S3 compatible object storage, the value is a complete presigned URL that grants access without authentication until it expires. The lifetime comes from the guestpass_ttl_secs setting of the bucket and is one hour by default, so the value stops working after that and is not suitable for long-term storage. Keep the storage_url instead, and treat the presigned URL as confidential for as long as it is valid.

To store a file that is already on disk without reading it yourself, use new_storage_file_from_name() and pass the path:

result = client.new_storage_file_from_name(bucket="project_pictures", filename="project-logo.png")

That method reads the file for you, but it does not carry the file name through to the bucket. To control the stored file name, use new_storage_file() as described in the next section.

Control the Stored File Name#

By default, Squirro assigns a generated file name to a stored file, so the name you upload is not the name under which the file is kept. That behavior keeps file names unique within a bucket.

To keep your own file names, ask a Squirro administrator to set preserve_filename = True on the bucket, then pass the name you want through the filename argument:

result = client.new_storage_file(bucket="reports", data=data, filename="quarterly-report.pdf")

The returned storage_url ends with the name you passed:

storage://reports/00/fd/bd01d3ee1d72f18c/quarterly-report.pdf

On a bucket without preserve_filename = True, the filename argument has no effect. The same call stores the file under a generated name instead:

storage://reports/00/fd/bd01d3ee1d72f18c/00fdbd01d3ee1d72f18cbc8b972fab81c7e25d67

The newsletter bucket ignores preserve_filename and always prefixes the stored file name with a generated value. A file uploaded with a filename argument is stored as <generated>_<filename>, so the name you pass is kept and prefixed. A file uploaded without that argument is stored under the generated value alone.

Note

new_storage_file_from_name() does not set the stored file name from the path you give it. On a bucket with preserve_filename = True, use new_storage_file() with an explicit filename argument instead.

If you need the stored file name to be predictable, confirm the bucket configuration with a Squirro administrator before you rely on it. For the setting itself, see the storage.ini page. In every case, use the returned storage_url to refer to the file rather than building the URL yourself.

Download a File#

Use the get_storage_file() method in your Python script to retrieve the content of a stored file. The method takes the storage URL and returns the raw bytes.

content = client.get_storage_file(storage_url)

with open("downloaded-logo.png", "wb") as handle:
    handle.write(content)

The method returns the file content as bytes, not text, which is why the target file is opened in binary mode. Printing the type and the first bytes of a PNG file gives:

<class 'bytes'>
b'\x89PNG\r\n\x1a\n\x00\x00\x00\rIHDR'

Downloads are not restricted by the enable_api setting, so you can read a file from any bucket. That setting governs uploads and deletions only.

Delete a File#

Use the delete_storage_file() method in your Python script to remove a stored file. The method takes the storage URL.

result = client.delete_storage_file(storage_url)

The call returns an empty dictionary, because a successful deletion is confirmed by the status code rather than by a response body:

{}

Treat the absence of an error as confirmation of the deletion. Deleting a file that is not present raises a not-found error instead:

squirro_client.exceptions.NotFoundError: (404, 'Not Found')

Deletion requires the same enable_api = True setting as uploading, and fails the same way when the bucket name does not exist or carries no enable_api setting.

Deleting a file from the newsletter bucket also removes its storage_url from the squirro_newsletter email template, as described in the Choose a Bucket section.

For the generated method reference, see the APIs by Topic page.


Contents