Automate the Backup of Elasticsearch

Automate the Backup of Elasticsearch#

This page describes how to configure and automate the backup of Elasticsearch. Creating regular backups (called snapshots) is a critical part of any deployment, production or not.

The snapshot system of Elasticsearch is very efficient as its incremental and hence doing frequent backups is not a problem. Even on a development environment, you can protect against accidental data loss or human error by doing frequent backups.

Prerequisites#

The squirro-ansible role will do the heavy lifting for you, but you need to do some preparation work, as there are many different snapshot repositories types and each comes with its own requirements.

Always refer to the official Elasticsearch Snapshot/Restore documentation for the most up to date information.

We will cover an S3 repository and a shared filesystem repository.

S3 Repository#

See the official Elasticsearch S3 repository plugin documentation for the most up-to-date information.

You will need to create an S3 repository and make it available to your AWS EC2 instance using IAM EC2 roles. If your instances are not on AWS, then you need to use API keys.

Once the bucket is ready, you can configure the repository in Elasticsearch like so:

es_curl -XPUT https://localhost:9200/_snapshot/my_repository -H 'Content-Type: application/json' -d '
{
    "type": "s3",
    "settings": {
        "bucket": "my_bucket",
        "region": "us-east-1",
        "path": "my_backup_folder"
    }
}'

In this URL my_repository is the name of the repository, which you will need to refer to later in the configuration again.

Its highly recommended that you don’t use the root of the bucket, but instead also set a path. This will allow you to use the same bucket for multiple repositories. This is especially important during migrations.

This is a one-time operation, and needs to be done only once for the entire cluster, e.g. on the master node.

Shared Filesystem Repository#

See the official ElasticsearchShared file system repository documentation for the most up-to-date information.

You need a shared file system like NFS that is mounted on each instance, in the same location. The location must be writeable by the Linux user elasticsearch, which consequently should also have the same uid on every node in the cluster.

The folder needs to to be configured in the elasticsearch.yml file. You can do this by adding the following variable in your Ansible playbook:

elasticsearch_repo_paths: ["/mnt/my_share/my_repository"]

Apply the playbook once on all instances. This will restart the Elasticsearch service.

Once this is done and the files system is ready, you can configure the repository in Elasticsearch like so:

es_curl -XPUT  https://localhost:9200/_snapshot/my_repository -H 'Content-Type: application/json' -d '
{
    "type": "fs",
    "settings": {
        "location": "/mnt/my_share/my_repository"
    }
}'

In this URL my_repository is the name of the repository, which you will need to refer to later in the configuration again.

This is a one-time operation, and needs only to be done once for the entire cluster, e.g. on the master node.

Its highly recommended that you don’t use the root of the shared file system, but instead create a folder for each repository. This will allow you to use the same files ystem for multiple repositories. This is especially important during migrations.

Configure the Backup#

Once the repository is configured, you can configure the backups.

To enable backups, you need to set the following variable in your Ansible playbook:

elasticsearch_snapshot_repository_name: "my_repository"

The name must match the name of the repository you configured in the previous steps.

By default hourly backups are done every hour on the hour from 1-23 hours. Forty-eight of these backups are kept in the repository. The default configuration also creates daily backups every day at 00:00. Thirty of these backups are kept in the repository.

This results in hourly protection for the last two days and daily protection for the last thirty days.

You can change all of these settings.

For example to back up twice per hour during work hours and keep ninety-six of these backups around, the use the following settings:

elasticsearch_hourly_snapshot_cron_minute: "0,30"
elasticsearch_hourly_snapshot_cron_hour: "8-18"
elasticsearch_hourly_snapshot_cron_day: "*"
elasticsearch_hourly_snapshot_cron_month: "*"
elasticsearch_hourly_snapshot_cron_weekday: "*"
elasticsearch_hourly_snapshots_to_keep: 96

elasticsearch_daily_snapshot_cron_minute: "0"
elasticsearch_daily_snapshot_cron_hour: "0"
elasticsearch_daily_snapshot_cron_day: "*"
elasticsearch_daily_snapshot_cron_month: "*"
elasticsearch_daily_snapshot_cron_weekday: "*"
elasticsearch_daily_snapshots_to_keep: 30

Ensure that no two backups are happening at the same time. Also consider that a snapshot can take a long time if you have a lot of index pressure and a slow repository.

Restore a single index#

The squirro-ansible role also provides a helper tool to restore a single index from a snapshot. This is not documented yet.

In the meantime consult the official Elasticsearch Restore a snapshot documentation on how to restore data.

Restore / Bootstrap an entire cluster#

The squirro-ansible role includes a helper tool to restore an entire cluster safely and automatically. This is useful in scenarios where you want to downscale a deployment to save cost.

This is not documented, yet. Reach out to Squirro Support in the meantime for more information if you need it.

Contents