WoodWing Help Center

Backing up and restoring the Elvis 6 index

 

Backing up and restoring the Elvis 6 index

It is recommended to regularly create a backup of the Elasticsearch data in Elvis 6 Server. This article describes how to create backups (referred to as 'snapshots') and how to restore them at a later stage.

Automated backups

A default Elvis 6 Server installation automatically creates daily Elasticsearch backups at 1 o'clock at night.

To change the moment when automatic backups are created, do the following:

Step 1. Change the following option:

  • File: cluster-config.properties.txt
  • Option: elasticsearch.backup.cron
  • Value: A cron-like expression as a list of 6 single space-separated fields representing second, minute, hour, day, month and weekday. Month and weekday names can be given as the first 3 letters of the English names. The expression is resolved against the server's local time zone. Leave the setting empty to disable automated backups.
  • Example:
  • Every day at 1 o'clock at night: 0 0 1 * * *
  • Every hour at 10 minutes past the whole hour: 0 10 * * * *

elasticsearch.backup.cron=0 0 1 * * *

Notes:

  • This feature is disabled when cluster.elasticsearch.external = true
  • Leave the option empty to disable automated backups

Step 2. Restart Elvis Server.

Automated backup cleanup

Snapshots will be automatically removed after 90 days. Segments that are no longer referenced by any snapshot are removed at that point too.

The following can be controlled:

  • The number of days after which a snapshot should be removed

Example: Delete a snapshot when it is older than 90 days.

  • The maximum number of snapshots to keep

Example: Keep no more than 30 snapshots at all times. When a new snapshot is created by which this maximum number is exceeded, the oldest snapshot is removed.

You can use each option individually without using the other option but it is recommended to combine both options. When doing so, the condition of both options need to be met in order for a snapshot to be deleted, thereby preventing that snapshots are inadvertently deleted especially when a backup fails for some reason.

Example: 

The options are set as follows:

  • Delete all snapshots older than 90 days
  • Keep a maximum number of 30 snapshots

In this scenario, a snapshot will only be deleted when it is older than 90 days AND more than 30 snapshots exist.

Controlling the number of days after which a snapshot should be removed

Step 1. Change the following option:

  • File: cluster-config.properties.txt
  • Option: elasticsearch.backup.rotateAgeInDays
  • Value: A number in days. Snapshots older than this number of days will be removed.
  • Example:

elasticsearch.backup.rotateAgeInDays=90

Note: To ignore this setting, set it to -1.

Step 2. Restart Elvis Server.

Controlling the maximum number of snapshots to keep

Step 1. Change the following option:

  • File: cluster-config.properties.txt
  • Option: elasticsearch.backup.rotateMaxToKeep
  • Value: Any number. When the number of available snapshots exceeds this number, the oldest snapshot is removed. Default value: the same number as specified in elasticsearch.backup.rotateAgeInDays.
  • Example:

elasticsearch.backup.rotateMaxToKeep=30

Note: To ignore this setting, set it to -1.

Step 2. Restart Elvis Server.

Creating a snapshot manually

Manually creating snapshots of the indices can be done by using the Elasticsearch API. The process involves setting up a repository and then manually creating the snapshot.

Step 1. Creating a snapshot repository

The repository is a folder that is shared between all nodes. It is used for storing all backup files: the indices together with additional information that is needed to restore the backups.

Best practice is to place the index snapshots within the Elvis Server/Elvis Shared Data folder.

It is advised to use a path that is the same on all nodes by setting the elasticsearch.backup.location setting:

  • File: cluster-config.properties.txt
  • Option: elasticsearch.backup.location
  • Value: A path to a folder that is shared between all nodes.
  • Example: The default setting defines a 'backups' folder in the shared data location (Elvis Shared Data):

elasticsearch.backup.location=${sharedDataLocation}/backups

A backup of the shared data will contain both files and an index.

Use the following commands:

Note: Here, a folder named "hotDataBackup" is created within the Elvis Shared Data folder. Modify the path if you want to create a repository elsewhere.

curl -XPUT ''http://localhost:9200/_snapshot/elvis-shared-data' -d '{

"type": "fs",

"settings": {

"location": "/Library/Elvis Server/Elvis Shared Data/backups"

}

}'

This will create the folder that is specified as the snapshot location and Elasticsearch will return with an acknowledge response.

To review information about the created repository use the following command:

curl -XGET 'http://localhost:9200/_snapshot/elvis-shared-data?pretty'

Step 2. Creating a snapshot

Use a unique name for the snapshot so that it can be easily identified, for example by including a date-timestamp in the file name.

Use the following command:

Note: Modify the path and date-timestamp as needed.

curl -XPUT 'localhost:9200/_snapshot/elvis-shared-data/2016-06-29t07_07_20.578z?wait_for_completion=true'

To list all snapshots in the repository, use the following command:

curl -XGET 'localhost:9200/_snapshot/elvis-shared-data/_all?pretty=true'

Note: Files in the Elvis Hot Data folder (such as the activation_state file) are not backed up.

Restoring a snapshot

Note: Elvis Server cannot be used while a backup is restored.

The following scenarios are described:

  • Restoring index files on the same cluster
  • Restoring index files on a new cluster or when replacing an existing cluster

Restoring index files on the same cluster

In this scenario, the index is restored without deleting the existing index.

Step 1. Use one of the following methods:

Step 2. Monitor the restore status through the following URL:

http://localhost:9200/_plugin/head/

  • Check the restore progress in the top bar (on the same row as Cluster Overview). Use auto refresh to check the progress if needed.
  • After the restore is finished, ensure that all indices have green shards (shards are represented as numbered boxes).
  • Ensure that all indices are assigned to the node-client.
  • Ensure that the cluster health is green (see the top-most bar).

Note: The restore process should just take a few minutes for a 500MB index file.

Step 3. Restart Elvis Server.

The process is now complete.

Restoring index files on a new cluster or when replacing an existing cluster

In this scenario, the current indices are first removed.

Step 1. Create a backup of the following Elvis Server folders:

  • Config
  • Elvis Hot Data

Step 2. (Optional, only when not using a clean installation) Clean the existing index by doing the following:

Step 2a. Stop Elvis Server.

Step 2b. Remove every node index from your cluster by removing all folders located in the following location:

/Elvis Server/Elvis Hot Data/esdata/your_elvis_cluster_name/nodes/

Step 2c. Start Elvis Server.

All indices will be regenerated but obviously without containing records for any assets.

Step 3. Use one of the following methods:

Step 4. Monitor the restore status through the following URL:

http://localhost:9200/_plugin/head/

  • Check the restore progress in the top bar (on the same row as Cluster Overview). Use auto refresh to check the progress if needed.
  • After the restore is finished, ensure that all indices have green shards (shards are represented as numbered boxes).
  • Ensure that all indices are assigned to the node-client.
  • Ensure that the cluster health is green (see the top-most bar).

Note: The restore process should just take a few minutes for a 500MB index file.

Step 5. Restart Elvis Server.

The process is now complete.

Deleting a snapshot

Important:

  • Always remove snapshots using the Elasticsearch API this way; it ensures that any file that is necessary for restoring other snapshots is left in place.
  • Because the backups are incremental, manually removing snapshots may corrupt other snapshots.

Step 1. Locate the snapshot to delete by listing all snapshots that are in the repository. Do this by using the following command:

curl -XGET 'localhost:9200/_snapshot/elvis-shared-data/_all?pretty=true'

Step 2. Delete the snapshot by using the following command:

Note: Modify the repository path as needed.

curl -XDELETE 'localhost:9200/_snapshot/elvis-shared-data/2016-06-29t07_07_20.578z'

Path references

The following is an overview of the path references

  • Elasticsearch snapshot repository: localhost:9200/_snapshot/elvis-shared-data
  • Elasticsearch backup location: <Elvis Shared Data>/backups/esbackup
  • Elasticsearch snapshot name: 2016-06-29t07_07_20.578z

Rescuing lost index entries

After restoring a backed-up index, the assets in the file store and the references to them in the index might not be synchronized. This is especially the case when only a slightly older version of the index is restored instead of restoring a matching backup of the file store.

To validate if a repair is necessary and to optionally perform the repair, the File Store Rescue Tool can be used. For more information, see Rescuing lost index entries for assets in the File Store folder of Elvis 6 Server.

Additional information

For more information about Elasticsearch backup, additional settings and how to monitor processes see the Elasticsearch documentation.

Document history

  • 4 September 2017: Updated commands in ‘Step 1. Creating a snapshot repository’.
Was this article helpful?
0 out of 0 found this helpful / Created: / Updated:
Have more questions? Submit a request

0 Comments

Please sign in to leave a comment.