Follow

Running Elvis 6 in the Amazon Web Services (AWS) cloud

Comments

4 comments

  • Avatar
    Siebrand Mazeland

    If one is to create a snapshot with name "elvis-shared-data" and a settings location or type that differs from what is set initially, will Elvis still create snapshots at the scheduled time simply using that identifier?

  • Avatar
    Nico Krijnen

    Hi Sybrand. Looking at the code, the Elvis server will delete and add the snapshot repository before each backup, to ensure that it uses the settings as configured in the Elvis server. For example in case you change the backup location. This operation does not affect the contents of the snapshot repository, but it does force the "elvis-shared-data" to be an 'fs' type repository. So your proposed trick won't work.

  • Avatar
    Siebrand Mazeland

    Nico, do you think a feature request makes sense? Currently we use EFS as a shared file system, and the only reason we need it, is for the ES snapshot repo. We currently schedule an " s3 sync --delete" to get the snapshot somewhere sane. With a 14:1 cost ratio per GB/month for EFS:S3, it's very much worth being able to write directly to an S3 based snapshot repo.

  • Avatar
    Vincent Bergervoet

    Hello Siebrand. The S3 backup plugin for Elasticsearch has an issue, making it not really the ideal solution. The incremental backup tries to read all existing backup snapshot files before it makes a new backup. This causes it to become slower and slower at performing incremental backups as the backup get's larger. 

    So either you have to delete old snapshots, or make full backups every time.

    We use the last method for doing hourly backups on Swivle. But you will definitely want to add some monitoring and keep an eye on the backup duration and put some alarms on it when it takes too long.

    Including the monitoring, it might make more sense if you set up Elasticsearch S3 backup yourself.
    You may still file a feature request for it.

Please sign in to leave a comment.