Using Amazon S3 as a storage engine for Elvis 5

Info: This feature requires Elvis 5.1.1 or higher.

When running Elvis 5 within Amazon Web Services (AWS) it is possible to use S3 as a storage engine.

Note: This method replaces the shared-volume storage engine. Knowledge about setting up Amazon EC2 and S3 is required. This article briefly describes the steps needed to create such a setup, but will not go in-depth regarding AWS specifics.

Configuration considerations

When using Amazon S3 as a storage engine for Elvis, please take note of the following:

• When running the complete Elvis installation on AWS, Amazon S3 is the recommended storage solution. As well as being cost-effective, it ensures availability and proper performance.

• When running Elvis together with a WoodWing Enterprise integration and both are set up on AWS, please also view Integrating Elvis 5 in Enterprise Server 10 on AWS - Load balancer information.

• When running the Elvis installation in a non-Amazon environment (such as a local office or a data center), consider the network speed between the Elvis installation and Amazon S3 and how this relates to the speed of uploading and downloading assets:
• A slow connection to AWS is suitable only for storing archived files that are accessed on occasion
• A fast connection allows storing highly requested production files

• When using Amazon S3 as a storage engine, be aware that the feature that automatically creates an Elasticsearch backup by default references sharedDataLocation in its setting elasticsearch.backup.location. If the Search nodes will not have a file system based shared data location, the backup should be disabled.

Setup and configuration

Setting up Amazon S3 for use with Elvis 5 is a 3-step process:

1. Setting up AWS
2. Installing Elvis 5
3. Configuring Elvis

1. Setting up AWS

Step 1. Create a VPC (or re-use an existing one).

See the AWS documentation: Getting Started with Amazon VPC.

Step 2. Create a subnet within the created VPC (or re-use an existing one).

See the AWS documentation: Subnets in Your VPC.

Step 3. (Optional, only when using multiple nodes) Set up a load balancer (ELB).

See the AWS documentation: Getting Started With Elastic Load Balancing.

Note: Make sure that sticky sessions are turned ON.

Step 4. Set up an S3 bucket for use by Elvis.

See the AWS documentation: Getting Started with Amazon Simple Storage Service.

Step 5. Set up the security between nodes. Several ways of doing this exist, make sure that the following ports are open to Elvis and that TCP is chosen as the protocol:

• 5701 - 5710
• 9300 - 9301
• 80 or 443, depending on the use of https

Step 6. Create the first EC2 instance.

2. Installing Elvis 5

The installation process is the same as for an on-premises installation. See Installing Elvis Server 5.x.

3. Configuring Elvis 5

On each node, configure specific Elvis S3 options. We recommend setting up a node completely and creating an AMI to speed up configuration of additional nodes.

Step 1. Configure the node-config.properties.txt file:

• cluster.join.tcpip.members. Fill in all internal IP’s of the nodes so that you do not have to use the .aws. settings.
• sharedDataLocation. Use fast _local_ storage.
• fileStoreType. Set it to local, otherwise it will wait until shared-date is mounted.

# Fill in all internal IP’s of the nodes, so you don’t have to use the .aws. settings
cluster.join.tcpip.members=

# Just used for S3 temp file storage, use fast _local_ storage
sharedDataLocation=

# Set it to local, otherwise it will wait until shared-date is mounted
fileStoreType=local

Step 2. Configure the cluster-config.properties.txt file:

• storage.engine.type. Set to S3.
• storage.engine.s3.bucket. The name of your created elvis bucket.
• storage.engine.s3.endpoint. The S3 endpoint. For example: s3-eu-west-1.amazonaws.com.
• storage.engine.s3.accessKeyId. The S3 access key ID.
• storage.engine.s3.secretKey. The S3 secret key.

storage.engine.type=S3

# The name of your created elvis bucket
storage.engine.s3.bucket=

# S3 endpoint e.g. s3-eu-west-1.amazonaws.com
storage.engine.s3.endpoint=

# The s3 credentials
storage.engine.s3.accessKeyId=
storage.engine.s3.secretKey=

Document history

• 12 November 2018; Updated section 'Configuration considerations' with information about load balancers.