Using Amazon S3 as a storage engine for Elvis 5
Info: This feature requires Elvis 5.1.1 or higher.
When running Elvis 5 within Amazon Web Services (AWS) it is possible to use S3 as a storage engine.
Note: This method replaces the shared-volume storage engine. Knowledge about setting up Amazon EC2 and S3 is required. This article briefly describes the steps needed to create such a setup, but will not go in-depth regarding AWS specifics.
When using Amazon S3 as a storage engine for Elvis, please take note of the following:
- When running the complete Elvis installation on AWS, Amazon S3 is the recommended storage solution. As well as being cost-effective, it ensures availability and proper performance.
- When running the Elvis installation in a non-Amazon environment (such as a local office or a data center), consider the network speed between the Elvis installation and Amazon S3 and how this relates to the speed of uploading and downloading assets:
- A slow connection to AWS is suitable only for storing archived files that are accessed on occasion
- A fast connection allows storing highly requested production files
- When using Amazon S3 as a storage engine, be aware that the feature that automatically creates an Elasticsearch backup by default references sharedDataLocation in its setting elasticsearch.backup.location. If the Search nodes will not have a file system based shared data location, the backup should be disabled.
Setup and configuration
Setting up Amazon S3 for use with Elvis 5 is a 3-step process:
- Setting up AWS
- Installing Elvis 5
- Configuring Elvis
1. Setting up AWS
Step 1. Create a VPC (or re-use an existing one).
See the AWS documentation: Getting Started with Amazon VPC.
Step 2. Create a subnet within the created VPC (or re-use an existing one).
See the AWS documentation: Subnets in Your VPC.
Step 3. (Optional, only when using multiple nodes) Set up a load balancer (ELB).
See the AWS documentation: Getting Started With Elastic Load Balancing.
Note: Make sure that sticky sessions are turned ON.
Step 4. Set up an S3 bucket for use by Elvis.
See the AWS documentation: Getting Started with Amazon Simple Storage Service.
Step 5. Set up the security between nodes. Several ways of doing this exist, make sure that the following ports are open to Elvis and that TCP is chosen as the protocol:
- 5701 - 5710
- 9300 - 9301
- 80 or 443, depending on the use of https
Step 6. Create the first EC2 instance.
Note: The same hardware requirements apply as for on-premise installations. The following operating systems are supported:
2. Installing Elvis 5
The installation process is the same as for an on-premises installation. See Installing Elvis Server 5.x.
3. Configuring Elvis 5
On each node, configure specific Elvis S3 options. We recommend setting up a node completely and creating an AMI to speed up configuration of additional nodes.
Step 1. Configure the node-config.properties.txt file:
- cluster.join.tcpip.members. Fill in all internal IP’s of the nodes so that you do not have to use the .aws. settings.
- sharedDataLocation. Use fast _local_ storage.
- fileStoreType. Set it to local, otherwise it will wait until shared-date is mounted.
# Fill in all internal IP’s of the nodes, so you don’t have to use the .aws. settings cluster.join.tcpip.members= # Just used for S3 temp file storage, use fast _local_ storage sharedDataLocation= # Set it to local, otherwise it will wait until shared-date is mounted fileStoreType=local
Step 2. Configure the cluster-config.properties.txt file:
- storage.engine.type. Set to S3.
- storage.engine.s3.bucket. The name of your created elvis bucket.
- storage.engine.s3.endpoint. The S3 endpoint. For example: s3-eu-west-1.amazonaws.com.
- storage.engine.s3.accessKeyId. The S3 access key ID.
- storage.engine.s3.secretKey. The S3 secret key.
storage.engine.type=S3 # The name of your created elvis bucket storage.engine.s3.bucket= # S3 endpoint e.g. s3-eu-west-1.amazonaws.com storage.engine.s3.endpoint= # The s3 credentials storage.engine.s3.accessKeyId= storage.engine.s3.secretKey=
- 15 February 2017: Added a note to section 'Configuration considerations' about using Amazon S3 as a storage engine.
- 22 December 2016: Added note to section 1. 'Setting up AWS', opening ports about using TCP as the protocol to use.