Multi-tiered storage in Assets Server

Info: Performing some of the steps described in this article requires direct server access. Depending upon how your system is hosted and the level of access you have to that system, coordination may be required with your Partner or WoodWing Support team. For a full overview of the steps that need to be done by WoodWing and how to request them, see WoodWing Cloud - Change management.

With the high number of assets that companies use nowadays, considerations need to be made about where all these assets should be stored. Assets that are used in production obviously need to be made available on fast disks while assets that are not used anymore can be moved to slower, less expensive disks to keep them available for reference and possible later use.

High-end storage solutions such as AWS Storage Gateway or ceph usually provide functionality to spread data over different disks transparently within the storage hardware.

Using such solutions is preferred because it makes it possible for Assets Server to address its storage as a single volume. This means that Assets Server can just move files around without having to copy actual data, an operation that is time consuming to perform. In these setups, the storage solution will move data between online storage and near-line storage in the background without affecting application performance.

Multiple volumes

If your storage hardware does not support the above, Assets Server makes it possible for you to spread the "Elvis Data" over multiple volumes. This can be used in the following cases:

• Store live production data on fast disks and store rarely accessed archive data on slower, less expensive disks (near-line storage), even if the storage hardware does not provide this by itself.
• When limitations on volume sizes exist. These can either be related to the storage hardware, the operating system or a virtualization layer. By spreading the data over multiple smaller volumes you can still store all your data on one Assets Server.

Before you start

Before implementing a multi-tier environment, please take note of the following:

• When implementing a multi-tier environment in an existing Assets Server installation, be aware that existing assets will not be moved and might therefore become irretrievable. Ensure therefore that your new storage tiers do not match on existing folders, or alternatively move your existing assets / folders manually beforehand.

• It is currently not supported to change the paths after they have been initially configured and files are stored in that location. Doing so results in bad behavior because the existing files will not have moved but are still placed in the previously defined location.

• Assets Server will try to move files where possible for optimal performance. Because files cannot be moved between volumes, a copy and delete action is used to spread data over multiple volumes. This dramatically impacts performance since a copy action is very slow compared to a move operation. Note that the tempFilestore (used for newly uploaded files) is always in the standard dataLocation, so make sure to keep critical production files on the default location for best import performance.

How it works

In the Assets Server multi-volume environment, assets are stored to a storage volume by a Storage Engine.

The multi-tier Storage Engine is configured as a top-level storage engine, with a default engine and typically one or more additional engines. It will delegate storage to its children, depending on defined patterns. This will be explained in more detail below.

The Storage Engine itself is of type 'multi-tier' while all following Storage Engines are of type 'shared-volume' or 's3'.

storage.engine {
  type = "multi-tier"
  default-engine {
    type = "shared-volume"
    ...
  }
  engines : [
    {
      name = "fast-storage"
      type = "shared-volume"
      ...
    }
    {
      name = "PDFs"
      type = "shared-volume"
      ...
    }
    {
      name = "archive"
      type = "s3"
      ...
    }
  ]
}

One of the main properties of each sub-engine (with the exception of the default engine) is the 'pattern' property. With it you can control which assets to store on the volume, for example all assets uploaded to a particular directory.

Patterns and order of definition

When folder paths are matched by multiple patterns, the order of storage engine definitions plays an important part. Precedence goes from top to bottom, with the default engine being an exception. In the following example for instance, the patterns will be matched in the following order.

1. pattern = "/fast-storage/**"
2. pattern = "/**/pdf/**"
3. pattern = "/archive/**"

storage.engine {
  type = "multi-tier"
  default-engine {
    type = "shared-volume"
    ...
  }
  engines : [
    {
      name = "fast-storage"
      type = "shared-volume"
      pattern = "/fast-storage/**"
      ...
    }
    {
      name = "PDFs"
      type = "shared-volume"
      pattern = "/**/pdf/**"
      ...
    }
    {
      name = "archive"
      type = "s3"
      pattern = "/archive/**"
      ...
    }
  ]
}

This means that files uploaded to 'fast-storage/pdf' will be written to the location defined for 'fast-storage' because the 'fast-storage' pattern sits higher in the order than the 'PDFs' pattern. Similarly, files uploaded to 'archive/pdf' will be written to the storage location defined for 'PDFs' because the 'PDFs' pattern sits above the archive pattern in the order of definition.

Non-matching patterns

If none of the patterns can be matched, the assets are stored in the location that is defined for the default Storage Engine.

Multiple engine patterns

Multiple patterns can be defined for the same engine. Each pattern should be separated by a comma.

    {
      name = "fast-storage"
      type = "shared-volume"
      pattern = "/fast-storage/**,**/pdf/**"
      ...
    }

Example

In the following example (taken from the example configuration file that is provided with Assets Server), the following is defined:

• All assets that do not match any of the patterns of the sub-engines are stored in a default shared volume (see lines 3 to 7)
• All assets uploaded to the "/fast-storage/**" directory are stored on a fast storage shared volume (see lines 12 to 18). This includes files uploaded to a 'pdf' subdirectory.
• All files in a 'pdf' subdirectory are uploaded to a separate shared volume (see lines 21 to 27) even when they are uploaded to, for example, 'archive/pdf', but not when they are uploaded to, for example, 'fast-storage/pdf'.
• All assets uploaded to the "/archive/**" directory are stored on Amazon S3 (see lines 30 to 34), with the exception of files uploaded to 'pdf' sub directories, such as '/archive/pdf', which are always stored on the 'PDFs' volume.

storage.engine {
  type = "multi-tier"
  default-engine {
    type = "shared-volume"
    assetFilestore = "/Library/Elvis Server/fs/default/assets"
    renditionFilestore = "/Library/Elvis Server/fs/default/renditions"
    tempFilestore = "/Library/Elvis Server/fs/default/tmp"
    mounted = false
  }
  engines : [
    {
      name = "fast-storage"
      type = "shared-volume"
      pattern = "/fast-storage/**"
      assetFilestore = "/Library/Elvis Server/fs/SSD1/assets"
      renditionFilestore = "/Library/Elvis Server/fs/SSD1/renditions"
      tempFilestore = "/Library/Elvis Server/fs/SSD1/tempFilestore"
      mounted = true
    }
    {
      name = "PDFs"
      type = "shared-volume"
      pattern = "/**/pdf/**"
      assetFilestore = "/Library/Elvis Server/fs/PDF/assets"
      renditionFilestore = "/Library/Elvis Server/fs/PDF/renditions"
      tempFilestore = "/Library/Elvis Server/fs/PDF/tempFilestore"
      mounted = true
    }
    {
      name = "archive"
      type = "s3"
      pattern = "/archive/**"
      region = "eu-west-1"
      bucket = "elvis-archive"
    }
  ]
}

Configuration

Configuring multi-tiered storage is done in the storage-engine-config.conf file (located in the default Assets Server/Config folder).

Define any of the engines and their patterns as described below and save the file.

Config syntax

The configuration of the file is specified in HOCON, a superset of JSON that is more human oriented and lenient. For more information about the syntax, see the HOCON specification.

Adding additional engines

Adding additional engines is a typical real-world scenario: you might start off with one or two main storage locations and decide to add an additional location at some later point — for example for archive purposes — and subsequently move files to that location.

To add an additional engine to your configuration, add another block ("{ }") to your list ("[ ]") of engines.

Copy the code

engines : [
    ...
    {
      ...
    }
    ...]

Note: The order in which engines are added matters to the outcome, see Patterns and order of definition.

Default Storage Engine

The default Storage Engine differs from the other Storage Engines in the following ways:

• It is defined outside the list of sub-engines.
• No name is needed.
• No pattern is needed: it is the default engine when no match is found for all other engine patterns.

Copy the code

default-engine {
   type = "shared-volume"
   assetFilestore = "/Library/Elvis Server/fs/default/assets"
   renditionFilestore = "/Library/Elvis Server/fs/default/renditions"
   tempFilestore = "/Library/Elvis Server/fs/default/tmp"
   mounted = false
 }

Shared volume Storage Engine

Step 1. Add a descriptive name, define the type as "shared-volume" and set "mounted" to "true".

Setting "mounted = true" will cause Assets Server to check whether the volume is actually mounted during startup and it will stop the Server if this check fails.

Note: For a local volume, set "mounted" to "false". Only do this for a standalone (single-node) installation, and only if there should really be no mounted volume in this location.

Copy the code

{
  name = "fast-storage"
  type = "shared-volume"
  ...
  mounted = true
}

Step 2. Define the locations on this mounted share where the assets (assetFilestore), their renditions (renditionFilestore) and the temporary files (tempFilestore) are stored.

Copy the code

{
  ...
  assetFilestore = "/Library/Elvis Server/fs/SSD1/assets"
  renditionFilestore = "/Library/Elvis Server/fs/SSD1/renditions"
  tempFilestore = "/Library/Elvis Server/fs/SSD1/tempFilestore"
  ...
}

Note: On Windows, use '/' or '\\' instead of '\'.

Tip: You can also use your Assets Server config properties here, such as sharedDataLocation.

Copy the code

{
  ...
  assetFilestore = "${sharedDataLocation}/assets"
  renditionFilestore = "${renditionFilestoreLocation}"
  tempFilestore = "${tempFilestoreLocation}"
  ...
}

Step 3. Provide a pattern to match the folderPath of your assets on. Only the matched assets will go to this storage engine.

The example below matches all your files that you store in sub directories called 'pdf' (for example: "/foo/bar/pdf").

Copy the code

{
  ...
  pattern = "/**/pdf/**"
  ...
}

S3 Storage Engine

Note: Although the use of S3 storage is fully supported, please be aware of possible performance issues such as long upload and download times.

Step 1. Provide a descriptive name and define the type as "s3" .

Copy the code

{
  name = "archive"
  type = "s3"
  ...
}

Step 2. Define the following:

Copy the code

{
  ...
  region = "eu-west-1"
  bucket = "elvis-archive"
  ...
}

Step 3. Provide a pattern to match the folderPath of your assets on. Only the matched assets will go to this storage engine.

The example below matches all files that you store in the "/archive" folder in Assets Server.

Copy the code

{
  ...
  pattern = "/archive/**"
  ...
}

Step 4. (Optional) Add authentication specific to this bucket.

Note: The default AWS authentication settings will be used if you don't supply them here (therefore it is optional).

Copy the code

accessKeyId = "${storage.engine.s3.accessKeyId}"
secretKey = "${storage.engine.s3.secretKey}"

Patterns

The patterns must be defined as "glob patterns" (see the Antglob Reference Guide). These will match on the assetPath of your asset, ignoring case.

Example: To match on files in '/Users/admin', use a pattern such as '/users/admin/**'.

Fallback

During the installation of Assets Server, the fileStoreType was defined as either 'local' or 'shared' in the node-config.properties.txt file.

These configuration values are used as fallback options in case no multi-tier storage is defined.

• storage.engine.type will be set as the type of your default-engine.
• fileStoreType will become the mounted property: 'shared' becomes 'true', 'local' becomes 'false'.
• sharedDataLocation is by default the base of assetFilestoreLocation, renditionFilestoreLocation and tempFilestoreLocation. The default values of these properties will be used as your default-engine's assetFilestore, renditionFilestore and tempFilestore.

Note: You could set it directly to ${assetFilestoreLocation} and so on, if you changed those properties already.

• For an S3 default-engine, your storage.engine.s3 properties will get used as default, as seen below.

Default configuration

The default configuration defines both the shared-volume and S3 configuration in the default-config block. This is fine, as only the properties for the defined type will be used.

For readability purposes it is however advised to only define the properties relevant to the type you configure.

Copy the code

storage.engine {
  type = "multi-tier"
  default-engine {
    type = "${storage.engine.type}"
 
    # the settings below are only used when type is 'shared-volume'
    sharedDataLocation = "${sharedDataLocation}"
    assetFilestore = "${assetFilestoreLocation}"
    renditionFilestore = "${renditionFilestoreLocation}"
    tempFilestore = "${tempFilestoreLocation}"
    mounted = ${mounted} # from the `filestoreType` legacy config option (`shared`=true,`local`=false)
 
    # the settings below are only used when type is 's3'
    region = "${storage.engine.s3.region}"
    bucket = "${storage.engine.s3.bucket}"
    accessKeyId = "${storage.engine.s3.accessKeyId}"
    secretKey = "${storage.engine.s3.secretKey}"
  }
  engines : []
}

Full configuration

A full configuration is shown in the example below.

Note: You can use placeholders such as ${sharedDataLocation} to use values from your other Assets Server configuration.

Notable properties:

• The default-engine is configured just like s3 or a shared-volume (depending on its type), but the name and pattern are left out (they would have no added value).
• Defining a multi-tier engine with only a default-engine and no sub-engines will work like configuring either of the other engines, with very little overhead.
• Be careful when defining the mounted option. If this property is set to false, the storage engine will not perform a mount check on the location and happily create directories for it if it doesn't exist. In case of a shared-volume, this will lead to data being spread amongst nodes, which will yield unreadable files depending on the Assets Server node that is being used to retrieve them.
• Optional values are commented out in the example below:

• accessKeyId / secretKey will be used as credentials when both are defined. Otherwise the default AWS credential provider will be used which will get this information from several other places in the system. See the Amazon documentation Class DefaultAWSCredentialsProviderChain.

• tempDirLocation will be used to store your files when they get downloaded from S3 or before they get uploaded to S3. The default is to use the system's temporary directory (as defined in java.io.tmpdir).

Note: Make sure that this location is on a fast local disk.

storage.engine {
  type = "multi-tier"
  default-engine {
    type = "shared-volume"
    assetFilestore = "/Library/Elvis Server/fs/default/assets"
    renditionFilestore = "/Library/Elvis Server/fs/default/renditions"
    tempFilestore = "/Library/Elvis Server/fs/default/tmp"
    mounted = false
  }
  engines : [
    {
      name = "fast-storage"
      type = "shared-volume"
      pattern = "/fast-storage/**"
      assetFilestore = "/Library/Elvis Server/fs/SSD1/assets"
      renditionFilestore = "/Library/Elvis Server/fs/SSD1/renditions"
      tempFilestore = "/Library/Elvis Server/fs/SSD1/tempFilestore"
      mounted = true
    }
    {
      name = "PDF"
      type = "shared-volume"
      pattern = "/**/pdf/**"
      assetFilestore = "/Library/Elvis Server/fs/PDF/assets"
      renditionFilestore = "/Library/Elvis Server/fs/PDF/renditions"
      tempFilestore = "/Library/Elvis Server/fs/PDF/tempFilestore"
      mounted = true
    }
    {
      name = "archive"
      type = "s3"
      pattern = "/archive/**"
      region = "eu-west-1"
      bucket = "elvis-archive"
    }
  ]
}

Troubleshooting

The following error appears: ... has type STRING rather than BOOLEAN*

A placeholder was probably defined that resolves to some configuration property which is in fact a string instead of a boolean. ${mounted} is a special property, resolved to the legacy filestoreType configuration property, with 'shared' resolving to true and 'local' to false.

The following error appears: java.lang.IllegalArgumentException: 'pattern' property must end with '/*' or '/**': pattern '**/*png*.*' for engine 'PNG'

You have defined a pattern as shown in the error which would cause a match on file name or extension. Such a pattern is currently not supported.

All patterns must end with one of the following:

• '/*' to match the assets in the matched folder only
• or '/**' to match the assets in the matched folder and in all its sub folders