Importing files in Elvis 5 in bulk

A bulk import of assets into Elvis is typically done when the system has just been set up and assets need to be moved from your existing file store location(s) into Elvis.

This will also be a good moment to analyze the assets that have accumulated in your file store and to exclude unwanted files from the import to ensure an Elvis system that is as clean as possible.

Elvis 5 provides various methods for importing assets in bulk:

• The Bulk Import tool of the Elvis 5 Pro client. (Requires Elvis 5.17 or higher) This improved import tool imports large numbers of assets from an existing file storage, including from Amazon S3. It moves or copies assets and retains the folder structure of the original file store. For more information, see Importing files in bulk using the Import tool of the Elvis Management Console.

• A scripted import through an API. This allows you to perform additional tasks on the imported assets such as:
• Adding or enriching metadata
• Including metadata from sidecar files (such as .xmp files that accompany image files)
• Re-defining the folder structure
• Adding assets to Collections

For more information, see Elvis 5 REST API - create.

• The Bulk Import tool of Elvis 5 Server. This tool moves assets and retains the folder structure of the original file store. For more information, see Importing files in bulk using the Bulk Import tool of Elvis 5 Server.

How long does a bulk import take?

When it would take only 1 second for importing 1 file, importing 50,000 assets would take 15 hours while a typical file store ranging anywhere from 500,000 to 10 million assets would take weeks or even months to import.

Improving the import time

To overcome this time problem, consider doing the following:

• Split your bulk import into stages:

1. First only import the business-critical assets: these are needed straight away
2. Then import all archived files; it is not very likely that these files are needed immediately

• Increase system performance by:

• Adding Processing nodes to spread the load of extracting metadata, generating and transcoding previews and thumbnails, and embedding the Elvis ID.

• Adding Search nodes to speed up the process of indexing.

• Using systems with fast hard disks such as SSD disks.

• Making as much network bandwidth available as possible.

Note: Remember that these steps are only required for the initial bulk import. Once Elvis is up and running, you can scale down again.

Document history

• 20 December 2016: Re-written article and moved description of using the Bulk Import tool of Elvis 5 Server to Importing files in bulk using the Bulk Import tool of Elvis 5 Server.