Importing files to Assets Server in bulk

A bulk import of files to Assets Server is typically done when the system has just been set up and files need to be moved from your existing file store locations to Assets Server.

This will also be a good moment to analyze the files that have accumulated in your file store and to exclude unwanted files from the import to make sure that the Assets Server system that is imported to is as clean as possible.

Assets Server provides various methods for importing files in bulk:

• The Bulk Import tool. This import tool imports large numbers of files from an existing file storage, including from Amazon S3. It moves or copies files and retains the folder structure of the original file store. For more information, see Importing files in bulk using the Bulk Import tool in Assets Server.

• A scripted import through an API. This makes it possible to perform additional tasks on the imported files such as:
• Adding or enriching metadata.
• Including metadata from sidecar files (such as .xmp files that accompany image files).
• Re-defining the folder structure.
• Adding files to Collections.

For more information, see Assets Server REST API - create.

How long does a bulk import take?

When it would take only 1 second for importing 1 file, importing 50,000 assets would take 15 hours while a typical file store ranging anywhere from 500,000 to 10 million assets would take weeks or even months to import.

Improving the import time

To overcome this time problem, consider doing the following:

• Split your bulk import into stages:

1. First only import the business-critical files: these are needed straight away.
2. Then import all archived files; it is not very likely that these files are needed immediately.

• Increase system performance by:

• Adding Processing nodes to spread the load of extracting metadata, generating and transcoding previews and thumbnails, and embedding the Assets Server ID.

• Adding Search nodes to speed up the process of indexing.

• Using systems with fast hard disks such as SSD disks.

• Making as much network bandwidth available as possible.

Note: Remember that these steps are only required for the initial bulk import. Once Assets Server is up and running, the system can be scaled down again.