Archiving files from Studio Server 10.11.0 to Assets Server

In a typical production process, publications reach their end of life once they have been published. They are subsequently archived to make room in the system for other publications while still having access to the files for reference or re-use.

When Assets Server is integrated with Studio Server, files can be archived from Studio Server to Assets Server.

This article describes the process and configuration.

Note: New archiving functionality has been progressively added to Studio Server. The steps that are described here apply to Studio Server 10.11.0. For other versions, see the following articles: For version 10.13.1 or any higher version of Studio Server. For version 10.13.0. For version 10.12.x. For version 10.11.4 or any higher version of 10.11. For versions 10.11.1 – 10.11.3. For version 10.10.0.

Supported files

Studio Server 10.11.0 supports archiving for the following files:

• Digital articles and their placed images.
• Layouts and their placed files, and optionally their output files in PDF and EPS format.

Note: Archiving Dossiers requires Studio Server 10.11.1 or higher.

How it works

The basic process of archiving files works as follows:

1. A check is done to see if any files are available for archiving.
2. When files are available for archiving, the archiving process is performed.
3. The archived files are removed from Studio Server.
4. When archiving a file failed or could not be performed, notifications are given to the user.
5. Archived files can be restored at a later stage.

The main reasons why archiving could fail are:

• The Workflow Status of a file is set to 'Personal'.
• The Archive Status of a file is set to 'Excluded'.
• One of the files is a Layout Module.

More detailed information is given later in this article.

Requirements

It is assumed here that an integration between Studio Server and Assets Server is in place and fully working.

Configuration

Configuration steps need to be performed in Studio Server and optionally in Assets Server.

Info: Use the filter to only show information for one step. All steps 1. Configuring archiving settings in Studio Server 2. Setting up Studio Server Jobs 3. (Optional, for Digital articles only) Configuring versioning settings in Studio Server 4. Testing the implementation 5. Additional optional steps

1. Configuring archiving settings in Studio Server

Archiving settings are configured in Studio Server for each Brand individually on the Brand Maintenance page.

The archiving options are configured in the Assets Server section.

Note: These options only appear when the WoodWing Assets Content Source Server plug-in is enabled, as part of integrating Studio Server and Assets Server.

Step 1. In Studio Server, access the Brand Maintenance page for which Digital articles and layouts are to be archived. Set the fields as described in the following steps.

Note: The Production Zone field is not used for archiving and can be ignored here.

Step 2. (Only when archiving Digital articles) When archived Digital articles and their images are restored, a default Workflow Status can be assigned. Set these in the following fields:

• Default Status Restored Digital Article
• Default Status Restored Image

Notes: The default setting is 'First status' meaning that the first Workflow Status that is configured for that file type in the Brand is assigned. The Personal status is not supported. When a configured Workflow Status is deleted in Studio Server, the field which has this status set is re-set to 'First status'.

Step 3. Set the Archive Schema field as follows:

For layouts:  Select Issue based. This defines an archiving process in which layouts are archived when the publication date of an Issue has passed, as is the case for layouts. For Digital articles: Select Continuous. This defines a workflow in which publications are continuously published (such as Digital articles that are published frequently to a publication channel).

Step 4. For each Archive Schema, set its options.

• Archive Zone. Set the location in Assets Server in which to archive the files

The structure of the location in which layouts and Digital articles and their files are archived to in Assets Server are outlined below.

Archive Schema - Digital /<base folder>/<story folder>/<version folder>/ Base folder: A folder consisting of a custom name, a name based on a variable — such as the name of the Brand, a Category, or a date or time — or a combination of a custom name and a variable. Story folder: The name of the Dossier, or (when the Digital article is not part of a Dossier) the name of the Digital article. Version folder: The date of archiving followed by an em-dash and the name of the Dossier or Digital article. Example 1: (Digital article is not part of a Dossier, 'Include unplaced objects in Dossier' is selected) Base folder: 'Archive Zone' Digital article name: 'Amsterdam' Archiving date: 2 January 2020 /Archive Zone/Amsterdam/2020-01-02 — Amsterdam/ Example 2: (Digital article is part of a Dossier, 'Include unplaced objects in Dossier' is selected) Base folder: 'Archive Zone' Dossier name: 'Cities' Digital article name: 'Amsterdam' Archiving date: 2 January 2020 /Archive Zone/Cities/2020-01-02 — Cities/ Example 3: (Digital article is part of a Dossier, 'Include unplaced objects in Dossier' is not selected) Base folder: 'Archive Zone' Dossier name: 'Cities' Digital article name: 'Amsterdam' Archiving date: 2 January 2020 /Archive Zone/Cities/2020-01-02 — Cities/

Archive Schema - Dossiers /<base folder>/<story folder>/<version folder>/ Base folder: A folder consisting of a custom name, a name based on a variable — such as the name of the Brand, a Category, or a date or time — or a combination of a custom name and a variable. Story folder: The name of the Dossier. Version folder: Not applicable. Example: (Dossier containing some objects but not a Digital article or layouts) Base folder: 'Archive Zone' Dossier name: 'Cities' Archiving date: 2 January 2020 /Archive Zone/Cities/

Archive Schema - Print /<base folder>/<story folder>/<version folder>/ Base folder: A folder consisting of a custom name, a name based on a variable — such as the name of the Brand, a Category, or a date or time — or a combination of a custom name and a variable. Story folder: The name of the layout (regardless of whether the 'Include unplaced objects in Dossier' option is selected or not). Version folder: Not applicable. Example 1: (Layout is not part of a Dossier, 'Include unplaced objects in Dossier' is selected) Base folder: 'Archive Zone' Layout name: 'Amsterdam' Archiving date: 2 January 2020 Archive Zone/Amsterdam/ Example 2: (Layout is part of a Dossier, 'Include unplaced objects in Dossier' is selected) Base folder: 'Archive Zone' Dossier name: 'Cities' Layout name: 'Amsterdam' Archiving date: 2 January 2020 Archive Zone/Amsterdam/ Example 3: (Layout is part of a Dossier, 'Include unplaced objects in Dossier' is not selected) Base folder: 'Archive Zone' Dossier name: 'Cities' Layout name: 'Amsterdam' Archiving date: 2 January 2020 Archive Zone/Amsterdam/

Setting up the structure can be done manually or automatically.

Manual setup

In this setup, a folder is manually created and linked to in the Studio Server settings.

Step 4a. Using Assets, create a folder on Assets Server and assign Permissions to it.

Note: It is recommended to not allow users to edit archived files.

Step 4b. Add the path to the Archive Zone field.

Example: /Archive Zone/My Brand/My publication date

Automatic setup

In this setup, a folder structure is constructed in the Brand settings on Studio Server. When archiving is performed for the first time for that Brand, the folder is automatically created in that location in Assets Server (when it does not exist yet). User permissions are set once the folder exists.

Step 4a. Construct the path by using the following variables.

Notes: The same wildcard may be used more than once. A few date and time wildcards are used as an example; all other notations are supported as well. Empty values which may lead to double slashes are automatically resolved by leaving out the sub folder and the superfluous slash. Example:foo/${issue}/bar results in foo/bar and not foo//bar.

Wildcard	Description
${brand}	Adds the name of the Brand to the folder name.
${category}	Adds the name of the Category to the folder name.
${issue}	Adds the name of the Issue to the folder name. Notes: This wildcard is ignored when the layout or Digital article is not targeted for an Issue. When a layout or Digital article is targeted to more than one Issue, the name of the first Issue is taken.
${archived:Y-m}	Adds the year and month to the folder name of the moment when the archiving takes place. Example: (When date is 1 January 2020): ${archived:Y-m} becomes 2020-01.
${date:Y-m}	Adds the year and month to the folder name of the moment when the archiving takes place. Example: (When date is 1 January 2020): ${date:Y-m} becomes 2020-01.
${created:Y-m}	Adds the year and month of the moment when the layout or Digital article was created to the folder name. Example: (When date is 1 January 2020): ${created:Y-m} becomes 2020-01.
${modified:Y-m}	Adds the year and month to the folder name of the moment when the layout or Digital article was modified. Example: (When date is 1 January 2020): ${modified:Y-m} becomes 2020-01.
${C_<name>:Y-m}	Adds the year and month to the folder name of a custom workflow property (specified by C_<name>) of which the type is set to 'Date' or 'Datetime'. Example: (When custom property is C_MY_PROPERTY and date is 1 January 2020): ${C_MY_PROPERTY:Y-m} becomes 2020-01.
${C_<name>}	Adds the value of a custom workflow object property (specified by C_<name>) to the folder name. Replace C_<name> with your custom property name.

Example: The following definition: /Archive Zone/${brand}/${category}/${created:Y}/${created:m} results in the following path (when the Brand is 'WW News', the Category is 'Sport', and the current date is 1 January 2020): /Archive Zone/WW News/Sport/2020/01

Step 4b. Once a first archiving action is performed for the Brand and the defined folder exists in Assets Server, assign Permissions to the folder.

• Archive after (days). Define the number of days after which the archiving should take place. The day that is referenced is as follows:

• For layouts: The number of days after the publication date. Default is 7.
• For Digital articles: The most recent date when an object in the Dossier was modified. Default is 30.

Note: Carefully determine this time period: when set too short, users might still be working on files which will make the archiving process fail; setting it too long will clutter the system with unused files.

• Objects to archive. (Only applies to archiving layouts) Define if only the layout or also any existing output files need to be archived by choosing from one of the following options:

• Layouts. Only the layout and any files that are placed on it are archived.
• Layouts, PDFs, and EPS files. The layout and any files that are placed on it are archived, as well as any output files in PDF or EPS format that have been automatically created by Studio Server.

Note about PDF and EPS files. Publishing the created layout is done by converting it to a PDF or EPS file. In a Studio Server workflow, an output file is typically created automatically as soon as the layout is set to a specific Workflow Status (see Workflow Status settings in Studio Server). When these output files are available, they can optionally be archived together with the layout.

Step 5. Click Update to save the changes.

2. Setting up Studio Server Jobs

In this step, scheduled tasks are set up in Studio Server which perform the archiving process.

Note: It is assumed here that the Studio Server Jobs feature is set up and fully functioning. Fore more information, see Working with Studio Server Jobs in Studio Server.

This process is performed by the following Studio Server Jobs:

• SelectForArchive. Determines if any objects (a layout, Digital article, or Dossier) are available for archiving. When found, it creates a SendToArchive job for that object.
• SendToArchive. Performs the actual archiving for that object.

These jobs need to be configured and a recurring job needs to be set up.

Step 1. On the Studio Server Job Config page click on the Server Job names SelectForArchive and SendToArchive to access their settings page and in the User name field set a user who has system admin privileges. Click Update when done.

Step 2. Set up a recurring job for the SelectForArchive job.

Example: A crontab configuration for running the job every hour: 0 * * * * curl "http://myserver.com/StudioServer/jobindex.php?createrecurringjob=SelectForArchive&serverplugin=Elvis"

3. (Optional, for Digital articles only) Configuring versioning settings in Studio Server

Info: This step only has to be performed when archiving Digital articles.

When making use of this archiving method, the Create Permanent Version option should be disabled for images, videos, and audio files.

This can be done in the Workflow Status settings for these objects.

4. Testing the implementation

A simple demo tool is available for testing the archiving feature and thereby better understanding the system behavior. It is explained in the following file as part of the Studio Server package:

StudioServer/server/plugins/Elvis/demo/sendtoarchive/README.md

Note: Do not use the tool in a production environment.

Use the criteria described in the archiving workflow section in this article for setting up your testing scenarios.

5. Optional steps

After setting up the integration itself, the following optional steps may be required to perform:

1. (Optional) Adding the 'Archive Status' property to the clients

In certain scenarios, a Brand administrator may want to manually modify the 'Archive Status' property of a file.

Examples: It may be changed from its empty value to 'Excluded' to prevent that file from being archived. It may be changed from 'Failed' to its empty value for a file that was inadvertently locked for editing while the archiving process took place which prevented it from being archived.

To make this possible, the 'Archive Status' property needs to be made visible in the client applications. For information about how this is done, see Adding a property to a dialog box in Studio Server.

2. (Optional, for layouts only) Configuring how images should be restored

When an archived layout is restored to Studio Server from Assets Server, any images that are placed can be restored in one of the following ways:

• As a copy in Studio Server that is not linked to the file in Assets Server.
• As a Production shadow object in Studio Server that is linked to a copy of the original file in Assets Server.

For information about configuring this, see Controlling how files are stored in the Assets Server - Studio Server integration.

3. (Optional) Making it possible to preview archived Digital articles in Assets

Info: This step only has to be performed when archiving Digital articles.

Digital articles make use of various resources such as Component Sets, styles, custom swatches, and so on. These resources are stored in a central location in the cloud in a system named WoodWing Cloud Services. To make it possible to preview a Digital article in Assets, Assets Server and Studio Server need to be connected to WoodWing Cloud Services so that these resources can be shared.

This is done through a so-called tenant registration. For Assets Server this registration takes place in the Management Console.

Perform this process by following the steps in Connecting Studio Servers and Assets Server to WoodWing Cloud Services.

The archiving workflow

The archiving process takes place in two parts:

1. At a scheduled time, a check is done to see if a layout or Digital article is available for archiving. Such objects are referred to as 'candidate objects'.

2. For each found candidate object, a Server Job is created. When this job is run, the following processes are performed:

1. The objects to archive are determined and verified.

2. The archiving process is performed.

1. Determining candidate files

Specific rules apply for determining if a layout or Digital article is scheduled for archiving.

Process for layouts

A layout is included for archiving when all of the following conditions are met:

• The layout is assigned to a Brand for which the Archive Schema is set to 'Issue based' and the 'Archive after (days)' option is filled in.
• The layout is assigned to an Issue for which a publication date is set.
• The difference between the current date and the publication date is bigger than the number set in the 'Archive after (days)' option.
• The difference between the current date and the layout creation date is bigger than the number set in the 'Archive after (days)' option.
• The 'Archive Status' property of the layout is set to the 'Empty' value (this indicates that the layout is not already processed by another job), nor failed during archiving, nor manually excluded from archiving.

When candidate layouts are found, the following takes place:

• The 'Archive Status' property of the layout is set to 'Candidate'.
• The 'Archive Status Modified' property of the layout is set to the current date and time.
• Studio Server Jobs are created for archiving the layouts (one job for each layout).
• The Studio Server Job that was used for determining the candidate layouts is automatically removed from the job queue.

Process for Digital articles

A Digital article is included for archiving when all of the following conditions are met:

• The Digital article is placed in a Dossier.
• The Digital article is part of a Brand for which the Archive Schema is set to 'Continuous'.
• The 'Archive Status' property of the Dossier is empty.
• The most recent modified date of a file in the Dossier is older than the value set in the 'Archive after (days)' option for the Brand.

When candidate Digital articles are found, the following actions take place:

• The 'Archive Status' property of the Dossier is set to 'Candidate'.
• The 'Archive Status Modified' property of the Dossier is set to the current date and time.
• Studio Server Jobs are created for archiving the Digital articles (one Job for each candidate Dossier)
• The Studio Server Job that was used for determining the candidate Digital articles is automatically removed from the job queue.

2. Performing the archiving process

When candidate objects are found and all validation rules are met, archiving takes place.

Specific steps apply for each archived object.

Process for layouts

• The layout and all its placed objects (including those on the pasteboard) are archived. These files are re-linked to the layout to make it possible to restore the layout when needed.

• When 'Objects to archive' is set to 'Layouts, PDFs and EPS files', any output files are also archived.

Note that an output file is created for each page and for each Edition. Many output files can therefore exist for each layout. Each output file is included in the layout folder and will have the following file name structure:

layout name — Edition name— page number.file extension

Example: Amsterdam — North — 1.pdf

• When a Word or Excel file is placed on the layout, an InDesign preference controls whether the original file is placed or the placement is a link to the original file (see Preferences > File Handling > Create Links When Placing Text and Spreadsheet Files).

Depending on how this preference is set, Word and Excel files are treated as follows:

• Option is disabled: The Word or Excel file is automatically converted to an InCopy file by Studio when it is placed. During archiving, this InCopy file is archived.
• Option is enabled. A link is created to the original Word or Excel file. During archiving, this original file is archived.

• The archived files are removed from Studio Server.

Note: A Dossier may be left in place if it still contains files that have not been archived.

• The name of the folder in which the files are archived is the name of the layout.

• For files that could not be archived, the Archive Status property is set to 'Failed' and messages are added to the files to inform the user.

Layouts are not archived when:

• The Archive Status of the layout or one of its placed files is set to 'Excluded' or 'Failed'.
• The Workflow Status of the layout is set to 'Personal'.
• The layout or one of its placed files is still checked-out (locked for editing).
• The layout contains a Layout Module (a layout placed on a layout).

Handling files that are also placed on other layouts

By default, all files that are placed on the layout are archived to Assets Server and removed from Studio Server.

However, when a placed file is also placed on another layout, the following rules apply:

• The file remains in Studio Server or (when applicable) in the Production Zone in Assets Server until there are no other layouts on which it is also placed. The links between the file and these layouts remain intact.

• The file is unlinked from the layout that is archived.

• The file is copied to the Archive Zone (rather than moved). The copied file is linked to the layout.

Layout archiving and Dossiers

It is very common for layouts and their supporting files to be added to a Dossier in Studio Server. As part of archiving layouts though, Dossiers are ignored.

This means that:

• Layouts and the files that are placed on it are removed from all Dossiers they are part of.
• Dossiers are not deleted, even when they end up empty when the archiving process is completed.

Process for Digital articles

• The Dossier is added as a folder to the archive environment in Assets Server. The folder contains the Digital articles and any images that are part of the articles, plus all images, audio files, and video files that are present in the Dossier. Files of any other types that are present in the Dossier are ignored (not archived).

• The name of the folder in which the files are archived is the name of the Digital article, or (when the 'Include unplaced objects in Dossier' option is selected and the Digital article is part of a Dossier) the name of the Dossier.

• The Dossier and all the files that are archived are removed from Studio Server.

Note: A Dossier may be left in place if it still contains files that have not been archived.

• (For Assets Server 6.50 and higher) The names of the default components that the Digital article contains are stored in a metadata property named 'Article Components'. This property can be used for finding Digital articles containing a specific component, for example by adding the property to the Filter in Assets.

Note: Names of custom properties are not stored.

• Any shadow objects that were stored in the Production Zone of Assets Server are moved to the Archive Zone in Assets Server; when these files are still used elsewhere, they are copied instead of moved. These images are relinked in the Digital article.

Digital articles are not archived when:

• The Dossier is set to the 'Personal' status.
• At least one of the Digital articles in the Dossier has the 'Archive Status' property set to 'Excluded'.
• Any of the files in the Dossier are locked for editing.

Notifications when archiving fails

When for some reason the archiving process cannot be performed, users are notified in the following ways:

• A Flag icon is shown to the user for the candidate object:
• In Studio for InDesign in the Flag (Icon) column of the Studio panel when viewing files in List View mode.
• In Studio in the Flag column of the search results panel when viewing files in List View mode.

When hovering the mouse pointer over the icon, a tooltip with information about why the object could not be archived is shown.

• For Dossiers of a Digital article, a Dossier note is added which can be viewed in Studio.

Preventing files from being archived

Preventing layouts and Digital articles from being archived can be done in various ways.

• For layouts:

• For all layouts assigned to a Brand:
• On the Brand Maintenance page, make sure that the Archive Schema option is not set to Issued Based.

• For all layouts assigned to an Issue:
• On the Issue Maintenance page, clear the Publication Date field.

• For a specific layout:
• Set the 'Archive Status' property of the layout to 'Excluded'.

• For Digital articles:

• For all Digital articles assigned to a Brand:
• On the Brand Maintenance page, make sure that the Archive Schema option is not set to Continuous.

• For a specific Digital article:
• Set the 'Archive Status' property of the Digital article to 'Excluded'. (Doing this for one Digital article in the Dossier prevents the whole Dossier from being archived, even when additional Digital articles exist in the Dossier for which this property is not set.)

Preventing the archiving from failing

The exact conditions in which archiving fails are described above, but in general follow these guidelines:

• When Layout Modules are used, consider not using the archive schema 'Issue based'.
• Do not enable archive schemas for objects that are not used (such as 'Continuous' when the Brand is used for print only).
• Do not keep files checked out for editing.
• Do not use the Workflow Status 'Personal' for files that can be picked up for archiving.
• Set the 'Archive after (days)' option to a suitable period of time (not too short and not too long).

Locating files that have failed to archive

It is important to be aware of files that have failed to archive and to rectify them. For this purpose, a Custom Search can be set up in Studio to locate these files on a regular basis.

For more information, see Locating files in Studio that failed to archive.

Resolving failed archiving

When the archiving process fails, the objects need to be manually corrected to have them picked up as a candidate again. Do the following:

• Clear the Archive Status field for that object (it will be set to 'Failed').
• Remove an object from a Dossier when the Workflow Status of that object is set to 'Personal'.

Restoring archived files

Information about restoring archived files back to Studio Server can be found in the following articles:

• Digital articles: By opening them in Assets.
• Layouts: By opening them in Assets.