Skip to main content
English (United States) Nederlands

Working with Studio Server Jobs in Studio Server

Updated

A Studio Server Job is a task that is performed by Studio Server, such as clearing all objects of the Trash Can at set times or processing images during an import or export process.

This task does not necessarily have to be performed on the workflow machine: it can also be delegated to a different instance of Studio Server. Possible reasons for doing this include:

  • Running the process is expected to take too long on the workflow machine, which would result in end users having to wait for the process to complete.
  • Running the process is expected to consume too many resources (memory) on that machine, possibly causing memory swapping of other processes servicing other users, resulting in unpredictable performance.
  • The workflow machine is not able to perform the task because the required tools (such as Image Magick, Tika, and so on) are not available.

Other scenarios in which Server Jobs are useful include:

  • The process requires more access rights than the acting user has, and therefore there is a need to configure a user with more access rights for the job.
  • The process has an administrative purpose and needs to be repeated periodically. The job can then be scheduled to run at certain times.

In all instances, Server Jobs are queued and processed asynchronously in the background. End-users therefore do not have to wait for Studio Server to process a job and are not affected by heavy processing or memory consumption. These are the main goals of the Server Jobs concept.

Terminology

When working with Server Jobs, the following terminology is used:

Scheduled job

A job scheduled to be repeated at set intervals. For example: clearing a temp folder every day.

User-initiated job

A job triggered by an end-user. For example: uploading an image, which results in generating a preview of that image.

Server Job Queue

Server Jobs are placed in the Server Job Queue from which they can be picked up by any Studio Server machine.

Server Job configuration

The settings that define a Server Job, including:

  • The time interval of a recurring Server Job
  • How often a recurring Server Job should run
  • The number of attempts that a failed process should be retried

Background versus foreground processing

From a Studio Server point of view, the process of running a Server Job is done in the foreground; however, from an end-user’s point of view, the process is run in the background. This way, users can continue their work without having to wait for the process to be completed first.

Studio Server farm

A cluster of Studio Servers working together.

Server Job Processor

The Server Job Processor is the part of the system that picks up available Server Jobs from the Server Job Queue.

Co-worker machine

A Server machine running Studio Server that is dedicated to pick-up some or all jobs from the queue.

Note: By design, a Co-worker machine is not accessible by client applications and therefore does not act as a Workflow Server. Only under specific circumstances (for testing, demos, development, or even small offices) should a Co-worker machine be made available to client applications; doing so introduces the old issues of unpredictable performance due to heavy workload on the system.

Workflow Server

An instance of Studio Server set up to offer the workflow features (assigning a status, routing to users, and so on) to client applications.

Workflow Server versus Co-worker machines

There is no difference between installing Studio Server as a Workflow Server or as a Co-worker machine; each Studio Server system can be configured to run as a Co-worker, a Workflow Server, or both:

  • When Studio Server is not configured to pick up any Server Jobs, it is automatically considered to be a dedicated Workflow server.
  • When Studio Server is configured to pick up some or all Server Jobs, it is automatically considered to be a Co-worker machine.

Scheduler

A Scheduler is a continuously running engine that drives the Server Jobs mechanism for a Studio Server farm. It manages the following tasks:

  • Continuous processing of jobs from the queue.
  • Periodic creation of jobs (instances) based on recurring job configurations.

How Server Jobs are processed

Once a Server Job is created, it needs to be picked up. This can be done by each instance of Studio Server and is performed by the Server Job Processor: a PHP server module triggered by a scheduler.

For each process, the scheduler starts a new Job Processor. Each process runs for a very limited time (for instance a few minutes), after which the process no longer picks up new jobs from the queue and ends by itself. During the time the process runs, the scheduler creates new processes, thereby creating a pool of processes.

Idle times

When no jobs are available to be run, one process will be connected to watch for jobs arriving in the queue. This way, very low resources are used, enabling the whole server machine to become idle. When the waiting process picks up a job that just arrived from the queue, it gives way for other processes to come into action again to monitor and process the queue.

High workloads

When more jobs are available than the system can currently handle, the Server Job Processor will keep these processes on hold until the jobs that are being processed are completed.

Reattempts

In case a job fails to be processed, the system tries to understand the reason for the failure and decides if the job can be re-run. This can be repeated a set number of times after which the process is removed.

Identical jobs

In a typical setup, multiple Studio Servers are used. For practical reasons, the scheduling setup of recurring jobs for one Studio Server is usually copied to all other Studio Servers. As a result, multiple servers are triggering the queuing of the same Studio Server Job at the same time.

In an ideal setup, only one Studio Server is used for triggering a particular Server Job.

To prevent identical jobs from being created, Studio Server prevents the creation of a Server Job when an identical job has already been created within the last 60 seconds.

Setting up and configuring Studio Server Jobs

The following steps are to be performed:

1. Setting up Studio Server instances

2. Configuring existing Server Jobs

3. Installing a Scheduler

4. Setting up the job queue processor

5. (Optional) Setting up recurring jobs

6. Validating the installation

1. Setting up Studio Server instances

Each individual Studio Server can play several roles:

  • Application Server (Workflow Server)
  • File transfer Server
  • Co-worker (background processor)

Some servers in a Studio Server farm can have multiple roles, other servers can be clustered with an Apache dispatcher, while yet other servers might be dedicated to a certain role.

Regardless of the intended server role, for each different URL through which those servers (or clusters) can be reached, follow these steps:

Step 1. In Studio Server, access the Studio Servers page by doing the following:

Step 1a. Click Integrations in the Maintenance menu or on the Home page.

Step 1b. In the Studio Server Jobs section, click Studio Servers. The Studio Servers page appears.

The Studio Servers page

Step 2. Click New Server.

The Studio Server Maintenance page appears.

The Studio Server Maintenance page

Step 3. In the Name box, add a descriptive name.

Step 4. In the URL field, type the URL of the server as seen from the client applications.

Example: http://127.0.0.1/StudioServer

Step 5. (Optional) In the Description field, add a description for the server.

Step 6. From the Handle Jobs list, choose All.

Step 7. Click Update.

The Studio Servers page appears.

2. Configuring existing Server Jobs

The following Server Jobs are available by default:

  • TransferServerCleanUp. For clearing all files that are older than 24 hours from the File Transfer folder.
  • AutoPurgeTrashCan. For permanently deleting all files from the Trash Can.

Note: The email address of this user is used to report the results of the Auto Purge operation by e-mail to that user. The e-mail shows how many objects there are/were in the Trash Can, how long the operation took, and any errors that occurred. For information about customizing the e-mail, see Customizing notification e-mails in Studio Server.

To make these operational, follow these steps:

Step 1. Access the Studio Server Job Config page by doing the following:

Step 1a. In Studio Server, click Integrations in the Maintenance menu or on the Home page.

Step 1b. In the Studio Server Jobs section, click Job Configurations. The Studio Server Job Config page appears.

The Studio Server Job Config page

Step 2. Click the name of each job and set its options:

  • Attempts: Enter a number which specifies how many attempts should be made to re-try the job until running the job should be canceled.
  • Username: Choose a user with system admin privileges.
  • Active: Select to activate this job.

Step 3. Click Update.

3. Installing a Scheduler

Info: Performing the described step requires direct server access. Depending upon how your system is hosted and the level of access you have to that system, coordination may be required with your Partner or WoodWing Support team. For a full overview of the steps that need to be done by WoodWing and how to request them, see WoodWing Cloud - Change management.

A Scheduler should be set up on every Studio Server instance that needs to act as a Co-worker machine. This enables the Co-workers to pick up (heavy) jobs from the queue.

Step 1. Make sure that Studio Server is accessible from the command line, for example through the use of cURL.

Instructions for installing cURL

Linux

cURL is pre-installed on these platforms; no further action is needed.

Windows

Step 1a. Download the 32-bit or 64-bit binary package from http://curl.haxx.se.

Step 1b. Extract the package into any folder, for example c:\curl.

Step 1c. Test if cURL works by starting the Command Prompt and typing the following command:

c:\Curl\curl.exe --version

The installed URL version information should be displayed.

Step 2. Use a scheduler of your choice.

  • Task Scheduler for Windows. Task Scheduler is pre-installed for Windows. No further action is therefore needed.
  • Crontab for Linux. Crontab is pre-installed for Linux. Therefore, there is no further action needed.

4. Setting up the job queue processor

Info: Performing the described step requires direct server access. Depending upon how your system is hosted and the level of access you have to that system, coordination may be required with your Partner or WoodWing Support team. For a full overview of the steps that need to be done by WoodWing and how to request them, see WoodWing Cloud - Change management.

For performance reasons, a Scheduler would not be needed for machines running Studio Server that are configured exclusively as workflow application servers. Nevertheless, for Studio Server there are only light administrative job types available. Therefore any server machine can be used, even a workflow application server. More so because those job types are typically run at night time, and therefore do not affect production at all. Nevertheless, due to its great potential, the Server Jobs concept is likely to get used more and more. This includes co-worker jobs that could take away the heavy tasks from the workflow application servers. In other terms: for Studio Server it does not matter greatly which server is picked to install a Scheduler on, but for future versions it might make a big difference in performance and the deployment of additional servers becomes more important.

For continuous queue processing, the Scheduler needs to run the jobindex.php module periodically, whereby the jobindex.php runs until the Scheduler calls it again.

Example: In the following example the interval is one minute (for the Scheduler), and therefore also the duration is also one minute (for the jobindex.php). Running this URL makes the jobindex.php process jobs from the queue for one minute (60 seconds):

http://127.0.0.1/StudioServer/jobindex.php?maxexectime=60&maxjobprocesses=3

 

Configuration steps

Steps for Linux

Step 1. Edit the Crontab configuration by typing the following command:

crontab -e

Step 2. Press the [i] key to enter ‘insert’ mode and add one line below the header as follows:

Notes:

  • Make sure to add the URL in double quotes.
  • Use the [tab] key to jump to the next column.

# min hour mday month wday command

* * * * * curl "http://127.0.0.1/StudioServer/jobindex.php?maxexectime=60&maxjobprocesses=3"

Step 3. Press Shift+Z twice to exit and save changes.

The saved changes are directly effective in Crontab.

Steps for Windows

Step 1. Go to Control Panel > Administrative Tools.

Step 2. Start the Task Scheduler.

Step 3. From the Action menu, choose Create Task.

Step 4. Access the General tab.

Step 5. In the Name box, type Studio Server - Job Queue Processor.

Step 6. Select the option Run whether user is logged on or not.

Step 7. Access the Triggers tab.

Step 8. Create a new trigger with the following settings:

  • Begin the task: On a schedule.
  • Settings: Daily, set start date & time, Recur every 1 days.
  • Advanced settings:
  • Repeat task every 1 minute, for a duration of 1 day, Enabled.
  • Clear the option Stop all running tasks at end of repetition duration.

Step 9. Click OK.

Step 10. Access the Actions tab.

Step 11. Create a new action.

Step 12. In the Program/script box, type:

"c:\Curl\curl.exe"

Step 13. In the Add argument box, type the following (make sure to add the URL in double quotes):

"http://127.0.0.1/StudioServer/jobindex.php?maxexectime=60&maxjobprocesses=3"

Step 14. Access the Settings tab and do the following:

  • Clear the option If the running task does not end when requested, force it to stop
  • Set the option If the task is already running, then the following rule applies: to Run a new instance in parallel

Some notes on the URL:

  • Important: The double quotes ("...") around the URL are required to pass the whole URL to the curl command line tool.
  • You can trigger remote servers too, but you might lose overview. Even worse, when the local server is down, the remote server will not be triggered anymore. A more robust and granular solution is to configure localhost (127.0.0.1) only to run on the local machine itself.

Having that in place, the machine running jobindex.php is continuously watching the job queue. When it picks up a job, that job could take longer than one minute. So there is a danger that the next run of the Scheduler takes place before the first one is completed. Therefore, there is another parameter ‘maxjobprocesses’ that tells there should never be more than 3 jobs running in parallel. Doing so, the machine will not get overloaded with processes that are running jobs and swapping each other out of memory. This would be wasting processing time and leading to less throughput. Instead of picking up all jobs, you can make a specific selection: open the Server Maintenance admin page and check the Handle Jobs / Server Jobs options to specify the job types to be picked up by this machine.

5. (Optional) Setting up recurring jobs

Info: Performing the described step requires direct server access. Depending upon how your system is hosted and the level of access you have to that system, coordination may be required with your Partner or WoodWing Support team. For a full overview of the steps that need to be done by WoodWing and how to request them, see WoodWing Cloud - Change management.

For a whole Studio Server farm, there should be only one Studio Server instance responsible for creating recurring server jobs. Calling the jobindex.php file to create recurring jobs from multiple application servers (each having an exact copy of the crontab) is NOT wanted. Studio Server Jobs are not designed to run in parallel; it is not wanted to have 5 identical cleanup jobs in the queue simply because 5 application servers are installed. Studio Server jobs therefore have to be created from one application server instance only.

Running a recurring server job is a very light operation so it can be performed on any machine without performance issues. (Note that this operation does not execute jobs.)

For recurring job creation, the Scheduler needs to run the jobindex.php module periodically, whereby the jobindex.php simply creates one job instance for a given recurring job type.

The following recurring Server Job types are supported:

  • AutoPurgeTrashCan. For permanently deleting all files from the Trash Can.
  • AutoCleanWebEditDir. For cleaning up the WebEdit folder. Define the number of days after which the files should be removed by using the AUTOCLEAN_WEBEDITDIR_DAYS setting in the configserver.php file (recommended: config_overrule.php file):

As an example, the following steps explain how to configure the TransferServerCleanUp job type. For other job types, please repeat all the steps but then fill in the appropriate job type name.

Here, we let the schedule run a configured URL once a day at 23:00h after working hours. Running this URL makes the jobindex.php create a Transfer Server Cleanup job in the queue:

http://127.0.0.1/StudioServer/jobindex.php?createrecurringjob=TransferServerCleanUp

Steps for Linux

Step 1. Edit the Crontab configuration by typing the following command:

crontab -e

Step 2. Press the [i] key to enter ‘insert’ mode and add one line below the header as follows:

# min hour mday month wday command

* * * * * curl "http://127.0.0.1/StudioServer/jobindex.php?maxexectime=60&maxjobprocesses=3"

0 23 * * 1,2,3,4,5 curl "http://127.0.0.1/StudioServer/jobindex.php?createrecurringjob=TransferServerCleanUp"

Notes:

  • Make sure to add the URL in double quotes.
  • Use the [tab] key to jump to the next column.

Step 3. Press Shift+Z twice to exit and save changes.

Steps for Windows

Step 1. Go to Control Panel > Administrative Tools.

Step 2. Start the Task Scheduler.

Step 3. From the Action menu, choose Create Task.

Step 4. Access the General tab.

Step 5. In the Name field, type Studio Server - Transfer Server Cleanup.

Step 6. Select the option Run whether user is logged on or not.

Step 7. Access the Triggers tab.

Step 8. Create a new trigger with the following settings:

  • Begin the task: On a schedule.
  • Settings: Weekly, set start date & time, Recur every 1 weeks on Monday, Tuesday, Wednesday, Thursday, Friday;.
  • Advanced settings: Enabled.

Step 9. Click OK.

Step 10. Access the Actions tab.

Step 11. Create a new action.

Step 12. In the Program/script box, enter:

"c:\Curl\curl.exe"

Step 13. In the Add argument field, type the following (make sure to add the URL in double quotes and to adjust the Server URL):

"http://127.0.0.1/StudioServer/jobindex.php?createrecurringjob=TransferServerCleanUp"

Step 14. Deselect the option Stop all recurring tasks at end of repetition duration.

Some notes on the URL:

  • Important: The double quotes ("...") around the URL are required to pass the whole URL to the curl command line tool.
  • You can trigger remote servers too, but you might lose overview. Even worse, when the local server is down, the remote server will not be triggered anymore. A more robust and granular solution is to configure localhost (127.0.0.1) only to run on the local machine itself.

Having that in place, the job simply gets created in the queue, waiting for any Studio Server to pick it up. Therefore, at least one Studio Server needs to be configured to pick up the TransferServerCleanUp job type. Open the Studio Server Maintenance admin page for a configured Studio Server and check the Handle Jobs / Studio Server Jobs options to specify the job type.

Step 15. (Optional, for IIS on Windows only) In the FastCGI settings of IIS, increase the value of the Activity Timeout setting. The default setting of 30 is potentially too short for letting Server Jobs to be picked up.

The FastCGI settings

6. Validating the installation

Info: Performing the described step requires direct server access. Depending upon how your system is hosted and the level of access you have to that system, coordination may be required with your Partner or WoodWing Support team. For a full overview of the steps that need to be done by WoodWing and how to request them, see WoodWing Cloud - Change management.

Once the Scheduler is set up, check its working by following these following steps:

Step 1. Run each of the recurring jobs manually.

Steps for Linux

Run any of the configured recurring jobs. For example:

curl "http://127.0.0.1/StudioServer/jobindex.php?createrecurringjob=TransferServerCleanUp"

Steps for Windows

Step 1a. Run the Task Scheduler.

Step 1b. Select a configured recurring job.

Step 1c. Click Run.

Step 2. In Studio Server, access the Job Queue page by doing the following:

Step 2a. In Studio Server, click Integrations in the Maintenance menu or on the Home page.

Step 2b. In the Studio Server Jobs section, click Job Queue. The Studio Server Job Queue page appears.

The page should show the following:

  • There should be one instance of each recurring job type listed.
  • The Status of each job should change to Completed within one minute (after Queued Since).

Note: Refresh the page when jobs are not completed yet.

Step 3. Wait for the next day to see if the recurring tasks are automatically created at 23:00h.

Related articles

Comments

0 comments