Migrate workflow.xml from ibrain2 to Bee Specification

• Delete all:
  • <param name=“processing_name” value=“xxx” />
  • <class>xxx</class>

• Substitute: <module processing_name=“xxx” through <module name=“xxx”

• Change <module cluster_queue=“xxx”> by <module required_runtime_minutes=“xxx”> and /or <module required_memory_mb=“xxx”> corresponding to the mappings in the system.config:
  • cluster.rms.queuemapping = 30 = test_queue
  • cluster.rms.queuemapping = 720 = fs_long
  • cluster.rms.queuemapping = 100000 = fs_very_long

Examples:

cluster_queue="SHORT" ---->  required_runtime_minutes="30"
cluster_queue="LONG" ---->  required_runtime_minutes="720"

• Add <module version=“1.*.*” class=“ch.systemsx.bee.workflowmanager.module.ClusterModule”

• Substitute <params> and </params> through <executable> and </executable>

• Substitute <param name=“command_script” value=“xxx”> through <path>xxx</path>

• Add an <args> element to contain the different <arg..> elements.

• Substitute each <param name=“argument_x”…> through an equivalent <arg…> element in the same sort order.

• Add the necessary parameters for parallel executed modules.

Example:

<params>				
   <param name="indexbuilder_dataset" value="hcs_plate" />
   <param name="indexbuilder_regex" value=".*_cDAPI.*\.(TIF|JP2)$" />
   <param name="indexes_per_job" value="400" />
   <param name="indexes_start" value="2" />
</params>

• Substitute all the <input from_module=“0”> through workflow input datasets.

Example:

<input from_module="0">
  <resources>
    <resource name="hcs_plate" value="(^i)([^_]+)(_m)([^_]+)(_b)([^_]+)(_u)([^_]+)" />
  </resources>
</input>
  ----> 
<workflow>
  <input>
    <datasets>
      <dataset name="hcs_plate" id="${RawImages}" type="HCS_IMAGE_RAW" stage="true" />
    </datasets>
  </input>...

• Substitute all the <input from_module=“n”> where n > 0, through an output dataset in the n-module

Example:

<input from_module="1">
  <resources>
    <resource name="cpv1_create_batch_file_output_directory" value="Cpv1BatchFile" />
  </resources>
</input>
 ---->  
<output>
  <datasets>
    <dataset name="Cpv1BatchFile" store="false" />...

• Substitute each <storage><dataset_type> element through a corresponding <output><datasets><dataset> element in the same module.

Example:

<storage enabled="true">
  <dataset_type name="CLUSTER_JOB_LOGS">
    <files in_dir="" regex_pattern=".*\.STDERR$" out_dir="" />
    <files in_dir="" regex_pattern=".*\.STDOUT$" out_dir="" />
  </dataset_type>
</storage> 
  ---->  
<output>
  <datasets>
    <dataset type="CLUSTER_JOB_LOGS" store="true">
      <files in_dir="" regex=".*\.stderr" />
      <files in_dir="" regex=".*\.stdout" />
    </dataset>

• Substitute <output_checks mode=“task_level”> through <validations level=“task”>, and <output_checks mode=“module_level”> through <validations level=“module”>, and move them into the <output> element, after the <datasets> definition.

• In the <validations> sections substitute:
  • <validate> by <validation>
  • delete the resource_type=“xxx” attributes
  • resource_regex=“{IB2_STDOUT_LOG}” by regex=“${task.log_stdout}”
  • resource_regex=“{IB2_STDERR_LOG}” by regex=“${task.log_stderr}”
  • remaining resource_regex=“xxx” by regex=“xxx”
  • comparator=“is_equal_to” by comparator=“equal”
  • comparator=“is_superior_to” by comparator=“greater” (other allowed comparators are: less, not_equal, greater_or_equal, less_or_equal)
  • pattern_regex by content_regex
  • fail_status=“VALIDATION_ERROR” by fail_status=“validation_error”
  • fail_status=“VALIDATION_WARNING” by fail_status=“validation_warning”

Please, check the details in the workflow.xml specification syntax document: Workflow Template Specification