User Tools

Site Tools


beewm:devel:migrate_workflow.xml_from_ibrain2_to_bee_specification

Migrate workflow.xml from ibrain2 to Bee Specification

  • Delete all:
    • <param name=“processing_name” value=“xxx” />
    • <class>xxx</class>
  • Substitute: <module processing_name=“xxx” through <module name=“xxx”
  • Change <module cluster_queue=“xxx”> by <module required_runtime_minutes=“xxx”> and /or <module required_memory_mb=“xxx”> corresponding to the mappings in the system.config:
    • cluster.rms.queuemapping = 30 = test_queue
    • cluster.rms.queuemapping = 720 = fs_long
    • cluster.rms.queuemapping = 100000 = fs_very_long

Examples:

cluster_queue="SHORT" ---->  required_runtime_minutes="30"
cluster_queue="LONG" ---->  required_runtime_minutes="720"
  • Add <module version=“1.*.*” class=“ch.systemsx.bee.workflowmanager.module.ClusterModule”
  • Substitute <params> and </params> through <executable> and </executable>
  • Substitute <param name=“command_script” value=“xxx”> through <path>xxx</path>
  • Add an <args> element to contain the different <arg..> elements.
  • Substitute each <param name=“argument_x”…> through an equivalent <arg…> element in the same sort order.
  • Add the necessary parameters for parallel executed modules.

Example:

<params>				
   <param name="indexbuilder_dataset" value="hcs_plate" />
   <param name="indexbuilder_regex" value=".*_cDAPI.*\.(TIF|JP2)$" />
   <param name="indexes_per_job" value="400" />
   <param name="indexes_start" value="2" />
</params>
  • Substitute all the <input from_module=“0”> through workflow input datasets.

Example:

<input from_module="0">
  <resources>
    <resource name="hcs_plate" value="(^i)([^_]+)(_m)([^_]+)(_b)([^_]+)(_u)([^_]+)" />
  </resources>
</input>
  ----> 
<workflow>
  <input>
    <datasets>
      <dataset name="hcs_plate" id="${RawImages}" type="HCS_IMAGE_RAW" stage="true" />
    </datasets>
  </input>...
  • Substitute all the <input from_module=“n”> where n > 0, through an output dataset in the n-module

Example:

<input from_module="1">
  <resources>
    <resource name="cpv1_create_batch_file_output_directory" value="Cpv1BatchFile" />
  </resources>
</input>
 ---->  
<output>
  <datasets>
    <dataset name="Cpv1BatchFile" store="false" />...
  • Substitute each <storage><dataset_type> element through a corresponding <output><datasets><dataset> element in the same module.

Example:

<storage enabled="true">
  <dataset_type name="CLUSTER_JOB_LOGS">
    <files in_dir="" regex_pattern=".*\.STDERR$" out_dir="" />
    <files in_dir="" regex_pattern=".*\.STDOUT$" out_dir="" />
  </dataset_type>
</storage> 
  ---->  
<output>
  <datasets>
    <dataset type="CLUSTER_JOB_LOGS" store="true">
      <files in_dir="" regex=".*\.stderr" />
      <files in_dir="" regex=".*\.stdout" />
    </dataset>
  • Substitute <output_checks mode=“task_level”> through <validations level=“task”>, and <output_checks mode=“module_level”> through <validations level=“module”>, and move them into the <output> element, after the <datasets> definition.
  • In the <validations> sections substitute:
    • <validate> by <validation>
    • delete the resource_type=“xxx” attributes
    • resource_regex=“{IB2_STDOUT_LOG}” by regex=“${task.log_stdout}”
    • resource_regex=“{IB2_STDERR_LOG}” by regex=“${task.log_stderr}”
    • remaining resource_regex=“xxx” by regex=“xxx”
    • comparator=“is_equal_to” by comparator=“equal”
    • comparator=“is_superior_to” by comparator=“greater” (other allowed comparators are: less, not_equal, greater_or_equal, less_or_equal)
    • pattern_regex by content_regex
    • fail_status=“VALIDATION_ERROR” by fail_status=“validation_error”
    • fail_status=“VALIDATION_WARNING” by fail_status=“validation_warning”

:!: Please, check the details in the workflow.xml specification syntax document: Workflow Template Specification

beewm/devel/migrate_workflow.xml_from_ibrain2_to_bee_specification.txt · Last modified: 2016/05/17 16:17 by 127.0.0.1