screeningBee Data Analysis Tools

This page is read only. You can view the source, but not change it. Ask your administrator if you think this is wrong.
====== Migrate workflow.xml from ibrain2 to Bee Specification ======


   * Delete all:
     * ''<param name="processing_name" value="xxx" />''
     * ''<class>xxx</class>''

   * Substitute: ''<module processing_name="xxx"''  through  ''<module name="xxx"''

   * Change ''<module cluster_queue="xxx">'' by ''<module required_runtime_minutes="xxx">'' and /or ''<module required_memory_mb="xxx">'' corresponding to the mappings in the ''system.config'':
     * cluster.rms.queuemapping = 30 = test_queue 
     * cluster.rms.queuemapping = 720 = fs_long
     * cluster.rms.queuemapping = 100000 = fs_very_long

Examples: 
<code xml>
cluster_queue="SHORT" ---->  required_runtime_minutes="30"
cluster_queue="LONG" ---->  required_runtime_minutes="720"
</code>

   * Add ''<module version="1.*.*" class="ch.systemsx.bee.workflowmanager.module.ClusterModule"''

   * Substitute ''<params>'' and ''</params>'' through ''<executable>'' and ''</executable>''

   * Substitute ''<param name="command_script" value="xxx">'' through  ''<path>xxx</path>''

   * Add an ''<args>'' element to contain the different ''<arg..>'' elements.

   * Substitute each ''<param name="argument_x"...>'' through an equivalent ''<arg...>'' element in the same sort order.

   * Add the necessary parameters for parallel executed modules.
Example:
<code xml>			
<params>				
   <param name="indexbuilder_dataset" value="hcs_plate" />
   <param name="indexbuilder_regex" value=".*_cDAPI.*\.(TIF|JP2)$" />
   <param name="indexes_per_job" value="400" />
   <param name="indexes_start" value="2" />
</params>
</code>

   * Substitute all the ''<input from_module="0">'' through workflow input datasets.

Example:
<code xml>	
<input from_module="0">
  <resources>
    <resource name="hcs_plate" value="(^i)([^_]+)(_m)([^_]+)(_b)([^_]+)(_u)([^_]+)" />
  </resources>
</input>
  ----> 
<workflow>
  <input>
    <datasets>
      <dataset name="hcs_plate" id="${RawImages}" type="HCS_IMAGE_RAW" stage="true" />
    </datasets>
  </input>...
</code>


   * Substitute all the ''<input from_module="n">''  where n > 0, through an output dataset in the n-module

Example:
<code xml>
<input from_module="1">
  <resources>
    <resource name="cpv1_create_batch_file_output_directory" value="Cpv1BatchFile" />
  </resources>
</input>
 ---->  
<output>
  <datasets>
    <dataset name="Cpv1BatchFile" store="false" />...
</code>

   * Substitute each ''<storage><dataset_type>'' element through a corresponding ''<output><datasets><dataset>'' element in the same module.
Example:
<code xml>
<storage enabled="true">
  <dataset_type name="CLUSTER_JOB_LOGS">
    <files in_dir="" regex_pattern=".*\.STDERR$" out_dir="" />
    <files in_dir="" regex_pattern=".*\.STDOUT$" out_dir="" />
  </dataset_type>
</storage> 
  ---->  
<output>
  <datasets>
    <dataset type="CLUSTER_JOB_LOGS" store="true">
      <files in_dir="" regex=".*\.stderr" />
      <files in_dir="" regex=".*\.stdout" />
    </dataset>
</code>

   * Substitute ''<output_checks mode="task_level">'' through ''<validations level="task">,'' and ''<output_checks mode="module_level">'' through <''validations level="module">'', and move them into the ''<output>'' element, after the ''<datasets>'' definition.

   * In the ''<validations>'' sections substitute:
     * ''<validate>'' by ''<validation>''
     * delete the ''resource_type="xxx"'' attributes
     * ''resource_regex="{IB2_STDOUT_LOG}"'' by ''regex="${task.log_stdout}"''
     * ''resource_regex="{IB2_STDERR_LOG}"'' by ''regex="${task.log_stderr}"''
     * remaining ''resource_regex="xxx"'' by ''regex="xxx"''
     * ''comparator="is_equal_to"'' by ''comparator="equal"''
     * ''comparator="is_superior_to"'' by ''comparator="greater"''  (other allowed comparators are: less, not_equal, greater_or_equal, less_or_equal)
     * ''pattern_regex'' by ''content_regex''
     * ''fail_status="VALIDATION_ERROR"'' by ''fail_status="validation_error"''
     * ''fail_status="VALIDATION_WARNING"'' by ''fail_status="validation_warning"''


:!: Please, check the details in the ''workflow.xml'' specification syntax document: [[:beewm:devel:workflow_specification_syntax|Workflow Template Specification]]
screeningBee Data Analysis Tools

User Tools

Site Tools

Page Tools
