User Tools

Site Tools


beewm:devel:feature_list

Differences

This shows you the differences between two versions of the page.

Link to this comparison view

Both sides previous revisionPrevious revision
Next revision
Previous revision
devel:feature_list [2013/07/25 10:34] epujadasbeewm:devel:feature_list [2016/05/17 16:17] (current) – external edit 127.0.0.1
Line 2: Line 2:
  
 ===== Use existing storage datasets as input for processing ===== ===== Use existing storage datasets as input for processing =====
-See: [[devel:workflow_dataset_combination|Workflow Dataset Combination]]+See: \\ [[:beewm:devel:web_interface|User Interface]] \\ [[:beewm:devel:workflow_dataset_combination|Workflow Dataset Combination]]
  
 ==== Concept ==== ==== Concept ====
Line 17: Line 17:
  
 ===== Use single dataset(s) as input for processing (for example CP pipeline) ===== ===== Use single dataset(s) as input for processing (for example CP pipeline) =====
-See: [[devel:workflow_dataset_combination|Workflow Dataset Combination]] +See: \\ [[:beewm:devel:web_interface|User Interface]] \\ [[:beewm:devel:workflow_dataset_combination|Workflow Dataset Combination]]
  
 This is specifically a request for the user interface. When starting a specific workflow This is specifically a request for the user interface. When starting a specific workflow
Line 38: Line 37:
  
 ===== Deduce which results need to be computed automatically ===== ===== Deduce which results need to be computed automatically =====
-See: [[devel:dataset_equivalence_classes|Dataset Equivalence Classes]]+See: [[:beewm:devel:dataset_equivalence_classes|Dataset Equivalence Classes]]
  
 ==== Concept ==== ==== Concept ====
Line 66: Line 65:
  
 ===== Tight Acquisition Integration ===== ===== Tight Acquisition Integration =====
-See: [[devel:tight_acquisition_integration|Tight Acquisition Integration]]+See: [[:beewm:devel:tight_acquisition_integration|Tight Acquisition Integration]]
  
 Make use the time of the acquisition. There are steps should be performed per image Make use the time of the acquisition. There are steps should be performed per image
Line 104: Line 103:
  
 ===== Automatic resolution of parallelism (run parallel whatever possible) ===== ===== Automatic resolution of parallelism (run parallel whatever possible) =====
-See: [[devel:dependency_resolution|Dependency Resolution]]+See: [[:beewm:devel:dependency_resolution|Dependency Resolution]]
  
 ===== Trigger processing when data becomes available ===== ===== Trigger processing when data becomes available =====
-See: [[devel:trigger_of_processings|Trigger of Processings]]+See: [[:beewm:devel:trigger_of_processings|Trigger of Processings]]
  
 ====== Staging, Resource Handling and Cluster Integration ====== ====== Staging, Resource Handling and Cluster Integration ======
  
 ===== Clever handling of resource bottlenecks (i.e. never stage two plates at once, stage smaller datasets first, ...) ===== ===== Clever handling of resource bottlenecks (i.e. never stage two plates at once, stage smaller datasets first, ...) =====
-See: [[devel:resource_optimization|Resource Optimization]]+See: [[:beewm:devel:resource_optimization|Resource Optimization]]
  
 Typical resource bottlenecks and rules for clever handling of resource bottlenecks are: Typical resource bottlenecks and rules for clever handling of resource bottlenecks are:
Line 126: Line 125:
  
 ===== Allow manual prioritization ===== ===== Allow manual prioritization =====
-See: [[devel:prioritization|Prioritization]]+See: [[:beewm:devel:prioritization|Prioritization]]
  
   - Allow setting initial priority for each processing   - Allow setting initial priority for each processing
Line 132: Line 131:
  
 ===== Prioritize processing by Resource Bottlenecks ===== ===== Prioritize processing by Resource Bottlenecks =====
-See: [[devel:prioritization|Prioritization]]+See: [[:beewm:devel:prioritization|Prioritization]]
  
   - Prefer processings for which more datasets are already staged   - Prefer processings for which more datasets are already staged
Line 141: Line 140:
  
 ===== Stage repeatedly used data only once ===== ===== Stage repeatedly used data only once =====
-See: [[devel:cluster_cache|Reusable Datasets in Cluster]]+See: [[:beewm:devel:cluster_cache|Reusable Datasets in Cluster]]
  
 Maintain information about the pool of datasets on the cluster. Maintain information about the pool of datasets on the cluster.
  
 ===== Free cluster scratch space immediately after use ===== ===== Free cluster scratch space immediately after use =====
-See: [[devel:cluster_cache|Reusable Datasets in Cluster]]+See: [[:beewm:devel:cluster_cache|Reusable Datasets in Cluster]]
  
 Delete data from the pool of datasets on the cluster as soon as the dataset Delete data from the pool of datasets on the cluster as soon as the dataset
Line 152: Line 151:
  
 ===== During Startup Recover or Kill Running Cluster Jobs ===== ===== During Startup Recover or Kill Running Cluster Jobs =====
-See: [[devel:cluster_job_management_and_visualization|Cluster Job Management and Visualization]]+See: [[:beewm:devel:cluster_job_management_and_visualization|Cluster Job Management and Visualization]]
  
 Make sure that when the workflow manager starts, it either kills all running Make sure that when the workflow manager starts, it either kills all running
Line 158: Line 157:
  
 ===== Stop running processes (include killing of jobs) ===== ===== Stop running processes (include killing of jobs) =====
-See: [[devel:cluster_job_management_and_visualization|Cluster Job Management and Visualization]]+See: [[:beewm:devel:cluster_job_management_and_visualization|Cluster Job Management and Visualization]]
  
 Allow stopping of jobs, including forced stop where cluster processes Allow stopping of jobs, including forced stop where cluster processes
Line 164: Line 163:
  
 ===== Optimize number of jobs for Cluster Queuing System ===== ===== Optimize number of jobs for Cluster Queuing System =====
-See: [[devel:cluster_queue_optimization|Cluster Queue Optimization]]+See: [[:beewm:devel:cluster_queue_optimization|Cluster Queue Optimization]]
  
 Avoid overloading the cluster queuing system by starting unnecessary large Avoid overloading the cluster queuing system by starting unnecessary large
Line 180: Line 179:
  
 ===== Automatic Restart or Email notification in case of daemon crash ===== ===== Automatic Restart or Email notification in case of daemon crash =====
-See: [[devel:watchdog_and_user_notification|Watchdog and User Notification]]+See: [[:beewm:devel:watchdog_and_user_notification|Watchdog and User Notification]]
  
 A watchdog process could monitor the current status and responsiveness of the A watchdog process could monitor the current status and responsiveness of the
Line 189: Line 188:
  
 ===== Email notification for all possible error conditions ===== ===== Email notification for all possible error conditions =====
-See: [[devel:watchdog_and_user_notification|Watchdog and User Notification]]+See: [[:beewm:devel:watchdog_and_user_notification|Watchdog and User Notification]]
  
 ==== Concept ==== ==== Concept ====
Line 206: Line 205:
  
 ===== Avoid Database duplication ===== ===== Avoid Database duplication =====
 +See: [[:beewm:devel:combining_processing_and_storage_information|Combining Processing Information with Storage Information]]
 +
 Many properties of datasets can and should be annotated with the datasets in the Many properties of datasets can and should be annotated with the datasets in the
 storage. For these properties, the authority should be the storage and not the storage. For these properties, the authority should be the storage and not the
Line 225: Line 226:
  
 ===== Resolve separation of workflows and module configuration ===== ===== Resolve separation of workflows and module configuration =====
-See: [[devel:workflow_specification_syntax|Workflow Specification Syntax]]+See: [[:beewm:devel:workflow_specification_syntax|Workflow Specification Syntax]]
  
 Modules are partially configured in: Modules are partially configured in:
Line 233: Line 234:
  
 ===== Simple workflow description ===== ===== Simple workflow description =====
-See: [[devel:workflow_specification_syntax|Workflow Specification Syntax]]+See: [[:beewm:devel:workflow_specification_syntax|Workflow Specification Syntax]]
  
 ===== Clean (chroot) Cluster environment for modules ===== ===== Clean (chroot) Cluster environment for modules =====
  
-See: [[devel:chroot-like_cluster_environment|CHROOT-like Cluster Environment]]+See: [[:beewm:devel:chroot-like_cluster_environment|CHROOT-like Cluster Environment]]
  
   - Executed modules should not be able to overwrite installation files   - Executed modules should not be able to overwrite installation files
beewm/devel/feature_list.1374741262.txt.gz · Last modified: 2013/07/25 10:34 by epujadas