This shows you the differences between two versions of the page.
Both sides previous revisionPrevious revisionNext revision | Previous revision | ||
devel:feature_list [2013/07/25 10:34] – epujadas | beewm:devel:feature_list [2016/05/17 16:17] (current) – external edit 127.0.0.1 | ||
---|---|---|---|
Line 2: | Line 2: | ||
===== Use existing storage datasets as input for processing ===== | ===== Use existing storage datasets as input for processing ===== | ||
- | See: [[devel: | + | See: \\ [[: |
==== Concept ==== | ==== Concept ==== | ||
Line 17: | Line 17: | ||
===== Use single dataset(s) as input for processing (for example CP pipeline) ===== | ===== Use single dataset(s) as input for processing (for example CP pipeline) ===== | ||
- | See: [[devel: | + | See: \\ [[: |
This is specifically a request for the user interface. When starting a specific workflow | This is specifically a request for the user interface. When starting a specific workflow | ||
Line 38: | Line 37: | ||
===== Deduce which results need to be computed automatically ===== | ===== Deduce which results need to be computed automatically ===== | ||
- | See: [[devel: | + | See: [[:beewm:devel: |
==== Concept ==== | ==== Concept ==== | ||
Line 66: | Line 65: | ||
===== Tight Acquisition Integration ===== | ===== Tight Acquisition Integration ===== | ||
- | See: [[devel: | + | See: [[:beewm:devel: |
Make use the time of the acquisition. There are steps should be performed per image | Make use the time of the acquisition. There are steps should be performed per image | ||
Line 104: | Line 103: | ||
===== Automatic resolution of parallelism (run parallel whatever possible) ===== | ===== Automatic resolution of parallelism (run parallel whatever possible) ===== | ||
- | See: [[devel: | + | See: [[:beewm:devel: |
===== Trigger processing when data becomes available ===== | ===== Trigger processing when data becomes available ===== | ||
- | See: [[devel: | + | See: [[:beewm:devel: |
====== Staging, Resource Handling and Cluster Integration ====== | ====== Staging, Resource Handling and Cluster Integration ====== | ||
===== Clever handling of resource bottlenecks (i.e. never stage two plates at once, stage smaller datasets first, ...) ===== | ===== Clever handling of resource bottlenecks (i.e. never stage two plates at once, stage smaller datasets first, ...) ===== | ||
- | See: [[devel: | + | See: [[:beewm:devel: |
Typical resource bottlenecks and rules for clever handling of resource bottlenecks are: | Typical resource bottlenecks and rules for clever handling of resource bottlenecks are: | ||
Line 126: | Line 125: | ||
===== Allow manual prioritization ===== | ===== Allow manual prioritization ===== | ||
- | See: [[devel: | + | See: [[:beewm:devel: |
- Allow setting initial priority for each processing | - Allow setting initial priority for each processing | ||
Line 132: | Line 131: | ||
===== Prioritize processing by Resource Bottlenecks ===== | ===== Prioritize processing by Resource Bottlenecks ===== | ||
- | See: [[devel: | + | See: [[:beewm:devel: |
- Prefer processings for which more datasets are already staged | - Prefer processings for which more datasets are already staged | ||
Line 141: | Line 140: | ||
===== Stage repeatedly used data only once ===== | ===== Stage repeatedly used data only once ===== | ||
- | See: [[devel: | + | See: [[:beewm:devel: |
Maintain information about the pool of datasets on the cluster. | Maintain information about the pool of datasets on the cluster. | ||
===== Free cluster scratch space immediately after use ===== | ===== Free cluster scratch space immediately after use ===== | ||
- | See: [[devel: | + | See: [[:beewm:devel: |
Delete data from the pool of datasets on the cluster as soon as the dataset | Delete data from the pool of datasets on the cluster as soon as the dataset | ||
Line 152: | Line 151: | ||
===== During Startup Recover or Kill Running Cluster Jobs ===== | ===== During Startup Recover or Kill Running Cluster Jobs ===== | ||
- | See: [[devel: | + | See: [[:beewm:devel: |
Make sure that when the workflow manager starts, it either kills all running | Make sure that when the workflow manager starts, it either kills all running | ||
Line 158: | Line 157: | ||
===== Stop running processes (include killing of jobs) ===== | ===== Stop running processes (include killing of jobs) ===== | ||
- | See: [[devel: | + | See: [[:beewm:devel: |
Allow stopping of jobs, including forced stop where cluster processes | Allow stopping of jobs, including forced stop where cluster processes | ||
Line 164: | Line 163: | ||
===== Optimize number of jobs for Cluster Queuing System ===== | ===== Optimize number of jobs for Cluster Queuing System ===== | ||
- | See: [[devel: | + | See: [[:beewm:devel: |
Avoid overloading the cluster queuing system by starting unnecessary large | Avoid overloading the cluster queuing system by starting unnecessary large | ||
Line 180: | Line 179: | ||
===== Automatic Restart or Email notification in case of daemon crash ===== | ===== Automatic Restart or Email notification in case of daemon crash ===== | ||
- | See: [[devel: | + | See: [[:beewm:devel: |
A watchdog process could monitor the current status and responsiveness of the | A watchdog process could monitor the current status and responsiveness of the | ||
Line 189: | Line 188: | ||
===== Email notification for all possible error conditions ===== | ===== Email notification for all possible error conditions ===== | ||
- | See: [[devel: | + | See: [[:beewm:devel: |
==== Concept ==== | ==== Concept ==== | ||
Line 206: | Line 205: | ||
===== Avoid Database duplication ===== | ===== Avoid Database duplication ===== | ||
+ | See: [[: | ||
+ | |||
Many properties of datasets can and should be annotated with the datasets in the | Many properties of datasets can and should be annotated with the datasets in the | ||
storage. For these properties, the authority should be the storage and not the | storage. For these properties, the authority should be the storage and not the | ||
Line 225: | Line 226: | ||
===== Resolve separation of workflows and module configuration ===== | ===== Resolve separation of workflows and module configuration ===== | ||
- | See: [[devel: | + | See: [[:beewm:devel: |
Modules are partially configured in: | Modules are partially configured in: | ||
Line 233: | Line 234: | ||
===== Simple workflow description ===== | ===== Simple workflow description ===== | ||
- | See: [[devel: | + | See: [[:beewm:devel: |
===== Clean (chroot) Cluster environment for modules ===== | ===== Clean (chroot) Cluster environment for modules ===== | ||
- | See: [[devel: | + | See: [[:beewm:devel: |
- Executed modules should not be able to overwrite installation files | - Executed modules should not be able to overwrite installation files |