Prioritization

Different processing workflows or the processing of different datasets can have different importance. We should support this on the fine enough level.

iBrain2 has a very simple prioritization solution. Processes have a priority value which is set 5 by default, but can be changed via rest. The process with higher priority value will be scheduled earlier.

We can define priority values for each important factors of the submission process:

• User
• Workflow
• Dataset (or maybe more fine grained, assay, project, dataset,…)

Of coures each item can have a default priority.
The priority values can be additive, so the priority of the running proces would be the sum of the above listed priorities.

• Prioritization should affect the priority of the submitted cluster job too
• As the dataset related information most probably would come from the storage this would mean the priority of the datasets would be stored there. (as an optional attribuite of the dataset)
• Maybe the prioritization policy should be also configurable (maybe specialized implementation fits better in some cases).
• There was an idea that the priority of the process should also be dependent on the size of the input data to prioritize the shorter processing. This could be also feasible, but much complicated. What's the size of the dataset (files or bytes)? It isn't too expensive to compute the size of the dataset? How to implement this policy not to lead to starvation of large datasets (with some randomness)?

1. Allow setting initial priority for each processing
2. Allow dynamic changing of priority for each processing

1. Prefer processings for which more datasets are already staged
2. Prefer processing for which smaller amounts of data need to be staged
3. Prefer processings after which larger amounts of data can be deleted
4. Prefer processings of workflows with fewer remaining steps
5. Prefer processings which are quicker