User Tools

Site Tools


beewm:devel:resource_optimization

Resource Optimization

The cluster workflow manager should employ clever handling of resource bottlenecks.

Typical resource bottlenecks and rules for clever handling of resource bottlenecks are:

Network transfer of data

  1. staging of smaller datasets before larger datasets (log10-scale)
  2. do not stage more than N datasets at once (for N = 2 or 3)

Hard disk reading/writing

  1. do not move more than N datasets at once (for N = 2 or 3)
  2. use move operations for input/output style processings (on the fly processing during copy)

Cluster queuing system

beewm/devel/resource_optimization.txt ยท Last modified: 2016/05/17 16:17 by 127.0.0.1