User Tools

Site Tools


beewm:devel:chroot-like_cluster_environment

CHROOT-like Cluster Environment

It is important to make sure that the following data hazzard conditions are avoided:

  1. Executed modules should not be able to overwrite installation files
  2. Executed modules should not overwrite output files from previous executions or other modules

This can be achieved by a two-fold process:

Executing Modules as a Dedicated User with Limited Permissions

Executing modules as a user that has write permissions only to result files ensures that accidental overwrites of installation files, executables and the user home directory will not take place.

Setting a strict CHROOT-like Environment

Setting a strict environment of directories and symlinks individually for each process makes sure that each process can not by default see files of other processes. Note that for example, current iBRAIN2 supports re-executing a job that failed before, and that iBRAIN2 does not guarantee that each repeated execution has a clean environment. This can lead to problems when the cluster job tries to overwrite or merge output files.

Ideas from iPortal

  • There is a modul definition and usage solution on Brutus, users/user groups can have their own modules and this can be loaded via the module load command. This is used to install all the necessary tools, binaries, scripts, environment settings for the proteomics portal. There is a module called imsbtools this loads other modules (like jython, tpp,..) set up environment variables and also extend the PATH to make new tools available. Only administrators can change or overwrite this settings/ files.
  • iPortal runs each job under the user who submitted the processing. For this we need user management then we can generate keys which can be connected to the users portal account and can be used to log in to the cluster. In this way users cannot overwrite files from others.
beewm/devel/chroot-like_cluster_environment.txt ยท Last modified: 2013/07/24 18:11 by epujadas