Tools for Computation on Batch Systems

As a successor of the packages 'BatchJobs' and 'BatchExperiments', this package provides a parallel implementation of the Map function for high performance computing systems managed by schedulers 'IBM Spectrum LSF' (<>), 'OpenLava' (<>), 'Univa Grid Engine'/'Oracle Grid Engine' (<>), 'Slurm' (<>), 'TORQUE/PBS' (<>), or 'Docker Swarm' (<>). A multicore and socket mode allow the parallelization on a local machines, and multiple machines can be hooked up via SSH to create a makeshift cluster. Moreover, the package provides an abstraction mechanism to define large-scale computer experiments in a well-organized and reproducible way.

JOSS Publicatoin CRAN Status Badge Build Status Build Status Coverage Status

As a successor of the packages BatchJobs and BatchExperiments, batchtools provides a parallel implementation of Map for high performance computing systems managed by schedulers like Slurm, Sun Grid Engine, OpenLava, TORQUE/OpenPBS, Load Sharing Facility (LSF) or Docker Swarm (see the setup section in the vignette).

Main features:

  • Convenience: All relevant batch system operations (submitting, listing, killing) are either handled internally or abstracted via simple R functions
  • Portability: With a well-defined interface, the source is independent from the underlying batch system - prototype locally, deploy on any high performance cluster
  • Reproducibility: Every computational part has an associated seed stored in a data base which ensures reproducibility even when the underlying batch system changes
  • Abstraction: The code layers for algorithms, experiment definitions and execution are cleanly separated and allow to write readable and maintainable code to manage large scale computer experiments


Install the stable version from CRAN:


For the development version, use devtools:


Next, you need to setup batchtools for your HPC (it will run sequentially otherwise). See the vignette for instructions.

Why batchtools?

The development of BatchJobs and BatchExperiments is discontinued for the following reasons:

  • Maintainability: The packages BatchJobs and BatchExperiments are tightly connected which makes maintenance difficult. Changes have to be synchronized and tested against the current CRAN versions for compatibility. Furthermore, BatchExperiments violates CRAN policies by calling internal functions of BatchJobs.
  • Data base issues: Although we invested weeks to mitigate issues with locks of the SQLite data base or file system (staged queries, file system timeouts, ...), BatchJobs kept working unreliable on some systems with high latency under certain conditions. This made BatchJobs unusable for many users.

BatchJobs and BatchExperiments will remain on CRAN, but new features are unlikely to be ported back. The vignette contains a section comparing the packages.



Please cite the JOSS paper using the following BibTeX entry:

  doi = {10.21105/joss.00135},
  url = {},
  year  = {2017},
  month = {feb},
  publisher = {The Open Journal},
  volume = {2},
  number = {10},
  author = {Michel Lang and Bernd Bischl and Dirk Surmann},
  title = {batchtools: Tools for R to work on batch systems},
  journal = {The Journal of Open Source Software}

Related Software

Contributing to batchtools

This R package is licensed under the LGPL-3. If you encounter problems using this software (lack of documentation, misleading or wrong documentation, unexpected behaviour, bugs, ...) or just want to suggest features, please open an issue in the issue tracker. Pull requests are welcome and will be included at the discretion of the author. If you have customized a template file for your (larger) computing site, please share it: fork the repository, place your template in inst/templates and send a pull request.


batchtools 0.9.11

  • Removed deprecated function chunkIds().
  • New default for argument fs.timeout in the cluster function constructor is 0 (was NA before).
  • Fixed a unit test for OSX.
  • Improved stability and documentation.

batchtools 0.9.10

  • Exported functions findConfFile() and findTemplateFile().
  • Dropped support for providing a template file directly as string. A valid file is now always required.
  • Fixed writing to TMPDIR instead of the R session's temporary directory.

batchtools 0.9.9

  • RDS files are explicitly stored in version 2 to ensure backward compatibility with R versions prior to 3.5.0.
  • Package fs is now used internally for all file system operations.
  • Support for per-site configuration files and templates to be set up by system administrators.
  • The print of getStatus() now includes a time stamp.
  • chunk() now optionally shuffles the ids before chunking.
  • Support for setting per-job resources in submitJobs().
  • Example templates now include resources for blas.threads and omp.threads.
  • Some bug fixes regarding read-only registries.

batchtools 0.9.8

  • Renamed column "memory" in the status table to "mem.used" to avoid name clashes with the resource specification.
  • Exported function assertRegistry().
  • New function unwrap() as alias to flatten(). The latter causes a name clash with package purrr and will be deprecated in a future version.
  • Registries now contain a unique hash which is updated each time the registry is altered. Can be utilized to invalidate caches, e.g. the cache of knitr.

batchtools 0.9.7

  • Added a workaround for a test to be compatible with testthat v2.0.0.
  • Better and more customizable handling of expired jobs in waitForJobs().
  • Package foreach is now supported for nested parallelization as an alternative to parallelMap.
  • Depreciated argument flatten has been removed.
  • New helper function flatten() to manually unnest/unwrap lists in data frames.
  • Removed functions getProblemIds() and getAlgorithmIds(). Instead, you can just access reg$problems or reg$algorithms, respectively.
  • The number of the maximum concurrent jobs can now also be controlled via setting resources.
  • Internal data base changes to speed up some operations. Old registries are updated on first load by loadRegistry().
  • Fixed a bug where the sleep mechanism between queries was not working.
  • Fixed a bug where submit errors on SLURM and TORQUE were not detected as temporary.

batchtools 0.9.6

  • Fixed a bug where the wrong problem was retrieved from the cache. This was only triggered for chunked jobs in combination with an ExperimentRegistry.

batchtools 0.9.5

  • Added a missing routine to upgrade registries created with batchtools prior to v0.9.3.
  • Fixed a bug where the registry could not be synced if jobs failed during initialization (#135).
  • The sleep duration for waitForJobs() and submitJobs() can now be set via the configuration file.
  • A new heuristic will try to detect if the registry has been altered by a simultaneously running R session. If this is detected, the registry in the current session will be set to a read-only state.
  • waitForJobs() has been reworked to allow control over the heuristic to detect expired jobs. Jobs are treated as expired if they have been submitted but are not detected on the system for expire.after iterations (default 3 iterations, before 1 iteration).
  • New argument writeable for loadRegistry() to allow loading registries explicitly as read-only.
  • Removed argument update.paths from loadRegistry(). Paths are always updated, but the registry on the file system remains unchanged unless loaded in read-write mode.
  • ClusterFunctionsSlurm now come with an experimental nodename argument. If set, all communication with the master is handled via SSH which effectively allows you to submit jobs from your local machine instead of the head node. Note that mounting the file system (e.g., via SSHFS) is mandatory.

batchtools 0.9.4

  • Fixed handling of file.dir with special chars like whitespace.
  • All backward slashes will now be converted to forward slashes on windows.
  • Fixed order of arguments in findExperiments() (argument ids is now first).
  • Removed code to upgrade registries created with versions prior to v0.9.0 (first CRAN release).
  • addExperiments() now warns if a design is passed as data.frame with factor columns and stringsAsFactors is TRUE.
  • Added functions setJobNames() and getJobNames() to control the name of jobs on batch systems. Templates should be adapted to use instead of job.hash for naming.
  • Argument flatten of getJobResources(), getJobPars() and getJobTable() is deprecated and will be removed. Future versions of the functions will behave like flatten is set to FALSE explicitly. Single resources/parameters must be extracted manually (or with tidyr::unnest()).

batchtools 0.9.3

  • Running jobs now are also included while querying for status "started". This affects findStarted(), findNotStarted() and getStatus().
  • findExperiments() now performs an exact string match (instead of matching substrings) for patterns specified via and For substring matching, use prob.pattern or algo.pattern, respectively.
  • Changed arguments for reduceResultsDataTable()
    • Removed fill, now is always TRUE
    • Introduced flatten to control if the result should be represented as a column of lists or flattened as separate columns. Defaults to a backward-compatible heuristic, similar to getJobPars.
  • Improved heuristic to lookup template files. Templates shipped with the package can now be used by providing just the file name (w/o extension).
  • Updated CITATION

batchtools 0.9.2

  • Full support for array jobs on Slurm and TORQUE.
  • Array jobs have been disabled for SGE and LSF (due to missing information about the output format) but will be re-enable in a future release. Note that the variable has been removed from JobCollection in favor of the new variable (logical).
  • findExperiments() now has two additional arguments to match using regular expressions. The possibility to prefix a string with "~" to enable regular expression matching has been removed.
  • New function batchReduce().
  • New function estimateRuntimes().
  • New function removeRegistry().
  • Missing result files are now handled more consistently, raising an exception in its defaults if the result is not available. The argument missing.val has been added to reduceResultsList() and reduceResultsDataTable() and removed from loadResult() and batchMapResults().
  • Timestamps are now stored with sub-second accuracy.
  • Renamed Torque to TORQUE. This especially affects the constructor makeClusterFunctionsTorque which now must be called via makeClusterFunctionsTORQUE()
  • chunkIds() has been deprecated. Use chunk(), lpt() or binpack() instead.
  • Fixed listing of jobs for ClusterFunctionsLSF and ClusterFunctionsOpenLava (thanks to @phaverty).
  • Job hashes are now prefixed with the literal string 'job' to ensure they start with a letter as required by some SGE systems.
  • Fixed handling of NULL results in reduceResultsList()
  • Fixed key lookup heuristic join functions.
  • Fixed a bug where getJobTable() returned difftimes with the wrong unit (e.g., in minutes instead of seconds).
  • Deactivated swap allocation for ClusterFunctionsDocker.
  • The package is now more patient while communicating with the scheduler or file system by using a timeout-based approach. This should make the package more reliable and robust under heavy load.

batchtools 0.9.0

Initial CRAN release. See the vignette for a brief comparison with BatchJobs/BatchExperiments.

Reference manual

It appears you don't have a PDF plugin for this browser. You can click here to download the reference manual.