Evaluate Function Calls on HPC Schedulers (LSF, SGE, SLURM, PBS/Torque)

Evaluate arbitrary function calls using workers on HPC schedulers in single line of code. All processing is done on the network without accessing the file system. Remote schedulers are supported via SSH.


  • 0.8.6

    • Progress bar is now shown before any workers start (#107)
    • Socket connections are now authenticated using a session password (#125)
    • Marked internal functions with @keywords internal
    • Added vignettes for the User Guide and Technical Documentation
  • 0.8.5

    • Added experimental support as parallel foreach backend (#83)
    • Moved templates to package inst/ directory (#85)
    • Added send_call to worker to evaluate arbitrary expressions (drake#501; #86)
    • Option clustermq.scheduler is now respected if set after package load (#88)
    • System interrupts are now handled correctly (rzmq#44; #73, #93, #97)
    • Number of workers running/total is now shown in progress bar (#98)
    • Unqualified (short) host names are now resolved by default (#104)
  • 0.8.4

    • Fix error for qsys$reusable when using n_jobs=0/local processing (#75)
    • Scheduler-specific templates are deprecated. Use clustermq.template instead
    • Allow option clustermq.defaults to fill default template values (#71)
    • Errors in worker processing are now shut down cleanly (#67)
    • Progress bar now shows estimated time remaining (#66)
    • Progress bar now also shown when processing locally
    • Memory summary now adds estimated memory of R session (#69)
  • 0.8.3

    • Support rettype for function calls where return type is known (#59)
    • Reduce memory requirements by processing results when we receive them
    • Fix a bug where cleanup, log_worker flag were not working for SGE/SLURM
  • 0.8.2

    • Fix a bug where never-started jobs are not cleaned up
    • Fix a bug where tests leave processes if port binding fails (#60)
    • Multicore no longer prints worker debug messages (#61)
  • 0.8.1

    • Fix performance issues for a high number of function calls (#56)
    • Fix bug where multicore workers were not shut down properly (#58)
    • Fix default templates for SGE, LSF and SLURM (misplaced quote)
  • 0.8.0

    • Templates changed: clustermq:::worker now takes only master as argument
    • Fix a bug where copies of common_data are collected by gc too slowly (#19)
    • Creating workers is now separated from Q, enabling worker reuse (#45)
    • Objects in the function environment must now be exported explicitly (#47)
    • Messages on the master are now processed in threads (#42)
    • Added multicore qsys using the parallel package (#49)
    • New function Q_rows using data.frame rows as iterated arguments (#43)
    • Jobs will now be submitted as array if possible
    • Job summary will now report max memory as reported by gc (#18)
  • 0.7.0

    • Initial release on CRAN

Reference manual

It appears you don't have a PDF plugin for this browser. You can click here to download the reference manual.

install.packages("clustermq") by Michael Schubert, a year ago


Report a bug at https://github.com/mschubert/clustermq/issues

Browse source code at https://github.com/cran/clustermq

Authors: Michael Schubert [aut, cre]

Documentation:   PDF Manual  

Task views: High-Performance and Parallel Computing with R

Apache License (== 2.0) | file LICENSE license

Imports methods, narray, progress, purrr, R6, Rcpp, utils

Suggests callr, devtools, dplyr, foreach, iterators, knitr, parallel, rmarkdown, roxygen2, testthat, tools

Linking to Rcpp

System requirements: C++11, ZeroMQ (libzmq)

Suggested by drake, raveio, targets.

See at CRAN