Iterated Racing for Automatic Algorithm Configuration

Iterated race is an extension of the Iterated F-race method for the automatic configuration of optimization algorithms, that is, (offline) tuning their parameters by finding the most appropriate settings given a set of instances of an optimization problem.


CRAN_Status_Badge CRAN Downloads

Maintainers: Manuel López-Ibáñez, Leslie Pérez Cáceres

Creators: Manuel López-Ibáñez, Jérémie Dubois-Lacoste

Contributors: Jérémie Dubois-Lacoste, Thomas Stützle, Mauro Birattari, Eric Yuan and Prasanna Balaprakash.

Contact: https://groups.google.com/d/forum/irace-package

Introduction

The irace package implements the Iterated Race method, which is a generalization of the Iterated F-race method for the automatic configuration of optimization algorithms, that is, the tuning of their parameters by finding the most appropriate settings given a set of instances of an optimization problem. It builds upon the race package by Birattari and it is implemented in R.

Keywords: automatic configuration, offline tuning, parameter tuning, racing, F-race.

Relevant literature:

  1. M. López-Ibáñez, J. Dubois-Lacoste, L. Pérez Cáceres, T. Stützle, and M. Birattari. The irace package: Iterated Racing for Automatic Algorithm Configuration.. Operations Research Perspectives, 3:43–58, 2016.
    [ bibtex | doi:10.1016/j.orp.2016.09.002 ]

  2. Manuel López-Ibáñez, Jérémie Dubois-Lacoste, Thomas Stützle, and Mauro Birattari. The irace package, Iterated Race for Automatic Algorithm Configuration. Technical Report TR/IRIDIA/2011-004, IRIDIA, Université libre de Bruxelles, Belgium, 2011.
    [ bibtex | PDF ]

  3. Manuel López-Ibáñez. The irace software package: A tutorial. COMEX Workshop on Practical Automatic Algorithm Configuration, 2014.
    [ workshop webpage | PDF ]

Requisites

  • R (https://www.r-project.org) is required for running irace, but you don't need to know the R language to use it. Versions that work: >= 2.15.0

User guide

A complete user guide comes with the package. You can access it online or, after installing the irace package, invoking from R the following command:

        R> vignette("irace-package")

The following is a quick-start guide. The user guide gives more detailed instructions.

Installing R

The official instructions are available at https://cran.r-project.org/doc/manuals/r-release/R-admin.html. We give below a quick R installation guide that will work in most cases.

GNU/Linux

You should install R from your package manager. On a Debian/Ubuntu system it will be something like:

$ sudo apt-get install r-base

Once R is installed, you can launch R from the Terminal and from the R prompt install the irace package. See instructions below.

OS X

You can install R directly from a CRAN mirror (https://cran.r-project.org/bin/macosx/).

Alternatively, if you use homebrew, you can just brew the R formula from the science tap (unfortunately it does not come already bottled so you need to have Xcode installed to compile it):

    $ brew tap homebrew/science
    $ brew install r

Once R is installed, you can launch R from the Terminal (or from your Applications), and from the R prompt install the irace package. See instructions below.

Windows

You can install R from a CRAN mirror (https://cran.r-project.org/bin/windows/). Once R is installed, you can launch the R console and install the irace package from it. See instructions below.

Installing the irace package

There are two methods for installing the irace R package on your computer:

  1. Install within R (automatic download):
        $ R
        R> install.packages("irace")

select a mirror close to you, and test the installation with

        R> library(irace)
        R> CTRL+d
  1. Manually download the package from CRAN and invoke at the command-line:
        $ R CMD INSTALL <package>

where <package> is one of the three versions available: .tar.gz (Unix/BSD/GNU/Linux), .tgz (MacOS X), or .zip (Windows).

If the package fails to install because of insufficient permissions, you need to force a local installation by doing:

    $ mkdir ~/R
    $ R CMD INSTALL --library=~/R irace.tar.gz
    $ export R_LIBS=~/R:${R_LIBS}

Once installed, test that it is working by doing:

    $ R
    R> library(irace)
    R> system.file(package="irace")
    [1] "~/R/irace"

The last command tells you the installation directory of irace.

GNU/Linux and OS X

Save the installation directory of irace to a variable, and add it to your .bash_profile, .bashrc or .profile:

    export IRACE_HOME=~/R/irace/ # Path given by system.file(package="irace") 
    export PATH=${IRACE_HOME}/bin/:$PATH

After adding this and opening a new terminal, you should be able to invoke irace as follows:

    $ irace --help

Windows

If the installation directory of irace is C:/R/irace/, you can invoke irace by opening a terminal (launch the program cmd.exe) and executing:

    C:\> C:\R\irace\bin\irace.bat --help

You can also launch irace by opening the R console and executing:

    R> library(irace)
    R> irace.cmdline("--help")

Usage

  1. Create a directory for storing the tuning scenario setup
        $ mkdir ~/tuning
        $ cd ~/tuning
  1. Copy the template and example files to the scenario directory
        $ cp $IRACE_HOME/templates/*.tmpl .
where `$IRACE_HOME` is the path to the installation directory of
`irace`. It can be obtained by doing:
        $ R
        > system.file(package="irace")
  1. For each template in your tuning directory, remove the .tmpl suffix, and modify them following the instructions in each file. In particular,

    • The scripts target-runner and target-evaluator (if you need it at all) should be executable. The output of target-runner (or target-evaluator if you use a separate evaluation step) is minimized by default. If you wish to maximize it, just multiply the value by -1 within the script.
    • In scenario.txt, uncomment and assign only the parameters for which you need a value different than the default one.

    There are examples in $IRACE_HOME/examples/.

  2. Put the instances in ~/tuning/Instances/. In addition, you can create a file that specifies which instances from that directory should be run and which instance-specific parameters to use. See scenario.txt.tmpl and instances-list.tmpl for examples. The command irace will not attempt to create the execution directory (execDir), so it must exist before calling irace. The default execDir is the current directory.

  3. Calling the command:

        $ cd ~/tuning/ && $IRACE_HOME/bin/irace
performs one run of Iterated Race. See the output of `irace --help` for
additional irace parameters. Command-line parameters override the
scenario setup specified in the `scenario.txt` file.

Many tuning runs in parallel

For executing several repetitions of irace in parallel, call the program

    $ cd ~/tuning/ && $IRACE_HOME/bin/parallel-irace N

where N is the number of repetitions. By default, the execution directory of each run of irace will be set to ./execdir-dd, where dd is a number padded with zeroes.

Be careful, parallel-irace will create these directories from scratch, deleting them first if they already exist.

Check the help of parallel-irace by running it without parameters.

Parallelize one tuning

A single run of irace can be done much faster by executing the calls to targetRunner (the runs of the algorithm being tuned) in parallel. See the user guide for the details.

Frequently Asked Questions

The user guide contains a list of frequently asked questions.

News

2.4

  • The output of irace now specifies in which order, if any, configurations are printed. (Manuel L�pez-Ib��ez, suggested by Markus Wagner)

  • Several fixes for handling paths in Windows. (Manuel L�pez-Ib��ez)

  • readConfigurationsFile() now has a text= argument, which allows reading configurations from a string. (Manuel L�pez-Ib��ez)

  • User-provided functions (targetRunner, targetEvaluator and repairConfiguration) and user-provided conditions for forbidden configurations are now byte-compiled when read, which should make their evaluation noticeably faster. (Manuel L�pez-Ib��ez)

  • The argument 'experiment' passed to the R function targetRunner does not contain anymore an element 'extra.params'. Similarly, the 'scenario' structure does not contain anymore the elements 'instances.extra.params' and 'testInstances.extra.params'. Any instance-specific parameters values now form part of the character string that defines an instance and it is up to the user-defined targetRunner to parse them appropriately. These changes make no difference when targetRunner is an external script, or when instances and instance-specific parameter values are read from a file. (Manuel L�pez-Ib��ez)

2.3

  • Fix bug that will cause iraceResults$experimentLog to count calls to targetEvaluator as experiments, even if no call to targetRunner was performed. This does not affect the computation of the budget consumed and, thus, it does not affect the termination criteria of irace. The bug triggers an assertion that terminates irace, thus no run that was successful with version 2.2 is affected. (Manuel L�pez-Ib��ez)

2.2

  • Command-line parameters are printed to stdout (useful for future replications). (Manuel L�pez-Ib��ez, suggested by Markus Wagner)

  • Users may provide a function to repair configurations before being evaluated. See the scenario variable repairConfiguration. (Manuel L�pez-Ib��ez)

  • The option --sge-cluster (sgeCluster) was removed and replaced by --batchmode (batchmode). It is now the responsibility of the target-runner to parse the output of the batch job submission command (e.g., qsub or squeue), and return just the job ID. Values supported are: "sge", "torque", "pbs" and "slurm". (Manuel L�pez-Ib��ez)

  • The option --parallel can now be combined with --batchmode to limit the number of jobs submitted by irace at once. This may be useful in batch clusters that have a small queue of jobs. (Manuel L�pez-Ib��ez)

  • New examples under inst/examples/batchmode-cluster/. (Manuel L�pez-Ib��ez)

  • It is now possible to include scenario definition files from other scenario files by using:

    eval.parent(source("scenario-common.txt", chdir = TRUE, local = TRUE))

    This feature is VERY experimental and the syntax is likely to change in the future. (Manuel L�pez-Ib��ez)

  • Fix a bug that re-executed elite results under some circumstances. (Leslie P�rez C�ceres)

  • Restrict the number of maximum configurations per race to 1024. (Leslie P�rez C�ceres)

  • Do not warn if the last line in the instance file does not terminate with a newline. (Manuel L�pez-Ib��ez)

  • Fix bug when deterministic == 1. (Manuel L�pez-Ib��ez, Leslie P�rez C�ceres)

  • Update manual and vignette with details about the expected arguments and return value of targetRunner and targetEvaluator. (Manuel L�pez-Ib��ez)

  • Many updates to the User Guide vignette. (Manuel L�pez-Ib��ez)

  • Fix \dontrun example in irace-package.Rd (Manuel L�pez-Ib��ez)

  • Fix bug: If testInstances contains duplicates, results of testing are not correctly saved in iraceResults$testing$experiments nor reported correctly at the end of a run. Now unique IDs of the form 1t, 2t, ... are used for each testing instance. These IDs are used for the rownames of iraceResults$testing$experiments and the names of the scenario$testInstances and iraceResults$testing$seeds vectors. (Manuel L�pez-Ib��ez)

  • Fix bug where irace keeps retrying the target-runner call even if it succeeds. (Manuel L�pez-Ib��ez)

  • New command-line parameter

    --only-test FILE
    

    which just evaluates the configurations given in FILE on the testing instances defined by the scenario. Useful if you decide on the testing instances only after running irace. (Manuel L�pez-Ib��ez)

  • Bugfix: When using maxTime != 0, the number of experiments performed may be miscounted in some cases. (Manuel L�pez-Ib��ez)

2.1

  • Fix CRAN errors in tests. (Manuel L�pez-Ib��ez)

  • Avoid generating too many configurations at once if the initial time estimation is too small. (Manuel L�pez-Ib��ez)

2.0

  • Minimum R version is 2.15.

  • Elitist irace by default, it can be disabled with parameter --elitist 0. (Leslie P�rez C�ceres, Manuel L�pez-Ib��ez)

  • The parameter --test-type gains two additional values:

    t-test-bonferroni (t-test with Bonferroni's correction for multiple comparisons), t-test-holm (t-test with Holm's correction for multiple comparisons)

    (Manuel L�pez-Ib��ez)

  • MPI does not create log files with --debug-level 0. (Manuel L�pez-Ib��ez)

  • For simplicity, the parallel-irace-* scripts do not use an auxiliary tune-main script. For customizing them, make a copy and edit them directly. (Manuel L�pez-Ib��ez)

  • New parameters:

    --target-runner-retries : Retry target-runner this many times in case of error. (Manuel L�pez-Ib��ez)

  • We print diversity measures after evaluating on each instance: (Leslie P�rez C�ceres)

    • Kendall's W (also known as Kendall's coefficient of concordance) If 1, all candidates have ranked in the same order in all instances. If 0, the ranking of each candidate on each instance is essentially random.

          W = Friedman / (m * (k-1))
      
    • Spearman's rho: average (Spearman) correlation coefficient computed on the ranks of all pairs of raters. If there are no repeated data values, a perfect Spearman correlation of +1 or -1 occurs when each of the variables is a perfect monotone function of the other.

  • Many internal and external interfaces have changed. For example, now we consistently use 'scenario' to denote the settings passed to irace and 'configuration' instead of 'candidate' to denote the parameter settings passed to the target algorithm. Other changes are:

    parameters$boundary -> parameters$domain hookRun -> targetRunner hookEvaluate -> targetEvaluator tune-conf -> scenario.txt instanceDir -> trainInstancesDir instanceFile -> trainInstancesFile testInstanceDir -> testInstancesDir testInstanceFile -> testInstancesFile

  • Minimal example of configuring a MATLAB program (thanks to Esteban Diaz Leiva)

  • Paths to files or directories given in the scenario file are relative to the scenario file (except for --log-file, which is an output file and it is relative to --exec-dir). Paths given in the command-line are relative to the current working directory. Given

    $ cat scenario/scenario.txt targetRunner <- "./target-runner" $ irace -s scenario/scenario.txt

    irace will search for "./scenario/target-runner", but given

    $ irace -s scenario/scenario.txt --target-runner ./target-runner

    irace will search for "./target-runner". (Manuel L�pez-Ib��ez)

  • New command-line wrapper for Windows installed at 'system.file("bin/irace.bat", package="irace")' (thanks to Anthony Antoun)

  • Budget can be specified as maximum time (maxTime, --max-time) consumed by the target algorithm. See the documentation for the details about how this is handled. (Leslie P�rez C�ceres, Manuel L�pez-Ib��ez)

1.07

  • The best configurations found, either at the end or at each iteration of an irace run, can now be applied to a set of test instances different from the training instances. See options testInstanceDir, testInstanceFile, testNbElites, and testIterationElites. (Leslie P�rez C�ceres, Manuel L�pez-Ib��ez)

  • The R interfaces of hookRun, hookEvaluate and hookRunParallel have changed. See help(hook.run.default) and help(hook.evaluate.default) for examples of the new interfaces.

  • Printing of race progress now reports the actual configuration and instance IDs, and numbers are printed in a more human-readable format. (Leslie P�rez C�ceres, Manuel L�pez-Ib��ez)

  • Reduce memory use for very large values of maxExperiments.
    (Manuel L�pez-Ib��ez, thanks to Federico Caselli for identifying the issue)

  • New option --load-balancing (loadBalancing) for disabling load-balancing when executing jobs in parallel. Load-balancing makes better use of computing resources, but increases communication overhead. If this overhead is large, disabling load-balancing may be faster. (Manuel L�pez-Ib��ez, thanks to Federico Caselli for identifying the issue)

  • The option --parallel in Windows now uses load-balancing by default. (Manuel L�pez-Ib��ez)

  • The wall-clock time after finishing each task is printed in the output. (Manuel L�pez-Ib��ez, thanks to Federico Caselli for providing an initial patch)

1.06

  • Fix bug that could introduce spurious whitespace when printing the final configurations. (Manuel L�pez-Ib��ez)

  • Fix bug if there are more initial candidates than needed for the first race. (Leslie P�rez C�ceres, Manuel L�pez-Ib��ez)

  • New configuration options, mainly for R users:

    • hookRunParallel: Optional R function to provide custom parallelization of hook.run.

    • hookRunData: Optional data passed to hookRun. This is ignored by the default hookRun function, but it may be used by custom hookRun R functions to pass persistent data around. (Manuel L�pez-Ib��ez)

1.05

  • New option --version. (Manuel L�pez-Ib��ez)

  • Terminate early if there is no sufficient budget to run irace with the given settings. (Manuel L�pez-Ib��ez)

  • The option --parallel (without --mpi) now works under Windows. (Manuel L�pez-Ib��ez, thanks to Pablo Valledor Pellicer for testing it)

  • Improved error handling when running under Rmpi. Now irace will terminate as soon as the master node detects at least one failed slave node. This avoids irace reporting two times the same error. Also, irace will print all the unique errors returned by all slaves and not just the first one. (Manuel L�pez-Ib��ez)

  • Forbidden configurations may be specified in terms of constraints on their values. Forbidden configurations will never be evaluated by irace. See --forbidden-file and inst/templates/forbidden.tmpl. (Manuel L�pez-Ib��ez)

  • New option --recovery-file (recoveryFile) allows resuming a previous irace run. (Leslie P�rez C�ceres)

  • The confidence level for the elimination test is now configurable with parameter --confidence. (Leslie P�rez C�ceres)

  • Much more robust handling of relative/absolute paths. Improved support for Windows. (Leslie P�rez C�ceres, Manuel L�pez-Ib��ez)

  • Provide better error messages for incorrect parameter descriptions. (Manuel L�pez-Ib��ez) Examples:

    x "" i (0, 0) # lower and upper bounds are the same x "" r (1e-4, 5e-4) # given digits=2, ditto x "" i (-1, -2) # lower bound must be smaller than upper bound x "" c ("a", "a") # duplicated values

  • Print elapsed time for calls to hook-run if debugLevel >=1. (Manuel L�pez-Ib��ez)

  • examples/hook-run-python/hook-run: A multi-purpose hook-run written in Python. (Franco Mascia)

  • Parallel mode in an SGE cluster (--sge-cluster) is more robust. (Manuel L�pez-Ib��ez)

1.04

  • Replace obsolete package multicore by package parallel (requires R >= 2.14.0)

  • Use load-balancing (mc.preschedule = FALSE) in mclapply.

1.03

  • Use reg.finalizer to finish Rmpi properly without clobbering .Last().

  • Remove uses of deprecated as.real().

  • Nicer error handling in readParameters.

  • Add hypervolume (multi-objective) example.

  • Fix several bugs in the computation of similar candidates.

1.02

  • More concise output.

  • The parameters expName and expDescription are now useless and they were removed.

  • Faster computation of similar candidates (Jeremie Dubois-Lacoste and Leslie P�rez C�ceres).

  • Fix bug when saving instances in tunerResults$experiments.

  • irace.cmdline ("--help") does not try to quit R anymore.

1.01

  • Fix bug caused by file.exists (and possibly other functions) not handling directory names with a trailing backslash or slash on Windows.

  • Fix bug using per-instance parameters (Leslie P�rez C�ceres).

  • Fix bug when reading initial candidates from a file.

Reference manual

It appears you don't have a PDF plugin for this browser. You can click here to download the reference manual.

install.packages("irace")

3.1 by Manuel López-Ibáñez, 2 months ago


http://iridia.ulb.ac.be/irace


Browse source code at https://github.com/cran/irace


Authors: Manuel López-Ibáñez [aut, cre] (<https://orcid.org/0000-0001-9974-1295>), Jérémie Dubois-Lacoste [aut], Leslie Pérez Cáceres [aut], Thomas Stützle [aut], Mauro Birattari [aut], Eric Yuan [ctb], Prasanna Balaprakash [ctb]


Documentation:   PDF Manual  


Task views: Optimization and Mathematical Programming


GPL (>= 2) license


Imports stats, utils, compiler

Suggests Rmpi, parallel, knitr, testthat


Suggested by MOEADr, ParamHelpers, mlr.


See at CRAN