Matheuristics 2016 : irace tutorial

by Leslie Perez Caceres - Iridia, ULB

Table of Contents
  1. Introduction
  2. Installing R
  3. Installing irace
  4. Executing irace
  5. irace options
  6. References


Automatic algorithm configurators are powerful tools for supporting the algorithm design process. irace is an automatic algorithm configuration technique that is developed at IRIDIA laboratory and it is freely available in

The objective of the following tutorial is to set up an execution of irace, explore its options and analyze the obtained results. Feel free to use your own algorithm or the provided example.

This tutorial will use the version 2.0 of irace, this version is not publicly available yet, it can be downloaded in:

A user guide is distributed with the package, the guide is available in: irace-package.pdf. In this guide you can find a complete guide of the options of irace and all the information needed to set up a new scenario.

If you have questions or suggestions about irace, please contact us in the irace google group:
irace package google group

irace slides:

1. Installing R

In order to execute irace you must install R. R can be downloaded in You can follow the instructions below.

2. Installing irace

Download the irace 2.0 R package acording the system you are running, see the links in the introduction.

Install the package:

Check the installation is succesful and obtain the installation path

> library(irace)
> system.file(package="irace")
> CTRL+d

3. Executing irace:

Download the ACOTSP example acotsp-example.tar.gz. The target algorithm of this example is ACOTSP, a framework that provides ant colony optimization algorithms for solving the TSP. Extract the files contained in the example:

  1. scenario.txt: File that constains the irace options.
  2. parameters-acotsp.txt: Parameter search space definition.
  3. targetRunner: Bash script for executing ACOTSP (GNU/Linux/OS X).
  4. targetRunner.bat: Batch script for executing ACOTSP (Windows). *You must set the execution path.
  5. training.txt, test.txt: Training and testing instance lists.
  6. Instances: Folder that contains the instances.
  7. default.txt: File that constains initial configurations.
  8. forbidden.txt: File that constains constraints that forbid parameter combinations.

You can check the configuration scenario files are correct before executing irace using the check option of irace.

  1. In the command line:

    # $IRACE_HOME is the installation directory of irace.
    $ $IRACE_HOME/bin/irace --scenario scenario.txt --check
  2. In the R console:

    > library("irace")
    > scenario <- readScenario(filename="scenario.txt",
    +                          scenario=defaultScenario())
    > checkIraceScenario(scenario=scenario)

Now you can execute irace and observe the output:

You should obtain an output similar to this:

# 2016-09-04 01:00:29 CEST: Elitist race
# Elitist new instances: 1
# Elitist limit: 2

# 2016-09-04 01:00:29 CEST: Initialization
# nbIterations: 5
# minNbSurvival: 5
# nbParameters: 11
# seed: 876536678
# confidence level: 0.95
# budget: 300
# mu: 5
# deterministic: FALSE

# 2016-09-04 01:00:29 CEST: Iteration 1 of 5
# experimentsUsedSoFar: 0
# remainingBudget: 300
# currentBudget: 60
# nbConfigurations: 10
     x No test is performed.
     - The test is performed and some configurations are discarded.
     = The test is performed but no configuration is discarded.
     ! The test is performed and configurations could be discarded but elite configurations are preserved.

| |   Instance|      Alive|       Best|      Mean best| Exp so far|  W time|  rho|KenW|  Qvar|
|x|          1|         10|          4|    29041929.00|         10|00:00:12|   NA|  NA|    NA|
|x|          2|         10|          4|    33245942.50|         20|00:00:16|+0.92|0.96|0.0041|
|x|          3|         10|          4|    33240741.00|         30|00:00:14|+0.94|0.96|0.0035|
|x|          4|         10|          4|    33175378.50|         40|00:00:14|+0.95|0.97|0.0034|
|-|          5|          1|          4|    34060122.40|         50|00:00:16|   NA|  NA|    NA|
Best configuration:           4    mean value:     34060122.40
Description of the best configuration:
  .ID. algorithm localsearch alpha beta  rho ants nnls dlb q0 rasrank elitistants .PARENT.
4    4       ras           3  1.94 5.97 0.46   78   12   1 NA      22          NA       NA

# 2016-09-04 01:01:43 CEST: Elite configurations:
  algorithm localsearch alpha beta  rho ants nnls dlb q0 rasrank elitistants
4       ras           3  1.94 5.97 0.46   78   12   1 NA      22          NA
On each iteration irace reports the best configuration and a set of elite configurations. The elite configurations (in this case only one) are configurations that are not statistically different to the best one according to the experiments performed.

5. irace options

A list of the options available in irace can be found in the user guide irace-package.pdf.
The ACOTSP example is set up in a simple way, test adding the following options:

  1. Add an initial candidate to the configuration process, this will give iracea starting point for the search. You can find an initial configuration example in the default.txt file. Give this configuration to irace using the option configurationsFile in the scenario file or the --configurations-file flag in the command line.
  2. Add forbidden configurations constrainst to the configuration process, this will avoid irace certain paramtere values combinations. This is very useful when algorithms have components with unstable behaviour or when there are combinations that are known to be bad. You can find an example of a constraint in the forbidden.txt file. Provide this constraint to irace using the option forbiddenFile in the scenario file or the --forbidden-file flag in the command line.
  3. irace has the option to perform the testing of the final configuration after the configuration process has finalized. To activate this feature you must provide a test instance set. There are two kinds of testing that irace can perform, the option testNbElites defines how many final elite configurations will be tested. This option is useful when many final configurations are provided and we would like to compare them. The option testIterationElites enables the testing of the best configuration of each iteration.

    You can also execute the testing of final configurations once irace was execute using the generated log file.

    > library("irace")
    > testing.main(logFile="exec-dir/irace.Rdata")


[1] Manuel López-Ibáñez, Jérémie Dubois-Lacoste, Thomas Stützle, and Mauro Birattari: The irace package, Iterated Race for Automatic Algorithm Configuration. Technical Report TR/IRIDIA/2011-004, IRIDIA, Université libre de Bruxelles, Belgium, 2011.