by Leslie Perez Caceres - Iridia, ULB
| Table of Contents |
Automatic algorithm configurators are powerful tools for supporting the algorithm design process. irace is an automatic algorithm configuration technique that is developed at IRIDIA laboratory and it is freely available in http://iridia.ulb.ac.be/irace.
The objective of the following tutorial is to set up an execution of irace, explore its options and analyze the obtained results. Feel free to use your own algorithm or the provided example.
This tutorial will use the version 2.0 of irace, this version is not publicly available yet, it can be downloaded in:
irace_2.0.tar.gzirace_2.0.tar.gzirace_2.0.zipirace-package.pdf. In this guide you can find a complete guide of the options of irace and all the information needed to set up a new scenario.
irace, please contact us in the irace google group:irace package google group
irace slides:
In order to execute irace you must install R. R can be downloaded in
http://www.r-project.org.
You can follow the instructions below.
$ sudo apt-get install r-base
Once R is installed, you can launch R from the terminal and from the R prompt install the irace package. See instructions below.
You can install R from a CRAN mirror (the link is of the Belgian one):
http://cran.freestatistics.org/bin/windows/
Once R is installed, you can launch the R console and install the irace package from it. See instructions below.
You can install R directly from a CRAN mirror (the link is of the Belgian one):
http://cran.freestatistics.org/bin/macosx/
Alternatively, if you use homebrew, you can just brew the R formula from the science tap (unfortunately it does not come already bottled so you need to have Xcode installed to compile it):
$ brew tap homebrew/science
$ brew install r
Once R is installed, you can launch the R console and install the irace package from it. See instructions below.
iraceDownload the irace 2.0 R package acording the system you are running, see the links in the introduction.
Install the package:
Install within R:
$ R
> install.packages("irace2.0.tar.gz")
Install in the command line:
$ R CMD INSTALL <irace2.0.tar.gz>
Check the installation is succesful and obtain the installation path
> library(irace) > system.file(package="irace") > CTRL+d
irace command
Save the installation directory path of irace to a variable, and add it to your
.bash_profile, .bashrc or
.profile:
export IRACE_HOME=~/R/irace/ # Path given by system.file(package="irace")
export PATH=${IRACE_HOME}/bin/:$PATH
# export R_LIBS=~/R:${R_LIBS} # Only if local installation was forced
After adding this and opening a new terminal, you
should be able to invoke irace as follows:
$ irace --help
irace:Download the ACOTSP example acotsp-example.tar.gz.
The target algorithm of this example is ACOTSP, a framework that provides ant colony optimization algorithms for solving the TSP. Extract the files contained in the example:
scenario.txt: File that constains the irace options. parameters-acotsp.txt: Parameter search space definition. targetRunner: Bash script for executing ACOTSP (GNU/Linux/OS X). targetRunner.bat: Batch script for executing ACOTSP (Windows). *You must set the execution path. training.txt, test.txt: Training and testing instance lists.Instances: Folder that contains the instances.default.txt: File that constains initial configurations.forbidden.txt: File that constains constraints that forbid parameter combinations. You can check the configuration scenario files are correct before executing irace using the check option of irace.
In the command line:
# $IRACE_HOME is the installation directory of irace. $ $IRACE_HOME/bin/irace --scenario scenario.txt --check
In the R console:
> library("irace")
> scenario <- readScenario(filename="scenario.txt",
+ scenario=defaultScenario())
> checkIraceScenario(scenario=scenario)
Now you can execute irace and observe the output:
In the command line:
# $IRACE_HOME is the installation directory of irace. $ $IRACE_HOME/bin/irace --scenario scenario.txt
In R console:
> library("irace")
> parameters <- readParameters("parameters.txt")
> scenario <- readScenario(filename="scenario.txt",
+ scenario=defaultScenario())
> irace(scenario = scenario, parameters = parameters)
You should obtain an output similar to this:
# 2016-09-04 01:00:29 CEST: Elitist race
# Elitist new instances: 1
# Elitist limit: 2
# 2016-09-04 01:00:29 CEST: Initialization
# nbIterations: 5
# minNbSurvival: 5
# nbParameters: 11
# seed: 876536678
# confidence level: 0.95
# budget: 300
# mu: 5
# deterministic: FALSE
# 2016-09-04 01:00:29 CEST: Iteration 1 of 5
# experimentsUsedSoFar: 0
# remainingBudget: 300
# currentBudget: 60
# nbConfigurations: 10
Markers:
x No test is performed.
- The test is performed and some configurations are discarded.
= The test is performed but no configuration is discarded.
! The test is performed and configurations could be discarded but elite configurations are preserved.
+-+-----------+-----------+-----------+---------------+-----------+--------+-----+----+------+
| | Instance| Alive| Best| Mean best| Exp so far| W time| rho|KenW| Qvar|
+-+-----------+-----------+-----------+---------------+-----------+--------+-----+----+------+
|x| 1| 10| 4| 29041929.00| 10|00:00:12| NA| NA| NA|
|x| 2| 10| 4| 33245942.50| 20|00:00:16|+0.92|0.96|0.0041|
|x| 3| 10| 4| 33240741.00| 30|00:00:14|+0.94|0.96|0.0035|
|x| 4| 10| 4| 33175378.50| 40|00:00:14|+0.95|0.97|0.0034|
|-| 5| 1| 4| 34060122.40| 50|00:00:16| NA| NA| NA|
+-+-----------+-----------+-----------+---------------+-----------+--------+-----+----+------+
Best configuration: 4 mean value: 34060122.40
Description of the best configuration:
.ID. algorithm localsearch alpha beta rho ants nnls dlb q0 rasrank elitistants .PARENT.
4 4 ras 3 1.94 5.97 0.46 78 12 1 NA 22 NA NA
# 2016-09-04 01:01:43 CEST: Elite configurations:
algorithm localsearch alpha beta rho ants nnls dlb q0 rasrank elitistants
4 ras 3 1.94 5.97 0.46 78 12 1 NA 22 NA
On each iteration irace reports the best configuration and a set of elite configurations. The elite configurations (in this case only one) are configurations that are not statistically different to the best one according to the experiments performed.
irace optionsA list of the options available in irace can be found in the user guide
irace-package.pdf.
The ACOTSP example is set up in a simple way, test adding the following options:
iracea starting point for the search. You can find an initial configuration example in the default.txt file. Give this configuration to irace using the option configurationsFile in the scenario file or the --configurations-file flag in the command line.irace certain paramtere values combinations. This is very useful when algorithms have components with unstable behaviour or when there are combinations that are known to be bad. You can find an example of a constraint in the forbidden.txt file. Provide this constraint to irace using the option forbiddenFile in the scenario file or the --forbidden-file flag in the command line.
irace has the option to perform the testing of the final configuration after the configuration process has finalized. To activate this feature you must provide a test instance set. There are two kinds of testing that irace can perform, the option testNbElites defines how many final elite configurations will be tested. This option is useful when many final configurations are provided and we would like to compare them. The option testIterationElites enables the testing of the best configuration of each iteration.
You can also execute the testing of final configurations once irace was execute using the generated log file.
> library("irace")
> testing.main(logFile="exec-dir/irace.Rdata")
[1] Manuel López-Ibáñez, Jérémie Dubois-Lacoste, Thomas Stützle, and Mauro Birattari: The irace package, Iterated Race for Automatic Algorithm Configuration. Technical Report TR/IRIDIA/2011-004, IRIDIA, Université libre de Bruxelles, Belgium, 2011.