An Analysis of Parameters of irace

by Leslie Pérez Cáceres, Manuel López Ibáñez, Thomas Stützle
2013

Table of Contents

Abstract
Configuration Scenarios
Experiments

Abstract:

The irace package implements a flexible tool for the automatic configuration of algorithms. However, irace itself has specific parameters that enable the customization of the search process according to the tuning scenario. In this paper, we analyze five parameters of irace: the number of iterations, the number of instances seen before the first elimination test, the maximum number of elite configurations, the statistical test and the confidence level of the statistical test. These parameters define some key aspects of the way irace searches and identifies good configurations. Originally, their values have been set based on rules of thumb and an intuitive understanding of the configuration process. This work aims at giving insights about the sensitivity of irace to these parameters in order to provide guidance for their settings and possible further improvements of irace.

Keywords: automatic algorithm configuration, parameter tuning, irace, racing algorithms

Configuration Scenarios:

Configuration Scenario	Parameters	Training Set	Test Set	Objective	Cut Off Time	Configuration Budget
ACOTSP	11 parameters	50 TSP instances	250 TSP instances	Solution Quality	20 sec.	5000 evaluations
SPEAR	26 parameters	302 SAT instances	302 SAT instances	Runtime	300 sec.	10000 evaluations
MOACO	16 parameters	60 TSP instances	60 TSP instances	Solution Quality (hypervolume)	4*(instance_size/100)^2 sec.	5000 evaluations

1. ACOTSP

ACOTSP is a software package that implements various ant colony optimization (ACO) algorithms for solving the Traveling Salesman Problem (TSP). We used the version 1.02, available in the following link.

The configured parameters were 11; 3 categorical, 4 integer and 4 continuous, the irace parameter file can be found here.
The training instance set is composed of 10 instances of each size 1000, 1500, 2000, 2500 and 3000, the irace file defining execution seeds used can be be found here.
The testing instance set is composed of 50 instances of each size 1000, 1500, 2000, 2500 and 3000.
The scenario minimize solution quality using a budget of 5000 evaluations and a cut off time of 20 seconds for ACOTSP executions. An example of the irace configuration file used is available here.

2. SPEAR

SPEAR is a tree search solver for SAT problems. We used the version 2.7, available in the following link.

The configured parameters were 26, all categoric. The irace parameter file can be found here.
The training instance set is composed of 302 SAT instances, they belong to the "spear-swv" benchmark from the sat benchmark collection for automatic algorithm configuration available here. The irace instance file defining execution seeds can be found here.
The testing instance set is composed of 302 SAT instances, they also belong to the "spear-swv" benchmark from the sat benchmark collection for automatic algorithm configuration available here.
The scenario minimize the runtime using a 10000 evaluation budget and a cut off time of 300 seconds for SPEAR executions. An example of the irace configuration file used is available here.

3. MOACO

MOACO is a framework of multi-objective ACO algorithms. We used the version 0.1, available in the following link.

The configured parameters were 16; 11 categorical, 1 integer, 4 continuous. The irace parameter file can be found here.
The training instance set is composed of 10 instance of each size 500, 600, 700, 800, 900 and 1000, the irace file defining execution seeds used can be found here.
The testing instance set is composed of 10 instance of each size 500, 600, 700, 800, 900 and 1000.
The scenario minimize solution quality of the Pareto front measured by the hypervolume. The budget was 5000 evaluations and the cut off time was dinamically adjusted using the formula 4*(instance_size/100)^2. An example of the irace configuration file used is available here.