Chapter 4: AutoMoDe-`Vanilla`

Supplementary Material

Gianpiero Francesca supervised by Mauro Birattari
January 2017

Videos

In the following we show, for each task, two videos, one for AutoMoDe-Vanilla and one EvoStick.
These videos are randomly chosen among the controllers designed using the design budget 200,000.
The complete collection of videos is available for download:

Aggregation - design budget: 10,000 50,000 200,000
Foraging - design budget: 10,000 50,000 200,000

Aggregation

AutoMoDe-Vanilla - controller nr. 1 of the design budget 200,000. EvoStick - controller nr. 7 of the design budget 200,000.

Foraging

AutoMoDe-Vanilla - controller nr. 17 of the design budget 200,000. EvoStick - controller nr. 9 of the design budget 200,000.

Statistical data

The data of all the experiments are available for download here. You can import them in R using read.table("results.txt",header=T) .

Futher exploratory experiments

To exclude the possibility that the differences observed between AutoMoDe-Vanilla and EvoStick are due to the different optimization processes adopted by the two methods rathern than by the different control architectures, we performed a further set of exploratory experiments.

We defined another automatic design method that we called FnnStick. This method uses the same control architecture used by EvoStick, a neural network; and the same optimization process used by AutoMoDe-Vanilla, the F-race algorithm. In the following analysis, we compared EvoStick and FnnStick. As the two methods use the same control architecture, the possible differences in performance will have to be ascribed solely to the optimization process.

We have compared EvoStick and FnnStick both on aggregation and foraging using the same setting described in the chapter. In simulation, we have considered the three design budget levels: 10,000, 50,000, and 200,000 simulation runs. In aggregation, the results obtaind by EvoStick and FnnStick are comparable. In foraging, EvoStick performs slightly better than FnnStick. Due to time constraints, we have performed robot experiments only on the control software obtained under the 50,000 design budget and we have tested only 10 randomly selected instances of the control software produced by the two methods (out of the 20 produced). The results show that, also in this case, EvoStick and FnnStick are comparable in aggregation, while EvoStick performs is better than FnnStick in foraging.

Our exploratory experiments confirm that in no case FnnStick performs better than EvoStick. In the context of our experiments, F-race is not more powerful than the evolutionary algorithm adopted in EvoStick. This excludes that the differences in performance between AutoMoDe-Vanilla and EvoStick are due to the different optimization processes adopted by the two methods.

In the following we report, the boxplots that describe our observations.