A Note on the Effects of Enforcing Bound Constrants on Algorithm Comparisons using the IEEE CEC'05 Benchmark Function Suite

by  Tianjun Liao, Daniel Molina, Marco A. Montes de Oca, and Thomas Stützle
April 2011

Paper Abstract

The benchmark functions proposed in the special session on real parameter optimization of the 2005 IEEE Congress on Evolutionary Computation (CEC'05) are playing an important role in performance comparisons of continuous optimizers and, consequently, in the assessment of the state of the art in continuous optimization. Unfortunately, the original wording for describing the experimental protocol used for the aforementioned session is rather vague about how bound constraints should be handled. During experiments with bound handling mechanisms in IPOP-CMA-ES, the algorithm with the best performance in the aforementioned session, and MA-LSch-CMA, a recent memetic algorithm, we noticed that the performance of these algorithms substantially varies depending on whether bound constraints are enforced or not. We found that if bound constraints are not enforced, the solutions generated may be associated with lower error values than if bounds are enforced. However, the solutions generated when bound constraints are not enforced are often infeasible. These results point toward a major problem in many published experimental evaluations. In particular, several claims about superior performance of an algorithm over IPOP-CMA-ES (or any other algorithm) may be flawed due to possible problems with the enforcement of bound constraints. Therefore, we advocate for reporting explicitly the bound handling mechanism used and the final solutions found in experimental studies involving the CEC'05 benchmark functions.

Tables for ncb vs fcb

Tables for the statement in the submitted paper: in all functions, where IPOP-CMA-ES and MA-LSch-CMA have all their final solutions outside the bounds, fcb is statistically significantly worse than ncb.

See the boundecjsupp.pdf

Examples of presenting final solutions

As a simple example, solutions of IPOP-CMA-ES-ncb and IPOP-CMA-ES-acb are presented for f18 of dimension 30. In this case, we can examine that IPOP-CMA-ES-ncb obtained the final solutions outside the bounds, but IPOP-CMA-ES-ncb outperforms IPOP-CMA-ES-acb.  A similar pattern arises for MA-LSch-CMA-ncb and MA-LSch-CMA-acb.

See the SolExample.tar.gz