BMOBench: Black-Box Multi-Objective Optimization
Benchmarking Platform

arXiv:1605.07009v2 [math.OC] 1 Jul 2017

Supplementary Materials for Multi-Objective Simultaneous
Optimistic Optimization
Abdullah Al-Dujaili, Suresh Sundaram
July 4, 2017
This document briefly describes the Black-Box Multi-Objective Optimization Benchmarking (BMOBench) platform. It presents the test problems,
evaluation procedure, and experimental setup. To this end, the BMOBench
is demonstrated by comparing recent multi-objective solvers from the literature, namely SMS-EMOA (Beume et al., 2007), DMS (Custódio et al., 2011), and
MO-SOO (Al-Dujaili and Suresh, 2017).

1

Test Problems

One-hundred multi-objective optimization problems from the literature are selected.1 These problems have simple bound constraints, that is to say, X =
[l, u] ⊂ Rn , where u  l. Table 1 presents a brief list of these problems with
number of dimensions/objectives. In order to have a better understanding of the
algorithm strength/weakness, the benchmark problems are categorized (wherever possible) according to three key characteristics, namely dimensionality:
low- or high-dimension decision space, separability: separable or non-separable
objectives, and modality: uni-modal or multi-modal objectives. Each of these
attributes imposes a different challenge in solving an MOO problem (Huband
et al., 2006).

2

Evaluation Budget

MO-SOO is a deterministic algorithm producing the same approximation set in
each run of the algorithm for a given problem, whereas the approximation sets
produced by the compared stochastic algorithms: DMS and SMS-EMOA can be
different every time they are run for a given problem. In practice, stochastic
algorithms are run several times per problem. To this end and to ensure a fair
comparison, given a computational budget of v function evaluations per run, the
stochastic algorithms are allocated 10 runs per problem instance. On the other
hand, the deterministic algorithms are run once per problem instance with the
accumulated 10 × v function evaluations.
1 retrieved

from http://www.mat.uc.pt/dms.

1

#

Problem Name

n

m

D

S

M

#

Problem Name

n

m

D

S

M

#

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34

BK1 (Huband et al., 2006)
CL1 (Cheng and Li, 1999)
Deb41 (Deb, 1999)
Deb512a (Deb, 1999)
Deb512b (Deb, 1999)
Deb512c (Deb, 1999)
Deb513 (Deb, 1999)
Deb521a (Deb, 1999)
Deb521b (Deb, 1999)
Deb53 (Deb, 1999)
DG01 (Huband et al., 2006)
DPAM1 (Huband et al., 2006)
DTLZ1 (Deb et al., 2002)
DTLZ1n2 (Deb et al., 2002)
DTLZ2 (Deb et al., 2002)
DTLZ2n2 (Deb et al., 2002)
DTLZ3 (Deb et al., 2002)
DTLZ3n2 (Deb et al., 2002)
DTLZ4 (Deb et al., 2002)
DTLZ4n2 (Deb et al., 2002)
DTLZ5 (Deb et al., 2002)
DTLZ5n2 (Deb et al., 2002)
DTLZ6 (Deb et al., 2002)
DTLZ6n2 (Deb et al., 2002)
ex005 (Hwang and Masud, 1979)
Far1 (Huband et al., 2006)
FES1 (Huband et al., 2006)
FES2 (Huband et al., 2006)
FES3 (Huband et al., 2006)
Fonseca (Fonseca and Fleming, 1998)
I1 (Huband et al., 2005)
I2 (Huband et al., 2005)
I3 (Huband et al., 2005)
I4 (Huband et al., 2005)

2
4
2
2
2
2
2
2
2
2
1
10
7
2
12
2
12
2
12
2
12
2
22
2
2
2
10
10
10
2
8
8
8
8

2
2
2
2
2
2
2
2
2
2
2
2
3
2
3
2
3
2
3
2
3
2
3
2
2
2
2
3
4
2
3
3
3
3

L
L
L
L
L
L
L
L
L
L
L
H
H
L
H
L
H
L
H
L
H
L
H
L
L
L
H
H
H
L
H
H
H
H

S
×
×
×
×
×
×
×
×
×
×
NS
×
×
×
×
×
×
×
×
×
×
×
×
×
NS
S
S
S
S
S
NS
NS
NS

U
×
×
×
×
×
×
×
×
×
M
×
M
M
U
U
M
M
U
U
U
U
U
U
U
M
U
U
U
U
U
U
U
U

35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67

I5 (Huband et al., 2005)
IKK1 (Huband et al., 2006)
IM1 (Huband et al., 2006)
Jin1 (Vicente and Custódio, 2012)
Jin2 (Vicente and Custódio, 2012)
Jin3 (Vicente and Custódio, 2012)
Jin4 (Vicente and Custódio, 2012)
Kursawe (Kolda et al., 2003)
L1ZDT4 (Deb et al., 2006)
L2ZDT1 (Deb et al., 2006)
L2ZDT2 (Deb et al., 2006)
L2ZDT3 (Deb et al., 2006)
L2ZDT4 (Deb et al., 2006)
L2ZDT6 (Deb et al., 2006)
L3ZDT1 (Deb et al., 2006)
L3ZDT2 (Deb et al., 2006)
L3ZDT3 (Deb et al., 2006)
L3ZDT4 (Deb et al., 2006)
L3ZDT6 (Deb et al., 2006)
LE1 (Huband et al., 2006)
lovison1 (Liuzzi et al., 2003)
lovison2 (Liuzzi et al., 2003)
lovison3 (Liuzzi et al., 2003)
lovison4 (Liuzzi et al., 2003)
lovison5 (Liuzzi et al., 2003)
lovison6 (Liuzzi et al., 2003)
LRS1 (Huband et al., 2006)
MHHM1 (Huband et al., 2006)
MHHM2 (Huband et al., 2006)
MLF1 (Huband et al., 2006)
MLF2 (Huband et al., 2006)
MOP1 (Huband et al., 2006)
MOP2 (Huband et al., 2006)

8
2
2
2
2
2
2
3
10
30
30
30
30
10
30
30
30
30
10
2
2
2
2
2
3
3
2
1
2
1
2
1
4

3
3
2
2
2
2
2
2
2
2
2
2
2
2
2
2
2
2
2
2
2
2
2
2
3
3
2
3
3
2
2
2
2

H
L
L
L
L
L
L
L
H
H
H
H
H
H
H
H
H
H
H
L
L
L
L
L
L
L
L
L
L
L
L
L
L

NS
×
×
×
×
×
×
×
×
×
×
×
×
×
×
×
×
×
×
S
×
×
×
×
×
×
S
×
S
×
NS
S
S

U
U
U
U
U
U
U
×
×
×
×
×
×
×
×
×
×
×
×
U
×
×
×
×
×
×
U
U
U
M
M
U
U

68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100

Problem Name

n

m

D

S

M

MOP3 (Huband et al., 2006)
MOP4 (Huband et al., 2006)
MOP5 (Huband et al., 2006)
MOP6 (Huband et al., 2006)
MOP7 (Huband et al., 2006)
OKA1 (Nocedal and Wright, 2006)
OKA2 (Nocedal and Wright, 2006)
QV1 (Huband et al., 2006)
Sch1 (Huband et al., 2006)
SK1 (Huband et al., 2006)
SK2 (Huband et al., 2006)
SP1 (Huband et al., 2006)
SSFYY1 (Huband et al., 2006)
SSFYY2 (Huband et al., 2006)
TKLY1 (Huband et al., 2006)
VFM1 (Huband et al., 2006)
VU1 (Huband et al., 2006)
VU2 (Huband et al., 2006)
WFG1 (Huband et al., 2006)
WFG2 (Huband et al., 2006)
WFG3 (Huband et al., 2006)
WFG4 (Huband et al., 2006)
WFG5 (Huband et al., 2006)
WFG6 (Huband et al., 2006)
WFG7 (Huband et al., 2006)
WFG8 (Huband et al., 2006)
WFG9 (Huband et al., 2006)
ZDT1 (Zitzler et al., 2000)
ZDT2 (Zitzler et al., 2000)
ZDT3 (Zitzler et al., 2000)
ZDT4 (Zitzler et al., 2000)
ZDT6 (Zitzler et al., 2000)
ZLT1 (Huband et al., 2006)

2
3
2
2
2
2
3
10
1
1
4
2
2
1
4
2
2
2
8
8
8
8
8
8
8
8
8
30
30
30
10
10
10

2
2
3
2
3
2
2
2
2
2
2
2
2
2
2
3
2
2
3
3
3
3
3
3
3
3
3
2
2
2
2
2
3

L
L
L
L
L
L
L
H
L
L
L
L
L
L
L
L
L
L
H
H
H
H
H
H
H
H
H
H
H
H
H
H
H

×
S
NS
S
×
×
×
S
×
S
×
NS
S
×
×
S
S
S
S
NS
NS
S
S
NS
S
NS
NS
S
S
S
S
S
S

×
×
×
×
U
×
×
M
×
M
×
U
U
×
×
U
U
U
U
×
U
M
×
U
U
U
×
U
U
×
×
M
U

Table 1: Test problems definition and properties. Symbols: D : dimensionality
∈ {L : low-dimensionality, H : high-dimensionality}; S : separability ∈ {S :
separable, NS : non-separable}; M : modality ∈ {U : uni-modal, M : multimodal}; × : uncategorized/mixed.
Quality Indicator (I)

Pareto-Compliant

Reference Set Required

Target

−
Hypervolume Difference (IH
)

Yes

Yes

Minimize

Generational Distance (IGD )

No

Yes

Minimize

Inverted Generational Distance (IIGD )

No

Yes

Minimize

1
Additive -Indicator (I+
)

Yes

Yes

Minimize

Table 2: Employed Quality Indicators. Adapted from (Hadka, 2012) (for more
details, see Knowles and Corne, 2002; Coello et al., 2002).
In our experiments, the evaluation budget v is made proportional to the
search space dimension n and is set to 102 ·n. The overall computational budget
used by an algorithm on BMOBench is the product of the evaluation budget
per run, the number of problems, and the number of runs per problem.
With n = 2, for instance, the overall computational budget used by MO-SOO
on BMOBench is 103 · 2 · 100 · 1 = 2 × 105 function evaluations. Each of the
other algorithms uses also a computational budget of 102 · 2 · 100 · 10 = 2 × 105
function evaluations.

3

Benchmark Procedure

Similar to (Brockhoff et al., 2015), a set of targets are defined for each problem in
terms of four popular quality indicators (Knowles et al., 2006; Zitzler et al., 2003)
listed in Table 2. A solver (algorithm) is then evaluated based on its runtime
with respect to each target: the number of function evaluations used until the
target is reached. We present the recorded runtime values in terms of data
profiles (Moré and Wild, 2009). A data profile can be regarded as an empirical
cumulative distribution function of the observed number of function evaluations
in which the y-axis tells how many targets—over the set of problems and quality

2

indicators—have been reached by each algorithm for a given evaluation budget
(on the x-axis). Mathematically, a data profile for a solver s on a problem class
P has the form


tp,s
1
ds (α) =
p∈P
≤α ,
|P |
np
where tp,s is the observed runtime of solver s on solving problem p (hitting
a target) over a decision space X ⊆ Rnp . The data profile approach captures
several benchmarking aspects, namely the convergence behavior over time rather
than a fixed budget, which can as well be aggregated over problems of similar
category (see, for more details, Brockhoff et al., 2015). In our experiments,
70 linearly spaced values in the logarithmic scale from 10−0.8 to 10−3 and from
−
1
10−0.1 to 10−2 were used as targets for IH
, IGD , and IIGD ; and I+
, respectively.
−
1
The IH , IGD , IIGD and I+ values are computed for each algorithm at any
point of its run based on the set of all (normalized) non-dominated vectors found
so far—i.e., the archive—with respect to a (normalized) reference set R ∈ Ω.
We computed the reference set for calculating the quality indicators by aggregating(union) the approximation sets generated by the evolutionary algorithms
used in (Custódio et al., 2011).
As mentioned in the previous section, we aim to provide a fair comparison
between deterministic and stochastic solvers and accommodate the multiple-run
practice for stochastic algorithms, at the same time. This has been reflected in
the evaluation budget allocation (see Section 2). Likewise, we need to adapt
the data profiles. To this end, given a problem instance and for each one of
the stochastic solvers, we consider the best reported runtime for each target
from the solver’s 10 runs, rather than the mean value. With this setting in
hand, the data profile of MO-SOO at 103 function evaluations, for instance, can
be compared to that of SMS-EMOA at 102 function evaluations.

4

Results

Proportion of (function+target)

Figures 1 and 2 show the data profiles of the compared algorithms as a function
of the number of functione evaluations used.
1.0

all

All Indicators

0.8

MO-SOO

0.6

SMS-EMOA

0.4
DMS

0.2
0.0 0
10

101

102

# f-evaluations

103

Figure 1: Data profiles aggregated over all the problems across all the quality indicators computed for each of the compared algorithms. The symbol ×
indicates the maximum number of function evaluations.

3

DMS

0.4
SMS-EMOA

0.2
0.0 0
10

101

102

103

0.8

MO-SOO

0.6

SMS-EMOA

0.4
DMS

0.2
0.0 0
10

101

102

0.6

SMS-EMOA

0.4
DMS

0.2
0.0 0
10

101

102

103

# f-evaluations / dimension

uni-modal

103

Proportion of (function+target)

Proportion of (function+target)

# f-evaluations / dimension

1.0

MO-SOO

# f-evaluations / dimension

1.0
0.8

MO-SOO
SMS-EMOA

0.4
DMS

0.2
0.0 0
10

101

102

separable
MO-SOO

0.6

SMS-EMOA

0.4
DMS

0.2
0.0 0
10

101

102

103

# f-evaluations / dimension

multi-modal

0.6

1.0
0.8

Proportion of (function+target)

0.6

0.8

103

# f-evaluations / dimension

1.0
0.8

MO-SOO

0.6

SMS-EMOA

0.4
DMS

0.2
0.0 0
10

101

102

1.0

non-separable

0.8

MO-SOO

0.6

SMS-EMOA

0.4
DMS

0.2
0.0 0
10

101

102

103

# f-evaluations / dimension

mixed

103

Proportion of (function+target)

MO-SOO

high-dimensional

Proportion of (function+target)

0.8

1.0

Proportion of (function+target)

low-dimensioanl

Proportion of (function+target)

Proportion of (function+target)

Hypervolume (HV)

1.0

# f-evaluations / dimension

1.0

all

0.8

MO-SOO

0.6

SMS-EMOA

0.4
DMS

0.2
0.0 0
10

101

102

103

# f-evaluations / dimension

0.4
SMS-EMOA

0.2
0.0 0
10

101

102

0.8

MO-SOO

0.6

SMS-EMOA

0.4
DMS

0.2
0.0 0
10

101

SMS-EMOA

0.4
DMS

0.2
0.0 0
10

101

102

# f-evaluations / dimension

uni-modal

102

Proportion of (function+target)

Proportion of (function+target)

# f-evaluations / dimension

1.0

0.6

# f-evaluations / dimension

1.0
0.8

MO-SOO
SMS-EMOA

0.4
DMS

0.2
0.0 0
10

101

MO-SOO

0.6

SMS-EMOA

0.4
DMS

0.2
0.0 0
10

101

102

# f-evaluations / dimension

multi-modal

0.6

separable

0.8

Proportion of (function+target)

DMS

MO-SOO

102

Proportion of (function+target)

0.6

high-dimensional

0.8

1.0

# f-evaluations / dimension

1.0
0.8

MO-SOO

0.6

SMS-EMOA

0.4
DMS

0.2
0.0 0
10

101

1.0

non-separable

0.8

MO-SOO

0.6

SMS-EMOA

0.4
DMS

0.2
0.0 0
10

101

102

# f-evaluations / dimension

mixed

102

Proportion of (function+target)

MO-SOO

1.0

Proportion of (function+target)

low-dimensioanl

0.8

Proportion of (function+target)

Proportion of (function+target)

Additive Epsilon

1.0

# f-evaluations / dimension

1.0

all

0.8

MO-SOO

0.6

SMS-EMOA

0.4
DMS

0.2
0.0 0
10

101

102

# f-evaluations / dimension

DMS

0.4
SMS-EMOA

0.2
0.0 0
10

101

102

MO-SOO

0.6

SMS-EMOA

0.4
DMS

0.2
0.0 0
10

101

MO-SOO

0.6

SMS-EMOA

0.4
DMS

0.2
0.0 0
10

101

102

# f-evaluations / dimension

uni-modal

102

Proportion of (function+target)

Proportion of (function+target)

# f-evaluations / dimension

1.0
0.8

high-dimensional

# f-evaluations / dimension

1.0

MO-SOO

0.6

SMS-EMOA

0.4
DMS

0.2
0.0 0
10

101

separable
MO-SOO

0.6

SMS-EMOA

0.4
DMS

0.2
0.0 0
10

101

102

# f-evaluations / dimension

multi-modal

0.8

1.0
0.8

Proportion of (function+target)

0.6

1.0
0.8

102

# f-evaluations / dimension

1.0

MO-SOO

0.6

SMS-EMOA

0.4
DMS

0.2
0.0 0
10

101

1.0

non-separable

0.8

MO-SOO

0.6

DMS

0.4
SMS-EMOA

0.2
0.0 0
10

101

102

# f-evaluations / dimension

mixed

0.8

102

Proportion of (function+target)

MO-SOO

Proportion of (function+target)

low-dimensioanl

Proportion of (function+target)

1.0
0.8

Proportion of (function+target)

Proportion of (function+target)

Generational Distance (GD)

# f-evaluations / dimension

1.0

all

0.8

MO-SOO

0.6

SMS-EMOA

0.4
DMS

0.2
0.0 0
10

101

102

# f-evaluations / dimension

0.4
SMS-EMOA
101

102

103

0.8

MO-SOO

0.6

SMS-EMOA

0.4
DMS

0.2
0.0 0
10

101

102

# f-evaluations / dimension

0.4
DMS

0.2
0.0 0
10

101

102

# f-evaluations / dimension

uni-modal

Proportion of (function+target)

Proportion of (function+target)

# f-evaluations / dimension

1.0

SMS-EMOA

1.0
0.8

MO-SOO

0.6

SMS-EMOA

0.4
DMS

0.2
0.0 0
10

101

102

separable

0.8

MO-SOO

0.6

SMS-EMOA

0.4
DMS

0.2
0.0 0
10

101

102

# f-evaluations / dimension

multi-modal

Proportion of (function+target)

0.2
0.0 0
10

MO-SOO

Proportion of (function+target)

DMS

high-dimensional

0.8
0.6

1.0

# f-evaluations / dimension

1.0
0.8

MO-SOO

0.6

SMS-EMOA

0.4
DMS

0.2
0.0 0
10

101

102

1.0

non-separable

0.8

MO-SOO

0.6

SMS-EMOA

0.4
DMS

0.2
0.0 0
10

101

102

# f-evaluations / dimension

mixed

103

# f-evaluations / dimension

Proportion of (function+target)

MO-SOO

1.0

Proportion of (function+target)

low-dimensioanl

0.8
0.6

Proportion of (function+target)

Proportion of (function+target)

Inverted Generational Distance (IGD)

1.0

1.0

all

0.8

MO-SOO

0.6

SMS-EMOA

0.4
DMS

0.2
0.0 0
10

101

102

103

# f-evaluations / dimension

Figure 2: Data profiles aggregated over problem categories for each of the quality
indicators computed. The symbol × indicates the maximum number of function
evaluations.

5

Empirical Runtime Evaluation

In order to evaluate the complexity of the algorithms (measured in runtime), we
have run the algorithms on a representative set of the problems. The empirical
4

runtime/f-evaluation (seconds)

DMS
MO-SOO
SMS-EMOA

−2

10

10−3

101

102
#f-evaluations

103

Figure 3: A log-log plot visualizing the runtime per one function evaluation
(in seconds) of the compared algorithms. All the algorithms were run on a
selected set of problems over a set of evaluation budgets, namely BK1, DPAM1,
L3ZDT1, DTLZ3, and FES3; with an evaluation budget ∈ {10, 100, 1000} per
problem on a PC with: 64-bit Windows 7, Intel Xeon E5 CPU @ 3.20GHz,
16GB of memory.
complexity of an algorithm is then computed as the running time (in seconds) of
the algorithm summed over all the problems given an evaluation budget (#FE).
The results are shown in Figure 3.

References
Abdullah Al-Dujaili and S. Suresh. Multi-objective simultaneous optimistic
optimization. Manuscript submitted for publication to Information Sciences,
2017.
Nicola Beume, Boris Naujoks, and Michael Emmerich. Sms-emoa: Multiobjective selection based on dominated hypervolume. European Journal of Operational Research, 181(3):1653–1669, 2007.
Dimo Brockhoff, Thanh-Do Tran, and Nikolaus Hansen. Benchmarking Numerical Multiobjective Optimizers Revisited. In Genetic and Evolutionary
Computation Conference (GECCO 2015), Madrid, Spain, July 2015. doi:
10.1145/2739480.2754777. URL https://hal.inria.fr/hal-01146741.
FY Cheng and XS Li. Generalized center method for multiobjective engineering
optimization. Engineering Optimization, 31(5):641–661, 1999.
Carlos A Coello Coello, David A Van Veldhuizen, and Gary B Lamont. Evolutionary algorithms for solving multi-objective problems, volume 242. Springer,
2002.
5

Ana Luı́sa Custódio, JF Aguilar Madeira, A Ismael F Vaz, and Luı́s N Vicente. Direct multisearch for multiobjective optimization. SIAM Journal on
Optimization, 21(3):1109–1140, 2011.
Kalyanmoy Deb. Multi-objective genetic algorithms: Problem difficulties and
construction of test problems. Evolutionary computation, 7(3):205–230, 1999.
Kalyanmoy Deb, Lothar Thiele, Marco Laumanns, and Eckart Zitzler. Scalable multi-objective optimization test problems. In Proceedings of the
Congress on Evolutionary Computation (CEC-2002),(Honolulu, USA), pages
825–830. Proceedings of the Congress on Evolutionary Computation (CEC2002),(Honolulu, USA), 2002.
Kalyanmoy Deb, Ankur Sinha, and Saku Kukkonen. Multi-objective test problems, linkages, and evolutionary methodologies. In Proceedings of the 8th annual conference on Genetic and evolutionary computation, pages 1141–1148.
ACM, 2006.
Carlos M Fonseca and Peter J Fleming. Multiobjective optimization and multiple constraint handling with evolutionary algorithms. i. a unified formulation.
Systems, Man and Cybernetics, Part A: Systems and Humans, IEEE Transactions on, 28(1):26–37, 1998.
D Hadka. Moea framework a free and open source java framework for multiobjective optimization, 2012.
S. Huband, P. Hingston, L. Barone, and L. While. A review of multiobjective
test problems and a scalable test problem toolkit. Evolutionary Computation,
IEEE Transactions on, 10(5):477–506, Oct 2006. ISSN 1089-778X. doi: 10.
1109/TEVC.2005.861417.
Simon Huband, Luigi Barone, Lyndon While, and Phil Hingston. A scalable
multi-objective test problem toolkit. In Evolutionary multi-criterion optimization, pages 280–295. Springer, 2005.
C.-L. Hwang and A. S. MD. Masud. Multiple objective decision making—
methods and applications: A state-of-the-art survey. Lecture Notes in
Econom. Math. Systems, 164, 1979. Springer-Verlag, Berlin.
J. Knowles and D. Corne. On metrics for comparing nondominated sets. In
Evolutionary Computation, 2002. CEC ’02. Proceedings of the 2002 Congress
on, volume 1, pages 711–716, May 2002. doi: 10.1109/CEC.2002.1007013.
J.D. Knowles, L. Thiele, and E. Zitzler. A tutorial on the performance assessment of stochastic multi-objective optimizers. TIK-Report 214, Computer
Engineering and Networks Laboratory, ETH Zurich, Gloriastrasse 35, ETHZentrum, 8092 Zurich, Switzerland, February 2006.
Tamara G Kolda, Robert Michael Lewis, and Virginia Torczon. Optimization
by direct search: New perspectives on some classical and modern methods.
SIAM review, 45(3):385–482, 2003.
Giampaolo Liuzzi, Stefano Lucidi, Francesco Parasiliti, and Marco Villani. Multiobjective optimization techniques for the design of induction motors. IEEE
Transactions on Magnetics, 39(3):1261–1264, 2003.
6

Jorge J Moré and Stefan M Wild. Benchmarking derivative-free optimization
algorithms. SIAM Journal on Optimization, 20(1):172–191, 2009.
J. Nocedal and Stephen J Wright. Numerical optimization, volume 2. SpringerVerlag, Berlin, 2 edition, 2006.
Luı́s N Vicente and AL Custódio. Analysis of direct searches for discontinuous
functions. Mathematical programming, 133(1-2):299–325, 2012.
Eckart Zitzler, Kalyanmoy Deb, and Lothar Thiele. Comparison of multiobjective evolutionary algorithms: Empirical results. Evolutionary computation, 8
(2):173–195, 2000.
Eckart Zitzler, Lothar Thiele, Marco Laumanns, Carlos M Fonseca, and Viviane Grunert Da Fonseca. Performance assessment of multiobjective optimizers: an analysis and review. IEEE Transactions on Evolutionary Computation, 7(2):117–132, 2003.

7

