Random Orthogonal Search with Triangular and Quadratic Distributions (TROS and QROS): Parameterless Algorithms for Global Optimization

Abstract

In this paper, the behavior and performance of Pure Random Orthogonal Search (PROS), a parameter-free evolutionary algorithm (EA) that outperforms many existing EAs on the well-known benchmark functions with finite-time budget, are analyzed. The sufficient conditions to converge to the global optimum are also determined. In addition, we propose two modifications to PROS, namely Triangular-Distributed Random Orthogonal Search (TROS) and Quadratic-Distributed Random Orthogonal Search (QROS). With our local search mechanism, both modified algorithms improve the convergence rates and the errors of the obtained solutions significantly on the benchmark functions while preserving the advantages of PROS: parameterless, excellent computational efficiency, ease of applying to all kinds of applications, and high performance with finite-time search budget. The experimental results show that both TROS and QROS are competitive in comparison to several classic metaheuristic optimization algorithms.

1. Introduction

Black-box optimization refers to optimizing an objective function in the absence of prior knowledge of the function. The only knowledge of the objective function is the observed outputs of the given inputs. Metaheuristics such as evolutionary computation (EC) techniques and evolutionary algorithms (EAs) are widely employed for black-box optimization of various fields across science and engineering [1,2,3] due to their certain primary advantages over the traditional techniques (such as Newton’s method and gradient descent) including high robustness in solving complex real-world problems, and domain knowledge and regularity (e.g., convexity, continuity, differentiability) of the functions to be optimized are generally not required [4].

However, these algorithms often require a large number of function evaluations to evolve or update the candidate solutions in order for errors reduced to a satisfactory level [5]. For many real-world problems, evaluating the objective function is often costly. These function evaluations may involve conducting computational intensive numerical simulations or expensive physical experiments [6,7,8,9] When only a limited number of objective function evaluations is affordable, EC’s trial and error approach becomes unattractive.

Moreover, EC techniques are commonly recognized as heuristic search algorithms because their theoretical analysis often lags behind the development of the algorithms [4,10]. Their performance is often sensitive to the problem-dependent hyper-parameters [11]. Various approaches have been suggested to overcome these drawbacks. For example, building an inexpensive surrogate model for evaluation so as to reduce the number of objective function evaluations [5,12], and automatic or self-adaptive tuning of the hyper-parameters [11,13,14].

In this paper, we propose two algorithms for expensive black-box optimization. The two algorithms are modified from Pure Random Orthogonal Search (PROS) [15]. PROS is a (1 + 1) Evolutionary Strategy ((1 + 1) ES), a kind of EA, that has only one parent in the population and generates one child in each generation that competes with its parent. The finite-time performance of PROS is promising and outperforms several existing EC techniques on most benchmark functions [15]. Unlike those well known EAs, PROS is free of parameters and thus the tuning of parameters is not required. To the best of our knowledge, there were no mathematical analyses on the behaviour and performance of the algorithm reported in the literature. In this paper, we aim to fill this gap. The major contributions of this research work are summarized as follows:

1.

The behavior and performance analysis of PROS, which are not available in [15], are provided here.

2.

Two effective novel (1 + 1) ES based on PROS, namely TROS and QROS, are proposed and they outperform PROS on a set of benchmark problems.

3.

The performance of TROS and QROS are found competitive with three well-known optimization algorithms (GA, PSO and DE) on a set of benchmark problems.

Problem Formulation

In this paper, we restrict attention to global optimization of expensive black-box functions. The goal of global optimization is to find ${\mathbf{x}}^{*}$ which is a global minimum of f,

$${\mathbf{x}}^{*}=arg\underset{\mathbf{x}\in \Omega }{min}f\left(\mathbf{x}\right)$$

where $f:{\mathbb{R}}^D\to \mathbb{R}$ is a scalar-valued objective function defined on the decision space $\Omega \subseteq {\mathbb{R}}^D$ and $\mathbf{x}=(x_1,x_2,...,x_D)$ represents a vector of decision variables with D dimensions.

We have made the following assumptions.

Assumption 1.

f has a single global optimum ${\displaystyle f^{*}=\underset{\mathbf{x}\in \Omega }{min}f\left(\mathbf{x}\right)}$.

Assumption 2.

f can be evaluated at all points of Ω in an arbitrary order.

Assumption 3.

The slope of f is bounded with a Lipschitz constant L such that

$$|f\left({\mathbf{x}}_1\right)-f\left({\mathbf{x}}_2\right)|\le L\parallel {\mathbf{x}}_1-{\mathbf{x}}_2\parallel ,\forall {\mathbf{x}}_1,{\mathbf{x}}_2\in \Omega ,$$

where $L>0$ and $\parallel ·\parallel$ denotes the Euclidean norm.

It has to be noted that although f is assumed a Lipschitz function, the corresponding Libschitz constant L is unknown to an optimization algorithm.

2. Analysis of the Pure Random Orthogonal Search (PROS) Algorithm

The PROS algorithm was originally proposed by Plevris et al. in [15]. We describe it with our revised notation as follows:

Initially, a candidate solution vector ${\mathbf{x}}^{\left(0\right)}$ is generated randomly from $\Omega$ (lines 1 and 2). Then, for each iteration t, j is chosen randomly from 1 to D (line 4) and a real number r is drawn randomly with uniform distribution within the search space of the j-th decision variable (line 5). A new candidate solution vector $\mathbf{y}$ is obtained by replacing the value of the j-th decision variable of the current best solution vector ${\mathbf{x}}^{\left(t\right)}$ with r (lines 6 and 7). The new candidate solution vector $\mathbf{y}$ is evaluated and the new best solution vector ${\mathbf{x}}^{(t+1)}$ is then updated based on the comparison result between $f\left(\mathbf{y}\right)$ and $f\left({\mathbf{x}}^{\left(t\right)}\right)$. If $f\left(\mathbf{y}\right)$ is smaller, $\mathbf{y}$ is accepted as the new best solution vector. Otherwise, $\mathbf{y}$ is rejected (lines 8 to 12). The iteration counter t is updated and the search process is repeated until a termination criterion is met (lines 13 and 14). Finally, the best solution vector found by the algorithm is returned as the final result (line 15).

In the following analysis, we assume the algorithm runs continuously until the global optimum is found (defined below of this section). It is because we are interested in analyzing its ultimate performance. In practice, it may take an infinitely long time to reach the global optimum. Therefore, other termination criteria are usually adopted. For example, the algorithm stops when no improvement is made after a certain number of iterations or when a pre-defined maximum number of iterations is reached.

The error of the algorithm after running for t iterations is given by

$$e_t=f\left({\mathbf{x}}^{\left(t\right)}\right)-f\left({\mathbf{x}}^{*}\right).$$

Here, ${\mathbf{x}}^{\left(t\right)}$ is the solution vector found by the algorithm in the t-th iteration and is also the best solution vector found by the algorithm after t iterations. It is because the values of the solution vectors found by PROS is monotonically non-increasing. That is,

$$f\left({\mathbf{x}}^{\left(t\right)}\right)\le f\left({\mathbf{x}}^{(t-1)}\right)\le \cdots \le f\left({\mathbf{x}}^{\left(0\right)}\right),$$

Consider the pseudo code from line 8 to line 12 of Algorithm 1. In the t-th iteration, when a better solution $\mathbf{y}$ is found, it will be assigned to ${\mathbf{x}}^{(t+1)}$. In this case, $f\left({\mathbf{x}}^{(t+1)}\right)<f\left({\mathbf{x}}^{\left(t\right)}\right)$. When no better solution is found in the t-th iteration, ${\mathbf{x}}^{\left(t\right)}$ will be assigned to ${\mathbf{x}}^{(t+1)}$. In this case, $f\left({\mathbf{x}}^{(t+1)}\right)=f\left({\mathbf{x}}^{\left(t\right)}\right)$. Combining the two conditions, $f\left({\mathbf{x}}^{(t+1)}\right)\le f\left({\mathbf{x}}^{\left(t\right)}\right)$ and the inequalities then follow.

Algorithm 1: Pure Random Orthogonal Search (PROS)

input: nil output: the best solution vector ${\mathbf{x}}^{\left(t\right)}$ found by the algorithm $t\leftarrow 0$ Initialize ${\mathbf{x}}^{\left(t\right)}=(x_1,x_2,\dots ,x_D)$ randomly from $\Omega =[a_1,b_1]\times [a_2,b_2]\times \cdots \times [a_D,b_D]$ repeat

Definition 1.

(Region of Global Optimum). It is said that a candidate solution vector $\mathbf{x}$ is in the region of global optimum $R_{ϵ}$ if the error of $\mathbf{x}$ is not larger than ϵ. That is,

$$R_{ϵ}=\{\mathbf{x}:\mathbf{x}\in \Omega \mathrm{and}f\left(\mathbf{x}\right)-f^{*}\le ϵ\},$$

where ϵ is a small positive real number denoting the tolerance of error.

Definition 2.

(Convergence Time). The convergence time T of an algorithm is defined to be the first time t when

$${\mathbf{x}}^{\left(t\right)}\in R_{ϵ}.$$

2.1. One-Dimensional Functions

In this section, we study the performance of PROS on one-dimensional functions defined on the domain $\Omega =[a,b]$. Since we are interested in the case where $ϵ$ is small, we assume that $\frac{2ϵ}{L}<b-a$.

Lemma 1.

For any $ϵ>0$, the region of the global minimum contains an interval of minimum length $\frac{ϵ}{L}$ for all one-dimensional functions.

Proof. 

Let ${\mathbf{x}}^{*}=\left(x_1^{*}\right)$ be the global optimum, ${\mathbf{x}}^l=\left(x_1^l\right)\in R_{ϵ}$ be a point on the left of ${\mathbf{x}}^{*}$ and ${\mathbf{x}}^r=\left(x_1^r\right)\in R_{ϵ}$ be a point on the right of ${\mathbf{x}}^{*}$. We are going to find a sufficient condition to ensure that $[x_1^l,x_1^r]$ is in the region of global optimum. Since f is a Lipschitz function, the interval $[x_1^l,x_1^{*}]$ belongs to $R_{ϵ}$ if $x_1^l$ satisfies

$$f\left({\mathbf{x}}^l\right)-f\left({\mathbf{x}}^{*}\right)\le L\parallel {\mathbf{x}}^l-{\mathbf{x}}^{*}\parallel =L(x_1^{*}-x_1^l)\le ϵ,$$

or equivalently, $x_1^l\ge x_1^{*}-\frac{ϵ}{L}$. We want to make $x_1^l$ as small as possible, so we let

$$x_1^l=max\{a,x_1^{*}-\frac{ϵ}{L}\}.$$

Similarly, the interval $[x_1^{*},x_1^r]$ belongs to $R_{ϵ}$ if

$$x_1^r=min\{b,x_1^{*}+\frac{ϵ}{L}\}.$$

Since we have assumed $\frac{2ϵ}{L}<b-a$, it is impossible that $x_1^l=a$ and $x_1^r=b$ hold simultaneously. Therefore, at least one of the intervals has length $\frac{ϵ}{L}$. Hence, the length of the region of global optimum is at least $\frac{ϵ}{L}$. □

Theorem 1.

The expected convergence time of PROS for one-dimensional functions is bounded above by $\frac{L(b-a)}{ϵ}$.

Proof. 

Let l be the length of the region of global optimum, $R_{ϵ}$. The probability that a point chosen uniformly at random falls in $R_{ϵ}$ is given by

$$p=\frac{l}{b-a}\ge \frac{ϵ}{L(b-a)}.$$

The convergence time T is geometrically distributed with parameter p. Its expected value is given by

$$\mathbb{E}\left[T\right]=\frac{1}{p}\le \frac{L(b-a)}{ϵ}.$$

□

Corollary 1.

PROS converges in probability to the region of global optimum for all one-dimensional functions, i.e.,

$$\underset{t\to \infty }{lim}P\{{\mathbf{x}}^{\left(t\right)}\in R_{ϵ}\}=1,\forall ϵ>0.$$

Proof. 

For any realization $\{{\mathbf{x}}^{\left(t\right)}:t=0,1,2,\dots \}$, $f\left({\mathbf{x}}^{\left(t\right)}\right)$ is monotonically non-increasing in t. If ${\mathbf{x}}^{\left(t\right)}\in R_{ϵ}$, then ${\mathbf{x}}^{(t+\tau )}\in R_{ϵ}$ for all non-negative integer $\tau$. Therefore, $P\{{\mathbf{x}}^{\left(t\right)}\in R_{ϵ}\}$ is monotonically non-decreasing in t. Since the sequence is bounded above by 1, it is convergent.

Given any $ϵ>0$, the expected convergence time is bounded according to Theorem 1. If ${lim}_{t\to \infty }P\{{\mathbf{x}}^{\left(t\right)}\notin R_{ϵ}\}$ was non-zero, then the expected convergence time would be unbounded, which leads to a contradiction. □

2.2. Multi-Dimensional Functions

In this section, we study the performance of PROS on D-dimensional functions with domain $\Omega =[a_1,b_1]\times [a_2,b_2]\times \cdots \times [a_D,b_D]$, where $D>1$. In particular, we focus on the class of totally separable functions as defined below. For notation simplicity, define ${\mathbf{x}}_{-i}\triangleq (x_1,x_2,\dots ,x_{i-1},x_{i+1},\dots ,x_D)$.

Definition 3.

(Partially Separable Function). A function $f\left(\mathbf{x}\right)$ is partially separable with coordinate i if ${\displaystyle arg\underset{x_i}{min}f(x_i,{\mathbf{x}}_{-i})}$ is independent of ${\mathbf{x}}_{-i}$.

An example of partially separable functions with coordinate $i=1$ is:

$$f\left(\mathbf{x}\right)=x_1^2+\sum _{i=2}^{D-1}{(x_i-x_{i+1})}^2.$$

Definition 4.

(Totally Separable Function). A function $f\left(\mathbf{x}\right)$ is totally separable if it is partially separable with every of the D coordinates.

An example of totally separable functions is:

$$f\left(\mathbf{x}\right)=\prod _{i=1}^D(x_i^2+1).$$

In each iteration, PROS minimizes f in one of the random coordinate i. As f is totally separable, each coordinate i can be minimized independently. The following result shows that PROS converges to the region of global optimum in probability.

Theorem 2.

PROS converges in probability to the region of global optimum for all totally separable functions, i.e.,

$$\underset{t\to \infty }{lim}P\{{\mathbf{x}}^{\left(t\right)}\in R_{ϵ}\}=1,\forall ϵ>0.$$

Proof. 

Since f is Lipschitz, given any ${\mathbf{x}}_{-i}$, we have

$$|f(x_i,{\mathbf{x}}_{-i})-f({\tilde{x}}_i,{\mathbf{x}}_{-i})|\le L|x_i-{\tilde{x}}_i|.$$

Therefore, f can be regarded as a one-dimensional Lipschitz function in $x_i$ with the same constant L. As shown in the previous subsection, there is an interval $V_{{ϵ}_i}$ of minimum length $\frac{{ϵ}_i}{L}$ such that for all $x_i\in V_{{ϵ}_i}$,

$$|f(x_i,{\mathbf{x}}_{-i})-f(x_i^{*},{\mathbf{x}}_{-i})|\le {ϵ}_i.$$

Note that $V_{{ϵ}_i}$ is independent of ${\mathbf{x}}_{-i}$. Under PROS, it is clear that

$$\underset{t\to \infty }{lim}P\{x_i^{\left(t\right)}\in V_{{ϵ}_i}\}=1,\forall {ϵ}_i>0.$$

Since f is Lipschitz, it is a continuous function. Given any $ϵ>0$, there exists sufficiently small positive ${ϵ}_i$’s such that when $x_i^{\left(t\right)}\in V_{{ϵ}_i}$ for all i, we must have ${\mathbf{x}}^{\left(t\right)}\in R_{ϵ}$. The statement then follows from

$$P\{{\mathbf{x}}^{\left(t\right)}\in R_{ϵ}\}\ge P\{x_i^{\left(t\right)}\in V_{{ϵ}_i},\forall i\}=\prod _{i=1}^{D}P\{x_i^{\left(t\right)}\in V_{{ϵ}_i}\}.$$

□

The expected convergence time can be obtained for the following subclass of totally separable functions.

Definition 5.

(Additively Separable Function). A function f is additively separable if $f\left(\mathbf{x}\right)$ can be written in the form of $f_1\left(x_1\right)+f_2\left(x_2\right)+\cdots +f_D\left(x_D\right)$, where $\mathbf{x}=(x_1,x_2,\dots ,x_D)$ and $f_1,f_2,\dots ,f_D$ are one-dimensional functions.

An example of additively separable functions is the sum-of-spheres function:

$$f_{sphere}\left(\mathbf{x}\right)=\sum _{i=1}^D{x}_i^2.$$

For the optimization of D-dimensional additively separable functions, it is equivalent to optimize D one-dimensional functions independently, i.e.,

$$\underset{\mathbf{x}\in \Omega }{min}f\left(\mathbf{x}\right)=\sum _{i=1}^D\underset{x_i\in [a_i,b_i]}{min}{f}_i\left(x_i\right).$$

Theorem 3.

The expected convergence time of PROS for D-dimensional additively separable functions is bounded above by

$$\mathbb{E}\left[T\right]\le \frac{D^{2}L}{ϵ}\sum _{i=1}^D(b_i-a_i).$$

Proof. 

A point $\mathbf{x}=(x_1,x_2,\dots ,x_D)$ belongs to $R_{ϵ}$ if

$$\sum _{i=1}^D(f_i\left(x_i\right)-f_i^{*})\le ϵ,$$

where $f_i^{*}$ is the global minimum of the i-th one-dimensional function. Then,

$$f_i\left(x_i\right)-f_i^{*}\le \frac{ϵ}{D},\mathrm{for}i\in \{1,\dots ,D\}$$

(1)

is a sufficient condition for $\mathbf{x}\in R_{ϵ}$.

Let $S_i$ be the iteration time PROS enters the region for $x_i$ as stated in (1). Under PROS, at each iteration t, the coordinate to be optimized is chosen uniformly at random. As in the proof of Theorem 1, $S_i$ is a geometric random variable with parameter

$$p_i\ge \frac{1}{D}\left(\frac{ϵ/D}{L(b_i-a_i)}\right)=\frac{ϵ}{D^{2}L(b_i-a_i)}.$$

Note that $S_i$’s are not independent. We bound the convergence time T as follows:

$$T\le max\{S_1,\dots ,S_D\}\le S_1+\cdots +S_D.$$

Hence,

$$\mathbb{E}\left[T\right]\le \sum _{i=1}^D\mathbb{E}\left[S_i\right]=\sum _{i=1}^D\frac{1}{p_i}\le \frac{D^{2}L}{ϵ}\sum _{i=1}^D(b_i-a_i).$$

□

3. Modified PROS with Local Search Mechanism

Although the PROS algorithm converges to the global optimum, provided that the sufficient conditions are satisfied, it converges slowly when compared with other well-known EC algorithms [15]. The major reason is PROS simply performs uniform orthogonal search in every iteration with no local search mechanism. The probability of finding an improved solution (defined in the below subsection) diminishes as it moves closer to the global optimum. Consider the situation that in the t-th iteration, ${\mathbf{x}}^{\left(t\right)}$ is already very close to ${\mathbf{x}}^{*}$ but has not yet fallen into in the region of global optimum. There is a high chance that ${\mathbf{x}}^{(t+1)}$ would reach the region of the global optimum if a narrow-range local search is performed in the $(t+1)$-th iteration. However, with uniform orthogonal search, the chance to reach the global optimum is relatively low, thus making it converge slowly. One may consider using a sampling policy other than uniform to perform local search.

3.1. Triangular-Distributed Random Orthogonal Search (TROS)

In this section, we present our first proposed algorithm called Triangular-Distributed Random Orthogonal Search (TROS). The TROS algorithm (Algorithm 2) is presented as follows:

Algorithm 2: Triangular-Distributed Random Orthogonal Search (TROS)

input: nil output: the best solution vector ${\mathbf{x}}^{\left(t\right)}$ found by the algorithm $t\leftarrow 0$ Initialize ${\mathbf{x}}^{\left(t\right)}=(x_1,x_2,\dots ,x_D)$ randomly from $\Omega =[a_1,b_1]\times [a_2,b_2]\times \cdots \times [a_D,b_D]$ repeat

Compared with PROS, TROS has one change in line 5 of Algorithm 1. Instead of sampling the next point of the j-th decision variable using uniform distribution, the triangular distribution $\mathcal{T}$ is used in the TROS algorithm. The probability density function of the triangular distribution $\mathcal{T}(a,b,c)$ is

$$f_{\mathcal{T}}\left(x\right)=\left\{\begin{array}{cc}\frac{2}{b-a}\frac{x-a}{c-a}\hfill & ,a<x\le c\hfill \\ \frac{2}{b-a}\frac{b-x}{b-c}\hfill & ,c<x<b\hfill \\ 0\hfill & ,x\le a\mathrm{or}x\ge b\hfill \end{array}\right.$$

(2)

where a, b and c are the parameters of the distribution that represents the lower limit of x, upper limit of x and mode of x, respectively. The triangular distribution is illustrated in Figure 1.

In each iteration of TROS, the next point is sampled using the triangular distribution with the settings $a=a_j,b=b_j,c=x_j^{\left(t\right)}$ where $j\in \{1,2,\dots ,D\}$ is the randomly chosen decision variable for the current iteration, $a_j$ and $b_j$ are the lower bound and upper bound of the j-th decision variable, and $x_j^{\left(t\right)}$ is the value of the j-th decision variable of the current best solution vector. With this distribution, there is a higher chance to draw a sample that is near to the current position $x_j^{\left(t\right)}$ than is far from the current position. As a result, the algorithm performs exploitation (that is, encourages local search) on the j-th decision variable.

3.2. Quadratic-Distributed Random Orthogonal Search (QROS)

In this section, we present our second proposed algorithm called Quadratic-Distributed Random Orthogonal Search (QROS). The QROS algorithm (Algorithm 3) is presented as follows:

Algorithm 3: Quadratic-Distributed Random Orthogonal Search (QROS)

input: nil output: the best solution vector ${\mathbf{x}}^{\left(t\right)}$ found by the algorithm $t\leftarrow 0$ Initialize ${\mathbf{x}}^{\left(t\right)}=(x_1,x_2,\dots ,x_D)$ randomly from  $\Omega =[a_1,b_1]\times [a_2,b_2]\times \cdots \times [a_D,b_D]$ repeat

Compared with TROS, QROS has one change in line 5 of Algorithm 2. The value of the j-th decision variable is sampled using the quadratic distribution $\mathcal{Q}$. The probability density function of the quadratic distribution $\mathcal{Q}(a,b,c)$ is

$$f_{\mathcal{Q}}\left(x\right)=\left\{\begin{array}{cc}{\alpha }_{\mathcal{Q}}{(x-a)}^2\hfill & ,a<x\le c\hfill \\ {\beta }_{\mathcal{Q}}{(x-b)}^2\hfill & ,c<x<b\hfill \\ 0\hfill & ,x\le a\mathrm{or}x\ge b\hfill \end{array}\right.$$

(3)

where ${\alpha }_{\mathcal{Q}}=\frac{3}{(b-a){(c-a)}^2}$, ${\beta }_{\mathcal{Q}}=\frac{3}{(b-a){(c-b)}^2}$, a, b and c are the lower limit of x, upper limit of x and mode of x, respectively. The quadratic distribution is illustrated in Figure 1.

In each iteration of QROS, the next point is sampled using the quadratic distribution with the settings $a=a_j,b=b_j,c=x_j^{\left(t\right)}$ where $j\in \{1,2,\dots ,D\}$ is the randomly chosen decision variable for the current iteration, $a_j$ and $b_j$ are the lower bound and upper bound of the j-th decision variable, and $x_j^{\left(t\right)}$ is the value of the j-th decision variable of the current best solution vector. With this distribution, there is a higher chance to draw a sample that is near to the current position $x_j^{\left(t\right)}$ than is far from the current position. Compared with TROS, QROS encourages exploitation even more than TROS.

It has to be noted that both TROS and QROS are still “parameter-free” algorithms, as a and b are fixed boundaries, and c is determined based on the current best solution vector. With a simple modification, both TROS and QROS are able to improve the convergence speed of PROS and keep the major properties of PROS: parameterless, excellent computational efficiency, easy to apply to all kinds of applications, and high performance with finite-time search budget.

3.3. Analysis of the Modified Algorithms

In this subsection, we are going to explain the motivation and analyze the performance of the two algorithms. Sampling using uniform distribution can be considered as performing global search on a decision variable, while sampling using the triangular distribution and the quadratic distribution can be considered as performing local search on a decision variable.

Definition 6.

(Improved Solution). The candidate solution vector ${\mathbf{x}}^{(t+1)}$ found by the algorithm is said to be improved if its error is less than the error of the current best solution. That is,

$$f\left({\mathbf{x}}^{(t+1)}\right)-f\left({\mathbf{x}}^{*}\right)<f\left({\mathbf{x}}^{\left(t\right)}\right)-f\left({\mathbf{x}}^{*}\right)$$

or simply

$$f\left({\mathbf{x}}^{(t+1)}\right)<f\left({\mathbf{x}}^{\left(t\right)}\right)$$

where t denotes the current iteration.

For continuous (or Lipschitz) unimodal functions defined on a bounded domain, finding an improved solution vector always lead to the convergence to the global optimum. However, it may not be true for multimodal functions. Therefore, we are not interested in simply finding an improved solution vector. Here, we define a restrictive subset of improved solution vectors.

Definition 7.

(Tame Solution). The candidate solution vector ${\mathbf{x}}^{(t+1)}$ found by the algorithm is said to be a $tame$ solution if the errors of all points in between the line containing the candidate solution and the optimum solution are all less than the error of the current best solution. That is,

$$f(\alpha {\mathbf{x}}^{(t+1)}+(1-\alpha ){\mathbf{x}}^{*})<f({\mathbf{x}}^{\left(t\right)}),\forall \alpha \in [0,1]$$

where t denotes the current iteration.

A local optimum in a multimodal function is not considered as a $tame$ solution if there exists a high hill in between the local optimum and the global optimum. The purpose of defining $tame$ solution is when a $tame$ solution is found, it does not only mean the solution is improved but also implies one could apply some simple techniques to converge to the global optimum. Therefore, we are interested in knowing the probability of finding a $tame$ solution by each of the algorithms.

We define following terms for the analysis of the performance of the proposed algorithms. Let ${\mathbf{x}}^{\left(t\right)}=(x_1^{\left(t\right)},x_2^{\left(t\right)},\dots ,x_D^{\left(t\right)})$ be the current best solution vector, and $j\in \{1,\dots ,D\}$ be the chosen decision variable of the current iteration. Given ${\mathbf{x}}_{-j}^{\left(t\right)}=(x_1^{\left(t\right)},\dots ,x_{j-1}^{\left(t\right)},x_{j+1}^{\left(t\right)},\dots ,$$x_D^{\left(t\right)})$, assume $f({\mathbf{x}}_{-j}^{\left(t\right)},x_j)$ has a unique global minimum and is denoted as $x_j^{*}$. That is,

$$f({\mathbf{x}}_{-j}^{\left(t\right)},x_j^{*})\le f({\mathbf{x}}_{-j}^{\left(t\right)},x_j),\forall x_j\in [a_j,b_j].$$

Let $R_j$ be the set of values of the j-th decision variable that belong to improved solution vectors. That is,

$$R_j=\left\{x_j\right|f({\mathbf{x}}_{-j}^{\left(t\right)},x_j)<f({\mathbf{x}}_{-j}^{\left(t\right)},x_j^{\left(t\right)}),x_j\in [a_j,b_j]\}.$$

Let $S_j$ be the set of values of the j-th decision variable that belong to $tame$ solution vectors. That is,

$$S_j=\left\{x_j\right|(\alpha x_j+(1-\alpha )x_j^{*})\in R_j,\forall \alpha \in [0,1],x_j\in [a_j,b_j]\}.$$

Let $x_j^l$ be the smallest $x_j$ value and $x_j^r$ be the largest $x_j$ value of the $tame$ solutions, respectively. That is,

$$x_j^l=inf{S}_j$$

and

$$x_j^r=sup{S}_j.$$

Let $l_j$ be the length of the interval $S_j$. That is,

$$l_j=x_j^r-x_j^l.$$

By Lemma 1, $l_j$ is non-negative and is bounded below by

$$l_j\ge \frac{e}{L}$$

where e is the error of $f({\mathbf{x}}_{-j}^{\left(t\right)},x_j^{\left(t\right)})$ with respect to $f({\mathbf{x}}_{-j}^{\left(t\right)},x_j^{*})$. That is,

$$e=f({\mathbf{x}}_{-j}^{\left(t\right)},x_j^{\left(t\right)})-f({\mathbf{x}}_{-j}^{\left(t\right)},x_j^{*}).$$

Figure 2 shows two examples of multimodal functions with the corresponding set of $tame$ solutions denoted by $S_j$.

Definition 8.

(Probability of $Tame$ Convergence). The probability of $tame$ convergence is defined as the probability of an algorithm with sampling policy π to find a $tame$ solution in its next iteration, given the current best solution vector. That is,

$$P\{x_j^{(t+1)}\in S_j|{\mathbf{x}}^{\left(t\right)},x_j^{(t+1)}\sim {\pi }_{{\mathbf{x}}^{\left(t\right)}}\}.$$

Lemma 2.

The probability of $tame$ convergence of PROS is $\frac{l_j}{b_j-a_j}$.

Proof. 

Let $P_j^U$ be the probability that a $tame$ solution is found by sampling using uniform distribution on the j-th decision variable in the $(t+1)$-th iteration, given the best solution vector in the t-th iteration. Then

$$P_j^U=P\{x_j^{(t+1)}\in S_j|{\mathbf{x}}_{-j}^{\left(t\right)},x_j^{(t+1)}\sim \mathcal{U}(a_j,b_j)\}=\frac{l_j}{b_j-a_j}.$$

□

Theorem 4.

The probability of $tame$ convergence of TROS is $\frac{l_j}{b_j-a_j}\left(\frac{l_j+2v_j}{u_j+l_j+v_j}\right)$ when $x_j^{\left(t\right)}<x_j^{*}$ and is $\frac{l_j}{b_j-a_j}\left(\frac{l_j+2q_j}{q_j+l_j+r_j}\right)$ when $x_j^{\left(t\right)}>x_j^{*}$ where $u_j=x_j^l-x_j^{\left(t\right)}$, $v_j=b_j-x_j^r$, $q_j=x_j^l-a_j$ and $r_j=x_j^{\left(t\right)}-x_j^r$.

Proof. 

Let $P_j^T$ be the probability that a $tame$ solution is found by sampling using the triangular distribution on the j-th decision variable in the $(t+1)$-th iteration, given the best solution vector in the t-th iteration. That is,

$$P_j^T=P\{x_j^{(t+1)}\in S_j|{\mathbf{x}}_{-j}^{\left(t\right)},x_j^{(t+1)}\sim \mathcal{T}(a_j,b_j,x_j^{\left(t\right)})\}.$$

For case 1, $x_j^{\left(t\right)}<x_j^{*}$:

$$P_j^T=\frac{l_j}{2}(\frac{2}{b_j-a_j}\frac{v_j}{u_j+l_j+v_j}+\frac{2}{b_j-a_j}\frac{l_j+v_j}{u_j+l_j+v_j})$$

$$=\frac{l_j}{b_j-a_j}\left(\frac{l_j+2v_j}{u_j+l_j+v_j}\right).$$

Similarly, for case 2, $x_j^{\left(t\right)}>x_j^{*}$:

$$P_j^T=\frac{l_j}{2}(\frac{2}{b_j-a_j}\frac{q_j}{q_j+l_j+r_j}+\frac{2}{b_j-a_j}\frac{l_j+q_j}{q_j+l_j+r_j})$$

$$=\frac{l_j}{b_j-a_j}\left(\frac{l_j+2q_j}{q_j+l_j+r_j}\right).$$

□

Corollary 2.

The conditions that TROS has a higher probability of $tame$ convergence than that of PROS, the same probability as that of PROS, and a lower probability than that of PROS are as follows, respectively.

For case 1, $x_j^{\left(t\right)}<x_j^{*}$:

$$P_j^T\left\{\begin{array}{cc}>P_j^U\hfill & ,v_j>u_j\hfill \\ =P_j^U\hfill & ,v_j=u_j\hfill \\ <P_j^U\hfill & ,v_j<u_j\hfill \end{array}\right..$$

For case 2, $x_j^{\left(t\right)}>x_j^{*}$:

$$P_j^T\left\{\begin{array}{cc}>P_j^U\hfill & ,q_j>r_j\hfill \\ =P_j^U\hfill & ,q_j=r_j\hfill \\ <P_j^U\hfill & ,q_j<r_j\hfill \end{array}\right..$$

Corollary 3.

The probability of $tame$ convergence of TROS is higher than or equal to that of PROS for all convex functions.

Proof. 

For convex functions, $u_j=0$ (for case 1) and $r_j=0$ (for case 2). An example convex function is shown in Figure 3.

For case 1, $x_j^{\left(t\right)}<x_j^{*}$:

$$P_j^T=\frac{l_j}{b_j-a_j}\left(\frac{l_j+2v_j}{u_j+l_j+v_j}\right)$$

$$=P_j^U(1+\frac{v_j}{l_j+v_j})$$

$$\therefore P_j^T\left\{\begin{array}{cc}>P_j^U\hfill & ,v_j>0\hfill \\ =P_j^U\hfill & ,v_j=0\hfill \end{array}\right..$$

Similarly, for case 2, $x_j^{\left(t\right)}>x_j^{*}$:

$$P_j^T=\frac{l_j}{b_j-a_j}\left(\frac{l_j+2q_j}{q_j+l_j+r_j}\right)$$

$$=P_j^U(1+\frac{q_j}{q_j+l_j})$$

$$\therefore P_j^T\left\{\begin{array}{cc}>P_j^U\hfill & ,q_j>0\hfill \\ =P_j^U\hfill & ,q_j=0\hfill \end{array}\right..$$

Therefore, for convex functions, $P_j^T$ can never be less than $P_j^U$. □

Theorem 5.

The probability of $tame$ convergence of QROS is $\frac{l_j}{b_j-a_j}\frac{(l_j+v_j)(l_j+2v_j)+v_j^2}{{(u_j+l_j+v_j)}^2}$ when $x_j^{\left(t\right)}<x_j^{*}$ and is $\frac{l_j}{b_j-a_j}\frac{(q_j+l_j)(2q_j+l_j)+q_j^2}{{(q_j+l_j+r_j)}^2}$ when $x_j^{\left(t\right)}>x_j^{*}$ where $u_j=x_j^l-x_j^{\left(t\right)}$, $v_j=b_j-x_j^r$, $q_j=x_j^l-a_j$ and $r_j=x_j^{\left(t\right)}-x_j^r$.

Proof. 

Let $P_j^Q$ be the probability that a $tame$ solution is found by sampling using the quadratic distribution on the j-th decision variable in the $(t+1)$-th iteration, given the best solution vector in the t-th iteration. That is,

$$P_j^Q=P\{x_j^{(t+1)}\in S_j|{\mathbf{x}}_{-j}^{\left(t\right)},x_j^{(t+1)}\sim \mathcal{Q}(a_j,b_j,x_j^{\left(t\right)})\}.$$

For case 1, $x_j^{\left(t\right)}<x_j^{*}$:

$$P_j^Q={\int }_{x_j^l}^{x_j^r}{f}_Q\left(x\right)dx$$

$$={\int }_{x_j^l}^{x_j^r}\frac{3{(x-b_j)}^2}{(b_j-a_j){(x_j^{\left(t\right)}-b_j)}^2}dx$$

$$=\frac{l_j}{b_j-a_j}\frac{(l_j+v_j)(l_j+2v_j)+v_j^2}{{(u_j+l_j+v_j)}^2}.$$

Similarly, for case 2, $x_j^{\left(t\right)}>x_j^{*}$:

$$P_j^Q={\int }_{x_j^l}^{x_j^r}{f}_Q\left(x\right)dx$$

$$={\int }_{x_j^l}^{x_j^r}\frac{3{(x-a_j)}^2}{(b_j-a_j){(x_j^{\left(t\right)}-a_j)}^2}dx$$

$$=\frac{l_j}{b_j-a_j}\frac{(q_j+l_j)(2q_j+l_j)+q_j^2}{{(q_j+l_j+r_j)}^2}.$$

□

Corollary 4.

The probability of $tame$ convergence of QROS is higher than or equal to that of TROS and PROS for all convex functions.

Proof. 

For convex functions, $u_j=0$ (for case 1) and $r_j=0$ (for case 2).

For case 1, $x_j^{\left(t\right)}<x^{*}$:

$$p_j^Q=\frac{l_j}{b_j-a_j}\frac{(l_j+v_j)(l_j+2v_j)+v_j^2}{{(u_j+l_j+v_j)}^2}$$

$$=\underset{P_j^T}{\underbrace{P_j^U(1+\frac{v_j}{l_j+v_j}}}+\frac{v_j^2}{{(l_j+v_j)}^2})$$

$$\therefore \left\{\begin{array}{cc}>P_j^T>P_j^U\hfill & ,v_j>0\hfill \\ =P_j^T=P_j^U\hfill & ,v_j=0\hfill \end{array}\right..$$

For case 2, $x_j^{\left(t\right)}>x^{*}$:

$$P_j^Q=\frac{l_j}{b_j-a_j}\frac{(q_j+l_j)(2q_j+l_j)+q_j^2}{{(q_j+l_j+r_j)}^2}$$

$$=\underset{P_j^T}{\underbrace{P_j^U(1+\frac{q_j}{q_j+l_j}}}+\frac{q_j^2}{{(q_j+l_j)}^2})$$

$$\therefore \left\{\begin{array}{cc}>P_j^T>P_j^U\hfill & ,q_j>0\hfill \\ =P_j^T=P_j^U\hfill & ,q_j=0\hfill \end{array}\right..$$

Therefore, for convex functions, $P_j^Q$ can never be less than $P_j^T$ and $P_j^U$. □

The probability density function of distributions $\mathcal{U}$, $\mathcal{T}$ and $\mathcal{Q}$ are characterized by a degree-0, degree-1 and degree-2 polynomials, respectively. From Corollary 4, the increment in the probablity of $tame$ convergence diminishes as the degree of the polynomial increases. We expect sampling using an even higher degree distribution (e.g., cubic distribution with the mode narrower than $\mathcal{Q}$ and $\mathcal{T}$) would give higher probability of $tame$ convergence, but the gain diminishes as the degree grows.

4. Experiments

In order to evaluate the merits of the two modified algorithms (TROS and QROS), a set of benchmark test problems is selected from the literature [15,16,17] and shown in

Table 1. Benchmark test problems. (Unimodal functions: $f_1$–$f_5$; multimodal functions: $f_6$–$f_{12}$; convex functions: $f_1$–$f_2,f_5$; non-convex functions: $f_3$–$f_4,f_6$–$f_{12}$; separable functions: $f_1$–$f_2,f_6$–$f_7,f_{12}$; non-separable functions: $f_3$–$f_5,f_8$–$f_{11}$).

No.	Function	Formulation and Global Optimum	Search Space
$f_1$	Sphere	${\sum }_{i=1}^D{x}_i^2$	${[-10,10]}^D$
		${\mathbf{x}}^{*}=(0,0,\cdots ,0),f\left({\mathbf{x}}^{*}\right)=0$
$f_2$	Ellipsoid	${\sum }_{i=1}^{D}i{x}_i^2$	${[-10,10]}^D$
		${\mathbf{x}}^{*}=(0,0,\cdots ,0),f\left({\mathbf{x}}^{*}\right)=0$
$f_3$	Schwefel 1.2	${\sum }_{i=1}^D{\left({\sum }_{j=1}^i{x}_j\right)}^2$	${[-5.12,5.12]}^D$
		${\mathbf{x}}^{*}=(0,0,\cdots ,0),f\left({\mathbf{x}}^{*}\right)=0$
$f_4$	Rosenbrock	${\sum }_{i=1}^{D-1}[100{(x_{i+1}-x_i^2)}^2+{(x_i-1)}^2]$	${[-2.048,2.048]}^D$
		${\mathbf{x}}^{*}=(1,1,\cdots ,1),f\left({\mathbf{x}}^{*}\right)=0$
$f_5$	Zakharov	${\sum }_{i=1}^D{x}_i^2+{\left(\frac{1}{2}{\sum }_{i=1}^{D}i{x}_i\right)}^2+{\left(\frac{1}{2}{\sum }_{i=1}^{D}i{x}_i\right)}^4$	${[-10,10]}^D$
		${\mathbf{x}}^{*}=(0,0,\cdots ,0),f\left({\mathbf{x}}^{*}\right)=0$
$f_6$	Alpine 1	${\sum }_{i=1}^D|x_{i}sin\left(x_i\right)+0.1x_i|$	${[-10,10]}^D$
		${\mathbf{x}}^{*}=(0,0,\cdots ,0),f\left({\mathbf{x}}^{*}\right)=0$
$f_7$	Rastrigin	$10D+{\sum }_{i=1}^D[x_i^2-10cos\left(2\pi x_i\right)]$	${[-5.12,5.12]}^D$
		${\mathbf{x}}^{*}=(0,0,\cdots ,0),f\left({\mathbf{x}}^{*}\right)=0$
$f_8$	Ackley	$20+exp\left(1\right)-20exp(-0.2\sqrt{\frac{1}{D}{\sum }_{i=1}^D{x}_i^2})$	${[-32.768,32.768]}^D$
		$-exp(\frac{1}{D}{\sum }_{i=1}^{D}cos\left(2\pi x_i\right))$
		${\mathbf{x}}^{*}=(0,0,\cdots ,0),f\left({\mathbf{x}}^{*}\right)=0$
$f_9$	Griewank	${\sum }_{i=1}^D\frac{x_i^2}{4000}-{\prod }_{i=1}^{D}cos\left(\frac{x_i}{\sqrt{i}}\right)+1$	${[-600,600]}^D$
		${\mathbf{x}}^{*}=(0,0,\cdots ,0),f\left({\mathbf{x}}^{*}\right)=0$
$f_{10}$	HGBat	$|{\left({\sum }_{i=1}^D{x}_i^2\right)}^2-{\left({\sum }_{i=1}^D{x}_i\right)}^2{|}^{1/2}+$	${[-15,15]}^D$
		$(0.5{\sum }_{i=1}^D{x}_i^2+{\sum }_{i=1}^D{x}_i)/D+0.5$
		${\mathbf{x}}^{*}=(-1,-1,\cdots ,-1),f\left({\mathbf{x}}^{*}\right)=0$
$f_{11}$	HappyCat	$|{\sum }_{i=1}^D{x}_i^2{-D|}^{1/4}+$	${[-20,20]}^D$
		$(0.5{\sum }_{i=1}^D{x}_i^2+{\sum }_{i=1}^D{x}_i)/D+0.5$
		${\mathbf{x}}^{*}=(-1,-1,\cdots ,-1),f\left({\mathbf{x}}^{*}\right)=0$
$f_{12}$	Weierstrass	${\sum }_{i=1}^D[{\sum }_{k=0}^{20}{0.5}^{k}cos\left(2\pi 3^k(x_i+0.5)\right)$	${[-0.5,0.5]}^D$
		$-D{\sum }_{k=0}^{20}{0.5}^{k}cos\left(\pi 3^k\right)]$
		${\mathbf{x}}^{*}=(0,0,\cdots ,0),f\left({\mathbf{x}}^{*}\right)=0$

. The benchmark problems include unimodal and highly multimodal functions, convex and non-convex functions, and separable and non-separable functions. TROS and QROS were compared with the original PROS algorithm and three well-known and widely used EC algorithms for optimization: genetic algorithm (GA) [18], particle swarm optimization (PSO) [19] and differential evolution (DE) [20]. PROS, TROS and QROS were implemented in Java by the authors. GA, PSO and DE were run using pymoo, an open source framework for multi-objective optimization in Python, version 0.6.0 [21]. The code for the benchmark problems running on pymoo were revised by the authors (for experiments of shifted search space). The hyper-parameters of GA, PSO and DE are selected using the default values from the pymoo package (they are the typical values suggested in the literature).

The experiments were carried out on a 3.20 GHz computer with 16GB RAM under a Windows 10 platform. Multiple runs were conducted for each problem by each algorithm with a different seed for the generation of random numbers for each run. For fair comparison, all algorithms end with the same maximum number of objective function evaluations. We follow the experimental settings of the PROS paper [15] for the population size and the maximum number of objective function evaluations. In order to increase the reliability of the statistical results, the number of runs was increased to 100, 100 and 30 for 5D, 10D, and 50D problems, respectively (instead of 10 for all dimensions in [15]). The settings are summarized in

Table 2. Settings of the numerical experiments.

Settings		$\mathit{D}=5$	$\mathit{D}=10$	$\mathit{D}=50$
Population Size	$10D$	50	100	500
Max. No. of Generations	$(20D-50)$	50	150	950
Max. No. of Objective	$10D(20D-50)$	2500	15,000	475,000
Function Evaluations
No. of Runs		100	100	30

.

4.1. Experiment I: Basic Benchmark Problems

Figure 4 shows the experimental results of the mean error for 100 independent runs of each of the algorithms on each benchmark problem for dimension $D=5$. The x-axes are the number of objective function evaluations (in log scale) and the y-axes are the average error of the algorithm after certain objective function evaluations. As shown in the figure, QROS converged faster than the other five algorithms except on $f_3$ and $f_5$ where PSO converged the fastest. TROS and PROS had similar trends as QROS on the benchmark functions: they converged faster than GA, PSO and DE on most benchmark functions except on $f_3$ and $f_5$. Figure 5 and Figure 6 show the experimental results for dimension $D=10$ and $D=50$, respectively. Similar to the results for $D=5$, QROS converged faster than the other five algorithms except on $f_3$ (for 10D) and $f_5$ (for 10D and 50D) where PSO converged the fastest.

Table 3. Statistical results of 100 runs on 5D benchmark functions.

f	PROS	TROS	QROS
$f_1$	4.65e-03 (4.85e-03)	1.06e-03 (8.88e-04)	4.13e-04 (4.54e-04)
$f_2$	1.38e-02 (1.68e-02)	3.13e-03 (2.65e-03)	1.21e-03 (1.33e-03)
$f_3$	2.17e-01 (1.80e-01)	7.61e-02 (5.05e-02)	4.32e-02 (2.99e-02)
$f_4$	1.34e+00 (1.17e+00)	1.48e+00 (1.16e+00)	1.55e+00 (1.17e+00)
$f_5$	5.13e+00 (6.46e+00)	5.33e-01 (8.45e-01)	1.28e-01 (1.81e-01)
$f_6$	4.81e-03 (3.68e-03)	2.38e-03 (9.69e-04)	1.62e-03 (7.19e-04)
$f_7$	2.41e-01 (2.49e-01)	5.30e-02 (4.49e-02)	2.91e-02 (3.01e-02)
$f_8$	7.48e-01 (4.80e-01)	2.81e-01 (1.65e-01)	1.53e-01 (8.95e-02)
$f_9$	2.69e-01 (1.41e-01)	1.33e-01 (6.76e-02)	8.98e-02 (3.69e-02)
$f_{10}$	4.01e-01 (1.40e-01)	3.34e-01 (1.24e-01)	2.83e-01 (1.26e-01)
$f_{11}$	4.67e-01 (1.34e-01)	4.05e-01 (1.20e-01)	3.72e-01 (9.48e-02)
$f_{12}$	3.49e-01 (1.04e-01)	2.22e-01 (6.44e-02)	1.77e-01 (4.67e-02)

shows the mean of the errors of the final solution vectors returned by PROS, TROS and QROS for 100 independent runs of on each benchmark problem for dimension $D=5$. The mean errors represent the medium to long term performance of the algorithms. The best results (the smallest mean final errors) for each benchmark problem are highlighted in bold font. The corresponding standard deviations are placed next with parenthesis. Owing to limited space, the results of GA, PSO and DE are not shown here, as they have been reported in [15] already. As seen from the table, QROS had the smallest mean final errors on most of the benchmark problems. Similar results can be seen from

Table 4. Statistical results of 100 runs on 10D benchmark functions.

f	PROS	TROS	QROS
$f_1$	8.62e-04 (7.26e-04)	2.38e-04 (1.85e-04)	1.03e-04 (7.51e-05)
$f_2$	4.45e-03 (3.33e-03)	1.34e-03 (1.23e-03)	5.68e-04 (4.63e-04)
$f_3$	6.84e-01 (3.62e-01)	2.52e-01 (1.35e-01)	1.35e-01 (7.28e-02)
$f_4$	3.40e+00 (3.28e+00)	3.72e+00 (2.99e+00)	4.53e+00 (2.91e+00)
$f_5$	4.69e+01 (2.47e+01)	1.40e+01 (9.29e+00)	6.26e+00 (4.77e+00)
$f_6$	3.10e-03 (9.52e-04)	1.70e-03 (5.30e-04)	1.10e-03 (3.52e-04)
$f_7$	4.48e-02 (3.77e-02)	1.24e-02 (9.46e-03)	5.24e-03 (3.56e-03)
$f_8$	1.61e-01 (8.23e-02)	7.13e-02 (3.00e-02)	4.32e-02 (1.55e-02)
$f_9$	1.78e-01 (6.53e-02)	9.15e-02 (3.68e-02)	6.77e-02 (2.74e-02)
$f_{10}$	4.73e-01 (2.46e-01)	4.35e-01 (1.97e-01)	4.31e-01 (1.99e-01)
$f_{11}$	4.90e-01 (1.65e-01)	4.30e-01 (1.49e-01)	3.76e-01 (1.17e-01)
$f_{12}$	3.40e-01 (8.16e-02)	2.20e-01 (4.97e-02)	1.68e-01 (3.85e-02)

and

Table 5. Statistical results of 30 runs on 50D benchmark functions.

f	PROS	TROS	QROS
$f_1$	1.12e-04 (3.30e-05)	2.82e-05 (7.60e-06)	1.24e-05 (4.68e-06)
$f_2$	2.86e-03 (9.68e-04)	7.25e-04 (2.22e-04)	3.29e-04 (1.32e-04)
$f_3$	1.34e+01 (3.13e+00)	6.45e+00 (1.57e+00)	3.75e+00 (8.21e-01)
$f_4$	8.06e+01 (3.80e+01)	6.05e+01 (3.41e+01)	5.48e+01 (2.46e+01)
$f_5$	9.66e+02 (1.63e+02)	7.58e+02 (1.51e+02)	6.44e+02 (1.36e+02)
$f_6$	2.46e-03 (3.48e-04)	1.32e-03 (1.93e-04)	8.85e-04 (1.26e-04)
$f_7$	5.81e-03 (1.72e-03)	1.32e-03 (2.79e-04)	6.25e-04 (1.60e-04)
$f_8$	2.06e-02 (3.28e-03)	9.90e-03 (1.13e-03)	6.49e-03 (9.52e-04)
$f_9$	2.63e-02 (1.51e-02)	1.17e-02 (1.01e-02)	1.39e-02 (1.77e-02)
$f_{10}$	5.99e-01 (2.48e-01)	5.80e-01 (2.25e-01)	5.66e-01 (2.27e-01)
$f_{11}$	6.51e-01 (1.28e-01)	6.17e-01 (1.11e-01)	6.21e-01 (1.08e-01)
$f_{12}$	5.34e-01 (4.80e-02)	3.43e-01 (2.93e-02)	2.61e-01 (2.84e-02)

which show the means of the final errors for dimension $D=10$ and $D=50$, respectively. In general, QROS and TROS are more efficient in reducing the mean final errors on most benchmark problems when compared with PROS.

4.2. Experiment II: Random Shifted Benchmark Problems

One may notice that both TROS and QROS are in favor of objective functions with the global optimum located at the center of the search space. For example, the global optimum of $f_1,f_2,f_3,f_5,f_6,f_7,f_8,f_9,f_{12}$ is $(0,0,\cdots ,0)$. In order to test the effectiveness of TROS and QROS in a general case, another experiment was conducted. The same set of benchmark problems were used in this experiment, but the search space was shifted so that the global optimum may be located anywhere within the bounded search space. A random point $\mathbf{o}=(o_1,o_2,\cdots ,o_D)$ is drawn uniformly from ${\Omega }^{\prime }=[a_1-x_1^{*},b_1-x_1^{*}]\times [a_2-x_2^{*},b_2-x_2^{*}]\times \cdots \times [a_D-x_D^{*},b_D-x_D^{*}]$ for each benchmark function for each run where $a_i,b_i,x_i^{*}$ are the lower limit, upper limit and the optimum value of the i-th decision variable in the original search space of the benchmark function being optimized and $i\in \{1,2,\cdots ,D\}$. The shifted search space becomes ${\Omega }^{\prime \prime }=[a_1-o_1,b_1-o_1]\times [a_2-o_2,b_2-o_2]\times \cdots \times [a_D-o_D,b_D-o_D]$. It has to be noted that the benchmark problems are carefully chosen in this experiment to ensure for each benchmark problem, $x^{*}$ is still the global optimum in the shifted search space but $x^{*}$ is not necessarily be located at the center of the new search space.

Figure 7, Figure 8 and Figure 9 show the experimental results of the mean error of each of the algorithms on each benchmark problem for dimension $D=5$, $D=10$ and $D=50$, respectively. Similar to the non-shifted experiments, PROS, TROS and QROS converged faster than GA, PSO and DE on most benchmark problems except on $f_5$ when $D=50$. When only considering the three random orthogonal search algorithms, PROS usually converged quickly initially, then TROS surpassed PROS, followed by QROS surpassed TROS on most of the benchmark problems.

Table 6. Statistical results of 100 runs on 5D benchmark functions with shifted global optimum.

f	PROS	TROS	QROS
$f_1$	4.31e-03 (4.38e-03)	1.67e-03 (3.92e-03)	1.15e-03 (5.90e-03)
$f_2$	1.24e-02 (1.20e-02)	4.41e-03 (7.05e-03)	2.60e-03 (9.36e-03)
$f_3$	2.64e-01 (2.23e-01)	9.43e-02 (8.36e-02)	5.54e-02 (4.39e-02)
$f_4$	1.38e+00 (3.51e+00)	9.47e-01 (3.44e+00)	1.82e+00 (1.12e+01)
$f_5$	7.76e+00 (1.24e+01)	8.39e-01 (1.16e+00)	2.60e-01 (4.88e-01)
$f_6$	4.62e-03 (2.28e-03)	6.57e-03 (9.53e-03)	1.21e-02 (1.96e-02)
$f_7$	2.23e-01 (2.23e-01)	1.39e-01 (3.07e-01)	2.97e-01 (5.00e-01)
$f_8$	7.34e-01 (4.52e-01)	3.27e-01 (2.86e-01)	2.34e-01 (3.01e-01)
$f_9$	2.94e-01 (1.97e-01)	1.54e-01 (7.26e-02)	1.18e-01 (1.06e-01)
$f_{10}$	3.90e-01 (1.68e-01)	3.18e-01 (1.22e-01)	3.13e-01 (1.38e-01)
$f_{11}$	4.82e-01 (1.21e-01)	4.28e-01 (1.43e-01)	4.15e-01 (1.93e-01)
$f_{12}$	3.47e-01 (1.03e-01)	2.64e-01 (1.33e-01)	2.79e-01 (2.32e-01)

,

Table 7. Statistical results of 100 runs on 10D benchmark functions with shifted global optimum.

f	PROS	TROS	QROS
$f_1$	8.71e-04 (5.89e-04)	2.55e-04 (1.99e-04)	1.64e-04 (2.97e-04)
$f_2$	4.64e-03 (3.58e-03)	1.33e-03 (1.18e-03)	9.91e-04 (2.60e-03)
$f_3$	7.99e-01 (4.64e-01)	2.78e-01 (1.76e-01)	1.62e-01 (9.67e-02)
$f_4$	1.35e+00 (6.26e+00)	8.60e-01 (2.81e+00)	8.20e-01 (2.68e+00)
$f_5$	6.23e+01 (4.86e+01)	1.80e+01 (1.70e+01)	7.34e+00 (7.04e+00)
$f_6$	3.12e-03 (9.80e-04)	2.38e-03 (2.74e-03)	6.05e-03 (9.72e-03)
$f_7$	4.53e-02 (3.06e-02)	6.38e-02 (2.58e-01)	2.65e-01 (5.19e-01)
$f_8$	1.64e-01 (7.02e-02)	8.10e-02 (3.39e-02)	6.33e-02 (1.06e-01)
$f_9$	1.85e-01 (6.35e-02)	9.56e-02 (3.76e-02)	7.97e-02 (3.49e-02)
$f_{10}$	3.84e-01 (2.06e-01)	3.61e-01 (1.80e-01)	3.37e-01 (1.96e-01)
$f_{11}$	4.50e-01 (1.46e-01)	4.51e-01 (2.05e-01)	4.45e-01 (1.92e-01)
$f_{12}$	3.31e-01 (7.39e-02)	2.52e-01 (9.44e-02)	3.03e-01 (2.26e-01)

and

Table 8. Statistical results of 30 runs on 50D benchmark functions with shifted global optimum.

f	PROS	TROS	QROS
$f_1$	1.13e-04 (2.63e-05)	2.88e-05 (1.00e-05)	1.30e-05 (4.43e-06)
$f_2$	2.90e-03 (7.79e-04)	7.43e-04 (3.28e-04)	3.31e-04 (1.23e-04)
$f_3$	1.78e+01 (4.20e+00)	8.55e+00 (1.75e+00)	5.46e+00 (1.20e+00)
$f_4$	2.02e+01 (3.98e+01)	2.59e+00 (8.90e+00)	3.84e+00 (1.35e+01)
$f_5$	1.93e+03 (4.23e+02)	1.55e+03 (3.24e+02)	1.30e+03 (2.85e+02)
$f_6$	2.62e-03 (3.48e-04)	1.69e-03 (5.51e-04)	3.19e-03 (2.83e-03)
$f_7$	5.89e-03 (1.37e-03)	1.58e-03 (4.07e-04)	2.99e-01 (5.23e-01)
$f_8$	2.09e-02 (2.66e-03)	9.85e-03 (1.31e-03)	6.68e-03 (1.06e-03)
$f_9$	2.77e-02 (1.27e-02)	1.92e-02 (1.70e-02)	5.13e-02 (4.70e-02)
$f_{10}$	3.68e-01 (1.11e-01)	3.70e-01 (1.55e-01)	3.32e-01 (9.46e-02)
$f_{11}$	5.82e-01 (1.11e-01)	5.74e-01 (9.44e-02)	6.06e-01 (1.34e-01)
$f_{12}$	5.51e-01 (4.15e-02)	3.93e-01 (1.21e-01)	5.14e-01 (2.79e-01)

show the statistical results of the final error of each of the algorithms on each benchmark problem for dimension $D=5$, $D=10$ and $D=50$, respectively. Similar to the non-shifted experiments, both QROS and TROS often improve the mean final errors. The performance of QROS and TROS are quite promising on the shifted benchmark functions.

5. Future Work

In our future work, we plan to extend the algorithms in two directions. The first one is to allow switching between various sampling polices (e.g., uniform, triangular, quadratic) in an adaptive way. This is particular useful for black-box optimization where the form of the function is unknown to the optimization algorithm. One could learn the best sampling policy for that particular function and adapt gradually from the results of previous function evaluations. The second direction is to gradually rotate the objective function based on the current best solution set [22,23]. By doing so, a complex function would be transformed to a simpler convex function in the small area of concern. Any simple techniques such as TROS and QROS would quickly converge to the global optimum if it is in the area of concern. We are interested in investigating the convergence of TROS and QROS with the addition of transformation techniques.

6. Conclusions

In this paper, we perform an analysis of PROS. PROS is a parameterless EA that outperforms several well-known optimization algorithms on benchmark functions when the search budget is limited. It is attractive for expensive optimizing problems. However, to the best of our knowledge, there is no mathematical analysis of the performance and behavior of PROS reported in the literature. In this paper, we fill this gap.

Moreover, we propose TROS and QROS, which preserve the advantages of PROS and outperform it. We perform an analysis of $tame$ convergence for PROS, TROS and QROS. We conduct two sets of experiments to evaluate the performance of TROS and QROS. We increase the number of runs compared to those conducted in [15] in order to increase the reliability of the statistical results. Experimental results illustrate that TROS and QROS outperform PROS to a large extent. Both TROS and QROS converge quickly on the benchmark problems. Although GA, PSO and DE usually have better performance in terms of smaller error in the final solution when a considerable amount of objective function evaluation is given, the performances of QROS and TROS are not much inferior to that of GA, PSO and DE. In fact, QROS and TROS are even sensible choices when we consider the computational complexity.