A Direct Search Algorithm for Global Optimization

Abstract

A direct search algorithm is proposed for minimizing an arbitrary real valued function. The algorithm uses a new function transformation and three simplex-based operations. The function transformation provides global exploration features, while the simplex-based operations guarantees the termination of the algorithm and provides global convergence to a stationary point if the cost function is differentiable and its gradient is Lipschitz continuous. The algorithm’s performance has been extensively tested using benchmark functions and compared to some well-known global optimization algorithms. The results of the computational study show that the algorithm combines both simplicity and efficiency and is competitive with the heuristics-based strategies presently used for global optimization.

1. Introduction

An optimization problem consists of finding an element of a given set that minimizes (or maximizes) a certain value associated with each element of the set. In this paper, we are interested in minimizing the value of a real valued function defined over the n-dimensional Euclidean space. Without loss of generality, our problem is equivalent to minimizing a real valued function over the open unit n-box ${(0,1)}^n\subset {\mathbb{R}}^n$.

Let f denote a real function defined over the open unit n-box ${(0,1)}^n\subset {\mathbb{R}}^n$, and consider the following optimization problem:

$$\begin{array}{c}\hfill min\{f\left(\mathbf{x}\right):\mathbf{x}\in {(0,1)}^n\}\end{array}$$

(1)

The function to be minimized f is called the cost function and the unit n-box ${(0,1)}^n$ is the search space of the problem. Our aim is to obtain a point ${\mathbf{x}}^{*}\in {(0,1)}^n$, such that the cost function f attains a minimum at ${\mathbf{x}}^{*}$, i.e., $f\left({\mathbf{x}}^{*}\right)\le f\left(x\right)$, $\forall \mathbf{x}\in {(0,1)}^n$. We shall make the following assumption:

A1 

The cost function f attains its minimum at a point of the search space.

If the function f is continuously differentiable in ${(0,1)}^n$, then:

$$\begin{array}{c}\hfill \nabla f\left({\mathbf{x}}^{*}\right)=0,{\mathbf{x}}^{*}\in (0,1)\end{array}$$

(2)

Many optimization methods make use of the stationarity Equation (2) to compute candidate optimizer points. These methods require the explicit knowledge of the gradient, but they do not guarantee that the point obtained is an optimizer, except for a pseudoconvex function.

We shall also make an additional assumption that will restrict our alternatives to design a procedure to compute the optimum.

A2 

The gradient of the cost function is not available for the optimization mechanism.

Consequently, only function evaluations can be used to compute the minimum.

Remark 1. 

The problem Equation (1) is equivalent to unconstrained optimization. Consider the following unconstrained optimization problem:

$$\begin{array}{c}\hfill min\{f\left(\mathit{\xi }\right):\mathit{\xi }\in {\mathbb{R}}^n\}\end{array}$$

(3)

and let $\mathit{\xi }={\left[\begin{array}{cccc}{\xi }_1& {\xi }_2& \cdots & {\xi }_n\end{array}\right]}^T$ be the vector of variables of the cost function. The invertible transformation:

$$\begin{array}{c}\hfill x_i=\frac{1}{1+e^{-{\xi }_i}}\end{array}$$

(4)

transforms the real line into the unit interval. Consequently, we can convert an unconstrained minimization problem into an open unit n-box constrained minimization problem by transforming each of its variables. Similarly, the linear transformation:

$$\begin{array}{c}\hfill x_i=\frac{{\xi }_i-a}{b-a}\end{array}$$

(5)

converts the open interval $(a,b)$ into the open unit interval. Consequently, both unconstrained and arbitrary open n-box optimization problems are included in our formulation. Furthermore, under Assumption A2 and by choosing appropriate values of a and b, the global minimum of function f is attained in an interior point of the open n-box ${(a,b)}^n$.

The rest of this paper is organized as follows. Section 2 is a review of the literature on direct search and global optimization algorithms. The direct search algorithm for global optimization is designed in Section 3. The algorithm makes use of a set of simplex-based operations for a transformed cost function. We begin by reviewing some basic results about n-dimensional simplices that will be used later to define the algorithm and study its properties. Then, we will introduce a transformation of the cost function that is crucial to improving the exploratory properties of the algorithm. Finally, we conclude this section by explaining the algorithm in detail and analyzing its convergence properties. In Section 4, an experimental study of the algorithm’s performance is accomplished by using three well-known test functions. Its performance is also compared to other global optimization strategies. The conclusions are presented in Section 5.

2. Direct Search Methods and Global Optimization

A direct search method solves optimization problems without requiring any information about the gradient of the cost function [1,2]. Unlike many traditional optimization algorithms that use information about the gradient or higher derivatives to search for an optimal point, a direct search algorithm searches a set of points around the current point, looking for one where the value of the cost function is lower than the value at the current point. Direct search methods solve problems for which the cost function is not differentiable or not even continuous. Direct search optimization has been receiving increasing attention over recent years within the optimization community, including the establishment of solid mathematical foundations for many of the methods considered in practice [1,3,4,5,6,7]. These techniques do not attempt to compute approximations to the gradient, but rather, the improvement is derived from a model of the cost function or using geometric strategies. There exists a great number of problems in different fields, such as engineering, mathematics, physics, chemistry, economics, medicine, computer science or operational research, where derivatives are not available, but there is a need for optimization. Some relevant examples are: tuning of algorithmic parameters, automatic error analysis, structural design, circuit design, molecular geometry or dynamic pricing. See Chapter 1 of [6] for a detailed description of the applications.

Initially, direct search methods have been dismissed for several reasons [8]. Some of them are the following: they are based on heuristics, but not on rigorous mathematical theory; they are slow to converge and only appropriate for small-sized problems; and finally, there were no mathematical analysis tools to accompany them. However, these drawbacks can be refuted today. Regarding slow convergence, in practice, one only needs improvement rather than optimality. In such a case, direct methods can be a good option because they can obtain an improved solution even faster than local methods that require gradient information. In addition, direct search methods are straightforward to implement. If we not only consider the elapsed time for a computer to run, but the total time needed to formulate the problem, program the algorithm and obtain an answer, then a direct search is also a compelling alternative. Regarding the problem size, it is usually considered that direct search methods are best suited for problems with a small number of variables. However, they have been successfully applied to problems with a few hundred variables. Finally, regarding analytical tools, several convergence analysis studies for a direct search algorithm were published starting in the early 1970s [9,10,11,12]. These results prove that it is possible to provide rigorous guarantees of convergence for a large number of direct search methods [4], even in the presence of heuristics. Consequently, direct search methods are regarded today as a respectable choice, and sometimes the only option, for solving certain classes of difficult and practical optimization problems, and they should not be dismissed as a compelling alternative in these cases. Direct search methods are broadly classified into local and global optimization approaches.

2.1. Local Optimization Approaches

Since there does not exist a suitable algorithmic characterization of global optima, most optimizers are based on local optimization. The most important derivative-free local based methods are pattern search methods and model-based search methods. Pattern search methods perform local exploration of the cost function on a pattern (a set of points conveniently selected). The Hooke–Jeeves [13] method and the Nelder–Mead simplex method [14] are two well-known examples of pattern search local methods.

The method of Hooke and Jeeves consists of a sequence of exploratory moves from a base point in the coordinate directions, which, if successful, are followed by pattern moves. The purpose of an exploratory move is to acquire information about the cost function in the neighborhood of the current base point. A pattern move attempts to speed up the search by using the information already acquired about the function to identify the best search direction.

The Nelder–Mead simplex algorithm belongs to the class of simplex-based direct search methods introduced by Spendley et al. in [15]. This class of methods evolves a pattern set of $n+1$ vectors on an n-dimensional search space that is interpreted as the vertex set of a simplex, i.e., a convex n-dimensional polytope. Different simplex-based methods use different operators to evolve the vertex set. The Nelder–Mead algorithm starts by ordering the vertices of the simplex and uses five operators: reflection, expansion, outer contraction, inner contraction and shrinkage. At each iteration, only one of these operations is performed. As a result, a new simplex is obtained such that either contains a better vertex (a vertex that is better than the worst one and substitutes it) or has a smaller volume. From the point of view of the function value, every operator, except for the shrinkage, is productive and obtains a vertex that improves the worst one. The shrinkage operator reduces the volume of the simplex to guarantee algorithm termination. Unfortunately, the convergence of the Nelder–Mead simplex algorithm to a local minimizer is not guaranteed [16], and a number of modifications have been introduced to get convergence. Nevertheless, the Nelder–Mead simplex algorithm remains as one of the most popular direct search algorithms.

Model-based search methods are based on the idea of using function evaluations to compute a model of the cost function and to obtain derivative approximations from this model. Many model-based methods are trust-region algorithms [17] that interpolate the cost function in some region of appropriate shape and size to obtain a good approximation of the cost function that can later be easily optimized on that region. Typically, the model to be adjusted is a fixed-order polynomial, and the trust region is a sphere in some norm of the search space. The model function is sequentially optimized in the trust region, and the trust region is updated for the next iteration until the approximated model satisfies a first-order optimality condition. Model-based methods also include [18,19,20]. During the last decade, there has been a considerable amount of work to handle constraints in the field of direct search methods. New algorithms have been developed to deal with bounds and linear inequalities [21,22] smooth nonlinear constraints [23,24] or non-smooth constraints [25,26,27]. In this work, we do not deal with constraints. Our aim here is to develop a simple and efficient algorithm that improves the heuristics-based optimization methods that are presently used to obtain the global minimum in a multidimensional and multimodal continuous function.

2.2. Global Optimization Approaches

Local optimization methods are the best option for convex, or at least unimodal, optimization problems. However, most real-world problems are not convex, and the solution provided by a local method is not guaranteed to be global. Unfortunately, even if derivatives are available, it is not possible in general to prove that a local optimum is global. A rigorous global optimization algorithm for a multimodal cost function requires in-depth exploration of the search space, and this is only possible for functions with a small number of variables, because the computational time for space exploration increases exponentially with the dimension of the search space.

In order to circumvent the curse of dimensionality, many global optimization algorithms use random search [28] and heuristics [29,30]. Some successful classes of global optimization algorithms are simulated annealing algorithms [31], evolutionary algorithms [32], estimation of distribution algorithms [33], particle swarm optimization algorithms [34,35], differential evolution algorithms [36,37], tabu search algorithms [38,39] or ant colony optimization algorithms [40,41].

In recent years, there has been a great interest in derivative-free methods for global optimization; see the monographs [42,43,44,45,46] and the references therein. In particular, there are two classes of algorithms that are of special interest. The first class is Lipschitz optimization methods. Lipschitz methods substitute the cost function by an auxiliary function that underestimates it. The auxiliary function is linear piecewise and is obtained by partitioning the search space and repeatedly using the Lipschitz property [47,48,49,50]. An interesting Lipschitz method is DIRECT (DIviding RECTangles), that was introduced in [51]. In this method, the cost function is evaluated at several sample points by using all possible weights on local versus global search expressed by the Lipschitz constant. The DIRECT algorithm has gained popularity due to its simplicity. However, it has two weaknesses. First, it is slow in reaching the solution with high accuracy, and second, it spends an excessive number of evaluations exploring local minima. Some variants have been developed to improve these weaknesses [52,53,54,55].

Another class of global optimization methods uses a filled function to avoid local minima. The basic idea is to design an auxiliary function, called the filled function, such that when the algorithm reaches a local minimum, the minimization of the filled function provides a point that moves away from the basin of the local minimum. The filled function method was introduced by Ge [56,57]. It has had increasing interest in filled methods in the last few years [58,59,60].

3. A Direct Search Algorithm for Global Optimization

The direct search algorithm for global optimization proposed here has two main ingredients: a transformed cost function and a simplex-based mechanism to evolve the initial point. The transformed cost function improves global exploration, while the evolution mechanism guarantees the termination and global convergence to a stationary point. To the best of our knowledge, the transformed cost function has not been previously used in the literature, and it is a key ingredient of our approach, because it provides global exploration of the search space. The new algorithm is an efficient alternative to heuristics-based optimization methods for global optimization and preserves the convergence properties of a well-designed direct search method.

The algorithm makes use of an initial point that is allocated as a qualified vertex of an n-dimensional initial simplex. This simplex is evolved by a sequence of operations that depend on the value of the transformed cost function at each vertex. Each operation produces a similar simplex of a different size. The sequence of operations is designed in such a way that the simplex tends to asymptotically collapse in a single point. The algorithm terminates when the simplex vertices are close enough to each other.

Before explaining the algorithm and its operators in detail, we shall review some basic geometric results about n-simplices. Later, we shall introduce the cost function transformation and its main properties.

3.1. Notation

A brief summary of the notation used in this paper is included next for quick reference.

Set of integer numbers in interval $\mathcal{I}$: ${\mathbb{Z}}_{\mathcal{I}}:=\{z\in \mathbb{Z}:z\in \mathcal{I}\}$.

Closed convex hull of a set $\mathcal{A}$: $\mathbf{Co}\left(\mathcal{A}\right)$.

Difference set $\mathcal{A}\setminus \mathcal{B}=\{x\in \mathcal{A}:x\notin \mathcal{B}\}$.

Larger integer less than or equal to x: $\lfloor x\rfloor$.

Inner product of Euclidean space: $\langle \mathbf{x},\mathbf{y}\rangle ={\mathbf{x}}^T\mathbf{y}$.

p-norm of Euclidean space: ${\parallel \mathbf{x}\parallel }_p={\left({\displaystyle \sum _{i=1}^n}{\left|x_i\right|}^p\right)}^{\frac{1}{p}}$, $p\in {\mathbb{Z}}_{[1,\infty )}$.

∞-norm of Euclidean space: ${\parallel \mathbf{x}\parallel }_{\infty }=\underset{p\to \infty }{lim}{\parallel \mathbf{x}\parallel }_p=max\left\{\right|x_i|:\forall i\}$.

3.2. Review of the Basic Geometric Results about n-Simplices

Let $\mathcal{V}=\{{\mathbf{x}}_i:i\in {\mathbb{Z}}_{[0,n]}\}$ be a set of $n+1$ affinely-independent points in ${\mathbb{R}}^n$, then the closed convex hull of $\mathcal{V}$ is an n-simplex Σ with vertex set $\mathcal{V}$, and it is denoted as follows:

$$\begin{array}{c}\hfill \Sigma =\mathbf{Co}\left(\mathcal{V}\right)\end{array}$$

(6)

Next, we introduce certain specific types of n-simplices that are non-standard in the literature.

Definition 1 

(Right n-simplex). A right n-simplex has a vertex, such that the n edges intersecting at it are pairwise orthogonal. This vertex is called the right vertex.

Definition 2 

(Isosceles right n-simplex). A right n-simplex, such that every edge intersecting at the right vertex has the same length Δ, is an isosceles right n-simplex of length Δ.

Definition 3 

(Standard n-simplex). The standard n-simplex is an isosceles right n-simplex of unit length.

By locating the zero of ${\mathbb{R}}^n$ in the right vertex, the standard n-simplex ${\Sigma }_n^0$ is the n-dimensional polytope with vertex set $\mathcal{V}=\{{\mathbf{b}}_i\in {\mathbb{R}}^n:i\in {\mathbb{Z}}_{[0,n]}\}$ where ${\mathbf{b}}_0$ is the zero vector, and ${\mathbf{b}}_i$ is the i-th element of the standard basis of ${\mathbb{R}}^n$. Alternatively, the standard n-simplex can be defined as the intersection of the closed unit ball in the one-norm of ${\mathbb{R}}^n$ and the non-negative orthant, i.e., ${\Sigma }_n^0{=\{\mathbf{x}\in [0,1]}^n{:\parallel \mathbf{x}\parallel }_1\le 1\}$.

The content of an n-dimensional object is the measure of the amount of space inside its boundary. For instance, the content of a two-simplex is its area, and the content of a three-simplex is its volume. It can be proven by induction that the content of a standard n-simplex is $\frac{1}{n!}$.

A regular n-box is an n-dimensional box with edges of equal length. If the content of an n-simplex is not zero, then a geometric object of nonzero content can fit in its interior. The following lemma provides the edge length of the maximum regular n-box that can fit in the interior of a standard n-simplex.

Lemma 1. 

The regular n-box of maximum size that can be contained inside a standard n-simplex has edge length $\frac{1}{n}$ and content ${\left(\frac{1}{n}\right)}^n$.

Proof. 

The standard n-simplex is given as ${\Sigma }_n^0{=\{\mathbf{x}\in [0,1]}^n{:\parallel \mathbf{x}\parallel }_1\le 1\}$. The maximum n-box that can be allocated inside the standard n-simplex has a vertex in the origin, and it is oriented along the coordinate directions. The opposite vertex to the origin of this regular n-box has coordinates $x_i=\frac{1}{n}$ for $i\in {\mathbb{Z}}_{[1,n]}$; therefore, its edge length is $\frac{1}{n}$, and its content is ${\left(\frac{1}{n}\right)}^n$. ☐

The edges that intersect at the right vertex of a standard n-simplex are oriented along the coordinate directions, so these edge directions define a nonnegative orthant. Any direction inside this orthant can be expressed by a nonnegative n-dimensional vector of unit norm, i.e., $\langle \mathbf{d},\mathbf{d}\rangle =1$. A direction $\mathbf{d}$ forms angles ${\theta }_i$ with the coordinate directions whose cosines are called the directional cosines and are given by $cos{\theta }_i=\langle \mathbf{d},{\mathbf{b}}_i\rangle$. The following lemma proves that any direction $\mathbf{d}$ contained in the nonnegative orthant has at least one directional cosine that never is less than $\frac{1}{\sqrt{n}}$.

Lemma 2. 

The smallest directional cosine of any direction contained in the orthant formed by the edges intersecting at the right vertex of a standard n-simplex is never less than $\frac{1}{\sqrt{n}}$.

Proof. 

Without loss of generality, suppose that the right vertex is the zero point and that the edges are in the coordinate directions, characterized by the elements of the standard basis of ${\mathbb{R}}^n$, i.e., ${\mathbf{b}}_i$ for $i\in {\mathbb{Z}}_{[0,n]}$. Consider the direction ${\mathbf{d}}^{*}=\left[\begin{array}{ccc}\frac{1}{\sqrt{n}}& \cdots & \frac{1}{\sqrt{n}}\end{array}\right]$ whose directional cosines are $cos{\theta }_i^{*}=\langle {\mathbf{d}}^{*},{\mathbf{b}}_i\rangle =\frac{1}{\sqrt{n}}$ for all $i\in {\mathbb{Z}}_{[1,n]}$. Suppose that unlike the lemma statement, there exists a direction $\mathbf{d}$ with unit norm $\langle \mathbf{d},\mathbf{d}\rangle =1$, such that $cos{\theta }_i=\langle \mathbf{d},{\mathbf{b}}_i\rangle <\frac{1}{\sqrt{n}}$ for all $i\in {\mathbb{Z}}_{[1,n]}$, then $\langle \mathbf{d},\mathbf{d}\rangle <n{\sum }_{i\in {\mathbb{Z}}_{[1,n]}}\frac{1}{n}=1$. This contradicts the fact that $\mathbf{d}$ has unit norm, and the lemma is proven. ☐

A consequence of this lemma is that any direction contained in the nonnegative n-orthant always forms an angle less than $\frac{\pi }{2}$ radians with at least one of the coordinate directions that form the orthant. In addition, if we consider an arbitrary direction $\mathbf{d}\in {\mathbb{R}}^n$, then there always exists an n-simplex similar to the standard n-simplex, such that the edge directions of its right vertex define an orthant, such that the smallest directional cosine is never less than $\frac{1}{\sqrt{n}}$. This n-simplex is obtained by rotating π radians the edges of the standard n simplex corresponding to the negative components of the direction vector $\mathbf{d}$.

3.3. Function Transformation

A key ingredient of our direct search global optimization algorithm is a transformation of the cost function that improves its exploratory features, providing the capability of jumping out of local minimizers, while preserving termination and convergence properties.

Let $\mathbf{x}\in {\mathbb{R}}^n$ be an n-dimensional vector; $\lfloor \mathbf{x}\rfloor$ denotes the n-dimensional vector whose components are the greatest integers less than or equal to the components of $\mathbf{x}$. The fractional part of $\mathbf{x}$ is denoted as $\mathrm{frac}\left(\mathbf{x}\right)$ and is defined as follows:

$$\begin{array}{cc}\hfill \mathrm{frac}\left(\mathbf{x}\right)& =\mathbf{x}-\lfloor \mathbf{x}\rfloor \hfill \end{array}$$

(7)

Let ξ be an arbitrary n-dimensional vector of real numbers and P a positive integer. Let us introduce the following map:

$$\begin{array}{c}\hfill \mathbf{x}\left(\mathit{\xi }\right)=\mathrm{frac}\left(P\mathit{\xi }\right)\end{array}$$

(8)

This map is onto and transforms the n-dimensional Euclidean space ${\mathbb{R}}^n$ in the n-dimensional unit interval ${(0,1)}^n$. Let f be the cost function of the minimization problem given in Equation (1) and introduce the following function transformation:

$$\begin{array}{c}\hfill \varphi \left(\mathit{\xi }\right):=f\left(\mathbf{x}\right(\mathit{\xi }\left)\right)=f\left(\mathrm{frac}\right(P\mathit{\xi }\left)\right)\end{array}$$

(9)

where P is a positive integer. Since the cost function f is not necessarily defined outside of the open unit n-box, the domain of the transformed cost function φ is:

$$\begin{array}{c}\hfill \mathrm{dom}\varphi =\{\mathit{\xi }\in {\mathbb{R}}^n:P\mathit{\xi }\notin {\mathbb{Z}}^n\}\end{array}$$

(10)

The reason is that if $P\mathit{\xi }\in {\mathbb{Z}}^n$, then $\mathrm{frac}\left(P\mathit{\xi }\right)=0$, and f is not necessarily defined for $\mathit{\xi }=0$.

The transformed function φ has certain interesting properties. The most important are the periodicity of period $P^{-1}$ and piecewise continuity, both on each variable.

Lemma 3. 

The transformed function $\varphi \left(\mathit{\xi }\right)=f\left(\mathrm{frac}\right(P\mathit{\xi }\left)\right)$ satisfies the following properties:

(i) 

The function φ is continuous for any $\mathit{\xi }\in {\mathbb{R}}^n$, such that $P\mathit{\xi }\in {(0,1)}^n$.

(ii) 

Let $\mathit{\xi },\mathit{\eta }\in {\mathbb{R}}^n$ be such that $P\mathit{\xi }\in {(0,1)}^n$ and $P\mathit{\eta }\in {\mathbb{Z}}^n$, then $\varphi (\mathit{\xi }+\eta )=\varphi \left(\mathit{\xi }\right)$.

Proof. 

(i) If $P\mathit{\xi }\in {(0,1)}^n$, then $\mathrm{frac}\left(P\mathit{\xi }\right)=P\mathit{\xi }$ and $\varphi \left(\mathit{\xi }\right)=f\left(P\mathit{\xi }\right)$. Since f is continuous for any $\mathbf{x}\in {(0,1)}^n$, then φ is continuous for any $\mathit{\xi }\in {\mathbb{R}}^n$, such that $P\mathit{\xi }\in {(0,1)}^n$; (ii) If ξ and η are n-dimensional vectors, such that $P\mathit{\xi }\in {(0,1)}^n$ and $P\mathit{\eta }\in {\mathbb{Z}}^n$, respectively, then $\mathrm{frac}\left(P\mathit{\eta }\right)=0$ and $\mathrm{frac}\left(P\right(\mathit{\xi }+\mathit{\eta }\left)\right)=\mathrm{frac}\left(P\mathit{\xi }\right)=P\mathit{\xi }$; consequently, $\varphi (\mathit{\xi }+\mathit{\eta })=f\left(\mathrm{frac}\right(P\mathit{\xi }\left)\right)=\varphi \left(\mathit{\xi }\right)$. ☐

Let $\mathbf{x}\in {\mathbb{R}}^n$ and ${\mathcal{B}}_P\left(\mathbf{x}\right)$ denote the following n-box:

$$\begin{array}{c}\hfill {\mathcal{B}}_P\left(\mathbf{x}\right)=\{\mathit{\xi }\in {\mathbb{R}}^n:\lfloor P\mathit{\xi }\rfloor =\lfloor P\mathbf{x}\rfloor ,P\mathit{\xi }\notin {\mathbb{Z}}^n\}\end{array}$$

(11)

The transformed function φ is continuous on ${\mathcal{B}}_P\left(\mathbf{x}\right)$, and there always exists ${\mathit{\xi }}^{*}\in {\mathcal{B}}_P\left(\mathbf{x}\right)$, such that $\varphi \left({\mathit{\xi }}^{*}\right)=min\{\varphi \left(\mathit{\xi }\right):\mathit{\xi }\in \mathrm{dom}\varphi \}$. Note that the n-box $\mathcal{B}\left(\mathbf{x}\right)$ is given by ${\mathcal{B}}_P\left(\mathbf{x}\right)=\{\mathit{\xi }=P^{-1}(\mathbf{z}+\lfloor P\mathbf{x}\rfloor ):\mathbf{z}\in {(0,1)}^n\}$. If $\mathbf{x}=0$, then ${\mathcal{B}}_P\left(\mathbf{0}\right)$ is the fundamental n-box of the function φ, i.e., $B_P\left(\mathbf{0}\right)={(0,P^{-1})}^n$. For another $\mathbf{x}\in {\mathbb{R}}^n$, ${\mathcal{B}}_P\left(\mathbf{x}\right)$ is an n-box of length $P^{-1}$, where the function φ takes the same values as in the fundamental n-box ${\mathcal{B}}_P\left(\mathbf{0}\right)$. Note that there are $P^n$ different n-boxes for $\mathbf{x}\in {(0,1)}^n$; then, the function φ repeats $P^n$ times in the unit n-box ${(0,1)}^n$.

Example 1. 

The function $f\left(x\right)=\left|x\right|-|\sqrt{x}sin\left(3\pi x\right)|$, and the transformed function $\varphi \left(\mathit{\xi }\right)=f\left(\mathrm{frac}\right(P\mathit{\xi }\left)\right)$ for $P=10$ are depicted for the open unit interval in Figure 1. The function f is continuous on the open unit interval $(0,1)$ and has three local minima. The transformed function φ is periodic, of period $P^{-1}=0.1$, and piecewise continuous on the open unit interval. The fundamental period of φ is $\mathcal{B}\left(0\right)=(0,0.1)$, and the function value repeats ten times on the unit interval. The function φ is continuous on each interval $\mathcal{B}\left(x\right)=\{\xi :\lfloor 10\xi \rfloor =\lfloor 10x\rfloor ,10\xi \notin \mathbb{Z}\}$ for $x\in \mathbb{R}$, but discontinuous at the boundary points of these intervals, which are given by $\xi =0.1k$ for any integer k. This discontinuity always occurs if $f(0+ϵ)\ne f(1-ϵ)$ for any $ϵ>0$, as occurs in this example and which is clearly visible in Figure 1. Moreover, the function φ is not necessarily defined for these points of discontinuity.

Consider the unconstrained optimization problem:

$$\begin{array}{c}\hfill min\left\{\varphi \right(\mathit{\xi }):\mathit{\xi }\in \mathrm{dom}\varphi \}\end{array}$$

(12)

The function φ has a global optimizer at ${\mathit{\xi }}^{*}$ if $\varphi \left({\mathit{\xi }}^{*}\right)\le \varphi \left(\mathit{\xi }\right)$ for all $\mathit{\xi }\in \mathrm{dom}\varphi$. The following theorem proves that the transformed unconstrained minimization problem Equation (12) also provides the minimum value of the original optimization problem Equation (1).

Theorem 1. 

Let f be a real value continuous function on the open unit n-box, and let φ be defined as $\varphi \left(\mathit{\xi }\right)=f\left(\mathrm{frac}\right(P\mathit{\xi }\left)\right)$, where P is a positive integer, then ${\lambda }^{*}=min\{f\left(\mathbf{x}\right):\mathbf{x}\in {(0,1)}^n\}$ if and only if ${\lambda }^{*}=min\{\varphi \left(\mathit{\xi }\right):\mathit{\xi }\in \mathrm{dom}\varphi \}$.

Proof. 

(If): Let ${\mathbf{x}}^{*}$ be such that ${\lambda }^{*}=f\left({\mathbf{x}}^{*}\right)=min\{f\left(\mathbf{x}\right):\mathbf{x}\in {(0,1)}^n\}$ and $\varphi \left(\mathit{\xi }\right)=f\left(\mathrm{frac}\right(P\mathit{\xi }\left)\right)$, where P is a positive integer; then define ${\mathit{\xi }}^{*}=P^{-1}({\mathbf{x}}^{*}+\mathit{\eta })$ for any $\mathit{\eta }\in {\mathbb{Z}}^n$; then $\varphi \left({\mathit{\xi }}^{*}\right)=f\left({\mathbf{x}}^{*}\right)={\lambda }^{*}$. (Only if): Let ${\mathit{\xi }}^{*}$ be such that ${\lambda }^{*}=\varphi \left({\mathit{\xi }}^{*}\right)=min\{\varphi \left(\mathit{\xi }\right):\mathit{\xi }\in \mathrm{dom}\varphi \}$; define ${\mathbf{x}}^{*}=\mathrm{frac}\left(P{\mathit{\xi }}^{*}\right)$; then $f\left({\mathbf{x}}^{*}\right)=\varphi \left({\mathit{\xi }}^{*}\right)={\lambda }^{*}$. ☐

Theorem 1 allows us to obtain a global minimizer of the original function f on the open unit n-box by solving the unconstrained optimization problem (12). If we have a method that computes a global minimizer ${\mathit{\xi }}^{*}$ of the unconstrained optimization problem (12), then ${\mathbf{x}}^{*}=\mathrm{frac}\left(P{\mathit{\xi }}^{*}\right)\in {(0,1)}^n$, and it satisfies $\varphi \left({\mathit{\xi }}^{*}\right)=f\left({\mathbf{x}}^{*}\right)$.

The following lemma proves that any regular n-box of a size greater than or equal to $P^{-1}$ always contains a point ${\mathit{\xi }}^{*}$ that attains the global minimum of the unconstrained optimization problem (12).

Lemma 4. 

Let $\mathcal{B}$ be a regular n-box of edge length ${\Delta }_B$, such that $P{\Delta }_B\ge 1$, and let ${\lambda }^{*}=min\{f\left(\mathbf{x}\right):\mathbf{x}\in (0,1)\}$; then, the n-box $\mathcal{B}$ always contains a point ${\mathit{\xi }}^{*}\in {\mathbb{R}}^n$, such that $\varphi \left({\mathit{\xi }}^{*}\right)={\lambda }^{*}$.

Proof. 

Let ${\mathbf{x}}^{*}\in {(0,1)}^n$ be such that $f\left({\mathbf{x}}^{*}\right)={\lambda }^{*}\le f\left(\mathbf{x}\right)$ for any $\mathbf{x}\in {(0,1)}^n$, and let $\overline{\mathit{\xi }}\in \mathcal{B}$, such that $\lfloor P\overline{\mathit{\xi }}\rfloor =\lfloor P\overline{\mathit{\xi }}+{\mathbf{x}}^{*}\rfloor$. Consider the open n-box ${\mathcal{B}}_P\left(\overline{\mathit{\xi }}\right)=\{\mathit{\xi }\in {\mathbb{R}}^n:\lfloor P\mathit{\xi }\rfloor =\lfloor P\overline{\mathit{\xi }}\rfloor ,P\mathit{\xi }\notin {\mathbb{Z}}^n\}$, then ${\mathcal{B}}_P\left(\overline{\mathit{\xi }}\right)$ has edge length $P^{-1}$, and the point ${\mathit{\xi }}^{*}=P^{-1}({\mathbf{x}}^{*}+\lfloor P\overline{\mathit{\xi }}\rfloor )$ satisfies ${\mathit{\xi }}^{*}\in \mathcal{B}\left(\overline{\mathit{\xi }}\right)\cap \mathcal{B}$ and $\varphi \left({\mathit{\xi }}^{*}\right)=f\left({\mathbf{x}}^{*}\right)={\lambda }^{*}$. ☐

We are also interested in determining the size of an n-simplex similar to the standard n-simplex that always contains a global minimizer of $\left\{\varphi \right(\mathit{\xi }):\mathit{\xi }\in \mathrm{dom}\varphi \}$. Since Lemma 4 establishes that such a point is always contained in a regular n-box of length not less than $P^{-1}$, then the solution is an isosceles right n-simplex that contains a regular n-box of length $P^{-1}$. From Lemma 1, this n-simplex has edge length $n{P}^{-1}$ and content $P^{-n}/(n-1)!$. The following lemma states the result.

Theorem 2. 

Let Σ be an isosceles right n-simplex of length Δ, such that $P\Delta \ge n$, and let ${\lambda }^{*}=min\{f\left(\mathbf{x}\right):\mathbf{x}\in {(0,1)}^n\}$; then, the n-simplex Σ always contains a point ${\mathit{\xi }}^{*}\in \mathrm{dom}\varphi$, such that $f\left(\mathrm{frac}\left({\mathit{\xi }}^{*}\right)\right)={\lambda }^{*}$.

Proof. 

A regular n-box of edge length ${\Delta }_B=\Delta /n$ can be inscribed inside any isosceles right n-simplex of length Δ. Since $P\Delta \ge n$, then $P{\Delta }_B\ge 1$, and the theorem statement is a straightforward consequence of Lemma 4. ☐

In the rest of this section, we shall design a direct search algorithm to obtain a solution for the optimization problem (1). We distinguish two algorithms, the basic algorithm and the complete algorithm. The basic algorithm is a simplex-based algorithm for the transformed cost function. An initial point is embedded at the right vertex of an isosceles right n-simplex of edge length ${\Delta }_0$. This n-simplex evolves by a sequence of operations. As a result of each iteration, another n-simplex is obtained that is similar to the standard n-simplex and has the point with minimum value of the transformed cost function located in the right vertex. The edge length of the n-simplex decreases by a constant factor if no operations performed during the iteration produce a vertex with a smaller value of the transformed cost function. Whenever the edge length is no less than $n{P}^{-1}$, we say that the algorithm is in exploratory phase, because the n-simplex always contains a global minimizer of $\left\{\varphi \right(\mathit{\xi }):\mathit{\xi }\in \mathrm{dom}\varphi \}$. When the edge length of the n-simplex is less than $n{P}^{-1}$, then we say that the algorithm is in convergence phase, and it aims to approach a stationary point. Thus, the basic algorithm is designed to preserve the good properties of direct search algorithms, such as convergence to a stationary point, but increases the possibility to reach the global optimum at the end of the exploratory phase. Another feature of the basic algorithm is that there is no interruption or change from one phase to the other, it is a straightforward consequence of using the transformed cost function.

3.4. The Basic Algorithm

The basic algorithm is a simplex-based algorithm for the transformed optimization problem Equation (12). It performs a sequence of operations over an initial simplex until a termination condition is satisfied. The main operators of this algorithm are expansive translation, rotation and shrinkage. The expansive translation is performed after a sufficient decrease of the transformed cost function to ensure that the best point is located at the right vertex. The rotation operation provides exploration of the search space, and the shrinkage operator guarantees termination and global convergence to a stationary point of the transformed cost function. The global improvement is a consequence of using the transformed cost function Equation (8). A sufficient decrease condition of the cost function to accept a successful iteration is a key aspect to prove the termination and global convergence to a stationary point. The main components of the basic algorithm are explained below.

3.4.1. Function Transformation

The cost function transformation is given by:

$$\begin{array}{c}\hfill \varphi \left(\mathit{\xi }\right)=f\left(\mathrm{frac}\right(P\mathit{\xi }\left)\right)\end{array}$$

(13)

This transformation aims to perform global improvement at each iteration $k\in {\mathbb{Z}}_{[0,\infty )}$ whenever the n-simplex has an edge length, such that $P{\Delta }_k\ge n$.

3.4.2. Initial Simplex

Let $\mathrm{vset}(\mathit{\xi },\Delta )$ where $\mathit{\xi }\in {\mathbb{R}}^n$ and $\Delta \in \mathbb{R}$ be defined as follows:

$$\mathrm{vset}(\mathit{\xi },\Delta ):=\{{\mathit{\xi }}_i:{\mathit{\xi }}_i=\mathit{\xi }+\Delta {\mathbf{b}}_i,i\in {\mathbb{Z}}_{[1,n]}\}$$

(14)

where ${\mathbf{b}}_0$ is an n-dimensional zero vector and ${\mathbf{b}}_i$ for $i\in {\mathbb{Z}}_{[1,n]}$ is the i-th element of the standard basis of ${\mathbb{R}}^n$. The initial n-simplex is obtained by varying an initial point at a distance ${\Delta }_0$ in each coordinate direction, such that $P{\Delta }_0>n$. Let ${\mathit{\xi }}_0\in {(0,1)}^n$ be an initial point; then, the vertex set of the initial simplex is given by:

$${\mathcal{V}}_0=\mathrm{vset}({\mathit{\xi }}_0,{\Delta }_0)$$

(15)

If the initial point ${\mathit{\xi }}_0$ is not provided as an input of the algorithm, then it can be randomly chosen. Consequently, the initial simplex is an isosceles right n-simplex with edge length ${\Delta }_0$ and content $\frac{1}{n!}{\Delta }_0^n$ whose right vertex is indexed by zero.

3.4.3. Expansive Translation

This operation aims to position the point with lowest value of the transformed cost function found so far at the right vertex. Let $\mathcal{V}$ be the vertex set of an isosceles right n-simplex with edge length Δ, and let ${\mathit{\xi }}_i\in \mathcal{V}$ be such that $\varphi \left({\mathit{\xi }}_i\right)\le \varphi \left({\mathit{\xi }}_j\right)$ for ${\mathit{\xi }}_j\in \mathcal{V}$; then, the expansive translation of the right n-simplex $\mathcal{V}$ with respect to the vertex i is given by the following expression:

$$\begin{array}{c}\hfill {\mathit{\xi }}_j^{\prime }={\mathit{\xi }}_i+\rho ({\mathit{\xi }}_j-{\mathit{\xi }}_0),{\mathit{\xi }}_j\in \mathcal{V}\end{array}$$

(16)

where $1\le \rho <\infty$. If the vertex set of the current simplex is given by $\mathcal{V}=\mathrm{vset}({\mathit{\xi }}_0,\Delta )$, then the transformed n-simplex has the vertex set ${\mathcal{V}}^{\prime }=\mathrm{vset}({\mathit{\xi }}_j,{\Delta }^{\prime })$, where ${\Delta }^{\prime }=\rho \Delta$, and its edge length is $|{\Delta }^{\prime }|$. The new n-simplex, after the expansive translation, preserves its shape, but increases its content by a factor of ${\rho }^n$. From a practical point of view, it is convenient to choose the expansive factor ρ close to one, but slightly greater, because this avoids obtaining candidate points that were already visited in previous iterations, in the next rotation operation.

Let c be an arbitrary positive constant; if $\varphi \left({\mathit{\xi }}_i^{\prime }\right)<\varphi \left({\mathit{\xi }}_0\right)-c{\Delta }^2$ for some ${\mathit{\xi }}_i^{\prime }\in {\mathcal{V}}^{\prime }\}$, then the algorithm proceeds to a new iteration and performs an expansive translation operation. Otherwise, the algorithm executes a rotation operation.

The introduction of the positive term $c{\Delta }^2$ guarantees that the iteration is considered successful only if it provides a point with a sufficient decrease of the cost function.

3.4.4. Rotation

This operation produces a similar n-simplex, but where each edge is rotated π radians with respect to the right vertex. A graphical interpretation is depicted in Figure 2.

Let $\mathcal{V}$ denote the vertex set of an n-simplex. The vertices of the rotated n simplex are given by the following expression:

$$\begin{array}{c}\hfill {\mathit{\xi }}_j^{\prime }={\mathit{\xi }}_0-({\mathit{\xi }}_j-{\mathit{\xi }}_0),{\mathit{\xi }}_j\in \mathcal{V}\end{array}$$

(17)

If the vertex set of the current simplex is given by $\mathcal{V}=\mathrm{vset}({\mathit{\xi }}_0,\Delta )$, then the transformed n-simplex has the vertex set ${\mathcal{V}}^{\prime }=\mathrm{vset}({\mathit{\xi }}_0,{\Delta }^{\prime })$, where ${\Delta }^{\prime }=-\Delta$, and its edge length is $|{\Delta }^{\prime }|$. If $\varphi \left({\mathit{\xi }}_i^{\prime }\right)<\varphi \left({\mathit{\xi }}_0\right)-c{\Delta }^2$ for some ${\mathit{\xi }}_i^{\prime }\in {\mathcal{V}}^{\prime }$, then the rotation operation succeeds, and the algorithm proceeds to start a new iteration and performs an expansive translation operation. Otherwise, the rotation operation fails; the rotated vertex set is discarded; and the algorithm executes a shrinkage operation with the original vertex set $\mathcal{V}$.

3.4.5. Shrinkage

This operation performs a contraction of the simplex by fixing the right vertex and placing the remaining vertices along the same directions, but at a distance that is reduced by a constant factor $\sigma \in (0,1)$. A graphical interpretation of the shrinkage operation is depicted in Figure 2 for ${\mathbb{R}}^2$.

Let $\mathcal{V}$ denote the vertex set of an n-simplex. The vertices of the transformed n-simplex are given by:

$${\mathit{\xi }}_j^{\prime }={\mathit{\xi }}_0+\sigma ({\mathit{\xi }}_j-{\mathit{\xi }}_0),{\mathit{\xi }}_j\in \mathcal{V}$$

(18)

If the vertex set of the current simplex is given by $\mathcal{V}=\mathrm{vset}({\mathit{\xi }}_0,\Delta )$, then the transformed n-simplex has the vertex set ${\mathcal{V}}^{\prime }=\mathrm{vset}({\mathit{\xi }}_0,{\Delta }^{\prime })$, where ${\Delta }^{\prime }=\sigma \Delta$, and its edge length is $|{\Delta }^{\prime }|$. After the shrinkage operation, the algorithm proceeds to start a new iteration by performing an expansive translation operation.

3.4.6. Sufficient Decrease Condition

The expansive translation and rotation operations are considered successful if some vertex of the transformed n-simplex satisfies the following sufficient decrease condition:

$$\begin{array}{c}\hfill \varphi \left({\mathit{\xi }}_i^{\prime }\right)<\varphi \left({\mathit{\xi }}_0\right)-c{\Delta }^2,\mathrm{for}\mathrm{some}{\mathit{\xi }}_i^{\prime }\in {\mathcal{V}}^{\prime }\end{array}$$

where c is a positive constant. This condition is key to proving the termination of the basic algorithm and global convergence to a stationary point. The constant c is arbitrary, but usually, a small value is chosen.

3.4.7. Stopping Criterion

The sufficient decrease condition and the shrinkage operation guarantee that the size of the n-simplex converges to zero. Consequently, a stopping criterion can be defined by taking a sufficiently small number $ϵ>0$. Let $|\Delta |$ be the edge length of the n-simplex; then, the basic algorithm stops when the following condition is satisfied:

$$\begin{array}{c}\hfill P|\Delta |\le ϵ\end{array}$$

(19)

3.4.8. Space Transformation

The search space transformation is given by the map:

$$\begin{array}{c}\hfill x\left(\mathit{\xi }\right)=\mathrm{frac}\left(P\mathit{\xi }\right)\end{array}$$

(20)

Therefore, when the n-simplex with vertex set $\mathcal{V}$ reaches edge length $\Delta \le ϵ$, the basic algorithm stops, and the point obtained is $\overline{\mathbf{x}}=\mathrm{frac}\left(P\overline{\mathit{\xi }}\right)$, where $\overline{\mathit{\xi }}\in argmin\{\mathit{\xi }:\mathit{\xi }\in \mathcal{V}\}$; and it attains the function value $f\left(\overline{\mathbf{x}}\right)=\varphi \left(\overline{\mathit{\xi }}\right)$.

3.4.9. Implementation

An implementation of the basic algorithm in pseudocode is given in Algorithm 1.

Algorithm 1 Basic algorithm.
1: function BGDS($f,{\mathbf{x}}_0$)
2:     parameters: $ϵ>0$, $P\in \mathbb{N}$
3:     definitions: $s\left(\mathit{\xi }\right):=P\mathit{\xi }-\lfloor P\mathit{\xi }\rfloor$, $\varphi \left(\mathit{\xi }\right):=f\left(s\right(\mathit{\xi }\left)\right)$
4:     $\mathbf{z}\leftarrow \mathrm{rand}(dim\left(x_0\right))$
5:     ${\mathit{\xi }}_0\leftarrow (\lfloor P\mathbf{z}\rfloor +{\mathbf{x}}_0)/P$
6:     $ϵ\leftarrow ϵ/P$
7:     $\mathit{\xi }\leftarrow \mathrm{LDS}(\varphi ,{\mathit{\xi }}_0,ϵ)$	▹ Local direct search
8:     return $s\left(\mathit{\xi }\right)$
9: end function
10: function LDS($\varphi ,{\mathit{\xi }}_0,ϵ$)
11:     parameters: $\rho ,\sigma ,c\in \mathbb{R}$ : $1\le \rho ,0<\sigma <1,0<c,0<\Delta$
12:     definitions: $\mathrm{vset}(\mathit{\xi },\Delta ):=\mathit{\xi }\in {\mathbb{R}}^n,\Delta \in \mathbb{R}\}$, $\mathrm{vmin}\left(\mathcal{V}\right)\in argmin\{\varphi \left({\mathit{\xi }}_i\right):{\mathit{\xi }}_i\in \mathcal{V}\}$
13:     $\mathit{\xi }\leftarrow {\mathit{\xi }}_0$
14:     $\mathcal{V}\leftarrow \mathrm{vset}(\mathit{\xi },\Delta )$
15:     ${\mathit{\xi }}_{min}\leftarrow \mathrm{vmin}\left(\mathcal{V}\right)$
16:     while $ϵ<|\Delta |$ do
17:         if $\varphi \left({\mathit{\xi }}_{min}\right)<\varphi \left(\mathit{\xi }\right)-c{\Delta }^2$ then	▹ Expansive translation
18:            $\mathit{\xi }\leftarrow {\mathit{\xi }}_{min}$
19:            $\Delta \leftarrow \rho \Delta$
20:            $\mathcal{V}\leftarrow \mathrm{vset}(\mathit{x},\Delta )$
21:            ${\mathit{\xi }}_{min}\leftarrow \mathrm{vmin}\left(\mathcal{V}\right)$
22:         end if
23:         if $\varphi \left(\mathit{\xi }\right)-c{\Delta }^2\le \varphi \left({\mathit{\xi }}_{min}\right)$ then	▹ Rotation
24:            $\Delta \leftarrow -\Delta$
25:            $\mathcal{V}\leftarrow \mathrm{vset}(\mathit{\xi },\Delta )$
26:            ${\mathit{\xi }}_{min}\leftarrow \mathrm{vmin}\left(\mathcal{V}\right)$
27:         end if
28:         if $\varphi \left(\mathit{\xi }\right)-c{\Delta }^2\le \varphi \left({\mathit{\xi }}_{min}\right)$ then	▹ Shrinkage
29:            $\Delta \leftarrow \sigma \Delta$
30:            $\mathcal{V}\leftarrow \mathrm{vset}(\mathit{\xi },\Delta )$
31:            ${\mathit{\xi }}_{min}\leftarrow \mathrm{vmin}\left(\mathcal{V}\right)$
32:         end if
33:     end while
34:     return ${\mathit{\xi }}_{min}$
35: end function

3.5. Convergence of the Basic Algorithm

The basic algorithm has been designed to provide termination and improvement by performing a sequence of simplex transformations. If in addition, the cost function is differentiable and its gradient is Lipschitz continuous, then the basic algorithm converges to a stationary point.

Theorem 3. 

Let f be the cost function of a minimization problem on the open unit n-box; then the basic algorithm always terminates. If, in addition, f is differentiable and its gradient $\Delta f$ is Lipschitz continuous on the unit n-box, then the sequence of points generated at each iteration by the basic algorithm converges to a stationary point.

Proof. 

Let $\mathcal{S}$ and $\mathcal{U}$ denote the index sets of successful and unsuccessful iterations of the basic algorithm, respectively. Let ${\mathbf{x}}_k=\mathrm{frac}\left(P{\mathit{\xi }}_k\right)$, then $f\left({\mathbf{x}}_k\right)=\varphi \left({\mathit{\xi }}_k\right)$. If $k\in \mathcal{S}$, then $f\left({\mathbf{x}}_{k+1}\right)<f\left({\mathbf{x}}_k\right)-c{\Delta }_k^2$ and ${\Delta }_{k+1}=\rho {\Delta }_k$, while if $k\in \mathcal{U}$, then $f\left({\mathbf{x}}_{k+1}\right)\ge f\left({\mathbf{x}}_k\right)-c{\Delta }_k^2$ and ${\Delta }_{k+1}=\sigma {\Delta }_k$. Let us begin by proving that ${lim}_{k\to \infty }\left|{\Delta }_k\right|=0$. Suppose not; then, for any $k\in {\mathbb{Z}}_{[0,\infty ]}$, there exists $\ell \ge k$, such that $\ell \in \mathcal{S}$ because, if such an ℓ does not exist, then the set of successful iterations would be finite, and when k approaches infinity, the update rule ${\Delta }_{k+1}=\sigma {\Delta }_k$ yields ${lim}_{k\to \infty }\left|{\Delta }_k\right|=0$. In addition, there exists $\underset{¯}{\Delta }>0$, such that for each successful iteration $\ell \in \mathcal{S}$, $|{\Delta }_{\ell }|\ge \underset{¯}{\Delta }$ and:

$$f\left({\mathbf{x}}_{\ell +1}\right)<f\left({\mathbf{x}}_{\ell }\right)-c{\Delta }_{\ell }^2\le f\left({\mathbf{x}}_{\ell }\right)-c{\underset{¯}{\Delta }}^2$$

However, since ℓ approaches infinity, this implies that $f\left({\mathbf{x}}_{\ell +1}\right)\to -\infty$, which contradicts the fact that the function f is bounded below on ${(0,1)}^n$. Consequently, the boundedness of f implies that $\underset{¯}{\Delta }=0$ and ${lim}_{k\to \infty }\left|{\Delta }_k\right|=0$. This proves the termination of the basic algorithm.

For the second part of the theorem, if f is continuous and differentiable on ${(0,1)}^n$, then so is $\varphi \left(\mathit{\xi }\right)=f\left(\mathrm{frac}\right(P\mathit{\xi }\left)\right)$ on the n-box ${\mathcal{B}}_P\left(\mathbf{x}\right)=\{\mathit{\xi }\in {\mathbb{R}}^n:\lfloor P\mathit{\xi }\rfloor =\lfloor P\mathbf{x}\rfloor ,P\mathit{\xi }\notin {\mathbb{Z}}^n\}$ for any $\mathbf{x}\in {\mathbb{R}}^n$. Furthermore, if $\nabla f$ is Lipschitz continuous with constant M on ${(0,1)}^n$, then so is $\nabla \varphi$ in ${\mathcal{B}}_P\left(\mathbf{x}\right)$ with constant $PM$. Since we proved that ${lim}_{k\to \infty }\left|{\Delta }_k\right|=0$, then there always exists a positive integer L, such that $\lfloor P{\mathit{\xi }}_k\rfloor =\lfloor P{\mathit{\xi }}_L\rfloor$ for any positive integer $k>L$. The basic algorithm searches points along the coordinate directions; then, from Lemma 2 and for any positive integer $k>L$, no matter the value of $\nabla \varphi \left({\mathit{\xi }}_k\right)$, there is always at least one (positive or negative) coordinate direction ${\mathbf{d}}_i\in \{{\mathbf{b}}_i,-{\mathbf{b}}_i\}$, such that:

$$\frac{1}{\sqrt{n}}\le cos\theta =\frac{\langle -\nabla \varphi \left({\mathit{\xi }}_k\right),{\mathbf{b}}_i\rangle }{\parallel \nabla \varphi \left({\mathit{\xi }}_k\right)\parallel \parallel {\mathbf{d}}_i\parallel }$$

or equivalently:

$$\frac{1}{\sqrt{n}}\parallel \nabla \varphi \left({\mathit{\xi }}_k\right)\le -\langle \nabla \varphi \left({\mathit{\xi }}_k\right),{\mathbf{d}}_i\rangle$$

(21)

because $\parallel {\mathbf{d}}_i\parallel =1$. For any unsuccessful iteration $k\in \mathcal{U}$, $\varphi ({\mathit{\xi }}_k+|{\Delta }_k\left|{\mathbf{d}}_k\right)\ge \varphi \left({\mathit{\xi }}_k\right)-c{\Delta }_k^2$, the mean value theorem establishes that:

$$\begin{array}{cc}\hfill -c{\Delta }_k^2& \le \varphi ({\mathit{\xi }}_k+|{\Delta }_k\left|{\mathbf{d}}_i\right)-\varphi \left({\mathit{\xi }}_k\right)\hfill \\ & =\langle \nabla \varphi ({\mathit{\xi }}_k+{\alpha }_k|{\Delta }_k|{\mathbf{d}}_i),{\mathbf{d}}_i\rangle |{\Delta }_k|\hfill \end{array}$$

for some ${\alpha }_k\in [0,1]$. Subtracting $\langle \nabla \varphi \left({\mathit{\xi }}_k\right),{\mathbf{d}}_i\rangle \left|{\Delta }_k\right|$ from both sides leads to:

$$\begin{array}{cc}\hfill -c{\Delta }_k^2-\langle \nabla \varphi \left({\mathit{\xi }}_k\right),{\mathbf{d}}_i\rangle \left|{\Delta }_k\right|& \le \langle \nabla \varphi ({\mathit{\xi }}_k+{\alpha }_k|{\Delta }_k|{\mathbf{d}}_i)-\nabla \varphi \left({\mathit{\xi }}_k\right),{\mathbf{d}}_i\rangle |{\Delta }_k|\hfill \end{array}$$

Applying Equation (21) yields:

$$\begin{array}{cc}\hfill \frac{1}{\sqrt{n}}\parallel \nabla \varphi \left({\mathit{\xi }}_k\right)\parallel & \le \langle \nabla \varphi ({\mathit{\xi }}_k+{\alpha }_k|{\Delta }_k|{\mathbf{d}}_i)-\nabla \varphi \left({\mathit{\xi }}_k\right),{\mathbf{d}}_i\rangle +c|{\Delta }_k|\hfill \end{array}$$

Taking into account that φ is continuously differentiable in ${\mathcal{B}}_P\left({\mathit{\xi }}_k\right)$ and its gradient $\nabla \varphi$ is Lipschitz with constant $PM$ in ${\mathcal{B}}_P\left({\mathit{\xi }}_k\right)$,

$$\frac{1}{\sqrt{n}}\parallel \nabla \varphi \left({\mathit{\xi }}_k\right)\parallel \le PM\parallel {\alpha }_k{\Delta }_k{\mathbf{d}}_i\parallel +c|{\Delta }_k|\le (PM+c)\left|{\Delta }_k\right|$$

because ${\alpha }_k\in (0,1)$ and $\parallel {\mathbf{b}}_i\parallel =1$. Consequently,

$$\parallel \nabla \varphi \left({\mathit{\xi }}_k\right)\parallel \le \sqrt{n}(PM+c)\left|{\Delta }_k\right|$$

and taking the limit when k approaches infinity, it is possible to conclude that:

$$\underset{k\to \infty }{lim}\parallel \nabla \varphi \left({\mathit{\xi }}_k\right)\parallel \le \sqrt{n}(PM+c)\underset{k\to \infty }{lim}\left|{\Delta }_k\right|=0,\mathrm{for}k\in \mathcal{U}$$

and finally, by making ${\mathbf{x}}_k=\mathrm{frac}\left({\mathit{\xi }}_k\right)$:

$$\begin{array}{cc}\hfill \underset{k\to \infty }{lim}\parallel \nabla f\left({\mathbf{x}}_k\right)\parallel & =P\underset{k\to \infty }{lim}\parallel \nabla \varphi \left({\mathit{\xi }}_k\right)\parallel =0,\mathrm{for}k\in \mathcal{U}.\hfill \end{array}$$

Consequently, the sequence of points generated by the basic algorithm converges to a stationary point. ☐

Remark 2. 

The basic algorithm can be applied to any cost function f, not necessarily continuous or differentiable in the search space ${(0,1)}^n$. For a continuous and differentiable function with a Lipschitz continuous gradient, the basic algorithm converges to a stationary point. If in addition the cost function is strictly convex, then the unique optimizer is a stationary point, and Theorem 3 guarantees that the basic algorithm converges to the unique optimizer. Note that we do not need to know the value of the Lipschitz constant M.

Example 2. 

The evolution of the basic algorithm for the function $f\left(x\right)=\left|x\right|-|\sqrt{x}sin\left(3\pi x\right)|$ is depicted in Figure 3. The parameters of the algorithm are $P={10}^4$, $\sigma =0.5$, $\rho =1.05$, $c=0.01$, $ϵ={10}^{-4}$. Since this problem is unidimensional, the simplex is an interval. The left plots represent the vertex points of the simplex in the original space (above) and their corresponding function values (below) after each iteration. The right plots represent the values of the worst (above) and best (below) points of the simplex in the original space that were obtained during the execution of the algorithm. Note that the algorithm explores the whole range of the unit interval. This exploration is performed whenever the length of the simplex edge ${\Delta }_k$ satisfies $P{\Delta }_k\ge 1$, which corresponds to the exploratory phase of the algorithm. Once ${\Delta }_k$ is $P{\Delta }_k<1$, then the algorithm enters the phase of convergence to a stationary point. In this example, the algorithm stops after 24 iterations. From Iterations 1 to 12, the algorithm is in the exploratory phase, and from Iterations 13 to 24, the algorithm is in the convergence phase to a stationary point. The point obtained corresponds to $f(0.1721)=-0.2422$, which is the global minimal value within the space tolerance ϵ.

3.6. The Complete Algorithm

The complete algorithm, that we call GDS (an acronym for Global Direct Search), is obtained by repeating the basic algorithm a number of times. A single application of the basic algorithm may obtain the global optimum within a space tolerance ϵ only in easy problems. For more difficult optimization problems, a new execution of the basic algorithm, starting with the best point obtained in the previous execution, may improve the result. Each execution produces a local minimizer; however, if there are better points in some coordinate direction, the algorithm may jump to one of these points during the exploratory phase of a new execution. Unfortunately, not every optimization problem satisfies the property that, for any local minimizer, there are points in some coordinate direction where the cost function is improved. In such a case, a strategy that turns out to be more effective is to start from a new random initial point. The class of cost functions that we are facing is not usually known in advance, and we do not know which strategy is more appropriate for a new execution of the basic algorithm. A trade-off solution is to repeat the basic algorithm N times, but starting from a new random initial point after a number $R\le N$ of repetitions.

3.6.1. Initial Point

Let N denote the total number of executions of the basic algorithm and R the number of repetitions without choosing a new initial point. Let $\overline{\mathbf{x}}$ denote the point obtained in the previous execution of the basic algorithm and ${\mathit{\xi }}_r$ a random vector in ${(0,1)}^n$ with a uniform probability distribution. The initial point for the n-th execution of the basic algorithm is obtained as follows:

$$\begin{array}{c}\hfill {\mathit{\xi }}_0=\left\{\begin{array}{cc}\mathrm{frac}(P^{-1}\mathrm{int}\left(P{\mathit{\xi }}_r\right)+\overline{\mathbf{x}})\hfill & \mathrm{if}n\mathrm{mod}(R+1)\ne 0\hfill \\ {\mathit{\xi }}_r\hfill & \mathrm{if}n\mathrm{mod}(R+1)=0\hfill \end{array}\right.\end{array}$$

(22)

where $\mathrm{mod}$ is the modulo operator that represents the remainder of the integer division of their arguments. Note that if $R=0$, the initial point is always a random point; however, if $R=N$, the initial point is always the point obtained in the previous execution of the basic algorithm.

3.6.2. Best Point Storage

The minimum point found during the execution of the complete algorithm is preserved. The point obtained after each basic algorithm execution is compared to the best solution stored and, in the case of improvement, the new point replaces it. Let $\mathbf{x}$ denote the point obtained after the last execution of the basic algorithm and ${\mathbf{x}}_{min}$ the best point stored so far; the storage rule is:

$$\begin{array}{c}\hfill {\mathbf{x}}_{min}^{\prime }=\left\{\begin{array}{cc}\overline{\mathbf{x}}\hfill & \text{if}f\left(\overline{\mathbf{x}}\right)<f\left({\mathbf{x}}_{min}\right)\hfill \\ {\mathbf{x}}_{min}\hfill & \text{otherwise}\hfill \end{array}\right.\end{array}$$

(23)

3.6.3. Implementation of the Complete Algorithm

An implementation of the complete algorithm GDS in pseudocode is given in Algorithm 2.

Algorithm 2 Complete algorithm GDS.

1: function GDS($f,{\mathbf{x}}_0$) 2:     parameters: $N,R\in \mathbb{N}$, such that $R\le N$ 3:     ${\mathbf{x}}_{min}\leftarrow {\mathbf{x}}_0$ 4:     for $k=1$ to N do 5:         $\mathbf{x}\leftarrow {\mathbf{x}}_{min}$ 6:         if $kmod(R+1)=0$ then 7:            $\mathbf{x}\leftarrow \mathrm{rand}(dim\left({\mathbf{x}}_0\right))$ 8:         end if 9:         $\mathbf{x}\leftarrow \mathrm{BGDS}(f,\mathbf{x})$        ▹ Basic global direct search 10:         if $f\left(\mathbf{x}\right)<f\left({\mathbf{x}}_{min}\right)$ then 11:            ${\mathbf{x}}_{min}\leftarrow \mathbf{x}$ 12:         end if 13:     end for 14:     return ${\mathbf{x}}_{min}$ 15: end function

4. Experimental Study

4.1. Test Functions

Global optimization is not a trivial problem. Theoretical results, such as the No Free Lunch (NFL) theorem [61], state that an efficient general-purpose universal optimization algorithm is not possible. In other words, this result says that one algorithm can outperform another only for a certain class of problems. It is a usual practice to analyze the performance of an optimization algorithm by using test functions. A large number of standard test functions have been used to study the performance of our direct search global optimization algorithm. In this section, we present the results for the 24 functions given in

Table 1. Description of the functions used in the experimental study.

Func #	Description
1.01	Sphere
1.02	Ellipsoid separable with monotone x-transformation, condition 1e + 06
1.03	Rastrigin separable with both x-transformations condition 10
1.04	Skew Rastrigin–Bueche separable, condition 10, skew-condition 100
1.05	Linear slope, neutral extension outside the domain (not flat)
2.06	Attractive sector function
2.07	Step-ellipsoid, condition 100
2.08	Rosenbrock, original
2.09	Rosenbrock, rotated
3.10	Ellipsoid with monotone x-transformation, condition 1e6
3.11	Discus with monotone x-transformation, condition 1e6
3.12	Bent cigar with asymmetric x-transformation, condition 1e6
3.13	Sharp ridge, slope 1:100, condition 10
3.14	Sum of different powers
4.15	Rastrigin with both x-transformations, condition 10
4.16	Weierstrass with monotone x-transformation, condition 100
4.17	Schaffer F7 with asymmetric x-transformation, condition 10
4.18	Schaffer F7 with asymmetric x-transformation, condition 1000
4.19	F8F2 composition of 2D Griewank–Rosenbrock
5.20	Schwefel x sin(x) with tridiagonal transformation, condition 10
5.21	Gallagher 101 Gaussian peaks, condition up to 1000
5.22	Gallagher 21 Gaussian peaks, condition up to 1000, 1000 for global opt
5.23	Katsuuras repetitive rugged function
5.24	Lunacek bi-Rastrigin, condition 100

. These function are described in detail in [62]. The functions are classified into five groups: separable functions (func # 1.xx), functions with low or moderate conditioning (func # 2.xx), unimodal functions with high conditioning (func # 3.xx), multimodal functions with an adequate global structure (func # 4.xx) and multimodal functions with a weak global structure (func # 5.xx). All of the functions are scalable with the dimension. All of the functions are defined and can be evaluated over ${\mathbb{R}}^n$, and the search space is ${[-5,5]}^n$, where n is the dimension of the function domain. The global optimal value is attained in this search space.

4.2. Selection of the GDS Parameters

The performance of the GDS algorithm for different parameter values has been empirically analyzed by means of extensive computer studies. Our algorithm has been implemented in the scientific software MATLAB [63], and its performance has been successfully compared to the global optimization strategies that are implemented in the Global Optimization Toolbox of MATLAB. The GDS algorithm has a number of configurable parameters, and this comparative study allowed us to select a set of parameters that are appropriate for many problems. The expansive and contractive factors are set to $\rho =1.05$ and $\sigma =0.5$. It is convenient to take the expansive value slightly greater than one to avoid evaluating points already checked in previous iterations. However, large values of ρ usually produce slow convergence. Consequently, values in the interval $[1.01,1.10]$ are usually a good option for most problems. Regarding the contractive factor σ, the closer to one, the larger the number of iterations of the basic GDS algorithm will be. From our experiments, $\sigma =0.5$ is a convenient value that produces a satisfactory trade-off between the efficiency of the algorithm and the convergence time. The constant c of the sufficient decrease condition is chosen as $c=0.01$. Any positive value of c provides the termination of the algorithm, but small values are recommended to improve the algorithm’s performance during the initial iterations when Δ is large. The parameters P and R were fixed to $P=1000$ and $R=5$. These values were selected after a massive experimentation. In our experiments, we analyzed the mean value of the CPU time for one execution of the basic GDS algorithm for different values of the parameter R. The empirical results showed that the larger the value of R, the lower the CPU time for execution. This means that when R increases, the number of executions N can also be increased to a certain extent without compromising the total CPU time. The reduction is larger for smaller values of R, e.g., the reduction is about 40% from $R=0$ to $R=1$ and about 30% from $R=1$ to $R=2$. Since increasing R is also beneficial for functions that can be globally optimized along the coordinate directions, a good option that trades off between moderate CPU time and efficiency in computing the global optimum is to choose $R\in [2,10]$ and increase the number of repetitions of the basic algorithm N as much as possible, to maintain a moderate total CPU time. In fact, we have checked that a good selection of P and R could improve the results depending on the problem. However, since a practitioner usually does not know in advance what class its problem belongs to, it is important to fix a set of parameters that trade off for a large class of problems. In conclusion, the parameter values of the GDS algorithm for the experimental study were selected as $P=1000$, $R=5$, $\rho =1.05$, $\sigma =0.5$, $c=0.01$ and $N=1e9$, and the stopping criterion is the number of function evaluations.

4.3. Performance Evaluation

The evaluation of the GDS algorithm has been accomplished by solving the 24 functions in Table 1 using the fixed set of parameters values explained in the previous subsection. The performance of GDS is compared to DIRECT [51]. DIRECT is a deterministic global optimization algorithm based on the domain partition strategy. Each iteration includes two phases. The first phase identifies hyper-rectangles that potentially contain a global optima. In the second phase, the hyper-rectangles identified in the first phase are partitioned into smaller hyper-rectangles, and the cost function is evaluated at the centers of the hyper-rectangles. The convergence properties guarantee that if no termination criteria are included, DIRECT will exhaustively sample the domain. The implementation of DIRECT in MATLAB by Finkel [64] has been used in this study. The experiments have been performed in a laptop PC equipped with a dual core processor Intel (R) Core (TM) i5-3317U CPU @1.70 GHz running MATLAB R2014a under the Windows operating system.

The comparative study has been performed as follows: For both algorithms, GDS and DIRECT, a fixed number of function evaluations is used as a stopping condition, and the error in computing the global minimum value is obtained. Each problem has been solved 15 times, and the median of the error is tabulated for each function. The study has been accomplished for the 24 functions of Table 1 and for different dimensions of the function domain and different numbers of function evaluations. The default parameters of DIRECT [64] have been used, but using the number of function evaluations as stopping conditions.

In

Table 2. Comparative study of the median of the error for problems of dimensions 5 and 10 and the number of evaluations 1e + 03 and 1e + 04.

Func	GDS	DIRECT	GDS	DIRECT	GDS	DIRECT	GDS	DIRECT
#	dim = 5	nev = 1e+03	dim = 5	nev = 1e+04	dim = 10	nev = 1e+03	dim = 10	nev = 1e+04
1.01	3.647e+01	5.094e−04	9.920e−09	1.347e−04	1.422e+02	1.862e−01	9.826e−09	1.076e−03
1.02	2.400e+06	2.044e+00	8.064e−06	3.053e−01	1.072e+07	1.475e+05	2.020e−05	2.129e+00
1.03	3.649e+02	7.015e+00	2.002e-02	5.970e+00	6.338e+02	4.234e+01	7.859e−02	1.692e+01
1.04	2.507e+02	1.249e+01	1.696e−02	6.967e+00	9.518e+02	5.252e+01	1.058e+00	2.274e+01
1.05	6.767e+01	1.643e−02	7.809e−05	1.643e-02	1.788e+02	2.271e+01	1.986e−04	1.909e−02
2.06	6.348e+04	2.306e+00	3.497e−07	7.272e−01	5.778e+05	2.520e+01	1.435e−01	1.185e+01
2.07	4.044e+00	6.738e−01	5.543e−01	5.216e−03	2.796e+01	6.227e+00	5.254e+00	1.815e+00
2.08	3.644e+04	2.107e+00	1.647e+00	1.794e−02	2.066e+05	9.353e+01	3.141e+00	8.302e+00
2.09	7.835e+03	5.858e-01	2.433e−01	2.747e−02	8.654e+04	1.551e+01	5.383e+00	6.010e+00
3.10	1.426e+05	8.102e+02	9.892e+02	3.345e+01	7.788e+06	1.312e+04	3.999e+03	1.640e+03
3.11	1.003e+02	1.997e+01	5.722e+01	4.615e+00	1.647e+06	5.343e+01	1.023e+02	3.198e+01
3.12	4.213e+07	6.904e+00	4.372e+00	2.012e−02	2.597e+08	1.284e+05	5.751e+00	7.904e−01
3.13	8.841e+02	2.545e+00	1.067e+00	5.285e−02	2.388e+03	9.999e+01	2.036e+00	6.097e+00
3.14	2.356e+01	8.957e−03	8.020e−04	7.238e−04	5.779e+01	2.685e−01	2.232e−03	1.452e−02
4.15	1.920e+02	7.971e+00	5.014e+00	2.988e+00	7.494e+02	5.364e+01	6.365e+01	1.991e+01
4.16	3.869e+00	6.053e−01	6.144e−01	4.247e−03	3.089e+01	5.140e+00	2.428e+00	5.762e−01
4.17	1.069e+01	1.104e−01	9.849e−01	6.272e−03	2.350e+01	1.457e+00	1.265e+01	9.732e−02
4.18	4.694e+01	2.134e−01	2.395e+00	2.908e−02	5.608e+01	6.410e+00	5.172e+01	4.363e−01
4.19	2.775e+00	8.478e-02	5.901e−01	4.858e-02	2.127e+01	1.990e-01	2.630e+00	1.620e−01
5.20	1.983e+04	1.265e+00	2.392e−01	8.295e−01	7.714e+04	2.707e+00	9.283e-01	1.829e+00
5.21	1.751e+01	2.052e−04	9.300e−01	6.972e−06	7.886e+01	2.338e+00	6.676e+00	2.239e−03
5.22	4.032e+01	6.928e−01	6.024e−02	2.746e−04	8.485e+01	1.651e+00	7.272e+00	6.928e−01
5.23	1.395e+00	9.682e−01	6.205e−01	8.782e−01	2.247e+00	1.438e+00	7.203e−01	9.066e−01
5.24	9.572e+01	2.122e+01	1.108e+01	1.071e+01	2.673e+02	7.556e+01	9.847e+01	6.139e+01

, the results for functions of dimensions five and 10 and a number of function evaluations equal to ${10}^3$ and ${10}^4$ are shown, while in

Table 3. Comparative study of the median of the error for problems of dimensions 20 and 40 and the number of evaluations 1e + 04 and 1e + 05.

Func	GDS	DIRECT	GDS	DIRECT	GDS	DIRECT	GDS	DIRECT
#	dim = 20	nev = 1e+04	dim = 20	nev = 1e+05	dim = 40	nev = 1e+04	dim = 40	nev = 1e+05
1.01	6.690e−2	1.456e−01	9.434e-09	2.626e−03	3.187e+02	2.533e+01	8.897e−09	5.976e−02
1.02	3.171e+03	1.570e+06	3.806e−05	3.471e+01	1.195e+07	2.904e+06	4.865e−02	2.904e+06
1.03	9.586e+00	5.187e+01	3.545e−02	3.620e+01	1.719e+03	5.029e+02	2.008e+00	9.027e+01
1.04	9.342e+00	6.536e+01	7.299e−02	6.174e+01	4.448e+03	5.542e+02	1.487e+00	1.165e+02
1.05	5.537e−01	3.190e+01	4.178e−04	2.201e−02	5.900e+02	3.495e+02	8.092e−04	4.421e+01
2.06	3.202e+01	6.464e+01	1.873e+00	3.152e+01	9.308e+05	1.415e+03	2.211e+00	1.177e+02
2.07	3.453e+01	1.654e+01	1.371e+01	6.260e+00	1.376e+02	1.234e+02	6.765e+01	4.261e+01
2.08	7.405e+01	6.866e+01	4.320e+00	3.332e+01	6.045e+05	7.049e+03	8.010e+01	1.602e+02
2.09	2.030e+01	5.129e+01	8.313e+00	3.241e+01	3.287e+05	2.260e+02	3.715e+01	1.424e+02
3.10	4.927e+04	1.210e+04	2.472e+03	4.680e+03	1.048e+07	2.325e+05	2.234e+04	6.582e+04
3.11	2.337e+02	8.049e+01	2.610e+02	6.898e+01	5.563e+02	1.885e+02	4.841e+02	1.792e+02
3.12	1.716e+04	1.010e+05	6.880e+00	1.499e+03	6.739e+08	2.964e+07	7.067e+00	1.240e+05
3.13	3.187e+01	1.404e+02	1.805e+00	1.050e+02	3.588e+03	9.766e+02	2.019e+00	1.607e+02
3.14	2.114e−02	2.780e−01	6.335e−04	8.335e−02	1.016e+02	6.893e+00	2.241e−03	3.609e−01
4.15	1.349e+03	1.125e+02	9.561e+01	6.304e+01	3.134e+03	4.190e+02	7.482e+02	2.579e+02
4.16	6.812e+00	5.866e+00	3.164e+00	3.113e+00	2.283e+01	1.475e+01	9.467e+00	8.788e+00
4.17	1.885e+01	1.854e+00	2.314e+01	4.799e−1	2.714e+01	5.368e+00	2.041e+01	1.716e+00
4.18	7.823e+01	5.327e+00	8.362e+01	1.387e+00	1.084e+02	2.034e+01	1.062e+02	8.653e+00
4.19	6.546e+00	2.504e−01	4.953e+00	2.307e−01	2.069e+01	2.504e−01	8.861e+00	2.504e−01
5.20	1.108e+00	1.973e+00	6.419e−01	1.837e+00	1.273e+05	2.104e+02	9.083e−01	2.075e+00
5.21	7.598e+00	5.096e−01	1.803e+00	1.227e−01	8.327e+01	7.596e+00	4.714e+00	1.086e+00
5.22	1.114e+01	2.024e+00	6.919e−01	7.305e−1	8.487e+01	1.260e+01	8.905e+00	6.475e+00
5.23	1.038e+00	1.229e+00	6.147e−01	1.175e+00	2.385e+00	2.263e+00	1.078e+00	1.798e+00
5.24	5.915e+02	1.532e+02	9.995e+01	1.532e+02	1.333e+03	3.299e+02	8.494e+02	3.055e+02

, the same results are shown for dimensions 10 and 20 and ${10}^4$ and ${10}^5$ function evaluations. GDS and DIRECT obtained different results for different functions. In general, for a small number of function evaluations, DIRECT gets a smaller median of the error. However, as the number of function evaluations increases, the improvement in approaching the minimum value is higher in GDS. One of the outstanding features of GDS is its simplicity. This can be checked by comparing the CPU time with DIRECT. The CPU time for all of the problems, dimensions and function evaluations is much smaller for GDS; see Figure 4 and Figure 5. Note that GDS is more than 100-times faster than DIRECT for several functions of dimension 40 and ${10}^5$ function evaluations.

The results of a study of the GDS algorithm for functions of high dimension are presented in Figure 6. The GDS algorithm is executed on the functions 1.02, 1.02, 1.04 and 1.05 of dimensions 100 and 200 to search the global minimum. The error to the global minimum in the logarithmic scale and the CPU time are shown for ${10}^4$, ${10}^5$, ${10}^6$ and ${10}^7$. It is interesting to remark that the CPU time grows linearly with the dimension of the problem.

Since GDS is simple and computationally efficient, it can be successfully applied to problems of high dimensions where the number of function evaluations to obtain a good result is large for any direct search algorithm.

5. Conclusions

A direct search algorithm for the global optimization of multivariate functions has been reported. The algorithm performs a sequence of operations for an initial isosceles right n-simplex. The result of these operations is always a similar n-simplex of a different size. The operations depend on the values of the vertices of the n-simplex, where these values are computed using a transformed cost function that improves the exploratory features of the algorithm. The convergence properties of the algorithm have been proven. The results of an extensive experimental study demonstrate that the algorithm is capable of approaching the global optimum for black box problems with moderate computation time. It is competitive with most of the heuristic-based global optimization strategies presently used. Furthermore, it has been compared to DIRECT, a well-known Lipschitzian-based direct search algorithm. The new algorithm is very simple and computationally cost-effective and can be applied to problems of high dimension, because the empirical results demonstrate that its computation time increases linearly with the dimension of the problem. In future work, we plan to extend the algorithm to constrained optimization.