A Stochastic Convergence Result for the Nelder–Mead Simplex Method

Abstract

We prove that the Nelder–Mead simplex method converges in the sense that the simplex vertices converge to a common limit point with a probability of one. The result may explain the practical usefulness of the Nelder–Mead method.

1. Introduction

The Nelder–Mead (NM) simplex method [1] is a direct method for the solution of the minimization problem

$$f\left(x\right)\to min\left(f:{\mathbb{R}}^n\to \mathbb{R}\right),$$

where f is continuous. It is an “incredibly popular method” (see [2]) in derivative-free optimization [3,4,5,6,7] and in various application areas (see, e.g., [8]). The Nelder–Mead simplex method became popular especially in the computational chemistry area as shown by the book [3] and the references therein. The original Nelder–Mead paper [1] has 36,809 references in Google Scholar as of 26 February 2023, showing a great variety of applications and a great number of mainly heuristic variants occasionally combined with other techniques. The Nelder–Mead algorithm can be found in many software libraries or systems as well, such as IMSL, NAG, Matlab, Scilab, Python SciPy and R [9]. The popularity of the method is due to its observed good performance in practice. In spite of this, only a few theoretical results are known on its convergence (see, e.g., [2,10,11,12]).

The counterexample of McKinnon [13] is a strictly convex function $f:{\mathbb{R}}^2\to \mathbb{R}$ with continuous derivatives on which the Nelder–Mead algorithm converges to a nonstationary point of f. For strictly convex functions $f:{\mathbb{R}}^2\to \mathbb{R}$ with bounded level sets, Lagarias, Reeds, Wright and Wright [10] proved that the function values at all simplex vertices converged to the same value and the diameters of the simplices are converging to zero. Kelley [4,14] gave a sufficient-decrease condition for the average of the objective function values (evaluated at the simplex vertices) and proved that if this condition is satisfied during the process, then any accumulation point of the simplices is a critical point of f. Han and Neumann [15] investigated the convergence to 0 and the effect of dimensionality on the function $f\left(x\right)=x^{T}x$ ($x\in {\mathbb{R}}^n$). For the restricted Nelder–Mead algorithm, Lagarias, Poonen and Wright [11] later proved that if $f:{\mathbb{R}}^2\to \mathbb{R}$ was a twice-continuously differentiable function with bounded level sets and everywhere positive definite Hessian, then it converged to the unique minimizer of f.

If the objective function f does not satisfy the conditions of [10] or [11], then a number of counterexamples show that the Nelder–Mead method may have different types of convergence behavior. It is possible that the function values at the simplex vertices converge to a common value, while the function f has no finite minimum and the simplex sequence is unbounded (Examples 1 and 2 of [16]). It is also possible that the simplex vertices converge to the same point, but the limit point is not a stationary point of f ([13], Examples 3 and 4 of [16]). Other examples indicate that the simplex sequence may converge to a limit simplex of positive diameter resulting in different limit values of f at the vertices at the limit simplex (Examples 4 and 5 of [16]).

Here, we study the convergence of the simplex vertices to a common limit point. In papers [16,17], we proved this type of convergence under sufficient conditions for $1\le n\le 3$ and $1\le n\le 8$, respectively. However, the key assumption of papers [16,17] was related to an algorithmically undecidable problem and required ways to circumvent it.

In this paper, we prove two new theorems for the convergence of the Nelder–Mead method in low dimensional spaces ($n=2,3,4,5,6$). Theorem 1 replaces the key assumption of [16,17] with an algorithmically computable one. It is the basis of Theorem 2 of Section 6, which proves that the Nelder–Mead method converges with a probability of one. This result may explain the good behavior of the Nelder–Mead method experienced in practice. The case of two-dimensional strictly convex functions is observed in Remarks 2 and 4, respectively.

2. The Nelder–Mead Simplex Method

There are several forms and variants of the Nelder–Mead method. We use the version of Lagarias, Reeds, Wright and Wright [10]. The vertices of the initial simplex $S^{\left(0\right)}$ are denoted by $x_1^{\left(0\right)},x_2^{\left(0\right)},\dots ,x_{n+1}^{\left(0\right)}\in {\mathbb{R}}^n$. It is assumed that vertices $x_1^{\left(0\right)},x_2^{\left(0\right)},\dots ,x_{n+1}^{\left(0\right)}$ are ordered such that

$$f\left(x_1^{\left(0\right)}\right)\le f\left(x_2^{\left(0\right)}\right)\le \cdots \le f\left(x_{n+1}^{\left(0\right)}\right)$$

(1)

and this condition is maintained during the iterations of the Nelder–Mead algorithm. The simplex of iteration k is denoted by $S^{\left(k\right)}=\left[x_1^{\left(k\right)},x_2^{\left(k\right)},\dots ,x_{n+1}^{\left(k\right)}\right]\in {\mathbb{R}}^{n\times \left(n+1\right)}$. Define $x_c^{\left(k\right)}=\frac{1}{n}{\sum }_{i=1}^n{x}_i^{\left(k\right)}$ and $x^{\left(k\right)}\left(\lambda \right)=\left(1+\lambda \right)x_c^{\left(k\right)}-\lambda x_{n+1}^{\left(k\right)}$. The reflection, expansion and contraction points of simplex $S^{\left(k\right)}$ are defined by

$$x_r^{\left(k\right)}=x^{\left(k\right)}\left(1\right),x_e^{\left(k\right)}=x^{\left(k\right)}\left(2\right),x_{oc}^{\left(k\right)}=x^{\left(k\right)}\left(\frac{1}{2}\right),x_{ic}^{\left(k\right)}=x^{\left(k\right)}\left(-\frac{1}{2}\right),$$

respectively. The function values at the vertices $x_j^{\left(k\right)}$ and the points $x_r^{\left(k\right)}$, $x_e^{\left(k\right)}$, $x_{oc}^{\left(k\right)}$ and $x_{ic}^{\left(k\right)}$ are denoted by $f\left(x_j^{\left(k\right)}\right)=f_j^{\left(k\right)}$ ($j=1,\dots ,n+1$), $f_r^{\left(k\right)}=f\left(x_r^{\left(k\right)}\right)$, $f_e^{\left(k\right)}=f\left(x_e^{\left(k\right)}\right)$, $f_{oc}^{\left(k\right)}=f\left(x_{oc}^{\left(k\right)}\right)$ and $f_{ic}^{\left(k\right)}=f\left(x_{ic}^{\left(k\right)}\right)$, respectively.

The Nelder–Mead simplex method (Algorithm 1) is a nonstationary iterative method where the kernel of the iteration loop consists of ordering the simplex vertices and exactly one of four possible operations. The logical conditions for these operations (reflection, expansion, contraction and shrinking) are mutually exclusive.

For the order operation, there are two rules that apply to reindexing after each iteration. If a nonshrink step occurs, then $x_{n+1}^{\left(k\right)}$ is replaced by a new point $v\in \left\{x_r^{\left(k\right)},x_e^{\left(k\right)},x_{oc}^{\left(k\right)},x_{ic}^{\left(k\right)}\right\}$. The following cases are possible:

$$f\left(v\right)<f\left(x_1^{\left(k\right)}\right),f\left(x_1^{\left(k\right)}\right)\le f\left(v\right)\le f\left(x_n^{\left(k\right)}\right),f\left(v\right)<f\left(x_{n+1}^{\left(k\right)}\right).$$

If

$$j=\left\{\begin{array}{cc}1,\hfill & \mathrm{if}f\left(v\right)<f\left(x_1^{\left(k\right)}\right)\hfill \\ \underset{2\le \ell \le n+1}{max}\left\{f\left(x_{\ell -1}^{\left(k\right)}\right)\le f\left(v\right)\le f\left(x_{\ell }^{\left(k\right)}\right)\right\},\hfill & \mathrm{otherwise}\hfill \end{array}\right.,$$

then the new simplex vertices are

$$x_i^{\left(k+1\right)}=x_i^{\left(k\right)}\left(1\le i\le j-1\right),x_j^{\left(k+1\right)}=v,x_i^{\left(k+1\right)}=x_{i-1}^{\left(k\right)}\left(i=j+1,\dots ,n+1\right).$$

(2)

This rule inserts v into the ordering with the highest possible index. If shrinking occurs, then

$$z_1=x_1^{\left(k\right)},z_i=\left(x_i^{\left(k\right)}+x_1^{\left(k\right)}\right)/2(i=2,\dots ,n+1)$$

plus a reordering takes place. If $f\left(x_1^{\left(k\right)}\right)\le f\left(z_i\right)$ ($i=2,\dots ,n+1$), then by convention $x_1^{\left(k+1\right)}=x_1^{\left(k\right)}$. Hence, it is guaranteed that

$$f\left(x_1^{\left(k\right)}\right)\le f\left(x_2^{\left(k\right)}\right)\le \cdots \le f\left(x_{n+1}^{\left(k\right)}\right)\left(k\ge 0\right).$$

(3)

The insertion rule (2) implies that if function f is bounded below on ${\mathbb{R}}^n$ and only a finite number of shrink iterations occur, then each sequence $\left\{f_i^{\left(k\right)}\right\}$ converges to some $f_i^{\infty }$ for $i=1,\dots ,n+1$ (see Lemma 3.3 of [10]).

Algorithm 1: Nelder–Mead algorithm

3. A Matrix Form of the Nelder–Mead Method

Assume that simplex $S^{\left(k\right)}=\left[x_1^{\left(k\right)},x_2^{\left(k\right)},\dots ,x_{n+1}^{\left(k\right)}\right]$ is such that condition (3) holds. If the incoming vertex v is of the form

$$v=\frac{1+\alpha }{n}\sum _{i=1}^n{x}_i^{\left(k\right)}-\alpha x_{n+1}^{\left(k\right)}=x^{\left(k\right)}\left(\alpha \right)$$

for some $\alpha \in \left\{1,2,\frac{1}{2},-\frac{1}{2}\right\}$, we can define the transformation matrix

$$T\left(\alpha \right)=\left[\begin{array}{cc}{I}_n& \frac{1+\alpha }{n}e\\ 0& -\alpha \end{array}\right]\left(e={\left[1,1,\dots ,1\right]}^T\right).$$

Since $S^{\left(k\right)}T\left(\alpha \right)=\left[x_1^{\left(k\right)},\dots ,x_n^{\left(k\right)},x\left(\alpha \right)\right]$, we have to reorder the matrix columns according to the insertion rule (2). Define the permutation matrix

$$P_j=\left[e_1,\dots ,e_{j-1},e_{n+1},e_j,\dots ,e_n\right]\in {\mathbb{R}}^{\left(n+1\right)\times \left(n+1\right)}\left(j=1,\dots ,n+1\right).$$

Then, $S^{\left(k\right)}T\left(\alpha \right)P_j$ is the new simplex $S^{\left(k+1\right)}$. The following cases are possible

Operation	New simplex
Reflection ($v=x_r^{\left(k\right)}$)	$S^{\left(k+1\right)}=S^{\left(k\right)}T\left(1\right)P_j$ ($j=2,\dots ,n$)
Expansion ($v=x_e^{\left(k\right)}$)	$S^{\left(k+1\right)}=S^{\left(k\right)}T\left(2\right)P_1$
Expansion ($v=x_r^{\left(k\right)}$)	$S^{\left(k+1\right)}=S^{\left(k\right)}T\left(1\right)P_1$
Outside contraction ($v=x_{oc}^{\left(k\right)}$)	$S^{\left(k+1\right)}=S^{\left(k\right)}T\left(\frac{1}{2}\right)P_j$ ($j=1,\dots ,n+1$)
Inside contraction ($v=x_{ic}^{\left(k\right)}$)	$S^{\left(k+1\right)}=S^{\left(k\right)}T\left(-\frac{1}{2}\right)P_j$ ($j=1,\dots ,n+1$)

For shrinking, the new simplex is

$$S^{\left(k+1\right)}=S^{\left(k\right)}{T}_{shr}P,$$

where

$$T_{shr}=\frac{1}{2}{I}_{n+1}+\frac{1}{2}{e}_1{e}^T,$$

the permutation matrix $P\in {\mathcal{P}}_{n+1}$ is defined by the ordering condition (3), and ${\mathcal{P}}_{n+1}$ is the set of all possible permutation matrices of order $n+1$.

Hence, for $k\ge 1$,

$$S^{\left(k\right)}=S^{\left(k-1\right)}{T}_k{P}^{\left(k\right)}=S^{\left(0\right)}{B}_k,$$

(4)

where

$$B_k=\prod _{i=1}^k{T}_i{P}^{\left(i\right)}\left(T_i{P}^{\left(i\right)}\in \mathcal{T}\right)$$

(5)

and

$$\begin{array}{c}\mathcal{T}=\left\{T\left(\alpha \right)P_j:\alpha \in \left\{-\frac{1}{2},\frac{1}{2}\right\},j=1,\dots ,n+1\right\}\hfill \\ \hfill \cup \left\{T_{shr}P:P\in {\mathcal{P}}_{n+1}\right\}\cup \left\{T\left(1\right)P_j:j=1,\dots ,n\right\}\cup \left\{T\left(2\right)P_1\right\}\end{array}$$

(6)

Note that $\mathcal{T}$ contains $3n+3+\left(n+1\right)!$ matrices.

Observe that the transformation matrices $T\left(\alpha \right)$, $T_{shr}$, $T\left(\alpha \right)P$ and $T_{shr}P$ ($P\in {\mathcal{P}}_{n+1}$) are nonsingular and have the property that their column sums are equal to one (see, e.g., [16,17]). The latter property implies that $∥TP∥\ge 1$ ($TP\in \mathcal{T}$) holds in any induced matrix norm.

We seek for conditions that guarantee that

$$\underset{k\to \infty }{lim}{x}_i^{\left(k\right)}\to \widehat{x}\left(i=1,2,\dots ,n+1\right)$$

holds for some vector $\widehat{x}$. If so, then ${lim}_{k\to \infty }{S}^{\left(k\right)}=\widehat{x}{e}^T$ and for $k\to \infty$, both $f_i^{\left(k\right)}\to f\left(\widehat{x}\right)$ ($i=1,2,\dots ,n+1$) and diam$\left(S^{\left(k\right)}\right)\to 0$ follow. In fact, we prove the convergence of the right infinite matrix product

$$\prod _{i=1}^{\infty }{T}_i{P}^{\left(i\right)}(T_i{P}^{\left(i\right)}\in \mathcal{T})$$

to a rank-one matrix of the form $B=w{e}^T$, from which $S^{\left(k\right)}=S^{\left(0\right)}{B}_k\to \widehat{x}{e}^T$ ($\widehat{x}=S^{\left(0\right)}w$) and the speed estimate

$$\mathrm{diam}\left(S^{\left(k\right)}\right)\le \sqrt{2}∥S^{\left(0\right)}∥∥B_k-B∥$$

(7)

also follow (see also [16,17]).

Counterexamples of [16,17,18] show that even if the simplex sequence $S^{\left(k\right)}$ converges to some limit $S^{\infty }$, it may happen that diam$\left(S^{\infty }\right)>0$. If diam$\left(S^{\infty }\right)=0$ holds, that is $S^{\infty }=\widehat{x}{e}^T$ for some vector, $\widehat{x}$, it may happen that $\widehat{x}$ is not a stationary or minimum point (see McKinnon [13] and also [16,17,18]).

4. Properties of the Transformation Matrices

The spectra of the transformation matrices $T\left(\alpha \right)P_j$ and $T_{shr}P$ are fully characterized in Section 3 of [16]. Furthermore, these matrices have a common similarity form (9). Define the matrix

$$F=\left[\begin{array}{cc}1& -e^T\\ 0& I_n\end{array}\right].$$

(8)

Lemma 1 

([16,17]). For all $T_i{P}^{\left(i\right)}\in \mathcal{T}$, matrix $F^{-1}{T}_i{P}^{\left(i\right)}F$ has the form

$$F^{-1}{T}_i{P}^{\left(i\right)}F=\left[\begin{array}{cc}1& 0\\ b_i& C_i\end{array}\right],$$

(9)

where $b_i\in {\mathbb{R}}^n$ and $C_i\in {\mathbb{R}}^{n\times n}$ depends on $T_i{P}^{\left(i\right)}$.

For a more general result, see Hartfiel [19]. Note that a constant $\gamma >0$ exist such that $∥b_i∥\le \gamma$ holds for all $T_i{P}^{\left(i\right)}\in \mathcal{T}$. For later use, we make the following numbering of the elements $T_s{P}^{\left(s\right)}\in \mathcal{T}$ and their corresponding matrices $C_s$:

$\hspace{1em}{T}_s{P}^{\left(s\right)}\in \mathcal{T}$	↔	$\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}{C}_s$
$T\left(1\right)P_{j+1}$	↔	$C_j(j=1,\dots ,n-1)$
$T\left(2\right)P_1$	↔	$C_n$
$T\left(1\right)P_1$	↔	$C_{n+1}$
$T\left(\frac{1}{2}\right)P_j$	↔	$C_{n+1+j}(j=1,\dots ,n+1)$
$T\left(-\frac{1}{2}\right)P_j$	↔	$C_{2n+2+j}(j=1,\dots ,n+1)$
$T_{shr}P$$\left(P\in {\mathcal{P}}_{n+1}\right)$	↔	$C_{3n+3+j}(j=1,\dots ,\left(n+1\right)!)$

where the numbering of the permutations $P\in {\mathcal{P}}_{n+1}$ follows the perms function of Matlab (in actual computations).

5. An Improved Convergence Result

Here, we prove a new convergence theorem where the key condition is numerically checkable at least for low dimensions unlike in [16,17], where the key assumption was algorithmically undecidable and required ways to circumvent this problem. The new result is the basis of the stochastic convergence result of Section 6. The latter theorem may explain, in a sense, the experienced good behavior of the Nelder–Mead method in practice.

Formula (9) implies that

$$B_k=\prod _{j=1}^k{T}_{i_j}{P}^{\left(i_j\right)}=F{L}_k{F}^{-1}\left(T_{i_j}{P}^{\left(i_j\right)}\in \mathcal{T}\right),$$

(10)

where

$$L_k=\prod _{j=1}^k\left[\begin{array}{cc}1& 0\\ b_{i_j}& C_{i_j}\end{array}\right],$$

(11)

and $B_k$ is convergent if and only if $L_k$ is convergent to some matrix

$$\left[\begin{array}{cc}1& 0\\ \tilde{x}& \tilde{Y}\end{array}\right].$$

We use the following simple result (see, e.g., [16,17]). For $i\ge 1$, let

$$A_i=\left[\begin{array}{cc}1& 0\\ b_i& C_i\end{array}\right]\in {\mathbb{R}}^{\left(n+1\right)\times \left(n+1\right)}\left(C_i\in {\mathbb{R}}^{n\times n}\right).$$

(12)

Lemma 2. 

Assume that $∥{\prod }_{j=1}^k{C}_j∥\le c_k$, ${\sum }_{k=1}^{\infty }{c}_k$ is convergent ($<\infty$) and $∥b_k∥\le \gamma$ for all k. Then, $L_k={\prod }_{j=1}^k{A}_j$ converges and

$$\underset{k\to \infty }{lim}{L}_k=\left[\begin{array}{cc}1& 0\\ \tilde{x}& 0\end{array}\right]$$

(13)

for some $\tilde{x}$.

Proof. 

It is easy to see that

$$L_k=\prod _{j=1}^k{A}_j=\left[\begin{array}{cc}1& 0\\ \sum _{i=1}^k\left(\prod _{j=1}^{i-1}{C}_j\right)b_i& \prod _{j=1}^k{C}_j\end{array}\right]=\left[\begin{array}{cc}1& 0\\ x_k& \prod _{j=1}^k{C}_j\end{array}\right].$$

(14)

If ${\sum }_{k=1}^{\infty }{c}_k$ is convergent, then $c_k\to 0$. Hence, ${\prod }_{j=1}^k{C}_j\to 0$ as $k\to \infty$. Since $s_k={\sum }_{j=1}^k{c}_j$ is convergent, for any $\epsilon >0$ there is a number $k_0=k_0\left(\epsilon \right)$ such that for $m>k\ge k_0$, $\left|s_m-s_k\right|<\epsilon$. Thus, for $m>k\ge k_0$, we obtain

$$∥x_m-x_k∥\le \sum _{i=k+1}^m∥\prod _{j=1}^{i-1}{C}_j∥∥b_i∥\le \gamma \sum _{i=k+1}^m{c}_{i-1}\le \gamma \epsilon .$$

Hence, $x_k\to$$\tilde{x}$ for some $\tilde{x}$. □

If $∥C_j∥\le q<1$ for $j\ge 1$, then $∥{\prod }_{j=1}^k{C}_j∥\le q^k$ and the series ${\sum }_{i=1}^{\infty }{q}^i$ is convergent.

Assume now that $n\ge 2$ and consider all possible products of $T_{i_j}{P}^{\left(i_j\right)}$ of fixed length ℓ ($\ell \ge 2$). For each product ${\prod }_{j=1}^{\ell }{T}_{i_j}{P}^{\left(i_j\right)}$, there is a corresponding product ${\prod }_{j=1}^{\ell }{C}_{i_j}$. Define the sets $\mathcal{C}=\left\{C_i:T_i{P}^{\left(i\right)}\in \mathcal{T}\right\}$ and

$${\mathcal{C}}_{\ell }=\left\{\prod _{j=1}^{\ell }{C}_{i_j}:C_{i_j}\in \mathcal{C}\right\}=\left\{Y_i:i=1,2,\dots ,N\right\},$$

where $N={\left(3n+3+\left(n+1\right)!\right)}^{\ell }$. Consider the norm of the elements of ${\mathcal{C}}_{\ell }$ and decompose the set ${\mathcal{C}}_{\ell }$ in the form ${\mathcal{C}}_{\ell }=$${\mathcal{C}}_{\ell }^q\cup {\mathcal{C}}_{\ell }^Q$, where

$${\mathcal{C}}_{\ell }^q=\left\{\prod _{j=1}^{\ell }{C}_{i_j}\in {\mathcal{C}}_{\ell }:∥\prod _{j=1}^{\ell }{C}_{i_j}∥\le q\right\}\left(q<1\right),$$

$${\mathcal{C}}_{\ell }^Q=\left\{\prod _{j=1}^{\ell }{C}_{i_j}\in {\mathcal{C}}_{\ell }:q<∥\prod _{j=1}^{\ell }{C}_{i_j}∥\le Q\right\}\left(Q>1\right),$$

$0<q<1$ is a fixed number, and for simplicity, Q is selected such that $∥C_i∥\le Q$ also holds for all $C_i\in \mathcal{C}$. That such a q exists, ${\mathcal{C}}_{\ell }^Q$ and ${\mathcal{C}}_{\ell }^q$ are not empty follow from [16]. This fact is also indicated by Equation (21).

We investigate the product (11). For any $k\ge \ell$, write $k=m\ell +r$ with $m,r\in \mathbb{N}$ and $0\le r<\ell$. Note that $m=⌊\frac{k}{\ell }⌋$, where $⌊·⌋$ stands for the floor function. Then,

$$\prod _{j=1}^k{C}_{i_j}=\left[\prod _{j=1}^m\left(C_{i_{\left(j-1\right)\ell +1}}\cdots C_{i_{j\ell }}\right)\right]C_{i_{m\ell +1}}\cdots C_{i_{m\ell +r}}$$

(15)

and

$$∥\prod _{j=1}^k{C}_{i_j}∥\le \left(\prod _{i=1}^m∥C_{i_{\left(j-1\right)\ell +1}}\cdots C_{i_{j\ell }}∥\right)Q^r.$$

(16)

Assume that $r_1\left(m\right)$ℓ-products belong to ${\mathcal{C}}_{\ell }^q$ and $r_2\left(m\right)$ ℓ-product belong to ${\mathcal{C}}_{\ell }^Q$. Clearly $r_1\left(m\right)+r_2\left(m\right)=m$. Then, $∥{\prod }_{j=1}^k{C}_{i_j}∥\le q^{r_1\left(m\right)}{Q}^{r_2\left(m\right)+\ell -1}$. There exists an integer $\kappa \ge 1$ such that $q^{1-\kappa }\le Q\le q^{-\kappa }$. Hence, $∥{\prod }_{j=1}^k{C}_{i_j}∥\le Q^{\ell -1}{q}^{r_1\left(m\right)-\kappa r_2\left(m\right)}$. Moreover, assume that $r_1\left(m\right)-\kappa r_2\left(m\right)\ge \mu m$ for some $\mu \in \left(0,1\right)$. This assumption guarantees that the elements from ${\mathcal{C}}_{\ell }^q$ counterbalance the effect of those from ${\mathcal{C}}_{\ell }^Q$. It also implies the density inequalities

$$r_1\left(m\right)\ge \frac{\mu +\kappa }{1+\kappa }m>\frac{1-\mu }{1+\kappa }m\ge r_2\left(m\right).$$

It follows that

$$∥\prod _{j=1}^k{C}_{i_j}∥\le Q^{\ell -1}{q}^{\mu m}=Q^{\ell -1}{\left(q^{\mu }\right)}^m=:c_k.$$

Now, $q^{\mu }<1$ and $\frac{k}{\ell }-1<m=\frac{k-r}{\ell }\le \frac{k}{\ell }$. Hence,

$$c_k:=Q^{\ell -1}{\left(q^{\mu }\right)}^m\le Q^{\ell -1}{\left(q^{\mu }\right)}^{\frac{k}{\ell }-1}=\left[Q^{\ell -1}{q}^{-\mu }\right]{\left(q^{\frac{\mu }{\ell }}\right)}^k={\Gamma }_1{\left(q^{\frac{\mu }{\ell }}\right)}^k\to 0,$$

since $q^{\frac{\mu }{\ell }}<1$ and ${\sum }_{k=1}^{\infty }{c}_k\le {\Gamma }_1{\sum }_{k=1}^{\infty }{\left(q^{\frac{\mu }{\ell }}\right)}^k<\infty$. Hence, by Lemma 2

$$\underset{k\to \infty }{lim}{L}_k=\left[\begin{array}{cc}1& 0\\ \tilde{x}& 0\end{array}\right]=\tilde{L}$$

holds for some vector $\tilde{x}$. Since there exists a constant $\gamma >0$ such that $∥b_i∥\le \gamma$,

$$∥\tilde{x}-x_k∥=∥\sum _{i=k+1}^{\infty }\left(\prod _{j=1}^{i-1}{C}_{i_j}\right)b_i∥\le \gamma {\Gamma }_1\sum _{i=k}^{\infty }{\left(q^{\frac{\mu }{\ell }}\right)}^i\le {\Gamma }_2{\left(q^{\frac{\mu }{\ell }}\right)}^k,$$

$${∥L_k-\tilde{L}∥}_{\vartheta }\le {\Gamma }_3{\left(q^{\frac{\mu }{\ell }}\right)}^k$$

holds with a suitable constant ${\Gamma }_3>0$. It follows that

$$B_k\to F\left[\begin{array}{cc}1& 0\\ \tilde{x}& 0\end{array}\right]F^{-1}=\left[\begin{array}{c}1-e^T\tilde{x}\\ \tilde{x}\end{array}\right]e^T=w{e}^T=B$$

(17)

and

$${∥B_k-B∥}_{\vartheta }\le {\Gamma }_4\mathrm{cond}\left(F\right){\left(q^{\frac{\mu }{\ell }}\right)}^k.$$

(18)

We can summarize the obtained results in the following.

Theorem 1. 

Assume that $n\ge 2$, $S^{\left(0\right)}$ is nondegenerate, $\ell \ge 2$ is fixed and ${\mathcal{C}}_{\ell }^q$ is not empty. Let $r_1\left(⌊\frac{k}{\ell }⌋\right)$ be the number of ℓ-products that belong to ${\mathcal{C}}_{\ell }^q$ and $r_2\left(⌊\frac{k}{\ell }⌋\right)$ be the number of those ℓ-products that belong to ${\mathcal{C}}_{\ell }^Q$ during the first k iterations of the Nelder–Mead method. Moreover, assume that for $\kappa \in \mathbb{N}$, $q^{1-\kappa }\le Q\le q^{-\kappa }$ and for some $\mu \in \left(0,1\right)$, $r_1\left(⌊\frac{k}{\ell }⌋\right)\ge \mu ⌊\frac{k}{\ell }⌋+\kappa r_2\left(⌊\frac{k}{\ell }⌋\right)$ holds ($k\ge k_0$). Then, the Nelder–Mead algorithm converges in the sense that

$$\underset{k\to \infty }{lim}{x}_j^{\left(k\right)}=\widehat{x}(j=1,\dots ,n+1)$$

(19)

with a convergence speed proportional to $O\left(q^{\mu k}\right)$. If f is continuous at $\widehat{x}$, then

$$\underset{k\to \infty }{lim}f\left(x_j^{\left(k\right)}\right)=f\left(\widehat{x}\right)(j=1,\dots ,n+1)$$

(20)

holds as well.

A simple but time-consuming computation of the elements of ${\mathcal{C}}_{\ell }$ shows the feasibility of the assumptions of Theorem 1. Equation (21) shows the ratios of $\left|{\mathcal{C}}_{\ell }^q\right|/\left|{\mathcal{C}}_{\ell }\right|$ for the specified $\left(n,\ell \right)$ pairs, in case of spectral norm and $q=0.99$.

$$\begin{array}{|ccccc|}\hline n\setminus \ell \hfill & 2\hfill & 3\hfill & 4\hfill & 5\hfill \\ 2\hfill & 0.7111\hfill & 0.8361\hfill & 0.9020\hfill & 0.9409\hfill \\ 3\hfill & 0.8518\hfill & 0.9374\hfill & 0.9738\hfill & 0.9891\hfill \\ 4\hfill & 0.8507\hfill & 0.9760\hfill & 0.9956\hfill & \\ 5\hfill & 0.9641\hfill & 0.9973\hfill & \hfill & \\ 6\hfill & 0.9935\hfill & & & \\ \hline\end{array}$$

(21)

The greater the ratio $\left|{\mathcal{C}}_{\ell }^q\right|/\left|{\mathcal{C}}_{\ell }\right|$, the better the chance for convergence since more elements can be selected from ${\mathcal{C}}_{\ell }^q$ than from ${\mathcal{C}}_{\ell }^Q$. For any $n\ge 2$, there are problems on which the NM algorithm does not converge in the above sense (see [16]). Hence, in the general case, the density assumption seems to be necessary.

Corollary 1. 

For $n=2,3,4,5,6$, Equation (21) implies the convergence of the Nelder–Mead method under the density condition of Theorem 1.

For strictly convex functions $f:{\mathbb{R}}^n\to \mathbb{R}$, Lagarias et al. proved ([10] Lemma 3.5) that no shrinking occurs when the Nelder–Mead algorithm applied to f. The following observation has some importance, if we can rule out certain operations or steps of the Nelder–Mead method when it is applied to a function f.

Remark 1. 

Theorem 1 remains true, if sets $\mathcal{T}$, $\mathcal{C}$, ${\mathcal{C}}_{\ell }^q$ and ${\mathcal{C}}_{\ell }^Q$ are replaced by the subsets $\tilde{\mathcal{T}}\subset \mathcal{T}$, $\tilde{\mathcal{C}}=\left\{C_i:T_i{P}^{\left(i\right)}\in \tilde{\mathcal{T}}\right\}$ and ${\tilde{\mathcal{C}}}_{\ell }=\left\{{\prod }_{j=1}^{\ell }{C}_{i_j}:C_{i_j}\in \tilde{\mathcal{C}}\right\}$, such that ${\tilde{\mathcal{C}}}_{\ell }={\tilde{\mathcal{C}}}_{\ell }^q\cup {\tilde{\mathcal{C}}}_{\ell }^Q$,

$${\tilde{\mathcal{C}}}_{\ell }^q=\left\{\prod _{j=1}^{\ell }{C}_{i_j}\in {\tilde{\mathcal{C}}}_{\ell }:∥\prod _{j=1}^{\ell }{C}_{i_j}∥\le q\right\},$$

$${\tilde{\mathcal{C}}}_{\ell }^Q=\left\{\prod _{j=1}^{\ell }{C}_{i_j}\in {\tilde{\mathcal{C}}}_{\ell }:q<∥\prod _{j=1}^{\ell }{C}_{i_j}∥\le Q\right\}$$

and ${\tilde{\mathcal{C}}}_{\ell }^q$ is nonempty.

Remark 2. 

Assume that $n=2$ and the operation set $\mathcal{T}$ is restricted to

$$\tilde{\mathcal{T}}=\mathcal{T}\setminus \left\{T_{shr}P:P\in {\mathcal{P}}_{n+1}\right\}$$

(no shrinking occurs). For $q=0.99$, the ratios $\left|{\tilde{\mathcal{C}}}_{\ell }^q\right|/\left|{\tilde{\mathcal{C}}}_{\ell }\right|$ are given in the next equation

$$\begin{array}{|ccccccc|}\hline \ell \hfill & 2\hfill & 3\hfill & 4\hfill & 5\hfill & 6\hfill & 7\hfill \\ \left|{\tilde{\mathcal{C}}}_{\ell }^q\right|/\left|{\tilde{\mathcal{C}}}_{\ell }\right|\hfill & 0.3456\hfill & 0.4691\hfill & 0.5468\hfill & 0.6143\hfill & 0.6715\hfill & 0.7187\hfill \\ \hline\end{array}$$

(22)

Hence, we have convergence for strictly convex functions $f:{\mathbb{R}}^2\to \mathbb{R}$ under the conditions of Theorem 1.

Theorem 1 is based on the behavior of ℓ consecutive steps of the Nelder–Mead method. The difference between Theorem 1 and Theorem 9 of [16] is the following. In the case of Theorem 9 of [16], we identified a subset ${\mathcal{W}}_1\subset \mathcal{T}$ and constructed a matrix norm ${∥·∥}_{\vartheta }$, such that for every $T_i{P}^{\left(i\right)}\in {\mathcal{W}}_1$, ${∥C_i∥}_{\vartheta }\le q<1$ held. The existence of such norm was related to an algorithmically undecidable problem. The ratio of operations from ${\mathcal{W}}_1$ and $\mathcal{T}\setminus {\mathcal{W}}_1$ then decided the convergence. Here, in Theorem 1, we avoided the construction of the matrix norm ${∥·∥}_{\vartheta }$, by computing the (spectral) norms of ℓ consecutive steps (products of ℓ operators) and sorting them into two subsets ${\mathcal{C}}_{\ell }^q$ and ${\mathcal{C}}_{\ell }^Q$. The rest of the proof was quite similar to that of Theorem 9 of [16]. The norms of ℓ-products can be easily computed, although the computation time and required memory quickly increase with n and ℓ.

If one can show that for a given function f, the NM method takes only steps that belong to ${\mathcal{C}}_{\ell }^q$ for some ℓ, then the convergence immediately follows. However, if it is not the case, then either one has to make some additional condition as in Theorem 1 or seek some statistical characterization such as Theorem 2 of the next section.

6. A Random Convergence Result

The counterexamples of [16,17,18] showed that there was no sure convergence for the NM method. One may ask, however, if Theorem 1 is sharp enough in some sense. Here, we study a simple random model using the proof of Theorem 1. Formulas (15) and (16) imply that it is enough to study the infinite product

$$\prod _{j=1}^{\infty }{C}_{i_j}=\left[\prod _{j=1}^{\infty }\left(C_{i_{\left(j-1\right)\ell +1}}\cdots C_{i_{j\ell }}\right)\right]=\prod _{j=1}^{\infty }{Y}_{i_j}(Y_{i_j}\in {\mathcal{C}}_{\ell }).$$

Assume that the ℓ-products $Y_i\in {\mathcal{C}}_{\ell }$ are randomly chosen with the probability $p_i\ge 0$ and ${\sum }_{i=1}^N{p}_i=1$. Moreover, assume that the subsequent ℓ-products $C_{i_{\left(j-1\right)\ell +1}}\cdots C_{i_{j\ell }}$ ($T_{i_{\left(j-1\right)\ell +1}}\cdots T_{i_{j\ell }}$) are randomly chosen independently of each other.

Let $a_i={∥Y_i∥}_2$ ($Y_i\in {\mathcal{C}}_{\ell }$, $i=1,\dots ,N$). Note that $a_i>0$ for $1\le i\le N$ and there are numbers $a_j$ such that $a_j>1$ (see also Section 3 of [16]). Let X be a random variable and let $a_1,a_2,\dots ,a_N$ be the values which it assumes. Let $P\left(X=a_i\right)=p_i$ ($1\le i\le N$) be the probability distribution of X. Assume that for the expected value of X, $\mu =EX={\sum }_{i=1}^N{p}_i{a}_i<1$ holds.

If X is uniformly distributed, the expected values $\mu =EX$ belonging to the cases of Equation (21) are given in the following equation (with four-decimal digit precision).

$$\begin{array}{|ccccc|}\hline n\setminus \ell \hfill & 2\hfill & 3\hfill & 4\hfill & 5\hfill \\ 2\hfill & 0.8512\hfill & 0.6515\hfill & 0.4931\hfill & 0.3725\hfill \\ 3\hfill & 0.7435\hfill & 0.4891\hfill & 0.3202\hfill & 0.2093\hfill \\ 4\hfill & 0.5963\hfill & 0.3305\hfill & 0.1825\hfill & \\ 5\hfill & 0.5704\hfill & 0.2918\hfill & \hfill & \\ 6\hfill & 0.6015\hfill & \hfill & & \\ \hline\end{array}$$

(23)

Hence, the condition $\mu =EX<1$ holds in these cases.

We need the following simple results.

Lemma 3. 

Let the positive random variables ${\left\{X_i\right\}}_{i=1}^{\infty }$ be independent and identically distributed with the same distribution as X, and assume that $\mu =EX<1$. Then, $Z_k={\prod }_{i=1}^k{X}_i\to 0$ holds with a probability of one. Moreover, there exist numbers ${\left\{c_k\right\}}_{k=1}^{\infty }$ such that ${\sum }_{k=1}^{\infty }{c}_k<\infty$ and only a finite number of the events $\left\{Z_k\ge c_k\right\}$ can occur with a probability of one.

Proof. 

The independence of $X_i$’s implies that $E{Z}_k={\prod }_{i=1}^k\left(E{X}_i\right)={\mu }^k$. For any $c_k>0$, it follows from the Markov inequality that $P\left(Z_k\ge c_k\right)\le \frac{E{Z}_k}{c_k}=\frac{{\mu }^k}{c_k}$. Select $c_k={\mu }^{k/2}$ ($k\ge 1$). Then,

$$\sum _{k=1}^{\infty }P\left(Z_k\ge c_k\right)\le \sum _{k=1}^{\infty }{c}_k=\frac{\sqrt{\mu }}{1-\sqrt{\mu }}<\infty .$$

It follows from the Borel–Cantelli lemma (see, e.g., Theorem 1.5.1 of [20] or Borovkov [21]) that $Z_n\to 0$ with a probability of one and only a finite number of the events $\left\{Z_k\ge c_k\right\}$ ($k=1,2,\dots$) can occur. □

Corollary 2. 

Under the conditions of Lemma 3 there exists numbers ${\tilde{c}}_k\ge 0$ ($k\ge 1$) such that $Z_k\le {\tilde{c}}_k$ ($k\ge 1$) and ${\sum }_{k=1}^{\infty }{\tilde{c}}_k<\infty$.

Proof. 

There is a random index $n\left(\omega \right)$ such that $P\left(n\left(\omega \right)<\infty \right)$ with a probability of one, and $Z_k<c_k$ if $k\ge n\left(\omega \right)$. Since $\tilde{a}={max}_i{a}_i>1$ and for any k, $P\left(Z_k={\tilde{a}}^k\right)\ge P\left(X_1=\tilde{a},\dots ,X_k=\tilde{a}\right)>0$, there is no fixed number $n_0$ such that $Z_k<\epsilon$, for $k\ge n_0$ and for all $\epsilon >0$. However, for $k<n\left(\omega \right)$, $Z_k\le {\tilde{c}}_k:={\tilde{a}}^k$ and $Z_k<{\tilde{c}}_k:=c_k$ for $k\ge n\left(\omega \right)$. Thus, it follows that $Z_k\le {\tilde{c}}_k$ ($k\ge 1$) and ${\sum }_{k=1}^{\infty }{\tilde{c}}_k<\infty$. □

Theorem 2. 

Assume that $n\ge 2$, $S^{\left(0\right)}$ is nondegenerate, $\ell \ge 2$ is fixed, ${\mathcal{C}}_{\ell }^q\ne \varnothing$, and the ℓ-products $Y_i\in {\mathcal{C}}_{\ell }$ are randomly chosen with probability $p_i\ge 0$ and ${\sum }_{i=1}^N{p}_i=1$. Furthermore, assume that the subsequent ℓ-products are randomly chosen independently of each other. If $\mu ={\sum }_{i=1}^N{a}_i{p}_i<1$, then for the Nelder–Mead algorithm,

$$\underset{k\to \infty }{lim}{x}_j^{\left(k\right)}=\widehat{x}(j=1,\dots ,n+1)$$

(24)

holds with a probability of one. If, in addition, f is continuous at $\widehat{x}$, then

$$\underset{k\to \infty }{lim}f\left(x_j^{\left(k\right)}\right)=f\left(\widehat{x}\right)(j=1,\dots ,n+1)$$

(25)

also holds with a probability of one.

Proof. 

Let X be a random variable defined on the positive numbers ${\left\{a_i\right\}}_{i=1}^N$ with the probability distribution ${\left\{p_i\right\}}_{i=1}^N$. By assumption, $\mu =EX={\sum }_{i=1}^N{a}_i{p}_i<1$. Lemma 3 and Corollary 2 imply that

$$∥\prod _{j=1}^{k\ell }{C}_{i_j}∥\le \prod _{j=1}^k∥C_{i_{\left(j-1\right)\ell +1}}\cdots C_{i_{j\ell }}∥=\prod _{j=1}^k{a}_{s_j}\le {\tilde{c}}_k\left(k\ge 1\right)$$

and ${\sum }_{i=1}^{\infty }{\tilde{c}}_k<\infty$. The result now follows from Lemma 2, the proof of Theorem 1 and the “continuity theorem” (see, e.g., Borovkov [21], Theorem 6.1.4, p. 134). □

Note that assumption $r_1\left(⌊\frac{k}{\ell }⌋\right)\ge \mu ⌊\frac{k}{\ell }⌋+\kappa r_2\left(⌊\frac{k}{\ell }⌋\right)$ does not occur here. Instead, we use the assumption $\mu =EX<1$. Furthermore, note that the smaller the $\mu$, the faster the convergence.

If we assume a uniform distribution for X, then we have the following.

Corollary 3. 

Under the assumption of uniform distribution, Equation (23) implies Theorem 2 for $n=2,3,4,5,6$.

Theorem 2 and the Corollary 3 ensure the convergence of the NM algorithm with a probability of one although the limit point is not necessarily a minimum point of f. Concerning the speed of convergence, we note that some classical stochastic approximation algorithms (Robbins–Monro, Kiefer–Wolfowitz methods) also converge with a probability of one (see, e.g., [22]).

Remark 3. 

Theorem 2 remains true, if the sets ${\mathcal{C}}_{\ell }^q$ and ${\mathcal{C}}_{\ell }$ are replaced by the subsets ${\tilde{\mathcal{C}}}_{\ell }^q$ and ${\tilde{\mathcal{C}}}_{\ell }$, respectively, and the probability assumptions for these also hold.

Remark 4. 

Assume that $n=2$ and the operation set $\mathcal{T}$ is restricted to

$$\tilde{\mathcal{T}}=\mathcal{T}\setminus \left\{T_{shr}P:P\in {\mathcal{P}}_{n+1}\right\}$$

(no shrinking occurs). For $q=0.99$ and a uniform distribution, the expected values μ are given in the next equation

$$\begin{array}{|ccccccc|}\hline \ell \hfill & 2\hfill & 3\hfill & 4\hfill & 5\hfill & 6\hfill & 7\hfill \\ \mu \hfill & 1.2961\hfill & 1.2139\hfill & 1.1215\hfill & 1.0334\hfill & 0.9489\hfill & 0.8701\hfill \\ \hline\end{array}$$

(26)

Hence, Theorem 2 holds for $\ell =6,7$. It follows that for strictly convex functions $f:{\mathbb{R}}^2\to \mathbb{R}$, the Nelder–Mead method converges with a probability of one under the assumption of uniform distribution.

7. Conclusions

Although the convergence of the Nelder–Mead simplex method to a minimum point of a function f cannot be guaranteed in general, the main result indicates that in the stochastic sense, it converges to a point at which the value of f is less than the best function value at the start. It also may explain the good behavior of the Nelder–Mead method on average.