Randomized Simplicial Hessian Update

Abstract

Recently, a derivative-free optimization algorithm was proposed that utilizes a minimum Frobenius norm (MFN) Hessian update for estimating the second derivative information, which in turn is used for accelerating the search. The proposed update formula relies only on computed function values and is a closed-form expression for a special case of a more general approach first published by Powell. This paper analyzes the convergence of the proposed update formula under the assumption that the points from ${\mathbb{R}}^n$ where the function value is known are random. The analysis assumes that the $N+2$ points used by the update formula are obtained by adding $N+1$ vectors to a central point. The vectors are obtained by transforming a prototype set of $N+1$ vectors with a random orthogonal matrix from the Haar measure. The prototype set must positively span a $N\le n$ dimensional subspace. Because the update is random by nature we can estimate a lower bound on the expected improvement of the approximate Hessian. This lower bound was derived for a special case of the proposed update by Leventhal and Lewis. We generalize their result and show that the amount of improvement greatly depends on N as well as the choice of the vectors in the prototype set. The obtained result is then used for analyzing the performance of the update based on various commonly used prototype sets. One of the results obtained by this analysis states that a regular n-simplex is a bad choice for a prototype set because it does not guarantee any improvement of the approximate Hessian.

1. Introduction

Derivative-free optimization algorithms have attracted much attention due to the fact that in many optimization problems, the evaluation of the gradients of the function subject to optimization and constraints is expensive. Such optimization problems can be often formulated as constrained black-box optimization (BBO) [1] problems of the form

$$\begin{array}{c}\hfill minf\left(\mathbf{x}\right)\mathrm{subject}\mathrm{to}\end{array}$$

(1)

$$\begin{array}{c}\hfill c_i\left(\mathbf{x}\right)\le 0i=1,2,\dots ,n_{\mathrm{C}}\end{array}$$

(2)

Functions f and $c_i$ are maps from ${\mathbb{R}}^n$ to $\mathbb{R}$. The objective is to minimize f subject to $n_{\mathrm{C}}$ nonlinear constraints defined by functions $c_i$. The method for computing f and $c_i$ is treated as a black-box, and the gradients are usually not available. Such problems often arise in engineering optimization when simulation is used for obtaining the function values. BBO often relies on models of the function and of the constraints. Various approaches to building black-box models were developed in the past, such as linear [2] and quadratic models [3], radial-basis functions [4], support vector machines [5], neural networks [6], etc.

In this paper, we focus on the quadratic models of f and $c_i$. The most challenging task in building these models is the computation of the Hessian matrix. Instead of using the exact Hessian, the model can utilize an approximate Hessian. The approximation can be improved gradually by applying an update formula based on the function and the gradient values at points visited in the algorithm’s past. As the algorithm converges towards a solution, the approximate Hessian converges to the true Hessian.

For derivative-based optimization, several approaches for updating the approximate Hessian are well studied and tested in practice (e.g., BFGS update, SR1 update [7]). Unfortunately, these approaches rely on the gradient of the function (constraints), which, by assumption, is not available in derivative-free optimization.

Let n denote the dimension of the search space. For derivative-free optimization, a Hessian update formula based on the function values computed at $m\ge n+2$ points visited in the algorithm’s past was proposed by Powell in [8]. The update formula was obtained by minimizing the Frobenius norm of the update applied to the approximate Hessian subject to linear constraints imposed by the function values at m points in the search space. The paper proposed an efficient way for computing the update and explored some of its properties. The convergence rate of the update formula was not studied.

In a later paper, a simple update formula that uses three collinear points for computing the updated approximate Hessian [9] was examined. The normalized direction along which the three points lie was assumed to be uniformly distributed on the unit sphere. With this assumption, the convergence rate of the update was analyzed and shown to be linear. This update formula was successfully used in a derivative-free algorithm from the family of mesh adaptive direct search algorithms (MADS) [10]. A similar Hessian updating approach was used for speeding up global optimization in [11].

The assumption that the points taking part in an update must be collinear is a significant limitation for the underlying derivative-free algorithm. With this in mind, a new simplicial update formula was proposed in [12]. The formula relies on $m\le n+2$ points. The reason for choosing the term simplicial Hessian update is the fact that the $m-1$ points form a simplex centered around the first point. For $m=n+2$, the formula is a special case of the update formula proposed in [8]. By imposing some restrictions on the positions of the m points, the update formula can be used for any m that satisfies $3\le m\le n+2$. The case $m=3$ corresponds to the update formula proposed in [9].

To illustrate the approach for obtaining the update formula, let us assume that the current quadratic model of function f is given by

$$m\left(\mathbf{x}\right)=\frac{1}{2}{\mathbf{x}}^{\mathrm{T}}\mathbf{B}\mathbf{x}+{\widehat{\mathbf{g}}}^{\mathrm{T}}\mathbf{x}+\widehat{c}$$

(3)

$\mathbf{B}$ is the current approximate Hessian. Let the points where the function is known be denoted by ${\mathbf{x}}_i$. For the sake of simplicity, let $f_i$ denote $f\left({\mathbf{x}}_i\right)$. Based on these points, we are looking for an updated model:

$$m_{+}\left(\mathbf{x}\right)=\frac{1}{2}{\mathbf{x}}^{\mathrm{T}}{\mathbf{B}}_{+}\mathbf{x}+{\widehat{\mathbf{g}}}_{+}^{\mathrm{T}}\mathbf{x}+{\widehat{c}}_{+}.$$

(4)

The model must satisfy m constraints

$$m_{+}\left({\mathbf{x}}_i\right)=f_i$$

(5)

that are linear in $\widehat{c}$ and the components of ${\mathbf{B}}_{+}$ and ${\widehat{\mathbf{g}}}_{+}$. Based on these constraints, we are looking for an updated approximate Hessian ${\mathbf{B}}_{+}$. Because we have fewer constraints than there are unknowns, we also require that $\parallel {\mathbf{B}}_{+}{-\mathbf{B}\parallel }_{\mathrm{F}}$ is minimal (${\parallel ·\parallel }_{\mathrm{F}}$ denotes the Frobenius norm). The update formula we obtain in this way is a minimum Frobenius norm update formula.

For computing the expected improvement of the approximate Hessian, we first assume f itself is quadratic. We also assume the aforementioned m points are obtained by applying a random orthogonal transformation to $m-1$ vectors that form a prototype set and adding the resulting vectors to a central point. As in [9], the convergence rate of the update is linear. The speed with which the approximate Hessian converges to the true Hessian depends on the choice of the prototype set. Our result is a generalization of the result published in [9].

This paper is divided as follows. In Section 2, some basic properties of minimum Frobenius norm updates are explored. The Frobenius product is revisited with the purpose of simplifying the notation, and the update formula is derived. In the next section, uniformly distributed orthogonal matrices are introduced. Some auxiliary results are derived that are later used for computing the expected improvement of the approximate Hessian. Section 4 analyzes the convergence of the proposed update and derives the expected value of the improvement in the sense of the Frobenius norm of the difference between the approximate Hessian and the true Hessian. The expected improvement is computed for several prototype sets. The section is followed by an example demonstrating the convergence of the proposed update and concluding remarks.

Notation. Components of vectors ($\mathbf{a}$) and matrices ($\mathbf{A}$) are denoted by subscripts (i.e., $a_i$ and $a_{ij}$, respectively). The i-th column of matrix $\mathbf{A}$ is denoted by ${\mathbf{a}}_i$. The unit vectors forming an orthogonal basis for ${\mathbb{R}}^n$ are denoted by ${\mathbf{e}}_i$. Vectors are assumed to be column vectors, and the inner product of two vectors is written in matrix notation as ${\mathbf{a}}^{\mathrm{T}}\mathbf{b}$. The Frobenius norm and the trace of a matrix are denoted by ${\parallel ·\parallel }_{\mathrm{F}}$ and $\mathrm{tr}(·)$, respectively. The expected value of a random variable is denoted by $\mathrm{E}\left[·\right]$.

2. Obtaining the Update Formula

Let $\mathbf{H}$ denote the Hessian of a function. Minimum Frobenius norm (MFN) update formulas replace the current Hessian approximation $\mathbf{B}$ with a new (better) approximation ${\mathbf{B}}_{+}$ in such manner that the Frobenius norm of the change (i.e., ${\mathbf{B}}_{+}-\mathbf{B}$) is minimal, subject to constraints imposed on ${\mathbf{B}}_{+}$.

The Frobenius norm is a norm induced by the Frobenius (inner) product on the space of n-by-n matrices. The Frobenius product of two matrices is given by

$$\mathbf{A}:\mathbf{B}=\sum _{i=1}^n\sum _{j=1}^n{a}_{ij}{b}_{ij}=\mathrm{tr}\left({\mathbf{A}}^{\mathrm{T}}\mathbf{B}\right)=\mathrm{tr}\left({\mathbf{B}}^{\mathrm{T}}\mathbf{A}\right).$$

(6)

Using the Frobenius product, one can express the Frobenius norm of matrix $\mathbf{A}$ as

$$\begin{array}{ccc}\hfill {\parallel \mathbf{A}\parallel }_{\mathrm{F}}^2& =& \mathbf{A}:\mathbf{A}.\hfill \end{array}$$

(7)

Quadratic terms can be expressed with the Frobenius product as

$$\begin{array}{ccc}\hfill {\mathbf{x}}^{\mathrm{T}}\mathbf{A}\mathbf{x}& =& \mathbf{A}:\left(\mathbf{x}{\mathbf{x}}^{\mathrm{T}}\right).\hfill \end{array}$$

(8)

The Frobenius product introduces the notion of perpendicularity into the set of matrices (not to be confused with the orthogonality of matrices, which is equivalent to ${\mathbf{Q}}^{\mathrm{T}}\mathbf{Q}=\mathbf{I}$).

Definition 1.

Two nonzero matrices $\mathbf{A}$ and $\mathbf{B}$ are perpendicular (denoted by $\mathbf{A}\perp \mathbf{B}$) if $\mathbf{A}:\mathbf{B}=0$.

The Frobenius product can also be used for expressing linear constraints. A linear equality constraint on matrix $\mathbf{X}$ can be formulated as

$$\mathbf{A}:\mathbf{X}=a.$$

(9)

The following Lemma provides motivation for the use of minimum Frobenius norm updating.

Lemma 1.

Let $\mathbf{H}$, $\mathbf{B}$, and ${\mathbf{B}}_{+}$ denote the exact, the current approximate, and the updated approximate Hessian, respectively. Suppose we have m linear equality constraints of the form

$${\mathbf{A}}_i:{\mathbf{B}}_{+}=a_i,i=1,\dots ,m.$$

(10)

imposed on ${\mathbf{B}}_{+}$. Let ${\mathcal{P}}_{\perp }$ denote the subspace spanned by matrices ${\mathbf{A}}_i$. Then, the corresponding MFN update satisfies

•  $({\mathbf{B}}_{+}-\mathbf{B})\in {\mathcal{P}}_{\perp }$, and
•  $\parallel {\mathbf{B}}_{+}{-\mathbf{H}\parallel }_{\mathrm{F}}\le {\parallel \mathbf{B}-\mathbf{H}\parallel }_{\mathrm{F}}$.

Proof. 

Finding the MFN update is equivalent to minimizing the Frobenius norm of ${\mathbf{B}}_{+}-\mathbf{B}$ subject to linear equality constraints (10). These constraints define an affine subspace in the $n(n+1)/2$ dimensional space of Hessian matrices, and ${\mathbf{B}}_{+}$ is a member of this affine subspace. Because the true Hessian also satisfies constraints (10), it is also a member of the aforementioned affine subspace.

To simplify the problem, we can translate it in such manner that $\mathbf{H}$ becomes $\mathbf{0}$. When we do this, the linear constraints become homogeneous, and instead of an affine subspace, they now define an ordinary subspace $\mathcal{P}$. Its orthogonal complement ${\mathcal{P}}_{\perp }$ is spanned by matrices ${\mathbf{A}}_i$. Due to translation, $\mathbf{B}$ and ${\mathbf{B}}_{+}$ are replaced by $\mathbf{B}-\mathbf{H}$ and ${\mathbf{B}}_{+}-\mathbf{H}$, of which the latter is a member of $\mathcal{P}$. Points with constant $\parallel {\mathbf{B}}_{+}{-\mathbf{B}\parallel }_{\mathrm{F}}={\parallel {\mathbf{B}}_{+}-\mathbf{H}-(\mathbf{B}-\mathbf{H})\parallel }_{\mathrm{F}}$ lie on a sphere centered at $\mathbf{B}-\mathbf{H}$. Matrix ${\mathbf{B}}_{+}-\mathbf{H}$ that corresponds to the smallest $\parallel {\mathbf{B}}_{+}{-\mathbf{B}\parallel }_{\mathrm{F}}$ lies on a sphere centered at $\mathbf{B}-\mathbf{H}$ that is tangential to subspace $\mathcal{P}$. Therefore, ${\mathbf{B}}_{+}-\mathbf{B}$ must be perpendicular to $\mathcal{P}$, i.e., ${\mathbf{B}}_{+}-\mathbf{B}\in {\mathcal{P}}_{\perp }$. This proves the first claim.

Due to ${\mathbf{B}}_{+}-\mathbf{H}\in \mathcal{P}$, we can see that ${\mathbf{B}}_{+}-\mathbf{H}$ and ${\mathbf{B}}_{+}-\mathbf{B}$ are perpendicular. From $\mathbf{B}-\mathbf{H}={\mathbf{B}}_{+}-\mathbf{H}-({\mathbf{B}}_{+}-\mathbf{B})$, we have

$${\parallel \mathbf{B}-\mathbf{H}\parallel }_{\mathrm{F}}^2\hspace{0.17em}=\hspace{0.17em}\parallel {\mathbf{B}}_{+}{-\mathbf{H}\parallel }_{\mathrm{F}}^2+{\parallel {\mathbf{B}}_{+}-\mathbf{B}\parallel }_{\mathrm{F}}^2$$

(11)

The second claim immediately follows from this result. □

Consider a quadratic function

$$q\left(\mathbf{x}\right)=\frac{1}{2}{\mathbf{x}}^{\mathrm{T}}\mathbf{H}\mathbf{x}+{\mathbf{g}}^{\mathrm{T}}\mathbf{x}+c$$

(12)

where $\mathbf{H}$ is its Hessian and $\mathbf{g}$ its gradient at $\mathbf{x}=\mathbf{0}$. Let the current and the updated approximation to $q\left(\mathbf{x}\right)$ be given by

$$m\left(\mathbf{x}\right)=\frac{1}{2}{\mathbf{x}}^{\mathrm{T}}\mathbf{B}\mathbf{x}+{\widehat{\mathbf{g}}}^{\mathrm{T}}\mathbf{x}+\widehat{c}$$

(13)

and

$$m_{+}\left(\mathbf{x}\right)=\frac{1}{2}{\mathbf{x}}^{\mathrm{T}}{\mathbf{B}}_{+}\mathbf{x}+{\widehat{\mathbf{g}}}_{+}^{\mathrm{T}}\mathbf{x}+{\widehat{c}}_{+},$$

(14)

respectively. In MFN, updating ${\mathbf{B}}_{+}$ is obtained by minimizing $\parallel {\mathbf{B}}_{+}{-\mathbf{B}\parallel }_{\mathrm{F}}$. The following lemma introduces one such update based on the case when the value of q is known at $N+2$ points.

Lemma 2.

Let $q_0,\dots ,q_{N+1}$, where $N\le n$ denote the values of $q\left(\mathbf{x}\right)$ corresponding to distinct points ${\mathbf{x}}_0,\dots ,{\mathbf{x}}_{N+1}$, respectively. Let ${\mathbf{v}}_i={\mathbf{x}}_i-{\mathbf{x}}_0$ and assume ${\sum }_{i=1}^{N+1}{\alpha }_i{\mathbf{v}}_i=\mathbf{0}$ with at least one ${\alpha }_i\ne 0$. Then the simplicial MFN update satisfying the interpolation conditions $m_{+}\left({\mathbf{x}}_i\right)=q_i$ for $i=0,1,\dots ,N+1$ can be computed as

$${\mathbf{B}}_{+}=\mathbf{B}+\beta \mathbf{A}$$

(15)

where

$$\begin{array}{ccc}\mathbf{A}& =& \frac{1}{2}\sum _{i=1}^{N+1}{\alpha }_i{\mathbf{v}}_i{\mathbf{v}}_i^{\mathrm{T}},\end{array}$$

(16)

$$\begin{array}{ccc}\beta & =& \frac{{\sum }_{i=1}^{N+1}{\alpha }_i\left({\mathbf{v}}_i^{\mathrm{T}}(\mathbf{H}-\mathbf{B}){\mathbf{v}}_i\right)}{{2\parallel \mathbf{A}\parallel }_{\mathrm{F}}^2}=\frac{{\sum }_{i=1}^{N+1}{\alpha }_i\left(2(q_i-q_0)-{\mathbf{v}}_i^{\mathrm{T}}\mathbf{B}{\mathbf{v}}_i\right)}{{2\parallel \mathbf{A}\parallel }_{\mathrm{F}}^2}.\end{array}$$

(17)

Proof. 

By assumption we have

$$\begin{array}{ccc}{q}_i=q\left({\mathbf{x}}_i\right)& =& \frac{1}{2}{\mathbf{x}}_i^{\mathrm{T}}\mathbf{H}{\mathbf{x}}_i+{\mathbf{g}}^{\mathrm{T}}{\mathbf{x}}_i+c\hfill \\ & =& \frac{1}{2}{\left({\mathbf{x}}_0+{\mathbf{v}}_i\right)}^{\mathrm{T}}\mathbf{H}\left({\mathbf{x}}_0+{\mathbf{v}}_i\right)\hspace{0.17em}+\hspace{0.17em}{\mathbf{g}}^{\mathrm{T}}\left({\mathbf{x}}_0+{\mathbf{v}}_i\right)\hspace{0.17em}+\hspace{0.17em}c\hfill \end{array}$$

(18)

Due to the interpolation conditions, we have $N+2$ constraints

$$q_i=m_{+}\left({\mathbf{x}}_i\right)=\frac{1}{2}{\left({\mathbf{x}}_0+{\mathbf{v}}_i\right)}^{\mathrm{T}}{\mathbf{B}}_{+}\left({\mathbf{x}}_0+{\mathbf{v}}_i\right)\hspace{0.17em}+\hspace{0.17em}{\widehat{\mathbf{g}}}_{+}^{\mathrm{T}}\left({\mathbf{x}}_0+{\mathbf{v}}_i\right)\hspace{0.17em}+\hspace{0.17em}{\widehat{c}}_{+}$$

(19)

By subtraction, we eliminate ${\widehat{c}}_{+}$ and obtain $N+1$ constraints

$$q_i-q_0=\frac{1}{2}{\mathbf{v}}_i^{\mathrm{T}}{\mathbf{B}}_{+}{\mathbf{v}}_i+{({\widehat{\mathbf{g}}}_{+}+{\mathbf{B}}_{+}{\mathbf{x}}_0)}^{\mathrm{T}}{\mathbf{v}}_{i}i=1,..,N+1.$$

(20)

Multiplying (20) with ${\alpha }_i$ and adding the resulting equations yields

$$\frac{1}{2}\sum _{i=1}^{N+1}{\alpha }_i{\mathbf{v}}_i^{\mathrm{T}}{\mathbf{B}}_{+}{\mathbf{v}}_i+{({\widehat{\mathbf{g}}}_{+}+{\mathbf{B}}_{+}{\mathbf{x}}_0)}^{\mathrm{T}}\sum _{i=1}^{N+1}{\alpha }_i{\mathbf{v}}_i=\sum _{i=1}^{N+1}{\alpha }_i(q_i-q_0).$$

(21)

By assumption, the second term on the left-hand side of (21) vanishes (thus, ${\widehat{\mathbf{g}}}_{+}$ is eliminated). We are left with a single linear constraint on ${\mathbf{B}}_{+}$:

$$\frac{1}{2}\sum _{i=1}^{N+1}{\alpha }_i{\mathbf{v}}_i^{\mathrm{T}}{\mathbf{B}}_{+}{\mathbf{v}}_i=\sum _{i=1}^{N+1}{\alpha }_i(q_i-q_0)$$

(22)

which can be rewritten by recalling (8) as

$$\mathbf{A}:{\mathbf{B}}_{+}=\sum _{i=1}^{N+1}{\alpha }_i(q_i-q_0),$$

(23)

where

$$\mathbf{A}=\frac{1}{2}\sum _{i=1}^{N+1}{\alpha }_i{\mathbf{v}}_i{\mathbf{v}}_i^{\mathrm{T}}.$$

(24)

Equation (23) is a linear constraint on the updated Hessian approximation ${\mathbf{B}}_{+}$. This is the only constraint on ${\mathbf{B}}_{+}$. From Lemma 1, we can see that ${\mathcal{P}}_{\perp }$ is spanned by $\mathbf{A}$. Therefore, we can write

$${\mathbf{B}}_{+}-\mathbf{B}=\beta \mathbf{A}.$$

(25)

By computing the Frobenius product of (25) with $\mathbf{A}$ and taking into account (23), we arrive at

$$\sum _{i=1}^{N+1}{\alpha }_i(q_i-q_0)-\mathbf{A}:\mathbf{B}=\beta \mathbf{A}:\mathbf{A}.$$

(26)

Now we can compute $\beta$:

$$\beta =\frac{{\sum }_{i=1}^{N+1}{\alpha }_i(q_i-q_0)-\mathbf{A}:\mathbf{B}}{\mathbf{A}:\mathbf{A}}\hspace{1em}=\hspace{1em}\frac{{\sum }_{i=1}^{N+1}{\alpha }_i\left(2(q_i-q_0)-{\mathbf{v}}_i^{\mathrm{T}}\mathbf{B}{\mathbf{v}}_i\right)}{{2\parallel \mathbf{A}\parallel }_{\mathrm{F}}^2}.$$

 □

The simplicial update formula introduced by Lemma 2 is the closed-form solution of the equations arising from the MFN update in [8] for $N=n$. One can see this by comparing the interpolation conditions to those in [8]. Due to the assumption ${\sum }_{i=1}^{N+1}{\alpha }_i{\mathbf{v}}_i=\mathbf{0}$, we can also apply it when $N<n$. The assumption implies the points ${\mathbf{x}}_1,\dots ,{\mathbf{x}}_{N+1}$ are positioned in a specific manner with respect to ${\mathbf{x}}_0$ (i.e., there exists a nontrivial linear combination ${\sum }_{i=1}^{N+1}{\alpha }_i({\mathbf{x}}_i-{\mathbf{x}}_0)=\mathbf{0}$).

By choosing $N=1$, we obtain a special case of the simplicial MFN update, where all three distinct points must be collinear to satisfy ${\sum }_{i=1}^{N+1}{\alpha }_i{\mathbf{v}}_i=\mathbf{0}$. Suppose ${\mathbf{v}}_1=-{\mathbf{v}}_2=\mathbf{v}$ and ${\alpha }_1={\alpha }_2=1$. Then,

$$\begin{array}{ccc}\hfill \mathbf{A}& =& \mathbf{v}{\mathbf{v}}^{\mathrm{T}},\hfill \end{array}$$

(27)

$$\begin{array}{ccc}\hfill \beta & =& \frac{(q_1+q_2-2q_0)-{\mathbf{v}}^{\mathrm{T}}\mathbf{B}\mathbf{v}}{{\parallel \mathbf{v}\parallel }^4}=\frac{q_{\mathbf{v}}^{\left(2\right)}\left({\mathbf{x}}_0\right)-{\mathbf{v}}^{\mathrm{T}}\mathbf{B}\mathbf{v}}{{\parallel \mathbf{v}\parallel }^4}\hfill \end{array}$$

(28)

where $q_{\mathbf{v}}^{\left(2\right)}\left({\mathbf{x}}_0\right)={\mathbf{v}}^{\mathrm{T}}\mathbf{H}\mathbf{v}$ is the second directional derivative of q along direction $\mathbf{v}$. The convergence properties of this MFN update formula were analyzed in [9]. The formula was used in the derivative-free optimization algorithm proposed in [10].

3. Uniformly Distributed Orthogonal Matrices

The notion of a uniform distribution over the group of orthogonal matrices (${\mathcal{O}}_n$) can be introduced via the Haar measure [13]. Let $\mathbf{A}$ denote a matrix with independent normally distributed elements with zero mean and variance 1. A random orthogonal matrix from the Haar measure ($\mathbf{O}$) can then be obtained with Algorithm 1.

Algorithm 1 Constructing a random orthogonal matrix from the Haar measure.

Perform QR decomposition $\mathbf{A}=\mathbf{Q}\mathbf{R}$.  Construct a diagonal matrix $\mathbf{D}$ with $d_{ii}=1$ if $r_{ii}\ge 0$ and $d_{ii}=-1$ otherwise.  $\mathbf{O}=\mathbf{Q}\mathbf{D}$.

Multiplying $\mathbf{O}$ with any unit vector results in a random unit vector that is uniformly distributed on the unit sphere (${\mathcal{S}}_n$) [14]. It can be shown that ${\mathbf{O}}^{\mathrm{T}}$ is also a uniformly distributed orthogonal matrix. Consequently, every column and every row of $\mathbf{O}$ are a random unit vector with uniform distribution on ${\mathcal{S}}_n$.

The results of this section are obtained with the help of the following lemma.

Lemma 3.

Let $x\in {\mathbb{R}}^n$ and let $\mathrm{d}\sigma$ denote the surface element of ${\mathcal{S}}_n$. Then,

$$V_n(r_1,r_2,\dots ,r_n)={\int }_{\parallel \mathbf{x}\parallel =1}{x}_1^{2r_1}{x}_2^{2r_2}·\cdots ·x_n^{2r_n}\mathrm{d}\sigma =\frac{2{\prod }_{i=1}^n\Gamma \left(r_i+1/2\right)}{\Gamma \left(n/2+{\sum }_{i=1}^n{r}_i\right)}$$

(29)

Proof. 

See [15], Appendix B. □

From Lemma 3, we can obtain the surface area of ${\mathcal{S}}_n$ by choosing $r_1=\dots =r_n=0$.

$$S_n=\frac{2\Gamma {(1/2)}^n}{\Gamma (n/2)}$$

(30)

Let ${\mathbf{o}}_i$ and ${\mathbf{o}}_j$ denote two random vectors that correspond to the i-th and the j-th column of $\mathbf{O}$. If $i\ne j$, then ${\mathbf{o}}_i^{\mathrm{T}}{\mathbf{o}}_j=0$. We denote the k-th component of ${\mathbf{o}}_i$ as $o_{ki}$.

Lemma 4.

Let $\mathbf{O}$ be a uniformly distributed random orthogonal matrix. Then,

$$\begin{array}{ccc}\hfill \mathrm{E}\left[{\left({\mathbf{e}}_k^{\mathrm{T}}{\mathbf{o}}_i\right)}^2\right]\hspace{0.17em}=\hspace{0.17em}\mathrm{E}\left[o_{ki}^2\right]& =& \frac{1}{n},\hfill \end{array}$$

(31)

$$\begin{array}{ccc}\hfill \mathrm{E}\left[{\left({\mathbf{e}}_k^{\mathrm{T}}{\mathbf{o}}_i\right)}^2{\left({\mathbf{e}}_l^{\mathrm{T}}{\mathbf{o}}_i\right)}^2\right]\hspace{0.17em}=\hspace{0.17em}\mathrm{E}\left[o_{ki}^2{o}_{li}^2\right]& =& \left\{\begin{array}{cc}\frac{3}{n(n+2)}\hfill & k=l\hfill \\ \frac{1}{n(n+2)}\hfill & k\ne l\hfill \end{array}\right.,\hfill \end{array}$$

(32)

$$\begin{array}{ccc}\hfill \mathrm{E}\left[{\left({\mathbf{e}}_k^{\mathrm{T}}{\mathbf{o}}_i\right)}^2{\left({\mathbf{e}}_l^{\mathrm{T}}{\mathbf{o}}_j\right)}^2\right]\hspace{0.17em}=\hspace{0.17em}\mathrm{E}\left[o_{ki}^2{o}_{lj}^2\right]& =& \left\{\begin{array}{cc}\frac{1}{n(n+2)}\hfill & k=l,i\ne j\hfill \\ \frac{n+1}{(n-1)n(n+2)}\hfill & k\ne l,i\ne j\hfill \end{array}\right..\hfill \end{array}$$

(33)

Proof. 

For proving (31), we can assume without loss of generality $k=i=1$. Because ${\mathbf{o}}_1=\mathbf{u}$ is uniformly distributed on ${\mathcal{S}}_n$, the expected value of $f\left(\mathbf{u}\right)$ can be obtained by computing the mean value of $f\left(\mathbf{u}\right)$ over ${\mathcal{S}}_n$. We use Lemma 3 for expressing the integral over the surface of ${\mathcal{S}}_n$.

$$\begin{array}{ccc}\hfill \mathrm{E}\left[o_{ki}^2\right]& =& \mathrm{E}\left[o_{11}^2\right]\hspace{0.17em}=\hspace{0.17em}\mathrm{E}\left[u_1^2\right]\hspace{0.17em}=\hspace{0.17em}{S}_n^{-1}{\int }_{\parallel \mathbf{u}\parallel =1}{u}_1^2\mathrm{d}\sigma \hfill \end{array}$$

(34)

$$\begin{array}{ccc}& =& S_n^{-1}\frac{2\Gamma (3/2)\Gamma {(1/2)}^{n-1}}{\Gamma (1+n/2)}=\frac{1}{n}.\hfill \end{array}$$

(35)

Regarding (32) for $k=l$, we have

$$\begin{array}{ccc}\hfill \mathrm{E}\left[o_{ki}^2{o}_{li}^2\right]& =& \mathrm{E}\left[o_{ki}^4\right]\hspace{0.17em}=\hspace{0.17em}{S}_n^{-1}{\int }_{\parallel \mathbf{u}\parallel =1}{u}_1^4\mathrm{d}\sigma \hfill \end{array}$$

(36)

$$\begin{array}{ccc}& =& S_n^{-1}\frac{2\Gamma (5/2)\Gamma {(1/2)}^{n-1}}{\Gamma (2+n/2)}=\frac{3}{n(n+2)}.\hfill \end{array}$$

(37)

For $k\ne l$, we assume without loss of generality $k=1$, $l=2$. From Lemma 3, we have

$$\begin{array}{ccc}\hfill \mathrm{E}\left[o_{ki}^2{o}_{li}^2\right]& =& S_n^{-1}{\int }_{\parallel \mathbf{u}\parallel =1}{u}_1^2{u}_2^2\mathrm{d}\sigma \hfill \end{array}$$

(38)

$$\begin{array}{ccc}& =& S_n^{-1}\frac{2\Gamma {(3/2)}^2\Gamma {(1/2)}^{n-2}}{\Gamma (2+n/2)}=\frac{1}{n(n+2)}\hfill \end{array}$$

(39)

For (33) with $k=l$, we can show that it is identical to (32) with $k\ne l$. We have

$$\begin{array}{c}\hfill {\mathbf{e}}_k^{\mathrm{T}}{\mathbf{o}}_i={\mathbf{e}}_k^{\mathrm{T}}\mathbf{O}{\mathbf{e}}_i={\mathbf{e}}_i^{\mathrm{T}}{\mathbf{O}}^{\mathrm{T}}{\mathbf{e}}_k={\mathbf{e}}_i^{\mathrm{T}}{\left({\mathbf{O}}^{\mathrm{T}}\right)}_k\end{array}$$

(40)

where ${\left({\mathbf{O}}^{\mathrm{T}}\right)}_k$ is the k-th column of ${\mathbf{O}}^{\mathrm{T}}$. This implies

$$\begin{array}{c}\hfill {\left({\mathbf{e}}_k^{\mathrm{T}}{\mathbf{o}}_i\right)}^2{\left({\mathbf{e}}_k^{\mathrm{T}}{\mathbf{o}}_j\right)}^2={\left({\mathbf{e}}_i^{\mathrm{T}}{\left({\mathbf{O}}^{\mathrm{T}}\right)}_k\right)}^2{\left({\mathbf{e}}_j^{\mathrm{T}}{\left({\mathbf{O}}^{\mathrm{T}}\right)}_k\right)}^2\end{array}$$

(41)

To confirm (33) for $k=l$, we take into account that ${\mathbf{O}}^{\mathrm{T}}$ is also a random orthogonal matrix from the Haar measure, replace $\mathbf{O}$ with ${\mathbf{O}}^{\mathrm{T}}$ in (32), and rename i, k, and l to k, i, and j, respectively.

Finally, to prove (33) for $k\ne l$, we can assume without loss of generality $i=k=1$ and $j=l=2$. The cosine of the angle between ${\mathbf{o}}_1$ and ${\mathbf{e}}_2$ can be expressed as $o_{21}={\mathbf{e}}_2^{\mathrm{T}}{\mathbf{o}}_1=cos\varphi$. Random vector ${\mathbf{o}}_2$ is orthogonal to ${\mathbf{o}}_1$. Its realizations cover a unit sphere in an $n-1$-dimensional subspace $\mathcal{B}$ orthogonal to ${\mathbf{o}}_1$. Unit vectors ${\mathbf{b}}_1,\dots ,{\mathbf{b}}_{n-1}$ form an orthogonal basis for this subspace. Note that ${\mathbf{b}}_i^{\mathrm{T}}{\mathbf{o}}_1=0$. The conditional probability density distribution of ${\mathbf{o}}_2$ is uniform on the aforementioned unit sphere in $\mathcal{B}$. Vector ${\mathbf{o}}_2$ can be expressed as

$${\mathbf{o}}_2=\sum _{i=1}^{n-1}{\eta }_i{\mathbf{b}}_i,\sum _{i=1}^{n-1}{\eta }_i^2=1,$$

(42)

where vector $\left[{\eta }_1,\dots ,{\eta }_{n-1}\right]$ is uniformly distributed on ${\mathcal{S}}_{n-1}$. Without loss of generality we can choose vectors ${\mathbf{b}}_i$ in such manner that ${\mathbf{e}}_2={\mathbf{o}}_{1}cos\varphi +{\mathbf{b}}_{1}sin\varphi$, where $\varphi$ is the angle between ${\mathbf{e}}_2$ and ${\mathbf{o}}_1$. Now we have

$$o_{22}={\mathbf{e}}_2^{\mathrm{T}}{\mathbf{o}}_2=\sum _{i=1}^{n-1}{\left({\mathbf{o}}_{1}cos\varphi +{\mathbf{b}}_{1}sin\varphi \right)}^{\mathrm{T}}{\eta }_i{\mathbf{b}}_i={\eta }_{1}sin\varphi$$

(43)

and

$$o_{11}^2{o}_{22}^2={\left({\mathbf{e}}_1^{\mathrm{T}}{\mathbf{o}}_1\right)}^2{\left({\mathbf{e}}_2^{\mathrm{T}}{\mathbf{o}}_2\right)}^2=o_{11}^2{\eta }_1^2{sin}^2\varphi ={\eta }_1^2{o}_{11}^2\left(1-o_{21}^2\right).$$

(44)

Next, we can express

$$\mathrm{E}\left[o_{ki}^2{o}_{lj}^2\right]\hspace{0.17em}=\hspace{0.17em}\mathrm{E}\left[{\eta }_1^2\right]\mathrm{E}\left[o_{11}^2-o_{11}^2{o}_{21}^2\right],$$

(45)

where the first expected value refers to $\left[{\eta }_1,\dots ,{\eta }_{n-1}\right]\in {\mathcal{S}}_{n-1}$ and the second one to ${\mathbf{o}}_1\in {\mathcal{S}}_n$. Using Lemma 3 and previously proven (32), we arrive at

$$\mathrm{E}\left[{\eta }_1^2\right]\mathrm{E}\left[o_{11}^2-o_{11}^2{o}_{21}^2\right]\hspace{0.17em}=\hspace{0.17em}\frac{1}{n-1}\left(\frac{1}{n}-\frac{1}{n(n+2)}\right)\hspace{0.17em}=\hspace{0.17em}\frac{n+1}{(n-1)n(n+2)}$$

(46)

 □

4. Convergence of the Proposed Update

Multiplying vectors in a prototype set $\mathcal{D}=\{{\mathbf{d}}_1,\dots ,{\mathbf{d}}_{N+1}\}$ with a uniformly distributed random orthogonal matrix $\mathbf{O}$ results in a set of random vectors $\mathcal{V}=\{{\mathbf{v}}_1,\dots ,{\mathbf{v}}_{N+1}\}$ such that every $\mathbf{v}/\parallel \mathbf{v}\parallel$ is uniformly distributed on ${\mathcal{S}}_n$. The angles between the vectors in a realization of such a set are identical to the angles between the corresponding vectors from the prototype set.

Suppose one is interested in the expected amount of improvement resulting from one application of the update formula from Lemma 2. We assume that the $N+2$ points where the function value is computed comprise ${\mathbf{x}}_0$ and additional $N+1$ points generated using a random orthogonal matrix $\mathbf{O}$ and a prototype set of vectors $\{{\mathbf{d}}_1,\dots ,{\mathbf{d}}_{N+1}\}$ in the following manner.

$${\mathbf{x}}_i={\mathbf{x}}_0+\mathbf{O}{\mathbf{d}}_i={\mathbf{x}}_0+{\mathbf{v}}_i,i=1,\dots ,N+1.$$

(47)

First, we prove an auxiliary lemma.

Lemma 5.

Let $\mathbf{a}$, $\mathbf{b}$, and $\mathbf{O}$ denote two unit vectors with $cos\phi ={\mathbf{a}}^{\mathrm{T}}\mathbf{b}$ and a uniformly distributed orthogonal matrix, respectively. Let $\mathbf{u}=\mathbf{O}\mathbf{a}$ and $\mathbf{v}=\mathbf{O}\mathbf{b}$. Then,

$$\mathrm{E}\left[u_k^2{v}_l^2\right]\hspace{0.17em}=\hspace{0.17em}\left\{\begin{array}{cc}\frac{1+2{cos}^2\phi }{n(n+2)}\hfill & k=l\hfill \\ \frac{n+1-2{cos}^2\phi }{(n-1)n(n+2)}\hfill & k\ne l\hfill \end{array}\right.$$

(48)

Proof. 

Without loss of generality, the coordinate system can be rotated in such manner that $\mathbf{a}={\mathbf{e}}_1$ and $\mathbf{b}={\mathbf{e}}_{1}cos\phi +{\mathbf{e}}_{2}sin\phi$. Then, we have

$$\begin{array}{ccc}\hfill \mathbf{u}& =& {\mathbf{o}}_1,\hfill \end{array}$$

(49)

$$\begin{array}{ccc}\hfill \mathbf{v}& =& {\mathbf{o}}_{1}cos\phi +{\mathbf{o}}_{2}sin\phi .\hfill \end{array}$$

(50)

For $k=l$, we have

$$\begin{array}{ccc}\hfill \mathrm{E}\left[u_k^2{v}_k^2\right]& =& \mathrm{E}\left[o_{k1}^2{\left(o_{k1}cos\phi +o_{k2}sin\phi \right)}^2\right]\hfill \end{array}$$

(51)

$$\begin{array}{ccc}& =& \mathrm{E}\left[o_{k1}^4\right]{cos}^2\phi +\mathrm{E}\left[o_{k1}^2{o}_{k2}^2\right]{sin}^2\phi +2\mathrm{E}\left[o_{k1}^3{o}_{k2}\right]cos\phi sin\phi \hfill \end{array}$$

(52)

The last term vanishes because the integral of odd powers of $o_{ij}$ over ${\mathcal{S}}_n$ is zero. By invoking Lemma 4, we arrive at

$$\mathrm{E}\left[u_k^2{v}_k^2\right]\hspace{0.17em}=\hspace{0.17em}\frac{3{cos}^2\phi }{n(n+2)}+\frac{{sin}^2\phi }{n(n+2)}=\frac{1+2{cos}^2\phi }{n(n+2)}$$

(53)

For $k\ne l$,

$$\begin{array}{ccc}\hfill \mathrm{E}\left[u_k^2{v}_l^2\right]& =& \mathrm{E}\left[o_{k1}^2{\left(o_{l1}cos\phi +o_{l2}sin\phi \right)}^2\right]\hfill \\ & =& \mathrm{E}\left[o_{k1}^2{o}_{l1}^2\right]{cos}^2\phi +\mathrm{E}\left[o_{k1}^2{o}_{l2}^2\right]{sin}^2\phi \hfill \end{array}$$

(54)

$$\begin{array}{ccc}& +& 2\mathrm{E}\left[o_{k1}^2{o}_{l1}{o}_{l2}\right]cos\phi sin\phi \hfill \end{array}$$

(55)

The last term vanishes due to odd powers of $o_{ij}$. Together with Lemma 4, we have

$$\mathrm{E}\left[u_k^2{v}_l^2\right]\hspace{0.17em}=\hspace{0.17em}\frac{{cos}^2\phi }{n(n+2)}+\frac{(n+1){sin}^2\phi }{(n-1)n(n+2)}=\frac{n+1-2{cos}^2\phi }{(n-1)n(n+2)}.$$

(56)

 □

Lemma 6.

Let $\{{\mathbf{d}}_1,\dots ,{\mathbf{d}}_{N+1}\}$ be a prototype set of vectors satisfying ${\sum }_{i=1}^{N+1}{\alpha }_i{\mathbf{d}}_i=0$, where all ${\alpha }_i\ge 0$ and at least one ${\alpha }_i\ne 0$. Let $\mathbf{O}$ be a uniformly distributed random orthogonal matrix, and let ${\mathbf{v}}_i=\mathbf{O}{\mathbf{d}}_i$ with $\parallel {\mathbf{d}}_i\parallel \parallel {\mathbf{d}}_j\parallel cos{\phi }_{ij}={\mathbf{d}}_i^{\mathrm{T}}{\mathbf{d}}_j={\mathbf{v}}_i^{\mathrm{T}}{\mathbf{v}}_j$. Then, the MFN update formula from Lemma 2 involving $N+2$ points (${\mathbf{x}}_0$ and the additional $N+1$ points constructed according to (47)) satisfies

$$\mathrm{E}\left[\parallel {\mathbf{B}}_{+}{-\mathbf{B}\parallel }_{\mathrm{F}}^2\right]\hspace{0.17em}=\hspace{0.17em}\mathrm{E}\left[{\beta }^2{\parallel \mathbf{A}\parallel }_{\mathrm{F}}^2\right]\hspace{0.17em}=\hspace{0.17em}({\gamma }_1-{\gamma }_2){\parallel \mathbf{B}-\mathbf{H}\parallel }_{\mathrm{F}}^2+{\gamma }_2\mathrm{tr}{\left(\mathbf{B}-\mathbf{H}\right)}^2$$

(57)

where

$${\gamma }_1=\frac{\mu +2}{n(n+2)}$$

(58)

$${\gamma }_2=\frac{(n+1)\mu -2}{(n-1)n(n+2)}$$

(59)

$$\begin{array}{ccc}\hfill \mu & =& {\left(\frac{{\sum }_{i=1}^{N+1}{\alpha }_i{\parallel {\mathbf{d}}_i\parallel }^2}{{2\parallel \mathbf{A}\parallel }_{\mathrm{F}}}\right)}^2=\frac{{\left({\sum }_{i=1}^{N+1}{\alpha }_i{\parallel {\mathbf{d}}_i\parallel }^2\right)}^2}{{\sum }_{i=1}^{N+1}{\sum }_{j=1}^{N+1}{\alpha }_i{\alpha }_j\parallel {\mathbf{d}}_i{\parallel }^2{\parallel {\mathbf{d}}_j\parallel }^2{cos}^2{\phi }_{ij}}\hfill \end{array}$$

(60)

Proof. 

By repeating the reasoning in the proof of Lemma 2 on (18), we obtain

$$\mathbf{A}:\mathbf{H}=\sum _{i=1}^{N+1}{\alpha }_i(q_i-q_0)$$

(61)

which yields together with the expression for $\beta$ from Lemma 2

$$\beta =\frac{\left(\mathbf{H}-\mathbf{B}\right)\hspace{0.17em}:\hspace{0.17em}\mathbf{A}}{\mathbf{A}:\mathbf{A}}=\frac{{\sum }_{i=1}^{N+1}{\alpha }_i{\mathbf{v}}_i^{\mathrm{T}}\left(\mathbf{H}-\mathbf{B}\right){\mathbf{v}}_i}{{2\parallel \mathbf{A}\parallel }_{\mathrm{F}}^2}.$$

(62)

Now we can express

$${\beta }^2{\parallel \mathbf{A}\parallel }_{\mathrm{F}}^2=\frac{{\left({\sum }_{i=1}^{N+1}{\alpha }_i{\mathbf{v}}_i^{\mathrm{T}}\left(\mathbf{B}-\mathbf{H}\right){\mathbf{v}}_i\right)}^2}{{4\parallel \mathbf{A}\parallel }_{\mathrm{F}}^2}.$$

(63)

Because vectors ${\mathbf{v}}_i$ are uniformly distributed on the unit sphere, we can rotate the coordinate system without affecting $\mathrm{E}\left[{\beta }^2{\parallel \mathbf{A}\parallel }_{\mathrm{F}}^2\right]$ so that $\mathbf{B}-\mathbf{H}$ is diagonalized.

$$\mathrm{E}\left[{\beta }^2{\parallel \mathbf{A}\parallel }_{\mathrm{F}}^2\right]\hspace{0.17em}=\hspace{0.17em}\mathrm{E}\left[\frac{{\left({\sum }_{i=1}^{N+1}{\alpha }_i{\mathbf{v}}_i^{\mathrm{T}}\mathbf{D}{\mathbf{v}}_i\right)}^2}{{4\parallel \mathbf{A}\parallel }_{\mathrm{F}}^2}\right].$$

(64)

Let $v_{ik}$ denote the k-th component of vector ${\mathbf{v}}_i$ and ${\lambda }_k$ the k-th eigenvalue of $\mathbf{B}-\mathbf{H}$ (k-th diagonal element of $\mathbf{D}$).

$$\mathrm{E}\left[{\beta }^2{\parallel \mathbf{A}\parallel }_{\mathrm{F}}^2\right]\hspace{0.17em}=\hspace{0.17em}\mathrm{E}\left[\frac{{\sum }_{i=1}^{N+1}{\sum }_{j=1}^{N+1}{\alpha }_i{\alpha }_j{\sum }_{k=1}^n{\sum }_{l=1}^n{v}_{ik}^2{v}_{jl}^2{\lambda }_k{\lambda }_l}{{4\parallel \mathbf{A}\parallel }_{\mathrm{F}}^2}\right].$$

(65)

We can rewrite (65) as

$$\mathrm{E}\left[{\beta }^2{\parallel \mathbf{A}\parallel }_{\mathrm{F}}^2\right]\hspace{0.17em}=\hspace{0.17em}\frac{{\sum }_{k=1}^n{\sum }_{i=1}^{N+1}{\sum }_{j=1}^{N+1}{\alpha }_i{\alpha }_j{\mathrm{E}}_{ikjk}{\lambda }_k^2+{\sum }_{k\ne l}{\sum }_{i=1}^{N+1}{\sum }_{j=1}^{N+1}{\alpha }_i{\alpha }_j{\mathrm{E}}_{ikjl}{\lambda }_k{\lambda }_l}{{4\parallel \mathbf{A}\parallel }_{\mathrm{F}}^2}$$

(66)

The expected value of $v_{ik}^2{v}_{jl}^2$ depends on $cos{\phi }_{ij}={\mathbf{v}}_i^{\mathrm{T}}{\mathbf{v}}_j$. From Lemma 5, we have

$${\mathrm{E}}_{ikjl}=\mathrm{E}\left[v_{ik}^2{v}_{jl}^2\right]\hspace{0.17em}=\hspace{0.17em}\left\{\begin{array}{cc}\frac{1+2{cos}^2{\phi }_{ij}}{n(n+2)}\parallel {\mathbf{v}}_i{\parallel }^2{\parallel {\mathbf{v}}_j\parallel }^2\hfill & k=l\hfill \\ \frac{n+1-2{cos}^2{\phi }_{ij}}{(n-1)n(n+2)}\parallel {\mathbf{v}}_i{\parallel }^2{\parallel {\mathbf{v}}_j\parallel }^2\hfill & k\ne l\hfill \end{array}\right.$$

(67)

Because the eigenvalues of $\mathbf{D}$ are the same as the eigenvalues of $\mathbf{B}-\mathbf{H}$, we have

$$\begin{array}{ccc}\hfill {\parallel \mathbf{B}-\mathbf{H}\parallel }_{\mathrm{F}}^2& =& {\parallel \mathbf{D}\parallel }_{\mathrm{F}}^2=\sum _{k=1}^n{\lambda }_k^2,\hfill \end{array}$$

(68)

$$\begin{array}{ccc}\hfill \mathrm{tr}{\left(\mathbf{B}-\mathbf{H}\right)}^2& =& \mathrm{tr}{\left(\mathbf{D}\right)}^2={\left(\sum _{k=1}^n{\lambda }_k\right)}^2=\sum _{k=1}^n\sum _{l=1}^n{\lambda }_k{\lambda }_l\hfill \end{array}$$

(69)

and

$$\begin{array}{ccc}\hfill ({\gamma }_1-{\gamma }_2){\parallel \mathbf{B}-\mathbf{H}\parallel }_{\mathrm{F}}^2+{\gamma }_2\mathrm{tr}{\left(\mathbf{B}-\mathbf{H}\right)}^2& =& ({\gamma }_1-{\gamma }_2)\sum _{k=1}^n{\lambda }_k^2+{\gamma }_2\sum _{k=1}^n\sum _{l=1}^n{\lambda }_k{\lambda }_l\hfill \\ \hfill & =& {\gamma }_1\sum _{k=1}^n{\lambda }_k^2+{\gamma }_2\sum _{k\ne l}{\lambda }_k{\lambda }_l\hfill \end{array}$$

(70)

Note that $\parallel {\mathbf{v}}_i\parallel \hspace{0.17em}=\hspace{0.17em}\parallel {\mathbf{d}}_i\parallel$. Taking into account (66), (67), and (70) yields

$$\begin{array}{ccc}\hfill {\gamma }_1& =& \frac{{\sum }_{i=1}^{N+1}{\sum }_{j=1}^{N+1}{\alpha }_i{\alpha }_j\parallel {\mathbf{d}}_i{\parallel }^2{\parallel {\mathbf{d}}_j\parallel }^2\left(1+2{cos}^2{\phi }_{ij}\right)}{{4n(n+2)\parallel \mathbf{A}\parallel }_{\mathrm{F}}^2},\hfill \end{array}$$

(71)

$$\begin{array}{ccc}\hfill {\gamma }_2& =& \frac{{\sum }_{i=1}^{N+1}{\sum }_{j=1}^{N+1}{\alpha }_i{\alpha }_j\parallel {\mathbf{d}}_i{\parallel }^2{\parallel {\mathbf{d}}_j\parallel }^2\left(n+1-2{cos}^2{\phi }_{ij}\right)}{{4(n-1)n(n+2)\parallel \mathbf{A}\parallel }_{\mathrm{F}}^2}.\hfill \end{array}$$

(72)

The Frobenius norm of $\mathbf{A}$ can be expressed as

$$\begin{array}{ccc}{\parallel \mathbf{A}\parallel }_{\mathrm{F}}^2\hfill & =\hfill & \mathrm{tr}\left({\mathbf{A}}^{\mathrm{T}}\mathbf{A}\right)\hspace{0.17em}=\hspace{0.17em}\mathrm{tr}\left({\left(\frac{1}{2}{\sum }_{i=1}^{N+1}{\alpha }_i{\mathbf{v}}_i{\mathbf{v}}_i^{\mathrm{T}}\right)}^{\mathrm{T}}\left(\frac{1}{2}{\sum }_{j=1}^{N+1}{\alpha }_j{\mathbf{v}}_j{\mathbf{v}}_j^{\mathrm{T}}\right)\right)\hfill \\ \hfill & =\hfill & \frac{1}{4}{\sum }_{i=1}^{N+1}{\sum }_{j=1}^{N+1}{\alpha }_i{\alpha }_j\mathrm{tr}\left({\mathbf{v}}_i{\mathbf{v}}_i^{\mathrm{T}}{\mathbf{v}}_j{\mathbf{v}}_j^{\mathrm{T}}\right)\hspace{0.17em}=\hspace{0.17em}\frac{1}{4}{\sum }_{i=1}^{N+1}{\sum }_{j=1}^{N+1}{\alpha }_i{\alpha }_j{\left({\mathbf{v}}_i^{\mathrm{T}}{\mathbf{v}}_j\right)}^2\hfill \\ \hfill & =\hfill & \frac{1}{4}{\sum }_{i=1}^{N+1}{\sum }_{j=1}^{N+1}{\alpha }_i{\alpha }_j\parallel {\mathbf{d}}_i{\parallel }^2{\parallel {\mathbf{d}}_j\parallel }^2{cos}^2{\phi }_{ij}\hfill \end{array}$$

(73)

By substituting (73) in (71) and (72), we arrive at

$$\begin{array}{ccc}\hfill {\gamma }_1& =& \frac{{8\parallel \mathbf{A}\parallel }_{\mathrm{F}}^2+{\sum }_{i=1}^{N+1}{\sum }_{j=1}^{N+1}{\alpha }_i{\alpha }_j\parallel {\mathbf{d}}_i{\parallel }^2{\parallel {\mathbf{d}}_j\parallel }^2}{{4n(n+2)\parallel \mathbf{A}\parallel }_{\mathrm{F}}^2},\hfill \end{array}$$

(74)

$$\begin{array}{ccc}\hfill {\gamma }_2& =& \frac{(n+1){\sum }_{i=1}^{N+1}{\sum }_{j=1}^{N+1}{\alpha }_i{\alpha }_j\parallel {\mathbf{d}}_i{\parallel }^2\parallel {\mathbf{d}}_j{\parallel }^2-8{\parallel \mathbf{A}\parallel }_{\mathrm{F}}^2}{{4(n-1)n(n+2)\parallel \mathbf{A}\parallel }_{\mathrm{F}}^2}.\hfill \end{array}$$

(75)

We also have

$$\sum _{i=1}^{N+1}\sum _{j=1}^{N+1}{\alpha }_i{\alpha }_j\parallel {\mathbf{d}}_i{\parallel }^2{\parallel {\mathbf{d}}_j\parallel }^2={\left(\sum _{i=1}^{N+1}{\alpha }_i{\parallel {\mathbf{d}}_i\parallel }^2\right)}^2$$

(76)

Substituting (76) into (74) and (75) concludes the proof. □

Theorem 1.

Let ${\gamma }_1$, ${\gamma }_2$, and μ be defined as in Lemma 6. Then,

$$\frac{\mathrm{E}\left[\parallel {\mathbf{B}}_{+}{-\mathbf{H}\parallel }_{\mathrm{F}}^2\right]}{{\parallel \mathbf{B}-\mathbf{H}\parallel }_{\mathrm{F}}^2}\le \left\{\begin{array}{cc}1-\frac{2(n-\mu )}{(n-1)n(n+2)}\hfill & \mu \ge \frac{2}{n+1}\hfill \\ 1-\frac{\mu }{n}\hfill & \mu <\frac{2}{n+1}\hfill \end{array}\right..$$

(77)

Proof. 

We start with the following identity.

$${\mathbf{B}}_{+}-\mathbf{H}+\mathbf{B}-{\mathbf{B}}_{+}=\mathbf{B}-\mathbf{H}.$$

(78)

Computing the Frobenius norm on both sides and considering $({\mathbf{B}}_{+}-\mathbf{B})\perp ({\mathbf{B}}_{+}-\mathbf{H})$ results in

$$\parallel {\mathbf{B}}_{+}{-\mathbf{H}\parallel }_{\mathrm{F}}^2\hspace{0.17em}+\hspace{0.17em}\parallel {\mathbf{B}}_{+}{-\mathbf{B}\parallel }_{\mathrm{F}}^2={\parallel \mathbf{B}-\mathbf{H}\parallel }_{\mathrm{F}}^2.$$

(79)

Taking into account (15) results in

$$\parallel {\mathbf{B}}_{+}{-\mathbf{H}\parallel }_{\mathrm{F}}^2={\parallel \mathbf{B}-\mathbf{H}\parallel }_{\mathrm{F}}^2-{\beta }^2{\parallel \mathbf{A}\parallel }_{\mathrm{F}}^2.$$

(80)

After Lemma 6 is applied, we have

$$\frac{\mathrm{E}\left[\parallel {\mathbf{B}}_{+}{-\mathbf{H}\parallel }_{\mathrm{F}}^2\right]}{{\parallel \mathbf{B}-\mathbf{H}\parallel }_{\mathrm{F}}^2}=1-\left({\gamma }_1-{\gamma }_2+{\gamma }_2\frac{\mathrm{tr}{\left(\mathbf{B}-\mathbf{H}\right)}^2}{{\parallel \mathbf{B}-\mathbf{H}\parallel }_{\mathrm{F}}^2}\right)$$

(81)

By definition, $\mu \ge 0$ and ${\gamma }_1\ge 0$. For ${\gamma }_2\ge 0$, we must have $\mu \ge 2/(n+1)$. By considering $\mathrm{tr}{(\mathbf{B}-\mathbf{H})}^2/{\parallel \mathbf{B}-\mathbf{H}\parallel }_{\mathrm{F}}^2\ge 0$, we arrive at

$$\frac{\mathrm{E}\left[\parallel {\mathbf{B}}_{+}{-\mathbf{H}\parallel }_{\mathrm{F}}^2\right]}{{\parallel \mathbf{B}-\mathbf{H}\parallel }_{\mathrm{F}}^2}\le 1-\left({\gamma }_1-{\gamma }_2\right)\hspace{0.17em}=\hspace{0.17em}1-\frac{2(n-\mu )}{(n-1)n(n+2)}.$$

(82)

For ${\gamma }_2<0$, we must have $\mu <2/(n+1)$. Invoking Lemma A1 yields

$$\frac{\mathrm{E}\left[\parallel {\mathbf{B}}_{+}{-\mathbf{H}\parallel }_{\mathrm{F}}^2\right]}{{\parallel \mathbf{B}-\mathbf{H}\parallel }_{\mathrm{F}}^2}\le 1-\left({\gamma }_1-{\gamma }_2+n{\gamma }_2\right)\hspace{0.17em}=\hspace{0.17em}1-\left({\gamma }_1+(n-1){\gamma }_2\right)\hspace{0.17em}=\hspace{0.17em}1-\frac{\mu }{n}.$$

(83)

 □

From Theorem 1, several results can be derived. First, we will assume the prototype set is a regular N-simplex (i.e., comprises $N+1$ vectors positively spanning an N-dimensional subspace). This case is interesting because the update formula in [9] is obtained for $N=1$. We are going to show that our estimate of the expected Hessian improvement is identical to the one published in [9]. This update formula (with $N=1$) was used in an optimization algorithm published in [10].

Next, we are going to show that using a regular n-simplex as the prototype set is a bad choice. According to Theorem 1, no improvement of the Hessian is guaranteed. Even worse, we show that improvement occurs only at the first application of the update formula.

Finally, we will analyze the case where the prototype set is what we refer to as an augmented set of N orthonormal vectors. Such a prototype set with $N=n$ was used in the optimization algorithm published in [14].

Corollary 1.

Let $\mathcal{D}$ be a regular N-simplex ($N\le n$). Then,

$$\frac{\mathrm{E}\left[\parallel {\mathbf{B}}_{+}{-\mathbf{H}\parallel }_{\mathrm{F}}^2\right]}{{\parallel \mathbf{B}-\mathbf{H}\parallel }_{\mathrm{F}}^2}\le 1-\frac{2(n-N)}{(n-1)n(n+2)}$$

(84)

Proof. 

For all $i\ne j$, we have $cos{\phi }_{ij}=-N^{-1}$ and $\parallel {\mathbf{d}}_i\parallel =1$. Because the sum of all vectors in a regular N-simplex is $\mathbf{0}$, we conclude ${\alpha }_i=1$ and

$$\begin{array}{c}\hfill \mu =\frac{{\left({\sum }_{i=1}^{N+1}{\alpha }_j\right)}^2}{{\sum }_{i=1}^{N+1}{\sum }_{j=1}^{N+1}{\alpha }_i{\alpha }_j{cos}^2{\phi }_{ij}}=\frac{{(N+1)}^2}{(N+1)·1+N(N+1)·N^{-2}}=N\end{array}$$

(85)

Because $2/(n+1)\le 1$ for all $n\ge 1$, we have $\mu \ge 2/(n+1)$, and the result follows from Theorem 1. □

Corollary 1 implies that the most efficient approach to MFN updating with a regular simplex in the role of the prototype set of unit vectors is to use a regular 1-simplex (three collinear points).

Corollary 2.

For $N=1$ and ${\mathbf{d}}_1=-{\mathbf{d}}_2$, set $\mathcal{D}$ is a regular 1-simplex and

$$\frac{\mathrm{E}\left[\parallel {\mathbf{B}}_{+}{-\mathbf{H}\parallel }_{\mathrm{F}}^2\right]}{{\parallel \mathbf{B}-\mathbf{H}\parallel }_{\mathrm{F}}^2}\le 1-\frac{2}{n(n+2)}.$$

(86)

This result was proven in [9] with a less general approach. Here, we obtain it as a special case of Corollary 1 for $N=1$.

According to Corollary 1, there is no guaranteed improvement of ${\parallel \mathbf{B}-\mathbf{H}\parallel }_{\mathrm{F}}$ if a regular n-simplex ($N=n$) is used in the update process. In fact, the situation is even worse as we show in the following Lemma.

Lemma 7.

If $\mathcal{D}$ is a regular n-simplex ($N=n$), then the MFN update from Lemma 2 improves the Hessian approximation only in its first application.

Proof. 

From Lemma 2, we can see that

$${\mathbf{B}}_{+}=\mathbf{B}+\beta \mathbf{A}$$

(87)

where (see (62))

$$\beta =\frac{{\sum }_{i=1}^{N+1}{\alpha }_i{\mathbf{v}}_i^{\mathrm{T}}\left(\mathbf{H}-\mathbf{B}\right){\mathbf{v}}_i}{{2\parallel \mathbf{A}\parallel }_{\mathrm{F}}^2}.$$

(88)

For a regular simplex, ${\alpha }_i=1$ and $\parallel {\mathbf{d}}_i\parallel =1$. The Frobenius norm of $\mathbf{A}$ is

$${\parallel \mathbf{A}\parallel }_{\mathrm{F}}^2=\frac{1}{4}\sum _{i=1}^{N+1}\sum _{j=1}^{N+1}{cos}^2{\phi }_{ij}=\frac{1}{4}·\left(1·(n+1)+\frac{1}{n^2}·n(n+1)\right)\hspace{0.17em}=\hspace{0.17em}\frac{{(n+1)}^2}{4n}$$

(89)

Due to Lemma A3 (see Appendix A for proof), we have

$$\sum _{i=1}^{N+1}{\alpha }_i{\mathbf{v}}_i^{\mathrm{T}}\left(\mathbf{H}-\mathbf{B}\right){\mathbf{v}}_i=\frac{n+1}{n}\mathrm{tr}(\mathbf{H}-\mathbf{B})$$

(90)

and

$$\beta =\frac{2}{n+1}\mathrm{tr}(\mathbf{H}-\mathbf{B})$$

(91)

From definition of $\mathbf{A}$, we obtain

$$\mathrm{tr}\left(\mathbf{A}\right)=\frac{1}{2}\sum _{i=1}^{n+1}{\alpha }_i\mathrm{tr}\left({\mathbf{v}}_i{\mathbf{v}}_i^{\mathrm{T}}\right)\hspace{0.17em}=\hspace{0.17em}\frac{1}{2}\sum _{i=1}^{n+1}{\alpha }_i{\parallel {\mathbf{v}}_i\parallel }^2=\frac{n+1}{2}$$

(92)

From ${\mathbf{B}}_{+}-\mathbf{H}=\mathbf{B}-\mathbf{H}+\beta \mathbf{A}$, we can express

$$\mathrm{tr}({\mathbf{B}}_{+}-\mathbf{H})=\mathrm{tr}(\mathbf{B}-\mathbf{H})+\beta \mathrm{tr}\left(\mathbf{A}\right)=0.$$

(93)

Let ${\mathbf{B}}_{++}$ denote the approximate Hessian after the second application of the update formula.

$${\mathbf{B}}_{++}={\mathbf{B}}_{+}+{\beta }_{+}{\mathbf{A}}_{+}$$

(94)

Because ${\beta }_{+}=2\mathrm{tr}(\mathbf{H}-{\mathbf{B}}_{+})/(n+1)=0$, we have ${\mathbf{B}}_{++}={\mathbf{B}}_{+}$, and the proof is complete. □

Intuition can mislead one into considering the regular n-simplex as the best choice for positioning $n+1$ points around an origin ${\mathbf{x}}_0$ when computing an MFN update based on Lemma 2. Lemma 7 shows the exact opposite—a regular n-simplex is the worst choice because the update formula does not improve the Hessian approximation in its second and all subsequent applications.

Definition 2.

An augmented set of $1\le N\le n$ orthonormal vectors is a set comprising N mutually orthogonal unit vectors ${\mathbf{e}}_1,\dots ,{\mathbf{e}}_N$ and their normalized negative sum $-N^{-1/2}({\mathbf{e}}_1+\dots +{\mathbf{e}}_N)$.

Note that an augmented set of $N=1$ orthonormal vectors is equivalent to a regular 1-simplex. Now, we have $\parallel {\mathbf{d}}_i\parallel \hspace{0.17em}=\hspace{0.17em}1$ and $cos{\phi }_{ii}=1$. For $i\ne j$, we have $cos{\phi }_{ij}=0$ except for $i=n+1$ or $j=n+1$ when $cos{\phi }_{ij}=-N^{-1/2}$. Because ${\mathbf{d}}_{N+1}$ is the normalized negative sum of the first N vectors, ${\alpha }_1=\dots ={\alpha }_n=1$ and ${\alpha }_{n+1}=N^{1/2}$.

Corollary 3.

If the prototype set of unit vectors is an augmented set of N orthonormal vectors, then

$$\frac{\mathrm{E}\left[\parallel {\mathbf{B}}_{+}{-\mathbf{H}\parallel }_{\mathrm{F}}^2\right]}{{\parallel \mathbf{B}-\mathbf{H}\parallel }_{\mathrm{F}}^2}\le 1-\frac{2n-N-N^{1/2}}{(n-1)n(n+2)}.$$

(95)

Proof. 

$$\begin{array}{ccc}\hfill \mu =\frac{{\left({\sum }_{j=1}^{N+1}{\alpha }_i\right)}^2}{{\sum }_{i=1}^{N+1}{\sum }_{j=1}^{N+1}{\alpha }_i{\alpha }_j{cos}^2{\phi }_{ij}}& =& \frac{{\left(N·1+N^{1/2}\right)}^2}{N·1·1·1^2+1·N^{1/2}·N^{1/2}·1^2+2N·1·N^{1/2}·N^{-1}}\hfill \\ & =& \frac{N{(N^{1/2}+1)}^2}{2N^{1/2}(N^{1/2}+1)}=\frac{N+N^{1/2}}{2}\hfill \end{array}$$

(96)

Because $2/(n+1)\le 1$ for all $n\ge 1$, we conclude $\mu \ge 2/(n+1)$, and the result follows from Theorem 1. □

A special case of Corollary 1 is the following result.

Corollary 4.

If the prototype set of unit vectors is an augmented set of $N=n$ orthonormal vectors, then

$$\frac{\mathrm{E}\left[\parallel {\mathbf{B}}_{+}{-\mathbf{H}\parallel }_{\mathrm{F}}^2\right]}{{\parallel \mathbf{B}-\mathbf{H}\parallel }_{\mathrm{F}}^2}\le 1-\frac{1}{(n+n^{1/2})(n+2)}.$$

(97)

Corollaries 2 and 4 indicate that for an augmented set of $N=n$ orthonormal vectors (used in [12]), the expected improvement of the approximate Hessian approaches half of the improvement obtained using a regular 1-simplex (introduced in [9]) when n approaches infinity.

Corollaries 1–4 indicate that the update formula yields a greater improvement of the approximate Hessian when the prototype set of vectors exhibits more directionality, in the sense that the vectors are confined to an $N<n$ dimensional subspace of the search space. Lower values of N result in faster convergence.

5. Example

We illustrate the proposed update with a simple example. The sequence of uniformly distributed orthogonal matrices is generated as in [14]. Three prototype sets are examined—regular 1-dimensional and $(n-1)$-dimensional simplex and the augmented set of n orthonormal vectors. The true Hessian $\mathbf{H}$ is chosen randomly and the initial Hessian approximation is set to $\mathbf{B}=\mathbf{0}$. The progress of the update is measured by the normalized Frobenius distance between $\mathbf{H}$ and $\mathbf{B}$.

Figure 1 depicts the progress of the proposed update with various prototype sets for $n=5$ and $n=10$. It is clearly visible that the convergence of the update is linear and depends on the choice of the prototype set. The convergence rate of the update using an augmented set of n orthonormal vectors is approximately half of the convergence rate exhibited by the update using a regular 1-simplex. It can also be seen that the bound on the amount of progress obtained from one update (Theorem 1) is fairly conservative. The actual progress of the update is much better in practice.

6. Discussion

The convergence of a Hessian update formula that requires only function values for computing the update was analyzed. The update formula is based on the formula published in [8] that generally requires the function values at $m\ge n+2$ points. The proposed update is based on the case where $m=n+2$. An additional requirement is introduced, namely that the $m-1$ vectors from the central point to the remaining $m-1$ points must positively span a $m-2$ dimensional subspace of ${\mathbb{R}}^n$. This requirement extends the usability of the proposed update to sets of points with $3\le m\le n+2$ members. The set of m points used by the update is generated by adding $m-1$ vectors to a central point in the set. The vectors are obtained by applying a random orthogonal transformation to a prototype set of vectors that spans a $m-2$ dimensional subspace of ${\mathbb{R}}^n$.

A lower bound on the expected improvement of the Hessian approximation was derived (Theorem 1). Up to now, no such result was published for the update from [8] and $m=n+2$. The obtained result was applied to several different prototype sets. The general result obtained for the case when the prototype set is a regular $m-2$-dimensional simplex (Corollary 1) shows that the expected improvement of the Hessian approximation is greatest for $m=3$ (i.e., 1-dimensional regular simplex) and decreases as the dimensionality of the simplex increases. The special case when $m=3$ (1-dimensional regular simplex) corresponds to the update from [9]. The lower bound on the expected improvement obtained with our general result (Corollary 2) matches the one that was published in [9]. For the n-dimensional regular simplex, our result indicates that the lower bound on expected improvement of the Hessian approximation is 0. Furthermore, it was shown that the Hessian approximation is possibly improved only by the first application of the proposed update formula (Lemma 7). Therefore, the use of the n-dimensional regular simplex in the role of the prototype set is a bad choice.

Next, the expected improvement of the approximate Hessian for a prototype set comprising $N\le n$ orthogonal vectors and their normalized negative sum was derived. Such a prototype set with $N=n$ was used in the optimization algorithm published in [12]. It was shown that using this kind of prototype set does guarantee a positive lower bound on the expected improvement of the Hessian approximation (Corollary 4). The general result (Corollary 3), however, again indicates that using a prototype set of lower dimensionality results in faster convergence. The result for $N=1$ (two collinear vectors in the role of the prototype set) is the same as the one obtained for the update from [9].

Finally, the results were illustrated by running the proposed update on a quadratic function with a randomly chosen Hessian for several choices of the prototype set. The observed progress was compared to the lower bound predicted by Theorem 1. The results indicate that the lower bound is quite pessimistic, and that the actual progress is faster. The observed performance was closest to the predicted lower bound for the update formula from [9].