Properties and Application of Incomplete Orthogonalization in the Directions of Gradient Difference in Optimization Methods

Abstract

This paper considers the problem of unconstrained minimization of smooth functions. Despite the high efficiency of quasi-Newton methods such as BFGS, their performance degrades in ill-conditioned problems with unstable or rapidly varying Hessians—for example, in functions with curved ravine structures. This necessitates alternative approaches that rely not on second-derivative approximations but on the topological properties of level surfaces. As a new methodological framework, we propose using a procedure of incomplete orthogonalization in the directions of gradient differences, implemented through the iterative least-squares method (ILSM). Two new methods are constructed based on this approach: a gradient method with the ILSM metric (HY_g) and a modification of the Hestenes–Stiefel conjugate gradient method with the same metric (HY_XS). Both methods are shown to have linear convergence on strongly convex functions and finite convergence on quadratic functions. A numerical experiment was conducted on a set of test functions. The results show that the proposed methods significantly outperform BFGS (2 times for HY_g and 3.5 times for HY_XS in terms of iterations number) when solving ill-posed problems with varying Hessians or complex level topologies, while providing comparable or better performance even in high-dimensional problems. This confirms the potential of using topology-based metrics alongside classical quasi-Newton strategies.

1. Introduction

We consider the problem of minimizing a differentiable function f(x) on a finite-dimensional Euclidean space. This problem can be stated as

f(x) → min, x ∈ Rⁿ.

Well-known methods [1,2] were developed to solve an unconstrained minimization problem, including the gradient method, which is based on the idea of function local linear approximation. Conjugate gradient methods (CGM) generate search directions that are consistent with the geometry of the minimized function. In practice, the CGM shows faster convergence rates than gradient descent algorithms, so CGM is widely used in machine learning.

Quasi-Newton methods (QNM) are based on the idea of using a matrix of second derivatives reconstructed from the gradients of a function. The commonly used matrix update formula for quasi-Newton methods is BFGS [3,4,5,6]. Two quasi-Newton algorithms for solving unconstrained optimization problems based on two modified secant relations to achieve reliable approximations of the Hessian matrices of the objective function were presented in [7].

Ultra-high dimensions and strong nonlinearity can lead to extremely complex optimization landscapes, for which gradient-based optimization solvers will perform poorly or fail easily [8]. Without denying the merits of the gradient descent method, it must be said that it turns out to be very slow when moving along a ravine, and as the number of variables of the objective function increases, such behavior of the method becomes typical. Performance of QNM degrades in ill-conditioned problems with unstable or rapidly varying Hessians—for example, in functions with curved ravine structures.

Relaxation subgradient methods (RSM) have been widely used in optimization practice for many years and have found practical application in such areas as signal and image processing [9,10], classification [11], network design [12], maintenance routing [13], dynamic process modeling [14], and many others.

In our earlier studies, it turned out that the problem of finding the direction of descent in the RSM can be reduced to the problem of solving a system of inequalities on subgradient sets and mathematically formulated as a solution to the problem of minimizing some quality functional. In this case, the properties of the learning algorithm determined the convergence rate of the minimization method.

In relaxation processes of ε-subgradient type, successive approximations are constructed as follows:

$$x_{k+1}=x_k-{\gamma }_k{s}_k, {\gamma }_k=\mathit{arg} \underset{\gamma }{min}f(x_k-\gamma s_k), k=0,1,2 \dots .$$

(1)

Here, k is the iteration number, γ_k is the stepsize, and descent direction s_k is selected from the set ${S(\partial }_{\epsilon }f(x_i))$ [1,15,16,17,18,19], ${\partial }_{\epsilon }f(x_i)$ is ε-subgradient set at a point x_k, and $S(G)=\{s\in R^n|\underset{g\in G}{min}(s,g)>0\},$ $G\subset R^n$ is a set of feasible directions, g are gradients.

Denote by $\partial f(x)\equiv \partial f_{\epsilon =0}(x)$ a subgradient set at a point x. If set S(G) is not empty, then any vector $s\in S\left(G\right)$ is a solution to the set of inequalities:

$$(s,g)>0,\hspace{0.33em}\forall g\in G,$$

(2)

in other words, it sets the normal of the dividing plane of the origin and the set G. Here, (s, g) is a dot product of vectors. One of the solutions to (2) is a vector of minimal length from G denoted as η(G).

The subgradient method was first proposed by Shor [15,20]. A number of effective approaches arose as a result of the development of the first subgradient methods with a space dilation [21,22]. The first relaxation subgradient methods were suggested in [23,24,25].

In Ref. [26], a method to solve the convex problems of nondifferentiable optimization relying on the basic philosophy of the conjugate gradient method and coinciding with it in the case of quadratic functions was presented. Authors in [27] propose a family of adaptive subgradient methods that dynamically incorporate knowledge of the geometry of the data observed in earlier iterations to perform more informative gradient-based learning. Spectral-step subgradient method for solving nonsmooth unconstrained optimization problems was demonstrated in [28]. In Ref. [29], a subgradient method based on step size adjusting is developed to solve the system of nonsmooth equations. Subgradient projection methods are often applied to large-scale problems with decomposition techniques. Adaptive projection subgradient methods was proposed in [30,31]. Authors in [32] consider a projected-subgradient type method, derived from using a general distance-like function instead of the usual Euclidean squared distance.

As in QNM, RSM uses ratios to calculate the required characteristics. These relations represent equalities, solving which it is possible to determine the desired values. Due to the multiplicity of these equalities, it is necessary to resort to their solution by gradient methods as information becomes available. Thus, the use of the iterative least squares method (ILSM) is possible in this case.

In this paper, we consider modifications of subgradient methods applied to solving smooth minimization problems. In this case, the system of inequalities (2) will be formed by a set of gradients of some neighborhood of the current minimum. The descent direction s satisfying (2) will allow the method to go beyond this neighborhood as a result of the minimization step along this direction.

The main contributions of this paper are as follows:

(1) A methodological approach was proposed based on a procedure of incomplete orthogonalization in the directions of gradient differences, implemented through the iterative least-squares method. This approach uses the structural characteristics of the level surfaces instead of second-derivative approximations.

(2) Two methods were constructed based on this approach: a gradient method with the ILSM metric and a modification of the Hestenes–Stiefel conjugate gradient method with the same metric. Both methods were implemented and numerically studied comparing with the quasi-Newton BFGS method on various types of smooth functions. The test results indicate the efficiency of the proposed methods, especially when solving poorly conditioned problems with complex curved ravines and unstable characteristics of the second derivatives.

(3) The noted linear transformation of coordinates eliminates the linear background that worsens the convergence of the gradient method. In this work, we proved that algorithm HY_g, where the gradient method acquires accelerating abilities due to the use of the proposed metric transformation, has similar properties to Newton’s method, quasi-Newton methods, and subgradient methods with a change in the space metric. The qualitative nature of the convergence rate estimates for algorithm HY_g and Newton’s method coincide. Also, the equivalence of the conjugate gradient method with the new metric (HY_XS) to the conjugate gradient method on quadratic functions is proven.

The results obtained allow us to conclude that it is possible to use the studied methods along with quasi-Newton methods to solve smooth optimization problems with a high degree of conditionality.

The rest of the paper is organized as follows. Section 2 describes our variation in the gradient minimization method. In Section 3, we analyze the properties and convergence rate of the proposed algorithm. In Section 4, we study acceleration properties of the space dilation algorithm in the direction of the gradient difference. In Section 5, we propose the Hestenes–Stiefel method in a metric with incomplete orthogonalization in the direction of the gradient difference. In Section 6, we present the results of numerical experiments. Section 7 concludes the work.

2. Gradient Minimization Method with Incomplete Orthogonalization in the Direction of the Gradient Difference

For the gradients on the descent trajectory of (1), we use the notation $\nabla f(x_k)=g(x_k)=g_k$. The idea of creating an iterative method for solving the system of inequalities (2) is based on its transformation into a system of equalities [17,18,19].

To derive the formulas of the iterative process, we use an idealized model of the set G [17,18,19]. Let $G\subset R^n$ belongs to a certain hyperplane, and the vector η(G) is also the vector of minimal length of this hyperplane. Then there is a solution to the system of equalities:

$$(s,g)=1,\forall g\in G,$$

(3)

which simultaneously satisfies (2). Here, s is a descent direction, g is the gradient. The solution of system (3) can be obtained as the solution of the system of equalities:

$$\left(s,g_i\right)=1, i=0,1,\dots ,k.$$

(4)

One of the possible solutions to the system (4) can be found in the form of $s_{k+1}=\mathit{arg} \underset{s}{min}{F}_k(s)$, where $F_k\left(s\right)$ is the sum-of-squares function of the residuals:

$$F_k(s)={\sum }_{i=0}^k{w}_i{Q}_i(s)+{\displaystyle \frac{1}{2}}{\sum }_{i=1}^n{s}_i^2, \cdots Q_i(s)={\displaystyle \frac{1}{2}}(1-(s,g_i){)}^2.$$

Here, w_i are weighting factors. Such a solution, taking into account the regularizing component ${\displaystyle \frac{1}{2}}{\sum }_{i=1}^n{s}_i^2$, can be obtained by the iterative least squares method (ILSM) [33].

In Ref. [19], based on ILSM, an iterative process of finding the descent direction for the minimization method (1) using training information (4) was obtained using special weighting factors w_i in $F_k(s)$:

$$s_{k+1}=s_k+{\displaystyle \frac{H_k{g}_k(1-(s_k,g_k))}{(g_k{H}_k{g}_k)}}, s_0=0.$$

(5)

$$H_{k+1}=H_k-(1-{\displaystyle \frac{1}{{\alpha }_k^2}}){\displaystyle \frac{H_k{g}_k{g}_k^T{H}_k^T}{(g_k,H_k{g}_k)}}, H_0=I,$$

(6)

where α_k > 1 is space dilation parameter, H_k is metric matrix. In (5), the direction adjustment is set so that the learning ratio $(s_{k+1},g_k)=1$ is fulfilled.

Using (5), (6), we derive the iterative process obtained earlier on the basis of heuristic considerations for solving problems of non-smooth optimization (r-algorithm [15]) in the form proposed in [16].

To obtain the formulas of the iterative process at iteration k, we transform the data (4) by subtracting adjacent equalities. We will obtain a new system:

$$\left(s,y_i\right)=0, i=0,1,\dots k-1, (s,g_k)=1, y_i=g_i-g_{i+1}.$$

(7)

For a part of the data at i = 0, 1, …k − 1, we perform transformations (5), (6). Transformation (5) for $(s,y_i)=0$ will take the following form:

$$s_{i+1}=s_i+{\displaystyle \frac{H_i{y}_i(0-(s_i,y_i))}{(y_i{H}_i{y}_i)}}, s_0=0.$$

(8)

As a result of transformations (8), regardless of the values of the matrices H_i, by virtue of equality s₀ = 0 we obtain s_i = 0, i = 0, 1, …k − 1. Together with (8), we will carry out the transformation (6) sequentially:

$$H_{i+1}=H_i-(1-{\displaystyle \frac{1}{{\alpha }_k^2}}){\displaystyle \frac{H_i{y}_i{y}_i^T{H}_i^T}{(y_i,H_i{y}_i)}}, H_0=I, i=0,1,\dots k-1.$$

Transformation (6) for the last equality $(s,g_k)=1$ from (7) can be omitted, because, as a result of transformation (5) for the last equality $(s,g_k)=1$, a vector s_k collinear to the vector H_kg_k will be obtained. Therefore, the iterative minimization process using transformations (5), (6) for data (7) at k = 0, 1, 2, …, will have the form (1), where the descent direction and the metric matrix are iterating using the following formulas:

$$s_k=H_k{g}_k,$$

(9)

$$H_{k+1}=H_k-(1-{\displaystyle \frac{1}{{\alpha }_k^2}}){\displaystyle \frac{H_k{y}_k{y}_k^T{H}_k^T}{(y_k,H_k{y}_k)}}, H_0=I, y_k=g_k-g_{k+1},$$

(10)

with fixed values of ${\alpha }_k^2={\alpha }^2>1$.

Thus, using the idea of obtaining the descent direction satisfying the system of inequalities (2), we derived the iterative process of the r-algorithm [15] in the form of [16], which was previously obtained based on heuristic considerations.

In Ref. [4], to speed up the process of finding the descent direction in method (5), (6), one of its special cases is presented. An algorithm for obtaining a direction s_k satisfying, in contrast to (5), simultaneously the last two learning relations $(s_{k+1},g_k)=1$ and $(s_{k+1},g_{k-1})=1$ is proposed and theoretically justified. In this algorithm, the formulas for obtaining the direction of descent are as follows:

$$p_k=\{\begin{array}{c}{g}_k, if \left(k\ge 1 and \left(g_k,H_k{g}_{k-1}\right)>0\right) or k=0,\\ g_k-\frac{g_{k-1}\left(g_k,H_k{g}_{k-1}\right)}{g_{k-1},H_k{g}_{k-1}}, otherwise.\end{array}$$

(11)

$$s_{k+1}=s_k+H_k{p}_k[1-(s_k,g_k)]/(g_k,H_k{p}_k).$$

(12)

In this case, the metric matrix is adjusted according to formulas (10) with a varying space dilation parameter ${\alpha }_k^2>1$. Note that in order to ensure convergence when solving non-smooth problems, the choice of the descent direction is not always carried out according to Formulas (11) and (12).

Here and below, we will denote an algorithm by specifying the sequence of actions indicated by the corresponding equations. In Ref. [17], estimates of optimal parameter ${\alpha }_k^2>1$values are obtained, and it is shown that the algorithm (1)–(11)–(12)–(10) in the quadratic case generates a sequence of conjugate descent vectors. In the case of smooth functions, it becomes possible to use in the algorithm (1)–(11)–(12)–(10) the conjugate gradient method instead of formulas for generating the descent direction (11), (12).

The transformation of the ILSM metric (10) in the minimization algorithm has the property of partially orthogonalizing the descent vectors to the directions of the adjacent gradients difference on the descent trajectory. In the case of minimizing the quadratic function, the use of transformations (10) to form the descent direction increases the degree of conjugacy.

In Ref. [18], estimates of the linear convergence rate for the algorithm (1)–(9)–(10) are obtained on strongly convex functions with a Lipschitz gradient. It is proven that on such functions, this method has accelerating properties qualitatively similar to those of Newton’s method. The fact that the Newton method [18] and quasi-Newton methods have accelerating properties is explained by using either matrices of second derivatives or their approximations. Accelerating properties of the algorithm (1)–(9)–(10) due to the property of matrices (10) in the form of partial orthogonalization of the descent vectors to the directions of the adjacent gradients difference on the descent trajectory.

It was shown in [18] that the minimization algorithm (1) with the calculation of the descent direction based on (11), (12), and the transformation of matrices (10) is identical to the conjugate gradient method when minimizing quadratic functions. In this paper, the Hestenes–Stiefel conjugate gradient method with metric matrices (10) was developed to solve problems of minimizing smooth functions. The finite convergence of such a method on quadratic functions is proved.

3. Properties and Convergence Rate of the Gradient Minimization Method with Incomplete Orthogonalization in the Direction of the Gradient Difference

Here, we analyze the properties of the algorithm (1)–(9)–(10). The algorithm consists of the following three steps:

1. Compute the next minimum point;

$$x_{k+1}=x_k-{\gamma }_k{s}_k, {\gamma }_k=\mathit{arg} \underset{\gamma }{min}f(x_k-\gamma s_k), k=0,1,2\dots .$$

2. Compute descent direction $s_k=H_k{g}_k$;

3. Compute metric matrix update:

$$H_{k+1}=H_k-(1-{\displaystyle \frac{1}{{\alpha }_k^2}}){\displaystyle \frac{H_k{y}_k{y}_k^T{H}_k^T}{(y_k,H_k{y}_k)}}, H_0=I, y_k=g_k-g_{k+1}.$$

Formulas for correction vector p_k, and metric matrix update H_k+1 are as follows:

$$p_k=x_{k+1}=x_k-{\gamma }_k{s}_k, s_k=H_k{g}_k, {\gamma }_k=\underset{\gamma }{argmin}f(x_k-\gamma s_k),k=0,1,2,\dots ,$$

(13)

$$H_{k+1}=H_k-(1-{\displaystyle \frac{1}{{\alpha }_k^2}}){\displaystyle \frac{H_k{y}_k{y}_k^T{H}_k^T}{(y_k,H_k{y}_k)}}, {\alpha }_M\ge {\alpha }_k\ge {\alpha }_m>1, y_k=g_k-g_{k+1},$$

(14)

where k is iteration number, x_k is current minimum point, s_k is descent direction, H_k is metric matrix at iteration k, g_k is gradient at iteration k, $\gamma$ is stepsize, ${\alpha }_k$is space dilation parameter, y_k is gradient difference, x₀ is initial point and H₀ is initial matrix.

As will be shown below, method (13)–(14) is invariant with respect to linear transformation of coordinates. By invariance, we mean the similarity of the process type in different systems. This means that when the iterations of the method are transferred to a new coordinate system, the iterative process is preserved. In this case, no conditions are imposed on the function, the only condition is its differentiability. For instance, in the case of a quadratic function, after transferring to a coordinate system where the Hessian has eigenvalues equal to one, the gradient differences $y_k=g_k-g_{k+1}$ will coincide with the method offsets $v_k=x_k-x_{k+1}$ during the iteration. Consequently, in the new coordinate system, the choice of the descent direction in (13) looks like the process of partial orthogonalization of the gradient to the previous descent directions using matrices (14). With complete orthogonalization, such a minimization process along the orthogonal system of directions for a quadratic function with unit Hessian will be finite [1].

The limiting variant of algorithm (13)–(14) for ${\alpha }_k^2=\infty$ takes the form of the previously known conjugate gradient method [1]. In this method, the descent directions are orthogonalized to all previous differences in adjacent gradients. This property does not allow the method to optimize in a subspace defined by gradient differences, that is, it excludes it from the optimization process. Such a method will not be effective, especially for large dimensions. In algorithm (13)–(14), only a partial reduction in the components of the descent direction in the subspace of the preceding differences in adjacent gradients is performed. This does not block the possibility of minimization in the examined subspace. Transformation (14) suppresses to a greater extent the gradient components directed orthogonally to the ravine directions. This is one of the qualitative justifications explaining the accelerating properties of the transformation (14), on the basis of which the r-algorithm was developed [15].

Cases of low efficiency of quasi-Newton methods are associated with functions in which the Hessian eigenvalues decrease as they approach the minimum. In this case, the inverse matrices of quasi-Newton methods increase in the examined subspace, which makes it difficult for the method to enter the unexplored subspace. In algorithms (13)–(14), by reducing the influence of the examined subspace, the unexplored subspace always has an advantage. For this reason, for some classes of functions, algorithm (13)–(14) may be more effective than quasi-Newton methods, which will be confirmed by numerical tests.

Transformation (14) in algorithm (13)–(14) can be understood as a scaling transformation for the gradient method. This transformation may be useful as a scaler in other methods. In this paper, we will consider the use of transformation (14) in the Hestenes–Stiefel conjugate gradient method.

We investigate the convergence rate of algorithm (13)–(14). Let us consider the conditions under which the convergence rate and accelerating properties of the r-algorithm are estimated.

Condition 1. 

We will assume that the function f(x), x∈ Rⁿ differentiable and strongly convex in Rⁿ and is minimized, i.e., there exists ρ > 0 such that∀x, y∈ Rⁿ and the following inequality holds:

$$f(\alpha x+(1-\alpha )y)\le \alpha f(x)+(1-\alpha )f(y)-\alpha (1-\alpha )\rho {\Vert x-y\Vert }^2/2,$$

and its gradient g(x) =∇ f(x) satisfies the Lipschitz condition:

$$‖g(x)-g(y)‖\le L‖x-y‖\hspace{1em}\forall x,y\in R^n, L>0.$$

Here, L is Lipschitz parameter.

Denote by x^* the minimum point of function f(x), f^* = f(x^*), f_k = f(x_k). The iteration of the gradient-matched method for exact one-dimensional descent has the form:

$$x_{k+1}=x_k-{\beta }_k{s}_k, {\beta }_k=\underset{\beta \ge 0}{argmin}f\left(x_k-\beta s_k\right), (s_k,g_k)>0,k=0,1,2,\dots ,$$

(15)

where the initial point x₀ is set. Formula (15) does not specify the descent direction S, they only impose the condition $(s_k,g_k)>0$ on it. Although the direction $s_k=H_k{g}_k$ from (13) also satisfies the condition from (1) due to the positive definiteness of the metric matrices.

The following theorem shows that changes in the gradient as a result of method (15) iterations with an exact one-dimensional descent lead to a decrease.

Theorem 1 

[18]. Let the function satisfy Condition 1. Then, for the sequence f_k, k = 0, 1, 2… specified by (15), the following estimate takes place:

$$f_{k+1}-f*\le (f_0-f*)\mathit{exp}\left[-{\displaystyle \frac{{\rho }^2}{L^2}}{\sum }_{i=0}^k{\displaystyle \frac{{‖y_i‖}^2}{{‖g_i‖}^2}}\right],$$

(16)

where $y_i=g_{i+1}-g_i$, f* is optimal function value, f₀ is initial function value, L is Lipschitz parameter, ρ is defined in Condition 1.

Here we derive the recurrent formulas for converting the determinants and trace of the metric matrices under consideration. Denote $A_k=H_k^{-1}$. Denote by Sp(A) a trace of matrix A and by det(A) a determinant of matrix A. For an arbitrary matrix A > 0, we denote by $A^{1/2}$ a symmetric matrix for which $A^{1/2}>0$ and $A^{1/2}{A}^{1/2}=A$.

Lemma 1. 

Let H_k > 0 and matrix H_k+1 is obtained as a result of the transformation (14):

$$H_{k+1}=H_k-(1-{\displaystyle \frac{1}{{\alpha }_k^2}}){\displaystyle \frac{H_k{y}_k{y}_k^T{H}_k^T}{(y_k,H_k{y}_k)}}, {\alpha }_M\ge {\alpha }_k\ge {\alpha }_m>1, y_k=g_k-g_{k+1}.$$

Then, H_k+1 > 0 and

$$A_{k+1}=A_k+({\alpha }_k^2-1){\displaystyle \frac{y_k{y}_k^T}{(y_k,H_k{y}_k)}},$$

(17)

$$Sp{A}_{k+1}=Sp{A}_k+({\alpha }_k^2-1){\displaystyle \frac{(y_k,y_k)}{(y_k,H_k{y}_k)}},$$

(18)

$${\mathit{detH}}_{k+1}={\mathit{detH}}_k/{\alpha }_k^2, {\mathit{detA}}_{k+1}={\alpha }_k^{2}det{A}_k.$$

(19)

Proof of Lemma 1. 

We transform the right-hand side of expression (14) using the notation introduced earlier:

$$H_{k+1}=H_k^{1/2}\left[I-(1-{\displaystyle \frac{1}{{\alpha }_k^2}}){\displaystyle \frac{H_k^{1/2}{y}_k{y}_k^T{H}_k^{1/2}}{(H_k^{1/2}{y}_k,H_k^{1/2}{y}_k)}}\right]H_k^{1/2}=H_k^{1/2}\left[I-(1-{\displaystyle \frac{1}{{\alpha }_k^2}}){\displaystyle \frac{v_k{v}_k^T}{(v_k,v_k)}}\right]H_k^{1/2},v_k=H_k^{1/2}{y}_k$$

(20)

Inverting the matrix (20) element by element, we obtain (17). Formula (18) follows from (17). Calculating the determinant element by element for the matrices in (20), we obtain (19). □

The following theorem substantiates the linear convergence rate of algorithm (13)–(14).

Theorem 2. 

Let the function f(x) satisfy Condition 1. Then, for the sequence {f_k}, k = 0, 1, 2, … given by the algorithm (13)–(14), with a limited initial matrix H₀:

$$m_0\le (H_{0}z,z)/(z,z)\le M_0,\forall z\in R^n,z\ne 0,$$

(21)

where M₀ and m₀ are maximal and minimal eigenvalues of the matrix H₀, respectively, the following estimation takes place:

$$f_{k+1}-f*\le (f_0-f*)\mathit{exp}\left\{-{\displaystyle \frac{{\rho }^2(k+1)}{L^{2}n}}\left[{\displaystyle \frac{\mathit{ln}({\alpha }_m^2)}{({\alpha }_M^2-1)}}+{\displaystyle \frac{n\mathit{ln}(m_0/M_0)}{(k+1)({\alpha }_M^2-1)}}\right]\right\}.$$

(22)

Proof of Theorem 2. 

We transform (18):

$$Sp(A_{k+1})=Sp(A_k)\left[1+{\displaystyle \frac{({\alpha }_k^2-1)(y_k,y_k)}{Sp(A_k)(H_k{y}_k,y_k)}}\right].$$

(23)

Due to the exact one-dimensional descent in (13), the following condition is satisfied:

$$(s_k,g_{k+1})=(H_k{g}_k,g_{k+1})=0.$$

The matrix H_k is positive definite, therefore, using the last equality, we obtain the following:

$$(H_k{y}_k,y_k)=(H_k{g}_k,g_k)+(H_k{g}_{k+1},g_{k+1})-2(H_k{g}_k,g_{k+1})\ge (H_k{g}_k,g_k).$$

Hence, taking into account the inequality

$$Sp(A_k)\ge M_k,$$

where M_k is the maximum eigenvalue of the matrix A_k, we find:

$$Sp(A_k)(H_k{y}_k,y_k)\ge Sp(A_k)(H_k{g}_k,g_k)\ge {\displaystyle \frac{Sp(A_k)}{M_k}}(g_k,g_k)\ge (g_k,g_k).$$

The last estimate allows us to reduce inequality (23) to the following form:

$$Sp(A_{k+1})\le Sp(A_k)\left[1+({\alpha }_k^2-1){\displaystyle \frac{{‖y_k‖}^2}{{‖g(x_k)‖}^2}}\right].$$

(24)

Based on the ratio between the arithmetic mean and geometric mean of the eigenvalues of the matrix A > 0, we have

$$Sp(A)/n\ge [\mathit{det}(A){]}^{1/n}.$$

Using the last inequality in (24), taking into account (19), we obtain:

$${\displaystyle \frac{Sp(A_0)}{n}}\prod _{i=0}^k\left[1+({\alpha }_M^2-1){\displaystyle \frac{{‖y_i‖}^2}{{‖g(x_i)‖}^2}}\right]\ge {\displaystyle \frac{Sp(A_{k+1})}{n}}\ge (\mathit{det}(A_{k+1}){)}^{1/n}=[({\alpha }_m^2{)}^{k+1}\mathit{det}(A_0){]}^{1/n}.$$

The latter ratio, based on inequality

$$1+p\le \mathit{exp}(p),$$

we convert to the following form:

$${\displaystyle \frac{Sp(A_0)}{n}}\mathit{exp}\left[({\alpha }_M^2-1){\sum }_{i=0}^k{\displaystyle \frac{{‖y_i‖}^2}{{‖g(x_i)‖}^2}}\right]\ge ({\alpha }_m^2{)}^{(k+1)/n}(\mathit{det}(A_0){)}^{1/n}.$$

(25)

By virtue of condition (21), inequalities hold:

$$Sp(A_0)/n\le 1/m_0,(\mathit{det}(A_0){)}^{1/n}\ge 1/M_0.$$

Taking these relations into account, we transform inequality (25):

$${\displaystyle \frac{1}{m_0}}\mathit{exp}\left[({\alpha }_M^2-1)\sum _{i=0}^k{\displaystyle \frac{{‖y_i‖}^2}{{‖g(x_i)‖}^2}}\right]\ge {\displaystyle \frac{{\alpha }_m^{2(k+1)/n}}{M_0}}.$$

Taking the logarithm of the latter, we find:

$$\left[({\alpha }_M^2-1){\sum }_{i=0}^k{\displaystyle \frac{{‖y_i‖}^2}{{‖g(x_i)‖}^2}}\right]\ge {\displaystyle \frac{(k+1)\mathit{ln}({\alpha }_m^2)}{n}}+\mathit{ln}({\displaystyle \frac{1}{M_0}})-\mathit{ln}\left({\displaystyle \frac{1}{m_0}}\right).$$

From this follows the inequality:

$${\sum }_{i=0}^k{\displaystyle \frac{{‖y_i‖}^2}{{‖g(x_i)‖}^2}}\ge {\displaystyle \frac{(k+1)\mathit{ln}({\alpha }_m^2)}{n({\alpha }_M^2-1)}}+{\displaystyle \frac{\mathit{ln}(m_0/M_0)}{({\alpha }_M^2-1)}},$$

which, together with estimate (16) of Theorem 1, proves (22). □

The obtained convergence rate estimates do not explain the method’s high convergence rate, for example, on quadratic functions. To justify the method’s acceleration properties, we have to demonstrate its invariance under a linear coordinate transformation and then use estimate (22) in the coordinate system in which the ρ/L ratio is maximal. It is possible to increase this ratio, for example, in the case of quadratic functions, where its maximum value is 1.

4. Acceleration Properties of the Space Dilation Algorithm in the Direction of the Gradients Difference

To estimate the convergence rate greater than linear in minimization methods, the minor variability of the Hessian in the extremum region is primarily used, which can be estimated based on the Hessian properties. We examine the acceleration properties of the algorithm without assumptions about the function’s second derivative matrices.

According to estimate (22), the convergence rate of the r-algorithm is determined by the magnitude of the ρ/L ratio. In our study, similar to study [18], we will assume the existence of a linear transformation of coordinates that increases the magnitude of this ratio.

Let a function f(x) satisfy Condition 1. We define a linear transformation of variables

$$\widehat{x}=Px,$$

(26)

where $P\in R^n$ is a non-singular matrix. In the new coordinate system, the function being minimized will have the following form:

$$f_P(\widehat{x})=f\left(P^{-1}\widehat{x}\right)=f(x).$$

(27)

The function (27) formed in this way also satisfies Condition 1 with parameter of strong convexity ρ and Lipschitz parameter L.

Denote $V\in R^{n\times n}$ a non-singular matrix such that for parameters ρ and L of functions $f_V(\widehat{x})$ with

$$\widehat{x}=Vx$$

(28)

and $f_P(\widehat{x})$ for an arbitrary non-singular matrix $P\in R^{n\times n}$ in (26), the inequality holds

$${\displaystyle \frac{{\rho }_V}{L_V}}\ge {\displaystyle \frac{{\rho }_p}{L_p}}.$$

(29)

As was established in [18], transformation (28) plays the role of a distinguished coordinate system, which is the best from the point of view of the convergence rate of gradient methods. For a non-degenerate quadratic function, we can consider a coordinate system where ρ/L = 1 and the maximum and minimum eigenvalues are equal. In the general case, we only assume the possibility of eliminating the differences in the elongation of the level surfaces.

Next, we will show that algorithm (13)–(14), applied to the function f(x), and algorithm (13)–(14), applied to the function $f_P(\widehat{x})$ defined in (22), under appropriate initial conditions, construct sequences of minimization points related by the transformation (26). This will allow us to use the estimates of the convergence rate of algorithm (13)–(14) in the preferred coordinate system.

Theorem 3. 

Let the initial conditions of the algorithm (13)–(14), applied to minimize the functions f(x) and $f_P(\widehat{x})$, defined in (27), be related by the equalities:

$${\widehat{x}}_0=P{x}_0, {\widehat{H}}_0=P{H}_0{P}^T.$$

(30)

Then the characteristics of these processes are related by the following relations:

$$f_P({\widehat{x}}_k)=f(x_k), {\widehat{x}}_k=P{x}_k, \nabla f_P({\widehat{x}}_k)=P^{-T}\nabla f(x_k), {\widehat{H}}_k=P{H}_k{P}^T,(k=0,1,2\dots ).$$

(31)

Proof of Theorem 3. 

For the gradients of functions $f_P\left(\widehat{x}\right)$ and f(x) the following relation holds:

$$\nabla f_P(\widehat{x})=P^{-T}\nabla f(x).$$

From this and (30) follows (31) for k = 0. Let us assume that equalities (31) are satisfied ∀k = 0, 1, …, i. Let us show their feasibility for k = i + 1. From (13) for k = i after left-hand multiplying by P, taking into account the proved equalities (31), we obtain:

$$P{x}_{i+1}=P{x}_i-{\gamma }_{i}P{H}_i{P}^T{P}^{-T}\nabla f\left(x_i\right)={\overline{x}}_i-{\gamma }_i{\widehat{H}}_i\nabla f_P\left({\widehat{x}}_i\right).$$

(32)

Hence, according to the definition of the function f_p, at the stage of one-dimensional minimization the equality ${\gamma }_i={\overline{\gamma }}_i$ is satisfied. Therefore, the right-hand side of (32) is the implementation of the step in the new coordinate system. Consequently:

$${\widehat{x}}_i=P{x}_i,\nabla f_P({\widehat{x}}_i)=P^{-T}\nabla f(x_i),{\widehat{y}}_i=\nabla f_P({\widehat{x}}_{i+1})-\nabla f_P({\widehat{x}}_i)=P^{-T}{y}_i.$$

(33)

Multiplying (14) with the current indices on the left by P, and on the right by P^T, taking into account (33) and the equality:

$$(y_i,P^{-1}P{H}_i{P}^T{P}^{-T}{y}_i)=({\widehat{H}}_i{\widehat{y}}_i,{\widehat{y}}_i),$$

we get

$$P{H}_{i+1}{P}^T=P{H}_i{P}^T-\left(1-{\displaystyle \frac{1}{{\alpha }_i^2}}\right){\displaystyle \frac{P{H}_i{P}^T{P}^{-T}{y}_i{y}_i^T{P}^{-1}P{H}_i^T{P}^T}{\left(y_i,P^{-1}P{H}_i{P}^T{P}^{-T}{y}_i\right)}}={\widehat{H}}_i-\left(1-{\displaystyle \frac{1}{{\alpha }_i^2}}\right){\displaystyle \frac{{\widehat{H}}_i{\widehat{y}}_i{\widehat{y}}_i^T{\widehat{H}}_i^T}{\left({\widehat{H}}_i{\widehat{y}}_i,{\widehat{y}}_i\right)}},$$

where the right side is the implementation of (14) in the new coordinate system.

Finally, we get $P{H}_{i+1}{P}^T={\widehat{H}}_{i+1}$. Therefore, equalities (31) will also be valid for k = i + 1. Continuing the process of induction, we obtain a proof of the theorem. □

In the following theorem, we use algorithms (13)–(14) in a distinguished coordinate system (28) with property (29).

Theorem 4. 

Let the function f(x) satisfy Condition 1. Then, for the sequence {f_k}, k = 0, 1, 2, … given by the algorithm (13)–(14), with a limited initial matrix H₀, according to (21) the following estimation takes place:

$$f_{k+1}-f*\le (f_0-f*)\mathit{exp}\left\{-{\displaystyle \frac{{\rho }_V^2(k+1)}{L_V^{2}n}}\left[{\displaystyle \frac{\mathit{ln}({\alpha }_m^2)}{({\alpha }_M^2-1)}}+{\displaystyle \frac{n\mathit{ln}({\widehat{m}}_0/{\widehat{M}}_0)}{(k+1)({\alpha }_M^2-1)}}\right]\right\}.$$

(34)

where ${\widehat{m}}_0,{\widehat{M}}_0$ are the minimum and maximum eigenvalues of the matrix ${\widehat{H}}_0=V{H}_0{V}^T$, respectively, in the selected coordinate system (28) having the property (29).

Proof of Theorem 4.

According to the results of Theorem 3, we can choose an arbitrary coordinate system to estimate the convergence rate of the minimization process in the algorithm. Therefore, using estimate (22) in a coordinate system with matrix P = V, we obtain estimates (34). □

The first term in square brackets (34) characterizes the constant in the method’s convergence rate estimate, and the second term represents the cost of adjusting the metric matrix.

Let us consider the acceleration effect of algorithm (13)–(14) compared to well-known methods: steepest descent and Newton’s method. For these methods, the convergence rate estimate is as follows:

$$f_{k+1}-f*\le q^{k+1}(f_0-f*).$$

(35)

Under Condition 1 imposed on the function, for the steepest descent, where s_k = g_k, the decay rate is as follows [1]:

$$q=1-{\displaystyle \frac{\rho }{L}}.$$

(36)

For Newton’s method $s_k=[{\nabla }^{2}f(x_k){]}^{-1}\nabla f\left(x_k\right)$ and decay rate is [18]:

$$q=1-{\displaystyle \frac{{\rho }^2}{L^2}}.$$

(37)

The convergence rate for Newton’s method, due to its invariance with respect to the linear transformation of coordinates, has the form [18]:

$$q=1-{\rho }_V^2/L_V^2.$$

(38)

Given that

$$l_V/L_V>>l/L,$$

(39)

Taking into account (36)–(38), the convergence rate of Newton’s method will be significantly higher than that of the steepest descent.

Estimate (34) for the algorithm (13)–(14) is equivalent to the estimate for Newton’s method (37) in terms of the influence of constants on the convergence rate. Under condition (39), the convergence rate estimate for the algorithm (13)–(14) is preferable to that for the steepest descent method, which is confirmed further by a computational experiment.

Thus, algorithm (13)–(14) on strongly convex functions, without assuming the existence of second derivatives, exhibits acceleration properties compared to the steepest descent method.

5. Hestenes–Stiefel Method in a Metric with Incomplete Orthogonalization in the Direction of Gradient Difference

Transformation (14) in the algorithm (13)–(14) can act as a scaling transformation for the steepest descent. This transformation may also be useful in other methods.

In Ref. [17], it is shown that the sequence of approximations of the minimum generated by the algorithm (1)–(11)–(12), with the transformation of matrices (10), or equivalently with (14), coincides with the sequence generated by the conjugate gradient method. Transformations (11), (12) are based on the simultaneous solution of the system of equalities $\left(s_{k+1},g_k\right)=1$ and $(s_{k+1},g_{k-1})=1$. The equality $\left(s_{k+1},g_k\right)-\left(s_{k+1},g_{k-1}\right)=\left(s_{k+1},g_k-g_{k-1}\right)=0$ holds for their difference. In the Hestenes–Stiefel method [30], the new direction is also chosen based on the equality $(s_{k+1},g_k-g_{k-1})=0$. In the quadratic case, the new direction in the Hestenes–Stiefel method will be conjugate to the current descent direction. In the case of smooth functions, we can replace the transformations (11), (12) for obtaining the descent direction with the transformations in the Hestenes–Stiefel method.

The Hestens–Stiefel conjugate gradient method [34] has the form:

$$x_{k+1}=x_k-{\gamma }_k{s}_k, {\gamma }_k=\underset{\gamma }{argmin}f(x_k-\gamma s_k),k=0,1,2\dots ,$$

(40)

$$s_0=g_0, s_{k+1}=g_{k+1}-{\displaystyle \frac{(g_{k+1},y_k)}{(s_k,y_k)}}{s}_k, y_k=g_k-g_{k+1}.$$

(41)

From (41), it follows that the new descent direction of the current gradient difference is orthogonal $(s_{k+1},y_k)=0$. For a quadratic function, this implies conjugacy of adjacent descent vectors, regardless of the accuracy of the one-dimensional search.

In the Hestenes–Stiefel method with a space metric transformation (14), we also use the property $(s_{k+1},y_k)=0$ to obtain the descent vector transformation formulas:

$$x_{k+1}=x_k-{\gamma }_k{s}_k, {\gamma }_k=\underset{\gamma }{argmin}f(x_k-\gamma s_k),k=0,1,2\dots ,$$

(42)

$$s_0=H_0{g}_0, s_{k+1}=H_k{g}_{k+1}-{\displaystyle \frac{(H_k{g}_{k+1},y_k)}{(s_k,y_k)}}{s}_k, y_k=g_{k+1}-g_k,$$

(43)

$$H_0=I, H_{k+1}=H_k-(1-{\displaystyle \frac{1}{{\alpha }_k^2}}){\displaystyle \frac{H_k{y}_k{y}_k^T{H}_k^T}{(y_k,H_k{y}_k)}}, {\alpha }_M\ge {\alpha }_k\ge {\alpha }_m>1.$$

(44)

Algorithms (42)–(44) has the properties of the conjugate gradient method. To justify this property, we need the following theorem.

Theorem 5. 

Let the matrices H_i, i = 1, 2, …k, be obtained as a result of transformations (44), and let the vectors y_i, i = 0, 1, …k − 1, used in (44), be orthogonal to the vector g_k+1. Then,$H_k{g}_{k+1}=g_{k+1}$.

Proof of Theorem 5. 

We will carry out the proof by induction. Let $H_i{g}_{k+1}=g_{k+1}$ for i < k − 1. Then, taking into account the orthogonality of the vectors y_i, i = 0, 1, …, k − 1, to the vector g_k+1, we obtain

$$(H_{i+1},g_{k+1})=\left(H_i-(1-{\displaystyle \frac{1}{{\alpha }_k^2}}){\displaystyle \frac{H_i{y}_i{y}_i^T{H}_i^T}{(y_i,H_i{y}_i)}},g_{k+1}\right)=g_{k+1}-(1-{\displaystyle \frac{1}{{\alpha }_k^2}}){\displaystyle \frac{(y_i,H_i{g}_{k+1})}{(y_i,H_i{y}_i)}}{H}_i{y}_i=g_{k+1}-(1-{\displaystyle \frac{1}{{\alpha }^2}}){\displaystyle \frac{(y_i,g_{k+1})}{(y_i,H_i{y}_i)}}{H}_i{y}_i=g_{k+1}.$$

Continuing the process of induction, we obtain the statement of the theorem. □

Regarding the convergence of algorithms (42)–(44) on quadratic functions, the following theorem holds.

Theorem 6. 

When minimizing quadratic functions, the sequences of points x_k, k = 0, 1, 2, …, of algorithms (40)–(41), (42)–(44) with the same initial points and initial matrix $H_0=I$ coincide.

Proof of Theorem 6. 

We will prove the theorem by induction. Let the points x_k of algorithms (40)–(41), (42)–(44) coincide for i < k. In conjugate gradient methods, when minimizing a quadratic function using exact one-dimensional descent, the gradient vectors are mutually orthogonal. Therefore, the current gradient g_i at a new point is the same for both algorithms and orthogonal to all previous gradients. The vector s_i is obtained by formula (43) using the matrix $H_{i-1}{g}_i$. The matrix H_i−1 is obtained using vectors y_j, j = 0, 1, …i − 2, in which the gradient vectors g_j, j = 0, 1, …i − 1, orthogonal to the vector g_i, are used. From here, according to Theorem 5, we obtain $H_{i-1}{g}_i=g_i$. Using this equality in (43), we obtain the coincidence of the descent direction with the Hestenes–Stiefel method

$$s_i=H_{i-1}{g}_i-{\displaystyle \frac{(H_{i-1}{g}_i,y_{i-1})}{(s_{i-1},y_{i-1})}}{s}_{i-1}=g_i-{\displaystyle \frac{(g_i,y_{i-1})}{(s_{i-1},y_{i-1})}}{s}_{i-1}.$$

Consequently, the new iteration of each process will be performed by minimization along the same directions and from the same point. Continuing the induction process further, we obtain a proof of the theorem. □

The level surface topology of the function being minimized may be similar in many ways to the topology of the quadratic function in a neighborhood significantly larger than the neighborhood in which the function and its quadratic representation are similar. In such a case, it can be assumed that algorithms (13)–(14) and (42)–(44), which do not use approximations of the Hessians, will be more effective than quasi-Newton methods.

6. Numerical Experiment

To identify the potential of transformation (14) for solving problems of minimizing smooth functions and to test the effectiveness of the presented algorithms (13)–(14) and (42)–(44), a numerical study was conducted. The methods are compared with the quasi-Newton BFGS method.

All methods used in the study utilized the one-dimensional search described in [35], where the gradient and function value are used as information for organizing the method. This is particularly advantageous in conditions where the costs of computing the gradient and function are comparable.

In the following tables we will denote:

(1)

BFGS—quasi-Newtonian BFGS method;

(2)

HY_g—algorithm (13)–(14);

(3)

HY_XS—algorithm (42)–(44).

When solving an ill-conditioned minimization problem with high accuracy, the quasi-Newton method BFGS is typically used, or, if possible, Newton’s method. For this reason, we chose the BFGS method as the benchmark for comparison with the methods under study.

The set of test functions includes a quadratic function. Since we know the function’s condition number, we can estimate its complexity for minimization. Also, based on the results of minimizing the quadratic function, we obtain information about the method’s behavior in a certain neighborhood of the current minimization point of the real function, where its quadratic representation is valid.

The tests include functions with both linear and curvilinear ravines. With curvilinear ravines, the Hessian eigenvectors change as we move toward the minimum, which leads to obsolescence of the metric matrices and a decrease in the convergence rate. Further successful progress requires reconfiguring the method’s parameters. On such functions we will observe the behavior of the method with the variability of the quadratic representation of the function as we move towards the minimum.

We also consider a function whose level surface topology matches that of the quadratic function. In this case, it is of interest to compare the algorithms (13)–(14) and (42)–(44) with quasi-Newton methods, which rely less on the properties of the Hessian matrices.

The final test problem reflects the variability of scaling across variables as we move towards the extremum. In this case, there is an active change in the level surface topology due to changes in the scales along the coordinate axes.

In all methods, the function and gradient were calculated simultaneously. Tables show the number of iterations and the total number of function and gradient calculations for the selected methods. The problem dimension in each experiment varies from 100 to 1000. The stopping criterion was $f(x^k)-f^{*}\le \epsilon$.

6.1. Quadratic Function

We use the following quadratic function:

$$f_Q(x,[a\mathit{max}])={\displaystyle \frac{1}{2}}{\sum }_{i=1}^n{a}_i{x}_i^2,a_i=a{max}^{{\displaystyle \frac{i-1}{n-1}}}.$$

(45)

The limits of eigenvalues a_i of this function are λ_min = 1 and λ_max = a_max. Starting point is x₀ = (100,100,…, 100). The stopping criterion was $f(x^k)-f^{*}\le 10^{-10}$.

Table 1. Results of function f_Q(x, [a_max = 10⁴]) minimization.

n	BFGS		HY_g		HY_XS
	Number of Iterations	Number of Function and Gradient Calculations	Number of Iterations	Number of Function and Gradient Calculations	Number of Iterations	Number of Function and Gradient Calculations
100	107	324	197	463	166	408
200	176	464	304	666	203	472
300	234	575	404	865	254	573
400	284	672	494	1045	292	650
500	328	760	577	1211	327	720
600	367	837	656	1370	360	787
700	403	909	728	1515	392	851
800	437	978	795	1649	424	915
900	466	1036	856	1771	454	975
1000	495	1093	912	1884	482	1032

and

Table 2. Results of function f_Q(x, [a_max = 10⁸]) minimization.

n	BFGS		HY_g		HY_XS
	Number of Iterations	Number of Function and Gradient Calculations	Number of Iterations	Number of Function and Gradient Calculations	Number of Iterations	Number of Function and Gradient Calculations
100	136	443	474	1111	476	1155
200	263	785	725	1668	651	1559
300	377	1086	967	2219	796	1838
400	481	1342	1217	2745	934	2118
500	576	1547	1455	3303	1080	2429
600	675	1783	1695	3863	1253	2818
700	769	2018	1932	4409	1414	3169
800	853	2198	2164	4942	1551	3484
900	944	2428	2389	5432	1686	3778
1000	1026	2608	2620	5996	1813	4067

show the results of function minimization for different degrees of conditionality a_max = 10⁴ and a_max = 10⁸. Best results are given in bold.

For function with a low degree of conditionality the BFGS quasi-Newton method does not have enough number of iterations to create a suitable metric matrix. Perhaps for this reason, the HY_XS algorithm is equivalent in performance to the BFGS. On this test, HY_XS method outperforms the HY_g method.

In Table 2, the problem conditionality is higher, and the required number of iterations to solve the problem is commensurate with the problem dimension. BFGS manages to construct the metric matrix and, as a result, achieves a high convergence rate in the final iterations and outperforms other algorithms. Here, HY_XS outperforms HY_g.

This is an example of an ill-conditioned problem with a fixed Hessian, where the quasi-Newton method (BFGS) is significantly more efficient than other methods. Based on the results of this example, we can conclude that if the matrix of second derivatives in a real-world problem is ill-conditioned and does not change significantly over a sufficiently wide range, then the quasi-Newton method will outperform the HY_g and HY_XS algorithms.

6.2. Function with Multidimensional Ellipsoidal Ravine

The following function has a multidimensional ellipsoidal ravine. Minimization occurs when moving along a curvilinear ravine to the minimum point.

$$f_E(x,[a\mathit{max},b\mathit{max}])=(1-x_1{)}^2+a {\mathit{max}\left(1-{\sum }_{i=1}^n{x}_i^2/b_i\right)}^2,b_i=b{\mathit{max}}^{{\displaystyle \frac{i-1}{n-1}}}$$

(46)

The stopping criterion was $f(x^k)-f^{*}\le 10^{-4}$.

Table 3. Results of function f_E(x, [a_max = 10², b_max = 10³]) minimization from starting point x₀¹.

n	BFGS		HY_g		HY_XS
	Number of Iterations	Number of Function and Gradient Calculations	Number of Iterations	Number of Function and Gradient Calculations	Number of Iterations	Number of Function and Gradient Calculations
100	58	196	354	1008	46	125
200	114	372	682	1837	58	150
300	135	463	949	2495	37	118
400	154	485	1052	2691	37	95
500	154	496	1162	2991	49	132
600	185	599	1017	2524	39	118
700	185	573	917	2200	39	104
800	207	675	1168	2858	42	117
900	224	739	490	1171	40	110
1000	236	750	1042	2457	36	104

and

Table 4. Results of function f_E(x, [a_max = 10², b_max = 10³]) minimization from starting point x₀².

n	BFGS		HY_g		HY_XS
	Number of Iterations	Number of Function and Gradient Calculations	Number of Iterations	Number of Function and Gradient Calculations	Number of Iterations	Number of Function and Gradient Calculations
100	937	2465	242	689	74	288
200	1434	3729	364	1008	139	507
300	1729	4526	603	1575	146	513
400	1929	5042	661	1661	258	857
500	2127	5590	819	2019	180	602
600	2267	5945	840	2034	204	679
700	2356	6169	1017	2454	261	852
800	2428	6381	1160	2769	271	924
900	2532	6676	1355	3249	207	697
1000	2631	6932	1540	3697	248	817

demonstrate results of function f_E minimization. In Table 3 starting point is x₀¹ = (−1, 0.1, …, 0.1), in Table 4 starting point is x₀² = (−1, 2, 3, …, n). Best results are given in bold.

For this starting point, the BFGS and HY_XS methods reach a minimum almost immediately, where the conditioning level increases as the minimum is approached. Therefore, the BFGS and HY_XS algorithms are more efficient than the HY_g method.

At the minimum point, the function is degenerate. Therefore, as the degree of conditioning increases while the minimum is approached, the HY_g method, which does not use conjugate directions, turns out to be the worst performer.

For this starting point, the methods initially enter a ravine far from the minimum point and move along the bottom of the curvilinear ravine. The elongation of the isosurfaces and the directions of elongation in the curvilinear ravine change, preventing the BFGS method from effectively and quickly adjusting the metric matrix. Here, the HY_XS and HY_g algorithms have an advantage. Moreover, the conjugacy of the descent directions in the HY_XS algorithm allows for obtaining a solution much faster in the neighborhood of a degenerate minimum point.

6.3. Function with Multidimensional Ellipsoidal Ravine and Non-Degenerate Minimum Point

The next function also has a multi-dimensional ellipsoidal ravine.

$$f_{EX}(x,[a\mathit{max},b\mathit{max}]])=(1-x_1{)}^2+a {\mathit{max}\left(1-{\sum }_{i=1}^n{\displaystyle \frac{{x_i}^2}{b_i}}\right)}^2+{\displaystyle \frac{1}{2}}{\sum }_{i=1}^n{\displaystyle \frac{{x_i}^2}{b_i}},\hspace{1em}{b}_i=b {max}^{{\displaystyle \frac{i-1}{n-1}}}.$$

(47)

The starting point is x₀ = (−1, 2, 3, …, n). Stopping criterion is $f(x^k)-f^{*}\le 10^{-10}$. The function has an additional quadratic term, so the function ceases to be degenerate at the minimum point. Due to this, gradient methods are able to find the minimum of function f_EX with higher accuracy than for function f_E.

Table 5. Results of function f_EX(x, [a_max = 10², b_max = 10³]) minimization.

n	BFGS		HY_g		HY_XS
	Number of Iterations	Number of Function and Gradient Calculations	Number of Iterations	Number of Function and Gradient Calculations	Number of Iterations	Number of Function and Gradient Calculations
100	862	2198	281	817	295	1005
200	1354	3436	433	1097	349	1180
300	1704	4356	617	1438	490	1257
400	1913	4909	816	1848	623	1561
500	2167	5581	879	1946	454	1339
600	2315	5967	992	2173	649	1572
700	2471	6397	1154	2511	563	1540
800	2576	6683	1333	2888	679	1724
900	2683	6969	1441	3105	802	1934
1000	2765	7184	1601	3441	804	1951

shows the results of function f_EX minimizing. Best results are given in bold.

Despite the fact that the minimum point is non-degenerate, BFGS performs worse due to its slow progress along the ravine bottom. The HY_g algorithm spends significant time reaching the minimum in the neighborhood of the minimum. In the HY_XS algorithm, the minimization stage in the minimum region is completed more quickly by using conjugate directions. Here, HY_XS, which combines the ability to more efficiently progress along the bottom of a curvilinear ravine and the conjugacy of descent vectors, proves more effective than other algorithms.

6.4. Non-Quadratic Function

The following function has topological similarity of level surfaces to the quadratic function.

$$f_{Q^2}(x,[a\mathit{max}])={\left({\sum }_{i=1}^n{a}_i{x}_i^2\right)}^2, a_i=a {\mathit{max}}^{{\displaystyle \frac{i-1}{n-1}}}.$$

(48)

Starting point is $x_0=(1, 1, \dots , 1)$. Stopping criterion is $f(x^k)-f^{*}\le 10^{-10}$. This is equivalent to reducing the quadratic term ${\sum }_{i=1}^n{a}_i{x}_i^2$ to a value smaller than 10⁻⁵.

Table 6. Results of function f_Q²(x, [a_max = 10⁴]) minimization.

n	BFGS		HY_g		HY_XS
	Number of Iterations	Number of Function and Gradient Calculations	Number of Iterations	Number of Function and Gradient Calculations	Number of Iterations	Number of Function and Gradient Calculations
100	1028	2639	141	381	171	544
200	1848	4787	224	565	201	599
300	2511	6553	297	701	234	646
400	3056	7982	359	814	337	696
500	3527	9232	420	933	346	714
600	3973	10,429	474	1040	355	732
700	4335	11,378	520	1120	365	752
800	4644	12,204	567	1209	378	778
900	4951	13,034	611	1286	390	802
1000	5204	13,697	653	1354	402	826

demonstrates the results of function f_Q^2 minimization. Best results are given in bold.

Here, a low convergence rate of the BFGS method is observed. As the method approaches a minimum, the Hessian elements tend to zero. Consequently, the elements of the inverse matrix increase. In Ref. [17], it was noted that as the matrix of second derivatives in the surveyed space grows, the approximated matrices in quasi-Newton methods also grow. This complicates the exit to the unexplored subspace, which slows the convergence rate. In the HY_g and HY_XS algorithms, on the contrary, the possibility of entering the unexplored subspace always increases, which explains the advantage of these algorithms for this function. The results for the HY_XS algorithm confirm the advantages of using the metric in the conjugate gradient method.

6.5. Function with Scaling by Variables

This function used additional variables c_i to change the scales a_i for each variable.

$$f_{abc}(x,[a\mathit{max},b\mathit{max}])={\displaystyle \frac{1}{2}}\sum _{i=1}^n{a}_i{c}_i{x}_i^2, a_i=a {max}^{{\displaystyle \frac{i-1}{n-1}}}, c_i={\displaystyle \frac{b max}{b_i}}\left({\displaystyle \frac{x_i^2}{1+x_i^2}}\right)+b_i\left(1-{\displaystyle \frac{x_i^2}{1+x_i^2}}\right),b_i=b {max}^{{\displaystyle \frac{i-1}{n-1}}}$$

This function near the extremum will have the form

$$f_{abc}(x,[a\mathit{max},b \mathit{max}])\approx {\displaystyle \frac{1}{2}}{\sum }_{i=1}^n{a}_i{b}_i{x}_i^2.$$

Far from the extremum, we obtain a function in which the coefficients are used in reverse order:

$$f_{abc}(x,[a\mathit{max},b \mathit{max}])\approx {\displaystyle \frac{1}{2}}{\sum }_{i=1}^n{a}_i{\displaystyle \frac{b max}{b_i}}{x}_i^2.$$

Changes in the scales of the coefficients according to the presented limiting variants of the function can be represented by the following transition:

$$a_i{\displaystyle \frac{b max}{b_i}}\to a_i{b}_i.$$

It follows that the first coefficient decreases by a factor of $b max$, and the last coefficient increases by a factor of $b max$. The starting point is $x_0=(100, 100, \dots , 100)$. The stopping criterion is $f(x^k)-f^{*}\le 10^{-10}$.

Table 7. Results of function f_abc(x, [a_max = 10⁴, b_max = 10³]) minimization.

n	BFGS		HY_g		HY_XS
	Number of Iterations	Number of Function and Gradient Calculations	Number of Iterations	Number of Function and Gradient Calculations	Number of Iterations	Number of Function and Gradient Calculations
100	1819	4735	502	1407	587	1635
200	2992	7450	716	2033	763	2016
300	4004	10,030	922	2430	956	2498
400	4483	11,193	1154	2987	1124	2856
500	4961	12,371	1431	3651	1262	3205
600	5263	13,130	1397	3862	1422	3590
700	5577	13,949	1554	4241	1526	3842
800	5850	14,737	1858	4760	1677	4189
900	6026	15,107	1917	4949	1802	4489
1000	6154	15,439	2124	5376	1914	4783

demonstrates the results of function f_abc minimization. Best results are given in bold.

The BFGS method performs the worst on this function. This is due to the high variability of the Hessian. Here, the space scanning strategy with excludes previously explored subspaces, which is used in the HY_g and HY_XS algorithms, proves suitable. The use of vector conjugacy in the HY_XS algorithm has little effect on the algorithm’s performance compared to the HY_g method.

It should be noted that the metrics in QNM and in RSM have different meanings. There are obstacles to good convergence in QNM: the first is a rapid change in the directions of the Hessian eigenvectors, which is observed in tests, Table 2 and Table 3; the second is a rapid change in the scales of variables (Table 6 and Table 7). Another case of poor convergence is the tendency of the second derivatives to zero as they approach the minimum (Table 5 illustrates this case).

The following

Table 8. Results of function minimization, n = 1000.

Function	BFGS		HY_g		HY_XS
	Number of Iterations	Number of Function and Gradient Calculations	Number of Iterations	Number of Function and Gradient Calculations	Number of Iterations	Number of Function and Gradient Calculations
fQ, amax = 104	495	1093	912	1884	482	1032
fQ, amax = 108	1026	2608	2620	5996	1813	4067
fE, x01	236	750	1042	2457	36	104
fE, x02	2631	6932	1540	3697	248	817
fEX	2765	7184	1601	3441	804	1951
fQ^2	5204	13,697	653	1354	402	826
fabc	6154	15,439	2124	5376	1914	4783

summarizes the results of minimization for all functions with n = 1000.

Let us draw the main conclusions based on the results of Table 8.

• The BFGS method significantly outperforms the HY_g and HY_XS algorithms in ill-posed problems, where the Hessian does not undergo significant changes depending on the position in the minimization space. This thesis is confirmed by the data for the function f_Q with a_max = 10⁸. For such a function, BFGS manages to construct a suitable coordinate transformation, after which the algorithm’s convergence rate significantly increases. In such situations, the HY_XS method has advantages over the HY_g algorithm due to the conjugate gradient method.
• Function f_E has an ellipsoidal ravine and a degenerate minimum point. From the first starting point, with a steeper ravine than from the second starting point, the main obstacle is the degeneracy of the minimum. Algorithms BFGS and HY_XS, which use the properties of a quadratic function, complete the minimization stage in the extremum region significantly faster than HY_g. As the degeneracy of the ravine increases, BFGS’s progress along the ravine becomes slow due to the high variability of the Hessian. Algorithms HY_g and HY_XS are more successful here, and the properties of the conjugate gradient method of HY_XS lead to a significant acceleration of the convergence rate. A similar situation arises when minimizing the function f_EX.
• The level surfaces of the function f_Q^2 are typologically equivalent to the level surfaces of a quadratic function. In this situation, based on an analysis of the data in Table 7, we can conclude that the metric transformation in algorithms HY_g and HY_XS is more effective than the metric transformation of the quasi-Newton method.
• It can be seen that the overheads of the HY_g method are reduced by approximately 2 times compared to the quasi-Newton method, and the overheads of the HY_XS method were 3.5 times lower on average than those of the quasi-Newton method. Overall, the computational experiment confirms the acceleration properties of the algorithms HY_g and HY_XS, obtained by using the metric transformation (14), and their applicability alongside quasi-Newton methods in solving complex, ill-conditioned minimization problems.

7. Conclusions

The conditionality of the minimization problem determines the spread of the isosurface elongations in different directions, which in turn determines the complexity of the problem’s solution. In minimization practice, it is often possible to reduce the isosurface elongations through a linear coordinate transformation, thereby increasing the convergence rate of the gradient method used in the new coordinate system. For strongly convex functions with a Lipschitz gradient, such a coordinate transformation reduces the difference between the strong convexity and Lipschitz constants. If the function is twice differentiable, such a transformation reduces the spread of the Hessian’s eigenvalues.

The noted linear transformation of coordinates eliminates the linear background that worsens the convergence of the gradient method. The property of eliminating such background is possessed by Newton’s method, quasi-Newton methods, and subgradient methods with a change in the space metric. In this work, we proved that algorithm HY_g, where the gradient method acquires accelerating abilities due to the use of the metric transformation (14), has similar properties. The qualitative nature of the convergence rate estimates for algorithm HY_g and Newton’s method coincide.

Taking into account the efficiency of transformation (14), we propose the conjugate gradient method HY_XS, also based on this transformation. The equivalence of this method to the conjugate gradient method on quadratic functions is proven. To evaluate the efficiency of the proposed method on test functions, we compare it with the quasi-Newton BFGS and the algorithm HY_g.

The test results demonstrate the efficiency of algorithm HY_XS. On almost all test problems, its convergence rate is significantly higher than that of the HY_g method. A comparison with the quasi-Newton method shows that method HY_XS is more efficient in the case of a high degree of variability of the Hessian matrices of the function being minimized.

A computational experiment shows that methods HY_g and HY_XS are effective in solving ill-conditioned problems with complex curvilinear ravines and unstable characteristics of the second derivatives. The main conclusion is that these methods, along with quasi-Newton methods, are applicable to solving problems of minimizing smooth functions with a high degree of conditioning.

The present study was conducted on smooth functions. It is probably of interest to extend these metric transformations to increase the convergence rate of the method on non-smooth functions. It is clear, purely speculatively, that by converting the metric, the elongation of the level surfaces in different directions can be equalized in the non-smooth case. However, we retain a detailed study of this issue for future research.