An Efficient Limited Memory Multi-Step Quasi-Newton Method

Abstract

This paper is dedicated to the development of a novel class of quasi-Newton techniques tailored to address computational challenges posed by memory constraints. Such methodologies are commonly referred to as “limited” memory methods. The method proposed herein showcases adaptability by introducing a customizable memory parameter governing the retention of historical data in constructing the Hessian estimate matrix at each iterative stage. The search directions generated through this novel approach are derived from a modified version closely resembling the full memory multi-step BFGS update, incorporating limited memory computation for a singular term to approximate matrix–vector multiplication. Results from numerical experiments, exploring various parameter configurations, substantiate the enhanced efficiency of the proposed algorithm within the realm of limited memory quasi-Newton methodologies category.

1. Introduction

Unconstrained Optimization is concerned with the minimization of a specific objective function as follows:

minimize $f\left(x\right)$, where $f:$ $R^n$ $\to$ $R$.

The aforementioned problem can be solved by employing a class of methods referred to as the quasi-Newton techniques for unconstrained optimization. Only the function and its first derivatives are required for quasi-Newton (QN) methods [1]. The Hessian does not need to be available or even coded. To integrate updates in the function and the corresponding gradient, an approximation matrix to the actual Hessian is retained and updated across the iterations.

Given $B_i$, the current Hessian approximation, a new Hessian estimation $B_{i+1}$ needs to be constructed, for the new solution estimate $x_{i+1}$. To find $B_{i+1}$, we can utilize the Taylor series of order one to the gradient vector about the point $x_{i+1}$ to obtain the following relation (known as the Secant equation) [2,3,4,5,6]:

$$B_{i+1}{s}_i=y_i,$$

(1)

where

$$s_i=x_{i+1}-x_i,$$

and

$$y_i=g_{i+1}-g_i,$$

where $g_i\equiv \mathrm{g}\left(x_i\right), \mathrm{i}\mathrm{s} \mathrm{t}\mathrm{h}\mathrm{e} \mathrm{g}\mathrm{r}\mathrm{a}\mathrm{d}\mathrm{i}\mathrm{e}\mathrm{n}\mathrm{t} \mathrm{a}\mathrm{t} \mathrm{i}\mathrm{t}\mathrm{e}\mathrm{r}\mathrm{a}\mathrm{t}\mathrm{e} x_i$. The computed Hessian matrix approximations must satisfy (1). To estimate the next Hessian approximation matrix $B_{i+1}$ using the current matrix $B_i$ and the vectors $s_i$ and $y_i$, an update of the form $B_{i+1}=$ $B_i+C_i$ is applied, where $C_i$ is a correctional update matrix. It may be preferable to use $H_{i+1}=H_i+D_i$, where $D_i$ is an update term and $H_{i+1}=B_{i+1}^{-1}$. The update can be of rank one or two. A widely used rank two correction formula is referred to as the BFGS formula. The BFGS is given by

$$B_{i+1}^{BFGS}=B_i+\frac{y_i{y}_i^T}{y_i^T{s}_i}-\frac{B_i{s}_i{s}_i^T{B}_i}{s_i^T{B}_i{s}_i},$$

and the corresponding inverse matrix as

$$H_{i+1}^{BFGS}=H_i+\left[1+\frac{{\mathrm{y}}_i^T{H}_i{y}_i}{y_i^T{s}_i}\right]\frac{s_i{s}_i^T}{y_i^T{s}_i}-\frac{{\mathrm{s}}_i{\mathrm{y}}_i^T{H}_i+H_i{y}_i{s}_i^T}{y_i^T{s}_i}.$$

(2)

Reported numerical outcomes show that this formula outperforms other updating formulas, particularly when the line search is inexact. BFGS is regarded as a standard update formula [3,5,7,8].

The search direction for the standard quasi-Newton methods is computed using

$$B_i{p}_i=-g_i.$$

2. Literature Review

Limited memory QN methods (L-BFGS) have gained considerable prominence in optimization due to their ability to handle large-scale problems efficiently, especially with memory constraints. These iterative methods are mostly classified as quasi-Newton techniques [9,10,11,12]. The versatility of L-BFGS methods is evident in their application across various domains. In machine learning, they are integral to training support vector machines (SVMs), optimizing logistic regression models, and fine-tuning neural networks. Their memory-efficient design makes them invaluable when working with large datasets. Additionally, L-BFGS plays a pivotal role in structural optimization, a field where finite element analysis involves high-dimensional design spaces. In computational chemistry, L-BFGS aids in optimizing molecular structures, facilitating breakthroughs in drug discovery and materials science.

Nevertheless, L-BFGS distinguishes itself from the classical quasi-Newton methods by storing only a limited number of recent iterations data, often referred to as memory pairs ($s_i$, $y_i$), where $s_i$, represents the alteration in the optimization variable, and $y_i$ denotes the change in the gradient (see (1)). These memory pairs are instrumental in updating the approximate Hessian matrix, achieving a balance between computational efficiency and optimization accuracy.

In the past decade, L-BFGS has seen significant methodological advancements and the emergence of new variants tailored to address specific optimization challenges. For example, L-BFGS-B extends the method to handle bound-constrained optimization problems effectively [10]. Innovations like L-BFGS-TF and L-BFGS++ have been introduced to enhance scalability and convergence properties. These methodological innovations have substantially widened the scope of L-BFGS applications, encompassing areas such as machine learning, deep learning, and numerical simulations.

Limited memory BFGS algorithms often derive using the updated identity for the inverse Hessian given as [10]

$$H_{i+1}=\left(I-{\rho }_i{s}_i{y}_i^T\right)H_i\left(I-{\rho }_i{y}_i{s}_i^T\right)+{\rho }_i{s}_i{s}_i^T,$$

where $s_i$ and $y_i$ are as in (1) and ${\rho }_i=1/s_i^T{y}_i$.

For a given identity matrix I, a sequence of vectors $q_{i-m},\dots .., q_m$ is defined such that $q_i\equiv \left(I-{\rho }_i{s}_i{y}_i^T\right)q_{i+1}$. Then, the quantities $q_i$ and $q_{i+1}$ are recursively computed. For more details see [10].

Next, we present some successful limited-memory BFGS formulas. These formulas are all variations of the standard L-BFGS formula, and they are designed to enhance the numerical behavior of the L-BFGS algorithm for certain types of problems. One of the well-known limited memory methods is the Liu–Nocedal formula [9]. The updated Hessian matrix is the previous Hessian matrix updated with a correction term. The correction term is a weighted sum of the two recent steps and gradient differences, where the weights are chosen to make certain that the updated Hessian matrix is positive-definite. The formula is given as

$$H_i=H_{i-1}+\frac{s_{i-1}{s}_{i-1}^T}{s_{i-1}^T{y}_{i-1}}-\frac{H_{i-1}{y}_{i-1}{y}_{i-1}^T{H}_{i-1}}{y_{i-1}^T{H}_{i-1}{y}_{i-1}}+\frac{{\rho }_i{s}_{i-2}{s}_{i-2}^T}{s_{i-2}^T{y}_{i-2}}-\frac{{\rho }_i{H}_{i-1}{y}_{i-2}{y}_{i-2}^T{H}_{i-1}}{y_{i-2}^T{H}_{i-1}{y}_{i-2}}$$

Another successful formula is the Byrd–Nocedal–Schnabel formula [13]. The correction term is a weighted sum of the latest m steps and gradient differences, where the weights are chosen to ascertain that the newly computed Hessian matrix is positive-definite. The formula is defined as

$$H_i=H_{i-1}+\frac{s_{i-1}{s}_{i-1}^T}{s_{i-1}^T{y}_{i-1}}-\frac{H_{i-1}{y}_{i-1}{y}_{i-1}^T{H}_{i-1}}{y_{i-1}^T{H}_{i-1}{y}_{i-1}}+{\sum }_{j=i-m}^{i-1}\frac{{\rho }_i{s}_j{s}_j^T}{s_j^T{y}_j}-{\sum }_{j=i-m}^{i-1}\frac{{\rho }_i{H}_{i-1}{y}_j{y}_j^T{H}_{i-1}}{y_j^T{H}_{i-1}{y}_j}.$$

(3)

A variation of (3) is the Zhu–Byrd–Nocedal formula [14]. The correction component is a weighted sum of the latest m steps and gradient differences, where the weights are also chosen to make certain that the newly computed Hessian matrix is positive-definite. Additionally, the correction term includes a term that is designed to improve the behavior of the L-BFGS algorithm for problems with non-smooth objective functions [15]. The formula is defined by

$$H_i=H_{i-1}+\frac{s_{i-1}{s}_{i-1}^T}{s_{i-1}^T{y}_{i-1}}-\frac{H_{i-1}{y}_{i-1}{y}_{i-1}^T{H}_{i-1}}{y_{i-1}^T{H}_{i-1}{y}_{i-1}}+{\sum }_{j=i-m}^{i-1}\frac{{\rho }_i{s}_j{s}_j^T}{s_j^T{y}_j}-{\sum }_{j=i-m}^{i-1}\frac{{\rho }_i{H}_{i-1}{y}_j{y}_j^T{H}_{i-1}}{y_j^T{H}_{i-1}{y}_j}+\frac{{\rho }_i{s}_{i-m}{s}_{i-m}^T}{s_{i-m}^T{y}_{i-m}}-\frac{{\rho }_i{H}_{i-1}{y}_{i-m}{y}_{i-m}^T{H}_{i-1}}{y_{i-m}^T{H}_{i-1}{y}_{i-m}}.$$

(4)

In (3) and (4), the parameter m controls the number of past update terms that are used to update the Hessian matrix. A larger value of m will result in a more precise approximation of the Hessian matrix, closer to that of the standard QN methods.

Another idea, presented in [16], focuses on applying regularized Newton methods in a versatile category of unconstrained optimization algorithms. The method they propose has the potential to combine favorable characteristics from both approaches. The primary emphasis is on the integration of regularization with limited memory quasi-Newton methods, leveraging the unique structure inherent in limited memory algorithms. The paper explores an alternative globalization technique, akin to the less familiar siblings of line search and trust-region methods, known as regularized Newton methods. The derived methods are referred to as regularized quasi-Newton methods. The methods utilize the relationship:

$${(B}_i+{\mu }_{i}I)p_i=-g_i,$$

which is used in the computation of the search direction $p_i$, instead of the one used by the standard quasi-Newton methods. The parameter ${\mu }_i$ serves as the regularization parameter. The derived methods integrate features from both line search and trust region techniques and update a limited memory version of the Hessian matrix estimate. The method displays good behavior overall though it involves the solution of a system of linear equations at each iteration, which increases the time complexity of the algorithm.

A recent paper also introduces a limited memory quasi-Newton type method for unconstrained multi-objective optimization problems [17], which is suitable for large-scale scenarios. The proposed algorithm approximates the convex combination of the Hessian matrices of the objective function using a positive definite matrix. The update formula for the approximation matrix is an extension of the one used in the L-BFGS method for scalar optimization [18]. The method is well defined even in the nonconvex case and exhibits global and R-linear convergence to Pareto optimality for twice continuously differentiable strongly convex problems. In the method derived in [17], the matrix used to update the inverse Hessian approximation is given by

$$H_{i+1}=\left(I-{\rho }_i{s}_i{u}_i^T\right)H_i\left(I-{\rho }_i{u}_i{s}_i^T\right)+{\rho }_i{s}_i{s}_i^T,$$

where ${\rho }_i=1/s_i^T{u}_i$ and $u_i$ is the summation of the previous $y_i$ dictated by the number of vectors retained in memory. Those $y$ values are multiplied by optimal Lagrange multipliers needed to achieve global convergence.

The performance of the algorithm is evaluated through computational comparisons with state-of-the-art Newton and quasi-Newton approaches in multi-objective optimization. The results show that the proposed approach is generally efficient. While testing the method, it failed to converge on some of our test functions, but it did well on others and managed to find global minimizers.

The paper in [19] presents a new numerical method for solving large-scale unconstrained optimization problems, derived from a modified BFGS-type update. The update formula is extended to the framework of a limited memory scheme. The update utilized in that paper takes the form

$$H_{i+1}=\left(I-{\rho }_i{s}_i{u}_i^T\right)H_i\left(I-{\rho }_i{u}_i{s}_i^T\right)+{\rho }_i{s}_i{s}_i^T,$$

where ${\rho }_i=1/s_i^T{u}_i$ and $u_i=y_i+{\gamma }_i{s}_i$, for some scalar ${\gamma }_i$ to ensure convergence. The paper discusses the global convergence and convergence rate of the algorithm with weak Wolfe–Powell line search.

A modified q-BFGS algorithm is proposed for nonlinear unconstrained optimization problems [20]. It uses a simple symmetric positive definite matrix and a new q-quasi-Newton equation to build an approximate q-Hessian. The method preserves global convergence properties without assuming the convexity of the objective function. Numerical results show improvement over the original q-BFGS method. A limited memory quasi-Newton method is introduced for large-scale unconstrained multi-objective optimization problems. It approximates the convex combination of the Hessian matrices of the objectives and exhibits global and R-linear convergence to Pareto optimality. Empirical comparisons demonstrate the efficiency and effectiveness of the proposed approach [21]. A new L-BFGS method with regularization techniques is proposed for large-scale unconstrained optimization. It guarantees global convergence and is robust in terms of solving more problems [4]. The momentum-accelerated quasi-Newton (MoQ) method approximates Nesterov’s accelerated gradient as a linear combination of past gradients, and its performance is evaluated on a function approximation problem.

3. A New Multi-Step Limited Memory Method

In this section, we devise a new QN-limited memory method inspired by the methods in (3) and (4). The method is characterized by its utilization of more of already computed past data to better the quality of the generated Hessian (inverse) approximations.

In the basic Secant equation, a straight line L is utilized to find a new iteration x_i+1 given the iterate x_i, whereas higher-order interpolants are used in the multi-step approaches [22,23]. The main advantage of using polynomials is that they exploit more of the already computed data in the matrix update rather than simply discarding them. This is expected to result in better Hessian approximations.

Multi-step methods generally update the Hessian estimates to satisfy the form

$$B_{i+1}{u}_i=w_i,$$

(5)

for which the classical Secant relation simply refers to the choices $u_k=s_k$ and $w_k=y_k$, respectively. One of the alternatives to $u_i$ and $w_i$ proposed in [24] is

$$u_i=s_i-{\mu }_{i-1}{s}_{i-1} \mathrm{and} {w_i=y}_i-{\mu }_{i-1}{y}_{i-1},$$

(6)

where ${\mu }_{i-1}={\delta }^2/(1+2\delta )$ and $\delta =‖s_i‖/‖s_{i-1}‖$.

Thus, Equation (5) becomes:

$$H_{i+1}\left(y_i-{\mu }_{i-1}{y}_{i-1}\right)=s_i-{\mu }_{i-1}{s}_{i-1}$$

Concentrating on the BFGS updating method (2), the inverse Hessian approximation at iteration $i$ may be expressed after manipulation as

$$H_{i+1}=H_i+{\left({\mathrm{u}}_i^T{\mathrm{w}}_i\right)}^{-1}\left\{\left[1+\frac{{\mathrm{w}}_i^T{r}_i}{{\mathrm{u}}_i^T{\mathrm{w}}_i}\right]{\mathrm{u}}_i{\mathrm{u}}_i^T-u_i{r}_i^T-r_i{u}_i^T\right\}$$

(7)

where $r_i=H_i{w}_i.$

Contained memory techniques are developed using the identity

$$p_{i+1}=-H_{i+1}{g}_{i+1}$$

which is equivalent to (using (7))

$$p_{i+1}=p_i-{H_{i}y}_i-{\left({\mathrm{u}}_i^T{\mathrm{w}}_i\right)}^{-1}\left\{\left(\left[1+\frac{{\mathrm{w}}_i^T{r}_i}{{\mathrm{u}}_i^T{\mathrm{w}}_i}\right]u_i^T{g}_{i+1}-r_i^T{g}_{i+1}\right)u_i+\left(u_i^T{g}_{i+1}\right)r_i\right\}.$$

(8)

However, the problem in using (7) is that of computing $r_i$ and ${H_{i}y}_i$ without actually storing $H_i.$ The simplest option would be to make the choice $H_i=I$ at each iteration and, thus, $r_i=y_i$ and ${H_{i}y}_i$, in that case, reduces to $y_i$. However, this is a numerically improper choice since it abandons previous information collected in the updated matrices. Another proposal is considered here and that is to retain some approximation of the vector $r_i$ and correct every cycle. This can be performed using:

Start with $r_0=H_0{w}_0$ $=w_0$ (since $H_0=I$)

$$r_1=H_1{w}_1=w_1+{\left(u_0^T{w}_0\right)}^{-1}\left\{\left(\left[1+\frac{w_0{w}_0^T}{u_0^T{w}_0}\right]u_0^T{w}_1-\left(w_0^T{w}_1\right)\right)u_0-\left(u_0^T{w}_1\right)w_0\right\}$$

(9)

and, recurrently, for any $i$, we use

$$r_i=H_{i-1}{w}_i+{\left(u_{i-1}^T{w}_{i-1}\right)}^{-1}\left\{\left(\left[1+\frac{w_{i-1}^T{r}_{i-1}}{u_{i-1}^T{y}_{i-1}}\right]u_{i-1}^T{w}_i-\left(r_{i-1}^T{w}_i\right)\right)u_{i-1}-\left(u_{i-1}^T{w}_i\right)r_{i-1}\right\}.$$

(10)

Relation (10) can then be utilized in (8) to compute the next direction vector. Still, the term $H_{i-1}{w}_i$ in (10) is not accessible. The obvious option is to set, $r_{i-1}$ to $w_{i-1}$ in (8). Such a choice is expected to affect negatively the updated matrix. Thus, our goal next is to build an expression for both $H_{i-1}{w}_i$ and ${H_{i}y}_i$ to complete the derivation. By doing so, we will then have derived a method that is expected to be computationally superior to that of setting $H_i$ to be just the identity matrix in (8). This is because more of the previously accumulated information is utilized in updating the matrix. Therefore, the preference computationally is to approximate the aforementioned matrix vector products as will be shown next.

To proceed with the derivation, it should be noted that for a diagonal matrix $H_0$, the next matrix in recurrence can be expressed as

$$H_1=H_0+U\left(H_0,u_0,w_0\right)$$

and

$$\begin{gathered} H_2=H_0+U\left(H_0,u_0,w_0\right)+U\left(H_1,u_1,w_1\right) \\ {H}_3=H_0+U\left(H_0,u_0,w_0\right)+U\left(H_1,u_1,w_1\right)+U\left(H_2,u_2,w_2\right)\dots . \mathrm{and} \mathrm{so} \mathrm{on} \end{gathered}$$

for vectors u and w, as in (5).

Let $\eta$ refer to the upper limit of the correction terms $U$ that can be saved. Because $H_0$ is diagonal, the limit count of $n$-vectors that can be utilized to build the matrix approximation is $2\eta +1.$ We have hit our storage limit once $H_{\eta }$ is produced. In this case, newer vectors start replacing older ones:

$$H_{\eta }=H_0+U\left(H_0,u_0,w_0\right)+\dots +U\left(H_{\eta -1},u_{\eta -1},w_{\eta -1}\right).$$

This idea was influenced by Nocedal’s algorithm [10]. Nocedal’s technique is founded on the idea that the latest data should be provided in the update, and replacing the oldest information is one compelling option. However, there is no assurance that the computed matrices will be positive-definite. Loss of positive-definiteness leads to search directions that are not necessarily descent, thus threatening the method’s convergence. To avoid this problem, the standard BFGS updating formula is expressed in the form [2]:

$$H_{i+1}=V_i^T{H}_i{V}_i+{\rho }_i{u}_i{u}_i^T,$$

(11)

where ${\rho }_i={\left(u_i^T{w}_i\right)}^{-1}$ and $V_i=I+{\rho }_i$ $w_i{u}_i^T$, where ${\rho }_i$ is a parameter that is chosen to ascertain that the newly computed Hessian matrix is positive-definite and m is a parameter that controls the number of past updates that are used to update the Hessian matrix.

Thus, given the above-proposed strategy, Nocedal suggests a form of the update as follows:

$$\begin{array}{l}\mathrm{For} i+1\le \eta ,\\ H_{i+1}=V_i^T{V}_{i-1}^T\dots V_0{H}_0\dots V_{i-1}{V}_i\\ +V_i^T\dots V_1^T{\rho }_0{u}_0{u}_0^T{V}_1\dots V_i+\dots \\ +V_i^T\dots V_{i-1}^T{\rho }_{i-2}{u}_{i-2}{u}_{i-2}^T{V}_{i-1}\dots V_i\\ +V_i^T{\rho }_{i-1}{u}_{i-1}{u}_{i-1}^T{V}_i+{\rho }_i{u}_i{u}_i^T\\ \mathrm{For} i+1>\eta ,\\ H_{i+1}=V_i^T{V}_{i-1}^T\dots V_{i-\eta +1}{H}_0\dots V_{i-\eta +1}{V}_{i-1}{V}_i\\ +V_i^T..V_{i-\eta +2}^T{\rho }_{i-\eta +1}{u}_{i-\eta +1}{u}_{i-\eta +1}^T\dots V_i+\dots \\ +V_i^T..V_{i-1}^T{\rho }_{i-2}{u}_{i-2}{u}_{i-2}^T{V}_{i-1}\dots V_i\\ +V_i^T{\rho }_{i-1}{u}_{i-1}{u}_{i-1}^T{\mathrm{V}}_i++{\rho }_i{u}_i{u}_i^T.\end{array}$$

(12)

Back to our problem, we adopt Nocedal’s approach with a suitably chosen small $\eta$ to build an approximation to the product $H_{i-1}{w}_i$ in (8). For example, for $\eta =1$, we have

$$\begin{array}{l} H_{i-1}{w}_i=V_{i-2}^T{H}_0{V}_{i-2}{w}_i+\left({\rho }_{i-1}{u}_{i-2}^T{w}_i\right)u_{i-2}\\ =H_0{w}_i-\left[{\rho }_{i-2}{u}_{i-2}^T{w}_i\right]H_0{w}_{i-2}\\ +{\rho }_{i-2}^2\left[\left(w_{i-2}^T{H}_0{w}_{i-2}\right)\left(u_{i-2}^T{w}_i\right)-\left(u_{i-2}^T{H}_0{w}_i\right)\left(u_{i-2}^T{w}_{i-2}\right)u_{i-2}\right]\\ +\left({\rho }_{i-2}{s}_{i-2}^T{w}_i\right)u_{i-2}.\end{array}$$

Similarly, to compute the product ${H_{i}y}_i$ the following may be used:

$$\begin{array}{l}{ H}_i{y}_i=V_{i-1}^T{H}_0{V}_{i-1}{y}_i+\left({\rho }_{i-1}{u}_{i-1}^T{y}_i\right)u_{i-1}\\ =H_0{y}_i-\left[{\rho }_{i-1}{u}_{i-1}^T{y}_i\right]H_0{w}_{i-1}\\ +{\rho }_{i-1}^2\left[\left(w_{i-1}^T{H}_0{w}_{i-1}\right)\left(u_{i-1}^T{y}_i\right)-\left(u_{i-1}^T{H}_0{y}_i\right)\left(u_{i-1}^T{w}_{i-1}\right)u_{i-1}\right]\\ +\left({\rho }_{i-1}{s}_{i-1}^T{y}_i\right)u_{i-1}.\end{array}$$

In general, the following is employed to build the above products for a given $\eta$ and an appropriate replacement of the matrix/vectors involved (Algorithm 1):

Algorithm 1 MSCBFGS

0. LET $k=i-1;$ 1. IF $k\le \eta \mathrm{THEN} \left(incr=0;bound=k\right)$        ELSE $\left(incr=k-\eta ;bound=\eta \right)$ 2. $q_{bound}=w_i$ 3. FOR $l=bound-1\dots 0$       $\{j=l+incr$;       ${\alpha }_l={\rho }_j{u}_j^T{q}_{l+1}$;       $q_l=q_{l+1}-{\alpha }_l{w}_j$;} 4. $z_0=H_0{q}_0$ 5. FOR $l=0,1,\dots ,bound-1;$        $\{j=l+incr;$        ${\beta }_j={\rho }_j{w}_j^T{z}_l;$        $\mathrm{IF} \left(k\succ \eta \right) \mathrm{THEN} z_{l+1}=z_l+u\left({\alpha }_l-{\beta }_j\right)$

$$\mathrm{ELSE} z_{l+1}=z_l+u\left({\alpha }_l-{\beta }_l\right)\}.$$ (13)

Our multi-step limited memory BFGS (MSCBFGS) algorithm can be stated as: Start with an approximate point x0 to the solution Start with H0 = I (or a scalar multiple of it) i ← 0 Find $g_0=\mathrm{g}\left(x_0\right)$ Repeat Step 1. Let $p_i=-H_i{g}_i$ Step 2. Minimize $f\left({\mathrm{x}}_i+\alpha p_i\right)$ where $\alpha \in R$ to find a step length ${\alpha }_i$ along $p_i$. To make certain that the obtained search direction is adequately descent, the parameter ${\alpha }_i$ is required to satisfy the strong Wolfe conditions [25] $$f(x_k+{\alpha }_k{d}_k)\le f\left(x_k\right)+\delta {\alpha }_k{g}_k^T{d}_k$$ And $$d_k^{T}g\left(x_k+{\alpha }_k{d}_k\right)\ge \sigma d_k^T{g}_k,$$ where $0<\mathsf{\delta }<\mathsf{\sigma }<1$ Step 3. $x_{i+1}=x_i+{\alpha }_i{p}_i$ Step 4. Compute $s_i=x_{i+1}-x_i y_i=g_{i+1}-g_i, w_i=y_i-{\mu }_i{y}_{i-1} \mathrm{and}$ $u_i=s_i-{\mu }_i{s}_{i-1}$. Step 5. If ${{\mathrm{w}}_i^T\mathrm{u}}_i>0$, then compute $H_{i+1}^{BFGS}$ using the recurrence in (8). Step 6. $i=i+1$ until ${\parallel g_i\parallel }_2<\epsilon$, (where $\epsilon \in R$ is a convergence parameter).

We now need to establish that (8) produces a descent direction vector. To do so, it suffices to show that the matrix used in the computation of (8) is positive-definite.

Theorem 1.

The matrix $H_{i+1}$  update in (7) is positive-definite if  $H_i$ is positive-definite and if  $u_i^T{w}_i>0.$

Proof of Theorem 1.

We first show that if $H_{i+1}$ is positive-definite, then $u_i^T{\mathrm{w}}_i>0$. From (5), it follows that

$$w_i^T{u}_i=w_i^T{H}_{i+1}{w}_i>0.$$

Now, given that $w_i^T{u}_i>0$ and $r_i=H_i{w}_i$ (from (7)), we proceed to prove that $H_{i+1}$ is positive-definite by showing that $v^T{H}_{i+1}v>0,$ for any vector $v\ne 0.$ Using (7) and (11), we have

$${v^{T}H}_{i+1}\mathrm{v}={v^{T}V}_i^T{H}_i{V}_i\mathrm{v}+{\rho }_i{\left(u_i^{T}v\right)}^2,$$

(14)

where $\rho ={\left(u_i^T{w}_i\right)}^{-1}$, $u_i=s_i-{\mu }_{i-1}{s}_{i-1}$, $V_i=I+{\rho }_i$ $w_i{u}_i^T$, and ${\rho }_i$ is a parameter that is chosen to make certain that the newly computed Hessian matrix is positive-definite. Since $H_i$ is positive-definite, $V_i^T{H}_i{V}_i$ is also positive-definite. The second term is also positive. Thus, the whole expression in (14) is positive-definite. □

It should be noted here that conditions (13) ensure only that $s_i^T{y}_i>0$ while there are no guarantees for $w_i^T{u}_i$. In our implementations, the condition $w_i^T{u}_i>0$ is tested, and if it is not satisfied, we revert to using the classical Secant update at that specific iteration with $u_i=s_i$ and $w_i=y_i$.

As the new algorithm is not stripping away the main structure of the BFGS update, the derived method possesses the same convergence properties as those that belong to the same class such as those in [10,26]. The R-linear convergence rate of the MSCBFGS method is proved under the assumption that the objective function is strongly convex. However, the method can also converge R-linearly for non-convex objective functions. The convergence rate of the MSCBFGS method can be affected by the choice of the memory parameter η. A larger value of η can lead to a faster convergence rate, but it can also make the method more sensitive to noise. The proof for the convergence rate is very similar to that conducted in [26].

Estimating the time complexity of the described optimization algorithm involves several factors. The number of iterations (k) in the optimization process, along with the computational costs of algorithmic components (O(f)) and convergence conditions (O(g)), collectively shape the overall time complexity. A very rough estimate of the algorithm’s time complexity could be expressed as O(k × (f + g)), considering the iterative nature of the optimization and the associated computations. However, a more precise analysis would require a detailed breakdown of specific operations and their computational costs. As a result, the per iteration complexity and storage requirement is O($\eta$n) where $\eta$ ≤ n is the size of the stored frame and n is the problem dimension, thus, reducing the O($n^2$) computational complexity and memory requirements of standard quasi-Newton methods.

The main feature of this algorithm is that it computes the search vector at each iteration using a formula that is almost the full memory multi-step BFGS version with only an approximation applied to the term $H_{i-1}{w}_i$ in (10). This is expected to produce results close to the full multi-step BFGS version, as the results reveal.

4. Numerical Results

The new method (7) is benchmarked against the methods in (3) and (4). Those are experimented on thirty distinct test problems with dimensions varying from 2 to 10,000. The total number of tested problems is 900. The functions tested are classified into four categories and can be found in [7,13,27,28,29]. The categories are as follows:

a.

Low dimension $\left(2\le n\le 15\right)$

b.

Medium dimension $\left(16\le n\le 45\right)$

c.

Moderate-High dimension $\left(46\le n\le 10000\right)$

d.

High dimension $\left(81\le n\le 10000\right)$

Each of the test problems possesses one or more of the following properties:

(a)

Non-convexity;

(b)

Global minimum;

(c)

Multi-modality;

(d)

Ill-conditioned;

(e)

Highly non-linear;

(f)

Symmetry around the global minimum;

(g)

Periodic;

(h)

Discrete optimization domain.

The functions and their properties are listed in

Table 1. Set of test problems.

Function Name (Dimension)	Properties
Rosenbrock $\left(n=2\right)$	a, b, c, d, f
Quadratic function $\left(n=2\right)$	b
Watson function $\left(3\le n\le 31\right)$	c, e
Extended Rosenbrock $(2\le n\le 10000,n$ even$)$	a, c, d
Extended Powell $(2\le n\le 10000,n$ divisible by $4)$	a, c
Penalty function I $\left(2\le n\le 10000\right)$	a, c, e
Variably dimensioned function $\left(2\le n\le 10000\right)$	a, c
Trigonometric function $\left(2\le n\le 10000\right)$	a, c, g
Modified Trigonometric function $\left(2\le n\le 10000\right)$	a, c, g
Broyden Tridiagonal function $\left(2\le n\le 10000\right)$	a, c
Discrete Boundary value function $\left(2\le n\le 10000\right)$	a, c, h
Oren and Spedicato Power function $\left(2\le n\le 10000\right)$	a, c, e
Full Set of Distinct Eigen Values Problem $\left(2\le n\le 10000\right)$	a (if all eigenvalues are negative)
Tridiagonal function $\left(2\le n\le 10000\right)$	a, c, f
Wolfe function $\left(2\le n\le 10000\right)$	a, c, e
Diagonal Rosenbrock’s function $(2\le n\le 10000,$ $n$ even$)$	c, e
Generalized Shallow function $(2\le n\le 10000,$ $n$ even$)$	c, e
Freudenstein and Roth $\left(n=2\right)$	a, c, e
Powell Badly Scaled $\left(n=2\right)$	a, d, e
Brown Badly Scaled $\left(n=2\right)$	a, d, e
Beale $\left(n=2\right)$	a, b, c
Bard $\left(n=3\right)$	a, b, c
Freudenstein and Roth $\left(n=2\right)$	a, b, c
Powell Badly Scaled $\left(n=2\right)$	a, b, c
Brown Badly Scaled $\left(n=2\right)$	a, b, c
Beale $\left(n=2\right)$	a, b, c
Bard $\left(n=3\right)$	a, b, c
Gaussian (n = 3)	B

.

Table 2. Large problems test results.

Method	Evaluations	Iterations	Time (s)	Scores
Byrd et al. [13]	25,039	22,106	426.42	32
	100%	100%	100%
Zhu and Byrd [14]	22,699	19,838	392.08	51
	90.66%	89.7%	91.95%
MSCBFGS	21,609	18,461	319.95	71
	86.30%	83.5%	75.03%

,

Table 3. Moderate high dimension problems test results.

Method	Evaluations	Iterations	Time (s)	Scores
Byrd et al. [13]	12,841	9673	3348.5	90
	100%	100%	100%
Zhu and Byrd [14]	12,343	8864	3075.9	129
	96.12%	91.64%	91.8%
MSCBFGS	11,654	8717	2980.4	162
	90.76%	90.12%	89.1%

,

Table 4. Medium-size dimension problems test results.

Method	Evaluations	Iterations	Time (s)	Scores
Byrd et al. [13]	11,745	11,061	18,645.71	66
	100%	100%	100%
Zhu and Byrd [14]	11,603	11,132	17,422.51	78
	98.79%	100.64%	93.44%
MSCBFGS	10,869	9850	13,916.57	110
	92.54%	89.06%	74.64%

and

Table 5. Small dimension problems test results.

Method	Evaluations	Iterations	Time (s)	Scores
Byrd et al. [13]	9597	5049	10,744.0	44
	100%	100%	100%
Zhu and Byrd [14]	9443	4953	10,503.0	41
	98.39%	98.11%	97.76%
MSCBFGS	9754	5180	10,779.4	26
	101.64%	102.61%	100.3%

report the results for each category.

Table 6. Overall scores for the test results for 900 problems.

Method	Evaluations	Iterations	Time (s)	Scores
Byrd et al. [13]	59,079	47,960	31,942	232
	100%	100%	100%
Zhu and Byrd [14]	56,087	44,787	31,394	299
	94.94%	93.38%	98.28%
MSCBFGS	53,887	42,209	27,996	369
	91.21%	88.01%	87.65%

reports the totals. The figures presented in each table indicate the total number of iterations, function/gradient evaluations, the timings, and the points for every algorithm. A point is granted to a method if it obtains the minimal evaluations count in solving a problem. Ties are resolved using the count of iterations. The percentages in each table indicate the improvements (or otherwise) of each method on each of the evaluation categories (iterations, evaluations, and time score) as opposed to the benchmark method of Byrd et al. A lower percentage indicate the savings obtained in comparison to the benchmark method that is assigned the full percentage. Our results for the above method seem to bias method 3 for $\eta =3$. Bigger $\eta$ have not introduced worthwhile improvements in performance over the other methods to justify the extra memory expense. Therefore, the test results reported on the new method are conducted with $\eta =3$.

The results reveal clearly that $\mathrm{M}\mathrm{S}\mathrm{C}\mathrm{B}\mathrm{F}\mathrm{G}\mathrm{S}$ (corresponding to $\eta =3)$ method is superior to the other two, especially on large problems. Figure 1, Figure 2 and Figure 3 report the results comparisons on each criterion while Figure 4 presents an overall overview of those.

The analysis of the numerical scores for the newly developed limited memory MSCBFGS method in comparison to the Byrd et al. [13] and Zhu and Byrd [14] methods reveals notable improvements in efficiency and effectiveness. In terms of evaluations, the MSCBFGS method requires only 91.21% of the evaluations compared to the Byrd et al. [13] method, indicating enhanced computational efficiency. Additionally, it achieves a further reduction in iterations, requiring only 88.01% of the iterations compared to the original method, demonstrating improved convergence properties. Moreover, the MSCBFGS method significantly reduces computation time, utilizing only 87.65% of the time compared to the Byrd et al. [13] method, while outperforming both the original method and the Zhu and Byrd method in terms of overall effectiveness, as evidenced by its higher score of 369 compared to 232 and 299, respectively. Overall, these results suggest that the newly developed MSCBFGS method offers substantial improvements in efficiency and effectiveness, making it a promising approach for optimization problems.

5. Conclusions

In conclusion, this paper has presented a contribution to the field of optimization through the development of a novel quasi-Newton method known as memory-sensitive or limited BFGS (MSCBFGS). The method has been tailored specifically to address the challenges posed by memory constraints in solving complex large-scale problems. By allowing for flexibility in the choice of a storage variable that governs the retention of past data in the formation of the new Hessian approximation, MSCBFGS exhibits adaptability and versatility.

The outcomes of extensive numerical experiments have underscored the efficacy of the MSCBFGS algorithm. These findings not only affirm the algorithm’s potential but also pave the way for its practical application in a range of memory-limited optimization tasks.

One notable advantage of the proposed method lies in its adaptability to varying memory constraints, offering a customizable parameter for controlling the retention of past data. This flexibility enables the algorithm to effectively navigate memory-limited environments, enhancing its applicability to a wide range of optimization problems.

6. Discussion

By utilizing a modified version of the full memory multi-step BFGS update, the method derived in this paper maintains a balance between computational efficiency and accuracy in approximating the Hessian matrix. However, it is crucial to acknowledge certain limitations. One such drawback is the potential sensitivity of the algorithm’s performance to the selection of parameters, which may require careful tuning for optimal results. Additionally, while the method demonstrates improved effectiveness within constrained memory contexts, its performance in comparison to alternative approaches warrants further investigation, particularly in scenarios with highly complex or nonlinear objective functions. Overall, the presented method offers promising advancements in addressing memory limitations in quasi-Newton optimization methods, yet ongoing research is essential to fully understand its capabilities and limitations in diverse optimization scenarios.

Future research can deeper investigate the optimization of the memory parameter selection process within MSCBFGS. Exploring adaptive or learning-based techniques to dynamically adjust this parameter during optimization could enhance the algorithm’s performance further, especially when taking the nature of the problem being solved into account [30].