Efficiency and Stability of a New Hybrid Unconstrained Optimization Algorithm with Quasi-Newton Updates and Higher-Order Methods

Abstract

We propose the higher-order quasi-Newton (HOQN) method, a hybrid algorithm for unconstrained optimization that combines Newtonian predictors with higher-order correctors derived from vector extensions of the Traub, Chun, and Ostrowski methods, along with quasi-Newton updates of the inverse Hessian using Broyden–Fletcher–Goldfarb–Shanno (BFGS) or Davidon–Fletcher–Powell (DFP) formulas. We demonstrate that the resulting scheme achieves cubic local convergence order, representing a substantial improvement over the superlinear convergence typical of classical quasi-Newton methods, while maintaining a cost of $O\left(n^2\right)$ per iteration. We also analyze variants that incorporate two successive quasi-Newton updates, and show that they retain the same cubic order. Numerical experiments with the benchmark functions of Himmelblau and Freudenstein–Roth confirm the theoretical convergence order and show that the hybrid variants consistently require fewer iterations than BFGS, DFP, and Symmetric Rank-One (SR1). In the case of the Booth function, given its strictly convex quadratic structure, the proposed hybrid methods reach the global minimum in just two iterations and exhibit numerical accuracy superior to that of classical quasi-Newton methods. In addition, limited-memory variants (L-HOQN) are introduced; these are evaluated during the training of a convolutional neural network on the MNIST dataset, where they achieve test accuracies exceeding $99\%$ and outperform L-BFGS and standard stochastic gradient descent (SGD) at all tested learning rates.

1. Introduction

The development of quasi-Newton methods is one of the most important lines of research in unconstrained numerical optimization, due to their ability to balance computational efficiency and accuracy in large-scale problems. These methods seek to replicate the local efficiency of the Newton method while avoiding the explicit computation of the inverse of the Hessian matrix.

Let us first consider the classical Newton method for unconstrained optimization. Let f: ${\mathbb{R}}^n\to \mathbb{R}$ be a twice continuously differentiable function, where $\nabla f\left(x\right)$ and ${\nabla }^{2}f\left(x\right)$ denote the gradient and the Hessian of f at a point $x\in {\mathbb{R}}^n$, respectively. Newton’s method generates a sequence $\left\{x^{\left(k\right)}\right\}$ of approximations to a minimizer of f according to

$$d^{\left(k\right)}=-{\left[{\nabla }^{2}f\left(x^{\left(k\right)}\right)\right]}^{-1}\nabla f\left(x^{\left(k\right)}\right),x^{(k+1)}=x^{\left(k\right)}+{\alpha }^{\left(k\right)}{d}^{\left(k\right)},$$

where $x^{\left(k\right)}$ denotes the current iterate, $d^{\left(k\right)}$ is the Newton search direction, and ${\alpha }^{\left(k\right)}>0$ is a step length obtained by means of a line search procedure.

Under standard regular assumptions, the method exhibits local quadratic convergence. However, its implementation requires the evaluation and subsequent inversion of the Hessian at each iteration, which implies a computational cost of the order $\mathcal{O}\left(n^3\right)$, making it impractical for large-scale problems. Furthermore, the Newton direction guarantees descent only when the Hessian is positive definite in a neighborhood of the solution, a condition that may not be satisfied in nonconvex problems.

With the aim of reducing computational cost while maintaining good convergence properties, quasi-Newton methods replace the exact Hessian with an iteratively updated approximation. In general, these methods generate the sequence $\left\{x^{\left(k\right)}\right\}$ according to the scheme

$$\begin{array}{c}{d}^{\left(k\right)}=-H^{\left(k\right)}\nabla f\left(x^{\left(k\right)}\right),x^{(k+1)}=x^{\left(k\right)}+{\alpha }^{\left(k\right)}{d}^{\left(k\right)},\\ s^{\left(k\right)}=x^{(k+1)}-x^{\left(k\right)},y^{\left(k\right)}=\nabla f\left(x^{(k+1)}\right)-\nabla f\left(x^{\left(k\right)}\right),\end{array}$$

where $H^{\left(k\right)}$ denotes an approximation of the inverse Hessian matrix ${\left[{\nabla }^{2}f\left(x^{\left(k\right)}\right)\right]}^{-1}$, $s^{\left(k\right)}$ represents the displacement between two successive iterates, and $y^{\left(k\right)}$ denotes the variation of the gradient between those iterates. The matrix $H^{(k+1)}$ is computed through a quasi-Newton update formula satisfying the secant equation

$$H^{(k+1)}{y}^{\left(k\right)}=s^{\left(k\right)}.$$

Different quasi-Newton methods arise from different update formulas. Among them, the DFP method, introduced in the 1960s, is one of the first designed for this purpose. The DFP update of the inverse Hessian is given by

$$H^{(k+1)}=H^{\left(k\right)}+{\displaystyle \frac{s^{\left(k\right)}{\left(s^{\left(k\right)}\right)}^T}{{\left(s^{\left(k\right)}\right)}^T{y}^{\left(k\right)}}}-{\displaystyle \frac{H^{\left(k\right)}{y}^{\left(k\right)}{\left(y^{\left(k\right)}\right)}^T{H}^{\left(k\right)}}{{\left(y^{\left(k\right)}\right)}^T{H}^{\left(k\right)}{y}^{\left(k\right)}}},$$

(1)

and was first presented in [1].

Later, in the 1970s, the BFGS method [2] was developed and is known for its efficiency and stability. In terms of the inverse Hessian approximation, the update is

$$H^{(k+1)}=\left(I-{\displaystyle \frac{s^{\left(k\right)}{\left(y^{\left(k\right)}\right)}^T}{{\left(y^{\left(k\right)}\right)}^T{s}^{\left(k\right)}}}\right)H^{\left(k\right)}\left(I-{\displaystyle \frac{y^{\left(k\right)}{\left(s^{\left(k\right)}\right)}^T}{{\left(y^{\left(k\right)}\right)}^T{s}^{\left(k\right)}}}\right)+{\displaystyle \frac{s^{\left(k\right)}{\left(s^{\left(k\right)}\right)}^T}{{\left(y^{\left(k\right)}\right)}^T{s}^{\left(k\right)}}},$$

(2)

where I denotes the identity matrix of the same dimension as $H^{\left(k\right)}$.

Another well-known quasi-Newton method is the symmetric rank-one (SR1) method [3]. The SR1 formula is considered one of the earliest quasi-Newton updates and is often associated with the ideas of Davidon [4].

However, its theoretical development and modern use in optimization were established primarily during the period 1991–1996. In terms of the approximation of the inverse of the Hessian, its update is defined by

$$H^{(k+1)}=H^{\left(k\right)}+{\displaystyle \frac{\left(s^{\left(k\right)}-H^{\left(k\right)}{y}^{\left(k\right)}\right){\left(s^{\left(k\right)}-H^{\left(k\right)}{y}^{\left(k\right)}\right)}^T}{{\left(s^{\left(k\right)}-H^{\left(k\right)}{y}^{\left(k\right)}\right)}^T{y}^{\left(k\right)}}}.$$

(3)

Quasi-Newton methods reduce the iterative cost to $\mathcal{O}\left(n^2\right)$ [5,6], but may suffer from loss of superlinear behavior and accumulation of numerical inaccuracies. In this context, hybrid algorithms [7,8,9,10] have emerged as a promising alternative by combining the strengths of different approaches to overcome the limitations of classical methods.

This article presents the Higher-Order Quasi-Newton (HOQN) method, a new hybrid algorithm with cubic local convergence order, which combines Newton-type techniques with high-order correctors constructed as vector extensions of the scalar methods of Chun, Ostrowski, and Traub. These vector extensions satisfy the optimality condition established by the Cordero–Torregrosa conjecture [11], according to which the convergence order of any Newton-type iterative method for solving nonlinear systems without memory cannot exceed

$$2^{k_1+k_2-1},$$

where $k_1$ represents the number of functional evaluations appearing in the entries of the Jacobian matrix per iteration and $k_2$ represents the number of evaluations of the nonlinear function, with $k_1\le k_2$. If a method reaches this upper bound, it is said to be optimal in the vectorial sense. In the proposed scheme, these high-order correctors are combined with quasi-Newton updates of the inverse Hessian matrix. Although the theoretical convergence of the method has been demonstrated in the context of convex functions, numerical experiments show that, in practice, the scheme also performs excellently in the minimization of nonconvex functions.

The HOQN scheme is applicable to any update formula, even allowing different update formulas after each hybrid step or nonlinear combinations of them, offering increased robustness and adaptability in solving complex optimization problems. In the HOQN scheme, a single update formula for the inverse of the Hessian matrix can be used after the high-order step, or two quasi-Newton update formulas can be used: one after the Newton step and another after the high-order corrector. This flexibility provides greater robustness and adaptability for solving complex optimization problems.

The main contribution of this work is to show that the HOQN method provides faster and more stable convergence compared to classical methods, due to the integration of higher-order corrections together with quasi-Newton updates. In addition, the dependence of the computational performance of the HOQN method on the initial estimation is studied by representing the basins of attraction in three benchmark functions: Himmelblau, Freudenstein–Roth, and Booth [12]. New variants of the hybrid algorithm with limited memory are also proposed, which are validated in the training of convolutional neural networks, obtaining excellent overall performance.

This manuscript is organized as follows. Section 1 provides an introduction to the study, outlining the state of the art in quasi-Newton methods and presenting the preliminary concepts that form the theoretical foundation of the research. Section 2 reviews the vector extensions of the scalar methods of Chun, Ostrowski, and Traub, and introduces a new optimal vector variant of Traub’s method inspired by the construction proposed by Singh et al. in [13]. Section 3 presents the general HOQN framework, including the one-update and two-update hybrid quasi-Newton schemes. Section 4 establishes the local convergence analysis of the proposed methods and proves the cubic convergence order under suitable smoothness and inverse-Hessian consistency assumptions. Section 5 reports the numerical experiments, including full-memory tests on classical benchmark functions, high-dimensional quadratic experiments, Dolan–Moré performance profiles, dynamical-plane analysis, and the limited-memory HOQN variants. Section 6 discusses the practical application of the proposed limited-memory hybrid optimizers to the training of a convolutional neural network on the MNIST dataset. Finally, Section 7 presents the main conclusions and outlines possible directions for future research.

Preliminary Concepts

In order to establish the theoretical foundations for the study of the proposed hybrid optimization methods, it is necessary to recall some classical definitions from optimization theory. These notions provide the mathematical framework for guaranteeing convergence properties and characterizing the behavior of the algorithms. In the sections devoted to vector extensions of iterative methods for nonlinear systems, we use the notation $F:{\mathbb{R}}^n\to {\mathbb{R}}^n$ for a nonlinear mapping and $F^{\prime }\left(x\right)$ for its Jacobian. In the particular case of unconstrained optimization, this mapping is identified with the gradient, that is, $F\left(x\right)=\nabla f\left(x\right)$. Hence, the first-order necessary condition for a local minimizer of f can be written as the nonlinear system $F\left(x\right)=0$.

Definition 1

(Convex Function). A function $f:{\mathbb{R}}^n\to \mathbb{R}$ is convex [14] if, for any pair of points $x_1,x_2\in {\mathbb{R}}^n$ and any $\lambda \in [0,1]$, the following inequality holds:

$$f\left(\lambda x_1+(1-\lambda )x_2\right)\le \lambda f\left(x_1\right)+(1-\lambda )f\left(x_2\right).$$

In addition to convexity, it is often useful to quantify how rapidly a function can change. This leads to the notion of Lipschitz continuity, which provides a bound on the variation of a function with respect to its inputs.

Definition 2

(Lipschitz Continuity of the Gradient). Let $f:{\mathbb{R}}^n\to \mathbb{R}$ be a continuously differentiable function. We say that the gradient $\nabla f$ is Lipschitz continuous [15,16] on a set $D\subseteq {\mathbb{R}}^n$ if there exists a constant $L\ge 0$ such that, for all $x,y\in D$,

$$\parallel \nabla f\left(x\right)-\nabla f\left(y\right)\parallel \le L\parallel x-y\parallel ,$$

where $\parallel ·\parallel$ denotes a norm in ${\mathbb{R}}^n$, and L is the Lipschitz constant.

When analyzing iterative optimization algorithms, line search techniques play a key role in guaranteeing sufficient progress at each iteration. For this purpose, the strong Wolfe conditions are frequently imposed.

Definition 3

(Strong Wolfe Conditions). An optimization algorithm satisfies the strong Wolfe conditions [17,18,19] if, at each iteration k, the search direction $d^{\left(k\right)}$ (with $\nabla f{\left(x^{\left(k\right)}\right)}^{\mathrm{T}}{d}^{\left(k\right)}<0$) and the step size ${\alpha }^{\left(k\right)}>0$ jointly meet:

$$\begin{array}{cc}\hfill \mathit{\text{Sufficient decrease (Armijo):}}& f\left(x^{\left(k\right)}+{\alpha }^{\left(k\right)}{d}^{\left(k\right)}\right)\le f\left(x^{\left(k\right)}\right)+c_1{\alpha }^{\left(k\right)}\nabla f{\left(x^{\left(k\right)}\right)}^{\mathrm{T}}{d}^{\left(k\right)},\hfill \\ \hfill \mathit{\text{Curvature condition:}}& \left|\nabla f{\left(x^{\left(k\right)}+{\alpha }^{\left(k\right)}{d}^{\left(k\right)}\right)}^{\mathrm{T}}{d}^{\left(k\right)}\right|\le c_2\left|\nabla f{\left(x^{\left(k\right)}\right)}^{\mathrm{T}}{d}^{\left(k\right)}\right|,\hfill \end{array}$$

(4)

where $x^{\left(k\right)}$ is the current iterate, $\nabla f\left(x^{\left(k\right)}\right)$ is the gradient of the objective function at $x^{\left(k\right)}$, and the constants satisfy $0<c_1<c_2<1$ (typically $c_1\in [{10}^{-6},{10}^{-4}]$ and $c_2\approx 0.9$).

Finally, to ensure that quasi-Newton updates preserve desirable convergence properties, one of the most widely used criteria is the Dennis–Moré condition, which characterizes the superlinear behavior of these methods. This condition will play a fundamental role in demonstrating that the quasi-Newton updates embedded in the HOQN scheme preserve its local order of convergence.

Definition 4

(Dennis–Moré Condition [5,20]). Let $\left\{x^{\left(k\right)}\right\}$ be a sequence generated by a quasi-Newton method converging to a local minimizer ξ, and let $\left\{H^{\left(k\right)}\right\}$ denote the corresponding sequence of inverse Hessian approximations. The sequence $\left\{H^{\left(k\right)}\right\}$ is said to satisfy the Dennis–Moré condition at ξ if

$$\underset{k\to \infty }{\lim }{\displaystyle \frac{\parallel \left(H^{(k+1)}-{\left[{\nabla }^{2}f\left(\xi \right)\right]}^{-1}\right)s^{\left(k\right)}\parallel }{\parallel s^{\left(k\right)}\parallel }}=0,$$

2. Vector Extensions of Traub’s, Chun’s, and Ostrowski’s Methods

Although the methods of Chun, Ostrowski and Traub were originally developed for solving nonlinear equations, their extension to the context of unconstrained optimization arises naturally by imposing the condition $\nabla f\left(x\right)=0$. Likewise, the methods of Chun [21] and Ostrowski [22], widely recognized for their high-order properties, provide a relevant conceptual basis for the development of vector extensions oriented toward optimization problems. For the design of optimal vectorial extensions of Chun’s and Ostrowski’s methods, the strategy proposed by Cordero et al. [23] is particularly relevant. This strategy is based on the fundamental idea introduced by Singh et al. [13], who present a two-step iterative scheme with fifth-order convergence for solving nonlinear systems. This scheme includes a first step based on a standard Newton iteration.

$$z^{\left(k\right)}=x^{\left(k\right)}-{\left[F^{\prime }\left(x^{\left(k\right)}\right)\right]}^{-1}F\left(x^{\left(k\right)}\right),$$

which computes a predictor $z^{\left(k\right)}$ that is refined in the subsequent step. Here, F : ${\mathbb{R}}^n\to {\mathbb{R}}^n$ is a nonlinear system of equations, and $F^{\prime }\left(x\right)$ denotes the Jacobian matrix of F evaluated at x. Unless otherwise stated, all norms considered throughout this work are Euclidean norms. The scheme then includes a second step, where a weighted corrective term is introduced, based on the ratio of squared norms of the function evaluated at $z^{\left(k\right)}$ and $x^{\left(k\right)}$, expressed as ${\displaystyle \frac{{∥F\left(z^{\left(k\right)}\right)∥}^2}{{∥F\left(x^{\left(k\right)}\right)∥}^2}}$.

$$x^{(k+1)}=z^{\left(k\right)}-\left(1+{\displaystyle \frac{{∥F\left(z^{\left(k\right)}\right)∥}^2}{{∥F\left(x^{\left(k\right)}\right)∥}^2}}\right){\left[F^{\prime }\left(z^{\left(k\right)}\right)\right]}^{-1}F\left(z^{\left(k\right)}\right).$$

In [23], Cordero et al. extend this idea by parametrizing the iterative scheme through the Ermakov Hyperfamily for Systems, with expression:

$$\begin{array}{cc}\hfill & z^{\left(k\right)}=x^{\left(k\right)}-{\left[F^{\prime }\left(x^{\left(k\right)}\right)\right]}^{-1}F\left(x^{\left(k\right)}\right),\hfill \\ \hfill & x^{(k+1)}=z^{\left(k\right)}-{\left[F^{\prime }\left(x^{\left(k\right)}\right)\right]}^{-1}\left(p_{k}F\left(z^{\left(k\right)}\right)+q_{k}F\left(x^{\left(k\right)}\right)\right),k=0,1,\dots \hfill \end{array}$$

(5)

where

$${\nu }_k={\displaystyle \frac{{∥F\left(z^{\left(k\right)}\right)∥}^2}{{∥F\left(x^{\left(k\right)}\right)∥}^2}}={\displaystyle \frac{F{\left(z^{\left(k\right)}\right)}^{T}F\left(z^{\left(k\right)}\right)}{F{\left(x^{\left(k\right)}\right)}^{T}F\left(x^{\left(k\right)}\right)}},{\mathcal{K}}_k={\displaystyle \frac{1}{1+\lambda {\nu }_k}},p_k={\mathcal{K}}_k(1+\psi {\nu }_k),q_k=2{\mathcal{K}}_k{\nu }_k,$$

(6)

obtaining, from (5), as special cases, the optimal vector versions of the scalar methods of Chun and Ostrowski, which we have denoted, respectively, by $M_1$ and $M_2$. For the optimal vector version $M_1$, when setting $\lambda =0$ and $\psi =1$, the iteration is expressed as:

$$x^{(k+1)}=z^{\left(k\right)}-{\left[F^{\prime }\left(x^{\left(k\right)}\right)\right]}^{-1}\left(\left(1+{\nu }_k\right)F\left(z^{\left(k\right)}\right)-2{\nu }_{k}F\left(x^{\left(k\right)}\right)\right).$$

(7)

On the other hand, by choosing $\lambda =-4$ and $\psi =0$, we obtain the optimal vector version $M_2$, whose iterative expression is

$$x^{(k+1)}=z^{\left(k\right)}-{\displaystyle \frac{1}{1-4{\nu }_k}}{\left[F^{\prime }\left(x^{\left(k\right)}\right)\right]}^{-1}\left(F\left(z^{\left(k\right)}\right)+2{\nu }_{k}F\left(x^{\left(k\right)}\right)\right).$$

(8)

On the other hand, Traub’s method [24], in its multidimensional form,

$$\begin{array}{cc}\hfill z^{\left(k\right)}& =x^{\left(k\right)}-{\left[F^{\prime }\left(x^{\left(k\right)}\right)\right]}^{-1}F\left(x^{\left(k\right)}\right),\hfill \\ \hfill x^{(k+1)}& =z^{\left(k\right)}-{\left[F^{\prime }\left(x^{\left(k\right)}\right)\right]}^{-1}F\left(z^{\left(k\right)}\right),\hfill \end{array}$$

(9)

constitutes a third-order multipass scheme that serves as one of the fundamental precursors to the hybrid constructions considered in this work. To align the Traub method with the idea introduced by Singh et al. in [13], we employ a technique based on weight functions depending on the acceleration parameter ${\nu }_k$, through which a new fourth-order variant of this method is generated. The incorporation of scalar weight functions defined in terms of ${\nu }_k$ allows us to preserve the essential structure of Traub’s method while, at the same time, endowing it with an additional acceleration mechanism that improves its local behavior. In this sense, the resulting variant can be interpreted as a natural extension of Traub’s method within the class of multistep methods with acceleration parameters. The following theorem formalizes this construction, establishes the corresponding error equation, and rigorously proves that the resulting method possesses fourth-order local convergence.

Theorem 1.

Let $F:D\subset {\mathbb{R}}^n\to {\mathbb{R}}^n$ be a sufficiently differentiable function in an open convex neighborhood D of ξ, where ξ is a simple solution of $F\left(x\right)=0$; that is, $F\left(\xi \right)=0,F^{\prime }\left(\xi \right)$ is nonsingular. Assume that $x^{\left(0\right)}$ is an initial approximation sufficiently close to ξ. Consider the iterative scheme

$$\left\{\begin{array}{c}{z}^{\left(k\right)}=x^{\left(k\right)}-{\left[F^{\prime }\left(x^{\left(k\right)}\right)\right]}^{-1}F\left(x^{\left(k\right)}\right),\hfill \\ x^{(k+1)}=x^{\left(k\right)}-H\left({\nu }_k\right){\left[F^{\prime }\left(x^{\left(k\right)}\right)\right]}^{-1}F\left(x^{\left(k\right)}\right)-G\left({\nu }_k\right){\left[F^{\prime }\left(x^{\left(k\right)}\right)\right]}^{-1}F\left(z^{\left(k\right)}\right),\hfill \end{array}\right.$$

(10)

where $e^{\left(k\right)}=x^{\left(k\right)}-\xi ,{\nu }_k={\displaystyle \frac{{∥F\left(z^{\left(k\right)}\right)∥}^2}{{∥F\left(x^{\left(k\right)}\right)∥}^2}}$, and $C_j={\displaystyle \frac{1}{j!}}{\left[F^{\prime }\left(\xi \right)\right]}^{-1}{F}^{\left(j\right)}\left(\xi \right),j\ge 2$.

Then, the sequence ${\left\{x^{\left(k\right)}\right\}}_{k\ge 0}$ converges locally to ξ with order of convergence 4 if and only if

$$H_0=H\left(0\right)=1,G_0=G\left(0\right)=1,H_1=H^{\prime }\left(0\right)=2,|G_1=G^{\prime }\left(0\right)|<+\infty .$$

In this case, the error equation is

$$e^{(k+1)}=\left(2C_2^3-6C_2{C}_3+3C_4-G_1{C}_2^3+2P^{-1}Q{C}_2^2\right)e^{{\left(k\right)}^4}+\mathcal{O}\left(e^{{\left(k\right)}^5}\right).$$

(11)

and, therefore, the method is of order 4.

Proof. 

Let $e^{\left(k\right)}=x^{\left(k\right)}-\xi$ denote the error. The Taylor expansions of $F\left(x^{\left(k\right)}\right)$ and $F^{\prime }\left(x^{\left(k\right)}\right)$ around $\xi$ are

$$F\left(x^{\left(k\right)}\right)=F^{\prime }\left(\xi \right)\left(e^{\left(k\right)}+C_2{e}^{{\left(k\right)}^2}+C_3{e}^{{\left(k\right)}^3}+C_4{e}^{{\left(k\right)}^4}\right)+\mathcal{O}\left(e^{{\left(k\right)}^5}\right),$$

and

$$F^{\prime }\left(x^{\left(k\right)}\right)=F^{\prime }\left(\xi \right)\left(I+2C_2{e}^{\left(k\right)}+3C_3{e}^{{\left(k\right)}^2}+4C_4{e}^{{\left(k\right)}^3}\right)+\mathcal{O}\left(e^{{\left(k\right)}^4}\right),$$

where $C_j={\displaystyle \frac{1}{j!}}{\left[F^{\prime }\left(\xi \right)\right]}^{-1}{F}^{\left(j\right)}\left(\xi \right),j\ge 2$. Furthermore, assuming that the inverse of the Jacobian satisfies

$${\left[F^{\prime }\left(x^{\left(k\right)}\right)\right]}^{-1}{F}^{\prime }\left(x^{\left(k\right)}\right)=F^{\prime }\left(x^{\left(k\right)}\right){\left[F^{\prime }\left(x^{\left(k\right)}\right)\right]}^{-1}=I,$$

we obtain

$${\left[F^{\prime }\left(x^{\left(k\right)}\right)\right]}^{-1}=\left(I+X_2{e}^{\left(k\right)}+X_3{e}^{{\left(k\right)}^2}+X_4{e}^{{\left(k\right)}^3}\right){\left[F^{\prime }\left(\xi \right)\right]}^{-1}+\mathcal{O}\left(e^{{\left(k\right)}^4}\right),$$

(12)

where

$$X_2=-2C_2,X_3=4C_2^2-3C_3,X_4=-2\left(4C_2^3+2C_4-3C_2{C}_3-3C_3{C}_2\right).$$

Therefore, the expansion of $A_k={\left[F^{\prime }\left(x^{\left(k\right)}\right)\right]}^{-1}F\left(x^{\left(k\right)}\right)$ is given by

$$A_k=e^{\left(k\right)}-C_2{e}^{{\left(k\right)}^2}+2\left(-C_3+C_2^2\right)e^{{\left(k\right)}^3}+B{e}^{{\left(k\right)}^4}+\mathcal{O}\left(e^{{\left(k\right)}^5}\right),$$

(13)

where $B=-4C_2^3-3C_4+4C_2{C}_3+3C_3{C}_2$.

Let us now calculate the error of Newton’s step. Since $z^{\left(k\right)}=x^{\left(k\right)}-A_k$, it follows that $e_z^{\left(k\right)}=z^{\left(k\right)}-\xi =e^{\left(k\right)}-A_k$. Therefore,

$$e_z^{\left(k\right)}=C_2{e}^{{\left(k\right)}^2}-2\left(-C_3+C_2^2\right)e^{{\left(k\right)}^3}-B{e}^{{\left(k\right)}^4}+\mathcal{O}\left(e^{{\left(k\right)}^5}\right),$$

that is,

$$e_z^{\left(k\right)}=C_2{e}^{{\left(k\right)}^2}-2\left(-C_3+C_2^2\right)e^{{\left(k\right)}^3}+\left(-4C_2^3-3C_4+4C_2{C}_3+3C_3{C}_2\right)e^{{\left(k\right)}^4}+\mathcal{O}\left(e^{{\left(k\right)}^5}\right).$$

(14)

Now let us expand $F\left(z^{\left(k\right)}\right)$ around $\xi$ using the result obtained in (14):

$$\begin{array}{cc}\hfill F\left(z^{\left(k\right)}\right)=& F^{\prime }\left(\xi \right)\left(e_z^{\left(k\right)}+C_2{e}_z^{{\left(k\right)}^2}+C_3{e}_z^{{\left(k\right)}^3}+C_4{e}_z^{{\left(k\right)}^4}\right)+\mathcal{O}\left(e_z^{{\left(k\right)}^5}\right)\hfill \\ \hfill =& F^{\prime }\left(\xi \right)\left(C_2{e}^{{\left(k\right)}^2}+A{e}^{{\left(k\right)}^3}+\left(B+C_2^3\right)e^{{\left(k\right)}^4}\right)+\mathcal{O}\left(e^{{\left(k\right)}^5}\right).\hfill \end{array}$$

(15)

where $A=-2\left(-C_3+C_2^2\right)$,    $B=4C_2^3+3C_4-4C_2{C}_3-3C_3{C}_2$. Therefore,

$$F\left(z^{\left(k\right)}\right)=F^{\prime }\left(\xi \right)\left(C_2{e}^{{\left(k\right)}^2}+2\left(C_3-C_2^2\right)e^{{\left(k\right)}^3}+\left(5C_2^3+3C_4-4C_2{C}_3-3C_3{C}_2\right)e^{{\left(k\right)}^4}\right)+\mathcal{O}\left(e^{{\left(k\right)}^5}\right).$$

(16)

Consequently,

$$B_k={\left[F^{\prime }\left(x^{\left(k\right)}\right)\right]}^{-1}F\left(z^{\left(k\right)}\right)=C_2{e}^{{\left(k\right)}^2}+2\left(C_3-2C_2^2\right)e^{{\left(k\right)}^3}+B_4{e}^{{\left(k\right)}^4}+\mathcal{O}\left(e^{{\left(k\right)}^5}\right),$$

(17)

where $B_4=3C_4-8C_2{C}_3-6C_3{C}_2+13C_2^3.$

Let us now proceed to determine the Taylor expansion of ${\nu }_k$.

$${\nu }_k={\displaystyle \frac{{∥F\left(z^{\left(k\right)}\right)∥}^2}{{∥F\left(x^{\left(k\right)}\right)∥}^2}}={\displaystyle \frac{F{\left(z^{\left(k\right)}\right)}^{T}F\left(z^{\left(k\right)}\right)}{F{\left(x^{\left(k\right)}\right)}^{T}F\left(x^{\left(k\right)}\right)}}.$$

(18)

Let $F:{\mathbb{R}}^n\to {\mathbb{R}}^n$ be sufficiently differentiable. For every $x={(x_1,x_2,\dots ,x_n)}^T\in {\mathbb{R}}^n$, it can be written as

$$F\left(x\right)={\left(f_1\left(x\right),f_2\left(x\right),\dots ,f_n\left(x\right)\right)}^T,$$

where each coordinate function $f_i:{\mathbb{R}}^n\to \mathbb{R}$ is scalar-valued. Observe that, although $F\left(x^{\left(k\right)}\right)$ and $F\left(z^{\left(k\right)}\right)$ are vectors in ${\mathbb{R}}^n$, the quotient ${\nu }_k$ is a scalar quantity. To manipulate it explicitly, Singh et al. [13] express it as

$${\displaystyle \frac{F{\left(z^{\left(k\right)}\right)}^{T}F\left(z^{\left(k\right)}\right)}{F{\left(x^{\left(k\right)}\right)}^{T}F\left(x^{\left(k\right)}\right)}}={\displaystyle \frac{{\displaystyle {\displaystyle \sum _{i=1}^n}{f}_i^2\left(z^{\left(k\right)}\right)}}{{\displaystyle {\displaystyle \sum _{i=1}^n}{f}_i^2\left(x^{\left(k\right)}\right)}}}.$$

(19)

Expanding $f_i\left(x^{\left(k\right)}\right)$ and $f_i\left(z^{\left(k\right)}\right)$ in Taylor series about $\xi$, we obtain

$$f_i\left(x^{\left(k\right)}\right)=f_i^{\prime }\left(\xi \right)e^{\left(k\right)}+{\displaystyle \frac{1}{2}}{f}_i^{\prime \prime }\left(\xi \right)e^{{\left(k\right)}^2}+{\displaystyle \frac{1}{6}}{f}_i^{\prime \prime \prime }\left(\xi \right)e^{{\left(k\right)}^3}+\mathcal{O}\left(e^{{\left(k\right)}^4}\right),$$

(20)

$$f_i\left(z^{\left(k\right)}\right)=f_i^{\prime }\left(\xi \right)e_z^{\left(k\right)}+{\displaystyle \frac{1}{2}}{f}_i^{\prime \prime }\left(\xi \right)e_z^{{\left(k\right)}^2}+{\displaystyle \frac{1}{6}}{f}_i^{\prime \prime \prime }\left(\xi \right)e_z^{{\left(k\right)}^3}+\mathcal{O}\left(e_z^{{\left(k\right)}^4}\right),$$

(21)

where $f_i^{\prime }\left(x\right)=\left({\displaystyle \frac{\partial f_i}{\partial x_1}},\dots ,{\displaystyle \frac{\partial f_i}{\partial x_n}}\right)$ denotes the row vector of $f_i$, and $f_i^{\prime \prime }\left(x\right)={\left[{\displaystyle \frac{{\partial }^2{f}_i}{\partial x_j\partial x_k}}\right]}_{n\times n}$ is the Hessian matrix of $f_i$. Higher order terms correspond to the multilinear derivatives of third order.

If we rewrite (20) and (21) as

$$f_i\left(x^{\left(k\right)}\right)=R_i{e}^{\left(k\right)}+H_i{e}^{{\left(k\right)}^2}+K_i{e}^{{\left(k\right)}^3}+\mathcal{O}\left(e^{{\left(k\right)}^4}\right),$$

(22)

$$f_i\left(z^{\left(k\right)}\right)=R_i{e}_z^{\left(k\right)}+H_i{e}_z^{{\left(k\right)}^2}+K_i{e}_z^{{\left(k\right)}^3}+\mathcal{O}\left(e^{{\left(k\right)}^4}\right),$$

(23)

where $R_i=f_i^{\prime }\left(\xi \right),H_i={\displaystyle \frac{1}{2}}{f}_i^{\prime \prime }\left(\xi \right),K_i={\displaystyle \frac{1}{6}}{f}_i^{\prime \prime \prime }\left(\xi \right)$. Then

$$f_i^2\left(x^{\left(k\right)}\right)=P_i{e}^{{\left(k\right)}^2}+Q_i{e}^{{\left(k\right)}^3}+S_i{e}^{{\left(k\right)}^4}+\mathcal{O}\left(e^{{\left(k\right)}^5}\right),$$

(24)

$$f_i^2\left(z^{\left(k\right)}\right)=P_i{e}_z^{{\left(k\right)}^2}+Q_i{e}_z^{{\left(k\right)}^3}+S_i{e}_z^{{\left(k\right)}^4}+\mathcal{O}\left(e^{{\left(k\right)}^5}\right),$$

(25)

where

$$\begin{array}{cccccc}\hfill P_i& =R_i^T{R}_i,\hfill & \hfill Q_i& =R_i^T{H}_i+H_i^T{R}_i,\hfill & \hfill S_i& =R_i^T{K}_i+K_i^T{R}_i+H_i^T{H}_i,\hfill \\ \hfill P& ={\displaystyle \sum _{i=1}^n}{P}_i,\hfill & \hfill Q& ={\displaystyle \sum _{i=1}^n}{Q}_i,\hfill & \hfill S& ={\displaystyle \sum _{i=1}^n}{S}_i.\hfill \end{array}$$

We use $e_z^{\left(k\right)}=C_2{e}^{{\left(k\right)}^2}+A{e}^{{\left(k\right)}^3}+B{e}^{{\left(k\right)}^4}+\mathcal{O}\left(e^{{\left(k\right)}^5}\right)$ and compute

$$e_z^{{\left(k\right)}^2}=C_2^2{e}^{{\left(k\right)}^4}+\left(C_{2}A+A{C}_2\right)e^{{\left(k\right)}^5}+\left(A^2+C_{2}B+B{C}_2\right)e^{{\left(k\right)}^6}+\mathcal{O}\left(e^{{\left(k\right)}^7}\right).$$

(26)

For $e_z^{{\left(k\right)}^3}$, the dominant term is of order six, hence $e_z^{{\left(k\right)}^3}=C_2^3{e}^{{\left(k\right)}^6}+\mathcal{O}\left(e^{{\left(k\right)}^7}\right)$. Since $e_z^{\left(k\right)}=\mathcal{O}\left(e^{{\left(k\right)}^2}\right)$, in order to compute ${\nu }_k$ up to order four it suffices to consider $P{e}_z^{{\left(k\right)}^2}$ up to $e^{{\left(k\right)}^6}$, $Q{e}_z^{{\left(k\right)}^3}$ at the term $C_2^3{e}^{{\left(k\right)}^6}$, while $S{e}_z^{{\left(k\right)}^4}$ starts at order $e^{{\left(k\right)}^8}$ and does not contribute. The numerator and the denominator, respectively, become:

$$\begin{array}{cc}\hfill {\displaystyle \sum _{i=1}^n}{f}_i^2\left(z^{\left(k\right)}\right)=& P{C}_2^2{e}^{{\left(k\right)}^4}+P\left(C_{2}A+A{C}_2\right)e^{{\left(k\right)}^5}\hfill \\ \hfill +& \left(P\left(A^2+C_{2}B+B{C}_2\right)+2Q{C}_2^3\right)e^{{\left(k\right)}^6}+\mathcal{O}\left(e^{{\left(k\right)}^7}\right),\hfill \end{array}$$

(27)

$${\displaystyle \sum _{i=1}^n}{f}_i^2\left(x^{\left(k\right)}\right)=P{e}^{{\left(k\right)}^2}+Q{e}^{{\left(k\right)}^3}+S{e}^{{\left(k\right)}^4}+\mathcal{O}\left(e^{{\left(k\right)}^5}\right).$$

(28)

Substituting (27) and (28) in (19) and expanding the quotient up to order four, we obtain

$$\begin{array}{cc}\hfill {\nu }_k=& C_2^2{e}^{{\left(k\right)}^2}+\left(2C_2{C}_3+2C_3{C}_2-4C_2^3-P^{-1}Q{C}_2^2\right)e^{{\left(k\right)}^3}\hfill \\ \hfill & +\left(12C_2^4+4C_3^2+3\left(C_2{C}_4+C_4{C}_2\right)-8C_2^2{C}_3-7C_3{C}_2^2-7C_2{C}_3{C}_2\right.\hfill \\ \hfill & \left.+5P^{-1}Q{C}_2^3-2P^{-1}Q{C}_2{C}_3-2P^{-1}Q{C}_3{C}_2+\left({\left(P^{-1}Q\right)}^2-P^{-1}S\right)C_2^2\right)e^{{\left(k\right)}^4}+\mathcal{O}\left(e^{{\left(k\right)}^5}\right).\hfill \end{array}$$

(29)

Since ${\nu }_k\to 0$ as $k\to \infty$, the functions H and G can be expanded in Taylor series about 0 as follows:

$$\begin{array}{cc}\hfill H\left({\nu }_k\right)& =H_0+H_1{\nu }_k+H_2{\nu }_k^2+\mathcal{O}\left({\nu }_k^3\right),\hfill \\ \hfill G\left({\nu }_k\right)& =G_0+G_1{\nu }_k+G_2{\nu }_k^2+\mathcal{O}\left({\nu }_k^3\right).\hfill \end{array}$$

For convenience, we introduce

$$A_k={\left[F^{\prime }\left(x^{\left(k\right)}\right)\right]}^{-1}F\left(x^{\left(k\right)}\right),B_k={\left[F^{\prime }\left(x^{\left(k\right)}\right)\right]}^{-1}F\left(z^{\left(k\right)}\right),$$

so that the iterative error equation can be written as

$$e^{(k+1)}=e^{\left(k\right)}-H\left({\nu }_k\right)A_k-G\left({\nu }_k\right)B_k.$$

(30)

Using the previous expansions, we have

$$A_k=e^{\left(k\right)}+\mathcal{O}\left(e^{{\left(k\right)}^2}\right),B_k=\mathcal{O}\left(e^{{\left(k\right)}^2}\right),$$

and, since ${\nu }_k=\mathcal{O}\left(e^{{\left(k\right)}^2}\right)$,

$${\nu }_k^2{A}_k=\mathcal{O}\left(e^{{\left(k\right)}^5}\right),{\nu }_k^2{B}_k=\mathcal{O}\left(e^{{\left(k\right)}^6}\right).$$

Therefore, the terms involving ${\nu }_k^2$ and higher powers do not affect the error equation up to order four.

Substituting these expansions into (30) and collecting powers of $e^{\left(k\right)}$, we obtain

$$\begin{array}{cc}\hfill e^{(k+1)}=& \left(1-H_0\right)e^{\left(k\right)}+\left(H_0-G_0\right)C_2{e}^{{\left(k\right)}^2}+\left(2\left(H_0-G_0\right)C_3+\left(4G_0-2H_0-H_1\right)C_2^2\right)e^{{\left(k\right)}^3}\hfill \\ \hfill & +\left(3\left(H_0-G_0\right)C_4+\left(-4H_0+8G_0-2H_1\right)C_2{C}_3+\left(-3H_0+6G_0-H_1\right)C_3{C}_2\right.\hfill \\ \hfill & \left.+\left(4H_0-13G_0+5H_1-G_1\right)C_2^3+H_1{P}^{-1}Q{C}_2^2\right)e^{{\left(k\right)}^4}+\mathcal{O}\left(e^{{\left(k\right)}^5}\right).\hfill \end{array}$$

(31)

Therefore, the order-four conditions are obtained by canceling the coefficients of $e^{\left(k\right)},e^{{\left(k\right)}^2},e^{{\left(k\right)}^3}$, namely

$$1-H_0=0,H_0-G_0=0,4G_0-2H_0-H_1=0,$$

which yield

$$H_0=1,G_0=1,H_1=2,$$

while $G_1$ remains a free finite parameter.

Consequently,

$$e^{(k+1)}=\left(2C_2^3-6C_2{C}_3+3C_4-G_1{C}_2^3+2P^{-1}Q{C}_2^2\right)e^{{\left(k\right)}^4}+\mathcal{O}\left(e^{{\left(k\right)}^5}\right),$$

(32)

and, therefore, method (10) is of order 4.    □

From (10), with $H_0=1$, $G_0=1$, $H_1=2$ and $|G_1|<+\infty$, a new optimal vectorial variant of the scalar Traub method, denoted by $M_3$. The iterative scheme is given by

$$\left\{\begin{array}{c}{z}^{\left(k\right)}=x^{\left(k\right)}-{\left[F^{\prime }\left(x^{\left(k\right)}\right)\right]}^{-1}F\left(x^{\left(k\right)}\right),\hfill \\ x^{(k+1)}=x^{\left(k\right)}-{\left[F^{\prime }\left(x^{\left(k\right)}\right)\right]}^{-1}\left(\left(1+2{\nu }_k\right)F\left(x^{\left(k\right)}\right)+\left(1+G_1{\nu }_k\right)F\left(z^{\left(k\right)}\right)\right).\hfill \end{array}\right.$$

(33)

where

$${\nu }_k={\displaystyle \frac{{∥F\left(z^{\left(k\right)}\right)∥}^2}{{∥F\left(x^{\left(k\right)}\right)∥}^2}},$$

The coefficient ${\nu }_k$ provides the $M_1,M_2$, and $M_3$ methods with greater computational efficiency compared to their classical counterparts, which, although preserving fourth-order convergence, are computationally more expensive. This is particularly important in optimization, where reducing computational cost is essential for effectively tackling large-scale and complex problems. If $F:{\mathbb{R}}^n\to {\mathbb{R}}^n$ denotes the nonlinear system under consideration, then, in the particular case of optimization, F is identified with the gradient vector of a scalar function $f:{\mathbb{R}}^n\to \mathbb{R}$, that is, $F\left(x\right)=\nabla f\left(x\right)$. Accordingly, these methods can be rewritten in terms of the gradient and the Hessian of f, and thus applied to approximate local extrema of f. Given an initial point $x_0\in {\mathbb{R}}^n$, the $M_1,M_2$, and $M_3$ methods generate sequences of points $\left\{x^{\left(k\right)}\right\}$ according to the following iterative formulas:

$$\begin{array}{cc}\hfill M_1:x^{(k+1)}& =z^{\left(k\right)}-{\left[{\nabla }^{2}f\left(x^{\left(k\right)}\right)\right]}^{-1}\left((1+{\nu }_k)\nabla f\left(z^{\left(k\right)}\right)-2{\nu }_k\nabla f\left(x^{\left(k\right)}\right)\right),\hfill \end{array}$$

(34)

$$\begin{array}{cc}\hfill M_2:x^{(k+1)}& =z^{\left(k\right)}-{\displaystyle \frac{1}{1-4{\nu }_k}}{\left[{\nabla }^{2}f\left(x^{\left(k\right)}\right)\right]}^{-1}\left(\nabla f\left(z^{\left(k\right)}\right)+2{\nu }_k\nabla f\left(x^{\left(k\right)}\right)\right),\hfill \end{array}$$

(35)

$$\begin{array}{cc}\hfill M_3:x^{(k+1)}& =z^{\left(k\right)}-{\left[{\nabla }^{2}f\left(x^{\left(k\right)}\right)\right]}^{-1}\left((1+2{\nu }_k)\nabla f\left(x^{\left(k\right)}\right)+(1+G_1{\nu }_k)\nabla f\left(z^{\left(k\right)}\right)\right).\hfill \end{array}$$

(36)

where $z^{\left(k\right)}=x^{\left(k\right)}-{\left[{\nabla }^{2}f\left(x^{\left(k\right)}\right)\right]}^{-1}\nabla f\left(x^{\left(k\right)}\right)$ is the Newton predictor, and ${\displaystyle {\nu }_k=\frac{\parallel \nabla f\left(z^{\left(k\right)}\right){\parallel }^2}{\parallel \nabla f\left(x^{\left(k\right)}\right){\parallel }^2}}$.

3. Hybrid-Quasi-Newton Method (HOQN)

This algorithm combines Newton’s method with high order corrections given by (34)–(36) and a quasi-Newton update of the inverse Hessian based on the DFP scheme; see Algorithm 1. Accordingly, the $M_1\mathrm{DFP}$, $M_2-\mathrm{DFP}$, and $M_3\mathrm{DFP}$ variants are obtained. At each iteration, a Newton direction followed by a high order correction is computed to define the search direction; the step size is determined via an inexact line search (Armijo or Wolfe), and the Hessian approximation is updated through a quasi-Newton formula at the end of the iteration. Additionally, the scheme can be extended by incorporating an intermediate quasi-Newton update after the Newton step, for instance using BFGS, leading to the variants described in the Remark 1. The higher-order corrections employed in the algorithm allow a local cubic convergence order to be achieved without significantly increasing the computational cost per iteration, which results in a faster reduction of the gradient norm and, consequently, in fewer iterations and function evaluations in nonconvex problems, where a purely Newton or a standard quasi-Newton method is less competitive. The general algorithm of the HOQN method is given by:

Algorithm 1 HOQN iteration with line search

Inputs:  $x^{\left(k\right)}$, inverse Hessian approximation $H^{\left(k\right)}$, objective f, gradient $\nabla f$, $|G_1|<+\infty$

1:  $g^{\left(k\right)}\leftarrow \nabla f\left(x^{\left(k\right)}\right)$

2:  $d_N^{\left(k\right)}\leftarrow -H^{\left(k\right)}{g}^{\left(k\right)}$

3:  ${\alpha }_N^{\left(k\right)}\leftarrow \arg {\min }_{\alpha \in (0,{\alpha }_{\max }]}f(x^{\left(k\right)}+\alpha d_N^{\left(k\right)})$

4:  $z^{\left(k\right)}\leftarrow x^{\left(k\right)}+{\alpha }_N^{\left(k\right)}{d}_N^{\left(k\right)}$

5:  $g_z^{\left(k\right)}\leftarrow \nabla f\left(z^{\left(k\right)}\right)$

6:  ${\nu }_k\leftarrow {\displaystyle \frac{\parallel g_z^{\left(k\right)}{\parallel }^2}{\parallel g^{\left(k\right)}{\parallel }^2}}$

7:  if method = $M_1$ DFP then

8:         $d_{HO}^{\left(k\right)}\leftarrow -H^{\left(k\right)}\left((1+{\nu }_k)g_z^{\left(k\right)}-2{\nu }_k{g}^{\left(k\right)}\right)$

9:         ${\alpha }_{HO}^{\left(k\right)}\leftarrow \arg {\min }_{\alpha \in (0,{\alpha }_{\max }]}f(z^{\left(k\right)}+\alpha d_{HO}^{\left(k\right)})$

10:         $x^{(k+1)}\leftarrow z^{\left(k\right)}+{\alpha }_{HO}^{\left(k\right)}{d}_{HO}^{\left(k\right)}$

11:  else if method = $M_2$ DFP then

12:         $d_{HO}^{\left(k\right)}\leftarrow -{\displaystyle \frac{1}{1-4{\nu }_k}}{H}^{\left(k\right)}\left(g_z^{\left(k\right)}+2{\nu }_k{g}^{\left(k\right)}\right)$

13:         ${\alpha }_{HO}^{\left(k\right)}\leftarrow \arg {\min }_{\alpha \in (0,{\alpha }_{\max }]}f(z^{\left(k\right)}+\alpha d_{HO}^{\left(k\right)})$

14:         $x^{(k+1)}\leftarrow z^{\left(k\right)}+{\alpha }_{HO}^{\left(k\right)}{d}_{HO}^{\left(k\right)}$

15:  else if method = $M_3$ DFP then

16:         $d_{HO}^{\left(k\right)}\leftarrow -H^{\left(k\right)}\left((1+2{\nu }_k)g^{\left(k\right)}+(1+G_1{\nu }_k)g_z^{\left(k\right)}\right)$

17:         ${\alpha }_{HO}^{\left(k\right)}\leftarrow \arg {\min }_{\alpha \in (0,{\alpha }_{\max }]}f(x^{\left(k\right)}+\alpha d_{HO}^{\left(k\right)})$

18:         $x^{(k+1)}\leftarrow x^{\left(k\right)}+{\alpha }_{HO}^{\left(k\right)}{d}_{HO}^{\left(k\right)}$

19:  end if

20:  $g^{(k+1)}\leftarrow \nabla f\left(x^{(k+1)}\right)$

21:  $s^{\left(k\right)}\leftarrow x^{(k+1)}-x^{\left(k\right)}$

22:  $y^{\left(k\right)}\leftarrow g^{(k+1)}-g^{\left(k\right)}$

23:  $H^{(k+1)}\leftarrow \mathrm{DFP}\mathrm{Update}(H^{\left(k\right)},s^{\left(k\right)},y^{\left(k\right)})$

Output:  $x^{(k+1)},H^{(k+1)}$

Remark 1.

If an intermediate quasi-Newton update is incorporated, that is, if an additional update is performed after the Newton step in Algorithm 1, for instance using a BFGS update, the following variants are obtained: $B{M}_{1}D$, $B{M}_{2}D$ and $B{M}_{3}D$.

4. Convergence Analysis of the HOQN Method

Theorem 2

(Local cubic error equation for the $M_i$ DFP-HOQN variants). Let $F=\nabla f:{\mathbb{R}}^n\to {\mathbb{R}}^n$ be sufficiently differentiable in a neighborhood of a strict local minimizer ξ, so that the Taylor expansions used in Theorem 1 hold. Assume that

$$F\left(\xi \right)=0,F^{\prime }\left(\xi \right)={\nabla }^{2}f\left(\xi \right)\succ 0.$$

Let

$$e^{\left(k\right)}=x^{\left(k\right)}-\xi ,C_j={\displaystyle \frac{1}{j!}}{\left[F^{\prime }\left(\xi \right)\right]}^{-1}{F}^{\left(j\right)}\left(\xi \right),j\ge 2.$$

As in Theorem 1, the notation $\mathcal{O}\left(e^{{\left(k\right)}^p}\right)$ is understood in the vectorial sense; for a matrix sequence, $E^{\left(k\right)}=\mathcal{O}\left(e^{\left(k\right)}\right)$ means $\parallel E^{\left(k\right)}\parallel \le C\parallel e^{\left(k\right)}\parallel$.

Consider the one-update variants $M_1\mathrm{DFP}$, $M_2\mathrm{DFP}$, and $M_3\mathrm{DFP}$, obtained from the correctors $M_1$, $M_2$, and $M_3$ in (34), (35), and (36), respectively, by replacing ${\left[F^{\prime }\left(x^{\left(k\right)}\right)\right]}^{-1}$ with $H^{\left(k\right)}$. Assume that the iteration is in the local full-step regime,

$${\alpha }_N^{\left(k\right)}={\alpha }_{HO}^{\left(k\right)}=1,z^{\left(k\right)}=x^{\left(k\right)}-H^{\left(k\right)}F\left(x^{\left(k\right)}\right),{\nu }_k={\displaystyle \frac{\parallel F\left(z^{\left(k\right)}\right){\parallel }^2}{\parallel F\left(x^{\left(k\right)}\right){\parallel }^2}},$$

for all sufficiently large k. For the $M_2$-variant, assume $1-4{\nu }_k\ne 0$, which holds locally. For the $M_3$-variant, the corrector is understood in the $x^{\left(k\right)}$-based form

$$x^{(k+1)}=x^{\left(k\right)}-H^{\left(k\right)}\left((1+2{\nu }_k)F\left(x^{\left(k\right)}\right)+(1+G_1{\nu }_k)F\left(z^{\left(k\right)}\right)\right),\left|G_1\right|<+\infty .$$

Assume the first-order local inverse-Hessian consistency condition

$$H^{\left(k\right)}={\left[F^{\prime }\left(x^{\left(k\right)}\right)\right]}^{-1}+E^{\left(k\right)},E^{\left(k\right)}=\mathcal{O}\left(e^{\left(k\right)}\right).$$

(37)

Define

$$R_k:=E^{\left(k\right)}{F}^{\prime }\left(\xi \right),Z_k:=C_2{e}^{{\left(k\right)}^2}-R_k{e}^{\left(k\right)},$$

and

$${\vartheta }_k:={\displaystyle \frac{\parallel F^{\prime }\left(\xi \right)Z_k{\parallel }^2}{\parallel F^{\prime }\left(\xi \right)e^{\left(k\right)}{\parallel }^2}}.$$

(38)

Then

$$Z_k=\mathcal{O}\left(e^{{\left(k\right)}^2}\right),{\vartheta }_k=\mathcal{O}\left(e^{{\left(k\right)}^2}\right),{\nu }_k={\vartheta }_k+\mathcal{O}\left(e^{{\left(k\right)}^3}\right).$$

Moreover, the explicit local error recurrences are

$$\begin{array}{cc}\hfill M_1\mathrm{DFP}:e^{(k+1)}& ={\mathcal{C}}_{1,k}^{\left(3\right)}+\mathcal{O}\left(e^{{\left(k\right)}^4}\right),\hfill \end{array}$$

(39)

$$\begin{array}{cc}\hfill M_2\mathrm{DFP}:e^{(k+1)}& ={\mathcal{C}}_{2,k}^{\left(3\right)}+\mathcal{O}\left(e^{{\left(k\right)}^4}\right),\hfill \end{array}$$

(40)

$$\begin{array}{cc}\hfill M_3\mathrm{DFP}:e^{(k+1)}& ={\mathcal{C}}_{3,k}^{\left(3\right)}+\mathcal{O}\left(e^{{\left(k\right)}^4}\right),\hfill \end{array}$$

(41)

where the cubic contributions are explicitly given by

$$\begin{array}{cc}\hfill {\mathcal{C}}_{1,k}^{\left(3\right)}& =2C_2{e}^{\left(k\right)}\left(C_2{e}^{{\left(k\right)}^2}-R_k{e}^{\left(k\right)}\right)-R_k\left(C_2{e}^{{\left(k\right)}^2}-R_k{e}^{\left(k\right)}\right)+2{\vartheta }_k{e}^{\left(k\right)},\hfill \end{array}$$

(42)

$$\begin{array}{cc}\hfill {\mathcal{C}}_{2,k}^{\left(3\right)}& =2C_2{e}^{\left(k\right)}\left(C_2{e}^{{\left(k\right)}^2}-R_k{e}^{\left(k\right)}\right)-R_k\left(C_2{e}^{{\left(k\right)}^2}-R_k{e}^{\left(k\right)}\right)-2{\vartheta }_k{e}^{\left(k\right)},\hfill \end{array}$$

(43)

$$\begin{array}{cc}\hfill {\mathcal{C}}_{3,k}^{\left(3\right)}& ={\mathcal{C}}_{2,k}^{\left(3\right)}.\hfill \end{array}$$

(44)

Consequently, there exists $K>0$ such that

$$\parallel e^{(k+1)}\parallel \le K\parallel e^{\left(k\right)}{\parallel }^3$$

for all sufficiently large k. Hence, each $M_i\mathrm{DFP}$-HOQN variant has local convergence of order at least three. If

$$\underset{k\to \infty }{\mathrm{lim\; inf}}{\displaystyle \frac{\parallel {\mathcal{C}}_{i,k}^{\left(3\right)}\parallel }{\parallel e^{\left(k\right)}{\parallel }^3}}>0,$$

then the corresponding local convergence order is exactly three.

Proof. 

We use the Taylor expansions established in Theorem 1. In particular,

$${\left[F^{\prime }\left(x^{\left(k\right)}\right)\right]}^{-1}F\left(x^{\left(k\right)}\right)=e^{\left(k\right)}-C_2{e}^{{\left(k\right)}^2}+2(-C_3+C_2^2)e^{{\left(k\right)}^3}+\mathcal{O}\left(e^{{\left(k\right)}^4}\right).$$

Using (37), $R_k=E^{\left(k\right)}{F}^{\prime }\left(\xi \right)$, and $R_k=\mathcal{O}\left(e^{\left(k\right)}\right)$, we obtain

$$H^{\left(k\right)}F\left(x^{\left(k\right)}\right)=e^{\left(k\right)}-C_2{e}^{{\left(k\right)}^2}+2(-C_3+C_2^2)e^{{\left(k\right)}^3}+R_k{e}^{\left(k\right)}+R_k{C}_2{e}^{{\left(k\right)}^2}+\mathcal{O}\left(e^{{\left(k\right)}^4}\right).$$

(45)

Hence, for $e_z^{\left(k\right)}=z^{\left(k\right)}-\xi$,

$$e_z^{\left(k\right)}=Z_k+2(C_3-C_2^2)e^{{\left(k\right)}^3}-R_k{C}_2{e}^{{\left(k\right)}^2}+\mathcal{O}\left(e^{{\left(k\right)}^4}\right),$$

(46)

and therefore

$$Z_k=\mathcal{O}\left(e^{{\left(k\right)}^2}\right),e_z^{\left(k\right)}=\mathcal{O}\left(e^{{\left(k\right)}^2}\right).$$

Since $e_z^{\left(k\right)}=\mathcal{O}\left(e^{{\left(k\right)}^2}\right)$,

$$F\left(z^{\left(k\right)}\right)=F^{\prime }\left(\xi \right)e_z^{\left(k\right)}+\mathcal{O}\left(e^{{\left(k\right)}^4}\right),{\left[F^{\prime }\left(x^{\left(k\right)}\right)\right]}^{-1}{F}^{\prime }\left(\xi \right)=I-2C_2{e}^{\left(k\right)}+\mathcal{O}\left(e^{{\left(k\right)}^2}\right).$$

Consequently,

$$e_z^{\left(k\right)}-H^{\left(k\right)}F\left(z^{\left(k\right)}\right)=2C_2{e}^{\left(k\right)}{Z}_k-R_k{Z}_k+\mathcal{O}\left(e^{{\left(k\right)}^4}\right).$$

(47)

Moreover, from (46),

$$F\left(z^{\left(k\right)}\right)=F^{\prime }\left(\xi \right)Z_k+\mathcal{O}\left(e^{{\left(k\right)}^3}\right),F\left(x^{\left(k\right)}\right)=F^{\prime }\left(\xi \right)e^{\left(k\right)}+\mathcal{O}\left(e^{{\left(k\right)}^2}\right),$$

which gives

$${\nu }_k={\vartheta }_k+\mathcal{O}\left(e^{{\left(k\right)}^3}\right),{\vartheta }_k=\mathcal{O}\left(e^{{\left(k\right)}^2}\right).$$

(48)

Also,

$$H^{\left(k\right)}F\left(x^{\left(k\right)}\right)=e^{\left(k\right)}+\mathcal{O}\left(e^{{\left(k\right)}^2}\right),H^{\left(k\right)}F\left(z^{\left(k\right)}\right)=\mathcal{O}\left(e^{{\left(k\right)}^2}\right).$$

Thus,

$${\nu }_k{H}^{\left(k\right)}F\left(x^{\left(k\right)}\right)={\vartheta }_k{e}^{\left(k\right)}+\mathcal{O}\left(e^{{\left(k\right)}^4}\right),{\nu }_k{H}^{\left(k\right)}F\left(z^{\left(k\right)}\right)=\mathcal{O}\left(e^{{\left(k\right)}^4}\right).$$

(49)

For $M_1\mathrm{DFP}$, (47) and (49) yield

$$e^{(k+1)}=2C_2{e}^{\left(k\right)}{Z}_k-R_k{Z}_k+2{\vartheta }_k{e}^{\left(k\right)}+\mathcal{O}\left(e^{{\left(k\right)}^4}\right),$$

which proves (39) and (42). For $M_2\mathrm{DFP}$, using

$${(1-4{\nu }_k)}^{-1}=1+4{\nu }_k+\mathcal{O}\left(e^{{\left(k\right)}^4}\right),$$

together with (47) and (49), gives

$$e^{(k+1)}=2C_2{e}^{\left(k\right)}{Z}_k-R_k{Z}_k-2{\vartheta }_k{e}^{\left(k\right)}+\mathcal{O}\left(e^{{\left(k\right)}^4}\right),$$

which proves (40) and (43).

For $M_3\mathrm{DFP}$, using the $x^{\left(k\right)}$-based form and $e^{\left(k\right)}-H^{\left(k\right)}F\left(x^{\left(k\right)}\right)=e_z^{\left(k\right)}$, we obtain

$$e^{(k+1)}=e_z^{\left(k\right)}-H^{\left(k\right)}F\left(z^{\left(k\right)}\right)-2{\nu }_k{H}^{\left(k\right)}F\left(x^{\left(k\right)}\right)-G_1{\nu }_k{H}^{\left(k\right)}F\left(z^{\left(k\right)}\right).$$

The last term is $\mathcal{O}\left(e^{{\left(k\right)}^4}\right)$, since $|G_1|<+\infty$, ${\nu }_k=\mathcal{O}\left(e^{{\left(k\right)}^2}\right)$, and $H^{\left(k\right)}F\left(z^{\left(k\right)}\right)=\mathcal{O}\left(e^{{\left(k\right)}^2}\right)$.

Therefore,

$$e^{(k+1)}=2C_2{e}^{\left(k\right)}{Z}_k-R_k{Z}_k-2{\vartheta }_k{e}^{\left(k\right)}+\mathcal{O}\left(e^{{\left(k\right)}^4}\right),$$

which proves (41) and (44).

Finally, since $R_k=\mathcal{O}\left(e^{\left(k\right)}\right)$, $Z_k=\mathcal{O}\left(e^{{\left(k\right)}^2}\right)$, and ${\vartheta }_k=\mathcal{O}\left(e^{{\left(k\right)}^2}\right)$, each ${\mathcal{C}}_{i,k}^{\left(3\right)}$ is $\mathcal{O}\left(e^{{\left(k\right)}^3}\right)$.

Hence,

$$\parallel e^{(k+1)}\parallel \le K\parallel e^{\left(k\right)}{\parallel }^3$$

for some $K>0$ and all sufficiently large k. If the normalized cubic contribution does not vanish asymptotically, the order is exactly three; otherwise, it is at least three.    □

Remark 2

(Interpretation of the cubic error equation). The cubic contributions of $M_2\mathrm{DFP}$ and $M_3\mathrm{DFP}$ coincide:

$${\mathcal{C}}_{2,k}^{\left(3\right)}={\mathcal{C}}_{3,k}^{\left(3\right)}.$$

This is due to the common dominant term $-2{\nu }_k{H}^{\left(k\right)}F\left(x^{\left(k\right)}\right)$; their first method-dependent difference appears at fourth order. Indeed, if

$$A_k:=H^{\left(k\right)}F\left(x^{\left(k\right)}\right),B_k:=H^{\left(k\right)}F\left(z^{\left(k\right)}\right),$$

then

$$e_{M_2}^{(k+1)}-e_{M_3}^{(k+1)}=(G_1-4){\nu }_k{B}_k+\mathcal{O}\left(e^{{\left(k\right)}^5}\right)=(G_1-4){\vartheta }_k{Z}_k+\mathcal{O}\left(e^{{\left(k\right)}^5}\right).$$

The quasi-Newton perturbation enters the dominant cubic term through $R_k=E^{\left(k\right)}{F}^{\prime }\left(\xi \right)$. If locally

$$\parallel E^{\left(k\right)}\parallel \le \eta \parallel e^{\left(k\right)}\parallel ,\rho :=\eta \parallel F^{\prime }\left(\xi \right)\parallel ,$$

then

$$\parallel R_k\parallel \le \rho \parallel e^{\left(k\right)}\parallel ,\parallel Z_k\parallel \le a\parallel e^{\left(k\right)}{\parallel }^2,a:=\parallel C_2\parallel +\rho .$$

With

$${\kappa }_{*}:=\parallel F^{\prime }\left(\xi \right)\parallel \parallel {\left[F^{\prime }\left(\xi \right)\right]}^{-1}\parallel ,$$

we obtain

$${\vartheta }_k\le {\kappa }_{*}^2{a}^2{\parallel e^{\left(k\right)}\parallel }^2.$$

Thus, one may take

$$\parallel e^{(k+1)}\parallel \le K_{HOQN}\parallel e^{\left(k\right)}{\parallel }^3+\mathcal{O}(\parallel e^{\left(k\right)}{\parallel }^4),K_{HOQN}=2\parallel C_2\parallel a+\rho a+2{\kappa }_{*}^2{a}^2.$$

This bound shows that the cubic constant depends on the local curvature coefficient $C_2$, the conditioning of $F^{\prime }\left(\xi \right)$, and the first-order inverse-Hessian perturbation.

For comparison, a pure quasi-Newton step

$$x^{(k+1)}=x^{\left(k\right)}-H^{\left(k\right)}F\left(x^{\left(k\right)}\right)$$

satisfies, under (37),

$$e^{(k+1)}=C_2{e}^{{\left(k\right)}^2}-R_k{e}^{\left(k\right)}+\mathcal{O}\left(e^{{\left(k\right)}^3}\right).$$

Hence, even under $H^{\left(k\right)}={\left[F^{\prime }\left(x^{\left(k\right)}\right)\right]}^{-1}+\mathcal{O}\left(e^{\left(k\right)}\right)$, a pure quasi-Newton step is at most quadratic in this expansion. Under the Dennis–Moré condition alone, one obtains superlinear convergence, but no fixed cubic order is guaranteed in general. The HOQN correction cancels the quadratic residual and shifts the leading term to order three. Although exact high-order correctors may reach fourth order, the quasi-Newton approximation and the line-search setting reduce the effective HOQN local order to cubic, which still improves on the usual superlinear behavior of classical quasi-Newton schemes.

The next lemma shows that the cubic local order obtained in Theorem 2 is preserved by the two-update BFGS–DFP strategy.

Lemma 1

(Invariance of cubic local convergence under BFGS–DFP double updating). Assume the hypotheses of Theorem 2. Let $F=\nabla f$, and let $H_N^{\left(k\right)}$ denote the inverse Hessian approximation at the beginning of the k-th HOQN iteration. We consider the complete two-update variants $B{M}_{1}D$, $B{M}_{2}D$, and $B{M}_{3}D$, consisting of a Newton-type predictor, an intermediate BFGS update, a high-order corrector, and a final DFP update. In the local full-step regime, $z^{\left(k\right)}=x^{\left(k\right)}-H_N^{\left(k\right)}F\left(x^{\left(k\right)}\right),v_k={\displaystyle \frac{{∥F\left(z^{\left(k\right)}\right)∥}^2}{{∥F\left(x^{\left(k\right)}\right)∥}^2}}$. After this predictor, a BFGS update (2) is applied with

$$s_N^{\left(k\right)}=z^{\left(k\right)}-x^{\left(k\right)},y_N^{\left(k\right)}=F\left(z^{\left(k\right)}\right)-F\left(x^{\left(k\right)}\right),$$

producing ${\widehat{H}}^{\left(k\right)}$. The high-order correction is then computed using ${\widehat{H}}^{\left(k\right)}$. In the local full-step regime, the correction stages of the complete two-update variants are represented by

$$\begin{array}{cc}\hfill B{M}_{1}D:x^{(k+1)}& =z^{\left(k\right)}-{\widehat{H}}^{\left(k\right)}\left((1+{\nu }_k)F\left(z^{\left(k\right)}\right)-2{\nu }_{k}F\left(x^{\left(k\right)}\right)\right),\hfill \end{array}$$

(50)

$$\begin{array}{cc}\hfill B{M}_{2}D:x^{(k+1)}& =z^{\left(k\right)}-{\displaystyle \frac{1}{1-4{\nu }_k}}{\widehat{H}}^{\left(k\right)}\left(F\left(z^{\left(k\right)}\right)+2{\nu }_{k}F\left(x^{\left(k\right)}\right)\right),\hfill \end{array}$$

(51)

$$\begin{array}{cc}\hfill B{M}_{3}D:x^{(k+1)}& =x^{\left(k\right)}-{\widehat{H}}^{\left(k\right)}\left((1+2{\nu }_k)F\left(x^{\left(k\right)}\right)+(1+G_1{\nu }_k)F\left(z^{\left(k\right)}\right)\right),\left|G_1\right|<+\infty .\hfill \end{array}$$

(52)

Here, $B{M}_{i}D$ denotes the complete two-update hybrid iteration; the displayed formulas are its local full-step correction maps. For $B{M}_{2}D$, $1-4{\nu }_k\ne 0$ locally.

After $x^{(k+1)}$ is computed, a DFP update (1) is applied using ${\widehat{H}}^{\left(k\right)}$ and

$$s_D^{\left(k\right)}=x^{(k+1)}-x^{\left(k\right)},y_D^{\left(k\right)}=F\left(x^{(k+1)}\right)-F\left(x^{\left(k\right)}\right),$$

yielding $H_N^{(k+1)}$. Assume that the BFGS and DFP updates are well defined, that their curvature conditions hold, and that

$$H_N^{\left(k\right)}={\left[F^{\prime }\left(x^{\left(k\right)}\right)\right]}^{-1}+E_{N,k},{\widehat{H}}^{\left(k\right)}={\left[F^{\prime }\left(x^{\left(k\right)}\right)\right]}^{-1}+{\widehat{E}}_k,$$

with

$$E_{N,k}=\mathcal{O}\left(e^{\left(k\right)}\right),{\widehat{E}}_k=\mathcal{O}\left(e^{\left(k\right)}\right).$$

Assume also that the final DFP update propagates first-order local consistency:

$$H_N^{(k+1)}={\left[F^{\prime }\left(x^{(k+1)}\right)\right]}^{-1}+\mathcal{O}\left(e^{(k+1)}\right).$$

Define

$$R_{N,k}:=E_{N,k}{F}^{\prime }\left(\xi \right),{\widehat{R}}_k:={\widehat{E}}_k{F}^{\prime }\left(\xi \right),Z_{N,k}:=C_2{e}^{{\left(k\right)}^2}-R_{N,k}{e}^{\left(k\right)},$$

and

$${\vartheta }_{N,k}:={\displaystyle \frac{\parallel F^{\prime }\left(\xi \right)Z_{N,k}{\parallel }^2}{\parallel F^{\prime }\left(\xi \right)e^{\left(k\right)}{\parallel }^2}}.$$

(53)

Then

$$Z_{N,k}=\mathcal{O}\left(e^{{\left(k\right)}^2}\right),{\vartheta }_{N,k}=\mathcal{O}\left(e^{{\left(k\right)}^2}\right),{\nu }_k={\vartheta }_{N,k}+\mathcal{O}\left(e^{{\left(k\right)}^3}\right).$$

Moreover,

$$\begin{array}{cc}\hfill B{M}_{1}D:e^{(k+1)}& ={\widehat{\mathcal{C}}}_{1,k}^{\left(3\right)}+\mathcal{O}\left(e^{{\left(k\right)}^4}\right),\hfill \end{array}$$

(54)

$$\begin{array}{cc}\hfill B{M}_{2}D:e^{(k+1)}& ={\widehat{\mathcal{C}}}_{2,k}^{\left(3\right)}+\mathcal{O}\left(e^{{\left(k\right)}^4}\right),\hfill \end{array}$$

(55)

$$\begin{array}{cc}\hfill B{M}_{3}D:e^{(k+1)}& ={\Delta }_k+{\widehat{\mathcal{C}}}_{2,k}^{\left(3\right)}+\mathcal{O}\left(e^{{\left(k\right)}^4}\right),\hfill \end{array}$$

(56)

where

$${\Delta }_k:=\left(H_N^{\left(k\right)}-{\widehat{H}}^{\left(k\right)}\right)F\left(x^{\left(k\right)}\right)$$

and

$$\begin{array}{cc}\hfill {\widehat{\mathcal{C}}}_{1,k}^{\left(3\right)}& =2C_2{e}^{\left(k\right)}{Z}_{N,k}-{\widehat{R}}_k{Z}_{N,k}+2{\vartheta }_{N,k}{e}^{\left(k\right)},\hfill \end{array}$$

(57)

$$\begin{array}{cc}\hfill {\widehat{\mathcal{C}}}_{2,k}^{\left(3\right)}& =2C_2{e}^{\left(k\right)}{Z}_{N,k}-{\widehat{R}}_k{Z}_{N,k}-2{\vartheta }_{N,k}{e}^{\left(k\right)}.\hfill \end{array}$$

(58)

Consequently, $B{M}_{1}D$ and $B{M}_{2}D$ preserve the cubic local order. For $B{M}_{3}D$, the same conclusion holds provided that

$${\Delta }_k=\left(H_N^{\left(k\right)}-{\widehat{H}}^{\left(k\right)}\right)F\left(x^{\left(k\right)}\right)=\mathcal{O}\left(e^{{\left(k\right)}^3}\right).$$

Equivalently, at this order,

$$\left(R_{N,k}-{\widehat{R}}_k\right)e^{\left(k\right)}=\mathcal{O}\left(e^{{\left(k\right)}^3}\right).$$

In particular,

$${\widehat{H}}^{\left(k\right)}-H_N^{\left(k\right)}=\mathcal{O}\left(e^{{\left(k\right)}^2}\right)$$

implies the compatibility condition above. Under this condition, all three complete two-update variants preserve the cubic local convergence order.

Proof. 

Let

$$e=e^{\left(k\right)},e_z^{\left(k\right)}=z^{\left(k\right)}-\xi .$$

Repeating the derivation of (46) for the predictor matrix $H_N^{\left(k\right)}$ gives

$$e_z^{\left(k\right)}=Z_{N,k}+2(C_3-C_2^2)e^3-R_{N,k}{C}_2{e}^2+\mathcal{O}\left(e^4\right).$$

(59)

Thus, $Z_{N,k}=\mathcal{O}\left(e^2\right)$ and $e_z^{\left(k\right)}=\mathcal{O}\left(e^2\right)$.

Next, repeating the cancellation argument leading to (47), now using ${\widehat{H}}^{\left(k\right)}$, yields

$$e_z^{\left(k\right)}-{\widehat{H}}^{\left(k\right)}F\left(z^{\left(k\right)}\right)=2C_{2}e{Z}_{N,k}-{\widehat{R}}_k{Z}_{N,k}+\mathcal{O}\left(e^4\right).$$

(60)

Similarly, by the arguments leading to (48) and (49),

$${\nu }_k={\vartheta }_{N,k}+\mathcal{O}\left(e^3\right),{\vartheta }_{N,k}=\mathcal{O}\left(e^2\right),$$

and

$${\nu }_k{\widehat{H}}^{\left(k\right)}F\left(x^{\left(k\right)}\right)={\vartheta }_{N,k}e+\mathcal{O}\left(e^4\right),{\nu }_k{\widehat{H}}^{\left(k\right)}F\left(z^{\left(k\right)}\right)=\mathcal{O}\left(e^4\right).$$

(61)

For $B{M}_{1}D$, substituting (60) and (61) into (50) gives

$$e^{(k+1)}=2C_{2}e{Z}_{N,k}-{\widehat{R}}_k{Z}_{N,k}+2{\vartheta }_{N,k}e+\mathcal{O}\left(e^4\right),$$

which proves (54). For $B{M}_{2}D$, using

$${(1-4{\nu }_k)}^{-1}=1+4{\nu }_k+\mathcal{O}\left(e^4\right)$$

in (51) gives

$$e^{(k+1)}=2C_{2}e{Z}_{N,k}-{\widehat{R}}_k{Z}_{N,k}-2{\vartheta }_{N,k}e+\mathcal{O}\left(e^4\right),$$

which proves (55).

For $B{M}_{3}D$, since the predictor was computed with $H_N^{\left(k\right)}$,

$$e-{\widehat{H}}^{\left(k\right)}F\left(x^{\left(k\right)}\right)=e_z^{\left(k\right)}+\left(H_N^{\left(k\right)}-{\widehat{H}}^{\left(k\right)}\right)F\left(x^{\left(k\right)}\right)=e_z^{\left(k\right)}+{\Delta }_k.$$

Using this identity in (52), together with (60) and (61), and using $G_1{\nu }_k{\widehat{H}}^{\left(k\right)}F\left(z^{\left(k\right)}\right)=\mathcal{O}\left(e^4\right)$, we obtain

$$e^{(k+1)}={\Delta }_k+2C_{2}e{Z}_{N,k}-{\widehat{R}}_k{Z}_{N,k}-2{\vartheta }_{N,k}e+\mathcal{O}\left(e^4\right),$$

which proves (56).

Since

$$Z_{N,k}=\mathcal{O}\left(e^2\right),R_{N,k}=\mathcal{O}\left(e\right),{\widehat{R}}_k=\mathcal{O}\left(e\right),{\vartheta }_{N,k}=\mathcal{O}\left(e^2\right),$$

the cubic order of $B{M}_{1}D$ and $B{M}_{2}D$ follows. For $B{M}_{3}D$, the same conclusion holds if ${\Delta }_k=\mathcal{O}\left(e^3\right)$. Moreover,

$$F\left(x^{\left(k\right)}\right)=F^{\prime }\left(\xi \right)e+\mathcal{O}\left(e^2\right)$$

implies

$${\Delta }_k=(E_{N,k}-{\widehat{E}}_k)F\left(x^{\left(k\right)}\right)=(R_{N,k}-{\widehat{R}}_k)e+\mathcal{O}\left(e^3\right),$$

which gives the stated equivalence. The sufficient condition ${\widehat{H}}^{\left(k\right)}-H_N^{\left(k\right)}=\mathcal{O}\left(e^{{\left(k\right)}^2}\right)$ immediately implies ${\Delta }_k=\mathcal{O}\left(e^{{\left(k\right)}^3}\right)$.

Finally, the DFP update is applied only after $x^{(k+1)}$ has been computed, so it does not enter the current local error equation. Its role is to generate $H_N^{(k+1)}$, and the assumed first-order consistency of this matrix supplies the induction hypothesis for the next iteration. Therefore, the BFGS–DFP double updating strategy preserves the cubic local convergence order under the stated assumptions.    □

To continue the formal analysis of the convergence properties of the HOQN method, we adopt the following notation and assumptions over the bounded level set

$$\mathcal{L}:=\left\{x\in {\mathbb{R}}^n:f\left(x\right)\le f\left(x^{\left(0\right)}\right)\right\}.$$

Throughout this work, the symbols ⪯, ⪰, and $\succ 0$ are understood in the Loewner sense for symmetric matrices; in particular, $H^{\left(k\right)}⪰\lambda I$ means that $H^{\left(k\right)}-\lambda I$ is positive semidefinite, whereas $H^{\left(k\right)}\succ 0$ means that $H^{\left(k\right)}$ is symmetric positive definite.

(B1)

The objective function $f:{\mathbb{R}}^n\to \mathbb{R}$ is twice continuously differentiable on an open neighborhood of $\mathcal{L}$, i.e., $f\in C^2\left({\mathbb{R}}^n\right)$.

(B2)

There exist constants $0<m\le M$ such that $mI⪯{\nabla }^{2}f\left(x\right)⪯MI,\forall x\in \mathcal{L}$. In particular, $\nabla f$ is Lipschitz continuous on $\mathcal{L}$.

(B3)

The initial matrix $H^{\left(0\right)}$ is symmetric positive definite, and the sequence $\left\{H^{\left(k\right)}\right\}$ generated by the quasi-Newton updates satisfies $H^{\left(k\right)}⪰\lambda I,\forall k\ge 0$, for some constant $\lambda >0$.

(B4)

The step sizes ${\alpha }_N^{\left(k\right)}$ and ${\alpha }_{HO}^{\left(k\right)}$ associated with the Newton and higher–order directions, respectively, are obtained by inexact line searches satisfying the strong Wolfe conditions with constants $0<c_1<c_2<1$.

Lemma 2

(Strong Descent Condition). Let

$$\underline{\lambda }:=\underset{k\ge 0}{\inf }{\lambda }_{\min }\left(H^{\left(k\right)}\right)>0,$$

where ${\lambda }_{\min }\left(H^{\left(k\right)}\right)$ denotes the smallest eigenvalue of the symmetric positive–definite (SPD) matrix $H^{\left(k\right)}$. Under assumptions(B1)–(B4), the Newton direction $d_N^{\left(k\right)}$ and the high–order correction $d_{HO}^{\left(k\right)}$$z^{\left(k\right)}$ at the k-th iteration of Algorithm 1 satisfy

$$\begin{array}{cc}\hfill \nabla f{\left(x^{\left(k\right)}\right)}^{\mathrm{T}}{d}_N^{\left(k\right)}& \le -\underline{\lambda }{∥\nabla f\left(x^{\left(k\right)}\right)∥}^2,\hfill \end{array}$$

(62)

$$\begin{array}{cc}\hfill \nabla f{\left(z^{\left(k\right)}\right)}^{\mathrm{T}}{d}_{HO}^{\left(k\right)}& \le -\underline{\lambda }{∥\nabla f\left(z^{\left(k\right)}\right)∥}^2.\hfill \end{array}$$

(63)

Consequently, the step sizes ${\alpha }_N^{\left(k\right)},{\alpha }_{HO}^{\left(k\right)}>0$ returned by two independent strong–Wolfe line searches yield the one–step decrease

$$f\left(x^{(k+1)}\right)\le f\left(x^{\left(k\right)}\right)-c_1\underline{\lambda }\left({\alpha }_N^{\left(k\right)}{∥\nabla f\left(x^{\left(k\right)}\right)∥}^2+{\alpha }_{HO}^{\left(k\right)}{∥\nabla f\left(z^{\left(k\right)}\right)∥}^2\right).$$

(64)

Proof. 

Let $g^{\left(k\right)}:=\nabla f\left(x^{\left(k\right)}\right)$ and $g_z^{\left(k\right)}:=\nabla f\left(z^{\left(k\right)}\right)$ with $z^{\left(k\right)}:=x^{\left(k\right)}+{\alpha }_N^{\left(k\right)}{d}_N^{\left(k\right)}$. Let $H^{\left(k\right)}\succ 0$ be the (inverse–Hessian) approximation at $x^{\left(k\right)}$ and assume $H^{\left(k\right)}⪰\underline{\lambda }I$ with $\underline{\lambda }:={\inf }_{k\ge 0}{\lambda }_{\min }\left(H^{\left(k\right)}\right)>0$. By the Rayleigh inequality,

$${\left(g^{\left(k\right)}\right)}^{\mathrm{T}}{H}^{\left(k\right)}{g}^{\left(k\right)}\ge {\lambda }_{\min }\left(H^{\left(k\right)}\right)\parallel g^{\left(k\right)}{\parallel }^2\ge \underline{\lambda }{\parallel g^{\left(k\right)}\parallel }^2.$$

Defining the Newton direction by $d_N^{\left(k\right)}:=-H^{\left(k\right)}{g}^{\left(k\right)}$, we obtain

$${\left(g^{\left(k\right)}\right)}^{\mathrm{T}}{d}_N^{\left(k\right)}=-{\left(g^{\left(k\right)}\right)}^{\mathrm{T}}{H}^{\left(k\right)}{g}^{\left(k\right)}\le -\underline{\lambda }{\parallel g^{\left(k\right)}\parallel }^2.$$

which proves (62).

We consider the three high–order directions used in Algorithm 1 and show, in each case, that

$$g_z^{\mathrm{T}}{d}_{HO}^{\left(\mathrm{k}\right)}\le -\underline{\lambda }{\parallel g_z\parallel }^2.$$

To analyze the uniform spectral bound associated with the $M_1$, $M_2$, and $M_3$ directions, we proceed separately for each high-order correction.

Since ${\nu }_k={\displaystyle \frac{{∥g_z^{\left(k\right)}∥}^2}{{∥g^{\left(k\right)}∥}^2}}$ and the iteration is considered before termination, it follows that ${\nu }_k>0$. Moreover, since $H^{\left(k\right)}⪰\underline{\lambda }I$, we have

$${\left(g_z^{\left(k\right)}\right)}^{\mathrm{T}}{H}^{\left(k\right)}{g}_z^{\left(k\right)}\ge \underline{\lambda }{\parallel g_z^{\left(k\right)}\parallel }^2.$$

In addition, we assume that, in the local regime,

$${\left(g_z^{\left(k\right)}\right)}^{\mathrm{T}}{H}^{\left(k\right)}{g}^{\left(k\right)}\ge 0.$$

(i)

$M_1$ (HOChun): For the Chun-type direction, define

$$d_{HOChun}^{\left(k\right)}:=-H^{\left(k\right)}\left((1+{\nu }_k)g_z^{\left(k\right)}-2{\nu }_k{g}^{\left(k\right)}\right).$$

Then,

$$\begin{array}{cc}\hfill -{\left(g_z^{\left(k\right)}\right)}^{\mathrm{T}}{d}_{HOChun}^{\left(k\right)}& =(1+{\nu }_k){\left(g_z^{\left(k\right)}\right)}^{\mathrm{T}}{H}^{\left(k\right)}{g}_z^{\left(k\right)}-2{\nu }_k{\left(g_z^{\left(k\right)}\right)}^{\mathrm{T}}{H}^{\left(k\right)}{g}^{\left(k\right)}\hfill \\ & \ge (1+{\nu }_k)\underline{\lambda }{\parallel g_z^{\left(k\right)}\parallel }^2-2{\nu }_k{\left(g_z^{\left(k\right)}\right)}^{\mathrm{T}}{H}^{\left(k\right)}{g}^{\left(k\right)}\hfill \\ & \ge \underline{\lambda }{\parallel g_z^{\left(k\right)}\parallel }^2.\hfill \end{array}$$

Therefore,

$${\left(g_z^{\left(k\right)}\right)}^{\mathrm{T}}{d}_{HOChun}^{\left(k\right)}\le -\underline{\lambda }{\parallel g_z^{\left(k\right)}\parallel }^2.$$

(ii)

$M_2$ (HOOS): For the Ostrowski-type direction, define

$$d_{HOOS}^{\left(k\right)}:=-{\displaystyle \frac{1}{1-4{\nu }_k}}{H}^{\left(k\right)}\left(g_z^{\left(k\right)}+2{\nu }_k{g}^{\left(k\right)}\right).$$

Then,

$$\begin{array}{cc}\hfill -{\left(g_z^{\left(k\right)}\right)}^{\mathrm{T}}{d}_{HOOS}^{\left(k\right)}& ={\displaystyle \frac{1}{1-4{\nu }_k}}{\left(g_z^{\left(k\right)}\right)}^{\mathrm{T}}{H}^{\left(k\right)}{g}_z^{\left(k\right)}+{\displaystyle \frac{2{\nu }_k}{1-4{\nu }_k}}{\left(g_z^{\left(k\right)}\right)}^{\mathrm{T}}{H}^{\left(k\right)}{g}^{\left(k\right)}\hfill \\ & \ge {\displaystyle \frac{\underline{\lambda }}{1-4{\nu }_k}}{\parallel g_z^{\left(k\right)}\parallel }^2,\hfill \end{array}$$

provided that $1-4{\nu }_k>0$. If the factor ${\displaystyle \frac{1}{1-4{\nu }_k}}$ is absorbed into the line-search step size, we obtain

$${\left(g_z^{\left(k\right)}\right)}^{\mathrm{T}}{d}_{HOOS}^{\left(k\right)}\le -\underline{\lambda }{\parallel g_z^{\left(k\right)}\parallel }^2.$$

(iii)

$M_3$ (HOT): For the Traub-type direction, define

$$d_{HOT}^{\left(k\right)}:=-H^{\left(k\right)}\left((1+2{\nu }_k)g^{\left(k\right)}+(1+G_1{\nu }_k)g_z^{\left(k\right)}\right),G_1\ge 0.$$

Then,

$$\begin{array}{cc}\hfill -{\left(g_z^{\left(k\right)}\right)}^{\mathrm{T}}{d}_{HOT}^{\left(k\right)}& =(1+2{\nu }_k){\left(g_z^{\left(k\right)}\right)}^{\mathrm{T}}{H}^{\left(k\right)}{g}^{\left(k\right)}+(1+G_1{\nu }_k){\left(g_z^{\left(k\right)}\right)}^{\mathrm{T}}{H}^{\left(k\right)}{g}_z^{\left(k\right)}\hfill \\ & \ge (1+2{\nu }_k){\left(g_z^{\left(k\right)}\right)}^{\mathrm{T}}{H}^{\left(k\right)}{g}^{\left(k\right)}+(1+G_1{\nu }_k)\underline{\lambda }{\parallel g_z^{\left(k\right)}\parallel }^2\hfill \\ & \ge \underline{\lambda }{\parallel g_z^{\left(k\right)}\parallel }^2.\hfill \end{array}$$

Therefore,

$${\left(g_z^{\left(k\right)}\right)}^{\mathrm{T}}{d}_{HOT}^{\left(k\right)}\le -\underline{\lambda }{\parallel g_z^{\left(k\right)}\parallel }^2.$$

In all three cases, (63) holds. Applying the first Wolfe condition at $x^{\left(k\right)}$, we obtain

$$f\left(x^{\left(k\right)}+{\alpha }_N^{\left(k\right)}{d}_N^{\left(k\right)}\right)\le f\left(x^{\left(k\right)}\right)+c_1{\alpha }_N^{\left(k\right)}{\left(g^{\left(k\right)}\right)}^{\mathrm{T}}{d}_N^{\left(k\right)}.$$

Using (62), it follows that

$$f\left(z^{\left(k\right)}\right)\le f\left(x^{\left(k\right)}\right)-c_1\underline{\lambda }{\alpha }_N^{\left(k\right)}{\parallel g^{\left(k\right)}\parallel }^2.$$

(65)

Now apply the first Wolfe condition at $z^{\left(k\right)}$ for the high-order step:

$$f\left(z^{\left(k\right)}+{\alpha }_{HO}^{\left(k\right)}{d}_{HO}^{\left(k\right)}\right)\le f\left(z^{\left(k\right)}\right)+c_1{\alpha }_{HO}^{\left(k\right)}{\left(g_z^{\left(k\right)}\right)}^{\mathrm{T}}{d}_{HO}^{\left(k\right)}.$$

Substituting (63) and then (65), we obtain

$$f\left(x^{(k+1)}\right)\le f\left(x^{\left(k\right)}\right)-c_1\underline{\lambda }\left({\alpha }_N^{\left(k\right)}\parallel g^{\left(k\right)}{\parallel }^2+{\alpha }_{HO}^{\left(k\right)}{\parallel g_z^{\left(k\right)}\parallel }^2\right).$$

which is exactly (64). The proof is complete.    □

Lemma 3

(Asymptotic gradient nulling). Let ${\left\{x^{\left(k\right)}\right\}}_{k\ge 0}$ be the sequence produced by Algorithm 1. Suppose(B1)–(B4)hold, and that the search directions generated by the algorithm satisfy the uniform–descent and bounded–direction properties stated in Lemma 2. Then

$$\underset{k\to \infty }{\lim }∥\nabla f\left(x^{\left(k\right)}\right)∥=0.$$

Proof. 

We now prove this result.

(i)

By Lemma 2, we know that

$$\begin{array}{cc}\hfill f\left(x^{(k+1)}\right)& \le f\left(x^{\left(k\right)}\right)-\underline{\lambda }\left(c_1{\alpha }_N^{\left(k\right)}\parallel g^{\left(k\right)}{\parallel }^2+{\overline{c}}_1{\alpha }_{HO}^{\left(k\right)}{\parallel g_{z^{\left(k\right)}}\parallel }^2\right).\hfill \end{array}$$

(ii)

Using (B2), f attains a minimum on the set $\mathcal{L}=\{x\in {\mathbb{R}}^n:f\left(x\right)\le f\left(x^{\left(0\right)}\right)\}$. Let $f_{\inf }:={\inf }_{x\in \mathcal{L}}f\left(x\right)$. Then $f_{\inf }\le f\left(x\right)$ for all $x\in \mathcal{L}$. Hence, for the iterates $\left\{x^{\left(k\right)}\right\}$ we obtain

$$\begin{array}{cc}\hfill f\left(x^{\left(1\right)}\right)& \le f\left(x^{\left(0\right)}\right)-\underline{\lambda }\left(c_1{\alpha }_0^N\parallel g_0{\parallel }^2+{\overline{c}}_1{\alpha }_0^{HO}{\parallel g_{z_0}\parallel }^2\right),\hfill \\ \hfill f\left(x^{\left(2\right)}\right)& \le f\left(x^{\left(1\right)}\right)-\underline{\lambda }\left(c_1{\alpha }_1^N\parallel g_1{\parallel }^2+{\overline{c}}_1{\alpha }_1^{HO}{\parallel g_{z_1}\parallel }^2\right)\hfill \\ \hfill & \le f\left(x^{\left(0\right)}\right)-\underline{\lambda }{\displaystyle \sum _{j=0}^1}\left(c_1{\alpha }_j^N\parallel g_j{\parallel }^2+{\overline{c}}_1{\alpha }_j^{HO}{\parallel g_{z_j}\parallel }^2\right),\hfill \\ \hfill f\left(x^{\left(3\right)}\right)& \le f\left(x^{\left(2\right)}\right)-\underline{\lambda }\left(c_1{\alpha }_2^N\parallel g_2{\parallel }^2+{\overline{c}}_1{\alpha }_2^{HO}{\parallel g_{z_2}\parallel }^2\right)\hfill \\ \hfill & \le f\left(x^{\left(0\right)}\right)-\underline{\lambda }{\displaystyle \sum _{j=0}^2}\left(c_1{\alpha }_j^N\parallel g_j{\parallel }^2+{\overline{c}}_1{\alpha }_j^{HO}{\parallel g_{z_j}\parallel }^2\right),\hfill \end{array}$$

(66)

and, in general,

$$f\left(x^{(k+1)}\right)\le f\left(x^{\left(0\right)}\right)-\underline{\lambda }{\displaystyle \sum _{j=0}^k}\left(c_1{\alpha }_j^N\parallel g_j{\parallel }^2+{\overline{c}}_1{\alpha }_j^{HO}{\parallel g_{z_j}\parallel }^2\right).$$

Letting $k\to \infty$ yields

$$f_{\inf }\le f\left(x^{\left(0\right)}\right)-\underline{\lambda }\left(c_1{\displaystyle \sum _{j=0}^{\infty }}{\alpha }_j^N\parallel g_j{\parallel }^2+{\overline{c}}_1{\displaystyle \sum _{j=0}^{\infty }}{\alpha }_j^{HO}{\parallel g_{z_j}\parallel }^2\right),$$

hence both series

$${\displaystyle \sum _{j=0}^{\infty }}{\alpha }_j^N\parallel g_j{\parallel }^2\mathrm{and}{\displaystyle \sum _{j=0}^{\infty }}{\alpha }_j^{HO}{\parallel g_{z_j}\parallel }^2$$

are convergent.

(iii)

Now, let us prove that the step sizes ${\alpha }_N^{\left(k\right)}$ and ${\alpha }_{HO}^{\left(k\right)}$ are bounded away from zero whenever the corresponding gradients remain bounded away from zero. Let $\varphi \left(\alpha \right):=f(x+\alpha d)$, so

$${\varphi }^{\prime }\left(\alpha \right)=\nabla f{(x+\alpha d)}^{\mathrm{T}}d\mathrm{and}{\varphi }^{\prime }\left(0\right)=\nabla f{\left(x\right)}^{\mathrm{T}}d=g^{\mathrm{T}}d.$$

Assume $\nabla f$ is L–Lipschitz, that is,

$$\parallel \nabla f\left(u\right)-\nabla f\left(v\right)\parallel \le L\parallel u-v\parallel ,\forall u,v.$$

We consider $u=x+\alpha d$ and $v=x$. Then,

$$\parallel \nabla f(x+\alpha d)-\nabla f\left(x\right)\parallel \le L\parallel (x+\alpha d)-x\parallel =L\alpha \parallel d\parallel .$$

Then, for any $\alpha >0$,

$$\begin{array}{cc}\hfill {\varphi }^{\prime }\left(\alpha \right)-{\varphi }^{\prime }\left(0\right)& ={(\nabla f(x+\alpha d)-\nabla f\left(x\right))}^{\mathrm{T}}d\hfill \\ \hfill & \ge -\parallel \nabla f(x+\alpha d)-\nabla f\left(x\right)\parallel \parallel d\parallel \hfill \\ \hfill & \ge -{L\alpha \parallel d\parallel }^2.\hfill \end{array}$$

(67)

Hence,

$${\varphi }^{\prime }\left(\alpha \right)\ge {\varphi }^{\prime }\left(0\right)-L\alpha {\parallel d\parallel }^2.$$

(68)

Let $\alpha$ satisfy the strong Wolfe curvature condition

$$|{\varphi }^{\prime }\left(\alpha \right)|\le c_2\left|{\varphi }^{\prime }\left(0\right)\right|,0<c_2<1.$$

Then

$${\varphi }^{\prime }\left(\alpha \right)\ge -|{\varphi }^{\prime }\left(\alpha \right)|\ge -c_2\left|{\varphi }^{\prime }\left(0\right)\right|.$$

Combining with (68), we obtain

$${\varphi }^{\prime }\left(0\right)-{L\alpha \parallel d\parallel }^2\ge -c_2\left|{\varphi }^{\prime }\left(0\right)\right|.$$

Thus,

$${\varphi }^{\prime }\left(0\right)+c_2|{\varphi }^{\prime }{\left(0\right)|\ge L\alpha \parallel d\parallel }^2.$$

Since d is a descent direction, ${\varphi }^{\prime }\left(0\right)=g^{\mathrm{T}}d<0$ and thus $|{\varphi }^{\prime }\left(0\right)|=-{\varphi }^{\prime }\left(0\right)$. Replacing, we get

$${\varphi }^{\prime }\left(0\right)+c_2\left|{\varphi }^{\prime }\left(0\right)\right|={\varphi }^{\prime }\left(0\right)-c_2{\varphi }^{\prime }\left(0\right)=(1-c_2){\varphi }^{\prime }\left(0\right).$$

Therefore, the equivalent inequality is

$$-(1-c_2)|{\varphi }^{\prime }\left(0\right){|\ge -L\alpha \parallel d\parallel }^2⟹(1-c_2)|{\varphi }^{\prime }\left(0\right){|\le L\alpha \parallel d\parallel }^2.$$

$$\alpha \ge {\displaystyle \frac{(1-c_2)\left|{\varphi }^{\prime }\left(0\right)\right|}{{L\parallel d\parallel }^2}}={\displaystyle \frac{(1-c_2)\left|g^{\mathrm{T}}d\right|}{{L\parallel d\parallel }^2}}.$$

(69)

Assume there exist constants $\underline{\lambda }>0$ and $C>0$ such that

$$|g^{\mathrm{T}}d|\ge \underline{\lambda }{\parallel g\parallel }^2,\parallel d\parallel \le C\parallel g\parallel .$$

(70)

Then ${\parallel d\parallel }^2\le C^2{\parallel g\parallel }^2$ and, substituting (70) into (69), we obtain

$$\alpha \ge {\displaystyle \frac{(1-c_2)\underline{\lambda }{\parallel g\parallel }^2}{L{C}^2{\parallel g\parallel }^2}}={\displaystyle \frac{(1-c_2)\underline{\lambda }}{L{C}^2}}={\alpha }_{\min }>0.$$

Hence, if $\parallel g^{\left(k\right)}\parallel \ge \epsilon$, then ${\alpha }_N^{\left(k\right)}\ge {\alpha }_{\min }$; similarly, if $\parallel g\left(z^{\left(k\right)}\right)\parallel \ge \epsilon$, then ${\alpha }_{HO}^{\left(k\right)}\ge {\alpha }_{\min }$. ]

Assume by contradiction that ${\mathrm{lim\; sup}}_{k\to \infty }\parallel g^{\left(k\right)}\parallel =\eta >0$. Then, for any $\epsilon >0$, there exist infinitely many indices k such that $\parallel g^{\left(k\right)}\parallel \ge \eta -\epsilon$; choosing $\epsilon =\eta /2$ yields a subsequence $\left\{k_{\ell }\right\}$ satisfying

$$\parallel g^{\left(k_{\ell }\right)}\parallel \ge \eta /2,\forall \ell .$$

By (iii), ${\alpha }_N^{\left(k_{\ell }\right)}\ge {\alpha }_{\min }$. Therefore,

$${\alpha }_N^{\left(k_{\ell }\right)}{\parallel g^{\left(k_{\ell }\right)}\parallel }^2\ge {\alpha }_{\min }{\left({\displaystyle \frac{\eta }{2}}\right)}^2,\forall \ell .$$

Summing over ℓ yields

$${\displaystyle \sum _{\ell =1}^{\infty }}{\alpha }_N^{\left(k_{\ell }\right)}{\parallel g^{\left(k_{\ell }\right)}\parallel }^2\ge {\displaystyle \sum _{\ell =1}^{\infty }}{\alpha }_{\min }{\left({\displaystyle \frac{\eta }{2}}\right)}^2=\infty .$$

On the other hand, by (ii) we have

$${\displaystyle \sum _{k=0}^{\infty }}{\alpha }_N^{\left(k\right)}{\parallel g^{\left(k\right)}\parallel }^2<\infty ,$$

which contradicts the divergence of the subseries above. Since

$$g^{\left(k\right)}:=\nabla f\left(x^{\left(k\right)}\right),$$

the result follows immediately. Hence,

$$\underset{k\to \infty }{\lim }\parallel g^{\left(k\right)}\parallel =\underset{k\to \infty }{\lim }\parallel g\left(z^{\left(k\right)}\right)\parallel =0.$$

   □

Remark 3

(Worst-case complexity for first-order stationarity). The descent estimate in Lemma 2 and the step-size lower bound established in Lemma 3 yield a standard worst-case complexity bound. Let

$$g^{\left(k\right)}:=\nabla f\left(x^{\left(k\right)}\right),g_z^{\left(k\right)}:=\nabla f\left(z^{\left(k\right)}\right),f_{\inf }:=\underset{x\in \mathcal{L}}{\inf }f\left(x\right),$$

where $\mathcal{L}=\{x\in {\mathbb{R}}^n:f\left(x\right)\le f\left(x^{\left(0\right)}\right)\}$. If, before termination, If, before termination,

$${\alpha }_N^{\left(k\right)}\ge {\alpha }_{\min }>0,{\alpha }_{HO}^{\left(k\right)}\ge {\alpha }_{\min }>0,\sigma :=\underline{\lambda }{\alpha }_{\min }\min \{c_1,{\overline{c}}_1\},$$

then Lemma 2 givesthen Lemma 2 gives

$$f\left(x^{(k+1)}\right)\le f\left(x^{\left(k\right)}\right)-\sigma \left(\parallel g^{\left(k\right)}{\parallel }^2+{\parallel g_z^{\left(k\right)}\parallel }^2\right).$$

Summing from $k=0$ to $N-1$, we obtain

$$\sigma {\displaystyle \sum _{k=0}^{N-1}}\left(\parallel g^{\left(k\right)}{\parallel }^2+{\parallel g_z^{\left(k\right)}\parallel }^2\right)\le f\left(x^{\left(0\right)}\right)-f_{\inf }.$$

In particular,

$$\underset{0\le k\le N-1}{\min }\parallel g^{\left(k\right)}\parallel \le \sqrt{{\displaystyle \frac{f\left(x^{\left(0\right)}\right)-f_{\inf }}{\sigma N}}}.$$

Therefore, to guarantee an index $0\le k\le N-1$ such that

$$\parallel \nabla f\left(x^{\left(k\right)}\right)\parallel \le \epsilon ,$$

it is sufficient to take

$$N\ge {\displaystyle \frac{f\left(x^{\left(0\right)}\right)-f_{\inf }}{\sigma {\epsilon }^2}}.$$

Hence, the worst-case number of outer iterations required to reach first-order stationarity is

$$N_{\epsilon }=\mathcal{O}\left({\epsilon }^{-2}\right).$$

Since each full-memory HOQN iteration involves a finite number of matrix-vector products and rank-two quasi-Newton updates, its dense algebraic cost is $O\left(n^2\right)$. Consequently, the dense worst-case arithmetic complexity is

$$\mathcal{O}\left(n^2{\epsilon }^{-2}\right).$$

Lemma 4

(Dennis–Moré condition). Let $f:{\mathbb{R}}^n\to \mathbb{R}$ be of class $C^2$ near ξ, with ${\nabla }^{2}f\left(\xi \right)\succ 0$, and let $\left\{x^{\left(k\right)}\right\}$ be generated by Algorithm 1. Set

$$e^{\left(k\right)}:=x^{\left(k\right)}-\xi ,s^{\left(k\right)}:=x^{(k+1)}-x^{\left(k\right)},$$

and define

$$d^{\left(k\right)}:=-H^{\left(k\right)}g\left(x^{\left(k\right)}\right),d_N^{\left(k\right)}:=-{\nabla }^{2}f{\left(\xi \right)}^{-1}g\left(x^{\left(k\right)}\right).$$

Assume that $\left\{H^{\left(k\right)}\right\}$ is updated by (1), that $\left\{H^{\left(k\right)}\right\}$ is bounded, that $x^{\left(k\right)}\to \xi$ superlinearly, and that the Dennis–Moré condition holds:

$$\underset{k\to \infty }{\lim }{\displaystyle \frac{\parallel (H^{\left(k\right)}-{\nabla }^{2}f{\left(\xi \right)}^{-1})s^{\left(k\right)}\parallel }{\parallel s^{\left(k\right)}\parallel }}=0.$$

Then

$${\displaystyle \frac{\parallel d^{\left(k\right)}-d_N^{\left(k\right)}\parallel }{\parallel d_N^{\left(k\right)}\parallel }}\to 0.$$

Proof. 

Let

$$H^{*}:={\nabla }^{2}f{\left(\xi \right)}^{-1},g^{\left(k\right)}:=g\left(x^{\left(k\right)}\right).$$

By definition,

$$d^{\left(k\right)}-d_N^{\left(k\right)}=-\left(H^{\left(k\right)}-H^{*}\right)g^{\left(k\right)}.$$

(71)

Using the first-order expansion of the gradient around $\xi$, and the superlinear convergence of $x^{\left(k\right)}$, we have

$$g^{\left(k\right)}={\nabla }^{2}f\left(\xi \right)e^{\left(k\right)}+\mathcal{O}(\parallel e^{\left(k\right)}{\parallel }^2),s^{\left(k\right)}=e^{(k+1)}-e^{\left(k\right)}=-e^{\left(k\right)}+o(\parallel e^{\left(k\right)}\parallel ).$$

Hence

$$e^{\left(k\right)}=-s^{\left(k\right)}+o(\parallel s^{\left(k\right)}\parallel ),g^{\left(k\right)}=-{\nabla }^{2}f\left(\xi \right)s^{\left(k\right)}+o(\parallel s^{\left(k\right)}\parallel ).$$

Substituting this expression for $g^{\left(k\right)}$ into (71) gives

$$d^{\left(k\right)}-d_N^{\left(k\right)}=\left(H^{\left(k\right)}-H^{*}\right){\nabla }^{2}f\left(\xi \right)s^{\left(k\right)}+o(\parallel s^{\left(k\right)}\parallel ).$$

Since ${\nabla }^{2}f\left(\xi \right)$ is fixed and $\left\{H^{\left(k\right)}\right\}$ is bounded, there exists $C>0$ such that

$$\parallel d^{\left(k\right)}-d_N^{\left(k\right)}\parallel \le C\parallel (H^{\left(k\right)}-H^{*})s^{\left(k\right)}\parallel +o(\parallel s^{\left(k\right)}\parallel ).$$

Dividing by $\parallel s^{\left(k\right)}\parallel$ and using the Dennis–Moré condition yields

$${\displaystyle \frac{\parallel d^{\left(k\right)}-d_N^{\left(k\right)}\parallel }{\parallel s^{\left(k\right)}\parallel }}\to 0.$$

Moreover, since

$$d_N^{\left(k\right)}=-H^{*}{g}^{\left(k\right)}=s^{\left(k\right)}+o(\parallel s^{\left(k\right)}\parallel ),$$

we have $\parallel d_N^{\left(k\right)}\parallel \sim \parallel s^{\left(k\right)}\parallel$. Therefore,

$${\displaystyle \frac{\parallel d^{\left(k\right)}-d_N^{\left(k\right)}\parallel }{\parallel d_N^{\left(k\right)}\parallel }}\to 0,$$

which proves the result.    □

5. Numerical Tests

In this section, we numerically compare the proposed HOQN variants ($M_1\mathrm{DFP}$, $M_2\mathrm{DFP}$, $M_3\mathrm{DFP}$, $\mathrm{B}{M}_1\mathrm{D}$, $\mathrm{B}{M}_2\mathrm{D}$, and $\mathrm{B}{M}_3\mathrm{D}$) with the classical quasi-Newton methods BFGS, DFP, and SR1, using the Booth test function $f_B$, the Himmelblau function $f_H$, and the Freudenstein–Roth function $f_{FR}$. All numerical experiments were carried out in MATLAB R2024. The iterative process was terminated when the gradient norm satisfied the stopping criterion $∥\nabla f\left(x^{\left(k\right)}\right)∥<{10}^{-6}$.

Although the global convergence analysis is developed under standard inexact line-search assumptions of Armijo/Wolfe type, the numerical implementation reported in this section computes the step sizes in both the predictor and corrector stages by bounded one-dimensional minimization using MATLAB’s fminbnd routine over the interval $[0,10]$. Thus, the theoretical framework provides sufficient conditions for the convergence analysis, whereas fminbnd is adopted as a practical step-selection procedure in the computational experiments. Next, we define the test functions:

• $f_B\left(x_1,x_2\right)={\left(x_1+2x_2-7\right)}^2+{\left(2x_1+x_2-5\right)}^2$;
• $f_H\left(x_1,x_2\right)={\left(x_1^2+x_2-11\right)}^2+{\left(x_1+x_2^2-7\right)}^2$;
• $f_{FR}\left(x_1,x_2\right)={\left(x_1+x_2(x_2(x_2+1)-14)-29\right)}^2+{\left(-x_1+x_2(x_2(x_2-5)+2)+13\right)}^2$.

The Booth function $f_B$ is strictly convex and has a unique global minimizer at ${(1,3)}^T$ with value $f_B\left({(1,3)}^T\right)=0$. The Himmelblau function has four local minima which are also global minima, all satisfying $f_H=0$: ${(3,2)}^T$, ${(-2.805118,3.131312)}^T$, ${(-3.779310,-3.283186)}^T$, and ${(3.584428,-1.848126)}^T$. The Freudenstein–Roth function has a global minimizer at ${(5,4)}^T$ with $f_{FR}\left({(5,4)}^T\right)=0$, a well-known local minimizer at approximately ${(11.4128,-0.8968)}^T$, and it also exhibits a saddle point at approximately ${(23.9206,2.2300)}^T$. The 3D landscapes in Figure 1 visualize these geometries: Figure 1a corresponds to the convex, unimodal $f_B$, whereas Figure 1b and Figure 1c show the multimodal landscapes of $f_H$ and $f_{FR}$, respectively, providing intuition about the basins and valleys against which the HOQN variants are assessed.

A particularly important metric in numerical tests of iterative methods is the Approximate Computational Order of Convergence (ACOC), as it allows for the experimental validation of the theoretical order of convergence. In the context of hybrid optimization algorithms, estimating the convergence order, denoted by $\rho$, using the classical formula proposed in [25], often leads to unreliable results due to significant fluctuations between consecutive iterations. To obtain a more stable and consistent estimate of $\rho$, we adopt an approach based on the linear regression of the logarithms of successive errors $(\ln E_k,\ln E_{k+1})$. This methodology leverages global information from multiple iterations within the asymptotic regime, thereby reducing sensitivity to local fluctuations in individual errors and providing a more robust estimate than formulas based solely on the most recent iterations. In particular, the ACOC is calculated as the slope of the least-squares regression line that fits the data $\ln E_{k+1}$ versus $\ln E_k$, and is given by:

$$\rho ={\displaystyle \frac{{\displaystyle {\displaystyle \sum _{k=k_0}^{m-1}}\left(\ln E_k-\overline{\ln E_k}\right)\left(\ln E_{k+1}-\overline{\ln E_{k+1}}\right)}}{{\displaystyle {\displaystyle \sum _{k=k_0}^{m-1}}{\left(\ln E_k-\overline{\ln E_k}\right)}^2}}},$$

(72)

where $E_k=\parallel x^{\left(k\right)}-\xi \parallel$ denotes the error at iteration k, m is the total number of iterations considered, $\overline{\ln E_k}={\displaystyle \frac{1}{N}}{\sum }_{k=k_0}^{m-1}\ln E_k$, and $\overline{\ln E_{k+1}}={\displaystyle \frac{1}{N}}{\sum }_{k=k_0}^{m-1}\ln E_{k+1}$, with $N=m-k_0$ denoting the number of data points used in the regression. Furthermore, $k_0$ denotes the first iteration from which the iterates are considered to lie in the asymptotic regime.

5.1. Hybrid Optimization with Complete Memory

We begin by analyzing the full memory variants of the proposed hybrid methods, comparing their numerical performance with classical quasi-Newton schemes on selected test functions.

Table 1. Numerical tests for the Himmelblau function with two different initial estimates.

Himmelblau Function $\left({\mathit{f}}_{\mathit{H}}\right)$, $\mathit{\xi }={(-3.7793,-3.2832)}^{\mathit{T}}$
$x^{\left(0\right)}={(-2.2920,-2.6501)}^T$
Method	Iter	$\rho$	$\parallel \nabla f\left(x^{(k+1)}\right)\parallel$	CT (s)
BFGS	5	1.8	$3.8318\times {10}^{-7}$	0.0021
DFP	5	1.8	$3.8871\times {10}^{-7}$	0.0013
SR1	5	1.8	$3.8989\times {10}^{-7}$	0.0024
$M_1\mathrm{DFP}$	10	2.9	$2.0383\times {10}^{-12}$	0.0047
$M_2\mathrm{DFP}$	4	2.3	$1.43\times {10}^{-14}$	0.0038
$M_3\mathrm{DFP}$	4	2.5	$7.9947\times {10}^{-8}$	0.0019
$\mathrm{B}{M}_1\mathrm{D}$	3	2.8	$5.7620\times {10}^{-11}$	0.0013
$\mathrm{B}{M}_2\mathrm{D}$	3	2.3	$4.20\times {10}^{-11}$	0.0058
$\mathrm{B}{M}_3\mathrm{D}$	4	2.5	$5.5011\times {10}^{-9}$	0.0040
$x^{\left(0\right)}={(-1.956,-2.667)}^T$
BFGS	6	1.6	$1.0194\times {10}^{-8}$	0.0044
DFP	6	1.6	$1.0500\times {10}^{-8}$	0.0040
SR1	6	1.6	$1.0491\times {10}^{-8}$	0.0015
$M_1\mathrm{DFP}$	12	3.0	$3.9892\times {10}^{-7}$	0.0029
$M_2\mathrm{DFP}$	4	2.4	$3.81\times {10}^{-7}$	0.0015
$M_3\mathrm{DFP}$	4	2.7	$5.9150\times {10}^{-7}$	0.0039
$\mathrm{B}{M}_1\mathrm{D}$	3	3.2	$3.8079\times {10}^{-8}$	0.0058
$\mathrm{B}{M}_2\mathrm{D}$	3	2.8	$6.24\times {10}^{-7}$	0.0050
$\mathrm{B}{M}_3\mathrm{D}$	4	2.4	$9.3926\times {10}^{-8}$	0.0012

and

Table 2. Numerical tests for the Freudenstein–Roth function with two different initial estimates.

Freudenstein–Roth Function $\left({\mathit{f}}_{\mathit{FR}}\right)$, $\mathit{\xi }={(5,4)}^{\mathit{T}}$
$x^{\left(0\right)}={(3.5081,4.0087)}^T$
Method	Iter	$\rho$	$\parallel \nabla f\left(x^{(k+1)}\right)\parallel$	CT (s)
BFGS	4	1.3	$2.2231\times {10}^{-7}$	0.0068
DFP	4	1.3	$2.2179\times {10}^{-7}$	0.0066
SR1	4	1.3	$2.2171\times {10}^{-7}$	0.0044
$M_1\mathrm{DFP}$	5	3.0	$6.3499\times {10}^{-10}$	0.0043
$M_2\mathrm{DFP}$	3	2.8	$1.01\times {10}^{-8}$	0.0017
$M_3\mathrm{DFP}$	3	2.6	$7.0095\times {10}^{-7}$	0.0088
$\mathrm{B}{M}_1\mathrm{D}$	2	–	$8.7848\times {10}^{-8}$	0.0064
$\mathrm{B}{M}_2\mathrm{D}$	2	–	$4.00\times {10}^{-7}$	0.0038
$\mathrm{B}{M}_3\mathrm{D}$	3	2.6	$7.0595\times {10}^{-7}$	0.0087
$x^{\left(0\right)}={(4.3,4.0001)}^T$
Method	Iter	$\rho$	$\parallel \nabla f\left(x^{(k+1)}\right)\parallel$	CT (s)
BFGS	4	1.2	$2.8710\times {10}^{-7}$	0.0014
DFP	4	1.2	$2.8639\times {10}^{-7}$	0.0010
SR1	4	1.2	$2.8612\times {10}^{-7}$	0.0014
$M_1\mathrm{DFP}$	4	2.5	$5.8292\times {10}^{-8}$	0.0093
$M_2\mathrm{DFP}$	4	3.0	$4.56\times {10}^{-7}$	0.0064
$M_3\mathrm{DFP}$	3	3.0	$1.9586\times {10}^{-7}$	0.0047
$\mathrm{B}{M}_1\mathrm{D}$	2	–	$2.3814\times {10}^{-7}$	0.0012
$\mathrm{B}{M}_2\mathrm{D}$	2	–	$3.40\times {10}^{-7}$	0.0015
$\mathrm{B}{M}_3\mathrm{D}$	3	3.0	$1.9628\times {10}^{-7}$	0.0063

present the numerical results obtained for the Himmelblau and Freudenstein–Roth functions, considering two different initial conditions. The classical quasi-Newton methods BFGS, DFP, and SR1 exhibit superlinear convergence, as is characteristic of this class of methods, requiring between five and six iterations to reach the minimum. On the other hand, with the exception of $M_1\mathrm{DFP}$, the proposed hybrid methods consistently converge in fewer iterations for both functions and initial conditions, achieving observed convergence orders consistent with the theoretical cubic order and computational times competitive with the classical methods considered. Furthermore, regarding the final gradient norm $∥\nabla f\left(x^{(k+1)}\right)∥$, it is observed that, in these nonconvex problems, this metric can vary significantly depending on the initial condition used; however, the smallest values were obtained by some of the proposed hybrid variants. This fact constitutes a clear indication of the greater numerical accuracy of these variants in approximating the minimizer. The reported computation time (CT(s)) in the numerical experiments corresponds to the total execution time of the iterative algorithm, measured in seconds, from the start of the process until the termination criterion is satisfied or the maximum number of iterations is reached. Consequently, this value includes the cost of all operations performed in each iteration. To obtain reliable time measurements, each method was run 15 times for each test problem, and the times reported in the tables correspond to the average of those runs. The numerical experiments were conducted on a 13-inch MacBook Air (2025) running macOS Sequoia 15.7.4, equipped with an Apple M4 chip and 16 GB of unified memory. This computational environment provides a consistent and up-to-date basis for the comparative evaluation of the methods under consideration.

The numerical results indicate that the efficiency of the methods strongly depends on the nature of the problem. In the nonconvex problems, such as the Himmelblau and Freudenstein–Roth functions, the hybrid methods reach the minimizer in fewer iterations and with a faster convergence rate than the classical BFGS and DFP methods, while also yielding smaller final gradient norms.

Let us now analyze the Booth function, which is a two-dimensional strictly convex quadratic problem with unique global minimizer $\xi ={(1,3)}^T$.

Table 3. Numerical tests for the Booth function $f_B$ with two different initial estimates.

Method	Iter	$\mathit{\rho }$	$\parallel \nabla \mathit{f}\left({\mathit{x}}^{(\mathit{k}+1)}\right)\parallel$	CT (s)
Booth Function   ${\mathit{f}}_{\mathit{B}}$,$\mathbf{\xi }={(\mathbf{1},\mathbf{3})}^{\mathit{T}}$
$x^{\left(0\right)}={(3.45,4.08)}^T$
BFGS	2	–	$1.4200\times {10}^{-14}$	0.0046
DFP	2	–	$2.2500\times {10}^{-14}$	0.0020
SR1	2	–	$1.5888\times {10}^{-14}$	0.0038
$M_1\mathrm{DFP}$	2	–	$7.1054\times {10}^{-15}$	0.0041
$M_2\mathrm{DFP}$	2	–	$1.5900\times {10}^{-14}$	0.0038
$M_3\mathrm{DFP}$	2	–	$1.0049\times {10}^{-14}$	0.0023
${\mathit{BM}}_{1}D$	2	–	$0.0000\times {10}^0$	0.0022
${\mathit{BM}}_{2}D$	2	–	$0.0000\times {10}^0$	0.0022
${\mathit{BM}}_{3}D$	2	–	$0.0000\times {10}^0$	0.0068
$x^{\left(0\right)}={(-3,9)}^T$
BFGS	2	–	$3.5500\times {10}^{-14}$	0.0007
DFP	2	–	$2.9300\times {10}^{-14}$	0.0007
SR1	2	–	$5.1238\times {10}^{-14}$	0.0029
$M_1\mathrm{DFP}$	2	–	$2.5600\times {10}^{-14}$	0.0031
$M_2\mathrm{DFP}$	2	–	$1.0000\times {10}^{-14}$	0.0019
$M_3\mathrm{DFP}$	2	–	$7.9441\times {10}^{-14}$	0.0056
${\mathit{BM}}_{1}D$	2	–	$0.0000\times {10}^0$	0.0006
${\mathit{BM}}_{2}D$	2	–	$0.0000\times {10}^0$	0.0022
${\mathit{BM}}_{3}D$	2	–	$0.0000\times {10}^0$	0.0059

reports the numerical results obtained from two different initial conditions. All the methods converge in two iterations and reach final gradient norms of order ${10}^{-14}$ or smaller. Since only two iterations are required, the approximate computational order of convergence cannot be reliably estimated; therefore, the value of $\rho$ is not reported.

The results in Table 3 should be interpreted as a consistency check for a simple strictly convex quadratic problem, not as evidence of a general optimality property of the proposed methods. In exact arithmetic, Newton’s method solves a strictly convex quadratic problem in one step when the exact inverse Hessian is used, while full-memory quasi-Newton schemes with line searches may display very fast finite termination on low-dimensional quadratic problems. Therefore, the two-iteration behavior observed for the Booth function is a finite-dimensional quadratic effect. Consequently, the Booth experiment confirms that all methods behave accurately on a convex quadratic benchmark, but it is not sufficient to assess scalability or general asymptotic behavior. For this reason, the Booth test is complemented below with high-dimensional quadratic experiments with prescribed Hessian condition numbers.

5.2. High-Dimensional Quadratic Scalability Test

Since the Booth function is only a two-dimensional strictly convex quadratic problem, its two-iteration behavior should not be interpreted as evidence of general scalability. To test the methods beyond the low-dimensional Booth case, we consider the strictly convex quadratic family

$$f_{n,\kappa }\left(x\right)={\displaystyle \frac{1}{2}}{x}^T{A}_{n,\kappa }x-b^{T}x,A_{n,\kappa }=Q^T{\mathsf{\Lambda }}_{n,\kappa }Q,{\mathsf{\Lambda }}_{n,\kappa }=\mathrm{diag}\left(1,{\kappa }^{1/(n-1)},{\kappa }^{2/(n-1)},\dots ,\kappa \right),$$

where Q is an orthogonal matrix generated with a fixed random seed. Hence,

$${\nabla }^2{f}_{n,\kappa }\left(x\right)=A_{n,\kappa },\kappa \left(A_{n,\kappa }\right)=\kappa .$$

The vector b was chosen as $b=A_{n,\kappa }\xi$, with $\xi ={\displaystyle \frac{1}{\sqrt{n}}}{(1,\dots ,1)}^T$ so that the exact minimizer is known. We used $x^{\left(0\right)}=0,H^{\left(0\right)}=I$,and the stopping criterion $∥\nabla f\left(x^{\left(k\right)}\right)∥\le {10}^{-6}$. The step length was computed by exact one-dimensional minimization along each search direction, in order to isolate the cost of the quasi-Newton updates from line-search variability. The tested dimensions and condition numbers were $n\in \{100,500,1000\},\kappa \in \left\{{10}^2,{10}^6\right\}$. To assess the expected dense $O\left(n^2\right)$ scaling,

Table 4. Scalability test on high-dimensional strictly convex quadratic problems.

n	$\mathit{\kappa }$	Iterations			Normalized Cost $\mathit{\eta }$
		BFGS	${\mathit{M}}_{\mathbf{1}}\mathbf{DFP}$	${\mathbf{BM}}_{\mathbf{1}}\mathbf{D}$	BFGS	${\mathit{M}}_{\mathbf{1}}\mathrm{DFP}$	${\mathbf{BM}}_{\mathbf{1}}\mathbf{D}$
100	${10}^2$	57	58	34	$2.3662\times {10}^{-8}$	$2.2992\times {10}^{-8}$	$2.9310\times {10}^{-8}$
100	${10}^6$	98	112	58	$1.1032\times {10}^{-8}$	$8.9478\times {10}^{-9}$	$1.5510\times {10}^{-8}$
500	${10}^2$	82	70	49	$1.0261\times {10}^{-8}$	$1.0478\times {10}^{-8}$	$1.6158\times {10}^{-8}$
500	${10}^6$	394	434	220	$8.3910\times {10}^{-9}$	$8.9006\times {10}^{-9}$	$1.3985\times {10}^{-8}$
1000	${10}^2$	87	75	48	$8.2811\times {10}^{-9}$	$1.0092\times {10}^{-8}$	$1.3768\times {10}^{-8}$
1000	${10}^6$	770	800	408	$8.8353\times {10}^{-9}$	$1.5070\times {10}^{-8}$	$2.2218\times {10}^{-8}$

reports the number of outer iterations and the normalized cost $\eta ={\displaystyle \frac{CT}{\mathrm{Iter}{n}^2}}$.

All methods reached the prescribed tolerance in every tested case. The hybrid variant ${\mathit{BM}}_{1}D$ consistently required fewer outer iterations than both BFGS and $M_1\mathrm{DFP}$. In the most demanding case, $n=1000,\kappa ={10}^6$, BFGS required 770 iterations and $M_1\mathrm{DFP}$ required 800 iterations, whereas ${\mathit{BM}}_{1}D$ required 408 iterations. This confirms the reduction in outer steps produced by the predictor–corrector structure. Although ${\mathit{BM}}_{1}D$ performs two stages per outer iteration, the normalized cost $\eta$ remains of order ${10}^{-8}$ across all tested dimensions and condition numbers, consistently with the dense $O\left(n^2\right)$ cost of full-memory quasi-Newton updates. Hence, these experiments show that the proposed hybrid scheme extends beyond the two-dimensional Booth case, preserves the expected dense scaling, and substantially reduces the number of outer iterations on high-dimensional quadratic problems.

5.3. Dolan–Moré Performance Profiles and Robustness Assessment

To provide a broader numerical assessment, we complement the pointwise convergence results with Dolan–Moré performance profiles. Among the proposed hybrid variants, BM₁D, BM₂D and BM₃D were selected for the global benchmark because they showed the most stable behavior in the preliminary screening, with fewer non-descent directions, fewer line-search breakdowns and more reliable quasi-Newton updating. These methods correspond, respectively, to the modified Chun-type, Ostrowski-type and Traub-type hybrid quasi-Newton schemes, and are compared with the classical quasi-Newton methods DFP, BFGS and SR1 [19]. Dolan–Moré profiles are a standard tool for benchmarking optimization solvers over a common test set [26].

Let $\mathcal{P}$ be the set of test instances and $\mathcal{S}$ the set of solvers. For each $p\in \mathcal{P}$ and $s\in \mathcal{S}$, let $t_{p,s}$ denote the computational cost required by solver s. If the solver fails to satisfy the stopping criterion within the computational budget, or produces non-finite values, we set $t_{p,s}=+\infty$. The performance ratio is

$$r_{p,s}={\displaystyle \frac{t_{p,s}}{{\displaystyle \underset{\overline{s}\in \mathcal{S}}{\min }{t}_{p,\overline{s}}}}},$$

where the minimum is taken over the solvers that successfully solve problem p. If no solver solves a given instance, all ratios are treated as infinite and the instance contributes only to the failure-rate analysis. The Dolan–Moré profile of solver s is defined by

$${\rho }_s\left(\tau \right)={\displaystyle \frac{1}{\left|\mathcal{P}\right|}}\left|\left\{p\in \mathcal{P}:r_{p,s}\le \tau \right\}\right|,\tau \ge 1.$$

Thus, ${\rho }_s\left(1\right)$ measures relative efficiency, whereas the limiting value of ${\rho }_s\left(\tau \right)$ for large $\tau$ measures robustness, since failed runs keep infinite cost.

Following the reviewer’s recommendation, the benchmark set was enlarged beyond the basic two-dimensional tests. It consists of 30 base problems from classical unconstrained optimization benchmarks, the Moré–Garbow–Hillstrom family, the Wood problem, higher-dimensional extensions, and CUTEr/CUTEst-type analytic test instances [27,28,29,30]. The composition of the set is reported in

Table 5. Composition of the 30 base problems used in the Dolan–Moré benchmark.

Collection	Problems	Number
Moré–Garbow–Hillstrom/Wood	Rosenbrock, Freudenstein–Roth, Beale, Powell singular, Wood, Extended Rosenbrock $(n=10,20)$, Extended Powell $(n=12,20)$, Broyden tridiagonal $(n=10,20)$, Brown almost-linear, and variably dimensioned problems $(n=10,20)$	14
Classical unconstrained tests	Himmelblau, Booth, Sphere, Zakharov, Styblinski–Tang, Branin–Hoo, Matyas, Dixon–Price, Powell badly scaled, Three-hump camel, Extended Beale, and Extended Himmelblau	12
CUTEr/CUTEst-type analytic representatives	Trigonometric, Penalty I, ARWHEAD, and GENHUMPS	4

. Each base problem was tested in five variants: one smooth version, two deterministic noisy versions, and two piecewise-smooth nonsmooth versions. Hence, the complete benchmark contains $30\times 5=150$ instances and, with six solvers, $150\times 6=900$ numerical runs.

The primary performance measure was the total computational work $t_{p,s}=N_f+N_g$, where $N_f$ and $N_g$ are the numbers of objective-function and gradient evaluations. This metric is appropriate because BM₁D, BM₂D and BM₃D are two-stage schemes, whereas DFP, BFGS and SR1 are one-stage quasi-Newton methods. For completeness, we also report profiles based on the number of outer iterations, which highlight the reduction in iteration count achieved by the proposed methods. For smooth instances, a run was declared successful when $∥\nabla f\left(x^{\left(k\right)}\right)∥\le {10}^{-6}$. For noisy and nonsmooth variants, where the gradient norm may be less reliable, we used the function-reduction criterion

$$f\left(x^{\left(k\right)}\right)\le f_L+{\tau }_f\left(f\left(x^{\left(0\right)}\right)-f_L\right),{\tau }_f={10}^{-5}.$$

Here, $f_L$ denotes the best available reference value; when the exact value was unavailable, it was taken as the best value obtained over all solvers. This criterion follows the benchmarking philosophy of Moré and Wild for smooth, noisy and piecewise-smooth optimization problems [31]. For noisy variants, the solvers used perturbed information, but success was assessed with the corresponding unperturbed objective value. In addition to the performance profiles, we report the failure rate

$${\mathrm{FR}}_{s,\mathcal{C}}=1-{\displaystyle \frac{1}{\left|\mathcal{C}\right|}}\left|\left\{p\in \mathcal{C}:t_{p,s}<+\infty \right\}\right|,$$

where $\mathcal{C}\subseteq \mathcal{P}$ denotes the smooth, noisy, nonsmooth, or full benchmark class. This measure complements the profiles, since a solver may be efficient on the instances it solves while still being unreliable under perturbations or nonsmoothness.

Table 6. Dolan–Moré results for the 30 smooth test instances.

Performance measured by computational work $(N_f+N_g)$
Method	Solved	Solved (%)	Best at $\tau =1$	${\rho }_s\left(2\right)$	${\rho }_s\left(5\right)$	Median cost
DFP	29	96.667	16	0.8333	0.9667	159.0
BFGS	30	100.000	13	0.9667	1.0000	163.0
SR1	28	93.333	18	0.9333	0.9333	140.0
BM1D	30	100.000	5	0.8667	1.0000	169.5
BM2D	30	100.000	4	0.9333	1.0000	182.5
BM3D	30	100.000	5	0.6667	1.0000	245.0
Performance measured by number of outer iterations
Method	Solved	Solved (%)	Best at $\tau =1$	${\rho }_s\left(2\right)$	${\rho }_s\left(5\right)$	Median cost
DFP	29	96.667	4	0.7000	0.9667	7.0
BFGS	30	100.000	4	0.9333	1.0000	8.5
SR1	28	93.333	5	0.8000	0.9333	6.5
BM1D	30	100.000	21	1.0000	1.0000	4.5
BM2D	30	100.000	23	1.0000	1.0000	5.0
BM3D	30	100.000	13	1.0000	1.0000	5.5

summarizes the Dolan–Moré results for the smooth subset of 30 instances. The first block uses the total work $N_f+N_g$, whereas the second block uses the number of outer iterations. The column “Best at $\tau =1$” counts ties independently.

The smooth-subset results show that BFGS and the three proposed hybrid methods solve all 30 instances, whereas DFP and SR1 show one and two failures, respectively. In terms of outer iterations, BM₁D, BM₂D and BM₃D have median iteration counts of 4.5, 5.0 and 5.5, compared with 7.0, 8.5 and 6.5 for DFP, BFGS and SR1. BM₂D attains the best iteration count in 23 instances, followed by BM₁D in 21 and BM₃D in 13. When total work $N_f+N_g$ is used, the comparison becomes more balanced, as expected for two-stage methods. Even so, all three hybrid methods solve every smooth instance and reach ${\rho }_s\left(5\right)=1$. Among them, BM₂D gives the most balanced smooth-subset behavior, with ${\rho }_s\left(2\right)=0.9333$, full success, and lower median work than BM₃D. To assess robustness under perturbations and loss of smoothness,

Table 7. Failure rates by benchmark class and overall.

Method	Smooth	Noisy	Nonsmooth	Overall
DFP	0.0333	0.1833	0.0333	0.0933
BFGS	0.0000	0.1167	0.0000	0.0467
SR1	0.0667	0.1500	0.0333	0.0867
BM1D	0.0000	0.1500	0.0000	0.0600
BM2D	0.0000	0.1333	0.0000	0.0533
BM3D	0.0000	0.1333	0.0000	0.0533

reports failure rates over the smooth, noisy and nonsmooth subsets, together with the overall rate over all 150 instances.

The failure-rate results indicate that the noisy variants are the most demanding part of the benchmark. In the smooth and nonsmooth classes, BM₁D, BM₂D and BM₃D achieve zero failure rate, matching BFGS and improving over DFP and SR1. In the noisy class, BFGS has the lowest failure rate, $0.1167$, followed by BM₂D and BM₃D, both with $0.1333$. Overall, BM₂D and BM₃D are the most robust proposed methods, with global failure rates of $0.0533$, close to BFGS $\left(0.0467\right)$. Figure 2 reports the empirical success rates by problem class, complementing Table 7.

The heat map confirms that the proposed methods solve all smooth and nonsmooth instances, while maintaining competitive success rates in the noisy class: $85.0\%$ for BM₁D and $86.7\%$ for BM₂D and BM₃D.

Figure 3 shows the aggregated failure rates over the full benchmark. BFGS gives the lowest global failure rate, $4.67\%$, while BM₂D and BM₃D follow closely with $5.33\%$. The larger failure rates of SR1 and DFP, $8.67\%$ and $9.33\%$, respectively, indicate greater sensitivity under the present perturbation setting.

The global Dolan–Moré profiles over the full benchmark are shown in Figure 4. The left panel uses the primary metric $N_f+N_g$, whereas the right panel uses the number of outer iterations.

The work-based profile in Figure 4a gives the most balanced comparison because it accounts for the additional evaluations required by the two-stage hybrid schemes. In this metric, BFGS and SR1 are highly competitive for small values of $\tau$, reflecting their lower cost per iteration. However, the proposed methods remain close to the best solvers over a wide range of $\tau$, with limiting values consistent with the low failure rates reported in Table 7. The iteration-based profile in Figure 4b highlights the main advantage of the proposed schemes: BM₁D, BM₂D and BM₃D solve a large fraction of the test instances with fewer outer iterations than DFP, BFGS and SR1. Overall, Table 6 and Table 7, together with Figure 2, Figure 3 and Figure 4, show that the proposed hybrid schemes are robust and competitive on a heterogeneous benchmark set. BM₂D and BM₃D are the most robust proposed methods, whereas BM₁D provides strong iteration reduction on several smooth instances. BFGS remains a very strong classical baseline in terms of total work and failure rate; therefore, the proposed methods should be interpreted as competitive hybrid alternatives that reduce outer iterations while preserving robust behavior under smooth, noisy and nonsmooth scenarios.

5.4. Local Verification Setup on Ill-Conditioned Hessians

To validate the explicit cubic error recurrences derived in Theorem 2 and Lemma 1, we consider a family of smooth test functions with prescribed Hessian condition number at the solution. For $\kappa >1$, let

$$A_{\kappa }=Q^T{\mathsf{\Lambda }}_{\kappa }Q,{\mathsf{\Lambda }}_{\kappa }=\mathrm{diag}\left(1,{\kappa }^{1/(n-1)},{\kappa }^{2/(n-1)},\dots ,\kappa \right),$$

where Q is a fixed orthogonal matrix generated with a prescribed random seed. We consider

$$f_{\kappa }\left(x\right)={\displaystyle \frac{1}{2}}{x}^T{A}_{\kappa }x+{\displaystyle \frac{\gamma }{3}}{\displaystyle \sum _{i=1}^n}{x}_i^3+{\displaystyle \frac{\delta }{4}}{\displaystyle \sum _{i=1}^n}{x}_i^4,x\in {\mathbb{R}}^n.$$

Then

$$\xi =0,\nabla f_{\kappa }\left(0\right)=0,{\nabla }^2{f}_{\kappa }\left(0\right)=A_{\kappa },\kappa \left({\nabla }^2{f}_{\kappa }\left(0\right)\right)=\kappa .$$

The gradient and Hessian are

$$F\left(x\right)=\nabla f_{\kappa }\left(x\right)=A_{\kappa }x+\gamma x^{\circ 2}+\delta x^{\circ 3},F^{\prime }\left(x\right)={\nabla }^2{f}_{\kappa }\left(x\right)=A_{\kappa }+2\gamma \mathrm{diag}\left(x\right)+3\delta \mathrm{diag}\left(x^{\circ 2}\right).$$

where $x^{\circ 2}$ and $x^{\circ 3}$ denote componentwise powers. In the experiments, we used $n=20,\gamma ={10}^{-1},\delta ={10}^{-2},\kappa \in \left\{{10}^2,{10}^4,{10}^6,{10}^8\right\}$. Since the convergence theory is local, the verification was carried out in the full-step regime ${\alpha }_N^{\left(k\right)}={\alpha }_{HO}^{\left(k\right)}=1$, which corresponds to the asymptotic regime assumed in the local analysis. For each value of $\kappa$, 96 local samples were generated by taking $x^{\left(0\right)}=rv,\parallel v\parallel =1$,with small radii r and randomly generated unit directions v. The initial inverse Hessian approximation was chosen as $H_N^{\left(0\right)}=A_{\kappa }^{-1}$. For sufficiently small $x^{\left(0\right)}$, this choice satisfies

$$H_N^{\left(0\right)}={\left[F^{\prime }\left(x^{\left(0\right)}\right)\right]}^{-1}+\mathcal{O}(\parallel e^{\left(0\right)}\parallel ),$$

which is precisely the first-order inverse-Hessian consistency condition used in Theorem 2. For each sample, one HOQN iteration was performed and the observed error $e^{\left(1\right)}$ was compared with the cubic term ${\mathcal{C}}_0^{\left(3\right)}$. The local order ${\rho }_{\mathrm{loc}}$ was estimated by a log–log regression of $\parallel e^{\left(1\right)}\parallel$ versus $\parallel e^{\left(0\right)}\parallel$. We also report

$$K^{\mathrm{emp}}={\displaystyle \frac{\parallel e^{\left(1\right)}\parallel }{\parallel e^{\left(0\right)}{\parallel }^3}},K^{\mathrm{th}}={\displaystyle \frac{\parallel {\mathcal{C}}_0^{\left(3\right)}\parallel }{\parallel e^{\left(0\right)}{\parallel }^3}},Q={\displaystyle \frac{\parallel e^{\left(1\right)}-{\mathcal{C}}_0^{\left(3\right)}\parallel }{\parallel e^{\left(0\right)}{\parallel }^4}}.$$

Thus, bounded values of Q provide numerical evidence for $∥e^{\left(1\right)}-{\mathcal{C}}_0^{\left(3\right)}∥=\mathcal{O}\left({∥e^{\left(0\right)}∥}^4\right)$. For $M_1\mathrm{DFP}$ and ${\mathit{BM}}_{1}D$, respectively, we monitor the inverse-Hessian consistency quantities

$${\delta }_N={\displaystyle \frac{\parallel H_N^{\left(0\right)}-{\left[F^{\prime }\left(x^{\left(0\right)}\right)\right]}^{-1}\parallel }{\parallel e^{\left(0\right)}\parallel }},{\delta }_B={\displaystyle \frac{\parallel {\widehat{H}}^{\left(0\right)}-{\left[F^{\prime }\left(x^{\left(0\right)}\right)\right]}^{-1}\parallel }{\parallel e^{\left(0\right)}\parallel }},$$

where ${\widehat{H}}^{\left(0\right)}$ is the intermediate inverse Hessian approximation obtained after the BFGS update following the Newton-type predictor.

Table 8. Cubic error verification for $M_1\mathrm{DFP}$ and ${\mathit{BM}}_{1}D$.

Method	$\mathit{\kappa }$	Samples	${\mathit{\rho }}_{\mathbf{loc}}$	${\mathit{K}}^{\mathbf{emp}}$	${\mathit{K}}^{\mathbf{th}}$	Q	${\mathit{\delta }}_{\mathit{N}}$	${\mathit{\delta }}_{\mathit{B}}$
$M_1\mathrm{DFP}$	${10}^2$	96	3.0107	$2.5538\times {10}^{-6}$	$2.5327\times {10}^{-6}$	$3.1318\times {10}^{-5}$	$2.8409\times {10}^{-2}$	–
$M_1\mathrm{DFP}$	${10}^4$	96	3.9990	$8.9961\times {10}^{-8}$	$2.2590\times {10}^{-7}$	$1.3963\times {10}^{-5}$	$2.0813\times {10}^{-2}$	–
$M_1\mathrm{DFP}$	${10}^6$	96	4.0005	$1.0508\times {10}^{-8}$	$1.9845\times {10}^{-7}$	$8.9582\times {10}^{-6}$	$1.5872\times {10}^{-2}$	–
$M_1\mathrm{DFP}$	${10}^8$	96	3.9252	$2.5802\times {10}^{-9}$	$8.0759\times {10}^{-8}$	$3.7921\times {10}^{-6}$	$8.1706\times {10}^{-3}$	–
${\mathit{BM}}_{1}D$	${10}^2$	96	3.0007	$2.4177\times {10}^{-5}$	$2.4889\times {10}^{-5}$	$3.0458\times {10}^{-5}$	$2.8409\times {10}^{-2}$	$2.7995\times {10}^{-2}$
${\mathit{BM}}_{1}D$	${10}^4$	96	3.2511	$2.4725\times {10}^{-7}$	$3.5745\times {10}^{-7}$	$1.3954\times {10}^{-5}$	$2.0813\times {10}^{-2}$	$2.0811\times {10}^{-2}$
${\mathit{BM}}_{1}D$	${10}^6$	96	3.7883	$1.2633\times {10}^{-8}$	$2.0103\times {10}^{-7}$	$8.9578\times {10}^{-6}$	$1.5872\times {10}^{-2}$	$1.5872\times {10}^{-2}$
${\mathit{BM}}_{1}D$	${10}^8$	96	3.8858	$2.5274\times {10}^{-9}$	$8.0762\times {10}^{-8}$	$3.7922\times {10}^{-6}$	$8.1706\times {10}^{-3}$	$8.1706\times {10}^{-3}$

reports the local verification results for $M_1\mathrm{DFP}$ and ${\mathit{BM}}_{1}D$ on ill-conditioned Hessians.

The results show that the estimated local order remains close to three for both $M_1\mathrm{DFP}$ and ${\mathit{BM}}_{1}D$, even for $\kappa ={10}^8$. Orders higher than three are consistent with the theoretical result, which guarantees a cubic convergence order, and may occur when the leading cubic coefficient is small or is partially canceled out along some sampled directions. The empirical constants $K^{\mathrm{emp}}$ and the theoretical constants $K^{\mathrm{th}}$ remain finite across all tests. The small values of Q indicate that, after subtracting the explicit cubic contribution ${\mathcal{C}}_0^{\left(3\right)}$, the remaining error behaves as a fourth-order residual, which supports

$$∥e^{\left(1\right)}-{\mathcal{C}}_0^{\left(3\right)}∥=\mathcal{O}\left({∥e^{\left(0\right)}∥}^4\right).$$

Finally, the bounded values of ${\delta }_N$ and, for ${\mathit{BM}}_{1}D$, ${\delta }_B$, confirm that the approximations of the inverse Hessian before and after the intermediate BFGS update remain first-order consistent with ${\left[F^{\prime }\left(x^{\left(0\right)}\right)\right]}^{-1}$. This provides numerical evidence that the BFGS update following the Newton-type predictor and the DFP update following the high-order corrector preserve the local cubic regime for the two-update variant described as ${\mathit{BM}}_{1}D$.

5.5. Dynamical Planes

To analyze the dependence of the methods on the initial estimate, we generate dynamical planes [32]. Each plane is constructed on a uniform grid, using each grid point as an initial condition and coloring it according to the minimizer reached by the method, thus visualizing the corresponding basins of attraction. In all dynamical planes, the colors indicate the basins of attraction associated with the minimizers reached by the method, whereas the black symbols mark the corresponding local minimizers. Unlike root-finding dynamical planes, here the goal is to locate minimizers rather than zeros of a nonlinear operator; consequently, the observed equilibrium points may appear on the boundary of attraction regions. The configuration used was a $500\times 500$ grid, with a maximum of 500 iterations and tolerance ${10}^{-4}$. Figure 5, Figure 6 and Figure 7 show the dynamical planes for the Himmelblau function, and Figure 8, Figure 9 and Figure 10 show the corresponding results for the Freudenstein–Roth function [12]. Overall, BFGS, DFP, SR1, $M_3\mathrm{DFP}$, $\mathrm{B}{M}_1\mathrm{D}$, $\mathrm{B}{M}_2\mathrm{D}$, and $\mathrm{B}{M}_3\mathrm{D}$ exhibit similar and comparatively stable dynamical behavior, whereas $M_1\mathrm{DFP}$ and $M_2\mathrm{DFP}$ show more fragmented or chaotic basins. The Himmelblau function produces the most complex attraction structure, with highly fragmented regions where small changes in the initial estimate may lead to different local minima. By contrast, the Freudenstein–Roth planes are less fragmented.

For the strictly convex quadratic Booth function, the Hessian is constant positive definite and $\xi =(1,3)$ is the unique minimizer. Hence, the considered descent line-search variants do not generate additional attractors: all initial conditions converge to the same point and the basin of attraction coincides with the whole domain. Therefore, only the $\mathrm{B}{M}_1\mathrm{D}$ plane is shown in Figure 11, as it is representative of all methods for $f_B$.

5.6. Hybrid Optimization with Limited Memory

In general, optimization methods may require a significant amount of memory, making it essential to adopt computational strategies that efficiently manage such requirements. One of the most widely used approaches in this context is the limited-memory variant of the BFGS method (L-BFGS) [33]. Instead of storing or factorizing the full Hessian matrix, which entails a memory cost of order $\mathcal{O}\left(n^2\right)$ and becomes impractical for dimensions $n\gtrsim {10}^4$, L-BFGS retains only a limited number m of the most recent curvature pairs $\left(s_i,y_i\right)$. As a result, the overall memory cost is reduced to $\mathcal{O}\left(mn\right)$, while still providing an effective approximation of curvature information. Building upon this strategy, we propose the hybrid limited-memory method L-HOQN, whose iterative scheme is described in Algorithm 2.

In low-dimensional problems, such as the Himmelblau, Freudenstein–Roth, and Booth test functions with $n=2$, limited-memory optimization methods and, in particular, the L-HOQN method, are affected by a loss of curvature information and, consequently, exhibit slower convergence than their full-memory counterparts. In the case of the L-HOQN method variants, $M_1$–LBFGS, $M_2$-LBFGS, and $M_3$-LBFGS, this loss reduces the effectiveness of the high order corrections and slows down the convergence process, as observed in the results reported in

Table 9. Limited-memory results for the Himmelblau function with $m=10$.

Himmelblau Function $\left({\mathit{f}}_{\mathit{H}}\right)$
$x^{\left(0\right)}={(4.9,1.96)}^T$, $\xi ={(-2.81,3.13)}^T$
Method	Iter	$\rho$	$\parallel \nabla f\left(x^{(k+1)}\right)\parallel$	CT (s)
$M_1$-LBFGS	170	1.0034	$8.7304\times {10}^{-13}$	0.0161
$M_2$-LBFGS	530	1.0053	$2.2616\times {10}^{-9}$	0.0292
$M_3$-LBFGS	68	0.9730	$8.4545\times {10}^{-13}$	0.0143
L-BFGS	187	0.9993	$6.5113\times {10}^{-13}$	0.0069
$x^{\left(0\right)}={(1.95,3.19)}^T$, $\xi ={(3.58,-1.85)}^T$
Method	Iter	$\rho$	$\parallel \nabla f\left(x^{(k+1)}\right)\parallel$	CT (s)
$M_1$-LBFGS	293	1.0024	$9.0901\times {10}^{-13}$	0.0153
$M_2$-LBFGS	561	1.0052	$4.4543\times {10}^{-9}$	0.0350
$M_3$-LBFGS	95	0.9590	$9.8602\times {10}^{-13}$	0.0412
L-BFGS	106	0.9953	$6.9535\times {10}^{-13}$	0.0036

,

Table 10. Limited-memory results for the Freudenstein–Roth function with $m=10$.

Freudenstein—Roth Function $\left({\mathit{f}}_{\mathit{FR}}\right)$
$x^{\left(0\right)}={(18,1)}^T$, $\xi ={(5,4)}^T$
Method	Iter	$\rho$	$\parallel \nabla f\left(x^{(k+1)}\right)\parallel$	CT (s)
$M_1$-LBFGS	749	0.9336	$9.9704\times {10}^{-9}$	0.0263
$M_2$-LBFGS	792	0.9975	$2.5072\times {10}^{-6}$	0.0323
$M_3$-LBFGS	714	0.9929	$9.8866\times {10}^{-10}$	0.0318
L-BFGS	827	0.9929	$9.9511\times {10}^{-9}$	0.0230
$x^{\left(0\right)}={(9,0)}^T$, $\xi ={(11.41,-0.90)}^T$
Method	Iter	$\rho$	$\parallel \nabla f\left(x^{(k+1)}\right)\parallel$	CT (s)
$M_1$-LBFGS	1158	0.9912	$3.0208\times {10}^{-7}$	0.0431
$M_2$-LBFGS	175	1.0327	$2.4232\times {10}^{-10}$	0.0444
$M_3$-LBFGS	314	0.9789	$9.9611\times {10}^{-9}$	0.0152
L-BFGS	1158	0.9944	$8.4165\times {10}^{-10}$	0.0266

and

Table 11. Limited-memory results for the Booth function with $m=10$.

Booth Function $\left({\mathit{f}}_{\mathit{B}}\right)$, $\mathit{\xi }={(1,3)}^{\mathit{T}}$
$x^{\left(0\right)}={(6.01,-1.50)}^T$
Method	Iter	$\rho$	$\parallel \nabla f\left(x^{(k+1)}\right)\parallel$	CT (s)
$M_1$-LBFGS	5	1.6198	$0.0\times {10}^0$	0.0123
$M_2$-LBFGS	16	0.9297	$1.7990\times {10}^{-13}$	0.0069
$M_3$-LBFGS	5	2.2100	$2.2571\times {10}^{-9}$	0.0010
L-BFGS	3	0.2094	$0.0\times {10}^0$	0.0003
$x^{\left(0\right)}={(2,8)}^T$
Method	Iter	$\rho$	$\parallel \nabla f\left(x^{(k+1)}\right)\parallel$	CT (s)
$M_1$-LBFGS	24	1.0248	$9.0994\times {10}^{-14}$	0.0016
$M_2$-LBFGS	7	0.9539	$0.0\times {10}^0$	0.0024
$M_3$-LBFGS	5	1.2141	$6.7270\times {10}^{-9}$	0.0011
L-BFGS	3	0.2304	$0.0\times {10}^0$	0.0011

. Nevertheless, this behavior changes substantially in practical high-dimensional applications, such as neural network training, which will be analyzed in Section 6.

Algorithm 2 Hybrid method L-HOQN

Require:  $x^{\left(k\right)}$, $g^{\left(k\right)}=\nabla f\left(x^{\left(k\right)}\right)$, $H^{\left(k\right)}$, ${\alpha }_N^{\left(k\right)},{\alpha }_{HO}^{\left(k\right)}>0$, $\mathrm{type}$, $|G_1|<+\infty$, memory $\mathcal{M}$ with at most m pairs

Ensure:  $x^{(k+1)}$, $H^{(k+1)}$

1:  $d_N^{\left(k\right)}\leftarrow -H^{\left(k\right)}{g}^{\left(k\right)}$

2:  $z^{\left(k\right)}\leftarrow x^{\left(k\right)}+{\alpha }_N^{\left(k\right)}{d}_N^{\left(k\right)}$

3:  $g_z^{\left(k\right)}\leftarrow \nabla f\left(z^{\left(k\right)}\right)$

4:  ${\nu }_k\leftarrow {\displaystyle \frac{\parallel g_z^{\left(k\right)}{\parallel }^2}{\parallel g^{\left(k\right)}{\parallel }^2}}$

5:  if   $\mathrm{type}={\mathrm{M}}_1$   then

6:         $h^{\left(k\right)}\leftarrow (1+{\nu }_k)g_z^{\left(k\right)}-2{\nu }_k{g}^{\left(k\right)}$

7:         $d_{HO}^{\left(k\right)}\leftarrow -H^{\left(k\right)}{h}^{\left(k\right)}$

8:         $x^{(k+1)}\leftarrow z^{\left(k\right)}+{\alpha }_{HO}^{\left(k\right)}{d}_{HO}^{\left(k\right)}$

9:  else if   $\mathrm{type}={\mathrm{M}}_2$   then

10:         $h^{\left(k\right)}\leftarrow g_z^{\left(k\right)}+2{\nu }_k{g}^{\left(k\right)}$

11:         $d_{HO}^{\left(k\right)}\leftarrow -{\displaystyle \frac{1}{1-4{\nu }_k}}{H}^{\left(k\right)}{h}^{\left(k\right)}$

12:         $x^{(k+1)}\leftarrow z^{\left(k\right)}+{\alpha }_{HO}^{\left(k\right)}{d}_{HO}^{\left(k\right)}$

13:  else if   $\mathrm{type}={\mathrm{M}}_3$   then

14:         $h^{\left(k\right)}\leftarrow (1+2{\nu }_k)g^{\left(k\right)}+(1+G_1{\nu }_k)g_z^{\left(k\right)}$

15:         $d_{HO}^{\left(k\right)}\leftarrow -H^{\left(k\right)}{h}^{\left(k\right)}$

16:         $x^{(k+1)}\leftarrow x^{\left(k\right)}+{\alpha }_{HO}^{\left(k\right)}{d}_{HO}^{\left(k\right)}$

17:  end if

18:  $g^{(k+1)}\leftarrow \nabla f\left(x^{(k+1)}\right)$

19:  $s^{\left(k\right)}\leftarrow x^{(k+1)}-x^{\left(k\right)}$

20:  $y^{\left(k\right)}\leftarrow g^{(k+1)}-g^{\left(k\right)}$

21:  if$\mathrm{card}\left(\mathcal{M}\right)=m$then

22:         remove oldest pair from $\mathcal{M}$

23:  end if

24:  append $(s^{\left(k\right)},y^{\left(k\right)})$ to $\mathcal{M}$

25:  update $H^{(k+1)}$ via two-loop recursion on $\mathcal{M}$

6. Practical Application in Neural Networks for the MNIST Database

The MNIST (Modified National Institute of Standards and Technology) dataset [34] is a canonical benchmark in the fields of computer vision and deep learning. It consists of 70,000 grayscale images of handwritten digits, of which 60,000 are designated for training and 10,000 for evaluation, each with a resolution of $28\times 28$ pixels. Despite its apparent simplicity, the task of training neural networks on this corpus entails the resolution of a high-dimensional nonconvex optimization problem, characterized by the presence of multiple local minima and highly intricate error surfaces. Handwritten digit classification constitutes a benchmark problem of substantial practical relevance in optical character recognition (OCR) applications, including forms, postal codes, bank checks, vehicle license plates, and other environments involving the processing of handwritten data, and it has served for decades as a standard testbed for the validation of pattern recognition theories and the assessment of machine learning algorithms. In order to facilitate the comparability of results and to promote progress in the field, several benchmark databases have been developed in which handwritten digit samples are uniformly preprocessed through segmentation and normalization, thereby providing a common framework in which researchers may objectively compare the performance of their methods [35]. Consequently, MNIST is systematically employed as a testbed for quantifying and comparing the effectiveness of different optimization algorithms and classification architectures in handwritten digit recognition tasks.

6.1. Architecture of the Implemented Neural Network

A convolutional neural network (CNN) was implemented (see Figure 12) whose input is a single-channel grayscale image of size $28\times 28\times 1$, processed by two convolutional blocks: the first applies 32 kernels of size $3\times 3$ with ReLU activation followed by max-pooling of $2\times 2$ (yielding feature maps of size $13\times 13\times 32$), and the second uses 64 kernels of the same size with ReLU activation and identical pooling (yielding feature maps of size $5\times 5\times 64$). The resulting tensor is flattened into a 1600-dimensional vector, passed through a fully connected layer of 128 neurons with ReLU activation, and finally projected to 10 logit output corresponding to the digit classes 0–9. This architecture balances complexity and efficiency by capturing spatial patterns and reducing dimensionality in the convolutional stages, while the dense layers perform the final classification.

The training of the neural network is formulated as a large-scale unconstrained optimization problem, whose objective is to determine the set of parameters $\theta$ (network weights and biases) that minimizes a regularized objective function. In particular, we consider the problem

$$\underset{\theta }{\min }\mathcal{J}\left(\theta \right)=\mathcal{L}\left(\theta \right)+\lambda {\parallel \theta \parallel }_2^2,$$

where $\mathcal{L}\left(\theta \right)$ denotes the multiclass cross-entropy loss function [36,37] and $\lambda ={10}^{-4}$ is the $L_2$ regularization parameter. The cross-entropy loss is defined in terms of the direct network outputs (logits) as

$$\mathcal{L}\left(\theta \right)=-{\displaystyle \frac{1}{N}}{\displaystyle \sum _{i=1}^N}\log \left({\displaystyle \frac{e^{z_{i,y_i}}}{{\sum }_{j=1}^{10}{e}^{z_{i,j}}}}\right),$$

where N denotes the number of samples (or the minibatch size), $z_{i,j}$ represents the logit associated with sample i and class j, and $y_i\in \{0,\dots ,9\}$ is the corresponding true label.

Training the convolutional architecture shown in Figure 12 leads to a very high-dimensional optimization problem, in which storing the full Hessian matrix is computationally infeasible. Therefore, it is essential to employ optimization methods that preserve curvature information without incurring excessive memory consumption. In Section 6.2, we introduce three new memory-constrained hybrid optimization algorithms that combine high order corrections with L-BFGS updates to efficiently train the neural network on the MNIST dataset.

6.2. Proposed Limited-Memory Hybrid Optimizers for CNN Training

In this section, we evaluate the performance of the $M_1$-LBFGS, $M_2$-LBFGS, and $M_3$-LBFGS methods (see Algorithm 2) on a more complex problem, specifically in the training of the convolutional neural network shown in Figure 12, designed for automatic digit classification. These new hybrid algorithms are generally more accurate and, in most cases, more stable during training than the classical SGD and L-BFGS methods. To solve the optimization problem arising from the training of this network, we employ the PyTorch library [38], version 2.8.0, which is widely recognized in the field of deep learning. PyTorch provides native GPU support, which is essential for minibatch training in deep architectures, and offers a flexible API that enables the efficient implementation of hybrid optimizers such as those proposed in this work. All experiments were performed on a 13-inch MacBook Air equipped with an Apple M4 chip and 16 GB of memory, running macOS Sequoia 15.7.4. The code was implemented and executed using Python 3.12.3. Our implementations use a fixed learning rate $\alpha$, momentum values $\mu =0.95$ and $\mu =0.99$ (see

Table 12. MNIST performance versus memory size m, momentum $\mu =0.95$, and learning rate $lr$.

		${\mathit{M}}_1$-LBFGS		${\mathit{M}}_2$-LBFGS		${\mathit{M}}_3$-LBFGS		L-BFGS
$\mathit{lr}$	$\mathit{m}$	Acc.	Avg. Time (s)	Acc.	Avg. Time (s)	Acc.	Avg. Time (s)	Acc.	Avg. Time (s)
0.05	5	0.9880	14.27	0.9894	14.50	0.9900	14.10	0.9915	12.45
	10	0.9877	15.25	0.9880	17.04	0.9909	15.20	0.9918	13.09
	15	0.9894	16.19	0.9876	17.80	0.9901	16.20	0.9854	12.30
	20	0.9885	17.72	0.9891	18.55	0.9903	17.50	0.9899	12.32
0.10	5	0.9928	14.74	0.9921	15.33	0.9930	14.20	0.9904	12.17
	10	0.9931	15.99	0.9923	17.11	0.9935	15.30	0.9911	12.32
	15	0.9932	16.87	0.9931	18.00	0.9928	16.10	0.9916	12.17
	20	0.9934	17.92	0.9932	19.36	0.9931	17.10	0.9929	12.90

and

Table 13. Comparison of hybrid optimizers with SGD and L-BFGS on MNIST for different learning rates.

Methods	Learning Rate ($\mathit{lr}$)	Final Accuracy	Avg. Time (s)
$M_3$-LBFGS	0.01	0.9889	15.20
	0.025	0.9920	14.20
	0.08	0.9940	14.10
$M_1$-LBFGS	0.01	0.9888	14.20
	0.025	0.9922	14.15
	0.08	0.9925	13.95
$M_2$-LBFGS	0.01	0.9890	14.65
	0.025	0.9916	14.62
	0.08	0.9920	14.66
L-BFGS	0.01	0.9854	12.23
	0.025	0.9876	12.27
	0.08	0.9902	12.33
SGD	0.01	0.9857	11.93
	0.025	0.9879	11.99
	0.08	0.9920	12.02

), and a limited-memory L-BFGS buffer of size $m=5,10,15,$ and 20, allowing us to assess the impact of the learning rate, momentum, and memory size on the efficiency of the hybrid methods.

6.3. Efficiency of the $M_1$-LBFGS, $M_2$-LBFGS and $M_3$-LBFGS Methods for MNIST

One of the key parameters affecting the efficiency of limited-memory optimization methods for minimizing the cross-entropy loss in neural networks is the learning rate ($lr$).

Table 12 shows how this parameter, together with the memory size, influences the accuracy of the $M_1$-LBFGS, $M_2$-LBFGS, $M_3$-LBFGS, and L-BFGS optimization methods. The reported accuracy metrics correspond to the test dataset and are computed after each training epoch. In addition, the average time reported in the table represents the total time per epoch, including both the training and evaluation phases. The results indicate that, for $\mathrm{lr}=0.10$, the proposed hybrid variants consistently achieve accuracy levels above $99\%$, outperforming the classical L-BFGS method for all memory sizes.

In Table 13, we report the comparison between the proposed hybrid schemes and the classical SGD and L-BFGS methods in the training of the neural network on the MNIST dataset, considering the learning rates $lr=0.01$, $lr=0.025$, and $lr=0.08$. In all limited-memory algorithms, a memory size $m=5$ was used, while in the proposed hybrid methods, an optimization momentum parameter $\mu =0.99$ was incorporated.

The results correspond to the test set and consistently show that the hybrid methods achieve higher levels of final accuracy than the classical approaches across all learning-rate regimes analyzed. In particular, for $lr=0.08$, the hybrid schemes clearly outperform L-BFGS (0.9902) and SGD (0.9920), indicating superior generalization capability under more demanding training configurations. This trend persists for $lr=0.025$, where the hybrid methods retain a significant advantage over L-BFGS and SGD in terms of the achieved accuracy. Even in the conservative scenario $lr=0.01$, the results show that the hybrid algorithms perform competitively or better than the classical methods. Although the computational cost per epoch is moderately higher in the hybrid schemes, the increase is proportional and fully justified by the systematic improvement in accuracy observed. As shown in Table 13, when comparing the performance of the $M_1$-LBFGS, $M_2$–LBFGS, and $M_3$–LBFGS algorithms under the learning rates $\mathrm{lr}$ = 0.10 and $\mathrm{lr}=0.15$, it can be observed that, for the lower learning rate, the three hybrid algorithms achieve the highest accuracy levels among them and outperform the classical L-BFGS and SGD methods. Figure 13 shows the evolution of the MNIST test accuracy for the implemented neural network under two learning rate values, $\mathrm{lr}=0.10$ and $\mathrm{lr}=0.15$. When comparing the optimizers, the hybrid variants $M_1$-LBFGS, $M_2$-LBFGS, and $M_3$-LBFGS exhibit a more stable and consistent behavior across epochs, achieving final accuracies that are slightly higher or comparable to those obtained by the classical L-BFGS and SGD methods.

Although this paper proposes three new hybrid algorithms with limited memory, the $M_3$-LBFGS method may become saturated when learning rates greater than $\mathrm{lr}=0.15$ are used. For this reason, the use of the $M_1$-LBFGS and $M_2$-LBFGS algorithms is recommended in such cases. The curves in Figure 14 show the accuracy achieved by the $M_1$-LBFGS, $M_2$-LBFGS, L-BFGS, and SGD optimizers during the training and testing phases of the neural network, for the learning rate $\mathrm{lr}=0.20$. In the hybrid variants, a memory size of $m=5$ was used. It can be observed that the $M_1$-LBFGS and $M_2$-LBFGS methods exhibit greater stability and, in general, a final accuracy slightly higher than that of the classical L-BFGS and SGD methods. Furthermore, the graphs show that the hybrid methods maintain very similar behavior between the training and test sets, while L-BFGS and SGD exhibit less consistent performance across both sets.

Figure 15 illustrates the evolution of the batch loss during the training of the neural network using the hybrid optimizers $M_1$-LBFGS, $M_2$-LBFGS, and $M_3$-LBFGS, with memory size $m=5$ and $\mu =0.95$. Figure 15a–c correspond to a learning rate of $0.05$, whereas Figure 15d–f present the results for a learning rate of $0.1$. It is observed that, for the lower learning rate ($\mathrm{lr}=0.05$), the batch loss decreases gradually with noticeable fluctuations before reaching a stable regime, with $M_2$-LBFGS exhibiting higher variability. By comparison, for the higher learning rate ($\mathrm{lr}=0.1$), all three hybrid methods display a faster and more uniform reduction of the loss, with significantly reduced noise after the initial training stages, indicating improved stability and faster convergence. Moreover, $M_1$-LBFGS and $M_3$-LBFGS tend to reach slightly lower stabilized loss values compared to $M_2$-LBFGS. Accuracy levels above $99\%$ achieved by the $M_1$-LBFGS, $M_2$-LBFGS, and $M_3$-LBFGS variants for $\mathrm{lr}=0.1$, as reported in Table 12 and Table 13 together with the low and stable batch loss values observed during training in Figure 15, indicate that these hybrid methods exhibit highly competitive performance in this high-dimensional optimization scenario.

In general, the numerical results show that the three proposed limited-memory hybrid variants are promising strategies for MNIST classification, achieving high accuracy and demonstrating stable and reliable neural network training.

6.4. Additional MNIST Experiments with Adam and AdamW

Finally, we compared the proposed limited-memory HOQN variants with the widely used adaptive first-order optimizers Adam [39,40] and AdamW [41,42] . The variants $M_1\mathrm{LBFGS}$ and $M_2\mathrm{LBFGS}$ were selected because they showed the most stable behavior in preliminary experiments, particularly with respect to changes in the learning rate. For both methods, we used memory size $m=5$, momentum parameter $\mu =0.95$, $L_2$ regularization parameter ${10}^{-8}$, learning rate $0.1$, and batch size 64. Adam and AdamW were trained with learning rate $0.001$ under the same CNN architecture, batch size, number of epochs, and random seeds. The corresponding results are reported in

Table 14. MNIST loss and accuracy over three seeds after 10 epochs.

Method	lr	Loss	Accuracy
$M_1\mathrm{LBFGS}$	0.100	$0.0151\pm 0.0007$	$0.9921\pm 0.0006$
$M_2\mathrm{LBFGS}$	0.100	$0.0208\pm 0.0008$	$0.9917\pm 0.0007$
Adam	0.001	$0.0101\pm 0.0009$	$0.9913\pm 0.0010$
AdamW	0.001	$0.0088\pm 0.0006$	$0.9915\pm 0.0012$

.

The results show that the proposed limited-memory HOQN variants remain competitive with Adam and AdamW. In particular, $M_1\mathrm{LBFGS}$ attains the highest mean test accuracy, while $M_2\mathrm{LBFGS}$ achieves comparable performance with low variability across seeds. Although AdamW obtains the lowest mean training loss, the test accuracies of all methods remain close, indicating that the proposed L-HOQN variants provide stable and competitive classification performance under the same training protocol.

7. Conclusions

This work introduced the higher-order quasi-Newton (HOQN) framework for unconstrained optimization, combining Newton-type predictors, higher-order correction terms derived from vector extensions of the Chun, Ostrowski, and Traub methods, and quasi-Newton updates of the inverse Hessian approximation. The resulting formulation provides a flexible hybrid structure that allows one-update and two-update variants while preserving the dense full-memory quasi-Newton cost of order $\mathcal{O}\left(n^2\right)$ per iteration.

From the theoretical point of view, local cubic convergence was established for the one-update variants $M_1-\mathrm{DFP}$, $M_2-\mathrm{DFP}$, and $M_3-\mathrm{DFP}$ under standard smoothness, positive-definiteness, and first-order inverse-Hessian consistency assumptions. The analysis also shows that the BFGS–DFP two-update strategy preserves the cubic local regime under suitable compatibility conditions. These results show that the higher-order correction cancels the dominant quadratic residual of standard quasi-Newton iterations, shifting the leading error term to cubic order.

The numerical experiments confirm the practical relevance of the proposed hybrid schemes. On the Himmelblau and Freudenstein–Roth functions, the HOQN variants generally require fewer outer iterations than classical BFGS, DFP, and SR1 methods while maintaining competitive computational times. The Booth function, due to its strictly convex quadratic structure, is mainly interpreted as a consistency test, where all methods exhibit very fast convergence. Additional high-dimensional quadratic experiments further show that the predictor–corrector structure remains effective beyond two-dimensional problems, particularly in ill-conditioned settings, while preserving the expected dense $\mathcal{O}\left(n^2\right)$ scaling.

The Dolan–Moré performance profiles and dynamical-plane analyses provide additional evidence of robustness and stability. Over heterogeneous smooth, noisy, and nonsmooth benchmark instances, the proposed methods are competitive with classical quasi-Newton schemes; although BFGS remains a strong baseline when total computational work is considered, BM₂D and BM₃D achieve low failure rates and robust performance. The dynamical planes also show that the two-update variants BM₁D, BM₂D, and BM₃D exhibit stable basin structures, whereas some one-update variants are more sensitive to the initial approximation.

Limited-memory variants of the HOQN framework were also developed for large-scale optimization. Although these variants may lose curvature information in low-dimensional test functions, their performance in convolutional neural network training on MNIST is promising. The proposed $M_1$-LBFGS, $M_2$-LBFGS, and $M_3$-LBFGS methods achieve test accuracies above $99\%$ for suitable learning rates and show stable behavior across training and test sets. Comparisons with Adam and AdamW over several random seeds indicate that the limited-memory hybrid methods are competitive in classification accuracy, with small variability across initializations.

Overall, the results suggest that the HOQN framework is a promising strategy for combining the fast local behavior of higher-order methods with the computational practicality of quasi-Newton updates. Future work should address global convergence theory for nonconvex problems, trust-region and adaptive line-search globalization strategies, stochastic and mini-batch limited-memory variants, deeper neural architectures, and more challenging datasets. Another relevant direction is the construction of new hybrid quasi-Newton algorithms based on other optimal vector methods for nonlinear systems, following the ideas introduced by Singh et al. in [13], and their adaptation to unconstrained optimization.