A Chebyshev–Halley Method with Gradient Regularization and an Improved Convergence Rate

Abstract

High-order methods are particularly crucial for achieving highly accurate solutions or satisfying high-order optimality conditions. However, most existing high-order methods require solving complex high-order Taylor polynomial models, which pose significant computational challenges. In this paper, we propose a Chebyshev–Halley method with gradient regularization, which retains the convergence advantages of high-order methods while effectively addressing computational challenges in polynomial model solving. The proposed method incorporates a quadratic regularization term with an adaptive parameter proportional to a certain power of the gradient norm, thereby ensuring a closed-form solution at each iteration. In theory, the method achieves a global convergence rate of $O\left(k^{-3}\right)$ or even $O\left(k^{-5}\right)$, attaining the optimal rate of third-order methods without requiring additional acceleration techniques. Moreover, it maintains local superlinear convergence for strongly convex functions. Numerical experiments demonstrate that the proposed method compares favorably with similar methods in terms of efficiency and applicability.

1. Introduction

In this paper, we consider the following unconstrained convex optimization problem:

$$\underset{x\in {\mathbb{R}}^n}{min}f\left(x\right)$$

(1)

where $f:{\mathbb{R}}^n\to \mathbb{R}$ is a continuously differentiable function with a globally Lipschitz continuous third-order derivative.

From a theoretical standpoint, high-order optimization methods exploit $p(p\ge 3)$th-order derivative information to attain enhanced convergence rates compared to first- and second-order methods. Furthermore, in the context of non-convex optimization, these methods exhibit a distinct advantage of circumventing degenerate solutions induced by singular Hessian matrices, thereby yielding solutions with higher precision and superior quality [1,2]. The modern development of high-order methods can be traced back to the cubic regularization of Newton’s method (CNM) [3], which laid the foundation for subsequent extensions to pth-order tensor methods [4,5,6,7,8], as well as adaptive [9,10,11,12] and stochastic [13,14,15,16] variants. In the aforementioned studies, a pth-order Taylor polynomial model is employed to approximate the objective function $f\left(x\right)$ at $x_k$:

$$\begin{array}{c}\hfill {\mathrm{\Omega }}_{x_k,p,{\lambda }_k}\left(s\right)=f\left(x_k\right)+\sum _{i=1}^p{\displaystyle \frac{1}{i!}}{\nabla }^{i}f\left(x_k\right){\left[s\right]}^i+{\displaystyle \frac{{\lambda }_k}{(p+1)!}}{\parallel s\parallel }^{p+1},\end{array}$$

where ${\nabla }^{i}f\left(x_k\right)\in {\mathbb{R}}^{n^i}$ is the ith-order derivative and ${\nabla }^{i}f\left(x_k\right){\left[s\right]}^i\in \mathbb{R}$ is the ith-order derivative along $s\in {\mathbb{R}}^n$. The next iterate is updated as $x_{k+1}=x_k+s_k$, where $s_k=arg\underset{s\in {\mathbb{R}}^n}{min}{\mathrm{\Omega }}_{x_k,p,{\lambda }_k}\left(s\right)$, and it continues until a satisfactory solution is found.

The non-convex nature of the polynomial model ${\mathrm{\Omega }}_{x_k,p,{\lambda }_k}\left(s\right)$ presents a significant challenge for these methods in achieving minimization. Nesterov [17] demonstrated that a sufficiently large regularization parameter ${\lambda }_k$ can ensure the convexity of the model ${\mathrm{\Omega }}_{x_k,p,{\lambda }_k}\left(s\right)$ while preserving the theoretical advantages of local high-order approximations, particularly in settings where the ith-order derivative is Lipschitz continuous. Furthermore, Nesterov developed an efficient minimization scheme for a convex polynomial model with $p=3$, incorporating a linearly convergent inner iteration at each step. The scheme leverages automatic differentiation techniques [18,19] to compute third-order derivatives along specific directions, significantly improving computational efficiency. Doikov et al. [20] proposed an inexact strategy for tensor methods, which initially reduced the accuracy of the model’s solution and gradually improved it in later iterations, thus ensuring global convergence. Cartis et al. [21] utilized the cubic-quartic regularization (CQR) framework to successfully address the minimization of a non-convex cubic model with quartic regularization. Ahmadi et al. [22] introduced sum-of-squares (SoS) convex polynomials as an alternative to address the limitations of the Taylor model in intuitive representation, and they solved the resulting model via n-dimensional semidefinite programming. Subsequently, ref. [23] integrated SoS into an adaptive framework, extending its applicability to general non-convex problems. Xiao et al. [24] proposed a regularized Chebyshev–Halley method (RCH) for solving a third-order polynomial model, achieving a global convergence rate of $O\left(k^{-3}\right)$, which can be further accelerated to $O\left(k^{-5}\right)$. The method transforms the solution of a polynomial model into the simultaneous resolution of a regularized Newton equation and a Halley equation, while maintaining computational complexity comparable to that of second-order methods.

Another active line of research aims to improve convergence rates through acceleration techniques. In machine learning and deep learning applications, for example, efficient methods not only accelerate model convergence but also significantly reduce computational resource requirements. In this context, two primary directions have been explored. The first is Nesterov-type acceleration [17,25,26,27], which leverages the estimate sequence technique. However, this technique relies on intricate algebraic constructions, leading to a lack of a unified mathematical explanation for its acceleration mechanism. Furthermore, experimental studies have indicated that accelerated tensor methods may underperform in practice [28]. The second direction is based on the accelerated hybrid proximal extragradient (A-HPE) framework [29,30,31,32], which achieves an optimal convergence rate of $O(k^{-\frac{3p+1}{2}})$ for pth-order optimization methods. Despite its theoretical advantages, this framework requires multiple iterative sequences, additional hyperparameters, and a time-consuming bisection line search process that is sensitive to parameter choices, making it less suitable for high-dimensional problems.

In practical scenarios, researchers tend to prefer fundamental methods due to their simplicity, ease of implementation, and generalizability to various optimization problems. Polyak [33] introduced a quadratic regularization technique in his work and proposed the following iterative scheme:

$$x_{k+1}=x_k-{({\nabla }^{2}f\left(x_k\right)+{\lambda }_{k}I)}^{-1}\nabla f\left(x_k\right),$$

where ${\lambda }_k\propto \parallel \nabla f\left(x_k\right)\parallel$. This iteration is concise and admits a closed-form solution at each step, circumventing the need to solve the polynomial model ${\mathrm{\Omega }}_{x_k,2,{\lambda }_k}\left(s\right)$, but it only achieves the $O(k^{-\frac{1}{4}})$ global convergence rate. Ueda et al. [34] employed a similar technique and reconstructed the quadratic Taylor polynomial model as follows:

$$\begin{array}{c}\hfill {\mathrm{\Omega }}_{x_k,2,{\lambda }_k}^2\left(s\right)=f\left(x_k\right)+\sum _{i=1}^p{\displaystyle \frac{1}{i!}}{\nabla }^{i}f\left(x_k\right){\left[s\right]}^i+{\displaystyle \frac{{\lambda }_k}{2}}{\parallel s\parallel }^2.\end{array}$$

Although its minimization leads to the same iterative scheme, the global convergence rate remains suboptimal. The aforementioned studies suggest that the quadratic regularization strategy may establish a balance between computational efficiency and global convergence, offering a new pathway for developing fundamental methods.

Recently, Doikov et al. [35] combined quadratic regularization with the Bregman distance, designing a regularization parameter ${\lambda }_k\propto \sqrt{\parallel \nabla f\left(x_k\right)\parallel }$. The resulting regularized Newton method can be viewed as a generalized relaxation of CNM, which theoretically not only preserves the global convergence rate of $O\left(k^{-2}\right)$ but also significantly reduces the computational complexity through closed-form iterations. Mishchenko [36] explored the relationship between the Levenberg–Marquardt penalty and CNM, constructing a quadratic regularization model. Despite differences in theoretical analysis, the proposed method achieves the same $O\left(k^{-2}\right)$ convergence rate and also admits a closed-form solution. Building on this, ref. [37] further unified quadratic and cubic regularization forms while introducing novel regularization terms such as the elastic net. Refs. [38,39,40,41] have also explored simple methods for more general or non-convex problems. Meanwhile, significant progress has been made in fundamental methods based on stepsize adjustment. Hanzely et al. [42,43] designed a novel damping parameter update scheme for the damped Newton method by revealing the implicit duality between the Newton stepsize and Levenberg–Marquardt regularization. This scheme updates the stepsize using the product of the gradient and the Hessian matrix, achieving a global convergence rate of $O\left(k^{-2}\right)$ or $O\left(k^{-3}\right)$.

In this paper, we propose a Chebyshev–Halley method with gradient regularization, a third-order method that combines rapid convergence with iterative simplicity. The proposed method employs a quadratic regularization strategy, where the regularization parameter is adjusted based on a power of the gradient norm at the current iterate. Therefore, the iterate is updated directly through matrix inversion and vector multiplication, which removes the need for nested inner iterations to solve the polynomial model. Theoretically, the method can achieve a global convergence rate of $O\left(k^{-3}\right)$ or $O\left(k^{-5}\right)$ without requiring acceleration techniques, attaining the optimal convergence rate for third-order methods. Numerical experiments demonstrate its effectiveness and applicability across various scenarios.

This paper is organized as follows. Section 2 presents the assumptions and the algorithmic framework of the proposed method. Section 3 focuses on the convergence analysis. Section 4 provides numerical experiments to validate the effectiveness of the method. Section 5 summarizes this paper.

2. Gradient-Regularized Chebyshev–Halley Method

For $x,s\in {\mathbb{R}}^n$, define $\nabla f\left(x\right)\left[s\right]=\nabla f{\left(x\right)}^{T}s$, ${\nabla }^{2}f\left(x\right){\left[s\right]}^2=s^T{\nabla }^{2}f\left(x\right)s$ and ${\nabla }^{3}f\left(x\right){\left[s\right]}^3=s^T{\nabla }^{3}f\left(x\right)ss$, where $\nabla f\left(x_k\right),{\nabla }^{2}f\left(x_k\right)$ and ${\nabla }^{3}f\left(x_k\right)$ represent the gradient, Hessian matrix, and third-order derivative, respectively. The objective function $f\left(x\right)$ is assumed to satisfy the following standard conditions.

Assumption 1.

The objective function $f\left(x\right)$ is convex and differentiable on ${\mathbb{R}}^n$, with a Lipschitz continuous third-order derivative. Specifically, there exists a constant $0<L_3<\infty$ such that for any $x,y\in {\mathbb{R}}^n$,

$$∥{\nabla }^{3}f\left(x\right)-{\nabla }^{3}f\left(y\right)∥\le L_3\parallel x-y\parallel .$$

As a direct consequence, for any $x,s\in {\mathbb{R}}^n$, the following bounds hold:

$$\begin{array}{c}\hfill ∥\nabla f(x+s)-\nabla f\left(x_k\right)-{\nabla }^{2}f\left(x_k\right)s-{\displaystyle \frac{1}{2}}{\nabla }^{3}f\left(x_k\right)ss∥\le {\displaystyle \frac{L_3}{6}}{\parallel s\parallel }^3,\end{array}$$

(2)

$$\begin{array}{c}\hfill ∥{\nabla }^{2}f(x+s)-{\nabla }^{2}f\left(x_k\right)-{\nabla }^{3}f\left(x_k\right)s∥\le {\displaystyle \frac{L_3}{2}}{\parallel s\parallel }^2.\end{array}$$

(3)

These bounds in (2) and (3) quantify the approximation errors of the gradient and Hessian, respectively, and play a crucial role in analyzing the convergence behavior of the proposed method.

Assumption 2.

The level set $\mathcal{L}\left(x_0\right):=\{x\in {\mathbb{R}}^n:f\left(x\right)\le f\left(x_0\right)\}$ is bounded, satisfying

$$\underset{x,y\in \mathcal{L}\left(x_0\right)}{max}\parallel x-y\parallel \le D<+\infty .$$

The classical Chebyshev–Halley method, also known as the improved hyperbolic tangent method, is iteratively formatted as [44] $x_{k+1}=x_k+s_k^1+s_k^2$, where $s_k^1$, $s_k^2$ are the solutions to the Newton equation

$${\nabla }^{2}f\left(x_k\right)s_k^1=-\nabla f\left(x_k\right),$$

and the Halley equation

$${\nabla }^{2}f\left(x_k\right)s_k^2=-{\displaystyle \frac{1}{2}}{\nabla }^{3}f\left(x_k\right)s_k^1{s}_k^1,$$

respectively. To improve the global convergence of the method, ref. [24] approximate $f\left(x\right)$ using a third-order Taylor polynomial model with a quartic regularization term and incorporate an acceleration sequence to achieve a global convergence rate of $O\left(k^{-5}\right)$. However, this method requires nested inner iterations to solve the model, and the introduction of an acceleration sequence further complicates parameter tuning.

In this section, we consider the following third-order Taylor polynomial model with quadratic regularization:

$$\begin{array}{ccc}\hfill s_k& =arg\underset{s\in {\mathbb{R}}^n}{min}{\mathrm{\Omega }}_{x_k,3,{\lambda }_k}^2\left(s\right)\hfill & \hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\end{array}$$

(4)

$$\begin{array}{cc}\hfill & \triangleq arg\underset{s\in {\mathbb{R}}^n}{min}\left\{f\left(x_k\right)+\nabla f{\left(x_k\right)}^{T}s+{\displaystyle \frac{1}{2}}{s}^T{\nabla }^{2}f\left(x_k\right)s+{\displaystyle \frac{1}{6}}{s}^T{\nabla }^{3}f\left(x_k\right)ss+{\displaystyle \frac{{\lambda }_k}{2}}{\parallel s\parallel }^2\right\},\hfill \end{array}$$

(5)

where $x_k\in {\mathbb{R}}^n$, ${\lambda }_k\ge L_3$ is a regularization parameter proportional to a certain power of the gradient norm at $x_k$. The basic iterative framework of the algorithm is presented below.

Algorithm 1 incorporates a quadratic regularization term and third-order derivatives to enhance convergence properties. The quadratic regularization term ensures global convergence while also providing a closed-form solution, thereby avoiding the need for nested inner iterations to solve the high-order Taylor polynomial model. Simultaneously, the utilization of the third-order derivative accelerates convergence, and the computational cost of third-order derivatives is comparable to that of Newton’s method [45].

Algorithm 1 Chebyshev–Halley method with gradient regularization (CHGR).

Require: Initial point $x_0\in {\mathbb{R}}^n$, $\alpha \in [{\displaystyle \frac{2}{3}},1]$, $ϵ>0$ and $H>0$.   1: for  $k=0,1,2,\dots$ do   2:     Set ${\lambda }_k\ge H{\parallel \nabla f\left(x_k\right)\parallel }^{\alpha }$.   3:     Obtain the direction $s_k$ through the following two steps: $$\begin{array}{cc}\hfill \left({\nabla }^{2}f\left(x_k\right)+{\lambda }_{k}I\right)s_k^1& =-\nabla f\left(x_k\right),\hfill \end{array}$$ (6) $$\begin{array}{cc}\hfill \left({\nabla }^{2}f\left(x_k\right)+{\lambda }_{k}I\right)s_k& \hfill =-\nabla f\left(x_k\right)-{\displaystyle \frac{1}{2}}{\nabla }^{3}f\left(x_k\right)s_k^1{s}_k^1.\end{array}$$ (7)   4:     $x_{k+1}=x_k+s_k$.   5: end for

We now derive an upper bound for the regularized Newton direction.

Corollary 1.

Suppose that Assumption 1 holds. Thus,

$$∥s_k^1∥=∥{\displaystyle \frac{\nabla f\left(x_k\right)}{{\nabla }^{2}f\left(x_k\right)+{\lambda }_{k}I}}∥\le {\displaystyle \frac{\parallel \nabla f\left(x_k\right)\parallel }{{\lambda }_k}}\le {\displaystyle \frac{\parallel \nabla f\left(x_k\right){\parallel }^{1-\alpha }}{H}}.$$

(8)

In particular, if $H\ge \parallel \nabla f\left(x_k\right){\parallel }^{1-\alpha }$, then

$$∥s_k^1∥\le 1.$$

(9)

Define $s_k^2=s_k-s_k^1$. From (6) and (7), it follows that

$$∥s_k^2∥={\displaystyle \frac{1}{2}}∥{\nabla }^{3}f\left(x_k\right)s_k^1{s}_k^1∥\le {\displaystyle \frac{L_3}{2}}{∥s_k^1∥}^2.$$

Furthermore, assuming that there exists a constant $C>0$ such that $C∥s_k^1∥\le ∥s_k∥$, then, combining (9) and the norm triangle inequality, we obtain

$$C∥s_k^1∥\le ∥s_k∥\le ∥s_k^1∥+∥s_k^2∥\le ∥s_k^1∥+{\displaystyle \frac{L_3}{2}}{∥s_k^1∥}^2\le \left(1+{\displaystyle \frac{L_3}{2}}\right)∥s_k^1∥.$$

(10)

Thus, $\parallel s_k\parallel$ and $\parallel s_k^1\parallel$ are of the same order of magnitude, meaning that $\parallel s_k\parallel$ is primarily determined by $\parallel s_k^1\parallel$, while $\parallel s_k^2\parallel$ serves only as a secondary correction term, proportional to the square of $\parallel s_k^1\parallel$. As $\parallel s_k^1\parallel$ approaches zero, the effect of $\parallel s_k^2\parallel$ diminishes, making its contribution to the overall stepsize negligible.

3. Convergence Analysis

This section is dedicated to analyzing the convergence of the proposed method. We begin by estimating the bound for the third-order derivative ${\nabla }^{3}f\left(x\right)$ along the direction $h\in {\mathbb{R}}^n$.

Lemma 1.

Suppose that Assumption 1 holds. For any $x,h\in {\mathbb{R}}^n$ and $\tau >0$, we have

$$∥{\nabla }^{3}f\left(x\right)hh∥\le \sqrt{2L_3}∥h∥\sqrt{h^T{\nabla }^{2}f\left(x\right)h}.$$

(11)

Proof. 

For any $x,y,h\in {\mathbb{R}}^n$, it follows from (3) that

$$0\le h^T{\nabla }^{2}f\left(y\right)h\le h^T{\nabla }^{2}f\left(x\right)h+{(y-x)}^T{\nabla }^{3}f\left(x\right)hh+{\displaystyle \frac{L_3}{2}}{\parallel y-x\parallel }^2{\parallel h\parallel }^2.$$

Let $y=x-\tau u$, where $\tau >0$ and $u\in {\mathbb{R}}^n$ with $\parallel u\parallel =1$. Then,

$${\left(\tau u\right)}^T{\nabla }^{3}f\left(x\right)hh\le h^T{\nabla }^{2}f\left(x\right)h+{\displaystyle \frac{L_3}{2}}{\parallel \tau u\parallel }^2{\parallel h\parallel }^2.$$

By dividing both sides of the inequality by $\tau$ and taking the norm, we obtain

$$∥{\nabla }^{3}f\left(x\right)hh∥\le {\displaystyle \frac{1}{\tau }}{h}^T{\nabla }^{2}f\left(x\right)h+\tau {\displaystyle \frac{L_3}{2}}{\parallel h\parallel }^2.$$

By minimizing the right-hand side of the inequality with respect to $\tau$, we obtain the optimal parameter ${\tau }_{\ast }=\sqrt{\frac{2h^T{\nabla }^{2}f\left(x\right)h}{L_3{\parallel h\parallel }^2}}$. Substituting ${\tau }_{\ast }$ into the inequality above yields the upper bound (11). □

Lemma 2.

Suppose that Assumption 1 holds. For any $s_k\in {\mathbb{R}}^n$, we have

$$s_k^T{\nabla }^{2}f\left(x_k\right)s_k\le {\displaystyle \frac{3\parallel \nabla f\left(x_k\right){\parallel }^2}{16{\lambda }_k}}.$$

(12)

Proof. 

From (7), we have

$$\begin{array}{cc}\hfill s_k^T{\nabla }^{2}f\left(x_k\right)s_k& =s_k^T\left(-\nabla f\left(x_k\right)-{\lambda }_k{s}_k-{\displaystyle \frac{1}{2}}{\nabla }^{3}f\left(x_k\right)s_k^1{s}_k^1\right)\hfill \\ \hfill & \le \parallel s_k\parallel \parallel \nabla f\left(x_k\right)\parallel +{\displaystyle \frac{1}{2}}\parallel s_k\parallel \parallel {\nabla }^{3}f\left(x_k\right)\parallel {∥s_k^1∥}^2-{\lambda }_k{\parallel s_k\parallel }^2\hfill \\ \hfill & \le \parallel s_k\parallel \parallel \nabla f\left(x_k\right)\parallel +{\displaystyle \frac{L_3\parallel \nabla f\left(x_k\right)\parallel }{2{\lambda }_k}}∥s_k^1∥\parallel s_k\parallel -{\lambda }_k{\parallel s_k\parallel }^2\hfill \\ \hfill & \le {\displaystyle \frac{3}{2}}\parallel s_k\parallel \parallel \nabla f\left(x_k\right)\parallel -{\lambda }_k{\parallel s_k\parallel }^2,\hfill \end{array}$$

where the first inequality follows from the norm triangle inequality, the second inequality follows from Assumption 1 and (8), and the last inequality follows from ${\lambda }_k\ge L_3$ and (9). Maximizing the right-hand side of the above inequality with respect to $\parallel s_k\parallel$ yields (12). □

The following theorem establishes the descent property of the objective function, ensuring that each iteration sufficiently reduces the gradient norm.

Theorem 1.

Suppose that Assumption 1 holds and that the regularization parameter ${\lambda }_k$ satisfies

$${\lambda }_k\ge max\{c_1{\parallel \nabla f\left(x_k\right)\parallel }^{\frac{2}{3}},c_2{\parallel \nabla f\left(x_k\right)\parallel }^{\frac{4}{5}},1\},$$

(13)

where $c_1=max{\left\{\frac{75L_3}{8},\frac{5{(2+L_3)}^2{L}_3^2}{16C^2},\frac{5{(2+L_3)}^2{L}_3^2}{24C}\right\}}^{\frac{1}{3}}$, $c_2={\left(\frac{5{(2+L_3)}^4{L}_3^2}{576}\right)}^{\frac{1}{5}}$. Then, we have

$$〈\nabla f\left(x_{k+1}\right),-s_k〉\ge {\displaystyle \frac{1}{4{\lambda }_k}}{∥\nabla f\left(x_{k+1}\right)∥}^2.$$

(14)

Proof. 

From (2) and (7), we obtain

$$\begin{array}{cc}\hfill {\displaystyle \frac{L_3}{6}}{\parallel s_k\parallel }^3& \ge ∥\nabla f\left(x_{k+1}\right)-\nabla f\left(x_k\right)-{\nabla }^{2}f\left(x_k\right)s_k-{\displaystyle \frac{1}{2}}{\nabla }^{3}f\left(x_k\right)s_k{s}_k∥\hfill \\ \hfill & =∥\nabla f\left(x_{k+1}\right)+{\lambda }_k{s}_k+{\displaystyle \frac{1}{2}}{\nabla }^{3}f\left(x_k\right)s_k^1{s}_k^1-{\displaystyle \frac{1}{2}}{\nabla }^{3}f\left(x_k\right)s_k{s}_k∥\hfill \\ \hfill & \ge ∥\nabla f\left(x_{k+1}\right)+{\lambda }_k{s}_k-{\displaystyle \frac{1}{2}}{\nabla }^{3}f\left(x_k\right)s_k{s}_k∥-∥{\displaystyle \frac{1}{2}}{\nabla }^{3}f\left(x_k\right)s_k^1{s}_k^1∥.\hfill \end{array}$$

Squaring both sides of the above inequality yields

$$\begin{array}{cc}\hfill & {\left({\displaystyle \frac{1}{2}}\parallel {\nabla }^{3}f\left(x_k\right)s_k^1{s}_k^1\parallel +{\displaystyle \frac{L_3}{6}}{\parallel s_k\parallel }^3\right)}^2\hfill \\ \hfill & \ge \parallel \nabla f\left(x_{k+1}\right){\parallel }^2+{\lambda }_k^2{\parallel s_k\parallel }^2+{\displaystyle \frac{1}{4}}{∥{\nabla }^{3}f\left(x_k\right)s_k{s}_k∥}^2+2{\lambda }_k〈\nabla f\left(x_{k+1}\right),s_k〉\hfill \\ \hfill & -〈\nabla f\left(x_{k+1}\right)+{\lambda }_k{s}_k,{\nabla }^{3}f\left(x_k\right)s_k{s}_k〉.\hfill \end{array}$$

(15)

By using the Cauchy–Schwarz inequality and the norm triangle inequality, we obtain

$$\begin{array}{cc}\hfill 〈\nabla f\left(x_{k+1}\right),{\nabla }^{3}f\left(x_k\right)s_k{s}_k〉& \le ∥\nabla f\left(x_{k+1}\right)∥∥{\nabla }^{3}f\left(x_k\right)s_k{s}_k∥\hfill \\ \hfill & \le {\displaystyle \frac{{∥\nabla f\left(x_{k+1}\right)∥}^2+{∥{\nabla }^{3}f\left(x_k\right)s_k{s}_k∥}^2}{2}},\hfill \end{array}$$

and

$${\lambda }_k〈s_k,{\nabla }^{3}f\left(x_k\right)s_k{s}_k〉\le {\lambda }_k\parallel s_k\parallel ∥{\nabla }^{3}f\left(x_k\right)s_k{s}_k∥.$$

By substituting these inequalities into (15) and using ${\lambda }_k\ge 1$, we obtain

$$\begin{array}{cc}\hfill & \langle \nabla f\left(x_{k+1}\right),-s_k\rangle \hfill \\ \hfill & \ge {\displaystyle \frac{1}{4{\lambda }_k}}{∥\nabla f\left(x_{k+1}\right)∥}^2+{\displaystyle \frac{{\lambda }_k{∥s_k∥}^2}{2}}-{\displaystyle \frac{1}{8{\lambda }_k}}{∥{\nabla }^{3}f\left(x_k\right)s_k{s}_k∥}^2\hfill \\ \hfill & -{\displaystyle \frac{∥s_k∥}{2}}∥{\nabla }^{3}f\left(x_k\right)s_k{s}_k∥-{\displaystyle \frac{1}{2}}{\left({\displaystyle \frac{1}{2}}∥{\nabla }^{3}f\left(x_k\right)s_k^1{s}_k^1∥+{\displaystyle \frac{L_3}{6}}{\parallel s_k\parallel }^3\right)}^2\hfill \\ \hfill & ={\displaystyle \frac{1}{4{\lambda }_k}}{∥\nabla f\left(x_{k+1}\right)∥}^2+{\displaystyle \frac{{\lambda }_k{∥s_k∥}^2}{2}}-{\displaystyle \frac{1}{8{\lambda }_k}}{∥{\nabla }^{3}f\left(x_k\right)s_k{s}_k∥}^2\hfill \\ \hfill & -{\displaystyle \frac{\parallel s_k\parallel }{2}}∥{\nabla }^{3}f\left(x_k\right)s_k{s}_k∥-{\displaystyle \frac{1}{8}}{∥{\nabla }^{3}f\left(x_k\right)s_k^1{s}_k^1∥}^2\hfill \\ \hfill & -{\displaystyle \frac{L_3^2}{72}}\parallel s_k{\parallel }^6-{\displaystyle \frac{L_3}{12}}∥{\nabla }^{3}f\left(x_k\right)s_k^1{s}_k^1∥{\parallel s_k\parallel }^3.\hfill \end{array}$$

By combining (8), (10)–(12), we obtain

$$\parallel s_k\parallel \le \left(1+{\displaystyle \frac{L_3}{2}}\right)∥s_k^1∥\le \left(1+{\displaystyle \frac{L_3}{2}}\right){\displaystyle \frac{\parallel \nabla f\left(x_k\right)\parallel }{{\lambda }_k}},$$

and

$$∥{\nabla }^{3}f\left(x_k\right)s_k{s}_k∥\le {\displaystyle \frac{3L_3}{8{\lambda }_k}}\parallel \nabla f\left(x_k\right){\parallel }^2{\parallel s_k\parallel }^2.$$

Thus, we obtain

$$\begin{array}{cc}\hfill & \langle \nabla f\left(x_{k+1}\right),-s_k\rangle \hfill \\ \hfill & \ge {\displaystyle \frac{1}{4{\lambda }_k}}\parallel \nabla f\left(x_{k+1}\right){\parallel }^2+{\displaystyle \frac{{\lambda }_k{\parallel s_k\parallel }^2}{2}}-{\displaystyle \frac{3L_3}{64{\lambda }_k^2}}\parallel \nabla f\left(x_k\right){\parallel }^2{\parallel s_k\parallel }^2\hfill \\ \hfill & -\sqrt{{\displaystyle \frac{3L_3}{32{\lambda }_k}}}\parallel \nabla f\left(x_k\right)\parallel \parallel s_k{\parallel }^2-{\displaystyle \frac{{\left(1+\frac{L_3}{2}\right)}^2{L}_3^2}{8C^2{\lambda }_k^2}}\parallel \nabla f\left(x_k\right){\parallel }^2{\parallel s_k\parallel }^2\hfill \\ \hfill & -{\displaystyle \frac{{\left(1+\frac{L_3}{2}\right)}^4{L}_3^2}{72{\lambda }_k^4}}\parallel \nabla f\left(x_k\right){\parallel }^4\parallel s_k{\parallel }^2-{\displaystyle \frac{{\left(1+\frac{L_3}{2}\right)}^2{L}_3^2}{12C{\lambda }_k^2}}\parallel \nabla f\left(x_k\right){\parallel }^2{\parallel s_k\parallel }^2.\hfill \end{array}$$

To ensure that (14) holds, we divide ${\displaystyle \frac{{\lambda }_k{\parallel s_k\parallel }^2}{2}}$ into five parts, each satisfying the following conditions:

(i).

${\displaystyle \frac{{\lambda }_k{\parallel s_k\parallel }^2}{10}}\ge {\displaystyle \frac{3L_3}{64{\lambda }_k^2}}\parallel \nabla f\left(x_k\right){\parallel }^2{\parallel s_k\parallel }^2$, which leads to ${\lambda }_k\ge {\left({\displaystyle \frac{15L_3}{32}}\right)}^{\frac{1}{3}}{\parallel \nabla f\left(x_k\right)\parallel }^{\frac{2}{3}}$.

(ii).

${\displaystyle \frac{{\lambda }_k{\parallel s_k\parallel }^2}{10}}\ge \sqrt{{\displaystyle \frac{3L_3}{32{\lambda }_k}}}\parallel \nabla f\left(x_k\right)\parallel \parallel s_k{\parallel }^2$, which leads to ${\lambda }_k\ge {\left({\displaystyle \frac{75L_3}{8}}\right)}^{\frac{1}{3}}{\parallel \nabla f\left(x_k\right)\parallel }^{\frac{2}{3}}$.

(iii).

${\displaystyle \frac{{\lambda }_k{\parallel s_k\parallel }^2}{10}}\ge {\displaystyle \frac{{\left(1+\frac{L_3}{2}\right)}^2{L}_3^2}{8C^2{\lambda }_k^2}}\parallel \nabla f\left(x_k\right){\parallel }^2{\parallel s_k\parallel }^2$, which leads to ${\lambda }_k\ge {\left({\displaystyle \frac{5{(2+L_3)}^2{L}_3^2}{16C^2}}\right)}^{\frac{1}{3}}{\parallel \nabla f\left(x_k\right)\parallel }^{\frac{2}{3}}$.

(iv).

${\displaystyle \frac{{\lambda }_k{\parallel s_k\parallel }^2}{10}}\ge {\displaystyle \frac{{\left(1+\frac{L_3}{2}\right)}^4{L}_3^2}{72{\lambda }_k^4}}\parallel \nabla f\left(x_k\right){\parallel }^4{\parallel s_k\parallel }^2$, which leads to ${\lambda }_k\ge {\left({\displaystyle \frac{5{(2+L_3)}^4{L}_3^2}{576}}\right)}^{\frac{1}{5}}{\parallel \nabla f\left(x_k\right)\parallel }^{\frac{4}{5}}$.

(v).

${\displaystyle \frac{{\lambda }_k{\parallel s_k\parallel }^2}{10}}\ge {\displaystyle \frac{{\left(1+\frac{L_3}{2}\right)}^2{L}_3^2}{12C{\lambda }_k^2}}\parallel \nabla f\left(x_k\right){\parallel }^2{\parallel s_k\parallel }^2$, which leads to ${\lambda }_k\ge {\left({\displaystyle \frac{5{(2+L_3)}^2{L}_3^2}{24C}}\right)}^{\frac{1}{3}}{\parallel \nabla f\left(x_k\right)\parallel }^{\frac{2}{3}}$.

By combining these conditions, we obtain the requirement (13). Thus, the theorem is proven. □

Based on the previous results, we now establish the global convergence of the proposed method.

Theorem 2.

Suppose that Assumptions 1–2 hold. Thus, the sequence $\left\{x_k\right\}$ generated by the Algorithm 1 satisfies

$$f\left(x_k\right)-f^{\ast }\le \left\{\begin{array}{c}{\left({\displaystyle \frac{12c_1{D}^{\frac{4}{3}}}{k}}\right)}^3,if{\lambda }_k=c_1{\parallel \nabla f\left(x_k\right)\parallel }^{\frac{2}{3}},\hfill \\ {\left({\displaystyle \frac{20c_2{D}^{\frac{6}{5}}}{k}}\right)}^5,if{\lambda }_k=c_2{\parallel \nabla f\left(x_k\right)\parallel }^{\frac{4}{5}}.\hfill \end{array}\right.$$

(16)

Proof. 

From Assumption 2, for any $x\in \mathcal{L}\left(x_0\right)$, we have

$$〈\nabla f\left(x_k\right),x_k-x^{\ast }〉\ge f\left(x_k\right)-f^{\ast },$$

which implies

$$\parallel \nabla f\left(x_k\right)\parallel \ge {\displaystyle \frac{f\left(x_k\right)-f^{\ast }}{D}},$$

(17)

where D is the bound of the level set $\mathcal{L}\left(x_0\right)$.

For $0\le \beta \le 1$ and $x\ge 0$, the function $y\left(x\right)=x^{\beta }$ is concave. Consequently, we have

$$a^{\beta }-b^{\beta }\ge {\displaystyle \frac{\beta }{a^{1-\beta }}}(a-b),\forall a>b\ge 0.$$

(18)

Let ${\lambda }_k=c_1{\parallel \nabla f\left(x_k\right)\parallel }^{\frac{2}{3}}$. By combining this with (14), we obtain

$$\begin{array}{cc}\hfill f\left(x_k\right)-f\left(x_{k+1}\right)& \ge 〈\nabla f\left(x_{k+1}\right),x_k-x_{k+1}〉\hfill \end{array}$$

(19)

$$\begin{array}{cc}\hfill & \ge {\displaystyle \frac{1}{4{\lambda }_k}}{\parallel \nabla f\left(x_{k+1}\right)\parallel }^2={\displaystyle \frac{\parallel \nabla f\left(x_{k+1}\right){\parallel }^2}{4c_1{\parallel \nabla f\left(x_k\right)\parallel }^{\frac{2}{3}}}}.\hfill \end{array}$$

(20)

Let ${\delta }_k=f\left(x_k\right)-f^{\ast }$. Thus,

$$\begin{array}{cc}\hfill {\displaystyle \frac{1}{{\delta }_{k+1}^{\beta }}}-{\displaystyle \frac{1}{{\delta }_k^{\beta }}}& ={\displaystyle \frac{{\delta }_k^{\beta }-{\delta }_{k+1}^{\beta }}{{\delta }_{k+1}^{\beta }{\delta }_{k+1}^{\beta }}}\hfill \\ \hfill & \ge {\displaystyle \frac{\beta \left({\delta }_k-{\delta }_{k+1}\right)}{{\delta }_k{\delta }_{k+1}^{\beta }}}\hfill \\ \hfill & ={\displaystyle \frac{\beta \left(f\left(x_k\right)-f\left(x_{k+1}\right)\right)}{{\delta }_k{\delta }_{k+1}^{\beta }}}\hfill \\ \hfill & \ge {\displaystyle \frac{\beta \parallel \nabla f\left(x_{k+1}\right){\parallel }^2{\parallel \nabla f\left(x_k\right)\parallel }^{\frac{4}{3}}}{4c_1{\parallel \nabla f\left(x_k\right)\parallel }^2{\delta }_k{\delta }_{k+1}^{\beta }}}\hfill \\ \hfill & \ge {\displaystyle \frac{\beta \parallel \nabla f\left(x_{k+1}\right){\parallel }^2{\delta }_k^{\frac{4}{3}}}{4c_1{D}^{\frac{4}{3}}{\parallel \nabla f\left(x_k\right)\parallel }^2{\delta }_k{\delta }_{k+1}^{\beta }}},\hfill \end{array}$$

where the first inequality follows from (18), the second inequality follows from (19), and the third inequality follows from (17). By setting $\beta ={\displaystyle \frac{1}{3}}$ and using the fact that ${\delta }_{k+1}<{\delta }_k$, the above inequality is simplified to

$${\displaystyle \frac{1}{{\delta }_{k+1}^{\beta }}}-{\displaystyle \frac{1}{{\delta }_k^{\beta }}}\ge {\displaystyle \frac{1}{12c_1{D}^{\frac{4}{3}}}}{\displaystyle \frac{{∥\nabla f\left(x_{k+1}\right)∥}^2}{{∥\nabla f\left(x_k\right)∥}^2}}.$$

By summing these inequalities from 1 to k, we obtain

$$\begin{array}{cc}\hfill {\displaystyle \frac{1}{{\delta }_k^{\beta }}}-{\displaystyle \frac{1}{{\delta }_0^{\beta }}}& \ge {\displaystyle \frac{1}{12c_1{D}^{\frac{4}{3}}}}\sum _{i=0}^{k-1}{\displaystyle \frac{\parallel \nabla f\left(x_{i+1}\right){\parallel }^2}{\parallel \nabla f\left(x_i\right){\parallel }^2}}\hfill \\ \hfill & \ge {\displaystyle \frac{1}{12c_1{D}^{\frac{4}{3}}}}{\left(\prod _{i=0}^{k-1}{\displaystyle \frac{\parallel \nabla f\left(x_{i+1}\right)\parallel }{\parallel \nabla f\left(x_i\right)\parallel }}\right)}^{\frac{2}{k}}\hfill \\ \hfill & \ge {\displaystyle \frac{1}{12c_1{D}^{\frac{4}{3}}}}{\left({\displaystyle \frac{{\delta }_k}{D\parallel \nabla f\left(x_0\right)\parallel }}\right)}^{\frac{2}{k}}\hfill \\ \hfill & ={\displaystyle \frac{1}{12c_1{D}^{\frac{4}{3}}}}exp\left(-{\displaystyle \frac{2}{k}}ln{\displaystyle \frac{D\parallel \nabla f\left(x_0\right)\parallel }{{\delta }_k}}\right)\hfill \\ \hfill & \ge {\displaystyle \frac{1}{12c_1{D}^{\frac{4}{3}}}}\left(1-{\displaystyle \frac{2}{k}}ln{\displaystyle \frac{D\parallel \nabla f\left(x_0\right)\parallel }{{\delta }_k}}\right),\hfill \end{array}$$

where the second inequality follows from the geometric–arithmetic mean inequality, and the third inequality follows from (17).

As $k\to \infty$, the inequality $1<{\displaystyle \frac{2}{k}}ln{\displaystyle \frac{D\parallel \nabla f\left(x_0\right)\parallel }{{\delta }_k}}$ does not hold. Therefore, we have

$$1\ge {\displaystyle \frac{2}{k}}ln{\displaystyle \frac{D\parallel \nabla f\left(x_0\right)\parallel }{{\delta }_k}},$$

which implies

$${\displaystyle \frac{1}{{\delta }_k^{\beta }}}\ge {\displaystyle \frac{1}{12c_1{D}^{\frac{4}{3}}}}.$$

Thus, the first case in (16) is proven.

Let ${\lambda }_k=c_2{\parallel \nabla f\left(x_k\right)\parallel }^{\frac{4}{5}}$ and $\beta =\frac{1}{5}$. Following a similar derivation process, the second case in (16) can be proven. □

In Algorithm 1, we set ${\lambda }_k=H{\parallel \nabla f\left(x_k\right)\parallel }^{\alpha }$. Based on the convergence analysis, we know that $\alpha$ can be chosen as $\frac{2}{3}$ or $\frac{4}{5}$. Correspondingly, the value of H is given by $H=max{\left\{\frac{75L_3}{8},\frac{5{(2+L_3)}^2{L}_3^2}{16C^2},\frac{5{(2+L_3)}^2{L}_3^2}{24C}\right\}}^{\frac{1}{3}}$, or ${\left(\frac{5{(2+L_3)}^4{L}_3^2}{576}\right)}^{\frac{1}{5}}$.

Next, we investigate the local convergence of the proposed method.

Theorem 3.

Suppose that Assumption 1 holds and that $f\left(x\right)$ is a ${\sigma }_2$-strongly convex function on ${\mathbb{R}}^n$. If there exists a constant $k^{\ast }>0$ such that for all $k\ge k^{\ast }$, $H\parallel s_k^1\parallel \ge c_0$, where $c_0$ is a sufficiently small positive constant, then

$$\parallel \nabla f\left(x_{k+1}\right)\parallel \le \left({\displaystyle \frac{L_{3}H}{6{{\sigma }_2}^3}}+{\displaystyle \frac{H}{{\sigma }_2{c}_0^{\frac{1}{1-\alpha }}}}\right){\parallel \nabla f\left(x_k\right)\parallel }^{2+\alpha }.$$

(21)

Proof. 

From the strong convexity of $f\left(x\right)$, it follows that

$$〈\nabla f\left(x_k\right),x_k-x_{k+1}〉\ge 〈\nabla f\left(x_{k+1}\right),x_k-x_{k+1}〉+{\sigma }_2{\parallel x_k-x_{k+1}\parallel }^2.$$

By combining (14), we obtain

$$\parallel \nabla f\left(x_k\right)\parallel \ge {\sigma }_2\parallel s_k\parallel .$$

(22)

Further, by combining (2) and the first-order optimality condition of Model (4), we have

$$\begin{array}{cc}\hfill ∥\nabla f\left(x_{k+1}\right)∥& \le ∥\nabla f\left(x_{k+1}\right)-\left(\nabla f\left(x_k\right)+{\nabla }^{2}f\left(x_k\right)s+{\displaystyle \frac{1}{2}}{\nabla }^{3}f\left(x_k\right)ss\right)∥+{\lambda }_k\parallel s_k\parallel \hfill \\ \hfill & \le {\displaystyle \frac{L_3}{6}}\parallel s_k{\parallel }^3+{\lambda }_k\parallel s_k\parallel \hfill \\ \hfill & \le {\displaystyle \frac{L_3}{6{{\sigma }_2}^3}}\parallel \nabla f\left(x_k\right){\parallel }^3+{\displaystyle \frac{H}{{\sigma }_2}}{\parallel \nabla f\left(x_k\right)\parallel }^{1+\alpha }\hfill \\ \hfill & \le \left({\displaystyle \frac{L_3{\parallel \nabla f\left(x_k\right)\parallel }^{1-\alpha }}{6{{\sigma }_2}^3}}+{\displaystyle \frac{H}{{\sigma }_2(H\parallel s_k^1{\parallel )}^{\frac{1}{1-\alpha }}}}\right){\parallel \nabla f\left(x_k\right)\parallel }^{2+\alpha },\hfill \\ \hfill & \le \left({\displaystyle \frac{L_{3}H}{6{{\sigma }_2}^3}}+{\displaystyle \frac{H}{{\sigma }_2{c}_0^{\frac{1}{1-\alpha }}}}\right){\parallel \nabla f\left(x_k\right)\parallel }^{2+\alpha },\hfill \end{array}$$

where the third inequality follows from (22), the fourth inequality follows from (8), and the last inequality follows from Corollary 1. □

Under the strong convexity condition, most existing works (e.g., refs. [34,35,36,37,38]) have established the local quadratic convergence of various Newton-type methods. However, the third-order method proposed in this paper exhibits a faster convergence rate, achieving an order of $2+\alpha$, where $\alpha \in [{\displaystyle \frac{2}{3}},1]$.

4. Numerical Experiments

In this section, we evaluate the performance of CHGR (Algorithm 1) on two classical test problems: $l_2$-regularized logistic regression and log-sum-exp. The experiments compare CHGR with various gradient-regularized methods, including the regularized Newton method (RegN [35,36]), the super-universal Newton method (SUN [39]), CNM [3] and RCH [24]. Additionally, we focus on several damped Newton methods with simple iterations and reliable global convergence properties, such as the affine-invariant cubic Newton method (AICN [42]) and the root Newton stepsize schedule (RN [43]). All numerical experiments were conducted in the MATLAB R2019b environment on a laptop equipped with an Intel Core i7 processor and 8GB of memory.

Table 1. Summary of iteration schemes and convergence properties for third-order methods ($p=3$).

Algorithm	Expression	Global Convergence Rate
CNM	$x_{k+1}=x_k-{\left({\nabla }^{2}f\left(x_k\right)+H\parallel x_{k+1}-x_k\parallel I\right)}^{-1}\nabla f\left(x_k\right)$	$O\left(k^{-2}\right)$
SUN	$x_{k+1}=x_k-{\left({\nabla }^{2}f\left(x_k\right)+H{\parallel \nabla f\left(x_k\right)\parallel }^{\alpha }I\right)}^{-1}\nabla f\left(x_k\right)$	$O\left(k^{-3}\right)$
RegN	$x_{k+1}=x_k-{\left({\nabla }^{2}f\left(x_k\right)+\sqrt{H\parallel \nabla f\left(x_k\right)\parallel }I\right)}^{-1}\nabla f\left(x_k\right)$	$O\left(k^{-2}\right)$
AICN	$x_{k+1}=x_k-{\eta }_k{\left({\nabla }^{2}f\left(x_k\right)\right)}^{-1}\nabla f\left(x_k\right)$ ${\eta }_k={\displaystyle \frac{-1+\sqrt{1+2H\sqrt{\nabla f{\left(x_k\right)}^T{\left({\nabla }^{2}f\left(x_k\right)\right)}^{-1}\nabla f\left(x_k\right)}}}{\sqrt{\nabla f{\left(x_k\right)}^T{\left({\nabla }^{2}f\left(x_k\right)\right)}^{-1}\nabla f\left(x_k\right)}}}$	$O\left(k^{-2}\right)$
RN	$x_{k+1}=x_k-{\eta }_k{\left({\nabla }^{2}f\left(x_k\right)\right)}^{-1}\nabla f\left(x_k\right)$ ${\eta }_k={\displaystyle \frac{1}{{\left(9H\right)}^{\frac{1}{3}}{\left(\nabla f{\left(x_k\right)}^T{\left({\nabla }^{2}f\left(x_k\right)\right)}^{-1}\nabla f\left(x_k\right)\right)}^{\frac{1}{3}}}}$	$O\left(k^{-3}\right)$
RCH	$\left({\nabla }^{2}f\left(x_k\right)+{\displaystyle \frac{H}{2}}{\parallel s_k^1\parallel }^{2}I\right)s_k^1=-\nabla f\left(x_k\right)$ $\left({\nabla }^{2}f\left(x_k\right)+{\displaystyle \frac{H}{2}}{\parallel s_k\parallel }^{2}I\right)s_k=-\nabla f\left(x_k\right)-{\displaystyle \frac{1}{2}}{\nabla }^{3}f\left(x_k\right)s_k^1{s}_k^1$$x_{k+1}=x_k+s_k$	$O\left(k^{-3}\right)$
CHGR	(6), (7)	$O\left(k^{-3}\right)$ or $O\left(k^{-5}\right)$

summarizes the iteration schemes and global convergence properties of the aforementioned methods.

The following two problems are widely used in the fields of optimization and machine learning, commonly employed to evaluate the effectiveness and robustness of optimization algorithms in practical tasks.

Problem 1.

$l_2$-regularized logistic regression:

$$\begin{array}{c}\hfill \underset{x\in {\mathbb{R}}^n}{min}f\left(x\right)={\displaystyle \frac{1}{d}}\sum _{i=1}^{d}log\left(1+exp\left(-y_i\left({\omega }_i^{T}x\right)\right)\right)+{\displaystyle \frac{l_2}{2}}{\parallel x\parallel }^2,\end{array}$$

where ${\left(y_i,{\omega }_i\right)}_{i=1}^d$ are the given data samples, with ${\omega }_i\in {\mathbb{R}}^n$ representing the feature vectors and $y_i\in \{-1,1\}$ representing the class label. $l_2$ is the regularization coefficient that controls model complexity and prevents overfitting. If $l_2$ is too small, it may result in ill-conditioned problems, negatively affecting the stability and convergence of the methods.

The experiments are conducted on datasets provided by LIBSVM (https://www.csie.ntu.edu.tw/~cjlin/libsvmtools/datasets/, accessed on 14 April 2025), including the ionosphere, mushroom, a4a, and a9a datasets. All linear systems are solved using the MATLAB backslash operator “∖”. The initial point for AICN and RN is chosen as $x_0=10\ast {(1,\dots ,1)}^T\in {\mathbb{R}}^n$ [42,43], while the initial points for the remaining methods are randomly generated from a standard normal distribution. The termination criterion for all methods is set as $\parallel \nabla f\left(x_k\right)\parallel \le 1\times {10}^{-10}$, and H is selected to be either $0.1$ or $0.5$. To evaluate the adaptability of the methods under different regularization strengths, $l_2$ is set to $1\times {10}^{-2}$ or $1\times {10}^{-6}$.

Problem 2.

log-sum-exp:

$$\underset{x\in {\mathbb{R}}^n}{min}f\left(x\right)=\rho log\left(\sum _{i=1}^{d}exp\left({\displaystyle \frac{a_i^{T}x-b_i}{\rho }}\right)\right),$$

where the components of vectors $a_1,\dots ,a_d\in {\mathbb{R}}^n$ and $b\in {\mathbb{R}}^d$ are independently sampled from a uniform distribution over the interval $[-1,1]$. The parameter $\rho >0$ is a smoothing factor that controls the approximation accuracy. This function serves as a smooth approximation of $max\{a_1^{T}x-b_1,\dots ,a_n^{T}x-b_n\}$. When ρ is small, the function becomes poorly conditioned.

In the experiments, the initial points $x_0$ for all methods are generated from a standard normal distribution, with the initial parameter set as $H=0.1$. The termination criterion is set as $\parallel \nabla f\left(x_k\right)\parallel \le 1\times {10}^{-10}$. To comprehensively evaluate the performance of the methods, two problem sizes are tested: $n=300,d=100$ and $n=700,d=300$. Additionally, different smoothing parameters $\rho$ are selected to assess the convergence stability of the methods.

Figure 1, Figure 2, Figure 3 and Figure 4 illustrate the performance comparison of the methods on Problem 1. The detailed analysis is as follows:

• Efficient Convergence

  –

  In most cases, CHGR performs best with $\alpha ={\displaystyle \frac{4}{5}}$, which is consistent with theoretical results. In terms of both iteration count and computational efficiency, CHGR significantly outperforms second-order methods such as AICN, RN, RegN, and CNM. This is due to its two-step iterative framework, which accelerates convergence by incorporating third-order derivative information while maintaining computational complexity comparable to that of second-order methods.

  –

  For $l_2={10}^{-2}$, AICN has a slightly less or comparable computational time to CHGR, due to fewer matrix and vector multiplication operations. However, for $l_2={10}^{-6}$, AICN fails after only two steps of a normal iteration due to its high sensitivity to parameters. A similar issue also occurs in RN.

  –

  For $\alpha ={\displaystyle \frac{2}{3}}$, CHGR outperforms SUN in terms of the computational time on a9a ($l_2={10}^{-2}$) but performs slightly worse on a4a ($l_2={10}^{-2}$). SUN is a single-step Newton iteration with an adaptive update strategy for $L_3$, and it can accelerate convergence in some cases. However, the need to verify the descent condition for the adaptive strategy introduces additional computational costs.

• Numerical Stability

  –

  CHGR maintains stable convergence across different regularization strengths, while CNM and RCH fail to achieve satisfactory solutions within 100 steps for $l_2={10}^{-6}$. This may be due to the more flexible regularization strategy of CHGR, which dynamically adjusts the regularization parameter based on the gradient norm, thereby improving convergence.

  –

  For $l_2={10}^{-2}$, AICN and CHGR exhibit comparable performance. However, for $l_2={10}^{-6}$, the weaker regularization causes the problem to become ill-conditioned, leading to the failure of AICN and RN to converge. Additionally, these two methods are more sensitive to the initial point, often failing due to numerical instability when initialized with $x_0={(1,1,\dots ,1)}^T$ or a random initialization.

Figure 5 and Figure 6 present the comparative results of the methods on Problem 2. The experimental results demonstrate that CHGR consistently achieves fast and stable convergence, showcasing its excellent convergence performance and strong adaptability to various problems. In comparison, AICN and RN succeed only when the problem is well conditioned, failing to converge for ill-conditioned cases like $d=300,n=100,\rho =0.1$ or $d=700,n=300,\rho =0.5$. Additionally, CNM and RCH also diverge for $d=300$, $n=100,\rho =0.1$, indicating their limitations in handling ill-conditioned problems.

In summary, CHGR demonstrates excellent convergence and numerical stability across various test problems and different regularization strengths, indicating its strong flexibility and applicability in handling complex problems.

5. Results

This paper proposes a Chebyshev–Halley method with gradient regularization, which demonstrates excellent global convergence properties while maintaining iteration simplicity. The method employs a third-order Taylor polynomial model with a quadratic regularization term and adjusts the regularization parameter based on a power of the gradient norm. In each iteration, the method requires only matrix inversion and vector products to obtain a closed-form solution, avoiding the complexity of nested sub-iterations for solving the polynomial model. Despite not incorporating acceleration techniques, the method theoretically achieves a global convergence rate of $O\left(k^{-3}\right)$ or $O\left(k^{-5}\right)$ and exhibits locally superlinear convergence for strongly convex problems. Numerical experiments show that the proposed method outperforms gradient-regularized Newton-type methods, such as RegN and SUN. Moreover, it exhibits greater adaptability than damped Newton methods like AICN and RN.