An Accelerated Diagonally Structured CG Algorithm for Nonlinear Least Squares and Inverse Kinematics

Abstract

Nonlinear least squares (NLS) models are extensively used as optimization frameworks in various scientific and engineering disciplines. This work proposes a novel structured conjugate gradient (SCG) method that incorporates a structured diagonal approximation for the second-order term of the Hessian, particularly designed for solving NLS problems. In addition, an acceleration scheme for the SCG method is proposed and analyzed. The global convergence properties of the proposed method are rigorously established under specific assumptions. Numerical experiments were conducted on large-scale NLS benchmark problems to evaluate the performance of the method. The outcome of these experiments indicates that the proposed method outperforms other approaches using the established performance metrics. Moreover, the developed approach is utilized to address the inverse kinematics challenge in controlling the motion of a robotic system with four degrees of freedom (4DOF).

1. Introduction

Nonlinear least squares (NLS) problems can be theoretically described as follows:

$$\underset{x\in {\mathbb{R}}^n}{min}f\left(x\right),$$

(1)

where the definition of the function $f\left(x\right)$ is given by

$$f\left(x\right)={\displaystyle \frac{1}{2}}{\parallel u\left(x\right)\parallel }^2={\displaystyle \frac{1}{2}}\sum _{i=1}^m{\left(u_i\left(x\right)\right)}^2,x\in {\mathbb{R}}^n.$$

(2)

NLS methods are broadly classified into two categories: first-order and second-order methods. First-order methods such as the Steepest Descent (SD), Gauss–Newton (GN), and Levenberg–Marquardt (LM) depend solely on the first derivative or gradient of the objective function for their computations. In contrast, second-order methods incorporate the Hessian matrix of the objective function or employ models and approximations of the Hessian. Second-order methods include techniques such as Newton’s method, the Quasi-Newton (QN) method, and Trust-region (TR) methods [1]. A diagram illustrating this classification is presented in Figure 1.

NLS has recently gained more recognition for its outstanding performance in various cognitive applications in computational science and engineering, including robotic motion control [2,3], image restoration [4,5,6], parameter identification [7], data fitting [8,9], and meteorological applications [10]. Moreover, it is a special kind of unconstrained optimization problem. Therefore, a general unconstrained minimization technique can be employed to generate a sequence of iterations $\left\{x_k\right\}$, via the following:

$$x_{k+1}=x_k+{\alpha }_k{d}_k$$

(3)

In the direction of the search vector $d_k$, the step sizes ${\alpha }_k>0,k\ge 0$ are obtained using a line search scheme. In inexact line search, the Wolfe line search method finds the value of ${\alpha }_k$ that reduces the cost function along $d_k$ by satisfying certain conditions [11].

$$\begin{array}{cc}\hfill & f(x_k+{\alpha }_k{d}_k)\le f\left(x_k\right)+\delta {\alpha }_k{g}_k^T{d}_k,\hfill \end{array}$$

(4)

$$\begin{array}{cc}\hfill & g{(x_k+{\alpha }_k{d}_k)}^T{d}_k\ge \sigma g_k^T{d}_k.\hfill \end{array}$$

(5)

In contrast, the strong Wolfe line search finds the value of ${\alpha }_k$ that satisfies a further requirement in addition to the condition in Equation (4).

$$|g{(x_k+{\alpha }_k{d}_k)}^T{d}_k|\le \sigma \left|g_k^T{d}_k\right|,$$

(6)

where $\delta ,\sigma \in (0,1)$ and $d_k$ is the search direction generated using the CG method by the following formula.

$$d_{k+1}=\left\{\begin{array}{c}-g_{k+1}ifk=0\hfill \\ -g_{k+1}+{\beta }_k{d}_{k}ifk\ge 1.\hfill \end{array}\right.$$

(7)

where the scalar factor is ${\beta }_k$, commonly termed the CG parameter, and $g\left(x\right)=\nabla f\left(x\right)$ denotes the gradient of the function f. However, the selection of ${\beta }_k$ significantly affects the computational efficacy of CG methods, prompting extensive research into effective parameter choices; see, for example, [12,13,14,15,16,17,18]. Furthermore, the direction $d_k$ described in Equation (7) is generally needed to meet the sufficient descent condition, which stipulates that there must be a constant $c>0$ such that

$$g_k^T{d}_k\le -c{\parallel g_k\parallel }^2.$$

The Hestenes–Stiefel (HS) method [19], the Polak–Ribiere–Polyak (PRP) method [20,21], and the Liu–Storey (LS) method [22] share similar structures in their numerators, which yield some common computational benefits [23]. In addition, their conjugate parameters are defined as follows:

$$\begin{array}{c}\hfill {\beta }_k^{HS}={\displaystyle \frac{g_{k+1}^T{y}_k}{d_k^T{y}_k}};{\beta }_k^{PRP}={\displaystyle \frac{g_{k+1}^T{y}_k}{\parallel g_k{\parallel }^2}};{\beta }_k^{LS}=-{\displaystyle \frac{g_{k+1}^T{y}_k}{g_k^T{d}_k}},\end{array}$$

where $g_{k+1}=\nabla f\left(x_{k+1}\right)$ and $y_k=g_{k+1}-g_k$.

Due to the distinctive structure of NLS, the objective function, its gradient vector, and the Hessian matrix can be expressed as follows:

$$\nabla f\left(x\right):=\sum _{i=1}^m{u}_i\left(x\right)\nabla u_i\left(x\right)=V{\left(x\right)}^{T}u\left(x\right),$$

(8)

$${\nabla }^{2}f\left(x\right):=\underset{1\mathrm{st}\mathrm{Term}}{\underbrace{\sum _{i=1}^m\nabla u_i\left(x\right)\nabla u_i{\left(x\right)}^T}}+\stackrel{2\mathrm{nd}\mathrm{Term}}{\overbrace{\sum _{i=1}^m{u}_i\left(x\right){\nabla }^2{u}_i\left(x\right)}}=V{\left(x\right)}^{T}V\left(x\right)+S\left(x\right),$$

(9)

where $V\left(x\right)$ denotes the Jacobian matrix of $u\left(x\right)$ and $S\left(x\right)={\sum }_{i=1}^m{u}_i\left(x\right){\nabla }^2{u}_i\left(x\right)$, representing the second term in Equation (9). The GN and LM methods ignore the second-order components of the Hessian matrix associated with the objective function (1). As a result, they are generally effective in solving small residual problems, but they can struggle with larger residual problems [24,25]. To address this limitation, Brown and Dennis [26] introduced the Structured Quasi-Newton (SQN) method, which takes advantage of the structure of the Hessian of the objective function in Equation (1) by combining elements of both the GN and Quasi-Newton methods. Numerically, the SQN algorithm has shown superior performance compared to GN and LM [27]. However, a significant challenge remains: the search direction generated by SQN is not guaranteed to be a descent direction for f, which complicates global convergence and limits its applicability to large-scale problems. The Quasi-Newton approach to non-negative image restoration is effectively presented in the work of Hanke et al. [28], while Hochbruck and Honig [29] provide insights into the convergence of a regularizing LM scheme for nonlinear ill-posed problems. Additionally, the recent work by Pes and Rodriguez [30] introduces a doubly relaxed minimal-norm GN method for underdetermined NLS problems.

Recent research efforts in NLS have focused on approximating the second-order term of the Hessian while preserving the efficiency and robustness of Newton-type methods. Despite their effectiveness, these approaches often require substantial matrix storage, particularly for large-scale problems [2,31]. To overcome these limitations, structured matrix-free techniques have emerged as a more efficient alternative for solving Equation (1). For instance, ref. [27] suggested a matrix-free, structured approach to large-scale NLS problems through CG search directions. Furthermore, ref. [32] introduced an approximation strategy that takes advantage of the diagonal components of the first and second terms of the Hessian matrix, incorporating a safeguarding mechanism to ensure adequate descent. More recently, ref. [33] developed a scaled variant of the SCG method based on a structured gradient relation, proving its global convergence under standard assumptions.

In another development, ref. [34] developed a structured two-term spectral CG method derived from the HS formulation, showcasing strong numerical performance. Yet, their approach requires additional safeguarding strategies, particularly when the parameter is nonpositive during certain iterations. Furthermore, ref. [35] introduced a structured adaptive technique for NLS problems based on spectral data and Barzilai–Borwein (BB) parameters. While CG methods that incorporate secant conditions may not guarantee a descent direction from a theoretical standpoint, they have demonstrated strong practical performance in tackling large-scale unconstrained optimization problems [36]. Hence, designing a CG method that efficiently integrates second-order information from the objective function is crucial to ensuring a descent search direction and enhancing overall performance.

In this paper, we build on the concept of the accelerated gradient method and the diagonal estimate of the objective function’s Hessian matrix. We introduce a new formula for ${\beta }_k$ by leveraging the structure of the Hessian in NLS problems. This research makes the following main contributions:

• This work proposes a novel SCG method that incorporates a structured diagonal approximation of the second-order term of the Hessian, combined with an acceleration scheme.
• The resulting search directions are proven to satisfy the sufficient descent condition.
• The proposed method is shown to have global convergence properties, relying on a strong Wolfe line search strategy and mild assumptions.
• Numerical experiments are conducted on a broad scale to evaluate the performance of the proposed method against existing methods.
• To illustrate its practical use, the SCG algorithm is implemented to solve the inverse kinematics of a robotic problem with 4DOF.

The structure of this paper is outlined below: Section 2 presents the newly developed method along with its algorithmic framework. Section 3 explores the theoretical global convergence behavior of the proposed algorithm under specific assumptions. In Section 4, numerical simulations are conducted to evaluate the performance of the methods in comparison with existing techniques. Lastly, Section 5 demonstrates the application of the proposed strategy to an inverse kinematic robotic motion control problem involving a system with 4DOF.

2. Formulation of the Diagonally Structured Conjugate Gradient with Acceleration Scheme

This section introduces the accelerated diagonally SCG method, which is designed to enhance the computational efficiency of standard CG algorithms. The technique incorporates a diagonal structure to approximate second-order information. Unlike the work presented in [27,31], our method lies in the introduction of an adaptive acceleration parameter, denoted as ${\eta }_k$, which is derived via a higher-order Taylor series expansion. This parameter is dynamically computed at each iteration and serves to enhance the convergence rate by adaptively modifying the conjugate direction updates.

2.1. The Diagonally Structured CG Coefficient

Consider approximating the second-order term of the Hessian with a diagonal matrix

$$W_k=Z_{k-1}{S}_{k-1}^{-1}$$

(10)

where $Z_{k-1}=diag(z_k^1,z_k^2,\dots ,z_k^n)$ and $S_{k-1}=diag(s_k^1,s_k^2,\dots ,s_k^n)$; this is equivalent to a specific form of the secant equation:

$$W_k{S}_{k-1}=Z_{k-1}.$$

Let $z_{k-1}=(z_k^1,z_k^2,\dots ,z_k^n)$ and $s_{k-1}=(s_k^1,s_k^2,\dots ,s_k^n)$ be two vectors, where the structured vector is defined by

$$z_{k-1}=V_k^T{V}_k{s}_{k-1}+{(V_k-V_{k-1})}^T{u}_k.$$

(11)

But $W_k=diag(w_k^1,\dots ,w_k^n)$, with $w_k^i\approx z_{k-1}^i/s_{k-1}^i$. However, if $s_{k-1}^i=0$ then $w_k$ is not well-defined, so we devise another definition for the diagonal entries of $W_k$:

$$w_k^i=\left\{\begin{array}{c}{\displaystyle \frac{z_{k-1}^i}{s_{k-1}^i}},if{s}_{k-1}^i\ne 0\mathrm{and}{ϵ}_l\le {\displaystyle \frac{z_{k-1}^i}{s_{k-1}^i}}\le {ϵ}_u,\hfill \\ 1,\mathrm{otherwise}.\hfill \end{array}\right.$$

(12)

In our implementation, we follow the strategy adopted in [31] with slight modification, which suggests the practical use of ${ϵ}_l={10}^{-5}$ and ${ϵ}_u={10}^{-1}$. These values are chosen to allow for a broad but safe range for the ratio ${\displaystyle \frac{z_{k-1}^i}{s_{k-1}^i}}$, accommodating reasonable curvature information while avoiding extreme values that could distort the search direction.

The following defines the proposed diagonally SCG method search direction:

$$d_k=\left\{\begin{array}{c}-W_k{g}_{k}ifk=0,\hfill \\ -W_k{g}_k+{\beta }_k^{*}{d}_{k-1}ifk\ge 1.\hfill \end{array}\right.$$

(13)

where ${\beta }_k^{*}$ is defined as follows:

$${\beta }_k^{*}={\displaystyle \frac{g_k^T{d}_{k-1}}{\parallel d_{k-1}{\parallel }^2}}$$

(14)

2.2. The Acceleration Scheme

Motivated by the work presented by Andrei [37], which was originally designed to solve unconstrained optimization problems, we present the scheme for the SCG formula. Let $x_{k+1}=x_k+{\alpha }_k{d}_k$, where $s_k={\alpha }_k{d}_k$, and consider Equation (4) as follows:

$$f(x_k+{\alpha }_k{d}_k)\le f\left(x_k\right)+\delta {\alpha }_k{g}_k^T{d}_k$$

(15)

The enhanced accelerated conjugate gradient algorithm can be introduced via the following iterative framework:

$$x_{k+1}=x_k+{\eta }_k{\alpha }_k{d}_k$$

where ${\eta }_k>0$ is a parameter that needs to be formulated to enhance the algorithm’s performance. Presently, by higher-order Taylor expansion, we have the following:

$$f(x_k+{\alpha }_k{d}_k)\approx f\left(x_k\right)+{\alpha }_k{g}_k^T{d}_k+{\displaystyle \frac{1}{2!}}{\alpha }_k^2{d}_k^T{\nabla }^{2}f\left(x_k\right)d_k+o(\parallel {\alpha }_k{d}_k{\parallel }^2),$$

(16)

Conversely, when $\eta >0$, we derive the higher-order Taylor expansion of

$$f(x_k+\eta {\alpha }_k{d}_k)\approx f\left(x_k\right)+\eta {\alpha }_k{g}_k^T{d}_k+{\displaystyle \frac{1}{2!}}{\eta }^2{\alpha }_k^2{d}_k^T{\nabla }^{2}f\left(x_k\right)d_k+o(\parallel \eta {\alpha }_k{d}_k{\parallel }^2).$$

(17)

We can combine Equations (16) and (17) as follows:

$$f(x_k+\eta {\alpha }_k{d}_k)=f(x_k+{\alpha }_k{d}_k)+{\lambda }_k\left(\eta \right)$$

(18)

where

$${\lambda }_k\left(\eta \right)={\displaystyle \frac{1}{2}}({\eta }^2-1){\alpha }_k^2{d}_k^T{\nabla }^{2}f\left(x_k\right)d_k+(\eta -1){\alpha }_k{g}_k^T{d}_k+{\eta }^2{\alpha }_{k}o({\alpha }_k\parallel d_k{\parallel }^2)-{\alpha }_{k}o({\alpha }_k\parallel d_k{\parallel }^2)$$

(19)

Now, suppose that

$$\begin{array}{c}{a}_k={\alpha }_k{g}_k^T{d}_k,\hfill \\ b_k={\alpha }_k^2{d}_k^T{\nabla }^{2}f\left(x_k\right)d_k,\hfill \\ {\epsilon }_k=o({\alpha }_k\parallel d_k{\parallel }^2).\hfill \end{array}$$

Therefore, Equation (19) becomes

$${\lambda }_k\left(\eta \right)={\displaystyle \frac{1}{2}}({\eta }^2-1)b_k+(\eta -1)a_k+{\eta }^2{\alpha }_k{\epsilon }_k-{\alpha }_k{\epsilon }_k$$

(20)

$${\lambda }_k^{\prime }\left(\eta \right)=\eta b_k+a_k+2\eta {\alpha }_k{\epsilon }_k$$

(21)

and

$${\lambda }_k^{\prime }\left({\eta }_k\right)=0$$

implies that

$${\eta }_k=-{\displaystyle \frac{a_k}{b_k+2{\alpha }_k{\epsilon }_k}}$$

(22)

It can be seen that Equation (22) is a minimizer of ${\lambda }_k$, provided that $b_k+2{\alpha }_k{\epsilon }_k>0$. As discussed in [37], the contribution of ${\epsilon }_k$ can be ignored in Equation (22). Also, $b_k$ can be approximated by $b_k=-{\alpha }_k{y}_k^T{d}_k$, in which $y_k=g_k-g_w$ with $w_k=x_k+{\alpha }_k{d}_k$. Hence, ${\eta }_k$ is computed as follows:

$${\eta }_k=\left\{\begin{array}{c}-{\displaystyle \frac{a_k}{b_k}},if{b}_k>0,\hfill \\ 1,\mathrm{otherwise}.\hfill \end{array}\right.$$

(23)

3. Global Convergence Analysis

In this section, we utilize Equation (14) under certain assumptions to demonstrate the convergence properties at a global scale for the proposed algorithm. For this section, we assume that $g_k\ne 0,\forall k$. Concerning the objective function, we adopt the following assumptions:

Assumption 1.

The set of levels $\ell =\{x\in {\mathbb{R}}^n,|f\left(x\right)\le f\left(x_0\right)\}$ lies within a bounded domain. Consequently, there exists a positive constant $\overline{b}$ such that

$$\parallel x\parallel \le \overline{b}foreveryx\in \ell .$$

(24)

Assumption 2.

The Jacobian matrix $V\left(x\right)$ exhibits Lipschitz continuity, and $u\left(x\right)$ is continuously differentiable within a given neighborhood $\mathcal{N}$ of ℓ. Specifically, a positive constant $a_1$ exists such that

$$\parallel V\left(x\right)-V\left(y\right)\parallel \le a_1\parallel x-y\parallel ,\forall x,y\in \mathcal{N}.$$

(25)

To prove the following lemmas and theorems, we now give estimates for $u\left(x\right)$, $V\left(x\right)$, and $\nabla f\left(x\right)$ based on Assumptions 1 and 2. By applying Equations (24) and (25), it follows that

$$\parallel V\left(x\right)\parallel \le \parallel V\left(x\right)-V\left(x_0\right)\parallel +\parallel V\left(x_0\right)\parallel \le a_1\parallel x-x_0\parallel +\parallel V\left(x_0\right)\parallel \le 2a_1\parallel x-x_0\parallel +\parallel V\left(x_0\right)\parallel$$

There exists $a_2>0$, such that for any $x\in \ell$,

$$\parallel V\left(x\right)\parallel \le a_2\mathrm{for}\mathrm{all}x\in \ell .$$

(26)

Furthermore, using the mean value theorem and Equation (25), we have for all x

$$\begin{array}{cc}\hfill \parallel u\left(x\right)-u\left(y\right)-V\left(y\right)(x-y)\parallel & \le {\int }_0^1\parallel V(y+\xi (x-y))-V\left(y\right)\parallel \parallel x-y\parallel d\xi ,\hfill \\ \hfill & \le a_1{\parallel x-y\parallel }^2{\int }_0^1\xi d\xi ={\displaystyle \frac{a_1}{2}}{\parallel x-y\parallel }^2.\hfill \end{array}$$

We decompose the difference $u\left(x\right)-u\left(y\right)$ as follows:

$$u\left(x\right)-u\left(y\right)=\left[u\left(x\right)-u\left(y\right)-V\left(y\right)(x-y)\right]+V\left(y\right)(x-y),$$

which leads to the inequality

$$\parallel u\left(x\right)-u\left(y\right)\parallel \le \parallel u\left(x\right)-u\left(y\right)-V\left(y\right)(x-y)\parallel +\parallel V\left(y\right)(x-y)\parallel ,$$

This implies that

$$\begin{array}{cc}\hfill \parallel u\left(x\right)-u\left(y\right)\parallel & \le {\displaystyle \frac{a_1}{2}}{\parallel x-y\parallel }^2+\parallel V\left(y\right)(x-y)\parallel \hfill \\ \hfill & \le \left({\displaystyle \frac{a_1}{2}}\parallel x-y\parallel +\parallel V\left(y\right)\parallel \right)\parallel x-y\parallel \hfill \\ \hfill & \le (a_1\overline{b}+a_2)\parallel x-y\parallel \hfill \end{array}$$

From the above inequality, it follows that there exists $b_1>0$ such that

$$\parallel u\left(x\right)-u\left(y\right)\parallel \le b_1\parallel x-y\parallel \mathrm{for}\mathrm{all}x,y\in \ell .$$

(27)

Hence, there exists $b_2>0$ such that

$$\parallel u\left(x\right)\parallel \le b_2\mathrm{for}\mathrm{all}x\in \ell .$$

(28)

$$\begin{array}{cc}\hfill \parallel \nabla f\left(x\right)-\nabla f\left(y\right)\parallel =& {\parallel V\left(x\right)}^T(u\left(x\right)-u\left(y\right))+{(V\left(x\right)-V\left(y\right))}^{T}u\left(y\right)\parallel \hfill \\ \hfill & \le \parallel V\left(x\right)\parallel \parallel u\left(x\right)-u\left(y\right)\parallel +\parallel V\left(x\right)-V\left(y\right)\parallel \parallel u\left(y\right)\parallel \hfill \\ \hfill & \le b_1\parallel V\left(x\right)\parallel \parallel x-y\parallel +a_1\parallel x-y\parallel \parallel u\left(y\right)\parallel \hfill \\ \hfill & \le (b_1{a}_2+a_1{b}_2)\parallel x-y\parallel \hfill \end{array}$$

(29)

Therefore, from Equation (29), it follows that a constant $c_1:=(b_1{a}_2+a_1{b}_2)$

$$\parallel \nabla f\left(x\right)-\nabla f\left(y\right)\parallel \le c_1\parallel x-y\parallel ,\forall x,y\in \ell .$$

(30)

which implies that $c_2>0$ such that

$$\parallel \nabla f\left(x\right)\parallel \le c_2,\forall x\in \ell .$$

(31)

Assumption 3.

The gradient of Equation (2), denoted by $g\left(x\right)=V{\left(x\right)}^{T}u\left(x\right)$, exhibits uniform continuity in an open convex domain that contains the level set ℓ.

Lemma 1.

Suppose that Assumptions 1 and 2 are satisfied. Let $x_k$ and $d_k$ be sequences produced by Algorithm 1. Then, a positive constant L exists, ensuring that the following inequality is true for $k\ge 0$,   $\parallel z_{k-1}\parallel \le L\parallel s_{k-1}\parallel$.

Proof. 

From the definition of the structured vector defined in Equation (11), we obtain

$$\begin{array}{cc}\hfill \parallel z_{k-1}\parallel & =\parallel V_k^T{V}_k{s}_{k-1}+{(V_k-V_{k-1})}^T{u}_k\parallel \hfill \\ \hfill & \le \parallel V_k^T{V}_k{s}_{k-1}\parallel +\parallel {(V_k-V_{k-1})}^T{u}_k\parallel \hfill \\ \hfill & \le \parallel V_k{\parallel }^2\parallel s_{k-1}\parallel +\parallel V_k-V_{k-1}\parallel \parallel u_k\parallel \hfill \\ \hfill & \le a_2^2\parallel s_{k-1}\parallel +a_1\parallel x_k-x_{k-1}\parallel \parallel u_k\parallel \hfill \\ \hfill & \le a_2^2\parallel s_{k-1}\parallel +a_1{b}_2\parallel s_{k-1}\parallel \hfill \\ \hfill & =(a_2^2+a_1{b}_2)\parallel s_{k-1}\parallel .\hfill \end{array}$$

Therefore, setting $L:=a_2^2+a_1{b}_2,$ we obtain the inequality.□

The lemma that follows shows that the sufficient descent requirement is satisfied by the direction $d_k$, independent of the line search requirement.

Algorithm 1: Diagonally Structured Conjugate Gradient with Acceleration (DSCGA)

Step 1: Select the starting point $x_0$ from the domain of f. Set $d_0=-g_0$ and $k=0$. Scalars $\delta ,\sigma \in (0,1)$, and ${ϵ}_g,{ϵ}_f,{ϵ}_l,{ϵ}_u>0$. Compute $u_0=u\left(x_0\right)$, $V_0=V\left(x_0\right)$, and $g_0=V_0^T{u}_0$. Step 2: If $\parallel g_k\parallel \le ϵ$, stop; otherwise, proceed to Step 3. Step 3: Determine ${\alpha }_k$ using Equations (4) and (6). Step 4: Calculate $w=x_k+{\alpha }_k{d}_k$, $g_w=\nabla f\left(w\right)$, and $y_k=g_k-g_w$. Step 5: Evaluate ${\eta }_k=-{\displaystyle \frac{a_k}{b_k}},$ and rescale the search direction ${\widehat{d}}_k={\eta }_k{\alpha }_k{d}_k$. Update $x_{k+1}=x_k+{\alpha }_k{\widehat{d}}_k$, where ${\alpha }_k\in (0,1]$. Else set $x_{k+1}=w$. Step 6: Compute $s_{k-1}=x_k-x_{k-1}$ and $z_{k-1}=V_k^T{V}_k{s}_{k-1}+{(V_k-V_{k-1})}^T{u}_k,$ with $V_k=V\left(x_k\right)$ and $u_k=u\left(x_k\right)$. Form $W_k=\mathrm{diag}(w_k^1,\dots ,w_k^n)$, with $$w_k^i=\left\{\begin{array}{c}{\displaystyle \frac{z_{k-1}^i}{s_{k-1}^i}},if{s}_{k-1}^i=0\mathrm{and}{ϵ}_l\le {\displaystyle \frac{z_{k-1}^i}{s_{k-1}^i}}\le {ϵ}_u\hfill \\ 1,\mathrm{otherwise},\hfill \end{array}\right.$$ Step 7: Compute ${\beta }_k^{*}$ using Equation (14). Step 8: Evaluate $d_k=-W_k{g}_k+{\beta }_k^{*}{d}_{k-1}$ Step 9: Update $k:=k+1$ and return to Step 2.

Lemma 2.

Assume that all entries of the diagonal matrix $W_k=diag(w_k^1,\dots ,w_k^n)$ satisfy the safeguarding condition $0<w_k^i\le {\epsilon }_u$ for all $i=1,\dots ,n$ and iteration k, where ${\epsilon }_u>0$ is a constant. Then, the search direction $d_k$ defined in Equation (13) satisfies the descent condition:

$$g_k^T{d}_k\le -c{\parallel g_k\parallel }^2.$$

(32)

Note: 

In the proof below, we assume that each diagonal entry of $W_k$ satisfies $0<w_k^i\le {\epsilon }_u$, as enforced by the safeguarding condition in Equation (12). This ensures that ${\displaystyle \frac{1}{w_k^i}}\ge {\displaystyle \frac{1}{{\epsilon }_u}}$, which is used to derive the lower bound in the descent property.

Proof. 

It follows for $k=0$ that

$$g_0^T{d}_0=-g_0^T{W}_0{g}_0=-{\parallel g_0\parallel }^2.$$

$$\begin{array}{cc}\hfill g_k^T{d}_k=& g_k^T(-W_k{g}_k+{\beta }_k^{*}{d}_{k-1}),\hfill \\ \hfill & =g_k^{T}diag\left(-{\displaystyle \frac{1}{w_k^1}},-{\displaystyle \frac{1}{w_k^2}},\dots ,-{\displaystyle \frac{1}{w_k^n}}\right)g_k+{\displaystyle \frac{g_k^T{d}_{k-1}}{\parallel d_{k-1}{\parallel }^2}}{g}_k^T{d}_{k-1},\hfill \\ \hfill & \le -{\displaystyle \frac{1}{{\epsilon }_u}}\parallel g_k{\parallel }^2+{\displaystyle \frac{|g_k^T{d}_{k-1}|}{\parallel d_{k-1}{\parallel }^2}}\left|g_k^T{d}_{k-1}\right|,\hfill \\ \hfill & \le -{\displaystyle \frac{1}{{\epsilon }_u}}\parallel g_k{\parallel }^2+{\displaystyle \frac{\parallel g_k\parallel \parallel d_{k-1}\parallel }{\parallel d_{k-1}{\parallel }^2}}\parallel g_k\parallel \parallel d_{k-1}\parallel ,\hfill \\ \hfill & \le -\left({\displaystyle \frac{1}{{\epsilon }_u}}-1\right){\parallel g_k\parallel }^2.\hfill \end{array}$$

Thus, defining c as

$$c:=\left({\displaystyle \frac{1}{{\epsilon }_u}}-1\right),$$

we obtain $g_k^T{d}_k\le -c{\parallel g_k\parallel }^2$. Hence, the proof is completed.□

Lemma 3.

Let $\left\{x_k\right\}$ and $\left\{d_k\right\}$ be the sequences defined by Algorithm 1. If $ϵ>0$ is such that for every $k\ge 1,\parallel g\left(x_{k-1}\right)\parallel \ge ϵ$, then $\kappa >0$ is such that

$$\parallel d_k\parallel \le \kappa \parallel g_k\parallel .$$

(33)

Proof. 

If $k=0$, then $\parallel d_0\parallel =\parallel W_0{g}_0\parallel \le \parallel W_0\parallel \parallel g_0\parallel \le \sqrt{n}\parallel g_0\parallel$.

If $k\ge 1$, then it follows that

$$\begin{array}{cc}\hfill \parallel d_k\parallel =& ∥-W_k{g}_k+{\beta }_k^{*}{d}_{k-1}∥,\hfill \\ \hfill & \le \parallel W_k\parallel \parallel g_k\parallel +|{\beta }_k^{*}|\parallel d_{k-1}\parallel ,\hfill \end{array}$$

(34)

Taking the absolute value of the parameter ${\beta }_k^{*}$ from Equation (13) yields the following result:

$$|{\beta }_k^{*}|=\left|{\displaystyle \frac{g_k^T{d}_{k-1}}{\parallel d_{k-1}{\parallel }^2}}\right|\le {\displaystyle \frac{\parallel g_k\parallel \parallel d_{k-1}\parallel }{\parallel d_{k-1}{\parallel }^2}}={\displaystyle \frac{\parallel g_k\parallel }{\parallel d_{k-1}\parallel }}.$$

(35)

It is important to note that the derivation in Equation (35) assumes the non-negativity of the parameter ${\beta }_k^{*}$, as defined in Equation (13). This assumption is justified by the fact that ${\beta }_k^{*}$ is computed as a scalar projection of the gradient $g_k$ onto the previous direction $d_{k-1}$, normalized by the squared norm $\parallel d_{k-1}{\parallel }^2$, which is strictly positive. In practical implementations, ${\beta }_k^{*}$ tends to remain non-negative when the algorithm is in the vicinity of a minimizer and the directions $d_k$ preserve a descent component. Moreover, should ${\beta }_k^{*}$ ever become negative, it would still be controlled in the analysis by taking its absolute value, as shown in Equation (35). This ensures the bound remains valid regardless of the sign, but in our convergence analysis, we adopt the non-negativity assumption to simplify the derivation of norm bounds.

Substituting Equation (35) into Equation (34), we obtain the following:

$$\begin{array}{cc}\hfill \parallel d_k\parallel & \le \parallel W_k\parallel \parallel g_k\parallel +{\displaystyle \frac{\parallel g_k\parallel }{\parallel d_{k-1}\parallel }}\parallel d_{k-1}\parallel ,\hfill \\ \hfill & \le \sqrt{n}\parallel g_k\parallel +\parallel g_k\parallel ,\hfill \\ \hfill & =\left(\sqrt{n}+1\right)\parallel g_k\parallel .\hfill \end{array}$$

Therefore, defining $\kappa$ as

$$\kappa :=\left(\sqrt{n}+1\right),$$

we obtain $\parallel d_k\parallel \le \kappa \parallel g_k\parallel$. Hence, we have the result.□

Lemma 4.

Based on Assumptions 1 and 2, we have

$$\underset{k\to \infty }{lim}{\alpha }_k{g}_k^T{d}_k=0$$

(36)

Proof. 

Let $x_{k+1}$ be the new iterate produced by the DSCGA method, satisfying the inequality

$$f\left(x_{k+1}\right)\le f\left(x_k\right)+\delta {\alpha }_k{g}_k^T{d}_k.$$

(37)

Since the sequence $\left\{f\left(x_k\right)\right\}$ is monotonically decreasing and has a lower bound of 0, it converges to some limit, say $f^{*}$. Taking the limit inferior on both sides of the inequality yields

$$\underset{k\to \infty }{lim\; inf}f\left(x_{k+1}\right)\le \underset{k\to \infty }{lim\; inf}f\left(x_k\right)+\delta \underset{k\to \infty }{lim\; inf}{\alpha }_k{g}_k^T{d}_k.$$

(38)

Since $f\left(x_k\right)\to f^{*}$, we conclude that

$$\underset{k\to \infty }{lim\; inf}{\alpha }_k{g}_k^T{d}_k\ge 0.$$

(39)

On the other hand, since $d_k$ is a descent direction, it implies that $g_k^T{d}_k<0$ for all $k\ge 0$. This implies

$$\underset{k\to \infty }{lim\; sup}{\alpha }_k{g}_k^T{d}_k\le 0.$$

(40)

Thus, we conclude

$$\underset{k\to \infty }{lim\; sup}{\alpha }_k{g}_k^T{d}_k\le 0\le \underset{k\to \infty }{lim\; inf}{\alpha }_k{g}_k^T{d}_k.$$

(41)

From this, we deduce that

$$\underset{k\to \infty }{lim}{\alpha }_k{g}_k^T{d}_k=0.$$

(42)

□

The following lemma is essential to establish the global convergence of the DSCGA method under the Wolfe line search strategy. Its proof is provided in [38].

Lemma 5.

Assume that the sequences $\left\{x_k\right\}$, $\left\{g_k\right\}$, and $\left\{d_k\right\}$ are produced by Algorithm 1, where ${\alpha }_k$ is chosen according to the search conditions of the Wolfe line (4) and (6). If Assumptions 1 and 2 hold, then the following summation is finite:

$$\sum _{k=1}^{+\infty }{\displaystyle \frac{{\left(g_k^T{d}_k\right)}^2}{\parallel d_k{\parallel }^2}}<+\infty .$$

(43)

Using Lemma 2 and Equation (43), we arrive at the following corollary.

Corollary 1.

Assume that the sequences $\left\{x_k\right\}$, $\left\{g_k\right\}$, and $\left\{d_k\right\}$ are produced by Algorithm 1, where ${\alpha }_k$ is obtained using the Wolfe condition. If Assumptions 1 and 2 hold, then the following series is convergent:

$$\sum _{k=1}^{+\infty }{\displaystyle \frac{\parallel g_k{\parallel }^4}{\parallel d_k{\parallel }^2}}<+\infty .$$

(44)

Building on the earlier lemmas, we present the following global convergence result using the following theorem.

Theorem 1.

Suppose that the sequences $\left\{x_k\right\}$, $\left\{d_k\right\}$, and $\left\{g_k\right\}$ are generated by Algorithm 1, where ${\alpha }_k$ is obtained according to the Wolfe conditions (4) and (6). If Assumption 1 holds, then

$$\underset{k\to +\infty }{lim}\parallel g_k\parallel =0.$$

Proof. 

One can directly deduce from Equation (33) that

$${\displaystyle \frac{1}{{\kappa }^2}}\le {\displaystyle \frac{\parallel g_k{\parallel }^2}{\parallel d_k{\parallel }^2}}.$$

When both sides of this inequality are multiplied by $\parallel g_k{\parallel }^2$, we obtain the following:

$${\displaystyle \frac{\parallel g_k{\parallel }^2}{{\kappa }^2}}\le {\displaystyle \frac{\parallel g_k{\parallel }^4}{\parallel d_k{\parallel }^2}}.$$

Summing over k, thus, we have the following:

$${\displaystyle \frac{1}{{\kappa }^2}}\sum _{k=1}^{+\infty }{\parallel g_k\parallel }^2\le \sum _{k=1}^{+\infty }{\displaystyle \frac{\parallel g_k{\parallel }^4}{\parallel d_k{\parallel }^2}}.$$

From the inequality above and Equation (44), we deduce that

$${\displaystyle \frac{1}{{\kappa }^2}}\sum _{k=1}^{+\infty }{\parallel g_k\parallel }^2<+\infty .$$

This implies that

$$\underset{k\to +\infty }{lim}{\parallel g_k\parallel }^2=0.$$

Thus, we obtain

$$\underset{k\to +\infty }{lim}\parallel g_k\parallel =0,$$

This concludes the proof. □

4. Numerical Results

This section assesses the computational efficiency of the proposed algorithm by benchmarking it against two established methods: SNCG [33] and SSGM [39]. The evaluation metrics include the number of iterations (It), the number of function evaluations (Fe), the number of gradient evaluations (Ge), and the total CPU runtime. For the proposed algorithm, the parameters are configured as $\rho =0.9$, $\sigma =0.0001$, ${ϵ}_l={10}^{-5}$, and ${ϵ}_u={10}^{-1}$, while the parameters for SNCG and SSGM are directly sourced from their respective references [33,39]. The benchmark test functions used for this comparison are described in full in

Table 1. Benchmark problems sourced from [40,41,42,43,44].

No.	FUNCTION	No.	FUNCTION
1.	PENALTY FUNCTION 1	13.	EXPONENTIAL FUNCTION 2
2.	VARIABLY DIMENSIONED	14.	SINGULAR FUNCTION 2
3.	TRIGONOMETRIC FUNCTION	15.	EXT. FREUDENSTEIN AND ROTH
4.	DISCRETE BOUNDARY-VALUE	16.	EXT. POWELL SINGULAR FUNCTION
5.	LINEAR FULL RANK	17.	FUNCTION 21
6.	PROBLEM 202	18.	BROYDEN TRIDIAGONAL FUNCTION
7.	PROBLEM 206	19.	EXTENDED HIMMELBLAU
8.	PROBLEM 212	20.	FUNCTION 27
9.	RAYDAN 1	21.	TRILOG FUNCTION
10.	RAYDAN 2	22.	ZERO JACOBIAN FUNCTION
11.	SINE FUNCTION 2	23.	EXPONENTIAL FUNCTION
12.	EXPONENTIAL FUNCTION 1	24.	FUNCTION 18
		25.	BROWN ALMOST FUNCTION

.

To evaluate the performance of the three algorithms, we employed the performance profiles introduced by Dolan and More [45]. These profiles were used to assess the algorithms based on the number of iterations, as well as the total number of function and gradient evaluations performed. In their work, Dolan and More [45] proposed a methodology for evaluating and comparing the performance of a solver s over a set of problems f. This method involves $n_s$ solvers and $n_f$ problems, as described below, and determines the computational cost q for each solver–problem pair:

$$q_{f,s}=\cos \mathrm{t}\mathrm{of}\mathrm{solving}\mathrm{problem}a\mathrm{using}\mathrm{solver}s.$$

Based on the computed cost $q_{f,s}$, [45] proposed a metric to evaluate solver efficiency. The performance ratio, which serves as a comparative measure, is defined as

$$r_{f,s}={\displaystyle \frac{q_{f,s}}{min\{q_{f,s}:s\in S\}}}.$$

Additionally, the cumulative distribution function of performance can be expressed as

$${\rho }_s\left(\tau \right)={\displaystyle \frac{1}{n_f}}\mathrm{size}\{f\in F:{log}_2\left(r_{f,s}\right)\le \tau \}.$$

This methodology facilitates the generation of performance profile graphs for each solver $s\in S$, leveraging the available data. These performance profiles illustrate the proportion ${\rho }_s\left(\tau \right)$ of problems that a solver can solve within a factor $\tau$ of the best-performing solver. A solver is considered more efficient if it achieves a higher ${\rho }_s\left(\tau \right)$ for a given $\tau$. Consequently, the solver that maintains the highest performance profile curve across all values of $\tau$ is deemed the most efficient.

All algorithms were implemented and executed in MATLAB R2022a, and experiments were conducted on a Windows 11 platform equipped with a 10th Gen Intel Core i7-1065G7 processor (Santa Clara, CA, USA), with 16GB of RAM and a 512GB SSD. The Wolfe line search method was employed for the experiments, and the Wolfe condition parameters for the SNCG and SSGM algorithms adhered to the specifications outlined in their respective original studies. The termination criterion for all algorithms was defined as $\parallel g\left(x_k\right)\parallel <ϵ$, where $ϵ={10}^{-5}$, or if any of the following conditions were met:

• The algorithm ran for over 1000 iterations.
• More than 5000 evaluations of the function were performed.

If an algorithm was unable to solve a specific problem, it was marked as a failure, and the failure point was indicated by a symbol (⋆) in the table. A failure was defined as one of the following conditions: exceeding the maximum number of iterations, failing to reach the predefined convergence tolerance, or encountering numerical issues such as divergence or undefined values (e.g., NaNs or Infs).

From

Table 2. Numerical results from problems 1–14, the symbol (⋆) denotes failure.

METHODS			SNCG				SSGM				DSCGA
FUNCS	DIM	It	Fe	Ge	TIME	It	Fe	Ge	TIME	It	Fe	Ge	TIME
1.	3000	4	7	10	0.1762	3	5	8	0.0183	2	5	7	0.2012
	6000	4	7	10	0.0674	3	5	8	0.0195	2	5	7	0.0798
	9000	4	7	10	0.0482	3	5	8	0.0237	3	5	7	0.0630
	12,000	4	7	10	0.0413	3	5	8	0.0197	3	5	7	0.0315
	15,000	4	7	10	0.0324	3	5	8	0.0279	3	5	7	0.0956
2.	3000	⋆	⋆	⋆	⋆	138	153	215	0.78824	16	18	49	0.2485
	6000	⋆	⋆	⋆	⋆	133	185	250	1.0024	25	32	76	0.7239
	9000	⋆	⋆	⋆	⋆	154	261	263	2.8126	70	71	71	0.0698
	12,000	⋆	⋆	⋆	⋆	177	321	323	3.9750	32	95	97	0.2875
	15,000	⋆	⋆	⋆	⋆	183	387	350	5.2280	22	49	67	1.6995
3.	3000	50	446	151	0.4861	71	114	214	0.6545	45	47	136	0.4324
	6000	60	578	181	1.2966	82	146	247	2.3541	35	52	106	1.0266
	9000	48	253	145	0.8607	97	175	292	4.0113	73	94	220	2.623
	12,000	66	765	199	4.1457	106	184	319	4.6306	67	90	202	2.8439
	15,000	84	454	253	4.1474	98	185	295	5.9078	61	85	184	2.1456
4.	3000	5	25	22	0.1225	5	23	40	0.0923	2	6	7	0.0667
	6000	7	35	16	0.0753	7	35	25	0.0896	2	6	7	0.1102
	9000	11	37	4	0.0377	9	6	14	0.0256	3	8	8	0.1157
	12,000	15	39	4	0.0349	11	35	25	0.0954	4	11	16	0.0555
	15,000	20	46	4	0.0711	13	59	40	0.1001	5	26	18	0.0598
5.	3000	2	5	7	0.0514	2	5	7	0.0073	2	5	7	0.0139
	6000	2	5	7	0.0385	2	5	7	0.0092	2	5	7	0.0175
	9000	2	5	7	0.0165	2	5	7	0.0133	2	5	7	0.0428
	12,000	2	5	7	0.0176	2	5	7	0.0191	2	5	7	0.0425
	15,000	2	5	7	0.2577	2	5	7	0.0251	2	5	7	0.0817
6.	3000	4	9	13	0.00804	5	13	16	0.0126	5	11	16	0.0275
	6000	4	9	13	0.02215	5	13	16	0.040159	5	11	16	0.0439
	9000	4	9	13	0.0261	5	13	16	0.0329	5	11	16	0.0509
	12,000	4	9	13	0.0331	5	13	16	0.0427	5	11	16	0.1295
	15,000	4	9	13	0.0484	5	13	16	0.0593	5	11	16	0.1678
7.	3000	6	13	19	0.0222	6	16	19	0.2530	5	12	16	0.0760
	6000	6	13	19	0.1060	6	16	19	0.5337	5	12	16	0.0500
	9000	6	13	19	0.0659	6	16	19	0.6334	5	12	16	0.0667
	12,000	6	13	19	0.0603	6	16	19	0.6633	5	12	16	0.1283
	15,000	6	13	19	0.1582	6	16	19	0.0774	5	12	16	0.1472
8.	3000	10	21	31	0.0577	7	11	22	0.1155	4	9	13	0.0431
	6000	10	21	31	0.0898	7	11	22	0.1234	4	9	13	0.0774
	9000	10	21	31	0.22631	7	11	22	0.2435	4	9	13	0.2239
	12,000	10	21	31	0.2282	7	11	22	0.2355	4	9	13	0.1138
	15,000	10	21	31	0.2192	7	11	22	0.3259	4	9	13	0.3342
9.	3000	⋆	⋆	⋆	⋆	⋆	⋆	⋆	⋆	5	5	16	0.0641
	6000	⋆	⋆	⋆	⋆	⋆	⋆	⋆	⋆	5	5	16	0.2054
	9000	⋆	⋆	⋆	⋆	⋆	⋆	⋆	⋆	7	18	22	0.0152
	12,000	⋆	⋆	⋆	⋆	⋆	⋆	⋆	⋆	7	18	22	0.0199
	15,000	⋆	⋆	⋆	⋆	⋆	⋆	⋆	⋆	7	18	22	0.0357
10.	3000	5	11	16	0.0404	4	9	13	0.0095	4	9	13	0.0161
	6000	5	11	16	0.0224	4	9	13	0.0191	4	9	13	0.0312
	9000	5	11	16	0.0328	4	9	13	0.0214	4	9	13	0.0465
	12,000	5	11	16	0.0505	4	9	13	0.0317	4	9	13	0.0914
	15,000	5	11	16	0.0494	4	9	13	0.0465	4	9	13	0.0957
11.	3000	2	3	4	0.0332	1	3	4	0.0173	1	3	4	0.0146
	6000	2	3	4	0.0259	1	3	4	0.0119	1	3	4	0.0407
	9000	2	3	4	0.0251	1	3	4	0.0187	1	3	4	0.0494
	12,000	2	3	4	0.0247	1	3	4	0.0247	1	3	4	0.0094
	15,000	2	3	4	0.0255	1	3	4	0.0408	1	3	4	0.0249
12.	3000	⋆	⋆	⋆	⋆	4	7	8	0.0563	7	13	17	0.503
	6000	⋆	⋆	⋆	⋆	5	10	9	0.0192	6	14	16	0.0115
	9000	⋆	⋆	⋆	⋆	5	10	9	0.0280	13	25	34	0.3499
	12,000	⋆	⋆	⋆	⋆	7	12	13	0.0332	16	35	46	0.4575
	15,000	⋆	⋆	⋆	⋆	8	15	14	0.0680	18	38	55	0.0261
13.	3000	⋆	⋆	⋆	⋆	83	444	250	0.2925	13	34	40	0.3015
	6000	⋆	⋆	⋆	⋆	67	357	202	0.3742	15	31	46	0.6974
	9000	⋆	⋆	⋆	⋆	79	421	238	0.5503	46	43	49	0.8374
	12,000	⋆	⋆	⋆	⋆	37	259	112	0.0935	21	37	64	0.3054
	15,000	⋆	⋆	⋆	⋆	61	327	184	0.3767	50	64	51	0.3524
14.	3000	5	12	16	0.0465	7	17	22	0.1774	5	12	16	0.0259
	6000	5	12	16	0.0272	7	17	22	0.3432	5	12	16	0.0509
	9000	5	12	16	0.0343	7	17	22	0.4903	5	12	16	0.0657
	12,000	5	12	16	0.0409	7	17	22	0.0525	5	12	16	0.0755
	15,000	5	12	16	0.1378	7	17	22	0.7693	5	12	16	0.1830

and

Table 3. Numerical results from problems 15–25, the symbol (⋆) denotes failure.

METHODS			SNCG				SSGM				DSCGA
FUNCS	DIM	It	Fe	Ge	TIME	It	Fe	Ge	TIME	It	Fe	Ge	TIME
15.	3000	⋆	⋆	⋆	⋆	26	69	79	0.0637	10	17	31	0.0905
	6000	⋆	⋆	⋆	⋆	26	69	79	0.3639	10	17	31	0.2584
	9000	⋆	⋆	⋆	⋆	26	69	79	0.2277	10	23	31	0.4727
	12,000	⋆	⋆	⋆	⋆	26	69	79	0.2571	13	34	40	0.6318
	15,000	⋆	⋆	⋆	⋆	26	69	79	0.3489	14	38	41	0.4181
16.	3000	19	149	58	0.1235	5	18	16	0.2524	10	12	13	0.0617
	6000	19	149	58	0.1494	5	18	16	0.6322	10	12	13	0.3435
	9000	19	149	58	0.2219	5	18	16	0.4866	10	12	13	0.1647
	12,000	19	149	58	0.4396	5	18	16	0.5401	10	12	13	0.2115
	15,000	19	149	58	0.4359	5	18	16	0.6261	10	12	13	0.4103
17.	3000	67	432	202	0.5096	59	276	178	0.4913	66	428	199	0.8889
	6000	67	432	202	0.9914	59	276	178	0.8029	66	428	199	1.2145
	9000	67	432	202	1.9478	59	276	178	0.89849	66	428	199	1.8835
	12,000	67	432	202	1.8094	59	276	178	1.4494	66	428	199	2.3523
	15,000	67	432	202	2.0496	59	276	178	1.4754	66	428	199	2.5114
18.	3000	⋆	⋆	⋆	⋆	⋆	⋆	⋆	⋆	23	101	109	2.4467
	6000	⋆	⋆	⋆	⋆	⋆	⋆	⋆	⋆	39	245	116	2.8094
	9000	⋆	⋆	⋆	⋆	⋆	⋆	⋆	⋆	40	256	117	2.8123
	12,000	⋆	⋆	⋆	⋆	⋆	⋆	⋆	⋆	41	255	117	2.8332
	15,000	⋆	⋆	⋆	⋆	⋆	⋆	⋆	⋆	50	331	152	2.6772
19.	3000	52	335	157	0.31743	67	357	202	0.2436	17	114	52	0.14464
	6000	52	335	157	0.2549	83	444	250	0.4514	17	114	52	0.4019
	9000	52	335	157	0.4769	85	452	256	0.8718	17	114	52	0.2591
	12,000	52	335	157	0.4952	76	399	229	0.6840	17	114	52	0.3423
	15,000	52	335	157	0.9069	89	473	268	0.9816	17	114	52	0.3721
20.	3000	31	242	94	0.1337	61	327	184	0.2679	8	67	24	0.0534
	6000	32	246	97	0.2033	34	246	103	0.2648	8	68	25	0.1076
	9000	32	244	97	0.4123	43	300	130	0.5784	8	68	25	0.1785
	12,000	26	216	79	0.4675	42	347	127	0.9517	8	68	25	0.2726
	15,000	31	229	94	0.8068	34	218	103	0.5337	8	68	25	0.2123
21.	3000	5	12	16	0.0460	7	17	22	0.2688	5	12	16	0.0427
	6000	5	12	16	0.0338	7	17	22	0.4862	5	12	16	0.2084
	9000	5	12	16	0.0505	7	17	22	0.0599	5	12	16	0.1205
	12,000	5	12	16	0.0634	7	17	22	0.1868	5	12	16	0.0967
	15,000	5	12	16	0.0723	7	17	22	0.1079	5	12	16	0.1268
22.	3000	34	241	103	0.1115	40	266	121	0.1011	9	49	28	0.0387
	6000	32	227	97	0.1781	42	283	127	0.2281	7	43	22	0.0766
	9000	29	222	88	0.3336	39	282	118	0.6486	6	44	19	0.0174
	12,000	24	171	73	0.5712	41	273	124	0.7159	11	66	34	0.3608
	15,000	29	230	88	0.5215	42	305	127	0.6488	9	56	28	0.3549
23.	3000	26	120	79	0.2191	62	223	384	0.6480	28	272	85	0.3291
	6000	19	54	58	0.1244	62	223	384	0.6481	13	105	40	0.2024
	9000	20	58	61	0.15792	158	498	475	1.4069	18	173	55	0.4258
	12,000	25	168	76	0.4151	52	310	157	0.8004	14	103	43	0.5185
	15,000	26	231	79	0.6274	46	385	139	1.1442	14	81	43	0.3717
24.	3000	2	3	3	0.0380	1	2	2	0.0360	1	1	1	0.0042
	6000	2	3	3	0.0031	1	2	2	0.0249	1	1	1	0.0024
	9000	2	3	3	0.0034	1	2	2	0.0391	1	1	1	0.0033
	12,000	2	3	3	0.0037	1	2	2	0.0485	1	1	1	0.0055
	15,000	2	3	3	0.0193	1	2	2	0.0486	1	1	1	0.0075
25.	3000	17	29	37	0.0300	21	79	64	0.0510	13	21	22	0.0276
	6000	17	31	39	0.0391	23	86	70	0.1759	13	22	22	0.0530
	9000	17	32	39	0.058212	23	87	70	0.2089	14	24	22	0.0811
	12,000	18	33	40	0.0698	24	91	73	0.4886	15	27	22	0.0381
	15,000	19	33	40	0.1273	23	88	70	0.3694	17	29	29	0.4467

, we observe that of the 125 instances, DSCGA solves all (about 100%), while SNCG fails in 30 instances (about 24%) and SSGM fails in 10 (about 8%) problems. For example, as shown in Figure 2, when $\tau =1$, the DSCGA algorithm successfully solves approximately 96% of the test problems using a lower iteration count. In comparison, SSGM solves about 63%, and SNCG solves around 52% of the problems. As illustrated in Figure 3, for $\tau =1$, the DSCGA algorithm solves nearly 92% of the test problems with the fewest function evaluations. In contrast, SSGM and SNCG address approximately 63% and 45% of the problems, respectively, under the same metric. Figure 4 for $\tau =1$ shows that DSCGA effectively solves approximately 94% of the test problems with the fewest gradient evaluations. In comparison, SSGM and SNCG achieve success rates of 60% and 50%, respectively, based on gradient computations. Furthermore, considering the CPU time, Figure 5 highlights that DSCGA is highly efficient, solving around 71% of test problems in the least amount of time when $\tau =1$. In contrast, SSGM and SNCG address approximately 58% and 54% of the problems, respectively, under the same criterion. An important insight from these figures is that, while all algorithms exhibited comparable performance initially, the proposed method consistently outperformed the others in all metrics as $\tau$ increased. This underscores the robustness of the method and its strong competitive edge in solving the problems under study.

5. Applications in Inverse Kinematics

This section focuses primarily on the equation that governs the discrete kinematic model with four degrees of freedom, as described in the following. This equation characterizes the planar kinematic model of a four-joint system.

$$p\left(\theta \right)=\left[\begin{array}{c}{\sum }_{i=1}^4{l}_{i}cos({\theta }_1+{\theta }_2+\dots +{\theta }_i)\\ {\sum }_{i=1}^4{l}_{i}sin({\theta }_1+{\theta }_2+\dots +{\theta }_i)\end{array}\right],$$

(45)

Here, $p(·)$ represents the kinematic transformation, capturing the positioning of a robot’s endpoint or any specific component as a function of active adjustments in its joint angles, denoted by $\theta \in {\mathbb{R}}^4$. For $i=1,2,3,4$, the link is represented by $l_i$, which is the length of the relevant link. The position and orientation of the robot’s end effector are generally represented by $p\left(\theta \right)$ in the framework of robotic motion. On the other hand, this particular study depicts the robot’s x-y Cartesian position.

Let the vector corresponding to the intended path at a given time instant $t_k$ be represented as ${\phi }_{t_k}\in {\mathbb{R}}^2$. To be computed at each time segment $t_k$ within the interval $[0,t_f]$, we present the following least squares model. The following is the definition of the optimization problem:

$$\underset{\theta \in {\mathbb{R}}^4}{min}{\displaystyle \frac{1}{2}}\sum _{i=1}^4{\left(p_i\left(\theta \right)-{\phi }_{t_k,i}\right)}^2,$$

(46)

where ${\phi }_{t_k,i}$ indicates the goal route of a Lissajous curve at $t_k$. Additionally, several Lissajous curves in 4DOF have been used; refer to [46] for an example. The forward kinematic function $p\left(\theta \right)$ maps joint space to Cartesian space:

$$\psi =p\left(\theta \right),$$

(47)

The inverse kinematic problem is then formulated as an NLS problem:

$$\underset{\theta \in {\mathbb{R}}^n}{min}{\displaystyle \frac{1}{2}}{\parallel u\left(\theta \right)\parallel }^2,$$

(48)

where the residual function is defined as follows:

$$u\left(\theta \right)=p\left(\theta \right)-{\psi }_{t_k}$$

The gradient is then computed by the following:

$$g_{t_k}=V^T\left({\theta }_{t_k}\right)u\left({\theta }_{t_k}\right)$$

(49)

where $V\left({\theta }_{t_k}\right)\in {\mathbb{R}}^{m\times n}$ is the Jacobian of $u\left({\theta }_{t_k}\right)$ defined as $V\left({\theta }_{t_k}\right)={\displaystyle \frac{\delta p\left({\theta }_{t_k}\right)}{\delta \left({\theta }_{t_k}\right)}}$. We update the joint variables using the DSCGA iterative formula:

$${\theta }_{t_{k+1}}={\theta }_{t_k}+{\alpha }_{t_k}{d}_{t_k},$$

(50)

where $d_{t_k}$ is the search direction of the DSCGA formula:

$$d_{t_k}^{DSCGA}=-W_{t_k}{g}_{t_k}+{\beta }_{t_k}^{*}{d}_{t_{k-1}},\forall t_k\ge 1.$$

(51)

To simulate the results and solve the inverse kinematics problem, the following pseudo-code was utilized.

The following parameters were utilized in the process to put the strategy into practice and simulate the outcomes:

• The initial joint angular vector is ${\theta }_{t_0}=\left[0,{\displaystyle \frac{\pi }{4}},{\displaystyle \frac{\pi }{3}},{\displaystyle \frac{\pi }{2}}\right]$ at the starting time $t_0=0$.
• The length of the links is represented by $l_i$, where $i=1,\dots ,4$.
• The task should take $t_f=10$ seconds in total.

The selected Lissajous curve trajectories exhibit nonlinear and time-varying characteristics due to the presence of sinusoidal components in both the x and y directions. By varying the frequencies and phase shifts, the resulting paths are oscillatory, non-repetitive, and highly dynamic, making them particularly challenging for trajectory tracking, especially in inverse kinematics and motion control. These conditions often lead to ill-conditioned Jacobians and require solvers to handle rapid variations with high precision.

Problem 26.

The motion of the end-effector is defined by a Lissajous pattern, as given below.

$${\phi }_{t_k}=\left[\begin{array}{c}{\displaystyle \frac{3}{2}}+0.3sin\left(4t_k+{\displaystyle \frac{2\pi }{3}}\right)\\ {\displaystyle \frac{\sqrt{3}}{2}}+0.3cos(3t_k+{\displaystyle \frac{2\pi }{3}})\end{array}\right].$$

(52)

The results from the simulation of problem 26 are shown in the following figures. Figure 6a displays the robot trajectories generated using Algorithm 2. Figure 6b illustrates the optimal synthesis of the robot paths for the given task. Furthermore, Figure 7a,b present the residual errors for each method used in the numerical analysis.

Algorithm 2: Solution of the 4DOF Model Using the DSCGA Method

Step 1: Inputs: $t_0$, ${\theta }_{t_0}$, $t_f$, g, and $K_{max}$ Step 2: For $k=1$ to $K_{max}$, repeat $t_k=k·g$; Step 3: Compute ${\phi }_{t_k,i}$; Step 4: Calculate ${\theta }_{t_k}$ using the DSCGA $\left({\theta }_{t_0},{\phi }_{t_k,i}\right)$, as detailed in Algorithm 1; Step 5: Set ${\theta }_{\mathrm{new}}=\left[{\theta }_{t_0};{\theta }_{t_k}\right]$; Step 6: Return: ${\theta }_{\mathrm{new}}$

The findings of Problem 26 are summarized in

Table 4. The performance of the DSCGA algorithm compared to SSGM and SNCG algorithms on Problem 26.

Methods	DSCGA			SNCG			SSGM
	NI	Time (s)	Re	NI	Time (s)	Re	NI	Time (s)	Re
Problem 26	52	0.299	${10}^{-6}$	77	0.824	${10}^{-5}$	81	0.496	${10}^{-5}$

, which also includes residual tracking error, CPU time, and the number of iterations. With just 52 rounds needed, DSCGA shows the best performance among the methods examined. The iterations for SSGM and SNCG are as follows: 81 and 77, respectively. The other approaches, SSGM and SNCG, are slower than DSCGA, leading to computational efficiency, achieving the problem in the shortest CPU time of 0.299 s.

The residual error norm rates are shown in Figure 7a,b, which provide a comparison of the performance of the algorithms. Having the lowest observed error of roughly ${10}^{-6}$, the DSCGA algorithm exhibits better accuracy. The SSGM follows, with error rates of about ${10}^{-5}$ and ${10}^{-5}$, respectively. These results highlight how the DSCGA algorithm reduces residual error more precisely than the other approaches.

Problem 27.

The following Lissajous curve is described by

$${\phi }_{t_k}=\left[\begin{array}{c}{\displaystyle \frac{3}{2}}+{\displaystyle \frac{1}{5}}sin\left(t_k\right)\\ {\displaystyle \frac{\sqrt{3}}{2}}+{\displaystyle \frac{1}{5}}sin\left(4t_k\right)\end{array}\right].$$

(53)

The simulation results for Problem 27 are shown in the figures below. Figure 8a depicts the robot paths generated by Algorithm 2, while Figure 8b demonstrates the effective execution of the robot trajectories for the specified task. Furthermore, the residual errors for each method used in the experiment are presented in Figure 9a,b.

The results for Problem 27 are shown in

Table 5. The performance of the DSCGA algorithm compared to SSGM and SNCG algorithms on Problem 27.

Methods	DSCGA			SNCG			SSGM
	NI	Time (s)	Re	Ni	Time (s)	Re	NI	Time (s)	Re
Problem 27	58	0.433	${10}^{-6}$	76	0.469	${10}^{-4}$	89	0.604	${10}^{-5}$

, which includes the residual tracking error, CPU time, and the number of iterations. DSCGA is the most efficient method, only requiring 58 iterations to solve the problem. However, SSGM and SNCG require more iterations: 89 and 76, respectively. DSCGA also completes the task the fastest, with a minimal CPU time of 0.433 s.

The output data from the model solution is presented in the charts in Figure 8. As shown in Figure 8b, the end effector accurately followed the intended trajectory, effectively completing the task. The residual error norms are shown in Figure 9, with DSCGA achieving the lowest error rate of approximately ${10}^{-6}$. Following DSCGA, SSGM and SNCG showed error rates of around ${10}^{-5}$ and ${10}^{-4}$, respectively.

6. Conclusions

This study introduced an accelerated diagonally structured CG method designed to solve NLS optimization problems. The method guarantees a sufficient descent property without the need for line search. Its performance was evaluated using a range of benchmark problems from existing literature, with global convergence proven under strong Wolfe line search conditions. The findings emphasize the competitive edge of the method compared to other similar approaches documented in the literature. The method was also applied to solve the inverse kinematics problem in robotic motion control for a 4DOF system, where it outperformed both SNCG and SSGM in several performance metrics. Future work should focus on developing more advanced accelerated CG algorithms that can effectively handle high-dimensional problems commonly encountered in scientific and engineering applications.