Development of an Efficient Variable Step-Size Gradient Method Utilizing Variable Fractional Derivatives

Abstract

The fractional gradient method has garnered significant attention from researchers. The common view regarding fractional-order gradient methods is that they have a faster convergence rate compared to classical gradient methods. However, through conducting theoretical convergence analysis, we have revealed that the maximum convergence rate of the fractional-order gradient method is the same as that of the classical gradient method. This discovery implies that the superiority of fractional gradients may not reside in achieving fast convergence rates compared to the classical gradient method. Building upon this discovery, a novel variable fractional-type gradient method is proposed with an emphasis on automatically adjusting the step size. Theoretical analysis confirms the convergence of the proposed method. Numerical experiments demonstrate that the proposed method can converge to the extremum point both rapidly and accurately. Additionally, the Armijo criterion is introduced to ensure that the proposed gradient methods, along with various existing gradient methods, can select the optimal step size at each iteration. The results indicate that, despite the proposed method and existing gradient methods having the same theoretical maximum convergence speed, the introduced variable step size mechanism in the proposed method consistently demonstrates superior convergence stability and performance when applied to practical problems.

1. Introduction

It has been more than 300 years since fractional calculus was first proposed. For a long time, due to the lack of corresponding physical background, fractional calculus was regarded as a purely theoretical study of mathematics. With the development of science and technology, researchers found that fractional calculus can provide a good description of modeling some abnormal behaviors such as diffusion process [1], cybernetics [2,3], viscoelasticity [4,5], biology [6], etc. The long-memory properties of fractional calculus not only reflect local characteristics at the time of evaluation but also reflect the entire history of the function. Therefore, fractional calculus was extensively used in complex nonlinear dynamical systems with memory and genetic properties [7,8,9].

As a natural extension of the constant fractional derivative, variable fractional derivative is a relatively new branch in the field of fractional calculus and has also been widely applied in various mechanical problems such as contact dynamics [10], viscoelasticity [11], fracture mechanics [12], vibration control [13], etc. In the variable fractional derivative, the order can be continuously varied as a function of the integral (differential) independent variable, which can be classified into three categories according to the memory characteristics of the order [14].

As one of the standard optimization algorithms, the gradient method (GM) has been widely used in many engineering applications, such as adaptive filtering [15], image processing [16], and system identification [17]. Recently, the fractional gradient method has attracted the attention of many scholars. The basic principle of this method is to replace the first-order derivative with a fractional-order derivative [18]. However, Pu et al. [19] found that the fractional gradient method cannot guarantee to converge to the extreme point. This shortcoming has been overcome by introducing the variable initial value [20] and truncation technique [21]. After this, the above-improved methods were successfully applied in many scenarios, such as parameter identification [22,23,24,25,26,27], signal processing [28,29,30,31,32,33], and neural networks [34,35,36,37,38,39]. Wei et al. [40] recently divided the existing fractional-type gradient method into three categories, such as variable initial value type, truncation type, and variable fractional-order type. On this basis [40], they further developed a new method that can converge to the real extreme point by replacing the constant fractional order with a variable fractional order.

Research on the fractional gradient method has been progressing rapidly. It is important to note that many of these studies emphasize the higher convergence rate of the fractional gradient method compared to the classical gradient method. However, the research also suggests that the maximum convergence rate of the fractional gradient method is the same as that of the classical gradient method. This implies that the advantage of the fractional gradient method may not lie solely in its higher convergence rate.

Therefore, it is crucial to explore the advantages of the fractional gradient method from a different perspective. It is well-known that gradient-based methods require a small iteration step size to achieve high accuracy, while a large iteration step size is needed for rapid convergence. Consequently, the classical gradient method, with its fixed step size, cannot simultaneously achieve high precision and a rapid convergence rate. Hence, there is a need to propose a new gradient method that is capable of achieving both high precision and a rapid convergence rate.

To address this issue, this paper presents a novel gradient descent method based on variable fractional derivatives, aiming to achieve both rapid convergence rate and high convergence precision. The underlying principle of this method is to offer a larger iteration step size in the initial stages of the iteration, facilitating rapid convergence. Conversely, towards the end of the iteration, the step size gradually decreases to a fixed and smaller value, ensuring a higher level of convergence precision. The main contributions of this paper are outlined as follows:

• The fractional-type methods currently in existence can be broadly categorized into three types. Theoretical convergence analysis has revealed that the maximum convergence rate of fractional-type gradient methods is comparable to that of classical gradient methods. These findings suggest that the advantage of fractional gradients may not necessarily lie in their superior convergence rates compared to the classical gradient method.
• We reveal that the fractional order operator’s key impact on the algorithm lies in the variable step size mechanism. Based on this discovery, a novel variable step-size gradient method is proposed by combining the existing three kinds of gradient methods. This method can overcome the shortcoming that the classical gradient method cannot balance the rapid convergence rate and high convergence precision.
• Numerical simulations are provided to demonstrate the effectiveness of the proposed variable fractional-order gradient method. It can be observed that the proposed method converges rapidly and accurately to the extremum of convex optimization problems. Moreover, under optimal step-size conditions, the proposed method exhibits remarkable capability, maintaining a high convergence performance even when other methods struggle to achieve their maximum convergence speeds in practical simulations.

The structure of the paper is as follows: In Section 2, several definitions, such as the variable fractional derivative and a lemma about convex function, are introduced. The novel gradient method based on the variable fractional derivative is proposed in Section 3. In Section 4, the theoretical convergence analysis of the proposed algorithm is presented. Numerical simulations are presented to illustrate the effectiveness of the proposed method in Section 5. Finally, the paper is concluded in the Section 6.

2. Classification of Fractional Gradient Methods

Ever since fractional calculus was invented, the definition of fractional-order derivatives has been changing and improving. In recent years, scholars have put forward various definitions of fractional-order derivatives. However, there are three widely accepted definitions which are the Riemann–Liouville definition, the Grunwald–Letnikov definition, and the Caputo definition. Previous studies have shown that the fractional-order gradient method with Riemann–Liouville definition cannot converge to the true extreme point of the objective function [21,25]. Therefore, in this paper, we adopt the Caputo definition, given as follows:

$${}_{t_0}{}^C\mathit{𝒟}{}_{t_k}^{\alpha }f\left(t\right)=\frac{1}{\mathsf{\Gamma }\left(n-\alpha \right)}{{\displaystyle \int }}_{t_0}^t\frac{f^{\left(n\right)}\left(\tau \right)}{{\left(t-\tau \right)}^{\alpha -n+1}} \mathrm{d}\tau$$

(1)

where $n=\left[\alpha \right]$ is the upper integer bound on $\alpha$ and $\mathsf{\Gamma }(\cdot )$ is the gamma function. Alternatively, it can be rewritten in a similar form to the conventional Taylor series:

$${}_{t_0}{}^C\mathit{𝒟}{}_{t_k}^{\alpha }f\left(t\right)={{\displaystyle \sum }}_{i=n}^{\infty }\frac{f^{\left(i\right)}\left(t_0\right)}{\mathsf{\Gamma }\left(i+1-\alpha \right)}{\left(t-t_0\right)}^{i-\alpha }$$

(2)

The variable fractional-order Caputo derivative of $f\left(t\right)$ with the order $\alpha \left(t\right)>0$ on the interval $\left(t_0,t\right)$ is defined as:

$${}_{t_0}{}^C\mathit{𝒟}{}_{t_k}^{\alpha \left(t\right)}f\left(t\right)=\frac{1}{\mathsf{\Gamma }\left[n-\alpha \left(t\right)\right]}{{\displaystyle \int }}_{t_0}^t\frac{1}{{\left(t-\tau \right)}^{1+\alpha \left(t\right)-n}}\frac{{\mathrm{d}}^{n}f\left(\tau \right)}{\mathrm{d}{\tau }^n} \mathrm{d}\tau$$

(3)

where $n=\left[\alpha \left(t\right)\right]$ is the upper integer bound on α(t) and $\mathsf{\Gamma }(\cdot )$ is the gamma function. We emphasize that the dynamics of all the systems considered in this study start from $t=0$ without any prior history. Thus, all functions of interest are identically zero for all instants before $t=0$.

Suppose that $f\left(t\right)$ is a smooth convex function with a unique extreme point $\widehat{t}$. It is well known that each iteration step of the fractional-order gradient method (FOGM) is formulated as:

$$t_{k+1}=t_k-\rho {}_{t_0}{}^C\mathit{𝒟}{}_{t_k}^{\alpha }f\left(t\right)$$

(4)

where $\rho >0$ is the iteration step size. Note that the FOGM will degenerate to its counterpart integer-order gradient method as $\alpha \to 1$.

It is well known that the fractional-order gradient method cannot converge to the real extreme value. To circumvent this problem, scholars propose the following three improvement versions, listed as follows:

Type-I (variable initial value strategy): Let the initial value $t_0$ of the fractional order derivative ${}_{t_0}{}^C\mathit{𝒟}{}_{t_k}^{\alpha }f\left(t\right)$ change to $t_{k-1}$ with the iteration; this improved iteration format can be formulated as:

$$t_{k+1}=t_k-\rho {}_{t_{k-1}}{}^C\mathit{𝒟}{}_{t_k}^{\alpha }f\left(t\right)$$

(5)

Type-II (truncation strategy): According to [37], the Caputo fractional-order derivative can be also rewritten in another form as a Taylor series:

$${}_{t_0}{}^C\mathit{𝒟}{}_{t_k}^{\alpha }f\left(t\right)={{\displaystyle \sum }}_{i=n}^{\infty }(\begin{array}{c}{t}_0-n\\ i-n\end{array})\frac{f^{\left(i\right)}\left(t\right)}{\mathsf{\Gamma }\left(i+1-\alpha \right)}{\left(t-t_0\right)}^{i-\alpha }$$

(6)

Under this Taylor expansion of the scheme, by truncating the higher-order terms and preserving the first-order terms $\frac{1}{\mathsf{\Gamma }\left(2-\alpha \right)}\mathit{𝒟}f\left(t\right){\left(t-t_0\right)}^{1-\alpha }$, the iteration format of this method can be formulated as:

$$t_{k+1}=t_k-\rho \mathit{𝒟}f\left(t_k\right){\left(t_k-t_0\right)}^{1-\alpha }$$

(7)

Type-III (variable fractional-order strategy): By designing different variable fractional-order schemes, let the variable fractional-order derivative $\alpha \left(t\right)$ converge to $1$ when the function $f\left(t\right)$ converges to the real extreme value $f\left(\widehat{t}\right)$. The iteration format of this method can be formulated as:

$$t_{k+1}=t_k-\rho {}_{t_0}{}^C\mathit{𝒟}{}_{t_k}^{\alpha \left(t_k\right)}f\left(t\right)$$

(8)

with $\alpha \left(t_k\right)\to 1$ as the iteration converges. We have thus provided a summary of the main improvement strategies in the literature.

Remark 1.

It should be noted that both Type-I and Type-III require calculating the fractional derivative at each iteration, which is very time-consuming for the cases where the objective function cannot be expressed explicitly. Whereas, in Type-II, the convergence accuracy of the algorithm is affected due to the truncation of the higher-order terms of the fractional derivatives. In [21], Chen proposed a novel fractional-order gradient method that combines Type-I and Type-II. However, this hybrid method may lead to a numerical oscillator in some cases, such that it cannot always achieve a satisfactory convergence accuracy.

A significant amount of research conducted using numerical simulations has found that the fractional gradient method, compared to the traditional gradient method, has a faster convergence rate advantage. In the subsequent subsection, we will conduct an in-depth theoretical analysis of the convergence rate of the fractional gradient method to verify whether it possesses a greater convergence speed advantage compared to the traditional gradient method.

3. Convergence Analysis of Fractional Gradient Method

In this section, we will theoretically analyze the convergence rate of the fractional-type gradient algorithm. To do this, we introduce the following Lemma:

Lemma 1.

Suppose that $f(.)$ is a smooth convex function; then, it must satisfy the following Lipschitz continuity condition

$$f\left(y\right)\le f\left(x\right)+\mathit{𝒟}f{\left(x\right)}^T\left(y-x\right)+\frac{M}{2}{{\displaystyle \parallel }y-x{\displaystyle \parallel }}_2^2$$

(9)

and strong convex condition

$$f\left(y\right)\ge f\left(x\right)+\mathit{𝒟}f{\left(x\right)}^T\left(y-x\right)+\frac{m}{2}{{\displaystyle \parallel }y-x{\displaystyle \parallel }}_2^2$$

(10)

where M and m are arbitrary constants.

Firstly, we rewrite Type-I, -II and -III as the following unified form:

$$t_{k+1}=t_k-\rho \Phi \left(t_k\right)$$

(11)

where $\Phi \left(t_k\right)$ represents the iteration functions such as ${}_{t_{k-1}}{}^C\mathit{𝒟}{}_{t_k}^{\alpha }f\left(t\right)$, $\mathit{𝒟}f\left(t_k\right){\left(t_k-t_0\right)}^{1-\alpha }$ and ${}_{t_0}{}^C\mathit{𝒟}{}_{t_k}^{\alpha \left(t_k\right)}f\left(t\right)$. Obviously, $\Phi \left(t_k\right)$ is a bounded function. Comparing inequality (9) and algorithm (11), we let $y=t_{k+1}$, $x=t_k$ and substitute them into Equation (9). One arrives at:

$$\begin{array}{c}f\left(t_{k+1}\right)-f\left(t_k\right)\le -\mathit{𝒟}f\left(t_k\right)\rho \Phi \left(t_k\right)+\frac{M}{2}{{\displaystyle \parallel }\rho \Phi \left(t_k\right){\displaystyle \parallel }}_2^2\\ \le -\left[\rho {\mu }_k-\frac{M}{2}{\left(\rho {\mu }_k\right)}^2\right]{{\displaystyle \parallel }\mathit{𝒟}f\left(t_k\right){\displaystyle \parallel }}_2^2\end{array}$$

(12)

with ${\mu }_k=\Phi \left(t_k\right)/\mathit{𝒟}f\left(t_k\right)$. Note that Equation (12) degenerates to its counterpart integer-order gradient descent method as ${\mu }_k\to 1$. For this case, let $g\left(\rho {\mu }_k\right)=\rho {\mu }_k-\frac{M}{2}{\left(\rho {\mu }_k\right)}^2$. One can ascertain that $g\left(\rho {\mu }_k\right)$ has a minimum value of $g{\left(\rho {\mu }_k\right)}_{min}=\frac{1}{2M}$ when $\rho {\mu }_k=\frac{1}{M}$. By substituting $g({\left(\rho {\mu }_k\right)}_{min}=\frac{1}{2M}$ into inequality (12), one arrives at

$$\begin{array}{c}f\left(t_{k+1}\right)-f\left(t_k\right)\le -\frac{1}{2M}{\left|\left|\mathit{𝒟}f\left(t_k\right)\right|\right|}_2^2\end{array}$$

(13)

On the other hand, we let the right-hand side of Equation (10) be the function $\psi \left(y\right)$ that takes the extreme value when $y=x-\frac{1}{m}\mathit{𝒟}f\left(x\right)$. Substituting this into Equation (10), one can yield

$$\begin{array}{c}f\left(y\right)\ge f\left(x\right)-\frac{1}{2m}{\left|\left|\mathit{𝒟}f\left(x\right)\right|\right|}_2^2\end{array}$$

(14)

Let $y=\widehat{t}$ be the optimal point and $x=t_{k+1}$; Equation (14) becomes:

$$\begin{array}{c}f\left(\widehat{t}\right)\ge f\left(t_{k+1}\right)-\frac{1}{2m}{\left|\left|\mathit{𝒟}f\left(t_{k+1}\right)\right|\right|}_2^2\end{array}$$

(15)

Subtracting optimal value $f\left(\widehat{t}\right)$ from both sides of Equation (13) yields

$$\begin{array}{c}f\left(t_{k+1}\right)-f\left(\widehat{t}\right)\le f\left(t_k\right)-f\left(\widehat{t}\right)-\frac{1}{2M}{\left|\left|\mathit{𝒟}f\left(t_k\right)\right|\right|}_2^2\end{array}$$

(16)

Combining Equations (15) and (16) further yields

$${\displaystyle \parallel }\frac{f\left(t_{k+1}\right)-f\left(\widehat{t}\right)}{f\left(t_k\right)-f\left(\widehat{t}\right)}{\displaystyle \parallel }\le {\displaystyle \parallel }1-\frac{m}{M}{\displaystyle \parallel }<1$$

(17)

The difference between $f\left(t_{k+1}\right)-f\left(\widehat{t}\right)$ and $f\left(t_k\right)-f\left(\widehat{t}\right)$ is proportional and $f\left(t_{k+1}\right)$ is closer to the optimal value $f\left(\widehat{t}\right)$ than $f\left(t_k\right)$ as ${\displaystyle \parallel }1-\frac{m}{M}{\displaystyle \parallel }$ is always less than 1. Thus, we theoretically prove that the fractional-type gradient method is always convergence.

In addition, Equation (17) also provides an estimate of the maximum convergence rate of the fractional-type gradient method, which is the same as the classical gradient method. That is to say, the advantage of the fractional-type gradient method may not be that it has a faster convergence rate than the classical gradient method. However, a related question arises about the practical feasibility of the classical gradient method to achieve its theoretical maximum convergence rate in real-world problem-solving contexts. This prompts us to rediscover the advantage of the fractional gradient method.

Remark 2.

We further rewrite Equation (11) as:

$$t_{k+1}=t_k-\rho {\mu }_k\mathit{𝒟}f\left(t_k\right)$$

(18)

Herein, ${\mu }_k=\Phi \left(t_k\right)/\mathit{𝒟}f\left(t_k\right)$. Note that Equation (18), the fractional-type gradient method, adds an auxiliary iteration function ${\mu }_k$ when compared to the classical gradient method. Note that  ${\mu }_k$ is a variable that changes with the number of iterations k. Hence,   ${\mu }_k$ can be regarded as a regulation parameter to adjust the step size of the fractional-type gradient method at each iteration.

Remark 3.

It is widely acknowledged that the classical gradient method encounters difficulties in achieving a satisfactory trade-off between convergence rate and precision. Nonetheless, the fractional-type gradient method, with its inherent variable step-size characteristic, presents a prospective solution to address this dilemma. In view of this, our objective is to investigate a fresh standpoint regarding the merits of the fractional gradient method in comparison to the classical gradient method.

4. A Novel Variable Step-Size Gradient Method

4.1. Establishment of the Novel Gradient Method

To balance the trade-off between convergence rate and precision, we propose a novel fractional-type gradient method by combining the Type-I, -II and -III. We construct the novel method starting from Type-III:

$$t_{k+1}=t_k-\rho {}_{t_0}{}^C\mathit{𝒟}{}_{t_k}^{\alpha \left(t_k\right)}f\left(t\right)$$

(19)

where $\alpha \left(t_k\right)$ is designed as follows:

$$\alpha \left(t_k\right)=1-u{e}^{-\frac{v}{t_k-t_{k-1}}}$$

(20)

Here, $u$ and $v$ are control parameters that can adjust the convergence rate and range of the algorithm (19). Obviously, if only the proposed algorithm (19) is convergent, the iteration points $t_k$ and $t_{k-1}$ are close enough that the variable fraction-order $\alpha \left(t\right)$ tends to 1. Therefore, this scheme can guarantee that it converges to the true extreme point.

As noted above, it is very time-consuming to calculate the fractional derivative at each iteration. To avoid this problem, we further introduce the variable initial value strategy (Type-I) and truncation strategy (Type-II).

Firstly, by replacing the initial value $t_0$ with variable initial value $t_k$, Equation (19) can be rewritten as

$$t_{k+2}=t_{k+1}-\rho {}_{t_k}{}^C\mathit{𝒟}{}_{t_{k+1}}^{\alpha \left(t_k\right)}f\left(t\right)$$

(21)

where $0<\alpha \left(t\right)<1$ and $\rho >0$. According to Equation (6), the variable fractional derivative can be expressed as the following equivalent form:

$${}_{t_k}{}^C\mathit{𝒟}{}_{t_{k+1}}^{\alpha \left(t\right)}f\left(t\right)={{\displaystyle \sum }}_{i=0}^{\infty }\frac{{\mathit{𝒟}}^{\left(i+1\right)}f\left(t_k\right)}{\mathsf{\Gamma }\left(i+2-\alpha \left(t\right)\right)}{\left(t_{k+1}-t_k\right)}^{i+1-\alpha \left(t_k\right)}$$

(22)

Considering that the first-order item plays a dominant role, Algorithm (21) can be truncated as

$$t_{k+2}=t_{k+1}-\frac{\rho }{\mathsf{\Gamma }\left(2-\alpha \left(t\right)\right)}\mathit{𝒟}f\left(t_{k+1}\right){\left(t_{k+1}-t_k\right)}^{1-\alpha \left(t_k\right)}$$

(23)

Since $\frac{\rho }{\mathsf{\Gamma }\left(2-\alpha \left(t\right)\right)}$ can be seen as part of the step size, it can be simplified as

$$t_{k+2}=t_{k+1}-\rho \mathit{𝒟}f\left(t_{k+1}\right){\left(t_{k+1}-t_k\right)}^{1-\alpha \left(t_k\right)}$$

(24)

To force the step size to be positive and avoid singularity, the proposed method can be further modified as:

$$t_{k+2}=t_{k+1}-\rho \mathit{𝒟}f\left(t_{k+1}\right){\left(\left|t_{k+1}-t_k\right|+\delta \right)}^{1-\alpha \left(t_k\right)}$$

(25)

here, $\delta$ is selected as a very small positive constant so that it hardly affects the convergence of the proposed algorithm. Up to now, we have completed a description of the proposed algorithm, which can be called a truncation variable fractional-order gradient method (TVFOGM). The flow chart of the TVFOGM is shown in Algorithm 1.

Algorithm 1: TVFOGM

1: Initialize $\rho ,\delta ,u,v,k_{max}$ and $t_0$.

2: Calculate $t_1$ by conventional gradient method:

$t_1=t_0-\rho \mathit{𝒟}f\left(t_0\right)$

3: Calculate the variable fractional order by:

$\alpha \left(t_k\right)=1-u{e}^{-\frac{v}{t_k-t_{k-1}}}$

4: Calculate the next iteration point $t_k \left(k\ge 2\right)$ by:

$t_{k+2}=t_{k+1}-\rho \mathit{𝒟}\left(t_{k+1}\right){\left(\left|t_{k+1}-t_k\right|+\delta \right)}^{1-\alpha \left(t_k\right)}$

5: Check if $t_k$ meets the accuracy requirements or $k>k_{max}$,

if not, go to step 3.

4.2. Convergence Analysis of TVFOGM

In this section, we will theoretically prove that the proposed algorithm can converge to the true extremum point using the reduction to absurdity. Assume that $t_k$ converges to a different point $\tilde{t}\ne \widehat{t}$ and $\mathit{𝒟}f\left(\tilde{t}\right)\ne 0$. It is concluded that, for any sufficiently small positive scalar $\epsilon$, there exists a sufficient large number $N\in \mathbb{N}$ such that:

$$\left|t_k-\tilde{t}\right|<\epsilon <\left|\widehat{t}-\tilde{t}\right|,\forall k>N$$

Thus $d={ \inf }_{k>N}\left|\mathit{𝒟}f\left(t_k\right)\right|>0$ must hold. According to Inequality (24), one can obtain:

$$\begin{array}{c}\left|t_{k+2}-t_{k+1}\right|=\rho \left|\mathit{𝒟}f\left(t_{k+1}\right)\right|{\left(\left|t_{k+1}-t_k\right|+\delta \right)}^{1-\alpha \left(t_k\right)}\\ \ge \rho d{\left(\left|t_{k+1}-t_k\right|+\delta \right)}^{1-\alpha \left(t_k\right)}\end{array}$$

where $\delta$ is a given positive real number. Therefore, $\left|t_{k+2}-t_{k+1}\right|\ge \rho d{\left(\left|t_{k+1}-t_k\right|+\delta \right)}^{1-\alpha \left(t_k\right)}\ge \rho d{\left(\left|t_{k+1}-t_k\right|\right)}^{1-\alpha \left(t_k\right)}$. Take $\epsilon <\frac{{\left(\rho d\right)}^{1/\alpha \left(t_k\right)}}{2}$; thus, $\rho d>{\left(2\epsilon \right)}^{\alpha \left(t_k\right)}$, and then:

$$\begin{array}{c}\left|t_{k+2}-t_{k+1}\right|\ge \rho d{\left|t_{k+1}-t_k\right|}^{1-\alpha \left(t\right)}\\ >{\left(2\epsilon \right)}^{\alpha \left(t_k\right)}{\left|t_{k+1}-t_k\right|}^{1-\alpha \left(t_k\right)}\end{array}$$

Since $\left|t_{k+1}-t_k\right|<\left|t_{k+1}-\tilde{t}\right|+\left|\tilde{t}-t_k\right|<2\epsilon$, then ${\left(2\epsilon \right)}^{\alpha \left(t_k\right)}>{\left|t_{k+1}-t_k\right|}^{\alpha \left(t_k\right)}$, it finally yields:

$$\begin{array}{c}\left|t_{k+2}-t_{k+1}\right|>{\left(2\epsilon \right)}^{\alpha \left(t_k\right)}{\left|t_{k+1}-t_k\right|}^{1-\alpha \left(t_k\right)}\\ >{\left|t_{k+1}-t_k\right|}^{\alpha \left(t_k\right)}{\left|t_{k+1}-t_k\right|}^{1-\alpha \left(t_k\right)}\\ >\left|t_{k+1}-t_k\right|\end{array}$$

In this way, we obtain the conclusion $\left|t_{k+2}-t_{k+1}\right|>\left|t_{k+1}-t_k\right|$, which contradicts the above premise so that the proposed algorithm is convergent and converges to the only extreme point. In addition, the convergence rate of the proposed method can still be estimated by Equation (17). That is to say, the maximum convergence rate of the proposed method is also the same as the classical gradient method.

Remark 4.

The proposed method provides an additional auxiliary iterative function ${\mu }_k={\left(\left|t_{k+1}-t_k\right|+\delta \right)}^{1-\alpha \left(t_k\right)}$ to adjust the iteration step automatically according to the difference between adjacent iterations $t_{k+1}$ and $t_k$. Concretely, it can provide a larger iteration step size as there is a large difference between $t_{k+1}$ and $t_k$. As the iteration converges, the value of $t_{k+1}$ and $t_k$ are very close to each other; then, the step size of the proposed method will reduce to a fixed value $\rho$ as the auxiliary iterative function will tend to 1.

Remark 5.

By leveraging this characteristic, the suggested algorithm offers a swift convergence rate during the initial stage of iteration. Additionally, it can attain a high convergence accuracy as the iteration progresses when a suitably small value is assigned to $\rho$. In other words, the proposed method effectively addresses a fundamental limitation of the conventional gradient method, which fails to strike a balance between efficiency and precision. Moreover, the inclusion of a variable fractional-order operator augments the algorithm’s ability to achieve optimal convergence rates throughout the optimization process.

4.3. Incorporating the Armijo Criterion

To further emphasize the advantages of the proposed method, we introduce the Armijo Criterion, which is a mathematical condition used in optimization algorithms to determine an appropriate step size along a search direction. This ensures that each step taken during the optimization process results in a sufficient reduction in the objective function value. Algorithm 2 shows the specific flow of the algorithm.

Algorithm 2: Armijo Criterion

(1)  Input of current point $t_k.$ Search direction $d_k.$ Objective function $f\left(t\right).$ Current gradient $g_k$. Armijo constant $\beta$ (usually a value between 0 and 1). A positive constant $\sigma$ (typically between 0 and 0.5). (2)  If the inequality $f\left(t_k+\beta d_k\right)\le f\left(t_k\right)+\sigma \beta g_k^T{d}_k$ is satisfied, let the iteration step size $\rho =\beta$ and stop algorithm. If not, go to Step 3. (3)  Let $\beta =\beta /2$; turn to Step 2.

Remark 6.

By incorporating the Armijo Criterion into the TVFOGM, several significant enhancements can be achieved. Firstly, the convergence speed of the algorithm can be elevated to approach its theoretical maximum, thereby optimizing its efficiency. Additionally, the inclusion of the Armijo Criterion bolsters the stability of the algorithm, ensuring its robust performance across diverse functions. Most importantly, it efficiently fortifies the algorithm’s convergence abilities, enabling it to effectively tackle a wide array of functions.

5. Numerical Experiments

In this section, we validate the convergence performance of the proposed TVFOGM through several numerical experiments. To do this, we compare it with the classical gradient method (GM), the previous three types of FOGM, and FOGM in [21]. The optimal fractional order $\alpha$ of previous FOGMs is set as 1.5.

Experiment 1.

Firstly, we consider a simple quadratic function $f\left(t\right)={\left(t-5\right)}^2$. Obviously, the extreme point of this function is $\widehat{t}=5$. Parameter  $\rho$ is the iteration step size and $\epsilon$ is the convergence precision. In this case, the additional auxiliary parameter $v$ is sete asa fixed value of $v=\rho$, and parameter $u$ is set to be −1.5. The termination condition is $\left|\mathit{𝒟}f\left(t_k\right)\right|<\epsilon$ or the maximal iteration number equals 50,000. At this point, we consider the method to be invalid. Take the initial point $t_0 =$ 100;

Table 1. Convergence numbers of each method with different precisions $\epsilon$ and iteration step sizes $\rho$.

Iteration Step Size	Convergence Precision	GM	FOGM-Type-1	FOGM-Type-2	FOGM-Type-3	FOGM [21]	TVFOGM
$\rho =$	$\epsilon ={10}^{-3}$	45	50,000	443	50,000	36	21
$0.1$	$\epsilon ={10}^{-5}$	65	50,000	665	50,000	50,000	41
	$\epsilon ={10}^{-7}$	86	50,000	887	50,000	50,000	62
$\rho =$	$\epsilon ={10}^{-3}$	488	523	4497	50,000	178	30
$0.01$	$\epsilon ={10}^{-5}$	716	50,000	6739	50,000	183	221
	$\epsilon ={10}^{-7}$	944	50,000	8981	50,000	50,000	449

shows the convergence number of various algorithms converging to the same precision. Compared with classical GM, the previous three types of FOGM do not improve the convergence ability. However, the FOGM in [21] can reduce the number of convergences, but cannot achieve a higher convergence precision. In this case, the proposed TVFOGM can balance the rapid convergence rate and high convergence precision. To explore this characteristic more clearly, the iteration process of these methods is shown in Figure 1.

Figure 1 shows the convergence rate and precision of the proposed TVFOGM, previous FOGM, and classical GM, respectively. The previous FOGM can converge rapidly to the extreme point, but the convergence precision is limited. For the same $\rho$, the proposed TVFOGM has a larger convergence rate than other methods in the initial stage, and then reduces to the same convergence rate as the classical GM after convergence to a certain precision. This reflects that the proposed method has the characteristic of a variable step size, and the precision is at least two orders of magnitude higher than that of the classical GM. For the selection of parameter $u$, we adopted three different values, and the performance of the TVFOGM did not change greatly. The robustness of the TVFOGM will be further studied in the following calculation.

Figure 2 shows the required number of iteration steps that converge to the same precision (10⁻²) for the classical GM and proposed TVFOGM, respectively. The convergence rate of classical GM is very sensitive to the step size $\rho$ because its number of iterations strongly depends on the iteration step size, as shown in Figure 1A. However, the number of iteration steps for the proposed TVFOGM almost remains at the same small magnitude as step size $\rho$ varies. This means that the proposed TVFOGM is more robust and efficient.

Figure 3 further shows the effect of the$u$ on the convergence of the proposed TVFOGM. Subjecting this to the different ρ values shows that there seems to be a minimum number of iterations for the proposed TVFOGM, e.g., about $u=-1.5$. The difference between the minimum values of the four cases is very small. In addition, the influence of auxiliary parameter $u$ on the number of iterations of the TVFOGM is almost negligible. This indicates that the selection of parameter $u$ has a larger degree of freedom.

Through the above discussions, we verified that the proposed method can achieve both a high convergence rate and precision. In addition, the method is not sensitive to the step size $\rho$, so it is recommended to choose a smaller step size $\rho$ to guarantee higher convergence precision.

Experiment 2.

Here, we further consider a more complex objective function, given as $f\left(t\right)=\frac{1}{t}+lnt$ with extreme point is $\widehat{t}=1$. The characteristic of this objective function is that the gradient of the function is very small when t is given a large value. Therefore, if the initial value of the iteration is given as a large value, the iteration rate will be very slow in the initial stage.

In order to improve the convergence rate, a larger iteration step size $\rho$ should be selected for the classical gradient method. As shown in Figure 4A, when a larger iteration step size $\rho =$2 (green line) is chosen, although there is a rapid convergence rate in the initial stage, the final convergence precision is very low. When choosing a smaller iteration step size, the classical GM can achieve a rapid convergence precision size, but its convergence rate will be reduced. However, the proposed TVFOGM can achieve both rapid convergence rate and high convergence precision due to it essentially being a variable step-size method, as shown by the black line in Figure 4A. We also discuss the selection of auxiliary parameters $v$ and $u$. As shown in Figure 4B, the two additional auxiliary parameters have little effect on the convergence rate of the proposed method so that the parameters can be selected to be approximately the same as the previous Simulation I, such as $v=1$ and $u=$−1.5. Given the same $\rho$ and number of iterations, one can observe that the proposed TVFOGM can quickly converge to the extreme point, whereas the classical GM and FOGM only iterate forward a short distance in Figure 5.

Experiment 3.

Here, we consider a two-dimensional objective function $f\left(x,y\right)=2{\left(x-1\right)}^2+{\left(y-5\right)}^2$. Obviously, the target point here is at point $\left(1, 5\right)$. In this simulation, the iteration start point of all algorithms is set to the same point $\left(20, 20\right)$, the iteration step size $\rho$ is set to a small value given as $0.01$, the parameter $v$ is set to be equal to $\rho$, the parameter $u$ is also selected as $u=-1.5$ and the number of times that the iteration is terminated is set to 20. Figure 6 shows the iteration processes of GM, FOGM, and TVFOGM. It can be seen that the proposed TVFOGM can also rapidly converge to the target point. More importantly, it can also reach a high convergence precision, as listed in

Table 2. Convergence errors of GM, FOGM, and TVFOGM at iteration 20, respectively.

GM	FOGM	TVFOGM
257.4713	166.2380	0.0047

.

Experiment 4.

As a classical technique for chaos suppression, the time-delayed feedback controlling strategy has been widely developed by stabilizing unstable periodic orbits (UPOs) embedded in chaotic systems. A critical issue for achieving a high level of controlling precision is to search for an appropriate time delay. This section utilizes the proposed TVFOGM to determine the optimal time delay in the delayed feedback controller. The time delay is adjusted within the iterative scheme provided by the proposed method, and finally converges to the period of the target UPO. Also consider the paradigmatic Rössler system subjected to time-delayed feedback.

$$\begin{array}{ll}\hfill & \dot{x}=-y-z\hfill \\ \hfill & \dot{y}=x+ay+u\left(t\right)\hfill \\ \hfill & \dot{z}=b+z\left(x-c\right)\hfill \end{array}$$

(26)

Given the initial value $x\left(0\right)=1$, $y\left(0\right)=2$, $z\left(0\right)=1$, and parameters $a=0.2$, $b=0.2$, $c=5.7$, the uncontrolled system (i.e., u(t) = 0) will exhibit chaotic behavior, as shown in Figure 7A. Take the control function $u\left(t\right)=\eta \left[y\left(t-\tau \right)-y\left(t\right)\right]$. The results shown in Figure 7B,C are the periodic solutions of the controlled system with the time delay $\tau$ as either 5 or 6, both with the feedback gain of $\eta =0.2$.

It should be pointed out that the attained periodic solutions are not necessarily the UPO embedded in the uncontrolled system. Next, we will use the TVFOGM to obtain the optimal stable solution $\tau$ of the system. Take the target function (27) for the system:

$${\varphi }_T\left(\tau \right)={{\displaystyle \sum }}_{s=0}^S{\left[e\left(t_{\mathrm{s}}\right)\right]}^2={{\displaystyle \sum }}_{s=0}^S{\left[y\left(t_{\mathrm{s}}-\tau \right)-y\left(t_{\mathrm{s}}\right)\right]}^2$$

(27)

where $t_s$s are selected as timepoints that are uniformly distributed over a fixed interval [0, T]. By applying the proposed method to the above objective function, one can obtain:

$${\tau }_{k+2}={\tau }_{k+1}-\rho {\left(\left|{\tau }_{k+1}-{\tau }_k\right|+\delta \right)}^{1-\alpha \left({\tau }_k\right)}\frac{\partial {\varphi }_T\left({\tau }_{k+1}\right)}{\partial \tau }$$

where the gradient of ${\varphi }_T\left(\tau \right)$ with respect to $\tau$ is obtained using numerical difference such as $\frac{\partial {\varphi }_T\left({\tau }_{k+1}\right)}{\partial \tau }=\frac{{\varphi }_T\left({\tau }_{k+1}+\mathsf{\Delta }\tau \right)-{\varphi }_T\left({\tau }_{k+1}\right)}{\mathsf{\Delta }\tau }$ with $\mathsf{\Delta }\tau$ as a small quantity. In this case, we need to adjust the parameter $v$ slightly to ensure the convergence of the proposed algorithm as the objective function (27) is not a strictly convex function. For instance, the parameter selection of $v$ is set to $v=0.1$ for the identification of period-1 and period-2 orbits, while for period-3 orbits, it will also be selected as $v=\rho$. The selection of other parameters is the same as the above numerical simulation.

As shown in Figure 8, the proposed method can also quickly identify the periods of unstable period-1, period-2, and period-3 orbits embedded in a chaotic system (26), while the classical GM and FOGM are very inefficient. This shows that the proposed TVFOGM has better adaptability than the classical GM and FOGM. By using the obtained optimal time delays, we successfully achieved the stabilization of period-1, period-2, and period-3 unstable periodic orbits, as shown in Figure 9B–D. Figure 10 shows the corresponding control errors, which verify the precision of the obtained optimal time delays.

Table 3. Convergence series for the time delay provided by TVFOGM, GM, and FOGM.

Methods:	TVFOGM	GM	FOGM
1 iteration	8.000	8.000	8.000
2 iterations	7.868	7.868	7.868
3 iterations	7.330	7.738	7.511
4 iterations	6.994	7.573	7.251
5 iterations	6.521	7.399	6.944
6 iterations	6.277	7.229	6.704
14 iterations	5.890	6.282	5.890
15 iterations	5.890	6.213	5.885
36 iterations		5.890	5.895
37 iterations		5.890	5.883
38 iterations			5.895

and

Table 4. Convergence time of TVFOGM, GM, and FOGM, respectively.

TVFOGM	GM	FOGM
9.40 s	22.54 s	NAN

compare the number of iteration steps and iteration time of the three gradient methods, from which it can be seen that the proposed method has a higher convergence efficiency.

Through this example, the proposed method is also shown to efficiently handle non-convex optimization problems. This is because, in this example, only one parameter $v$ needs to be adjusted, and the selection of other parameters remains unchanged from the previous example.

Experiment 5.

In the preceding four simulations, we examined the performance of the algorithms under a fixed iteration step size. The outcomes indicate that through the regulation of this fixed step size, TVFOGM not only enhances convergence accuracy but also significantly improves convergence speed, rendering it an effective optimization technique. In Section 3, we theoretically prove that the theoretical maximum convergence rate of the classical gradient method is equivalent to that of the fractional-type methods. To evaluate the performance of these methods under optimal step-size conditions, we introduce the Armijo criterion shown in Section 4.3 in this simulation.

The Armijo criterion ensures that each iteration step of the algorithm converges toward the current optimal choice. Furthermore, we will employ 10 benchmark test functions [41,42] which are characterized as having a valley shape, bowl shape, plate shape and steep ridges to facilitate a comprehensive comparative analysis of the performance of these methods. The characteristics of these 10 benchmark test functions are documented in

Table 5. The 10 benchmark functions and their features.

No.	Function Name	Expression	Feature	$\mathbf{Extreme} \mathbf{Point} \widehat{\mathit{t}}$	$\mathbf{Min} \mathit{f}\left(\widehat{\mathit{t}}\right)$
1	Sphere Function	$f\left(t\right)={\displaystyle {\displaystyle \sum }_{i=1}^3}{t}_i^2$	Bowl-shaped	(0, 0, 0)	0
2	Schwefel’s Problem 2.22	$f\left(t\right)={\displaystyle {\displaystyle \sum }_{i=1}^3}\left|t_i\right|+{\displaystyle {\displaystyle \prod }_{i=1}^3}\left|t_i\right|$	Smooth, multi-peaked	(0, 0, 0)	0
3	Schwefel’s Problem 1.2	$f\left(t\right)={\displaystyle {\displaystyle \sum }_{i=1}^3}{\left({\displaystyle {\displaystyle \sum }_{j=1}^i}{t}_j\right)}^2$	Bowl-shaped	(0, 0, 0)	0
4	Schwefel’s Problem 2.21	$f\left(t\right)=\underset{i}{max}\left\{\left|t_i\right|,1\le i\le 3\right\}$	Valley-shaped	(0, 0, 0)	0
5	Rosenbrock’s Function	$f\left(t\right)={\displaystyle {\displaystyle \sum }_{i=1}^3}\left[100{\left(t_{i+1}-t_i^2\right)}^2+{\left(t_i-1\right)}^2\right]$	Valley-shaped	(1, 1, 1)	0
6	Sphere Function	$f\left(t\right)={\displaystyle {\displaystyle \sum }_{i=1}^3}{\left(\left|t_i+2\right|\right)}^2$	Bowl-shaped	(−2, −2, −2)	0
7	Booth	$f\left(t\right)={\left(t_1-2t_2-7\right)}^2+{\left(2t_1+t_2-5\right)}^2$	Plate-shaped,	(3.4, −1.8)	0
8	Goldstein-Price Function	$f\left(t\right)$	several local minima	(0, −1)	3
9	Hartman’s Function	$f\left(t\right)=-{\displaystyle {\displaystyle \sum }_{i=1}^4}{c}_i\exp \left[-{\displaystyle {\displaystyle \sum }_{j=1}^3}{a}_{ij}{\left(t_j-p_{ij}\right)}^2\right]$	Steep ridges/drops	(0.114, 0.556, 0.852)	−3.8628
10	McCormick Function	$f\left(t\right)=\sin \left(t_1+t_2\right)+{\left(t_1-t_2\right)}^2-1.5t_1+2.5t_2+1$	Plate-shaped	(−0.547, −1.547)	−1.9132

Notes: 1. The expression of function 8 is $\begin{array}{c}f\left(t\right)=\left[1+{\left(t_1+t_2+1\right)}^2\left(19-14t_1+3t_1^2-14t_2+6t_1{t}_2+3t_2^2\right)\right]\\ \times \left[30+{\left(2t_1-3t_2\right)}^2\left(18-32t_1+12t_1^2+48t_2-36t_1{t}_2+27t_2^2\right)\right]\end{array}$. 2. In the expression of function 9,$c_i=\left(1, 1.2, 3, 3.2\right)$, $a_{ij}=\left(3, 10, 30;0.1, 10, 35;3, 10, 30, 0.1, 10, 35\right)$,$p_{ij}=\left(0.3689, 0.1170, 0.2673;0.4699, 0.4387, 0.7470;0.1091, 0.8732, 0.5547;0.03815, 0.5743, 0.8828\right)$.

.

The convergence performance of the three methods is shown in

Table 6. Numerical results of TVFOGM on 10 benchmark functions.

No.	Initial Point	Method	Iteration Number	${\mathit{t}}_{\mathit{k}}$	$\mathit{f}\left({\mathit{t}}_{\mathit{k}}\right)$
1	(2, 2, 2)	GM	20,000	(0, 0, 0)	1.2741 × 10−5
		FOGM (α = 0.8)	8	(0, 0, 0)	3.0123 × 10−8
		FOGM (α = 1.2)	7	(0, 0, 0)	2.0317 × 10−7
		TVFOGM	6	(0, 0, 0)	1.6982 × 10−15
2	(2, 2, 2)	GM	27	(0, 0, 0)	5.1533 × 10−17
		FOGM (α = 0.8)	60	(0, 0, 0)	7.4117 × 10−17
		FOGM (α = 1.2)	45	(0, 0, 0)	1.5587 × 10−16
		TVFOGM	22	(0, 0, 0)	6.9001 × 10−17
3	(1, 1, 1)	GM	36	(0, 0, 0)	2.3833 × 10−7
		FOGM (α = 0.8)	44	(0, 0, 0)	7.3228 × 10−7
		FOGM (α = 1.2)	45	(0, 0, 0)	1.8001 × 10−9
		TVFOGM	31	(0, 0, 0)	2.8866 × 10−7
4	(10, 10, 10)	GM	43	(0, 0, 0)	1.2761 × 10−17
		FOGM (α = 0.8)	46	(0, 0, 0)	2.9384 × 10−17
		FOGM (α = 1.2)	75	(0, 0, 0)	4.4939 × 10−17
		TVFOGM	40	(0, 0, 0)	5.2108 × 10−17
5	(−1, 5, −2)	GM	421	(1, 1, 1)	8.7176 × 10−7
		FOGM (α = 0.8)	668	(1, 1, 1)	7.8917 × 10−7
		FOGM (α = 1.2)	609	(1, 1, 1)	2.3693 × 10−9
		TVFOGM	419	(1, 1, 1)	9.3046 × 10−7
6	(10, 5, 1)	GM	3465	(−2, −2, −2)	2.5029 × 10−7
		FOGM (α = 0.8)	15	(−2, −2, −2)	3.1444 × 10−8
		FOGM (α = 1.2)	11	(−2, −2, −2)	1.8627 × 10−9
		TVFOGM	6	(−2, −2, −2)	1.8650 × 10−9
7	(2, 2)	GM	8	(3.4, −1.8)	1.6006 × 10−8
		FOGM (α = 0.8)	12	(3.4, −1.8)	6.9082 × 10−10
		FOGM (α = 1.2)	10	(3.4, −1.8)	1.2007 × 10−8
		TVFOGM	9	(3.4, −1.8)	1.8144 × 10−8
8	(5, 10)	GM	28	(−0.6, −0.4)	30.0000
		FOGM (α = 0.8)	20,000	(1.8, 0.2)	84.0104
		FOGM (α = 1.2)	20,000	(1.8, 0.2)	84.0003
		TVFOGM	118	(0, −1)	3.0000
9	(0.5, 0.5, 0.5)	GM	320	(0.114, 0.556, 0.852)	−3.8628
		FOGM (α = 0.8)	303	(0.114, 0.556, 0.852)	−3.8628
		FOGM (α = 1.2)	230	(0.114, 0.556, 0.852)	−3.8628
		TVFOGM	285	(0.114, 0.556, 0.852)	−3.8628
10	(0.1, 0.3)	GM	22	(−0.547, −1.547)	−1.9132
		FOGM (α = 0.8)	12	(−0.547, −1.547)	−1.9132
		FOGM (α = 1.2)	13	(−0.548, −1.548)	−1.9132
		TVFOGM	9	(−0.547, −1.547)	−1.9132

. Iteration number $k$ indicates the number of iterations used by the method to converge to the extreme point $\widehat{t}$, and its limit is set at 20,000. When the number of iterations $k$ of the method exceeds 20,000, we consider this method to be invalid for this problem. $t_0$ is the initial point of the method iteration, $t_k$ is the final convergence point, and $f\left(t_k\right)$ is the final convergence function value. According to [25] the parameter α of FOGM is selected as α = 0.8 and α = 1.2. The parameter $u$ of TVFOGM is also selected as $u=-1.5$. As the iteration step size $\rho$ of each iteration is constantly changing, the parameter $v$ is set as 1 for convenience.

As depicted in Table 5, the incorporation of the Armijo criterion leads to significant results for the three methods, allowing them to achieve the fastest convergence rates in certain cases, such as functions 4, 7, and 9. However, when testing functions 2, 3, and 5, it becomes evident that the FOGM lags behind GM in terms of performance, whereas TVFOGM exhibits a slight advantage over GM. On the other hand, when testing functions 1, 6, and 10, it becomes apparent that the convergence ability of GM substantially declines. In contrast, the TVFOGM method demonstrates the ability to simultaneously achieve rapid convergence and increased convergence accuracy in these specific scenarios. While TVFOGM effectively addresses the limitation that classical GM faces in achieving both convergence speed and accuracy, an observation can be made regarding function 8. In scenarios characterized by multiple extreme points, it is clear that gradient-based methods struggle to converge to the global optimal point, resulting in random final convergence points. Therefore, our future efforts will focus on devising strategies to overcome this challenge and ensure that the algorithm converges to the global optimal point, which represents a critical area for future improvements.

6. Conclusions

In this paper, we conducted an analysis of existing fractional-type gradient methods in three categories. Through convergence analysis, we discovered that the maximum convergence rate of fractional-type gradient methods is the same as that of classical gradient methods. This indicates that the advantage of fractional gradients may not solely lie in their faster convergence rates compared to classical gradient methods. This realization motivated us to reexamine the benefits of fractional gradient methods.

Through theoretical analysis, it is evident that fractional-type gradient methods are essentially variable-step-size methods, offering an effective means to dynamically adjust the step size during optimization. Based on this insight, we propose a novel variable fractional gradient method that focuses on automatic step size adjustment. By doing so, this method overcomes the limitations of classical gradient methods in balancing rapid convergence rates with high convergence precision.

Theoretical analysis demonstrates the convergence of the proposed variable fractional gradient method. Numerical simulations are provided to illustrate the effectiveness of the proposed approach. It is evident that the method achieves rapid and accurate convergence to the extreme point. Moreover, the numerical analysis reveals that the proposed method exhibits insensitivity to additional auxiliary parameters. Thus, for most optimization problems, the auxiliary parameters u can be chosen as −1.5, while the selection of v should be appropriately adjusted based on the specific optimization problem. It is worth emphasizing that selecting a small value for the parameter ρ is recommended to achieve a high convergence precision.

Additionally, we conducted tests on the three methods using 10 benchmark test functions with the inclusion of the Armijo criterion. The findings demonstrate that, despite having the same theoretical maximum convergence speed, TVFOGM consistently displays superior convergence stability and performance when applied to real-world problems.

A critical focus for future enhancements is to ensure the method’s convergence towards the global optimal point in scenarios involving multiple extreme points. Additionally, it is essential to delve into the challenging task of stability analysis for the proposed TVFOGM and explore its application to more intricate optimization problems like neural networks and parameter identification. These advancements would broaden the method’s scope and effectiveness in practical applications. As a result, our proposed TVFOGM should be further extended to RNN-based and CNN-based networks.