The Improved Stochastic Fractional Order Gradient Descent Algorithm

Abstract

This paper mainly proposes some improved stochastic gradient descent (SGD) algorithms with a fractional order gradient for the online optimization problem. For three scenarios, including standard learning rate, adaptive gradient learning rate, and momentum learning rate, three new SGD algorithms are designed combining a fractional order gradient and it is shown that the corresponding regret functions are convergent at a sub-linear rate. Then we discuss the impact of the fractional order on the convergence and monotonicity and prove that the better performance can be obtained by adjusting the order of the fractional gradient. Finally, several practical examples are given to verify the superiority and validity of the proposed algorithm.

1. Introduction

In the field of machine learning, the gradient descent algorithm is one of the most fundamental methods for optimization problems [1,2]. Due to the continuous expansion of the scale of data, the traditional gradient descent algorithms cannot be used effectively to solve the optimization problems in large-scale machine learning [3,4,5]. To deal with this situation, the SGD algorithm was introduced, which is an online optimization algorithm and can be used to reduce the computational complexity by selecting one or several sample gradients randomly to replace the overall gradients in the iteration process [6]. Recently, to improve performance, the traditional SGD algorithm was modified from the different aspects, such as the adaptive gradient algorithm (Adagrad) [7], the Adadelta method [8] and adaptive moment estimation (Adam) [9], where the objective functions were assumed to be convex or strongly convex, which might be not satisfied in applications. Hence, the nonconvex learning algorithm was introduced [10].

In some practical problems, the bounds of the regret need to be minimized as far as possible. Hence, a fractional order gradient was introduced due to its good properties, such as long memory and multiple parameters [11,12]. However, the fractional order algorithm might fail to find the optimum [13]. Hence, some improved fractional order gradient algorithms were proposed by the technology of the truncation of high order terms [14,15,16]. By using a variable initial value strategy and truncation on high order terms, the fractional order gradient descent method was transformed into an adaptive learning rate method, the learning rate of which corresponds to the power of the previous terms. Its long memory property was transformed into a short memory property that can be tolerated. By combining the SGD algorithm and fractional order, some composite algorithms were introduced in machine learning [17,18,19,20,21]. However, the literature above only applied new algorithms with a fractional order gradient to applications without proving its convergence or proving its convergence on a rather specific algorithm, like an RBF neural network; to the best of our knowledge, there are few theoretical works on general fractional online SGD algorithms.

In this paper, we mainly propose three new SGD algorithms with a fractional order gradient for the online optimization problem, under which the bounds of the regret can be lowered by adjusting the fractional order. The main contributions of this paper include the following three aspects.

(1) Three improved stochastic fractional order gradient algorithms are proposed for the online optimization problem combing fractional gradients.

(2) Compared with the results in [20,21], where only simulation results were expressed, this paper proves the convergence of the proposed algorithms in mathematics. In addition, it is shown that the fractional order gradient can relieve the gradient exploding phenomenon, which often occurs in deep learning.

(3) The proposed algorithms are applied to the parameter identification problem and the classification problem to examine the effectiveness of the proposed algorithm.

Notation 1.

Let ${\mathbf{R}}^n$ be the Euclidean space with dimension n. N is the natural number of a set, given $A\in {\mathbf{R}}^{n\times m}$ and $x\in {\mathbf{R}}^n$, $\parallel A\parallel$ and $\parallel x\parallel$ are the 2-norm, and $\left|x\right|$ is the absolute value vector of each component of x. $\nabla f(x)$ denotes the gradient of $f(x)$.

2. Materials and Methods

Many researchers have given different definitions of a fractional derivate from different aspects; the Caputo type is the most common in applications due to its integer order initial values condition. Additionally, the Riemann—Liouville type also works in some situations [14]. For a smooth function $f(t)$, the Caputo derivate is defined by

$${}_{t_0}^C{\mathcal{D}}_t^{\alpha }f(t)=\frac{1}{\Gamma (n-\alpha )}{\int }_{t_0}^t\frac{f^{(n)}(\tau )}{{(t-\tau )}^{\alpha -n+1}}d\tau ,$$

(1)

where $n-1<\alpha <n,n\in \mathcal{N}$, and Formula (1) can be rewritten in Taylor series as follows:

$${}_{t_0}^C{\mathcal{D}}_t^{\alpha }f(t)=\sum _{i=1}^{\infty }\frac{f^{(i)}(t_0)}{\Gamma (i+1-\alpha )}{(t-t_0)}^{i-\alpha },0<\alpha <1.$$

(2)

Let $f(\theta )$ be a convex objective function, and the traditional gradient descent algorithm is

$${\theta }_{t+1}={\theta }_t-{\mu }_t{\mathbf{g}}_t,$$

(3)

where ${\mathbf{g}}_t=\nabla f({\theta }_t)$ and ${\mu }_t$ is the learning rate. Substituting the gradient by the fractional order gradient in (3), we get the following iteration algorithm:

$${\theta }_{t+1}={\theta }_t-{\mu }_t{}_{{\theta }_0}^C{\mathcal{D}}_{{\theta }_t}^{\alpha }f({\theta }_t).$$

(4)

To eliminate the effect of the global character of a fractional differential operator, we choose the following variable initial value strategy, i.e.,

$${\theta }_{t+2}={\theta }_{t+1}-{\mu }_t{}_{{\theta }_t}^C{\mathcal{D}}_{{\theta }_{t+1}}^{\alpha }f({\theta }_t).$$

(5)

Adopting the Tayor series of a fractional order differential, we have

$${\theta }_{t+2}={\theta }_{t+1}-{\mu }_t\sum _{i=0}^{\infty }\frac{f^{(i+1)}({\theta }_t)}{\Gamma (i+2-\alpha )}{({\theta }_{t+1}-{\theta }_t)}^{i+1-\alpha },0<\alpha <1.$$

(6)

By Reserving the first item and omitting the other items in the summation, we have

$${\theta }_{t+2}={\theta }_{t+1}-{\mu }_t\frac{\nabla f({\theta }_{t+1})}{\Gamma (2-\alpha )}{({\theta }_{t+1}-{\theta }_t)}^{1-\alpha }.$$

(7)

In fact, we can extend algorithm (7) to the case of $1<\alpha <2$ directly. In the following, we assume $0<\alpha <2$. To guarantee the feasibility of ${({\theta }_{t+1}-{\theta }_t)}^{1-\alpha }$, we modify (7) into the following form:

$${\theta }_{t+2}={\theta }_{t+1}-{\mu }_t\frac{\nabla f({\theta }_{t+1})}{\Gamma (2-\alpha )}{|{\theta }_{t+1}-{\theta }_t|}^{1-\alpha }.$$

(8)

Remark 1.

Notice the properties of the gamma function—singularities would occur when the orders of the fractional gradient are taken as other integers $\alpha =3,4,\cdots$. And the gamma function displays a poor property when the order is closed to its singularities.

For the SGD algorithm, each batch of samples has its own objective function $f_t(\theta )$. The task of this paper is to propose the SGD algorithm with a fractional order and analyze their convergence.

Let

$$R(T)=\sum _{t=1}^T{f}_t({\theta }_t)-mi{n}_{\theta }\sum _{t=1}^T{f}_t(\theta ),$$

where $R(T)$ is the empirical regret function of the optimization problem and $f_t(\theta )$ is the objective sub-function of one batch of samples at present. If $R(T)/T\to 0$ as $T\to \infty$, then we call that algorithm convergent. Next, we will analyze the asymptotic property of $R(T)$,

$$\begin{array}{cc}\hfill R(T)& =\sum _{t=1}^T{f}_t({\theta }_t)-mi{n}_{\theta }\sum _{t=1}^T{f}_t({\theta }_t)\hfill \\ \hfill & =\sum _{t=1}^T{f}_t({\theta }_t)-\sum _{t=1}^T{f}_t({\theta }^{*})\hfill \\ \hfill & =\sum _{t=1}^T[f_t({\theta }_t)-f_t({\theta }^{*})],\hfill \end{array}$$

(9)

where ${\theta }^{*}\triangleq argmi{n}_{\theta }{\sum }_{t=1}^T{f}_t(\theta )$. We need the following assumptions on parameter space and objective functions in the following analysis [22].

Assumption 1.

Suppose all $f_i(\theta )$ are convex. By using the property of the convex function, we conclude that $f(w)-f(v)\le \langle \mathbf{g},w-v\rangle$, where $\mathbf{g}=\nabla f(v)$.

Assumption 2.

Suppose that parameters ${\theta }^{*}\in {\theta }^d$ and $\parallel {\theta }_i-{\theta }_j\parallel \le D,\forall {\theta }_i,{\theta }_j\in {\theta }^n$, where ${\theta }^d$ is a bounded and closed set and $D>0$ is a positive scalar.

Assumption 3.

Suppose the gradient of the objective function is bounded, i.e., $\parallel {\mathbf{g}}_{\mathbf{t}}\parallel \le G,\forall t$, where ${\mathbf{g}}_{\mathbf{t}}$ is the gradient of function $f_t$ in the tth batch of data.

Remark 2.

In Assumption 1, we assume all objective functions $f_t(\theta )$ are convex. Hence, there must exist a global optimal parameter for the objective function. Assumption 2 implies that the Euclidean distance of any two temporary parameters would not be too long. Most of the results on online optimization require this assumption, because the optimal points of different objective functions can be far apart. Assumption 3 guarantees that the gradient of the objective function is bounded, which is very important for minimizing the empirical regret function [23]. When parameters between iterations are bounded, it can be proved that the gradient is also bounded since the objective function cannot vary too severely during a bounded interval. An example is Gaussian kernel function $f(x)=exp(-x^2)$. In addition, in machine learning, Assumption 3 can also be satisfied by clipping the gradient [24].

3. Main Results

In this section, we will analyze the convergence of the proposed three online algorithms with fractional order $\alpha$.

3.1. Standard SGD with Fractional Order Gradient

$${\theta }_{t+2}={\theta }_{t+1}-{\mu }_t\frac{\nabla f({\theta }_{t+1})}{\Gamma (2-\alpha )}(|{\theta }_{t+1}-{\theta }_t{|+\delta )}^{1-\alpha }.$$

(10)

To avoid the phenomena that the before and after iteration values are the same, we add a small positive parameter $\delta$ to the format of the algorithm to improve the performance.

Theorem 1.

Under Assumptions 1–3, Algorithm 1 is convergent, and the bound of$R(T)$can be adjusted by fractional order$\alpha$.

Proof. 

By Formula (10), we have

$$\begin{array}{cc}\hfill {\theta }_{t+1}-{\theta }^{*}& ={\theta }_t-{\theta }^{*}-{\mu }_t{\nabla }^{\alpha }{f}_t({\theta }_t)\hfill \\ \hfill \Rightarrow {\theta }_{t+1}-{\theta }^{*}& ={\theta }_t-{\theta }^{*}-\frac{{\mu }_t}{\Gamma (2-\alpha )}{g}_t(|{\theta }_t-{\theta }_{t-1}{|+\delta )}^{1-\alpha }\hfill \\ \hfill \Rightarrow \parallel {\theta }_{t+1}-{\theta }^{*}{\parallel }^2& =\parallel {\theta }_t-{\theta }^{*}-\frac{{\mu }_t}{\Gamma (2-\alpha )}{g}_t(|{\theta }_t-{\theta }_{t-1}{|+\delta )}^{1-\alpha }{\parallel }^2.\hfill \end{array}$$

Then,

$$\langle g_t,{\theta }_t-{\theta }^{*}\rangle =\frac{\Gamma (2-\alpha )}{2{\mu }_t}\frac{\parallel {\theta }_{t+1}-{\theta }^{*}{\parallel }^2-{\parallel {\theta }_t-{\theta }^{*}\parallel }^2}{\parallel |{\theta }_t-{\theta }_{t-1}{|+\delta \parallel }^{1-\alpha }}+\frac{{\mu }_t}{2\Gamma (2-\alpha )}\parallel g_t{\parallel }^2\parallel |{\theta }_t-{\theta }_{t-1}{|+\delta \parallel }^{1-\alpha }.$$

(11)

By applying the property of convex objective function $f_t({\theta }_t)-f_t({\theta }^{*})\le \langle g_t,{\theta }^{*}-{\theta }_t\rangle$ and the definition of R(T), we have $R(T)\le {\sum }_{t=1}^T\langle g_t,{\theta }^{*}-{\theta }_t\rangle$. Together with (11), we have

$$R(T)\le \sum _{t=1}^T\frac{\Gamma (2-\alpha )}{2{\mu }_t}\frac{\parallel {\theta }_{t+1}-{\theta }^{*}{\parallel }^2-{\parallel {\theta }_t-{\theta }^{*}\parallel }^2}{\parallel |{\theta }_t-{\theta }_{t-1}{|+\delta \parallel }^{1-\alpha }}+\sum _{t=1}^T\frac{{\mu }_t}{2\Gamma (2-\alpha )}\parallel g_t{\parallel }^2\parallel |{\theta }_t-{\theta }_{t-1}{|+\delta \parallel }^{1-\alpha }.$$

(12)

The first summation term of (12) can be enlarged by rearranging the order of the term as follows:

$$\begin{array}{cc}\hfill & \sum _{t=1}^T\frac{\Gamma (2-\alpha )}{2{\mu }_t}\frac{\parallel {\theta }_{t+1}-{\theta }^{*}{\parallel }^2-{\parallel {\theta }_t-{\theta }^{*}\parallel }^2}{\parallel |{\theta }_t-{\theta }_{t-1}{|+\delta \parallel }^{1-\alpha }}\hfill \\ \hfill \le & \frac{\Gamma (2-\alpha ){\delta }^{\alpha -1}}{2}[\frac{\parallel {\theta }_1-{\theta }^{*}{\parallel }^2}{{\mu }_1}+\sum _{t=2}^T(\frac{1}{{\mu }_t}-\frac{1}{{\mu }_{t-1}})\parallel {\theta }_t-{\theta }^{*}{\parallel }_2^2-\frac{\parallel {\theta }_{T+1}-{\theta }^{*}{\parallel }_2^2}{{\mu }_T}]\hfill \\ \hfill \le & \frac{\Gamma (2-\alpha ){\delta }^{\alpha -1}}{2}[\frac{1}{{\mu }_1}{D}^2+D^2\sum _{t=2}^T(\frac{1}{{\mu }_t}-\frac{1}{{\mu }_{t-1}})+0]\hfill \\ \hfill \le & \frac{\Gamma (2-\alpha )D^2{\delta }^{\alpha -1}}{2{\mu }_T}.\hfill \end{array}$$

(13)

By adopting Assumption 3, we can get the bounds of the regret function from different situations.

When fractional order $0<\alpha \le 1$:

$$\begin{array}{cc}\hfill R(T)& \le \frac{D^2{\delta }^{\alpha -1}}{2{\mu }_T}+\sum _{t=1}^T\frac{{\mu }_t}{2}\parallel g_t{\parallel }^2\parallel \parallel {\theta }_t-{\theta }_{t-1}{\parallel +\delta \parallel }^{1-\alpha }\hfill \\ \hfill & \le \frac{D^2{\delta }^{\alpha -1}\Gamma (2-\alpha )}{2{\mu }_T}+\frac{G^2{(D+\delta )}^{1-\alpha }}{2\Gamma (2-\alpha )}\sum _{t=1}^T{\mu }_t.\hfill \end{array}$$

(14)

When fractional order $1<\alpha <2$:

$$\begin{array}{cc}\hfill R(T)& \le \frac{D^2{(D+\delta )}^{\alpha -1}\Gamma (2-\alpha )}{2{\mu }_T}+\sum _{t=1}^T\frac{{\mu }_t}{2\Gamma (2-\alpha )}\parallel g_t{\parallel }^2\parallel |{\theta }_t-{\theta }_{t-1}{|+\delta \parallel }^{1-\alpha }\hfill \\ \hfill & \le \frac{D^2{(D+\delta )}^{\alpha -1}\Gamma (2-\alpha )}{2{\mu }_T}+\frac{G^2{\delta }^{1-\alpha }}{2\Gamma (2-\alpha )}\sum _{t=1}^T{\mu }_t.\hfill \end{array}$$

(15)

And the bounds of the regret function on $0<\alpha <2$ can be summarized as

$$\begin{array}{c}R(T)\le \left\{\begin{array}{cc}\frac{D^2{\delta }^{\alpha -1}\Gamma (2-\alpha )}{2{\mu }_T}+\frac{G^2{(D+\delta )}^{1-\alpha }}{2\Gamma (2-\alpha )}\sum _{t=1}^T{\mu }_t,\hfill & 0<\alpha \le 1,\hfill \\ \frac{D^2{(D+\delta )}^{\alpha -1}\Gamma (2-\alpha )}{2{\mu }_T}+\frac{G^2{\delta }^{1-\alpha }}{2\Gamma (2-\alpha )}\sum _{t=1}^T{\mu }_t,\hfill & 1<\alpha <2.\hfill \end{array}\right.\hfill \end{array}$$

(16)

Let $T\to \infty$; the convergence of Algorithm 1 is decided by $\frac{1}{{\mu }_{T}T}$ and $\frac{{\sum }_{t=1}^T{\mu }_t}{T}$. When we take the polynomial decay rate, such as ${\mu }_t=\frac{C}{t^p},p\ge 0$, especially when $p=\frac{1}{2}$, $R(T)$ obtains its optimal bound $O(\sqrt{T})$ for $\frac{1}{{\mu }_{T}T}=\frac{C}{\sqrt{T}}$ and ${\sum }_{t=1}^T\frac{C}{\sqrt{t}}\le 1+{\int }_1^T\frac{dt}{\sqrt{t}}=2\sqrt{T}-1$. Therefore, $R(T)/T=O(\frac{1}{\sqrt{T}})\to 0$; hence, Algorithm 1 is convergent.    □

Algorithm 1 SGD with fractional order.

1: Initialize $\alpha ,{\mu }_0,t_{max},t=0$ and ${\theta }_0$; 2: Calculate ${\theta }_1$ under ordinary SGD: ${\theta }_1={\theta }_0-{\mu }_0\nabla f_0({\theta }_0)$; 3: Update the parameters by ${\theta }_{t+2}={\theta }_{t+1}-{\mu }_{t+1}\frac{\nabla f({\theta }_{t+1})}{\Gamma (2-\alpha )}(|{\theta }_{t+1}-{\theta }_t{|+\delta )}^{1-\alpha }$ and ${\mu }_t=\frac{{\mu }_0}{\sqrt{t}}$; 4: Increase t to $t_{max}$ until all batches of samples have been used.

Remark 3.

Compared with the integer order algorithm, fractional term $\frac{(|{\theta }_{t+1}-{\theta }_t{|+\delta )}^{1-\alpha }}{\Gamma (2-\alpha )}$ would accelerate the convergence of algorithms. When $0<\alpha \le 1$, a larger value of fractional term $\frac{(|{\theta }_{t+1}-{\theta }_t{|+\delta )}^{1-\alpha }}{\Gamma (2-\alpha )}$ leads to an increase in learning rate when ${\theta }_{t+1}$ and ${\theta }_t$ differ significantly and the difference is close to D at the beginning of iterations; the situation is the opposite when $1<\alpha <2$. In particular, when $D<0.5$, $\frac{D^{1-\alpha }}{\Gamma (2-\alpha )}$ is an incremental function of $0<\alpha <1.5$.

Remark 4.

When learning rate ${\mu }_t={\mu }_0/t^p$, $R(T)=O(T^{max(p,1-p)})$. And if we take ${\mu }_t$ as a constant, $R(T)/T=O(1)$, the algorithm still works for other some situations in [20,25].

Remark 5.

The bound of the regret function mainly depends on the summation part of Formula (16). When we take parameters $\delta ={10}^{-3}$ and D in the algorithm of different orders, we find that the monotonicity of the regret function would vary as a fractional order. It is shown that the larger fractional order brings a smaller loss of function when $0<\alpha \le 1$, while the result is the opposite when $1<\alpha <2$ until the order is close to 2. We usually take the normalization of the training data to reduce the effect of data magnitude. Hence, the value of D used in Figure 1 is meaningful where the monotonicity of the coefficient is shown in Figure 1.

3.2. Adagrad Algorithm with Fractional Order

Notation 2.

For the Adagrad algorithm with a fractional order, we need to split parameters and variables by its dimension as ${\theta }_t={[{\theta }_{t,1},\cdots ,{\theta }_{t,d}]}^T$, $g_t={[g_{t,1},\cdots ,g_{t,d}]}^T$ and ${\mu }_t={[{\mu }_{t,1},\cdots ,{\mu }_{t,n}]}^T$.

Theorem 2.

Under Assumptions 1–3, Algorithm 2 is convergent, where · is the Hadamard product between vectors and$\mu$is a constant.

Proof. 

We split the objective function according to its dimension:

$$\sum _{t=1}^T\langle g_t,{\theta }_t-{\theta }^{*}\rangle =\sum _{t=1}^T\sum _{i=1}^d{g}_{t,i}({\theta }_{t,i}-{\theta }_i^{*})=\sum _{i=1}^d\sum _{t=1}^T{g}_{t,i}({\theta }_{t,i}-{\theta }_i^{*}).$$

(17)

We get the same form of $R(T)$ with Theorem 1 when $0<\alpha \le 1$:

$$\begin{array}{cc}\hfill R(T)& \le \sum _{i=1}^d[\sum _{t=1}^T\frac{\Gamma (2-\alpha )}{2{\mu }_{t,i}}\frac{{({\theta }_{t,i}-{\theta }_i^{*})}^2-{({\theta }_{t+1,i}-{\theta }_i^{*})}^2}{{({\theta }_{t+1,i}-{\theta }_{t,i})}^{1-\alpha }}\hfill \\ \hfill & +\sum _{t=1}^T\frac{{\mu }_{t,i}}{2\Gamma (2-\alpha )}{g}_{t,i}^2{({\theta }_{t+1,i}-{\theta }_{t,i})}^{1-\alpha }];\hfill \end{array}$$

(18)

hence,

$$\begin{array}{cc}\hfill R(T)& \le \sum _{i=1}^d[\frac{D_i^2{\delta }^{\alpha -1}}{2\mu }\sqrt{\sum _{s=1}^T{g}_{s,i}^2{\parallel {\theta }_{s,i}-{\theta }_{s-1,i}\parallel }^{2-2\alpha }}\hfill \\ \hfill & +\frac{\mu {(D_i+\delta )}^{1-\alpha }}{2}\sum _{i=1}^T\frac{g_{t,i}^2}{\sqrt{g_{s,i}^2{\parallel {\theta }_{s,i}-{\theta }_{s-1,i}\parallel }^{2-2\alpha }}}].\hfill \end{array}$$

(19)

The first term of Formula (19) can be enlarged to

$$\sqrt{\sum _{s=1}^T{g}_{s,i}^2{\parallel {\theta }_{s,i}-{\theta }_{s-1,i}\parallel }^{2-2\alpha }}\le G_i{D}_i^{1-\alpha }\sqrt{T}.$$

(20)

For the last term, we have

$$\begin{array}{cc}\hfill & \sum _{t=1}^T\frac{g_{t,i}^2(|{\theta }_{t,i}-{\theta }_{t-1,i}{|+\delta )}^{2-2\alpha }}{\sqrt{{\sum }_{s=1}^t{g}_{s,i}^2{({\theta }_{s,i}-{\theta }_{s-1,i})}^{2-2\alpha }}}\hfill \\ \hfill & =\sum _{t=1}^T\frac{2g_{t,i}^2(|{\theta }_{t,i}-{\theta }_{t-1,i}{|+\delta )}^{2-2\alpha }}{\sqrt{{\sum }_{s=1}^t{g}_{s,i}^2{({\theta }_{s,i}-{\theta }_{s-1,i})}^{2-2\alpha }}+\sqrt{{\sum }_{s=1}^t{g}_{s,i}^2{({\theta }_{s,i}-{\theta }_{s-1,i})}^{2-2\alpha }}}\hfill \\ \hfill & \le \frac{g_{1,i}^2(|{\theta }_{1,i}-{\theta }_{0,i}{|+\delta )}^{2-2\alpha }}{\sqrt{g_{1,i}^2}{({\theta }_{1,i}-{\theta }_{0,i})}^{2-2\alpha }}\hfill \\ \hfill & +\sum _{t=2}^T\frac{2g_{t,i}^2(|{\theta }_{t,i}-{\theta }_{t-1,i}{|+\delta )}^{2-2\alpha }}{\sqrt{{\sum }_{s=1}^t{g}_{s,i}^2{({\theta }_{a,i}-{\theta }_{a-1,i})}^{2-2\alpha }}+\sqrt{{\sum }_{s=1}^{t-1}{g}_{s,i}^2}{({\theta }_{s,i}-{\theta }_{s-1,i})}^{2-2\alpha }}\hfill \\ \hfill & \le \sqrt{g_{1,i}^2(|{\theta }_{1,i}-{\theta }_{0,i}{|+\delta )}^{2-2\alpha }}\hfill \\ \hfill & +\sum _{t=2}^T(2\sqrt{\sum _{s=1}^t{g}_{s,i}^2(|{\theta }_{s,i}-{\theta }_{s-1,i}{|+\delta )}^{2-2\alpha }}-2\sqrt{\sum _{s=1}^{t-1}{g}_{s,i}^2(|{\theta }_{s,i}-{\theta }_{s-1,i}{|+\delta )}^{2-2\alpha })}\hfill \\ \hfill & \le 2\sqrt{\sum _{s=1}^T{g}_{s,i}^2(|{\theta }_{s,i}-{\theta }_{s-1,i}{|+\delta )}^{2-2\alpha }}\le 2G_i{(D+\delta )}^{1-\alpha }\sqrt{T}.\hfill \end{array}$$

(21)

Finally, combining Formula (21) and the estimation of the first term of $R(T)$, we have

$$\begin{array}{cc}\hfill R(T)& \le \sum _{i=1}^d[\frac{D_i^2{G}_i{\delta }^{\alpha -1}{(D_i+\delta )}^{1-\alpha }}{2\mu }\sqrt{T}]+\sum _{i=1}^d[\mu {(D_i+\delta )}^{2-2\alpha }{G}_i\sqrt{T}].\hfill \\ \hfill & =\sum _{i=1}^d{G}_i([\frac{D_i^2{\delta }^{\alpha -1}{(D_i+\delta )}^{1-\alpha }}{2\mu }+\mu {(D_i+\delta )}^{2-2\alpha }]\sqrt{T}.\hfill \end{array}$$

(22)

Similarly, we get the bound of $R(T)$ when $1<\alpha <2$:

$$R(T)\le \sum _{i=1}^d{G}_i([\frac{D_i^2{(D_i+\delta )}^{\alpha -1}{\delta }^{1-\alpha }}{2\mu }+\mu {\delta }^{2-2\alpha }]\sqrt{T}.$$

(23)

Similar to the proof in Theorem 1, let $T\to \infty$, ${\displaystyle \frac{R(T)}{T}}=O(\frac{1}{\sqrt{T}})\to 0$, which means Adagrad with fractional order is convergent.    □

Algorithm 2 Adagrad with fractional order.

1: Initialize $\alpha ,{\mu }_0,t_{max},t=0$ and ${\theta }_0$; 2: Calculate ${\theta }_1$ under ordinary SGD: ${\theta }_1={\theta }_0-{\mu }_0\nabla f_0({\theta }_0)$; 3: Update the parameters by ${\theta }_{t+1}={\theta }_t-\frac{{\mu }_t}{\Gamma (2-\alpha )}·g_t\parallel |{\theta }_t-{\theta }_{t-1}{|+\delta \parallel }^{1-\alpha }$ and ${\mu }_{t,i}=\frac{\mu \Gamma (2-\alpha )}{\sqrt{{\sum }_{s=1}^t{g}_{s,i}^2{\parallel {\theta }_{s,i}-{\theta }_{s-1,i}\parallel }^{2-2\alpha }}}$; 4: Increase t to $t_{max}$ until all batches of samples have been used.

Remark 6.

The learning rate of Algorithm 2 is related to its history gradient information. When the number of iterations is increasing, the value of the summation of the former gradient is hard to compute; meanwhile, the step size decreases so that new samples become less important. Once we take fractional order α into the algorithm, history gradient accumulation can be relieved by selecting a bigger order but the extra cost of the computation of the fractional gradient would increase.

Remark 7.

Different from Algorithm 1, the fractional gradient part of adaptive step size ${\mu }_t$ does not need a parameter δ to avoid singularity due to the accumulation of the square of the history gradient. Actually, parameter δ might damage the convergence of the algorithm, especially when $1<\alpha <2$ as shown in Figure 2.

3.3. SGD with Momentum and Fractional Order Gradient

Theorem 3.

Under Assumptions 1–3, Algorithm 3 with momentum is convergent.

Algorithm 3 mSGD with fractional order

1: Initialize $\alpha ,{\mu }_0,{\beta }_0,t_{max},t=0$ and ${\theta }_0$, $m_0=0$; 2: Calculate ${\theta }_1$ under ordinary SGD: ${\theta }_1={\theta }_0-{\mu }_0\nabla f_0({\theta }_0)$; 3: Update the parameters by ${\theta }_{t+1}={\theta }_t-{\mu }_t{\widehat{m}}_t$, $m_t={\beta }_t{m}_{t-1}+(1-{\beta }_t)\frac{{\mathbf{g}}_t\parallel |{\theta }_t-{\theta }_{t-1}{|+\delta \parallel }^{1-\alpha }}{\Gamma (2-\alpha )}$ and ${\widehat{m}}_t=\frac{m_t}{1-{\prod }_{i=1}^t{\beta }_i}$;${\mu }_{t+1}=\frac{{\mu }_0}{\sqrt{t+1}}$,${\beta }_{t+1}=\frac{{\beta }_0}{\sqrt{t+1}}$; 4: Increase t to $t_{max}$ until all batches of samples have been used.

Proof. 

For simplicity, denote $h_{t,i}=\frac{g_{t,i}(|{\theta }_{i,t}-{\theta }_{i,t-1}{|+\delta )}^{1-\alpha }}{\Gamma (2-\alpha )}$. The proposed algorithm can be expressed in the form of a component.

$$\begin{array}{cc}\hfill {\theta }_{i,t+1}& ={\theta }_{i,t}-{\mu }_t{\widehat{m}}_{i,t}\hfill \\ \hfill & ={\theta }_{i,t}-{\mu }_t\frac{m_{i,t}}{1-{\prod }_{i=1}^t{\beta }_i}\hfill \\ \hfill & ={\theta }_{i,t}-{\mu }_t\frac{{\beta }_t{m}_{i,t-1}+(1-{\beta }_t)h_{t,i}}{1-{\prod }_{i=1}^t{\beta }_i}.\hfill \end{array}$$

(24)

Similar to [9], we set ${\beta }_i\le \sqrt{c}<1$ as a decreasing sequence which means momentum $m_i$ would be close to $h_i$.

Let ${\theta }_{i,t+1}={\theta }_{i,t}-{\mu }_t\frac{{\beta }_t{m}_{i,t-1}+(1-{\beta }_t)h_{t,i}}{1-{\prod }_{s=1}^t{\beta }_s}$, ${\gamma }_t=\frac{{\mu }_t}{1-{\prod }_{s=1}^t{\beta }_s}$. So the gradient term can be separated as below:

$$\begin{array}{cc}\hfill {\theta }_{i+1,t}={\theta }_{i,t}& -{\gamma }_t({\beta }_t+(1-{\beta }_t)h_{t,i})\hfill \\ \Rightarrow {({\theta }_{i,t+1}-{\theta }_i^{*})}^2\hfill & ={[({\theta }_{i,t+1}-{\theta }_i^{*})-{\gamma }_t(\beta m_{i,t-1}+(1-\beta )h_{t,i})]}^2\hfill \\ \hfill \Rightarrow h_{t,i}({\theta }_{i,t}-{\theta }_i^{*})& =\frac{{({\theta }_{i,t}-{\theta }_i^{*})}^2-{({\theta }_{i,t+1}-{\theta }_i^{*})}^2}{2{\gamma }_t(1-{\beta }_t)}-\frac{{\beta }_t}{1-{\beta }_t}{m}_{i,t-1}({\theta }_{i,t-1}-{\theta }_i^{*})+\frac{{\gamma }_t{m}_{i,t}^2}{2(1-{\beta }_t)}.\hfill \end{array}$$

(25)

The first term of Formula (25) can be changed into

$$\begin{array}{cc}\hfill g_{t,i}({\theta }_{i,t}-{\theta }_i^{*})=& \Gamma (2-\alpha )\sum _{t=1}^T\frac{{({\theta }_{i,t}-{\theta }_i^{*})}^2-{({\theta }_{i,t+1}-{\theta }_i^{*})}^2}{2{\gamma }_t(1-{\beta }_t)(|{\theta }_{i,t}-{\theta }_{i,t-1}{|+\delta )}^{1-\alpha }}\hfill \\ \hfill =& \Gamma (2-\alpha )\sum _{t=1}^T\frac{[{({\theta }_{i,t}-{\theta }_i^{*})}^2-{({\theta }_{i,t+1}-{\theta }_i^{*})}^2](1-{\prod }_{s=1}^t{\beta }_s)}{2{\mu }_t(1-{\beta }_t)(|{\theta }_{i,t}-{\theta }_{i,t-1}{|+\delta )}^{1-\alpha }}\hfill \\ \hfill \le & \Gamma (2-\alpha )\sum _{t=1}^T\frac{{({\theta }_{i,t}-{\theta }_i^{*})}^2-{({\theta }_{i,t+1}-{\theta }_i^{*})}^2}{2{\mu }_t(1-{\beta }_1)(|{\theta }_{i,t}-{\theta }_{i,t-1}{|+\delta )}^{1-\alpha }}\hfill \\ \hfill \le & \Gamma (2-\alpha ){\delta }^{\alpha -1}[\frac{{({\theta }_{1,i}-{\theta }_i^{*})}^2}{2{\mu }_1(1-{\beta }_1)}-\frac{{({\theta }_{T+1,i}-{\theta }_i^{*})}^2}{2{\mu }_T(1-{\beta }_1)}]\hfill \\ \hfill +& \Gamma (2-\alpha )\sum _{t=2}^T{\delta }^{\alpha -1}{({\theta }_{i,t}-{\theta }_i^{*})}^2·[\frac{1}{2{\mu }_t(1-{\beta }_1)}-\frac{1}{2{\mu }_{t-1}(1-{\beta }_1)}]\hfill \\ \hfill \le & \Gamma (2-\alpha ){\delta }^{\alpha -1}[\frac{D_i^2}{2{\mu }_t(1-{\beta }_1)}-\frac{D_i^2}{2{\mu }_{t-1}(1-{\beta }_1)}]+\frac{\Gamma (2-\alpha )D_i^2{\delta }^{\alpha -1}}{2{\mu }_1(1-{\beta }_1)}\hfill \\ \hfill \le & \frac{\Gamma (2-\alpha )D_i^2{\delta }^{\alpha -1}}{2{\mu }_T(1-{\beta }_1)}.\hfill \end{array}$$

(26)

The third inequality in (26) holds for ${\beta }_t<\cdots <{\beta }_1$ and the second term of Formula (25) can be changed into:

$$\Gamma (2-\alpha )\sum _{t=1}^T-\frac{{\beta }_t}{1-{\beta }_t}\frac{m_{i,t-1}({\theta }_{t,i}-{\theta }_i^{*})}{(|{\theta }_{i,t}-{\theta }_{i,t-1}{|+\delta )}^{1-\alpha }}\le \Gamma (2-\alpha )\sum _{t=1}^T\frac{{\beta }_t{\delta }^{\alpha -1}}{1-{\beta }_t}|-m_{i,t-1}|D_i.$$

(27)

As for the momentum term, we have

$$\begin{array}{cc}\hfill m_{i,t}& ={\beta }_t{\beta }_{t-1}\cdots {\beta }_1{m}_{i,0}+{\beta }_t{\beta }_{t-1}\cdots (1-{\beta }_1)h_{1,i}+\cdots +(1-{\beta }_t)h_{t,i}\hfill \\ \hfill & =\sum _{s=1}^t(1-{\beta }_s)(\prod _{k=s+1}^t{\beta }_k)h_{s,i}.\hfill \end{array}$$

(28)

$$\begin{array}{cc}\hfill |m_{t,i}|& \le \sum _{s=1}^t(1-{\beta }_s)(\prod _{r=s+1}^t{\beta }_r)\left|h_{s,i}\right|\hfill \\ \hfill & \le \frac{1}{\Gamma (2-\alpha )}\sum _{s=1}^t(1-{\beta }_s)(\prod _{r=s+1}^t{\beta }_r)G_i{(D_i+\delta )}^{1-\alpha }\hfill \\ \hfill & \le \frac{1}{\Gamma (2-\alpha )}(1-\prod _{s=1}^t{\beta }_s)G_i{(D_i+\delta )}^{1-\alpha }\le \frac{1}{\Gamma (2-\alpha )}{G}_i{(D_i+\delta )}^{1-\alpha }.\hfill \end{array}$$

(29)

Therefore, the second term of Formula (25) can be enlarged as:

$$\begin{array}{cc}\hfill & \sum _{t=1}^T-\frac{{\beta }_t}{1-{\beta }_t}{m}_{i,t-1}({\theta }_{t,i}-{\theta }_i^{*})\hfill \\ \hfill & \le G_i{D}_i{\delta }^{\alpha -1}{(D_i+\delta )}^{1-\alpha }\sum _{t=1}^T\frac{{\beta }_t}{1-{\beta }_t}.\hfill \end{array}$$

(30)

As for the third term, we have

$$\begin{array}{cc}\hfill & \sum _{t=1}^T\frac{{\gamma }_t}{2(1-{\beta }_t)}{m}_{i,t}^2\hfill \\ \hfill & \le \sum _{t=1}^T\frac{{\gamma }_t}{2(1-{\beta }_t)}·\sum _{s=1}^t{((1-{\beta }_s)(\prod _{r=s+1}^t{\beta }_r)h_{s,i})}^2\hfill \\ \hfill & \le \frac{1}{{\Gamma }^2(2-\alpha )}\sum _{t=1}^T\frac{{\gamma }_t}{2(1-{\beta }_t)}·\sum _{s=1}^t{({(1-{\beta }_s)}^2(\prod _{r=s+1}^t{\beta }_r))}^2{G}_i^2{(D_i+\delta )}^{2-2\alpha }.\hfill \\ \hfill & \le \frac{1}{{\Gamma }^2(2-\alpha )}\sum _{t=1}^T\frac{{\mu }_t}{2(1-{\beta }_t)(1-{\prod }_{s=1}^t{\beta }_s)}·\sum _{s=1}^t(\prod _{r=s+1}^t{\beta }_r){)}^2{G}_i^2{(D_i+\delta )}^{2-2\alpha }.\hfill \end{array}$$

(31)

Finally, we get the bound of the regret function of SGD with momentum and a fractional gradient order.

$$\begin{array}{cc}\hfill R(T)\le & \Gamma (2-\alpha )\sum _{i=1}^d\frac{D_i^2{\delta }^{\alpha -1}}{2{\mu }_T(1-{\beta }_1)}+\sum _{i=1}^d{G}_i{D}_i{\delta }^{\alpha -1}{(D_i+\delta )}^{1-\alpha }\sum _{t=1}^T\frac{{\beta }_t}{1-{\beta }_t}\hfill \\ \hfill +& \frac{1}{{\Gamma }^2(2-\alpha )}\sum _{i=1}^d{G}_i^2{(D_i+\delta )}^{2-2\alpha }\sum _{t=1}^T\frac{{\mu }_t}{2(1-{\beta }_t)(1-{\prod }_{s=1}^t{\beta }_s)}\sum _{s=1}^t\prod _{r=s+1}^t{(\sqrt{c})}^2\hfill \\ \hfill \le & \Gamma (2-\alpha )\frac{{\sum }_{i=1}^d{D}_i^2{\delta }^{\alpha -1}}{2{\mu }_T(1-{\beta }_1)}+(\sum _{i=1}^d{G}_i{D}_i{\delta }^{2-2\alpha })(\frac{{\sum }_{t=1}^T{\beta }_t}{1-{\beta }_1})\hfill \\ \hfill +& \frac{1}{{\Gamma }^2(2-\alpha )}(\sum _{i=1}^d{G}_i^2{(D_i+\delta )}^{2-2\alpha })\frac{{\sum }_{t=1}^T{\mu }_t}{2{(1-{\beta }_1)}^2(1-c)},\hfill \end{array}$$

(32)

where ${\beta }_t\le \sqrt{c}<1$. The result is similar when $1<\alpha <2$. It is shown that the bound has a connection with $\frac{1}{T{\mu }_T}$, $\frac{{\sum }_{t=1}^T{\beta }_t}{T}$, $\frac{{\sum }_{t=1}^T{\mu }_t}{T}$ and fractional order $\alpha$. If we take ${\beta }_t$ as $\frac{{\beta }_1}{\sqrt{t}}$, the order of magnitude of the regret function will be maintained in $O(\sqrt{T})$, then $R(T)/T=O(\frac{1}{\sqrt{T}})\to 0$, as $T\to \infty$. □

Remark 8.

In theoretical analysis, the three algorithms above have the same convergence speed $O(\frac{1}{\sqrt{T}})$. But in practical engineering, SGD with momentum has the fastest convergence speed for its history gradient information, which would be shown in the simulation. For the fractional gradient condition, we can take the order by comparing the effects of ${\beta }_t$ and ${\mu }_t$.

4. Simulations

In this section, we will solve two practical problems using the proposed stochastic fractional order gradient descent method to demonstrate the convergence and the relationship between convergence speed and fractional order. The neural network training and all other experiments are conducted on a computer with a GeForce RTX 4060 Laptop GPU and Intel i7 CPU @ 2.6 GHz.

4.1. Example 1

In this example, we will solve a system identification problem to examine the effectiveness of the proposed algorithm. The target model is an auto-regressive (AR) model with its coefficients to be recognized.

$$y(k)=\sum _{i=1}^p{a}_{i}y(k-i)+\xi (k),$$

(33)

where $y(k-i)$ is the output of the system at time of $k-i$ and $\xi (k)$ is the stochastic noise sequence. $a_i$ are parameters to be estimated. Our goal is to determine the coefficients of the model. The regret function of the system is

$$J_k(\widehat{\theta })=\frac{1}{2}{(y(k)-{\varphi }^T(k)·\widehat{\theta }(k))}^2,$$

(34)

where $\widehat{\theta }(k)={({\widehat{a}}_1(k),\cdots ,{\widehat{a}}_p(k))}^T$ and $\varphi (k)={(y(k-1),\cdots ,y(k-p))}^T$. For each sample $\{y(k),\widehat{\theta }(k),\varphi (k)\}$, the objective functions can be seen as the online optimization problem. Therefore, applying Algorithm 1 to the system, the algorithm degrades into the LMS algorithm. The iteration formula of the algorithm can be written as:

$$\widehat{\theta }(k+1)=\widehat{\theta }(k)+\frac{{\mu }_k}{\Gamma (2-\alpha )}(y(k)-{\varphi }^T(k)·\widehat{\theta }(k))\varphi (k)·\parallel |\widehat{\theta }(k)-\widehat{\theta }(k-1){|+\delta \parallel }^{1-\alpha }.$$

(35)

In particular, when $\alpha =1$, Formula (35) was turned into

$$\widehat{\theta }(k+1)=\widehat{\theta }(k)+{\mu }_k(y(k)-{\varphi }^T(k)·\widehat{\theta }(k))\varphi (k),$$

(36)

which became an ordinary LMS algorithm. We take ${\mu }_k=\frac{0.1}{\sqrt{k}}$ as the learning rate and analyze the convergence of the parameters under noise. We consider an AR model with order $p=2$:

$$y(k)=1.5y(k-1)-0.7y(k-2)+\xi (k),$$

(37)

when the noise sequence is Gaussian white noise with mean value 0 and variance 0.5.

The numerical simulation results are shown in Figure 3, Figure 4 and Figure 5, where the abscissa is the number of iterations. It is shown that the parameters of the AR model could be identified by the algorithms proposed in this article and the figures reflect the effectiveness of the algorithm and that the larger fractional order would bring a faster convergence speed. Meanwhile, the problem of gradient accumulation in the Adagrad method can be relieved by a bigger fractional order as Figure 6 shows.

As Figure 4 and Figure 5 shows, compared with existing algorithms with an integer order gradient—like standard SGD, Adagrad, where $\mu =0.1$, and the momentum method, where ${\beta }_t=\frac{0.1}{\sqrt{k}}$—the fractional counterpart performs better on convergence but the accuracy is not outstanding for the disturbance of noise. In particular, SGD with momentum and a fractional order has a better result than the other algorithms in terms of convergence speed. That is to say, the fractional gradient SGD method can be applied to other mature algorithms and is likely to perform better than them.

4.2. Example 2

A deep BP neural network with a fractional gradient order is built to test the validity of the proposed algorithm. The data we adopted are from the well-known MNIST dataset, which contains 60,000 handwritten numeral images for training [20]. Each sample of the dataset can be seen as a matrix of $28\ast 28$ and the label of each sample is a vector of $10\ast 1$, where the element being 1 is the classification result. The main application of the fractional order gradient is the process of error back propagation, where each layer of the network updates its value of weights from the error between samples and the network weights trained before, and this can be seen as a typical online optimization problem. The batch size set here is 200 and the maximum of number of iterations is 300, which can be checked in Figure 7. The designed network consists of 5 layers which have 64 nodes in a hidden layer and 10 nodes in an output layer. Under the fractional order gradient information, the update of the weight matrix between layers can be described as

$$W_{k+2}=W_{k+1}-\frac{1}{\Gamma (2-\alpha )}\frac{\eta }{n}\sum _{i=1}^n\frac{\partial E}{\partial W_{k+1}^i}(|W_{k+1}^i-W_k^i{|+\delta )}^{1-\alpha },$$

(38)

where $E=\frac{1}{2}{\sum }_{j=1}^l{\sum }_{i=1}^{n_l}{(a_{ij}^l-o_{ij})}^2$ with $L_2$ regularization, $\eta$ is the learning rate, and n is the sample size of each batch; set $\eta =\frac{3}{\sqrt{k}}$.

The results are shown in Figure 7, Figure 8 and Figure 9, which reveal the effectiveness of SGD with a fractional order gradient. The degrees of accuracy on the test set are 90.55%, 93.3%, and 94.4%, respectively. The neural network under order 1.2 has a faster convergence speed and higher accuracy on the test set, which is consistent with Theorem 1. Figure 8 shows that order 1.1 has the highest accuracy and other orders larger than 1 would achieve better results than the integer order.

5. Conclusions

In this article, we proposed some improved stochastic fractional order gradient descent algorithms for the online optimization problem; the convergence is given, where the objective functions were assumed to be convex and the gradients were assumed to be bounded. The bounds of the empirical regret functions of the improved SGD with a fractional order were built based on several assumptions, and the proposed algorithm can relieve the gradient exploding problem. Finally, it was shown how the fractional order affects the convergence of the algorithm by way of two practical applications including system identification and classification.

Future work will include studying the variable fractional order stochastic gradient descent algorithm and developing the decentralized version of FOSGD to accommodate larger-scale datasets. While the setting in this article considers an invariable fractional order in a centralized algorithm, which cannot give scope to the advantage of the fractional algorithm, developing the decentralized version with a variable fractional order is in our future agenda.