An Extended Gradient Method for Smooth and Strongly Convex Functions

Abstract

In this work, we introduce an extended gradient method that employs the gradients of the preceding two iterates to construct the search direction for the purpose of solving the centralized and decentralized smooth and strongly convex functions. Additionally, we establish the linear convergence for iterate sequences in both the centralized and decentralized manners. Furthermore, the numerical experiments demonstrate that the centralized extended gradient method can achieve faster acceleration than the compared algorithms, and the search direction also exhibits the capability to improve the convergence of the existing algorithms in both two manners.

1. Introduction

In this work, we consider the unconstrained optimization problem:

$$\underset{x\in {\mathbb{R}}^p}{min}f\left(x\right).$$

(1)

The cost function $f\left(x\right):{\mathbb{R}}^p\mapsto \mathbb{R}$ of this problem is a differentiable strongly convex function, and the gradient $\nabla f\left(x\right)$ is L-Lipschitz continuous. Starting from an initial point $x^0\in {\mathbb{R}}^p$ with $\nabla f\left(x^{-1}\right)=\mathbf{0}$, the extended gradient method updates the iterates $x^k$ for $k\ge 0$:

$$x^{k+1}=x^k-\alpha (\nabla f\left(x^k\right)+\nabla f\left(x^{k-1}\right)),$$

(2)

where the step size $\alpha$ ($\alpha >0$) is sufficiently small, and it will be analyzed below to ensure the convergence of the extended gradient algorithm.

Firstly, we discussed briefly the accelerated scheme of gradient methods, that is, the iterate $x^{k+1}$ updates using the information of the preceding two iterates. Such as the gradient method with extrapolation that has the following forms: $x^{k+1}=x^k-{\alpha }_k\nabla f\left(x^k\right)+{\beta }_k(x^k-x^{k-1})$, where ${\alpha }_k$ is the step size and ${\beta }_k\in [0,1)$ is a scalar. When ${\alpha }_k$ and ${\beta }_k$ are set as the constant scalar $\alpha$ and $\beta$, respectively, the method is equivalent to the heavy ball method [1]. In [2], the linear convergence rate was established for the heavy ball method applied to smooth, twice continuously differentiable and strongly convex functions. Further, the authors in [3] constructed an example to illustrate that even if the objective functions are strongly convex, the heavy ball method still can not converge when the functions are not twice continuously differentiable. Thus, the property that the objective function is twice continuously differentiable is a necessary condition in the convergence guarantees of the heavy ball method. Based on this fact, for the case of strictly convex quadratic functions, the recent work [4], obtained the iteration complexity of the heavy ball method under the assumption that the appropriate upper and lower bounds for eigenvalues of the Hessian matrix are known. In particular, several accelerated gradient descent methods with a similar structure to the heavy ball method also have been proposed in [5], and they converge linearly to the solution of the smooth and strongly convex functions with the optimal iteration complexity. Compared to the heavy ball method, the above accelerated methods fully utilize the difference between the preceding two iterates. Moreover, the main advantage of solving convex functions in practice is that the accelerated gradient descent methods can converge to the optimum more quickly [6].

On the other hand, the accelerated scheme also can update the variables employing the gradients of the preceding several iterates. As mentioned in [7,8], the optimistic gradient descent-ascent (OGDA) method and the inertial gradient algorithm with Hessian dampling (IGAHD) update the variable via the difference of the gradients of the preceding two iterates. Besides, for the least mean squares (LMS) estimation of graph signals, the extended LMS algorithm in [9] and proportionate-type graph LMS algorithm in [10] update the variable via the weighted sum of the gradients of the preceding several iterates. Further, the Anderson acceleration gradient descent method (AA-PGA) in [11] updates the variable by employing the convex combination of the gradients of the preceding several iterates. However, AA-PGA converges linearly to the optimal solution $x^{*}$ under the assumption ${\nabla }^{2}f\left(x^{*}\right)⪰\upsilon I$. Inspired by the above works, we here introduce an extended gradient method, in which the variables are updated along the direction of the sum of the gradients of the preceding two iterates. The main purpose of this work is to analyze the convergence of the extended gradient method for finding the optimal solution of problem (1). We will show that when the step size is less than a given upper bound, the linear convergence of the extended gradient method can be guaranteed.

Next, we consider a class of smooth and strongly convex optimization functions on the real vector space with Euclidean norm $\parallel ·\parallel$ and inner product $⟨·,·⟩$.

Definition 1.

A function $f:{\mathbb{R}}^p\mapsto \mathbb{R}$ is L-smooth if the gradient $\nabla f\left(x\right)$ is L-Lipschitz continuous, that is,

$$\begin{array}{c}\hfill \parallel \nabla f\left(x\right)-\nabla f\left(y\right)\parallel \le L\parallel x-y\parallel ,\forall x,y.\end{array}$$

If f is convex and L-smooth it also holds that $f\left(x\right)\le f\left(y\right)+⟨\nabla f\left(y\right),x-y⟩+\frac{L}{2}{\parallel x-y\parallel }^2$ [12].

Definition 2.

A function $f:{\mathbb{R}}^p\mapsto \mathbb{R}$ is μ-strongly convex if there exists $\mu >0$ such that

$$\begin{array}{c}\hfill f\left(x\right)\ge f\left(y\right)+⟨\nabla f\left(y\right),x-y⟩+\frac{\mu }{2}{\parallel x-y\parallel }^2.\end{array}$$

2. Analysis of the Extended Gradient Method

In this section, to establish the convergence of the extended gradient method, we first obtain an important inequality in Theorem 1, where the upper bound on $\parallel x^{k+1}-x^{*}\parallel$ will be given in terms of the linear combinations of $\parallel x^k-x^{*}\parallel$, $\parallel x^{k-1}-x^{*}\parallel$ and $\parallel x^{k-2}-x^{*}\parallel$. This inequality is vital and will be used in the following analysis.

Theorem 1.

Assume that f is μ-strongly convex, and the gradient $\nabla f\left(x\right)$ is L-Lipschitz continuous. Let the step size α satisfy the condition $0\le \alpha \le \frac{\sqrt{1+\frac{2\mu }{L}}-1}{2L}$, and ${\left\{x^k\right\}}_{k\in \mathbb{N}}$ be generated by the extended gradient method (2). Then, the following hold.

(i) The iterates $\left\{x^k\right\}$ satisfy the following linear inequality

$$\begin{array}{cc}\hfill {\parallel x^{k+1}-x^{*}\parallel }^2\le & (1+2{\alpha }^2{L}^2-\alpha \mu )\parallel x^k-x^{*}{\parallel }^2+(2{\alpha }^2{L}^2+2{\alpha }^3{L}^3-\alpha \mu ){\parallel x^{k-1}-x^{*}\parallel }^2\hfill \\ \hfill & +2{\alpha }^3{L}^3{\parallel x^{k-2}-x^{*}\parallel }^2,\hfill \end{array}$$

where $x^{*}$ is the optimal solution.

(ii) $\underset{2\le k\le K}{min}f\left(x^k\right)$ converges to the optimal value $f\left(x^{*}\right)$ at the sub-linear rate of $\mathcal{O}\left(\frac{1}{K}\right)$.

Proof. 

(i) According to the update equation in (2), for any $x\in {\mathbb{R}}^p$, we have

$$\begin{array}{cc}\hfill {\parallel x^{k+1}-x\parallel }^2=& {\parallel x^k-\alpha (\nabla f\left(x^k\right)+\nabla f\left(x^{k-1}\right))-x\parallel }^2\hfill \\ \hfill =& {\parallel x^k-x\parallel }^2+{\alpha }^2{\parallel \nabla f\left(x^k\right)+\nabla f\left(x^{k-1}\right)\parallel }^2\hfill \\ \hfill & -2\alpha {(\nabla f\left(x^k\right)+\nabla f\left(x^{k-1}\right))}^{\mathrm{T}}(x^k-x)\hfill \\ \hfill =& {\parallel x^k-x\parallel }^2+{\alpha }^2{\Gamma }_1-2\alpha ({\Gamma }_2+{\Gamma }_3),\hfill \end{array}$$

(3)

where ${\Gamma }_1:={\parallel \nabla f\left(x^k\right)+\nabla f\left(x^{k-1}\right)\parallel }^2$, ${\Gamma }_2:=\nabla f{\left(x^k\right)}^{\mathrm{T}}(x^k-x)$, and ${\Gamma }_3:=\nabla f{\left(x^{k-1}\right)}^{\mathrm{T}}(x^k-x)$.

Now, substituting $x=x^{*}$, and noting that $\nabla f\left(x^{*}\right)=0$, we first bound the term ${\Gamma }_1$,

$$\begin{array}{cc}\hfill {\Gamma }_1& \le 2\parallel \nabla f\left(x^k\right){\parallel }^2+2{\parallel \nabla f\left(x^{k-1}\right)\parallel }^2\hfill \\ \hfill & =2\parallel \nabla f\left(x^k\right)-\nabla f\left(x^{*}\right){\parallel }^2+2{\parallel \nabla f\left(x^{k-1}\right)-\nabla f\left(x^{*}\right)\parallel }^2\hfill \\ \hfill & \le 2L^2\parallel x^k-x^{*}{\parallel }^2+2L^2{\parallel x^{k-1}-x^{*}\parallel }^2.\hfill \end{array}$$

(4)

Secondly, we provide a lower bound of ${\Gamma }_2$ with the strongly convex property of $f\left(x\right)$,

$$\begin{array}{c}\hfill {\Gamma }_2=\nabla f{\left(x^k\right)}^{\mathrm{T}}(x^k-x^{*})\ge f\left(x^k\right)-f\left(x^{*}\right)+\frac{\mu }{2}{\parallel x^k-x^{*}\parallel }^2.\end{array}$$

(5)

Finally, we bound the term ${\Gamma }_3$ by employing ${\Gamma }_1$ and ${\Gamma }_2$. Based on the smoothness of f, we can obtain that

$$\begin{array}{c}\hfill f\left(x^k\right)\le f\left(x^{k-1}\right)+\nabla f{\left(x^{k-1}\right)}^{\mathrm{T}}(x^k-x^{k-1})+\frac{L}{2}{\parallel x^k-x^{k-1}\parallel }^2.\end{array}$$

(6)

Add and subtract the inner product $\nabla f{\left(x^{k-1}\right)}^{\mathbf{T}}{x}^{*}$ to the right-hand side of (6) and rearrange the terms to have

$$\begin{array}{c}\hfill f\left(x^k\right)\le f\left(x^{k-1}\right)+\nabla f{\left(x^{k-1}\right)}^{\mathrm{T}}(x^{*}-x^{k-1})+\frac{L}{2}{\parallel x^k-x^{k-1}\parallel }^2+\nabla f{\left(x^{k-1}\right)}^{\mathrm{T}}(x^k-x^{*}).\end{array}$$

(7)

Then by the bound of ${\Gamma }_2$ (5) (k is replaced by $k-1$), (7) can be rewritten as

$$\begin{array}{c}\hfill f\left(x^k\right)\le f\left(x^{*}\right)-\frac{\mu }{2}\parallel x^{k-1}-x^{*}{\parallel }^2+\frac{L}{2}{\parallel x^k-x^{k-1}\parallel }^2+\nabla f{\left(x^{k-1}\right)}^{\mathrm{T}}(x^k-x^{*}),\end{array}$$

(8)

wherein, in terms of the update Equation (2) and the bound of ${\Gamma }_1$ (4), the term $\parallel x^k-x^{k-1}{\parallel }^2$ in (8) can be further expressed as

$$\begin{array}{cc}\hfill \parallel x^k-x^{k-1}{\parallel }^2& ={\alpha }^2{\parallel \nabla f\left(x^{k-1}\right)+\nabla f\left(x^{k-2}\right)\parallel }^2\hfill \\ \hfill & \le 2{\alpha }^2{L}^2\parallel x^{k-1}-x^{*}{\parallel }^2+2{\alpha }^2{L}^2{\parallel x^{k-2}-x^{*}\parallel }^2.\hfill \end{array}$$

(9)

Therefore, the bound of ${\Gamma }_3$ can be yielded as

$$\begin{array}{cc}\hfill {\Gamma }_3=\nabla f{\left(x^{k-1}\right)}^{\mathrm{T}}(x^k-x^{*})\ge & f\left(x^k\right)-f\left(x^{*}\right)+(\frac{\mu }{2}-{\alpha }^2{L}^3){\parallel x^{k-1}-x^{*}\parallel }^2\hfill \\ \hfill & -{\alpha }^2{L}^3{\parallel x^{k-2}-x^{*}\parallel }^2\hfill \end{array}$$

(10)

Substituting (4), (5) and (10) (the bounds of ${\Gamma }_1$, ${\Gamma }_2$, and ${\Gamma }_3$) into (3), we have

$$\begin{array}{cc}\hfill {\parallel x^{k+1}-x^{*}\parallel }^2\le & (1+2{\alpha }^2{L}^2-\alpha \mu )\parallel x^k-x^{*}{\parallel }^2+(2{\alpha }^2{L}^2+2{\alpha }^3{L}^3-\alpha \mu ){\parallel x^{k-1}-x^{*}\parallel }^2\hfill \\ \hfill & +2{\alpha }^3{L}^3{\parallel x^{k-2}-x^{*}\parallel }^2-4\alpha (f\left(x^k\right)-f\left(x^{*}\right)).\hfill \end{array}$$

(11)

Considering that $f\left(x^k\right)\ge f\left(x^{*}\right)$ in (11), we complete the proof of (i).

(ii) Rearranging the terms in (11), it holds

$$\begin{array}{cc}\hfill 4\alpha (f\left(x^k\right)-f\left(x^{*}\right))\le & \left(\parallel x^k-x^{*}{\parallel }^2-{\parallel x^{k+1}-x^{*}\parallel }^2\right)\hfill \\ \hfill & -2{\alpha }^3{L}^3\left(\parallel x^{k-1}-x^{*}{\parallel }^2-{\parallel x^{k-2}-x^{*}\parallel }^2\right)\hfill \\ \hfill & +(2{\alpha }^2{L}^2-\alpha \mu ){\parallel x^k-x^{*}\parallel }^2\hfill \\ \hfill & +(2{\alpha }^2{L}^2+4{\alpha }^3{L}^3-\alpha \mu ){\parallel x^{k-1}-x^{*}\parallel }^2.\hfill \end{array}$$

(12)

For $0\le \alpha \le \frac{\sqrt{1+\frac{2\mu }{L}}-1}{2L}$, we obtain $2{\alpha }^2{L}^2+4{\alpha }^3{L}^3-\alpha \mu \le -(2{\alpha }^2{L}^2-\alpha \mu )$ and thus

$$\begin{array}{cc}\hfill 4\alpha (f\left(x^k\right)-f\left(x^{*}\right))\le & \left(\parallel x^k-x^{*}{\parallel }^2-{\parallel x^{k+1}-x^{*}\parallel }^2\right)\hfill \\ \hfill & -2{\alpha }^3{L}^3\left(\parallel x^{k-1}-x^{*}{\parallel }^2-{\parallel x^{k-2}-x^{*}\parallel }^2\right)\hfill \\ \hfill & -(2{\alpha }^2{L}^2-\alpha \mu )(\parallel x^{k-1}-x^{*}{\parallel }^2-\parallel x^k-x^{*}{\parallel }^2).\hfill \end{array}$$

(13)

Summing over $k=2,\cdots ,K$, we have

$$\begin{array}{cc}\hfill \sum _{k=2}^{K}4\alpha \left(f\left(x^k\right)-f\left(x^{*}\right)\right)\le & {\parallel x^2-x^{*}\parallel }^2-{\parallel x^{K+1}-x^{*}\parallel }^2\hfill \\ \hfill & -2{\alpha }^3{L}^3(\parallel x^{K-1}-x^{*}{\parallel }^2-\parallel x^0-x^{*}{\parallel }^2)\hfill \\ \hfill & -(2{\alpha }^2{L}^2-\alpha \mu )(\parallel x^1-x^{*}{\parallel }^2-\parallel x^K-x^{*}{\parallel }^2).\hfill \end{array}$$

(14)

Noticed that $0\le \alpha \le \frac{\sqrt{1+\frac{2\mu }{L}}-1}{2L}\le \frac{\mu }{2L^2}$, then $(2{\alpha }^2{L}^2-\alpha \mu ){\parallel x^K-x^{*}\parallel }^2<0$. Therefore, the following inequality holds

$$\begin{array}{c}\hfill \underset{2\le k\le K}{min}f\left(x^k\right)-f\left(x^{*}\right)\le \frac{{\parallel x^2-x^{*}\parallel }^2-(2{\alpha }^2{L}^2-\alpha \mu )\parallel x^1-x^{*}{\parallel }^2+2{\alpha }^3{L}^3{\parallel x^0-x^{*}\parallel }^2}{4\alpha (K-2)},\end{array}$$

(15)

which means that $\underset{2\le k\le K}{min}f\left(x^k\right)$ converges to the optimal value $f\left(x^{*}\right)$ at the sub-linear rate of $\mathcal{O}\left(\frac{1}{K}\right)$. □

The result in (ii) of Theorem 1 shows that the function value generated by the extended gradient method converges to the optimal solution of problem (1) at a sublinear rate. Further, in order to derive the linear convergence of the extended gradient method, we give two important lemmas as follows.

Lemma 1

([13] Corollary 8.1.29). Let $A\in {\mathbb{R}}^{n\times n}$ be a nonnegative matrix, and $\omega \in {\mathbb{R}}^n$ be a positive vector. If $A\omega <q\omega$, then $\rho \left(A\right)<q$.

Lemma 2

([14] Lemma 1 in Section 2.1). It holds that $\rho \left(A\right)=\underset{k\mapsto \infty }{lim}{\parallel A^k\parallel }^{\frac{1}{k}}$, i.e., the spectral radius of A gives the asymptotic growth rate of $\parallel A^k\parallel$: For every $ϵ>0$ there is $c=c\left(\epsilon \right)$ such that $\parallel A^k{\parallel \le c(\rho \left(A\right)+\epsilon )}^k$, for all k.

Together with (i) of Theorem 1, Lemma 1 and Lemma 2, we give the following Theorem 2. For notational convenience, we define $z^k:=(\parallel x^k-x^{*}{\parallel }^2,\parallel x^{k-1}-x^{*}{\parallel }^2,\parallel x^{k-2}-x^{*}{{\parallel }^2)}^{\top }$.

Theorem 2.

The same assumptions hold as in Theorem 1. Let the sequence $\left\{x^k\right\}$ generated by the extended gradient method and $x^{*}$ be the optimal solution. Then for every $\epsilon >0$, there is $c=c\left(ϵ\right)$ such that $||z^{k+1}{||\le c(\rho \left(A\right)+ϵ)}^k\parallel z^0\parallel$, for all k, where $\rho \left(A\right)<1$ is a constant.

Proof. 

By the result in (i) of Theorem 1, we first readily find that $z^{k+1}\le A{z}^k$, here $A=\left(\begin{array}{ccc}1+2{\alpha }^2{L}^2-\alpha \mu & 2{\alpha }^2{L}^2+2{\alpha }^3{L}^3-\alpha \mu & 2{\alpha }^3{L}^3\\ 1& 0& 0\\ 0& 1& 0\end{array}\right)$.

Under the assumption that the spectral radius of matrix A is less than 1 (i.e., $\rho \left(A\right)<1$), the statement of convergence can be obtained via Lemma 2. Thus, it remains to analyze the spectral radius of the matrix A and show that the spectral radius of A is less than 1. Find a positive vector $c={[c_1,c_2,c_3]}^{\mathbf{T}}$ satisfying $Ac<c$, then the following inequalities hold

$$\begin{array}{cc}\hfill & (2{\alpha }^2{L}^2-\alpha \mu )c_1+(2{\alpha }^2{L}^2+2{\alpha }^3{L}^3-\alpha \mu )c_2+2{\alpha }^3{L}^3{c}_3<0;\hfill \\ \hfill & c_1<c_2;\hfill \\ \hfill & c_2<c_3.\hfill \end{array}$$

(16)

To ensure that the first inequality in (16) holds, the necessary condition is that $2{\alpha }^2{L}^2-\alpha \mu <0$, which means $\alpha <\frac{\mu }{2L^2}$. Then, according to the sign of $2{\alpha }^2{L}^2+2{\alpha }^3{L}^3-\alpha \mu$, we further deduce the bounds of $\alpha$ by considering the following two cases.

(i) If $2{\alpha }^2{L}^2+2{\alpha }^3{L}^3-\alpha \mu >0$, we can derive the first inequality of (16) as

$$\begin{array}{cc}\hfill (2{\alpha }^2{L}^2-\alpha \mu )c_1+(2{\alpha }^2{L}^2+2{\alpha }^3{L}^3-\alpha \mu )c_2+2{\alpha }^3{L}^3{c}_3>& (4{\alpha }^2{L}^2+2{\alpha }^3{L}^3-2\alpha \mu )c_1\hfill \\ \hfill & +2{\alpha }^3{L}^3{c}_3,\hfill \end{array}$$

(17)

duo to $0<c_1<c_2$. Then, since the left-hand side of the (17) is less than 0, we have

$$\begin{array}{c}\hfill 2{\alpha }^3{L}^3{c}_3<-(4{\alpha }^2{L}^2+2{\alpha }^3{L}^3-2\alpha \mu )c_1<-(4{\alpha }^2{L}^2+2{\alpha }^3{L}^3-2\alpha \mu )c_3,\end{array}$$

(18)

due to $0<c_1<c_3$. Further, it follows from (18) that $-(4{\alpha }^2{L}^2+2{\alpha }^3{L}^3-2\alpha \mu )c_3-2{\alpha }^3{L}^3{c}_3>0$, namely, $2{\alpha }^2{L}^2+2{\alpha }^3{L}^3-\alpha \mu <0$, which leads to a contradiction to the assumption.

(ii) If $2{\alpha }^2{L}^2+2{\alpha }^3{L}^3-\alpha \mu \le 0$, the first inequality of (16) can be written as

$$\begin{array}{cc}\hfill (2{\alpha }^2{L}^2-\alpha \mu )c_1+(2{\alpha }^2{L}^2+2{\alpha }^3{L}^3-\alpha \mu )c_2+2{\alpha }^3{L}^3{c}_3>& (2{\alpha }^2{L}^2-\alpha \mu )c_1\hfill \\ \hfill & +(2{\alpha }^2{L}^2+4{\alpha }^3{L}^3-\alpha \mu )c_3,\hfill \end{array}$$

(19)

due to $0<c_2<c_3$. Then, based on the fact that the left-hand side of (19) is less than 0, we have

$$\begin{array}{cc}\hfill (2{\alpha }^2{L}^2+4{\alpha }^3{L}^3-\alpha \mu )c_3& <-(2{\alpha }^2{L}^2-\alpha \mu )c_1\hfill \\ \hfill & <-(2{\alpha }^2{L}^2-\alpha \mu )c_3,\hfill \end{array}$$

(20)

due to $0<c_1<c_3$. On rearranging the terms of (20), we obtain that $2(2{\alpha }^2{L}^2+2{\alpha }^3{L}^3-\alpha \mu )c_3<0$, which is consistent with the assumption. Therefore, the upper bound of step size $\alpha$ can be reduced as

$$\begin{array}{c}\hfill \alpha <\frac{\sqrt{1+\frac{2\mu }{L}-1}}{2L}=min\left\{\frac{\mu }{2L^2},\frac{\sqrt{1+\frac{2\mu }{L}-1}}{2L}\right\}.\end{array}$$

(21)

Finally, we have demonstrated that there exists a positive vector $c={[c_1,c_2,c_3]}^{\mathbf{T}}$ such that $Ac<c$, then it is clear $\rho \left(A\right)<1$ according to Lemma 1 for $a<\frac{\sqrt{1+\frac{2\mu }{L}-1}}{2L}$. □

Remark 1.

Choose $\alpha \le min\{\frac{1}{6L},\frac{\sqrt{1+\frac{2\mu }{L}-1}}{2L}\}$ and assume that $\parallel \nabla f\left(x^{k-2}\right)\parallel \le 2\parallel \nabla f\left(x^{k-1}\right)\parallel$ for some fix k. It follows that

$$\begin{array}{cc}\hfill \parallel \nabla f\left(x^{k-1}\right)\parallel & \le \parallel \nabla f\left(x^k\right)-\nabla f\left(x^{k-1}\right)\parallel +\parallel \nabla f\left(x^k\right)\parallel \hfill \\ \hfill & \le L\parallel x^k-x^{k-1}\parallel +\parallel \nabla f\left(x^k\right)\parallel \hfill \\ \hfill & =L\alpha \parallel \nabla f\left(x^{k-1}\right)+\nabla f\left(x^{k-2}\right)\parallel +\parallel \nabla f\left(x^k\right)\parallel \hfill \\ \hfill & \le L\alpha \parallel \nabla f\left(x^{k-1}\right)\parallel +\parallel \nabla f\left(x^{k-2}\right)\parallel +\parallel \nabla f\left(x^k\right)\parallel \hfill \\ \hfill & \le 3L\alpha \parallel \nabla f\left(x^{k-1}\right)\parallel +\parallel \nabla f\left(x^k\right)\parallel \hfill \\ \hfill & \le \frac{1}{2}\parallel \nabla f\left(x^{k-1}\right)\parallel +\parallel \nabla f\left(x^k\right)\parallel ,\hfill \end{array}$$

which means that $\parallel \nabla f\left(x^{k-1}\right)\parallel \le 2\parallel \nabla f\left(x^k\right)\parallel$. By induction and by the selections of initial points, it can be ensured that $\parallel \nabla f\left(x^{k-1}\right)\parallel \le 2\parallel \nabla f\left(x^k\right)\parallel$ for all $k\in \mathbb{N}$. As a consequence, by defining $g^k:=\nabla f\left(x^k\right)+\nabla f\left(x^{k-1}\right)$, we obtain $\parallel g^k-\nabla f\left(x^k\right)\parallel \le 2\parallel \nabla f\left(x^k\right)\parallel$. This is indeed similar to the inexact condition considered in [14] (Theorem 2 in Section 4.2.3), [15,16]. Note, that [14] considers the methods of the form

$$\begin{array}{c}\hfill x^{k+1}=x^k-{\alpha }_k{g}^k,where\parallel g^k-\nabla f\left(x^k\right)\parallel \le \eta \parallel \nabla f\left(x^k\right)\parallel ,\end{array}$$

and where $\eta \in (0,1)$ for the convex functions with Lipschitz continuous gradients. While [15,16] considers the methods of type $(0,\frac{2}{3})$ for smooth functions without convexity. The constant 2 in the extended gradient method can relax the inexact condition.

3. Analysis of the Decentralized Extended Gradient Method

The traditional (centralized) optimization methods are found on a large assumption that all data samples are stored and operated at a computing unit [17]. However, in the networked multi-agent systems, the data resources and function information related to optimization problems are stored in each agent distribution, and the distributed schemes are introduced into the framework of optimization [18]. In the distributed fashion, all data are transmitted to a fusion center that conducts the optimization operation and scatters the updated information to all agents. Nevertheless, the distributed operation is gradually replaced by the decentralized manner for the reasons of privacy and communication costs. The decentralized optimization is designed for the situations of networked systems which can fully utilize the resources of multiple computing units and improve the reliability of the networked systems.

In this section, we focus on distributed gradient methods. The earlier distributed gradient methods [19,20,21] slowly converge to the optimal solution by utilizing the diminishing step size. With the constant step size, these methods only converge to a neighborhood of the optimal solution. Recently, the distributed gradient methods [22,23,24] combined with more techniques can converge linearly to the optimal solution. The techniques include the multi-step consensus, the difference between the two consecutive iterates and the gradient tracking technique, which could cause more computation and communication burden. Thus, we try to investigate the behavior of the decentralized extended gradient method working with the simple update mechanism.

3.1. Decentralized Extended Gradient Method

This section develops a decentralized version of the extended gradient method for the following problem:

$$\underset{x\in {\mathbb{R}}^p}{min}f\left(x\right)=\frac{1}{n}\sum _{i=1}^n{f}_i\left(x\right),$$

(22)

where the local cost function $f_i$ is L-smooth and $\mu$-strongly convex, which is defined over a network consisting of n agents. To apply the extended gradient method to problem (22) under the decentralized situation, all the agents carry out the extended gradient method in parallel.

A set of local variables ${\{x_i\in {\mathbb{R}}^p\}}_{i=1}^n$ is introduced, and $x_i$ is assigned to agent i. Denote the aggregate variable X and the aggregate gradient $\nabla F\left(X\right)$ as $X={(x_1,x_2,\cdots ,x_n)}^{\top }$ and $\nabla F\left(X\right)={(\nabla f_1\left(x_1\right),\nabla f_2\left(x_2\right),\cdots ,\nabla f_n\left(x_n\right))}^{\top }$. The decentralized extended gradient method updates the iterates $X^{k+1}$:

$$X^{k+1}=W{X}^k-\alpha (\nabla F\left(X^K\right)+\nabla F\left(X^{k-1}\right)),$$

(23)

where the weight matrix W is symmetric and double stochastic, i.e., $W^{\top }=W$, $W{1}_n=1_n$ and $1_n^{\top }W=1_n^{\top }$, herein $1_n=(1,\cdots ,1)\in {\mathbb{R}}^p$. Weight the matrix W is generated based on the network. When the agent i and j are neighbors or $i=j$, $W_{ij}\ne 0$. In the decentralized setting, the local function $f_i$ is only accessed by agent i, $i=1,\cdots ,n$. All agents cooperate to optimize the global function f via local intra-node computation and extra-node communication. The local computation is operated on the individual agent, i.e., the local variable of agent i is updated by the gradient information of the local function $f_i$. The communication is operated over the network, i.e., the local variable is updated by combining the weighted average of its neighbor’s local variable, which enforces the consensus of all local variables.

3.2. Convergence Result

We need to define the average vector: ${\overline{X}}^k=\frac{1}{n}{1}_n^{\top }{X}^k=\frac{1}{n}{\sum }_{i=1}^n{x}_i^k\in {\mathbb{R}}^{1\times p}$. Then, we give the upper bounds on $\parallel {\overline{X}}^k-{\left(x^{*}\right)}^{\top }\parallel$ and $\parallel X^k-1_n{\overline{X}}^k\parallel$ by using the linear combinations of the values of the preceding two iterates.

Lemma 3.

Let ${\left\{X^k\right\}}_{k\in \mathbb{N}}$ be the iterates generated by the updates in (23). If $\xi \in (\alpha -\sqrt{1-(n-1)\alpha }$, $\alpha +\sqrt{1-(n-1)\alpha })$, then the following relation hold:

$$z^{k+1}\le A{z}^k,$$

(24)

where $z^k={\left(\parallel X^k-1_n{\overline{X}}^k\parallel ,\parallel {\overline{X}}^k-{\left(x^{*}\right)}^{\top }\parallel ,\parallel X^{k-1}-1_n{\overline{X}}^{k-1}\parallel ,\parallel {\overline{X}}^{k-1}-{\left(x^{*}\right)}^{\top }\parallel \right)}^{\top }\in {\mathbb{R}}^4$ and $A=\left(\begin{array}{cccc}\delta +\frac{\xi L\alpha \sqrt{n-1}}{\alpha -({\xi }^2-2\alpha \xi +n{\alpha }^2)}& \frac{\xi L\alpha \sqrt{n(n-1)}}{\alpha -({\xi }^2-2\alpha \xi +n{\alpha }^2)}& \frac{\xi L\alpha \sqrt{n-1}}{\alpha -({\xi }^2-2\alpha \xi +n{\alpha }^2)}& \frac{\xi L\alpha \sqrt{n(n-1)}}{\alpha -({\xi }^2-2\alpha \xi +n{\alpha }^2)}\\ \frac{\alpha L}{\sqrt{n}}& \lambda & \frac{\alpha L}{\sqrt{n}}& \alpha L\\ 1& 0& 0& 0\\ 0& 1& 0& 0\end{array}\right)\in {\mathbb{R}}^{4\times 4}$, here in $\lambda =max\left\{\right|1-\alpha L|,|1-\alpha \mu \left|\right\}$ and $\delta =\parallel W-\frac{1_n{1}_n^{\top }}{n}\parallel$.

If $\frac{1}{\mu +L}<\alpha <min\{\frac{2}{L},\frac{\sqrt{n}{\gamma }_2}{L({\gamma }_1+\sqrt{n}{\gamma }_2+{\gamma }_3+\sqrt{n}{\gamma }_4)}\}$ and $\xi \in (\alpha -\sqrt{1-(n-1)\alpha },min\{\alpha +\sqrt{1-(n-1)\alpha },\frac{\sqrt{b^2-4ac}}{2a}\})$, where $0<{\gamma }_1<{\gamma }_3$, $0<{\gamma }_2<{\gamma }_4$, $a=(1-\delta ){\gamma }_1$, $b=\alpha L\sqrt{n-1}({\gamma }_1+\sqrt{n}{\gamma }_2+{\gamma }_3+\sqrt{n}{\gamma }_4)+2(1-\delta )\alpha {\gamma }_1$ and $c=(1-\delta )(n{\alpha }^2-\alpha )$, then $\rho \left(A\right)<1$ and thus $\parallel X^k-1_n{\overline{X}}^k\parallel$ and $\parallel {\overline{X}}^k-{\left(x^{*}\right)}^{\top }\parallel$ converge to zero at the linear rate of $O\left(\rho {\left(A\right)}^k\right)$.

Proof. 

First, we establish a linear system by using the upper bounds on $\parallel X^{k+1}-1_n{\overline{X}}^{k+1}\parallel$ and $\parallel {\overline{X}}^{k+1}-{\left(x^{*}\right)}^{\top }\parallel$. Next, we can find the range of $\alpha$ such that $\rho \left(A\right)<1$. From the update in (23), we derive

$$\begin{array}{c}\hfill {\overline{X}}^{k+1}=\frac{1}{n}{1}_n^{\top }\left(W{X}^k-\alpha \left(\nabla F\left(X^k\right)+\nabla F\left(X^{k-1}\right)\right)\right)={\overline{X}}^k-\frac{\alpha }{n}{1}_n^{\top }\left(\nabla F\left(X^k\right)+\nabla F\left(X^{k-1}\right)\right).\end{array}$$

Step 1. Bound $\parallel X^{k+1}-1_n{\overline{X}}^{k+1}\parallel$.

From the above equation, it follows that

$$\begin{array}{cc}\hfill \parallel X^{k+1}-1_n{\overline{X}}^{k+1}\parallel & =\parallel (I-\frac{1}{n}{1}_n{1}_n^{\top })(W{X}^k-\alpha (\nabla F\left(X^K\right)+\nabla F\left(X^{k-1}\right)))\parallel \hfill \\ \hfill & \le \parallel (I-\frac{1}{n}{1}_n{1}_n^{\top })W{X}^k\parallel +\parallel I-\frac{1}{n}{1}_n{1}_n^{\top }\parallel \parallel \alpha \nabla F\left(X^K\right)+\alpha \nabla F\left(X^{k-1}\right)\parallel \hfill \end{array}$$

(25)

Notice that $\left(I-\frac{1}{n}{1}_n{1}_n^{\top }\right)W=\left(W-\frac{1}{n}{1}_n{1}_n^{\top }\right)\left(I-\frac{1}{n}{1}_n{1}_n^{\top }\right)$ and $\parallel I-\frac{1}{n}{1}_n{1}_n^{\top }\parallel =\sqrt{n-1}$. Therefore, it follows

$$\begin{array}{c}\hfill \parallel X^{k+1}-1_n{\overline{X}}^{k+1}\parallel \le \delta \parallel X^k-1_n{\overline{X}}^k\parallel +\sqrt{n-1}\left(\parallel \alpha \nabla F\left(X^k\right)\parallel +\parallel \alpha \nabla F\left(X^{k-1}\right)\parallel \right),\end{array}$$

(26)

where $\delta =\parallel W-\frac{1}{n}{1}_n{1}_n^{\top }\parallel <1$. For the term $\parallel \alpha \nabla F\left(X^k\right)\parallel$, we can further bound as

$$\begin{array}{cc}\hfill \parallel \alpha \nabla F\left(X^k\right)\parallel =& \parallel \alpha \nabla F\left(X^k\right)-\frac{\xi }{n}{1}_n{1}_n^{\top }\nabla F\left(X^k\right)+\frac{\xi }{n}{1}_n{1}_n^{\top }\nabla F\left(X^k\right)-\xi 1_n\nabla f{\left({\overline{X}}^k\right)}^{\top }\hfill \\ \hfill & +\xi 1_n\nabla f{\left({\overline{X}}^k\right)}^{\top }-\xi 1_n\nabla f{\left(x^{*}\right)}^{\top }\parallel \hfill \\ \hfill \le & \parallel (\alpha I-\frac{\xi }{n}{1}_n{1}_n^{\top })\nabla F\left(X^k\right)\parallel +\xi \parallel 1_n{\left(\frac{1}{n}\sum _{i=1}^n\left(\nabla f_i\left(x_i^k\right)-\nabla f_i\left({\overline{X}}^k\right)\right)\right)}^{\top }\parallel \hfill \\ \hfill & +\xi \parallel 1_n{\left(\frac{1}{n}\sum _{i=1}^n\left(\nabla f_i\left({\overline{X}}^k\right)-\nabla f_i\left(x^{*}\right)\right)\right)}^{\top }\parallel \hfill \\ \hfill \le & \parallel \alpha I-\frac{\xi }{n}{1}_n{1}_n^{\top }\parallel \parallel \nabla F\left(X^k\right)\parallel +\xi \parallel 1_n\parallel \parallel \frac{1}{n}\sum _{i=1}^n\left(\nabla f_i\left(x_i^k\right)-\nabla f_i\left({\overline{X}}^k\right)\right)\parallel \hfill \\ \hfill & +\xi \parallel 1_n\parallel \parallel \frac{1}{n}\sum _{i=1}^n\left(\nabla f_i\left({\overline{X}}^k\right)-\nabla f_i\left(x^{*}\right)\right)\parallel \hfill \\ \hfill \le & ({\xi }^2-2\alpha \xi +n{\alpha }^2)\parallel \nabla F\left(X^k\right)\parallel +\xi \sqrt{n}\frac{1}{n}\sum _{i=1}^n\parallel \nabla f_i\left(x_i^k\right)-\nabla f_i\left({\overline{X}}^k\right)\parallel \hfill \\ \hfill & +\xi \sqrt{n}\frac{1}{n}\sum _{i=1}^n\parallel \nabla f_i\left({\overline{X}}^k\right)-\nabla f_i\left(x^{*}\right)\parallel \hfill \\ \hfill \le & ({\xi }^2-2\alpha \xi +n{\alpha }^2)\parallel \nabla F\left(X^k\right)\parallel +\xi L\sqrt{n}\frac{1}{n}\sum _{i=1}^n\parallel x_i^k-{\overline{X}}^k\parallel \hfill \\ \hfill & +\xi L\sqrt{n}\parallel {\overline{X}}^k-{\left(x^{*}\right)}^{\top }\parallel \hfill \\ \hfill \le & ({\xi }^2-2\alpha \xi +n{\alpha }^2)\parallel \nabla F\left(X^k\right)\parallel +\xi L\sqrt{n}\sqrt{\frac{{\sum }_{i=1}^n{\parallel x_i^k-{\overline{X}}^k\parallel }^2}{n}}\hfill \\ \hfill & +\xi L\sqrt{n}\parallel {\overline{X}}^k-\left(x^{*}\right)\top \parallel \hfill \\ \hfill \le & ({\xi }^2-2\alpha \xi +n{\alpha }^2)\parallel \nabla F\left(X^k\right)\parallel +\xi L\parallel X^k-1_n{\overline{X}}^k\parallel +\xi L\sqrt{n}\parallel {\overline{X}}^k-{\left(x^{*}\right)}^{\top }\parallel ,\hfill \end{array}$$

(27)

where choosing $\xi \in (\alpha -\sqrt{1-(n-1)\alpha },\alpha +\sqrt{1-(n-1)\alpha })$ such that $\alpha -({\xi }^2-2\alpha \xi +n{\alpha }^2)>0$. So

$$\begin{array}{c}\hfill \parallel \nabla F\left(X^k\right)\parallel \le \frac{1}{\alpha -({\xi }^2-2\alpha \xi +n{\alpha }^2)}(\xi L\parallel X^k-1_n{\overline{X}}^k\parallel +\xi L\sqrt{n}\parallel {\overline{X}}^k-{\left(x^{*}\right)}^{\top }\parallel ).\end{array}$$

(28)

Similarly, we have

$$\begin{array}{c}\hfill \parallel \nabla F\left(X^{k-1}\right)\parallel \le \frac{1}{\alpha -({\xi }^2-2\alpha \xi +n{\alpha }^2)}(\xi L\parallel X^{k-1}-1_n{\overline{X}}^{k-1}\parallel +\xi L\sqrt{n}\parallel {\overline{X}}^{k-1}-\left(x^{*}\right)\top \parallel ).\end{array}$$

(29)

Substituting (28) and (29) into (26) yields

$$\begin{array}{cc}\hfill \parallel X^{k+1}-1_n{\overline{X}}^{k+1}\parallel \le & (\delta +\frac{\xi L\alpha \sqrt{n-1}}{\alpha -({\xi }^2-2\alpha \xi +n{\alpha }^2)})\parallel X^k-1_n{\overline{X}}^k\parallel \hfill \\ \hfill & +\frac{\xi L\alpha \sqrt{n(n-1)}}{\alpha -({\xi }^2-2\alpha \xi +n{\alpha }^2)}\parallel {\overline{X}}^k-{\left(x^{*}\right)}^{\top }\parallel \hfill \\ \hfill & +\frac{\xi L\alpha \sqrt{n-1}}{\alpha -({\xi }^2-2\alpha \xi +n{\alpha }^2)}\parallel X^{k-1}-1_n{\overline{X}}^{k-1}\parallel \hfill \\ \hfill & +\frac{\xi L\alpha \sqrt{n(n-1)}}{\alpha -({\xi }^2-2\alpha \xi +n{\alpha }^2)}\parallel {\overline{X}}^{k-1}-{\left(x^{*}\right)}^{\top }\parallel .\hfill \end{array}$$

(30)

Step 2. Bound $\parallel \overline{X}-{\left(x^{*}\right)}^{\top }\parallel$.

$$\begin{array}{cc}\hfill \parallel {\overline{X}}^{k+1}-{\left(x^{*}\right)}^{\top }\parallel =& \parallel {\overline{X}}^k-\alpha \frac{1_n^{\top }}{n}(\nabla F\left(X^k\right)+\nabla F\left(X^{k-1}\right))-{\left(x^{*}\right)}^{\top }\parallel \hfill \\ \hfill =& \parallel {\overline{X}}^k-\alpha \frac{1_n^{\top }}{n}\nabla F\left(1_n{\overline{X}}^k\right)-{\left(x^{*}\right)}^{\top }+\alpha \frac{1_n^{\top }}{n}(\nabla F\left(1_n{\overline{X}}^k\right)-\nabla F\left(X^k\right))\hfill \\ \hfill & -\alpha \frac{1_n^{\top }}{n}\nabla F\left(X^{k-1}\right)\parallel \hfill \\ \hfill \le & \parallel {\overline{X}}^k-\alpha \nabla f\left({\overline{X}}^k\right)-{\left(x^{*}\right)}^{\top }\parallel +\alpha \parallel \frac{1_n^{\top }}{n}(\nabla F\left(1_n{\overline{X}}^k\right)-\nabla F\left(X^k\right))\parallel \hfill \\ \hfill & +\alpha \parallel \frac{1_n^{\top }}{n}\nabla F\left(X^{k-1}\right)\parallel .\hfill \end{array}$$

(31)

Consider the first term in (31), we have

$$\begin{array}{c}\hfill \parallel {\overline{X}}^k-\alpha \nabla f\left({\overline{X}}^k\right)-{\left(x^{*}\right)}^{\top }\parallel \le \lambda \parallel {\overline{X}}^k-{\left(x^{*}\right)}^{\top }\parallel ,\end{array}$$

(32)

where $\lambda =max\left\{\right|1-\alpha L|,|1-\alpha \mu \left|\right\}$. For the second term and third term in (31), we bound them as

$$\begin{array}{c}\hfill \parallel \frac{1_n^{\top }}{n}(\nabla F\left(1_n{\overline{X}}^k\right)-\nabla F\left(X^k\right))\parallel \le \frac{L}{\sqrt{n}}\parallel X^k-1_n{\overline{X}}^k\parallel .\end{array}$$

(33)

and

$$\begin{array}{cc}\hfill \parallel \frac{1_n^{\top }}{n}\nabla F\left(X^{k-1}\right)\parallel =& \parallel \frac{1_n^{\top }}{n}\nabla F\left(X^{k-1}\right)-\frac{1_n^{\top }}{n}\nabla F\left(1_n{\overline{X}}^{k-1}\right)+\frac{1_n^{\top }}{n}\nabla F\left(1_n{\overline{X}}^{k-1}\right)\hfill \\ \hfill & -\frac{1_n^{\top }}{n}\nabla F\left(1_n{\left(x^{*}\right)}^{\top }\right)\parallel \hfill \\ \hfill \le & \frac{L}{\sqrt{n}}\parallel X^{k-1}-1_n{\overline{X}}^{k-1}\parallel +L\parallel {\overline{X}}^{k-1}-{\left(x^{*}\right)}^{\top }\parallel ,\hfill \end{array}$$

(34)

due to the smoothness of global function f. Substituting (32)–(34) into (31), it holds

$$\begin{array}{cc}\hfill \parallel {\overline{X}}^{k+1}-{\left(x^{*}\right)}^{\top }\parallel \le & \frac{L\alpha }{\sqrt{n}}\parallel X^k-1_n{\overline{X}}^k\parallel +\lambda \parallel {\overline{X}}^k-{\left(x^{*}\right)}^{\top }\parallel \hfill \\ \hfill & +\frac{L\alpha }{\sqrt{n}}\parallel X^{k-1}-1_n{\overline{X}}^{k-1}\parallel +L\alpha \parallel {\overline{X}}^{k-1}-{\left(x^{*}\right)}^{\top }\parallel .\hfill \end{array}$$

(35)

The linear system in (24) is obtained by using the results of (30) and (35).

Next, According to Lemma 1, we give the range of $\alpha$ and $\xi$ and a positive vector $\gamma ={({\gamma }_1,{\gamma }_2,{\gamma }_3,{\gamma }_4)}^{\top }$ such that $A\gamma <\gamma$, which is equivalent to the following inequalities

$$\begin{array}{c}\hfill \xi L\alpha \sqrt{n-1}{\gamma }_1+\xi L\alpha \sqrt{n(n-1)}{\gamma }_2+\xi L\alpha \sqrt{n-1}{\gamma }_3+\xi L\alpha \sqrt{n(n-1)}{\gamma }_4\\ \hfill <(1-\delta )(\alpha -({\xi }^2-2\alpha \xi +n{\alpha }^2)){\gamma }_1,\end{array}$$

(36)

$$\begin{array}{cc}\hfill \frac{L\alpha }{\sqrt{n}}{\gamma }_1+\frac{L\alpha }{\sqrt{n}}{\gamma }_3+\alpha L{\gamma }_4& <(1-\lambda ){\gamma }_2,\hfill \end{array}$$

(37)

$$\begin{array}{cc}\hfill {\gamma }_1& <{\gamma }_3,\hfill \end{array}$$

(38)

$$\begin{array}{cc}\hfill {\gamma }_2& <{\gamma }_4.\hfill \end{array}$$

(39)

If $\frac{1}{L+\mu }<\alpha <\frac{2}{L}$, according to the definition that $\lambda =max\left\{\right|1-\alpha \mu |,|1-\alpha L\left|\right\}$, we have that $l-\lambda =2-\alpha L>0$. Since $\alpha$ satisfies (37), we require

$$\begin{array}{c}\hfill \frac{1}{\mu +L}<\alpha <min\{\frac{2}{L},\frac{\sqrt{n}{\gamma }_2}{L({\gamma }_1+\sqrt{n}{\gamma }_2+{\gamma }_3+\sqrt{n}{\gamma }_4)}\}\end{array}$$

(40)

The range of $\xi$ follows (36), that is,

$$\begin{array}{c}\hfill \xi \in (\alpha -\sqrt{1-(n-1)\alpha },min\{\alpha +\sqrt{1-(n-1)\alpha },\frac{\sqrt{b^2-4ac}}{2a}\}),\end{array}$$

(41)

where $a=(1-\delta ){\gamma }_1$, $b=\alpha L\sqrt{n-1}({\gamma }_1+\sqrt{n}{\gamma }_2+{\gamma }_3+\sqrt{n}{\gamma }_4)+2(1-\delta )\alpha {\gamma }_1$ and $c=(1-\delta )(n{\alpha }^2-\alpha )$. Then, consider that ${\gamma }_1,{\gamma }_2,{\gamma }_3$ and ${\gamma }_4$ are positive, we can select proper ${\gamma }_1$, ${\gamma }_2$, ${\gamma }_3(>{\gamma }_1)$ and ${\gamma }_4(>{\gamma }_2)$ to make the set in (41) be nonempty. □

4. Numerical Experiments

In this section, we perform computational experiments on both the centralized and the decentralized problems to demonstrate the efficiency of the extended gradient method. We compare our proposed method with many well-known algorithms. All algorithms are programmed in Matlab (R2018b, MathWorks, Natick, MA, USA) and run on a PC (Lenovo-M430, Beijing, China) with a 2.90 GHz Intel Core i5-10400 CPU and 8 GB of memory.

4.1. Centralized Problem

We consider the performance of the extended gradient method when it is applied to the LASSO problem:

$$\begin{array}{c}\hfill \underset{x\in {\mathbf{R}}^p}{min}f\left(x\right)=\frac{1}{2}{\parallel Mx-b\parallel }^2+\nu {\parallel x\parallel }_1,\end{array}$$

(42)

where the random matrix $M\in {\mathbb{R}}^{m\times p}$ is generated by the uniform distribution $U(0,1)$, $b=Mx$ where x also follows an uniform distribution $U(0,1)$ with sparsity r. In particular, we set the parameters $m=512$, $p=1024$, $r=0.1$, and the initial condition $x^0\in U(0,1)$. The termination criterion is set to reach the maximum number of iterations or $|f\left(x^k\right)-f\left(x^{k-1}\right)|\le {10}^{-15}$. When the regularization parameter $\nu$ is zero, the LASSO problem becomes the least squares problem. Otherwise, the objective function $f\left(x\right)$ is nonsmooth. Considering that the nonsmooth term ${\parallel x\parallel }_1$ is actually the sum of the absolute values of each component of x, we employ the proximal gradient method and the approximation method based on the Huber function, respectively, to solve the LASSO problem. The Huber function becomes closer to the absolute value function as the parameter $\theta$ approaches 0.

$$\begin{array}{c}\hfill \begin{array}{c}\hfill h_{\theta }=\left\{\begin{array}{cc}\frac{1}{2\theta }{x}^2,\hfill & |x_i|<\theta \hfill \\ |x_i|-\frac{\theta }{2},\hfill & otherwise\hfill \end{array}\right.\end{array}\end{array}$$

(43)

In order to study the influence of the previous gradients on the search direction of the optimization algorithm, we employ the gradient (denoted as $g_1$), the sum of the gradients of the preceding two iterates (denoted as $g_2$), and the sum of the gradients of the preceding three iterates (denoted as $g_3$) to construct search directions, respectively. Figure 1 shows the performance of the gradient descent method(GD) for the least square problem, the proximal gradient method(PG) and the approximation method based on the Huber function (AH) for the LASSO problem ($\nu =0.001$) when the above three search directions are employed. We set the parameter $\theta$ as 0.02. For these algorithms with the three search directions, we set the step sizes as $\frac{1}{L}$, $0.8\frac{1}{L}$ and $0.4\frac{1}{L}$, respectively. Observing that when the sum of the gradients of the preceding two iterates ($g_2$) is selected to construct the search direction, the optimization algorithms achieve better acceleration than them with the other two search directions ($g_1$ and $g_3$).

Figure 1. Performance of optimization algorithms with three search directions.

On the other hand, we compare the extended gradient method (our algorithm) with the gradient descent method, heavy ball method (iPiasco) [2] and Nesterov accelerated gradient descent method [5]. Specifically, for the gradient descent method, the step size $\alpha$ is $\frac{1}{L}$. For the extended gradient method, the step size $\alpha$ is $0.8\frac{1}{L}$. For the heavy ball method, the step size $\alpha$ is $\frac{4}{{(\sqrt{\mu }+\sqrt{L})}^2}$ and the extrapolation parameter $\beta$ is ${\left(\frac{\sqrt{L}-\sqrt{\mu }}{\sqrt{\mu }+\sqrt{L}}\right)}^2$. For the Nesterov accelerated gradient descent method, the extrapolation parameter is $\frac{k-2}{k+1}$ and step size $\alpha$ is $\frac{1}{L}$. Figure 2 depicts the comparison result of all compared methods for the least squares problem. We notice that the extended gradient method outperforms the other algorithms in terms of the number of iterations. Moreover, in order to show that the sum of two gradients ($g_2$) can accelerate the convergence of algorithms, we use it to construct the new search direction of the Nesterov accelerated gradient descent method and iPiasco. We set step sizes $\alpha$ and $\nu$ as $0.8\frac{1}{L}$ and $0.001$. From Figure 3, we observe that when using the sum of two gradients to construct the new search direction, both the proximal Nesterov accelerated gradient descent method and the proximal iPiasco can achieve better acceleration performance.

Figure 2. Comparison of different algorithms for the least squares problem.

Figure 3. Performance of Nesterov accelerated gradient descent method and iPiasco with two search directions.

4.2. Decentralized Problem

We illustrate the performance of the decentralized extended gradient method ($DG{D}_{g2}$) on the following problem

$$\begin{array}{c}\hfill \underset{x\in {\mathbf{R}}^p}{min}f\left(x\right)=\sum _{i=1}^n{f}_i\left(x\right),\mathrm{with}{f}_i\equiv \frac{1}{2}\parallel M_{i}x-b_i{\parallel }^2+\nu {\parallel x\parallel }_1,\end{array}$$

(44)

where the random matrix $M_i\in {\mathbb{R}}^{m\times p}$ is generated by the uniform distribution $U(0,1)$, and each column of $M_i$ is normalized to be 1. We set the parameters $m=512$, $p=1024$, $r=0.1$, $\nu =0.0001$, and $b_i=M_{i}x$ where x also follows the uniform distribution $U(0,1)$ with sparsity r. The undirected graph is generated by Erdös-Rényi model [25] where each pair of nodes has the connection probability $p=0.8$, and the weight matrix $W=\frac{I+M}{2}$, here M is generated with the Metropolis constant edge rules [26]. To demonstrate the effect of the previous gradient on the search direction of decentralized optimization algorithms, we use the sum of the gradients of the preceding two iterates ($g_2$) to construct the search direction in the accelerated penalty method with consensus ($AP{M}_C$) [27], the decentralized gradient descent (DGD) [21] and the decentralized stochastic gradient descent (DSGD) [28]. For DGD, DSGD, $DG{D}_{g2}$ and $DSG{D}_{g2}$, we tune the optimized step size by hand, which are $0.09$, $0.9$, $0.058$ and $0.58$, respectively. For $AP{M}_C$ and $AP{M}_{Cg2}$, the step sizes are $\frac{1}{L}$ and $0.1\frac{1}{L}$. The initial condition $X^0$ is $0_{n\times p}$. Compared to the original $AP{M}_C$, DGD and DSGD, we can see from Figure 4 that the algorithms with the gradients of the preceding two iterates ($g_2$) to construct search direction outperform than those with the current negative gradient as the search direction.

Figure 4. Comparison of different algorithms for the LASSO problem.

Finally, we extend the proposed method to a classification problem. First, we introduce the Radial Basis Function Network (RBFN) that uses a radial basis function $\psi (s;c)$ as the activation function, where s is a sample and c is a center. The output of the RBFN is a linear combination of radial basis functions $\psi (s;c)$ and the neuron weight parameters x. The training error that evaluates the performance of the RBFN can be written as (44). The training samples are $\{(s_1,1),\cdots ,(s_N/2,1),(s_{N/2+1},-1),\cdots ,(s_N,-1)\}$. The centers $\{c_1,\cdots ,c_p\}$ are randomly chosen from the training samples. The radial basis function is Gauss function $\psi (s,c)=e^{\frac{{\parallel s-c\parallel }^2}{2}}$. The matrix $M_i$ represents $\left[\begin{array}{ccc}\psi (s_{i_1};c_1)& \cdots & \psi (s_{i_{N/n}};c_1)\\ ⋮& \ddots & ⋮\\ \psi (s_{i_1};c_p)& \cdots & \psi (s_{i_{N/n}};c_p)\end{array}\right]$, and $b_i\in {\mathbb{R}}^{N/n}$ is the corresponding sample label vector. We set $N=4000$, $p=300$ and $n=20$. Each node holds samples set ${\{s_{i_m},b_{i_m}\}}_{m=1}^{200}$. For the decentralized extended gradient method, the step size is set as $\frac{1}{L+\mu }$. Figure 5 shows the classification result, and the accuracy of classing the samples is 1, which further indicates the efficiency of our method.

Figure 5. Classification result of the extended gradient method.

5. Conclusions

The gradient methods have been widely employed to solve the optimization problems. In this paper, we first introduce an extended gradient method for the centralized and decentralized smooth and strongly convex problems. Secondly, we establish the linear convergence of the extended gradient method in a centralized and decentralized manner, respectively. Finally, the numerical examples are used to demonstrate the convergence and to validate the efficiency compared to current classical methods. In the future, the acceleration technique used in this work can be applied to more optimization problems.