A Distributed Optimization Accelerated Algorithm with Uncoordinated Time-Varying Step-Sizes in an Undirected Network

Abstract

In recent years, significant progress has been made in the field of distributed optimization algorithms. This study focused on the distributed convex optimization problem over an undirected network. The target was to minimize the average of all local objective functions known by each agent while each agent communicates necessary information only with its neighbors. Based on the state-of-the-art algorithm, we proposed a novel distributed optimization algorithm, when the objective function of each agent satisfies smoothness and strong convexity. Faster convergence can be attained by utilizing Nesterov and Heavy-ball accelerated methods simultaneously, making the algorithm widely applicable to many large-scale distributed tasks. Meanwhile, the step-sizes and accelerated momentum coefficients are designed as uncoordinate, time-varying, and nonidentical, which can make the algorithm adapt to a wide range of application scenarios. Under some necessary assumptions and conditions, through rigorous theoretical analysis, a linear convergence rate was achieved. Finally, the numerical experiments over a real dataset demonstrate the superiority and efficacy of the novel algorithm compared to similar algorithms.

1. Introduction

In recent years, with the rapid development of artificial intelligence, big data, etc., there has been much attention to distributed optimization problems in multi-agent systems. As one of the most important fields, distributed optimization methods have gained significant growing interest due to the widespread applications in science and engineering areas such as the transmission of information in wireless sensor networks [1,2,3], the collaboration of vehicles in formation control [4,5], speeding up the optimization process in distributed machine learning [6,7], distributed resource allocation in smart-grid networks [8,9,10], distributed control in nonlinear dynamical systems [11,12], etc. Specifically, a distributed optimization framework can avoid the establishment of long-distance communication between agents while providing better load balancing for the network. Compared to traditional centralized optimization, agents in a multi-agent system communicate information only with their neighbors for distributed optimization. At the same time, the local objective function of each agent is known only by itself.

Literature Review: Since the DGD (Distributed gradient descent) algorithm was proposed by Nedic [13] for solving distributed convex problem in multiagent systems, great progress has been made in the distributed optimization field. Especially, the distributed first-order methods have attracted many researchers’ attention. Based on consensus theory [14] and gradient-descent technology, diminishing step-sizes were introduced into the algorithm DGD [13], which made the algorithm converge to the exact optimal solution but with a sublinear rate. When there were constraints of a decision variable, by utilizing the projection method, Sundhar [15] proposed a stochastic subgradient projection algorithm. Similar to [13,15], refs. [16,17,18] also employed the diminishing step-sizes, and these algorithms could converge linearly. However, diminishing step-sizes will lead to a much slower convergence rate. Then, the distributed algorithms with constant step-sizes were developed in [19,20,21,22,23,24,25,26,27,28,29,30,31,32] to overcome the shortcoming. The algorithm EXTRA [19] (Extra: An exact first-order algorithm for decentralized consensus optimization) and its improvement [20,21,22] modified the update rule of DGD by taking the difference of two consecutive iterations of formulas. Compared to DGD, the linear convergence rate can be verified in EXTRA, and even the step-size was fixed to a constant; EXTRA was more stable, but two weight matrices in EXTRA must obey strict conditions called the Mixing Matrix. A different type of distributed optimization algorithm HSADO (harnessing smoothness to accelerate distributed optimization) was proposed by Qu and Li [26] when the local objective functions were convex and smooth. HSADO adopted a gradient-tracking mechanism, which replaced the gradient term in DGD with a tracking gradient that was the gradient estimation of the average gradient of the whole network. If the step-sizes were set to constants, HSADO also can converge to the optimal solution linearly. Based on HSADO, researchers modified to adapt different scenarios, such as time-varying networks [27,28] and node-varying [29,30] and accelerated methods [31,32]. Further, researchers studied the primal-dual method in distributed optimization by utilizing the Augmented Lagrangian function; the original problem was reformulated in a dual problem. It has been demonstrated that EXTRA was equivalent to the algorithms in  [33,34] by introducing dual variables, and  [27] also provided a primal-dual interpretation for HSADO. Recently, the primal-dual algorithm UG (A unification and generalization of exact distributed first-order methods) proposed in [35] unified and generalized the methods EXTRA and HSADO, while it also converged linearly.

Motivations: Among these studies, EXTRA, HSADO, and UG are most related to our research. The algorithm UG can be regard as a generalization and unification of DGD, EXTRA, and HSADO. However, each local objective function of the agent in the network requires to be twice continuously differentiable, which is a rigorous condition in actual scenarios. In order to obtain linear convergence, fixed constant step-sizes were frequently adopted in distributed optimization algorithms such as  [19,26,35], etc. Unfortunately, uncoordinated step-sizes for different agents are required rather than the same constant step-sizes. This situation was first studied in  [36], in which an augmented distributed-gradient method was proposed, but it converged sublinearly. Then, by employing uncoordinated step-sizes, Lü [30] and Jakovetic [28] both established a global linear convergence of their algorithms in time-varying undirected and directed networks, respectively. To endow more independence, time-varying and nonidentical step-sizes of each agent were studied. A primal-dual fixed-point algorithm with nonidentical step-sizes was proposed by Li [37] when the object function of each agent was twice differentiable and nonsmooth. Xin  [32] also adopted nonidentical step-sizes in a directed network. With a more relaxed step-size and network topolog, a distributed primal-dual optimization method in  [38] was proposed by utilizing time-varying step-sizes, which was proved to converge linearly. Until now, to the best knowledge of the authors, no related studies for the widely used algorithm UG with uncoordinated, time-varying, and nonidentical step-sizes in an undirected network were studied. Recently, as optimization processes of large-scale tasks such as deep learning are getting slower, the convergence rate of distributed optimization algorithms need further improvement. With the help of Nesterov [39] and Heavy-ball [40] accelerated methods, the convergence rate of distributed optimization algorithms can be improved. In [32], a Heavy-ball distributed accelerated method with gradient-tracking technology was proposed to accelerate the well-known row-stochastic and column-stochastic algorithm [41]. In [31,42], a better convergence rate was shown by utilizing the Nesterov accelerated method. Moreover, both the Heavy-ball and Nesterov accelerated methods were introduced to improve the convergence rate in directed networks for machine learning in  [43]. For the widely used algorithm UG, it is challenging to study whether the simultaneous inclusion of Heavy-ball and Nesterov momentum can bring about a faster convergence rate in large-scale computing and communication tasks.

Statement of Contributions: Throughout this article, we mainly focus on the application of distributed convex optimization method over an undirected network. We propose a novel distributed optimization algorithm with uncoordinated, time-varying, and nonidentical step-sizes and accelerated momentum terms, which has a faster linear convergence rate and can apply to more scenarios. To summarize, three contributions are as follows:

• Based on the distributed optimization methods [19,26,35], we designed and discussed a faster distributed optimization accelerated algorithm, named UGNH (UG with Nesterov and Heavy-ball accelerated methods), which solves the distributed convex problems over an undirected network. In particular, the momentum with the Nesterov and Heavy-ball methods together improve the convergence rate, which can be seen in the numerical experiments.
• Compared to related algorithms, in our algorithm, not only the step-sizes but the coefficients of momentum terms (for convenience, we call them coefficients for short later) are uncoordinated, time-varying, and nonidentical, which are locally chosen for each agent. Through convergence analysis, the step-sizes and coefficients are more flexible than most existing methods. Meanwhile, if the local objective functions satisfy the conditions that are smooth and strongly convex, we can obtain an upper bound of step-sizes and coefficients. Under the upper bounds, the sequences generated by UGNH converge to the exact optimal solutions linearly.
• In contrast to related algorithms, the upper bounds of the largest step-sizes and coefficients of UGNH are more relaxed, which only depend on the parameters of objective functions and the topology of the network. Meanwhile, there can be zero step-sizes and coefficients (not all) among agents.

Organization: The rest of this article is arranged as follows. In Section 2, we describe the distributed problem and provide some necessary assumptions. In Section 3, we discuss the development of relevant distributed optimization algorithms and two classical accelerated methods and then propose a new distributed accelerated algorithm. Convergence analysis is detailed in Section 4. In Section 5, numerical experiments are provided to demonstrate the superiority and efficiency of our algorithm. Finally, Section 6 concludes this article and provides some research directions for the future.

Basic Notation: Throughout the rest of this article, unless otherwise specified, all vectors are considered as column vectors, and n is the number of agents in network. The real-number set, the natural-number set, and the m-dimensional real column vector are denoted by $\mathbb{R}$, $\mathbb{N}$, and ${\mathbb{R}}^m$, respectively. The subscript notations $i,j\in \left\{1,2,\cdots ,n\right\}$ represent the indices of the agents, while the superscript notation t represents an index for the iteration step, e.g., $x_i^t$ represents the ith agent’s decision variable at the jth iteration; $0_n\in {\mathbb{R}}^n$, $1_n\in {\mathbb{R}}^n$, and $I_n\in {\mathbb{R}}^{n\times n}$ denote an n-dimensional zero vector, one vector, and an identity matrix, respectively. For a matrix P, $p_{ij}$ denotes the element at the i-th row and the j-th column of P, while its spectral radius and spectral norm are defined as $\rho \left(P\right)$ and $\parallel P\parallel$, respectively. Similarly, $\parallel x\parallel$ denotes the 2-norm for vector x. The transpose of a vector x and a matrix P are denoted by $x^T$ and $P^T$, respectively. For a vector $r={\left[r_1,r_2,\cdots ,r_n\right]}^T$, diag(r) represents a diagonal matrix, the diagonal elements of which equal to the vector r. The notation ⊗ represents the Kronecker product. Let $\nabla f\left(x\right):{\mathbb{R}}^m\to {\mathbb{R}}^m$ denote the gradient of $f\left(x\right)$ at x.

2. Preliminaries

This section describes the formulation of the distributed optimization problem and some necessary basic assumptions related to network and function.

2.1. Problem Formulation

Consider an undirected network of n agents, which cooperatively solve the optimization problem written in the following form over a common variable $x\in {\mathbb{R}}^m$:

$$\begin{array}{c}\hfill \underset{x\in {\mathbb{R}}^m}{min}f\left(x\right)=\frac{1}{n}\sum _{i=1}^n{f}_i\left(x\right)\end{array}$$

(1)

Here, each local objective function $f_i:{\mathbb{R}}^m\to \mathbb{R}$ with a convex property is possessed by agent i, which exchanges local information only with its neighbors. Our main target was to design a distributed optimization algorithm, a decision variable of which can linearly converge to the optimal solution that minimizes the average of all local objective functions. The optimal average objective value of problem (1) is defined as $f\left({\tilde{x}}^{*}\right)$, where ${\tilde{x}}^{*}\in {\mathbb{R}}^m$ is the optimal decision variable. Then, the global optimal solution of (1) is denoted by $x^{*}\in {\mathbb{R}}^{nm}$, where $x^{*}=1_n\otimes {\tilde{x}}^{*}$.

As a local copy of the global decision variable is saved at each agent, optimization problem (1) can be solved in a distributed way by iterating the decision variable. In this study, network is described as $\mathcal{G}=\left\{V,E\right\}$, where $V=\left\{1,2,\cdots ,n\right\}$ is the vertex set that represents the agents of the network, and $E=\left\{\left(i,j\right)|i,j\in V\right\}$ is the edges set. In an undirected network, an edge $\left(i,j\right)\in E$ implies that an edge $\left(j,i\right)\in E$ too. Meanwhile, agent i and agent j can exchange information with each other. Let $N_i=\left\{j|\left(i,j\right)\in E\right\}\cup \left\{i\right\}$ denote the set of all neighbors of agent i. Then, formulation (1) can be rewritten as follows:

$$\begin{array}{c}\hfill \underset{x\in {\mathbb{R}}^{nm}}{min}f\left(x\right)=\frac{1}{n}\sum _{i=1}^n{f}_i\left(x_i\right)\end{array}$$

(2)

where $x={\left[x_1^T,x_2^T,\cdots ,x_n^T\right]}^T\in {\mathbb{R}}^{nm}$, $x_i=x_j$ for $\forall i,j\in V$. Recently, it has been proved that a new equality $\frac{1}{\alpha }{L}^{\frac{1}{2}}x=0$ is equivalent to the consensus condition $x_i=x_j$ in  [33], where $\alpha$ is the step-size, and $L=I-P$ is a Laplace matrix. Then, the primal-dual method can be introduced to solve (2) by utilizing the Augmented Lagrangian function, which is also a cornerstone of our algorithm.

Next, some necessary assumptions about the underlying graph and local objective functions are formalized, which are a common standard in related distributed optimization studies.

2.2. Assumptions

Assumption 1

([35]). The network $\mathcal{G}=\left\{V,E\right\}$ is connected, undirected, and simple. In particular, there are no self-loops of any agent and no multiple links between any two agents.

Assumption 2

([19]). A non-negative symmetric doubly stochastic weight matrix $P=\left\{p_{ij}\right\}\in {\mathbb{R}}^{n\times n}$ is defined to represent network $\mathcal{G}$. The weight of the matrix P satisfies the following three conditions:

• Non-negative: $p_{ij}=\left\{\begin{array}{c}>0,j\in N_i\hfill \\ =0,otherwise\hfill \end{array}\right.$
• Symmetric: $p_{ij}=p_{ji}$
• Doubly stochastic: ${\sum }_{i=1}^n{p}_{ij}={\sum }_{j=1}^n{p}_{ij}=1$

Assumption 3.

Each local objective function $f_i:{\mathbb{R}}^m\to \mathbb{R},i\in V$ is smooth with Lipschitz constant ${\psi }_i$ and strongly convex with parameter ${\mu }_i$. Mathematically, there exits ${\psi }_i>0$, ${\mu }_i\ge 0\left(\sum {\mu }_i>0\right)$, for any $x,y\in {\mathbb{R}}^m$ such that:

$$\begin{array}{cc}\hfill & \parallel \nabla f_i\left(x\right)-\nabla f_i\left(y\right)\parallel \le {\psi }_i\parallel x-y\parallel \hfill \\ \hfill & f_i\left(x\right)-f_i\left(y\right)\ge \nabla f_i{\left(x\right)}^T\left(x-y\right)+\frac{{\mu }_i}{2}\parallel x-y\parallel \hfill \end{array}$$

Remark 1.

Assumption 1 ensures that each agent can directly or indirectly affect other agents in the network. Assumption 3 is a standard assumption in convergence analysis of distributed optimization methods. Especially, under the assumption of strongly convex for each function, there exists a unique global optimal solution to problem (1). Moreover, for the global objective function f, we define $\overline{\psi }=\left(\frac{1}{n}\right){\sum }_{i=1}^n{\psi }_i$ as the global Lipschitz constant and $\overline{\mu }=\left(\frac{1}{n}\right){\sum }_{i=1}^n{\mu }_i$ as the global strongly convex parameter.

3. Algorithm Development

In this section, Section 3.1 describes the development of some related algorithms. Section 3.2 describes the Nesterov and Heavy-ball accelerated methods for the distributed optimization algorithm. Section 3.3 describes the proposed algorithm UGNH and the relationship between UGNH and the previous algorithms.

3.1. Related Algorithms

In this subsection, we focus on some classical algorithms DGD, EXTRA, HSADO, and UG, which are related to the proposed algorithm and give them a simple explanation.

In  [13], Nedic and Ozdaglar proposed a standard distributed gradient descent method DGD. The method updated the decision variable at each agent through its neighbors and the local negative gradient’s direction, as follows:

$$\begin{array}{c}\hfill x_i^{t+1}=\sum _{j\in N_i}{p}_{ij}{x}_j^t-{\alpha }^t\nabla f_i\left(x_i^t\right)\end{array}$$

(3)

where ${\alpha }^t$ was the step-size, which satisfied ${\alpha }^t>0$, ${\sum }_{t=0}^{\infty }{\alpha }^t=\infty$, and $\sum {\left({\alpha }^t\right)}^2<\infty$; the matrix P satisfied Assumption 2. The variable $x_i^t$ stored in the agents is the local estimate of x at the t-th iteration. It was proved that sequences generated by DGD cannot converge to the exact optimal solution $x^{*}$ when employing a fixed step-size, i.e., ${\alpha }^t=\alpha$. By taking an appropriately diminishing step-sizes, DGD can converge accurately, but the convergence rate was sublinear.

To acquire linear convergence, Shi [19] proposed a new method EXTRA by modifying the update rule of DGD (3). There were two steps performed as follows:

$$\begin{array}{ccc}\hfill x_i^1=& \sum _{j\in N_i}{p}_{ij}{x}_j^0-\alpha \nabla f_i\left(x_i^0\right)\hfill & \hfill \left(4\right)\\ \hfill x_i^{t+1}=& x_i^t+\sum _{j\in N_i}{p}_{ij}{x}_j^t-\sum _{j\in N_i}{\tilde{p}}_{ij}{x}_j^{t-1}-\alpha \left(\nabla f_i\left(x_i^t\right)-\nabla f_i\left(x_i^{t-1}\right)\right)\hfill & \hfill \left(5\right)\end{array}$$

where the step-size $\alpha >0$ was a constant, the matrix P satisfied Assumption 2, while $\tilde{P}=\frac{I+P}{2}$ was appropriate. Compared to DGD (3), an initial condition (4) and one more iteration (5) were added. Notably, the step-size was a constant, but EXTRA can linearly converge to the exact optimal solution as long as the step-size was chosen appropriately.

Based on DGD (3), Qu and Li [26] proposed a novel distributed algorithm HSADO by using gradient-tracking technology. An auxiliary variable $z_i^t$ was introduced to estimate the network-wide gradient average $\frac{1}{n}{\sum }_{i=1}^n\nabla f_i\left(x_i^t\right)$ at the t-th iteration for agent i. As a result, the gradient contribution $-\alpha \nabla f_i\left(x_i^t\right)$ in (3) was replaced by $-\alpha z_i^t$. The specific updating rules were as follows:

$$\begin{array}{ccc}\hfill x_i^{t+1}& =\sum _{j\in N_i}{p}_{ij}{x}_j^t-\alpha z_i^t\hfill & \hfill \left(6\right)\\ \hfill z_i^{t+1}& =\sum _{j\in N_i}{p}_{ij}{z}_j^t+\nabla f_i\left(x_i^{t+1}\right)-\nabla f_i\left(x_i^t\right)\hfill & \hfill \left(7\right)\end{array}$$

where the step-size $\alpha >0$ was a constant, and the matrix P satisfied Assumption 2. Under the previous assumptions, initialized with $x_i^0\in {\mathbb{R}}^m$ and $z_i^0=\nabla f_i\left(x_i^0\right)$, a global linear convergence rate could be gained when choosing an appropriate fixed step-size.

Recently, a novel distributed optimization algorithm UG was proposed in [35], which used the primal-dual method to solve the equivalent problem (2). Through tuning parameters, the algorithm subsumed the well-known algorithm EXTRA and HSADO. Updating rules were as follows:    

$$\begin{array}{ccc}\hfill x_i^{t+1}& =\sum _{j\in N_i}{p}_{ij}{x}_j^t-\alpha \left(\nabla f_i\left(x_i^t\right)+z_i^t\right)\hfill & \hfill \left(8\right)\\ \hfill z_i^{t+1}& =z_i^t-\sum _{j\in N_i}{l}_{ij}\left(\nabla f_j\left(x_j^t\right)+z_j^t-\sum _{q\in N_j}{k}_{jq}{x}_q^t\right)\hfill & \hfill \left(9\right)\end{array}$$

where the step-size $\alpha >0$ was a constant; $x_i^t$ was primal variable; and $z_i^t$ was dual variable, which were initialized to $x_i^0\in {\mathbb{R}}^m$ and $z_i^0=0_m$, respectively. For using more-compact notation, we defined $P=\left\{p_{ij}\right\}$, $\mathbb{L}=\left\{l_{ij}\right\}$, and $\mathcal{K}=\left\{k_{ij}\right\}$. The matrix $\mathbb{L}=I_n-P$ and the matrix $\mathcal{K}\in {\mathbb{R}}^{n\times n}$ are symmetric with the property that there exists some constant $\lambda$ such that $\mathcal{K}{1}_n=\lambda 1_n$.

By analyzing when the matrix $\mathcal{K}$ is chosen properly, the algorithm UG is equivalent to: (1) EXTRA, when $\mathcal{K}=\frac{1}{\alpha }P$ and (2) HSADO, when $\mathcal{K}=0_n$. For others, $\mathcal{K}=\frac{\overline{\mu }+\overline{\psi }}{2}{I}_n$ and $\mathcal{K}=\frac{\overline{\mu }+\overline{\psi }}{1+{\lambda }_n}P$(${\lambda }_n$ is the smallest eigenvalue of matrix P) are appropriate for $\mathcal{K}=k{I}_n$ and $\mathcal{K}=kP$, respectively. There are no extra computational variables and no communication relationships with other agents in the formula $\mathcal{K}$, so (8) and (9) are easy to implement. As UG unifies and generalizes previous methods, we mainly focused on it.

3.2. Distributed Accelerated Methods

In this section, centralized Nesterov and Heavy-ball accelerated methods will be introduced. With them, many distributed optimization algorithms can converge faster.

For the gradient-descent algorithm, i.e., $x_i^{t+1}=x_i^t-\alpha \nabla f_i\left(x_i^t\right)$, the best achievable convergence is $O\left({\left(\frac{\kappa -1}{\kappa +1}\right)}^t\right)$; $\kappa =\frac{\overline{\psi }}{\overline{\mu }}$ denotes the condition number of the objective function. If $\overline{\psi }$ is much larger than $\overline{\mu }$ so that $\kappa$ is large, then the gradient descent becomes quite slow. To accelerate the gradient descent, Polyak [40] proposed a method called Heavy-ball for updating decision variable. The specifics was as follows:

$$\begin{array}{c}\hfill x_i^{t+1}=x_i^t-\alpha \nabla f_i\left(x_i^t\right)+\gamma \left(x_i^t-x_i^{t-1}\right)\end{array}$$

(10)

where $\gamma$ was the momentum-accelerated coefficient, and the term $\gamma \left(x_i^t-x_i^{t-1}\right)$ was used to accelerate the convergence of the decision variable. It had been proved that under the appropriate step-size $\alpha$ and the coefficient $\gamma$, the momentum-accelerated method could achieve a convergence rate of $O\left({\left(\frac{\sqrt{\kappa }-1}{\sqrt{\kappa }+1}\right)}^t\right)$, which was obviously faster.

Inspired by conjugate gradient methods [44], history gradient information can improve the convergence rate for distributed first-order optimization algorithms. Nesterov proposed a method called CNGD [40] (Centralised Nesterov Gradient Descent method) as follows:

$$\begin{array}{ccc}\hfill x_i^{t+1}& =y_i^t-\alpha \nabla f_i\left(y_i^t\right)\hfill & \hfill \left(11\right)\\ \hfill y_i^{t+1}& =x_i^{t+1}+\gamma \left(x_i^{t+1}-x_i^t\right)\hfill & \hfill \left(12\right)\end{array}$$

where $\alpha =\sqrt{\frac{\overline{\mu }}{\overline{\psi }}}$, $\gamma =\frac{\sqrt{\overline{\psi }}-\sqrt{\overline{\mu }}}{\sqrt{\overline{\psi }}+\sqrt{\overline{\mu }}}$. It had been proved that CNGD achieved the best convergence rate among all centralized gradient methods within first-order algorithms. Under the previous assumptions, CNGD achieved a faster convergence rate $O\left({\left(1-\sqrt{\frac{\overline{\mu }}{\overline{\psi }}}\right)}^t\right)$, compared to the CGD’s convergence rate $O\left({\left(1-\frac{\overline{\mu }}{\overline{\psi }}\right)}^t\right)$.

It is notable that the two accelerated methods have been adapted in many distributed algorithms, such as  [42,43], etc. In this study, we devoted ourselves to studying the two accelerated methods on UG.

3.3. The Proposed Algorithm

The recent studies  [35,42] are the most relevant to our work. Based on these works, considering that the Nesterov and Heavy-ball accelerated methods are very helpful for achieving a faster convergence, we added them into UG simultaneously. Meanwhile, in order to apply in many more scenarios, the step-sizes and coefficients were designed as uncoordinated, time-varying, and nonidentical. Combining together, we propose a new distributed optimization algorithm named UGNH as follows:

$$\begin{array}{ccc}\hfill x_i^{t+1}& =\sum _{j=1}^n{p}_{ij}{y}_j^t-{\alpha }_i^t\left(\nabla f_i\left(y_i^t\right)+z_i^t\right)+{\gamma }_i^t\left(x_i^t-x_i^{t-1}\right)\hfill & \hfill \left(13\right)\\ \hfill y_i^{t+1}& =x_i^{t+1}+{\gamma }_i^t\left(x_i^{t+1}-x_i^t\right)\hfill & \hfill \left(14\right)\\ \hfill z_i^{t+1}& =z_i^t-\sum _{j=1}^n{l}_{ij}\left(\nabla f_j\left(y_j^t\right)+z_i^t-\sum _{q=1}^n{k}_{jq}{y}_q^t\right)\hfill & \hfill \left(15\right)\end{array}$$

where $i,j\in V$, $t\in \mathbb{N}$, the step-sizes ${\alpha }_i^t>0$, and accelerated momentum coefficients ${\gamma }_i^t\ge 0$ are uncoordinated, time-varying, and nonidentical, which are locally chosen at each agent. At the t-th iteration, each agent stores three variables: the primal decision variable $x_i^t\in {\mathbb{R}}^m$, the temporary variable $y_i^t\in {\mathbb{R}}^m$, and the dual variable $z_i^t\in {\mathbb{R}}^m$, which start with initial states: $x_i^0\in {\mathbb{R}}^m$, $y_i^0\in {\mathbb{R}}^m$ and $z_i^0=0_m$. The update of UGNH at each agent i is formally described in Algorithm 1.

Algorithm 1 The update of the algorithm UGNH at each agent i

1: Initialization: each agent starts with: $x_i^0\in {\mathbb{R}}^m$, $y_i^0\in {\mathbb{R}}^m$ and $z_i^0=0_m$. 2: for$t=0,1,2,\cdots$do 3:    Update the primal decision variable $x_i$ as follows:

$x_i^{t+1}={\sum }_{j=1}^n{p}_{ij}{y}_j^t-{\alpha }_i^t\left(\nabla f_i\left(y_i^t\right)+z_i^t\right)+{\gamma }_i^t\left(x_i^t-x_i^{t-1}\right)$

4:    Update the temporary variable $y_i^{t+1}$ as follows:

$y_i^{t+1}=x_i^{t+1}+{\gamma }_i^t\left(x_i^{t+1}-x_i^t\right)$

5:    for $j=1,2,\cdots ,n$ do 6:      for $q=1,2,\cdots ,n$ do 7:         Calculate $z_{temp}={\sum }_{j=1}^n{l}_{ij}\left(\nabla f_j\left(y_j^t\right)+z_i^t-{\sum }_{q=1}^n{k}_{jq}{y}_q^t\right)$ 8:      end for 9:      Update the dual variable $z_i$ as follows:

$z_i^{t+1}=z_i^t-z_{temp}$

10:    end for 11: end for

It is clear that UGNH is a primal-dual method; ${\gamma }_i^t\left(x_i^t-x_i^{t-1}\right)$ is the Heavy-ball accelerated term in (13), (14) is the Nesterov accelerated term, and (15) is the dual variable iteration. It also can be easy to verify that UGNH is equivalent to UG if ${\alpha }_i^t=\alpha$, ${\gamma }_i^t=0_m$. Further, it can be equal to EXTRA and HSADO if the matrix $\mathcal{K}$ is chosen properly.

Remark 2.

For the sake of compaction and brevity, let the dimension $m=1$. Other multiple dimensions can be similarly proved.

As a result, we define:$x^t={\left[x_1^t,x_2^t,\cdots ,x_n^t\right]}^T\in {\mathbb{R}}^n$, $y^t={\left[y_1^t,y_2^t,\cdots ,y_n^t\right]}^T\in {\mathbb{R}}^n$, $z^t={\left[z_1^t,z_2^t,\cdots ,z_n^t\right]}^T\in {\mathbb{R}}^n$ and $\nabla F\left(y^t\right)={\left[\nabla f_1\left(y_1^t\right),\nabla f_2\left(y_2^t\right),\cdots ,\nabla f_n\left(y_n^t\right)\right]}^T\in {\mathbb{R}}^n$, other notations latter used are defined as before. Then, UGNH can be compactly reformulated in a martix form as follows:

$$\begin{array}{ccc}\hfill x^{t+1}& =P{y}^t-{\Gamma }_{\alpha }^t\left(\nabla F\left(y^t\right)+z^t\right)+{\Gamma }_{\gamma }^t\left(x^t-x^{t-1}\right)\hfill & \hfill \left(16\right)\\ \hfill y^{t+1}& =x^{t+1}+{\Gamma }_{\gamma }^t\left(x^{t+1}-x^t\right)\hfill & \hfill \left(17\right)\\ \hfill z^{t+1}& =z^t-\mathbb{L}\left(\nabla F\left(y^t\right)+z^t-\mathcal{K}{y}^t\right)\hfill & \hfill \left(18\right)\end{array}$$

where ${\alpha }^t={\left[{\alpha }_1^t,{\alpha }_2^t,\cdots ,{\alpha }_n^t\right]}^T\in {\mathbb{R}}^n$ and ${\gamma }_{}^t={\left[{\gamma }_1^t,{\gamma }_2^t,\cdots ,{\gamma }_n^t\right]}^T\in {\mathbb{R}}^n$ represent step-sizes and coefficients, respectively. Furthermore, we define ${\Gamma }_{\alpha }^t=diag\left({\alpha }^t\right)\in {\mathbb{R}}^{n\times n}$ and ${\Gamma }_{\gamma }^t=diag\left({\gamma }^t\right)\in {\mathbb{R}}^{n\times n}$.

4. Convergence Analysis

This section analyzes in detail the linear convergence of decision variable sequences generated by UGNH when step-sizes and coefficients are chosen properly. First, we define some notations that may frequently be used later.

$$\begin{array}{c}{\overline{x}}^t=\frac{1}{n}{1}_n^T{x}^t,{\overline{z}}^t=\frac{1}{n}{1}_n^T{z}^t,J_n=\frac{1}{n}{1}_n{1}_n^T,\tilde{\psi }=\underset{i\in V}{max}\left\{{\psi }_i\right\},\\ a=\parallel P-J_n\parallel ,b=\parallel I_n-J_n\parallel ,c=\parallel P-I_n\parallel ,d=\parallel \mathbb{L}\parallel \left(\tilde{\psi }+\parallel \mathcal{K}\parallel \right)\end{array}$$

Moreover, considering that the step-sizes and coefficients are uncoordinated, time-varying, and nonidentical, there are many possible numerical values that may be difficult to handle. By employing a small trick, we only studied the supremum and infimum of the step-sizes and coefficients. The specific definitions are as follows:

$${\alpha }_{max}=\underset{t\ge 0}{sup}\underset{i\in V}{max}\left\{{\alpha }_i^t\right\},{\alpha }_{min}=\underset{t\ge 0}{inf}\underset{i\in V}{min}\left\{{\alpha }_i^t\right\},\tilde{\gamma }=\underset{t\ge 0}{sup}\underset{i\in V}{max}\left\{{\gamma }_i^t\right\}$$

In addition, let ${\xi }_{\alpha }={\alpha }_{max}-{\alpha }_{min}$ be the difference between ${\alpha }_{max}$ and ${\alpha }_{min}$, and let $\Phi =\frac{{\alpha }_{max}}{{\alpha }_{min}}$ be the condition number.

Before giving the main results, we introduce some helpful supporting lemmas for the convergence analysis.

4.1. Supporting Lemmas

Lemma 1

([26]). Under Assumption 3, the global objective function f is $\overline{\psi }$-smooth and $\overline{\mu }$-strongly convex. For any $x\in \mathbb{R}$ and $0<\alpha <\frac{2}{\overline{\psi }}$, we have:

$$\parallel x-\alpha \nabla f\left(x\right)-x^{*}\parallel \le \zeta \parallel x-x^{*}\parallel$$

where $\zeta =max\left\{\left|1-\overline{\psi }\alpha \right|,\left|1-\overline{\mu }\alpha \right|\right\}$.

Lemma 2

([19]). Assumption null $\left\{I_n-P\right\}=span\left\{1\right\}$, matrix P satisfies Assumption 2, $x^{*}$ is the optimal solution when $x^{*}$ satisfies the following conditions:

• $x^{*}=P{x}^{*}$ (consensus)
• $1_n^T\nabla F\left(x^{*}\right)=0$ (optimality)

Lemma 3

([32]). Assume that a matrix $P\in {\mathbb{R}}^{n\times n}$ and a vector $\epsilon \in {\mathbb{R}}^n$ are non-negative and positive, respectively; if $P\epsilon <\varrho \epsilon$ with $\varrho >0$, we have $\rho \left(P\right)<\varrho$.

4.2. Main Results

In this section, the linear-convergence analysis of the proposed algorithm is carried out in detail. Similar to relevant studies, we mainly focus on the following four mathematical expressions at the $(t+1)$-th iteration: $x^{t+1}-1_n\otimes {\overline{x}}^{t+1}$, $1_n\otimes {\overline{x}}^{t+1}-x^{*}$, $x^{t+1}-x^t$, and $z^{t+1}-z^{*}$. For convenience, let ${\Xi }_1^{t+1}$, ${\Xi }_2^{t+1}$, ${\Xi }_3^{t+1}$, and ${\Xi }_4^{t+1}$ represent the four expressions, respectively. Among them, by introducing the norm, $\parallel {\Xi }_1^{t+1}\parallel$ is described as a consensus violation, $\parallel {\Xi }_2^{t+1}\parallel$ as an optimal residual, $\parallel {\Xi }_3^{t+1}\parallel$ as a state difference, and $\parallel {\Xi }_4^{t+1}\parallel$ as a dual error.

Next, we spared no effort to bound the four norm expressions at the $(t+1)$-th iteration through their estimates at the t-th iteration in terms of linear combinations. Subsequently, based on Assumptions 1–3, we established a linear inequalities system for convergence analysis. In what follows, consensus violation $\parallel {\Xi }_1^{t+1}\parallel$ is bounded first.

Lemma 4.

$\forall t>0$, the following inequality holds:

$$\begin{array}{cc}\hfill \parallel {\Xi }_1^{t+1}\parallel \le & \left(a+b{\alpha }_{max}\tilde{\psi }\right)\parallel {\Xi }_1^t\parallel +b{\alpha }_{max}\tilde{\psi }\parallel {\Xi }_2^t\parallel +\left(b{\alpha }_{max}\tilde{\psi }+a+b\right)\tilde{\gamma }\parallel {\Xi }_3^t\parallel +b{\alpha }_{max}\parallel {\Xi }_4^t\parallel \hfill \end{array}$$

(19)

Proof of Lemma 4.

Considering (16) and (17), we have:

$$\begin{array}{cc}\hfill x^{t+1}=& P{x}^t-{\Gamma }_{\alpha }^t\left(\nabla F\left(y^t\right)+z^t\right)+P{\Gamma }_{\gamma }^{t-1}{\Xi }_3^t+{\Gamma }_{\gamma }^t{\Xi }_3^t\hfill \end{array}$$

(20)

Note that $\left(I_n-J_n\right)x^{t+1}={\Xi }_1^{t+1}$, $\left(I_n-J_n\right)P=P-J_n$ and $\left(P-J_n\right)1_n=0_n$, multiplying $\left(I_n-J_n\right)$ on both sides of (20), then:

$$\begin{array}{cc}\hfill {\Xi }_1^{t+1}=& \left(P-J_n\right){\Xi }_1^t+\left(I_n-J_n\right){\Gamma }_{\gamma }^t{\Xi }_3^t-\left(I_n-J_n\right){\Gamma }_{\alpha }^t\left(\nabla F\left(y^t\right)-\nabla F\left(x^{*}\right)\right)\hfill \\ \hfill & -\left(I_n-J_n\right){\Gamma }_{\alpha }^t\left(z^t+\nabla F\left(x^{*}\right)\right)+\left(P-J_n\right){\Gamma }_{\gamma }^{t-1}{\Xi }_3^t\hfill \end{array}$$

(21)

Based on the fact $z^t-z^{*}=z^t+\nabla F\left(x^{*}\right)$ [35] and Assumption 3, taking the norm on both sides of (21), then:

$$\begin{array}{cc}\hfill \parallel {\Xi }_1^{t+1}\parallel \le & \parallel P-J_n\parallel \parallel {\Xi }_1^t\parallel +\parallel I_n-J_n\parallel {\alpha }_{max}\tilde{\psi }\left(\parallel {\Xi }_1^t\parallel +\parallel {\Xi }_2^t\parallel \right)+\parallel I_n-J_n\parallel {\alpha }_{max}\tilde{\psi }\tilde{\gamma }\parallel {\Xi }_3^t\parallel \hfill \\ \hfill & +\parallel I_n-J_n\parallel {\alpha }_{max}\parallel {\Xi }_4^t\parallel +\parallel P-J_n\parallel \tilde{\gamma }\parallel {\Xi }_3^t\parallel +\parallel I_n-J_n\parallel \tilde{\gamma }\parallel {\Xi }_3^t\parallel \hfill \end{array}$$

(22)

Recalling the definition of a and b, then:

$$\begin{array}{cc}\hfill \parallel {\Xi }_1^{t+1}\parallel \le & a\parallel {\Xi }_1^t\parallel +b{\alpha }_{max}\tilde{\psi }\parallel {\Xi }_1^t\parallel +b{\alpha }_{max}\tilde{\psi }\parallel {\Xi }_2^t\parallel +b{\alpha }_{max}\tilde{\psi }\tilde{\gamma }\parallel {\Xi }_3^t\parallel \hfill \\ \hfill & +b{\alpha }_{max}\parallel {\Xi }_4^t\parallel +\left(a+b\right)\tilde{\gamma }\parallel {\Xi }_3^t\parallel \hfill \end{array}$$

(23)

Rearranging the terms in (23), the result in Lemma 4 is obtained. □

Lemma 5.

$\forall t>0$, the following inequality holds:

$$\begin{array}{cc}\hfill \parallel {\Xi }_2^{t+1}\parallel \le & \left({\alpha }_{max}\tilde{\psi }+{\xi }_{\alpha }\tilde{\psi }\right)\parallel {\Xi }_1^t\parallel +\left(\zeta +{\xi }_{\alpha }\tilde{\psi }\right)\parallel {\Xi }_2^t\parallel +\left({\alpha }_{max}\tilde{\psi }+{\xi }_{\alpha }\tilde{\psi }+2\right)\tilde{\gamma }\parallel {\Xi }_3^t\parallel +{\xi }_{\alpha }\parallel {\Xi }_4^t\parallel \hfill \end{array}$$

(24)

Proof of Lemma 5.

Multiplying $J_n$ on both of (16), and substituting $y^t=x^t+{\Gamma }_{\gamma }^{t-1}{\Xi }_3^t$, we have:

$$\begin{array}{cc}\hfill J_n{x}^{t+1}=& J_n{x}^t-J_n{\Gamma }_{\alpha }^t\left(\nabla F\left(y^t\right)+z^t\right)+J_n{\Gamma }_{\gamma }^{t-1}{\Xi }_3^t+J_n{\Gamma }_{\gamma }^t{\Xi }_3^t\hfill \end{array}$$

(25)

To get the related terms, recalling the fact that ${\overline{z}}^{t+1}={\overline{z}}^t=\cdots ={\overline{z}}^0=0$ (e.g., $J_n{z}^t=0$) in [35], we add some useful items and delete them in (25) as follows:

$$\begin{array}{cc}\hfill J_n{x}^{t+1}=& J_n{x}^t-{\alpha }_{max}{J}_n\nabla F\left(J_n{x}^t\right)+{\alpha }_{max}{J}_n\left(\nabla F\left(J_n{x}^t\right)-\nabla F\left(y^t\right)\right)+J_n{\Gamma }_{\gamma }^{t-1}{\Xi }_3^t+J_n{\Gamma }_{\gamma }^t{\Xi }_3^t\hfill \\ \hfill & +J_n\left(1_n\otimes {\alpha }_{max}-{\Gamma }_{\alpha }^t\right)\left(\nabla F\left(y^t\right)-\nabla F\left(x^{*}\right)\right)+J_n\left(1_n\otimes {\alpha }_{max}-{\Gamma }_{\alpha }^t\right)\left(z^t+\nabla F\left(x^{*}\right)\right)\hfill \end{array}$$

(26)

By applying $\nabla f\left(x\right)=\frac{1}{n}{1}_n\nabla F\left(x\right)$, subtracting $x^{*}$ on the sides of (26), we then obtain:

$$\begin{array}{cc}\hfill {\Xi }_2^{t+1}=& 1_n\left({\overline{x}}^t-{\tilde{x}}^{*}-{\alpha }_{max}\nabla f\left({\overline{x}}^t\right)\right)+{\alpha }_{max}{J}_n\left(\nabla F\left(J_n{x}^t\right)-\nabla F\left(y^t\right)\right)+J_n{\Gamma }_{\gamma }^{t-1}{\Xi }_3^t+J_n{\Gamma }_{\gamma }^t{\Xi }_3^t\hfill \\ \hfill & +J_n\left(1_n\otimes {\alpha }_{max}-{\Gamma }_{\alpha }^t\right)\left(\nabla F\left(y^t\right)-\nabla F\left(x^{*}\right)\right)+J_n\left(1_n\otimes {\alpha }_{max}-{\Gamma }_{\alpha }^t\right)\left(z^t+\nabla F\left(x^{*}\right)\right)\hfill \end{array}$$

(27)

Taking the norm on both sides of (27) and using Lemma 1, then:

$$\begin{array}{cc}\hfill \parallel {\Xi }_2^{t+1}\parallel \le & \zeta \parallel {\Xi }_2^t\parallel +{\alpha }_{max}\parallel J_n\parallel \tilde{\psi }\parallel {\Xi }_1^t-{\Gamma }_{\gamma }^{t-1}{\Xi }_3^t\parallel +\parallel J_n\parallel \left({\alpha }_{max}-{\alpha }_{min}\right)\tilde{\psi }\parallel y^t-x^{*}\parallel \hfill \\ \hfill & +\parallel J_n\parallel \left({\alpha }_{max}-{\alpha }_{min}\right)\parallel {\Xi }_4^t\parallel +2\parallel J_n\parallel \tilde{\gamma }\parallel {\Xi }_3^t\parallel \hfill \\ \hfill \le & \zeta \parallel {\Xi }_2^t\parallel +{\alpha }_{max}\tilde{\psi }\parallel {\Xi }_1^t\parallel +{\alpha }_{max}\tilde{\psi }\tilde{\gamma }\parallel {\Xi }_3^t\parallel +{\xi }_{\alpha }\tilde{\psi }\parallel {\Xi }_1^t\parallel +{\xi }_{\alpha }\tilde{\psi }\parallel {\Xi }_2^t\parallel \hfill \\ \hfill & +{\xi }_{\alpha }\tilde{\psi }\tilde{\gamma }\parallel {\Xi }_3^t\parallel +{\xi }_{\alpha }\parallel {\Xi }_4^t\parallel +2\tilde{\gamma }\parallel {\Xi }_3^t\parallel \hfill \end{array}$$

(28)

Rearranging the terms in (28), the desired results can be obtained. □

Lemma 6.

$\forall t>0$, the following inequality holds:

$$\begin{array}{cc}\hfill \parallel {\Xi }_3^{t+1}\parallel \le & \left(c+{\alpha }_{max}\tilde{\psi }\right)\parallel {\Xi }_1^t\parallel +{\alpha }_{max}\tilde{\psi }\parallel {\Xi }_2^t\parallel +\left({\alpha }_{max}\tilde{\psi }\tilde{\gamma }+2\tilde{\gamma }\right)\parallel {\Xi }_3^t\parallel +{\alpha }_{max}\parallel {\Xi }_4^t\parallel \hfill \end{array}$$

(29)

Proof of Lemma 6.

Substituting $y^t=x^t+{\Gamma }_{\gamma }^{t-1}{\Xi }_3^t$ in (16), then subtracting $x^t$ on both sides, then:

$$\begin{array}{cc}\hfill {\Xi }_3^{t+1}=& P\left(x^t+{\Gamma }_{\gamma }^{t-1}{\Xi }_3^t\right)-x^t-{\Gamma }_{\alpha }^t\left(\nabla F\left(y^t\right)+z^t\right)+{\Gamma }_{\gamma }^t{\Xi }_3^t\hfill \\ \hfill =& \left(P-I_n\right){\Xi }_1^t-{\Gamma }_{\alpha }^t\left(\nabla F\left(y^t\right)-\nabla F\left(x^{*}\right)\right)+{\Gamma }_{\alpha }^t{\Xi }_4^t+\left(P{\Gamma }_{\gamma }^{t-1}+{\Gamma }_{\gamma }^t\right){\Xi }_3^t\hfill \end{array}$$

(30)

The second equality is based on $\left(P-I_n\right)1_n=0_n$; recalling the definition of c and taking the norm on both sides of (30), we have:

$$\begin{array}{cc}\hfill \parallel {\Xi }_3^{t+1}\parallel \le & c\parallel {\Xi }_1^t\parallel +{\alpha }_{max}\tilde{\psi }\parallel y^t-x^{*}\parallel +{\alpha }_{max}\parallel {\Xi }_4^t\parallel +2\tilde{\gamma }\parallel {\Xi }_3\parallel \hfill \\ \hfill =& c\parallel {\Xi }_1^t\parallel +{\alpha }_{max}\tilde{\psi }\parallel {\Xi }_1^t\parallel +{\alpha }_{max}\tilde{\psi }\parallel {\Xi }_2^t\parallel +{\alpha }_{max}\tilde{\psi }\tilde{\gamma }\parallel {\Xi }_3^t\parallel +{\alpha }_{max}\parallel {\Xi }_4^t\parallel +2\tilde{\gamma }\parallel {\Xi }_3^t\parallel \hfill \end{array}$$

(31)

Rearranging the terms in (31), the result in Lemma 6 is obtained. □

Lemma 7.

Let Assumptions 2–3 and Lemma 2 hold. $\forall t>0$, the following inequality holds:

$$\begin{array}{cc}\hfill & \parallel {\Xi }_4^{t+1}\parallel \le d\parallel {\Xi }_1^t\parallel +d\parallel {\Xi }_2^t\parallel +d\tilde{\gamma }\parallel {\Xi }_3^t\parallel +a\parallel {\Xi }_4^t\parallel \hfill \end{array}$$

(32)

Proof of Lemma 7.

Noting $\left(P-J_n\right)1_n=0_n$ and adding $\nabla F\left(x^{*}\right)$ on both sides of (18), we have:

$$\begin{array}{cc}\hfill z^{t+1}+\nabla F\left(x^{*}\right)=& z^t+\nabla F\left(x^{*}\right)-\mathbb{L}\left(\nabla F\left(y^t\right)+z^t-\mathcal{K}{y}^t\right)\hfill \\ \hfill =& P\left(z^t+\nabla F\left(x^{*}\right)\right)+\mathbb{L}\mathcal{K}{y}^t-\left(I_n-P\right)\left(\nabla F\left(y^t\right)-\nabla F\left(x^{*}\right)\right)\hfill \\ \hfill =& \left(P-J_n\right)\left(z^t-z^{*}\right)+\mathbb{L}\mathcal{K}\left(y^t-x^{*}\right)-\mathbb{L}\left(\nabla F\left(y^t\right)-\nabla F\left(x^{*}\right)\right)\hfill \end{array}$$

(33)

The third equality of (33) is from the following fact in [35] and Lemma 2:

${\overline{z}}^{t+1}={\overline{z}}^t=\cdots ={\overline{z}}^0=0$, $J_n\nabla F\left(x^{*}\right)=0_n$ and $\mathbb{L}\mathcal{K}{1}_n=0_n$.

Recalling the definition of d, taking the norm on both sides of (33),we have:

$$\begin{array}{cc}\hfill \parallel {\Xi }_4^{t+1}\parallel & \le \parallel P-J_n\parallel \parallel {\Xi }_4^t\parallel +\left(\parallel \mathbb{L}\parallel \tilde{\psi }+\parallel \mathbb{L}\parallel \parallel \mathcal{K}\parallel \right)\parallel y^t-x^{*}\parallel =a\parallel {\Xi }_4^t\parallel +d\parallel y^t-x^{*}\parallel \hfill \end{array}$$

(34)

Substituting $y^t=x^t+{\Gamma }_{\gamma }^{t-1}{\Xi }_3^t$ in (34) and rearranging the terms can yield the desired result. □

With the Lemmas 4–7 above, we established the main convergence result as follows.

Theorem 1.

Suppose that Assumptions 1–3 hold. Considering the sequences $\left\{x^t\right\}$, $\left\{y^t\right\}$, and $\left\{z^t\right\}$ generating by the proposed algorithm UGNH and combining Lemmas 4–7 in a linear-inequalities system, we have:

$$\begin{array}{c}\hfill \left[\begin{array}{c}\parallel {\Xi }_1^{t+1}\parallel \\ \parallel {\Xi }_2^{t+1}\parallel \\ \parallel {\Xi }_3^{t+1}\parallel \\ \parallel {\Xi }_4^{t+1}\parallel \end{array}\right]\le \mathbb{H}\left[\begin{array}{c}\parallel {\Xi }_1^t\parallel \\ \parallel {\Xi }_2^t\parallel \\ \parallel {\Xi }_3^t\parallel \\ \parallel {\Xi }_4^t\parallel \end{array}\right]\end{array}$$

(35)

where the matrix$\mathbb{H}\in {\mathbb{R}}^{4\times 4}$is given as below:

$$\begin{array}{c}\hfill \mathbb{H}=\left[\begin{array}{cccc}a+b{\alpha }_{max}\tilde{\psi }& b{\alpha }_{max}\tilde{\psi }& b{\alpha }_{max}\tilde{\psi }\tilde{\gamma }+a\tilde{\gamma }+b\tilde{\gamma }& b{\alpha }_{max}\\ {\alpha }_{max}\tilde{\psi }+{\xi }_{\alpha }\tilde{\psi }& \zeta +{\xi }_{\alpha }\tilde{\psi }& {\alpha }_{max}\tilde{\psi }\tilde{\gamma }+{\xi }_{\alpha }\tilde{\psi }\tilde{\gamma }+2\tilde{\gamma }& {\xi }_{\alpha }\\ c+{\alpha }_{max}\tilde{\psi }& {\alpha }_{max}\tilde{\psi }& {\alpha }_{max}\tilde{\psi }\tilde{\gamma }+2\tilde{\gamma }& {\alpha }_{max}\\ d& d& d\tilde{\gamma }& a\end{array}\right]\end{array}$$

The largest step-size satisfies:

$$\begin{array}{c}\hfill {\alpha }_{max}<min\left\{\frac{{\epsilon }_1-a{\epsilon }_1}{b\tilde{\psi }{\epsilon }_1+b\tilde{\psi }{\epsilon }_2+b{\epsilon }_4},\frac{{\epsilon }_3-c{\epsilon }_1}{\tilde{\psi }{\epsilon }_1+\tilde{\psi }{\epsilon }_2+{\epsilon }_4},\frac{1}{\overline{\psi }}\right\}\end{array}$$

(36)

The maximum momentum coefficient satisfies:

$$\begin{array}{cc}\hfill \tilde{\gamma }<min& \left\{\frac{{\epsilon }_1-a{\epsilon }_1-b{\alpha }_{max}\tilde{\psi }{\epsilon }_1-b{\alpha }_{max}\tilde{\psi }{\epsilon }_2-b{\alpha }_{max}{\epsilon }_4}{b{\alpha }_{max}\tilde{\psi }{\epsilon }_3+a{\epsilon }_3+b{\epsilon }_3}\right.,\hfill \\ \hfill & \frac{\overline{\mu }{\alpha }_{max}{\epsilon }_2-{\xi }_{\alpha }\tilde{\psi }{\epsilon }_2-{\alpha }_{max}\tilde{\psi }{\epsilon }_1-{\xi }_{\alpha }\tilde{\psi }{\epsilon }_1-{\xi }_{\alpha }{\epsilon }_4}{{\alpha }_{max}\tilde{\psi }{\epsilon }_3+{\xi }_{\alpha }\tilde{\psi }{\epsilon }_3+2{\epsilon }_3},\hfill \\ \hfill & \left.\frac{{\epsilon }_3-{\alpha }_{max}\tilde{\psi }{\epsilon }_2-c{\epsilon }_1-{\alpha }_{max}\tilde{\psi }{\epsilon }_1-{\alpha }_{max}{\epsilon }_4}{{\alpha }_{max}\tilde{\psi }{\epsilon }_3+2{\epsilon }_3},\frac{{\epsilon }_4-d{\epsilon }_1-d{\epsilon }_2-a{\epsilon }_4}{d{\epsilon }_3}\right\}\hfill \end{array}$$

(37)

And the conditional number satisfies:

$$\begin{array}{cc}\hfill & 1\le \Phi <\frac{{\epsilon }_4+\tilde{\psi }{\epsilon }_2+\tilde{\psi }{\epsilon }_1}{{\epsilon }_4+\tilde{\psi }{\epsilon }_2+2\tilde{\psi }{\epsilon }_1-\overline{\mu }{\epsilon }_2}\hfill \end{array}$$

(38)

where ${\epsilon }_1$, ${\epsilon }_2$, ${\epsilon }_3$, and ${\epsilon }_4$ are arbitrary constants, which obey the following picking rules:

$$\begin{array}{cc}\hfill & {\epsilon }_2>0,{\epsilon }_1<\frac{\overline{\mu }{\epsilon }_2}{\tilde{\psi }},{\epsilon }_3>c{\epsilon }_1,{\epsilon }_4>\frac{d{\epsilon }_1+d{\epsilon }_2}{1-a}\hfill \end{array}$$

(39)

Then, the spectral radius of the matrix $\mathbb{H}$ is strictly less than 1, i.e., $\rho \left(\mathbb{H}\right)<1$, which is the desired result.

Proof of Theorem 1.

According to Lemmas 4–7, we can immediately get the inequalities (35). Then, we provide some necessary conditions for parameters $\tilde{\psi }$, $\tilde{\gamma }$ and $\Phi$, such that $\rho \left(\mathbb{H}\right)<1$. Based on Lemma 3, let $\epsilon ={\left[{\epsilon }_1,{\epsilon }_2,{\epsilon }_3,{\epsilon }_4\right]}^T\in {\mathbb{R}}^4$ be a positive vector, if $\mathbb{H}\epsilon <\epsilon$, then $\rho \left(\mathbb{H}\right)<1$. According to the definition of $\mathbb{H}$ above, the inequality $\mathbb{H}\epsilon <\epsilon$ is equivalent to the following four inequalities:

$$\begin{array}{cc}\hfill & \left(b{\alpha }_{max}\tilde{\psi }+a+b\right)\tilde{\gamma }{\epsilon }_3<{\epsilon }_1-a{\epsilon }_1-b{\alpha }_{max}\tilde{\psi }{\epsilon }_1-b{\alpha }_{max}\tilde{\psi }{\epsilon }_2-b{\alpha }_{max}{\epsilon }_4\hfill \end{array}$$

(40)

$$\begin{array}{cc}\hfill & \left({\alpha }_{max}\tilde{\psi }+{\xi }_{\alpha }\tilde{\psi }+2\right)\tilde{\gamma }{\epsilon }_3<{\epsilon }_2-{\alpha }_{max}\tilde{\psi }{\epsilon }_1-{\xi }_{\alpha }\tilde{\psi }{\epsilon }_1-\zeta {\epsilon }_2-{\xi }_{\alpha }\tilde{\psi }{\epsilon }_2-{\xi }_{\alpha }{\epsilon }_4\hfill \end{array}$$

(41)

$$\begin{array}{cc}\hfill & \left({\alpha }_{max}\tilde{\psi }+2\right)\tilde{\gamma }{\epsilon }_3<{\epsilon }_3-c{\epsilon }_1-{\alpha }_{max}\tilde{\psi }{\epsilon }_1-{\alpha }_{max}\tilde{\psi }{\epsilon }_2-{\alpha }_{max}{\epsilon }_4\hfill \end{array}$$

(42)

$$\begin{array}{cc}\hfill & d\tilde{\gamma }{\epsilon }_3<{\epsilon }_4-d{\epsilon }_1-d{\epsilon }_2-a{\epsilon }_4\hfill \end{array}$$

(43)

According to Lemma 1, if $0<{\alpha }_{max}<\frac{1}{\overline{\psi }}$, $\zeta =1-\overline{\mu }{\alpha }_{max}$, then (41) is equivalent to the following inequality:

$$\begin{array}{cc}\hfill & \left({\alpha }_{max}\tilde{\psi }+{\xi }_{\alpha }\tilde{\psi }+2\right)\tilde{\gamma }{\epsilon }_3<\overline{\mu }{\alpha }_{max}{\epsilon }_2-{\alpha }_{max}\tilde{\psi }{\epsilon }_1-{\xi }_{\alpha }\tilde{\psi }{\epsilon }_1-{\xi }_{\alpha }\tilde{\psi }{\epsilon }_2-{\xi }_{\alpha }{\epsilon }_4\hfill \end{array}$$

(44)

To make sure that the parameter $\tilde{\gamma }$ is positive, it implies that the right sides of (40) and (42)–(44) are positive. Immediately, we can get the following conditions:

$$\begin{array}{cc}\hfill & {\alpha }_{max}<\frac{{\epsilon }_1-a{\epsilon }_1}{b\tilde{\psi }{\epsilon }_1+b\tilde{\psi }{\epsilon }_2+b{\epsilon }_4}\hfill \end{array}$$

(45)

$$\begin{array}{cc}\hfill & {\xi }_{\alpha }<\frac{\overline{\mu }{\alpha }_{max}{\epsilon }_2-{\alpha }_{max}\tilde{\psi }{\epsilon }_1}{{\epsilon }_4+\tilde{\psi }{\epsilon }_2+\tilde{\psi }{\epsilon }_1};{\epsilon }_1<\frac{\overline{\mu }{\epsilon }_2}{\tilde{\psi }}\hfill \end{array}$$

(46)

$$\begin{array}{cc}\hfill & {\alpha }_{max}<\frac{{\epsilon }_3-c{\epsilon }_1}{\tilde{\psi }{\epsilon }_1+\tilde{\psi }{\epsilon }_2+{\epsilon }_4};{\epsilon }_3>c{\epsilon }_1\hfill \end{array}$$

(47)

$$\begin{array}{cc}\hfill & {\epsilon }_4>\frac{d{\epsilon }_1+d{\epsilon }_2}{1-a}\hfill \end{array}$$

(48)

Recalling that ${\xi }_{\alpha }={\alpha }_{max}-{\alpha }_{min}$, $\Phi =\frac{{\alpha }_{max}}{{\alpha }_{min}}$ as the conditional number, (46) further implies that:

$$\begin{array}{cc}\hfill & 1\le \Phi <\frac{{\epsilon }_4+\tilde{\psi }{\epsilon }_2+\tilde{\psi }{\epsilon }_1}{{\epsilon }_4+\tilde{\psi }{\epsilon }_2+2\tilde{\psi }{\epsilon }_1-\overline{\mu }{\epsilon }_2}\hfill \end{array}$$

(49)

Now, we attempt to select the proper vector $\epsilon ={\left[{\epsilon }_1,{\epsilon }_2,{\epsilon }_3,{\epsilon }_4\right]}^T$ such that the parameters ${\alpha }_{max}$, $\tilde{\gamma }$ and $\Phi$ are available. Based on (46)–(48), an arbitrary positive constant ${\epsilon }_2$ is chosen first, and then we choose ${\epsilon }_1$ from (46), finally choosing ${\epsilon }_3$ and ${\epsilon }_4$ from (47) and (48), respectively. Hence, according to (45) and (47), and the requirement of $0<{\alpha }_{max}<\frac{1}{\overline{\psi }}$ in (44), the upper bound of the largest step-size ${\alpha }_{max}$ shown in (36) can be obtained. Furthermore, according to (46), the upper bound of the conditional number $\Phi$ demonstrated in (30) can be obtained. Besides, the upper bound of the maximum coefficient $\tilde{\gamma }$ can yield from (40) and (42)–(44). Above all, the proof is finished. □

Remark 3.

According to Theorem 1, a linear convergence rate of the proposed algorithm can be easily obtained if the parameters ${\alpha }_{max}$, $\tilde{\gamma }$ and Φ follow the conditions (36)–(38), respectively. It is noteworthy that these parameters only depend on the topology of the network and objective functions. Although some global parameters such as $\overline{\mu }$, $\overline{\psi }$ and $\tilde{\psi }$ are needed when designing step-sizes and the coefficients, these parameters can be easily pre-calculated without much effort.

Remark 4.

Being uncoordinated and being nonidentical are two important characteristics often designed in many related studies, considering that step-sizes and coefficients might be changed with time variance in some practical scenarios. In our algorithm, step-sizes and coefficients were designed as uncoordinated, time-varying, and nonidentical. Furthermore, the largest step-size and coefficients were chosen according to their bounds shown in Theorem 1, which only depend on the the communication network and the objective functions. Notably, there is a bound of a conditional number, such that when the largest step-size is chosen, the smallest step-size needs to be chosen carefully.

5. Numerical Experiments

In this section, some necessary numerical experiments in a real dataset are provided to illustrate the efficiency and superiority of our algorithm. In the experiments, we considered a binary-classification logistic-regression problem in the Wisconsin breast cancer dataset provided in the UCI Machine Learning Repository [45]. The problem can be described in the following form:

$$\begin{array}{cc}\hfill & \underset{x\in {\mathbb{R}}^d}{min}\frac{1}{n}\sum _{i=1}^n\frac{1}{n_i}\sum _{j=1}^{n_i}ln\left(1+exp\left(-y_{ij}{a}_{ij}^{T}x\right)\right)+\frac{\tau }{2}\parallel x\parallel \hfill \end{array}$$

(50)

with each local objective function $f_i$ written as follows:

$$\begin{array}{cc}\hfill & f_i\left(x\right)=\frac{1}{n_i}\sum _{j=1}^{n_i}ln\left(1+exp\left(-y_{ij}{a}_{ij}^{T}x\right)\right)+\frac{\tau }{2}\parallel x\parallel \hfill \end{array}$$

(51)

where n is the number of the agent in network, and d is the dimension of the decision variable. Each agent i is assumed to have an equal data samples $n_i$, i.e., $n_i=\frac{N}{n}$(N is the total data samples). $a_{ij}\in {\mathbb{R}}^d$ represents the feature vector of the jth data sample at the ith agent, while $y_{ij}\in \left\{-1,1\right\}$ denotes the corresponding label. The regularization term $\frac{\tau }{2}\parallel x\parallel$ with parameter $\lambda =1$ was set to avoid over-fitting.

In the experiments, we set $N=200$ as training data, and $d=9$ represents the feature in the real dataset. Meanwhile, we simulated a randomly undirected network generated by the Erdos–Renyi network with $n=10$ nodes and edge probability $p=0.7$. Then, we compared the proposed algorithm UGNH to relevant algorithms: EXTRA, HSADO, and UG.

Figure 1, Figure 2, Figure 3, Figure 4 and Figure 5 show the results of our experiments, and the main conclusions are as follows:

• Figure 1 indicates that the proposed algorithm UGNH promotes the convergence rate compared to the related algorithms in the real dataset; thus, UGNH is effective and superior. From Figure 2, the sequences generated by UGNH, EXTRA, UG, and HSADO can converge to the optimal solutions as expected. Avoiding confusion of the figure, only one dimension of each decision variable is exhibited.
• Figure 3 means that UGNH with the Nesterov momentum and the Heavy-ball momentum improved the convergence rate compared to the algorithm with only one or no momentum.
• In Figure 4, we can conclude that step-size is usually chosen very small; the larger step-size leads to a faster convergence rate if it is chosen under the upper bound. For the coefficient, a similar result can be obtained in Figure 5. Comparing the two figures, it can be concluded that small changes in step-size are more influential than that of the coefficient.

6. Conclusions

In this study, a novel uncoordinated, time-varying, and nonidentical distributed optimization accelerated algorithm was proposed. It was mainly applied to handle the distributed optimization convex problem in an undirected network, where all agents are in an effort to optimize the average of all local objective functions collaboratively. When the largest step-size and the maximum coefficient do not exceed some estimated upper bounds, which have been provided in Theorem 1, the convergence rate of UGNH is linear under the condition that each local objective function is smooth and strongly convex. Besides, these parameters only depend on the topology of the network and the local objective function.

It is worth noting that to achieve a faster linear convergence rate, the Heavy-ball and Nesterov accelerated methods were simultaneously added into the algorithm, which provides a new way for accelerating convergence of other distributed optimization algorithms. Furthermore, the experiment results verified the effective and superior performance in a real dataset. However, UGNH is not suitable for all scenarios, and there are some more in-depth areas worth studying, such as the time-varying network architecture, random link failures, asynchronous communication between agents, directed networks, and so on. In all, these problems are worthy of further study and are our future research direction.