A Flexible Framework for Decentralized Composite Optimization with Compressed Communication

Abstract

This paper addresses the decentralized composite optimization problem, where a network of agents cooperatively minimize the sum of their local objective functions with non-differentiable terms. We propose a novel communication-efficient decentralized ADMM framework, termed as CE-DADMM, by combining the ADMM framework with the three-point compressed (3PC) communication mechanism. This framework not only covers existing mainstream communication-efficient algorithms but also introduces a series of new algorithms. One of the key features of the CE-DADMM framework is its flexibility, allowing it to adapt to different communication and computation needs, balancing communication efficiency and computational overhead. Notably, when employing quasi-Newton updates, CE-DADMM becomes the first communication-efficient second-order algorithm based on compression that can efficiently handle composite optimization problems. Theoretical analysis shows that, even in the presence of compression errors, the proposed algorithm maintains exact linear convergence when the local objective functions are strongly convex. Finally, numerical experiments demonstrate the algorithm’s impressive communication efficiency.

1. Introduction

The recent increase in the number of mobile devices with enhanced computing and communication capabilities has led to significant development of multiagent systems [1,2]. Within these systems, many applications, including smart grid management [3,4], wireless communications [5], multi-robot coordination [6,7], large-scale machine learning [8], etc., can be cast to decentralized optimization problems, in which a network of nodes cooperatively solve a finite-sum optimization problem using local information.

A vast of decentralized optimization algorithms have been proposed, since the pioneer work DGD [9], in which each node performs gradient descent and simultaneously communicates decision vector with its neighbors for consensus. As DGD requires diminishing stepsize, which might slower the convergence rate, gradient tracking (GT) based algorithms using constant stepsizes are then developed [10,11] and have been extensively investigated under various scenarios [12,13,14,15], to name a few. However, GT-based methods require to transmit both the decision vector and an additional gradient estimation vector, which increase the communication cost. In parallel, another type of decentralized algorithms based on alternating direction method of multipliers (ADMM) are proposed and analyzed [16,17]. Compared with GT-based algorithms, ADMM-type algorithms can achieve the same convergence rate but require the transmission of only decision vector, which can be more communication-efficient. Following this line, some decentralized optimization algorithms are proposed for accelerating the convergence rate by introducing second-order information [18,19,20,21]. More recently, Ref. [22] proposed a family of decentralized curvature aided primal dual algorithms, which can include gradient, Newton, and BFGS type of updates.

In the decentralized algorithms, it is of great significance to improve communication-efficiency. The methods can be classified into two types. One method is to adopt compressed communication scheme, using quantization [23] or sparsification [24,25] techniques to reduce communication overhead per transmission. Recently, the compressed communication scheme has been combined with DGD [26,27], GT-based algorithms [28,29], and ADMM-type algorithms [30,31]. Another method is to employ intermittent communication scheme which aims to reduce the communication frequency. Such type of methods includes event-triggered communication [32,33,34], lazy aggregation scheme (LAG) [35], etc. Besides, there are also some works combining the both methods [36,37,38]. In particular, ref. [39] combined event-triggered and compressed communication scheme with an ADMM-type algorithm and proposed a communication-efficient decentralized second-order optimization algorithm, which improve both computation and communication efficiency. It is worthy noting that the information distortion arisen by compressed communication scheme may have a negative effect on the convergence performance of the decentralized optimization algorithms. To overcome this shortage, Ref. [40] developed an error-feedback communication scheme (EF21) to avoid the negative effect of information distortion. Recently, a more general efficient communication scheme termed as three point compressor (3PC) is proposed in [41], which provides a unified method including EF21 and LAG as special cases. However, 3PC scheme is only investigated in distributed gradient descent algorithms under the parameter-server framework.

Despite of the progress, the development of communication-efficient decentralized optimization algorithms over the general networks is still lack an in-deep analysis, especially for the objective with non-differentiable part, i.e., decentralized composite optimization problems. Note that such problems have widely applications in the field of machine learning due to the existence of the non-differentiable regularization terms. Currently, some decentralized composite optimization algorithms have been proposed [22,42,43,44], but without employing efficient-communication schemes. Moreover, to the best of our knowledge, no work has been reported on communication-efficient decentralized composite optimization using second-order information. To fill this gap, in this paper, we incorporate the general efficient communication scheme 3PC [41] into the ADMM-based decentralized optimization framework [22], which result in a family of communication-efficient decentralized composite optimization algorithms with theoretical guarantees. It is worthy noting that such incorporation is not trivial as we need to overcome the negative effect arisen by the propagation of communication error over networks. The main contribution of this work can be summarized in the following two aspects:

• First, we propose a flexible framework termed as CE-DADMM for communication-efficient decentralized composite optimization problems. The framework not only encompasses some existing algorithms, such as COLA [32] and CC-DQM [39], but also introduces several new algorithms. Specifically, by incorporating quasi-Newton updates into CE-DADMM, we derive CE-DADMM-BFGS, the first communication-efficient decentralized second-order algorithm for composite optimization. Compared with CC-DQM, it avoids computing the Hessian matrix and its inversion, significantly reducing the computational cost. Compared with DRUID [22], CE-DADMM can reduce the communication cost due to the efficient communication scheme.
• Second, we theoretically prove that CE-DADMM can achieve exact linear convergence under the assumption of strong convexity by carefully analyzing the mixing error arisen by the efficient communication scheme and the disagreement of decision vectors. The dependency of the convergence rate on the parameters of the compression mechanism is also established. Additionally, sufficient numerical experiments are presented to substantiate the superior performance of our algorithms in terms of the communication efficiency.

Notation. If not specified, $\Vert \mathbf{x}\Vert$ and $\Vert \mathbf{A}\Vert$ represent the Euclidean norm and the spectral norm, respectively. For a positive definite matrix $\mathbf{P}\succ 0$, let ${\Vert \mathbf{x}\Vert }_{\mathbf{P}}:=\sqrt{{\mathbf{x}}^{\top }\mathbf{P}\mathbf{x}}$. Use $\left[n\right]$ to denote the set $\{1,\cdots ,n\}$. The proximal mapping for a function $g(·):{\mathbb{R}}^d\to \mathbb{R}$ is defined by ${\mathbf{prox}}_{g/\mathsf{\mu }}\left(\mathbf{v}\right)={arg\; min}_{\mathit{\theta }\in {\mathbb{R}}^d}\{g\left(\mathit{\theta }\right)+{\displaystyle \frac{\mathsf{\mu }}{2}}{\Vert \mathit{\theta }-\mathbf{v}\Vert }^2\}$. Let ${\mathbf{I}}_d$ represent the d-dimensional identity matrix, and $\mathbf{A}\otimes \mathbf{B}$ represent the Kronecker product of matrices $\mathbf{A}$ and $\mathbf{B}$.

2. Problem Setting

In this paper, we study the decentralized composite optimization problem on an undirected connected network with n agents (or, nodes), which takes the form

$$\begin{array}{c}\hfill \underset{\mathbf{x}\in {\mathbb{R}}^d}{min}\left\{\sum _{i=1}^n{f}_i\left(\mathbf{x}\right)+g\left(\mathbf{x}\right)\right\},\end{array}$$

(1)

where $\mathbf{x}$ refers to the decision vector and $f_i(·):{\mathbb{R}}^d\to \mathbb{R}$ is a convex and smooth function accessible only by node i and $g(·):{\mathbb{R}}^d\to \mathbb{R}$ is a convex (possibly non-smooth) regularizer.

Next, we equivalently reformulate problem (1) into a compact form in terms of the whole network following the same idea in [22]. Denote the communication graph as $\mathcal{G}=\left\{\mathcal{V},\mathcal{E}\right\}$, where $\mathcal{V}:=\left[n\right]$ is the set of agents and $\mathcal{E}\subseteq \mathcal{V}\times \mathcal{V}$ is the set of edges containing the pair $(i,j)$ if and only if agent i can communicate with agent j. There is no self-loops in $\mathcal{G}$, i.e., $(i,i)\notin \mathcal{E}$ for any $i\in \left[n\right]$. Note that the edges in $\mathcal{E}$ are enumerated in arbitrary order, with $e_k:=(i,j)\in \mathcal{E}$ denoting the k-th edge, where $k\in \left[m\right]$ and $m:=\left|\mathcal{E}\right|$ is the number of edges. The neighbor set of agent i is ${\mathcal{N}}_i:=\{j\in \mathcal{V}:(i,j)\in \mathcal{E}\}$. Let ${\mathbf{x}}_i\in {\mathbb{R}}^d$ and ${\mathbf{z}}_k\in {\mathbb{R}}^d$ be the local decision vectors corresponding the ith-node kth-edge, respectively. We assume that $\mathcal{G}$ is connected. Then, problem (1) is equivalent to the following constrained form:

$$\begin{array}{cc}\underset{\left\{{\mathbf{x}}_i\right\},\mathit{\theta },\left\{{\mathbf{z}}_{ij}\right\}}{min}& \left\{\sum _{i=1}^n{f}_i\left({\mathbf{x}}_i\right)+g\left(\mathit{\theta }\right)\right\},\hfill \\ \mathrm{s}.\mathrm{t}.& {\mathbf{x}}_i={\mathbf{x}}_j={\mathbf{z}}_k\forall e_k=(i,j)\in \mathcal{E},\hfill \\ & {\mathbf{x}}_l=\mathit{\theta },\mathrm{for}\mathrm{one}\mathrm{arbitrary}l\in \mathcal{V},\hfill \end{array}$$

(2)

where $\mathit{\theta }\in {\mathbb{R}}^d$ is an auxiliary variable for decoupling the smooth and non-smooth functions. Denote the optimal solution of problem (1) and (2) as ${\mathbf{x}}^{★}$ and $\{{\mathbf{x}}_i^{★},{\mathbf{z}}_k^{★},{\mathit{\theta }}^{★}\}$, respectively. It is straightforward to verify that ${\mathbf{x}}^{★}={\mathbf{x}}_i^{★}={\mathbf{z}}_k^{★}={\mathit{\theta }}^{★}$ for all $i\in \left[n\right]$ and $k\in \left[m\right]$.

In what follows, define $\tilde{\mathbf{x}}:=[{\mathbf{x}}_1;{\mathbf{x}}_2;\cdots ;{\mathbf{x}}_n]\in {\mathbb{R}}^{nd}$ and $\tilde{\mathbf{z}}:=[{\mathbf{z}}_1;{\mathbf{z}}_2;\cdots ;{\mathbf{z}}_m]\in {\mathbb{R}}^{md}$. In addition, define two matrices ${\widehat{\mathbf{A}}}_s$ and ${\widehat{\mathbf{A}}}_d\in {\mathbb{R}}^{m\times n}$ as follows: the k-th row of both ${\widehat{\mathbf{A}}}_s$ and ${\widehat{\mathbf{A}}}_d$ represents the k-th edge $e_k$. Specifically, the entries ${\left[\widehat{\mathbf{A}}s\right]}_{ki}$ and ${\left[\widehat{\mathbf{A}}d\right]}_{kj}$ are both equal to 1 if and only if the edge $e_k=(i,j)$; otherwise, they are 0. We also define $\mathbf{S}:=({\mathbf{s}}_l\otimes {\mathbf{I}}_d)\in {\mathbb{R}}^{nd\times d}$, where ${\mathbf{s}}_l\in {\mathbb{R}}^n$ is a vector with a 1 at its l-th position and 0 elsewhere. Clearly, the matrix ${\mathbf{S}}^{\top }$ extracts the component of $\tilde{\mathbf{x}}$ that corresponds to agent l, meaning that ${\mathbf{S}}^{\top }\tilde{\mathbf{x}}={\mathbf{x}}_l$. Let $F\left(\tilde{\mathbf{x}}\right):={\sum }_{i=1}^n{f}_i\left({\mathbf{x}}_i\right)$. Then, problem (2) can be written as

$$\begin{array}{cc}\hfill \underset{\tilde{\mathbf{x}},\mathit{\theta },\tilde{\mathbf{z}}}{min}& F\left(\tilde{\mathbf{x}}\right)+g\left(\mathit{\theta }\right)\hfill \\ \hfill \mathrm{s}.\mathrm{t}.& \mathbf{A}\tilde{\mathbf{x}}=\mathbf{B}\tilde{\mathbf{z}},{\mathbf{S}}^{\top }\tilde{\mathbf{x}}=\mathit{\theta },\mathrm{where}\mathbf{A}=\left[\begin{array}{c}{\widehat{\mathbf{A}}}_s\otimes {\mathbf{I}}_d\\ {\widehat{\mathbf{A}}}_d\otimes {\mathbf{I}}_d\end{array}\right]\mathrm{and}\mathbf{B}=\left[\begin{array}{c}{\mathbf{I}}_{md}\\ {\mathbf{I}}_{md}\end{array}\right].\hfill \end{array}$$

(3)

Note that problem (3) is written from the network level, which will be the basis for designing our algorithm.

3. Algorithm Formulation

In this section, we first introduce the basic iterations of our algorithm based on ADMM method. Then, by combining compressed communication techniques with the ADMM-based algorithm, we will devise our algorithm and discuss its relationship with existing algorithms.

3.1. Background: ADMM-Based Algorithm

ADMM is a powerful tool to solve an optimization problem with several blocks of variables. To apply ADMM for solving problem (3), define its augmented Lagrangian as:

$$\begin{array}{cc}\hfill \mathcal{L}(\tilde{\mathbf{x}},\mathit{\theta },\tilde{\mathbf{z}};\mathit{\nu },\mathit{\lambda }):=& F\left(\tilde{\mathbf{x}}\right)+g\left(\mathit{\theta }\right)+{\mathit{\nu }}^{\top }(\mathbf{A}\tilde{\mathbf{x}}-\mathbf{B}\tilde{\mathbf{z}})+{\mathit{\lambda }}^{\top }({\mathbf{S}}^{\top }\tilde{\mathbf{x}}-\mathit{\theta })\hfill \\ & +{\displaystyle \frac{{\mathsf{\mu }}_z}{2}}{∥\mathbf{A}\tilde{\mathbf{x}}-\mathbf{B}\tilde{\mathbf{z}}∥}^2+{\displaystyle \frac{{\mathsf{\mu }}_{\theta }}{2}}{∥{\mathbf{S}}^{\top }\tilde{\mathbf{x}}-\mathit{\theta }∥}^2,\hfill \end{array}$$

(4)

where ${\mathsf{\mu }}_z$ and ${\mathsf{\mu }}_{\theta }$ are positive constants, $\mathit{\nu }\in {\mathbb{R}}^{2md}$ and $\mathit{\lambda }\in {\mathbb{R}}^d$ are Lagrange multipliers. Then, the kth-iteration in ADMM is written as

$$\begin{array}{cc}\hfill {\tilde{\mathbf{x}}}_{t+1}& =arg\underset{\tilde{\mathbf{x}}}{min}\mathcal{L}(\tilde{\mathbf{x}},{\mathit{\theta }}_t,{\tilde{\mathbf{z}}}_t;{\mathit{\nu }}_t,{\mathit{\lambda }}_t)\hfill \end{array}$$

(5a)

$$\begin{array}{cc}\hfill {\mathit{\theta }}_{t+1}& =arg\underset{\mathit{\theta }}{min}\mathcal{L}({\tilde{\mathbf{x}}}_{t+1},\mathit{\theta },{\tilde{\mathbf{z}}}_t;{\mathit{\nu }}_t,{\mathit{\lambda }}_t)\hfill \end{array}$$

(5b)

$$\begin{array}{cc}\hfill {\tilde{\mathbf{z}}}_{t+1}& =arg\underset{\tilde{\mathbf{z}}}{min}\mathcal{L}({\tilde{\mathbf{x}}}_{t+1},{\mathit{\theta }}_{t+1},\tilde{\mathbf{z}};{\mathit{\nu }}_t,{\mathit{\lambda }}_t)\hfill \end{array}$$

(5c)

$$\begin{array}{cc}\hfill {\mathit{\nu }}_{t+1}& ={\mathit{\nu }}_t+{\mathsf{\mu }}_z(\mathbf{A}{\tilde{\mathbf{x}}}_{t+1}-\mathbf{B}{\tilde{\mathbf{z}}}_{t+1})\hfill \end{array}$$

(5d)

$$\begin{array}{cc}\hfill {\mathit{\lambda }}_{t+1}& ={\mathit{\lambda }}_t+{\mathsf{\mu }}_{\theta }({\mathbf{S}}^{\top }{\tilde{\mathbf{x}}}_{t+1}-{\mathit{\theta }}_{t+1}).\hfill \end{array}$$

(5e)

Define ${\widehat{\mathbf{E}}}_s={\widehat{\mathbf{A}}}_s-{\widehat{\mathbf{A}}}_d$, ${\widehat{\mathbf{E}}}_u={\widehat{\mathbf{A}}}_s+{\widehat{\mathbf{A}}}_d$, ${\widehat{\mathbf{L}}}_s={\widehat{\mathbf{E}}}_s^{\top }{\widehat{\mathbf{E}}}_s$, ${\widehat{\mathbf{L}}}_u={\widehat{\mathbf{E}}}_u^{\top }{\widehat{\mathbf{E}}}_u$, $\widehat{\mathbf{D}}=\frac{1}{2}({\widehat{\mathbf{L}}}_s+{\widehat{\mathbf{L}}}_u)$. In addition, define ${\mathbf{E}}_s={\widehat{\mathbf{E}}}_s\otimes {\mathbf{I}}_d$, and similarly for ${\mathbf{E}}_u,{\mathbf{L}}_s,{\mathbf{L}}_u$, and $\mathbf{D}$. Similar as [22], if we initialize the multiplier ${\mathit{\nu }}_t:=[{\mathit{\alpha }}_t;{\mathit{\beta }}_t]\in {\mathbb{R}}^{2md}$ with ${\mathit{\alpha }}_0=-{\mathit{\beta }}_0$, ${\mathbf{E}}_u{\tilde{\mathbf{x}}}_0=2{\tilde{\mathbf{z}}}_0$, there is ${\mathbf{E}}_u{\tilde{\mathbf{x}}}_t=2{\tilde{\mathbf{z}}}_t$. Let ${\mathit{\varphi }}_t={\mathbf{E}}_s^{\top }{\mathit{\alpha }}_t$, and approximate the augmented Lagrangian $\mathcal{L}(·)$ in (5a) by employing a second-order expansion at ${\mathbf{x}}_t$ as

$$\begin{array}{cc}\hfill & \widehat{\mathcal{L}}(\tilde{\mathbf{x}},{\mathit{\theta }}_t,{\tilde{\mathbf{z}}}_t;{\mathit{\nu }}_t,{\mathit{\lambda }}_t)={\mathcal{L}}_t\left({\tilde{\mathbf{x}}}_t\right)+(\tilde{\mathbf{x}}-{\tilde{\mathbf{x}}}_t){\nabla }_{\tilde{\mathbf{x}}}{\mathcal{L}}_t\left({\tilde{\mathbf{x}}}_t\right)+{\displaystyle \frac{1}{2}}{∥\tilde{\mathbf{x}}-{\tilde{\mathbf{x}}}_t∥}_{{\mathbf{H}}_t}^2,\hfill \end{array}$$

where ${\mathcal{L}}_t\left({\tilde{\mathbf{x}}}_t\right)$ is short for $\mathcal{L}({\tilde{\mathbf{x}}}_t,{\mathit{\theta }}_t,{\tilde{\mathbf{z}}}_t;{\mathit{\nu }}_t,{\mathit{\lambda }}_t)$, and ${\mathbf{H}}_t$ is an invertible matrix representing the approximated Hessian of the augmented Lagrangian, then the iteration (5) can be simplified as

$$\begin{array}{cc}\hfill {\tilde{\mathbf{x}}}_{t+1}& ={\tilde{\mathbf{x}}}_t-{\mathbf{H}}_t^{-1}\left(\nabla F\left({\tilde{\mathbf{x}}}_t\right)+{\mathit{\varphi }}_t+\mathbf{S}{\mathit{\lambda }}_t+{\displaystyle \frac{{\mathsf{\mu }}_z}{2}}{\mathbf{L}}_s{\tilde{\mathbf{x}}}_t+{\mathsf{\mu }}_{\theta }\mathbf{S}\left({\mathbf{S}}^{\top }{\tilde{\mathbf{x}}}_t-{\mathit{\theta }}_t\right)\right)\hfill \end{array}$$

(6a)

$$\begin{array}{cc}\hfill {\mathit{\theta }}_{t+1}& ={\mathbf{prox}}_{g/{\mathsf{\mu }}_{\theta }}\left({\mathbf{S}}^{\top }{\tilde{\mathbf{x}}}_{t+1}+{\mathsf{\mu }}_{\theta }^{-1}{\mathit{\lambda }}_t\right)\hfill \end{array}$$

(6b)

$$\begin{array}{cc}\hfill {\mathit{\varphi }}_{t+1}& ={\mathit{\varphi }}_t+{\displaystyle \frac{{\mathsf{\mu }}_z}{2}}{\mathbf{L}}_s{\tilde{\mathbf{x}}}_{t+1}\hfill \end{array}$$

(6c)

$$\begin{array}{cc}\hfill {\mathit{\lambda }}_{t+1}& ={\mathit{\lambda }}_t+{\mathsf{\mu }}_{\theta }({\mathbf{S}}^{\top }{\tilde{\mathbf{x}}}_{t+1}-{\mathit{\theta }}_{t+1}).\hfill \end{array}$$

(6d)

Compared with (5), the iteration (6) contains fewer vectors by eliminating the vector $\tilde{\mathbf{z}}$ and replacing $\mathit{\nu }$ by $\mathit{\varphi }$ to halve the dimension of $\mathit{\nu }$. Note that the iteration (6) is written in terms of the whole network. To implement (6) in a decentralized manner, we require the matrix ${\mathbf{H}}_t$ be block-diagonal, so that each block can be computed independently by each agent. Here, we assume that ${\mathbf{H}}_t=diag\{{\mathbf{H}}_{1,t},{\mathbf{H}}_{2,t},\cdots ,{\mathbf{H}}_{n,t}\}$. The choice of ${\mathbf{H}}_{i,t}$ will be discussed later. Then, agent i will perform the following iteration:

$$\begin{array}{cc}\hfill {\mathbf{x}}_{i,t+1}& ={\mathbf{x}}_{i,t}-{\mathbf{H}}_{i,t}^{-1}\left(\nabla f\left({\mathbf{x}}_{i,t}\right)+{\mathit{\varphi }}_{i,t}+{\displaystyle \frac{{\mathsf{\mu }}_z}{2}}\sum _{j\in {\mathcal{N}}_i}({\mathbf{x}}_{i,t}-{\mathbf{x}}_{j,t})+{\delta }_{il}\left({\mathit{\lambda }}_t+{\mathsf{\mu }}_{\theta }\left({\mathbf{x}}_{i,t}-{\mathit{\theta }}_t\right)\right)\right)\hfill \end{array}$$

(7a)

$$\begin{array}{cc}\hfill {\mathit{\varphi }}_{i,t+1}& ={\mathit{\varphi }}_{i,t}+{\displaystyle \frac{{\mathsf{\mu }}_z}{2}}\sum _{j\in {\mathcal{N}}_i}({\mathbf{x}}_{i,t+1}-{\mathbf{x}}_{j,t+1})\hfill \end{array}$$

(7b)

$$\begin{array}{cc}\hfill {\mathit{\theta }}_{t+1}& ={\delta }_{il}{\mathbf{prox}}_{g/{\mathsf{\mu }}_{\theta }}\left({\mathbf{x}}_{i,t+1}+{\mathsf{\mu }}_{\theta }^{-1}{\mathit{\lambda }}_t\right)\hfill \end{array}$$

(7c)

$$\begin{array}{cc}\hfill {\mathit{\lambda }}_{t+1}& ={\delta }_{il}\left({\mathit{\lambda }}_t+{\mathsf{\mu }}_{\theta }({\mathbf{x}}_{i,t+1}-{\mathit{\theta }}_{t+1})\right),\hfill \end{array}$$

(7d)

where ${\delta }_{il}=1$ if $i=l$, otherwise ${\delta }_{il}=0$.

3.2. Communication-Efficient Decentralized ADMM

Recalling the iteration (7a) and (7b), it can be seen that agent i will communicate the information on $\mathbf{x}$ to its neighbors at each iteration. However, such communication might not be realized for scenarios with limited communication resources. To reduce the communication overhead, we introduce the idea of compressed communication scheme into iteration (7), which results in our algorithm, termed as communication-efficient decentralized ADMM for composite optimization (CE-DADMM).

To compress the communication, we first give a definition of compressor.

Definition 1

(Compressor). A randomized map $\mathcal{C}:{\mathbb{R}}^d\to {\mathbb{R}}^d$ is called a compressor if there exists a constant $\delta \in [0,1)$ such that $\mathbb{E}{∥\mathcal{C}\left(\mathbf{x}\right)-\mathbf{x}∥}^2\le (1-\delta ){∥\mathbf{x}∥}^2$ holds for any $\mathbf{x}\in {\mathbb{R}}^d$.

In Definition 1, the compressor is characterized using the relationship between the compression error and the original state. Clearly, $\delta$ refers to the compression ratio. We can also call $\mathcal{C}$ a $\delta$-compressor. In our algorithm, we do not apply $\mathcal{C}(·)$ directly to the transmitted $\mathbf{x}$, as it will lead to a non-dismissing compression error. Instead, we will adopt a general compressor termed as Three Point Compressor (3PC) [41], whose definition is given below.

Definition 2

(Three Point Compressor, see [41]). A randomized map ${\mathcal{C}}_{\mathbf{h},\mathbf{y}}:{\mathbb{R}}^d\times {\mathbb{R}}^d\times {\mathbb{R}}^d\to {\mathbb{R}}^d$ is called a three point compressor (3PC) if there exist constants $0<A\le 1$ and $B\ge 0$ such that

$$\begin{array}{c}\hfill \mathbb{E}\left[{∥{\mathcal{C}}_{\mathbf{h},\mathbf{y}}\left(\mathbf{x}\right)-\mathbf{x}∥}^2\right]\le (1-A){∥\mathbf{h}-\mathbf{y}∥}^2+B{∥\mathbf{x}-\mathbf{y}∥}^2,\forall \mathbf{x},\mathbf{y},\mathbf{h}\in {\mathbb{R}}^d,\end{array}$$

(8)

where $\mathbf{y}$ and $\mathbf{h}$ are parameters of the compressor.

3PC can be realized using $\delta$-compressor $\mathcal{C}$. Two examples of ${\mathcal{C}}_{\mathbf{h},\mathbf{y}}$ are given below [41]:

$$\begin{array}{cc}\mathrm{E}\mathrm{F}21:\hfill & \hfill {\mathcal{C}}_{\mathbf{h},\mathbf{y}}\left(\mathbf{x}\right):=\mathbf{h}+\mathcal{C}(\mathbf{x}-\mathbf{h}),\end{array}$$

(9)

$$\begin{array}{cc}\mathrm{C}\mathrm{L}\mathrm{A}\mathrm{G}:\hfill & \hfill {\mathcal{C}}_{\mathbf{h},\mathbf{y}}\left(\mathbf{x}\right):=\left\{\begin{array}{cc}\mathbf{h}+\mathcal{C}(\mathbf{x}-\mathbf{h}),\hfill & {\mathrm{if}\Vert \mathbf{x}-\mathbf{h}\Vert }^2>\varsigma {\Vert \mathbf{x}-\mathbf{y}\Vert }^2\hfill \\ \mathbf{h},\hfill & \mathrm{otherwise}\hfill \end{array}\right..\end{array}$$

(10)

It can be checked that $A:=1-\sqrt{\delta }$ and $B:={\displaystyle \frac{\delta }{1-\sqrt{\delta }}}$ for EF21, and $A:=1-\sqrt{\delta }$ and $B:=max\left\{{\displaystyle \frac{\delta }{1-\sqrt{\delta }}},\varsigma \right\}$ for CLAG.

Next, we formulate our algorithm based on 3PC, whose pseudo code is presented in Algorithm 1. We introduce a new state ${\mathbf{y}}_{i,t}$ relating to ${\mathbf{x}}_{i,t}$, which refers to the estimation on ${\mathbf{x}}_{i,t}$ of agent i’s neighbors. The computation of ${\mathbf{H}}_{i,t}{\left(y_{i,t}\right)}^{-1}$ relies on ${\mathbf{y}}_{i,t}$, and since ${\mathbf{y}}_{i,t}$ consists of compressed information, this significantly reduces the computational cost associated with the inverse of ${\mathbf{H}}_{i,t}\left(y_{i,t}\right)$. Then, at iteration t, agent i transmits the compressed vector ${\mathcal{C}}_{{\mathbf{y}}_{i,t-1},{\mathbf{x}}_{i,t-1}}\left({\mathbf{x}}_{i,t}\right)$ rather than ${\mathbf{x}}_{i,t}$ to its neighbors, which lead to the new iterations as below:

$$\begin{array}{cc}\hfill {\mathbf{x}}_{i,t+1}=& {\mathbf{x}}_{i,t}-{({\mathbf{H}}_{i,t}({\mathbf{y}}_{i,t}))}^{-1}(\nabla f\left({\mathbf{x}}_{i,t}\right)+{\mathit{\varphi }}_{i,t}+{\displaystyle \frac{{\mathsf{\mu }}_z}{2}}\sum _{j\in {\mathcal{N}}_i}({\mathbf{y}}_{i,t}-{\mathbf{y}}_{j,t})\hfill \end{array}$$

$$\begin{array}{c}+{\delta }_{il}\left({\mathit{\lambda }}_t+{\mathsf{\mu }}_{\theta }({\mathbf{y}}_{i,t}-{\mathit{\theta }}_t)\right))\hfill \end{array}$$

(11a)

$$\begin{array}{cc}\hfill {\mathit{\varphi }}_{i,t+1}=& {\mathit{\varphi }}_{i,t}+{\displaystyle \frac{{\mathsf{\mu }}_z}{2}}\sum _{j\in {\mathcal{N}}_i}({\mathbf{y}}_{i,t+1}-{\mathbf{y}}_{j,t+1})\hfill \end{array}$$

(11b)

$$\begin{array}{cc}\hfill {\mathit{\theta }}_{t+1}=& {\delta }_{il}{\mathbf{prox}}_{g/{\mathsf{\mu }}_{\theta }}\left({\mathbf{y}}_{l,t+1}+{\mathsf{\mu }}_{\theta }^{-1}{\mathit{\lambda }}_t\right)\hfill \end{array}$$

(11c)

$$\begin{array}{cc}\hfill {\mathit{\lambda }}_{t+1}=& {\delta }_{il}\left({\mathit{\lambda }}_t+{\mathsf{\mu }}_{\theta }({\mathbf{y}}_{i,t+1}-{\mathit{\theta }}_{t+1})\right).\hfill \end{array}$$

(11d)

Algorithm 1 CE-DADMM
1: Initialization: ${\mathbf{x}}_{i,0}=\mathbf{0}$, ${\mathbf{y}}_{i,0}=\mathbf{0}$, ${\mathit{\varphi }}_{i,0}=\mathbf{0}$, ${\mathit{\theta }}_0=\mathbf{0}$, ${\mathit{\lambda }}_0=\mathbf{0}$, ${\mathbf{p}}_{i,t}=\nabla f_i\left(\mathbf{0}\right)$.
2: for t = 0,1,… do
3:     for agent i do
4:         Compute ${\mathbf{H}}_{i,t}^{-1}$ using (13), (14), or (15) according to its choice;
5:         Compute ${\mathbf{x}}_{i,t+1}$ using (11a);
6:         ${\mathbf{y}}_{i,t+1}={\mathcal{C}}_{{\mathbf{y}}_{i,t},{\mathbf{x}}_{i,t}}\left({\mathbf{x}}_{i,t+1}\right)$	// Compressing information
7:         Broadcast ${\mathbf{y}}_{i,t+1}$ to neighbors
8:         ${\mathit{\varphi }}_{i,t+1}={\mathit{\varphi }}_{i,t}+{\displaystyle \frac{{\mathsf{\mu }}_z}{2}}{\sum }_{j\in {\mathcal{N}}_i}({\mathbf{y}}_{i,t+1}-{\mathbf{y}}_{j,t+1})$
9:         if $i=l$ then	// Dealing with the non-smooth function
10:            ${\mathit{\theta }}_{t+1}={\mathbf{prox}}_{g/{\mathsf{\mu }}_{\theta }}({\mathbf{y}}_{l,t+1}+{\displaystyle \frac{1}{{\mathsf{\mu }}_{\theta }}}{\mathit{\lambda }}_t)$
11:            ${\mathit{\lambda }}_{t+1}={\mathit{\lambda }}_t+{\mathsf{\mu }}_{\theta }({\mathbf{y}}_{l,t+1}-{\mathit{\theta }}_{t+1})$
12:         end if
13:     end for
14: end for

3.3. Discussion

Our algorithm CE-DADMM presents a flexible framework that accommodates gradient updates, Newton updates, and quasi-Newton updates, depending on the choice of matrix ${\mathbf{H}}_{i,t}^{-1}$. Here, ${\mathbf{H}}_{i,t}^{-1}$ has the following general structure:

$$\begin{array}{c}\hfill {\mathbf{H}}_{i,t}^{-1}={\left({\mathbf{J}}_{i,t}+\left({\mathsf{\mu }}_z\left|{\mathcal{N}}_i\right|+{\delta }_{il}{\mathsf{\mu }}_{\theta }+ϵ\right){\mathbf{I}}_d\right)}^{-1},\end{array}$$

(12)

where $ϵ>0$ is used to provide additional robustness and ${\mathbf{J}}_{i,t}$ is a matrix to be determined. A detailed discussion is presented below.

Case 1: Gradient Updates. By choosing ${\mathbf{J}}_{i,t}\equiv \mathbf{0}$, (12) equals to

$$\begin{array}{c}\hfill {\mathbf{H}}_{i,t}^{-1}={\left(\left({\mathsf{\mu }}_z\left|{\mathcal{N}}_i\right|+{\delta }_{il}{\mathsf{\mu }}_{\theta }+ϵ\right){\mathbf{I}}_d\right)}^{-1}.\end{array}$$

(13)

Clearly, ${\mathbf{H}}_t$ is diagonal. The computation of ${\mathbf{H}}_t^{-1}$ requires $\mathcal{O}\left(d\right)$ computational cost. Compared with COLA [45], CE-DADMM considers the presence of the non-smooth term $g(·)$ and allows for more options in the choice of the compression mechanism $\mathcal{C}$. When $g(·)$ is excluded, only the lazy aggregation compression mechanism is applied, and Gradient Updates are used, CE-DADMM aligns with the form of COLA.

Case 2: Newton Updates. By choosing ${\mathbf{J}}_{i,t}={\nabla }^2{f}_i\left({\mathbf{y}}_{i,t}\right)$, (12) equals to

$$\begin{array}{c}\hfill {\mathbf{H}}_{i,t}^{-1}={\left({\nabla }^2{f}_i\left({\mathbf{y}}_{i,t}\right)+\left({\mathsf{\mu }}_z\left|{\mathcal{N}}_i\right|+{\delta }_{il}{\mathsf{\mu }}_{\theta }+ϵ\right){\mathbf{I}}_d\right)}^{-1}.\end{array}$$

(14)

According to the definition of $F\left({\tilde{\mathbf{y}}}_t\right)$, ${\nabla }^{2}F\left({\tilde{\mathbf{y}}}_t\right)$ is a block diagonal matrix with the ith block being ${\nabla }^2{f}_i\left({\mathbf{y}}_{i,t}\right)$. The computation of ${\mathbf{H}}_t^{-1}$ incurs $\mathcal{O}\left(d^3\right)$ computational cost. When CE-DADMM uses Newton updates, excludes $g(·)$, and adopts the same communication compression approach as CC-DQM [39], it recovers the form consistent with CC-DQM.

Case 3: Quasi-Newton Updates. Inspire by the distributed BFGS scheme in [22], we can derive a novel decentralized algorithm termed as CE-DADMM-BFGS, which combines the BFGS method with communication-efficient mechanisms. According to secant condition, each agent i constructs a model of the inverse Hessian directly using the pairs ${\{{\mathbf{q}}_{i,t},{\mathbf{s}}_{i,t}\}}_{i=1}^n$ defined as

$$\begin{array}{c}\hfill {\mathbf{q}}_{i,t}:=(\nabla f_i\left({\mathbf{y}}_{i,t+1}\right)-\nabla f_i\left({\mathbf{y}}_{i,t}\right))+({\mathsf{\mu }}_z\left|{\mathcal{N}}_i\right|+{\delta }_{il}{\mathsf{\mu }}_{\theta }+ϵ){\mathbf{s}}_{i,t}\mathrm{and}{\mathbf{s}}_{i,t}:={\mathbf{y}}_{i,t+1}-{\mathbf{y}}_{i,t}.\end{array}$$

The Hessian inverse approximation is then iteratively updated as:

$$\begin{array}{c}\hfill {\mathbf{H}}_{i,t+1}^{-1}={\displaystyle \frac{{\mathbf{s}}_{i,t}{\left({\mathbf{s}}_{i,t}\right)}^{\top }}{{\mathbf{q}}_{i,t}^{\top }{\mathbf{s}}_{i,t}}}+\left({\mathbf{I}}_d-{\displaystyle \frac{{\mathbf{s}}_{i,t}{\left({\mathbf{q}}_{i,t}\right)}^{\top }}{{\mathbf{q}}_{i,t}^{\top }{\mathbf{s}}_{i,t}}}\right){\left({\mathbf{H}}_{i,t}\right)}^{-1}\left({\mathbf{I}}_d-{\displaystyle \frac{{\mathbf{q}}_{i,t}{\left({\mathbf{s}}_{i,t}\right)}^{\top }}{{\mathbf{q}}_{i,t}^{\top }{\mathbf{s}}_{i,t}}}\right).\end{array}$$

(15)

Notably, the explicit inverse of ${\mathbf{H}}_{i,t+1}$ is unnecessary, as this expression serves merely as a formal representation. Consequently, the computational cost for each agent is reduced from $\mathcal{O}\left(d^3\right)$ to $\mathcal{O}\left(d^2\right)$.

4. Convergence Analysis

In this section, we propose a unified framework to analyze the proposed algorithms that incorporate gradient, Newton, and BFGS updates, along with a communication-efficient mechanism. First, we make the following assumptions throughout of the paper.

Assumption 1.

Each $f_i$ is twice continuously differentiable, $m_f$-strongly convex, and $M_f$–smooth, i.e., $m_f{\mathbf{I}}_d⪯{\nabla }^2{f}_i\left({\mathbf{x}}_i\right)⪯M_f{\mathbf{I}}_d$, where $M_f\ge m_f>0$. $g(·):{\mathbb{R}}^d\to \mathbb{R}$ is proper, closed, and convex, i.e., ${(\mathbf{x}-\mathbf{y})}^{\top }({\mathbf{s}}_{\mathbf{x}}-{\mathbf{s}}_{\mathbf{y}})\ge 0$ holds for any subgradients ${\mathbf{s}}_{\mathbf{x}}\in \partial g\left(\mathbf{x}\right)$ and ${\mathbf{s}}_{\mathbf{y}}\in \partial g\left(\mathbf{y}\right)$.

Assumption 2.

Each ${\nabla }^2{f}_i$ is Lipschitz continuous with constant $L_f$, i.e., $\Vert {\nabla }^2{f}_i\left(\mathbf{x}\right)-{\nabla }^2{f}_i\left(\mathbf{y}\right)\Vert \le L_f\Vert \mathbf{x}-\mathbf{y}\Vert$ holds for any $\mathbf{x},\mathbf{y}\in {\mathbb{R}}^d.$

Assumption 3.

Each ${\mathbf{H}}_{i,t}$ is uniformly upper bounded, i.e., for any $t\ge 0$, there exists a constant $\mathit{\psi }>0$ such that ${\mathbf{H}}_{i,t}⪯\mathit{\psi }{\mathbf{I}}_d.$

It is worthy noting that Assumption 3 is only required for the quasi-Newton update case. Next, we introduce the optimal condition of problem (3), which is independent of the algorithm and has been proved in [22]. The result is given below.

Lemma 1

(optimal condition, see Lemma 2 in [22]). Suppose $({\tilde{\mathbf{x}}}^{★},{\tilde{\mathbf{z}}}^{★},{\mathit{\alpha }}^{★},{\mathit{\theta }}^{★},{\mathit{\lambda }}^{★})$ is a primal-dual optimal pair of problem (3), if and only if the following holds:

$$\begin{array}{cc}\hfill \nabla F\left({\tilde{\mathbf{x}}}^{★}\right)+{\mathbf{E}}_s^{\top }{\mathit{\alpha }}^{★}+\mathbf{S}{\mathit{\lambda }}^{★}& =0,\hfill \end{array}$$

(16a)

$$\begin{array}{cc}\hfill \partial g\left({\mathit{\theta }}^{★}\right)-{\mathit{\lambda }}^{★}& \ni 0,\hfill \end{array}$$

(16b)

$$\begin{array}{cc}\hfill {\mathbf{E}}_s{\tilde{\mathbf{x}}}^{★}& =0,\hfill \end{array}$$

(16c)

$$\begin{array}{cc}\hfill {\mathbf{E}}_u{\tilde{\mathbf{x}}}^{★}& =2{\tilde{\mathbf{z}}}^{★},\hfill \end{array}$$

(16d)

$$\begin{array}{cc}\hfill {\mathbf{S}}^{\top }{\tilde{\mathbf{x}}}^{★}& ={\mathit{\theta }}^{★}.\hfill \end{array}$$

(16e)

Moreover, there exists a unique dual optimal pair $\left[{\mathit{\alpha }}^{★};{\mathit{\lambda }}^{★}\right]\in {\mathbb{R}}^{(m+1)d}$ that lies in the column space of $\mathbf{C}:=\left[\begin{array}{c}{\mathbf{E}}_s\\ {\mathbf{S}}^{\top }\end{array}\right]\in {\mathbb{R}}^{(m+1)d\times nd}$.

Now, we are ready to analyze our algorithm CE-DADMM. First, we write (11) into a compact form:

$$\begin{array}{cc}\hfill {\tilde{\mathbf{x}}}_{t+1}& ={\tilde{\mathbf{x}}}_t-{\left({\mathbf{H}}_t\right)}^{-1}\left(\nabla F\left({\tilde{\mathbf{x}}}_t\right)+{\mathit{\varphi }}_t+{\displaystyle \frac{{\mathsf{\mu }}_z}{2}}{\mathbf{L}}_s{\tilde{\mathbf{y}}}_t+\mathbf{S}\left({\mathit{\lambda }}_t+{\mathsf{\mu }}_{\theta }\left({\mathbf{S}}^{\top }{\tilde{\mathbf{y}}}_t-{\mathit{\theta }}_t\right)\right)\right)\hfill \end{array}$$

(17a)

$$\begin{array}{cc}\hfill {\mathit{\varphi }}_{t+1}& ={\mathit{\varphi }}_t+{\displaystyle \frac{{\mathsf{\mu }}_z}{2}}{\mathbf{L}}_s{\tilde{\mathbf{y}}}_{t+1}\hfill \end{array}$$

(17b)

$$\begin{array}{cc}\hfill {\mathit{\theta }}_{t+1}& ={\mathbf{prox}}_{g/{\mathsf{\mu }}_{\theta }}({\mathbf{S}}^{\top }{\tilde{\mathbf{y}}}_{t+1}+{\displaystyle \frac{1}{{\mathsf{\mu }}_{\theta }}}{\mathit{\lambda }}_t)\hfill \end{array}$$

(17c)

$$\begin{array}{cc}\hfill {\mathit{\lambda }}_{t+1}& ={\mathit{\lambda }}_t+{\mathsf{\mu }}_{\theta }({\mathbf{S}}^{\top }{\tilde{\mathbf{y}}}_{t+1}-{\mathit{\theta }}_{t+1}).\hfill \end{array}$$

(17d)

According to the discussion in Section 3.1, we have ${\mathbf{E}}_u{\tilde{\mathbf{x}}}_t=2{\tilde{\mathbf{z}}}_t$ and ${\mathit{\varphi }}_t={\mathbf{E}}_s^{\top }{\mathit{\alpha }}_t$ hold in (17). Then, it follows from Lemma 1 that the convergence of CE-DADMM can be obtained by showing $({\tilde{\mathbf{x}}}^t,{\tilde{\mathbf{z}}}^t,{\mathit{\alpha }}^t,{\mathit{\theta }}^t,{\mathit{\lambda }}^t)$ converges to $({\tilde{\mathbf{x}}}^{★},{\tilde{\mathbf{z}}}^{★},{\mathit{\alpha }}^{★},{\mathit{\theta }}^{★},{\mathit{\lambda }}^{★})$.

Due to the existence of the efficient communication scheme, we need to analyze the impact of communication error on the convergence of CE-DADMM. Define $\tilde{\mathbf{e}}:=[{\mathbf{e}}_1;{\mathbf{e}}_2;\cdots ;{\mathbf{e}}_n]$, where ${\mathbf{e}}_i:={\mathbf{y}}_i-{\mathbf{x}}_i$ for all agent i. Clearly, $\tilde{\mathbf{e}}$ describes the error caused by efficient communication scheme. Regarding $\tilde{\mathbf{e}}$, the following result holds.

Lemma 2.

The error ${\tilde{\mathbf{e}}}_{t+1}$ in CE-DADMM satisfies ${∥{\tilde{\mathbf{e}}}_{t+1}∥}^2\le (1-A){∥{\tilde{\mathbf{e}}}_t∥}^2+B{∥{\tilde{\mathbf{x}}}_{t+1}-{\tilde{\mathbf{x}}}_t∥}^2$, where A and B are the parameters of 3PC.

Proof. 

According to the definition of 3PC, we have

$$\begin{array}{cc}\hfill \Vert {\tilde{\mathbf{e}}}_{t+1}{\Vert }^2& =\Vert {\tilde{\mathbf{y}}}_{t+1}-{\tilde{\mathbf{x}}}_{t+1}{\Vert }^2\le (1-A)\Vert {\tilde{\mathbf{e}}}_t{\Vert }^2+B{\Vert {\tilde{\mathbf{x}}}_{t+1}-{\tilde{\mathbf{x}}}_t\Vert }^2,\hfill \end{array}$$

which completes the proof. □

Clearly, if 3PC is set as EF21 (9), it follows from Lemma 2 that

$$\begin{array}{c}\hfill {∥{\tilde{\mathbf{e}}}_{t+1}∥}^2\le \sqrt{\delta }{∥{\tilde{\mathbf{e}}}_t∥}^2+{\displaystyle \frac{\delta }{1-\sqrt{\delta }}}{∥{\tilde{\mathbf{x}}}_{t+1}-{\tilde{\mathbf{x}}}_t∥}^2.\end{array}$$

If 3PC is set as CLAG (10), we have

$$\begin{array}{c}\hfill {∥{\tilde{\mathbf{e}}}_{t+1}∥}^2\le \sqrt{\delta }{∥{\tilde{\mathbf{e}}}_t∥}^2+max\left\{{\displaystyle \frac{\delta }{1-\sqrt{\delta }}},\varsigma \right\}{∥{\tilde{\mathbf{x}}}_{t+1}-{\tilde{\mathbf{x}}}_t∥}^2.\end{array}$$

Then, to characterize the suboptimality of the iterates when (5a) is replaced by (17a), we introduce the following error term:

$$\begin{array}{c}\hfill {\mathbf{r}}_t:=\nabla F\left({\tilde{\mathbf{x}}}_t\right)-\nabla F\left({\tilde{\mathbf{x}}}_{t+1}\right)+{\mathbf{J}}_t({\tilde{\mathbf{x}}}_{t+1}-{\tilde{\mathbf{x}}}_t).\end{array}$$

(18)

The bound of the error term (18) is give below, which is important for our main result.

Lemma 3.

It holds that $\Vert {\mathbf{r}}_t\Vert \le {\tau }_t\Vert {\tilde{\mathbf{x}}}_{t+1}-{\tilde{\mathbf{x}}}_t\Vert +{\gamma }_t(\Vert {\tilde{\mathbf{e}}}_{t+1}\Vert +\Vert {\tilde{\mathbf{e}}}_t\Vert )$, where ${\tau }_t$ and ${\gamma }_t$ correspond to the update case as below:

$$\begin{array}{cccc}Case1:\hfill & \hfill {\tau }_t=& M_f,\hfill & \hfill {\gamma }_t=0.\end{array}$$

(19)

$$\begin{array}{cccc}Case2:\hfill & \hfill {\tau }_t=& min\left\{2M_f,{\displaystyle \frac{L_f}{2}}\Vert {\tilde{\mathbf{x}}}_{t+1}-{\tilde{\mathbf{x}}}_t\Vert +L_f\Vert {\tilde{\mathbf{e}}}_t\Vert \right\},\hfill & \hfill {\gamma }_t=0.\end{array}$$

(20)

$$\begin{array}{ccccc}\hfill & Case3:\hfill & \hfill {\tau }_t\le & 2\mathit{\psi },\hfill & \hfill {\gamma }_t\le 2(M_f+\mathit{\psi }).\end{array}$$

(21)

Proof. 

See Appendix A. □

Lemma 3 extends the results in [17,18] by providing an upper bound on the error introduced when replacing the exact sub-optimization step (5a) with a one-step update using the compressed variable (17a). Under Newton updates, as the error $\Vert {\tilde{\mathbf{e}}}_t\Vert$ approaches zero, the term ${\displaystyle \frac{L_f}{2}}\Vert {\tilde{\mathbf{x}}}_{t+1}-{\tilde{\mathbf{x}}}_t\Vert +L_f\Vert {\tilde{\mathbf{e}}}_t\Vert$ becomes smaller than $2M_f$ in (20). In the case of Quasi-Newton updates, the error $\Vert {\mathbf{r}}_t\Vert$ remains bounded by a constant, ensuring it do not grow indefinitely.

Let ${\sigma }_{max}^{{\mathbf{L}}_u}$ and ${\sigma }_{min}^{{\mathbf{L}}_u}$ denote the maximum and minimum eigenvalues of ${\mathbf{L}}_u$, respectively. Let ${\sigma }_{max}^{{\mathbf{L}}_s}$ denote the maximum eigenvalue of ${\mathbf{L}}_s$. Denote by ${\sigma }_{min}^{+}$ the smallest positive eigenvalue of $\mathbf{C}{\mathbf{C}}^{\top }$, where $\mathbf{C}$ is given in Lemma 1. Define ${\tilde{\mathbf{v}}}_t:={[{\tilde{\mathbf{x}}}_t,{\tilde{\mathbf{z}}}_t,{\mathit{\alpha }}_t,{\mathit{\theta }}_t,{\mathit{\lambda }}_t]}^{\top }$, ${\tilde{\mathbf{v}}}^{★}:={[{\mathbf{x}}_{★},{\mathbf{z}}_{★},{\mathit{\alpha }}_{★},{\mathit{\theta }}_{★},{\mathit{\lambda }}_{★}]}^{\top }$, and ${\mathcal{H}}_1=diag[ϵ,2{\mathsf{\mu }}_z,{\displaystyle \frac{2}{{\mathsf{\mu }}_z}},{\mathsf{\mu }}_{\theta },{\displaystyle \frac{1}{{\mathsf{\mu }}_{\theta }}}]$. Consider the following Lyapunov function:

$$\begin{array}{c}\hfill V_t=\Vert {\tilde{\mathbf{v}}}_t-{\tilde{\mathbf{v}}}_{★}{\Vert }_{{\mathcal{H}}_1}^2+\xi {\Vert {\tilde{\mathbf{e}}}_t\Vert }^2.\end{array}$$

(22)

Clearly, $V_t$ converges to zero implies that ${\tilde{\mathbf{v}}}_t$ converges to the optimal solution. We will use $V_t$ to establish the convergence result of our algorithm, which is given below.

Theorem 1.

Suppose Assumptions 1–3 hold. Let ${\mathsf{\mu }}_z=2{\mathsf{\mu }}_{\theta }$, $c_1=7{\sigma }_{max}^{{\mathbf{L}}_s}{\mathsf{\mu }}_{\theta }+25{\mathsf{\mu }}_{\theta }$, $c_2=3{\sigma }_{max}^{{\mathbf{L}}_s}{\mathsf{\mu }}_{\theta }+17{\mathsf{\mu }}_{\theta }$, and $c_3=2{\mathsf{\mu }}_{\theta }^2\left(1+{\left({\sigma }_{max}^{{\mathbf{L}}_s}\right)}^2\right)+4{\gamma }_t^2$. If ζ satisfies

$$\begin{array}{cc}\hfill \zeta \in & \left({\displaystyle \frac{2m_f{M}_f}{m_f+M_f}}-{\mathsf{\mu }}_{\theta },min\left\{{\displaystyle \frac{ϵ+B(6-c_1-{\mathsf{\mu }}_{\theta })}{{\tau }_t^2+2B{\gamma }_t^2}},{\displaystyle \frac{4-c_2+(1-A)(6-c_1-{\mathsf{\mu }}_{\theta })}{2(2-A){\tau }_t^2}}\right\}\right),\hfill \end{array}$$

then the iterates generated by CE-DADMM satisfy $V_{t+1}\le {\displaystyle \frac{1}{1+{\eta }_t}}{V}_t$, where

$$\begin{array}{cc}\hfill {\eta }_t=& min\left\{{\displaystyle \frac{ϵ+B(6-c_1-{\mathsf{\mu }}_{\theta })-\zeta ({\tau }_t^2+2B{\gamma }_t^2)}{8B+\frac{7}{{\mathsf{\mu }}_{\theta }{\sigma }_{min}^{+}}({ϵ}^2+2{\tau }_t^2+c_{3}B)}},{\displaystyle \frac{\frac{2m_f{M}_f}{m_f+M_f}-\frac{1}{\zeta }-{\mathsf{\mu }}_{\theta }}{ϵ+4{\mathsf{\mu }}_{\theta }+{\sigma }_{max}^{{\mathbf{L}}_u}{\mathsf{\mu }}_{\theta }}},{\displaystyle \frac{3{\sigma }_{min}^{+}}{{\sigma }_{max}^{{\mathbf{L}}_u}}},\right.\hfill \\ & \left.{\displaystyle \frac{4-c_2+(1-A)(6-c_1-{\mathsf{\mu }}_{\theta })-2(2-A)\zeta {\tau }_t^2}{(1-A)8+\frac{7}{{\mathsf{\mu }}_{\theta }{\sigma }_{min}^{+}}(2-A)c_3}},{\displaystyle \frac{{\mathsf{\mu }}_{\theta }{\sigma }_{min}^{+}}{7(m_f+M_f)}},{\displaystyle \frac{{\sigma }_{min}^{+}}{7}},{\displaystyle \frac{3}{16}}\right\}.\hfill \end{array}$$

(23)

Proof. 

See Appendix B. □

Clearly, Theorem 1 implies that CE-DADMM can achieve exact linear convergence at a rate of $\mathcal{O}\left({\mathit{\rho }}^t\right)$ with $\mathit{\rho }={\displaystyle \frac{1}{1+{\eta }_t}}$. The larger ${\eta }_t$ is, the faster CE-DADMM converges. To ensure positive ${\eta }_t$ in (23), it can be obtain that the step size ${\mathsf{\mu }}_{\theta }$ should not be too large. Besides, excessive compression of ${\mathbf{x}}_k$ should be avoided. This is because, when the compression ratio A is very small, B approaches infinity. According to (23), as $B\to \infty$, the first term in (23) becomes negative. It is worth noting that when ${\eta }_t,{\gamma }_t=0$, the compression ratio A can be arbitrarily small, making the approach highly applicable in scenarios with extremely limited bandwidth. Also, at this situation, we can get the fastest convergence rate and the smallest communication cost. However, setting ${\eta }_t,{\gamma }_t=0$ implies solving the subproblem (5a) exactly, which may result in high computational cost. This shows a trade-off between communication cost and computation cost in decentralized optimization.

5. Numerical Experiments

In this section, CE-DADMM is compared with existing state-of-the-art algorithms, including DRUID [22], PG-EXTRA [42], P2D2 [43], and CC-DQM [39], in distributed logistic/ridge/lasso regression problems. Noting that CC-DQM do not support non-smooth terms.

Datasets. We use real-world datasets from the LIBSVM library (Available Online: https://www.csie.ntu.edu.tw/~cjlin/libsvmtools/datasets/binary.html, accessed on 3 December 2024): a9a (32,561 samples, 123 dimensions) and ijcnn1 (49,990 samples, 22 dimensions). The samples are evenly distributed across n agents. The distribution of samples across agents for the a9a dataset (left) and ijcnn1 (right) is shown in Figure 1. Our experiments are implemented in Python 3.10.13.

Figure 1. Distribution of samples across agents for the a9a dataset (left) and ijcnn1 (right).

Experimental setting. The communication graph is randomly generated, with connections based on a Bernoulli distribution ($p=0.5$) among $n=10$ agents, as shown in Figure 2. We evaluate performance based on the total communication bits and the number of iterations. CE-DADMM employs compression mechanisms EF21 and CLAG, using a Top-K compressor to reduce dimensions to 30 dimensions for the a9a dataset and 6 dimensions for the ijcnn1 dataset. In the experiment, we examine the algorithms from two perspectives: the number of iterations and total communication bits. The number of iterations refers to the number of times the algorithm runs, while total communication bits is calculated based on the cumulative number of bits of variable $\mathbf{y}$ transmitted between agents. Additionally, we define $er{r}_t:={\displaystyle \frac{\Vert {\mathbf{x}}_t-{\mathbf{x}}^{★}\Vert }{\Vert {\mathbf{x}}_0-{\mathbf{x}}^{★}\Vert }}$ to measure the algorithm’s convergence.

Figure 2. Random communication graph of network with 10 agents.

5.1. Distributed Logistic Regression

The distributed logistics regression solves problem (1) with $g\left(\mathbf{x}\right)={\gamma }_2{\Vert \mathbf{x}\Vert }_1$, $f_i(·):{\mathbb{R}}^d\to \mathbb{R}$ defined as:

$$\begin{array}{c}\hfill f_i\left(\mathbf{x}\right):={\displaystyle \frac{1}{m_i}}\sum _{j=1}^{m_i}log\left(1+e^{-b_{ij}{\mathbf{a}}_{ij}^{\top }\mathbf{x}}\right)+{\displaystyle \frac{{\gamma }_1}{2}}{\Vert \mathbf{x}\Vert }^2,\end{array}$$

(24)

where ${\mathbf{a}}_{ij}\in {\mathbb{R}}^d$ represents the feature vector, $b_{ij}\in \{-1,1\}$ denotes the label, and $m_i$ is the number of sample data for agent i. The parameters ${\gamma }_1={10}^{-2}$ and ${\gamma }_2={10}^{-6}$ are regularization terms.

In Figure 3 and Figure 4, when measured by the number of iterations, CE-DADMM with EF21 and CLAG compression mechanisms performs on par with DRUID without compression, and surpasses P2D2 and PG-EXTRA in both convergence speed and accuracy. Additionally, we observe that CE-DADMM, when using (quasi) Newton methods, significantly reduces the number of iterations required to reach a given accuracy compared to first-order methods. When measured by total communication bits, the introduction of EF21 and CLAG in CE-DADMM allows for a substantial reduction in communication overhead compared to DRUID, even under the same update scheme. Notably, when using quasi-Newton updates, CE-DADMM requires fewer communication bits to achieve the same accuracy than DRUID-Newton updates, and also outperforms P2D2 and PG-EXTRA in this regard. When achieving the same convergence accuracy as shown in Table 1, the detailed numerical results are presented in Table 2 and Table 3. Note that for P2D2 and PG-EXTRA, since they fail to reach the predefined convergence accuracy, we use the number of iterations required for them to converge to their optimal values as a benchmark.

Figure 3. Performance comparison of distributed logistic regression the on a9a dataset: Plots of iteration number (left) and total communication bits (right) versus distance error.

Figure 4. Performance comparison of distributed logistic regression the on ijcnn1 dataset: Plots of iteration number (left) and total communication bits (right) versus distance error.

Table 1. Convergence accuracy ($er{r}_t$) for different experiments.

	Distributed Logistic Regression		Distributed Ridge Regression		Distributed LASSO
	a9a	ijcnn1	a9a	ijcnn1	a9a	ijcnn1
Convergence Error ($er{r}_t$)	$1\times {10}^{-4.5}$	$1\times {10}^{-5.5}$	$1\times {10}^{-6.8}$	$1\times {10}^{-6.8}$	$1\times {10}^{-4.5}$	$1\times {10}^{-5.5}$

Table 2. Comparison of iterations.

Method	Distributed		Distributed		Distributed
	Logistic Regression		Ridge Regression		LASSO
	a9a	ijcnn1	a9a	ijcnn1	a9a	ijcnn1
P2D2	468	811	-	-	4230	409
PG-EXTRA	470	813	-	-	4231	410
CC-DQM	-	-	246	79	-	-
DRUID-Gradient	845	884	977	176	3348	445
CE-DADMM-Gradient:EF21	845	884	977	171	3348	446
CE-DADMM-Gradient:CLAG	845	884	980	170	3349	446
DRUID-Newton	154	559	197	60	1910	318
CE-DADMM-Newton:EF21	155	559	252	60	1912	318
CE-DADMM-Newton:CLAG	155	559	194	64	1912	318
DRUID-BFGS	684	566	330	77	2890	399
CE-DADMM-BFGS:EF21	478	567	325	73	2890	400
CE-DADMM-BFGS:CLAG	478	566	327	69	2890	399

Table 3. Comparison of communication bits.

Method	Distributed		Distributed		Distributed
	Logistic Regression		Ridge Regression		LASSO
	a9a	ijcnn1	a9a	ijcnn1	a9a	ijcnn1
P2D2	116,056,832	38,539,264	-	-	862,720,000	29,344,768
PG-EXTRA	125,088,000	30,080,000	-	-	861,520,000	20,101,504
CC-DQM	-	-	20,782,080	1,219,680	-	-
DRUID-Gradient	146,340,480	27,382,784	169,200,768	5,451,776	579,820,032	13,784,320
CE-DADMM-Gradient:EF21	35,692,800	7,468,032	41,268,480	1,444,608	141,419,520	3,767,808
CE-DADMM-Gradient:CLAG	35,650,560	7,459,584	41,352,960	1,427,712	141,419,520	3,759,360
DRUID-Newton	26,670,336	17,315,584	34,117,248	1,858,560	330,781,440	9,850,368
CE-DADMM-Newton:EF21	6,547,200	4,722,432	10,644,480	506,880	80,762,880	2,686,464
CE-DADMM-Newton:CLAG	6,504,960	4,713,984	8,152,320	532,224	80,720,640	2,678,016
DRUID-BFGS	118,457,856	17,532,416	57,150,720	2,385,152	500,501,760	12,359,424
CE-DADMM-BFGS:EF21	20,190,720	4,790,016	13,728,000	616,704	122,073,600	3,379,200
CE-DADMM-BFGS:CLAG	20,148,480	4,773,120	13,770,240	574,464	122,031,360	3,362,304

5.2. Distributed Ridge Regression

The distributed ridge regression solves problem (1), whose $f_i(·)$ is the same as in (24) but with $g\left(x\right)=0$.

In Figure 5 and Figure 6, we observe that when measured by the number of iterations, CE-DADMM with EF21 and CLAG compression mechanisms performs similarly to DRUID without compression. However, CE-DADMM using (quasi) Newton updates converges faster compared to CC-DQM, which also employs an communication-efficient mechanism. On the other hand, when CE-DADMM employs first-order method, its convergence is slower than second-order method, CC-DQM, due to the latter benefiting from additional Hessian information. When measured by total communication bits, CE-DADMM with (quasi) Newton updates requires fewer bits to achieve the same accuracy compared to CC-DQM. Additionally, CE-DADMM benefits from communication-efficient mechanisms, resulting in a substantial reduction in communication bits needed to achieve the same convergence accuracy compared to DRUID. Similarly, the corresponding numerical results are also presented in Table 2 and Table 3.

Figure 5. Performance comparison of distributed ridge regression on the a9a dataset: Plots of iteration number (left) and total communication bits (right) versus distance error.

Figure 6. Performance comparison of distributed ridge regression on the ijcnn1 dataset: Plots of iteration number (left) and total communication bits (right) versus distance error.

5.3. Distributed LASSO

The distributed LASSO solves problem (1) with $g\left(\mathbf{x}\right)={\gamma }_2{\Vert \mathbf{x}\Vert }_1$, $f_i(·):{\mathbb{R}}^d\to \mathbb{R}$ defined as:

$$\begin{array}{c}\hfill f_i\left(\mathbf{x}\right):={\displaystyle \frac{1}{2m_i}}\sum _{j=1}^{m_i}\Vert {\mathbf{a}}_{ij}\mathbf{x}-b_j{\Vert }^2+{\displaystyle \frac{{\gamma }_1}{2}}{\Vert \mathbf{x}\Vert }^2,\end{array}$$

(25)

where ${\mathbf{a}}_{ij}$, $b_{ij}$, and $m_i$ are as defined in Section 5.1. The parameters ${\gamma }_1={10}^{-2}$ and ${\gamma }_2={10}^{-6}$ are regularization terms.

In Figure 7 and Figure 8, when measured by the number of iterations, CE-DADMM performs similarly to DRUID and outperforms P2D2 and PG-EXTRA in terms of convergence speed and accuracy, indicating that the introduction of the 3PC compression mechanism does not negatively affect the convergence speed of the CE-DADMM algorithm. When measured by total communication bits, after introducing the EF21 and CLAG compression mechanisms, CE-DADMM significantly reduces communication overhead compared to DRUID, while also outperforming P2D2 and PG-EXTRA, demonstrating that the communication compression mechanisms can significantly lower communication costs between agents. Similarly, the corresponding numerical results are also presented in Table 2 and Table 3.

Figure 7. Performance comparison of distributed LASSO on the a9a dataset: Plots of iteration number (left) and total communication bits (right) versus distance error.

Figure 8. Performance comparison of distributed LASSO on the ijcnn1 dataset: Plots of iteration number (left) and total communication bits (right) versus distance error.

6. Conclusions

This paper presents a communication-efficient ADMM algorithm for composite optimization, named as CE-DADMM. The algorithm utilizes the ADMM framework with the 3PC communication mechanism, effectively adapting to various communication and computational demands while balancing communication efficiency and computational cost. Notably, when employing quasi-Newton updates, CE-DADMM becomes the first compression-based second-order communication-efficient algorithm. Theoretical analysis demonstrates that the proposed algorithm achieves linear convergence when the local objective functions are strongly convex. Numerical experiments further validate the effectiveness and superior performance of the algorithm. Future work will focus on extending the algorithm to fully asynchronous settings and stochastic problems.