Distributed Sparse Precision Matrix Estimation via Alternating Block-Based Gradient Descent

Abstract

Precision matrices can efficiently exhibit the correlation between variables and they have received much attention in recent years. When one encounters large datasets stored in different locations and when data sharing is not allowed, the implementation of high-dimensional precision matrix estimation can be numerically challenging or even infeasible. In this work, we studied distributed sparse precision matrix estimation via an alternating block-based gradient descent method. We obtained a global model by aggregating each machine’s information via a communication-efficient surrogate penalized likelihood. The procedure chooses the block coordinates using the local gradient, to guide the global gradient updates, which can efficiently accelerate precision estimation and lessen communication loads on sensors. The proposed method can efficiently achieve the correct selection of non-zero elements of a sparse precision matrix. Under mild conditions, we show that the proposed estimator achieved a near-oracle convergence rate, as if the estimation had been conducted with a consolidated dataset on a single computer. The promising performance of the method was supported by both simulated and real data examples.

1. Introduction

Estimating an inverse covariance (or precision) matrix in high dimensions naturally arises in a wide variety of application domains, such as clustering analysis [1,2], discriminant analysis [3], and so on. When the dimension p is much larger than the sample size N, the precision matrix cannot be estimated using the inverse of the sample covariance matrix, due to the singularity of the sample covariance matrix, and estimating the precision matrix is ill-posed and time-consuming, as the number of parameters to be estimated is of the order $O\left(p^2\right)$. As an illustration, in the prostate cancer RNA-Seq dataset we analyze in this paper, genetic activity measurements have been documented for 102 subjects, with 50 normal control subjects and 52 prostate cancer patients. Given that there are over $D={6033}^2$ parameters to estimate, the analytical challenges associated with simultaneous discriminant analysis and estimation are significantly amplified. Accurate and fast precision estimation is becoming increasingly important in statistical learning. Among the many high-dimensional inference problems, a variety of precision estimating methods have been proposed to enrich the theory of this field. Friedman et al. [4] developed an $l_1$ penalized likelihood approach to directly estimate the precision matrix, namely graphical Lasso (GLasso); Cai et al. [5] proposed a constrained $l_1$-minimization procedure to seek a sparse precision matrix under a matrix inversion constraint; Liu and Luo [6] developed a penalized column-wise procedure for estimating a precision matrix; Zhang and Zou [7] advocated a new empirical loss termed the D-trace loss, to avoid computing the log determinant term. For more details refer to [8,9].

However, the rapid emergence of massive datasets poses a serious challenge for high-dimensional precision estimation, where the dimensionality p and the sample size N are both huge. In addition, the computing power, memory constraints, and privacy considerations often make it difficult to pool the separate collections of massive data into a single dataset. Communication is prohibitively expensive due to the limited bandwidth, and direct data sharing raises concerns about privacy and loss of ownership. For example, hospitals may collect the information of tens of thousands of patients, and directly transferring the raw data can be inefficient due to storage bottlenecks. Moreover, in practice, the hospitals are unwilling to share their raw data directly when scientists need to locate relevant genes corresponding to a certain disease from massive data, owing to privacy considerations. The accelerated growth of data sizes and joint analysis of data collected by different parties make statistical inferences on a single computer no longer sufficient, which in addition makes high-dimensional precision estimation a challenging task.

To resolve the above difficulties, one natural strategy is to consider using a “divide-and-conquer” strategy. In such a strategy, a large problem is first divided into smaller manageable subproblems, and the final output is obtained by combining the corresponding sub-outputs. Following this idea, statisticians can improve computing efficiency and reduce privacy risks, while obtaining a global method by aggregating the statistics of each machine. Many distributed statistical methods have been rebuilt for processing massive datasets. Lee et al. [10] proposed a debiasing approach to allow aggregation of local estimates in a distributed setting; Jordan et al. [11] developed an approximate likelihood approach for solving distributed statistical inference problems; and Fan et al. [12] extended their idea and presented two communication-efficient accurate statistical estimators (CEASE). For more details refer to [13,14].

Due to the importance of estimating a precision matrix, some studies have begun to focus on distributed estimation for the precision matrix, where the datasets are distributed over multiple machines, due to size limitations or privacy considerations. Arroyo and Hou [15] estimated the precision matrix for Gaussian graphical models via a simple averaging method; Wang and Cui [16] developed a distributed estimator of the sparse precision matrix by debiasing a D-trace Lasso-type estimator and aggregated estimator by simple averaging. Under distributed data storage, one needs to carefully address two crucial questions for estimating the precision matrix: (a) Estimation-effectiveness: The estimation suffers non-negligible information loss of the whole data, and one should design a distributed procedure to conduct an effective global high-dimensional precision matrix estimation, as if the data were used with a consolidated dataset on a single computer; (b) Communication-efficiency: Estimating a precision matrix suffers from high communication costs under a distributed setup, and the communication costs increases $O\left(p^2\right)$ with the dimensionality p from each machine, and one should design an efficient method to reduce the communication costs incurred by transferring matrices of $O\left(p^2\right)$.

To ease the implementation difficulties and communication costs of estimating a precision matrix, we propose an alternating block-based gradient descent (Bgd) method for distributed precision matrix estimation. In detail, we optimize a surrogate loss function, with all the machines participating to optimize their corresponding gradient-enhanced loss functions and evaluate gradients. In each iteration, we only update the block coordinates of the precision matrix, and the block is chosen using the largest $\kappa$ sizes of the local gradient in a random machine m. By setting $\kappa \ll p^2$, we can develop an accurate statistical estimation for the precision matrix under a distributed setup, which can lessen the communication costs and computation budget by using a random machine to evaluate the choice of block. Under mild conditions, we show that Bgd led to a consistent estimator, it could even achieve a similar performance as debiased lasso penalized D-trace estimation [7]. The promising performance of the method was supported by both simulated and real data examples.

The rest of this paper is organized as follows: In Section 2, we formulate the research problem and introduce the Bgd framework. In Section 3, we investigate the theoretical properties of Bgd. In Section 4, we demonstrate the promising performance of Bgd through Monte Carlo simulations and a real data example. Concluding remarks are given in Section 5. Technical details are presented in Appendix A.

Throughout this paper, we use c and C to represent certain positive constants, which may be different from line to line. Let $\left[p\right]$ mean the set of $\{1,2,\dots ,p\}$, and use ${\parallel \mathit{a}\parallel }_0={\sum }_{j=1}^{p}I(a_j\ne 0)$, ${\parallel \mathit{a}\parallel }_{\infty }={max}_j\left|a_j\right|$, and ${\parallel \mathit{a}\parallel }_2=\sqrt{{\sum }_{j=1}^p{a}_j^2}$ to denote its ${\mathcal{l}}_0$, ${\mathcal{l}}_{\infty }$, ${\mathcal{l}}_2$ norms for a vector $\mathit{a}={(a_1,\dots ,a_p)}^{\top }\in {\mathbb{R}}^p$, respectively. For a matrix $\mathit{A}=\left(a_{ij}\right)\in {\mathbb{R}}^{p_1\times p_2}$, let ${\parallel \mathit{A}\parallel }_{max}={max}_{i,j}\left|a_{ij}\right|$, ${\parallel \mathit{A}\parallel }_2={sup}_{{\parallel \mathit{x}\parallel }_2\le 1}{\parallel \mathit{A}\mathit{x}\parallel }_2$, ${\parallel \mathit{A}\parallel }_{\infty }={max}_{i=1,\cdots ,p_1}{\sum }_{j=1}^{p_2}\left|a_{ij}\right|$, ${\parallel \mathit{A}\parallel }_F=\sqrt{{\sum }_{i=1}^{p_1}{\sum }_{j=1}^{p_2}{a}_{ij}^2}$ be its max, spectral, inf, and Frobenius norms, respectively.

2. Distributed Sparse Precision Matrix Estimation

2.1. Model Setups

Assume that we have N independent and identically distributed p-dimensional random variables with a covariance matrix $\mathbf{\Sigma }$ or its corresponding precision matrix $\mathbf{\Omega }:={\mathbf{\Sigma }}^{-1}$. Each nonzero entry of $\mathbf{\Omega }$ corresponds to an edge in a Gaussian graphical model for describing the conditional dependence structure of the observed variables. In particular, if a p-dimensional random vector $\mathit{X}\sim N_p(\mathit{\mu },\mathbf{\Sigma })$, the conditional independence between ${\mathit{X}}_j$ and ${\mathit{X}}_l$ given other features is equivalent to ${\Omega }_{jl}=0$, $j,l\in \left[p\right]$. A sparse structure of the precision matrix $\mathbf{\Omega }$ provides a concise relationship between features and also gives a meaningful interpretation of the conditional independence among the features; thus, one needs to achieve a sparse and stable estimation for the precision matrix $\mathbf{\Omega }$.

Throughout this paper, we assume that the number of features p can be much larger than the total sample size N, but the true precision matrix is sparse, so there are few non-zero entries in the high-dimensional setting. We use $s^{*}=\{(j_1,j_2):{\omega }_{j_1{j}_2}\ne 0 \mathrm{for} j_1,j_2\in \left[p\right]\}$ to denote the index set of the true nonzero components in the precision matrix ${\mathbf{\Omega }}^{*}$. Given N independent observations of ${\left\{{\mathit{X}}_i\right\}}_{i=1}^N$, we suppose it is partitioned into M subsets ${\mathcal{D}}_1,\cdots ,{\mathcal{D}}_M$ completely at random and stored on M local clients. Without loss of generality, assume that $\mathit{X}$ is sub-Gaussian and the data are equally partitioned into M machines. In the high-dimensional setting, a common approach to obtaining a sparse estimator of $\mathbf{\Omega }$ is by minimizing the following $l_1$-regularized negative log-likelihood, known as a graphical lasso, which is defined as

$$\mathrm{tr}(\mathit{S}\mathbf{\Omega })-\log \det (\mathbf{\Omega })+{\lambda \parallel \mathbf{\Omega }\parallel }_1,$$

where $\mathit{S}$ is the sample covariance matrix. Many algorithms have been developed to solve the above problem. However, eigendecomposition or calculation of the determinant of a $p\times p$ matrix is inevitable in these algorithms. Motivated by [6,7], under the distributed scenario, the global and the local loss functions can be written as

$$\begin{array}{cc}\hfill {\mathcal{l}}_N(\mathbf{\Omega })& =\frac{1}{M}\sum _{m=1}^M{\mathcal{l}}_{\left(m\right)}(\mathbf{\Omega })\hfill \end{array}$$

(1)

$$\begin{array}{cc}\hfill {\mathcal{l}}_{\left(m\right)}(\mathbf{\Omega })& =:\frac{1}{2}\mathrm{tr}({\mathbf{\Omega }}^{\top }{\mathit{S}}_m\mathbf{\Omega })-\mathrm{tr}(\mathbf{\Omega }),\hfill \end{array}$$

(2)

where ${\mathit{S}}_m$ is the local sample covariance matrix in the client m. For a single machine, many algorithms have been developed to solve the above problem, and some authors have shown that their estimators are asymptotically consistent. The goal of this study is to estimate the high-dimensional precision matrix $\mathbf{\Omega }$ in a distributed system, where the communication cost and the accuracy of estimation are the major considerations.

2.2. Block-Gradient Descent Algorithm

To develop a communication-efficient method for learning a high-dimensional precision matrix, we first review the proposal of Jordan et al. [11]. Starting from an initial estimator, the gradient can be communicated and the parameters can be obtained based on a communication-efficient surrogate likelihood framework. Note that, in [11], only the first machine solved optimization problems, and the global Hessian matrix was replaced by the first local Hessian matrix. To fully utilize the information on each machine, we choose a random machine m to solve optimization problems in every iteration. In this strategy, we define the loss function for a random machine m as

$$\begin{array}{c}\hfill \tilde{L}(\mathbf{\Omega })={\mathcal{l}}_{\left(m\right)}(\mathbf{\Omega })-\mathrm{tr}\left[({\nabla }_{\mathbf{\Omega }}{\mathcal{l}}_{\left(m\right)}\left(\overline{\mathbf{\Omega }}\right)-{\nabla }_{\mathbf{\Omega }}{\mathcal{l}}_N\left(\overline{\mathbf{\Omega }}\right))\mathbf{\Omega }\right]+p\left(\right|\mathbf{\Omega }|,\lambda ),\end{array}$$

(3)

where $\overline{\mathbf{\Omega }}$ is an initial estimator of $\mathbf{\Omega }$ and ${\nabla }_{\mathbf{\Omega }}{\mathcal{l}}_N\left(\overline{\mathbf{\Omega }}\right)={\sum }_{m=1}^M{\nabla }_{\mathbf{\Omega }}{\mathcal{l}}_{\left(m\right)}\left(\overline{\mathbf{\Omega }}\right)$, $p(,\lambda )$ is a concave penalty function with a tuning parameter $\lambda >0$. In high-dimensional regimes, it is impossible to derive the closed-form solution of $\mathbf{\Omega }$. A naive method to remedy this is to add a strict convex quadratic regularization term $\frac{\nu }{2}{\parallel \mathbf{\Omega }-\overline{\mathbf{\Omega }}\parallel }_2^2$, and use $g(\mathbf{\Omega }|\overline{\mathbf{\Omega }})$ to approximate the surrogate loss function $\tilde{L}(\mathbf{\Omega })$. Then, $g(\mathbf{\Omega }|\overline{\mathbf{\Omega }})$ can be defined as

$$\begin{array}{cc}g(\mathbf{\Omega }|\overline{\mathbf{\Omega }})=& {\mathcal{l}}_{\left(m\right)}\left(\overline{\mathbf{\Omega }}\right)+\mathrm{tr}\left[{\nabla }_{\mathbf{\Omega }}{\mathcal{l}}_{\left(m\right)}\left(\overline{\mathbf{\Omega }}\right)(\mathbf{\Omega }-\overline{\mathbf{\Omega }})\right]-\mathrm{tr}\left[({\nabla }_{\mathbf{\Omega }}{\mathcal{l}}_{\left(m\right)}\left(\overline{\mathbf{\Omega }}\right)-{\nabla }_{\mathbf{\Omega }}{\mathcal{l}}_N\left(\overline{\mathbf{\Omega }}\right))\mathbf{\Omega }\right]\hfill \\ & +\frac{\nu }{2}\parallel \mathbf{\Omega }-\overline{\mathbf{\Omega }}\}{\parallel }_F^2+p\left(\right|\mathbf{\Omega }|,\lambda ).\hfill \end{array}$$

(4)

Using (4), if we set $\overline{\mathbf{\Omega }}$ as the current t-th iteration ${\mathbf{\Omega }}^{\left(t\right)}$, an approximate solution to (3) is obtained by the following iterative procedure

$${\mathbf{\Omega }}^{(t+1)}=argming(\mathbf{\Omega }|{\mathbf{\Omega }}^{\left(t\right)}).$$

At each iteration, the regularization term prevents its minimizer from moving too far away from ${\mathbf{\Omega }}^{\left(t\right)}$. This feature performs a non-greedy update in searching for the estimation of a high-precision matrix [17]. We can use gradient descent to optimize $g(\mathbf{\Omega }|{\mathbf{\Omega }}^{\left(t\right)})$, and $g(\mathbf{\Omega }|{\mathbf{\Omega }}^{\left(t\right)})$ can well approximate $\tilde{L}(\mathbf{\Omega })$ for ${\mathbf{\Omega }}^{\left(t\right)}$ close to $\mathbf{\Omega }$ when the stepsize $\nu$ is chosen appropriately by the local sample covariance matrix ${\mathit{S}}_m$ in machine m. However, the gradient descent method needs to transmit $O\left(p^2\right)$ bits from each machine, which results in high communication costs and computation burden per round. In intuition, we should choose the block that can best update the global gradient rapidly with communication constraints. In this paper, we use the local gradient to guide the choice of the block, where the block in the $t$-th iteration is chosen using

$$\begin{array}{c}\hfill C_t=\{(j,l),|{\mathit{B}}_{j,l}{|\ge \rho =\mathrm{vec}(\left|\mathit{B}\right|)}_{\left(\kappa \right)}\},\end{array}$$

(5)

where $\mathrm{vec}{\left(\mathit{B}\right)}_{\left(\kappa \right)}$ denotes $\kappa$-th largest component of the vector $\mathrm{vec}\left(\mathit{B}\right)$, and $\mathit{B}={\nabla }_{\mathbf{\Omega }}{\mathcal{l}}_{\left(k\right)}\left({\mathbf{\Omega }}^{\left(t\right)}\right)$ is the local gradient in machine k and k is a random machine chosen to optimize the surrogate negative likelihood ${\mathcal{l}}_{\left(k\right)}(\mathbf{\Omega })-\mathrm{tr}\left[({\nabla }_{\mathbf{\Omega }}{\mathcal{l}}_{\left(k\right)}\left(\overline{\mathbf{\Omega }}\right)-{\nabla }_{\mathbf{\Omega }}{\mathcal{l}}_N\left(\overline{\mathbf{\Omega }}\right))\mathbf{\Omega }\right]+p\left(\right|\mathbf{\Omega }|,\lambda )$. In every iteration, every machine transfers $O\left(\kappa \right)$ bits rather than $O\left(p^2\right)$ gradient matrices, and we just update the gradient and precision matrix in block $C_t$. Details can be found in Algorithm 1. With the aid of surrogate negative likelihood, we can efficiently train a global model that aggregates information from other machines. Thus, this has good potential to provide more reliable estimation results and degrade the communication loads and costs.

Regarding the update $\mathbf{\Omega }$, we need to design a distributed algorithm with communication constraints. To update $\mathbf{\Omega }$, we need to solve the following minimization problem:

$$\begin{array}{c}\hfill {\mathbf{\Omega }}^{(t+1)}=arg\underset{\mathbf{\Omega }}{min}{\displaystyle \frac{\nu }{2}}\parallel \mathbf{\Omega }-{\mathbf{\Omega }}^{\left(t\right)}+{\nu }^{-1}{C}_t\left[{\nabla }_{\mathbf{\Omega }}{\mathcal{l}}_N\left({\mathbf{\Omega }}^{\left(t\right)}\right)\right]{\parallel }^2+\sum _{(j,l)\in C_t}p(\parallel {\mathbf{\Omega }}_{jl}\parallel ,\lambda ),\end{array}$$

(6)

where

$$\begin{array}{c}\hfill C_t{\left[{\nabla }_{\mathbf{\Omega }}{\mathcal{l}}_N\left({\mathbf{\Omega }}^{\left(t\right)}\right)\right]}_{jl}=\left\{\begin{array}{cc}{\nabla }_{{\Omega }_{jl}}{\mathcal{l}}_N\left({\mathbf{\Omega }}^{\left(t\right)}\right)\hspace{1em}\hfill & (j,l)\in C_t\hfill \\ 0\hspace{1em}\hfill & (j,l)\notin C_t.\hfill \end{array}\right.\end{array}$$

(7)

The penalty function in (6) for updating $\mathbf{\Omega }$ can be chosen from Lasso, MCP, or SCAD off-diagonal penalties [18,19]. Closed-form solutions exist for the Lasso, SCAD, and MCP penalties for (6). For example, let $S(\mathit{a},\lambda )=\mathrm{sign}\left(a\right){\left(\right|a|-\lambda )}_{+}$ be the soft thresholding rule, and ${\left(b\right)}_{+}=b$ if $b>0$, and ${\left(b\right)}_{+}=0$ otherwise. Denote ${\mathit{\vartheta }}^{(t+1)}={\mathbf{\Omega }}^{\left(t\right)}-{\nu }^{-1}{C}_t[{\nabla }_{\mathbf{\Omega }}{\mathcal{l}}_N\left({\mathbf{\Omega }}^{\left(t\right)}\right)$]. The closed-form solution for the Lasso penalty is

$$\begin{array}{c}\hfill {\omega }_{jl}^{(t+1)}=\left\{\begin{array}{cc}S({\vartheta }_{jl}^{(t+1)},\lambda /\nu )\hspace{1em}\hfill & (j,l)\in C_t\hfill \\ {\omega }_{jl}^{\left(t\right)}\hspace{1em}\hfill & (j,l)\notin C_t.\hfill \end{array}\right.\end{array}$$

(8)

For the MCP penalty with $\varsigma >1/\nu$,

$$\begin{array}{c}\hfill {\omega }_{jl}^{(t+1)}=\left\{\begin{array}{cc}{\displaystyle \frac{S({\vartheta }_{jl}^{(t+1)},\lambda /\nu )}{1-1/\left(\varsigma \nu \right)}}\hspace{1em}\hfill & {\displaystyle \left|{\vartheta }_{jl}^{(t+1)}\right|\le \varsigma \lambda }\hfill \\ {\vartheta }_{jl}^{(t+1)}\hspace{1em}\hfill & {\displaystyle \left|{\vartheta }_{jl}^{(t+1)}\right|>\varsigma \lambda }\hfill \\ {\omega }_{jl}^{\left(t\right)}\hspace{1em}\hfill & (j,l)\notin C_t.\hfill \end{array}\right.\end{array}$$

(9)

For the SCAD penalty with $\varsigma >1/\nu +1$, the closed-form solution can be written as

$$\begin{array}{c}\hfill {\omega }_{jl}^{(t+1)}=\left\{\begin{array}{cc}S({\vartheta }_{jl}^{(t+1)},\lambda /\nu )\hspace{1em}\hfill & \left|{\vartheta }_{jl}^{(t+1)}\right|\le \lambda +\lambda /\nu \hfill \\ {\displaystyle \frac{S({\vartheta }_{jl}^{(t+1)},\varsigma \lambda /\left((\varsigma -1)\nu \right))}{1-1/\left(\right(\varsigma -1\left)\nu \right)}}\hspace{1em}\hfill & \lambda +\lambda /\nu <\left|{\vartheta }_{jl}^{(t+1)}\right|\le \varsigma \lambda \hfill \\ {\vartheta }_{jl}^{(t+1)}\hspace{1em}\hfill & \left|{\vartheta }_{jl}^{(t+1)}\right|>\varsigma \lambda \hfill \\ {\omega }_{jl}^{\left(t\right)}\hspace{1em}\hfill & (j,l)\notin C_t.\hfill \end{array}\right.\end{array}$$

(10)

where ${\mathbf{\Omega }}^{(t+1)}=\left({\omega }_{jl}^{(t+1)}\right)$, $\varsigma$ is a parameter that controls the concavity of the MCP and SCAD function. In particular, the SCAD converges to the Lasso penalty as $\varsigma \to \infty$. Following Fan and Li [20], we treat $\varsigma$ as a fixed constant, such as $\varsigma =3.7$. The SCAD not only enjoys sparsity as the $L_1$ penalty, but also has the property of unbiasedness, in that it does not shrink large estimated parameters, so that they remain unbiased throughout the iterations.

Note that the solution ${\mathbf{\Omega }}_1^{(t+1)}$ to (6) is not symmetric in general. To make the solution symmetric, following the symmetrization strategy of Cai et al. [5] and Cai et al. [21], the final estimator ${\mathbf{\Omega }}^{(t+1)}$ is constructed through comparison and assigning the one with the smallest magnitude at both entries of ${\mathbf{\Omega }}_1^{(t+1)}$, which is

$${\omega }_{jl}^{(t+1)}={\omega }_{1jl}^{(t+1)}I\left(\right|{\omega }_{1jl}^{(t+1)}|\le |{\omega }_{1lj}^{(t+1)}\left|\right)+{\omega }_{1lj}^{(t+1)}I\left(\right|{\omega }_{1lj}^{(t+1)}|<|{\omega }_{1jl}^{(t+1)}\left|\right).$$

This symmetrizing procedure is not ad hoc. The procedure ensures that the final estimator ${\mathbf{\Omega }}^{(t+1)}$ achieves the same entry-wise $L_{\infty }$ estimation error as ${\mathbf{\Omega }}_1^{(t+1)}$. For more details refer to Section 3 of Cai et al. [21].

We now discuss how to select the constraints $\kappa$ and the stepsize $\nu$. Regarding the setup for $\kappa$, a larger value of $\kappa$ often transmits more information of each local client in each iteration and leads to a more accurate and faster convergence of Bgd. Nevertheless, a larger $\kappa$ also means higher communication loads and costs. The choice of $\kappa$ is a great challenge. The value of $\kappa$ should not be too small or too large. Fortunately, we found the performance of Bgd was robust with a wide range of choices of $\kappa$ within certain a interval, which facilitates the use of Bgd by avoiding an elaborative specification on $\kappa$. In addition, many empirical studies have shown that a smaller value of $\nu$ often leads to a faster convergence of the above algorithm. Theorem 1 indicates that only if $\nu$ is larger than ${\Lambda }_{max}\left({\mathit{S}}_m\right)$, will the objective function be guaranteed to increase for every iteration. In practice, one can first use a tentatively small $\nu$ in local client m, and then check the condition based on the data in the local client m, as follows:

$$\begin{array}{c}\hfill \nu \parallel \Delta {\mathbf{\Omega }}^{\left(t\right)}{\parallel }_F^2\ge \mathrm{tr}({(\Delta {\mathbf{\Omega }}^{\left(t\right)})}^{\top }{\mathit{S}}_m\Delta {\mathbf{\Omega }}^{\left(t\right)}),\end{array}$$

(11)

where $\Delta {\mathbf{\Omega }}^{\left(t\right)}={\mathbf{\Omega }}^{(t+1)}-{\mathbf{\Omega }}^{\left(t\right)}$. If (11) is not satisfied, we take $\nu$ as twice its current value. The proposed Bgd algorithm is summarized in Algorithm 1.

Algorithm 1 Distributed sparse precision matrix estimation via Bgd

Input      Initial value ${\mathbf{\Omega }}^{\left(0\right)}$, number of iterations T. For $t=0,1,2,\cdots ,T-1$:              • choose a random machine m, update $C_t$;              • The machine m sends $C_t$ to machines through the central processor;              • Each machine evaluates ${\nabla }_{\mathbf{\Omega }}{\mathcal{l}}_{\left(k\right)}\left({\mathbf{\Omega }}^{\left(t\right)}\right)$ and sends ${\nabla }_{{\mathbf{\Omega }}_{C_t}}{\mathcal{l}}_{\left(k\right)}\left({\mathbf{\Omega }}^{\left(t\right)}\right)$ to the central processor;              • Then central processor transmit ${\nabla }_{{\mathbf{\Omega }}_{C_t}}{\mathcal{l}}_{\left(k\right)}\left({\mathbf{\Omega }}^{\left(t\right)}\right)$ to machine m and machine m computes $${\mathbf{\Omega }}^{(t+1)}=arg\underset{\mathbf{\Omega }}{min}{\displaystyle \frac{\nu }{2}}\parallel \mathbf{\Omega }-{\mathbf{\Omega }}^{\left(t\right)}-{\nu }^{-1}{C}_t\left[{\nabla }_{\mathbf{\Omega }}{\mathcal{l}}_N\left({\mathbf{\Omega }}^{\left(t\right)}\right)\right]{\parallel }^2+\sum _{(j,l)\in C_t}p(\parallel {\mathbf{\Omega }}_{jl}\parallel ,\lambda )$$ and broadcast to machines through the central processor, where the regularizer $\nu$ is chosen through the local covariance matrix ${\mathit{S}}_m$. Output       ${\mathbf{\Omega }}^{\left(T\right)}$.

Remark 1.

In the Bgd algorithm, we only transmit the gradient ${\nabla }_{{\mathbf{\Omega }}_{C_t}}{\mathcal{l}}_{\left(k\right)}\left({\mathbf{\Omega }}^{\left(t\right)}\right)$ and ${\mathbf{\Omega }}_{C_t}$ with block $C_t$ to the central processor and client m, by setting $\parallel C_t{\parallel }_0\ll p^2$, we can efficiently reduce the communication loads and costs by avoiding transferring $O\left(p^2\right)$ bits of the gradient matrix and the estimated precision matrix.

3. Theoretical Properties

We conducted a theoretical analysis to justify the proposed Bgd procedure. In particular, we studied the efficiency of the Bgd estimator in a distributed setup. To investigate the properties of the proposed Bgd algorithm, we required the following conditions:

(A1)

(Sparse matrix class) Suppose that ${\mathbf{\Omega }}^{*}\in \mathcal{U}$ with

$$\mathcal{U}=\left\{\mathbf{\Omega }\succ 0:\underset{1\le j\le p}{max}\sum _{l=1}^{p}1\{w_{jl}\ne 0\}\le s_p,{\parallel \mathbf{\Omega }\parallel }_{L_1}\le Q\right\}.$$

(A2)

(Irrepresentability condition) Let $\delta =\left\{(j,l)\right|{\Omega }_{jl}\ne 0\}$ be the set of all non-zero entries of 0, and ${\delta }^c$ be the complement of $\delta$, for some $0<\alpha <1$, the covariance matrix ${\mathbf{\Sigma }}^{*}$ satisfies

$$\underset{1\le i\le p}{max}{\parallel {\mathbf{\Sigma }}_{{\delta }_i^c{\delta }_i}^{*}\left({\mathbf{\Sigma }}_{{\delta }_i}^{*}{\mathbf{\Sigma }}_{{\delta }_i}^{*}\right)\parallel }_{\infty }\le 1-\alpha .$$

(A3)

(Bounded condition) There exists a constant $c_0\ge 1$ such that $c_0^{-1}\le {\Lambda }_{min}\left({\mathbf{\Omega }}^{*}\right)\le {\Lambda }_{max}\left({\mathbf{\Omega }}^{*}\right)\le c_0$, where ${\Lambda }_{min}\left(\mathit{A}\right)$ and ${\Lambda }_{max}\left(\mathit{A}\right)$ denote the smallest and largest eigenvalues of matrix $\mathit{A}$, respectively.

(A4)

(Restricted strong convexity for negative loglikelihood) There exists a positive constant $c_1$ and $\tilde{\mathbf{\Omega }}$ such that

$${\mathcal{l}}_N({\mathbf{\Omega }}^{*}+\mathbf{\Delta })-{\mathcal{l}}_N\left({\mathbf{\Omega }}^{*}\right)-\mathrm{tr}[{\dot{\mathcal{l}}}_N{\left({\mathbf{\Omega }}^{*}\right)}^{\top }\mathbf{\Delta }]=\mathrm{tr}[{\Delta }^{\top }{\ddot{\mathcal{l}}}_N\left(\tilde{\mathbf{\Omega }}\right)\mathbf{\Delta }]\ge c_4{\parallel \mathbf{\Delta }\parallel }_F^2,$$

for any $\mathbf{\Delta }\ne 0$ satisfying $\parallel {\mathbf{\Delta }}_{s_c^{*}}{\parallel }_1\le 3{\parallel {\mathbf{\Delta }}_{s^{*}}\parallel }_1$.

Condition (A1) indicates that the precision matrix has a sparse structure, it has been widely used in the literature on Gaussian graphical model estimation [5,6]. Condition (A2) is in the same spirit as the mutual incoherence or irrepresentable condition of Liu and Luo [6]. Condition (A3) requires that the smallest eigenvalue of the precision matrix ${\mathbf{\Omega }}^{*}$ is bounded below zero, and that its largest eigenvalue is finite. Condition (A3) also implies that $c_0^{-1}\le {\Lambda }_{min}\left({\mathbf{\Sigma }}^{*}\right)\le {\Lambda }_{max}\left({\mathbf{\Sigma }}^{*}\right)\le c_0$. This assumption is commonly imposed in the literature for the analysis of Gaussian graphical models [22]. Condition (A4) states that the negative log-likelihood ${\mathcal{l}}_N(\mathbf{\Omega })$ is restricted to strong convexity at $\parallel {\mathbf{\Delta }}_{s_c^{*}}{\parallel }_1\le 3{\parallel {\mathbf{\Delta }}_{s^{*}}\parallel }_1$.

Theorem 1.

Let ${\mathbf{\Omega }}^{\left(t\right)}$ be the sequence defined in the above Algorithm 1, if we use the local client m to compute 6, and set $\overline{\mathbf{\Omega }}={\mathbf{\Omega }}^{\left(t\right)}$, $\nu >{\Lambda }_{max}\left({\mathit{S}}_m\right)$, then

$$\tilde{L}\left({\mathbf{\Omega }}^{(t+1)}\right)\le \tilde{L}\left({\mathbf{\Omega }}^{\left(t\right)}\right).$$

Theorem 1, provided in Appendix A, indicates that with the appropriate scale $\nu$, we can ensure $\tilde{L}\left({\mathbf{\Omega }}^{(t+1)}\right)\le \tilde{L}\left({\mathbf{\Omega }}^{\left(t\right)}\right)$ with limited communication costs in every iteration. Theorem 1 provides an insight into choosing the stepsize $\nu$ with the local data under the distributed setting in a practical implementation.

Theorem 2.

Under the sub-Gaussian condition, suppose that assumptions (A1)–(A4) hold, if $s_p=o\left(\sqrt{N/logp}\right)$ with $\lambda \le C_{1}Q\sqrt{logp/N}$ for some $C_1>2$, and setting $\rho =C_{2}Q\sqrt{logp/N}$ for some $C_2\ge 0$,

$$\frac{1}{p}{\parallel \widehat{\mathbf{\Omega }}-{\mathbf{\Omega }}^{*}\parallel }_F^2=O_p\left(Q^2\frac{s_{p}logp}{N}\right)\to 0.$$

Theorem 2, provided in Appendix A, shows the convergence rate under the Frobenius norm.

4. Numerical Studies

In this section, we present several simulation studies and a real data example, to investigate the finite-sample performance of the proposed Bgd procedure in terms of its estimation accuracy. We compare the proposed method with several other distributed high-dimensional precision matrix estimation methods: naive estimation based on averaging the local estimation obtained from the R package “glasso”(Naive) using R version 4.1.0, debiased distributed D-trace loss penalized estimation (Dtrace, [16]), and debiased distributed graphical lasso estimation (Dglasso, [15]). For a benchmark, we set the debiased D-trace loss penalized estimation proposed by Zhang and Zou [7] with all data in a non-distributed setting as the global method. Each estimator was tuned by cross-validation and all numerical experiments were conducted using the software R on a Microsoft Windows computer with a sixteen-core 4.50 GHz CPU and 32 GB RAM. In addition, in this paper, for the penalty function in the objective function (3), we chose lasso and SCAD.

In our numerical studies, the Bgd model was implemented based on Algorithm 1 with ${\mathbf{\Omega }}^{\left(0\right)}=\mathit{I}$. We terminated the iterations when $\parallel {\mathbf{\Omega }}_{C_t}^{(t+1)}-{\mathbf{\Omega }}_{C_t}^{\left(t\right)}{\parallel }_F/{\parallel {\mathbf{\Omega }}_{C_t}^{\left(t\right)}\parallel }_F<{10}^{-3}$ or $T\ge T_{max}$, and we set $T_{max}=300$ in Bgd. We chose ${\displaystyle \kappa =⌞\sqrt{p}logp⌟}$, obviously, $\kappa \ll p^2$, which could efficiently reduce the communication loads and costs by avoiding transferring $O\left(p^2\right)$ bits of the gradient matrix and the estimated precision matrix. Here, ⌞a⌟ denotes the largest integer part of a. The Bgd model was implemented based on Algorithm 1 and we evaluated the estimation accuracy of each method using Frobenius loss and spectral loss, as follows:

$$L_F=\parallel \widehat{\mathbf{\Omega }}-{\mathbf{\Omega }}^{*}{\parallel }_F, L_2={\parallel \widehat{\mathbf{\Omega }}-{\mathbf{\Omega }}^{*}\parallel }_2.$$

Generally, the smaller $L_F$ and $L_2$ are, the higher the estimation accuracy. Moreover, to assess the accuracy with the sparseness of the true precision matrix recovered, we also evaluated false negative (FN) and false positive (FP) rates, as described below:

$$FN={\displaystyle \frac{{\sum }_{i<j}1({\widehat{w}}_{ij}=0,w_{ij}^{*}\ne 0)}{{\sum }_{i<j}1(w_{ij}^{*}\ne 0)}},FP={\displaystyle \frac{{\sum }_{i<j}1({\widehat{w}}_{ij}\ne 0,w_{ij}^{*}=0)}{{\sum }_{i<j}1(w_{ij}^{*}=0)}},$$

The false negative rate gives the percentage of nonzero-elements that are wrongly estimated to be zero. In contrast, the false positive rate gives the percentage of zero-elements that are wrongly estimated as nonzero. Both values are desired to be as small as possible. For each model under study, we set $M=\{5,10,20\}$. We further specify the parameter settings in the following sections, where the corresponding simulation results are also given.

(S1) We first assessed the performance of Bgd and its competitors based on their estimation accuracy across two different values of p (i.e., $p=200$ and $p=500$, and this design resulted in $D={200}^2$ and $D={500}^2$ parameters to be estimated, respectively). Here, we set ${\mathbf{\Omega }}^{*}$ as a band matrix, i.e., ${\omega }_{jl}^{*}=0.45$ for $|j-l|=1$, ${\omega }_{jj}^{*}=1.0$, and the other elements of ${\mathbf{\Omega }}^{*}$ were taken as zero, where $j,l=1,\cdots ,p$. The sample size was $N=140$.

(S2) We evaluated the performance of Bgd and its competitors across two different values of total sample size N. To this end, we considered the dimensional $p=350$, with sample sizes $N=160$ and $N=120$, respectively. Using a similar set as Wang and Jiang [23], we set ${\mathbf{\Sigma }}^{*}=0.4^{|j-l|}$, $j,l=1,\cdots ,p$. In this setting, ${\mathbf{\Omega }}^{*}$ had a sparse structure.

(S3) We considered a case with varying sparsity levels of the precision matrix. To this end, let ${\mathbf{\Omega }}_1=\left({\omega }_{jl}\right)$, where ${\omega }_{jl}=u_{jl}{\delta }_{jl}$, $u_{jj}=1.0$ and $u_{jl}=0.5$, for $j\ne l$, ${\delta }_{jl}$ is the Bernoulli random variable with a success probability of $0.01$ and $0.02$, respectively, and we chose $(N,p)=(300,400)$. We let ${\mathbf{\Omega }}^{*}={\mathbf{\Omega }}_1+1({\Lambda }_{min}\le 0)\left(\right|{\Lambda }_{min}|+0.02)I_p$ to ensure that the precision matrix ${\mathbf{\Omega }}^{*}$ was positive definite.

For each of the above presented cases, we generated $\mathbb{T}=100$ datasets. For each of the 100 datasets, the aforementioned methods were adopted to perform high-dimensional distributed precision estimation. The average Frobenius loss $L_F$, spectral loss $L_2$, FN, and FP for each method are reported in Table A1. We investigated the effect of the number of machines M and the local sample sizes n in terms of the estimation error. As shown in Table A1, (i) the naive method performance was poor in all cases; (ii) Dtrace and Dglasso exhibited improvement over the naive estimate by debiasing local lasso estimators and averaging the debiased local estimators; (iii) as the number of machines increased, the Dtrace and Dglasso methods deteriorated drastically; (iv) considering more complex structures of precision and varied machines, the proposed Bgd method with lasso and SCAD penalties still achieved smaller errors than the other methods, and the SCAD method had smaller $L_F$ values than the Lasso, in general, since SCAD had more accurate selection results and produced less biased estimates. In summary, the proposed Bgd method outperformed the other methods regardless of the machine number M, and the structure of the precision matrix, in that the values of $L_F$, $L_2$ and $FN$ for the former were smaller than those for the latter except the Global method, which was the benchmark.

To investigate the effect on the number of machines M, we replicated the aforementioned simulation study for cases S1–S3 and varied the number of machines M from 5 to 30, and plot the $L_F$ in Figure A1. From Figure A1, the performance of the Naive, Dtrace, and Dglasso deteriorated drastically with the number of machines. In contrast, the $L_F$ of Bgd versus M was almost flat and was very close to the global method, even when M was large. The proposed Bgd still showed accuracies that surpassed its competitors.

Real Data Analysis

In this subsection, we applied our method Bgd to a real data example. The prostate cancer data are available at http://bioinformatics.mdanderson.org/ (acessed on 1 March 2023). The data consist of genetic expression levels for $p=6033$ genes from 102 individuals (50 normal control subjects and 52 prostate cancer patients). This dataset has been analyzed in several articles for high-dimensional analysis [5,24].

To evaluate the performance of the proposed Bgd method for distributed precision estimation, we randomly partitioned $100\gamma \%$ samples as training data and the remaining $100(1-\gamma )\%$ samples as testing data, and the data were equally partitioned into $M=\{5,10,20\}$ data segments. Often having more than 50% training data is preferred [25], and we set $\gamma =0.6$ or $0.8$, which led to every client having only 3 or 4 observations for training when the total machines $M=20$. For simplicity of calculation, we selected $p=300$ genes from all the 16,386 genes using the Package “SIS” with the logistic model, which resulted in over 90,000 parameters to be estimated.

Our goal was to estimate the precision (inverse covariance) matrix in a distributed setting, and we could not use $L_2$ or $L_F$ to measure the estimation accuracy of each method as we did not know the true values of the ${\mathbf{\Omega }}^{*}$. Following the same analysis as [5], the normalized gene expression data were assumed to be normally distributed as $N({\mathit{\mu }}_k,\mathbf{\Sigma })$, where the two groups were assumed to have the same covariance matrix $\mathbf{\Sigma }$ but different means ${\mathit{\mu }}_k$, $k=1,2$. The estimated inverse covariance $\mathbf{\Omega }$ produced by the different methods was used in the linear discriminant scores:

$${\delta }_k\left(x\right)={\mathit{x}}^{\top }\widehat{\mathbf{\Omega }}{\widehat{\mathit{\mu }}}_k-\frac{1}{2}{\widehat{\mathit{\mu }}}_k^{\top }\widehat{\mathbf{\Omega }}{\widehat{\mathit{\mu }}}_k+log{\widehat{\pi }}_k.$$

The classification rule was taken to be $\widehat{k}\left(x\right)=argmax{\delta }_k\left(x\right)$ for $k=1,2$. For simplicity, the ${\widehat{\mathit{\mu }}}_k$ and ${\widehat{\pi }}_k$ in the linear discriminant scores were estimated by training data with the non-distributed setting, whereas $\widehat{\mathbf{\Omega }}$ was estimated by training data under a distributed setup. The classification performance was clearly associated with the estimation accuracy of $\mathbf{\Omega }$. The training dataset was used to perform parameter estimation, while the testing dataset was adopted to compute the classification error. We used the classification error in the testing dataset to assess the estimation performance and compare it with the existing results of other methods. A good estimation method for a precision matrix is expected to have a low misclassification (prediction error, Prr), high sensitivity (Sen), and specificity (Spe) for all partitions.

We summarized the assessment based on $T=100$ replications in terms of sensitivity (Sen), specificity (Spe), as well as the overall prediction error. The proposed Bgd method outperformed all the other methods and even had a similar performance to the global method. The models chosen by Bgd had a higher sensitivity and specificity, and lower misclassification error (Prr). From Table A2, the promising performance of Bgd is again observed.

Moreover, to demonstrate that Bgd is robust to a wide range of choices of $\kappa$ within a certain interval, we repeated the above procedure with $\gamma =0.8$ and calculated their corresponding Prr values. Figure A2 plots the Prr values by taking $\tau$ as the x-axis with $\kappa =⌞\tau \sqrt{p}logp⌟$. Inspection of Figure A2 indicates that the proposed Bgd method was robust against a wide choice of $\kappa$ and still performed better than the other methods, in that the Prr of Bgd was lower than the other methods with various $\kappa$.

5. Discussion

This paper proposes a novel method for high-dimensional precision estimation when the dataset is distributed into different machines. In this work, we studied distributed sparse precision matrix estimation via an alternating block-based gradient descent method, where the block was chosen by the local gradient. This procedure can increase the communication loads and costs for a reliable estimation. The proposed method showed good potential to improve the accuracy of estimation compared with the other distributed methods.

The current work focused on cases with homogeneous data analysis. It would be an interesting topic for future research to further extend the existing work to joint estimation of multiple precision matrices.

Appendix A.1. Tables and Figures

Table A1. Performance of the Bgd and competing methods in the simulation study.

				$\mathit{M}=5$					$\mathit{M}=10$					$\mathit{M}=20$
Setup	Meth	${\mathit{L}}_{\mathit{F}}$	${\mathit{L}}_{\mathbf{2}}$	FN	FP	Time	${\mathit{L}}_{\mathit{F}}$	${\mathit{L}}_{\mathbf{2}}$	FN	FP	Time	${\mathit{L}}_{\mathit{F}}$	${\mathit{L}}_{\mathbf{2}}$	FN	FP	Time
	Naive	5.36	0.77	0	0.01	7.18	7.03	0.81	0.01	0.03	5.42	8.86	1.34	0.08	0.01	3.10
	Dtrace	3.22	0.70	0	0	322	5.14	0.80	0.01	0	207	5.43	1.35	0.03	0	214
S1 (type1)	Dglasso	3.76	0.72	0.01	0	9.44	4.97	0.97	0.05	0	7.74	5.96	1.26	0.26	0	3.66
	Bgd-lasso	3.22	0.70	0.01	0	32.1	4.25	0.74	0.01	0	15.9	4.26	0.75	0.02	0	13.8
$D={200}^2$	Bgd-scad	3.20	0.68	0.01	0	45.1	3.82	0.73	0.01	0	33.8	3.90	0.74	0.02	0	29.2
	Global	2.80	0.68	0	0	62.6
	Naive	9.30	0.77	0	0.03	81.8	12.1	0.96	0.01	0.01	75.2	14.5	1.30	0.90	0	22.8
	Dtrace	7.56	0.90	0.04	0	923	8.15	0.84	0.02	0	872	9.03	1.68	0.03	0	864
S1 (type2)	Dglasso	6.04	0.85	0.05	0	108	7.75	0.90	0.18	0	115	10.2	1.39	0.38	0	122
	Bgd-lasso	6.05	0.74	0.02	0	447	6.50	0.75	0.02	0	301	6.40	0.74	0.02	0	144
$D={500}^2$	Bgd-scad	6.00	0.82	0.01	0	448	5.95	0.77	0.01	0	308	6.03	0.82	0.01	0	225
	Global	5.89	0.82	0	0	984
	Naive	12.1	1.21	0.01	0.06	12.8	13.5	1.29	1.00	0	15.2	13.5	1.84	1	0	22.2
	Dtrace	9.96	1.15	0.01	0.01	457	10.4	1.08	0.04	0	446	10.9	1.92	0.27	0	438
S2 (type1)	Dglasso	8.37	1.09	0.08	0	18.8	8.41	1.06	0.19	0	20.1	10.7	2.02	0.35	0	24.7
	Bgd-lasso	8.07	0.96	0.01	0.01	73.8	8.15	0.97	0.01	0.01	37.6	8.20	0.98	0.01	0.01	25.5
$N=160$	Bgd-scad	7.78	0.97	0.01	0	90.3	7.87	0.99	0.01	0.01	39.3	8.00	1.02	0.01	0.01	62.8
	Global	7.89	1.01	0.04	0	210
	Naive	11.9	1.12	0.19	0.04	9.79	12.5	1.24	0.17	0.04	10.4	16.2	4.28	1	0	12.6
	Dtrace	11.2	1.35	0.02	0.02	592	10.8	1.16	0.24	0	624	14.8	3.71	0.99	0	657
S2 (type2)	Dglasso	9.02	1.24	0.10	0	20.2	9.09	1.25	0.30	0	26.7	14.5	4.32	0.58	0	32.8
	Bgd-lasso	8.86	1.15	0.02	0.02	80.6	8.95	1.15	0.03	0.02	40.9	12.2	1.28	0.04	0.06	25.4
$N=120$	Bgd-scad	8.74	1.13	0.02	0.02	92.9	8.89	1.15	0.03	0.02	98.5	9.29	1.44	0.04	0.03	81.7
	Global	8.68	1.12	0.03	0	208
	Naive	15.2	2.02	0.40	0	18.2	16.5	2.42	0.79	0.01	19.1	17.0	2.64	0.92	0	25.9
	Dtrace	9.45	1.36	0.20	0	805	12.1	1.80	0.25	0.01	834	16.7	2.49	0.95	0	845
S3 (type1)	Dglasso	10.6	2.35	0.26	0	20.8	14.5	2.79	0.06	0.01	28.1	15.4	4.16	0.83	0	30.4
	Bgd-lasso	8.15	1.18	0.02	0.02	416	10.2	1.59	0.01	0	328	10.8	1.64	0	0.02	232
$pr=0.01$	Bgd-scad	6.32	0.85	0.02	0	428	6.39	0.86	0.04	0	388	6.74	0.91	0.06	0	280
	Global	5.54	0.95	0.04	0	239
	Naive	24.1	3.32	0.63	0	11.3	25.0	3.58	0.83	0	15.5	25.1	3.80	0.96	0	22.7
	Dtrace	16.1	2.15	0.30	0	874	16.2	2.26	0.36	0	892	21.9	2.79	0.80	0	924
S3 (type2)	Dglasso	16.4	2.40	0.01	0.06	11.7	17.8	3.60	0.18	0.09	25.8	22.4	4.01	0.88	0	32.1
	Bgd-lasso	10.6	1.42	0	0.05	394	10.6	1.39	0	0.05	313	11.9	1.58	0.01	0.05	223
$pr=0.02$	Bgd-scad	9.40	1.18	0	0.04	408	9.43	1.19	0	0.05	352	11.1	1.28	0.03	0.03	261
	Global	10.1	1.49	0.09	0	524

Table A1. Performance of the Bgd and competing methods in the simulation study.

				$\mathit{M}=5$					$\mathit{M}=10$					$\mathit{M}=20$
Setup	Meth	${\mathit{L}}_{\mathit{F}}$	${\mathit{L}}_{\mathbf{2}}$	FN	FP	Time	${\mathit{L}}_{\mathit{F}}$	${\mathit{L}}_{\mathbf{2}}$	FN	FP	Time	${\mathit{L}}_{\mathit{F}}$	${\mathit{L}}_{\mathbf{2}}$	FN	FP	Time
	Naive	5.36	0.77	0	0.01	7.18	7.03	0.81	0.01	0.03	5.42	8.86	1.34	0.08	0.01	3.10
	Dtrace	3.22	0.70	0	0	322	5.14	0.80	0.01	0	207	5.43	1.35	0.03	0	214
S1 (type1)	Dglasso	3.76	0.72	0.01	0	9.44	4.97	0.97	0.05	0	7.74	5.96	1.26	0.26	0	3.66
	Bgd-lasso	3.22	0.70	0.01	0	32.1	4.25	0.74	0.01	0	15.9	4.26	0.75	0.02	0	13.8
$D={200}^2$	Bgd-scad	3.20	0.68	0.01	0	45.1	3.82	0.73	0.01	0	33.8	3.90	0.74	0.02	0	29.2
	Global	2.80	0.68	0	0	62.6
	Naive	9.30	0.77	0	0.03	81.8	12.1	0.96	0.01	0.01	75.2	14.5	1.30	0.90	0	22.8
	Dtrace	7.56	0.90	0.04	0	923	8.15	0.84	0.02	0	872	9.03	1.68	0.03	0	864
S1 (type2)	Dglasso	6.04	0.85	0.05	0	108	7.75	0.90	0.18	0	115	10.2	1.39	0.38	0	122
	Bgd-lasso	6.05	0.74	0.02	0	447	6.50	0.75	0.02	0	301	6.40	0.74	0.02	0	144
$D={500}^2$	Bgd-scad	6.00	0.82	0.01	0	448	5.95	0.77	0.01	0	308	6.03	0.82	0.01	0	225
	Global	5.89	0.82	0	0	984
	Naive	12.1	1.21	0.01	0.06	12.8	13.5	1.29	1.00	0	15.2	13.5	1.84	1	0	22.2
	Dtrace	9.96	1.15	0.01	0.01	457	10.4	1.08	0.04	0	446	10.9	1.92	0.27	0	438
S2 (type1)	Dglasso	8.37	1.09	0.08	0	18.8	8.41	1.06	0.19	0	20.1	10.7	2.02	0.35	0	24.7
	Bgd-lasso	8.07	0.96	0.01	0.01	73.8	8.15	0.97	0.01	0.01	37.6	8.20	0.98	0.01	0.01	25.5
$N=160$	Bgd-scad	7.78	0.97	0.01	0	90.3	7.87	0.99	0.01	0.01	39.3	8.00	1.02	0.01	0.01	62.8
	Global	7.89	1.01	0.04	0	210
	Naive	11.9	1.12	0.19	0.04	9.79	12.5	1.24	0.17	0.04	10.4	16.2	4.28	1	0	12.6
	Dtrace	11.2	1.35	0.02	0.02	592	10.8	1.16	0.24	0	624	14.8	3.71	0.99	0	657
S2 (type2)	Dglasso	9.02	1.24	0.10	0	20.2	9.09	1.25	0.30	0	26.7	14.5	4.32	0.58	0	32.8
	Bgd-lasso	8.86	1.15	0.02	0.02	80.6	8.95	1.15	0.03	0.02	40.9	12.2	1.28	0.04	0.06	25.4
$N=120$	Bgd-scad	8.74	1.13	0.02	0.02	92.9	8.89	1.15	0.03	0.02	98.5	9.29	1.44	0.04	0.03	81.7
	Global	8.68	1.12	0.03	0	208
	Naive	15.2	2.02	0.40	0	18.2	16.5	2.42	0.79	0.01	19.1	17.0	2.64	0.92	0	25.9
	Dtrace	9.45	1.36	0.20	0	805	12.1	1.80	0.25	0.01	834	16.7	2.49	0.95	0	845
S3 (type1)	Dglasso	10.6	2.35	0.26	0	20.8	14.5	2.79	0.06	0.01	28.1	15.4	4.16	0.83	0	30.4
	Bgd-lasso	8.15	1.18	0.02	0.02	416	10.2	1.59	0.01	0	328	10.8	1.64	0	0.02	232
$pr=0.01$	Bgd-scad	6.32	0.85	0.02	0	428	6.39	0.86	0.04	0	388	6.74	0.91	0.06	0	280
	Global	5.54	0.95	0.04	0	239
	Naive	24.1	3.32	0.63	0	11.3	25.0	3.58	0.83	0	15.5	25.1	3.80	0.96	0	22.7
	Dtrace	16.1	2.15	0.30	0	874	16.2	2.26	0.36	0	892	21.9	2.79	0.80	0	924
S3 (type2)	Dglasso	16.4	2.40	0.01	0.06	11.7	17.8	3.60	0.18	0.09	25.8	22.4	4.01	0.88	0	32.1
	Bgd-lasso	10.6	1.42	0	0.05	394	10.6	1.39	0	0.05	313	11.9	1.58	0.01	0.05	223
$pr=0.02$	Bgd-scad	9.40	1.18	0	0.04	408	9.43	1.19	0	0.05	352	11.1	1.28	0.03	0.03	261
	Global	10.1	1.49	0.09	0	524

Table A2. Performance of the Bgd and competing methods in the real-data analysis with different partitions.

	$\mathit{M}=5$			$\mathit{M}=10$			$\mathit{M}=20$
		$\mathit{M}=\mathbf{5}$			$\mathit{M}=\mathbf{10}$			$\mathit{M}=\mathbf{20}$
$\mathit{\gamma }$	Meth.	Prr	Sen	Spe	Prr	Sen	Spe	Prr	Sen	Spe
	Naive	0.13	0.89	0.86	0.14	0.89	0.84	0.17	0.87	0.78
	Dtrace	0.11	0.88	0.89	0.12	0.89	0.87	0.15	0.87	0.84
	Dglasso	0.15	0.89	0.81	0.15	0.89	0.81	0.18	0.86	0.77
$0.6$	Bgd-lasso	0.08	0.91	0.91	0.11	0.89	0.89	0.13	0.88	0.86
	Bgd-scad	0.10	0.89	0.90	0.11	0.88	0.89	0.14	0.86	0.86
	Global	0.06	0.92	0.95
	Naive	0.14	0.87	0.83	0.15	0.88	0.84	0.17	0.87	0.78
	Dtrace	0.11	0.88	0.89	0.12	0.89	0.87	0.15	0.87	0.84
	Dglasso	0.14	0.88	0.82	0.15	0.89	0.80	0.17	0.87	0.78
$0.8$	Bgd-lasso	0.07	0.90	0.94	0.09	0.90	0.91	0.12	0.88	0.89
	Bgd-scad	0.07	0.90	0.94	0.08	0.89	0.95	0.13	0.87	0.87
	Global	0.04	0.96	0.97

Table A2. Performance of the Bgd and competing methods in the real-data analysis with different partitions.

	$\mathit{M}=5$			$\mathit{M}=10$			$\mathit{M}=20$
		$\mathit{M}=\mathbf{5}$			$\mathit{M}=\mathbf{10}$			$\mathit{M}=\mathbf{20}$
$\mathit{\gamma }$	Meth.	Prr	Sen	Spe	Prr	Sen	Spe	Prr	Sen	Spe
	Naive	0.13	0.89	0.86	0.14	0.89	0.84	0.17	0.87	0.78
	Dtrace	0.11	0.88	0.89	0.12	0.89	0.87	0.15	0.87	0.84
	Dglasso	0.15	0.89	0.81	0.15	0.89	0.81	0.18	0.86	0.77
$0.6$	Bgd-lasso	0.08	0.91	0.91	0.11	0.89	0.89	0.13	0.88	0.86
	Bgd-scad	0.10	0.89	0.90	0.11	0.88	0.89	0.14	0.86	0.86
	Global	0.06	0.92	0.95
	Naive	0.14	0.87	0.83	0.15	0.88	0.84	0.17	0.87	0.78
	Dtrace	0.11	0.88	0.89	0.12	0.89	0.87	0.15	0.87	0.84
	Dglasso	0.14	0.88	0.82	0.15	0.89	0.80	0.17	0.87	0.78
$0.8$	Bgd-lasso	0.07	0.90	0.94	0.09	0.90	0.91	0.12	0.88	0.89
	Bgd-scad	0.07	0.90	0.94	0.08	0.89	0.95	0.13	0.87	0.87
	Global	0.04	0.96	0.97

Figure A1. The $L_F$ norms for type1 of cases S1–S3 with varies machines.

Figure A2. The Prr for real data analysis with various $\tau$.

Appendix A.2. Proof of the Main Results

Proof of Theorem 1.

For $\tilde{L}\left({\mathbf{\Omega }}^{(t+1)}\right)$ and a random machine m, if we set the initial value as ${\mathbf{\Omega }}^{\left(t\right)}$, then we take the Taylor expansion of ${\mathcal{l}}_{\left(m\right)}(\mathbf{\Omega })$ at ${\mathbf{\Omega }}^{\left(t\right)}$ and define $\Delta {\mathbf{\Omega }}^{\left(t\right)}={\mathbf{\Omega }}^{(t+1)}-{\mathbf{\Omega }}^{\left(t\right)}$, $f\left({\mathbf{\Omega }}^{\left(t\right)}\right)=({\nabla }_{\mathbf{\Omega }}{\mathcal{l}}_{\left(m\right)}\left({\mathbf{\Omega }}^{\left(t\right)}\right)-{\nabla }_{\mathbf{\Omega }}{\mathcal{l}}_N\left({\mathbf{\Omega }}^{\left(t\right)}\right)){\mathbf{\Omega }}^{\left(t\right)}$, there exists a $\tilde{\mathbf{\Omega }}$ between ${\mathbf{\Omega }}^{\left(t\right)}$ and ${\mathbf{\Omega }}^{(t+1)}$ such that

$$\begin{array}{c}l\tilde{L}\left({\mathbf{\Omega }}^{(t+1)}\right)+f\left({\mathbf{\Omega }}^{\left(t\right)}\right)-{\mathcal{l}}_{\left(m\right)}\left({\mathbf{\Omega }}^{\left(t\right)}\right)\hfill \\ ={\left(\mathrm{vec}(\Delta {\mathbf{\Omega }}^{\left(t\right)})\right)}^{\top }{\nabla }_{\mathrm{vec}(\mathbf{\Omega })}{\mathcal{l}}_N\left({\mathbf{\Omega }}^{\left(t\right)}\right)+{\displaystyle \frac{1}{2}}{\left(\mathrm{vec}(\Delta {\mathbf{\Omega }}^{\left(t\right)})\right)}^{\top }{\nabla }_{\mathrm{vec}(\mathbf{\Omega })}^2{\mathcal{l}}_m\left(\tilde{\mathbf{\Omega }}\right)\mathrm{vec}(\Delta {\mathbf{\Omega }}^{\left(t\right)})+\hfill \\ p\left(\right|{\mathbf{\Omega }}^{t+1}|,\lambda ),\hfill \end{array}$$

where

$${\nabla }_{{\mathrm{vec}}^2(\mathbf{\Omega })}{\mathcal{l}}_m\left(\tilde{\mathbf{\Omega }}\right)=\mathrm{diag}({\mathit{S}}_m,\cdots ,{\mathit{S}}_m)=\left(\begin{array}{ccc}{\mathit{S}}_m& & \\ & \ddots & \\ & & {\mathit{S}}_m\end{array}\right).$$

By the fact that

$${\left(\mathrm{vec}(\Delta {\mathbf{\Omega }}^{\left(t\right)})\right)}^{\top }{\nabla }_{\mathrm{vec}(\mathbf{\Omega })}{\mathcal{l}}_m\left({\mathbf{\Omega }}^{\left(t\right)}\right)=\mathrm{tr}({\nabla }_{\mathbf{\Omega }}{\mathcal{l}}_m\left({\mathbf{\Omega }}^{\left(t\right)}\right)\Delta {\mathbf{\Omega }}^{\left(t\right)}),$$

$${\displaystyle \frac{\nu }{2}}{\left(\mathrm{vec}(\Delta {\mathbf{\Omega }}^{\left(t\right)})\right)}^{\top }\mathrm{vec}(\Delta {\mathbf{\Omega }}^{\left(t\right)})={\displaystyle \frac{\nu }{2}}{\parallel \Delta {\mathbf{\Omega }}^{\left(t\right)}\parallel }_F^2.$$

If we choose $\nu$ satisfying

$$\begin{array}{c}\hfill {\displaystyle \frac{1}{2}}{\left(\mathrm{vec}(\Delta {\mathbf{\Omega }}^{\left(t\right)})\right)}^{\top }{\nabla }_{\mathrm{vec}(\mathbf{\Omega })}^2{\mathcal{l}}_m\left(\tilde{\mathbf{\Omega }}\right)\mathrm{vec}(\Delta {\mathbf{\Omega }}^{\left(t\right)})\le {\displaystyle \frac{\nu }{2}}{\left(\mathrm{vec}(\Delta {\mathbf{\Omega }}^{\left(t\right)})\right)}^{\top }\mathrm{vec}(\Delta {\mathbf{\Omega }}^{\left(t\right)}),\end{array}$$

(A1)

we have

$$\tilde{L}\left({\mathbf{\Omega }}^{(t+1)}\right)+f\left({\mathbf{\Omega }}^{\left(t\right)}\right)\le g\left({\mathbf{\Omega }}^{(t+1)}\right|{\mathbf{\Omega }}^{\left(t\right)})+f\left({\mathbf{\Omega }}^{\left(t\right)}\right).$$

Note that ${\mathbf{\Omega }}^{(t+1)}=arg{min}_{\mathbf{\Omega }}g(\mathbf{\Omega }|{\mathbf{\Omega }}^{\left(t\right)})$ subject to $(j,l)\in C_t$, where $j,l\in \left[p\right]$, we have

$$g\left({\mathbf{\Omega }}^{(t+1)}\right|{\mathbf{\Omega }}^{\left(t\right)})\le g\left({\mathbf{\Omega }}^{\left(t\right)}\right|{\mathbf{\Omega }}^{\left(t\right)})=\tilde{L}\left({\mathbf{\Omega }}^{\left(t\right)}\right),$$

which means if $\nu \ge {\Lambda }_{max}\left({\mathit{S}}_m\right)$ and we use local client m to compute (6), one can obtain

$$\tilde{L}\left({\mathbf{\Omega }}^{(t+1)}\right)\le g\left({\mathbf{\Omega }}^{(t+1)}\right|{\mathbf{\Omega }}^{\left(t\right)})\le g\left({\mathbf{\Omega }}^{\left(t\right)}\right|{\mathbf{\Omega }}^{\left(t\right)})=\tilde{L}\left({\mathbf{\Omega }}^{\left(t\right)}\right).$$

We have completed the proof of Theorem 1. □

Lemma A1.

Given ${\mathit{X}}_1,{\mathit{X}}_2,\cdots ,{\mathit{X}}_N$ being i.i.d. sub-Gaussian random variables with $Var\left({\mathit{X}}_i\right)={\mathbf{\Sigma }}^{*}$, and suppose $\parallel {\mathit{X}}_i{\parallel }_{{\psi }_2}\le H$. Let ${\mathit{S}}_m$ be the local covariance matrix of machine m and define $\mathit{S}=\frac{1}{M}{\sum }_{m=1}^M{\mathit{S}}_m$. Then we have a constant C

$${\parallel \mathit{S}-{\mathbf{\Sigma }}^{*}\parallel }_{max}\le C{H}^2\sqrt{\frac{logp}{N}}.$$

The Lemma can be found in [26].

Proof of Theorem 2.

We prove the Theorem 2 by setting the penalty as Lasso, other penalties can simplify modified the proof. Considering

$$\check{\mathbf{\Omega }}=arg\underset{\mathbf{\Omega }}{min}{\mathcal{l}}_N(\mathbf{\Omega })+\lambda {\parallel \mathbf{\Omega }\parallel }_{1,\mathrm{off}}.$$

Define $\left\{{\tilde{\mathbf{\Omega }}}^{\left(t\right)}\right\}$ as the sequence generated by the following optimization problem without the block constraint $C_t$:

$$\begin{array}{c}\hfill arg\underset{\mathbf{\Omega }}{min}h(\mathbf{\Omega }|{\tilde{\mathbf{\Omega }}}^{\left(t\right)})={\mathcal{l}}_N\left({\tilde{\mathbf{\Omega }}}^{\left(t\right)}\right)+\mathrm{tr}\left[{\nabla }_{\mathbf{\Omega }}{\mathcal{l}}_N\left({\tilde{\mathbf{\Omega }}}^{\left(t\right)}\right)(\mathbf{\Omega }-{\tilde{\mathbf{\Omega }}}^{\left(t\right)})\right]+\frac{\nu }{2}\parallel \mathbf{\Omega }-{\tilde{\mathbf{\Omega }}}^{\left(t\right)}{\parallel }_F^2+p\left(\right|\mathbf{\Omega }|,\lambda ).\end{array}$$

Suppose the finial solution is $\tilde{\mathbf{\Omega }}$, then by theorem 3 of Beck and Teboulle [27], we have

$$\begin{array}{c}\hfill {\mathcal{l}}_N\left(\tilde{\mathbf{\Omega }}\right)+p\left(\right|\tilde{\mathbf{\Omega }}|,\lambda )\le {\mathcal{l}}_N\left({\mathbf{\Omega }}^{*}\right)+p\left(\right|{\mathbf{\Omega }}^{*}|,\lambda )+1/\left(2t\right){\Lambda }_{max}\left(\mathit{S}\right){\parallel {\tilde{\mathbf{\Omega }}}^{\left(0\right)}-{\mathbf{\Omega }}^{*}\parallel }_F^2.\end{array}$$

By Taylor’s expansion, we have

$${\mathcal{l}}_N\left(\tilde{\mathbf{\Omega }}\right)-{\mathcal{l}}_N\left({\mathbf{\Omega }}^{*}\right)=\mathrm{tr}\left[(\tilde{\mathbf{\Omega }}-{\mathbf{\Omega }}^{*}){\nabla }_{\mathbf{\Omega }}{\mathcal{l}}_N\left({\mathbf{\Omega }}^{*}\right)\right]+(1/2)\mathrm{tr}\left[(\tilde{\mathbf{\Omega }}-{\mathbf{\Omega }}^{*})\mathit{S}(\tilde{\mathbf{\Omega }}-{\mathbf{\Omega }}^{*})\right].$$

Note that using Lemma A1,

$$\parallel {\nabla }_{\mathbf{\Omega }}{\mathcal{l}}_N\left({\mathbf{\Omega }}^{*}\right){\parallel }_{\infty }=\parallel \mathit{S}{\mathbf{\Omega }}^{*}{-\mathit{I}\parallel }_{\infty }\le \parallel {\mathit{S}}^{*}{\mathbf{\Omega }}^{*}{-\mathit{I}\parallel }_{\infty }+{\parallel (\mathit{S}-{\mathit{S}}^{*}){\mathbf{\Omega }}^{*}\parallel }_{\infty }\le Q\sqrt{logp/N}\le \lambda /2.$$

When the iteration $t>4\lambda \parallel \mathrm{diag}({\mathbf{\Omega }}^{*}-\tilde{\mathbf{\Omega }}){\parallel }_1/({\Lambda }_{max}\left(\mathit{S}\right)\parallel \tilde{\mathbf{\Omega }}-{\mathbf{\Omega }}^{*}{\parallel }_F^2)$, we have

$$(1/t){\Lambda }_{max}\left(\mathit{S}\right)\parallel {\tilde{\mathbf{\Omega }}}^{\left(0\right)}-{\mathbf{\Omega }}^{*}{\parallel }_F^2\le 4\lambda {\parallel \mathrm{diag}({\mathbf{\Omega }}^{*}-\tilde{\mathbf{\Omega }})\parallel }_1.$$

Combining the above inequalities, and if we set $p\left(\right|\mathbf{\Omega }|,\lambda )=\lambda \parallel {\mathbf{\Omega }}^{*}{\parallel }_{1,\mathrm{off}}$, we have

$$\begin{array}{c}(1/2)\mathrm{tr}\left[{(\tilde{\mathbf{\Omega }}-{\mathbf{\Omega }}^{*})}^{\top }\mathit{S}(\tilde{\mathbf{\Omega }}-{\mathbf{\Omega }}^{*})\right]\hfill \\ \le \lambda \parallel \tilde{\mathbf{\Omega }}-{\mathbf{\Omega }}^{*}{\parallel }_1+2\lambda \parallel {\mathbf{\Omega }}^{*}{\parallel }_{1,\mathrm{off}}-2\lambda \parallel \tilde{\mathbf{\Omega }}{\parallel }_{1,\mathrm{off}}+2\lambda {\parallel \mathrm{diag}({\mathbf{\Omega }}^{*}-\tilde{\mathbf{\Omega }})\parallel }_1.\hfill \end{array}$$

(A2)

which further implies that

$$\begin{array}{c}(1/2)\mathrm{tr}\left[{(\tilde{\mathbf{\Omega }}-{\mathbf{\Omega }}^{*})}^{\top }\mathit{S}(\tilde{\mathbf{\Omega }}-{\mathbf{\Omega }}^{*})\right]+\lambda {\parallel \tilde{\mathbf{\Omega }}-{\mathbf{\Omega }}^{*}\parallel }_1\hfill \\ \le 2\lambda \parallel \tilde{\mathbf{\Omega }}-{\mathbf{\Omega }}^{*}{\parallel }_1+2\lambda \parallel {\mathbf{\Omega }}^{*}{\parallel }_{1,\mathrm{off}}-2\lambda \parallel \tilde{\mathbf{\Omega }}{\parallel }_{1,\mathrm{off}}+2\lambda {\parallel \mathrm{diag}({\mathbf{\Omega }}^{*}-\tilde{\mathbf{\Omega }})\parallel }_1\hfill \\ \le 4\lambda \sum _{j_1{j}_2\in S^{*}}{\parallel {(\tilde{\mathbf{\Omega }}-{\mathbf{\Omega }}^{*})}_{j_1{j}_2}\parallel }_1.\hfill \end{array}$$

By the fact that $\mathrm{tr}\left[{(\tilde{\mathbf{\Omega }}-{\mathbf{\Omega }}^{*})}^{\top }\mathit{S}(\tilde{\mathbf{\Omega }}-{\mathbf{\Omega }}^{*})\right]\ge 0$, we have

$$\parallel \tilde{\mathbf{\Omega }}-{\mathbf{\Omega }}^{*}{\parallel }_1\le 4{\parallel {(\tilde{\mathbf{\Omega }}-{\mathbf{\Omega }}^{*})}_{s^{*}}\parallel }_1.$$

Then, the condition (C4) is satisfied, and we have

$$C_3\parallel \tilde{\mathbf{\Omega }}-{\mathbf{\Omega }}^{*}{\parallel }_F^2\le (1/2)\mathrm{tr}\left[{(\tilde{\mathbf{\Omega }}-{\mathbf{\Omega }}^{*})}^{\top }\mathit{S}(\tilde{\mathbf{\Omega }}-{\mathbf{\Omega }}^{*})\right]\le \lambda \parallel {(\tilde{\mathbf{\Omega }}-{\mathbf{\Omega }}^{*})}_{s^{*}}{\parallel }_1\le \lambda \sqrt{p{s}_p}{\parallel \tilde{\mathbf{\Omega }}-{\mathbf{\Omega }}^{*}\parallel }_F.$$

Then, we have

$$\begin{array}{c}\hfill \frac{1}{p}{\parallel \tilde{\mathbf{\Omega }}-{\mathbf{\Omega }}^{*}\parallel }_F^2\le C{s}_p{M}^2{\lambda }^2=O_p\left(M^2\frac{s_{p}logp}{N}\right).\end{array}$$

(A3)

Now, we turn to the surrogate loss function $\tilde{L}(\mathbf{\Omega })$, when given the initial ${\mathbf{\Omega }}^{\left(t\right)}$ in t-th iteration, $h(\mathbf{\Omega }|{\mathbf{\Omega }}^{\left(t\right)})$ has the same gradient and solution path as $\tilde{L}(\mathbf{\Omega })$ without the block constraints using the gradient descent method. Note that for ${\mathbf{\Omega }}^{\left(t\right)}$, we have

$$\parallel \frac{1}{M}({\mathit{S}}_m{\mathbf{\Omega }}^{\left(t\right)}-\mathit{I}){\parallel }_{\infty }\le \rho \le C_{2}M\sqrt{\frac{logp}{n}}.$$

Using theorem 4 of [5] and the above statements, we have the same conclusion as (A3), that is

$$\begin{array}{c}\hfill \frac{1}{p}{\parallel \widehat{\mathbf{\Omega }}-{\mathbf{\Omega }}^{*}\parallel }_F^2=O_p\left(M^2\frac{s_{p}logp}{N}\right).\end{array}$$

□