A Useful Criterion on Studying Consistent Estimation in Community Detection

Abstract

In network analysis, developing a unified theoretical framework that can compare methods under different models is an interesting problem. This paper proposes a partial solution to this problem. We summarize the idea of using a separation condition for a standard network and sharp threshold of the Erdös–Rényi random graph to study consistent estimation, and compare theoretical error rates and requirements on the network sparsity of spectral methods under models that can degenerate to a stochastic block model as a four-step criterion SCSTC. Using SCSTC, we find some inconsistent phenomena on separation condition and sharp threshold in community detection. In particular, we find that the original theoretical results of the SPACL algorithm introduced to estimate network memberships under the mixed membership stochastic blockmodel are sub-optimal. To find the formation mechanism of inconsistencies, we re-establish the theoretical convergence rate of this algorithm by applying recent techniques on row-wise eigenvector deviation. The results are further extended to the degree-corrected mixed membership model. By comparison, our results enjoy smaller error rates, lesser dependence on the number of communities, weaker requirements on network sparsity, and so forth. The separation condition and sharp threshold obtained from our theoretical results match the classical results, so the usefulness of this criterion on studying consistent estimation is guaranteed. Numerical results for computer-generated networks support our finding that spectral methods considered in this paper achieve the threshold of separation condition.

1. Introduction

Networks with latent structure are ubiquitous in our daily life, for example, social networks from social platforms, protein–protein interaction networks, co-citation networks and co-authorship networks [1,2,3,4,5,6,7,8,9,10,11,12,13,14,15]. Community detection is a powerful tool to learn the latent community structure in networks and graphs in social science, computer science, machine learning, statistical science and complex networks [16,17,18,19,20,21,22]. The goal of community detection is to infer a node’s community information from the network.

Many models have been proposed to model networks with latent community structure; see [23] for a survey. The stochastic blockmodel (SBM) [24] stands out for its simplicity, and it has received increasing attention in recent years [25,26,27,28,29,30,31,32,33,34,35]. However, the SBM only models a non-overlapping network in which each node belongs to a single community. Estimating mixed memberships of the network whose node may belong to multiple communities has received a lot of attention [36,37,38,39,40,41,42,43,44]. To capture the structure of the network with mixed memberships, Ref. [36] proposed the popular mixed membership stochastic blockmodel (MMSB), which is an extension of SBM from non-overlapping networks to overlapping networks. It is well known that the degree-corrected stochastic blockmodel (DCSBM) [45] is an extension of SBM by considering the degree heterogeneity of nodes to fit real-world networks with various node degree. Similarly, Ref. [41] proposed a model named the degree-corrected mixed membership (DCMM) model as an extension of MMSB by considering the degree heterogeneity of nodes. There are alternative models based on MMSB, such as the overlapping continuous community assignment model (OCCAM) of [40] and the stochastic blockmodel with overlap (SBMO) proposed by [46], which can also model networks with mixed memberships. As discussed in Section 5, OCCAM equals DCMM, while SBMO is a special case of DCMM.

1.1. Spectral Clustering Approaches

For the four models SBM, DCSBM, MMSB and DCMM, many researchers focus on designing algorithms with provable consistent theoretical guarantees. Spectral clustering [47] is one of the most widely applied methods with guarantees of consistency for community detection.

Within the SBM and DCSBM frameworks for a non-overlapping network, spectral clustering has two steps. It first conducts the eigen-decomposition of the adjacency matrix or the Laplacian matrix [26,48,49]. Then it runs a clustering algorithm (typically, k-means) on some leading eigenvectors or their variants to infer the community membership. For example, Ref. [26] showed the consistency of spectral clustering designed based on Laplacian matrix under SBM. Ref. [48] proposed a regularized spectral clustering (RSC) algorithm designed based on regularized Laplacian matrix and shows its theoretical consistency under DCSBM. Ref. [30] studied the consistencies of two spectral clustering algorithms based on the adjacency matrix under SBM and DCSBM. Ref. [50] designed the spectral clustering on the ratios-of-eigenvectors (SCORE) algorithm with a theoretical guarantee under DCSBM. Ref. [49] studied the impact of regularization on a Laplacian spectral clustering under SBM.

Within the MMSB and DCMM frameworks for the overlapping network, broadly speaking, spectral clustering has the following three steps. One first conducts an eigen-decomposition of the adjacency matrix or the graph Laplacian, then hunts corners (also known as vertexes) using a convex hull algorithm, and finally has a membership reconstruction step by projection. The convex hull algorithms suggested in [41] differ in the k-means algorithm a lot. For example, Ref. [44] designed the sequential projection after cleaning (SPACL) algorithm based on the finding that there exists a simplex structure in the eigen-decomposition of the population adjacency matrix and studies the SPACL theoretical properties under MMSB. Meanwhile, SPACL uses the successive projection algorithm proposed in [51] to find the corners for its simplex structure. To fit DCMM, Ref. [41] designs the Mixed-SCORE algorithm based on the finding that there exists a simplex structure in the entry-wise ratio matrix obtained from the eigen-decomposition of the population adjacency matrix under DCMM. Ref. [41] also introduces several choices for convex hull algorithms to find corners for the simplex structure and show the estimation consistency of the Mixed-SCORE under DCMM. Ref. [43] finds the cone structure inherent in the normalization of eigenvectors of the population adjacency matrix under DCMM as well as OCCAM, and develops an algorithm to hunt corners in the cone structure.

1.2. Separation Condition, Alternative Separation Condition and Sharp Threshold

SBM with n nodes belonging to K equal (or nearly equal) size communities and vertices connect with probability $p_{\mathrm{in}}$ within clusters and $p_{\mathrm{out}}$ across clusters, denoted by $SBM(n,K,p_{\mathrm{in}},p_{\mathrm{out}})$, has been well studied in recent years, especially for the case when $K=2$; see [21] and the references therein. In this paper, we call the network generated from $SBM(n,K,p_{\mathrm{in}},p_{\mathrm{out}})$ the standard network for convenience. Without causing confusion, we also call $SBM(n,K,p_{\mathrm{in}},p_{\mathrm{out}})$ the standard network, occasionally. Let $p_{\mathrm{in}}={\alpha }_{\mathrm{in}}\frac{\log \left(n\right)}{n},p_{\mathrm{out}}={\alpha }_{\mathrm{out}}\frac{\log \left(n\right)}{n}$. Refs. [21,52] found that exact recovery in $SBM(n,2,{\alpha }_{\mathrm{in}}\frac{\log \left(n\right)}{n},{\alpha }_{\mathrm{out}}\frac{\log \left(n\right)}{n})$ is solvable, and efficiently so, if $|\sqrt{{\alpha }_{\mathrm{in}}}-\sqrt{{\alpha }_{\mathrm{out}}}|>\sqrt{2}$ (i.e., $|\sqrt{p_{\mathrm{in}}}-\sqrt{p_{\mathrm{out}}}|>\sqrt{\frac{2\log \left(n\right)}{n}}$) and unsolvable if $|\sqrt{{\alpha }_{\mathrm{in}}}-\sqrt{{\alpha }_{\mathrm{out}}}|<\sqrt{2}$ as summarized in Theorem 13 of [53]. This threshold can be achieved by semidefinite relaxations [21,54,55,56] and spectral methods with local refinements [57,58]. Unlike semidefinite relaxations, spectral methods have a different threshold, which was particularly pointed out by [21,52]: one highlight for $SBM(n,2,p_{\mathrm{in}},p_{\mathrm{out}})$ is a theorem by [59] which says that when $p_{\mathrm{in}}>p_{\mathrm{out}}$, if

$$\begin{array}{c}\hfill \frac{p_{\mathrm{in}}-p_{\mathrm{out}}}{\sqrt{p_{\mathrm{in}}}}\gg \sqrt{\frac{\log \left(n\right)}{n}}\end{array}$$

then spectral methods can exactly recover node labels with high probability as n goes to infinity (also known as consistent estimation [30,40,41,43,44,48,50]).

Consider a more general case $SBM(n,K,p_{\mathrm{in}},p_{\mathrm{out}})$ with $K=O\left(1\right)$; this paper finds that the above threshold can be extended as

$$\begin{array}{c}\hfill \frac{|p_{\mathrm{in}}-p_{\mathrm{out}}|}{\sqrt{\max (p_{\mathrm{in}},p_{\mathrm{out}})}}\gg \sqrt{\frac{\log \left(n\right)}{n}},\end{array}$$

(1)

which can be alternatively written as

$$\begin{array}{c}\hfill \frac{|{\alpha }_{\mathrm{in}}-{\alpha }_{\mathrm{out}}|}{\sqrt{\max ({\alpha }_{\mathrm{in}},{\alpha }_{\mathrm{out}})}}\gg 1.\end{array}$$

(2)

In this paper, when $K=O\left(1\right)$, the lower bound requirement on $\frac{|p_{\mathrm{in}}-p_{\mathrm{out}}|}{\sqrt{\max (p_{\mathrm{in}},p_{\mathrm{out}})}}$ (and $\frac{|{\alpha }_{\mathrm{in}}-{\alpha }_{\mathrm{out}}|}{\sqrt{\max ({\alpha }_{\mathrm{in}},{\alpha }_{\mathrm{out}})}}$) for the consistent estimation of spectral methods is called the separation condition (alternative separation condition). The network generated from $SBM(n,K,p_{\mathrm{in}},p_{\mathrm{out}})$ with $p_{\mathrm{in}}>p_{\mathrm{out}}$ is an assortative network in which nodes within the community have more edges than across communities [60]. The network generated from $SBM(n,K,p_{\mathrm{in}},p_{\mathrm{out}})$ with $p_{\mathrm{in}}<p_{\mathrm{out}}$ is a dis-assortative network in which nodes within the community have fewer edges than across communities [60]. Therefore, Equation (2) holds for both assortative and dis-assortative networks.

Meanwhile, when $K=1$ such that $p=p_{\mathrm{in}}=p_{\mathrm{out}}$, $SBM(n,K,p_{\mathrm{in}},p_{\mathrm{out}})=SBM(n,1,p,p)$ degenerates to Erdös–Rényi (ER) random graph $G(n,p)$ [53,61,62]. Ref. [61] finds that the ER random graph is connected with high probability if

$$\begin{array}{c}\hfill p\ge \frac{\log \left(n\right)}{n}.\end{array}$$

(3)

We call the lower bound requirement on p for generating a connected ER random graph the sharp threshold in this paper.

1.3. Inconsistencies on Separation Condition in Some Previous Works

In this paper, we focus on the consistency of spectral method in community detection. The study of consistency is developed by obtaining the theoretical upper bound of error rate for a spectral method through analyzing the properties of the population adjacency matrix under the statistical model. To compare the consistencies of the theoretical results under different models, it is meaningful to study whether the separation condition and sharp threshold obtained from upper bounds of theoretical error rates for different methods under different models are consistent or not. Meanwhile, the separation condition and sharp threshold can also be seen as alternative unified theoretical frameworks to compare all methods and model parameters mentioned in the concluding remarks of [30].

Based on the separation condition and sharp threshold, here we describe some phenomena of the inconsistency in the community detection area. We find that the separation conditions of $SBM(n,K,p_{\mathrm{in}},p_{\mathrm{out}})$ with $K=O\left(1\right)$ obtained from the error rates developed in [41,43,44] under DCMM or MMSB are not consistent with those obtained from the main results of [30] under SBM, and the sharp threshold obtained from the main results of [43,44] do not match the classical results. A summary of these inconsistencies is provided in Table 1 and Table 2. Furthermore, after delicate analysis, we find that the requirement on the network sparsity of [43,44] is stronger than that of [30,41], and [63] also finds that the requirement of Ref. [44] of network sparsity is sub-optimal.

Table 1. Comparison of separation condition and sharp threshold. Details of this table are given in Section 4. The classical result on separation condition given in Corollary 1 of [59] is $\sqrt{\frac{\log \left(n\right)}{n}}$ (i.e., Equation (1)). The classical result on sharp threshold is $\frac{\log \left(n\right)}{n}$ (i.e., Equation (3)) given in [61], Theorem 4.6 [62] and the first bullet in Section 2.5 [53]. In this paper, n is the number of nodes in a network, A is the adjacency matrix, $\Omega$ is the expectation of A under some models, $A_{\mathrm{re}}$ is a regularization of A, $\rho$ is the sparsity parameter such that $\rho \ge {\max }_{i,j}\Omega (i,j)$ and it controls the overall sparsity of a network, $\parallel ·\parallel$ denotes spectral norm, and $\xi >1$.

	Model	Separation Condition	Sharp Threshold
Ours using $\parallel A_{\mathrm{re}}-\Omega \parallel \le C\sqrt{\rho n}$	MMSB&DCMM	$\sqrt{\frac{\log \left(n\right)}{n}}$	$\frac{\log \left(n\right)}{n}$
Ours using $\parallel A-\Omega \parallel \le C\sqrt{\rho n\log \left(n\right)}$	MMSB&DCMM	$\sqrt{\frac{\log \left(n\right)}{n}}$	$\frac{\log \left(n\right)}{n}$
Ref. [41] using $\parallel A_{\mathrm{re}}-\Omega \parallel \le C\sqrt{\rho n}$ (original)	DCMM	$\sqrt{\frac{\log \left(n\right)}{n}}$	$\frac{\log \left(n\right)}{n}$
Ref. [41] using $\parallel A-\Omega \parallel \le C\sqrt{\rho n\log \left(n\right)}$	DCMM	$\sqrt{\frac{\log \left(n\right)}{n}}$	$\frac{\log \left(n\right)}{n}$
Refs. [43,44] using $\parallel A_{\mathrm{re}}-\Omega \parallel \le C\sqrt{\rho n}$ (original)	MMSB&DCMM	$\frac{{\log }^{\xi }\left(n\right)}{\sqrt{n}}$	$\frac{{\log }^{2\xi }\left(n\right)}{n}$
Refs. [43,44] using $\parallel A-\Omega \parallel \le C\sqrt{\rho n\log \left(n\right)}$	MMSB&DCMM	$\frac{{\log }^{\xi +0.5}\left(n\right)}{\sqrt{n}}$	$\frac{{\log }^{2\xi +1}\left(n\right)}{n}$
Ref. [30] using $\parallel A_{\mathrm{re}}-\Omega \parallel \le C\sqrt{\rho n}$ (original)	SBM&DCSBM	$\sqrt{\frac{1}{n}}$	$\frac{1}{n}$
Ref. [30] using $\parallel A-\Omega \parallel \le C\sqrt{\rho n\log \left(n\right)\log \left(n\right)}$	SBM&DCSBM	$\sqrt{\frac{\log \left(n\right)}{n}}$	$\frac{\log \left(n\right)}{n}$

Table 2. Comparison of alternative separation condition, where the classical result on alternative separation condition is 1 (i.e., Equation (2)).

	Model	Alternative Separation Condition
Ours using $\parallel A_{\mathrm{re}}-\Omega \parallel \le C\sqrt{\rho n}$	MMSB&DCMM	1
Ours using $\parallel A-\Omega \parallel \le C\sqrt{\rho n\log \left(n\right)}$	MMSB&DCMM	1
Ref. [41] using $\parallel A_{\mathrm{re}}-\Omega \parallel \le C\sqrt{\rho n}$ (original)	DCMM	1
Ref. [41] using $\parallel A-\Omega \parallel \le C\sqrt{\rho n\log \left(n\right)}$	DCMM	1
Refs. [43,44] using $\parallel A_{\mathrm{re}}-\Omega \parallel \le C\sqrt{\rho n}$ (original)	MMSB&DCMM	${\log }^{\xi -0.5}\left(n\right)$
Refs. [43,44] using $\parallel A-\Omega \parallel \le C\sqrt{\rho n\log \left(n\right)}$	MMSB&DCMM	${\log }^{\xi }\left(n\right)$
Ref. [30] using $\parallel A_{\mathrm{re}}-\Omega \parallel \le C\sqrt{\rho n}$ (original)	SBM&DCSBM	$\sqrt{\frac{1}{\log \left(n\right)}}$
Ref. [30] using $\parallel A-\Omega \parallel \le C\sqrt{\rho n\log \left(n\right)\log \left(n\right)}$	SBM&DCSBM	1

1.4. Our Findings

Recall that we reviewed several spectral clustering methods under SBM, DCSBM, MMSB and DCMM introduced in [26,30,41,43,44,48,49,50] and DCSBM, MMSB and DCMM are extensions of SBM (i.e., $SBM(n,K,p_{\mathrm{in}},p_{\mathrm{out}})$ is a special case of DCSBM, MMSB and DCMM). We have the following question:

Can these spectral clustering methods achieve the threshold in Equation (1) (or Equation (2)) for $SBM(n,K,p_{\mathrm{in}},p_{\mathrm{out}})$ with $K=O\left(1\right)$ and the threshold in Equation (3) for the Erdös–Rényi (ER) random graph $G(n,p)$?

The answer is yes. In fact, spectral methods for network with mixed memberships still achieve thresholds in Equations (1) and (2) for $MMSB(n,K,\Pi ,p_{\mathrm{in}},p_{\mathrm{out}})$ defined in Definition 2 when $K=O\left(1\right)$, where $MMSB(n,K,\Pi ,p_{\mathrm{in}},p_{\mathrm{out}})$ can be seen as a generalization of $SBM(n,K,p_{\mathrm{in}},p_{\mathrm{out}})$ such that there exist nodes belonging to multiple communities. Explanations for why these spectral clustering methods achieve thresholds in Equations (1)–(3) will be provided in Section 3, Section 4 and Section 5 via re-establishing theoretical guarantee for SPACL under MMSB and its extension under DCMM because we find that the main theoretical results of [43,44] are sub-optimal. Meanwhile, we can obtain (and cannot obtain) the separation condition and sharp threshold from the theoretical bounds of error rates for spectral methods analyzed in [30,41,43,44] ([26,30,48,49,50]) directly. Instead of re-establishing the theoretical guarantee for all spectral methods reviewed in this paper to show that they achieve thresholds in Equations (1) and (3) for $SBM(n,K,P_{\mathrm{in}},P_{\mathrm{out}})$ with $K=O\left(1\right)$, we mainly focus on the SPACL algorithm under MMSB and its extension under DCMM since MMSB and DCMM are more complex than SBM and DCSBM.

We then summarize the idea of using the separation condition and sharp threshold to study the consistencies, and compare the error rates and requirements on network sparsity of different spectral methods under different models as a four-step criterion, which we call the separation condition and sharp threshold criterion (SCSTC for short). With an application of this criterion, this paper provides an attempt to answer the questions of how the above inconsistency phenomena occur, and how to obtain consistent resultswith weaker requirements on the network sparsity of [43,44]. To answer the two questions, we use the recent techniques on row-wise eigenvector deviation developed in [64,65] to obtain consistent theoretical results directly related with model parameters for the SPACL and the SVM-cone-DCMMSB algorithm of [43]. The two questions are then answered by delicate analysis with an application of SCSTC to the theoretical upper bounds of error rates in this paper and some previous spectral methods. Using SCSTC for the spectral methods introduced and studied in [26,30,48,49,50] and some other spectral methods fitting models that can reduce to $SBM(n,K,P_{\mathrm{in}},P_{\mathrm{out}})$ with $K=O\left(1\right)$, one can prove that these spectral methods achieve thresholds in Equations (1)–(3). The main contributions in this paper are as follows:

(i)

We summarize the idea of using the separation condition of a standard network and sharp threshold of the ER random graph $G(n,p)$ to study the consistent estimations of different spectral methods designed via eigen-decomposition or singular value decomposition of the adjacency matrix or its variants under different models that can degenerate to SBM under mild conditions as a four-step criterion, SCSTC. The separation condition is used to study the consistency of the theoretical upper bound for the spectral method, and the sharp threshold can be used to study the network sparsity. The theoretical results of upper bounds for different spectral methods can be compared by SCSTC. Using this criterion, a few inconsistent phenomenons of some previous works are found.

(ii)

Under MMSB and DCMM, we study the consistencies of the SPACL algorithm proposed in [44] and its extended version using the recent techniques on row-wise eigenvector deviation developed in [64,65]. Compared with the original results of [43,44], our main theoretical results enjoy smaller error rates by lesser dependence on K and $\log \left(n\right)$. Meanwhile, our main theoretical results have weaker requirements on the network sparsity and the lower bound of the smallest nonzero singular value of the population adjacency matrix. For details, see Table 3 and Table 4.

Table 3. Comparison of error rates between our Theorem 1 and Theorem 3.2 [44] under $MMS{B}_n(K,\tilde{P},\Pi ,\rho )$. The dependence on K is obtained when $\kappa ({\Pi }^{\prime }\Pi )=O\left(1\right)$. For comparison, we have adjusted the $l_2$ error rates of Theorem 3.2 [44] into $l_1$ error rates. Note that as analyzed in the first bullet given after Lemma 2, whether using $\parallel A-\Omega \parallel \le C\sqrt{\rho n\log \left(n\right)}$ or $\parallel A_{\mathrm{re}}-\Omega \parallel \le C\sqrt{\rho n}$ does not change our $\varpi$, and has no influence on bound in Theorem 1. For [44], using $\parallel A_{\mathrm{re}}-\Omega \parallel \le \sqrt{\rho n}$, the power of $\log \left(n\right)$ in their Theorem 3.2 is $\xi$; using $\parallel A-\Omega \parallel \le \sqrt{\rho n\log \left(n\right)}$, the power of $\log \left(n\right)$ in their Theorem 3.2 is $\xi +0.5$.

	$\mathit{\rho }\mathit{n}$	${\mathit{\sigma }}_{\mathit{K}}\left(\mathbf{\Omega }\right)$	${\mathit{\lambda }}_{\mathit{K}}\left({\mathbf{\Pi }}^{\prime }\mathbf{\Pi }\right)$	Dependence on K	Dependence on $\mathbf{log}\left(\mathit{n}\right)$
Ours	$\ge \log \left(n\right)$	$⪰\sqrt{\rho n\log \left(n\right)}$	$>0$	$K^2$	${\log }^{0.5}\left(n\right)$
[44]	$\ge {\log }^{2\xi }\left(n\right)$	$⪰\sqrt{\rho n}{\log }^{\xi }\left(n\right)$	$\ge 1/\rho$	$K^{2.5}$	${\log }^{\xi }\left(n\right)$

Table 4. Comparison of error rates between our Theorem 2 and Theorem 3.2 [43] under $DCM{M}_n(K,P,\Pi ,\Theta )$. The dependence on K is obtained when $\kappa ({\Pi }^{\prime }\Pi )=O\left(1\right)$. For comparison, we adjusted the $l_2$ error rates of Theorem 3.2 [43] into $l_1$ error rates. Since Theorem 2 enjoys the same separation condition and sharp threshold as Theorem 1, and Theorem 3.2 [43] enjoys the same separation condition and sharp threshold as Theorem 3.2 [44], we do not report them in this table. Note that as analyzed in Remark 11, whether using $\parallel A-\Omega \parallel \le C\sqrt{{\theta }_{\max }{\parallel \theta \parallel }_1\log \left(n\right)}$ or $\parallel A_{\mathrm{re}}-\Omega \parallel \le C\sqrt{{\theta }_{\max }{\parallel \theta \parallel }_1}$ does not change our $\varpi$ under DCMM, and has no influence on the results in Theorem 2. For [43], using $\parallel A_{\mathrm{re}}-\Omega \parallel \sqrt{{\theta }_{\max }{\parallel \theta \parallel }_1}$, the power of $\log \left(n\right)$ in their Theorem 3.2 is $\xi$; using $\parallel A-\Omega \parallel \sqrt{{\theta }_{\max }{\parallel \theta \parallel }_1\log \left(n\right)}$, the power of $\log \left(n\right)$ in their Theorem 3.2 is $\xi +0.5$.

	$\mathbf{\Pi }(\mathit{i},:)$	${\mathit{\theta }}_{\mathbf{max}}{\parallel \mathit{\theta }\parallel }_1$	${\mathit{\sigma }}_{\mathit{K}}\left(\mathbf{\Omega }\right)$	$\mathit{\kappa }\left({\mathbf{\Pi }}^{\prime }{\mathbf{\Theta }}^2\mathbf{\Pi }\right)$	Dependence on K	Dependence on $\mathbf{log}\left(\mathit{n}\right)$
Ours	arbitrary	$\ge \log \left(n\right)$	$⪰{\theta }_{\max }\sqrt{n\log \left(n\right)}$	$\ge 1$	$K^6$	${\log }^{0.5}\left(n\right)$
[43]	$iid$ from Dirichlet	$\ge {\log }^{2\xi }\left(n\right)$	$⪰{\theta }_{\max }\sqrt{n}{\log }^{\xi }\left(n\right)$	$=O\left(1\right)$	$K^{6.5}$	${\log }^{\xi }\left(n\right)$

(iii)

Our results for DCMM are consistent with those for MMSB when DCMM degenerates to MMSB under mild conditions. Using SCSTC, under mild conditions, our main theoretical results under DCMM are consistent with those of [41]. This answers the question that the phenomenon that the main results of [43,44] do not match those of [41] occurs due to the fact that in Refs. [43,44], the theoretical results of error rates are sub-optimal. We also find that our theoretical results (as well as those of [41]) under both MMSB and DCMM match the classical results on the separation condition and sharp threshold, i.e., achieve thresholds in Equations (1)–(3). Using the bound of $\parallel A-\Omega \parallel$ instead of $\parallel A_{\mathrm{re}}-\Omega \parallel$ to establish the upper bound of error rate under SBM in [30], the two spectral methods studied in [30] achieve thresholds in Equations (1)–(3), which answers the question of why the separation condition obtained from error rate of [41] does not match that obtained from the error rate of [30]. Using $\parallel A_{\mathrm{re}}-\Omega \parallel$ or $\parallel A-\Omega \parallel$ influences the row-wise eigenvector deviations in Theorem 3.1 of [44] and Theorem I.3 of [43], and thus using $\parallel A_{\mathrm{re}}-\Omega \parallel$ or $\parallel A-\Omega \parallel$ influences the separation conditions and sharp thresholds of [43,44]. For comparison, our bound on row-wise eigenvector deviation is obtained by using the techniques developed in [64,65] and that of [41] is obtained by applying the modified Theorem 2.1 of [66]; therefore, using $\parallel A_{\mathrm{re}}-\Omega \parallel$ or $\parallel A-\Omega \parallel$ has no influence on the separation conditions and sharp thresholds of ours and that of [41]. For details, see Table 1 and Table 2. In a word, using SCSTC, the spectral methods proposed and studied in [26,30,41,43,44,48,49,50,67,68] or some other spectral methods fitting models that can reduce to $SBM(n,K,p_{\mathrm{in}},p_{\mathrm{out}})$ achieve thresholds in Equations (1)–(3).

(iv)

We verify our threshold in Equation (2) by some computer-generated networks in Section 6. The numerical results for networks generated under $MMSB(n,K,\Pi ,p_{\mathrm{in}},p_{\mathrm{out}})$ when $K=2$ and $K=3$ show that SPACL and its extended version achieve a threshold in Equation (2), and results for networks generated from $SBM(n,K,p_{\mathrm{in}},p_{\mathrm{out}})$ when $K=2$ and $K=3$ show that the spectral methods considered in [26,30,48,50] achieve the threshold in Equation (2).

The article is organized as follows. In Section 2, we give the formal introduction to the mixed membership stochastic blockmodel and review the algorithm SPACL considered in this paper. The theoretical results of consistency for the mixed membership stochastic blockmodel are presented and compared to related works in Section 3. After delicate analysis, the separation condition and sharp threshold criterion is presented in Section 4. Based on an application of this criterion, the improvement consistent estimation results for the extended version of SPACL under the degree corrected mixed membership model are provided in Section 5. Several computer-generated networks under MMSB and SBM are conducted to show that some spectral clustering methods achieve the threshold in Equation (2) in Section 6. The conclusion is given in Section 7.

Notations. We take the following general notations in this paper. Write $\left[m\right]:=\{1,2,\dots ,m\}$ for any positive integer m. For a vector x and fixed $q>0$, ${\parallel x\parallel }_q$ denotes its $l_q$-norm. We drop the subscript if $q=2$ occasionally. For a matrix M, $M^{\prime }$ denotes the transpose of the matrix M, $\parallel M\parallel$ denotes the spectral norm, ${\parallel M\parallel }_F$ denotes the Frobenius norm, ${\parallel M\parallel }_{2\to \infty }$ denotes the maximum $l_2$-norm of all the rows of M, and ${\parallel M\parallel }_{\infty }:={\max }_i{\sum }_j\left|M(i,j)\right|$ denotes the maximum absolute row sum of M. Let $\mathrm{rank}\left(M\right)$ denote the rank of matrix M. Let ${\sigma }_i\left(M\right)$ be the i-th largest singular value of matrix M, ${\lambda }_i\left(M\right)$ denote the i-th largest eigenvalue of the matrix M ordered by the magnitude, and $\kappa \left(M\right)$ denote the condition number of M. $M(i,:)$ and $M(:,j)$ denote the i-th row and the j-th column of matrix M, respectively. $M(S_r,:)$ and $M(:,S_c)$ denote the rows and columns in the index sets $S_r$ and $S_c$ of matrix M, respectively. For any matrix M, we simply use $Y=\max (0,M)$ to represent $Y_{ij}=\max (0,M_{ij})$ for any $i,j$. For any matrix $M\in {\mathbb{R}}^{m\times m}$, let $\mathrm{diag}\left(M\right)$ be the $m\times m$ diagonal matrix whose i-th diagonal entry is $M(i,i)$. $\mathbf{1}$ and $\mathbf{0}$ are column vectors with all entries being ones and zeros, respectively. $e_i$ is a column vector whose i-th entry is 1, while other entries are zero. In this paper, C is a positive constant which may vary occasionally. $f\left(n\right)=O\left(g\right(n\left)\right)$ means that there exists a constant $c>0$ such that $\left|f\right(n\left)\right|\le c\left|g\right(n\left)\right|$ holds for all sufficiently large n. $x⪰y$ means there exists a constant $c>0$ such that $\left|x\right|\ge c\left|y\right|$. $f\left(n\right)=o\left(g\right(n\left)\right)$ indicates that $\frac{f\left(n\right)}{g\left(n\right)}\to 0$ as $n\to \infty$.

2. Mixed Membership Stochastic Blockmodel

Let $A\in {\{0,1\}}^{n\times n}$ be a symmetric adjacency matrix such that $A(i,j)=1$ if there is an edge between node i to node j, and $A(i,j)=0$ otherwise. The mixed membership stochastic blockmodel (MMSB) [36] for generating A is as follows.

$$\begin{array}{c}\hfill \Omega :=\rho \Pi \tilde{P}{\Pi }^{\prime }A(i,j)\sim \mathrm{Bernoulli}(\Omega (i,j))i,j\in \left[n\right],\end{array}$$

(4)

where $\Pi \in {\mathbb{R}}^{n\times K}$ is called the membership matrix with $\Pi (i,k)\ge 0$ and ${\sum }_{k=1}^K\Pi (i,k)=1$ for $i\in \left[n\right]$ and $k\in \left[K\right]$, $\tilde{P}\in {\mathbb{R}}^{K\times K}$ is an non-negative symmetric matrix with ${\max }_{k,l\in \left[K\right]}\tilde{P}(k,l)=1$ for model identifiability under MMSB, $\rho$ is called the sparsity parameter which controls the sparsity of the network, and $\Omega \in {\mathbb{R}}^{n\times n}$ is called the population adjacency matrix since $\mathbb{E}\left[A\right]=\Omega$. As mentioned in [41,44], ${\sigma }_K\left(\tilde{P}\right)$ is a measure of the separation between communities, and we call it the separation parameter in this paper. $\rho$ and ${\sigma }_K\left(\tilde{P}\right)$ are two important model parameters directly related with the separation condition and sharp threshold, and they will be considered throughout this paper.

Definition 1.

Call model (4) the mixed membership stochastic blockmodel (MMSB), and denote it by $MMS{B}_n(K,\tilde{P},\Pi ,\rho )$.

Definition 2.

Let $MMSB(n,K,\Pi ,p_{\mathrm{in}},p_{\mathrm{out}})$ be a special case of $MMS{B}_n(K,\tilde{P},\Pi ,\rho )$ when $\rho \tilde{P}$ has diagonal entries $p_{\mathrm{in}}$ and non-diagonal entries $p_{\mathrm{out}}$, and $\kappa ({\Pi }^{\prime }\Pi )=O\left(1\right)$.

Call node i ‘pure’ if $\Pi (i,:)$ is degenerate (i.e., one entry is 1, all others $K-1$ entries are 0) and ‘mixed’ otherwise. When all nodes are pure in $\Pi$, we see that $MMSB(n,K,\Pi ,p_{\mathrm{in}},p_{\mathrm{out}})$ exactly reduces to $SBM(n,K,p_{\mathrm{in}},p_{\mathrm{out}})$. Thus, $MMSB(n,K,\Pi ,p_{\mathrm{in}},p_{\mathrm{out}})$ is a generalization of $SBM(n,K,p_{\mathrm{in}},p_{\mathrm{out}})$ with mixed nodes in each community. In this paper, we show that SPACL [44] fitting MMSB and SVM-cone-DCMMSB [43] and Mixed-SCORE [41] fitting DCMM also achieve thresholds in Equations (1)–(3) for $MMSB(n,K,\Pi ,p_{\mathrm{in}},p_{\mathrm{out}})$ with $K=O\left(1\right)$. By Theorems 2.1 and 2.2 [44], the following conditions are sufficient for the identifiability of MMSB, when $\rho \tilde{P}(k,l)\in [0,1]$ for all $k,l\in \left[K\right]$,

• (I1) $\mathrm{rank}\left(\tilde{P}\right)=K$.
• (I2) There is at least one pure node for each of the K communities.

Unless specified, we treat conditions (I1) and (I2) as the default from now on.

For $k\in \left[K\right]$, let ${\mathcal{I}}^{\left(k\right)}$ be the set of pure nodes in community k such that ${\mathcal{I}}^{\left(k\right)}=\{i\in \left[n\right]:\Pi (i,k)=1\}$. For $k\in \left[K\right]$, select one node from ${\mathcal{I}}^{\left(k\right)}$ to construct the index set $\mathcal{I}$, i.e., $\mathcal{I}$ is the index of nodes corresponding to K pure nodes, one from each community. Without loss of generality, let $\Pi (\mathcal{I},:)=I_K$ where $I_K$ is the $K\times K$ identity matrix. Recall that $\mathrm{rank}(\Omega )=K$. Let $\Omega =U\Lambda U^{\prime }$ be the compact eigen-decomposition of $\Omega$ such that $U\in {\mathbb{R}}^{n\times K},\Lambda \in {\mathbb{R}}^{K\times K}$, and $U^{\prime }U=I_K$. Lemma 2.1 [44] gives that $U=\Pi U(\mathcal{I},:)$, and such a form is called ideal simplex (IS for short) [41,44] since all rows of U form a K-simplex in ${\mathbb{R}}^K$ and the K rows of $U(\mathcal{I},:)$ are the vertices of the K-simplex. Given $\Omega$ and K, as long as we know $U(\mathcal{I},:)$, we can exactly recover $\Pi$ by $\Pi =U{U}^{-1}(\mathcal{I},:)$ since $U(\mathcal{I},:)\in {\mathbb{R}}^{K\times K}$ is a full rank matrix. As mentioned in [41,44], for such IS, the successive projection (SP) algorithm [51] (i.e., Algorithm A1) can be applied to U with K communities to exactly find the corner matrix $U(\mathcal{I},:)$. For convenience, set $Z=U{U}^{-1}(\mathcal{I},:)$. Since $\Pi =Z$, we have $\Pi (i,:)=\frac{Z(i,:)}{{\parallel Z(i,:)\parallel }_1}$ for $i\in \left[n\right]$.

Based on the above analysis, we are now ready to give the ideal SPACL algorithm with input $\Omega ,K$ and output $\Pi$.

• Let $\Omega =U\Lambda U^{\prime }$ be the top-K eigen-decomposition of $\Omega$ such that $U\in {\mathbb{R}}^{n\times K},\Lambda \in {\mathbb{R}}^{K\times K},U^{\prime }U=I$.
• Run SP algorithm on the rows of U assuming that there are K communities to obtain $\mathcal{I}$.
• Set $Z=U{U}^{-1}(\mathcal{I},:)$.
• Recover $\Pi$ by setting $\Pi (i,:)=\frac{Z(i,:)}{{\parallel Z(i,:)\parallel }_1}$ for $i\in \left[n\right]$.

With the given U and K, since the SP algorithm returns $U(\mathcal{I},:)$, we see that the ideal SPACL exactly (for detail, see Appendix B) returns $\Pi$.

Now, we review the SPACL algorithm of [44]. Set $\tilde{A}=\widehat{U}\widehat{\Lambda }{\widehat{U}}^{\prime }$ to be the top K eigen-decomposition of A such that $\widehat{U}\in {\mathbb{R}}^{n\times K},\widehat{\Lambda }\in {\mathbb{R}}^{K\times K},{\widehat{U}}^{\prime }\widehat{U}=I_K$, and $\widehat{\Lambda }$ contains the top K eigenvalues of A. For the real case, use $\widehat{Z},\widehat{\Pi }$ given in Algorithm 1 to estimate $Z,\Pi$, respectively. Algorithm 1 is the SPACL algorithm [44] where we only care about the estimation of the membership matrix $\Pi$, and omit the estimation of P and $\rho$. Meanwhile, Algorithm 1 is a direct extension of the ideal SPACL algorithm from the oracle case to the real case, and we omit the prune step in the original SPACL algorithm of [44].

Algorithm 1 SPACL [44]

Require: The adjacency matrix $A\in {\mathbb{R}}^{n\times n}$ and the number of communities K. Ensure: The estimated $n\times K$ membership matrix $\widehat{\Pi }$. 1: Obtain $\tilde{A}=\widehat{U}\widehat{\Lambda }{\widehat{U}}^{\prime }$, the top K eigen-decomposition of A. 2: Apply SP algorithm (i.e., Algorithm A1) on the rows of $\widehat{U}$ assuming there are K communities to obtain $\widehat{\mathcal{I}}$, the index set returned by SP algorithm. 3: Set $\widehat{Z}=\widehat{U}{\widehat{U}}^{-1}(\widehat{\mathcal{I}},:)$. Then set $\widehat{Z}=\max (0,\widehat{Z})$. 4: Estimate $\Pi (i,:)$ by $\widehat{\Pi }(i,:)=\widehat{Z}(i,:)/{\parallel \widehat{Z}(i,:)\parallel }_1,i\in \left[n\right]$.

3. Consistency under MMSB

Our main result under MMSB provides an upper bound on the estimation error of each node’s membership in terms of several model parameters. Throughout this paper, K is a known positive integer. Assume that

(A1)

$\rho n\ge \log \left(n\right)$.

Assumption (A1) provides a requirement on the lower bound of sparsity parameter $\rho$ such that it should be at least $\log \left(n\right)/n$. Then we have the following lemma.

Lemma 1.

Under $MMS{B}_n(K,\tilde{P},\Pi ,\rho )$, when Assumption (A1) holds, with probability at least $1-o\left(n^{-\alpha }\right)$ for any $\alpha >0$, we have

$$\begin{array}{c}\hfill \parallel A-\Omega \parallel \le \frac{\alpha +1+\sqrt{(\alpha +1)(\alpha +19)}}{3}\sqrt{\rho n\log \left(n\right)}.\end{array}$$

In Lemma 1, instead of simply using a constant $C_{\alpha }$ to denote $\frac{\alpha +1+\sqrt{(\alpha +1)(\alpha +19)}}{3}$, we keep the explicit form here.

Remark 1.

When Assumption (A1) holds, the upper bound of $\parallel A-\Omega \parallel$ in Lemma 1 is consistent with Corollary 6.5 in [69] since $\mathrm{Var}\left(A\right(i,j\left)\right)\le \rho$ under $MMS{B}_n(K,P,\Pi ,\rho )$.

Lemma 1 is obtained via Theorem 1.4 (Bernstein inequality) in [70]. For comparison, Ref. [44] applies Theorem 5.2 [30] to bound $\parallel A-\Omega \parallel$ (see, for example, Equation (14) of [44]) and obtains a bound as $C\sqrt{\rho n}$ for some $C>0$. However, $C\sqrt{\rho n}$ is the bound between a regularization of A and $\Omega$ as stated in the proof of Theorem 5.2 [30], where such regularization of A is obtained from A with some constraints in Lemmas 4.1 and 4.2 of the supplemental material [30]. Meanwhile, Theorem 2 [71] also gives that the bound between a regularization of A and $\Omega$ is $C\sqrt{\rho n}$, where such a regularization of A should also satisfy few constraints on A; see Theorem 2 [71] for detail. Instead of bounding the difference between a regularization of A and $\Omega$, we are interested in bounding $\parallel A-\Omega \parallel$ by the Bernstein inequality, which has no constraints on A. For convenience, use $A_{\mathrm{re}}$ to denote the regularization of A in this paper. Hence, $\parallel A_{\mathrm{re}}-\Omega \parallel \le C\sqrt{\rho n}$ with high probability, and this bound is model independent as shown by Theorem 5.2 [30] and Theorem 2 [71] as long as $\rho \ge {\max }_{i,j}\Omega (i,j)$ (here, let $\Omega =\mathbb{E}\left[A\right]$ without considering models, a $\rho$ satisfying $\rho \ge {\max }_{i,j}\Omega (i,j)$ is also the sparsity parameter which controls the overall sparsity of a network). Note that $A_{\mathrm{re}}$ is not $\tilde{A}$, where $\tilde{A}=\widehat{U}\Lambda {\widehat{U}}^{\prime }$ is obtained by the top K eigen-decomposition of A, while $A_{\mathrm{re}}$ is obtained by adding constraints on degrees of A; see Theorem 2 [71] for detail.

In [41,43,44], the main theoretical results for their proposed membership estimating methods hinge on a row-wise deviation bound for the eigenvectors of the adjacency matrix, whether under MMSB or DCMM. Different from the theoretical technique applied in Theorem 3.1 [44], which provides sup-optimal dependencies on $\log \left(n\right)$ and K, and needs sub-optimal requirements on the sparsity parameter $\rho$ and the lower bound of ${\sigma }_K(\Omega )$, to obtain row-wise deviation bound for the singular eigenvector of $\Omega$, we use Theorem 4.2 [64] and Theorem 4.2 [65].

Lemma 2

(Row-wise eigenspace error). Under $MMS{B}_n(K,\tilde{P},\Pi ,\rho )$, when Assumption (A1) holds, suppose ${\sigma }_K(\Omega )\ge C\sqrt{\rho n\log \left(n\right)}$, with probability at least $1-o\left(n^{-\alpha }\right)$,

• When we apply Theorem 4.2 of [64], we have

  $$\begin{array}{c}\hfill \parallel \widehat{U}{\widehat{U}}^{\prime }-U{U}^{\prime }{\parallel }_{2\to \infty }=O\left(\frac{\sqrt{K}(\kappa (\Omega )\sqrt{\frac{n}{K{\lambda }_K({\Pi }^{\prime }\Pi )}}+\sqrt{\log \left(n\right)})}{{\sigma }_K\left(\tilde{P}\right)\sqrt{\rho }{\lambda }_K({\Pi }^{\prime }\Pi )}\right),\end{array}$$
• When we apply Theorem 4.2 of [65], we have

  $$\begin{array}{c}\hfill \parallel \widehat{U}{\widehat{U}}^{\prime }-U{U}^{\prime }{\parallel }_{2\to \infty }=O\left(\frac{\sqrt{n\log \left(n\right)}}{{\sigma }_K\left(\tilde{P}\right)\sqrt{\rho }{\lambda }_K^{1.5}({\Pi }^{\prime }\Pi )}\right).\end{array}$$

For convenience, set $\varpi =\parallel \widehat{U}{\widehat{U}}^{\prime }-U{U}^{\prime }{\parallel }_{2\to \infty }$, and let ${\varpi }_1,{\varpi }_2$ denote the upper bound in Lemma 2 when applying Theorem 4.2 of [64] and Theorem 4.2 of [65], respectively. Note that when ${\lambda }_K({\Pi }^{\prime }\Pi )=O\left(\frac{n}{K}\right)$, we have ${\varpi }_1={\varpi }_2=O\left(\frac{K^{1.5}}{{\sigma }_K\left(\tilde{P}\right)}\frac{1}{\sqrt{n}}\sqrt{\frac{\log \left(n\right)}{\rho n}}\right)$, and therefore we simply let ${\varpi }_2$ be the bound since its form is slightly simpler than ${\varpi }_1$.

Compared with Theorem 3.1 of [44], since we apply Theorem 4.2 of [64] and Theorem 4.2 of [65] to obtain the bound of row-wise eigenspace error under MMSB, our bounds do not rely on $\min (K^2,{\kappa }^2(\Omega ))$ while Theorem 3.1 [44] does. Meanwhile, our bound in Lemma 2 is sharper with lesser dependence on K and $\log \left(n\right)$, has weaker requirements on the lower bounds of ${\sigma }_K(\Omega ),{\lambda }_K({\Pi }^{\prime }\Pi )$ and the sparsity parameter $\rho$. The details are given below:

• We emphasize that the bound of Theorem 3.1 of [44] should be $\parallel \widehat{U}{\widehat{U}}^{\prime }-U{U}^{\prime }{\parallel }_{2\to \infty }=O\left(\frac{\psi (\Omega )\sqrt{Kn}{\log }^{\xi }\left(n\right)}{{\sigma }_K\left(\tilde{P}\right)\sqrt{\rho }{\lambda }_K^{1.5}({\Pi }^{\prime }\Pi )}\right)$ instead of $\parallel \widehat{U}{\widehat{U}}^{\prime }-U{U}^{\prime }{\parallel }_{2\to \infty }=O\left(\frac{\psi (\Omega )\sqrt{Kn}}{{\sigma }_K\left(\tilde{P}\right)\sqrt{\rho }{\lambda }_K^{1.5}({\Pi }^{\prime }\Pi )}\right)$ for $\xi >1$ where the function $\psi$ is defined in Equation (7) of [44], and this is also pointed out by Table 2 of [63]. The reason is that in the proof part of Theorem 3.1 [44], from step (iii) to step (iv), they should keep the term ${\log }^{\xi }\left(n\right)$ since this term is much larger than 1. We can also find that the bound in Theorem 3.1 [44] should multiply ${\log }^{\xi }\left(n\right)$ from Theorem VI.1 [44] directly. For comparison, this bound $O\left(\frac{\psi (\Omega )\sqrt{Kn}{\log }^{\xi }\left(n\right)}{{\sigma }_K\left(\tilde{P}\right)\sqrt{\rho }{\lambda }_K^{1.5}({\Pi }^{\prime }\Pi )}\right)$ is $K^{0.5}{\log }^{\xi -0.5}\left(n\right)$ times our bound in Lemma 2. Meanwhile, by the proof of the bound in Theorem 3.1 of [44], we see that the bound depends on the upper bound of $\parallel A-\Omega \parallel$, and [44] applies Theorem 5.2 of [30] such that $\parallel A_{\mathrm{re}}-\Omega \parallel \le C\sqrt{\rho n}$ with high probability. Since $C\sqrt{\rho n}$ is the upper bound of the difference between a regularization of A and $\Omega$. Therefore, if we are only interested in bounding $\parallel A-\Omega \parallel$ instead of $\parallel A_{\mathrm{re}}-\Omega \parallel$, the upper bound of Theorem 3.1 [44] should be $O\left(\frac{\psi (\Omega )\sqrt{Kn}{\log }^{\xi +0.5}\left(n\right)}{{\sigma }_K\left(\tilde{P}\right){\lambda }_K^{1.5}({\Pi }^{\prime }\Pi )}\right)$, which is at least $K^{0.5}{\log }^{\xi }\left(n\right)$ times our bound in Lemma 2. Furthermore, the upper bound of the row-wise eigenspace error in Lemma 2 does not rely on the upper bound of $\parallel A-\Omega \parallel$ as long as ${\sigma }_K(\Omega )\ge C\sqrt{\rho n\log \left(n\right)}$ holds. Therefore, whether using $\parallel A_{\mathrm{re}}-\Omega \parallel \le C\sqrt{\rho n}$ or $\parallel A-\Omega \parallel \le C\sqrt{\rho n\log \left(n\right)}$ does not change the bound in Lemma 2.
• Our Lemma 2 requires ${\sigma }_K(\Omega )\ge C\sqrt{\rho n\log \left(n\right)}$, while Theorem 3.1 [44] requires ${\sigma }_K(\Omega )\ge 4\sqrt{\rho n}{\log }^{\xi }\left(n\right)$ by their Assumption 3.1. Therefore, our Lemma 2 has a weaker requirement on the lower bound of ${\sigma }_K(\Omega )$ than that of Theorem 3.1 [44]. Meanwhile, Theorem 3.1 [44] requires ${\lambda }_K({\Pi }^{\prime }\Pi )\ge \frac{1}{\rho }$ while our Lemma 2 has no lower bound requirement on ${\lambda }_K({\Pi }^{\prime }\Pi )$ as long as it is positive.
• Since $\parallel \Omega \parallel =\parallel \rho \Pi \tilde{P}{\Pi }^{\prime }\parallel \le C\rho n$ by basic algebra, the lower bound requirement on ${\sigma }_K(\Omega )$ in Assumption 3.1 of [44] gives that $4\sqrt{\rho n}{log}^{\xi }\left(n\right)\le {\sigma }_K(\Omega )\le \parallel \Omega \parallel \le C\rho n$, which suggests that Theorem 3.1 [44] requires $\rho n\ge C{\log }^{2\xi }\left(n\right)$, and this also matches with the requirement on $\rho n$ in Theorem VI.1 of [44] (and this is also pointed out by Table 1 of [63]). For comparison, our requirement on sparsity given in Assumption (A1) is $\rho n\ge \log \left(n\right)$, which is weaker than $\rho n\ge C{\log }^{2\xi }\left(n\right)$. Similarly, in our Lemma 2, the requirement ${\sigma }_K(\Omega )\ge C\sqrt{\rho n\log \left(n\right)}$ gives $C\sqrt{\rho n\log \left(n\right)}\le {\sigma }_K(\Omega )\le \parallel \Omega \parallel \le C\rho n$, thus we have $\log \left(n\right)\le C\rho n$ which is consistent with Assumption (A1).

If we further assume that $K=O\left(1\right),{\lambda }_K({\Pi }^{\prime }\Pi )=O\left(\frac{n}{K}\right)$ (i.e., $\kappa ({\Pi }^{\prime }\Pi )=O\left(1\right)$) and ${\sigma }_K\left(\tilde{P}\right)=O\left(1\right)$, the row-wise eigenspace error is of order $\frac{1}{\sqrt{n}}\sqrt{\frac{\log \left(n\right)}{\rho n}}$, which is consistent with the row-wise eigenvector deviation of the result of [63], shown in their Table 2. The next theorem gives the theoretical bounds on the estimations of memberships under MMSB.

Theorem 1.

Under $MMS{B}_n(K,\tilde{P},\Pi ,\rho )$, let $\widehat{\Pi }$ be obtained from Algorithm 1, and suppose the conditions in Lemma 2 hold; there exists a permutation matrix $\mathcal{P}\in {\mathbb{R}}^{K\times K}$ such that, with probability at least $1-o\left(n^{-\alpha }\right)$, we have

$$\begin{array}{c}\hfill {\max }_{i\in \left[n\right]}{\parallel e_i^{\prime }(\widehat{\Pi }-\Pi \mathcal{P})\parallel }_1=O\left(\varpi K\kappa ({\Pi }^{\prime }\Pi )\sqrt{{\lambda }_1({\Pi }^{\prime }\Pi )}\right).\end{array}$$

Remark 2

(Comparison to Theorem 3.2 [44]). Consider a special case by setting $\kappa ({\Pi }^{\prime }\Pi )=O\left(1\right)$, i.e., ${\lambda }_K({\Pi }^{\prime }\Pi )=O\left(\frac{n}{K}\right)$ and ${\lambda }_1({\Pi }^{\prime }\Pi )=O\left(\frac{n}{K}\right)$. We focus on comparing the dependencies on K in bounds of our Theorem 1 and Theorem 3.2 [44]. Under this case, the bound of our Theorem 1 is proportional to $K^2$ by basic algebra; since $\min (K^2,{\kappa }^2(\Omega ))=\min (K^2,O\left(1\right))=O\left(1\right)$ and the bound in Theorem 3.2 [44] should multiply $\sqrt{K}$ because (in [44]’s language) $\parallel {\widehat{V}}_p^{-1}{\parallel }_F\le \frac{\sqrt{K}}{{\sigma }_K\left({\widehat{V}}_p\right)}$ instead of $\parallel {\widehat{V}}_p^{-1}{\parallel }_F=\frac{1}{{\lambda }_K\left({\widehat{V}}_p\right)}$ in Equation (45) [44], the power of K is 2 by checking the bound of Theorem 3.2 [44]. Meanwhile, note that our bound in Theorem 2 is $l_1$ bound, while the bound in Theorem 3.2 [44] is $l_2$ bound. When we translate the $l_2$ bound of Theorem 3.2 [44] into $l_1$ bound, the power of K is 2.5 for Theorem 3.2 [44]. Hence, our bound in Theorem 1 has less dependence on K than that of Theorem 3.2 [44], and this is also consistent with the first bullet given after Lemma 2.

Table 3 summarizes the necessary conditions and dependence on the model parameters of the rates in Theorem 1 and Theorem 3.2 [44] for comparison. The following corollary is obtained by adding conditions on the model parameters similar to Corollary 3.1 in [44].

Corollary 1.

Under $MMSB(n,K,\Pi ,p_{\mathrm{in}},p_{\mathrm{out}})$ with $K=O\left(1\right)$, when the conditions of Lemma 2 hold, with probability at least $1-o\left(n^{-\alpha }\right)$, we have

$$\begin{array}{c}\hfill {\max }_{i\in \left[n\right]}{\parallel e_i^{\prime }(\widehat{\Pi }-\Pi \mathcal{P})\parallel }_1=O\left(\frac{1}{{\sigma }_K\left(\tilde{P}\right)}\sqrt{\frac{\log \left(n\right)}{\rho n}}\right).\end{array}$$

Remark 3.

Consider a special case in Corollary 1 by setting ${\sigma }_K\left(\tilde{P}\right)$ as a constant, we see that the error bound $O\left(\sqrt{\frac{\log \left(n\right)}{\rho n}}\right)$ in Corollary 1 is directly related to Assumption (A1), and for consistent estimation, ρ should shrink slower than $\frac{\log \left(n\right)}{n}$.

Remark 4.

Under the setting of Corollary 1, the requirement ${\sigma }_K(\Omega )\ge C\sqrt{\rho n\log \left(n\right)}$ in Lemma 2 holds naturally. By Lemma II.4 [44], we know that ${\sigma }_K(\Omega )\ge \rho {\sigma }_K\left(\tilde{P}\right){\lambda }_K({\Pi }^{\prime }\Pi )=C\rho n{\sigma }_K\left(\tilde{P}\right)$. To make the requirement ${\sigma }_K(\Omega )\ge C\sqrt{\rho n\log \left(n\right)}$ always hold, we just need $C\rho n{\sigma }_K\left(\tilde{P}\right)\ge C\sqrt{\rho n\log \left(n\right)}$, which gives that ${\sigma }_K\left(\tilde{P}\right)\ge C\sqrt{\frac{\log \left(n\right)}{\rho n}}$, and it just matches with the requirement of the consistent estimation of memberships in Corollary 1.

Remark 5

(Comparison to Theorem 3.2 [44]). When $K=O\left(1\right)$ and ${\lambda }_K({\Pi }^{\prime }\Pi )=O\left(\frac{n}{K}\right)$, by the first bullet in the analysis given after Lemma 2, the row-wise eigenspace error of Theorem 3.1 [44] is $O\left(\frac{{\log }^{\xi }\left(n\right)}{{\sigma }_K\left(\tilde{P}\right)\sqrt{\rho }n}\right)$, and it gives that their error bound on estimation membership given in their Equation (3) is $O\left(\frac{{\log }^{\xi }\left(n\right)}{{\sigma }_K\left(\tilde{P}\right)\sqrt{\rho n}}\right)$, which is ${\log }^{\xi -0.5}\left(n\right)$ times of the bound in our Lemma 1.

Remark 6

(Comparison to Theorem 2.2 [41]). Replacing the Θ in [41] by $\Theta =\sqrt{\rho }I$, their DCMM model degenerates to MMSB. Then their conditions in Theorem 2.2 are our Assumption (A1) and ${\lambda }_K({\Pi }^{\prime }\Pi )=O\left(\frac{n}{K}\right)$ for MMSB. When $K=O\left(1\right)$, the error bound in Theorem 2.2 in [41] is $O\left(\frac{1}{{\sigma }_K\left(\tilde{P}\right)}\sqrt{\frac{\log \left(n\right)}{\rho n}}\right)$, which is consistent with ours.

4. Separation Condition and Sharp Threshold Criterion

After obtaining Corollary 1 under MMSB, now we are ready to give our criterion after introducing the separation condition of $MMSB(n,K,\Pi ,p_{\mathrm{in}},p_{\mathrm{out}})$ with $K=O\left(1\right)$ and the sharp threshold of ER random graph $G(n,p)$ in this section.

Separation condition. Let $P=\rho \tilde{P}$ be the probability matrix for $MMSB(n,K,\Pi ,p_{\mathrm{in}},p_{\mathrm{out}})$ when $K=O\left(1\right)$, so P has diagonal (and non-diagonal) entries $p_{\mathrm{in}}$ (and $p_{\mathrm{out}}$) and ${\sigma }_K\left(P\right)=\rho {\sigma }_K\left(\tilde{P}\right)\equiv |p_{\mathrm{in}}-p_{\mathrm{out}}|$. Recall that ${\max }_{k,l\in \left[K\right]}\tilde{P}(k,l)=1$ under $MMS{B}_n(K,\tilde{P},\Pi ,\rho )$, we have ${\max }_{k,l\in \left[K\right]}P(k,l)=\rho \equiv \max (p_{\mathrm{in}},p_{\mathrm{out}})$. So, we have the separation condition $\frac{|p_{\mathrm{in}}-p_{\mathrm{out}}|}{\sqrt{\max (p_{\mathrm{in}},p_{\mathrm{out}})}}\equiv \sqrt{\rho }{\sigma }_K\left(\tilde{P}\right)$ (also known as the relative edge probability gap in [44]) and the alternative separation condition $\frac{|{\alpha }_{\mathrm{in}}-{\alpha }_{\mathrm{out}}|}{\sqrt{\max ({\alpha }_{\mathrm{in}},{\alpha }_{\mathrm{out}})}}\equiv \sqrt{\frac{\rho n}{\log \left(n\right)}}{\sigma }_K\left(\tilde{P}\right)$. Now, we are ready to compare the thresholds of the (alternative) separation condition obtained from different theoretical results.

• (a) By Corollary 1, we know that ${\sigma }_K\left(\tilde{P}\right)$ should shrink slower than $\sqrt{\frac{\log \left(n\right)}{\rho n}}$ for consistent estimation. Therefore, the separation condition $\frac{|p_{\mathrm{in}}-p_{\mathrm{out}}|}{\sqrt{\max (p_{\mathrm{in}},p_{\mathrm{out}})}}\equiv \sqrt{\rho }{\sigma }_K\left(\tilde{P}\right)$ should shrink slower than $\sqrt{\frac{\log \left(n\right)}{n}}$ (i.e., Equation (1)), and this threshold is consistent with Corollary 1 of [59] and Equation (17) of [49]. The alternative separation condition $\frac{|{\alpha }_{\mathrm{in}}-{\alpha }_{\mathrm{out}}|}{\sqrt{\max ({\alpha }_{\mathrm{in}},{\alpha }_{\mathrm{out}})}}\equiv \sqrt{\frac{\rho n}{\log \left(n\right)}}{\sigma }_K\left(\tilde{P}\right)$ should shrink slower than 1 (i.e., Equation (2)).
• (b) Undoubtedly, the (alternative) separation condition in (a) is consistent with that of [41], since Theorem 2.2 [41] shares the same error rate $O\left(\frac{1}{{\sigma }_K\left(\tilde{P}\right)}\sqrt{\frac{\log \left(n\right)}{\rho n}}\right)$ for $MMSB(n,K,\Pi ,p_{\mathrm{in}},p_{\mathrm{out}})$ with $K=O\left(1\right)$.
• (c) By Remark 5, using $\parallel A_{\mathrm{re}}-\Omega \parallel \le C\sqrt{\rho n}$, we know that in Ref. [44], Equation (3) is $O\left(\frac{{\log }^{\xi }\left(n\right)}{{\sigma }_K\left(\tilde{P}\right)\sqrt{\rho n}}\right)$, so $\sqrt{\rho }{\sigma }_K\left(\tilde{P}\right)$ should shrink slower than $\frac{{\log }^{\xi }\left(n\right)}{\sqrt{n}}$. Thus, for [44], the separation condition is $\frac{{\log }^{\xi }\left(n\right)}{\sqrt{n}}$, and the alternative separation condition is ${\log }^{\xi -0.5}\left(n\right)$, which are sub-optimal compared with ours in (a). Using $\parallel A-\Omega \parallel \le C\sqrt{\rho n\log \left(n\right)}$, and Equation (3) in Ref. [44], which is $O\left(\frac{{\log }^{\xi +0.5}\left(n\right)}{{\sigma }_K\left(\tilde{P}\right)\sqrt{\rho n}}\right)$, we see that for [44], now the separation condition is $\frac{{\log }^{\xi +0.5}\left(n\right)}{\sqrt{n}}$ and the alternative separation condition is ${\log }^{\xi }\left(n\right)$.
• (d) For comparison, the error bound of Corollary 3.2 [30] built under SBM for community detection is $O\left(\frac{1}{{\sigma }_K^2\left(\tilde{P}\right)\rho n}\right)$ for $SBM(n,K,p_{\mathrm{in}},p_{\mathrm{out}})$ with $K=O\left(1\right)$, so $\sqrt{\rho }{\sigma }_K\left(\tilde{P}\right)$ should shrink slower than $\frac{1}{\sqrt{n}}$. Thus the separation condition for [30] is $\frac{1}{\sqrt{n}}$. However, as we analyzed in the first bullet given after Lemma 2, [30] applied $\parallel A_{re}-\Omega \parallel \le C\sqrt{\rho n}$ to build their consistency results. Instead, we apply $\parallel A-\Omega \parallel \le C\sqrt{\rho n\log \left(n\right)}$ to the built theoretical results of [30], and the error bound of Corollary 3.2 [30] is $O\left(\frac{\log \left(n\right)}{{\sigma }_K^2\left(\tilde{P}\right)\rho n}\right)$, which returns the same separation condition as our Corollary 1 and Theorem 2.2 of [41] now. Following a similar analysis to (a)–(c), we can obtain an alternative separation condition for [30] immediately, and the results are provided in Table 2. Meanwhile, as analyzed in the first bullet given after Lemma 2, whether using $\parallel A-\Omega \parallel \le C\sqrt{\rho n\log \left(n\right)}$ or $\parallel A_{\mathrm{re}}-\Omega \parallel \le C\sqrt{\rho n}$ does not change our error rates. By carefully analyzing the proof of 2.1 of [41], we see that whether using $\parallel A-\Omega \parallel \le C\sqrt{\rho n\log \left(n\right)}$ or $\parallel A_{\mathrm{re}}-\Omega \parallel \le C\sqrt{\rho n}$ also does not change their row-wise large deviation, hence it does not influence their upper bound of the error rate for their Mixed-SCORE.

Sharp threshold. Consider the Erdös–Rényi (ER) random graph $G(n,p)$ [61]. To construct the ER random graph $G(n,p)$, let $K=1$ and $\Pi$ be an $n\times 1$ vector with all entries being ones. Since $K=1$ and the maximum entry of $\tilde{P}$ is assumed to be 1, we have $\tilde{P}=1$ in $G(n,p)$ and hence ${\sigma }_K\left(\tilde{P}\right)=1$. Then we have $\Omega =\Pi \rho \tilde{P}{\Pi }^{\prime }=\Pi \rho {\Pi }^{\prime }=\Pi p{\Pi }^{\prime }$, i.e, $p=\rho$. Since the error rate is $O\left(\frac{1}{{\sigma }_K\left(\tilde{P}\right)}\sqrt{\frac{\log \left(n\right)}{\rho n}}\right)=O\left(\sqrt{\frac{\log \left(n\right)}{pn}}\right)$, for consistent estimation, we see that p should shrink slower than $\frac{\log \left(n\right)}{n}$ (i.e., Equation (3)), which is just the sharp threshold in [61], Theorem 4.6 [62], strongly consistent with [72], and the first bullet in Section 2.5 [53] (called the lower bound requirement of p for the ER random graph to enjoy consistent estimation as the sharp threshold). Since the sharp threshold is obtained when $K=1$, which means a connected ER random graph $G(n,p)$, this is also consistent with the connectivity in Table 2 of [21]. Meanwhile, since our Assumption (A1) requires $\rho n\ge \log \left(n\right)$, it gives that p should shrink slower than $\frac{\log \left(n\right)}{n}$ since $p=\rho$ under $G(n,p)$, which is consistent with the sharp threshold. Since Theorem 2.2 of Ref. [41] enjoys the same error rate as ours under the settings in Corollary 1, [41] also reaches the sharp threshold as $\frac{\log \left(n\right)}{n}$. Furthermore, Remark 5 says that the bound for the error rate in Equation (3) [44] should be $O\left(\frac{{\log }^{\xi }\left(n\right)}{{\sigma }_K\left(\tilde{P}\right)\sqrt{\rho n}}\right)$ when using $\parallel A_{\mathrm{re}}-\Omega \parallel \le C\sqrt{\rho n}$; following a similar analysis, we see that the sharp threshold for [44] is $\frac{{\log }^{2\xi }\left(n\right)}{n}$, which is sub-optimal compared with ours. When using $\parallel A-\Omega \parallel \le C\sqrt{\rho n\log \left(n\right)}$, the sharp threshold for [44] is $\frac{{\log }^{2\xi +1}\left(n\right)}{n}$. Similarly, the error bound of Corollary 3.2 [30] is $O\left(\frac{1}{{\sigma }_K^2\left(\tilde{P}\right)\rho n}\right)\equiv O\left(\frac{1}{pn}\right)$ under ER $G(n,p)$ since $p=\rho ,{\sigma }_K\left(\tilde{P}\right)=1$ and $K=1$. Hence, the sharp threshold obtained from the theoretical upper bound for error rates of [30] is $\frac{1}{n}$, which does not match the classical result. Instead, we apply $\parallel A-\Omega \parallel \le C\sqrt{\rho n\log \left(n\right)}$ with a high probability to build the theoretical results of [30], and the error bound of Corollary 3.2 [30] is $O\left(\frac{\log \left(n\right)}{pn}\right)$, which returns the classical sharp threshold $\frac{\log \left(n\right)}{n}$ now.

Table 1 summarizes the comparisons of the separation condition and sharp threshold. Table 2 records the respective alternative separation condition. The delicate analysis given above supports our statement that the separation condition of a standard network (i.e., $SBM(n,K,p_{\mathrm{in}},p_{\mathrm{out}})$ with $K=O\left(1\right)$ or $MMSB(n,K,\Pi ,p_{\mathrm{in}},p_{\mathrm{out}})$ with $K=O\left(1\right)$) and the sharp threshold of ER random graph $G(n,p)$ can be seen as unified criteria to compare the theoretical results of spectral methods under different models. To conclude the above analysis, here, we summarize the main steps to apply the separation condition and sharp threshold criterion (SCSTC for short) to check the consistency of the theoretical results or compare the results of spectral methods under different models, where spectral methods mean methods developed based on the application of the eigenvectors or singular vectors of the adjacency matrix or its variants for community detection. The four-stage SCSTC is given below:

$ste{p}_1$

Check whether the theoretical upper bound of the error rate contains ${\sigma }_K\left(\tilde{P}\right)$ (note that $P=\rho \tilde{P}$ is probability matrix and maximum entries of $\tilde{P}$ should be set as 1), where the separation parameter ${\sigma }_K\left(\tilde{P}\right)$ always appears when considering the lower bound of ${\sigma }_K(\Omega )$. If it contains ${\sigma }_K\left(\tilde{P}\right)$, move to the next step. Otherwise, it suggests possible improvements for the consistency by considering ${\sigma }_K\left(\tilde{P}\right)$ in the proofs.

$ste{p}_2$

Let $K=O\left(1\right)$ and network degenerate to the standard network whose numbers of nodes in each community are in the same order and can been seen as $O\left(\frac{n}{K}\right)$ (i.e., a $SBM(n,K,p_{\mathrm{in}},p_{\mathrm{out}})$ with $K=O\left(1\right)$ in the case of a non-overlapping network or a $MMSB(n,K,\Pi ,p_{\mathrm{in}},p_{\mathrm{out}})$ with $K=O\left(1\right)$ in the case of an overlapping network, and we will mainly focus on $SBM(n,K,p_{\mathrm{in}},p_{\mathrm{out}})$ with $K=O\left(1\right)$ for convenience.). Let the model degenerate to $SBM(n,K,p_{\mathrm{in}},p_{\mathrm{out}})$ with $K=O\left(1\right)$, and then we obtain the new theoretical upper bound of the error rate. Note that if the model does consider degree heterogeneity, sparsity parameter $\rho$ should be considered in the theoretical upper bound of error rate in $ste{p}_1$. If the model considers degree heterogeneity, when it degenerates to $SBM(n,K,p_{\mathrm{in}},p_{\mathrm{out}})$ with $K=O\left(1\right)$, $\rho$ appears at this step. Meanwhile, if $\rho$ is not contained in the error rate of $ste{p}_1$ when the model does not consider degree heterogeneity, it suggests possible improvements by considering $\rho$.

$ste{p}_3$

Let $P=\rho \tilde{P}$ be the probability matrix when the model degenerates to $SBM(n,K,p_{\mathrm{in}},p_{\mathrm{out}})$ such that P has diagonal entries $p_{\mathrm{in}}$ and non-diagonal entries $p_{\mathrm{out}}$. So, ${\sigma }_K\left(P\right)=|p_{\mathrm{in}}-p_{\mathrm{out}}|=\rho {\sigma }_K\left(\tilde{P}\right)$ and separation condition $\frac{|p_{\mathrm{in}}-p_{\mathrm{out}}|}{\sqrt{\max (p_{\mathrm{in}},p_{\mathrm{out}})}}\equiv \sqrt{\rho }{\sigma }_K\left(\tilde{P}\right)$ since the maximum entry of $\tilde{P}$ is assumed to be 1. Compute the lower bound requirement of ${\sigma }_K\left(\tilde{P}\right)$ for consistency estimation through analyzing the new bound obtained in $ste{p}_2$. Compute separation condition $\frac{|p_{\mathrm{in}}-p_{\mathrm{out}}|}{\sqrt{\max (p_{\mathrm{in}},p_{\mathrm{out}})}}\equiv \sqrt{\rho }{\sigma }_K\left(\tilde{P}\right)$ using the lower bound requirement for ${\sigma }_K\left(\tilde{P}\right)$. The sharp threshold for the ER random graph $G(n,p)$ is obtained from the lower bound requirement on $\rho$ for the consistency estimation under the setting that $K=1,{\sigma }_K\left(\tilde{P}\right)=1$ and $p=\rho$.

$ste{p}_4$

Compare the separation condition and the sharp threshold obtained in $ste{p}_3$ with Equations (1) and (3), respectively. If the sharp threshold $\gg \frac{\log \left(n\right)}{n}$ or the separation condition $\gg \sqrt{\frac{\log \left(n\right)}{n}}$, then this leaves improvements on the requirement of the network sparsity or theoretical upper bound of the error rate. If the sharp threshold is $\frac{\log \left(n\right)}{n}$ and the separation condition is $\sqrt{\frac{\log \left(n\right)}{n}}$, the optimality of the theoretical results on both error rates and the requirement of network sparsity is guaranteed. Finally, if the sharp threshold $\ll \frac{\log \left(n\right)}{n}$ or separation condition $\ll \sqrt{\frac{\log \left(n\right)}{n}}$, this suggests that the theoretical result is obtained based on $\parallel A_{\mathrm{re}}-\Omega \parallel$ instead of $\parallel A-\Omega \parallel$.

Remark 7.

This remark provides some explanations on the four steps of SCSTC.

• In $ste{p}_1$, we give a few examples. When applying SCSTC to the main results of [40,48,67], we stop at $ste{p}_1$ as analyzed in Remark 8, suggesting possible improvements by considering ${\sigma }_K\left(\tilde{P}\right)$ for these works. Meanwhile, for the theoretical result without considering ${\sigma }_K\left(\tilde{P}\right)$, we can also move to $ste{p}_2$ to obtain the new theoretical upper bound of the error rate, which is related with ρ and n. Discussions on the theoretical upper bounds of error rates of [50,68] given in Remark 8 are examples of this case.
• In $ste{p}_2$, letting $K=O\left(1\right)$ and the model reduce to $SBM(n,K,p_{\mathrm{in}},p_{\mathrm{out}})$ for the non-overlapping network or $MMSB(n,K,\Pi ,p_{\mathrm{in}},p_{\mathrm{out}})$ for the overlapping network can always simplify the theoretical upper bound of error rate, as shown by our Corollaries 1 and 2. Here, we provide some examples about how to make a model degenerate to SBM. For $MMS{B}_n(K,\tilde{P},\Pi ,\rho )$ in this paper, when all nodes are pure, MMSB degenerates to SBM; for the $DCM{M}_n(K,\tilde{P},\Pi ,\Theta )$ model introduced in Section 5 or DCSBM considered in [30,48,50], setting $\Theta =\sqrt{\rho }I$ makes DCMM and DCSBM degenerates to SBM when all nodes are pure, similar to the ScBM and DCScBM considered in [67,68,71], the OCCAM model of [40], the stochastic blockmodel with the overlap proposed in [46], the extensions of SBM and DCSBM for hypergraph networks considered in [73,74,75], and so forth.
• In $ste{p}_3$ and $ste{p}_4$, the separation condition can be replaced by an alternative separation condition.
• When using SCSTC to build and compare theoretical results for the spectral clustering method, the key point is computing the lower bound for $\frac{|p_{\mathrm{in}}-p_{\mathrm{out}}|}{\sqrt{\max (p_{\mathrm{in}},p_{\mathrm{out}})}}$ when the probability matrix P has diagonal entries $p_{\mathrm{in}}$ and non-diagonal entries $p_{\mathrm{out}}$ from the theoretical upper bound of the error rate for a certain spectral method. If this lower bound is consistent with that of Equation (1), this suggests theoretical optimality, and otherwise it suggests possible improvements by following the four steps of SCSTC.

The above analysis shows that SCSTC can be used to study the consistent estimation of model-based spectral methods. Use SCSTC, the following remark lists a few works whose main theoretical results leave possible improvements.

Remark 8.

The unknown separation condition, or sub-optimal error rates, or a lack of requirement of network sparsity of some previous works, suggest possible improvements of their theoretical results. Here, we list a few works whose main results can be possibly improved until considering the separation condition.

• Theorem 4.4 of [48] proposes the upper bound of the error rate for their regularized spectral clustering (RSC) algorithm, designed based on a regularized Laplacian matrix under DCSBM. However, since [48] does not study the lower bound (in the [48] language) of ${\lambda }_K$ and m, we cannot directly obtain the separation condition from their main theorem. Meanwhile, the main result of [48] does not consider the requirement on the network sparsity, which leaves some improvements. Ref. [48] does not study the theoretical optimal choice for the RSC regularizer τ. After considering ${\sigma }_K\left(\tilde{P}\right)$ and sparsity parameter ρ, one can obtain the theoretical optimal choice for τ, and this is helpful for explaining and choosing the empirical optimal choice for τ. Therefore, the feasible network implementation of SCSTC is obtaining the theoretical optimal choices for some tuning parameters, such as regularizer τ of the RSC algorithm. By using SCSTC, we can find that RSC achieves thresholds in Equations (1)–(3), and we omit proofs for it in this paper.
• Refs. [26,49] study two algorithms designed based on the Laplacian matrix and its regularized version under SBM. They obtain meaningful results, but do not consider the network sparsity parameter ρ and separation parameter ${\sigma }_K\left(\tilde{P}\right)$. After obtaining improved error bounds which are consistent with separation condition $\sqrt{\frac{\log \left(n\right)}{n}}$ using SCSTC, one can also obtain the theoretical optimal choice for regularizer τ of the RSC-τ algorithm considered in [49] and find that the two algorithms considered in [26,49] achieve thresholds in Equations (1)–(3).
• Theorem 2.2 of [50] provides an upper bound of their SCORE algorithm under DCSBM. However, since they do not consider the influence of ${\sigma }_K\left(\tilde{P}\right)$, we cannot directly obtain the separation condition from their main result. Meanwhile, by setting their $\Theta =\sqrt{\rho }I$, DCSBM degenerates to SBM, which gives that their $er{r}_n=\frac{1}{{\rho }^{2}n}(1+\frac{\log \left(n\right)}{\rho n})=O\left(\frac{1}{{\rho }^{2}n}\right)$ by their assumption Equation (2.9). Hence, when $\Theta =\sqrt{\rho }I$, the upper bound of Theorem 2.2 in [50] is $O\left(\frac{{\log }^3\left(n\right)}{{\rho }^{2}n}\right)$. The upper bound of error rate in Corollary 3.2 of [30] is $O\left(\frac{\log \left(n\right)}{\rho n}\right)$ when using $\parallel A-\Omega \parallel \le C\sqrt{\rho n\log \left(n\right)}$ under the setting that $\kappa (\Pi )=O\left(1\right),K=O\left(1\right)$ and ${\sigma }_K\left(\tilde{P}\right)=O\left(1\right)$. We see that $\frac{\log {}^3\left(n\right)}{{\rho }^{2}n}$ grows faster than $\frac{\log \left(n\right)}{\rho n}$, which suggests that there is space to improve the main result of [50] in the aspects of the separation condition and error rates. Furthermore, using SCSTC, we can find that SCORE achieves thresholds in Equations (1)–(3) because its extension mixed-SCORE [41] achieves thresholds in Equations (1)–(3).
• Ref. [67] proposes two models, ScBM and DCScBM, to model the directed networks and an algorithm DI-SIM based on the directed regularized Laplacian matrix to fit DCScBM. However, similar to [48], their main theoretical result in their Theorem C.1 does not consider the lower bound of (in the language of Ref. [67]) ${\sigma }_K,m_y,m_z$ and ${\gamma }_z$, which causes that we cannot obtain the separation condition when DCScBM degenerates to SBM. Meanwhile, their Theorem C.1 also lacks a lower bound requirement on network sparsity. Hence, there is space to improve the theoretical guarantees of [67]. Similar to [48,49], we can also obtain the theoretical optimal choices for regularizer τ of the DI-SIM algorithm and prove that DI-SIM achieves the thresholds in Equations (1)–(3) since it is the directed version of RSC [48].
• Ref. [68] mainly studies the theoretical guarantee for the D-SCORE algorithm proposed by [14] to fit a special case of the DCScBM model for directed networks. By setting their $\theta \left(i\right)=\sqrt{\rho },\delta \left(j\right)=\sqrt{\rho }$ for $i,j\in \left[n\right]$, their directed-DCBM degenerates to SBM. Meanwhile, since their $er{r}_n=\frac{1}{\rho }$, their mis-clustering rate is $O\left(\frac{{\mathrm{T}}_{\mathrm{n}}^2\log \left(\mathrm{n}\right)}{\rho n}\right)$, which matches that of [30] under SBM when setting $T_n$ as a constant. However, if setting $T_n$ as $\log \left(n\right)$, then the error rate is $O\left(\frac{{\log }^3\left(n\right)}{\rho n}\right)$, which is sub-optimal compared with that of [30]. Meanwhile, similar to [50,68], the main result does not consider the influences of K and ${\sigma }_K\left(\tilde{P}\right)$, causing a lack of a separation condition. Hence, the main results of [68] can be improved by considering K, ${\sigma }_K\left(P\right)$, or a more optimal choice of $T_n$ to make their main results comparable with those of [30] when directed-DCBM degenerates to SBM. Using SCSTC, we can find that the D-SCORE also achieves thresholds in Equations (1)–(3) since it is the directed version of SCORE [50].

5. Degree Corrected Mixed Membership Model

Using SCSTC to Theorem 3.2 of [43], as shown in Table 1 and Table 2 results in Theorem 3.2 [43] being sub-optimal. To obtain the improvement theoretical results, we give a formal introduction of the degree corrected mixed membership (DCMM) model proposed in [41] first, then we review the SVM-cone-DCMMSB algorithm of [43] and provide the improvement theoretical results. A DCMM for generating A is as follows.

$$\begin{array}{c}\hfill \Omega :=\Theta \Pi \tilde{P}{\Pi }^{\prime }\Theta A(i,j)\sim \mathrm{Bernoulli}(\Omega (i,j))i,j\in \left[n\right],\end{array}$$

(5)

where $\Theta \in {\mathbb{R}}^{n\times n}$ is a diagonal matrix whose i-th diagonal entry is the degree heterogeneity of node i for $i\in \left[n\right]$. Let $\theta \in {\mathbb{R}}^{n\times 1}$ with $\theta \left(i\right)=\Theta (i,i)$ for $i\in \left[n\right]$. Set ${\theta }_{\max }={\max }_{i\in \left[n\right]}\theta \left(i\right),{\theta }_{\min }={\min }_{i\in \left[n\right]}\theta \left(i\right)$ and ${\tilde{P}}_{\max }={\max }_{k,l\in \left[K\right]}\tilde{P}(k,l),{\tilde{P}}_{\min }={\min }_{k,l\in \left[K\right]}\tilde{P}(k,l)$.

Definition 3.

Call model (5) the degree corrected mixed membership (DCMM) model, and denote it by $DCM{M}_n(K,\tilde{P},\Pi ,\Theta )$.

Note that if we set $\tilde{\Pi }=\Theta \Pi$ and choose $\Theta$ such that $\tilde{\Pi }\in {\{0,1\}}^{n\times K}$, then we have $\Omega =\tilde{\Pi }\tilde{P}{\tilde{\Pi }}^{\prime }$, which means that the stochastic blockmodel with overlap (SBMO) proposed in [46] is just a special case of DCMM. Meanwhile, if we write $\Theta$ as $\Theta =\tilde{\Theta }{D}_o$, where $\tilde{\Theta },D_o$ are two positive diagonal matrices and let ${\Pi }_o=D_o\Pi$, then we can choose $D_0$ such that $\parallel {\Pi }_o{(i,:)\parallel }_F=1$. By $\Omega =\Theta \Pi \tilde{P}{\Pi }^{\prime }\Theta =\tilde{\Theta }{\Pi }_o\tilde{P}{\Pi }_o^{\prime }\tilde{\Theta }$, we see that the OCCAM model proposed in [40] equals the DCMM model. By Equation (1.3) and Proposition 1.1 of [41], the following conditions are sufficient for the identifiability of DCMM when ${\theta }_{\max }{\tilde{P}}_{\max }\le 1$:

• (II1) $\mathrm{rank}\left(\tilde{P}\right)=K$ and $\tilde{P}$ has unit diagonals.
• (II2) There is at least one pure node for each of the K communities.

Note that though the diagonal entries of $\tilde{P}$ are ones, ${\tilde{P}}_{\max }$ may be larger than 1 as long as ${\theta }_{\max }{\tilde{P}}_{\max }\le 1$ under DCMM, and this is slightly different from the setting that ${\max }_{k,l\in \left[K\right]}\tilde{P}(k,l)=1$ under MMSB.

Without causing confusion, under $DCM{M}_n(K,\tilde{P},\Pi ,\Theta )$, we still let $\Omega =U\Lambda U^{\prime }$ be the top-K eigen-decomposition of $\Omega$ such that $U\in {\mathbb{R}}^{n\times K},\Lambda \in {\mathbb{R}}^{K\times K}$ and $U^{\prime }U=I_K$. Set $U_{*}\in {\mathbb{R}}^{n\times K}$ by $U_{*}(i,:)=\frac{U(i,:)}{{\parallel U(i,:)\parallel }_F}$ and let $N_U\in {\mathbb{R}}^{n\times n}$ be a diagonal matrix such that $N_U(i,i)=\frac{1}{{\parallel U(i,:)\parallel }_F}$ for $i\in \left[n\right]$. Then $U_{*}$ can be rewritten as $U_{*}=N_{U}U$. The existence of the ideal cone (IC for short) structure inherent in $U_{*}$ mentioned in [43] is guaranteed by the following lemma.

Lemma 3.

Under $DCM{M}_n(K,\tilde{P},\Pi ,\Theta )$, $U_{*}=Y{U}_{*}(\mathcal{I},:)$ where $Y=N_M\Pi {\Theta }^{-1}(\mathcal{I},\mathcal{I})N_U^{-1}(\mathcal{I},\mathcal{I})$ with $N_M$ being an $n\times n$ diagonal matrix whose diagonal entries are positive.

Lemma 3 gives $Y=U_{*}{U}_{*}^{-1}(\mathcal{I},:)$. Since $U_{*}=N_{U}U$ and $Y=N_M\Pi {\Theta }^{-1}(\mathcal{I},\mathcal{I})N_U^{-1}(\mathcal{I},\mathcal{I})$, we have

$$\begin{array}{c}\hfill N_U^{-1}{N}_M\Pi =U{U}_{*}^{-1}(\mathcal{I},:)N_U(\mathcal{I},\mathcal{I})\Theta (\mathcal{I},\mathcal{I}).\end{array}$$

(6)

Since $\Omega (\mathcal{I},\mathcal{I})=\Theta (\mathcal{I},\mathcal{I})\Pi (\mathcal{I},:)\tilde{P}{\Pi }^{\prime }(\mathcal{I},:)=\Theta (\mathcal{I},\mathcal{I})\tilde{P}\Theta (\mathcal{I},\mathcal{I})=U(\mathcal{I},:)\Lambda U^{\prime }(\mathcal{I},:)$, we have $\Theta (\mathcal{I},\mathcal{I})\tilde{P}\Theta (\mathcal{I},\mathcal{I})=U(\mathcal{I},:)\Lambda U^{\prime }(\mathcal{I},:)$. Then we have $\Theta (\mathcal{I},\mathcal{I})=\sqrt{\mathrm{diag}(U(\mathcal{I},:)\Lambda U^{\prime }(\mathcal{I},:))}$ when Condition (II1) holds such that $\tilde{P}$ has unit-diagonals. Set $J_{*}=N_U(\mathcal{I},\mathcal{I})\Theta (\mathcal{I},\mathcal{I})\equiv \sqrt{\mathrm{diag}(U_{*}(\mathcal{I},:)\Lambda U_{*}^{\prime }(\mathcal{I},:))},Z_{*}=N_U^{-1}{N}_M\Pi ,Y_{*}=U{U}_{*}^{-1}(\mathcal{I},:)$. By Equation (6), we have

$$\begin{array}{c}\hfill Z_{*}=Y_{*}{J}_{*}\equiv U{U}_{*}^{-1}(\mathcal{I},:)\mathrm{diag}(U_{*}(\mathcal{I},:)\Lambda U_{*}^{\prime }(\mathcal{I},:)).\end{array}$$

(7)

Meanwhile, since $N_U^{-1}{N}_M$ is an $n\times n$ positive diagonal matrix, we have

$$\begin{array}{c}\hfill \Pi (i,:)=\frac{Z_{*}(i,:)}{\parallel Z_{*}{(i,:)\parallel }_1},i\in \left[n\right].\end{array}$$

(8)

With given $\Omega$ and K, we can obtain $U,U_{*}$ and $\Lambda$. The above analysis shows that once $U_{*}(\mathcal{I},:)$ is known, we can exactly recover $\Pi$ by Equations (7) and (8). From Lemma 3, we know that $U_{*}=Y{U}_{*}(\mathcal{I},:)$ forms the IC structure. Ref. [43] proposes the SVM-cone algorithm (i.e., Algorithm A2) which can exactly obtain $U_{*}(\mathcal{I},:)$ from the ideal cone $U_{*}=Y{U}_{*}(\mathcal{I},:)$ with inputs $U_{*}$ and K.

Based on the above analysis, we are now ready to give the ideal SVM-cone-DCMMSB algorithm. Input $\Omega ,K$. Output: $\Pi$.

• Let $\Omega =U\Lambda U^{\prime }$ be the top-K eigen-decomposition of $\Omega$ such that $U\in {\mathbb{R}}^{n\times K},\Lambda \in {\mathbb{R}}^{K\times K},U^{\prime }U=I$. Let $U_{*}=N_{U}U$, where $N_U$ is a $n\times n$ diagonal matrix whose i-th diagonal entry is $\frac{1}{{\parallel U(i,:)\parallel }_F}$ for $i\in \left[n\right]$.
• Run SVM-cone algorithm on $U_{*}$ assuming that there are K communities to obtain $\mathcal{I}$.
• Set $J_{*}=\sqrt{\mathrm{diag}(U_{*}(\mathcal{I},:)\Lambda U_{*}^{\prime }(\mathcal{I},:))},Y_{*}=U{U}_{*}^{-1}(\mathcal{I},:),Z_{*}=Y_{*}{J}_{*}$.
• Recover $\Pi$ by setting $\Pi (i,:)=\frac{Z_{*}(i,:)}{\parallel Z_{*}{(i,:)\parallel }_1}$ for $i\in \left[n\right]$.

With given $U_{*}$ and K, since the SVM-cone algorithm returns $U_{*}(\mathcal{I},:)$, the ideal SVM-cone-DCMMSB exactly (for detail, see Appendix B) returns $\Pi$.

Now, we review the SVM-cone-DCMMSB algorithm of [43], where this algorithm can be seen as an extension of SPACL designed under MMSB to fit DCMM. For the real case, use ${\widehat{Y}}_{*},{\widehat{J}}_{*},{\widehat{Z}}_{*},{\widehat{\Pi }}_{*}$ given in Algorithm 2 to estimate $Y_{*},J_{*},Z_{*},\Pi$, respectively.

Algorithm 2 SVM-cone-DCMMSB [43]

Require: The adjacency matrix $A\in {\mathbb{R}}^{n\times n}$ and the number of communities K. Ensure: The estimated $n\times K$ membership matrix ${\widehat{\Pi }}_{*}$. 1: Obtain $\tilde{A}=\widehat{U}\widehat{\Lambda }{\widehat{U}}^{\prime }$, the top K eigen-decomposition of A. Let ${\widehat{U}}_{*}\in {\mathbb{R}}^{n\times K}$ such that ${\widehat{U}}_{*}(i,:)=\frac{\widehat{U}(i,:)}{\parallel \widehat{U}{(i,:)\parallel }_F}$ for $i\in \left[n\right]$. 2: Apply SVM-cone algorithm (i.e., Algorithm A2) on the rows of ${\widehat{U}}_{*}$ assuming there are K communities to obtain ${\widehat{\mathcal{I}}}_{*}$, the index set returned by SVM-cone algorithm. 3: Set ${\widehat{J}}_{*}=\sqrt{\mathrm{diag}\left({\widehat{U}}_{*}({\widehat{I}}_{*},:)\widehat{\Lambda }{\widehat{U}}_{*}^{\prime }({\widehat{\mathcal{I}}}_{*},:)\right)},{\widehat{Y}}_{*}=\widehat{U}{\widehat{U}}_{*}^{-1}({\widehat{\mathcal{I}}}_{*},:),{\widehat{Z}}_{*}={\widehat{Y}}_{*}{\widehat{J}}_{*}$ . Then set ${\widehat{Z}}_{*}=\max (0,{\widehat{Z}}_{*})$. 4: Estimate $\Pi (i,:)$ by ${\widehat{\Pi }}_{*}(i,:)={\widehat{Z}}_{*}(i,:)/{\parallel {\widehat{Z}}_{*}(i,:)\parallel }_1,i\in \left[n\right]$.

Consistency under DCMM

Assume that

(A2)

${\tilde{P}}_{\max }{\theta }_{\max }{\parallel \theta \parallel }_1\ge \log \left(n\right)$.

Since we let ${\tilde{P}}_{\max }\le C$, Assumption (A2) equals ${\theta }_{\max }{\parallel \theta \parallel }_1\ge \log \left(n\right)/C$. The following lemma bounds $\parallel A-\Omega \parallel$ under $DCM{M}_n(K,\tilde{P},\Pi ,\Theta )$ when Assumption (A2) holds.

Lemma 4.

Under $DCM{M}_n(K,\tilde{P},\Pi ,\Theta )$, when Assumption (A2) holds, with probability at least $1-o\left(n^{-\alpha }\right)$, we have

$$\begin{array}{c}\hfill \parallel A-\Omega \parallel \le \frac{\alpha +1+\sqrt{(\alpha +1)(\alpha +19)}}{3}\sqrt{{\tilde{P}}_{\max }{\theta }_{\max }{\parallel \theta \parallel }_1\log \left(n\right)}.\end{array}$$

Remark 9.

Consider a special case when $\Theta =\sqrt{\rho }I$ such that DCMM degenerates to MMSB, since ${\tilde{P}}_{\max }$ is assumed to be 1 under MMSB, Assumption (A2) and the upper bound of $\parallel A-\Omega \parallel$ in Lemma 4 are consistent with that of Lemma 1. When all nodes are pure, DCMM degenerates to DCSBM [45], then the upper bound of $\parallel A-\Omega \parallel$ in Lemma 4 is also consistent with Lemma 2.2 of [50]. Meanwhile, this bound is also consistent with Equation (6.34) in the first version of [41], which also applies the Bernstein inequality to bound $\parallel A-\Omega \parallel$. However, the bound is $C\sqrt{{\theta }_{\max }{\parallel \theta \parallel }_1}$ in Equation (C.53) of the latest version for [41], which applies Corollary 3.12 and Remark 3.13 of [76] to obtain the bound. Though the bound in Equation (C.53) of the latest version for [41] is sharper by a $\sqrt{\log \left(n\right)}$ term, Corollary 3.12 of [76] has constraints on $W(i,j)$ (here, $W=A-\Omega$) such that $W(i,j)$ can be written as $W(i,j)={\xi }_{ij}{b}_{ij}$, where $\{{\xi }_{i,j}:i\ge j\}$ are independent symmetric random variables with unit variance, and $\{b_{i,j}:i\ge j\}$ are given scalars; see the proof of Corollary 3.12 [76] for detail. Therefore, without causing confusion, we also use $A_{\mathrm{re}}$ to denote the constraint A used in [41] such that $\parallel A_{\mathrm{re}}-\Omega \parallel \le C\sqrt{{\theta }_{\max }{\parallel \theta \parallel }_1}$. Furthermore, if we set $\rho \ge {\max }_{i,j}\Omega (i,j)$ such that $\rho \ge {\theta }_{\max }^2$, the bound in Lemma 4 also equals $\parallel A-\Omega \parallel \le C\sqrt{\rho n\log \left(n\right)}$ and the assumption (A2) reads ${\tilde{P}}_{\max }\rho n\ge \log \left(n\right)$. The bound $\parallel A_{\mathrm{re}}-\Omega \parallel \le C\sqrt{{\theta }_{\max }{\parallel \theta \parallel }_1}$ in Equation (C.53) of [41] reads $\parallel A_{\mathrm{re}}-\Omega \left|\right|\le C\sqrt{\rho n}$.

Lemma 5.

(Row-wise eigenspace error) Under $DCM{M}_n(K,\tilde{P},\Pi ,\Theta )$, when Assumption (A2) holds, suppose ${\sigma }_K(\Omega )\ge C{\theta }_{\max }\sqrt{{\tilde{P}}_{\max }n\log \left(n\right)}$, with probability at least $1-o\left(n^{-\alpha }\right)$.

• When we apply Theorem 4.2 of [64], we have

  $$\begin{array}{c}\hfill \parallel \widehat{U}{\widehat{U}}^{\prime }-U{U}^{\prime }{\parallel }_{2\to \infty }=O\left(\frac{{\theta }_{\max }\sqrt{{\tilde{P}}_{\max }K}(\frac{{\theta }_{\max }\kappa (\Omega )}{{\theta }_{\min }}\sqrt{\frac{n}{K{\lambda }_K({\Pi }^{\prime }\Pi )}}+\sqrt{\log \left(n\right)})}{{\theta }_{\min }^2{\sigma }_K\left(\tilde{P}\right){\lambda }_K({\Pi }^{\prime }\Pi )}\right).\end{array}$$
• When we apply Theorem 4.2 of [65], we have

  $$\begin{array}{c}\hfill \parallel \widehat{U}{\widehat{U}}^{\prime }-U{U}^{\prime }{\parallel }_{2\to \infty }=O\left(\frac{{\theta }_{\max }\sqrt{{\tilde{P}}_{\max }{\theta }_{\max }{\parallel \theta \parallel }_1\log \left(n\right)}}{{\theta }_{\min }^3{\sigma }_K\left(\tilde{P}\right){\lambda }_K^{1.5}({\Pi }^{\prime }\Pi )}\right).\end{array}$$

Without causing confusion, we also use $\varpi ,{\varpi }_1,{\varpi }_2$ under DCMM as Lemma 2 for notation convenience.

Remark 10.

When $\Theta =\sqrt{\rho }I$ such that DCMM degenerates to MMSB, bounds in Lemma 5 are consistent with those of Lemma 2.

Remark 11

(Comparison to Theorem I.3 [43]). Note that the ρ in [43] is ${\theta }_{\max }^2$, which gives that the row-wise eigenspace concentration in Theorem I.3 [43] is $O\left(\frac{{\theta }_{\max }\sqrt{Kn}{\parallel U\parallel }_{2\to \infty }{\log }^{\xi }\left(n\right)}{{\sigma }_K(\Omega )}\right)$ when using $\parallel A_{\mathrm{re}}-\Omega \parallel \le C\sqrt{\rho n}$ and this value is at least $O\left(\frac{\sqrt{{\theta }_{\max }{\parallel \theta \parallel }_{1}K}{\parallel U\parallel }_{2\to \infty }{\log }^{\xi }\left(n\right)}{{\sigma }_K(\Omega )}\right)$. Since ${\parallel U\parallel }_{2\to \infty }\le \frac{{\theta }_{\max }}{{\theta }_{\min }\sqrt{{\lambda }_K({\Pi }^{\prime }\Pi )}}$ by Lemma II.1 of [43] and ${\sigma }_K(\Omega )\ge {\theta }_{\min }^2{\sigma }_K\left(\tilde{P}\right){\lambda }_K({\Pi }^{\prime }\Pi )$ by the proof of Lemma 5, we see that the upper bound of Theorem I.3 [43] is $O\left(\frac{{\theta }_{\max }\sqrt{K{\theta }_{\max }{\parallel \theta \parallel }_1}{\log }^{\xi }\left(n\right)}{{\theta }_{\min }^3{\sigma }_K\left(\tilde{P}\right){\lambda }_K^{1.5}({\Pi }^{\prime }\Pi )}\right)$, which is $\sqrt{K}{\log }^{\xi -0.5}\left(n\right)$ (recall that $\xi >1$) times than our ${\varpi }_2$. Again, Theorem I.3 [43] has stronger requirements on the sparsity of ${\theta }_{\max }{\parallel \theta \parallel }_1$ and the lower bound of ${\sigma }_K(\Omega )$ than our Lemma 5. When using the bound of $\parallel A-\Omega \parallel$ in our Lemma 4 to obtain the row-wise eigenspace concentration in Theorem I.3 [43], their upper bound is $\sqrt{K}{\log }^{\xi }\left(n\right)$ times than our ${\varpi }_2$. Similar to the first bullet given after Lemma 2, whether using $\parallel A-\Omega \parallel \le C\sqrt{{\theta }_{\max }{\parallel \theta \parallel }_1\log \left(n\right)}$ or $\parallel A_{\mathrm{re}}-\Omega \parallel \le C\sqrt{{\theta }_{\max }{\parallel \theta \parallel }_1}$ does not change our ϖ under DCMM.

Remark 12

(Comparison to Lemma 2.1 [41]). The fourth bullet of Lemma 2.1 [41] is the row-wise deviation bound for the eigenvectors of the adjacency matrix under some assumptions translated to our $\kappa ({\Pi }^{\prime }\Pi )=O\left(1\right)$, Assumption (A2) and lower bound requirement on ${\sigma }_K(\Omega )$ since they apply Lemma C.2 [41]. The row-wise deviation bound in the fourth bullet of Lemma 2.1 [41] reads $O\left(\frac{{\theta }_{\max }{K}^{1.5}\sqrt{{\theta }_{\max }{\parallel \theta \parallel }_1\log \left(n\right)}}{{\sigma }_K\left(\tilde{P}\right){\parallel \theta \parallel }_F^3}\right)$, where the denominator is ${\sigma }_K\left(\tilde{P}\right){\parallel \theta \parallel }_F^3$ instead of our ${\theta }_{\min }^3{\sigma }_K\left(\tilde{P}\right){\lambda }_K^{1.5}({\Pi }^{\prime }\Pi )$ due to the fact that [41] uses $\frac{{\sigma }_K\left(\tilde{P}\right){\parallel \theta \parallel }_F^2}{K}$ to roughly estimate ${\sigma }_K(\Omega )$ while we apply ${\theta }_{\min }^2{\sigma }_K\left(\tilde{P}\right){\lambda }_K({\Pi }^{\prime }\Pi )$ to strictly control the lower bound of ${\sigma }_K(\Omega )$. Therefore, we see that the row-wise deviation bound in the fourth bullet of Lemma 2.1 [41] is consistent with our bounds in Lemma 5 when $\kappa ({\Pi }^{\prime }\Pi )=O\left(1\right)$, while our row-wise eigenspace errors in Lemma 5 are more applicable than those of [41] since we do not need to add a constraint on ${\Pi }^{\prime }\Pi$ such that $\kappa ({\Pi }^{\prime }\Pi )=O\left(1\right)$. The upper bound of $\parallel A-\Omega \parallel$ of [41] is $C\sqrt{{\theta }_{\max }{\parallel \theta \parallel }_1}$ given in their Equation (C.53) under $DCM{M}_n(K,\tilde{P},\Pi ,\Theta )$, while ours is $C\sqrt{{\theta }_{\max }{\parallel \theta \parallel }_1\log \left(n\right)}$ in Lemma 4, since our bound of the row-wise eigenspace error in Lemma 5 is consistent with the fourth bullet of Lemma 2.1 [41], this supports the statement that the row-wise eigenspace error does not rely on $\parallel A-\Omega \parallel$ given in the first bullet after Lemma 2.

Let ${\pi }_{\min }={\min }_{1\le k\le K}{\mathbf{1}}^{\prime }\Pi e_k$ , where ${\pi }_{\min }$ measures the minimum summation of nodes belonging to a certain community. Increasing ${\pi }_{\min }$ makes the network tend to be more balanced, and vice versa. Meanwhile, the term ${\pi }_{\min }$ appears when we propose a lower bound of $\eta$ defined in Lemma A1 to keep track of the model parameters in our main theorem under $DCM{M}_n(K,\tilde{P},\Pi ,\Theta )$. The next theorem gives the theoretical bounds on estimations of memberships under DCMM.

Theorem 2.

Under $DCM{M}_n(K,\tilde{P},\Pi ,\Theta )$, let $\widehat{\Pi }$ be obtained from Algorithm 2, suppose conditions in Lemma 5 hold, and there exists a permutation matrix ${\mathcal{P}}_{*}\in {\mathbb{R}}^{K\times K}$ such that with probability at least $1-o\left(n^{-\alpha }\right)$, we have

$$\begin{array}{c}\hfill {\max }_{i\in \left[n\right]}{\parallel e_i^{\prime }({\widehat{\Pi }}_{*}-\Pi {\mathcal{P}}_{*})\parallel }_1=O\left(\frac{{\theta }_{\max }^{15}{K}^5\varpi {\kappa }^{4.5}({\Pi }^{\prime }\Pi ){\lambda }_1^{1.5}({\Pi }^{\prime }\Pi )}{{\theta }_{\min }^{15}{\pi }_{\min }}\right).\end{array}$$

For comparison, Table 4 summarizes the necessary conditions and dependence on model parameters of rates for Theorem 2 and Theorem 3.2 [43], where the dependence on K and $\log \left(n\right)$ are analyzed in Remark 13 given below.

Remark 13.

(Comparison to Theorem 3.2 [43]) Our bound in Theorem 2 is written as combinations of model parameters and Π can follow any distribution as long as Condition (II2) holds, where such model parameters’ related form of estimation bound is convenient for further theoretical analysis (see Corollary 2), while the bound in Theorem 3.2 [43] is built when Π follows a Dirichlet distribution and $\kappa ({\Pi }^{\prime }{\Theta }^2\Pi )=O\left(1\right)$. Meanwhile, since Theorem 3.2 [43] applies Theorem I.3 [43] to obtain the row-wise eigenspace error, the bound in Theorem 3.2 [43] should multiply ${\log }^{\xi }\left(n\right)$ by Remark 11, and this is also supported by the fact that in the proof of Theorem 3.1 [43], when computing bound of ${ϵ}_0$ (in the language in Ref. [43]) [43] ignores the ${\log }^{\xi }\left(n\right)$ term.

Consider a special case by setting ${\lambda }_K({\Pi }^{\prime }\Pi )=O\left(\frac{n}{K}\right),{\pi }_{\min }=O\left(\frac{n}{K}\right)$ and $\frac{{\theta }_{\max }}{{\theta }_{\min }}=O\left(1\right)$ with ${\theta }_{\max }=\sqrt{\rho }$, where such case matches the setting $\kappa ({\Pi }^{\prime }{\Theta }^2\Pi )=O\left(1\right)$ in Theorem 3.2 [43]. Now we focus on analyzing the powers of K in our Theorem 2 and Theorem 3.2 [43]. Under this case, the power of K in the estimation bound of Theorem 2 is 6 by basic algebra; since $\min (K^2,{\kappa }^2(\Omega ))=\min (K^2,O\left(1\right))=O\left(1\right),\frac{1}{{\lambda }_K^2({\Pi }^{\prime }{\Theta }^2\Pi )}=O\left(\frac{K^2}{{\rho }^2{n}^2}\right)$, $\frac{1}{\eta }=O\left(K\right)$ by Lemma A1 where η in Lemma A1 follows the same definition as that of Theorem 3.2 [43], and the bound in Theorem 3.2 [43] should multiply $\sqrt{K}$ because (in the language of Ref. [43]) $\parallel {\left({\widehat{Y}}_C{\widehat{Y}}_C^{\prime }\right)}^{-1}{\parallel }_F$ should be no larger than $\frac{\sqrt{K}}{{\lambda }_K\left({\widehat{Y}}_C{\widehat{Y}}_C^{\prime }\right)}$ instead of $\frac{1}{{\lambda }_K\left({\widehat{Y}}_C{\widehat{Y}}_C^{\prime }\right)}$ in the proof of Theorem 2.8 [43], the power of K is 6 by checking the bound of Theorem 3.2 [43]. Meanwhile, note that our bound in Theorem 2 is $l_1$ bound, while the bound in Theorem 3.2 [43] is $l_2$ bound, and when we translate the $l_2$ bound of Theorem 3.2 [43] into $l_1$ bound, the power of K is 6.5 for Theorem 3.2 [43], suggesting that our bound in Theorem 2 has less dependence on K than that of Theorem 3.2 [43].

The following corollary is obtained by adding some conditions on the model parameters.

Corollary 2.

Under $DCM{M}_n(K,\tilde{P},\Pi ,\Theta )$, when conditions of Lemma 5 hold, suppose ${\lambda }_K({\Pi }^{\prime }\Pi )=O\left(\frac{n}{K}\right),{\pi }_{\min }=O\left(\frac{n}{K}\right)$ and $K=O\left(1\right)$, with probability at least $1-o\left(n^{-\alpha }\right)$, we have

$$\begin{array}{c}\hfill {\max }_{i\in \left[n\right]}{\parallel e_i^{\prime }({\widehat{\Pi }}_{*}-\Pi {\mathcal{P}}_{*})\parallel }_1=O\left(\frac{{\theta }_{\max }^{16}\sqrt{{\tilde{P}}_{\max }{\theta }_{\max }{\parallel \theta \parallel }_1\log \left(n\right)}}{{\theta }_{\min }^{18}{\sigma }_K\left(\tilde{P}\right)n}\right).\end{array}$$

Meanwhile, when ${\theta }_{\max }=O\left(\sqrt{\rho }\right),{\theta }_{\min }=O\left(\sqrt{\rho }\right)$ (i.e., $\frac{{\theta }_{\min }}{{\theta }_{\max }}=O\left(1\right)$), we have

$$\begin{array}{c}\hfill {\max }_{i\in \left[n\right]}{\parallel e_i^{\prime }({\widehat{\Pi }}_{*}-\Pi {\mathcal{P}}_{*})\parallel }_1=O\left(\frac{1}{{\sigma }_K\left(\tilde{P}\right)}\sqrt{\frac{{\tilde{P}}_{\max }\log \left(n\right)}{\rho n}}\right).\end{array}$$

Remark 14.

When ${\lambda }_K({\Pi }^{\prime }\Pi )=O\left(\frac{n}{K}\right),K=O\left(1\right),{\theta }_{\max }=O\left(\sqrt{\rho }\right)$ and ${\theta }_{\min }=O\left(\sqrt{\rho }\right)$, the requirement ${\sigma }_K(\Omega )\ge C{\theta }_{\max }\sqrt{{\tilde{P}}_{\max }n\log \left(n\right)}$ in Lemma 5 holds naturally. By the proof of Lemma 5, ${\sigma }_K(\Omega )$ has a lower bound ${\theta }_{\min }^2{\sigma }_K\left(\tilde{P}\right){\lambda }_K({\Pi }^{\prime }\Pi )=O\left({\theta }_{\min }^2{\sigma }_K\left(P\right)n\right)$. To make the requirement ${\sigma }_K(\Omega )\ge C{\theta }_{\max }\sqrt{{\tilde{P}}_{\max }n\log \left(n\right)}$ always hold, we just need ${\theta }_{\min }^2{\sigma }_K\left(\tilde{P}\right)n\ge C{\theta }_{\max }\sqrt{{\tilde{P}}_{\max }n\log \left(n\right)}$, and it gives ${\sigma }_K\left(\tilde{P}\right)\ge C\sqrt{\frac{{\tilde{P}}_{\max }\log \left(n\right)}{\rho n}}$, which matches the requirement of consistent estimation in Corollary 2.

Using SCSTC to Corollary 2, let $\Theta =\sqrt{\rho }I$ such that DCMM degenerates to MMSB, and it is easy to see that the bound in Lemma 2 is consistent with that of Lemma 1. Therefore, the separation condition, alternative separation condition and sharp threshold obtained from Corollary 2 for the extended version of SPACL under DCMM are consistent with classical results, as shown in Table 1 and Table 2 (detailed analysis will be provided in next paragraph). Meanwhile, when ${\theta }_{\max }=O\left(\sqrt{\rho }\right),{\theta }_{\min }=O\left(\sqrt{\rho }\right)$ and settings in Corollary 2 hold, the bound in Theorem 2.2 [41] is of order $\frac{1}{{\sigma }_K\left(\tilde{P}\right)}\sqrt{\frac{\log \left(n\right)}{\rho n}}$, which is consistent with our bound in Corollary 2.

Consider a mixed membership network under the settings of Corollary 2 when $\Theta =\sqrt{\rho }I$ such that DCMM degenerates to MMSB. By Corollary 2, $\frac{{\sigma }_K\left(\tilde{P}\right)}{\sqrt{{\tilde{P}}_{\max }}}$ should shrink slower than $\sqrt{\frac{\log \left(n\right)}{\rho n}}$. We further assume that $\tilde{P}=(2-\beta )I_K+(\beta -1)\mathbf{1}{\mathbf{1}}^{\prime }$ for $\beta \in [1,2)\cup (2,\infty )$; we see that this $\tilde{P}$ with unit diagonals and $\beta -1$ as non-diagonal entries still satisfies Condition (II1). Meanwhile, ${\sigma }_K\left(\tilde{P}\right)=|\beta -2|={\tilde{P}}_{\max }-{\tilde{P}}_{\min }$ and ${\tilde{P}}_{\max }=\max (1,\beta -1)$, so $\frac{{\sigma }_K\left(\tilde{P}\right)}{\sqrt{{\tilde{P}}_{\max }}}=\frac{|\beta -2|}{\sqrt{\max (1,\beta -1)}}$ should shrink slower than $\sqrt{\frac{\log \left(n\right)}{\rho n}}$. Setting $P=\rho P$ as the probability matrix for such $\tilde{P}$, we have $p_{\mathrm{out}}=\rho (\beta -1),p_{\mathrm{in}}=\rho$, and $\max (p_{\mathrm{in}},p_{\mathrm{out}})=\rho \max (1,\beta -1)$. Sure, the separation condition $\frac{|p_{\mathrm{in}}-p_{\mathrm{out}}|}{\sqrt{\max (p_{\mathrm{in}},p_{\mathrm{out}})}}\equiv \frac{\sqrt{\rho }|\beta -2|}{\sqrt{\max (1,\beta -1)}}$ should shrink slower than $\sqrt{\frac{\log \left(n\right)}{n}}$, which satisfies Equation (1). For an alternative separation condition and sharp threshold, just follow a similar analysis as that of MMSB, and we obtain the results in Table 1 and Table 2.

6. Numerical Results

In this section, we present the experimental results for an overlapping network by plotting the phase transition behaviors for both SPACL and SVM-cone-DCMMSB to show that the two methods achieve the threshold in Equation (2) under $MMSB(n,K,\Pi ,p_{\mathrm{in}},p_{\mathrm{out}})$ when $K=2$ and $K=3$. We also use some experiments to show that the spectral methods studied in [26,30,48,50] achieve the threshold in Equation (2) under $SBM(n,K,p_{\mathrm{in}},p_{\mathrm{out}})$ when $K=2$ and $K=3$ for the non-overlapping network. To measure the performance of different algorithms, we use the error rate defined below:

$$\begin{array}{c}\hfill {\min }_{\mathcal{P}\in \{K\times K\mathrm{permutation}\mathrm{matrix}\}}\frac{1}{n}{\parallel \widehat{\Pi }-\Pi \mathcal{P}\parallel }_1.\end{array}$$

(9)

For all simulations, let $p_{\mathrm{in}}={\alpha }_{\mathrm{in}}\frac{\log \left(n\right)}{n}$ and $p_{\mathrm{out}}={\alpha }_{\mathrm{out}}\frac{\log \left(n\right)}{n}$ be diagonal and non-diagonal entries of P, respectively. Since P is the probability matrix, ${\alpha }_{\mathrm{in}}$ and ${\alpha }_{\mathrm{out}}$ should be located in $(0,\frac{n}{\log \left(n\right)}]$. After setting P and $\Pi$, each simulation experiment has the following steps:

(a)

Set $\Omega =\Pi P{\Pi }^{\prime }$.

(b)

Let W be an $n\times n$ symmetric matrix such that all diagonal entries of W are 0, and $W(i,j)$ are independent centered Bernoulli with parameters $\Omega (i,j)$. Let $A=\Omega -\mathrm{diag}(\Omega )+W$ be the adjacency matrix of a simulated network with mixed memberships under MMSB (so there are no loops).

(c)

Apply spectral clustering method to A with K communities. Record the error rate.

(d)

Repeat (b)–(c) 50 times, and report the mean of the error rates over the 50 times.

Experiment 1: Set $n=600,K=2$, and $n_0=250$, where $n_0$ is the number of pure nodes in each community. Let all mixed nodes have mixed membership $(1/2,1/2)$. Since ${\alpha }_{\mathrm{in}}$ and ${\alpha }_{\mathrm{out}}$ should be set to less than $\frac{n}{\log \left(n\right)}=\frac{600}{\log \left(600\right)}\approx 93.795$, we let ${\alpha }_{\mathrm{in}}$ and ${\alpha }_{\mathrm{out}}$ be in the range of $\{5,10,15,\dots ,90\}$. For each pair of $({\alpha }_{\mathrm{in}},{\alpha }_{\mathrm{out}})$, we generate P and then run steps (a)–(d) for this experiment. So, this experiment generates an adjacency matrix of network with mixed memberships under $MMSB(n,2,\Pi ,p_{\mathrm{in}},p_{\mathrm{out}})$. The numerical results are displayed in panels (a) and (b) of Figure 1. We can see that our theoretical bounds (red lines) are quite tight, and the threshold regions obtained from the boundaries of light white areas in panels (a) and (b) are close to our theoretical bounds. Meanwhile, both methods perform better when $\frac{|{\alpha }_{\mathrm{in}}-{\alpha }_{\mathrm{out}}|}{\sqrt{\max ({\alpha }_{\mathrm{in}},{\alpha }_{\mathrm{out}})}}$ increases and SVM-cone-DCMMSB outperforms SPACL for this experiment since panel (b) is darker than panel (a). Note that the network generated here is an assortative network when ${\alpha }_{\mathrm{in}}>{\alpha }_{\mathrm{out}}$, and the network is a dis-assortative network when ${\alpha }_{\mathrm{in}}<{\alpha }_{\mathrm{out}}$. So, the results of this experiment support our finding that SPACL and SVM-cone-DCMMSB achieves the threshold in Equation (2) for both assortative and dis-assortative networks.

Figure 1. Phase transition for SPACL and SVM-cone-DCMMSB under MMSB: darker pixels represent lower error rates. The red lines represent $\frac{|{\alpha }_{\mathrm{in}}-{\alpha }_{\mathrm{out}}|}{\sqrt{\max ({\alpha }_{\mathrm{in}},{\alpha }_{\mathrm{out}})}}=1$.

Experiment 2: Set $n=600,K=3$, and $n_0=150$. Let all mixed nodes have mixed membership $(1/3,1/3,1/3)$. Let ${\alpha }_{\mathrm{in}}$ and ${\alpha }_{\mathrm{out}}$ range in $\{5,10,15,\dots ,90\}$. Sure, this experiment is under $MMSB(n,3,\Pi ,p_{\mathrm{in}},p_{\mathrm{out}})$. The numerical results are displayed in panels (c) and (d) of Figure 1. We see that both methods perform poorly in the region between the two red lines, and the analysis of the numerical results for this experiment is similar to that of Experiment 1.

For visualization, we plot two networks generated from $MMSB(n,K,\Pi ,p_{\mathrm{in}},p_{\mathrm{out}})$ when $K=2$ and $K=3$ in Figure 2. We also plot two dis-assortative networks generated from $MMSB(n,K,\Pi ,p_{\mathrm{in}},p_{\mathrm{out}})$ when $K=2$ and $K=3$ in Figure A1 in Appendix A. In Experiments 1 and 2, there exist some mixed nodes for network generated under MMSB. The following two experiments only focus on network under SBM such that all nodes are pure. Meanwhile, we only consider four spectral algorithms studied in [26,30,48,50] for the non-overlapping network. For convenience, we call the spectral clustering method studied in [26] normalized principle component analysis (nPCA), and Algorithm 1 studied in [30] ordinary principle component analysis (oPCA), where nPCA and oPCA are also considered in [50]. Next, we briefly review nPCA, oPCA, RSC and SCORE.

Figure 2. Panel (a): a graph generated from the mixed membership stochastic blockmodel with $n=600$ nodes and 2 communities. Among the 600 nodes, each community has 250 pure nodes. For the 100 mixed nodes, they have mixed membership $(1/2,1/2)$. Panel (b): a graph generated from MMSB with $n=600$ nodes and 3 communities. Among the 600 nodes, each community has 150 pure nodes. For the 150 mixed nodes, they have mixed membership $(1/3,1/3,1/3)$. Nodes in panels (a,b) connect with probability $p_{\mathrm{in}}=60/600$ and $p_{\mathrm{out}}=1/600$, so the two networks in both panels are assortative networks. For panel (a), error rates for SPACL and SVM-cone-DCMMSB are 0.0285 and 0.0175, respectively, where error rate is defined in Equation (9). For panel (b), error rates for SPACL and SVM-cone-DCMMSB are 0.0709 and 0.0436, respectively. For both panels, dots in the same color are pure nodes in the same community and green square nodes are mixed.

The nPCA algorithm is as follows with input $A,K$ and output $\widehat{\Pi }$.

• Obtain the graph Laplacian $L=D^{-1/2}A{D}^{-1/2}$, where D is a diagonal matrix with $D(i,i)={\sum }_{j=1}^{n}A(i,j)$ for $i\in \left[n\right]$.
• Obtain $\widehat{U}\widehat{\Lambda }{\widehat{U}}^{\prime }$, the top K eigen-decomposition of L.
• Apply k-means algorithm to $\widehat{U}$ to obtain $\widehat{\Pi }$.

The oPCA algorithm is as follows with input $A,K$ and output $\widehat{\Pi }$.

• Obtain $\widehat{U}\widehat{\Lambda }{\widehat{U}}^{\prime }$, the top K eigen-decomposition of A.
• Apply k-means algorithm to $\widehat{U}$ to obtain $\widehat{\Pi }$.

The RSC algorithm is as follows with input $A,K$, regularizer $\tau$, and output $\widehat{\Pi }$.

• Obtain the regularized graph Laplacian $L_{\tau }=D_{\tau }^{-1/2}A{D}_{\tau }^{-1/2}$, where $D_{\tau }=D+\tau I$, and the default $\tau$ is the average node degree.
• Obtain $\widehat{U}\widehat{\Lambda }{\widehat{U}}^{\prime }$, the top K eigen-decomposition of $L_{\tau }$. Let ${\widehat{U}}_{*}$ be the row-normalized version of $\widehat{U}$.
• Apply k-means algorithm to ${\widehat{U}}_{*}$ to obtain $\widehat{\Pi }$.

The SCORE algorithm is as follows with input $A,K$, threshold $T_n$ and output $\widehat{\Pi }$.

• Obtain the K (unit-norm) leading eigenvectors of A: ${\widehat{\eta }}_1,{\widehat{\eta }}_2,\dots ,{\widehat{\eta }}_K$.
• Obtain an $n\times (K-1)$ matrix ${\widehat{R}}_{*}$ such that for $i\in \left[n\right],k\in [K-1]$,

  $${\widehat{R}}_{*}(i,k)=\left\{\begin{array}{cc}\widehat{R}(i,k),\hfill & \mathrm{if}|\widehat{R}(i,k)|\le T_n,\hfill \\ T_n,\hfill & \mathrm{if}\widehat{R}(i,k)>T_n,\hfill \\ -T_n,\hfill & \mathrm{if}\widehat{R}(i,k)<-T_n,\hfill \end{array}\right.$$

  where $\widehat{R}(i,k)=\frac{{\widehat{\eta }}_{k+1}\left(i\right)}{{\widehat{\eta }}_1}\left(i\right)$, and the default $T_n$ is $\log \left(n\right)$.
• Apply k-means algorithm to ${\widehat{R}}_{*}$ to obtain $\widehat{\Pi }$.

We now describe Experiments 3 and 4 under $SBM(n,K,p_{\mathrm{in}},p_{\mathrm{out}})$ when $K=2$ and $K=3$.

Experiment 3: Set $n=600,K=2$, and $n_0=300$, i.e., all nodes are pure and each community has 300 nodes. So this experiment generates the adjacency matrix of the network under $SBM(n,2,p_{\mathrm{in}},p_{\mathrm{out}})$. Numerical results are displayed in panels (a)–(d) of Figure 3. We can see that these spectral clustering methods achieve the threshold in Equation (2).

Figure 3. Phase transition for oPCA, nPCA, RSC and SCORE under SBM: darker pixels represent lower error rates. The red lines represent $\frac{|{\alpha }_{\mathrm{in}}-{\alpha }_{\mathrm{out}}|}{\sqrt{\max ({\alpha }_{\mathrm{in}},{\alpha }_{\mathrm{out}})}}=1$.

Experiment 4: Set $n=600,K=3$, and $n_0=200$, i.e., all nodes are pure and each community has 200 nodes. So, this experiment is under $SBM(n,3,p_{\mathrm{in}},p_{\mathrm{out}})$. The numerical results are displayed in panels (e)–(h) of Figure 3. The results show that these methods achieve threshold in Equation (2).

For visualization, we plot two assortative networks generated from $SBM(n,K,p_{\mathrm{in}},p_{\mathrm{out}})$ when $K=2$ and $K=3$ in Figure 4. We also plot two dis-assortative networks generated from $SBM(n,K,p_{\mathrm{in}},p_{\mathrm{out}})$ when $K=2$ and $K=3$ in Figure A2 in Appendix A.

Figure 4. Panel (a): a graph generated from $SBM(600,2,30/600,2/600)$. Panel (b): a graph generated from $SBM(600,3,30/600,2/600)$. So, in panel (a), there are 2 communities and each community has 300 nodes; in panel (b), there are 3 communities and each community has 200 nodes. Networks in panels (a,b) are assortative networks since $p_{\mathrm{in}}>p_{\mathrm{out}}$. For both panels, error rates for oPCA, nPCA, RSC and SCORE are 0. Colors indicate clusters.

7. Conclusions

In this paper, the four-step separation condition and sharp threshold criterion SCSTC is summarized as a unified framework to study the consistencies and compare the theoretical error rates of spectral methods under models that can degenerate to SBM in a community detection area. With an application of this criterion, we find some inconsistent phenomena of a few previous works. In particular, using SCSTC, we find that the original theoretical upper bounds on error rates of the SPACL algorithm under MMSB and its extended version under DCMM are sub-optimal for the error rates and requirements on network sparsity. To find how the inconsistent phenomena occur, we re-establish theoretical upper bounds of error rats for both SPACL and its extended version by using recent techniques on row-wise eigenvector deviation. The resulting error bounds explicitly keep track of seven independent model parameters $(K,\rho ,{\sigma }_K\left(\tilde{P}\right),{\lambda }_K({\Pi }^{\prime }\Pi ),{\lambda }_1({\Pi }^{\prime }\Pi ),{\theta }_{\min },{\theta }_{\max })$, which allow us to have a further delicate analysis. Compared with the original theoretical results, ours have smaller error rates with lesser dependence on K and $\log \left(n\right)$, weaker requirements on the network sparsity and the lower bound of the smallest nonzero singular value of population adjacency matrix under both MMSB and DCMM. For DCMM, we have no constraint on the distribution of the membership matrix as long as it satisfies the identifiability condition. When considering the separation condition of a standard network and the probability to generate a connected Erdös–Rényi (ER) random graph by using SCSTC, our theoretical results match the classical results. Meanwhile, our theoretical results also match those of Theorem 2.2 [41] under mild conditions, and when DCMM degenerates to MMSB, the theoretical results under DCMM are consistent with those under MMSB. Using the SCSTC criterion, we find that the reasons behind the inconsistent phenomena are the sup-optimality of the original theoretical upper bounds on error rates for SPACL as well as its extended version, and the usage of a regularization version of the adjacency matrix when building theoretical results for spectral methods designed to detect nodes labels for a non-mixed network. The processes of finding these inconsistent phenomena, sub-optimality theoretical results on error rates and the formation mechanism of these inconsistent phenomena guarantee the usefulness of the SCSTC criterion. As shown by Remark 8, the theoretical results of some previous works can be improved by applying this criterion. Using SCSTC, we find that spectral methods considered in [26,41,43,44,48,49,50,67,68] achieve thresholds in Equations (1)–(3), and this conclusion is verified by both theoretical analysis and the numerical results in this paper. A limitation of this criterion is that it is only used for studying the consistency of spectral methods for a standard network with a constant number of communities. It would be interesting to develop a more general criterion that can study the consistency of all methods besides spectral methods, and models besides those can degenerate to SBM for a non-standard network with large K. Finally, we hope that the SCSTC criterion developed in this paper can be widely applied to build and compare theoretical results for spectral methods in the community detection area and that the thresholds in Equations (1)–(3) can be seen as benchmark thresholds for spectral methods.   

Appendix C.1. Proof of Lemma 1

Proof. 

We apply Theorem 1.4 (Bernstein inequality) in [70] to bound $\parallel A-\Omega \parallel$, and this theorem is written as shown below.

Theorem A2.

Consider a finite sequence $\left\{X_k\right\}$ of independent, random, self-adjoint matrices with dimension d. Assume that each random matrix satisfies

$$\begin{array}{c}\hfill \mathbb{E}\left[X_k\right]=0,\mathrm{and}{\lambda }_{\max }\left(X_k\right)\le R\mathrm{almost}\mathrm{surely}.\end{array}$$

Then, for all $t\ge 0$,

$$\begin{array}{c}\hfill \mathbb{P}({\lambda }_{\max }\left(\sum _k{X}_k\right)\ge t)\le d·\exp \left(\frac{-t^2/2}{{\sigma }^2+Rt/3}\right),\end{array}$$

where ${\sigma }^2:=\parallel {\sum }_k\mathbb{E}\left[X_k^2\right]\parallel$.

Let $e_i$ be an $n\times 1$ vector, where $e_i\left(i\right)=1$ and 0 elsewhere, for $i\in \left[n\right]$. For convenience, set $W=A-\Omega$. Then we can write W as $W={\sum }_{i=1}^n{\sum }_{j=1}^{n}W(i,j)e_i{e}_j^{\prime }$. Set $W^{(i,j)}$ as the $n\times n$ matrix such that $W^{(i,j)}=W(i,j)(e_i{e}_j^{\prime }+e_j{e}_i^{\prime })$, which gives $W={\sum }_{1\le i<j\le n}{W}^{(i,j)}$ where $\mathbb{E}\left[W^{(i,j)}\right]=0$ and

$$\begin{array}{cc}\hfill \parallel W^{(i,j)}\parallel & = \parallel W(i,j)(e_i{e}_j^{\prime }+e_j{e}_i)\parallel =|W(i,j)|\parallel (e_i{e}_j^{\prime }+e_j{e}_i^{\prime })\parallel =|W(i,j)| =|A(i,j)-\Omega (i,j)|\le 1.\hfill \end{array}$$

For the variance parameter ${\sigma }^2:=\parallel {\sum }_{1\le i<j\le n}\mathbb{E}\left[{\left(W^{(i,j)}\right)}^2\right]\parallel$. We bound $\mathbb{E}\left(W^2(i,j)\right)$ as shown below:

$$\begin{array}{cc}\hfill \mathbb{E}\left(W^2(i,j)\right)& =\mathbb{E}\left({(A(i,j)-\Omega (i,j))}^2\right)=\mathrm{Var}\left(A(i,j)\right)=\Omega (i,j)(1-\Omega (i,j))\le \Omega (i,j)\hfill \\ \hfill & =\rho \Pi (i,:)\tilde{P}{\Pi }^{\prime }(j,:)\le \rho .\hfill \end{array}$$

Next we bound ${\sigma }^2$ as shown below:

$$\begin{array}{cc}\hfill {\sigma }^2& =\parallel \sum _{1\le i<j\le n}\mathbb{E}\left(W^2(i,j)\right)(e_i{e}_j^{\prime }+e_j{e}_i^{\prime })(e_i{e}_j^{\prime }+e_j{e}_i^{\prime })\parallel =\parallel \sum _{1\le i<j\le n}\mathbb{E}\left[W^2(i,j)(e_i{e}_i^{\prime }+e_j{e}_j^{\prime })\right]\parallel \hfill \\ \hfill & \le \underset{1\le i\le n}{\max }\left|\sum _{j=1}^n\mathbb{E}\left(W^2(i,j)\right)\right|\le \underset{1\le i\le n}{\max }\sum _{j=1}^n\rho =\rho n.\hfill \end{array}$$

Set $t=\frac{\alpha +1+\sqrt{(\alpha +1)(\alpha +19)}}{3}\sqrt{\rho n\log \left(n\right)}$ for any $\alpha >0$, combine Theorem A2 with ${\sigma }^2\le \rho n,R=1,d=n$, and we have

$$\begin{array}{cc}\hfill \mathbb{P}(\parallel W\parallel \ge t)& =\mathbb{P}(\parallel \sum _{1\le i<j\le n}{W}^{(i,j)}\parallel \ge t)\le n·\exp \left(\frac{-t^2/2}{{\sigma }^2+Rt/3}\right)\hfill \\ \hfill & \le n·\exp \left(\frac{-(\alpha +1)\log \left(n\right)}{\frac{18}{{(\sqrt{\alpha +1}+\sqrt{\alpha +19})}^2}+\frac{2\sqrt{\alpha +1}}{\sqrt{\alpha +1}+\sqrt{\alpha +19}}\sqrt{\frac{\log \left(n\right)}{\rho n}}}\right)\le \frac{1}{n^{\alpha }},\hfill \end{array}$$

where we use Assumption (A1) such that $\frac{18}{{(\sqrt{\alpha +1}+\sqrt{\alpha +19})}^2}+\frac{2\sqrt{\alpha +1}}{\sqrt{\alpha +1}+\sqrt{\alpha +19}}\sqrt{\frac{\log \left(n\right)}{\rho n}}\le$$\frac{18}{{(\sqrt{\alpha +1}+\sqrt{\alpha +19})}^2}$$+\frac{2\sqrt{\alpha +1}}{\sqrt{\alpha +1}+\sqrt{\alpha +19}}=1$. □

Appendix C.2. Proof of Lemma 2

Proof. 

Let $H={\widehat{U}}^{\prime }U$, and $H=U_H{\Sigma }_H{V}_H^{\prime }$ be the SVD decomposition of $H_{\widehat{U}}$ with $U_H,V_H\in {\mathbb{R}}^{n\times K}$, where $U_H$ and $V_H$ represent respectively the left and right singular matrices of H. Define $\mathrm{sgn}\left(H\right)=U_H{V}_H^{\prime }$. Since $\mathbb{E}\left(A\right(i,j)-\Omega (i,j\left)\right)=0$, $\mathbb{E}\left[{(A(i,j)-\Omega (i,j))}^2\right]\le \rho$ by the proof of Lemma 1, $\frac{1}{\sqrt{\rho n/\left(\mu \log \right(n\left)\right)}}\le O\left(1\right)$ holds by Assumption (A1) where $\mu$ is the incoherence parameter defined as $\mu =\frac{{n\parallel U\parallel }_{2\to \infty }^2}{K}$. By Theorem 4.2 [64], with high probability, we have

$$\begin{array}{c}\hfill \parallel \widehat{U}{\mathrm{sgn}\left(H\right)-U\parallel }_{2\to \infty }\le C\frac{\sqrt{K\rho }(\kappa (\Omega )\sqrt{\mu }+\sqrt{\log \left(n\right)})}{{\sigma }_K(\Omega )},\end{array}$$

provided that $c_1{\sigma }_K(\Omega )\ge \sqrt{\rho n\log \left(n\right)}$ for some sufficiently small constant $c_1$. By Lemma 3.1 of [44], we know that ${\parallel U\parallel }_{2\to \infty }^2\le \frac{1}{{\lambda }_K({\Pi }^{\prime }\Pi )}$, which gives

$$\begin{array}{c}\hfill \parallel \widehat{U}{\mathrm{sgn}\left(H\right)-U\parallel }_{2\to \infty }\le C\frac{\sqrt{K\rho }(\kappa (\Omega )\sqrt{\frac{n}{K{\lambda }_K({\Pi }^{\prime }\Pi )}}+\sqrt{\log \left(n\right)})}{{\sigma }_K(\Omega )}.\end{array}$$

(A1)

Remark A1.

By Theorem 4.2 of [65], when ${\sigma }_K(\Omega )\ge 4{\parallel A-\Omega \parallel }_{\infty }$, we have

$$\begin{array}{c}\hfill \parallel \widehat{U}{\mathrm{sgn}\left(H\right)-U\parallel }_{2\to \infty }\le 14\frac{{\parallel A-\Omega \parallel }_{\infty }}{{\sigma }_K(\Omega )}{\parallel U\parallel }_{2\to \infty }.\end{array}$$

By Lemma 3.1 [44], we have

$$\begin{array}{c}\hfill \parallel \widehat{U}{\mathrm{sgn}\left(H\right)-U\parallel }_{2\to \infty }\le 14\frac{{\parallel A-\Omega \parallel }_{\infty }}{{\sigma }_K(\Omega )}\sqrt{\frac{1}{{\lambda }_K({\Pi }^{\prime }\Pi )}}.\end{array}$$

Unlike Lemma V.1 [44] which bounds ${\parallel A-\Omega \parallel }_{\infty }$ via the Chernoff bound and obtains ${\parallel A-\Omega \parallel }_{\infty }\le C\rho n$ with high probability, we bound ${\parallel A-\Omega \parallel }_{\infty }$ by the Bernstein inequality using a similar idea as Equation (C.67) of [41]. Let $y={(y_1,y_2,\dots ,y_n)}^{\prime }$ be any $n\times 1$ vector; by Equation (C.67) [41], we know that with an application of the Bernstein inequality, for any $t\ge 0$ and $i\in \left[n\right]$, we have

$$\begin{array}{c}\hfill \mathbb{P}\left(\right|\sum _{j=1}^n(A(i,j)-\Omega (i,j))y\left(j\right)|>t)\le 2\exp (-\frac{t^2/2}{{\sum }_{j=1}^n\Omega (i,j)y^2\left(j\right)+\frac{{t\parallel y\parallel }_{\infty }}{3}}).\end{array}$$

By the proof of Lemma 1, we have $\Omega (i,j)\le \rho$. Set $y\left(j\right)$ as 1 or $-1$ such that $\left(A\right(i,j)-\Omega (i,j\left)\right)y\left(j\right)=\left|A\right(i,j)-\Omega (i,j\left)\right|$, we have

$$\begin{array}{c}\hfill {\mathbb{P}(\parallel A-\Omega \parallel }_{\infty }>t)\le 2\exp (-\frac{t^2/2}{\rho n+\frac{t}{3}}).\end{array}$$

Set $t=\frac{\alpha +1+\sqrt{(\alpha +1)(\alpha +19)}}{3}\sqrt{\rho n\log \left(n\right)}$ for any $\alpha >0$, by Assumption (A1), we have

$$\begin{array}{c}\hfill {\mathbb{P}(\parallel A-\Omega \parallel }_{\infty }>t)\le 2\exp (-\frac{t^2/2}{\rho n+\frac{t}{3}})\le n^{-\alpha }.\end{array}$$

Hence, when ${\sigma }_K(\Omega )\ge C_0\sqrt{\rho n\log \left(n\right)}$ where $C_0=4\frac{\alpha +1+\sqrt{(\alpha +1)(\alpha +19)}}{3}$, with probability at least $1-o\left(n^{-\alpha }\right)$,

$$\begin{array}{c}\hfill \parallel \widehat{U}\mathrm{sgn}\left(H_{\widehat{U}}\right){-U\parallel }_{2\to \infty }\le C\frac{\sqrt{\rho n\log \left(n\right)}}{{\sigma }_K(\Omega )}\sqrt{\frac{1}{{\lambda }_K({\Pi }^{\prime }\Pi )}}.\end{array}$$

Note that when ${\lambda }_K({\Pi }^{\prime }\Pi )=O\left(\frac{n}{K}\right)$, the above bound turns to be $C\frac{\sqrt{\rho K\log \left(n\right)}}{{\sigma }_K(\Omega )}$, which is consistent with that of Equation (A1). Also note that this bound $\frac{\sqrt{\rho n\log \left(n\right)}}{{\sigma }_K(\Omega )}\sqrt{\frac{1}{{\lambda }_K({\Pi }^{\prime }\Pi )}}$ is sharper than the $\frac{\rho n}{{\sigma }_K(\Omega )}\sqrt{\frac{1}{{\lambda }_K({\Pi }^{\prime }\Pi )}}$ of Lemma V.1 [44] by Assumption (A1).

Since $\widehat{U}$ and U have orthonormal columns, now we are ready to bound $\parallel \widehat{U}{\widehat{U}}^{\prime }-U{U}^{\prime }{\parallel }_{2\to \infty }$:

$$\begin{array}{cc}\hfill & \parallel \widehat{U}{\widehat{U}}^{\prime }-U{U}^{\prime }{\parallel }_{2\to \infty }={\max }_{i\in \left[n\right]}{\parallel e_i^{\prime }(U{U}^{\prime }-\widehat{U}{\widehat{U}}^{\prime })\parallel }_F\hfill \\ \hfill & ={\max }_{i\in \left[n\right]}{\parallel e_i^{\prime }(U{U}^{\prime }-\widehat{U}\mathrm{sgn}\left(H\right)U^{\prime }+\widehat{U}\mathrm{sgn}\left(H\right)U^{\prime }-\widehat{U}{\widehat{U}}^{\prime })\parallel }_F\hfill \\ \hfill & \le {\max }_{i\in \left[n\right]}\parallel e_i^{\prime }(U-\widehat{U}\mathrm{sgn}\left(H\right))U^{\prime }{\parallel }_F+{\max }_{i\in \left[n\right]}{\parallel e_i^{\prime }\widehat{U}(\mathrm{sgn}\left(H\right)U^{\prime }-{\widehat{U}}^{\prime })\parallel }_F\hfill \\ \hfill & ={\max }_{i\in \left[n\right]}\parallel e_i^{\prime }(U-\widehat{U}\mathrm{sgn}\left(H\right)){\parallel }_F+{\max }_{i\in \left[n\right]}{\parallel \widehat{U}(\mathrm{sgn}\left(H\right)U^{\prime }-{\widehat{U}}^{\prime })e_i\parallel }_F\hfill \\ \hfill & ={\max }_{i\in \left[n\right]}\parallel e_i^{\prime }(U-\widehat{U}\mathrm{sgn}\left(H\right)){\parallel }_F+{\max }_{i\in \left[n\right]}{\parallel (\mathrm{sgn}\left(H\right)U^{\prime }-{\widehat{U}}^{\prime })e_i\parallel }_F\hfill \\ \hfill & ={\max }_{i\in \left[n\right]}\parallel e_i^{\prime }(U-\widehat{U}\mathrm{sgn}\left(H\right)){\parallel }_F+{\max }_{i\in \left[n\right]}{\parallel e_i^{\prime }(U{\left(\mathrm{sgn}\left(H\right)\right)}^{\prime }-\widehat{U})\parallel }_F\hfill \\ \hfill & ={\max }_{i\in \left[n\right]}\parallel e_i^{\prime }(U-\widehat{U}\mathrm{sgn}\left(H\right)){\parallel }_F+{\max }_{i\in \left[n\right]}{\parallel e_i^{\prime }(U-\widehat{U}\mathrm{sgn}\left(H\right))\parallel }_F\hfill \\ \hfill & =2{\max }_{i\in \left[n\right]}\parallel e_i^{\prime }(U-\widehat{U}\mathrm{sgn}\left(H\right)){\parallel }_F=2{\parallel U-\widehat{U}\mathrm{sgn}\left(H\right)\parallel }_{2\to \infty }\hfill \\ \hfill & \le C\frac{\sqrt{K}(\kappa (\Omega )\sqrt{\frac{n}{K{\lambda }_K({\Pi }^{\prime }\Pi )}}+\sqrt{\log \left(n\right)})}{{\sigma }_K\left(\tilde{P}\right)\sqrt{\rho }{\lambda }_K({\Pi }^{\prime }\Pi )},\hfill \end{array}$$

where the last inequality holds since ${\sigma }_K(\Omega )\ge {\sigma }_K\left(\tilde{P}\right)\rho {\lambda }_K({\Pi }^{\prime }\Pi )$ under $MMS{B}_n(K,\tilde{P},\Pi ,\rho )$ by Lemma II.4 [44] This bound is $C\frac{\sqrt{n\log \left(n\right)}}{{\sigma }_K\left(\tilde{P}\right)\sqrt{\rho }{\lambda }_K^{1.5}({\Pi }^{\prime }\Pi )}$ if we use Theorem 4.2 of [65].

Remark A2.

By Theorem 4.5 [77], we have $\parallel \widehat{U}{\widehat{U}}^{\prime }-U{U}^{\prime }{\parallel }_{2\to \infty }\le \sqrt{n}(\parallel \widehat{U}{\parallel }_{2\to \infty }+{\parallel U\parallel }_{2\to \infty })\parallel U-\widehat{U}{\mathrm{sgn}\left(H\right)\parallel }_{2\to \infty }\le \sqrt{n}(\parallel U-\widehat{U}{\mathrm{sgn}\left(H\right)\parallel }_{2\to \infty }+{2\parallel U\parallel }_{2\to \infty })\parallel U-\widehat{U}{\mathrm{sgn}\left(H\right)\parallel }_{2\to \infty }$ $\le \sqrt{n}(\parallel U-\widehat{U}{\mathrm{sgn}\left(H\right)\parallel }_{2\to \infty }+\frac{2}{\sqrt{{\lambda }_K({\Pi }^{\prime }\Pi )}})\parallel U-\widehat{U}{\mathrm{sgn}\left(H\right)\parallel }_{2\to \infty }=O(\frac{2\sqrt{n}}{\sqrt{{\lambda }_K({\Pi }^{\prime }\Pi )}}\parallel U-\widehat{U}\mathrm{sgn}\left(H\right){\parallel }_{2\to \infty })$ . Sure our bound $\parallel \widehat{U}{\widehat{U}}^{\prime }-U{U}^{\prime }{\parallel }_{2\to \infty }\le 2{\parallel U-\widehat{U}\mathrm{sgn}\left(H\right)\parallel }_{2\to \infty }$ enjoys concise form. In particular, when ${\lambda }_K({\Pi }^{\prime }\Pi )=O\left(\frac{n}{K}\right)$ and $K=O\left(1\right)$, the two bounds give that $\parallel \widehat{U}{\widehat{U}}^{\prime }-U{U}^{\prime }{\parallel }_{2\to \infty }=O(\parallel U-\widehat{U}\mathrm{sgn}\left(H\right){\parallel }_{2\to \infty })$, which provides same error bound of the estimated memberships given in Corollary 1.

□

Appendix C.3. Proof of Theorem 1

Proof. 

Follow almost the same proof as Equation (3) of [44]. For $i\in \left[n\right]$, there exists a permutation matrix $\mathcal{P}\in {\mathbb{R}}^{K\times K}$ such that

$$\begin{array}{c}\hfill \parallel e_i^{\prime }(\widehat{Z}-Z\mathcal{P}){\parallel }_F=O\left(\varpi \kappa ({\Pi }^{\prime }\Pi )\sqrt{K{\lambda }_1({\Pi }^{\prime }\Pi )}\right).\end{array}$$

(A2)

Note that the bound in Equation (A2) is $\sqrt{K}$ times the bound in Equation (3) of [44], and this is because in Equation (3) of [44], (in the language of Ref. [44]) $\parallel {\widehat{V}}_p^{-1}\parallel$ denotes the Frobenius norm of ${\widehat{V}}_p^{-1}$ instead of the spectral norm. Since $\parallel {\widehat{V}}_p^{-1}{\parallel }_F\le \frac{\sqrt{K}}{{\sigma }_K\left({\widehat{V}}_p\right)}$, the bound in Equation (3) [44] should multiply $\sqrt{K}$.

Recall that $Z=\Pi ,\Pi (i,:)=\frac{Z(i,:)}{{\parallel Z(i,:)\parallel }_1},\widehat{\Pi }(i,:)=\frac{\widehat{Z}(j,:)}{\parallel \widehat{Z}{(j,:)\parallel }_1}$, for $i\in \left[n\right]$, since

$$\begin{array}{cc}\hfill \parallel e_i^{\prime }(\widehat{\Pi }-\Pi \mathcal{P}){\parallel }_1& =\parallel \frac{e_i^{\prime }\widehat{Z}}{\parallel e_i^{\prime }\widehat{Z}{\parallel }_1}-\frac{e_i^{\prime }Z\mathcal{P}}{\parallel e_i^{\prime }{Z\mathcal{P}\parallel }_1}{\parallel }_1={\parallel \frac{e_i^{\prime }\widehat{Z}\parallel e_i^{\prime }{Z\parallel }_1-e_i^{\prime }Z\mathcal{P}{\parallel e_i^{\prime }\widehat{Z}\parallel }_1}{\parallel e_i^{\prime }\widehat{Z}{\parallel }_1{\parallel e_i^{\prime }Z\parallel }_1}\parallel }_1\hfill \\ & \le \frac{\parallel e_i^{\prime }\widehat{Z}\parallel e_i^{\prime }{Z\parallel }_1-e_i^{\prime }\widehat{Z}\parallel e_i^{\prime }\widehat{Z}{\parallel }_1{\parallel }_1+\parallel e_i^{\prime }\widehat{Z}\parallel e_i^{\prime }\widehat{Z}{\parallel }_1-e_i^{\prime }Z\mathcal{P}\parallel e_i^{\prime }\widehat{Z}{{\parallel }_1\parallel }_1}{\parallel e_i^{\prime }\widehat{Z}{\parallel }_1{\parallel e_i^{\prime }Z\parallel }_1}\hfill \\ & =\frac{|\parallel e_i^{\prime }{Z\parallel }_1-\parallel e_i^{\prime }\widehat{Z}{\parallel }_1|+\parallel e_i^{\prime }\widehat{Z}-e_i^{\prime }{Z\mathcal{P}\parallel }_1}{\parallel e_i^{\prime }{Z\parallel }_1}\le \frac{2\parallel e_i^{\prime }(\widehat{Z}-Z\mathcal{P}){\parallel }_1}{\parallel e_i^{\prime }{Z\parallel }_1}\hfill \\ \hfill & =\frac{2\parallel e_i^{\prime }(\widehat{Z}-Z\mathcal{P}){\parallel }_1}{\parallel e_i^{\prime }{\Pi \parallel }_1}=2\parallel e_i^{\prime }(\widehat{Z}-Z\mathcal{P}){\parallel }_1\le 2\sqrt{K}{\parallel e_i^{\prime }(\widehat{Z}-Z\mathcal{P})\parallel }_F\hfill \\ \hfill & =O\left(\varpi K\kappa ({\Pi }^{\prime }\Pi )\sqrt{{\lambda }_1({\Pi }^{\prime }\Pi )}\right).\hfill \end{array}$$

□

Appendix C.4. Proof of Corollary 1

Proof. 

Under the conditions of Corollary 1, we have

$$\begin{array}{c}\hfill {\max }_{i\in \left[n\right]}{\parallel e_i^{\prime }(\widehat{\Pi }-\Pi \mathcal{P})\parallel }_1=O\left(\varpi \sqrt{n}\right).\end{array}$$

Under the conditions of Corollary 1, Lemma 2 gives $\varpi =O\left(\frac{1}{{\sigma }_K\left(\tilde{P}\right)}\frac{1}{\sqrt{n}}\sqrt{\frac{\log \left(n\right)}{\rho n}}\right)$, which gives that

$$\begin{array}{c}\hfill {\max }_{i\in \left[n\right]}{\parallel e_i^{\prime }(\widehat{\Pi }-\Pi \mathcal{P})\parallel }_1=O\left(\frac{1}{{\sigma }_K\left(\tilde{P}\right)}\sqrt{\frac{\log \left(n\right)}{\rho n}}\right).\end{array}$$

□

Appendix D.1. Proof of Lemma 3

Proof. 

Since $\Omega =U\Lambda U^{\prime }$, we have $U=\Omega U{\Lambda }^{-1}$ since $U^{\prime }U=I_K$. Recall that $\Omega =\Theta \Pi \tilde{P}{\Pi }^{\prime }\Theta$, we have $U=\Theta \Pi \tilde{P}{\Pi }^{\prime }\Theta U{\Lambda }^{-1}=\Theta \Pi B$, where we set $B=\tilde{P}{\Pi }^{\prime }\Theta U{\Lambda }^{-1}$ for convenience. Since $U(\mathcal{I},:)=\Theta (\mathcal{I},\mathcal{I})\Pi (\mathcal{I},:)B=\Theta (\mathcal{I},\mathcal{I})B$, we have $B={\Theta }^{-1}(\mathcal{I},\mathcal{I})U(\mathcal{I},:)$.

Set $M=\Pi B$. Then we have $U=\Theta M$, which gives that $U(i,:)=e_i^{\prime }U=\Theta (i,i)M(i,:)$ for $i\in \left[n\right]$. Therefore, $U_{*}(i,:)=\frac{U(i,:)}{{\parallel U(i,:)\parallel }_F}=\frac{M(i,:)}{{\parallel M(i,:)\parallel }_F}$, and combined with the fact that $B={\Theta }^{-1}(\mathcal{I},\mathcal{I})U(\mathcal{I},:)\equiv {\Theta }^{-1}(\mathcal{I},\mathcal{I})N_U^{-1}(\mathcal{I},\mathcal{I})N_U(\mathcal{I},\mathcal{I})U(\mathcal{I},:)\equiv {\Theta }^{-1}(\mathcal{I},\mathcal{I})N_U^{-1}(\mathcal{I},\mathcal{I})U_{*}(\mathcal{I},:)$, we have

$$\begin{array}{c}\hfill U_{*}=\left[\begin{array}{c}\Pi (1,:)/{\parallel M(1,:)\parallel }_F\\ \Pi (2,:)/{\parallel M(2,:)\parallel }_F\\ ⋮\\ \Pi (n,:)/{\parallel M(n,:)\parallel }_F\end{array}\right]B=\left[\begin{array}{c}\Pi (1,:)/{\parallel M(1,:)\parallel }_F\\ \Pi (2,:)/{\parallel M(2,:)\parallel }_F\\ ⋮\\ \Pi (n,:)/{\parallel M(n,:)\parallel }_F\end{array}\right]{\Theta }^{-1}(\mathcal{I},\mathcal{I})N_U^{-1}(\mathcal{I},\mathcal{I})U_{*}(\mathcal{I},:).\end{array}$$

Therefore, we have

$$\begin{array}{c}\hfill Y=N_M\Pi {\Theta }^{-1}(\mathcal{I},\mathcal{I})N_U^{-1}(\mathcal{I},\mathcal{I}),\end{array}$$

where $N_M$ is a diagonal matrix whose i-th diagonal entry is $\frac{1}{{\parallel M(i,:)\parallel }_F}$ for $i\in \left[n\right]$. □

Appendix D.2. Proof of Lemma 4

Proof. 

Similar to the proof of Lemma 1, set $W=A-\Omega$ and $W^{(i,j)}=W(i,j)(e_i{e}_j^{\prime }+e_j{e}_i^{\prime })$, we have $W={\sum }_{1\le i<j\le n}{W}^{(i,j)}$, $\mathbb{E}\left[W^{(i,j)}\right]=0$ and $\parallel W^{(i,j)}\parallel \le 1$. Since

$$\begin{array}{cc}\hfill \mathbb{E}\left(W^2(i,j)\right)& =\mathbb{E}\left({(A(i,j)-\Omega (i,j))}^2\right)=\mathrm{Var}\left(A(i,j)\right)=\Omega (i,j)(1-\Omega (i,j))\hfill \\ \hfill & \le \Omega (i,j)=\theta \left(i\right)\theta \left(j\right)\Pi (i,:)\tilde{P}{\Pi }^{\prime }(j,:)\le \theta \left(i\right)\theta \left(j\right){\tilde{P}}_{\max },\hfill \end{array}$$

we have

$$\begin{array}{cc}\hfill {\sigma }^2& =\parallel \sum _{1\le i<j\le n}\mathbb{E}\left(W^2(i,j)\right)(e_i{e}_j^{\prime }+e_j{e}_i^{\prime })(e_i{e}_j^{\prime }+e_j{e}_i^{\prime })\parallel =\parallel \sum _{1\le i<j\le n}\mathbb{E}\left[W^2(i,j)(e_i{e}_i^{\prime }+e_j{e}_j^{\prime })\right]\parallel \hfill \\ \hfill & \le \underset{1\le i\le n}{\max }|\sum _{j=1}^n\mathbb{E}\left(W^2(i,j)\right)|\le \underset{1\le i\le n}{\max }\sum _{j=1}^n\theta \left(i\right)\theta \left(j\right){\tilde{P}}_{\max }\le {\tilde{P}}_{\max }{\theta }_{\max }{\parallel \theta \parallel }_1.\hfill \end{array}$$

Set $t=\frac{\alpha +1+\sqrt{(\alpha +1)(\alpha +19)}}{3}\sqrt{{\tilde{P}}_{\max }{\theta }_{\max }{\parallel \theta \parallel }_1\log \left(n\right)}$ for any $\alpha >0$, combine Theorem A2 with ${\sigma }^2\le {\tilde{P}}_{\max }{\theta }_{\max }{\parallel \theta \parallel }_1,R=1,d=n$, we have

$$\begin{array}{cc}\hfill \mathbb{P}(\parallel W\parallel \ge t)& =\mathbb{P}(\parallel \sum _{1\le i<j\le n}{W}^{(i,j)}\parallel \ge t)\le n·\exp \left(\frac{-t^2/2}{{\sigma }^2+Rt/3}\right)\hfill \\ \hfill & \le n·\exp \left(\frac{-(\alpha +1)\log \left(n\right)}{\frac{18}{{(\sqrt{\alpha +1}+\sqrt{\alpha +19})}^2}+\frac{2\sqrt{\alpha +1}}{\sqrt{\alpha +1}+\sqrt{\alpha +19}}\sqrt{\frac{\log \left(n\right)}{{\tilde{P}}_{\max }{\theta }_{\max }{\parallel \theta \parallel }_1}}}\right)\le \frac{1}{n^{\alpha }},\hfill \end{array}$$

where we use Assumption (A2) such that $\frac{18}{{(\sqrt{\alpha +1}+\sqrt{\alpha +19})}^2}+\frac{2\sqrt{\alpha +1}}{\sqrt{\alpha +1}+\sqrt{\alpha +19}}\sqrt{\frac{\log \left(n\right)}{{\tilde{P}}_{\max }{\theta }_{\max }{\parallel \theta \parallel }_1}}\le \frac{18}{{(\sqrt{\alpha +1}+\sqrt{\alpha +19})}^2}+\frac{2\sqrt{\alpha +1}}{\sqrt{\alpha +1}+\sqrt{\alpha +19}}=1$. □

Appendix D.3. Proof of Lemma 5

Proof. 

The proof is similar to that of Lemma 2, so we omit most details. Since $\mathbb{E}\left(A\right(i,j)-\Omega (i,j\left)\right)=0$, $\mathbb{E}\left[{(A(i,j)-\Omega (i,j))}^2\right]\le \theta \left(i\right)\theta \left(j\right){\tilde{P}}_{\max }\le {\theta }_{\max }^2{\tilde{P}}_{\max }$, $\frac{1}{{\theta }_{\max }\sqrt{{\tilde{P}}_{\max }n/\left(\mu \log \left(n\right)\right)}}\le O\left(1\right)$ holds by Assumption (A2) where $\mu =\frac{{n\parallel U\parallel }_{2\to \infty }^2}{K}$. By Theorem 4.2 [64], with high probability, we have

$$\begin{array}{c}\hfill \parallel \widehat{U}{\mathrm{sgn}\left(H\right)-U\parallel }_{2\to \infty }\le C\frac{{\theta }_{\max }\sqrt{{\tilde{P}}_{\max }K}(\kappa (\Omega )\sqrt{\mu }+\sqrt{\log \left(n\right)})}{{\sigma }_K(\Omega )},\end{array}$$

provided that $c_{*}{\sigma }_K(\Omega )\ge {\theta }_{\max }\sqrt{{\tilde{P}}_{\max }n\log \left(n\right)}$ for some sufficiently small constant $c_{*}$. By Lemma H.1 of [43], we know that ${\parallel U\parallel }_{2\to \infty }^2\le \frac{{\theta }_{\max }^2}{{\lambda }_K({\Pi }^{\prime }{\Theta }^2\Pi )}\le \frac{{\theta }_{\max }^2}{{\theta }_{\min }^2{\lambda }_K({\Pi }^{\prime }\Pi )}$ under $DCM{M}_n(K,\tilde{P},\Pi ,\Theta )$, which gives

$$\begin{array}{c}\hfill \parallel \widehat{U}{\mathrm{sgn}\left(H\right)-U\parallel }_{2\to \infty }\le C\frac{{\theta }_{\max }\sqrt{{\tilde{P}}_{\max }K}(\frac{{\theta }_{\max }\kappa (\Omega )}{{\theta }_{\min }}\sqrt{\frac{n}{K{\lambda }_K({\Pi }^{\prime }\Pi )}}+\sqrt{\log \left(n\right)})}{{\sigma }_K(\Omega )},\end{array}$$

Remark A3.

Similar to the proof of Lemma 2, by Theorem 4.2 of [65], when ${\sigma }_K(\Omega )\ge 4{\parallel A-\Omega \parallel }_{\infty }$, we have

$$\begin{array}{c}\hfill \parallel \widehat{U}{\mathrm{sgn}\left(H\right)-U\parallel }_{2\to \infty }\le 14\frac{{\parallel A-\Omega \parallel }_{\infty }}{{\sigma }_K(\Omega )}{\parallel U\parallel }_{2\to \infty }\le \frac{14{\theta }_{\max }{\parallel A-\Omega \parallel }_{\infty }}{{\theta }_{\min }{\sigma }_K(\Omega )\sqrt{{\lambda }_K({\Pi }^{\prime }\Pi )}}.\end{array}$$

Let $y={(y_1,y_2,\dots ,y_n)}^{\prime }$ be any $n\times 1$ vector, and by the Bernstein inequality, for any $t\ge 0$ and $i\in \left[n\right]$, we have

$$\begin{array}{c}\hfill \mathbb{P}\left(\right|\sum _{j=1}^n(A(i,j)-\Omega (i,j))y\left(j\right)|>t)\le 2\exp (-\frac{t^2/2}{{\sum }_{j=1}^n\Omega (i,j)y^2\left(j\right)+\frac{{t\parallel y\parallel }_{\infty }}{3}}).\end{array}$$

By the proof of Lemma 4, we have $\Omega (i,j)\le \theta \left(i\right)\theta \left(j\right){\tilde{P}}_{\max }$, which gives ${\sum }_{j=1}^n\Omega (i,j)\le {\tilde{P}}_{\max }{\theta }_{\max }{\parallel \theta \parallel }_1$. Set $y\left(j\right)$ as 1 or $-1$ such that $\left(A\right(i,j)-\Omega (i,j\left)\right)y\left(j\right)=\left|A\right(i,j)-\Omega (i,j\left)\right|$, we have

$$\begin{array}{c}\hfill {\mathbb{P}(\parallel A-\Omega \parallel }_{\infty }>t)\le 2\exp (-\frac{t^2/2}{{\tilde{P}}_{\max }{\theta }_{\max }{\parallel \theta \parallel }_1+\frac{t}{3}}).\end{array}$$

Set $t=\frac{\alpha +1+\sqrt{(\alpha +1)(\alpha +19)}}{3}\sqrt{{\tilde{P}}_{\max }{\theta }_{\max }{\parallel \theta \parallel }_1\log \left(n\right)}$ for any $\alpha >0$, by Assumption (A2), we have

$$\begin{array}{c}\hfill {\mathbb{P}(\parallel A-\Omega \parallel }_{\infty }>t)\le 2\exp (-\frac{t^2/2}{{\tilde{P}}_{\max }{\theta }_{\max }{\parallel \theta \parallel }_1+\frac{t}{3}})\le n^{-\alpha }.\end{array}$$

Hence, when ${\sigma }_K(\Omega )\ge C_0\sqrt{{\tilde{P}}_{\max }{\theta }_{\max }{\parallel \theta \parallel }_1\log \left(n\right)}$ where $C_0=4\frac{\alpha +1+\sqrt{(\alpha +1)(\alpha +19)}}{3}$, with probability at least $1-o\left(n^{-\alpha }\right)$,

$$\begin{array}{c}\hfill \parallel \widehat{U}{\mathrm{sgn}\left(H\right)-U\parallel }_{2\to \infty }\le C\frac{{\theta }_{\max }\sqrt{{\tilde{P}}_{\max }{\theta }_{\max }{\parallel \theta \parallel }_1\log \left(n\right)}}{{\theta }_{\min }{\sigma }_K(\Omega )\sqrt{{\lambda }_K({\Pi }^{\prime }\Pi )}}.\end{array}$$

Meanwhile, since $\sqrt{{\tilde{P}}_{\max }{\theta }_{\max }{\parallel \theta \parallel }_1\log \left(n\right)}\le {\theta }_{\max }\sqrt{{\tilde{P}}_{\max }n\log \left(n\right)}$, for convenience, we let the lower bound requirement of ${\sigma }_K(\Omega )$ be $C{\theta }_{\max }\sqrt{{\tilde{P}}_{\max }n\log \left(n\right)}$.

Similar to the proof of Lemma 2, we have

$$\begin{array}{cc}\hfill & \parallel \widehat{U}{\widehat{U}}^{\prime }-U{U}^{\prime }{\parallel }_{2\to \infty }={\max }_{i\in \left[n\right]}\parallel e_i^{\prime }(U{U}^{\prime }-\widehat{U}{\widehat{U}}^{\prime }){\parallel }_F\le 2{\parallel U-\widehat{U}\mathrm{sgn}\left(H\right)\parallel }_{2\to \infty }\hfill \\ \hfill & \le C\frac{{\theta }_{\max }\sqrt{{\tilde{P}}_{\max }K}(\frac{{\theta }_{\max }\kappa (\Omega )}{{\theta }_{\min }}\sqrt{\frac{n}{K{\lambda }_K({\Pi }^{\prime }\Pi )}}+\sqrt{\log \left(n\right)})}{{\sigma }_K(\Omega )}\hfill \\ \hfill & \le C\frac{{\theta }_{\max }\sqrt{{\tilde{P}}_{\max }K}(\frac{{\theta }_{\max }\kappa (\Omega )}{{\theta }_{\min }}\sqrt{\frac{n}{K{\lambda }_K({\Pi }^{\prime }\Pi )}}+\sqrt{\log \left(n\right)})}{{\theta }_{\min }^2{\sigma }_K\left(\tilde{P}\right){\lambda }_K({\Pi }^{\prime }\Pi )},\hfill \end{array}$$

where the last inequality holds since ${\sigma }_K(\Omega )={\sigma }_K(\Theta \Pi \tilde{P}{\Pi }^{\prime }\Theta )\ge {\theta }_{\min }^2{\sigma }_K(\Pi P{\Pi }^{\prime })={\theta }_{\min }^2{\sigma }_K({\Pi }^{\prime }\Pi \tilde{P})\ge {\theta }_{\min }^2{\sigma }_K\left(\tilde{P}\right){\sigma }_K({\Pi }^{\prime }\Pi )={\theta }_{\min }^2{\sigma }_K\left(\tilde{P}\right){\lambda }_K({\Pi }^{\prime }\Pi )$. And this bound is $C\frac{{\theta }_{\max }\sqrt{{\tilde{P}}_{\max }{\theta }_{\max }{\parallel \theta \parallel }_1\log \left(n\right)}}{{\theta }_{\min }^3{\sigma }_K\left(\tilde{P}\right){\lambda }_K^{1.5}({\Pi }^{\prime }\Pi )}$ if we use Theorem 4.2 of [65]. □

Appendix D.4. Proof of Theorem 2

Proof. 

To prove this theorem, we follow similar procedures as Theorem 3.2 of [43]. For $i\in \left[n\right]$, recall that $Z_{*}=Y_{*}{J}_{*}\equiv N_U^{-1}{N}_M\Pi ,{\widehat{Z}}_{*}={\widehat{Y}}_{*}{\widehat{J}}_{*},\Pi (i,:)=\frac{Z(i,:)}{{\parallel Z(i,:)\parallel }_1}$ and ${\widehat{\Pi }}_{*}(i,:)=\frac{{\widehat{Z}}_{*}(i,:)}{\parallel {\widehat{Z}}_{*}{(i,:)\parallel }_1}$, where $N_M$ and M are defined in the proof of Lemma 3 such that $U=\Theta M\equiv \Theta \Pi B_{*}$ and $N_M(i,i)=\frac{1}{{\parallel M(i,:)\parallel }_F}$, we have

$$\begin{array}{c}\hfill \parallel e_i^{\prime }({\widehat{\Pi }}_{*}-\Pi {\mathcal{P}}_{*}){\parallel }_1\le \frac{2\parallel e_i^{\prime }({\widehat{Z}}_{*}-Z_{*}{\mathcal{P}}_{*}){\parallel }_1}{\parallel e_i^{\prime }{Z}_{*}{\parallel }_1}\le \frac{2\sqrt{K}{\parallel e_i^{\prime }({\widehat{Z}}_{*}-Z_{*}{\mathcal{P}}_{*})\parallel }_F}{\parallel e_i^{\prime }{Z}_{*}{\parallel }_1}.\end{array}$$

Now, we provide a lower bound of $\parallel e_i^{\prime }{Z}_{*}{\parallel }_1$ as below

$$\begin{array}{cc}\hfill \parallel e_i^{\prime }{Z}_{*}{\parallel }_1& =\parallel e_i^{\prime }{N}_U^{-1}{N}_M{\Pi \parallel }_1=\parallel N_U^{-1}(i,i)e_i^{\prime }{N}_M{\Pi \parallel }_1=N_U^{-1}(i,i){\parallel N_M(i,i)e_i^{\prime }\Pi \parallel }_1=\frac{N_M(i,i)}{N_U(i,i)}\hfill \\ \hfill & ={\parallel U(i,:)\parallel }_F{N}_M(i,i)={\parallel U(i,:)\parallel }_F\frac{1}{{\parallel M(i,:)\parallel }_F}={\parallel U(i,:)\parallel }_F\frac{1}{\parallel e_i^{\prime }{M\parallel }_F}\hfill \\ \hfill & ={\parallel U(i,:)\parallel }_F\frac{1}{\parallel e_i^{\prime }{\Theta }^{-1}{U\parallel }_F}={\parallel U(i,:)\parallel }_F\frac{1}{\parallel {\Theta }^{-1}(i,i)e_i^{\prime }{U\parallel }_F}=\theta \left(i\right)\ge {\theta }_{\min }.\hfill \end{array}$$

Therefore, by Lemma A3, we have

$$\begin{array}{cc}\hfill \parallel e_i^{\prime }({\widehat{\Pi }}_{*}-\Pi {\mathcal{P}}_{*}){\parallel }_1& \le \frac{2\sqrt{K}{\parallel e_i^{\prime }({\widehat{Z}}_{*}-Z_{*}{\mathcal{P}}_{*})\parallel }_F}{\parallel e_i^{\prime }{Z}_{*}{\parallel }_1}\le \frac{2\sqrt{K}{\parallel e_i^{\prime }({\widehat{Z}}_{*}-Z_{*}{\mathcal{P}}_{*})\parallel }_F}{{\theta }_{\min }}\hfill \\ \hfill & =O\left(\frac{{\theta }_{\max }^{15}{K}^5\varpi {\kappa }^{4.5}({\Pi }^{\prime }\Pi ){\lambda }_1^{1.5}({\Pi }^{\prime }\Pi )}{{\theta }_{\min }^{15}{\pi }_{\min }}\right).\hfill \end{array}$$

□

Appendix D.5. Proof of Corollary 2

Proof. 

Under conditions of Corollary 2, we have

$$\begin{array}{c}\hfill {\max }_{i\in \left[n\right]}{\parallel e_i^{\prime }({\widehat{\Pi }}_{*}-\Pi {\mathcal{P}}_{*})\parallel }_1=O\left(\frac{{\theta }_{\max }^{15}{K}^5\varpi {\kappa }^{4.5}({\Pi }^{\prime }\Pi ){\lambda }_1^{1.5}({\Pi }^{\prime }\Pi )}{{\theta }_{\min }^{15}{\pi }_{\min }}\right)=O\left(\frac{{\theta }_{\max }^{15}\varpi \sqrt{n}}{{\theta }_{\min }^{15}}\right).\end{array}$$

Under the conditions of Corollary 2, Lemma 5 gives $\varpi =O\left(\frac{{\theta }_{\max }\sqrt{{\tilde{P}}_{\max }{\theta }_{\max }{\parallel \theta \parallel }_1\log \left(n\right)}}{{\theta }_{\min }^3{\sigma }_K\left(\tilde{P}\right)n^{1.5}}\right)$, which gives that

$$\begin{array}{c}\hfill {\max }_{i\in \left[n\right]}{\parallel e_i^{\prime }({\widehat{\Pi }}_{*}-\Pi {\mathcal{P}}_{*})\parallel }_1=O\left(\frac{{\theta }_{\max }^{15}\varpi \sqrt{n}}{{\theta }_{\min }^{15}}\right)=O\left(\frac{{\theta }_{\max }^{16}\sqrt{{\tilde{P}}_{\max }{\theta }_{\max }{\parallel \theta \parallel }_1\log \left(n\right)}}{{\theta }_{\min }^{18}{\sigma }_K\left(\tilde{P}\right)n}\right).\end{array}$$

By basic algebra, this corollary follows. □

Appendix D.6. Basic Properties of Ω under DCMM

Lemma A1.

Under $DCM{M}_n(K,\tilde{P},\Pi ,\Theta )$, we have

$$\begin{array}{cc}\hfill & {\parallel U(i,:)\parallel }_F\ge \frac{{\theta }_{\min }}{{\theta }_{\max }\sqrt{K{\lambda }_1({\Pi }^{\prime }\Pi )}}\mathrm{for}i\in \left[n\right],\mathrm{and}\eta \ge \frac{{\theta }_{\min }^4{\pi }_{\min }}{{\theta }_{\max }^{4}K{\lambda }_1({\Pi }^{\prime }\Pi )},\hfill \end{array}$$

where $\eta ={\min }_{k\in \left[K\right]}\left({\left(U_{*}(\mathcal{I},:)U_{*}^{\prime }(\mathcal{I},:)\right)}^{-1}\mathbf{1}\right)\left(k\right)$.

Proof. 

Since $I=U^{\prime }U=U^{\prime }(\mathcal{I},:){\Theta }^{-1}(\mathcal{I},\mathcal{I}){\Pi }^{\prime }{\Theta }^2\Pi {\Theta }^{-1}(\mathcal{I},\mathcal{I})U(\mathcal{I},:)$ by the proof of Lemma 3, we have $(\left({\Theta }^{-1}(\mathcal{I},\mathcal{I})U(\mathcal{I},:)\right){\left({\left({\Theta }^{-1}(\mathcal{I},\mathcal{I})U(\mathcal{I},:)\right)}^{\prime }\right)}^{-1}={\Pi }^{\prime }{\Theta }^2\Pi$, which gives that

$$\begin{array}{cc}\hfill {\min }_k{\parallel e_k^{\prime }\left({\Theta }^{-1}(\mathcal{I},\mathcal{I})U(\mathcal{I},:)\right)\parallel }_F^2& ={\min }_k{e}_k^{\prime }\left({\Theta }^{-1}(\mathcal{I},\mathcal{I})U(\mathcal{I},:)\right){\left({\Theta }^{-1}(\mathcal{I},\mathcal{I})U(\mathcal{I},:)\right)}^{\prime }{e}_k\hfill \\ \hfill & \ge {\min }_{\parallel x\parallel =1}{x}^{\prime }\left({\Theta }^{-1}(\mathcal{I},\mathcal{I})U(\mathcal{I},:)\right){\left({\Theta }^{-1}(\mathcal{I},\mathcal{I})U(\mathcal{I},:)\right)}^{\prime }x\hfill \\ & ={\lambda }_K\left(\left({\Theta }^{-1}(\mathcal{I},\mathcal{I})U(\mathcal{I},:)\right){\left({\Theta }^{-1}(\mathcal{I},\mathcal{I})U(\mathcal{I},:)\right)}^{\prime }\right)=\frac{1}{{\lambda }_1({\Pi }^{\prime }{\Theta }^2\Pi )},\hfill \end{array}$$

where x is a $K\times 1$ vector whose $l_2$ norm is 1. Then, for $i\in \left[n\right]$, we have

$$\begin{array}{cc}\hfill {\parallel U(i,:)\parallel }_F& =\parallel {\theta }_i\Pi (i,:){\Theta }^{-1}{(\mathcal{I},\mathcal{I})U(\mathcal{I},:)\parallel }_F={\theta }_i{\parallel \Pi (i,:){\Theta }^{-1}(\mathcal{I},\mathcal{I})U(\mathcal{I},:)\parallel }_F\hfill \\ \hfill & \ge {\theta }_i{\min }_i{\parallel \Pi (i,:)\parallel }_F{\min }_i{\parallel e_i^{\prime }\left({\Theta }^{-1}(\mathcal{I},\mathcal{I})U(\mathcal{I},:)\right)\parallel }_F\hfill \\ \hfill & \ge {\theta }_i{\min }_i{\parallel e_i^{\prime }\left({\Theta }^{-1}(\mathcal{I},\mathcal{I})U(\mathcal{I},:)\right)\parallel }_F/\sqrt{K}\hfill \\ & \ge \frac{{\theta }_i}{\sqrt{K{\lambda }_1({\Pi }^{\prime }{\Theta }^2\Pi )}}\ge \frac{{\theta }_{\min }}{{\theta }_{\max }\sqrt{K{\lambda }_1({\Pi }^{\prime }\Pi )}},\hfill \end{array}$$

where we use the fact that ${\min }_i{\parallel \Pi (i,:)\parallel }_F\ge \frac{1}{\sqrt{K}}$ since ${\sum }_{k=1}^K\Pi (i,k)=1$ and all entries of $\Pi$ are non-negative.

Since $U_{*}=N_{U}U$, we have

$$\begin{array}{cc}\hfill {\left(U_{*}(\mathcal{I},:)U_{*}^{\prime }(\mathcal{I},:)\right)}^{-1}& =N_U^{-1}(\mathcal{I},\mathcal{I}){\Theta }^{-1}(\mathcal{I},\mathcal{I}){\Pi }^{\prime }{\Theta }^2\Pi {\Theta }^{-1}(\mathcal{I},\mathcal{I})N_U^{-1}(\mathcal{I},\mathcal{I})\hfill \\ \hfill & \ge \frac{{\theta }_{\min }^2}{{\theta }_{\max }^2{N}_{U,\max }^2}{\Pi }^{\prime }\Pi \ge \frac{{\theta }_{\min }^4}{{\theta }_{\max }^{4}K{\lambda }_1({\Pi }^{\prime }\Pi )}{\Pi }^{\prime }\Pi ,\hfill \end{array}$$

where we set $N_{U,\max }={\max }_{i\in \left[n\right]}{N}_U(i,i)$ and we use the fact that $N_U,\Theta$ are diagonal matrices and $N_{U,\max }\le \frac{{\theta }_{\max }\sqrt{K{\lambda }_1({\Pi }^{\prime }\Pi )}}{{\theta }_{\min }}$. Then we have

$$\begin{array}{cc}\hfill & \eta ={\min }_{k\in \left[K\right]}\left({\left(U_{*}(\mathcal{I},:)U_{*}^{\prime }(\mathcal{I},:)\right)}^{-1}\mathbf{1}\right)\left(k\right)\ge \frac{{\theta }_{\min }^4}{{\theta }_{\max }^{4}K{\lambda }_1({\Pi }^{\prime }\Pi )}{\min }_{k\in \left[K\right]}{e}_k^{\prime }{\Pi }^{\prime }\Pi \mathbf{1}\hfill \\ \hfill & =\frac{{\theta }_{\min }^4}{{\theta }_{\max }^{4}K{\lambda }_1({\Pi }^{\prime }\Pi )}{\min }_{k\in \left[K\right]}{e}_k^{\prime }{\Pi }^{\prime }\mathbf{1}=\frac{{\theta }_{\min }^4{\pi }_{\min }}{{\theta }_{\max }^{4}K{\lambda }_1({\Pi }^{\prime }\Pi )}.\hfill \end{array}$$

□

Appendix D.7. Bounds between Ideal SVM-cone-DCMMSB and SVM-cone-DCMMSB

The next lemma focuses on the 2nd step of SVM-cone-DCMMSB and is the cornerstone to characterize the behaviors of SVM-cone-DCMMSB.

Lemma A2.

Under $DCM{M}_n(K,\tilde{P},\Pi ,\Theta )$, when conditions of Lemma 5 hold, there exists a permutation matrix ${\mathcal{P}}_{*}\in {\mathbb{R}}^{K\times K}$ such that with probability at least $1-o\left(n^{-\alpha }\right)$, we have

$$\begin{array}{c}\hfill {\max }_{1\le k\le K}{\parallel e_k^{\prime }({\widehat{U}}_{*,2}({\widehat{\mathcal{I}}}_{*},:)-{\mathcal{P}}_{*}^{\prime }{U}_{*,2}(\mathcal{I},:))\parallel }_F=O\left(\frac{K^3{\theta }_{\max }^{11}\varpi {\kappa }^3({\Pi }^{\prime }\Pi ){\lambda }_1^{1.5}({\Pi }^{\prime }\Pi )}{{\theta }_{\min }^{11}{\pi }_{\min }}\right),\end{array}$$

where $U_{*,2}=U_{*}{U}^{\prime },{\widehat{U}}_{*,2}={\widehat{U}}_{*}{\widehat{U}}^{\prime }$, i.e., $U_{*,2},{\widehat{U}}_{*,2}$ are the row-normalized versions of $U{U}^{\prime }$ and $\widehat{U}{\widehat{U}}^{\prime }$, respectively.

Proof. 

Lemma G.1. of [43] says that using ${\widehat{U}}_{*,2}$ as input of the SVM-cone algorithm returns the same result as using ${\widehat{U}}_{*}$ as the input. By Lemma F.1 of [43], there exists a permutation matrix ${\mathcal{P}}_{*}\in {\mathbb{R}}^{K\times K}$ such that

$$\begin{array}{c}\hfill {\max }_{k\in \left[K\right]}{\parallel e_k^{\prime }({\widehat{U}}_{*,2}({\widehat{\mathcal{I}}}_{*},:)-{\mathcal{P}}_{*}^{\prime }{U}_{*,2}(\mathcal{I},:))\parallel }_F=O\left(\frac{\sqrt{K}\zeta {ϵ}_{*}}{{\lambda }_K^{1.5}\left(U_{*,2}(\mathcal{I},:)\right)U_{*,2}^{\prime }(\mathcal{I},:)}\right),\end{array}$$

where $\zeta \le \frac{4K}{\eta {\lambda }_K^{1.5}\left(U_{*,2}(\mathcal{I},:)U_{*,2}^{\prime }(\mathcal{I},:)\right)}=O\left(\frac{K}{\eta {\lambda }_K^{1.5}\left(U_{*}(\mathcal{I},:)U_{*}^{\prime }(\mathcal{I},:)\right)}\right),{ϵ}_{*}={\max }_{i\in \left[n\right]}{\parallel {\widehat{U}}_{*,2}(i,:)-U_{*,2}(i,:)\parallel }_F$ and $\eta ={\min }_{1\le k\le K}\left({\left(U_{*}(\mathcal{I},:)U_{*}^{\prime }(\mathcal{I},:)\right)}^{-1}\mathbf{1}\right)\left(k\right)$. Next we give upper bound of ${ϵ}_{*}$.

$$\begin{array}{cc}& \parallel {\widehat{U}}_{*,2}(i,:)-U_{*,2}{(i,:)\parallel }_F={\parallel \frac{{\widehat{U}}_2(i,:)\parallel U_2{(i,:)\parallel }_F-U_2(i,:){\parallel {\widehat{U}}_2(i,:)\parallel }_F}{\parallel {\widehat{U}}_2{(i,:)\parallel }_F{\parallel U_2(i,:)\parallel }_F}\parallel }_F\le \frac{2\parallel {\widehat{U}}_2(i,:)-U_2{(i,:)\parallel }_F}{\parallel U_2{(i,:)\parallel }_F}\hfill \\ \hfill & \le \frac{2\parallel {\widehat{U}}_2-U_2{\parallel }_{2\to \infty }}{\parallel U_2{(i,:)\parallel }_F}\le \frac{2\varpi }{\parallel U_2{(i,:)\parallel }_F}=\frac{2\varpi }{\parallel \left(U{U}^{\prime }\right)(i,:){\parallel }_F}=\frac{2\varpi }{\parallel U(i,:)U^{\prime }{\parallel }_F}=\frac{2\varpi }{{\parallel U(i,:)\parallel }_F}\hfill \\ \hfill & \le \frac{2{\theta }_{\max }\varpi \sqrt{K{\lambda }_1({\Pi }^{\prime }\Pi )}}{{\theta }_{\min }},\hfill \end{array}$$

where the last inequality holds by Lemma A1. Then, we have ${ϵ}_{*}=O\left(\frac{{\theta }_{\max }\varpi \sqrt{K{\lambda }_1({\Pi }^{\prime }\Pi )}}{{\theta }_{\min }}\right)$. By Lemma H.2. of [43], ${\lambda }_K\left(U_{*}(\mathcal{I},:)U_{*}^{\prime }(\mathcal{I},:)\right)\ge \frac{{\theta }_{\min }^2{\kappa }^{-1}({\Pi }^{\prime }\Pi )}{{\theta }_{\max }^2}$. By the lower bound of $\eta$ given in Lemma A1, we have

$$\begin{array}{c}\hfill {\max }_{k\in \left[K\right]}{\parallel e_k^{\prime }({\widehat{U}}_{*,2}({\widehat{\mathcal{I}}}_{*},:)-{\mathcal{P}}_{*}^{\prime }{U}_{*,2}(\mathcal{I},:))\parallel }_F=O\left(\frac{K^3{\theta }_{\max }^{11}\varpi {\kappa }^3({\Pi }^{\prime }\Pi ){\lambda }_1^{1.5}({\Pi }^{\prime }\Pi )}{{\theta }_{\min }^{11}{\pi }_{\min }}\right).\end{array}$$

□

Next the lemma focuses on the 3rd step of SVM-cone-DCMMSB and bounds ${\max }_{i\in \left[n\right]}{\parallel e_i^{\prime }({\widehat{Z}}_{*}-Z_{*}{\mathcal{P}}_{*})\parallel }_F$.

Lemma A3.

Under $DCM{M}_n(K,\tilde{P},\Pi ,\Theta )$, when the conditions of Lemma 5 hold, with a probability of at least $1-o\left(n^{-\alpha }\right)$, we have

$$\begin{array}{c}\hfill {\max }_{i\in \left[n\right]}{\parallel e_i^{\prime }({\widehat{Z}}_{*}-Z_{*}{\mathcal{P}}_{*})\parallel }_F=O\left(\frac{{\theta }_{\max }^{15}{K}^{4.5}\varpi {\kappa }^{4.5}({\Pi }^{\prime }\Pi ){\lambda }_1^{1.5}({\Pi }^{\prime }\Pi )}{{\theta }_{\min }^{14}{\pi }_{\min }}\right).\end{array}$$

Proof. 

For $i\in \left[n\right]$, since $Z_{*}=Y_{*}{J}_{*},{\widehat{Z}}_{*}={\widehat{Y}}_{*}{\widehat{J}}_{*}$ and $J_{*},{\widehat{J}}_{*}$ are diagonal matrices, we have

$$\begin{array}{cc}& \parallel e_i^{\prime }({\widehat{Z}}_{*}-Z_{*}{\mathcal{P}}_{*}){\parallel }_F=\parallel e_i^{\prime }(\max (0,{\widehat{Y}}_{*}{\widehat{J}}_{*})-Y_{*}{J}_{*}{\mathcal{P}}_{*}){\parallel }_F\le {\parallel e_i^{\prime }({\widehat{Y}}_{*}{\widehat{J}}_{*}-Y_{*}{J}_{*}{\mathcal{P}}_{*})\parallel }_F\hfill \\ \hfill & =\parallel e_i^{\prime }({\widehat{Y}}_{*}-Y_{*}{\mathcal{P}}_{*}){\widehat{J}}_{*}+e_i^{\prime }{Y}_{*}{\mathcal{P}}_{*}({\widehat{J}}_{*}-{\mathcal{P}}_{*}^{\prime }{J}_{*}{\mathcal{P}}_{*}){\parallel }_F\hfill \\ \hfill & \le \parallel e_i^{\prime }({\widehat{Y}}_{*}-Y_{*}{\mathcal{P}}_{*}){\parallel }_F\parallel {\widehat{J}}_{*}\parallel +\parallel e_i^{\prime }{Y}_{*}{\mathcal{P}}_{*}{\parallel }_F\parallel {\widehat{J}}_{*}-{\mathcal{P}}_{*}^{\prime }{J}_{*}{\mathcal{P}}_{*}\parallel \hfill \\ \hfill & =\parallel e_i^{\prime }({\widehat{Y}}_{*}-Y_{*}{\mathcal{P}}_{*}){\parallel }_F\parallel {\widehat{J}}_{*}\parallel +\parallel e_i^{\prime }{Y}_{*}{\parallel }_F\parallel {\widehat{J}}_{*}-{\mathcal{P}}_{*}^{\prime }{J}_{*}{\mathcal{P}}_{*}\parallel \hfill \\ \hfill & =\parallel e_i^{\prime }({\widehat{Y}}_{*}-Y_{*}{\mathcal{P}}_{*}){\parallel }_F\parallel {\widehat{J}}_{*}\parallel +\parallel e_i^{\prime }{Y}_{*}{\parallel }_F\parallel J_{*}-{\mathcal{P}}_{*}{\widehat{J}}_{*}{\mathcal{P}}_{*}^{\prime }\parallel .\hfill \end{array}$$

Therefore, the bound of $\parallel e_i^{\prime }({\widehat{Z}}_{*}-Z_{*}{\mathcal{P}}_{*}){\parallel }_F$ can be obtained as long as we bound $\parallel e_i^{\prime }({\widehat{Y}}_{*}-Y_{*}{\mathcal{P}}_{*}){\parallel }_F,\parallel {\widehat{J}}_{*}\parallel ,\parallel e_i^{\prime }{Y}_{*}{\parallel }_F$ and $\parallel J_{*}-{\mathcal{P}}_{*}{\widehat{J}}_{*}{\mathcal{P}}_{*}^{\prime }\parallel$. We bound the four terms as below:

• We bound $\parallel e_i^{\prime }({\widehat{Y}}_{*}-Y_{*}{\mathcal{P}}_{*}){\parallel }_F$ first. Set $U_{*}(\mathcal{I},:)=B_{*},{\widehat{U}}_{*}({\widehat{\mathcal{I}}}_{*},:)={\widehat{B}}_{*},U_{*,2}(\mathcal{I},:)=B_{2*},{\widehat{U}}_{*,2}({\widehat{\mathcal{I}}}_{*},:)={\widehat{B}}_{2*}$ for convenience. For $i\in \left[n\right]$, we have

  $$\begin{array}{cc}& \parallel e_i^{\prime }({\widehat{Y}}_{*}-Y_{*}{\mathcal{P}}_{*}){\parallel }_F={\parallel e_i^{\prime }(\widehat{U}{\widehat{B}}_{*}^{\prime }{\left({\widehat{B}}_{*}{\widehat{B}}_{*}^{\prime }\right)}^{-1}-U{B}_{*}^{\prime }{\left(B_{*}{B}_{*}^{\prime }\right)}^{-1}{\mathcal{P}}_{*})\parallel }_F\hfill \\ & =\parallel e_i^{\prime }(\widehat{U}-U\left(U^{\prime }\widehat{U}\right)){\widehat{B}}_{*}^{\prime }{\left({\widehat{B}}_{*}{\widehat{B}}_{*}^{\prime }\right)}^{-1}+e_i^{\prime }(U\left(U^{\prime }\widehat{U}\right){\widehat{B}}_{*}^{\prime }{\left({\widehat{B}}_{*}{\widehat{B}}_{*}^{\prime }\right)}^{-1}\hfill \\ \hfill & -U\left(U^{\prime }\widehat{U}\right){\left({\mathcal{P}}_{*}^{\prime }\left(B_{*}{B}_{*}^{\prime }\right){\left(B_{*}^{\prime }\right)}^{-1}\left(U^{\prime }\widehat{U}\right)\right)}^{-1}{)\parallel }_F\hfill \\ & \le \parallel e_i^{\prime }(\widehat{U}-U\left(U^{\prime }\widehat{U}\right)){\widehat{B}}_{*}^{\prime }{\left({\widehat{B}}_{*}{\widehat{B}}_{*}^{\prime }\right)}^{-1}{\parallel }_F+\parallel e_i^{\prime }U\left(U^{\prime }\widehat{U}\right)({\widehat{B}}_{*}^{\prime }{\left({\widehat{B}}_{*}{\widehat{B}}_{*}^{\prime }\right)}^{-1}\hfill \\ \hfill & -{\left({\mathcal{P}}_{*}^{\prime }\left(B_{*}{B}_{*}^{\prime }\right){\left(B_{*}^{\prime }\right)}^{-1}\left(U^{\prime }\widehat{U}\right)\right)}^{-1}{)\parallel }_F\hfill \\ \hfill & \le \parallel e_i^{\prime }(\widehat{U}-U\left(U^{\prime }\widehat{U}\right)){\parallel }_F\parallel {\widehat{B}}_{*}^{-1}{\parallel }_F+{\parallel e_i^{\prime }U\left(U^{\prime }\widehat{U}\right)({\widehat{B}}_{*}^{\prime }{\left({\widehat{B}}_{*}{\widehat{B}}_{*}^{\prime }\right)}^{-1}-{\left({\mathcal{P}}_{*}^{\prime }\left(B_{*}{B}_{*}^{\prime }\right){\left(B_{*}^{\prime }\right)}^{-1}\left(U^{\prime }\widehat{U}\right)\right)}^{-1})\parallel }_F\hfill \\ & \le \sqrt{K}\parallel e_i^{\prime }(\widehat{U}-U\left(U^{\prime }\widehat{U}\right)){\parallel }_F/\sqrt{{\lambda }_K\left({\widehat{B}}_{*}{\widehat{B}}_{*}^{\prime }\right)}+{\parallel e_i^{\prime }U\left(U^{\prime }\widehat{U}\right)({\widehat{B}}_{*}^{-1}-{\left({\mathcal{P}}_{*}^{\prime }{B}_{*}\left(U^{\prime }\widehat{U}\right)\right)}^{-1})\parallel }_F\hfill \\ \hfill & \stackrel{\left(i\right)}{=}\sqrt{K}\parallel e_i^{\prime }(\widehat{U}{\widehat{U}}^{\prime }-U{U}^{\prime })\widehat{U}{\parallel }_{F}O\left(\frac{{\theta }_{\max }\sqrt{\kappa ({\Pi }^{\prime }\Pi )}}{{\theta }_{\min }}\right)+{\parallel e_i^{\prime }U\left(U^{\prime }\widehat{U}\right)({\widehat{B}}_{*}^{-1}-{\left({\mathcal{P}}_{*}^{\prime }{B}_{*}\left(U^{\prime }\widehat{U}\right)\right)}^{-1})\parallel }_F\hfill \\ \hfill & \le \sqrt{K}\parallel e_i^{\prime }(\widehat{U}{\widehat{U}}^{\prime }-U{U}^{\prime }){\parallel }_{F}O\left(\frac{{\theta }_{\max }\sqrt{\kappa ({\Pi }^{\prime }\Pi )}}{{\theta }_{\min }}\right)+{\parallel e_i^{\prime }U\left(U^{\prime }\widehat{U}\right)({\widehat{B}}_{*}^{-1}-{\left({\mathcal{P}}_{*}^{\prime }{B}_{*}\left(U^{\prime }\widehat{U}\right)\right)}^{-1})\parallel }_F\hfill \\ \hfill & \le \sqrt{K}\varpi O\left(\frac{{\theta }_{\max }\sqrt{\kappa ({\Pi }^{\prime }\Pi )}}{{\theta }_{\min }}\right)+{\parallel e_i^{\prime }U\left(U^{\prime }\widehat{U}\right)({\widehat{B}}_{*}^{-1}-{\left({\mathcal{P}}_{*}^{\prime }{B}_{*}\left(U^{\prime }\widehat{U}\right)\right)}^{-1})\parallel }_F\hfill \\ & =O\left(\varpi \frac{{\theta }_{\max }\sqrt{K\kappa ({\Pi }^{\prime }\Pi )}}{{\theta }_{\min }}\right)+{\parallel e_i^{\prime }U\left(U^{\prime }\widehat{U}\right)({\widehat{B}}_{*}^{-1}-{\left({\mathcal{P}}_{*}^{\prime }{B}_{*}\left(U^{\prime }\widehat{U}\right)\right)}^{-1})\parallel }_F,\hfill \end{array}$$

  where we have used similar idea in the proof of Lemma VII.3 in [44] such that we apply $O\left(\frac{1}{{\lambda }_K\left(B_{*}{B}_{*}^{\prime }\right)}\right)$ to estimate $\frac{1}{{\lambda }_K\left({\widehat{B}}_{*}{\widehat{B}}_{*}^{\prime }\right)}$.
  Now we aim to bound $\parallel e_i^{\prime }U\left(U^{\prime }\widehat{U}\right)({\widehat{B}}_{*}^{-1}-{\left({\mathcal{P}}_{*}^{\prime }{B}_{*}\left(U^{\prime }\widehat{U}\right)\right)}^{-1}){\parallel }_F$. For convenience, set $T=U^{\prime }\widehat{U},S={\mathcal{P}}_{*}^{\prime }{B}_{*}T$. We have

  $$\begin{array}{cc}& \parallel e_i^{\prime }U\left(U^{\prime }\widehat{U}\right)({\widehat{B}}_{*}^{-1}-{\left({\mathcal{P}}_{*}^{\prime }{B}_{*}\left(U^{\prime }\widehat{U}\right)\right)}^{-1}){\parallel }_F={\parallel e_i^{\prime }UT{S}^{-1}(S-{\widehat{B}}_{*}){\widehat{B}}_{*}^{-1}\parallel }_F\hfill \\ & \le \parallel e_i^{\prime }UT{S}^{-1}(S-{\widehat{B}}_{*}){\parallel }_F\parallel {\widehat{B}}_{*}^{-1}{\parallel }_F\le {\parallel e_i^{\prime }UT{S}^{-1}(S-{\widehat{B}}_{*})\parallel }_F\frac{\sqrt{K}}{|{\lambda }_K\left({\widehat{B}}_{*}\right)|}\hfill \\ & =\parallel e_i^{\prime }UT{S}^{-1}(S-{\widehat{B}}_{*}){\parallel }_F\frac{\sqrt{K}}{\sqrt{{\lambda }_K\left({\widehat{B}}_{*}{\widehat{B}}_{*}^{\prime }\right)}}\le {\parallel e_i^{\prime }UT{S}^{-1}(S-{\widehat{B}}_{*})\parallel }_{F}O\left(\frac{{\theta }_{\max }\sqrt{K\kappa ({\Pi }^{\prime }\Pi )}}{{\theta }_{\min }}\right)\hfill \\ & =\parallel e_i^{\prime }UT{T}^{-1}{B}_{*}^{\prime }{\left(B_{*}{B}_{*}^{\prime }\right)}^{-1}{\mathcal{P}}_{*}(S-{\widehat{B}}_{*}){\parallel }_{F}O\left(\frac{{\theta }_{\max }\sqrt{K\kappa ({\Pi }^{\prime }\Pi )}}{{\theta }_{\min }}\right)\hfill \\ & =\parallel e_i^{\prime }U{B}_{*}^{\prime }{\left(B_{*}{B}_{*}^{\prime }\right)}^{-1}{\mathcal{P}}_{*}(S-{\widehat{B}}_{*}){\parallel }_{F}O\left(\frac{{\theta }_{\max }\sqrt{K\kappa ({\Pi }^{\prime }\Pi )}}{{\theta }_{\min }}\right)\hfill \\ & =\parallel e_i^{\prime }{Y}_{*}{\mathcal{P}}_{*}(S-{\widehat{B}}_{*}){\parallel }_{F}O\left(\frac{{\theta }_{\max }\sqrt{K{\lambda }_1({\Pi }^{\prime }\Pi )}}{{\theta }_{\min }}\right)\le \parallel e_i^{\prime }{Y}_{*}{\parallel }_F{\parallel S-{\widehat{B}}_{*}\parallel }_{F}O\left(\frac{{\theta }_{\max }\sqrt{K{\lambda }_1({\Pi }^{\prime }\Pi )}}{{\theta }_{\min }}\right)\hfill \\ \hfill & \stackrel{\mathrm{By}\mathrm{Equation}\left(A3\right)}{\le }\frac{{\theta }_{\max }^2\sqrt{K{\lambda }_1({\Pi }^{\prime }\Pi )}}{{\theta }_{\min }^2{\lambda }_K({\Pi }^{\prime }\Pi )}{\max }_{1\le k\le K}{\parallel e_k^{\prime }(S-{\widehat{B}}_{*})\parallel }_{F}O\left(\frac{{\theta }_{\max }K\sqrt{\kappa ({\Pi }^{\prime }\Pi )}}{{\theta }_{\min }}\right)\hfill \\ & ={\max }_{1\le k\le K}{\parallel e_k^{\prime }({\widehat{B}}_{*}-{\mathcal{P}}_{*}^{\prime }{B}_{*}{U}^{\prime }\widehat{U})\parallel }_{F}O\left(\frac{{\theta }_{\max }^3{K}^{1.5}\kappa ({\Pi }^{\prime }\Pi )}{{\theta }_{\min }^3\sqrt{{\lambda }_K({\Pi }^{\prime }\Pi )}}\right)\hfill \\ \hfill & ={\max }_{1\le k\le K}{\parallel e_k^{\prime }({\widehat{B}}_{*}{\widehat{U}}^{\prime }-{\mathcal{P}}_{*}^{\prime }{B}_{*}{U}^{\prime })\widehat{U}\parallel }_{F}O\left(\frac{{\theta }_{\max }^3{K}^{1.5}\kappa ({\Pi }^{\prime }\Pi )}{{\theta }_{\min }^3\sqrt{{\lambda }_K({\Pi }^{\prime }\Pi )}}\right)\hfill \\ & \le {\max }_{1\le k\le K}{\parallel e_k^{\prime }({\widehat{B}}_{*}{\widehat{U}}^{\prime }-{\mathcal{P}}^{\prime }{B}_{*}{U}^{\prime })\parallel }_{F}O\left(\frac{{\theta }_{\max }^3{K}^{1.5}\kappa ({\Pi }^{\prime }\Pi )}{{\theta }_{\min }^3\sqrt{{\lambda }_K({\Pi }^{\prime }\Pi )}}\right)\hfill \\ \hfill & ={\max }_{1\le k\le K}{\parallel e_k^{\prime }({\widehat{B}}_{2*}-{\mathcal{P}}_{*}^{\prime }{B}_{2*})\parallel }_{F}O\left(\frac{{\theta }_{\max }^3{K}^{1.5}\kappa ({\Pi }^{\prime }\Pi )}{{\theta }_{\min }^3\sqrt{{\lambda }_K({\Pi }^{\prime }\Pi )}}\right)\hfill \\ \hfill & \stackrel{\mathrm{By}\mathrm{Lemma}A2}{=}O\left(\frac{K^{4.5}{\theta }_{\max }^{14}\varpi {\kappa }^{4.5}({\Pi }^{\prime }\Pi ){\lambda }_1({\Pi }^{\prime }\Pi )}{{\theta }_{\min }^{14}{\pi }_{\min }}\right).\hfill \end{array}$$
  Then, we have

  $$\begin{array}{cc}\hfill \parallel e_i^{\prime }({\widehat{Y}}_{*}-Y_{*}{\mathcal{P}}_{*}){\parallel }_F& \le O\left(\varpi \frac{{\theta }_{\max }\sqrt{K\kappa ({\Pi }^{\prime }\Pi )}}{{\theta }_{\min }}\right)+\parallel e_i^{\prime }U\left(U^{\prime }\widehat{U}\right){({\widehat{B}}_{*}^{-1}-\left({\mathcal{P}}_{*}^{\prime }{B}_{*}{U}^{\prime }\widehat{U}\right))}^{-1}{)\parallel }_F\hfill \\ \hfill & \le O\left(\varpi \frac{{\theta }_{\max }\sqrt{K\kappa ({\Pi }^{\prime }\Pi )}}{{\theta }_{\min }}\right)+O\left(\frac{K^{4.5}{\theta }_{\max }^{14}\varpi {\kappa }^{4.5}({\Pi }^{\prime }\Pi ){\lambda }_1({\Pi }^{\prime }\Pi )}{{\theta }_{\min }^{14}{\pi }_{\min }}\right)\hfill \\ \hfill & =O\left(\frac{K^{4.5}{\theta }_{\max }^{14}\varpi {\kappa }^{4.5}({\Pi }^{\prime }\Pi ){\lambda }_1({\Pi }^{\prime }\Pi )}{{\theta }_{\min }^{14}{\pi }_{\min }}\right).\hfill \end{array}$$
• for $\parallel e_i^{\prime }{Y}_{*}{\parallel }_F$, since $Y_{*}=U{U}_{*}^{-1}(\mathcal{I},:)$, we have

  $$\begin{array}{c}\hfill \parallel e_i^{\prime }{Y}_{*}{\parallel }_F\le {\parallel U(i,:)\parallel }_F{\parallel U_{*}^{-1}(\mathcal{I},:)\parallel }_F\le \frac{\sqrt{K}{\parallel U(i,:)\parallel }_F}{\sqrt{{\lambda }_K\left(U_{*}(\mathcal{I},:)U_{*}^{\prime }(\mathcal{I},:)\right)}}\le \frac{{\theta }_{\max }^2\sqrt{K{\lambda }_1({\Pi }^{\prime }\Pi )}}{{\theta }_{\min }^2{\lambda }_K({\Pi }^{\prime }\Pi )}.\end{array}$$

  (A3)
• for $\parallel {\widehat{J}}_{*}\parallel$, recall that ${\widehat{J}}_{*}=\sqrt{\mathrm{diag}\left({\widehat{U}}_{*}({\widehat{I}}_{*},:)\widehat{\Lambda }{\widehat{U}}_{*}^{\prime }({\widehat{\mathcal{I}}}_{*},:)\right)}$, we have

  $$\begin{array}{cc}\hfill \parallel {\widehat{J}}_{*}{\parallel }^2& ={\max }_{k\in \left[K\right]}{\widehat{J}}_{*}^2(k,k)={\max }_{k\in \left[K\right]}{e}_k^{\prime }{\widehat{U}}_{*}({\widehat{I}}_{*},:)\widehat{\Lambda }{\widehat{U}}_{*}^{\prime }({\widehat{\mathcal{I}}}_{*},:)e_k\hfill \\ \hfill & ={\max }_{k\in \left[K\right]}\parallel e_k^{\prime }{\widehat{U}}_{*}({\widehat{I}}_{*},:)\widehat{\Lambda }{\widehat{U}}_{*}^{\prime }({\widehat{\mathcal{I}}}_{*},:)e_k\parallel \hfill \\ \hfill & \le {\max }_{k\in \left[K\right]}\parallel e_k^{\prime }{\widehat{U}}_{*}({\widehat{I}}_{*},:){\parallel }^2\parallel \widehat{\Lambda }\parallel \le {\max }_{k\in \left[K\right]}\parallel e_k^{\prime }{\widehat{U}}_{*}({\widehat{I}}_{*},:){\parallel }_F^2\parallel \widehat{\Lambda }\parallel =\parallel \widehat{\Lambda }\parallel ,\hfill \end{array}$$

  where we have used the fact that $\parallel {\widehat{U}}_{*}{(i,:)\parallel }_F=1$ for $i\in \left[n\right]$ in the last equality. Since we need ${\sigma }_K(\Omega )\ge C{\theta }_{\max }\sqrt{{\tilde{P}}_{\max }n\log \left(n\right)}\ge C\parallel A-\Omega \parallel$ in the proof of Lemma 5, we have $\parallel \widehat{\Lambda }\parallel =\parallel A\parallel =\parallel A-\Omega +\Omega \parallel \le \parallel A-\Omega \parallel +\parallel \Omega \parallel \le {\sigma }_K(\Omega )+\parallel \Omega \parallel \le 2\parallel \Omega \parallel =2\parallel \Theta \Pi \tilde{P}{\Pi }^{\prime }\Theta \parallel \le 2C{\theta }_{\max }^2{\tilde{P}}_{\max }{\lambda }_1({\Pi }^{\prime }\Pi )=O\left({\theta }_{\max }^2{\tilde{P}}_{\max }{\lambda }_1({\Pi }^{\prime }\Pi )\right)$. Then we have

  $$\begin{array}{c}\hfill \parallel {\widehat{J}}_{*}\parallel =O\left({\theta }_{\max }\sqrt{{\tilde{P}}_{\max }{\lambda }_1({\Pi }^{\prime }\Pi )}\right).\end{array}$$
• for $\parallel J_{*}-{\mathcal{P}}_{*}{\widehat{J}}_{*}{\mathcal{P}}_{*}^{\prime }\parallel$, we provide some simple facts first: $\parallel \widehat{\Lambda }\parallel =\parallel A\parallel ,\parallel \Lambda \parallel =\parallel \Omega \parallel ,\Omega =U\Lambda U^{\prime },\tilde{A}=\widehat{U}\widehat{\Lambda }{\widehat{U}}^{\prime },\parallel \widehat{U}\parallel =1,\parallel U\parallel =1,\parallel e_k^{\prime }{\mathcal{P}}_{*}{\widehat{B}}_{2*}\parallel =\parallel {\widehat{B}}_{2*}{e}_k\parallel =\parallel e_k^{\prime }{\widehat{B}}_{2*}\parallel \le \parallel e_k^{\prime }{\widehat{B}}_{2*}{\parallel }_F=1$. Since $\tilde{A}$ is the best rank K approximation to A in the spectral norm, and therefore $\parallel \tilde{A}-A\parallel \le \parallel \Omega -A\parallel$ since $\Omega =U\Lambda U^{\prime }$ with rank K and $\Omega$ can also be viewed as a rank K approximation to A. This leads to $\parallel \Omega -\tilde{A}\parallel =\parallel \Omega -A+A-\tilde{A}\parallel \le 2\parallel A-\Omega \parallel$. By Lemma H.2 [43], $\parallel B_{*}\parallel =\parallel U_{*}(\mathcal{I},:)\parallel =\sqrt{{\lambda }_1\left(U_{*}(\mathcal{I},:)U_{*}^{\prime }(\mathcal{I},:)\right)}\le \sqrt{\kappa ({\Pi }^{\prime }{\Theta }^2\Pi )}\le \frac{{\theta }_{\max }{\kappa }^{0.5}({\Pi }^{\prime }\Pi )}{{\theta }_{\min }}$. $\parallel A\parallel =\parallel A-\Omega +\Omega \parallel \le \parallel A-\Omega \parallel +\parallel \Omega \parallel \le {\sigma }_K(\Omega )+\parallel \Omega \parallel \le 2\parallel \Omega \parallel$ by the lower bound requirement of ${\sigma }_K(\Omega )$ in Lemma 5, and we also have $\parallel A-\Omega \parallel \le {\sigma }_K(\Omega )\le \parallel \Omega \parallel$. For $k\in \left[K\right]$, let ${\tau }_k=J_{*}(k,k),{\widehat{\tau }}_k=\left({\mathcal{P}}_{*}{\widehat{J}}_{*}{\mathcal{P}}_{*}^{\prime }\right)(k,k)$ for convenience. Based on the above facts and Lemma A2, we have

  $$\begin{array}{cc}\hfill & {\max }_{k\in \left[K\right]}|{\tau }_k^2-{\widehat{\tau }}_k^2|={\max }_{k\in \left[K\right]}\parallel e_k^{\prime }{U}_{*}(\mathcal{I},:)\Lambda U_{*}^{\prime }(\mathcal{I},:)e_k-e_k^{\prime }{\mathcal{P}}_{*}{\widehat{U}}_{*}({\widehat{\mathcal{I}}}_{*},:)\widehat{\Lambda }{\widehat{U}}_{*}^{\prime }({\widehat{\mathcal{I}}}_{*},:)P_{*}^{\prime }{e}_k\parallel \hfill \\ \hfill & ={\max }_{k\in \left[K\right]}\parallel e_k^{\prime }{B}_{2*}U\Lambda U^{\prime }{B}_{2*}{e}_k-e_k^{\prime }{\mathcal{P}}_{*}{\widehat{B}}_{2*}\widehat{U}\widehat{\Lambda }{\widehat{U}}^{\prime }{\widehat{B}}_{2*}^{\prime }{P}_{*}^{\prime }{e}_k\parallel \hfill \\ \hfill & \le \parallel e_k^{\prime }(B_{2*}-{\mathcal{P}}_{*}{\widehat{B}}_{2*})U\Lambda U^{\prime }{B}_{2*}^{\prime }{e}_k\parallel +\parallel e_k^{\prime }{\mathcal{P}}_{*}{\widehat{B}}_{2*}(U\Lambda U^{\prime }-\widehat{U}\widehat{\Lambda }{\widehat{U}}^{\prime })B_{2*}^{\prime }{e}_k\parallel \hfill \\ \hfill & +\parallel e_k^{\prime }{\mathcal{P}}_{*}{\widehat{B}}_{2*}\widehat{U}\widehat{\Lambda }{\widehat{U}}^{\prime }(B_{2*}^{\prime }-{\widehat{B}}_{2*}^{\prime }{\mathcal{P}}_{*}^{\prime })e_k\parallel \hfill \\ \hfill & \le \parallel e_k^{\prime }(B_{2*}-{\mathcal{P}}_{*}{\widehat{B}}_{2*})\parallel \parallel U\parallel \parallel \Lambda \parallel \parallel U^{\prime }\parallel \parallel B_{2*}^{\prime }{e}_k\parallel +\parallel e_k^{\prime }{\mathcal{P}}_{*}{\widehat{B}}_{2*}\parallel \parallel U\Lambda U^{\prime }-\widehat{U}\widehat{\Lambda }{\widehat{U}}^{\prime }\parallel \parallel B_{2*}^{\prime }{e}_k\parallel \hfill \\ & +\parallel e_k^{\prime }{\mathcal{P}}_{*}{\widehat{B}}_{2*}\parallel \parallel \widehat{U}\parallel \parallel \widehat{\Lambda }\parallel \parallel {\widehat{U}}^{\prime }\parallel \parallel (B_{2*}^{\prime }-{\widehat{B}}_{2*}^{\prime }{\mathcal{P}}_{*}^{\prime })e_k\parallel \hfill \\ \hfill & \le \parallel e_k^{\prime }(B_{2*}-{\mathcal{P}}_{*}{\widehat{B}}_{2*})\parallel \parallel \Lambda \parallel \parallel B_{2*}^{\prime }{e}_k\parallel +\parallel U\Lambda U^{\prime }-\widehat{U}\widehat{\Lambda }{\widehat{U}}^{\prime }\parallel \parallel B_{2*}^{\prime }{e}_k\parallel +\parallel \widehat{\Lambda }\parallel \parallel (B_{2*}^{\prime }-{\widehat{B}}_{2*}^{\prime }{\mathcal{P}}_{*}^{\prime })e_k\parallel \hfill \\ \hfill & =\parallel e_k^{\prime }(B_{2*}-{\mathcal{P}}_{*}{\widehat{B}}_{2*})\parallel (\parallel \Omega \parallel \parallel B_{2*}^{\prime }{e}_k\parallel +\parallel A\parallel )+\parallel \Omega -\tilde{A}\parallel \parallel B_{2*}^{\prime }{e}_k\parallel \hfill \\ \hfill & =\parallel e_k^{\prime }({\widehat{B}}_{2*}-{\mathcal{P}}_{*}^{\prime }{B}_{2*})\parallel (\parallel \Omega \parallel \parallel B_{2*}^{\prime }{e}_k\parallel +\parallel A\parallel )+\parallel \Omega -\tilde{A}\parallel \parallel B_{2*}^{\prime }{e}_k\parallel \hfill \\ \hfill & \le \parallel e_k^{\prime }({\widehat{B}}_{2*}-{\mathcal{P}}_{*}^{\prime }{B}_{2*})\parallel (\parallel \Omega \parallel \parallel B_{2*}^{\prime }{e}_k\parallel +\parallel A\parallel )+2\parallel A-\Omega \parallel \parallel B_{2*}^{\prime }{e}_k\parallel \hfill \\ \hfill & =\parallel e_k^{\prime }({\widehat{B}}_{2*}-{\mathcal{P}}_{*}^{\prime }{B}_{2*})\parallel (\parallel \Omega \parallel \parallel U{B}_{*}^{\prime }{e}_k\parallel +\parallel A\parallel )+2\parallel A-\Omega \parallel \parallel U{B}_{*}^{\prime }{e}_k\parallel \hfill \\ \hfill & \le \parallel e_k^{\prime }({\widehat{B}}_{2*}-{\mathcal{P}}_{*}^{\prime }{B}_{2*})\parallel (\parallel \Omega \parallel \parallel B_{*}^{\prime }{e}_k\parallel +\parallel A\parallel )+2\parallel A-\Omega \parallel \parallel B_{*}^{\prime }{e}_k\parallel \hfill \\ \hfill & \le \parallel e_k^{\prime }({\widehat{B}}_{2*}-{\mathcal{P}}_{*}^{\prime }{B}_{2*})\parallel (\parallel \Omega \parallel \parallel B_{*}^{\prime }{e}_k\parallel +2\parallel \Omega \parallel )+2\parallel \Omega \parallel \parallel B_{*}^{\prime }{e}_k\parallel \hfill \\ \hfill & =\parallel e_k^{\prime }({\widehat{B}}_{2*}-{\mathcal{P}}_{*}^{\prime }{B}_{2*})\parallel (\parallel B_{*}^{\prime }{e}_k\parallel +1)O\left({\theta }_{\max }^2{\tilde{P}}_{\max }{\lambda }_1({\Pi }^{\prime }\Pi )\right)+\parallel B_{*}^{\prime }{e}_k\parallel O\left({\theta }_{\max }^2{\tilde{P}}_{\max }{\lambda }_1({\Pi }^{\prime }\Pi )\right)\hfill \\ \hfill & \le \parallel e_k^{\prime }({\widehat{B}}_{2*}-{\mathcal{P}}_{*}^{\prime }{B}_{2*})\parallel (\parallel B_{*}\parallel +1)O\left({\theta }_{\max }^2{\tilde{P}}_{\max }{\lambda }_1({\Pi }^{\prime }\Pi )\right)+\parallel B_{*}\parallel O\left({\theta }_{\max }^2{\tilde{P}}_{\max }{\lambda }_1({\Pi }^{\prime }\Pi )\right)\hfill \\ \hfill & \le \parallel e_k^{\prime }({\widehat{B}}_{2*}-{\mathcal{P}}_{*}^{\prime }{B}_{2*})\parallel O({\theta }_{\max }^3{\tilde{P}}_{\max }{\kappa }^{0.5}({\Pi }^{\prime }\Pi ){\lambda }_1({\Pi }^{\prime }\Pi )/{\theta }_{\min })\hfill \\ \hfill & +O({\theta }_{\max }^3{\tilde{P}}_{\max }{\kappa }^{0.5}({\Pi }^{\prime }\Pi ){\lambda }_1({\Pi }^{\prime }\Pi )/{\theta }_{\min })\hfill \\ \hfill & \le \parallel e_k^{\prime }({\widehat{B}}_{2*}-{\mathcal{P}}_{*}^{\prime }{B}_{2*}){\parallel }_{F}O({\theta }_{\max }^3{\tilde{P}}_{\max }{\kappa }^{0.5}({\Pi }^{\prime }\Pi ){\lambda }_1({\Pi }^{\prime }\Pi )/{\theta }_{\min })\hfill \\ \hfill & +O({\theta }_{\max }^3{\tilde{P}}_{\max }{\kappa }^{0.5}({\Pi }^{\prime }\Pi ){\lambda }_1({\Pi }^{\prime }\Pi )/{\theta }_{\min })\hfill \\ \hfill & =O\left(\frac{K^3{\theta }_{\max }^{11}\varpi {\kappa }^3({\Pi }^{\prime }\Pi ){\lambda }_1^{1.5}({\Pi }^{\prime }\Pi )}{{\theta }_{\min }^{11}{\pi }_{\min }}\right)O({\theta }_{\max }^3{\tilde{P}}_{\max }{\kappa }^{0.5}({\Pi }^{\prime }\Pi ){\lambda }_1({\Pi }^{\prime }\Pi )/{\theta }_{\min })\hfill \\ \hfill & +O({\theta }_{\max }^3{\tilde{P}}_{\max }{\kappa }^{0.5}({\Pi }^{\prime }\Pi ){\lambda }_1({\Pi }^{\prime }\Pi )/{\theta }_{\min })=O\left(\frac{K^3{\theta }_{\max }^{14}{\tilde{P}}_{\max }\varpi {\kappa }^{3.5}({\Pi }^{\prime }\Pi ){\lambda }_1^{2.5}({\Pi }^{\prime }\Pi )}{{\theta }_{\min }^{12}{\pi }_{\min }}\right).\hfill \end{array}$$
  Recall that $J_{*}=N_U(\mathcal{I},\mathcal{I})\Theta (\mathcal{I},\mathcal{I})$, we have $\parallel J_{*}\parallel \le N_{U,\max }{\theta }_{\max }\le \frac{{\theta }_{\max }^2\sqrt{K{\lambda }_1({\Pi }^{\prime }\Pi )}}{{\theta }_{\min }}$ where the last inequality holds by Lemma A1. Similarly, we have $J_{*}(k,k)\ge {\theta }_{\min }{\min }_{i\in \left[n\right]}\frac{1}{{\parallel U(i,:)\parallel }_F}\ge \frac{{\theta }_{\min }^2\sqrt{{\lambda }_K({\Pi }^{\prime }\Pi )}}{{\theta }_{\max }}$ where the last inequality holds by the proof of Lemma 5. Then we have

  $$\begin{array}{cc}\hfill & \parallel J_{*}-{\mathcal{P}}_{*}{\widehat{J}}_{*}{\mathcal{P}}_{*}^{\prime }\parallel ={\max }_{k\in \left[K\right]}|{\widehat{\tau }}_k-{\tau }_k|={\max }_{k\in \left[K\right]}\frac{|{\widehat{\tau }}_k^2-{\tau }_k^2|}{{\widehat{\tau }}_k+{\tau }_k}\le {\max }_{k\in \left[K\right]}\frac{|{\widehat{\tau }}_k^2-{\tau }_k^2|}{{\tau }_k}\hfill \\ \hfill & \le \frac{{\theta }_{\max }}{{\theta }_{\min }^2\sqrt{{\lambda }_K({\Pi }^{\prime }\Pi )}}{\max }_{k\in \left[K\right]}|{\widehat{\tau }}_k^2-{\tau }_k^2|=O\left(\frac{K^3{\theta }_{\max }^{15}{\tilde{P}}_{\max }\varpi {\kappa }^{3.5}({\Pi }^{\prime }\Pi ){\lambda }_1^{2.5}({\Pi }^{\prime }\Pi )}{{\theta }_{\min }^{14}{\pi }_{\min }\sqrt{{\lambda }_K({\Pi }^{\prime }\Pi )}}\right).\hfill \end{array}$$
  Combining the above results, we have

  $$\begin{array}{cc}& \parallel e_i^{\prime }({\widehat{Z}}_{*}-Z_{*}{\mathcal{P}}_{*}){\parallel }_F\le \parallel e_i^{\prime }({\widehat{Y}}_{*}-Y_{*}{\mathcal{P}}_{*}){\parallel }_F\parallel {\widehat{J}}_{*}\parallel +\parallel e_i^{\prime }{Y}_{*}{\parallel }_F\parallel J_{*}-{\mathcal{P}}_{*}{\widehat{J}}_{*}{\mathcal{P}}_{*}^{\prime }\parallel \hfill \\ \hfill & \le O\left(\frac{K^{4.5}{\theta }_{\max }^{14}\varpi {\kappa }^{4.5}({\Pi }^{\prime }\Pi ){\lambda }_1({\Pi }^{\prime }\Pi )}{{\theta }_{\min }^{14}{\pi }_{\min }}\right)O\left({\theta }_{\max }\sqrt{{\tilde{P}}_{\max }{\lambda }_1({\Pi }^{\prime }\Pi )}\right)\hfill \\ \hfill & +\frac{{\theta }_{\max }^2\sqrt{K{\lambda }_1({\Pi }^{\prime }\Pi )}}{{\theta }_{\min }^2{\lambda }_K({\Pi }^{\prime }\Pi )}O\left(\frac{K^3{\theta }_{\max }^{15}{\tilde{P}}_{\max }\varpi {\kappa }^{3.5}({\Pi }^{\prime }\Pi ){\lambda }_1^{2.5}({\Pi }^{\prime }\Pi )}{{\theta }_{\min }^{14}{\pi }_{\min }\sqrt{{\lambda }_K({\Pi }^{\prime }\Pi )}}\right)\hfill \\ \hfill & =O\left(\frac{{\theta }_{\max }^{15}{K}^{4.5}\varpi {\kappa }^{4.5}({\Pi }^{\prime }\Pi ){\lambda }_1^{1.5}({\Pi }^{\prime }\Pi )}{{\theta }_{\min }^{14}{\pi }_{\min }}\right).\hfill \end{array}$$

□