Next Article in Journal
Palindromic Vectors, Symmetropy and Symmentropy as Symmetry Descriptors of Binary Data
Next Article in Special Issue
Relative Entropy of Distance Distribution Based Similarity Measure of Nodes in Weighted Graph Data
Previous Article in Journal
Monitoring the Evolution of Asynchrony between Mean Arterial Pressure and Mean Cerebral Blood Flow via Cross-Entropy Methods
Previous Article in Special Issue
Kolmogorov Basic Graphs and Their Application in Network Complexity Analysis
 
 
Order Article Reprints
Font Type:
Arial Georgia Verdana
Font Size:
Aa Aa Aa
Line Spacing:
Column Width:
Background:
Article

Structural Entropy of the Stochastic Block Models

1
Theory Lab, Central Research Institute, 2012 Labs, Huawei Tech. Co., Ltd., Hong Kong SAR, China
2
Department of Computer Science, School of Computing, National University of Singapore, Singapore 11741, Singapore
*
Author to whom correspondence should be addressed.
Entropy 2022, 24(1), 81; https://doi.org/10.3390/e24010081
Received: 24 November 2021 / Revised: 28 December 2021 / Accepted: 30 December 2021 / Published: 3 January 2022
(This article belongs to the Collection Graphs and Networks from an Algorithmic Information Perspective)

Abstract

:
With the rapid expansion of graphs and networks and the growing magnitude of data from all areas of science, effective treatment and compression schemes of context-dependent data is extremely desirable. A particularly interesting direction is to compress the data while keeping the “structural information” only and ignoring the concrete labelings. Under this direction, Choi and Szpankowski introduced the structures (unlabeled graphs) which allowed them to compute the structural entropy of the Erdős–Rényi random graph model. Moreover, they also provided an asymptotically optimal compression algorithm that (asymptotically) achieves this entropy limit and runs in expectation in linear time. In this paper, we consider the stochastic block models with an arbitrary number of parts. Indeed, we define a partitioned structural entropy for stochastic block models, which generalizes the structural entropy for unlabeled graphs and encodes the partition information as well. We then compute the partitioned structural entropy of the stochastic block models, and provide a compression scheme that asymptotically achieves this entropy limit.

1. Introduction

Shannon’s metric of “Entropy” of information is a foundational concept of information theory [1,2]. Given a discrete random variable X with support set (that is, the possible outcomes) x 1 , x 2 , , x n , which occurs with probability p 1 , p 2 , , p n , the entropy of X is defined as
H ( X ) : = i = 1 n p i log p i ,
where the logarithm here and throughout this paper is of base 2. Note that the entropy of X is a function of the probability distribution of X.
The entropy was originally created by Shannon in [3] as part of their theory of communication, where a data communication system consists of a data source X, a channel and a receiver. The fundamental problem of communication is for the receiver to reliably recover what data was generated by the source, based on the bits it receives through the channel. Shannon proved that the entropy of the source X plays a central role—in their source coding theorem it is shown that the entropy is the mathematical limit on how well the data can be losslessly compressed.
The question then arises: How to compress data that has structures, e.g., data in social networks? In Shannon’s 1953 less known paper [4] he argued for an extension of information theory, where data is considered as observations of a source, to “non-conventional data” (that is, lattices). Indeed, nowadays data appears in various formats and structures (e.g., sequences, expressions, interactions) and in drastically increasing amounts. In many scenarios, data is highly context-dependent and in particular, the structural information and the context information seem to be two conceptually different aspects. Therefore it is desirable to develop novel theory and efficient algorithms for extracting useful information from non-conventional data structures. Roughly speaking, such data consists of structural information, which, might be understood as the “shape” of the data, and context information which should be recognized as data labels.
It is well-known that complex networks (e.g., social networks) admit community structures [5]. That is, users within a group interact with each other more frequently than those outside the group. The stochastic block model (SBM) [6] is a celebrated random graph model that has been widely used to study the community structures in graphs and networks. It provides a good benchmark to evaluate the performance of community detection algorithms and inspires the design of many algorithms for community detection tasks. The theoretical underpinnings of the SBM have been extensively studied and sharp thresholds for exact recovery have been successively established [7,8,9,10,11,12]. We refer readers to [13] for a recent survey, where other interesting and important problems in SBM are also discussed.
In addition to the SBM discussed in [13], there are other angles to study compression of data with graph structures. Asadi et al. [14] investigated data compression on graphs with clusters. Zenil et al. [15] have surveyed information-theoretic methods, in particular Shannon entropy and algorithmic complexity, for characterizing graphs and networks.

1.1. Compression of Graphs

In recent years, graphical data and the network structures supporting them are becoming increasingly common and important in branches of engineering and sciences. To better represent and transmit graphical data, many works consider the problem of compressing the (random) graph up to isomorphism, i.e., compressing the structure of a graph. A graph G contains a finite set V of vertices and a set E of edges each of which connects two vertices. A graph can be represented by a binary matrix (the adjacency matrix) that further can be viewed as a binary sequence. Thus, encoding a labeled graph (that is, all vertices need to be distinguished) is equivalent to encoding the | V | 2 -digit binary sequence, given certain probability distribution on all | V | 2 possible edges. However, such a string does not reflect internal symmetries that are conveyed by the graph automorphism, and sometimes we are only interested in the local or global structures in the graph, rather than the exact vertex labelings. The structural entropy is defined when the graphs are considered unlabeled, or simply called structures, where the vertices are viewed as undistinguishable. The goal of this natural definition is to capture the information of the structure, and thus provides a fundamental measure in graph/structure compression schemes.
The problem actually has a strong theoretical background. Back to 1984, Turán [16] raised the question of finding an efficient coding method for general unlabeled graphs on n vertices, where a lower bound of n 2 n log n + O ( n ) bits is suggested. This lower bound can be seen by the number of unlabeled graphs [17]. The question was later answered by Naor [18] in 1990 who proposed such a representation that is optimal up to the first two leading terms when all unlabeled graphs are equally likely. In a recent paper Kieffer et al. [19] proved a structural complexity of a binary tree. There also have been some heuristic methods for real-world graph compression schemes, see [20,21,22,23,24]. Rather recently, Choi and Szpankowski [25] studied the structural entropy of the Erdős–Rényi random graph G ( n , p ) . They computed the structural entropy given that p is not (very) close to 0 or 1 and also gave a compression scheme that matches their computation. Later, the structural entropy for other randomly generated graphs, e.g., the preferential attachment graphs and web graphs are also studied [26,27,28,29].
However, it is well-known that the Erdős–Rényi model is too simplistic to model real networks, in particular due to its strong homogeneity and absence of community structure. In this paper, we consider the compression of graphical structures of the SBM, which in general model real networks better and circumvent the issues of the ER-model. In summary, our contributions are as follows:
  • We introduce the partitioned structural entropy which generalizes the structural entropy for unlabeled graphs and we show that it reflects the partition information of the SBM.
  • We provide an explicit formula for the partitioned structural entropy of the SBM.
  • We also propose a compression scheme that asymptotically achieves this entropy limit.
Semantic communications are considered as a key component of future generation networks, where a natural problem to consider is how to efficiently extract and transmit the “semantic information”. In the case of graph data, one may view the (partitioned) structures as the information that needs to be abstracted while the concrete labeling information is considered redundant. From this point of view, our result is a step for the study of semantic compression/communication under appropriate contexts.

1.2. Related Works

Finally, we would like to point out that there are some other information metrics defined on graphs. The term “graph entropy” has been defined and used in the history. For example, graph entropy introduced by Kőrner in [30] denotes the number of bits one has to convey to resolve the ambiguity of a vertex in a graph. This notion also turns out to be useful in other areas, including combinatorics. Chromatic entropy introduced in [31] is the lowest entropy of any coloring of a graph. It finds application in zero-error source coding. We remark that the structural entropy we considered is quite different from the Kőrner graph entropy and chromatic entropy.
On the other hand, a concept of graph entropy (also called topological information content of a graph) was introduced by Rashevsky [32] and Trucco [33], and later by Mowshowitz [34,35,36,37,38,39], which is defined as a function of (the structure of) a graph and an equivalence relation defined on its vertices or edges. Such a concept is a measure of the graph itself and does not involve any probability distribution.

2. Preliminaries

2.1. Structural Entropy of Unlabeled Graphs

Now let us formally define the structural entropy given a probability distribution on unlabeled graphs. In this subsection, we use notations borrowed from [25].
Given an integer n, define G n as the collection of all n-vertex labeled graphs.
Definition 1
(Entropy of Random Graph). Given an integer n and a random graph G distributed over G n , the entropy of G is defined as:
H G = E [ log P ( G ) ] = G G n P ( G ) log P ( G )
where P ( G ) P ( G = G ) is the probability of a graph G in G n .
Then the random structure model S n associated with the probability distribution G n , is defined as the unlabeled version of G n . For a given S S n , the probability of S can be computed as:
P ( S ) = G S , G G n P ( G ) .
Here G S means that G and S have the same structure, that is, S is isomorphic to G. Clearly if all isomorphic labeled graphs have the same probability, then for any labeled graph G S , one has:
P ( S ) = N ( S ) · P ( G )
where N ( S ) stands for the number of different labeled graphs that have the same structure as S.
Definition 2
(Structural Entropy). The structural entropy H S of a random graph G is defined as the entropy of a random structure S associated with G n , that is,
H S = E [ log P ( S ) ] = S S P ( S ) log P ( S )
where the sum is over all distinct structures.
The Erdos–Rényi random graph G ( n , p ) , also called the binomial random graph, is a fundamental random graph model, which has n vertices and each pair of vertices is connected with probability p, independent of other pairs. In 2012, Choi and Szpankowski [25] proved the following for the Erdős–Rényi random graphs.
Theorem 1
(Choi and Szpankowski, [25]). For large n and all p satisfying n 1 ln n p and 1 p n 1 ln n , the following holds:
1. 
The structural entropy H S of G ( n , p ) is:
H S = n 2 h ( p ) log n ! + O log n n α
for some α > 0 .
2. 
For a structure S of n vertices and ε > 0
P 1 n 2 log P ( S ) h ( p ) + log n ! n 2 < ε > 1 2 ε
where h ( p ) = p log p ( 1 p ) log ( 1 p ) is the entropy rate of a binary memoryless source.
Furthermore, they [25] also presented a compression algorithm for unlabeled graphs that asymptotically achieves the structural entropy up to an O ( n ) error term.

2.2. Stochastic Block Model–Our Result

As the ER model is not appropriate to model real networks, the stochastic block model is introduced on the assumption that vertices in a network connect independently but with probability based on their profiles, or equivalently, on their community assignment. For example, in the SBM with two communities and symmetric parameters, also known as the planted bisection model, denoted by G ( n , p , q ) , the vertex set is partitioned into two sets V 1 and V 2 , any pair of vertices inside V 1 or V 2 are connected with probability p and any pair of vertices across the clusters are connected with probability q, and all these connections are independent.
As an illuminating example, consider a context G where there are n / 2 users and n / 2 devices, and each pair of users and each pair of devices are connected with probability p, a user and a device is connected with probability q and each of these connections is independent of all other connections. Suppose that we need to compress the information of G. However, in the context it is not appropriate to view G as an unlabeled graph, that is, in addition to the structure information, it is also important to keep the “community” information – the compression also needs to encode the information that who is a user and who is device.
Definition 3
(Partition-respecting isomorphism, Partitioned Unlabeled Graphs). Let r n be integers. Suppose V is a set of n vertices and P = { V 1 , V 2 , , V r } is a partition of V into r parts. The partition-respecting isomorphism, denoted by “ P ” is defined as follows. For any two labeled graphs G and G , we write G P G if and only if G G they are isomorphic via an isomorphism function ϕ : V V such that ϕ ( V i ) = V i , for 1 i r . Then Γ P is defined as the collection of n-vertex graphs on V where we ignore the labels of vertices inside each V i , 1 i r , namely, the equivalence classes under partition-respecting isomorphism, with respect to P .
Note that every labeled graph G corresponds to a unique structure S Γ P , and we use G P S to denote this relation. Furthermore, under the above definition, general unlabeled graphs correspond to the case r = 1 .
Definition 4
(Partitioned Structural Entropy). Let V be a set of n vertices where n N . Suppose P = { V 1 , V 2 , , V r } is a partition of V into r parts and S n is a probability distribution over all partitioned unlabeled graphs on n vertices. Then the structural entropy H S associated to S n is defined by:
H S = E [ log P ( S ) ] = S S n P ( S ) log P ( S ) .
In this paper, we extend Theorem 1 to the structural entropy of the stochastic block model with any given number of blocks, and provide a compression algorithm that asymptotically matches this structural entropy. For ease of comprehension, we first give the result for the balanced bipartition case G ( n , p , q ) .
Theorem 2.
Let n be a positive even integer and let V = V 1 V 2 be a set of n vertices with | V 1 | = | V 2 | = n / 2 . Suppose G ( n , p , q ) is a probability distribution of graphs on V where every edge inside V 1 or V 2 is present with probability p and every edge between V 1 and V 2 is present with probability q, and these edges are mutually independent. For large even n and all p satisfying n 1 ln n p , q and 1 p n 1 ln n , the following holds:
(i) 
The partitioned structural entropy H S of G ( n , p , q ) is:
H S = 2 n / 2 2 h ( p ) + n 2 4 h ( q ) 2 log n 2 ! + O log n n α
for some α > 0 .
(ii) 
For a balanced bipartitioned structure S and ε > 0
P 1 n 2 log P ( S ) n 2 2 n 2 h ( p ) n 2 n 2 h ( q ) + 2 log ( n / 2 ) ! n 2 < 3 ε > 1 4 ε
where h ( p ) = p log p ( 1 p ) log ( 1 p ) is the entropy rate of a binary memoryless source.
Note that the structural entropy H S here is larger than that in Theorem 1 (even if p = q ), which reflects the fact that the SBM with “a planted (bi-)partition” contains prefixed structures, so has less symmetries than G ( n , p ) , the pure random model (For G ( n , p ) , when it is asymmetric, comparing with the completely labeled graphs, Theorem 1 saves a term as log n ! ; this saving becomes 2 log n / 2 ! for the planted balanced bipartition case in Theorem 2).

3. Proof of Theorem 2

One key ingredient in the proof of Theorem 1 in [25] is the following lemma on the symmetry of G ( n , p ) . A graph is called asymmetric if its automorphism group does not contain any permutation other than identity; otherwise it is called symmetric.
Lemma 1
(Kim, Sudakov and Vu, 2002). For all p satisfying n 1 ln n p and 1 p n 1 ln n , a random graph G G ( n , p ) is symmetric with probability O ( n w ) for any positive constant w.
Proof 
(Proof of Theorem 2). Note that every pair of vertices in V 1 or in V 2 should be considered as undistinguishable, but not the pairs of vertices in V 1 × V 2 . Recall that we write G P S for a graph G and a structure S if S represents the structure of G (with respect to the partition P ).
Let G : = G ( n , p , q ) . We first compute H G . Note that there are n 2 possible edges in G G , and we can view it as a binary sequence of length n 2 , where each digit is a Bernoulli random variable. Moreover, for edges inside V 1 or V 2 , the random variable, denoted by X 1 , has expectation p and for edges in V 1 × V 2 the random variable, denoted by X 2 , has expectation q. Thus, we have:
H G = E [ log X 1 2 n / 2 2 X 2 n 2 / 4 ] = 2 n / 2 2 E [ log X 1 ] n 2 4 E [ log X 2 ] = 2 n / 2 2 h ( p ) + n 2 4 h ( q ) .
Now write S n for the probability distribution on V over all partitioned unlabeled graphs inherited from G , namely, given S Γ P , P ( S ) = G P S P ( G ) . Let H S be the partitioned structural entropy of S n . Therefore, compared with our goal, it remains to show that:
H S H G = 2 log n / 2 ! + O log n n α .
Note that in G ( n , p , q ) , all labeled graphs G G such that G P S have the same probability P ( G ) . Thus, given a (labeled) graph G G , we have P ( G ) = P ( S ) / N ( S ) , where S S n is such that G P S . So the graph entropy of G = G ( n , p , q ) can be written as:
H G = G G P ( G ) log P ( G ) = S S n G P S , G G P ( G ) log P ( G ) = S S n G P S , G G P ( S ) N ( S ) log P ( S ) N ( S ) = S S n P ( S ) log P ( S ) N ( S ) = H S + S S P ( S ) log N ( S )
Define S [ W ] be be S restricted on W for W V . Now we split S into S 1 and S 2 , i.e., S 1 = S [ V 1 ] and S 2 = S [ V 2 ] . Write A u t ( S i ) for the automorphism group for S i , and we naturally have:
N ( S ) = ( n / 2 ) ! · ( n / 2 ) ! | A u t ( S 1 ) | | A u t ( S 2 ) | .
Combining this with (2) and (3), it remains to show that:
S S P ( S ) log | A u t ( S 1 ) | | A u t ( S 2 ) | = O log n n α .
In the summation above we only need to focus on S such that either S 1 or S 2 is symmetric, as otherwise log | A u t ( S 1 ) | | A u t ( S 2 ) | = log 1 = 0 . By Lemma 1, we conclude that the probability of S restricted on V 1 or V 2 is symmetric is O ( n 1 α ) for some α > 0 , and for such S we use the trivial bound log | A u t ( S 1 ) | | A u t ( S 2 ) | 2 log ( n / 2 ) ! 2 n log n . This gives us the desired estimate in (i)
S S P ( S ) log | A u t ( S 1 ) | | A u t ( S 2 ) | 2 n log n · O ( n 1 α ) = O log n n α .
To show (ii), for a set V of n vertices and a balanced bipartition P = ( V 1 , V 2 ) of V, we define the typical set T ε n as the set of structures S on n vertices satisfying:
(a)
S is asymmetric on V 1 and V 2 , respectively;
(b)
2 2 n / 2 2 h ( p ) n 2 4 h ( q ) n 2 ε P ( G ) 2 2 n / 2 2 h ( p ) n 2 4 h ( q ) + n 2 ε , for G P S .
Denote by T 1 n and T 2 n the sets of structures satisfying the properties (a) and (b), respectively, and thus we have T ε n = T 1 n T 2 n . Firstly, by the asymmetry of G ( n , p ) (Lemma 1), we conclude that P ( T 1 n ) > 1 2 ε for large n. Secondly, we use a binary sequence of length n 2 to represent a (labeled) instance G of G ( n , p , q ) , where the first n / 2 2 bits L 1 represent the induced subgraph on V 1 , the next n / 2 2 bits L 2 represent the induced subgraph on V 2 , and finally the rest n 2 / 4 bits L 12 represent the bipartite graph on V 1 × V 2 . Since all edges of G are generated independently, both L 1 and L 2 have in expectation n / 2 2 p 1’s and the AEP property of the binary sequences implies that:
2 n / 2 2 h ( p ) n 2 ε P ( G [ V 1 ] ) , P ( G [ V 2 ] ) 2 n / 2 2 h ( p ) + n 2 ε
holds with probability at least 1 2 ε . Similarly, L 12 has in expectation ( n 2 / 4 ) q 1’s and the AEP property of the binary sequences gives that with probability at least 1 ε ,
2 n 2 4 h ( q ) n 2 ε P ( G [ V 1 , V 2 ] ) 2 n 2 4 h ( q ) + n 2 ε
Since these edges are independent, we finally conclude that (b) holds with probability at least 1 3 ε . Thus, P ( T ε n ) 1 4 ε . Now we can compute P ( S ) for S T ε n . By (a), P ( S ) = ( n / 2 ) ! ( n / 2 ) ! P ( G ) for any G S . Together with (b) and straightforward computation, the assertion of (ii) follows. □

4. SBM Compression Algorithm

Given the computation of the structural entropy, a natural next step is to design efficient compression schemes that are close to or even (asymptotically) achieve this entropy limit. Choi and Szpankowski [25] presented such an algorithm (which they named Szip) for (unlabeled) random graphs, which uses in expectation at most n 2 h ( p ) n log n + O ( n ) bits and asymptotically achieves the structural entropy given in Theorem 1. Roughly speaking, Szip greedily peels off vertices from the graph and (efficiently) store the neighborhood information. This procedure can be simply reversed but the labeling of the recovered graph may be different from the original graph, which is the reason on why a saving of the codeword length is achieved. Refinements and analysis [25] are also provided to achieve the proposed performance.
Here we give an algorithm that optimally compresses SBMs which uses the Szip algorithm as building blocks and matches the structural entropy computation in Theorem 2. The algorithm consists of two stages. It first compresses S [ V 1 ] and S [ V 2 ] using Szip and then compresses S [ V 1 , V 2 ] using an arithmetic compression algorithm with the help of Szip decoding outputs.
To give a brief description of the compression algorithm, we again use the balanced bipartition V 1 V 2 as an example. The encoding and decoding procedure of the algorithm is illustrated in Figure 1. The algorithm encodes the observed S ( n , p , q ) into a binary string as follows. It uses Szip as a subroutine to compress S [ V 1 ] and S [ V 2 ] into binary sequences L 1 and L 2 . Then, as part of the encoder, we run the Szip decoder on L 1 and L 2 to obtain decoded structures S [ V 1 ] and S [ V 2 ] , respectively. We then compress S [ V 1 , V 2 ] as a labeled bipartite graph under the vertex labeling of S [ V 1 ] and S [ V 2 ] into L 12 . This “Labeled Encoder" can be done by treating it as a binary sequence of length n 2 / 4 and compressing using a standard arithmetic encoder [40,41,42]. The concatenation of Szip algorithms and the arithmetic encoder forms the cascade encoder of our algorithm and obtains the codeword ( L 1 , L 2 , L 12 ) . Upon receiving the codeword, we decode them parallelly using Szip decoder and the arithmetic decoder. This completes our algorithm.
The main challenge in the design of our algorithm is how the decoder can retrieve the consistency between the bipartite graph S [ V 1 , V 2 ] and the decoded version of S [ V 1 ] and S [ V 2 ] . A key observation here is that since Szip is a deterministic algorithm, although it may permute the vertex labelings, its output is an invariant given the same input. Given this, our solution here is to first run Szip (both encoding and decoding) at the encoder, and obtain structures S [ V 1 ] and S [ V 2 ] , respectively. We then compress S [ V 1 , V 2 ] (as a labeled bipartite graph) under the vertex labeling of S [ V 1 ] and S [ V 2 ] . This would guarantee that the decoded structures S ^ [ V 1 ] , S ^ [ V 2 ] and S ^ [ V 1 , V 2 ] share the same vertex labeling as S [ V 1 ] and S [ V 2 ] , namely, S is recovered.
Before discussing the performance of the algorithm, we first describe some useful properties of the arithmetic compression algorithm in the following lemma. We omit the proof of the lemma, which follows from the analysis in [40,41,42] and AEP properties in [1,2].
Lemma 2.
Let L be the codeword length of the arithmetic compression algorithm when compressing a binary sequence with length m and entropy rate h. For large m, the following holds:
(i) 
The expected codeword length asymptotically achieves the entropy of the message, i.e.,
E [ L ] = m h + O ( log m ) .
(ii) 
For any ϵ > 0 ,
P ( | L E [ L ] | ϵ log m ) 1 o ( 1 ) .
(iii) 
The arithmetic algorithm runs in time O ( m ) .
The following theorem characterizes the performance of our algorithm. It is immediate from Theorem 2 in [25] (performance of Szip) and 2, we omit the detailed proofs here.
Theorem 3.
Let V = V 1 V 2 be a set of n vertices and | V 1 | = | V 2 | = n / 2 . Given a partitioned unlabeled graph S on V, let L ( S ) be the codeword length given by our algorithm. For large n, our algorithm runs in time O ( n 2 ) , and satisfies the following:
(i) 
The algorithm asymptotically achieves the structural entropy in (1) (Note that ( n / 2 ) log ( n / 2 ) = n log n + O ( n ) .), i.e.,
E [ L ( S ) ] 2 n / 2 2 h ( p ) + n 2 4 h ( q ) n log n + O ( n ) .
(ii) 
For any ϵ > 0 ,
P ( | L ( S ) E [ L ( S ) ] | ϵ n log n ) 1 o ( 1 ) .

5. General SBM with R 2 Blocks

In previous sections, we discussed the structural entropy of SBM and the compression algorithm that asymptotically achieves this structural entropy for the balanced bipartition case ( r = 2 ). The corresponding results in Theorem 2 and 3 can be easily generalized to the general r-partition case. We briefly describe the generalizations below.

5.1. Structural Entropy

Our approach can deal with general SBMs similarly. In a general SBM with r 2 parts, the transition matrix, an r × r symmetric matrix P = ( p i j ) is used to describe the probabilities between and within the communities, where two vertices u V i and v V j are connected by an edge with probability p i j [ 0 , 1 ] (i and j are not necessarily distinct). We first give the result on the computation of the partitioned structural entropy of SBM.
Theorem 4.
Fix r reals x 1 , x 2 , , x r in ( 0 , 1 ) whose sum is 1. Let V = V 1 V 2 V r be a set of n vertices with a partition into r parts such that | V i | = x i n . Let S be a partitioned structure on V with transition matrix P = ( p i j ) . For large n and all 1 i r satisfying n 1 ln n p i , i and 1 p i , i n 1 ln n , the following holds:
(i) 
The r-partitioned structural entropy H S r for S is
H S r = i = 1 r x i n 2 h p i , i + 1 i < j r x i x j n 2 h p i , j i = 1 r log x i n ! + O log n n α
for some α > 0 .
(ii) 
For ε > 0 ,
P 1 n 2 · log P ( S ) i = 1 r x i n 2 h p i , i 1 i < j r x i x j n 2 h p i , j + i = 1 r log x i n ! < 3 ε > 1 4 ε .

5.2. Compression Algorithm

The compression algorithm for a general r with vertex partition { V 1 , V 2 , , V r } can be viewed as a union of the compression algorithms for S [ V i ] and S [ V i , V j ] ( i < j { 1 , 2 , , r } ). To be more precise, we describe the algorithm as follows. It first compresses all S [ V i ] into L i using Szip. Then run the Szip decoder with input L i to obtain the decoded structure S [ V i ] . With the indices of S [ V i ] , i = 1 , 2 , , r , we can compress S [ V 1 , V 2 , , V r ] as a labeled r-partite graph into L using an arithmetic encoder. This completes the encoding procedure and gives the codewords L 1 , , L r , L , for which we concatenate together and get the final codeword. The decoding is to simply run the Szip decoders and labeled (arithmetic) decoders parallelly. The correctness of the decoding output can also be argued accordingly.
The performance of the algorithm can be obtained similar to 3 as follows.
Theorem 5.
Fix r reals x 1 , x 2 , , x r in ( 0 , 1 ) whose sum is 1. Let V = V 1 V 2 V r be a set of n vertices with a partition into r parts such that | V i | = x i n . Given a partitioned unlabeled graph S on V with transition matrix P = ( p i j ) , let L ( S ) be the codeword length given by our algorithm. For large n, our algorithm runs in time O ( n 2 ) , and satisfies the following:
(i) 
The algorithm asymptotically achieves the structural entropy in (6), i.e.,
E [ L ( S ) ] i = 1 r x i n 2 h p i , i + 1 i < j r x i x j n 2 h p i , j n log n + O ( n ) .
(ii) 
For any ϵ > 0 ,
P ( | L ( S ) E [ L ( S ) ] | ϵ n log n ) 1 o ( 1 ) .

6. Conclusions

In this paper, we defined the partitioned unlabeled graphs and partitioned structural entropy, which generalize the structural entropy for unlabeled graphs introduced by Choi and Szpankowski [25]. We then computed the partitioned structural entropy for stochastic block models and gave a compression algorithm that asymptotically achieves this structural entropy limit. As mentioned earlier, we believe that in appropriate contexts the structural information of a graph or network can be interpreted as a kind of semantic information, in which case, the communication schemes may benefit from structural compressions which considerably reduce the cost.

Author Contributions

Conceptualization, J.H., W.H. and B.B.; methodology, J.H., T.G. and Q.Z.; software, J.H. and T.G.; validation, J.H., T.G., Q.Z. and B.B.; formal analysis, J.H. and T.G.; investigation, J.H., W.H. and B.B.; resources, W.H.; data curation, T.G.; writing—original draft preparation, J.H., T.G. and Q.Z.; writing—review and editing, J.H., T.G., Q.Z., W.H. and B.B.; visualization, J.H., T.G., W.H. and B.B.; supervision, W.H., B.B. and G.Z.; project administration, W.H., B.B. and G.Z.; funding acquisition, W.H., B.B. and G.Z. All authors have read and agreed to the published version of the manuscript.

Funding

This research received no external funding.

Institutional Review Board Statement

Not applicable.

Informed Consent Statement

Not applicable.

Conflicts of Interest

The authors declare no conflict of interest.

Abbreviations

The following abbreviations are used in this manuscript:
SBMStochastic Block Models
SzipCompression algorithm in [25]
AEPAsymptotic Equipartition

References

  1. Yeung, R.W. Information Theory and Network Coding; Springer: Berlin/Heidelberg, Germany, 2008. [Google Scholar]
  2. Cover, T.M.; Thomas, J.A. Elements of Information Theory, 2nd ed.; Wiley-Interscience: Hoboken, NJ, USA, 2006. [Google Scholar]
  3. Shannon, C.E. A Mathematical Theory of Communication. Bell Syst. Tech. J. 1948, 27, 379–423. [Google Scholar] [CrossRef][Green Version]
  4. Shannon, C.E. The lattice theory of information. Trans. Ire Prof. Group Inf. Theory 1953, 1, 105–107. [Google Scholar] [CrossRef]
  5. Palla, G.; Derényi, I.; Farkas, I.; Vicsek, T. Uncovering the overlapping community structure of complex networks in nature and society. Nature 2005, 435, 814–818. [Google Scholar] [CrossRef][Green Version]
  6. Holland, P.W.; Laskey, K.B.; Leinhardt, S. Stochastic blockmodels: First steps. Social Netw. 1983, 5, 109–137. [Google Scholar] [CrossRef]
  7. Abbe, E.; Bandeira, A.S.; Hall, G. Exact recovery in the stochastic block model. IEEE Trans. Inf. Theory 2015, 62, 471–487. [Google Scholar] [CrossRef][Green Version]
  8. Mossel, E.; Neeman, J.; Sly, A. Consistency thresholds for the planted bisection model. In Proceedings of the Forty-Seventh Annual ACM Symposium on Theory of Computing, Portland, OR, USA, 14–17 June 2015; pp. 69–75. [Google Scholar]
  9. Abbe, E.; Sandon, C. Community detection in general stochastic block models: Fundamental limits and efficient algorithms for recovery. In Proceedings of the 2015 IEEE 56th Annual Symposium on Foundations of Computer Science, Berkeley, CA, USA, 17–20 October 2015; pp. 670–688. [Google Scholar]
  10. Hajek, B.; Wu, Y.; Xu, J. Information limits for recovering a hidden community. IEEE Trans. Inf. Theory 2017, 63, 4729–4745. [Google Scholar] [CrossRef]
  11. Bianconi, G. Entropy of network ensembles. Phys. Rev. E 2009, 79, 036114. [Google Scholar] [CrossRef][Green Version]
  12. Peixoto, T.P. Entropy of stochastic blockmodel ensembles. Phys. Rev. E 2012, 85, 056122. [Google Scholar] [CrossRef][Green Version]
  13. Abbe, E. Community detection and stochastic block models: Recent developments. J. Mach. Learn. Res. 2017, 18, 6446–6531. [Google Scholar]
  14. Asadi, A.R.; Abbe, E.; Verdú, S. Compressing data on graphs with clusters. In Proceedings of the 2017 IEEE International Symposium on Information Theory (ISIT), Aachen, Germany, 25–30 June 2017; pp. 1583–1587. [Google Scholar] [CrossRef]
  15. Zenil, H.; Kiani, N.A.; Tegnér, J. A Review of Graph and Network Complexity from an Algorithmic Information Perspective. Entropy 2018, 20, 551. [Google Scholar] [CrossRef][Green Version]
  16. Turán, G. On the succinct representation of graphs. Discr. Appl. Math. 1984, 8, 289–294. [Google Scholar] [CrossRef][Green Version]
  17. Harary, F.; Palmer, E.M. Graphical Enumeration; Academic Press: Cambridge, MA, USA, 1973. [Google Scholar]
  18. Naor, M. Succinct representation of general unlabeled graphs. Discr. Appl. Math. 1990, 28, 303–307. [Google Scholar] [CrossRef][Green Version]
  19. Kieffer, J.C.; Yang, E.H.; Szpankowski, W. Structural complexity of random binary trees. In Proceedings of the 2009 IEEE International Symposium on Information Theory (ISIT), Seoul, Korea, 28 June–3 July 2009; pp. 635–639. [Google Scholar]
  20. Adler, M.; Mitzenmacher, M. Towards compressing Web graphs. In Proceedings of the DCC 2001. Data Compression Conference, Snowbird, Utah, 27–29 March 2001; pp. 203–212. [Google Scholar]
  21. Chierichetti, F.; Kumar, R.; Lattanzi, S.; Mitzenmacher, M.; Panconesi, A.; Raghavan, P. On Compressing Social Networks. In Proceedings of the 15th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, New York, NY, USA, 28 June–1 July 2009; pp. 219–228. [Google Scholar]
  22. Peshkin, L. Structure induction by lossless graph compression. In Proceedings of the 2007 Data Compression Conference (DCC’07), Snowbird, Utah, 27–29 March 2007; pp. 53–62. [Google Scholar]
  23. Savari, S.A. Compression of words over a partially commutative alphabet. IEEE Trans. Inf. Theory 2004, 50, 1425–1441. [Google Scholar] [CrossRef]
  24. Sun, J.; Bollt, E.; Ben-Avraham, D. Graph Compression-Save Information by Exploiting Redundancy. J. Statist. Mechan. Theory Exper. 2008, 2008, 06001. [Google Scholar] [CrossRef]
  25. Choi, Y.; Szpankowski, W. Compression of Graphical Structures: Fundamental Limits, Algorithms, and Experiments. IEEE Trans. Inf. Theory 2012, 58, 620–638. [Google Scholar] [CrossRef][Green Version]
  26. Łuczak, T.; Magner, A.; Szpankowski, W. Compression of Preferential Attachment Graphs. In Proceedings of the 2019 IEEE International Symposium on Information Theory (ISIT), Paris, France, 7–12 July 2019; pp. 1697–1701. [Google Scholar]
  27. Łuczak, T.; Magner, A.; Szpankowski, W. Asymmetry and structural information in preferential attachment graphs. Random Struct. Algorithms 2019, 55, 696–718. [Google Scholar] [CrossRef][Green Version]
  28. Sauerhoff, M. On the Entropy of Models for the Web Graph. 2016. Available online: https://www.researchgate.net/publication/255589046OntheEntropyofModelsfortheWebGraph (accessed on 1 November 2021).
  29. Kontoyiannis, I.; Lim, Y.H.; Papakonstantinopoulou, K.; Szpankowski, W. Symmetry and the Entropy of Small-World Structures and Graphs. In Proceedings of the 2021 IEEE International Symposium on Information Theory (ISIT), Victoria, Australia, 11–16 July 2021; pp. 3026–3031. [Google Scholar]
  30. Körner, J. Coding of an information source having ambiguous alphabet and the entropy of graphs. In 6th Prague Conference on Information Theory; Walter de Gruyter: Berlin, Germany, 1973; pp. 411–425. [Google Scholar]
  31. Alon, N.; Orlitsky, A. Source coding and graph entropies. IEEE Trans. Inf. Theory 1996, 42, 1329–1339. [Google Scholar] [CrossRef][Green Version]
  32. Rashevsky, N. Life information theory and topology. Bull. Math. Biophys. 1955, 17, 229–235. [Google Scholar] [CrossRef]
  33. Trucco, E. A note on the information content of graphs. Bull. Math. Biophys. 1956, 18, 129–135. [Google Scholar] [CrossRef]
  34. Mowshowitz, A. Entropy and the complexity of the graphs I: An index of the relative complexity of a graph. Bull. Math. Biophys. 1968, 30, 175–204. [Google Scholar] [CrossRef] [PubMed]
  35. Mowshowitz, A. Entropy and the complexity of graphs II: The information content of digraphs and infinite graphs. Bull. Math. Biophys. 1968, 30, 225–240. [Google Scholar] [CrossRef]
  36. Mowshowitz, A. Entropy and the complexity of graphs III: Graphs with prescribed information content. Bull. Math. Biophys. 1968, 30, 387–414. [Google Scholar] [CrossRef]
  37. Mowshowitz, A. Entropy and the complexity of graphs IV: Entropy measures and graphical structure. Bull. Math. Biophys. 1968, 30, 533–546. [Google Scholar] [CrossRef]
  38. Mowshowitz, A.; Dehmer, M. Entropy and the Complexity of Graphs Revisited. Entropy 2012, 14, 559–570. [Google Scholar] [CrossRef]
  39. Dehmer, M.; Mowshowitz, A. A History of Graph Entropy Measures. Inf. Sci. 2011, 181, 57–78. [Google Scholar] [CrossRef]
  40. Pasco, R.C. Source Coding Algorithms for Fast Data Compression. Ph.D. Thesis, Stanford University, Stanford, CA, USA, 1976. [Google Scholar]
  41. Rissanen, J.J. Generalized Kraft Inequality and Arithmetic Coding. IBM J. Res. Dev. 1976, 20, 198–203. [Google Scholar] [CrossRef]
  42. Willems, F.; Shtarkov, Y.; Tjalkens, T. The context-tree weighting method: Basic properties. IEEE Trans. Inf. Theory 1995, 41, 653–664. [Google Scholar] [CrossRef][Green Version]
Figure 1. Illustration of compression algorithm.
Figure 1. Illustration of compression algorithm.
Entropy 24 00081 g001
Publisher’s Note: MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Share and Cite

MDPI and ACS Style

Han, J.; Guo, T.; Zhou, Q.; Han, W.; Bai, B.; Zhang, G. Structural Entropy of the Stochastic Block Models. Entropy 2022, 24, 81. https://doi.org/10.3390/e24010081

AMA Style

Han J, Guo T, Zhou Q, Han W, Bai B, Zhang G. Structural Entropy of the Stochastic Block Models. Entropy. 2022; 24(1):81. https://doi.org/10.3390/e24010081

Chicago/Turabian Style

Han, Jie, Tao Guo, Qiaoqiao Zhou, Wei Han, Bo Bai, and Gong Zhang. 2022. "Structural Entropy of the Stochastic Block Models" Entropy 24, no. 1: 81. https://doi.org/10.3390/e24010081

Note that from the first issue of 2016, MDPI journals use article numbers instead of page numbers. See further details here.

Article Metrics

Back to TopTop