Information Theory of Networks

Abstract

The paper puts the emphasis on surveying information-theoretic network measures for analyzing the structure of networks. In order to apply the quantities interdisciplinarily, we also discuss some of their properties such as their structural interpretation and uniqueness.

1. Introduction

Information theory has been proven useful to solve interdisciplinary problems. For example, problems in biology, chemistry, computer science, ecology, electrical engineering, and neuroscience have been tackled by using information-theoretic methods such as entropy and mutual information, see [1,2,3,4,5]. In particular, advanced information-measures such as the Jensen–Shannon divergence have also been used for performing biological sequence analysis [6].

In terms of investigating networks, information-theoretic techniques have been applied in an interdisciplinary manner [7,8,9,10,11]. In this paper, we put the emphasis on reviewing information-theoretic measures to explore the network structure and shed light on some of their strong and weak points. But note that the problem of exploring the dynamics of networks by using information theory has also been tackled, see [12].

Interestingly, the problem of exploring graphs quantitatively emerged in the fifties when investigating structural aspects of biological and chemical systems [13,14]. In this context, an important problem is to quantify the structural information content of graphs by using Shannon’s information measure [14,15,16,17,18,19]. This groundbreaking work led to numerous measurements of network complexity by using Shannon’s information measure [15,20,21,22]. Particularly this task firstly appeared when studying the complexity of chemical and biological systems [13,23,24,25]. Besides studying chemical and biological questions, the structural complexity of networks have been also explored in computer science [1,26], ecology [10,27,28,29], information theory [30], linguistics [31,32], sociology [33,34], and mathematical psychology [33,35]. Also, information-theoretic approaches for investigating networks have been employed in network physics, see [7,36,37].

As the measures have been explored interdisciplinarily, it is particularly important to understand their strong and weak points. Otherwise, the results of applications involving the measures can not be understood properly. Besides surveying the most important measures, the main contribution is to highlight some of their strong and weak points. In this paper, this relates to better understand their structural interpretation and to gain insights about their uniqueness. The uniqueness, often called the discrimination power or degeneracy of an information-theoretic graph measure (and of course of any graph measure) relates to the property how well it can discriminate non-isomorphic graphs by its values, see [38,39,40]. An important problem is to evaluate the degree of degeneracy of a measure by several quantities such as the sensitivity measure due to Konstantinova [40]. Note that the discrimination power of a measure clearly depends on the graph class in question, see [38,41].

2. Graph Entropies

2.1. Measures Based on Equivalence Criteria and Graph Invariants

To find such measures, seminal work was done by Bonchev [8,42], Mowshowitz [15,16,17,18], Rashevsky [19] and Trucco [14]. Chronologically, Rashevsky [19], MacArthur [27] and Trucco [14] were the first who applied Shannon’s information measure to derive an entropy of a graph characterizing its topology. Then, Mowshowitz [15,16,17,18] called it structural information content of a graph and developed a theory to study the properties of such graph entropies under certain graph operations such as product, join etc. So far, numerous related quantities have been defined by applying the general approach of deriving partitions based on a graph invariant which is due to Mowshowitz [15]. As a result of these developments, Bertz [43], Basak et al. [44,45] and Bonchev [8,42,46] contributed various related measures which are all based on the idea of deriving partitions by using a graph invariant, e.g., vertices, edges, degrees, and distances.

Let $G=(V,E)$ be a graph, X be a graph invariant, and $\tau$ be an equivalence criterion. Then, G can be partitioned with respect to the elements of the graph invariant under consideration. From this procedure, one also obtains probability values for each partition [8,15] given by $p_i:=\frac{|X_i|}{\left|X\right|}$. By applying Shannon’s information measure [5], we yield the graph entropies as follows [8]:

$$\begin{array}{cc}\hfill I_t(G,\tau )& :=\left|X\right|log\left(\right|X\left|\right)-\sum _{i=1}^k|X_i|log(|X_i\left|\right)\hfill \end{array}$$

(1)

$$\begin{array}{cc}\hfill I_m(G,\tau )& :=-\sum _{i=1}^k\frac{|X_i|}{\left|X\right|}log\left(\frac{|X_i|}{\left|X\right|}\right)\hfill \end{array}$$

(2)

where k equals the number of different partitions. $I_t$ is called total information content and $I_m$ is called the mean information content of G, respectively.

In the following, we survey graph entropy measures exemplarily by applying this principle. Besides well-known quantities, we also mention more recently developed indices.

• Topological information content due to Rashevsky [19]:

  $$I_a\left(G\right):=-\sum _{i=1}^k\frac{|N_i|}{\left|V\right|}log\left(\frac{|N_i|}{\left|V\right|}\right)$$

  (3)

  $|N_i|$ denotes the number of topologically equivalent vertices in the i-th vertex orbit of G. k is the number of different orbits. This measure is based on symmetry in a graph as it relies on its automorphism group and vertex orbits. It can be easily shown that $I_a$ vanishes for vertex transitive graphs. Also, it attains maximum entropy for asymmetric graphs. However, it has been shown [41] that these symmetry-based measures possess little discrimination power. The reason for this is that many non-isomorphic graphs have the same orbit structure and, hence, they can not be distinguished by this index. Historically seen, the term topological information content was proposed by Rashevski [19]. Then, Trucco [14] redefined the measure in terms of graph orbits. Finally, Mowshowitz [15] studied extensively mathematical properties of this information measure for graphs (e.g., the behavior of $I_a$ under graph operations) and generalized it by considering infinite graphs [18].
• Symmetry index for graphs due to Mowshowitz et al. [47]:

  $$S\left(G\right):=\left(log\left(\right|V\left|\right)-I_a\left(G\right)\right)+log\left(\left|\mathrm{Aut}\right(G\left)\right|\right)$$

  (4)

  In [47], extremal values of this index and formulas for special graph classes such as wheels, stars and path graphs have been studied. As conjectured, the discrimination power of S turned out to be higher than by using $I_a$ as a discriminating term $log\left(\left|\mathrm{Aut}\right(G\left)\right|\right)$ has been added, see Equation (4). In particular, we obtained this result by calculating S on a set of 2265 chemical graphs whose order range from four to nineteen. A detailed explanation of the dataset can be found in [48].
• Chromatic information content due to Mowshowitz [15,16]:

  $$I_c\left(G\right):=\underset{\widehat{V}}{min}\left\{-\sum _{i=1}^h\frac{n_i\left(\widehat{V}\right)}{\left|V\right|}log\left(\frac{n_i\left(\widehat{V}\right)}{\left|V\right|}\right)\right\}$$

  (5)

  where $\widehat{V}:=\left\{V_i\right|1\le i\le h\},\left|V_i\right|:=n_i\left(\widehat{V}\right)$ denotes an arbitrary chromatic decomposition of a graph G. $h=\chi \left(G\right)$ is the chromatic number of G. Graph-theoretic properties of $I_c$ and its behavior on several graph classes have been explored by Mowshowitz [15,16]. To our knowledge, the structural interpretation of this measure as well as the uniqueness has not yet been explored extensively.
• Magnitude-based information indices due to Bonchev et al. [49]:

  $$\begin{array}{ccc}\hfill I_D\left(G\right)& :=& -\frac{1}{\left|V\right|}log\left(\frac{1}{\left|V\right|}\right)-\sum _{i=1}^{\rho \left(G\right)}\frac{2k_i}{{\left|V\right|}^2}log\left(\frac{2k_i}{{\left|V\right|}^2}\right)\hfill \end{array}$$

  (6)

  $$\begin{array}{ccc}\hfill I_D^W\left(G\right)& :=& W\left(G\right)log\left(W\left(G\right)\right)-\sum _{i=1}^{\rho \left(G\right)}i{k}_{i}log\left(i\right)\hfill \end{array}$$

  (7)

  where $k_i$ is the occurrence of a distance possessing value i in the distance matrix of G. The motivation to introduce these measures was to find quantities which detect branching well, see [49]. In this context, branching of a graph correlates with the number of terminal vertices. By using this model, Bonchev et al. [49] showed numerically and by means of inequalities that these indices detect branching meaningfully. Also, it turned out that magnitude-based information indices possess high discrimination power for trees. But recent studies [50] have shown that the uniqueness of the magnitude-based information indices deteriorate tremendously when being applied to large sets of graphs containing cycles. More precisely, Dehmer et al. [50] evaluated the uniqueness of several graph entropy measures and other topological indices by using almost 12 million non-isomorphic, connected and unweighted graphs possessing ten vertices.
• Vertex degree equality-based information index found by Bonchev [8]:

  $$I_{\mathrm{deg}}\left(G\right):=\sum _{i=1}^{\overline{k}}\frac{|N_i^{k_v}|}{\left|V\right|}log\left(\frac{|N_i^{k_v}|}{\left|V\right|}\right)$$

  (8)

  where $|N_i^{k_v}|$ is the number of vertices with degree equal to i and $\overline{k}:={max}_{v\in V}{k}_v$. Note that this quantity is easy to determine as the time complexity of the calculation of the degrees is clearly polynomial. But it is intuitive that a simple comparison of the degree distribution of graphs is not meaningful to discriminate their structure. In [50], it has been shown that this measure possesses little discrimination power when applying the quantity to several sets of graphs.
• Overall information indices found by Bonchev [46,51]:

  $$\begin{array}{cc}\hfill OX\left(G\right)& :=\sum _{k=0}^{\left|E\right|}{}^{k}X;\left\{X\right\}:=\{{}^{0}X,{}^{1}X,\dots ,{}^{\left|E\right|}X\}\hfill \end{array}$$

  (9)

  $$\begin{array}{cc}\hfill I(G,OX)& :=OXlog\left(OX\right)-\sum _{k=0}^{\left|E\right|}{}^{k}Xlog\left({}^{k}X\right)\hfill \end{array}$$

  (10)

  The index calculates the overall value $OX$ of a certain graph invariant X by summing up its values in all subgraphs, and partitioning them into terms of increasing orders (increasing number of subgraph edges k). In the simplest case, we have $OX=SC$, i.e., it is equal to the subgraph count [51]. Several more overall indices and their informational functionals have been calculated, such as overall connectivity (the sum of total adjacency of all subgraphs), overall Wiener index (the sum of total distances of all subgraphs), the overall Zagreb indices, and the overall Hosoya index [51]. They all share (with some inessential variations) the property to increase in value with the increase in graph complexity. The properties of most of these information functionals will not be studied here in detail.

Clearly, we only surveyed a subset of existing graph entropy measures. Further measures which are based on the same criterion can be found in [51,52,53]. Also, we would like to mention that information measures for graphs based on other entropy measures have been studied [54]. For instance, Passerini and Severini [54] explored the von Neumann entropy of networks in the context of network physics. Altogether, the variety of existing network measures bears great potential for analyzing complex networks quantitatively. But in the future, the usefulness and ability of these measures must be investigated more extensively to gain further theoretical insights in terms of their properties.

2.2. Körner Entropy

The definition of the Körner entropy is rooted in information theory and has been introduced to solve a particular coding problem, see [30,55]. Simony [55] discussed several definitions of this quantity which have been proven to be equivalent. One definition thereof is

$$H(G,P):=\underset{t⟶\infty }{lim}\underset{U\subseteq V^t,P^t\left(U\right)>1-ϵ}{min}\frac{1}{t}log\left(\chi \left(G^t\left(U\right)\right)\right)$$

(11)

For $V^{\prime }\subseteq V\left(G\right)$, the induced subgraph on $V^{\prime }$ is denoted by $G\left(V^{\prime }\right)$ and $\chi \left(G\right)$ is the chromatic number [56] of G, $G^t$ the t-th co-normal power [30] of G and

$$P^t\left(U\right):=\sum _{x\in U}{P}^t\left(x\right)$$

(12)

Note that $P^t\left(x\right)$ is the probability of the string x, see [55]. Examples and the interpretation of this graph entropy measure can be found in [30,55]. Due to the fact that its calculation relies on the stable set problem, its computational complexity may be insufficient. To our knowledge, the Körner entropy has not been used as a graph complexity measure in the sense of the quantities described in the previous section. That means, it does not express the structural information content of a graph (as the previously mentioned graph entropies) as it has been used in a different context, see [30,55]. Also, its computational complexity makes it impossible to apply this quantity on a large scale and to investigate properties such as correlation and uniqueness.

2.3. Entropy Measures Using Information Functionals

Information-theoretic complexity measures for graphs can also be inferred by assigning a probability value to each vertex of a graph in question [9,21]. Such probability values have been defined by using information functionals [9,21,48]. In order to define these information functionals, some key questions must be answered:

• What kind of structural features (e.g., vertices, edges, degrees, distances etc.) should be used to derive meaningful information functionals?
• In this context, what does “meaningful” mean?
• In case the functional is parametric, how can the parameters be optimized?
• What kind of structural information does the functional as well as the resulting entropy detect?

To discuss the first item, see [9,21,48] and note that metrical properties have been used to derive such information functionals. In order to prove whether a functional as well as the resulting entropy measures captures structural information meaningfully, an optimality criterion is needed. For example, suppose there exists a data set where the class labels of its entities (graphs) are known. By employing supervised machine learning techniques, the classification error can be optimized. Note that the last item relates to investigate the structural interpretation of the graph entropy measure. Indeed, this question could be raised for any topological index.

In order to reproduce some of these measures, we start with a graph $G=(V,E)$ and let f be an information functional representing a positive function that maps vertices to the positive reals. Note that f captures structural information of G. If we define the vertex probabilities as [9,21]

$$p\left(v_i\right):=\frac{f\left(v_i\right)}{{\sum }_{j=1}^{\left|V\right|}f\left(v_j\right)}$$

(13)

we yield the families of information-theoretic graph complexity measures [9,48]:

$$\begin{array}{cc}\hfill I_f\left(G\right)& :=-\sum _{i=1}^{\left|V\right|}\frac{f\left(v_i\right)}{{\sum }_{j=1}^{\left|V\right|}f\left(v_j\right)}log\left(\frac{f\left(v_i\right)}{{\sum }_{j=1}^{\left|V\right|}f\left(v_j\right)}\right)\hfill \end{array}$$

(14)

$$\begin{array}{cc}\hfill I_f^{\lambda }\left(G\right)& :=\lambda \left(log\left(\right|V\left|\right)+\sum _{i=1}^{\left|V\right|}\frac{f\left(v_i\right)}{{\sum }_{j=1}^{\left|V\right|}f\left(v_j\right)}log\left(\frac{f\left(v_i\right)}{{\sum }_{j=1}^{\left|V\right|}f\left(v_j\right)}\right)\right)\hfill \end{array}$$

(15)

$\lambda >0$ is a scaling constant. Typical information functionals are [9,21,48]

$$f_1\left(v_i\right):={\alpha }^{c_1|S_1(v_i,G)|+c_2|S_2(v_i,G)|+\cdots +c_{\rho \left(G\right)}\left|S_{\rho \left(G\right)}(v_i,G)\right|},c_k>0,1\le k\le \rho \left(G\right),\alpha >0$$

(16)

and

$$f_2\left(v_i\right):=c_1|S_1(v_i,G)|+c_2|S_2(v_i,G)|+\cdots +c_{\rho \left(G\right)}\left|S_{\rho \left(G\right)}(v_i,G)\right|,c_k>0,1\le k\le \rho \left(G\right)$$

(17)

The parameters $c_k>0$ to weight structural characteristics or differences of G in each sphere have to be chosen such that at least $c_i\ne c_j$ holds. Otherwise the probabilities become $\frac{1}{\left|V\right|}$ leading to maximum entropy $log\left(\right|V\left|\right)$. For instance, the setting $c_1>c_2>\cdots >c_{\rho \left(G\right)}$ have often been used, see [9,21,48]. Also, other schemes for the coefficients can be chosen but need to be interpreted in terms of the structural interpretation of the resulting entropy measure. As the measures are parametric (when using a parametric information functional), they can be interpreted as generalizations of the aforementioned partition-based measures.

By applying Equation (15), concrete information measures to characterize the structural complexity chemical structures have been derived in [48]. For example, if we choose the coefficients linearly decreasing, e.g.,

$$c_1:=\rho \left(G\right),c_2:=\rho \left(G\right)-1,\dots ,c_{\rho \left(G\right)}:=1$$

(18)

or exponentially decreasing, e.g.,

$$c_1:=\rho \left(G\right),c_2:=\rho \left(G\right)e^{-1},\dots ,c_{\rho \left(G\right)}:=\rho \left(G\right)e^{-\rho \left(G\right)+1}$$

(19)

the resulting measures are called $I_{f_{lin}^V}^{\lambda }$ and $I_{f_{exp}^V}^{\lambda }$, respectively. Importantly, it turned out that $I_{f_{lin}^V}^{\lambda }$ and $I_{f_{exp}^V}^{\lambda }$ possess high discrimination power when applying them to real and synthetic chemical graphs, see [48].

To obtain more advanced information functionals, the concept outlined above has been extended in [57]. The main idea for deriving these information functionals is based on the assumption that starting from an arbitrary vertex $v_i\in V$, information spreads out via shortest paths in the graph which can be determined by using Dijkstra’s algorithm [58]. Then, more sophisticated information functionals as well as complexity measures have been defined [57] by using local property measures, e.g., vertex centrality measures [59]. In particular, some of them turned out to be highly unique when applying the measures to almost 12 million non-isomorphic, connected and unweighted graphs possessing ten vertices [50]. Interestingly, the just mentioned information-theoretic complexity measures showed a constantly high uniqueness that does not depend much on the cardinality of the underlying graph set. This property is desirable as we found that the uniqueness of the most existing measures deteriorates dramatically if the cardinality of the underlying graph set increases.

2.4. Information-Theoretic Measures for Trees

In this section, we sketch a few entropic measures which have been developed to characterize trees structurally. For example, Emmert-Streib et al. [60] developed an approach to determine the structural information content of rooted trees by using the natural partitioning of the vertices in such a tree. That means the number of vertices can be counted on each tree level which leads to a probability distribution and, thus, to an entropy characterizing the topology of a rooted tree. Dehmer [57] used this idea to calculate the entropy of arbitrary undirected graphs by applying a decomposition approach. Mehler [31] also employed entropic measures as balance and imbalance measures of tree-like graphs in the context of social network analysis. Other aspects of tree entropy have been tackled by Lions [61].

2.5. Other Information-Theoretic Network Measures

Apart from information-theoretic measures mostly used in mathematical and structural chemistry, several other entropic networks measures for measuring disorder relations in complex networks have been explored in the context of network physics, see [62]. If $P\left(k_v\right)$ denotes the probability of a vertex v possessing degree k, the distribution of the so-called remaining degree was defined by [62]

$$q\left(k_v\right):=\frac{(k+1)P_{k_v+1}}{<k>}$$

(20)

$<k>:={\sum }_{k}kP\left(k_v\right)$. By applying Shannon’s information measure, the following graph entropy measure has been obtained [62]:

$$I\left(G\right):=\sum _{i=1}^{\left|V\right|}q\left(i\right)log\left(q\left(i\right)\right)$$

(21)

It can be interpreted as a measure for determining the heterogeneity of a complex network [62]. In order to develop information indices for weighted directed networks, Wilhelm et al. [63] defined the measure called Medium Articulation that obtains its maximum for networks with a medium number of edges. It has been defined by [63]

$$\mathrm{MA}\left(G\right):=R\left(G\right)·I\left(G\right)$$

(22)

where

$$R\left(G\right):=-\sum _{i,j}{T}_{v_i{v}_j}log\left(\frac{T_{v_i{v}_j}^2}{{\sum }_k{T}_{v_k{v}_j}{\sum }_l{T}_{v_i{v}_l}}\right)$$

(23)

represents the redundancy and [63]

$$I\left(G\right):=\sum _{i,j}{T}_{v_i{v}_j}log\left(\frac{T_{v_i{v}_j}}{{\sum }_k{T}_{v_k{v}_j}{\sum }_l{T}_{v_i{v}_l}}\right)$$

(24)

the mutual information.

Finally, the normalized flux from $v_i$ to $v_j$ is

$$T_{v_i{v}_j}:=\frac{t_{v_i{v}_j}}{{\sum }_{k,l}{t}_{v_k{v}_l}}$$

(25)

$t_{v_i{v}_j}$ is the flux (edge weight) between $v_i$ and $v_j$. It can be easily shown that R vanishes for a directed ring but attains its maximum for the complete graph [63]. The behavior of I is just converse. This implies that MA vanishes for extremal graphs and attains its maximum in between [63]. We remark that a critical discussion of MA and modified measures have been recently contributed by Ulanowicz et al. [64].

For finalizing this section, we also reproduce the so-called offdiagonal complexity ($OdC$) [65] that is based on determining the entropy of the offdiagonal elements of the vertex-vertex link correlation matrix [65,66]. Let $G=(V,E)$ be a graph and let ${\left(c_{ij}\right)}_{ij}$ be the vertex-vertex link correlation matrix, see [65]. Here $c_{ij}$ denotes the number of all neighbors possessing degree $j>i$ of all vertices with degree i [66]. $\overline{k}:={max}_{v\in V}{k}_v$ stands for the maximum degree of G. If one defines [66]

$$a_{\left|V\right|}:=\sum _{i=1}^{\overline{k}-\left|V\right|}{c}_{i,i+\left|V\right|}$$

(26)

and

$$b_{\left|V\right|}:=\frac{a_{\left|V\right|}}{{\sum }_{\left|V\right|=0}^{\overline{k}-1}{a}_{\left|V\right|}}$$

(27)

$OdC$ can be defined by [66]

$$OdC:=\frac{-\left({\sum }_{\left|V\right|=0}^{\overline{k}-1}{b}_{\left|V\right|}log\left(b_{\left|V\right|}\right)\right)}{log\left(\right|V|-1)}\in [0,1]$$

(28)

As the measure depends on correlations between degrees of pairs of vertices [65], it is not surprising that its discrimination power is low, see [41].

3. Structural Interpretation of Graph Measures

We already mentioned the problem of exploring the structural interpretation of topological graph measures exemplarily in the preceding sections. In general, this relates to explore what kind of structural complexity a particular measure does detect. The following listing shows a few such types of structural complexity of measures which have already been explored:

• Branching in trees [49,67,68]. Examples for branching measures are the Wiener index [69], the magnitude-based measures also known as Bonchev–Trinajstić indices [49] and others outlined by Janežić et al. [68].
• Linear tree complexity depending on their size and symmetry [68]. Examples for such measures are the MI and MB indices, TC and TC1 Indices etc., see [68].
• Balance and imbalance of tree-like graphs [31]. For examples, see [31].
• Cyclicity in graphs [23,38,68,70,71]. Note that in the context of mathematical chemistry, this graph property has been introduced and studied by Bonchev et al. [38]. Examples for branching measures are the BT and BI Indices, and the F index, see [70].
• Inner symmetry and symmetry in graphs [15,47,48,72]. Examples for such measures are $I_a$, S (see Section (2.1)) and $I_{f_2}$ (see Section (2.3)).

In view of the vast amount of topological measures developed so far, determining their structural interpretation is a daunting problem. Evidently, it is important to contribute to this problem as measures could be then classified by this property. This might be useful when designing new measures or finding topological indices for solving a particular problem.

4. Summary and Conclusion

In this paper, we surveyed information-theoretic measures for analyzing networks quantitatively. Also, we discussed some of their properties, namely the structural interpretation and uniqueness. Because a vast number of measures have been developed, the former problem has been somewhat overlooked when analyzing topological network measures. Also, the uniqueness of information-theoretic and non-information-theoretic measures is a crucial property. Applications thereof might be interesting for applications such as problems in combinatorial chemistry [73]. In fact, many papers exist to tackle this problem [40,74,75,76] but not on a large scale. Interestingly, a statistical analysis has been recently shown [50] that the uniqueness of many topological indices strongly depends on the cardinality of a graph set in question. Also, it is clear that the uniqueness property depends on a particular graph class. This implies that results may not be generalized when the measure gives feasible results for a special class only, e.g., trees, isomers etc.