Entropy and the Complexity of Graphs Revisited

Abstract

This paper presents a taxonomy and overview of approaches to the measurement of graph and network complexity. The taxonomy distinguishes between deterministic (e.g., Kolmogorov complexity) and probabilistic approaches with a view to placing entropy-based probabilistic measurement in context. Entropy-based measurement is the main focus of the paper. Relationships between the different entropy functions used to measure complexity are examined; and intrinsic (e.g., classical measures) and extrinsic (e.g., Körner entropy) variants of entropy-based models are discussed in some detail.

1. Introduction

Entropy applied to graphs is one of two major approaches to measuring the complexity of these protean mathematical structures. The aim of this paper is to classify and contextualize the various entropy-based measures that have been proposed since Rashevsky introduced the concept of topological information content in the mid-1950s [1]. Contemplating the complexity of graphs conjures up the parable of the blind men and the elephant. The approach taken depends on the objective and experience of the observer. Since the advent and spectacular growth of the Internet, networks have become a lightening rod for scientific investigation [2,3]. Efforts to assess and measure the complexity of networks are part of this gathering wave of interest. However, studies of graph complexity have a longer history and have proven important in a number of fields ranging from chemistry [4,5] to sociology [6,7]. Information-theoretic measures have been used to analyze ecological networks such as food webs [8,9]. Related measures have been applied to labeled chemical structures possessing bond types and hetero-atoms [10]. In this application a measure is used together with special weighting schemes allowing for efficient classification of chemical structures with the aid of support vector machines. The same type of measure can be applied to an arbitrary weighted graph [10].

The wide applicability of graphs and the absence of a complete set of invariants apart from the definition pose a great challenge to crafting useful measures of complexity. Nevertheless, such measures have been defined, applied and found useful [7,11,12,13,14,15]. Note that there has also been considerable effort in applying various kinds of graph entropies in network physics. For related work, see [16,17,18,19]. In this paper we present a systematic interpretation of entropy based measures of graph complexity.

To understand the significance and the limitations of the entropy approach we must first situate it in the broader context of efforts to characterize the complexity of graphs. Thus we begin by describing a general taxonomy and identifying the major threads of research in each area. Then we turn our attention to a detailed analysis of the different sub-branches of the taxonomy that rely on the use of the entropy concept.

2. Taxonomy

Two very different approaches to measuring the complexity of graphs have been developed. One may be termed deterministic, the other probabilistic.

• The deterministic category encompasses the encoding, substructure count and generative approaches. Dominant in the encoding approach is Kolmogorov complexity. The second includes measures which count the number of substructures of a specified kind [20]. Generative approaches consist of measures based on operations required to generate a graph [21].
• The probabilistic category includes measures that apply an entropy function to a probability distribution associated with a graph. This category is subdivided into intrinsic and extrinsic subcategories. Intrinsic measures use structural features of a graph to partition the graph (usually the set of vertices or edges) and thereby determine a probability distribution over the components of the partition. Extrinsic measures impose an arbitrary probability distribution on graph elements [22]. Both of these categories employ the probability distribution to compute an entropy value. Shannon’s entropy function is most commonly used, but several different families of entropy functions have been considered [23]. In the next section, we provide a brief overview of the main subcategories of the deterministic class of complexity measures. The probabilistic category is our main concern and will be examined in more detail in subsequent sections.

Table 1 summarizes the taxonomy described above.

Table 1. Taxonomy of graph complexity measures.

Deterministic Measures	Probabilistic Measures
Encoding	Intrinsic (probability distribution derived from structural features)
Substructure Count	Extrinsic (probability distribution externally imposed)
Generative

Table 1. Taxonomy of graph complexity measures.

Deterministic Measures	Probabilistic Measures
Encoding	Intrinsic (probability distribution derived from structural features)
Substructure Count	Extrinsic (probability distribution externally imposed)
Generative

3. Deterministic Complexity Measures

First, we examine the encoding variety of deterministic measures of graph complexity. The principal encoding-based measure is Kolmogorov complexity which applies to any object that can be represented mathematically. For a given encoding scheme, the Kolmogorov measure assigns an object a complexity value equal to the length of the word (i.e., the number of characters taken from the code alphabet) needed to encode that object [24]. The length of a code word for an object is independent of the coding scheme in the sense that the length differs by a constant from one alphabet to another [24]. Kolmogorov complexity can be applied to graphs in a straightforward way, see, e.g., [21,25]. A graph may be defined by an adjacency list. In particular an undirected graph with n vertices is completely described by a binary string of length $C(n,2)$ where each bit in the string indicates whether or not a particular pair of vertices is adjacent. Thus the Kolmogorov complexity of an n-vertex undirected graph with respect to a binary encoding scheme is at most $C(n,2)$. This figure can be treated as an upper bound since additional information about a graph (such as the symmetry structure) could be used to reduce the length of an adjacency list encoding [21].

An alternative encoding scheme for an n-vertex undirected graph could be obtained by labeling the vertices with the numbers 1 to n, and assigning a code word to each vertex whose length equals the size needed to encode the number n. The encoding for the entire graph would be the code word for n concatenated with the code word pairs corresponding to the edges in the graph. For example, the 4-vertex graph with five edges shown in Figure 1 could be assigned the following code word consisting of $33=3+5·(2·3)$ binary digits: 100001010001011001100010011011100. In general, the length of such a code for an n-vertex graph is given by $\left(2\right|E|+1)·C\left(n\right)$ where $\left|E\right|$ is the number of edges and $C\left(n\right)$ is the length of the code word for the number n. If there are relatively few edges in the graph this scheme might give a smaller code word length than the former code. Note, however, that the code word length for both schemes is fully determined by the number of vertices and edges.

A number of substructure measures to determine graph complexity have been proposed [12,26,27]. These measures are based on the substructure count of a given type in the graph. One such measure counts the number of distinct induced subgraphs. For example, the cycle $C_4$ has four distinct induced subgraphs and so does the complete graph $K_4$. Both of these graphs therefore have complexity 4 according to this measure. In general, it has been claimed that the more complex a graph is, the more subgraphs the graph contains [26,27]. Constantine [20] defined the complexity of a graph to be the number of its spanning trees. In addition, Bonchev [27] defined the subgraph count $SC$ for determining graph complexity as

$$SC\left(G\right):=\sum _{k=0}^{\left|E\right|}{}^{k}SC$$

(1)

In particular, ${}^{2}SC$ is the known Platt index [28] counting the number of two-edge subgraphs. A similar concept based on the use of information theory is incorporated in the overall complexity indices [29]. For additional related work, see [29,30,31].

Generative measures of graph complexity have also been developed. Such a measure specifies (i) a base set of elementary graph structures, and (ii) a set of operations that allow for combining elements of the base set to generate a subgraph H of G. The complexity of a graph G is the number of operations on the elements of the base required to generate H. The boolean functions approach to measuring complexity in [32] provides an example. Here complexity is defined as the minimum number of union and intersection operations required to obtain the complete edge set of a network given a set of generator graphs isomorphic to stars.

Figure 1. Variant Kolmogorov Encoding of a graph.

Figure 1. Variant Kolmogorov Encoding of a graph.

4. Probabilistic Measures of Graph Complexity

Probabilistic measures are typically associated with particular structural features of a graph. There are two different types of such measures, namely, intrinsic and extrinsic, both of which associate a probability distribution with elements (e.g., vertices or edges) of a graph. The numerical value of these measures is usually obtained by applying an entropy function to the probability distribution. These two types of probabilistic complexity measures differ in the way the probability distribution is determined. For intrinsic measures the distribution is induced by some structural feature of the graph. In the extrinsic case an arbitrary probability distribution is assigned to elements of the graph.

The entropy function adopted by Shannon [33] to analyze the performance of communication channels is the one most commonly used to measure the complexity of graphs. This particular function has been adopted because it satisfies the following three basic requirements of a measure of information interpreted as uncertainty removal: the amount of uncertainty removed by the receipt of messages from independent sources is the sum of their respective uncertainty removals; the amount of uncertainty removed by equiprobable messages increases monotonically with the number of messages; and the function is ‘well behaved’. More precisely, let $H\left(X\right)$ be a function of the random variable X with probability distribution $P_X=\{p_1,\cdots ,p_n\}$. Suppose $H\left(X\right)$ satisfies the following three properties: (i) $H(X,Y)=H\left(X\right)+H\left(Y\right)$, if X and Y are independent random variables; (ii) if $P_X$ is a uniform distribution, i.e., $p_i=1/n$, then $H\left(X\right)$ increases monotonically with n; and (iii) $H\left(X\right)$ is a continuous function of $P_X$. These three properties determine a function of the form $H\left(X\right)=-C{\sum }_{i=1}^n{p}_{i}log{p}_i$, where the constant C determines the base of the logarithm [34].

Shannon’s entropy function is a special case of Renyi entropy. The latter designates a family of functions determined by properties (ii) and (iii) only, i.e., by removing the additivity requirement. Renyi entropy has the form: $H_q\left(X\right)=\frac{1}{1-q}ln{\sum }_{i=1}^n{p}_i^q$. The limit of $H_q\left(X\right)$ as $q\to 1$ is the Shannon entropy function [34]. Another generalization of Shannon entropy is Kendall information content, which defines a family of functions that play a role in statistical analysis. Any of these functions could be used to measure the complexity of a graph [23]. All of them share a characteristic ability to distinguish between ensembles of events based on their respective probability distributions. For further discussion, see [6,35].

4.1. Classical Graph Entropies

The first measure of graph complexity to appear in the literature was an intrinsic measure based on the symmetries of a graph [1,15]. This measure of complexity equated “information content” with the Shannon entropy of a finite probability scheme associated with the graph. In general, such classical measures are defined in the following way. If X is an arbitrary graph invariant, τ an equivalence criterion for inducing equivalence classes $X_i$, the resulting graph entropy is

$$I(G,\tau ):=-\sum _{i=1}^k\frac{|X_i|}{\left|X\right|}log\left(\frac{|X_i|}{\left|X\right|}\right)$$

(2)

The quantity $\frac{|X_i|}{\left|X\right|}$ can be interpreted as a probability value for $X_i$. Mowshowitz [15] formalized the notion of topological information content introduced by Rashevsky [1] and traced the properties of the measure. This measure $I_a\left(G\right)$ is an index based on the orbits of the automorphism group of a graph.

$$I_a\left(G\right):=-\sum _{i=1}^k\frac{|V_i|}{\left|V\right|}log\left(\frac{|V_i|}{\left|V\right|}\right)$$

(3)

where $|V_i|$ is the cardinality of the i-th orbit of G, and k is the number of different orbits.

Another entropy-based measure, defined relative to the chromatic structure of a graph is called the chromatic information content [15,36]. This measure is defined as

$$I_c\left(G\right):=\underset{\widehat{V}}{min}\left\{-\sum _{i=1}^h\frac{n_i\left(\widehat{V}\right)}{\left|V\right|}log\left(\frac{n_i\left(\widehat{V}\right)}{\left|V\right|}\right)\right\}$$

(4)

where $\widehat{V}=\left\{V_i\right|1\le i\le h\},\left|V_i\right|=n_i\left(\widehat{V}\right)$ denotes an arbitrary chromatic decomposition of a graph G, and $h=\chi \left(G\right)$ is the chromatic number of G. Using simple graph invariants such as metrical properties and degrees, various other graph entropies such as the radial centric information measures [5]

$$I_{\text{cr}}\left(G\right):=\sum _{i=1}^{\rho }\frac{|N_i^{{\sigma }_v}|}{\left|V\right|}log\left(\frac{|N_i^{{\sigma }_v}|}{\left|V\right|}\right)$$

(5)

and the vertex degree equality-based information measure [5]

$$I_{\delta }\left(G\right):=\sum _{i=1}^{\overline{\delta }}\frac{|N_i^{{\delta }_v}|}{\left|V\right|}log\left(\frac{|N_i^{{\delta }_v}|}{\left|V\right|}\right)$$

(6)

have been developed. $|N_i^{{\sigma }_v}|$ is the number of vertices having the same eccentricity (σ). $|N_i^{{\delta }_v}|$ is the number of vertices with degree equal to i and $\overline{\delta }:={max}_{v\in V}{\delta }_v$. ${\delta }_v$ is the degree of $v\in V$. Bonchev [5] offers another alternative using weighted probability schemes based on such characteristics as distances and degrees resulting in so-called magnitude-based graph entropy measures. The following is an example.

$$I_D\left(G\right):=-\frac{1}{\left|V\right|}log\left(\frac{1}{\left|V\right|}\right)-\sum _{i=1}^{\rho \left(G\right)}\frac{2k_i}{{\left|V\right|}^2}log\left(\frac{2k_i}{{\left|V\right|}^2}\right)$$

(7)

where $k_i$ denotes the number of pairs of vertices at distance i from each other.

Extensions and related work on these measures can be found in [37,38].

The essential defining characteristic of classical entropy based measures is sensitivity to particular structural features of a graph. These measures differentiate graphs according to the structural feature (e.g., vertices, edges, degrees) used to partition the elements of the graph. A finite probability scheme consisting of the orbits of the automorphism group gives rise to a measure (topological information content) that differentiates graphs with different degrees of symmetry; a scheme based on color classes produces a measure (chromatic information content) that distinguishes between graphs with different independence characteristics. The relative character of graph complexity is evident in comparing the symmetry and color class measures applied to a complete graph on n vertices. The former measure gives the minimum value of 0 (since the group has but one orbit), while the latter gives the maximum value of $logn$ (since at least n colors are required). Clearly, these two different structural measures respond differently to increasing edge density. Topological information content is useful in applications (e.g., chemistry) [12,16] in which reactivity or complementarity of structures plays a central role. Chromatic information content is suited to applications in which the identification of independent sets of elements is the main desideratum.

4.2. Körner Entropy

Körner [40] introduced the first extrinsic measure of graph complexity, now appropriately called Körner entropy, which is given by

$$H(G,P):=\underset{t⟶\infty }{lim}\underset{U\subseteq V^t,P^t\left(U\right)>1-ϵ}{min}\frac{1}{t}log\left(\chi \left(G^t\left(U\right)\right)\right)$$

(8)

For $V^{\prime }\subseteq V\left(G\right)$, the induced subgraph on $V^{\prime }$ is denoted by $G\left(V^{\prime }\right)$ and $\chi \left(G\right)$ is the chromatic number [39] of G, $G^t$ the t-th co-normal power [40] of G and

$$P^t\left(U\right):=\sum _{x\in U}{P}^t\left(x\right)$$

(9)

Note that this quantity does not express the structural information content of a graph (see previous section) because it was developed as a solution of a coding problem in information theory. There exist several definitions of Körner’s entropy which have been proven equivalent [41]. According to the Körner’s first definition [40], this quantity is related to the stable set problem which in turn is related to minimum entropy colorings of graphs whose computation is known to be NP-hard, see [42]. This means that Körner entropy does not offer a practical tool for measuring the complexity of large scale networks. Because the sub-additivity inequality holds for this measure, it has several applications such as quantum sorting, perfect hashing, and obtaining lower bounds for the size of boolean formulas, see [41]. Further examples and more information-theoretic interpretations of this measure can be found in [40,41].

4.3. Parametric Graph Entropies

Another way to define extrinsic complexity measures for graphs using Shannon’s entropy is associated with assigning a probability value to each vertex by employing functions which capture structural information of a graph [14,22,43]. Various information functions have been defined based on metrical properties and other graph invariants. Typically, such functions are parameterized by weight differences and characterize structural spread in a graph [22]. Also, the parametrization of the measures allows for defining optimization problems based on given data sets.

Let $G=(V,E)$ be a graph and let f be an information function representing a positive function that maps vertices to the positive reals using structural features of a graph. Then, one obtains the measures [14,22,43]

$$\begin{array}{cc}\hfill I_f\left(G\right)& :=-\sum _{i=1}^{\left|V\right|}\frac{f\left(v_i\right)}{{\sum }_{j=1}^{\left|V\right|}f\left(v_j\right)}log\left(\frac{f\left(v_i\right)}{{\sum }_{j=1}^{\left|V\right|}f\left(v_j\right)}\right)\hfill \end{array}$$

(10)

$$\begin{array}{cc}\hfill I_{f_{☆}^V}^{\lambda }\left(G\right)& :=\lambda \left(log\left(\right|V\left|\right)+\sum _{i=1}^{\left|V\right|}\frac{f\left(v_i\right)}{{\sum }_{j=1}^{\left|V\right|}f\left(v_j\right)}log\left(\frac{f\left(v_i\right)}{{\sum }_{j=1}^{\left|V\right|}f\left(v_j\right)}\right)\right)\hfill \end{array}$$

(11)

representing families of graph entropy measures [14,22,43]. The vertex probabilities are defined by

$$p\left(v_i\right):=\frac{f\left(v_i\right)}{{\sum }_{j=1}^{\left|V\right|}f\left(v_j\right)}$$

(12)

In order to derive concrete graph entropies, the parametric information functions [14,22,43]

$$\begin{array}{c}\hfill f_1\left(v_i\right):={\alpha }^{c_1|S_1(v_i,G)|+c_2|S_2(v_i,G)|+\cdots +c_{\rho \left(G\right)}\left|S_{\rho \left(G\right)}(v_i,G)\right|}\\ \hfill c_k>0,1\le k\le \rho \left(G\right),\alpha >0\end{array}$$

(13)

and

$$\begin{array}{c}\hfill f_2\left(v_i\right):=c_1|S_1(v_i,G)|+c_2|S_2(v_i,G)|+\cdots +c_{\rho \left(G\right)}\left|S_{\rho \left(G\right)}(v_i,G)\right|\\ \hfill c_k>0,1\le k\le \rho \left(G\right)\end{array}$$

(14)

have been used. Criteria for selecting the parameters have been examined in [43]. Clearly, the interpretation of the resulting entropy depends on the information function chosen. When using $f_1$, Dehmer et al. found that $I_{f_1}$ measures a special kind of inner symmetry of the graph under consideration. This observation is based on the fact that the more the vertices differ with respect to their spherical neighborhoods, the smaller is the value of the measure and conversely [44].

Parametric entropy based measures rely on information functions to assign probabilities to the vertices of a graph. These vertex probabilities are then used to construct a finite probability scheme for a graph. This procedure gives greater scope to the definition of entropy based structural measures. Measures that are sensitive to very specific structures in a graph can be designed by choosing an appropriate information function. For example, the information function $f_2$ in Equation (10) gives rise to a measure $I_{f_2}$ that can discriminate reasonably well between non-isomorphic graphs [43]. In particular, the resulting measure has low degeneracy [43] for chemical graphs and exhaustively generated graphs up to 10 vertices.

4.4. Non-Parametric Graph Entropies

A novel information function is

$$f_3\left(v_i\right):=\left|{\lambda }_i\right|$$

(15)

leading to the extrinsic graph entropy [45]

$$I_{f_3}\left(G\right)=-\sum _{i=1}^k\frac{|{\lambda }_i|}{{\sum }_{j=1}^{\left|V\right|}\left|{\lambda }_j\right|}log\left(\frac{|{\lambda }_i|}{{\sum }_{j=1}^{\left|V\right|}\left|{\lambda }_j\right|}\right)$$

(16)

where G is an undirected graph and ${\lambda }_1,{\lambda }_2,\dots ,{\lambda }_k$ are the non-zero eigenvalues of its characteristic polynomial. Note that this non-parametric graph complexity measure quantifies the entropy of the underlying graph topology based on the spectrum of G. Obviously, $f_3$ can also be calculated by using the eigenvalues of various other graph polynomials such as the distance polynomial and Wiener polynomial [46,47]. This leads to graph entropies which have yet to be explored. Other than $f_3$, the information function

$$f_4\left(v_i\right):=d(v_i,v_1)+d(v_i,v_2)+\cdots +d(v_i,v_{\left|V\right|})$$

(17)

is also non-parametric but is based on metrical properties of an underlying graph.

Non-parametric differ from parametric entropy based measures in the way their respective finite probability schemes are constructed. Schemes in the latter case are directly related to the probabilities assigned to the vertices of a graph; those in the former case are only indirectly related to the probabilities assigned to the vertices. As indicated above the function $f_3\left(v_i\right)$, defined in terms of the eigenvalues of the graph-theoretical matrix of a graph (such as the adjacency and distance matrix), gives rise to a finite probability scheme consisting of the ratios of individual eigenvalues to the sum of all the eigenvalues.

Interestingly, this measure is also highly effective in distinguishing between non-isomorphic graphs, especially chemical graphs and exhaustively generated graphs [45]. The existence of cospectral graphs justifies the general observation that eigenvalue-based graph invariants are not effective in distinguishing non-isomorphic graphs. However, this result suggests the contrary for several classes of graphs [48,49].

5. Conclusions

Entropy functions have been used successfully to capture different aspects of graph complexity. Much of the work in this area has been undertaken by those directly concerned with applications in chemistry, biology and sociology, which accounts in part for the different approaches and entropy functions found in the literature. In this paper we have discussed the main entropy-based approaches to graph complexity measurement by introducing a general taxonomy. Coverage of the many different applications of entropy-based measures was not meant to be exhaustive, so, for example, the overview did not include applications to weighted graphs. Classical entropy-based measures can handle the weighted case by modifying the criterion for decomposing graph elements into equivalence classes. Elements (e.g., vertices or edges) equivalent by virtue of automorphisms of the underlying unweighted graph elements can be differentiated by applying numerical criteria. If, for example, edges e and f are equivalent under an edge automorphism, they might yet be distinguished if the difference in their respective weights is greater than some given threshold value. Proceeding in this way, a finite probability scheme could be constructed to which an entropy function is applied.

Formally speaking, intrinsic and extrinsic measures of graph complexity differ in the underlying finite probability scheme used for applying an entropy function to obtain a quantitative value. Both types of measures attempt to differentiate graphs according to some structural feature. The classical measures use global structures such as symmetry or independent sets of graph elements. Topological information content [1], for example, relies on symmetry to distinguish graphs representing the molecular structure of chemical compounds. Different structures give rise to different classifications of graphs. Extrinsic measures extend the possibilities for classifying graphs according to structural features. Parametric measures make explicit use of probabilities assigned to vertices to construct finite probability schemes. Non-parametric measures abstract the construction of such schemes by using values only indirectly related to the probabilities assigned to the vertices. The latter measures extend the possibilities for crafting “designer” measures that differentiate graphs according to ever more subtle structural features. All the entropy based measures are linked to particular structural features and are thus limited in their ability to differentiate graphs. The example given earlier showing the extreme difference in the respective values assigned to the complete graph by topological information content and chromatic information content illustrates this assertion. Those wishing to classify graphs using entropy-based measures must be content with an ordering relative to some particular structural feature of interest.