# Network Decomposition and Complexity Measures: An Information Geometrical Approach

## Abstract

**:**

## 1. Introduction

## 2. System Decomposition

**x**= (x

_{1}, ··· , x

_{n}) where x

_{i}∈ {0, 1} (1 ≤ i ≤ n). We denote the joint distribution of

**x**by p(

**x**). We define the decomposition p

^{dec}(

**x**) of p(

**x**) into two subsystems ${\mathbf{y}}^{\mathbf{1}}=\left({x}_{1}^{1},\cdots ,{x}_{{n}_{1}}^{1}\right)$ and ${\mathbf{y}}^{\mathbf{2}}=\left({x}_{1}^{2},\cdots ,{x}_{{n}_{2}}^{2}\right)$ (n

_{1}+ n

_{2}= n,

**y**∪

^{1}**y**=

^{2}**x**,

**y**∩

^{1}**y**= ϕ) as follows:

^{2}**y**) and p(

^{1}**y**) are the joint distributions of

^{2}**y**and

^{1}**y**, respectively. For simplicity, hereafter we denote the system decomposition using the smallest subscript of variables in each subsystem. For example, in case n = 4,

^{2}**y**= (x

^{1}_{1}, x

_{3}) and

**y**= (x

^{2}_{2}, x

_{4}), we describe the decomposed system p

^{dec}(

**x**) as < 1212 >. The system decomposition means to cut all statistical association between the two subsystems, which is expressed as setting the independent relation between them.

**x**), whose vertices V = {x

_{1}, ··· , x

_{n}} and edges E = V × V represent the variables and the statistical association, respectively. To express the system, we set the value of each vertex as the value of the corresponding variable, and the weight of each edge as the degree of dependency between the connected variables.

**x**) as follows:

**x**), both coordinates have the degrees of freedom ${\sum}_{k=1}^{n}{}_{n}{C}_{k}$.

**are simply the set of marginal distributions of each variable. The subcoordinates η**

^{1}**(1 < k ≤ n) include the statistical association among k variables, that can not be expressed with the coordinates less than the k-th order. This means that the different statistical associations exist independently in each order among the corresponding sets of the variables. The statistical association represented by the weight of a graph edge {x**

^{k}_{i}, x

_{j}} is therefore the superposition of the different dependencies defined on every subset of

**x**including x

_{i}and x

_{j}.

**; θ**

^{1}**; ··· ; θ**

^{2}**) that are the dual coordinates of η with respect to the Legendre transformation of the exponential family’s potential function ψ(θ) to its conjugate potential ϕ(η) as follows:**

^{n}**as follows [14].**

^{k}

^{k}^{−};0, ··· , 0) with no dependency above the k-th order. We denote the system specified with ζ

**and ${\zeta}_{\mathbf{0}}^{\mathbf{k}}$ as p(**

^{k}**x**,ζ

**) and p(**

^{k}**x**,${\zeta}_{\mathbf{0}}^{\mathbf{k}}$), respectively.

**x**,ζ): p(

**x**,${\zeta}_{\mathbf{0}}^{\mathbf{k}}$)].

_{i}, x

_{j}} with the KL-divergence, the above k-cut coordinates ζ

**are not appropriate to measure the information represented in each edge. We need to set another mixture coordinates so that to separate only the existing information between x**

^{k}_{i}and x

_{j}regardless of its order.

**Proposition 1.**The independence between the two decomposed systems

**y**= ( ${x}_{1}^{1}$, ··· , ${x}_{{n}_{1}}^{1}$) and

^{1}**y**=( ${x}_{1}^{2}$, ···, ${x}_{{n}_{2}}^{2}$) can be expressed on the new coordinates η

^{2}**as follows:**

^{dec}**take 0 elements as follows:**

^{dec}**Proof.**For simplicity, we show the cases of n = 2 and n = 3 for the first node separation.

**for the system decomposition < 12 > give its dual coordinates θ**

^{dec}**as follows:**

^{dec}**for the system decomposition < 122 > give its dual coordinates θ**

^{dec}**as follows:**

^{dec}**and θ**

^{dec}**coordinates.**

^{dec}**from η**

^{dec}**.**

^{dec}**means to decompose the hierarchical marginal distributions η into the products of the subsystems’ marginal distributions, in case the subscripts traverse the two subsystems. Therefore, only the statistical associations between two subsystems are set to be independent, while the internal dependencies of each subsystem remain unchanged. This is analytically equivalent to compose another mixture coordinates ξ, namely the < ··· >-cut coordinates, with proper description of the system decomposition with < ··· >. The ξ consists of the η coordinates with the subscripts that do not traverse between the decomposed subsystems, and the θ coordinates whose subscripts traverse between them.**

^{dec}**x**) is expressed with the < 1133 >-cut coordinates ξ as

**, which is, in any decomposition, equivalent to set all θ in ξ as 0:**

^{dec}**x**,ξ) : p(

**x**,ξ

**)] measures the information lost by the system decomposition. The following asymptotic agreement to χ**

^{dec}^{2}test also holds.

**Proposition 2**.

_{θ}(ξ) is the number of θ coordinates appearing in the ξ coordinates.

## 3. Edge Cutting

_{i}, x

_{j}} (1 ≤ i < j ≤ n) of the graph with n vertices. Hereafter we call this operation as the edge cutting i − j. In the same way as the system decomposition, the edge cutting corresponds to modify the η coordinates to produce η

**coordinates as follows:**

^{ec}**remains the same as those of η.**

^{ec}^{ec}from η consists of replacing the k-th order elements (k ≥ 3) of η including both i and j in its subscripts, with the product of the k − 1-th order η in maximum subgraphs (k − 1 vertices) each including i or j. This means that all orders of statistical association including the variables x

_{i}and x

_{j}are set to be independent only between them. Other relations that do not include simultaneously x

_{i}and x

_{j}remain unchanged.

_{i}, x

_{j}}. Although actual calculation can be performed only with η coordinates, this generalization is necessary to have a geometrical definition of the orthogonality. For simplicity, we only describe the ξ in the case of n = 4:

**of η**

^{ec}**, we can define the coordinates ξ**

^{ec}**of the system after the edge cutting 1 − 2 as follows:**

^{ec}**as 0.**

^{ec}**x**, ξ): p(

**x**, ξ

**)] represent the total amount of information represented by the edge 1 − 2.**

^{ec}^{2}test also holds:

**Proposition 3.**

^{2}value or the KL-divergence itself as edge information of edge 1 − 2.

## 4. Generalized Mutual Information as Complexity with Respect to the Total System Decomposition

## 5. Rectangle-Bias Complexity

**x**) I = const. } with different parameters but the same I value. This fact can be clearly explained with the use of information geometry. From the Pythagorean relation, we obtain the followings in case of n = 3:

_{r}:

_{r}:

## 6. Complementarity between Complexities Defined with Arithmetic and Geometric Means

_{r}comparing with other proposed measures of complexity.

_{r}is related to the geometrical mean. The C

_{r}can distinguish the two systems in Figure 2, giving relatively high C

_{r}value to the left system and low value to the right one.

_{r}has a finer resolution than other complexity measures. The constant conditions of complexity measures are the constraints on ${\sum}_{k=1}^{n}{}_{n}{C}_{k}$ degrees of freedom in model parameter space, which define different geometrical composition of corresponding submanifolds. We basically need ${\sum}_{k=1}^{n}{}_{n}{C}_{k}$ independent measures to assure the real-value resolution of network feature characterization. Complexities with arithmetic and geometric means are just giving complementary information on network heterogeneity, or different constant-complexity submanifolds structure in statistical manifold as depicted in Figure 3. Therefore, it is also possible to construct a class of systems that has identical I and C

_{r}values but different TSE complexity. Complexity measures should be utilized in combination, with respect to the non-linear separation capacity of network features of interest.

## 7. Cuboid-Bias Complexity with Respect to System Decompositionability

_{r}into general system size n. The n ≥ 4 situation is different from n = 3 and less in the existence of a hierarchical structure between system decompositions.

_{1}, x

_{2}, x

_{3}, x

_{4}) of a discrete distribution with 4 binary variables (x

_{1}, x

_{2}, x

_{3}, x

_{4}) (x

_{1}, x

_{2}, x

_{3}, x

_{4}∈{0, 1}) have 2

^{4}−1 = 15 parameters, which define the dual-flat coordinates of statistical manifold in information geometry.

_{1}, x

_{2}, x

_{3}, x

_{4}) parameters, the system decompositions and KL-divergences between them can be defined independently. This also holds even under the constant condition of I value or other complexity measures except the ones imposing dependency between system decompositions.

_{r}increases its value with such modification, but does not reflect the heterogeneity of KL-divergences according to the hierarchy of system decompositions. If we consider the system decompositionability as the mean facility to decompose the given system into its finest components with respect to the “all” possible system decompositions, such hierarchical difference also has a meaning in the definition of complexity.

_{r}does not reflect such characteristics since it does not distinguish between the hierarchical structure between the diameters of the green, red and blue dotted circles.

_{r}as the cuboid-bias complexity C

_{c}, which is defined as follows:

_{i}(i

_{s}) of Seq corresponds to the system decomposition, which is aligned according to the hierarchy with the following algorithmic procedure (based on [15]):

- (1)
- Initialization: Set the initial sets of system decomposition of all sequences in Seq as the whole system SD
_{1}(i_{s}):=< 111 ··· 1 > (1 ≤ i_{s}≤ |Seq|). - (2)
- Step i → i +1: If the system decomposition is the total system decomposition (SD
_{i}(i_{s}):=< 123 ··· n>), then stop. Otherwise, choose a non-decomposed subsystem SS_{i}(i_{s}) of the system decomposition SD_{i}(i_{s}), and further divide it into two independent subsystems $S{S}_{i}^{1}({i}_{s})$ and $S{S}_{i}^{2}({i}_{s})$ different for each i_{s}. SD_{i}_{+1}(i_{s}) is then defined as a system decomposition of total system that further separates independently subsystems $S{S}_{i}^{1}({i}_{s})$ and $S{S}_{i}^{2}({i}_{s})$, in addition to the previous decomposition SD_{i}(i_{s}). - (3)
- Go to the next step i +1 → i +2.

_{n}of |Seq| with system size n is obtained as the following recurrence formula:

_{1}:= 1.

_{r}, we took in the definition of C

_{c}the arithmetic average of cuboid volumes so that to renormalize the combinatorial increase of the decomposition paths (|Seq|) according to the system size n.

_{r}and the cuboid-bias complexity C

_{c}by taking the exact geometrical mean of each product of KL-divergences such as $\sqrt[n-1]{{\prod}_{i=1}^{n-1}D[S{D}_{i}({i}_{s}):S{D}_{i+1}({i}_{s})]}$. This is for further accessibility to theoretical analysis such as variational method (see “Further Consideration” section), and does not change qualitative behavior of C

_{r}and C

_{c}since the power root is a monotonically increasing function. This treatment can be interpreted as taking the (n − 1)-th power of the geometric means for the hierarchical sequences of KL-divergences.

_{c}with respect to the rectangle-biased one C

_{r}is shown in Figure 6. We consider the 6 nodes networks (n = 6) with the same I and C

_{r}values but different heterogeneity. The system in the top left figure has a circularly connected structure with medium intensity, while that of the top right figure has strongly connected 3 subsystems. These systems have qualitatively five different ways of system decomposition that are the basic generators of all hierarchical sequences Seq = {SD

_{1}(i

_{s}) →··· → SD

_{5}(i

_{s})|1 ≤ i

_{s}≤ |Seq|} for these networks. The five basial system decompositions are shown with the number ①, ②, ②′, ③ and ④ in top figures.

_{r}in both systems, the following condition is satisfied in the middle right figure: D[< 111111 >: ②] < D[< 111111 >: ①in Middle Left figure] < D[< 111111 >: ①] < D[< 111111 >: ②in Middle Left figure] <D[< 111111 >: ③] <D[< 111111 >: ④]. Furthermore, the total surface of right triangles sharing the circle diameter as hypotenuse in the middle left and the middle right figures are conditioned to be identical, therefore the rectangle-bias complexity C

_{r}fails to distinguish.

_{c}distinguishes between these two systems and gives higher value to the left one. The volume of 5-dimensional cuboids of the decomposition sequence $<111111>\stackrel{\u2460\u2461{\u2461}^{\u2462}\u2463}{\to}<123456>$ are schematically shown in the bottom figures, maintaining the quantitative difference between KL-divergences. Since the multi-information I is identical between the two systems, so is the values of KL-divergence D[< 111111 >:< 123456 >], which is the sum of the KL-divergences along the sequence $<111111>\stackrel{\u2460\u2461{\u2461}^{\u2462}\u2463}{\to}<123456>$ from the Pythagorean theorem. This means that the inequality between the cuboid volumes can be represented as the isoperimetric inequality of high-dimensional cuboid. As a consequence, the left system has quantitatively higher value of C

_{c}than the right one. The cuboid-bias complexity C

_{c}is also sensitive to such heterogeneity.

## 8. Regularized Cuboid-Bias Complexity with Respect to Generalized Mutual Information

_{c}with the multi-information I, which gives another measure of complexity namely “regularized cuboid-bias complexity${C}_{c}^{R}$.”

_{c}fails to distinguish. Both the blue and red systems are supposed to have the same C

_{c}value by adjusting the red system to have relatively smaller values of KL-divergences D[< 111 >:< 122 >] and D[< 113 >:< 123 >] than the blue one. Such conditioning is possible since the KL-divergences are independent parameters with each other.

_{c}value is identical, the two systems have different geometrical composition of system decompositions in the circle diagram. The red system has relatively easier way of decomposition < 111 >→< 122 > if renormalized with the total system decomposition < 111 >→< 123 >. This relative decompositionability with respect to the renormalization with the multi-information I can be clearly understood by superimposing the circle diagram of the two systems and comparing the angles between each and total decomposition paths (bottom figure). The red system has larger angle between the decomposition paths < 111 >→< 122 > and < 111 >→< 123 > than any others in the blue system, which represents the relative facility of the decomposition under renormalization with I. In this term, the paths < 111 >→< 121 > in the red and blue system do not change its relative facility, and the paths < 111 >→< 113 > are easier in the blue system.

## 9. Modular Complexity with Respect to the Easiest System Decomposition Path

_{m}as follows, which is the shortest path component of the cuboid-bias complexity C

_{c}:

_{min}of the sequence SD

_{1}(i

_{min}) → SD

_{2}(i

_{min}) → ··· → SD

_{n}(i

_{min}) is chosen as follows:

_{min}is unique if the system is completely heterogenous (i.e., D[SD

_{1}(i

_{k}): SD

_{2}(i

_{k})] ≠ D[SD

_{1}(i

_{l}) : SD

_{2}(i

_{l})], 1 ≤ i

_{k}<i

_{l}≤ |Seq|), otherwise plural decomposition paths that give the same C

_{m}value are possible according to the homogeneity of the system. Besides its value, the modular complexity C

_{m}should be utilized with the sequence information of the shortest decomposition path to evaluate the modularity structure of a system.

_{m}are identical but C

_{c}are different can be composed by varying the system decompositions other than in the shortest path SD

_{1}(i

_{min}) → SD

_{2}(i

_{min}) →··· → SD

_{n}(i

_{min}) without modifying the index i

_{m}in. There exist also inverse examples with identical C

_{c}and different C

_{m}, due to the complementarity between C

_{m}and C

_{c}.

_{c};

**Proposition 4.**The cuboid-bias complexities C

_{c}and${C}_{c}^{R}$ are bounded by the modular complexities C

_{m}and${C}_{m}^{R}$ respectively:

## 10. Numerical Comparison

_{c}, ${C}_{c}^{R}$, C

_{m}, and ${C}_{m}^{R}$. Since the minimum node number giving non-trivial meaning to these measures is n = 4, the corresponding dimension of parameter space is ${\sum}_{k=1}^{n}{}_{n}{C}_{k}=15$. The constant-complexity submanifolds are therefore difficult to visualize due to the high dimensionality. For simplicity, we focus on the 2-dimensional subspace of this parameter space whose first axis ranging from random to maximum dependencies of the system, and the second one representing the system decompositionability of < 1133 >.

^{−10}and η

_{0}= 0.5 was chosen for the calculation.

_{0}corresponds to the α = β = 1 condition in given parameters, whose η-coordinates become as follows:

_{0}= 1.0 and α = β = 1 gives I = 0.

_{0}, the boundary conditions of C

_{c}, ${C}_{c}^{R}$, C

_{m}and ${C}_{m}^{R}$ include that of the multi-information I, which vanish at the completely random or ordered state. This is common to other complexity measures such as the LMC complexity, and fit to the basic intuition on the concept of complexity situated equivalently far from the completely predictable and disordered states [21,22].

_{c}, ${C}_{c}^{R}$, C

_{m}and ${C}_{m}^{R}$ of a system become 0 if we add another independent variable. This property does not reflect the intuition of complexity defined by the arithmetic average of statistical measures. The proposed complexity can better find its meaning in comparison to other complexity measures such as the multi-information I, and by interactively changing the system scale to avoid trivial results with small independent subsystem. For example, the proposed complexities could be utilized as the information criteria for the model selection problems, especially with an approximative modular structure based on the statistical independency of data between subsystems. We insist that the complementarity principle between plural complexity measures of different foundation is the key to understand the complexity in a comprehensive manner.

_{c}, ${C}_{c}^{R}$, C

_{m}and ${C}_{m}^{R}$ in relation to the diverse composition of each system decomposition, it is useful to consider the geometry of their contour structure, as compared in Figure 9. The contour can be formalized as C

_{c}, ${C}_{c}^{R}$,C

_{m}, ${C}_{m}^{R}$ = const. for each complexity measure, and D[< 11 ··· 1 >: SD

_{i}(i

_{s})] = const. (1 ≤ i ≤ n−1, 1 ≤ i

_{s}≤ |Seq|) for each system decomposition. For that purpose, analysis with algebraic geometry can be considered as a prominent tool. Algebraic geometry investigates the geometrical property of polynomial equations [23]. The complexities C

_{c}, ${C}_{c}^{R}$, C

_{m}and ${C}_{m}^{R}$ can be interpreted as polynomial functions by taking each system decomposition as novel coordinates, therefore directly accessible to algebraic geometry. However, if we want to investigate the contour of the complexities on the

**p**parameter space, logarithmic function appears as the definition of KL-divergence, which is a transcendental function and outreach the analytical requirement of algebraic geometry. To introduce compatibility between the

**p**parameter space of information geometry and algebraic geometry, it suffices to describe the model by replacing the logarithmic functions as another n variables such as

**q**=log

**p**, and reconsider the intersection between the result from algebraic geometry on the coordinates (

**p**,

**q**) and

**q**=log

**p**condition. The contour of C

_{c}, ${C}_{c}^{R}$, C

_{m}and ${C}_{m}^{R}$ is also important to seek for the utility of these measures as a potential to interpret the dynamics of statistical association as geodesics.

## 11. Further Consideration

#### 11.1. Pythagorean Relations in System Decomposition and Edge Cutting

^{dec}parameters as 0 in mixture coordinate ξ

^{dec}. From the consistency of θ

^{dec}parameters in ξ

^{dec}being 0 in all system decompositions, we have the Pythagorean relation according to the inclusion relation of system decomposition. For example, the following holds:

^{ec}= 0 condition in mixture coordinates ξ

^{ec}. We have defined the η

^{ec}values of edge cutting based only on the orthogonal relation between η and θ coordinates, by generalizing the rule of system decomposition in η

^{ec}coordinates, and did not consider the Pythagorean relation between different edge cuttings.

^{ec}= 0 condition in ξ

^{ec}. Indeed, in k-cut mixture coordinates, θ

^{k}

^{+}= 0 condition is derived from the independent condition of the variables in all orders, and k-tuple statistical association is measured by reestablishing the η parameters for the statistical association up to k − 1-tuple order. In the same way, we can set θ

^{dec}= 0 condition for ξ

^{dec}of a system decomposition, and reestablish edges with respect to the η parameters, except the one in focus for edge cutting.

^{dec}for the system decomposition as follows:

^{dec}coordinates is equivalent to that of η coordinates.

^{EC}changes to the following:

^{EC}is also compatible with k-cut coordinates formalization for its simple θ

^{EC}= 0 conditions. To obtain ξ

^{EC}for arbitrary edge cutting i − j, one should take θ

^{EC}containing i and j in its subscript, set them to 0, and combine with η coordinates for the rest of the subscript. For plural edge cuttings i − j, ··· , k − l (1 ≤ i, j, k, l ≤ n), it suffices to take θ

^{EC}containing i and j, ... , k and l in its subscript respectively, then set them to 0.

^{i−j,··· , k−l}, the following holds for the example of system decomposition < 1222 >:

^{EC}values from θ

^{EC}= 0 conditions. We should call for some numerical algorithm to solve θ

^{EC}= 0 conditions with respect to η

^{EC}values, which are in general high-degree simultaneous polynomials. Furthermore, numerical convergence of the solution has to be very strict, since tiny deviation from the conditions can become non-negligible by passing fractional function and logarithmic function of θ coordinates.

^{ec}using the product between subgraphs’ η coordinates is analytically simple and does not need to consider the other edges’ recovery from system decomposition or independence hypothesis. We then chose the previous way of edge cutting for both calculability and clarity of the concept.

#### 11.2. Complexity of the Systems with Continuous Phase Space

_{i}, whose total number is n(A), defined as

_{i}}

_{T}(G) is defined simply with the number of decomposed subsystem elements by preimages as follows, without requiring ergodicity, therefore neither the existence of invariant measure μ:

_{T}(G) > 0, the dynamics of G contain chaotic orbits, but not necessary as attractive chaotic invariant set, since h

_{T}(G) ≥ h(μ, G) and the KS entropy can be negative.

_{g}(G) applying the same principle of taking geometric product between all hierarchical structure of the system decomposition A.

_{i}∈A is a

_{i}itself, meaning there exist a subsystem a

_{i}whose range is invariant under G, closed by itself. The system X can be completely divided into a

_{i}and the rest. This corresponds to the existence of an independent subsystem in cuboid-bias and modular complexities. In case such independent components do not exist, it still reflects the degree of orbit localization for all possible system decompositions in multiplicative manner. The condition σ(A) > 0 is to avoid trivial case such as the existence of unstable limit cycle, whose Lebesgue measure is 0.

_{g}(G)= 0 is the function having independent ergodic components, such as the Chirikov-Taylor map with appropriate parameter [25].

## 12. Conclusions and Discussion

_{C}, ${C}_{C}^{R}$, C

_{m}and ${C}_{m}^{R}$ generally reflect the minimum amount of information propagation rate spread entirely on the system without exception of isolated division.

_{C}, ${C}_{C}^{R}$, C

_{m}and ${C}_{m}^{R}$. Here, the shortest path selection of C

_{m}and ${C}_{m}^{R}$, and regularization of ${C}_{C}^{R}$ and ${C}_{m}^{R}$ with respect to multi-information I can be interpreted as the weight function of geometric mean. This perspective brings a definition of a generalized class of complexity measures based on the mixture coordinates and generalized mean of KL-divergence. Information discrepancy can also be generalized from KL-divergence to Bregman divergence, providing access to the concept of multiple centroids in large stochastic data analysis such as image processing [43]. The blank columns of the Table 1 imply the possibility of other complexity measures in this class. For example, the weighted geometric mean of KL-divergence defined between k-cut coordinates is expected to yield complexity measures that are sensitive to the heterogeneity of correlation orders. The weighted arithmetic mean of KL-divergence defined between < ··· >-cut coordinates should be sensitive to the mean decompositionability of arbitrary subsystem. Since these measures take analytically different form on mixture coordinates and/or mean functions, their derivatives do not coincide, which give independent information of the system on the complementary basis on statistical manifold, as long as the number of complexity measures are inferior to the freedom degree of the system.

## Acknowledgments

## Conflicts of Interest

## References

- Boccalettia, S.; Latorab, V.; Morenod, Y.; Chavezf, M.; Hwang, D.U. Complex Networks: Structure and Dynamics. Phys. Rep
**2006**, 424, 175–308. [Google Scholar] - Strogatz, S.H. Exploring Complex Networks. Nature
**2001**, 410, 268–276. [Google Scholar] - Wasserman, S.; Faust, K. Social Network Analysis; Cambridge University Press: Cambridge, UK, 1994. [Google Scholar]
- Funabashi, M.; Cointet, J.P.; Chavalarias, D. Complex Network. In Studies in Computational Intelligence; Springer: Berlin/Heidelberg, Germany, 2009; Volume 207, pp. 161–172. [Google Scholar]
- Badii, R.; Politi, A. Complexity: Hierarchical Structures and Scaling in Physics; Cambridge University Press: Cambridge, UK, 2008. [Google Scholar]
- Lempel, A.; Ziv, J. On the Complexity of Finite Sequences. IEEE Trans. Inf. Theory
**1976**, 22, 75–81. [Google Scholar] - Li, M.; Vitanyi, P. Texts in Computer Science. In An Introduction to Kolmogorov Complexity and Its Applications, 2nd ed; Springer: Berlin/Heidelberg, Germany, 1997. [Google Scholar]
- Cover, T.M.; Thomas, J.A. Elements of Information Theory; Wiley: New York, NY, USA, 2006. [Google Scholar]
- Bennett, C. On the Nature and Origin of Complexity in Discrete, Homogeneous, Locally-Interacting Systems. Found. Phys
**1986**, 16, 585–592. [Google Scholar] - Grassberger, P. Toward a Quantitative Theory of Self-Generated Complexity. Int. J. Theor. Phys
**1986**, 25, 907–938. [Google Scholar] - Crutchfield, J.P.; Feldman, D.P. Regularities Unseen Randomness Observed: The Entropy Convergence Hierarchy. Chaos
**2003**, 15, 25–54. [Google Scholar] - Crutchfield, J.P. Inferring Statistical Complexity. Phys. Rev. Lett
**1989**, 63, 105–108. [Google Scholar] - Prichard, D.; Theiler, J. Generalized Redundancies for Time Series Analysis. Physica D
**1995**, 84, 476–493. [Google Scholar] - Amari, S. Information Geometry on Hierarchy of Probability Distributions. IEEE Trans. Inf. Theory
**2001**, 47, 1701–1711. [Google Scholar] - Ay, N.; Olbrich, E.; Bertschinger, N.; Jost, J. A Unifying Framework for Complexity Measures of Finite Systems; Report 06-08-028; Santa Fe Institute: Santa Fe, NM, USA, 2006. [Google Scholar]
- MacKay, R.S. Nonlinearity in Complexity Science. Nonlinearity
**2008**, 21, T273–T281. [Google Scholar] - Tononi, G.; Sporns, O.; Edelman, M. A Measure for Brain Complexity: Relating Functional Segregation and Integration in the Nervous System. Proc. Natl. Acad. Sci. USA
**1994**, 91, 5033. [Google Scholar] - Feldman, D.P.; Crutchfield, J.P. Measures of statistical complexity: Why? Phys. Lett. A
**1998**, 238, 244–252. [Google Scholar] - Nakahara, H.; Amari, S. Information-Geometric Measure for Neural Spikes. Neural Comput
**2002**, 14, 2269–2316. [Google Scholar] - Olbrich, E.; Bertschinger, N.; Ay, N.; Jost, J. How Should Complexity Scale with System Size? Eur. Phys. J. B
**2008**, 63, 407–415. [Google Scholar] - Feldman, D.P.; Crutchfield, J.P. Measures of Statistical Complexity: Why? Phys. Lett. A
**1998**, 238, 244–252. [Google Scholar] - Lopez-Ruiz, R.; Mancini, H.; Calbet, X. A Statistical Measure of Complexity. Phys. Lett. A
**1995**, 209, 321–326. [Google Scholar] - Hodge, W.; Pedoe, D. Methods of Algebraic Geometry; Cambridge Mathematical Library, Cambridge University Press: Cambridge, UK, 1994; Volume 1–3. [Google Scholar]
- Demirel, G.; Vazquez, F.; Bohme, G.; Gross, T. Moment-closure Approximations for Discrete Adaptive Networks. Physica D
**2014**, 267, 68–80. [Google Scholar] - Fraser, G. (Ed.) The New Physics for the Twenty-First Century; Cambridge University Press: Cambridge, UK, 2006; p. 335.
- Scott, J. Social Network Analysis: A Handbook; SAGE Publications Ltd.: London, UK, 2000. [Google Scholar]
- Geier, F.; Timmer, J.; Fleck, C. Reconstructing Gene-Regulatory Networks from Time Series, Knock-Out Data, and Prior Knowledge. BMC Syst. Biol
**2007**, 1. [Google Scholar] [CrossRef] - Brown, E.N.; Kass, R.E.; Mitra, P.P. Multiple Neural Spike Train Data Analysis: State-of-the-Art and Future Challenges. Nat. Neurosci
**2004**, 7, 456–461. [Google Scholar] - Yee, T.W. The Analysis of Binary Data in Quantitative Plant Ecology. Ph.D. Thesis, The University of Auckland, New Zealand. 1993. [Google Scholar]
- Stanford Large Network Dataset Collection. Available online: http://snap.stanford.edu/data/ (accessed on 19 July 2014).
- BioGRID. Available online: http://thebiogrid.org/ (accessed on 19 July 2014).
- Neuroscience Information Framework. Available online: http://www.neuinfo.org/ (accessed on 19 July 2014).
- Global Biodiversity Information Facility. Available online: http://www.gbif.org/ (accessed on 19 July 2014).
- UCI Network Data Repository. Available online: http://networkdata.ics.uci.edu/index.php (accessed on 19 July 2014).
- Lewontin, R.C.; Cohen, D. On Population Growth in a Randomly Varying Environment. Proc. Natl. Acad. Sci. USA
**1969**, 62, 1056–1060. [Google Scholar] - Yoshimura, J.; Clark, C.W. Individual Adaptations in Stochastic Environments. Evol. Ecol
**1969**, 5, 173–192. [Google Scholar] - Wu, B.; Zhou, D.; Wang, L. Evolutionary Dynamics on Stochastic Evolving Networks for Multiple-Strategy Games. Phys. Rev. E
**2011**, 84, 046111. [Google Scholar] - Fu, F.; Wang, L. Coevolutionary Dynamics of Opinions and Networks: From Diversity to Uniformity. Phys. Rev. E
**2008**, 78, 016104. [Google Scholar] - Gross, T.; D’Lima, C.J.D.; Blasius, B. Epidemic Dynamics on an Adaptive Network. Phys. Rev. Lett
**2006**, 96, 208701. [Google Scholar] - Tsallis, C. Possible Generalization of Boltzmann-Gibbs Statistics. J. Stat. Phys
**1988**, 52, 479–487. [Google Scholar] - Quintana-Murci, L.; Alcais, A.; Abel, L.; Casanova, J.L. Immunology in natura: Clinical, Epidemiological and Evolutionary Genetics of Infectious Diseases. Nat. Immunol
**2007**, 8, 1165–1171. [Google Scholar] - Hardy, G.; Littlewood, J.; Polya, G. Inequalities; Cambridge University Press: Cambridge, UK, 1967; Chapter 3. [Google Scholar]
- Nielsen, F.; Nock, R. Sided and symmetrized Bregman centroids. IEEE Trans. Inf. Theory
**2009**, 55, 2882–2904. [Google Scholar]

**Figure 1.**Circle diagrams of system decomposition in 3-node network. Both systems have the same value of multi-information I that is expressed as the identical diameter length of the circles. 2 variations are shown, where the left system is more complex (C

_{r}high) in a sense any system decomposition requires to lose more information than the easiest one (< 122 >) in the right figure (C

_{r}low).

**Figure 2.**Schematic examples of stochastic systems with identical multi-information I where complexity measures with arithmetic mean fail to distinguish. (

**a**): Example 1 of stochastic system with homogeneous mean of edge information and symmetric fluctuation of its heterogeneity; (

**b**): Example 2 of heterogeneous stochastic system with bimodal edge information distribution and identical multi-information I and complexity based on arithmetic mean as example 1; (

**c**): schematic representation of the distribution of statistical association (edge information) in upper network; (

**d**): schematic representation of the distribution of statistical association (edge information) in upper network.

**Figure 3.**Schematic representation of complementarity between complexity measures based on arithmetic mean (C

_{a}) and geometric mean (C

_{g}) of informational distance. An example of the n − 1 dimensional constant-complexity submanifolds with respect to C

_{a}= const. and C

_{g}= const. conditions are depicted with yellow and orange surface, respectively. The dimension of the whole statistical manifold S is the parameter number n.

**Figure 4.**Hierarchy of system decomposition for 4 nodes network (n = 4). Possible sequences of Seq = {SD

_{1}(i

_{s}) → SD

_{2}(i

_{s}) → SD

_{3}(i

_{s}) → SD

_{4}(i

_{s})|1 ≤ i

_{s}≤ |Seq| = 18} are connected with the lines.

**Figure 5.**Hierarchical effect of sequential system decomposition on cuboid volume and rectangle surface on circle graph. We consider to increase the diameter of the green circle from dotted to dashed one without changing those of the red and blue circles, which gives different effect on the change of D[< 1222 >:< 1233 >] and D[< 1133 >:< 1134 >] according to the hierarchical structure of the decomposition sequences.

**Figure 6.**Meaning of taking geometric mean over the sequence of system decomposition in cuboid-bias complexity C

_{c}. (

**a**): Example of 6-node network with circularly connected structure with medium intensity. Edge width is proportional to edge information; (

**b**): Example of 6-node network with strongly connected 3 subsystems. Edge width is proportional to edge information. The multi-information I of the two systems in Top figures are conditioned to be identical; The dotted lines schematically represent possible system decompositions. (

**c,d**): Circle diagrams of each system decomposition in upper networks; The total surface of right triangles sharing the circle diameter as hypotenuse in (c) and (d) are conditioned to be identical, therefore the rectangle-bias complexity C

_{r}fails to distinguish. (

**e,f**): 5-dimensional cuboids of upper networks (Figure 6a,b) whose edges are the root of KL-divergences for the strain of system decomposition $<111111>\stackrel{\u2460\u2461{\u2461}^{\prime}\u2462\u2463}{\to}<123456>$. Only the first 3-dimensional part is shown with solid line, and the remaining 2-dimensional part is represented with dotted line. The volume of cuboid in (e) is larger than the one in (f), according to the isoperimetric inequality of high-dimensional cuboid. The total squared length of each side is identical between two cuboids, which represents multi-information I = D[< 111111 >:< 123456 >].

**Figure 7.**Examples of the 3-node systems with identical cuboid-bias complexity C

_{c}but different multi-information I on circle graph. (

**a**): System with smaller I but larger ${C}_{c}^{R}$; (

**b**): System with larger I but smaller ${C}_{c}^{R}$; (

**c**): Superposition of the above two systems. The regularized cuboid-bias complexity ${C}_{c}^{R}$ distinguishes between the blue and red systems.

**Figure 8.**Example of 16-node system < 11 ··· 1 > that has different levels of modularity. The four 4-node subsystems < 1111 > (blue blocks) are loosely connected and easy to be decomposed, while inside each component (red blocks) is tightly connected. The degree of connection represents statistical dependency or edge information between subsystems. Such hierarchical structure can be detected by observing the decomposition path of the modular complexity C

_{m}.

**Figure 9.**Contour plot of the complexity landscape of I, C

_{c}, C

_{m},${C}_{c}^{R}$, and${C}_{m}^{R}$ on α-β plane. (

**a**): Contour plot superposition of C

_{c}and C

_{m}.(

**b**): Contour plot superposition of ${C}_{c}^{R}$ and ${C}_{m}^{R}$.(

**c**): Contour plot of I. The color of contour plots corresponds to the color gradient of 3D plots in Figure 10;(

**d**): Schematic representation of the system in different regions of α-β plane. Edge width represents the degree of edge information, and independence is depicted with dotted line.

**Figure 10.**Landscape of complexities I, C

_{c}, C

_{m},${C}_{c}^{R}$, and${C}_{m}^{R}$ on α-β plane. (

**a**): Multi-information I; (

**b**): Cuboid-bias complexity C

_{c}. (

**c**): Modular complexity C

_{m}; (d): Regularized cuboid-bias complexity ${C}_{c}^{R}$;(

**e**): Regularized modular complexity ${C}_{m}^{R}$. All complexity measures show the complementarity intersecting with each other, satisfying the boundary conditions vanishing at α = 0 and β = 0 except the multi-information I. Note that regularized complexities ${C}_{c}^{R}$ and ${C}_{m}^{R}$ show singularity of convergence at α → 0 due to the regularization of infinitesimal value.

Generalized Mean of KL-Divergence | |||
---|---|---|---|

Mixture Coordinates | Arithmetic Mean | Geometric Mean | |

k-cut ζ | TSE complexity, I | ||

< ··· >-cut ξ | C_{C},
${C}_{C}^{R}$, C_{m},
${C}_{m}^{R}$ |

© 2014 by the authors; licensee MDPI, Basel, Switzerland This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution license (http://creativecommons.org/licenses/by/3.0/).

## Share and Cite

**MDPI and ACS Style**

Funabashi, M. Network Decomposition and Complexity Measures: An Information Geometrical Approach. *Entropy* **2014**, *16*, 4132-4167.
https://doi.org/10.3390/e16074132

**AMA Style**

Funabashi M. Network Decomposition and Complexity Measures: An Information Geometrical Approach. *Entropy*. 2014; 16(7):4132-4167.
https://doi.org/10.3390/e16074132

**Chicago/Turabian Style**

Funabashi, Masatoshi. 2014. "Network Decomposition and Complexity Measures: An Information Geometrical Approach" *Entropy* 16, no. 7: 4132-4167.
https://doi.org/10.3390/e16074132