#
The Thermodynamics of Network Coding, and an Algorithmic Refinement of the Principle of Maximum Entropy^{ †}

^{1}

^{2}

^{3}

^{4}

^{5}

^{6}

^{*}

^{†}

## Abstract

**:**

Thermodynamics is a funny subject. The first time you go through it, you don’t understand it at all. The second time you go through it, you think you understand it, except for one or two small points. The third time you go through it, you know you don’t understand it, but by that time you are so used to it, it doesn’t bother you any more.Arnold Sommerfeld [1].

## 1. Classical Thermodynamics and Related Work

- A measure of statistical disorder
- Some quantity or property that increases but never decreases
- A process that defines the direction of time
- A measure of statistical information

## 2. Notation and Definitions

**Definition**

**1.**

**Definition**

**2.**

**Definition**

**3.**

**Definition**

**4.**

**Definition**

**5.**

**Definition**

**6.**

**Definition**

**7.**

**Definition**

**8.**

**Definition**

**9.**

#### Graph Entropy

## 3. Algorithmic Information Dynamics

#### 3.1. Approximations to Algorithmic Complexity

#### 3.2. Block Decomposition Method

#### 3.3. Normalised BDM

#### 3.4. A Reprogrammability Calculus

**Definition**

**10.**

**Definition**

**11.**

## 4. The Thermodynamics of Computer Programs

#### Graphs as Computer Programs

## 5. Principle of Maximum Algorithmic Randomness (MAR)

#### 5.1. Maximal Algorithmic Randomness Preferential Attachment (MARPA) Algorithm

**Definition**

**12.**

#### 5.2. Supremacy of Algorithmic Maxent

**Theorem**

**1.**

**Theorem**

**2.**

#### 5.3. Numerical Examples

- Start with an empty or complete graph G.
- Perform all possible perturbations for a single operation (e.g., edge deletion or addition) on G to produce ${G}^{\prime}$.
- Keep only the perturbation that maximised algorithmic randomness, i.e., $C\left({G}^{\prime}\right)\ge C\left(G\right)$.
- Set $G:={G}^{\prime}$
- Repeat 1 until final target size is reached or $C\left(G\right)>C\left({G}^{\prime}\right)$.

#### Time Complexity and Terminating Criterion

## 6. Discussion

## 7. Conclusions

## Author Contributions

## Funding

## Conflicts of Interest

## References

- Angrist, S.W.; Helper, L.G. Order and Chaos. In Laws of Energy and Entropy; Basic Books: New York, NY, USA, 1967; p. 215. [Google Scholar]
- Hammer, D.; Romashchenko, A.; Shen, A.; Vereshchagin, N. Inequalities for Shannon entropies and Kolmogorov complexities. J. Comput. Syst. Sci.
**2000**, 60, 442–464. [Google Scholar] [CrossRef] - Grünwald, P.; Vitányi, P. Shannon Information and Kolmogorov Complexity. arXiv
**2004**, arXiv:cs/0410002. [Google Scholar] - Szilard, L. On the decrease of entropy in a thermodynamic system by the intervention of intelligent beings. Z. Phys.
**1929**, 53, 840–856. [Google Scholar] [CrossRef] - Lloyd, S.; Pagels, H. Complexity as thermodynamic depth. Ann. Phys.
**1988**, 188, 186–213. [Google Scholar] [CrossRef] - Bennett, C.H. Logical Depth and Physical Complexity. In The Universal Turing Machine: A Half-Century Survey; Herken, R., Ed.; Oxford University Press: Oxford, UK, 1988; pp. 227–257. [Google Scholar]
- Baez, J.C.; Stay, M. Algorithmic Thermodynamics, Computability of the Physical. Math. Struct. Comput. Sci.
**2012**, 22, 771–787. [Google Scholar] [CrossRef] - Crutchfield, J.P.; Shalizi, C.R. Thermodynamic depth of causal states: Objective complexity via minimal representations. Phys. Rev. E
**1999**, 59, 275–283. [Google Scholar] [CrossRef] [Green Version] - Zurek, W.H. Algorithmic randomness and physical entropy. Phys. Rev. A
**1989**, 40, 4731. [Google Scholar] [CrossRef] - Zurek, W.H. Thermodynamic cost of computation, algorithmic complexity, and the information metric. Nature
**1989**, 341, 119–124. [Google Scholar] [CrossRef] - Kolmogorov, A.N. Three approaches to the quantitative definition of information. Probl. Inf. Transm.
**1965**, 1, 1–7. [Google Scholar] [CrossRef] - Solomonoff, R.J. A formal theory of inductive inference: Parts 1 and 2. Inf. Control
**1964**, 7, 224–254. [Google Scholar] [CrossRef] - Chaitin, G.J. On the length of programs for computing finite binary sequences. J. ACM
**1966**, 13, 547–569. [Google Scholar] [CrossRef] - Levin, L.A. Laws of information conservation (non-growth) and aspects of the foundation of probability theory. Probl. Inf. Transm.
**1974**, 10, 206–210. [Google Scholar] - Zenil, H.; Kiani, N.A.; Tegnér, J. Low Algorithmic Complexity Entropy-deceiving Graphs. Phys. Rev. E
**2017**, 96, 012308. [Google Scholar] [CrossRef] [PubMed] - Erdös, P.; Rényi, A. On Random Graphs I. Publ. Math. Debrecen
**1959**, 6, 290–297. [Google Scholar] - Erdös, P.; Rényi, A. On the evolution of random graphs. Bull. Inst. Int. Stat
**1961**, 38, 343–347. [Google Scholar] - Gilbert, E.N. Random graphs. Ann. Math. Stat.
**1959**, 30, 1141–1144. [Google Scholar] [CrossRef] - Zenil, H.; Kiani, N.A.; Tegnér, J. Algorithmic Information Dynamics of Emergent, Persistent, and Colliding Particles in the Game of Life. In From Parallel to Emergent Computing; Adamatzky, A., Ed.; Taylor & Francis/CRC Press: Boca Raton, FL, USA, 2019; pp. 367–383. [Google Scholar]
- Zenil, H.; Kiani, N.A.; Zea, A.; Tegnér, J. Causal Deconvolution by Algorithmic Generative Models. Nat. Mach. Intell.
**2019**, 1, 58–66. [Google Scholar] [CrossRef] - Boccaletti, S.; Bianconi, G.; Criado, R.; Del Genio, C.I.; Gómez-Gardenes, J.; Romance, M.; Sendiña-Nadal, I.; Wang, Z.; Zanin, M. The structure and dynamics of multilayer networks. Phys. Rep.
**2014**, 544, 1–122. [Google Scholar] [CrossRef] [Green Version] - Chen, Z.; Dehmer, M.; Emmert-Streib, F.; Shi, Y. Entropy bounds for dendrimers. Appl. Math. Comput.
**2014**, 24, 462–472. [Google Scholar] [CrossRef] - Orsini, C.; Dankulov, M.M.; Colomer-de-Simón, P.; Jamakovic, A.; Mahadevan, P.; Vahdat, A.; Bassler, K.E.; Toroczkai, Z.; Boguñá, M.; Caldarelli, G.; et al. Quantifying randomness in real networks. Nat. Commun.
**2015**, 6, 8627. [Google Scholar] - Korner, J.; Marton, K. Random access communication and graph entropy. IEEE Trans. Inf. Theory
**1988**, 34, 312–314. [Google Scholar] [CrossRef] - Estrada, E.; José, A.; Hatano, N. Walk entropies in graphs. Linear Algebra Appl.
**2014**, 443, 235–244. [Google Scholar] [CrossRef] - Dehmer, M.; Mowshowitz, A. A history of graph entropy measures. Inf. Sci.
**2011**, 181, 57–78. [Google Scholar] [CrossRef] - Teixeira, A.; Matos, T.A.; Souto, A.; Antunes, L. Entropy Measures vs. Kolmogorov Complexity. Entropy
**2011**, 13, 595–611. [Google Scholar] [CrossRef] - Zenil, H.; Kiani, N.A.; Tegnér, J. Quantifying loss of information in network-based dimensionality reduction techniques. J. Complex Netw.
**2016**, 4, 342–362. [Google Scholar] [CrossRef] - Zenil, H.; Soler-Toscano, F.; Dingle, K.; Louis, A. Graph Automorphisms and Topological Characterization of Complex Networks by Algorithmic Information Content. Physica A
**2014**, 404, 341–358. [Google Scholar] [CrossRef] - Zenil, H.; Kiani, N.A.; Tegnér, J. Algorithmic complexity of motifs clusters superfamilies of networks. In Proceedings of the IEEE International Conference on Bioinformatics and Biomedicine, Shanghai, China, 18–21 December 2013. [Google Scholar]
- Kirchherr, W.W.; Li, M.; Vitányi, P.M.B. The miraculous universal distribution. J. Math. Intell.
**1997**, 19, 7–15. [Google Scholar] [CrossRef] [Green Version] - Calude, C.S. Information and Randomness: An Algorithmic Perspective, 2nd ed.; EATCS Series; Springer: New York, NY, USA, 2010. [Google Scholar]
- Cover, T.M.; Thomas, J.A. Elements of Information Theory, 2nd ed.; Wiley-Blackwell: New York, NY, USA, 2009. [Google Scholar]
- Buhrman, H.; Li, M.; Tromp, J.; Vitányi, P. Kolmogorov Random Graphs and the Incompressibility Method. SIAM J. Comput.
**1999**, 29, 590–599. [Google Scholar] [CrossRef] [Green Version] - Delahaye, J.-P.; Zenil, H. Numerical Evaluation of the Complexity of Short Strings: A Glance Into the Innermost Structure of Algorithmic Randomness. Appl. Math. Comput.
**2012**, 219, 63–77. [Google Scholar] [CrossRef] - Soler-Toscano, F.; Zenil, H.; Delahaye, J.-P.; Gauvrit, N. Calculating Kolmogorov Complexity from the Frequency Output Distributions of Small Turing Machines. PLoS ONE
**2014**, 9, e96223. [Google Scholar] [CrossRef] - Zenil, H.; Soler-Toscano, F.; Kiani, N.A.; Hernández-Orozco, S.; Rueda-Toicen, A. A Decomposition Method for Global Evaluation of Shannon Entropy and Local Estimations of Algorithmic Complexity. Entropy
**2018**, 20, 605. [Google Scholar] [CrossRef] - Zenil, H.; Kiani, N.A.; Marabita, F.; Deng, Y.; Elias, S.; Schmidt, A.; Ball, G.; Tegnér, J. An Algorithmic Information Calculus for Causal Discovery and Reprogramming Systems. bioRxiv
**2018**, 185637. [Google Scholar] [CrossRef] - Zenil, H.; Soler-Toscano, F.; Delahaye, J.-P.; Gauvrit, N. Two-Dimensional Kolmogorov Complexity and Validation of the Coding Theorem Method by Compressibility. PeerJ Comput. Sci.
**2015**. [Google Scholar] [CrossRef] - Zenil, H.; Kiani, N.A.; Tegnér, J. Methods of Information Theory and Algorithmic Complexity for Network Biology. Semin. Cell Dev. Biol.
**2016**, 51, 32–43. [Google Scholar] [CrossRef] [PubMed] - Bianconi, G. The entropy of randomized network ensembles. EPL (Europhys. Lett.)
**2007**, 81, 28005. [Google Scholar] [CrossRef] [Green Version] - Zenil, H.; Kiani, N.A.; Tegnér, J. Minimal Algorithmic Information Loss Methods for Dimension Reduction, Feature Selection and Network Sparsification. arXiv
**2018**, arXiv:1802.05843. [Google Scholar] - Zenil, H.; Badillo, L.; Hernández-Orozco, S.; Hernández-Quiroz, F. Coding-theorem Like Behaviour and Emergence of the Universal Distribution from Resource-bounded Algorithmic Probability. Int. J. Parallel Emerg. Distrib. Syst.
**2018**. [Google Scholar] [CrossRef] - Ackermann, W. Die Widerspruchsfreiheit der allgemeinen Mengenlehre. Math. Ann.
**1937**, 114, 305–315. [Google Scholar] [CrossRef] - Rado, R. Universal graphs and universal functions. Acta Arith.
**1964**, 9, 331–340. [Google Scholar] [CrossRef] [Green Version] - Gauvrit, N.; Zenil, H.; Soler-Toscano, F.; Delahaye, J.-P.; Brugger, P. Human Behavioral Complexity Peaks at Age 25. PLoS Comput. Biol.
**2017**, 13, e1005408. [Google Scholar] [CrossRef] - Soler-Toscano, F.; Zenil, H. A Computable Measure of Algorithmic Probability by Finite Approximations with an Application to Integer Sequences. Complexity
**2017**, 2017. [Google Scholar] [CrossRef] - Zenil, H. Algorithmic Data Analytics, Small Data Matters and Correlation versus Causation. In Berechenbarkeit der Welt? Philosophie und Wissenschaft im Zeitalter von Big Data (Computability of the World? Philosophy and Science in the Age of Big Data); Ott, M., Pietsch, W., Wernecke, J., Eds.; Springer: New York, NY, USA, 2017; pp. 453–475. [Google Scholar] [Green Version]
- Zenil, H.; Kiani, N.A.; Tegnér, J. A Review of Graph and Network Complexity from an Algorithmic Information Perspective. Entropy
**2018**, 20, 551. [Google Scholar] [CrossRef]

**Figure 1.**The thermodynamics of computer programming. (

**Top**) The programs producing simple versus random data have different reprogrammability properties. If repurposed to generate programs to print blocks of 0s, we only need a single intervention in the generative program of (

**1**), changing 1 to 0 inside the Print instruction indicating that 200 0s be printed instead of 200 1s. In contrast, asking a program that prints a random binary string s to print only 0s will require on average $\left|s\right|/2$ interventions to manually change every bit 1 to 0. Random perturbations can be seen as the exploration of the possible paths through which an object may evolve over time.

**Figure 2.**Graphs as produced by programs. All real-world networks lie between the extreme cases of being as simple as a complete graph whose algorithmic complexity C is minimal and grows by only $log\left|V\right(k\left)\right|$, and a random (also statistically random and thus E-R) graph whose algorithmic complexity is maximal and grows by its number of edges $\left|E\right(\text{E-R}\left)\right|$. If we ask what it takes to change the program producing k to produce $\text{E-R}$ and vice versa, in a random graph, any single algorithmic-random node or edge removal does not entail a major change in the program-size of its generating program, which is similar in size to the random graph itself, i.e., $\left|E\right(G\left)\right|$. The curve shows how, without loss of generality, the reprogramming capability of networks, as produced by computer programs, produces an asymmetry imposed by algorithmic complexity and reminiscent of traditional thermodynamics as based on classical probability. A maximally random network has only positive (blue) elements (Figure 3 and Figure 4) because there exists no perturbation that can increase the randomness of the network either by removing a node or an edge, as it is already random (and thus non-deterministic). Thus, changing its (near) minimal program-size length by edge or node removal is slow. However, a simple graph may have elements that incline its program-size length toward randomness. In each extreme case (simple vs. random), the distribution of sorted elements capable of shifting in each direction is shown in the form of what we call “signatures”, both for algorithmically random edge and node removal. The highest reprogrammability point is the place where a graph has as many elements to steer it in one direction as in the other.

**Figure 3.**Algorithmic Maxent in the generation of graphs (without loss of generality, no other restrictions are imposed). The paths to statistical and algorithmic randomness are different, and they determine different principles for different purposes. Algorithmic Maxent tells apart recursive cases from non-recursive (thus algorithmic random) ones. Classical Maxent quantifies statistical randomness but its algorithmic refinement quantifies both statistical and algorithmic randomness. This opens up the range of possibilities for moving toward and reaching a random graph, by not only considering whether it is random-looking but also whether it is actually algorithmically random. Because the worse case for algorithmic Maxent is not to distinguish recursive cases (due to, e.g., semi-computability) its worse performance is to retrieve the same Gibbs measure and produce the same ensemble as classical Maxent would do.

**Figure 4.**Reduction of the Gibbs/Boltzmann distribution. While objects may appear maximally statistically random in all other properties but the ones constrained by a problem of interest for which Maxent is relevant, there may actually be of different nature and some may actually be algorithmically random while others recursively generated hence they should not be placed in the same distribution. The original graph G has a shortest description $minG$ and perturbations to either the original and compressed version can lead to randomised graphs for different purposes. Some of them will have maximum entropy denoted by $max{G}_{n}^{\prime}$ but among them only some will also be of $maxK\left(G\right)$.

**Figure 5.**(

**Top left**) One can produce a MAR graph starting from an empty graph and adding one edge at a time (see Figure 6) or one can start from a complete graph and start deleting edge by edge keeping only those that maximise the algorithmic randomness of the resulting graph. (

**Bottom left**) Following this process, MAR graphs top E-R graphs meaning BDM effectively separate low algorithmic complexity (algorithmic randomness) from high entropy (statistical randomness), where entropy would simply be blind collapsing all recursive and non-recursive cases. (

**Right**) degree distribution comparison between E-R and MAR graphs.

**Figure 6.**(

**Top**) A MAR graph constructed by adding one by one every possible edge and keeping only those that maximise the uncompressibility/algorithmic complexity of the graph according to BDM. Shown are only eight steps from 10 edges to 150. (

**Bottom**) MAR graphs versus ensemble of pseudo-randomly generated E-R graphs (with edge density 0.5) versus the ZK graph [15] designed to have minimal adjacency matrix entropy but maximal degree sequence entropy.

© 2019 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (http://creativecommons.org/licenses/by/4.0/).

## Share and Cite

**MDPI and ACS Style**

Zenil, H.; Kiani, N.A.; Tegnér, J.
The Thermodynamics of Network Coding, and an Algorithmic Refinement of the Principle of Maximum Entropy. *Entropy* **2019**, *21*, 560.
https://doi.org/10.3390/e21060560

**AMA Style**

Zenil H, Kiani NA, Tegnér J.
The Thermodynamics of Network Coding, and an Algorithmic Refinement of the Principle of Maximum Entropy. *Entropy*. 2019; 21(6):560.
https://doi.org/10.3390/e21060560

**Chicago/Turabian Style**

Zenil, Hector, Narsis A. Kiani, and Jesper Tegnér.
2019. "The Thermodynamics of Network Coding, and an Algorithmic Refinement of the Principle of Maximum Entropy" *Entropy* 21, no. 6: 560.
https://doi.org/10.3390/e21060560