# Generalised Measures of Multivariate Information Content

^{1}

^{2}

^{*}

## Abstract

**:**

## 1. Introduction

## 2. Mutual Information Content

## 3. Marginal Information Sharing

**Theorem**

**1.**

**Proof.**

## 4. Synergistic Information Content

## 5. Properties of the Union and Intersection Information Content

**Property**

**1**

**.**The union and intersection information content are idempotent,

**Property**

**2**

**.**The union and intersection information content are commutative,

**Property**

**3**

**.**The union and intersection information content are associative,

**Property**

**4**

**.**The union and intersection information content are connected by absorption,

**Property**

**5**

**.**The union and intersection information content are distribute over each other,

**Theorem**

**2.**

**Theorem**

**3.**

**Proof.**

**Property**

**6**

**.**The union and intersection information content are given by at least one of

## 6. Generalised Marginal Information Sharing

**Theorem**

**4.**

**Proof.**

## 7. Multivariate Information Decomposition

**Theorem**

**5.**

**Proof.**

## 8. Union and Intersection Mutual Information

## 9. Conclusions

## Author Contributions

## Funding

## Acknowledgments

## Conflicts of Interest

## Appendix A

**Lemma**

**A1.**

**Proof.**

**Lemma**

**A2.**

**Proof.**

**Lemma**

**A3.**

**Proof.**

**Lemma**

**A4.**

**Proof.**

**Lemma**

**A5.**

**Proof.**

**Lemma**

**A6.**

**Proof.**

**Lemma**

**A7.**

**Proof.**

**Lemma**

**A8.**

**Proof.**

**Lemma**

**A9.**

**Proof.**

## References

- Reza, F. An Introduction to Information Theory, International student edition; McGraw-Hill: New York, NY, USA, 1961. [Google Scholar]
- Kuo Ting, H. On the Amount of Information. Theory Probab. Appl.
**1962**, 7, 439–447. [Google Scholar] [CrossRef] - Abramson, N. Information Theory and Coding; McGraw-Hill: New York, NY, USA, 1963. [Google Scholar]
- Campbell, L. Entropy as a measure. IEEE Trans. Inf. Theory
**1965**, 11, 112–114. [Google Scholar] [CrossRef] - Csiszar, I.; Körner, J. Information Theory: Coding Theorems for Discrete Memoryless Systems; Academic Press, Inc.: Cambridge, MA, USA, 1981. [Google Scholar]
- Yeung, R.W. A new outlook on Shannon’s information measures. IEEE Trans. Inf. Theory
**1991**, 37, 466–474. [Google Scholar] [CrossRef] - Yeung, R.W. Information Theory and Network Coding; Springer Science & Business Media: Berlin/Heidelberg, Germany, 2008. [Google Scholar]
- MacKay, D.J. Information Theory, Inference and Learning Algorithms; Cambridge University Press: Cambridge, UK, 2003. [Google Scholar]
- Fano, R.M. Transmission of Information: A Statistical Theory of Communication; M.I.T. Press: Cambridge, MA, USA, 1961. [Google Scholar]
- McGill, W. Multivariate information transmission. Trans. IRE Prof. Group Inf. Theory
**1954**, 4, 93–111. [Google Scholar] [CrossRef] - Bell, A.J. The co-information lattice. In Proceedings of the Fifth International Workshop on Independent Component Analysis and Blind Signal Separation: ICA, Granada, Spain, 22–24 September 2004; Volume 2003. [Google Scholar]
- Han, T.S. Linear dependence structure of the entropy space. Inf. Control
**1975**, 29, 337–368. [Google Scholar] [CrossRef][Green Version] - Cover, T.M.; Thomas, J.A. Elements of Information Theory; John Wiley & Sons: Hoboken, NJ, USA, 2012. [Google Scholar]
- Fano, R.M. The statistical theory of information. Il Nuovo Cimento
**1959**, 13, 353–372. [Google Scholar] [CrossRef] - Pinsker, M.S. Information and Information Stability of Random Variables and Processes; Holden-Day: San Francisco, CA, USA, 1964. [Google Scholar]
- Grätzer, G. General Lattice Theory; Springer Science & Business Media: Berlin/Heidelberg, Germany, 2002. [Google Scholar]
- Davey, B.A.; Priestley, H.A. Introduction to Lattices and Order; Cambridge University Press: Cambridge, UK, 2002. [Google Scholar]
- Comtet, L. Advanced Combinatorics: The Art of Finite and Infinite Expansions; Springer Science & Business Media: Berlin/Heidelberg, Germany, 2012. [Google Scholar]
- The OEIS Foundation Inc. The On-Line Encyclopedia of Integer Sequences. 2019. Available online: https://oeis.org/A000372 (accessed on 14 February 2020).
- Birkhoff, G. Lattice Theory; American Mathematical Soc.: Providence, RI, USA, 1940; Volume 25. [Google Scholar]
- Stanley, R.P. Enumerative Combinatorics; Cambridge University Press: Cambridge, UK, 1997; Volume 1. [Google Scholar]
- Sheldon, R. A first Course in Probability; Pearson Education India: Delhi, India, 2002. [Google Scholar]
- Williams, P.L.; Beer, R.D. Nonnegative decomposition of multivariate information. arXiv
**2010**, arXiv:1004.2515. [Google Scholar] - Lizier, J.T.; Bertschinger, N.; Jost, J.; Wibral, M. Information Decomposition of Target Effects from Multi-Source Interactions: Perspectives on Previous, Current and Future Work. Entropy
**2018**, 20, 307. [Google Scholar] [CrossRef][Green Version] - Crampton, J.; Loizou, G. The completion of a poset in a lattice of antichains. Int. Math. J.
**2001**, 1, 223–238. [Google Scholar] - Crampton, J.; Loizou, G. Two Partial Orders on the Set of Antichains. 2000. Available online: http://learninglink.ac.uk/oldsite/research/techreps/2000/bbkcs-00-09.pdf (accessed on 14 February 2020).
- Finn, C.; Lizier, J.T. Pointwise partial information decomposition using the specificity and ambiguity lattices. Entropy
**2018**, 20, 297. [Google Scholar] [CrossRef][Green Version] - Finn, C.; Lizier, J.T. Probability Mass Exclusions and the Directed Components of Mutual Information. Entropy
**2018**, 20, 826. [Google Scholar] [CrossRef][Green Version] - Chan, C.; Al-Bashabsheh, A.; Ebrahimi, J.B.; Kaced, T.; Liu, T. Multivariate mutual information inspired by secret-key agreement. Proc. IEEE
**2015**, 103, 1883–1913. [Google Scholar] [CrossRef] - Watanabe, S. Information theoretical analysis of multivariate correlation. IBM J. Res. Dev.
**1960**, 4, 66–82. [Google Scholar] [CrossRef] - Garner, W.R. Uncertainty and structure as psychological concepts. Science
**1963**, 140, 799. [Google Scholar] - Studenỳ, M.; Vejnarová, J. The multiinformation function as a tool for measuring stochastic dependence. In Learning in Graphical Models; Springer: Berlin/Heidelberg, Germany, 1998; pp. 261–297. [Google Scholar]
- Tononi, G.; Sporns, O.; Edelman, G.M. A measure for brain complexity: Relating functional segregation and integration in the nervous system. Proc. Natl. Acad. Sci. USA
**1994**, 91, 5033–5037. [Google Scholar] [CrossRef] [PubMed][Green Version] - Abdallah, S.A.; Plumbley, M.D. A measure of statistical complexity based on predictive information with application to finite spin systems. Phys. Lett. A
**2012**, 376, 275–281. [Google Scholar] [CrossRef] - Shannon, C. The lattice theory of information. Trans. IRE Prof. Group Inf. Theory
**1953**, 1, 105–107. [Google Scholar] [CrossRef] - Gács, P.; Körner, J. Common information is far less than mutual information. Probl. Control. Inf. Theory
**1973**, 2, 149–162. [Google Scholar] - Li, H.; Chong, E.K. On a connection between information and group lattices. Entropy
**2011**, 13, 683–708. [Google Scholar] [CrossRef][Green Version] - Yu, H.; Mineyev, I.; Varshney, L.R. A group-theoretic approach to computational abstraction: Symmetry-driven hierarchical clustering. arXiv
**2018**, arXiv:1807.11167. [Google Scholar] - Wolf, S.; Wultschleger, J. Zero-error information and applications in cryptography. In Proceedings of the IEEE Information Theory Workshop, San Antonio, TX, USA, 24–29 October 2004; pp. 1–6. [Google Scholar]
- Bertschinger, N.; Rauh, J.; Olbrich, E.; Jost, J. Shared information—New insights and problems in decomposing information in complex systems. In Proceedings of the European Conference on Complex Systems 2012; Springer: Berlin/Heidelberg, Germany, 2013; pp. 251–269. [Google Scholar]
- Harder, M.; Salge, C.; Polani, D. Bivariate measure of redundant information. Phys. Rev. E
**2013**, 87, 012130. [Google Scholar] [CrossRef] [PubMed][Green Version] - Harder, M. Information Driven Self-Organization of Agents and Agent Collectives. Ph.D. Thesis, University of Hertfordshire, Hatfield, UK, 2013. [Google Scholar]
- Bertschinger, N.; Rauh, J.; Olbrich, E.; Jost, J.; Ay, N. Quantifying unique information. Entropy
**2014**, 16, 2161–2183. [Google Scholar] [CrossRef][Green Version] - Griffith, V.; Koch, C. Quantifying Synergistic Mutual Information. In Guided Self-Organization: Inception; Emergence, Complexity and Computation; Prokopenko, M., Ed.; Springer: Berlin/Heidelberg, Germany, 2014; Volume 9, pp. 159–190. [Google Scholar]
- Griffith, V.; Chong, E.K.; James, R.G.; Ellison, C.J.; Crutchfield, J.P. Intersection information based on common randomness. Entropy
**2014**, 16, 1985–2000. [Google Scholar] [CrossRef][Green Version] - Rauh, J.; Bertschinger, N.; Olbrich, E.; Jost, J. Reconsidering unique information: Towards a multivariate information decomposition. In Proceedings of the 2014 IEEE International Symposium on Information Theory (ISIT), Honolulu, HI, USA, 29 June–4 July 2014; pp. 2232–2236. [Google Scholar]
- Barrett, A.B. Exploration of synergistic and redundant information sharing in static and dynamical Gaussian systems. Phys. Rev. E
**2015**, 91, 052802. [Google Scholar] [CrossRef][Green Version] - Griffith, V.; Ho, T. Quantifying redundant information in predicting a target random variable. Entropy
**2015**, 17, 4644–4653. [Google Scholar] [CrossRef][Green Version] - Olbrich, E.; Bertschinger, N.; Rauh, J. Information decomposition and synergy. Entropy
**2015**, 17, 3501–3517. [Google Scholar] [CrossRef][Green Version] - Perrone, P.; Ay, N. Hierarchical Quantification of Synergy in Channels. Front. Robot. AI
**2016**, 2, 35. [Google Scholar] [CrossRef][Green Version] - Rosas, F.; Ntranos, V.; Ellison, C.J.; Pollin, S.; Verhelst, M. Understanding interdependency through complex information sharing. Entropy
**2016**, 18, 38. [Google Scholar] [CrossRef][Green Version] - Faes, L.; Marinazzo, D.; Stramaglia, S. Multiscale information decomposition: Exact computation for multivariate Gaussian processes. Entropy
**2017**, 19, 408. [Google Scholar] [CrossRef][Green Version] - Ince, A.A.R. Measuring multivariate redundant information with pointwise common change in surprisal. Entropy
**2017**, 19, 318. [Google Scholar] [CrossRef][Green Version] - James, R.G.; Crutchfield, J.P. Multivariate dependence beyond shannon information. Entropy
**2017**, 19, 531. [Google Scholar] [CrossRef] - Kay, J.W.; Ince, R.A.; Dering, B.; Phillips, W.A. Partial and Entropic Information Decompositions of a Neuronal Modulatory Interaction. Entropy
**2017**, 19, 560. [Google Scholar] [CrossRef][Green Version] - Makkeh, A.; Theis, D.O.; Vicente, R. Bivariate Partial Information Decomposition: The Optimization Perspective. Entropy
**2017**, 19, 530. [Google Scholar] [CrossRef][Green Version] - Pica, G.; Piasini, E.; Chicharro, D.; Panzeri, S. Invariant components of synergy, redundancy, and unique information among three variables. Entropy
**2017**, 19, 451. [Google Scholar] [CrossRef][Green Version] - Quax, R.; Har-Shemesh, O.; Sloot, P. Quantifying synergistic information using intermediate stochastic variables. Entropy
**2017**, 19, 85. [Google Scholar] [CrossRef][Green Version] - Rauh, J.; Banerjee, P.; Olbrich, E.; Jost, J.; Bertschinger, N. On extractable shared information. Entropy
**2017**, 19, 328. [Google Scholar] [CrossRef][Green Version] - Rauh, J.; Banerjee, P.; Olbrich, E.; Jost, J.; Bertschinger, N.; Wolpert, D. Coarse-graining and the Blackwell order. Entropy
**2017**, 19, 527. [Google Scholar] [CrossRef] - Rauh, J. Secret sharing and shared information. Entropy
**2017**, 19, 601. [Google Scholar] [CrossRef][Green Version] - James, R.G.; Emenheiser, J.; Crutchfield, J.P. Unique information via dependency constraints. J. Phys. Math. Theor.
**2018**, 52, 014002. [Google Scholar] [CrossRef][Green Version] - Williams, P.L.; Beer, R.D. Generalized measures of information transfer. arXiv
**2011**, arXiv:1102.1507. [Google Scholar] - Flecker, B.; Alford, W.; Beggs, J.M.; Williams, P.L.; Beer, R.D. Partial information decomposition as a spatiotemporal filter. Chaos Interdiscip. J. Nonlinear Sci.
**2011**, 21, 037104. [Google Scholar] [CrossRef] [PubMed][Green Version] - Stramaglia, S.; Wu, G.R.; Pellicoro, M.; Marinazzo, D. Expanding the transfer entropy to identify information circuits in complex systems. Phys. Rev. E
**2012**, 86, 066211. [Google Scholar] [CrossRef] [PubMed][Green Version] - Lizier, J.T.; Flecker, B.; Williams, P.L. Towards a synergy-based approach to measuring information modification. In Proceedings of the 2013 IEEE Symposium on Artificial Life (ALIFE), Singapore, 15–19 April 2013; pp. 43–51. [Google Scholar]
- Stramaglia, S.; Cortes, J.M.; Marinazzo, D. Synergy and redundancy in the Granger causal analysis of dynamical networks. New J. Phys.
**2014**, 16, 105003. [Google Scholar] [CrossRef][Green Version] - Timme, N.; Alford, W.; Flecker, B.; Beggs, J.M. Synergy, redundancy, and multivariate information measures: An experimentalist’s perspective. J. Comput. Neurosci.
**2014**, 36, 119–140. [Google Scholar] [CrossRef] [PubMed] - Wibral, M.; Lizier, J.T.; Priesemann, V. Bits from brains for biologically inspired computing. Front. Robot.
**2015**, 2, 5. [Google Scholar] [CrossRef][Green Version] - Biswas, A.; Banik, S.K. Redundancy in information transmission in a two-step cascade. Phys. Rev. E
**2016**, 93, 052422. [Google Scholar] [CrossRef][Green Version] - Frey, S.; Williams, P.L.; Albino, D.K. Information encryption in the expert management of strategic uncertainty. arXiv
**2016**, arXiv:1605.04233. [Google Scholar] - Timme, N.M.; Ito, S.; Myroshnychenko, M.; Nigam, S.; Shimono, M.; Yeh, F.C.; Hottowy, P.; Litke, A.M.; Beggs, J.M. High-degree neurons feed cortical computations. PLoS Comput. Biol.
**2016**. [Google Scholar] [CrossRef] - Ghazi-Zahedi, K.; Langer, C.; Ay, N. Morphological computation: Synergy of body and brain. Entropy
**2017**, 19, 456. [Google Scholar] [CrossRef] - Maity, A.K.; Chaudhury, P.; Banik, S.K. Information theoretical study of cross-talk mediated signal transduction in MAPK pathways. Entropy
**2017**, 19, 469. [Google Scholar] [CrossRef][Green Version] - Sootla, S.; Theis, D.; Vicente, R. Analyzing information distribution in complex systems. Entropy
**2017**, 19, 636. [Google Scholar] [CrossRef][Green Version] - Tax, T.; Mediano, P.A.; Shanahan, M. The partial information decomposition of generative neural network models. Entropy
**2017**, 19, 474. [Google Scholar] [CrossRef] - Wibral, M.; Finn, C.; Wollstadt, P.; Lizier, J.T.; Priesemann, V. Quantifying Information Modification in Developing Neural Networks via Partial Information Decomposition. Entropy
**2017**, 19, 494. [Google Scholar] [CrossRef][Green Version] - Wibral, M.; Priesemann, V.; Kay, J.W.; Lizier, J.T.; Phillips, W.A. Partial information decomposition as a unified approach to the specification of neural goal functions. Brain Cogn.
**2017**, 112, 25–38. [Google Scholar] [CrossRef][Green Version] - Finn, C.; Lizier, J.T. Quantifying Information Modification in Cellular Automata using Pointwise Partial Information Decomposition. In Artificial Life Conference Proceedings; MIT Press: Cambridge, MA, USA, 2018; pp. 386–387. [Google Scholar]
- Rosas, F.; Mediano, P.A.; Ugarte, M.; Jensen, H. An information-theoretic approach to self-organisation: Emergence of complex interdependencies in coupled dynamical systems. Entropy
**2018**, 20, 793. [Google Scholar] [CrossRef][Green Version] - Wollstadt, P.; Lizier, J.T.; Vicente, R.; Finn, C.; Martínez-Zarzuela, M.; Mediano, P.A.; Novelli, L.; Wibral, M. IDTxl: The Information Dynamics Toolkit xl: A Python package for the efficient analysis of multivariate information dynamics in networks. J. Open Source Softw.
**2019**, 4, 1081. [Google Scholar] [CrossRef] - Biswas, A. Multivariate information processing characterizes fitness of a cascaded gene-transcription machinery. Chaos: Interdiscip. J. Nonlinear Sci.
**2019**, 29, 063108. [Google Scholar] [CrossRef] - James, R.G.; Emenheiser, J.; Crutchfield, J. Unique information and secret key agreement. Entropy
**2019**, 21, 12. [Google Scholar] [CrossRef][Green Version] - Kolchinsky, A. A novel approach to multivariate redundancy and synergy. arXiv
**2019**, arXiv:1908.08642. [Google Scholar] - Li, M.; Han, Y.; Aburn, M.J.; Breakspear, M.; Poldrack, R.A.; Shine, J.M.; Lizier, J.T. Transitions in information processing dynamics at the whole-brain network level are driven by alterations in neural gain. PLoS Comput. Biol.
**2019**. [Google Scholar] [CrossRef][Green Version] - Rosas, F.; Mediano, P.A.; Gastpar, M.; Jensen, H.J. Quantifying high-order interdependencies via multivariate extensions of the mutual information. Phys. Rev. E
**2019**, 100, 032305. [Google Scholar] [CrossRef] [PubMed][Green Version] - Ince, R.A. The Partial Entropy Decomposition: Decomposing multivariate entropy and mutual information via pointwise common surprisal. arXiv
**2017**, arXiv:1702.01591. [Google Scholar] - Lizier, J.T.; Heinzle, J.; Horstmann, A.; Haynes, J.D.; Prokopenko, M. Multivariate information-theoretic measures reveal directed information structure and task relevant changes in fMRI connectivity. J. Comput. Neurosci.
**2011**, 30, 85–107. [Google Scholar] [CrossRef] [PubMed] - Vakorin, V.A.; Krakovska, O.A.; McIntosh, A.R. Confounding effects of indirect connections on causality estimation. J. Neurosci. Methods
**2009**, 184, 152–160. [Google Scholar] [CrossRef] - Novelli, L.; Wollstadt, P.; Mediano, P.A.; Wibral, M.; Lizier, J.T. Large-scale directed network inference with multivariate transfer entropy and hierarchical statistical testing. Netw. Neurosci.
**2019**, 3, 827–847. [Google Scholar] [CrossRef] - Deutscher, D.; Meilijson, I.; Schuster, S.; Ruppin, E. Can single knockouts accurately single out gene functions? BMC Syst. Biol.
**2008**, 2, 50. [Google Scholar] [CrossRef][Green Version] - Anastassiou, D. Computational analysis of the synergy among multiple interacting genes. Mol. Syst. Biol.
**2007**, 3, 83. [Google Scholar] [CrossRef] - White, D.; Rabago-Smith, M. Genotype–phenotype associations and human eye color. J. Hum. Genet.
**2011**, 56, 5–7. [Google Scholar] [CrossRef][Green Version] - Chan, T.E.; Stumpf, M.P.; Babtie, A.C. Gene regulatory network inference from single-cell data using multivariate information measures. Cell Syst.
**2017**, 5, 251–267. [Google Scholar] [CrossRef][Green Version]

**Figure 1.**(

**Top left**) When depicting a measure on the union of two sets $\mu (A\cup B)$, the area of each section can be used to represent the inequality (5) and hence the values $\mu (A\backslash B)$, $\mu (B\backslash A)$ and $\mu (A\cap B)$ correspond to the area of each section. This correspondence can be generalised to consider an arbitrary number of sets. (

**Bottom left**) When depicting the joint entropy $H(X,Y)$, the area of each section can also be used to represent the inequality (1) and hence the values $H\left(X\right|Y)$, $H\left(Y\right|X)$ and $I(X;Y)$ correspond to the area of each section. However, this correspondence does not generalise beyond two variables. (

**Right**) For example, when considering the entropy of three variables, the multivariate mutual information $I(X;Y;Z)$ cannot be accurately represented using an area since, as represented by the hatching, it is not non-negative.

**Figure 2.**(

**Left**) Indiana assumes that Alice’s information $h\left(x\right)$ is independent of Bob’s information $h\left(y\right)$ such that her information is given by $h\left(x\right)+h\left(y\right)$. (

**Middle**) Johnny knows the joint distribution ${p}_{XY}$, and hence his information is given by the joint information content $h(x,y)$. (

**Right**) There is no inequality that requires Johnny’s information to be no greater than Indiana’s assumed information, or vice versa. On the one hand, Johnny can have more information than Indiana since a joint realisation can be more surprising than both of the individual marginal realisations. On the other hand, Indiana can have more information than Johnny since a joint realisation can be less surprising than both of the individual marginal realisations occurring independently. Thus, as represented by the hatching, the mutual information content $i(x;y)$ is not non-negative.

**Figure 3.**(

**Left**) If Alice’s information $h\left(x\right)$ is greater than Bob’s information $h\left(y\right)$, then Eve’s information $h(x\bigsqcup y)$ is equal to Alice’s information $h\left(x\right)$. In effect, Eve is pessimistically assuming that information provided by the least surprising marginal realisation $h(x\sqcap y)$ is already provided by the most surprising marginal realisation $h(x\bigsqcup y)$, i.e., Bob’s information $h\left(y\right)$ is a subset of Alice’s information $h\left(x\right)$. From this perspective, Eve gets unique information from Alice relative to Bob $h(x\backslash y)$, but does not get any unique information from Bob relative to Alice $h(y\backslash x)=0$. (

**Right**) Although for each realisation Eve can only get unique information from either Alice or Bob, it is possible that Eve can expect to get unique information from both Alice and Bob on average. (Do not confuse this representation of the union entropy with the diagram that represents the joint entropy in Figure 1).

**Figure 4.**(

**Left**) This Venn diagram shows how the synergistic information $h(x\oplus y)$ can be defined by comparing the joint information content $h(x,y)$ from Figure 2 to the union information content $h(x\bigsqcup y)$ from Figure 3. Note that, for this particular realisation, we are assuming that $h\left(x\right)>h\left(y\right)$. It also provides a visual representation of the decomposition (40) of the joint information content $h(x,y)$. (

**Right**) By rearranging the marginal entropies such that they match Figure 2 (albeit with different sizes here), it is easy to see why the mutual information content $i(x;y)$ is equal to the intersection information content $h(x\sqcap y)$ minus the synergistic information content $h(x\oplus y)$, c.f. (38).

**Figure 6.**(

**Bottom right**) The distributive lattices $\langle \mathit{x},h(\phantom{\rule{0.166667em}{0ex}}\bigsqcup \phantom{\rule{0.166667em}{0ex}}),h(\phantom{\rule{0.166667em}{0ex}}\sqcap \phantom{\rule{0.166667em}{0ex}})\rangle $ of information contents for two and three and three observers. It is also important to note that, by replacing h, x, y and z with H, X, Y and Z, respectively, we can obtain the distributive lattices for entropy. In fact, this is crucial since Property 6 enables us to reduce the distributive lattice of information contents to a mere total order; however, this property does not apply to the entropies, and hence we cannot further simplify the lattice of entropies. (

**Top left**) By the fundamental theorem of distributive lattices, the distributive lattices of marginal information contents has a one-to-one correspondence with the lattice of sets. Notice that the lattice for two sets corresponds to the Venn diagram for entropies in Figure 3.

**Figure 7.**(

**Left**) The total order of marginal information contents for two and three observers, whereby we have assumed that Alice’s information $h\left(x\right)$ is greater than Bob’s information $h\left(y\right)$, which is greater than Charlie’s information $h\left(z\right)$. It is important to note that taking the expectation value over these information contents for each realisation, which may each have a different total orders, yields entropies which are merely partially ordered. It is for this reason that Property 6 does not apply to entropies. (

**Right**) The Venn diagrams corresponding to the total order for for two and three observers and their corresponding information contents. Notice that the total order for two sets corresponds to the Venn diagram for information contents in Figure 3.

**Figure 8.**Similar to Figure 4, this Venn diagram shows how the synergistic information $h(x\oplus y\oplus z)$ can be defined by comparing the joint information content $h(x,y,z)$ to the union information content $h(x\bigsqcup y\bigsqcup z)$. Note that, for this particular realisation, we are assuming that $h\left(x\right)>h\left(y\right)>h\left(z\right)$.

**Figure 9.**(

**Top-middle and left**) The join semi-lattice $\u2329\mathit{h}\phantom{\rule{0.166667em}{0ex}};(\phantom{\rule{0.166667em}{0ex}},)\u232a$ for $n=2$ and $n=3$ marginal observers. Johnny’s information is always given by the joint information content at the top of the semi-lattice, while the information content of individuals such as Alice, Bob and Charlie who observe single realisations are found at the bottom of the semi-lattice. The information content of joint marginal observers such as Joanna, Jonas and Joan are found in between these two extremities. (

**Bottom-middle and right**) The meet semi-lattice $\u2329\mathit{h}\phantom{\rule{0.166667em}{0ex}};\sqcap \u232a$ for $n=2$ and $n=3$ marginal observers. Since these two semi-lattices are not connect by absorption, their combined structure is not a lattice.

**Figure 10.**(

**Top left**) The redundancy lattices $\langle \mathcal{A}\left(\mathit{x}\right),\u2aaf\rangle $ of information contents for two and three and three observers. Each note in the lattice corresponds to an element in $\mathcal{A}\left(\mathit{x}\right)$ from (69), while the ordering between elements is given by ⪯ from (70). (

**Bottom right**) The partial information contents ${h}_{\partial}\left(\alpha \right)$ corresponding to the redundancy lattices of information contents for two and three observers.

**Figure 11.**This Venn provides a visual representation of the decomposition of the joint entropy $H(X,Y,Z)$. This decomposition is given by replacing x, y, z and h with X, Y, Z and H in (85), respectively.

© 2020 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (http://creativecommons.org/licenses/by/4.0/).

## Share and Cite

**MDPI and ACS Style**

Finn, C.; Lizier, J.T. Generalised Measures of Multivariate Information Content. *Entropy* **2020**, *22*, 216.
https://doi.org/10.3390/e22020216

**AMA Style**

Finn C, Lizier JT. Generalised Measures of Multivariate Information Content. *Entropy*. 2020; 22(2):216.
https://doi.org/10.3390/e22020216

**Chicago/Turabian Style**

Finn, Conor, and Joseph T. Lizier. 2020. "Generalised Measures of Multivariate Information Content" *Entropy* 22, no. 2: 216.
https://doi.org/10.3390/e22020216