# Exact Partial Information Decompositions for Gaussian Systems Based on Dependency Constraints

^{1}

^{2}

^{*}

## Abstract

**:**

## 1. Introduction

#### 1.1. The Partial Information Decomposition

- red, the information about Y that is shared, common or redundant between ${X}_{0}$ and ${X}_{1}$,
- unq0, the information about Y that is available only from ${X}_{0}$,
- unq1, the information about Y that is available only from ${X}_{1}$,
- syn, the information about Y that is only available when ${X}_{0}$ and ${X}_{1}$ are observed together.

#### 1.1.1. The Partial Information Decomposition for Gaussian Variables

#### 1.2. Unique Information via Dependency Constraints

## 2. An ${I}_{\mathrm{dep}}$ PID for Univariate Gaussian Predictors and Target

#### 2.1. Gaussian Graphical Models

#### 2.2. Maximum Entropy Distributions

**Proposition**

**1.**

#### 2.3. Mutual Information

#### 2.4. The ${I}_{\mathrm{dep}}$ PID for Univariate Gaussian Predictors and Target

**Proposition**

**2.**

- (a)
- The ${I}_{\mathit{dep}}$ PID possesses consistency as well as the core axioms of non-negativity, self-redundancy, monotonicity, symmetry and identity.
- (b)
- When unq0 is equal to b or d, the the redundancy component is zero.
- (c)
- When unq0 is equal to i, the redundancy and both unique informations are constant with respect to the correlation between the two predictors.
- (d)
- When the correlations between each predictor and the target are both non-zero, then unq0 is equal to either i or to k.
- (e)
- When unq0 is equal to k, the synergy component is zero.
- (f)
- The redundancy component in the ${I}_{\mathit{mmi}}$ PID is greater than or equal to the redundancy component in the ${I}_{\mathit{dep}}$ PID with equality if, and only if, at least one of the following conditions holds: (i) either predictor and the target are independent; (ii) either predictor is conditionally independent of the target given the other predictor.
- (g)
- The synergy component in the ${I}_{\mathit{mmi}}$ PID is greater than or equal to the synergy component in the ${I}_{\mathit{dep}}$ PID with equality if, and only if, at least one of the following conditions holds: (i) either predictor and the target are independent; (ii) either predictor is conditionally independent of the target given the other predictor.
- (h)
- The ${I}_{\mathit{dep}}$ and ${I}_{\mathit{mmi}}$ PIDs are identical when either ${X}_{0}$ and Y are conditionally independent given ${X}_{1}$ or ${X}_{1}$ and Y are conditionally independent given ${X}_{0}$, and in particular they are identical for models ${U}_{1}\dots {U}_{6}$. In model ${U}_{7}$ the synergy component of ${I}_{\mathit{dep}}$ is zero.

#### 2.5. Some Examples

**Example**

**1.**

**Example**

**2.**

**Example**

**3.**

**Example**

**4.**

PID | unq0 | unq1 | red | syn |

${I}_{\mathrm{dep}}$ | 0.2877 | 0.2877 | 0.1981 | 0.4504 |

${I}_{\mathrm{mmi}}$ | 0 | 0 | 0.4587 | 0.7380 |

PID | unq0 | unq1 | red | syn |

${I}_{\mathrm{dep}}$ | 0.2324 | 0.3068 | 0.1623 | 0.4921 |

${I}_{\mathrm{mmi}}$ | 0 | 0.0744 | 0.3948 | 0.7245 |

**Example**

**5.**

PID | unq0 | unq1 | red | syn |

${I}_{\mathrm{dep}}$ | 0.0048 | 0.1476 | 0.3726 | 0 |

${I}_{\mathrm{mmi}}$ | 0 | 0.1427 | 0.3775 | 0.0048 |

#### 2.6. Graphical Illustrations

## 3. Multivariate Continuous Predictors and Target

#### 3.1. Properties of the Matrices P, Q, R, and the Inverse Matrix of ${\Sigma}_{Z}$

**Lemma**

**1.**

**Lemma**

**2.**

#### 3.2. Block Gaussian Graphical Models

#### 3.3. Maximum Entropy Distributions

**Proposition**

**3.**

#### 3.4. Mutual Information

#### 3.5. The ${I}_{\mathrm{dep}}$ PID for Multivariate Gaussian Predictors and Targets

**Proposition**

**4.**

- (a)
- This ${I}_{\mathit{dep}}$ PID possesses consistency as well as the core axioms of non-negativity, self-redundancy, monotonicity, symmetry and identity.
- (b)
- When unq0 is equal to b or d, the the redundancy component is zero.
- (c)
- When unq0 is equal to i, the redundancy and both unique informations are constant with respect to the correlation matrix P between the two predictors, ${\mathbf{X}}_{0},{\mathbf{X}}_{1}$.
- (d)
- When neither predictor and the target are independent, then unq0 is equal to either i or to k.
- (e)
- When unq0 is equal to k, the synergy component is zero.
- (f)
- The redundancy component in the ${I}_{\mathit{mmi}}$ PID is greater than or equal to the redundancy component in the ${I}_{\mathit{dep}}$ PID with equality if, and only, if at least one of the following conditions holds: (i) either predictor and the target are independent; (ii) either predictor is conditionally independent of the target given the other predictor.
- (g)
- The synergy component in the ${I}_{\mathit{mmi}}$ PID is greater than or equal to the synergy component in the ${I}_{\mathit{dep}}$ PID with equality if, and only, if at least one of the following conditions holds: (i) either predictor and the target are independent; (ii) either predictor is conditionally independent of the target given the other predictor.
- (h)
- The ${I}_{\mathit{dep}}$ and ${I}_{\mathit{mmi}}$ PIDs are identical when either ${\mathbf{X}}_{0}$ and $\mathbf{Y}$ are conditionally independent given ${\mathbf{X}}_{1}$ or ${\mathbf{X}}_{1}$ and $\mathbf{Y}$ are conditionally independent given ${\mathbf{X}}_{0}$, and in particular they are identical for models ${M}_{1}\dots {M}_{6}$. In model ${M}_{7}$ the synergy component of ${I}_{\mathit{dep}}$ is zero.

#### 3.6. Examples and Illustrations

$({\mathit{n}}_{\mathbf{0}},{\mathit{n}}_{\mathbf{1}},{\mathit{n}}_{\mathbf{2}})$ | $(\mathit{p},\mathit{q},\mathit{r})$ | PID | unq0 | unq1 | red | syn |

$(3,4,3)$ | $(-0.15,0.15,0.15)$ | ${I}_{\mathrm{dep}}$ | 0.1227 | 0.1865 | 0.0406 | 2.4772 |

${I}_{\mathrm{mmi}}$ | 0 | 0.0638 | 0.1632 | 2.6000 | ||

$(4,4,2)$ | $(-0.2,-0.2,0.3)$ | ${I}_{\mathrm{dep}}$ | 0.0893 | 0.7293 | 0.1889 | 0.0087 |

${I}_{\mathrm{mmi}}$ | 0 | 0.6401 | 0.2782 | 0.0980 | ||

$(4,2,4)$ | $(-0.1,0.15,-0.2)$ | ${I}_{\mathrm{dep}}$ | 0.2336 | 0.1899 | 0.0883 | 0.0345 |

${I}_{\mathrm{mmi}}$ | 0.0437 | 0 | 0.2782 | 0.2234 |

**Example**

**6.**

PID | unq0 | unq1 | red | syn |

${I}_{\mathrm{dep}}$ | 0.4077 | 0.0800 | 0.0232 | 0.1408 |

${I}_{\mathrm{mmi}}$ | 0.3277 | 0 | 0.1032 | 0.2209 |

PID | unq0 | unq1 | red | syn |

${I}_{\mathrm{dep}}$ | 0.3708 | 0.0186 | 0.0601 | 0 |

${I}_{\mathrm{mmi}}$ | 0.3522 | 0 | 0.0787 | 0.0186 |

## 4. Discussion

## Supplementary Materials

## Acknowledgments

## Author Contributions

## Conflicts of Interest

## Appendix A. Proof of Proposition 2

## Appendix B. Proof of Matrix Lemmas

- (i)
- The matrix M is positive definite if and only if A and $C-{B}^{T}{A}^{-1}B$ are positive definite.
- (ii)
- The matrix M is positive definite if and only if C and $A-B{C}^{-1}{B}^{T}$ are positive definite.
- (iii)
- $\left|M\right|=\left|A\right||D-{B}^{T}{A}^{-1}B|.$

**Proof of Lemma 1**

**Proof of Lemma 2**

## Appendix C. Proof of Proposition 4

## Appendix D. Computation of the Multivariate I_{dep} PID

## Appendix E. Deviance Tests

## References

- Williams, P.L.; Beer, R.D. Nonnegative Decomposition of Multivariate Information. arXiv, 2010; arXiv:1004.2515. [Google Scholar]
- James, R.G.; Emenheiser, J.; Crutchfield, J.P. Unique Information via Dependency Constraints. arXiv, 2017; arXiv:1709.06653. [Google Scholar]
- Harder, M.; Salge, C.; Polani, D. Bivariate measure of redundant information. Phys. Rev. E
**2013**, 87. [Google Scholar] [CrossRef] [PubMed] [Green Version] - Griffith, V.; Koch, C. Quantifying synergistic mutual information. In Guided Self-Organization: Inception. Emergence, Complexity and Computation; Springer: Berlin/Heidelberg, Germany, 2014; Volume 9, pp. 159–190. [Google Scholar]
- Bertschinger, N.; Rauh, J.; Olbrich, E.; Jost, J.; Ay, N. Quantifying Unique Information. Entropy
**2014**, 16, 2161–2183. [Google Scholar] [CrossRef] - Ince, R.A.A. Measuring multivariate redundant information with pointwise common change in surprisal. Entropy
**2017**, 19, 318. [Google Scholar] [CrossRef] - Chicharro, D. Quantifying multivariate redundancy with maximum entropy decompositions of mutual information. arXiv, 2017; arXiv:1708.03845. [Google Scholar]
- Finn, C.; Lizier, J.T. Pointwise Information Decomposition using the Specificity and Ambiguity Lattices. arXiv, 2018; arXiv:1801.09010. [Google Scholar]
- Rauh, J.; Bertschinger, N.; Olbrich, E.; Jost, J. Reconsidering unique information: Towards a multivariate information decomposition. In Proceedings of the 2014 IEEE International Symposium on Information Theory (ISIT), Honolulu, HI, USA, 29 June–4 July 2014; pp. 2232–2236. [Google Scholar]
- Rauh, J. Secret sharing and shared information. arXiv, 2017; arXiv:1706.06998. [Google Scholar]
- Rauh, J.; Banerjee, P.K.; Olbrich, E.; Jost, J.; Bertschinger, N. On extractable shared information. arXiv, 2017; arXiv:1701.07805. [Google Scholar]
- Barrett, A.B. An exploration of synergistic and redundant information sharing in static and dynamical Gaussian systems. Phys. Rev. E
**2015**, 91. [Google Scholar] [CrossRef] [PubMed] - Whittaker, J. Graphical Models in Applied Multivariate Statistics; Wiley: Chichester, UK, 2008. [Google Scholar]
- Edwards, D. Introduction to Graphical Modelling; Springer: New York, NY, USA, 2000. [Google Scholar]
- Lauritzen, S.L. Graphical Models; Oxford University Press: Oxford, UK, 1996. [Google Scholar]
- Dempster, A. Covariance Selection. Biometrics
**1972**, 28, 157–175. [Google Scholar] [CrossRef] - Harrison, L.; Penny, W.D.; Friston, K. Multivariate autoregressive modeling of fMRI time series. NeuroImage
**2003**, 19, 1477–1491. [Google Scholar] [CrossRef] - Schlögl, A.; Supp, G. Analyzing event-related EEG data with multivariate autoregressive parameters. Prog. Brain Res.
**2006**, 159, 135–147. [Google Scholar] [PubMed] - Magri, C.; Whittingstall, K.; Singh, V.; Logothetis, N.K.; Panzeri, S. A toolbox for the fast information analysis of multiple-site LFP, EEG and spike train recordings. BMC Neurosci.
**2009**, 10, 81. [Google Scholar] [CrossRef] [PubMed] - Ince, R.A.A.; Giordano, B.L.; Kayser, C.; Rousselet, G.A.; Gross, J.; Schyns, P.G. A Statistical Framework for Neuroimaging Data Analysis Based on Mutual Information Estimated via a Gaussian Copula. Hum. Brain Mapp.
**2017**, 38, 1541–1573. [Google Scholar] [CrossRef] [PubMed] - Olbrich, E.; Bertschinger, N.; Rauh, J. Information decomposition and synergy. Entropy
**2015**, 17, 3501–3517. [Google Scholar] [CrossRef] - Faes, F.; Marinazzo, D.; Stramaglia, S. Multiscale Information Decomposition: Exact Computation for Multivariate Gaussian Processes. Entropy
**2017**, 19, 408. [Google Scholar] [CrossRef] - Stramaglia, S.; Wu, G.-R.; Pellicoro, M.; Marinazzo, D. Expanding the transfer entropy to identify information circuits in complex systems. Phys. Rev. E
**2012**, 86. [Google Scholar] [CrossRef] [PubMed] [Green Version] - Mardia, K.V.; Kent, J.T.; Bibby, J.M. Multivariate Analysis; Academic Press: London, UK, 1979. [Google Scholar]
- Cover, T.M.; Thomas, J.A. Elements of Information Theory; Wiley-Interscience: New York, NY, USA, 1991. [Google Scholar]
- Horn, R.A.; Johnson, C.R. Matrix Analysis; Cambridge University Press: New York, NY, USA, 1985. [Google Scholar]
- Aitchison, J.; Kay, J.W.; Lauder, I.J. Statistical Concepts and Applications in Clinical Medicine; Chapman & Hall: Boca Raton, FL, USA, 2005. [Google Scholar]
- Seibold, D.R.; McPhee, R.D. Commonality Analysis: A Method for Decomposing Explained Variance in Multiple regression Analyses. Hum. Commun. Res.
**1979**, 5, 355–365. [Google Scholar] [CrossRef] - Ray-Mukherjee, J.; Nimon, K.; Mukherjee, S.; Morris, D.W.; Slotow, R.; Hamer, M. Using commonality analysis in multiple regressions: A tool to decompose regression effects in the face of multicollinearity. Methods Ecol. Evol.
**2014**, 5, 320–328. [Google Scholar] [CrossRef] - McPhee, R.D.; Seibold, D.R. Rationale, Procedures, and Applications for Decomposition of Explained Variation in Multiple Regression Analyses. Commun. Res.
**1979**, 6, 345–384. [Google Scholar] [CrossRef] - Pourahmadi, M.; Noorbaloochi, S. Multivariate time series analysis of neuroscience data: some challenges and opportunities. Curr. Opin. Neurobiol.
**2016**, 37, 12–15. [Google Scholar] [CrossRef] [PubMed] - Krumin, M.; Shoham, S. Multivariate Autoregressive Modeling and Granger Causality Analysis of Multiple Spike Trains. Comput. Intell. Neurosci.
**2010**, 2010. [Google Scholar] [CrossRef] [PubMed] - Dalhaus, R. Graphical interaction models of multivariate time series. Metrika
**2000**, 51, 157–172. [Google Scholar] [CrossRef] - Brillinger, D.R. Remarks concerning the graphical models for time series and point processes. Braz. Rev. Econ.
**1996**, 16, 1–23. [Google Scholar] [CrossRef] - Songsiri, J.; Dahl, J.; Vandenberghe, L. Graphical models of autoregressive processes. In Convex Optimization in Signal Processing and Communications; Palomar, D.P., Eldar, Y.C., Eds.; Cambridge University Press: Cambridge, UK, 2009; pp. 89–116. [Google Scholar]
- Meyer, C.D. Matrix Analysis and Applied Linear Algebra; Society for Applied and Industrial Mathematics: Philadelphia, PA, USA, 2000. [Google Scholar]

**Figure 1.**A dependency lattice of models (based on [2]). Edges coloured green (b, d, i, k) correspond to adding the constraint ${X}_{0}Y$ to the model immediately below. Edges coloured red (c, f, h, j) correspond to adding the constraint ${X}_{1}Y$ to the model immediately below.

**Figure 2.**A dependency lattice of models (based on [2]). Edges coloured green (b, d, i, k) correspond to adding the term ${X}_{0}Y$ to the model immediately below. Edges coloured red (c, f, h, j) correspond to adding the term ${X}_{1}Y$ to the model immediately below. The two relevant sub-lattices are shown here.

**Figure 3.**The ${I}_{\mathrm{mmi}}$ & ${I}_{\mathrm{dep}}$ PID components are plotted for a range of values of the correlation (p) between the two predictors. Two combinations of the correlations ($q,r$) between each predictor and the target are displayed. The total mutual information $I({X}_{0},{X}_{1};Y)$ is also shown as a dashed black curve.

**Figure 4.**In (

**a**,

**b**), the ${I}_{\mathrm{mmi}}$ & ${I}_{\mathrm{dep}}$ PIDs are plotted for a range of values of the correlation (p) between the two predictors. One combination of the correlations ($q,r$) between each predictor and the target are displayed. In (

**c**,

**d**), the ${I}_{\mathrm{mmi}}$ & ${I}_{\mathrm{dep}}$ PID are plotted for a range of allowable values of q, where q is equal to r, for $p=0.25$. The total mutual information $I({X}_{0},{X}_{1};Y)$ is also shown as a dashed black curve.

**Figure 5.**Regions in $(q,r)$ space in which the synergy component in the ${I}_{\mathrm{dep}}$ PID is equal to zero, plotted for four different values of p. Also, the determinant of ${\Sigma}_{Z}$ is positive.

**Figure 6.**An illustration of a block graphical model for the random vectors ${\mathbf{X}}_{0}$, ${\mathbf{X}}_{1}$ and $\mathbf{Y}$. ${\mathbf{X}}_{0}$ contains four random variables, while ${\mathbf{X}}_{1}$ has three and $\mathbf{Y}$ has two. This model expresses the conditional independence of ${\mathbf{X}}_{0}$ and $\mathbf{Y}$ given ${\mathbf{X}}_{1}$. In this model, the bivariate marginals ${\mathbf{X}}_{0}{\mathbf{X}}_{1}$ and ${\mathbf{X}}_{1}\mathbf{Y},$ as well as lower-order marginals, are fixed.

**Figure 7.**A dependency lattice of block graphical models. Edges coloured green (b, d, i, k) correspond to adding the set of constraints within ${\mathbf{X}}_{\mathbf{0}}\mathbf{Y}$ to the model immediately below. Edges coloured red (c, f, h, j) correspond to adding the set of constraints within ${\mathbf{X}}_{\mathbf{1}}\mathbf{Y}$ to the model immediately below. The two relevant sub-lattices are shown here.

**Figure 8.**The ${I}_{\mathrm{mmi}}$ and ${I}_{\mathrm{dep}}$ PID components are plotted for a range of values of the correlation (p) between the two predictors. Two combinations of the correlations ($q,r$) between each predictor and the target are displayed. The total mutual information $I({X}_{0},{X}_{1};Y)$ is also shown as a dashed black curve.

**Figure 9.**The ${I}_{\mathrm{mmi}}$ and ${I}_{\mathrm{dep}}$ PID components are plotted for a range of values of the correlation (q) between the predictor ${\mathbf{X}}_{0}$ and the target $\mathbf{Y}$. Two combinations of the correlations ($q,r$) between each predictor and the target are displayed. The total mutual information $I({X}_{0},{X}_{1};Y)$ is also shown as a dashed black curve.

**Table 1.**Graphical models and independences for the probability distribution of $\mathbf{Z}$. The vertices for random variables, ${X}_{0},{X}_{1},Y$ are denoted by $0,1,2$, respectively. Edges are denoted by pairs of vertices, such as $(1,2)$. In the column of independences, for example, $1\perp \phantom{\rule{0.0pt}{0ex}}\phantom{\rule{0.0pt}{0ex}}\perp 2|0$ indicates that ${X}_{1}$ and Y are conditionally independent given ${X}_{0}$ (based on [13], p. 61).

Model | Independences | Edge Set | Diagram | Description |
---|---|---|---|---|

${G}_{1}$ | $1\perp \phantom{\rule{0.0pt}{0ex}}\phantom{\rule{0.0pt}{0ex}}\perp 2|0,\phantom{\rule{0.166667em}{0ex}}0\perp \phantom{\rule{0.0pt}{0ex}}\phantom{\rule{0.0pt}{0ex}}\perp 2|1$ | $\left\{\right\}$ | Mutual independence | |

$1\perp \phantom{\rule{0.0pt}{0ex}}\phantom{\rule{0.0pt}{0ex}}\perp 2|0$ | ||||

${G}_{2}$ | $2\perp \phantom{\rule{0.0pt}{0ex}}\phantom{\rule{0.0pt}{0ex}}\perp 0|1,\phantom{\rule{0.166667em}{0ex}}2\perp \phantom{\rule{0.0pt}{0ex}}\phantom{\rule{0.0pt}{0ex}}\perp 1|0$ | $\left\{\right(0,1\left)\right\}$ | Independent subsets | |

${G}_{3}$ | $1\perp \phantom{\rule{0.0pt}{0ex}}\phantom{\rule{0.0pt}{0ex}}\perp 0|2,\phantom{\rule{0.166667em}{0ex}}1\perp \phantom{\rule{0.0pt}{0ex}}\phantom{\rule{0.0pt}{0ex}}\perp 2|0$ | $\left\{\right(0,2\left)\right\}$ | Independent subsets | |

${G}_{4}$ | $0\perp \phantom{\rule{0.0pt}{0ex}}\phantom{\rule{0.0pt}{0ex}}\perp 1|2,\phantom{\rule{0.166667em}{0ex}}0\perp \phantom{\rule{0.0pt}{0ex}}\phantom{\rule{0.0pt}{0ex}}\perp 2|1$ | $\left\{\right(1,2\left)\right\}$ | Independent subsets | |

${G}_{5}$ | $1\perp \phantom{\rule{0.0pt}{0ex}}\phantom{\rule{0.0pt}{0ex}}\perp 2|0$ | $\left\{\right(0,1),(0,2\left)\right\}$ | One independence | |

${G}_{6}$ | $0\perp \phantom{\rule{0.0pt}{0ex}}\phantom{\rule{0.0pt}{0ex}}\perp 2|1$ | $\left\{\right(0,1),(1,2\left)\right\}$ | One independence | |

${G}_{7}$ | $0\perp \phantom{\rule{0.0pt}{0ex}}\phantom{\rule{0.0pt}{0ex}}\perp 1|2$ | $\left\{\right(0,2),(1,2\left)\right\}$ | One independence | |

${G}_{8}$ | None | $\left\{\right(0,1),(0,2),(1,2\left)\right\}$ | Complete interdependence |

**Table 2.**Conditional independence constraints satisfied by the Gaussian graphical models ${G}_{1}\dots {G}_{7}$ that are applied when determining the maximum entropy models ${U}_{1}\dots {U}_{7}$.

${U}_{1}:p=qr,q=pr,r=pq$ | ||

${U}_{2}:q=pr,r=pq$ | ${U}_{3}:p=qr,r=pq$ | ${U}_{4}:p=qr,q=pr$ |

${U}_{5}:r=pq$ | ${U}_{6}:q=pr$ | ${U}_{7}:p=qr$ |

**Table 3.**Covariance matrices, with corresponding concentration matrices, for the Gaussian graphical models which were derived as maximum entropy probability models in Proposition 1.

Model | ${\widehat{\mathbf{\Sigma}}}_{\mathit{i}}$ | $\widehat{{\mathit{K}}_{\mathit{i}}}$ |
---|---|---|

${U}_{1}:{X}_{0},{X}_{1},Y$ | $\left[\begin{array}{ccc}1& 0& 0\\ 0& 1& 0\\ 0& 0& 1\end{array}\right]$ | $\left[\begin{array}{ccc}1& 0& 0\\ 0& 1& 0\\ 0& 0& 1\end{array}\right]$ |

${U}_{2}:{X}_{0}{X}_{1},Y$ | $\left[\begin{array}{ccc}1& p& 0\\ p& 1& 0\\ 0& 0& 1\end{array}\right]$ | $\frac{1}{1-{p}^{2}}\left[\begin{array}{ccc}1& -p& 0\\ -p& 1& 0\\ 0& 0& 1-{p}^{2}\end{array}\right]$ |

${U}_{3}:{X}_{0}Y,{X}_{1}$ | $\left[\begin{array}{ccc}1& 0& q\\ 0& 1& 0\\ q& 0& 1\end{array}\right]$ | $\frac{1}{1-{q}^{2}}\left[\begin{array}{ccc}1& 0& -q\\ 0& 1-{q}^{2}& 0\\ -q& 0& 1\end{array}\right]$ |

${U}_{4}:{X}_{1}Y,{X}_{0}$ | $\left[\begin{array}{ccc}1& 0& 0\\ 0& 1& r\\ 0& r& 1\end{array}\right]$ | $\frac{1}{1-{r}^{2}}\left[\begin{array}{ccc}1-{r}^{2}& 0& 0\\ 0& 1& -r\\ 0& -r& 1\end{array}\right]$ |

${U}_{5}:{X}_{0}{X}_{1},{X}_{0}Y$ | $\left[\begin{array}{ccc}1& p& q\\ p& 1& pq\\ q& pq& 1\end{array}\right]$ | $\frac{1}{(1-{p}^{2})(1-{q}^{2})}\left[\begin{array}{ccc}1-{p}^{2}{q}^{2}& ({q}^{2}-1)p& ({p}^{2}-1)q\\ ({q}^{2}-1)p& 1-{q}^{2}& 0\\ ({p}^{2}-1)q& 0& 1-{p}^{2}\end{array}\right]$ |

${U}_{6}:{X}_{0}{X}_{1},{X}_{1}Y$ | $\left[\begin{array}{ccc}1& p& pr\\ p& 1& r\\ pr& r& 1\end{array}\right]$ | $\frac{1}{(1-{p}^{2})(1-{r}^{2})}\left[\begin{array}{ccc}1-{r}^{2}& ({r}^{2}-1)p& 0\\ ({r}^{2}-1)p& 1-{p}^{2}{r}^{2}& ({p}^{2}-1)r\\ 0& ({p}^{2}-1)r& 1-{p}^{2}\end{array}\right]$ |

${U}_{7}:{X}_{0}Y,{X}_{1}Y$ | $\left[\begin{array}{ccc}1& qr& q\\ qr& 1& r\\ q& r& 1\end{array}\right]$ | $\frac{1}{(1-{q}^{2})(1-{r}^{2})}\left[\begin{array}{ccc}1-{r}^{2}& 0& ({r}^{2}-1)q\\ 0& 1-{q}^{2}& ({q}^{2}-1)r\\ ({r}^{2}-1)q& ({q}^{2}-1)r& 1-{q}^{2}{r}^{2}\end{array}\right]$ |

${U}_{8}:{X}_{0}{X}_{1},{X}_{0}Y,{X}_{1}Y$ | $\left[\begin{array}{ccc}1& p& q\\ p& 1& r\\ q& r& 1\end{array}\right]$ | $\frac{1}{|{\Sigma}_{Z}|}\left[\begin{array}{ccc}1-{r}^{2}& qr-p& pr-q\\ qr-p& 1-{q}^{2}& pq-r\\ pr-q& pq-r& 1-{p}^{2}\end{array}\right]$ |

${U}_{8}:\frac{1}{2}\mathrm{log}\left(\frac{1-{p}^{2}}{1-{p}^{2}-{q}^{2}-{r}^{2}+2pqr}\right)$ | ${U}_{4}:I({X}_{1};Y)$ |

${U}_{7}:\frac{1}{2}\mathrm{log}\left(\frac{1-{q}^{2}{r}^{2}}{(1-{q}^{2})(1-{r}^{2})}\right)$ | ${U}_{3}:I({X}_{0};Y)$ |

${U}_{6}:I({X}_{1};Y)=\frac{1}{2}\mathrm{log}\left(\frac{1}{1-{r}^{2}}\right)$ | ${U}_{2}:0$ |

${U}_{5}:I({X}_{0};Y)=\frac{1}{2}\mathrm{log}\left(\frac{1}{1-{q}^{2}}\right)$ | ${U}_{1}:0$ |

**Table 5.**Expression for the edge values in the dependency lattice in Figure 2 that are used to determine the unique informations.

$b=I({X}_{0};Y)=\frac{1}{2}\mathrm{log}\left(\frac{1}{1-{q}^{2}}\right)$ | $c=I({X}_{1};Y)=\frac{1}{2}\mathrm{log}\left(\frac{1}{1-{r}^{2}}\right)$ |

$d=I({X}_{0};Y)$ | $f=I({X}_{1};Y)$ |

$i=\frac{1}{2}\mathrm{log}\left(\frac{1-{q}^{2}{r}^{2}}{(1-{q}^{2})(1-{r}^{2})}\right)-I({X}_{1};Y)$ | $h=\frac{1}{2}\mathrm{log}\left(\frac{1-{q}^{2}{r}^{2}}{(1-{q}^{2})(1-{r}^{2})}\right)-I({X}_{0};Y)$ |

$k=\frac{1}{2}\mathrm{log}\left(\frac{1-{p}^{2}}{1-{p}^{2}-{q}^{2}-{r}^{2}+2pqr}\right)-I({X}_{1};Y)$ | $j=\frac{1}{2}\mathrm{log}\left(\frac{1-{p}^{2}}{1-{p}^{2}-{q}^{2}-{r}^{2}+2pqr}\right)-I({X}_{0};Y)$ |

**Table 6.**Conditional independence constraints satisfied by the block Gaussian graphical models ${G}_{1}\dots {G}_{8}$ that are applied when determining the maximum entropy models ${M}_{1}\dots {M}_{8}$.

${M}_{1}:P=Q{R}^{T},Q=PR,R={P}^{T}Q$ | ||

${M}_{2}:Q=PR,R={P}^{T}Q$ | ${M}_{3}:P=Q{R}^{T},R={P}^{T}Q$ | ${M}_{4}:P=Q{R}^{T},Q=PR$ |

${M}_{5}:R={P}^{T}Q$ | ${M}_{6}:Q=PR$ | ${M}_{7}:P=Q{R}^{T}$ |

Model | ${\widehat{\mathbf{\Sigma}}}_{\mathit{i}}$ | Model | ${\widehat{\mathbf{\Sigma}}}_{\mathit{i}}$ |
---|---|---|---|

${M}_{1}:{\mathbf{X}}_{0},{\mathbf{X}}_{1},\mathbf{Y}$ | $\left[\begin{array}{ccc}{I}_{{n}_{0}}& 0& 0\\ 0& {I}_{{n}_{1}}& 0\\ 0& 0& {I}_{{n}_{2}}\end{array}\right]$ | ${M}_{5}:{\mathbf{X}}_{0}{\mathbf{X}}_{1},{\mathbf{X}}_{0}\mathbf{Y}$ | $\left[\begin{array}{ccc}{I}_{{n}_{0}}& P& Q\\ {P}^{T}& {I}_{{n}_{1}}& {P}^{T}Q\\ {Q}^{T}& {Q}^{T}P& {I}_{{n}_{2}}\end{array}\right]$ |

${M}_{2}:{\mathbf{X}}_{0}{\mathbf{X}}_{1},\mathbf{Y}$ | $\left[\begin{array}{ccc}{I}_{{n}_{0}}& P& 0\\ {P}^{T}& {I}_{{n}_{1}}& 0\\ 0& 0& {I}_{{n}_{2}}\end{array}\right]$ | ${M}_{6}:{\mathbf{X}}_{0}{\mathbf{X}}_{1},{\mathbf{X}}_{1}\mathbf{Y}$ | $\left[\begin{array}{ccc}{I}_{{n}_{0}}& P& PR\\ {P}^{T}& {I}_{{n}_{1}}& R\\ {R}^{T}{P}^{T}& {R}^{T}& {I}_{{n}_{2}}\end{array}\right]$ |

${M}_{3}:{\mathbf{X}}_{0}\mathbf{Y},{\mathbf{X}}_{1}$ | $\left[\begin{array}{ccc}{I}_{{n}_{0}}& 0& Q\\ 0& {I}_{{n}_{1}}& 0\\ {Q}^{T}& 0& {I}_{{n}_{2}}\end{array}\right]$ | ${M}_{7}:{\mathbf{X}}_{0}\mathbf{Y},{\mathbf{X}}_{1}\mathbf{Y}$ | $\left[\begin{array}{ccc}{I}_{{n}_{0}}& Q{R}^{T}& Q\\ R{Q}^{T}& {I}_{{n}_{1}}& R\\ {Q}^{T}& {R}^{T}& {I}_{{n}_{2}}\end{array}\right]$ |

${M}_{4}:{\mathbf{X}}_{1}\mathbf{Y},{\mathbf{X}}_{0}$ | $\left[\begin{array}{ccc}{I}_{{n}_{0}}& 0& 0\\ 0& {I}_{{n}_{1}}& R\\ 0& {R}^{T}& {I}_{{n}_{2}}\end{array}\right]$ | ${M}_{8}:{\mathbf{X}}_{0}{\mathbf{X}}_{1},{\mathbf{X}}_{0}\mathbf{Y},{\mathbf{X}}_{1}\mathbf{Y}$ | $\left[\begin{array}{ccc}{I}_{{n}_{0}}& P& Q\\ {P}^{T}& {I}_{{n}_{1}}& R\\ {Q}^{T}& {R}^{T}& {I}_{{n}_{2}}\end{array}\right]$ |

${M}_{8}:\frac{1}{2}\mathrm{log}\frac{\left|\begin{array}{c}{I}_{{n}_{1}}-{P}^{T}P\end{array}\right|}{\left|\begin{array}{c}{\Sigma}_{Z}\end{array}\right|}$ | ${M}_{4}:I({\mathbf{X}}_{1};\mathbf{Y})$ |

${M}_{7}:\frac{1}{2}\mathrm{log}\frac{\left|\begin{array}{c}{I}_{{n}_{1}}-R{Q}^{T}Q{R}^{T}\end{array}\right|}{\left|\begin{array}{c}{I}_{{n}_{2}}-{Q}^{T}Q\end{array}\right|\left|\begin{array}{c}{I}_{{n}_{2}}-{R}^{T}R\end{array}\right|}$ | ${M}_{3}:I({\mathbf{X}}_{0};\mathbf{Y})$ |

${M}_{6}:I({\mathbf{X}}_{1};\mathbf{Y})=\frac{1}{2}\mathrm{log}\frac{1}{\left|\begin{array}{c}{I}_{{n}_{2}}-{R}^{T}R\end{array}\right|}$ | ${M}_{2}:0$ |

${M}_{5}:I({\mathbf{X}}_{0};\mathbf{Y})=\frac{1}{2}\mathrm{log}\frac{1}{\left|\begin{array}{c}{I}_{{n}_{2}}-{Q}^{T}Q\end{array}\right|}$ | ${M}_{1}:0$ |

**Table 9.**Expression for the edge values in the dependency lattice in Figure 7 that are used to determine the unique informations.

$b=d=I({\mathbf{X}}_{0};\mathbf{Y})=\frac{1}{2}\mathrm{log}\frac{1}{\left|\begin{array}{c}{I}_{{n}_{2}}-{Q}^{T}Q\end{array}\right|}$ | $c=f=I({\mathbf{X}}_{1};\mathbf{Y})=\frac{1}{2}\mathrm{log}\frac{1}{\left|\begin{array}{c}{I}_{{n}_{2}}-{R}^{T}R\end{array}\right|}$ |

$i=\frac{1}{2}\mathrm{log}\frac{\left|\begin{array}{c}{I}_{{n}_{1}}-R{Q}^{T}Q{R}^{T}\end{array}\right|}{\left|\begin{array}{c}{I}_{{n}_{2}}-{Q}^{T}Q\end{array}\right|\left|\begin{array}{c}{I}_{{n}_{2}}-{R}^{T}R\end{array}\right|}-I({\mathbf{X}}_{1};\mathbf{Y})$ | $h=\frac{1}{2}\mathrm{log}\frac{\left|\begin{array}{c}{I}_{{n}_{1}}-R{Q}^{T}Q{R}^{T}\end{array}\right|}{\left|\begin{array}{c}{I}_{{n}_{2}}-{Q}^{T}Q\end{array}\right|\left|\begin{array}{c}{I}_{{n}_{2}}-{R}^{T}R\end{array}\right|}-I({\mathbf{X}}_{0};\mathbf{Y})$ |

$k=\frac{1}{2}\mathrm{log}\frac{\left|\begin{array}{c}{I}_{{n}_{1}}-{P}^{T}P\end{array}\right|}{\left|\begin{array}{c}{\Sigma}_{Z}\end{array}\right|}-I({\mathbf{X}}_{1};\mathbf{Y})$ | $j=\frac{1}{2}\mathrm{log}\frac{\left|\begin{array}{c}{I}_{{n}_{1}}-{P}^{T}P\end{array}\right|}{\left|\begin{array}{c}{\Sigma}_{Z}\end{array}\right|}-I({\mathbf{X}}_{0};\mathbf{Y})$ |

© 2018 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (http://creativecommons.org/licenses/by/4.0/).