The Sampling Distribution of the Total Correlation for Multivariate Gaussian Random Variables

Abstract

The sampling distribution of the total correlation (TC) for a d-dimensional standardized multivariate Gaussian random variable with an identity covariance matrix is derived. It is shown to be the distribution of a sum of generalized beta random variables. It is also shown that, for large dimension and sample size, a central limit theorem holds, providing a Gaussian approximation to the sampling distribution for high dimensional data.

1. Introduction

Mutual information quantifies the information shared between two random variables [1,2,3]. This concept can be been generalized to d variables in a variety of ways [4,5,6,7], with the most direct generalization being Watanabe’s total correlation (TC),

$$T\left(\mathbf{X}\right)\equiv {\displaystyle \sum _{i=1}^d}h\left(X_i\right)-h\left(\mathbf{X}\right)$$

(1)

where $\mathbf{X}$ is a vector whose components are the d random variables $X_1,\dots ,X_d$, and for continuous random variables, $h\left(X_i\right)$ is the differential entropy of $X_i$ and $h\left(\mathbf{X}\right)$ is the joint differential entropy of $\mathbf{X}$.

Total correlation is also sometimes called multivariate mutual information, and it is the Kullback–Leibler divergence between the joint density of $\mathbf{X}$ and the density obtained by taking the product of the marginal densities of the $X_i$. Thus, the total correlation $T\left(\mathbf{X}\right)$ quantifies, in a quite general sense, the information shared among all the d random variables. The total correlation is non-negative and in the case where all d random variables are mutually independent we have $T\left(\mathbf{X}\right)=0$ [7,8]. For the special case where $\mathbf{X}$ is multivariate Gaussian with arbitrary mean and covariance matrix $\Sigma$, the total correlation can be written explicitly as

$$\begin{array}{cc}\hfill T\left(\mathbf{X}\right)& =\frac{1}{2}{\displaystyle \sum _{i=1}^d}log{\sigma }_{ii}^2-\frac{1}{2}log|\Sigma |\hfill \end{array}$$

(2)

where ${\sigma }_{ij}^2$ is the $ij$th entry of $\Sigma$. When the $X_i$ are independent we have ${\sigma }_{ij}^2=0$ for all $i\ne j$ and so $log|\Sigma |=log{\sigma }_{11}^2{\sigma }_{22}^2\cdots {\sigma }_{dd}^2$, giving $T\left(\mathbf{X}\right)=0$ in Equation (2) as expected.

The total correlation provides a natural way to quantify dependencies among a set of random variables. For example, often we seek to determine if a set of random variables are mutually independent because dependency among variables can indicate interesting and meaningful relationships in nature. To do so one can take a sample from the unknown distribution and compute the total correlation from this sample. Even if the random variables are mutually independent, however, the total correlation measured using such a finite sample will typically be positive (rather than zero) simply because of sampling variation. Therefore, it is of interest to know the sampling distribution of the total correlation under independence. Once we have the sampling distribution we can then perform statistical tests of independence. Here we derive the sampling distribution of (2) in the case where the $X_i$ are standardized (i.e., zero mean, unit variance), independent, Gaussian random variables.

Previous authors have proposed exact expressions for the mean and variance of the sample total correlation [9,10]. In fact, Guerrero (Section 2.1 of [9]) derived a moment generating function for the sample total correlation using the distribution of the log-determinant of a Wishart matrix (see Wilks [11,12]). Unfortunately the asymptotic approximation of Guerrero’s result does not match the results of Marralec [10] suggesting that one of the two is incorrect. We will resolve this discrepancy by deriving the moment generating function directly from our expression for the probability density function of the sample total correlation. In the limit of large sample size our results match those presented in Section 4.1 of Marralec [10], suggesting that the moment generating function of [9] is incorrect.

2. Definitions and Preliminaries

Let $\mathbf{X}$ represent a d-variate zero mean Gaussian random variable with covariance matrix $\Sigma =I_d$ where $I_d$ is the d-dimensional identity matrix. Let $\{{\mathbf{x}}_1,\cdots ,{\mathbf{x}}_n\}$ denote a sample of n draws from the distribution of $\mathbf{X}$. We focus on the case where $n\ge d$. The sample covariance matrix is $\widehat{\Sigma }=(1/n){\sum }_{i=1}^n{\mathbf{x}}_i{\mathbf{x}}_i^{\prime }=\left\{{\widehat{\sigma }}_{ij}\right\}$ and $n\widehat{\Sigma }$ is Wishart distributed with n degrees of freedom, which we denote as $n\widehat{\Sigma }\sim W(\Sigma ,d,n)$. From Equation (2) the sample total correlation is then also a random variable and is computed as

$$\begin{array}{cc}\hfill {\widehat{T}}_{d,n}\left(\mathbf{X}\right)& =\frac{1}{2}{\displaystyle \sum _{i=1}^d}log{\widehat{{\sigma }_{ii}}}^2-\frac{1}{2}log|\widehat{\Sigma }|\hfill \end{array}$$

(3)

where the subscripts d and n indicate that $\widehat{T}$ is a family of random variables indexed by dimension and sample size.

Odell and Feiveson’s 1966 result [13] provides a convenient way to characterize a Wishart-distributed matrix. Suppose that $V_i^{\left(n\right)}$$(1\le i\le d)$ are independent chi-square random variables with $n-i+1$ degrees of freedom. Suppose that $N_{ij}$ are independent standardized normal random variables for $1\le i<j\le d$, also independent of every $V_i^{\left(n\right)}$. Now construct the random variables

$$\begin{array}{cc}\hfill b_{11}& =V_1^{\left(n\right)}\hfill \\ \hfill b_{jj}& =V_j^{\left(n\right)}+{\displaystyle \sum _{i=1}^{j-1}}{N}_{ij}^2,2\le j\le d\hfill \\ \hfill b_{1j}& =N_{1j}\sqrt{V_1^{\left(n\right)}}2\le j\le d\hfill \\ \hfill b_{ij}& =N_{ij}\sqrt{V_i^{\left(n\right)}}+{\displaystyle \sum _{k=1}^{i-1}}{N}_{ki}{N}_{kj},2<i<j\le d.\hfill \end{array}$$

(4)

Then the matrix $B=\left\{b_{ij}\right\}$ (with $b_{ij}=b_{ji}$) is Wishart-distributed $W(I_d,d,n)$ and thus we have

$$\begin{array}{cc}\hfill n{\widehat{\sigma }}_{ii}^2\sim b_{ii}& \sim V_i^{\left(n\right)}+A_{i}1<1\le d\hfill \end{array}$$

(5)

where $A_i$ are independent chi-square random variables with $i-1$ degrees of freedom and we define $A_1=0$. Now following [14] we can also define the lower-triangular matrix $T=\left\{t_{ij}\right\}$ as

$$\begin{array}{cc}\hfill t_{ii}& =\sqrt{V_i^{\left(n\right)}}\hfill \\ \hfill t_{ij}& =N_{ji}1\le j<i\le d\hfill \\ \hfill t_{ij}& =0i<j\le d\hfill \end{array}$$

(6)

and thus $B=T{T}^{\prime }$. Furthermore, $\left|B\right|=|T{T}^{\prime }{|=|T|}^2={\prod }_{i=1}^d{t}_{ii}^2={\prod }_{i=1}^d{V}_i^{\left(n\right)}$, revealing that

$$n^d\left|\widehat{\Sigma }\right|\sim {\displaystyle \prod _{i=1}^d}{V}_i^{\left(n\right)}.$$

(7)

Result (7) is a special case of results found in Wilks [11]. For analogous results involving complex matrices see Goodman [15].

3. The Sampling Distribution of the Total Correlation

With the above preliminaries the we can now state the following theorem.

Theorem 1

(The Sampling Distribution of TC). Consider a sample of size n from a set of d independent, standardized, Gaussian random variables, with $n\ge d$. The total correlation (TC) is distributed as

$$\begin{array}{cc}\hfill {\widehat{T}}_{d,n}\left(\mathbf{X}\right)& \sim \frac{1}{2}{\displaystyle \sum _{i=1}^{d-1}}log\left(1+\frac{i}{n-i}{F}_{i,n-i}\right)\hfill \end{array}$$

(8)

where $F_{i,n-i}$ are independent F-distributed random variables with i and $n-i$ degrees of freedom. Equivalently, (8) can be written as

$$\begin{array}{cc}\hfill {\widehat{T}}_{d,n}\left(\mathbf{X}\right)& \sim {\displaystyle \sum _{i=1}^{d-1}}{Y}_{i,n}\hfill \end{array}$$

(9)

where $Y_{i,n}$ is a beta-exponential random variable with probability density

$$f_{Y_{i,n}}\left(y\right)=\lambda \frac{{(1-e^{-\lambda y})}^{\frac{i}{2}-1}{\left(e^{-\lambda y}\right)}^{\frac{n-i}{2}}}{B(\frac{i}{2},\frac{n-i}{2})}y>0$$

having parameter $\lambda =2$.

Proof. 

Writing Equation (3) as

$$\begin{array}{cc}\hfill {\widehat{T}}_{d,n}\left(\mathit{X}\right)& =\frac{1}{2}log\frac{{\prod }_{i=1}^d{\widehat{{\sigma }_{ii}}}^2}{|\widehat{\Sigma }|}\hfill \end{array}$$

(10)

and using result (5) and (7) one obtains

$$\begin{array}{cc}\hfill {\widehat{T}}_{d,n}\left(\mathbf{X}\right)& \sim \frac{1}{2}log\frac{{\prod }_{i=1}^d\left(V_i^{\left(n\right)}+A_i\right)}{{\prod }_{i=1}^d{V}_i^{\left(n\right)}}\hfill \\ \hfill & \sim \frac{1}{2}log{\displaystyle \prod _{i=1}^d}\left(1+\frac{A_i}{V_i^{\left(n\right)}}\right)\hfill \\ \hfill & \sim \frac{1}{2}{\displaystyle \sum _{i=1}^d}log\left(1+\frac{A_i}{V_i^{\left(n\right)}}\right).\hfill \end{array}$$

Scaling each chi-square by their corresponding degrees of freedom and re-indexing, yields (8). Equivalently, if we define $Y_{i,n}=\frac{1}{2}log\left(1+\frac{i}{n-i}{F}_{i,n-i}\right)$ then ${\widehat{T}}_{d,n}\left(\mathbf{X}\right)\sim {\sum }_{i=1}^{d-1}{Y}_{i,n}$, and using standard techniques it be can shown that the random variable $Y_{i,n}$ has probability density

$$f_{Y_{i,n}}\left(y\right)=2\frac{{(1-e^{-2y})}^{\frac{i}{2}-1}{\left(e^{-2y}\right)}^{\frac{n-i}{2}}}{B(\frac{i}{2},\frac{n-i}{2})}y>0$$

where $B(x,y)$ is the beta function. ☐

Corollary 1.

The moment generating function for ${\widehat{T}}_{d,n}\left(\mathbf{X}\right)$ is

$$\begin{array}{cc}\hfill M_{d,n}\left(t\right)& ={\left[\frac{\mathsf{\Gamma }\left(\frac{n}{2}\right)}{\mathsf{\Gamma }\left(\frac{n-t}{2}\right)}\right]}^{d-1}{\displaystyle \prod _{i=1}^{d-1}}\frac{\mathsf{\Gamma }\left(\frac{n-i-t}{2}\right)}{\mathsf{\Gamma }\left(\frac{n-i}{2}\right)}\hfill \end{array}$$

(11)

where $\mathsf{\Gamma }\left(x\right)$ is the gamma function. The mean and variance of ${\widehat{T}}_{d,n}\left(\mathbf{X}\right)$ are therefore

$$\begin{array}{cc}\hfill {\mu }_{d,n}& =\frac{d-1}{2}\psi (n/2)-\frac{1}{2}{\displaystyle \sum _{i=1}^{d-1}}\psi \left(\frac{n-i}{2}\right)\hfill \\ \hfill {\sigma }_{d,n}^2& =-\frac{d-1}{4}{\psi }^{\left(1\right)}(n/2)+\frac{1}{4}{\displaystyle \sum _{i=1}^{d-1}}{\psi }^{\left(1\right)}\left(\frac{n-i}{2}\right)\hfill \end{array}$$

(12)

where $\psi \left(x\right)={\mathsf{\Gamma }}^{\prime }\left(x\right)/\mathsf{\Gamma }\left(x\right)$ is the digamma function and ${\psi }^{\left(k\right)}\left(x\right)$ denotes its $k^{th}$ derivative.

Proof. 

Taking $Y_{i,n}=\frac{1}{2}log\left(1+\frac{i}{n-i}{F}_{i,n-i}\right)$, the moment generating function for $Y_{i,n}$ is

$${\varphi }_{i,n}\left(t\right)=\mathbb{E}\left[e^{t{Y}_{i,n}}\right]=\frac{\mathsf{\Gamma }\left[\frac{n}{2}\right]\mathsf{\Gamma }\left[\frac{n-i-t}{2}\right]}{\mathsf{\Gamma }\left[\frac{n-i}{2}\right]\mathsf{\Gamma }\left[\frac{n-t}{2}\right]}.$$

The random variables in the sum ${\sum }_{i=1}^{d-1}{Y}_{i,n}$ are independent, and therefore the moment generating function $M_{d,n}\left(t\right)$ for ${\widehat{T}}_{d,n}\left(\mathbf{X}\right)$ is the appropriate product of the functions ${\varphi }_{i,n}\left(t\right)$. Equation (12) then follow directly from the properties of moment generating functions. ☐

Guerrero [9] obtained a formula for the mean and variance of ${\widehat{T}}_{d,n}\left(\mathbf{X}\right)$ (except for a typo in the variance) using Wilks’ [12] moment generating function for the generalized variance. These are remarkably close to (12), but the proposed moment generating function for the sample total correlation information provided in Guerrero [9] appears to be incorrect.

4. A Central Limit Theorem for the Total Correlation

Girkos central limit theorem [16] implies asymptotic normality of the sample log-determinant, as seen in the work of Bao et al., and Cai et al. [17,18]. This suggests the existence of a central limit theorem result for ${\widehat{T}}_{d,n}\left(\mathbf{X}\right)$ when the dimension d and sample size n are large. Here we provide such a result.

Define the mean and variance of $Y_{i,n}$ as $m_{i,n}=\mathbb{E}\left[Y_{i,n}\right]$ and $s_{i,n}^2=\mathbb{E}\left[{(Y_{i,n}-{\mu }_{i,n})}^2\right]$, and the mean-centered random variables $Y_{i,n}^{*}=Y_{i,n}-m_{i,n}$. Note that ${\sigma }_{d,n}^2={\sum }_{i=1}^{d-1}{s}_{i,n}^2$.

Theorem 2

(Asymptotic normality of TC). Suppose $n\to \infty$ and $d\to \infty$ in such a way that $n/d\to k$ where $1\le k<\infty$. Then

$$\frac{1}{{\sigma }_{d,n}^2}{\displaystyle \sum _{i=1}^{d-1}}{Y}_{i,n}^{*}\stackrel{}{\to }N(0,1)$$

(13)

where convergence is in distribution. Thus, for large n and d (with $n\ge d$) the total correlation ${\widehat{T}}_{d,n}\left(\mathbf{X}\right)$ is approximately normally distributed with mean and variance given by ${\mu }_{d,n}$ and ${\sigma }_{d,n}^2$ in Equations (12).

Proof. 

The $Y_{i,n}^{*}$ are a triangular array of random variables such that, for any fixed n the $Y_{i,n}^{*}$ ($1\le i\le d-1$) are independent. Thus, (13) will hold provided that the Lyapunov condition is satisfied [19]; namely, that there exists a $\delta >0$ such that

$$\underset{d,n\to \infty }{lim}\frac{1}{{\sigma }_{d,n}^{2+\delta }}{\displaystyle \sum _{i=1}^{d-1}}\mathbb{E}\left[\right|Y_{i,n}^{*}{|}^{2+\delta }]=0.$$

For $\delta =2$ the entries in Lyapunov’s summation represent each $Y_{i,n}$’s fourth central moment, for which the generating function is $C_{i,n}\left(t\right)=e^{-m_{i,n}t}{\varphi }_{i,n}\left(t\right)$. The summation therefore becomes

$$\begin{array}{ccc}\hfill {\displaystyle \sum _{i=1}^{d-1}}\mathbb{E}\left[{\left(Y_{i,n}^{*}\right)}^4\right]& =& {\displaystyle \sum _{i=1}^{d-1}}\frac{1}{16}\left(3{\left({\psi }^{\left(1\right)}\left(\frac{n-i}{2}\right)-{\psi }^{\left(1\right)}(n/2)\right)}^2+{\psi }^{\left(3\right)}\left(\frac{n-i}{2}\right)-{\psi }^{\left(3\right)}(n/2)\right)\hfill \\ \hfill & =& \frac{3}{16}{\displaystyle \sum _{i=1}^{d-1}}{\left({\psi }^{\left(1\right)}\left(\frac{n-i}{2}\right)-{\psi }^{\left(1\right)}(n/2)\right)}^2+\frac{1}{16}{\displaystyle \sum _{i=1}^{d-1}}{\psi }^{\left(3\right)}\left(\frac{n-i}{2}\right)-\frac{d-1}{16}{\psi }^{\left(3\right)}(n/2)\hfill \end{array}$$

while the denominator in Lyapunov’s condition is

$${\sigma }_{d,n}^4={\left(\frac{1}{4}{\displaystyle \sum _{i=1}^{d-1}}{\psi }^{\left(1\right)}\left(\frac{n-i}{2}\right)-\frac{d-1}{4}{\psi }^{\left(1\right)}(n/2)\right)}^2.$$

In Appendix A we show that

$$0\le \frac{3}{16}{\displaystyle \sum _{i=1}^{d-1}}{\left({\psi }^{\left(1\right)}\left(\frac{n-i}{2}\right)-{\psi }^{\left(1\right)}(n/2)\right)}^2+\frac{1}{16}{\displaystyle \sum _{i=1}^{d-1}}{\psi }^{\left(3\right)}\left(\frac{n-i}{2}\right)-\frac{d-1}{16}{\psi }^{\left(3\right)}(n/2)\le \frac{48}{n-d+1}$$

and, for any fixed $1\le k<\infty$, and for sufficiently large d and n with $n/d$ sufficiently close to k,

$$\frac{1}{4}{\left(ln\left(\frac{n}{n-d+1}\right)+\frac{d-1}{n(n-d+1)}-\frac{d-1}{2}\left(\frac{2}{n}+\frac{4}{n^2}\right)\right)}^2\le {\left(\frac{1}{4}{\displaystyle \sum _{i=1}^{d-1}}{\psi }^{\left(1\right)}\left(\frac{n-i}{2}\right)-\frac{d-1}{4}{\psi }^{\left(1\right)}(n/2)\right)}^2.$$

Therefore, for any fixed $1\le k<\infty$, and for sufficiently large d and n with $n/d$ sufficiently close to k, we have

$$0\le \frac{1}{{\sigma }_{d,n}^4}{\displaystyle \sum _{i=1}^{d-1}}\mathbb{E}\left[\right|Y_{i,n}^{*}{|}^4]\le \frac{\frac{48}{n-d+1}}{\frac{1}{4}{\left(ln\left(\frac{n}{n-d+1}\right)+\frac{d-1}{n(n-d+1)}-\frac{d-1}{2}\left(\frac{2}{n}+\frac{4}{n^2}\right)\right)}^2}.$$

(14)

Now first consider the case where $n=d$ (and therefore $k=1$). Then (14) simplifies to

$$0\le \frac{1}{{\sigma }_{d,n}^4}{\displaystyle \sum _{i=1}^{d-1}}\mathbb{E}\left[\right|Y_{i,n}^{*}{|}^4]\le \frac{48}{\frac{1}{4}{\left(lnn+\frac{n-1}{n}-\frac{n-1}{2}\left(\frac{2}{n}+\frac{4}{n^2}\right)\right)}^2}.$$

Taking the limit $n\to \infty$ yields zero on the right-hand side, verifying Lyapunov’s condition for $k=1$. Next, consider the case where $n>d$. Taking the limit in (14) as $n\to \infty$ and $d\to \infty$ in such a way that $n/d\to k$ where $1<k<\infty$, again we see that the right-hand side is zero. This verifies Lyapunov’s condition in the case where $k>1$, thereby completing the proof. ☐

5. Conclusions

The total correlation of a multivariate random variable (sometimes called multivariate mutual information) is the Kullback–Leibler divergence between the joint density of the random variable and the product of its marginal densities. It therefore provides a natural measure of the degree of independence of a set of random variables. In this paper we derived the sampling distribution of the total correlation for a d-dimensional standardized multivariate Gaussian random variable with identity covariance matrix, and showed that it is the distribution of a sum of generalized beta random variables. We also proved that, for large dimension and sample size, a central limit theorem holds, providing a Gaussian approximation to the sampling distribution for high dimensional data.