Numerical Evaluation of Gaussian Mixture Entropy

Abstract

We develop an approximation method for the differential entropy $h\left(\mathbf{X}\right)$ of a q-component Gaussian mixture in ${\mathbb{R}}^n$. We provide two examples of approximations using our method denoted by ${\overline{h}}_{C,m}^{\mathrm{Taylor}}\left(\mathbf{X}\right)$ and ${\overline{h}}_C^{\mathrm{Polyfit}}\left(\mathbf{X}\right)$. We show that ${\overline{h}}_{C,m}^{\mathrm{Taylor}}\left(\mathbf{X}\right)$ provides an easy-to-compute lower bound to $h\left(\mathbf{X}\right)$, while ${\overline{h}}_C^{\mathrm{Polyfit}}\left(\mathbf{X}\right)$ provides an accurate and efficient approximation to $h\left(\mathbf{X}\right)$. ${\overline{h}}_C^{\mathrm{Polyfit}}\left(\mathbf{X}\right)$ is more accurate than known bounds and is conjectured to be much more resilient than other approximations in high dimensions.

1. Introduction

1.1. Differential Entropy of Gaussian Mixtures

A Gaussian mixture is a probability density function on ${\mathbb{R}}^n$ of the form $f\left(\mathbf{x}\right)={\sum }_{j=1}^q{p}_j{\mathcal{N}}_{{\mathbf{w}}_j,K_j}\left(\mathbf{x}\right)$, where $\mathbf{x}\in {\mathbb{R}}^n$, the $p_j>0$ are weights satisfying ${\sum }_{j=1}^q{p}_j=1$, and ${\mathcal{N}}_{\mathbf{w},K}$ stands for a Gaussian distribution with mean $\mathbf{w}$ and covariance matrix K. In other words, f is a mixture of q Gaussian pdfs, with arbitrary weights, allowing each Gaussian to be of different shape and displacement. A mixture like this occurs, e.g., when a stochastic process, with probability mass function $p_1,\dots ,p_q$, determines which distribution holds for $\mathbf{x}$. Such situations occur in many scientific areas. Furthermore, Gaussian mixtures are often employed as function approximators, by virtue of being smooth and localized. They have been used in a wide variety of studies, e.g., on diffusion models in physics and machine learning [1,2,3,4], non-adiabatic thermodynamics [5], wireless communication [6], wireless authentication [7], Byzantine attacks [8], gene expression [9,10], dark matter kinematics [11], and quasar spectra [12]. The differential entropy $h\left(X\right)$ of a continuous random variable $X\in \mathcal{X}$, with probability density function $f_X$, is defined as $h\left(X\right)=-{\int }_{\mathcal{X}}{f}_X\left(x\right)ln{f}_X\left(x\right)\mathrm{d}x$. It represents the continuum limit of the (discrete) Shannon entropy for the probability mass function $f_X\left(x_i\right)▵x$, where the volume $▵x$ is sent to zero and the infinite contribution $ln{\displaystyle \frac{1}{▵x}}$ is subtracted. The differential entropy of a random vector admits the same functional form as in the univariate case, and our results hold more generally in n dimensions.

In thermodynamics, the dissipation of heat associated with the irreversible erasure of information is at least $k_{b}Tln2$ as shown by Landauer [13]. In particular, Gaussian mixtures have been used to model the state of a bit before erasure, and their differential entropy is important to further bound the energy, hence improving upon the Landauer bound [14]. In [15], it is shown how neuronal populations encode information in the presence of an external source. The mutual information between the input signal and the stationary distribution of the neuronal populations is used to quantify the information encoded by the neurons, and positivity hinges on the time scale of the input. Although when the input evolves rapidly the mutual information vanishes, it is shown that when the input signal is of sufficiently low frequency, the stationary distribution of the neural populations is a Gaussian mixture, and evaluating its entropy accurately is crucial in that setting. In machine learning, the Active Diffusion Subsampling method [16] uses so-called particles whose distribution is approximated by a Gaussian mixture. The entropy of the mixture plays a crucial role in choosing new samples.

Analytically computing or estimating the differential entropy of a Gaussian mixture is a notoriously difficult problem. Even numerical evaluation can be problematic in high dimensions. The special case of a single Gaussian is simple and yields $h\left(X\right)={\displaystyle \frac{1}{2}}lndet\left(2\pi eK\right)$.

1.2. Related Work

Various methods have been proposed to approximate the differential entropy of a Gaussian mixture. A loose upper bound can be obtained from the fact that a Gaussian distribution maximizes entropy, given the first and second moments. This yields [17] $h\left(X\right)\le {\displaystyle \frac{1}{2}}lndet(2\pi e\Sigma )$, with $\Sigma ={\sum }_{i=1}^q{p}_i(w_i{w}_i^{\mathrm{T}}+K_i)-\left({\sum }_{i=1}^q{p}_i{w}_i\right){\left({\sum }_{j=1}^q{p}_j{w}_j\right)}^{\mathrm{T}}$. In [18], a numerical approximation was given for the case $q=2,n=1,w_2=-w_1$. A sequence of upper bounds for arbitrary q was obtained in [17], but only for $n=1$. Furthermore, the results are not in closed form, and the bounds in the sequence do not get progressively tighter. Another method is to replace the density f inside the logarithm by a single Gaussian $\overline{f}$ which has mean and covariance equal to the mixture. This leads to an exact expression containing the relative entropy, $h\left(X\right)={\displaystyle \frac{1}{2}}lndet(2\pi e\Sigma )-D\left(f\right||\overline{f})$; one can then find approximations or bounds for the relative entropy, as in [19,20]. This is done with Monte Carlo integration, which has the drawback of being computationally demanding and not giving an analytic expression.

In [21], an approximation was obtained by performing a Taylor expansion of $lnf\left(x\right)$ in the variables $x-w_j$. To avoid the need for expansion powers above 2, they introduced the trick of representing wide Gaussians approximately as the sum of several narrow Gaussians, in the f outside the logarithm. The Taylor expansion, as well as the splitting trick, introduces inaccuracies. As shown in [21], one can obtain a basic concavity deficit upper bound $h\left(X\right)\le {\sum }_{j=1}^q{p}_{j}ln{\displaystyle \frac{1}{p_j}}+{\sum }_{j=1}^q{p}_j{\displaystyle \frac{1}{2}}lndet\left(2\pi e{K}_j\right)$. This is significantly tighter than the above-mentioned bound ${\displaystyle \frac{1}{2}}ln[{\left(2\pi e\right)}^{n}det\Sigma ]$; it is exact in the case of a single Gaussian, and it gets arbitrarily close to the true value of $h\left(X\right)$ when the overlap between the components of the mixture becomes negligible. A refinement was also introduced by means of merging parts of the mixture that are clustered together. However, in configurations where the mixture components have significant overlap whilst keeping the mixture non-trivial, this bound becomes inaccurate.

In [22], tighter bounds are introduced for the differential entropy of general mixture distributions. In particular, the concavity deficit bound is refined as $h\left(X\right)\le ({sup}_i\left|\right|g_i-{\overline{g}}_i{\left|\right|}_{\mathrm{TV}}){\sum }_{j=1}^q{p}_{j}ln{\displaystyle \frac{1}{p_j}}+{\sum }_{j=1}^q{p}_j{\displaystyle \frac{1}{2}}lndet\left(2\pi e{K}_j\right)$, where $\left|\right|·{\left|\right|}_{\mathrm{TV}}$ denotes the total variation distance, $\left\{g_i\right\}$ are the components of the mixture, and ${\overline{g}}_j={\sum }_{i\ne j}{\displaystyle \frac{p_i}{1-p_j}}{g}_i$ is the mixture complement of $g_j$. Calculating the total variation distances between each component and its complement becomes computationally demanding when both the number of components and the dimension are large.

1.3. Contributions

We introduce a new method for estimating the differential entropy of a Gaussian mixture. We approximate $fln{\displaystyle \frac{1}{f}}$ by a polynomial $\mathrm{poly}\left(f\right)$. For any positive integer k, the power $f^k={(p_1{\mathcal{N}}_1+\dots +p_q{\mathcal{N}}_q)}^k$ can be written as a multinomial sum containing a product of powers of Gaussians. Hence, the integral $\int \mathrm{poly}\left(f\right(x\left)\right)\mathrm{d}x$ can be computed analytically. This yields a systematic way to approximate and/or lower bound $h\left(X\right)$ by analytic expressions.

• We consider polynomials that minimize the square error $\int \mathrm{weight}\left(s\right){[sln{\displaystyle \frac{1}{s}}-\mathrm{poly}\left(s\right)]}^2\mathrm{d}s$ on the range of f, with tunable function $\mathrm{weight}\left(s\right)=s^r$. We observe that negative r gives better results than positive r, which is to be expected since most of the volume in ${\mathbb{R}}^n$ has $f\left(x\right)$ close to zero. In particular, $r\approx -2$ performs best in the mixture configurations that we have studied, yielding relative errors in $h\left(X\right)$ of less than 1% already for the degree-3 polynomial fit.
• We consider the truncated Taylor series $-ln{\displaystyle \frac{f}{m}}\approx {\sum }_{k=1}^{\cdots }{\displaystyle \frac{1}{k}}{(1-{\displaystyle \frac{f}{m}})}^k$ for some constant m. This too gives rise to an approximation for $fln{\displaystyle \frac{1}{f}}$ that is polynomial in f. While not as accurate as the above-mentioned fit, it guarantees that each k-term has the same sign if $m\ge {max}_{x\in {\mathbb{R}}^n}f\left(x\right)$, and hence it gives rise to a sequence of increasingly accurate analytic lower bounds on $h\left(X\right)$. We observe that the relative error in $h\left(X\right)$ is still several percent even at polynomial degree 10.

In Section 2.1, we obtain a general recipe to approximate the differential entropy via a polynomial approximation as shown in Corollary 1. In Section 2.2, we apply Corollary 1 to the Taylor series of the logarithm to obtain a specific approximation for the differential entropy. Although it is not a very accurate approximation, it serves as an easy-to-compute lower bound. In Section 2.3, we apply Corollary 1 again using a polynomial approximation of $f\left(s\right)=-slns$, which gives an accurate approximation to the differential entropy. In Section 3, we show numerical results of our approximations for various configurations, and in Section 4, we assess our method and give pointers towards future work.

2. Polynomial Approximation

2.1. General Polynomial

Consider a random variable $\mathbf{X}\in {\mathbb{R}}^n$ whose probability density function (pdf) is a Gaussian mixture with weights ${\left\{p_i\right\}}_{i=1}^q$. The Gaussian pdfs have covariance matrices ${\left\{K_i\right\}}_{i=1}^q$, and they are centered on points ${\mathbf{w}}_1,\dots ,{\mathbf{w}}_q\in {\mathbb{R}}^n$.

$$\begin{array}{cc}\hfill & f_{\mathbf{X}}\left(\mathbf{x}\right)=\sum _{j=1}^q{p}_j{\mathcal{N}}_{{\mathbf{w}}_j,K_j}\left(\mathbf{x}\right)={\left(2\pi \right)}^{-{\displaystyle \frac{n}{2}}}\sum _{j=1}^q{p}_j{(det{K}_j)}^{-1/2}{e}^{-{\displaystyle \frac{1}{2}}{(\mathbf{x}-{\mathbf{w}}_j)}^{\mathrm{T}}{K}_j^{-1}(\mathbf{x}-{\mathbf{w}}_j)}.\hfill \end{array}$$

(1)

We let $g_i\left(\mathbf{x}\right)\equiv {\mathcal{N}}_{{\mathbf{w}}_i,K_i}\left(\mathbf{x}\right)$ denote the Gaussian pdf in the following. The differential entropy is $h\left(\mathbf{X}\right)=-{\mathbb{E}}_{\mathbf{X}}[ln{f}_{\mathbf{X}}]$, which can be written as follows:

$$\begin{array}{cc}\hfill & h\left(\mathbf{X}\right)=-{\mathbb{E}}_{\mathbf{X}}[ln{f}_{\mathbf{X}}]=-{\mathbb{E}}_{\mathbf{X}}[lnm{m}^{-1}{f}_{\mathbf{X}}]=-lnm-{\mathbb{E}}_{\mathbf{X}}\left[ln{\displaystyle \frac{f_{\mathbf{X}}}{m}}\right],\hfill \end{array}$$

(2)

where m can be chosen to enforce that the argument of the logarithm lies in a certain interval. We would like to approximate the entropy by means of a polynomial approximation in powers of $f_{\mathbf{X}}$:

$$-f_{\mathbf{X}}ln{\displaystyle \frac{f_{\mathbf{X}}}{m}}\approx \sum _{a=1}^C{c}_a{f}_{\mathbf{X}}^a,$$

(3)

In other words, we consider an order-C polynomial approximation. We set $c_0=0$ to avoid diverging integrals. Equation (3) is more relaxed than truncating a series of the form ${\sum }_{a=1}^{\infty }{c}_a{f}_{\mathbf{X}}^a$; i.e., we allow $\left\{c_a\right\}$ to depend on C. Using Equation (3), the differential entropy $h\left(\mathbf{X}\right)$ reads as follows:

$$h\left(\mathbf{X}\right)\approx -lnm+\sum _{a=1}^C{c}_a\int f_{\mathbf{X}}^a\left(\mathbf{x}\right)\mathrm{d}\mathbf{x},$$

(4)

and by choosing $\left\{c_a\right\}$ appropriately and analytically evaluating the integral, we obtain an efficient approximation for $h\left(\mathbf{X}\right)$. We start by rewriting the integral in Equation (4), and the result is Corollary 1.

Lemma 1. 

Let ${\left\{g_i\left(\mathbf{x}\right)\right\}}_{i=1}^q$ be given in accordance with Equation (1), $\widehat{t}\equiv (t_1,\dots ,t_q)$ s.t. $t_i\ge 0$ and ${\sum }_{i=1}^q{t}_i=a$. Furthermore, let $M^{-1}$ and $\mathit{\mu }$ be given by the following:

$$\begin{array}{cc}\hfill & M^{-1}=\sum _{j=1}^q{t}_j{K}_j^{-1},\hfill \end{array}$$

(5)

$$\begin{array}{cc}\hfill & \mathit{\mu }=\sum _{j=1}^q{t}_{j}M{K}_j^{-1}{\mathbf{w}}_j,\hfill \end{array}$$

(6)

and then we have the following:

$$\int \prod _{j=1}^q{g}_j^{t_j}\left(\mathbf{x}\right)\mathrm{d}\mathbf{x}=D\left(\widehat{t}\right),$$

(7)

where $D\left(\widehat{t}\right)$ is given by the following:

$$\begin{array}{cc}\hfill & D\left(\widehat{t}\right)={\left(2\pi \right)}^{-n(a-1)/2}\left(\prod _{i=1}^q{(det{K}_i)}^{-t_i/2}\right){(detM)}^{1/2}{e}^{-{\displaystyle \frac{1}{2}}\left({\sum }_{l=1}^q{t}_l{\mathbf{w}}_l^{\mathrm{T}}{K}_l^{-1}{\mathbf{w}}_l-{\mathit{\mu }}^{\mathrm{T}}{M}^{-1}\mathit{\mu }\right)}.\hfill \end{array}$$

(8)

Proof. 

See Appendix A. □

It follows that the integral in Equation (4) can be rewritten as shown in the following corollary.

Corollary 1. 

The differential entropy $h\left(\mathbf{X}\right)$ of the Gaussian mixture is approximated by the following:

$$\begin{array}{c}\hfill {\overline{h}}_{\overrightarrow{c},m}\left(\mathbf{X}\right)=-lnm+\sum _{a=1}^C{c}_a\sum _{\widehat{t}\in {\mathcal{T}}_a}\left({\displaystyle \genfrac{}{}{0pt}{}{a}{\widehat{t}}}\right)\left(\prod _{i=1}^q{p}_i^{t_i}\right)D\left(\widehat{t}\right),\end{array}$$

(9)

where ${\mathcal{T}}_a=\{\widehat{t}\in {\mathbb{Z}}_{+}^q:|\widehat{t}|=a\}$ and $\left({\displaystyle \genfrac{}{}{0pt}{}{a}{\widehat{t}}}\right)=\left({\displaystyle \genfrac{}{}{0pt}{}{a}{t_1\dots t_q}}\right)$.

Proof. 

See Appendix B. □

2.2. Taylor Series Approximation

We perform a Taylor series for the logarithm only, in order to obtain the coefficients $\left\{c_a\right\}$ in Equation (3). We define $Z\equiv 1-m^{-1}{f}_{\mathbf{X}}\left(\mathbf{X}\right)$ and we write:

$$\begin{array}{c}\hfill -f_{\mathbf{X}}ln{\displaystyle \frac{f_{\mathbf{X}}}{m}}=-f_{\mathbf{X}}ln(1-Z)=f_{\mathbf{X}}\sum _{k=1}^{\infty }{\displaystyle \frac{1}{k}}{Z}^k.\end{array}$$

(10)

The range of allowed m values is the one that keeps Z in the radius of convergence of the Taylor series: $\left|Z\right|<1$, i.e., $m\ge {max}_{\mathbf{x}\in {\mathbb{R}}^n}{f}_{\mathbf{X}}\left(\mathbf{x}\right)/2$. For $m={max}_{\mathbf{x}\in {\mathbb{R}}^n}{f}_{\mathbf{X}}\left(\mathbf{x}\right)/2$ then $Z\in [-1,1)$, while for $m={max}_{\mathbf{x}\in {\mathbb{R}}^n}{f}_{\mathbf{X}}\left(\mathbf{x}\right)$ then $Z\in [0,1)$. Furthermore, values above ${max}_{\mathbf{x}\in {\mathbb{R}}^n}{f}_{\mathbf{X}}\left(\mathbf{x}\right)$ such as $m={\sum }_i{p}_i{g}_i\left({\mathbf{w}}_i\right)$ shrink the range of Z from below towards 1. By performing the Taylor expansion for the logarithm, we obtain a series of the form given by Equation (3) as shown in Theorem 1.

Theorem 1. 

The procedure of performing the Taylor expansion of the logarithm as detailed in Equation (10) yields the following coefficients ${\left\{c_a^{\mathrm{Taylor}}\right\}}_{a=1}^C$ of the corresponding polynomial approximation in Equation (3):

$$c_a^{\mathrm{Taylor}}=\left\{\begin{array}{cc}{H}_{C-1},\hfill & a=1,\hfill \\ {\displaystyle \frac{{(-1)}^{a+1}}{m^{a-1}}}{\displaystyle \frac{1}{a-1}}\left({\displaystyle \genfrac{}{}{0pt}{}{C-1}{a-1}}\right),\hfill & \mathrm{otherwise}.\hfill \end{array}\right.,$$

(11)

and the corresponding differential entropy approximation as given by Equation (9) can be written as follows:

$${\overline{h}}_{C,m}^{\mathrm{Taylor}}\left(\mathbf{X}\right)=-lnm+H_{C-1}-m\sum _{a=1}^{C-1}{\displaystyle \frac{B_{a+1}}{a}}\left({\displaystyle \genfrac{}{}{0pt}{}{C-1}{a}}\right),$$

(12)

where $H_k$ is the k-th Harmonic number, and $\left\{B_a\right\}$ are given by the following:

$$B_a={\displaystyle \frac{{(-1)}^a}{m^a}}\sum _{\widehat{t}\in {\mathcal{T}}_a}\left({\displaystyle \genfrac{}{}{0pt}{}{a}{\widehat{t}}}\right)\left(\prod _{i=1}^q{p}_i^{t_i}\right)D\left(\widehat{t}\right).$$

(13)

Proof. 

See Appendix C. □

Note that one can show that $h\left(\mathbf{X}\right)$ is given by the infinite series:

$$h\left(\mathbf{X}\right)=-lnm-m\sum _{k=1}^{\infty }\sum _{a=0}^k{\displaystyle \frac{B_{a+1}}{k}}\left({\displaystyle \genfrac{}{}{0pt}{}{k}{a}}\right),$$

(14)

which is not the result of an infinite power series in $f_{\mathbf{X}}^a$ as evident by the a summation which is inside the k summation. Taking the first $C-1$ terms and changing the order of summations makes the coefficients $\left\{c_a^{\mathrm{Taylor}}\right\}$ dependent on C. Note that Theorem 1 with $m={max}_{\mathbf{x}\in {\mathbb{R}}^n}{f}_{\mathbf{X}}\left(\mathbf{x}\right)$ provides a lower bound for $h\left(\mathbf{X}\right)$. For some applications, having a bound is more important than having a good approximation.

2.3. Polyfit

2.3.1. General Polyfit

Suppose we have a function $f\left(s\right)$ defined on $\mathcal{I}=[a,b]$ that we wish to approximate via a polynomial expansion. We write

$$f\left(s\right)\approx \tilde{f}\left(s\right)=\sum _{i=1}^C{d}_{\mathcal{I},i}{s}^i,$$

(15)

and we define the error resulting from our estimation as follows:

$$E={\displaystyle \frac{1}{b-a}}{\int }_a^{b}w\left(s\right){(f\left(s\right)-\tilde{f}\left(s\right))}^2\mathrm{d}s,$$

(16)

where we consider the possibility of favoring portions of $\mathcal{I}$ more than others by choosing the weight function $w\left(s\right)$ appropriately, and $w\left(s\right)=1$ reduces E to the mean squared error. Our choice of w and $\mathcal{I}$ must be such that $M_{\mathcal{I}}$ in Lemma 2 is invertible.

Lemma 2. 

Let $M_{\mathcal{I}}$, ${\overrightarrow{d}}_{\mathcal{I}}$, and ${\overrightarrow{z}}_{\mathcal{I}}$ be given by the following:

$$\begin{array}{cc}\hfill & {\left(M_{\mathcal{I}}\right)}_{ij}\equiv {\int }_a^{b}w\left(s\right)s^{i+j}\mathrm{d}s,\hfill \end{array}$$

(17)

$$\begin{array}{cc}\hfill & {\left({\overrightarrow{d}}_{\mathcal{I}}\right)}_i\equiv d_{\mathcal{I},i},\hfill \end{array}$$

(18)

$$\begin{array}{cc}\hfill & {\left({\overrightarrow{z}}_{\mathcal{I}}\right)}_i\equiv {\int }_a^{b}w\left(s\right)f\left(s\right)s^i\mathrm{d}s,\hfill \end{array}$$

(19)

and then for invertible $M_{\mathcal{I}}$, the coefficients $d_{\mathcal{I},i}$ in Equation (15) that minimize the error function E in Equation (16) are given by the following:

$${\overrightarrow{d}}_{\mathcal{I}}=M_{\mathcal{I}}^{-1}{\overrightarrow{z}}_{\mathcal{I}}.$$

(20)

Proof. 

See Appendix D. □

2.3.2. Entropy Estimate Obtained from Polyfit

We now wish to use Lemma 2 to obtain an approximation for $h\left(\mathbf{X}\right)$. We first apply our estimator to the function $f\left(s\right)=-slns$ in the following lemma.

Lemma 3. 

Let $f\left(s\right)=-slns$ on $\mathcal{I}=(a,b]$, $r\in \mathbb{R}$ if $a>0$ and $r>-3$ if $a=0$. Let $w\left(s\right)=s^r$, and then the estimator $\tilde{f}\left(s\right)$ in Equation (15) is given by the following:

$$-slns\approx \sum _{i=1}^C{d}_{\mathcal{I},i}{s}^i,$$

(21)

where $d_{\mathcal{I},i}$ are obtained by solving Equation (20) with $M_{\mathcal{I}}$ and ${\overrightarrow{z}}_{\mathcal{I}}$ given by the following:

$$\begin{array}{cc}\hfill & {\left(M_{\mathcal{I}}\right)}_{ij}={\displaystyle \frac{b^{i+j+r+1}-a^{i+j+r+1}}{i+j+r+1}},\hfill \\ \hfill & {\left({\overrightarrow{z}}_{\mathcal{I}}\right)}_i={\displaystyle \frac{\left[b^{i+r+2}(1-(i+r+2)lnb)-(b\to a)\right]}{{(i+r+2)}^2}}.\hfill \end{array}$$

(22)

Proof. 

It follows from Equation (15), Lemma 2, and direct calculation. □

Corollary 2. 

Let $f\left(s\right)=-slns$ on $\mathcal{I}=(0,b]$, and $w\left(s\right)=s^r$ with $r>-3$; then, $\left\{d_{\mathcal{I},i}\right\}$ are given by the following:

$$d_{\mathcal{I},i}=\left\{\begin{array}{cc}{\tilde{d}}_1-lnb,\hfill & \mathrm{if}i=1\hfill \\ b^{1-i}{\tilde{d}}_i,\hfill & \mathrm{otherwise}\hfill \end{array}\right.,$$

(23)

where $\overrightarrow{\tilde{d}}$ is obtained by solving

$$\overrightarrow{\tilde{d}}={\tilde{M}}^{-1}\overrightarrow{\tilde{z}},$$

(24)

with $\tilde{M}$ and $\overrightarrow{\tilde{z}}$ given by the following:

$${\tilde{M}}_{ij}={\displaystyle \frac{1}{i+j+r+1}},{\tilde{z}}_i={\displaystyle \frac{1}{{(i+r+2)}^2}}.$$

(25)

Proof. 

See Appendix E. □

Corollary 2 shows that the problem of finding a polynomial fit for $f\left(s\right)=-slns$ on $(0,b]$ is equivalent to finding the inverse of $\tilde{M}$, which is independent of b. However, $\tilde{M}$ is an ill-conditioned matrix as the elements ${\tilde{M}}_{ij}$ become increasingly smaller for higher $i+j$. For example, if we take $r=-2$, then $\tilde{M}$ is the Hilbert matrix with inverse elements, ${\left(H^{-1}\right)}_{ij}={(-1)}^{i+j}(i+j-1)\left({\displaystyle \genfrac{}{}{0pt}{}{C+i-1}{C-j}}\right)\left({\displaystyle \genfrac{}{}{0pt}{}{C+j-1}{C-i}}\right){\left[\left({\displaystyle \genfrac{}{}{0pt}{}{i+j-2}{i-1}}\right)\right]}^2$, which become numerically unstable at $C\sim 10$. If we write $f_{\max }\equiv {max}_{\mathbf{x}\in {\mathbb{R}}^n}{f}_{\mathbf{X}}\left(\mathbf{x}\right)$, then for our Gaussian mixture $f_{\mathbf{X}}$, the possible values are in $(0,f_{\max }]$. In a very large part of ${\mathbb{R}}^n$, we find ourselves in the tails of the Gaussian mixture, which corresponds to $f_{\mathbf{X}}\left(\mathbf{x}\right)$ being close to zero. Hence, we want our weight function $w\left(s\right)$ to reflect this, and emphasize the region close to zero in the application of Corollary 2. Note that we can write

$$\begin{array}{cc}\hfill & h\left(\mathbf{X}\right)=-\int f_{\mathbf{X}}\left(\mathbf{x}\right)ln{f}_{\mathbf{X}}\left(\mathbf{x}\right)\mathrm{d}\mathbf{x}=-\int \left({\int }_0^{f_{\max }}\delta (s-f_{\mathbf{X}}\left(\mathbf{x}\right))slns\mathrm{d}s\right)\mathrm{d}\mathbf{x}\hfill \\ \hfill & =-{\int }_0^{f_{\max }}slns\left(\int \delta (s-f_{\mathbf{X}}\left(\mathbf{x}\right))\mathrm{d}\mathbf{x}\right)\mathrm{d}s={\int }_0^{f_{\max }}f\left(s\right)V\left(s\right)\mathrm{d}s,\hfill \end{array}$$

(26)

where $V\left(s\right)=\int \delta (s-f_{\mathbf{X}}\left(\mathbf{x}\right))\mathrm{d}\mathbf{x}$ can be thought of as the $(n-1)$-dimensional volume where the function $f_{\mathbf{X}}$ equals s. We see from Equation (26) that our entropy approximation is not only determined by how well we can estimate $f\left(s\right)$ using Corollary 2, but also by $V\left(s\right)$, which for Gaussian mixtures will put more weight around $s=0$. That is, we expect $r<0$ in Corollary 2 to produce better results for our entropy approximation than $r>0$, although they will produce worse estimates of $f\left(s\right)$ over any interval. Ideally, we set $w\left(s\right)=V\left(s\right)$ in Lemma 3, but $V\left(s\right)$ has no closed-form solution. This leads us to the following entropy approximation given by Theorem 2.

Theorem 2. 

Let $r>-3$, $\mathcal{I}=(0,f_{\max }]$; then, the entropy approximation ${\overline{h}}_C^{\mathrm{Polyfit}}\left(\mathbf{X}\right)$ based on Lemma 3, Corollary 2, and Corollary 1 is given by the following:

$${\overline{h}}_C^{\mathrm{Polyfit}}\left(\mathbf{X}\right)=\sum _{a=1}^C{c}_a^{\mathrm{Polyfit}}\sum _{\widehat{t}\in {\mathcal{T}}_a}\left({\displaystyle \genfrac{}{}{0pt}{}{a}{\widehat{t}}}\right)\left(\prod _{i=1}^q{p}_i^{t_i}\right)D\left(\widehat{t}\right),$$

(27)

where ${\overrightarrow{c}}^{\mathrm{Polyfit}}$ is given by the following:

$$c_a^{\mathrm{Polyfit}}=\left\{\begin{array}{cc}{\tilde{d}}_1-ln{f}_{\max },\hfill & \mathrm{if}a=1\hfill \\ f_{\max }^{1-a}{\tilde{d}}_a,\hfill & \mathrm{otherwise}\hfill \end{array}\right.,\overrightarrow{\tilde{d}}={\tilde{M}}^{-1}\overrightarrow{\tilde{z}},$$

(28)

with ${\tilde{M}}_{ij}$ and ${\tilde{z}}_i$ given by the following:

$${\tilde{M}}_{ij}={\displaystyle \frac{1}{i+j+r+1}},{\tilde{z}}_i={\displaystyle \frac{1}{{(i+r+2)}^2}}.$$

(29)

Proof. 

It follows from the direct application of Lemma 3, Corollary 2, and Corollary 1 to $-f_{\mathbf{X}}ln{f}_{\mathbf{X}}$. □

In Appendix F, we derive the volume function $V\left(s\right)$ for the trivial case $q=1$. Around $s\approx 0$, it blows up as $1/s$, with some additional logarithmic divergence; this gives some intuition as to why we get good results at $r\le -1$.

3. Numerical Results

3.1. Polyfit Results

We now present results of Theorem 2 applied to different Gaussian mixtures in order to estimate $h\left(\mathbf{X}\right)$. The performance metric we use is the percentage error compared to the exact value for the entropy. The mixtures we used to show the validity of our approximation are shown in

Table 1. The different Gaussian mixtures tested using Theorem 2. The resulting entropy errors appear in Figure 1.

q	n	Mixture Parameters
3	2	${\mathbf{w}}_1={(1,0)}^{\mathrm{T}},{\mathbf{w}}_2={(-1,0)}^{\mathrm{T}},{\mathbf{w}}_3={(0,1.5)}^{\mathrm{T}},K_i={\mathrm{I}}_2,\widehat{p}={(0.2,0.3,0.5)}^{\mathrm{T}}$
	2	${\mathbf{w}}_1={(0,0)}^{\mathrm{T}},{\mathbf{w}}_2={(-1.5,1.5)}^{\mathrm{T}},{\mathbf{w}}_3={(1.5,1.5)}^{\mathrm{T}},K_1=\left(\begin{array}{cc}1& 0\\ 0& 3\end{array}\right),K_2=\left(\begin{array}{cc}1& 0.2\\ 0.2& 1\end{array}\right)$, K3 = $\left(\begin{array}{cc}1& -0.2\\ -0.2& 1\end{array}\right)$, $\hat{p}={(0.2,0.3,0.5)}^{\mathrm{T}}$
4	3	${\mathbf{w}}_1={(0,0,0)}^{\mathrm{T}},{\mathbf{w}}_2={(-1.5,1.5,-1.5)}^{\mathrm{T}},{\mathbf{w}}_3={(1.5,1.5,1.5)}^{\mathrm{T}},{\mathbf{w}}_4={(1,1,1)}^{\mathrm{T}},K_i={\mathrm{I}}_3,$ $\hat{p}={(0.2,0.3,0.3,0.2)}^{\mathrm{T}}$
	8	${\mathbf{w}}_1={(0,\dots ,0)}^{\mathrm{T}},{\mathbf{w}}_2={(-1.5,1.5,\dots ,1.5)}^{\mathrm{T}},{\mathbf{w}}_3={(1.5,\dots ,1.5)}^{\mathrm{T}},{\mathbf{w}}_4={(1,\dots ,1)}^{\mathrm{T}},K_i={\mathrm{I}}_8,$ $\hat{p}={(0.2,0.3,0.3,0.2)}^{\mathrm{T}}$
5	4	${\mathbf{w}}_1={(0,0,0,0)}^{\mathrm{T}},{\mathbf{w}}_2={(-1.5,1.5,-1.5,1.5)}^{\mathrm{T}},{\mathbf{w}}_3={(1.5,1.5,1.5,1.5)}^{\mathrm{T}},{\mathbf{w}}_4={(1,1,1,1)}^{\mathrm{T}}$
		${\mathbf{w}}_5={(-3,3,-3,3)}^{\mathrm{T}},K_i={\mathrm{I}}_4,\widehat{p}={(0.2,0.3,0.3,0.1,0.1)}^{\mathrm{T}}$

. The examples cover various values of n and q. This is not meant to be all-encompassing, but it still covers a sufficient part of the parameter space to showcase the accuracy of our approximation.

From Figure 1, we see that for low-dimensional cases ($q=3,n=2$), our approximation converges quickly to the exact value of the entropy (within less than $1\%$ error), and we can fairly trust numerical accuracy for those cases. As for the optimal r value for the weight function, $r=-2$ is consistently the best choice for those parameters, with $r>0$ being less favorable, as predicted in Section 2.3.2.

At $q=4$, we start to see divergent behavior as n becomes larger. For the lower-dimensional case of $q=4,n=3$, the behavior is similar to previous cases with very high accuracy and $r=-2$ being the most favorable weight function. However, in the case of $q=4,n=8$, the result diverges as C grows larger, although up to $C=7$, the approximation is fairly accurate. Furthermore, the best-performing r in this case is $r=-2.5$ with an error percentage of around $-0.5\%$ at $C=6$. We conjecture that the divergent behavior in this case is caused by numerical inaccuracies in the evaluation of multinomial coefficients.

For $q=5,n=4$, we again have convergent behavior yielding an accurate approximation of the entropy with $r=-2$ being the best weight function. This prompts us to conclude that our method produces a very highly accurate approximation when $r\in [-2.5,-2]$ and C is kept between 3 and 8 in order to avoid computational inaccuracies. Note that the upper bound mentioned in the introduction $h\left(\mathbf{X}\right)\le {\sum }_{j=1}^q{p}_{j}ln{\displaystyle \frac{1}{p_j}}+{\sum }_{j=1}^q{p}_j{\displaystyle \frac{1}{2}}lndet\left(2\pi e{K}_j\right)$ produces values that are within 3–19% above the true entropy for the configurations in Table 1.

In Figure 2, we show Polyfit approximations of $f\left(s\right)=-slns$ as described in Lemma 3 for different values of r. We see clearly in Figure 2 that for $r=-2$, the approximation is overall inaccurate compared to $r=-1$ and $r=1$. However, $r=-2$ yields the best approximation for the entropy $h\left(\mathbf{X}\right)$. As we discussed in Section 2.3.2, the reason for this is that the value of $r=-2$ makes the approximation better around $s\approx 0$, which is where $V\left(s\right)$ is highest.

3.2. Taylor Series Results

We now show results for our approximation ${\overline{h}}_{C,m}^{\mathrm{Taylor}}\left(\mathbf{X}\right)$ based on the Taylor expansion of the logarithm as outlined in Theorem 1. We do not expect it to be as accurate as ${\overline{h}}_{C,m}^{\mathrm{Polyfit}}\left(\mathbf{X}\right)$; however, it provides us with an easy-to-compute lower bound for the entropy, as mentioned earlier in Section 2.2. We show the percentage error compared to the exact value of the entropy for different values of $\beta$ for two mixtures that we take from Table 1. We use the case of $q=3,n=2$ with non-spherical covariance matrices, as well as the case of $q=4,n=8$. The results are shown in Figure 3. As we can see from Figure 3, ${\overline{h}}_{C,m}^{\mathrm{Taylor}}\left(\mathbf{X}\right)$ is not as accurate as ${\overline{h}}_{C,m}^{\mathrm{Polyfit}}\left(\mathbf{X}\right)$ as expected, especially in the higher-dimensional case. However, it does provide monotonous and convergent behavior. Furthermore, the case $\beta =1/2$ is the fastest-converging case, which is also the case where computational inaccuracies start to appear the fastest as we increase C.

4. Discussion

Gaussian mixtures are one of the most applicable distributions in the literature, from wireless communications to physics and diffusion models. They have found success in either describing or modeling real life phenomena to a high degree of accuracy. One of the most important quantities in characterizing a distribution is the Shannon (differential) entropy, which, for Gaussian mixtures, does not have a known closed-form expression.

In this paper, we have derived a method by which the differential entropy of Gaussian mixtures can be approximated both accurately and efficiently. Our method enjoys a high degree of simplicity as it relies on the fact that for Gaussian mixtures “almost all the support has an image that is almost zero”. For various configurations, this trick allows our approximation to get very close to the true value for the differential entropy by summing only a small number of terms. Although some configurations are more difficult to approximate with a small number of terms, our method is accurate for a large number of randomly generated Gaussian mixtures (not reported in Section 3). This makes inaccuracy an exception that can be circumvented by avoiding numerical instabilities and taking into account higher-order terms.

It is not straightforward to benchmark against the existing literature. Many methods are at least partly numerical, whereas our approximation technique yields fully analytic expressions. The work [21] is closest to our technique in that it approximates $h\left(\mathbf{X}\right)$ with closed-form analytic expressions. However, there are significant qualitative differences. Since [21] tries to polynomially approximate $lnf\left(\mathbf{x}\right)$, the order C of their Taylor series has to increase with q as $C=2q$ (in the worst case) in order to correctly fit all the ‘bumps’ in the function. The error in $lnf\left(\mathbf{x}\right)$ then explodes as ${\left|\mathbf{x}\right|}^{2q}$ when $\left|\mathbf{x}\right|$ increases beyond a certain radius. This large error should be damped by the fact that it gets multiplied by the Gaussian tail of $f\left(\mathbf{x}\right)$. However, at large q, this can require drastic application of the splitting trick, which introduces errors. In contrast, our choice of polynomial order C is independent of q, and increasing q does not lead to errors. In our technique, inaccuracies are caused by the mismatch between the function $sln{\displaystyle \frac{1}{s}}$ and the polynomial fit around $s=0$. The performance of our technique is constrained by the fact that the function $sln{\displaystyle \frac{1}{s}}$ has infinite derivative at $s=0$ and hence cannot be fitted with a polynomial of finite degree at that point.

Our Polyfit method relies on choosing an appropriate weight function $w\left(s\right)$ as discussed in Section 2.3. Ideally for Gaussian mixtures, $w\left(s\right)$ should be optimally chosen as $V\left(s\right)$, which is difficult to estimate. However, from our experiments, we notice that a simple power $w\left(s\right)\propto s^r$ gives good results, especially for negative r around $r=-2$. At negative r, a lot of weight is given to small values of s. This makes sense because most of the n-dimensional volume in the integral $\int \mathrm{d}\mathbf{x}$ covers the tails of the distribution $f\left(\mathbf{X}\right)$. For a single Gaussian with spherical symmetry, it is easy to verify (see Appendix F) that the volume function $V\left(s\right)=\int \mathrm{d}\mathbf{x}\delta (s-f(\mathbf{x}\left)\right)$ behaves as $V\left(s\right)\propto {\displaystyle \frac{1}{s}}{(ln{\displaystyle \frac{f_{\max }}{s}})}^{{\displaystyle \frac{n}{2}}-1}$. At small s, this expression blows up faster than ${\displaystyle \frac{1}{s}}$ but slower than ${\displaystyle \frac{1}{s^2}}$. This suggests that r should lie in the vicinity of $[-2,-1]$. More extensive testing, with larger mixtures in higher dimensions, will show how generally applicable our method is. We have not exhaustively studied all possible polynomial fits. It is quite likely that some improvement can be gained by choosing a better weight function, e.g., one that more resembles $V\left(s\right)$.

Projection pursuit was introduced in 1974 as a method for analyzing multivariate data through its lower-dimensional projections [23]. However, exploring the full space of projections becomes increasingly impractical as the dimension grows [24]. Subsequent work addressed this issue by identifying “interesting” projections, namely those that reveal structure in the original data, through indices that measure departures from Gaussianity. One such index is differential entropy [25]. When the data arise from a finite mixture, skewness has also been used as a measure of non-normality for classifying projections [26,27]. Other approaches include the differential entropy estimator proposed in [28]. It would be of interest to compare these approaches with accurate estimation of differential entropy, using our method and taking our estimate as the projection index. In particular, an important question is whether maximizing skewness also leads to entropy optimization. A similar investigation could be carried out in relation to the work of Peña and Prieto [29,30,31]. These directions are left for future work.