Expected Logarithm of Central Quadratic Form and Its Use in KL-Divergence of Some Distributions

Abstract

In this paper, we develop three different methods for computing the expected logarithm of central quadratic forms: a series method, an integral method and a fast (but inexact) set of methods. The approach used for deriving the integral method is novel and can be used for computing the expected logarithm of other random variables. Furthermore, we derive expressions for the Kullback–Leibler (KL) divergence of elliptical gamma distributions and angular central Gaussian distributions, which turn out to be functions dependent on the expected logarithm of a central quadratic form. Through several experimental studies, we compare the performance of these methods.

1. Introduction

Expected logarithm of random variables usually appears in the expressions of important quantities like entropy and Kullback–Leibler (KL) divergence [1,2,3]. The second kind moment is an important statistics method used in estimation problems [4,5]. It also appears in an important class of inference algorithms called the variational Bayesian inference [6,7]. Furthermore, the geometric mean of a random variable which has been used in economics [8,9] is equal to the exponential of the expected logarithm of that random variable.

Central quadratic forms (CQFs) have many applications, and most of them stem from the fact that they are asymptotically equivalent to many statistics for testing null hypotheses. They are used for finding the number of components in mixtures of Gaussians [10], to test goodness-of-fit for some distributions [11] and as test statistics for dimensionality reduction in inverse regression [12].

In this paper, we develop three algorithms for computing the expected logarithm of CQFs. There is a need to develop special algorithms for it because CQFs do not have a closed-form probability density function, which makes the computation of their expected logarithms difficult. Although there is a vast literature on many different ways for calculating probability distributions of CQFs (see [13,14,15,16]), we have not found any work on calculating their expected logarithms. It is worth noting that one of our three algorithms is based upon works for computing the probability density function of CQFs using a series of gamma random variables [13,14]. We also derive expressions for the KL-divergence of two distributions that are subclasses of generalized elliptical distributions. These are zero-mean elliptical gamma (ZEG) distribution and angular central Gaussian (ACG) distribution. The only term in their KL-divergences that can not be computed in terms of elementary functions is an expected logarithm of a CQF, which can be computed by one of our developed algorithms.

The KL-divergence or the relative entropy was first introduced in [17] as a generalization of Shannon’s definition of information [18]. This divergence has been used extensively by statisticians and engineers. Many popular divergence classes like f-divergence and alpha-divergence have been introduced as generalizations to this divergence [19]. This divergence has several invariance properties like scale invariance that makes it an interesting dissimilarity measure in statistical inference problems [20]. KL-divergence is also used as a criterion for model selection [21], hypothesis testing [22], and merging in mixture models [23,24]. Additionally, it can be used as a measure of dissimilarity in classification problems, for example, text classification [25], speech recognition [26], and texture classification [27,28].

The wide applicability of KL-divergence as a useful dissimilarity measure persuade us to derive the KL-divergence for two important distributions. One of them is ZEG [29] that has a rich modeling power and allows heavy and light tail and different peak behaviors [30,31]. The other is ACG [32] which is a distribution on the unit sphere that has been used in many applications [33,34,35,36]. This distribution has many nice features; for example, its maximum likelihood estimator is asymptotically the most robust estimator of the scatter matrix of an elliptical distribution in the sense of minimizing the maximum asymptotic variance [37].

1.1. Contributions

To summarize, the key contributions of our paper are the following:

-

Introducing three methods for computing the expected logarithm of a CQF.

-

Proposing a procedure for computing the expected logarithm of an arbitrary positive random variable.

-

Deriving expressions for the entropy and the KL-divergence of ZEG and ACG distributions (the form of KL-divergence between ZEG distributions appeared in [38] but without its derivations).

The methods for computing the expected logarithm of a CQF differ in running-time and accuracy. Two of these, namely integral and series methods, are exact. The third method is a fast but inexact set of methods. The integral method is a direct application of our proposed procedure for computing the expected logarithm of positive random variables. We propose two fast methods that are based on approximating the CQF with a gamma random variable. We show that these fast methods give upper and lower bounds to the true expected logarithm. This leads us to develop another fast method based on a convex combination of the other two fast methods. Whenever the weights of the CQF are eigenvalues of a matrix as in the case of KL-divergences, the fast methods can be very efficient because they do not need eigenvalue computation.

1.2. Outline

The remainder of this paper is organized as follows. Section 2 proposes three different methods for computing the expected logarithm of a CQF. Furthermore, a theorem is stated at the beginning of this section that has a pivotal role in the first two methods. Then, we derive expressions for the KL-divergence and entropy of ZEG and ACG distributions in Section 3. Afterwards, in Section 4, multiple experiments are conducted to examine the performance of three methods for computing the aforementioned expected logarithm in terms of accuracy and computational time. Finally, Section 5 presents our conclusions. To improve the readability of the manuscript, the proofs of some theorems are presented in appendices.

2. Calculating the Expected Logarithm of a Central Quadratic Form

Suppose $N_i$ is the i-th random variable in the series of d independent standard normal random variables, i.e., normal random variables with zero-means and unit variances. Then the central (Gaussian) quadratic form is the following random variable:

$$U={\displaystyle \sum _{i=1}^d}{\lambda }_i{N}_i^2,$$

(1)

where ${\lambda }_i$s are non-negative real numbers. Note that $N_i^2$s are chi-square random variables with degree of freedoms equal to one; therefore, this random variable is also called the weighted sum of chi-square random variables. To the best of our knowledge, the expected logarithm of the random variable U does not have a closed-form expression using elementary mathematical functions. For its calculation, we propose three different approaches, namely an integral method, a series method and a set of fast methods. Each of them has its specific properties and does well in certain situations.

In the following theorem, a relation between the expected logarithm of two positive random variables distributed according to arbitrary densities and the Laplace transform of these two densities is given. This theorem is used in the integral method and the fast method. Note that the assumptions of the following theorem are unrestrictive. Therefore, it can also be used for computing the expected logarithm of other positive random variables.

Theorem 1. 

Let X and Y be two positive random variables, F and G be their cumulative distribution functions, and f and g be their probability density functions. Furthermore, suppose that $\mathcal{F}$ and $\mathcal{G}$ are the Laplace transform of f and g, respectively. If ${lim}_{x\to \infty }log(x)(G(x)-F(x))=0$, ${lim}_{x\to 0^{+}}log(x)(G(x)-F(x))=0$ and ${\int }_0^{\infty }\frac{\mathcal{G}(\sigma )-\mathcal{F}(\sigma )}{\sigma }d\sigma$ exists, then

$$𝔼[logX]-𝔼[logY]={\int }_0^{\infty }\frac{\mathcal{G}(\sigma )-\mathcal{F}(\sigma )}{\sigma }d\sigma .$$

(2)

Proof. 

Using the definition of Laplace transform, we have

$$\mathcal{G}(\mathit{s})-\mathcal{F}(\mathit{s})={\int }_0^{\infty }(g(x)-f(x))exp(-sx)dx.$$

(3)

Using the integration property of Laplace transform, we have

$$\frac{\mathcal{G}(\mathit{s})-\mathcal{F}(\mathit{s})}{s}={\int }_0^{\infty }\left[{\int }_0^x(g(\tau )-f(\tau ))d\tau \right]exp(-sx)dx.$$

Using the frequency integration property of Laplace transform, and the formulas $F(x)={\int }_0^{x}f(\tau )d\tau$, and $G(x)={\int }_0^{x}g(\tau )d\tau$ we further obtain

$${\int }_{\mathit{s}}^{\infty }\frac{\mathcal{G}(\sigma )-\mathcal{F}(\sigma )}{\sigma }d\sigma ={\int }_0^{\infty }\frac{1}{x}(G(x)-F(x))exp(-sx)dx.$$

Letting s in the above equation go to zero and since ${\int }_0^{\infty }\frac{\mathcal{G}(\sigma )-\mathcal{F}(\sigma )}{\sigma }d\sigma$ exists, we further obtain

$${\int }_0^{\infty }\frac{\mathcal{G}(\sigma )-\mathcal{F}(\sigma )}{\sigma }d\sigma ={\int }_0^{\infty }\frac{1}{x}(G(x)-F(x))dx.$$

Using the integration by parts formula having $log(x)$ and $G(x)-F(x)$ as its parts, we have

$${\int }_0^{\infty }\frac{\mathcal{G}(\sigma )-\mathcal{F}(\sigma )}{\sigma }d\sigma ={[log(x)(G(x)-F(x))]}_0^{\infty }-{\int }_0^{\infty }log(x)(g(x)-f(x))dx.$$

(4)

Since ${lim}_{x\to \infty }log(x)(G(x)-F(x))=0$, and ${lim}_{x\to 0^{+}}log(x)(G(x)-F(x))=0$, we obtain

$${[log(x)(G(x)-F(x))]}_0^{\infty }=0.$$

(5)

Hence by using (5) in (4), we have

$${\int }_0^{\infty }log(x)f(x)dx-{\int }_0^{\infty }log(x)g(x)dx={\int }_0^{\infty }\frac{\mathcal{G}(\sigma )-\mathcal{F}(\sigma )}{\sigma }d\sigma .$$

(6)

From the definition of expectation, relation (2) is obtained. ☐

2.1. Integral Method

In this part, we will use Theorem 1 for computing the expected logarithm of a CQF. To this end, we choose a random variable Y that has a closed-form formula for its expected logarithm and Laplace transform of its density. A possible candidate is the gamma random variable. The density of gamma random variable has the following Laplace transform:

$${(1+\theta s)}^{-k},$$

(7)

where k and θ are its shape and scale parameters, respectively. Also, the expected logarithm of this random variable is $\Psi (k)+log(\theta )$, where $\Psi (·)$ is digamma function.

Using the convolution property of Laplace transform, it is easy to see that the density function of the CQF given in (1) has the following closed-form Laplace transform:

$${\displaystyle \prod _{i=1}^d}{(1+2{\lambda }_{i}s)}^{-\frac{1}{2}}.$$

(8)

Lemmas 2 and 3 show that a CQF and a gamma random variable satisfy the conditions of Theorem 1. For proving Lemma 2, we need Lemma 1. First of all, let us express the following trivial proposition.

Proposition 1. 

Let $X_1,\cdots ,X_n$ be arbitrary real random variables. Suppose we have two many-to-one transformations $Y=h(X_1,\cdots ,X_n)$ and $Z=g(X_1,\cdots ,X_n)$. If the following inequality holds for any $x_i$s in the support of random variables $X_i$s:

$$h(x_1,\cdots ,x_n)\ge g(x_1,\cdots ,x_n),$$

(9)

then we have the following inequality between the cumulative distribution functions of random variables Y and Z:

$$F_Y(v)\le F_Z(v),forallv\in ℝ.$$

(10)

Lemma 1. 

Let F be the cumulative distribution function of a CQF, that is ${\sum }_{i=1}^d{\lambda }_i{N}_i^2$, where ${\lambda }_i$s are positive real numbers and $N_i$s are independent standard normal random variables. Also, let $G(x;k,\theta )$ be the cumulative distribution function of a gamma random variable with parameters k and θ, then the following inequalities hold:

$$G(x;\frac{d}{2},2{\lambda }_{max})\le F(x)\le G(x;\frac{d}{2},2{\lambda }_{min}),forallx\in {ℝ}^{+},$$

(11)

where ${\lambda }_{max}=max{\left\{{\lambda }_i\right\}}_{i=1}^d$, and ${\lambda }_{min}=min{\left\{{\lambda }_i\right\}}_{i=1}^d$.

Proof. 

This lemma is an immediate consequence of Proposition 1 and the following relation, knowing that $\lambda {\sum }_{i=1}^d{N}_i^2$ is a gamma random variable with the shape parameter $d/2$ and the scale parameter $2\lambda$:

$${\lambda }_{\text{min}}{\displaystyle \sum _{i=1}^d}{x}_i^2\le {\displaystyle \sum _{i=1}^d}{\lambda }_i{x}_i^2\le {\lambda }_{\text{max}}{\displaystyle \sum _{i=1}^d}{x}_i^2,\text{for}\text{all}{x}_i\in ℝ.$$

(12)

 ☐

Lemma 2. 

Let G be the cumulative distribution function of an arbitrary gamma random variable and F be the cumulative distribution function of random variable ${\sum }_{i=1}^d{\lambda }_i{N}_i^2$, where ${\lambda }_i$s are positive real numbers and $N_i$s are independent standard normal random variables, then ${lim}_{x\to \infty }log(x)(G(x)-F(x))=0$, and ${lim}_{x\to 0^{+}}log(x)(G(x)-F(x))=0$.

The proof of this lemma can be found in Appendix A.

Lemma 3. 

Let $\mathcal{G}$ be the Laplace transform of probability density function of an arbitrary gamma random variable and $\mathcal{F}$ be the Laplace transform of probability density function of ${\sum }_{i=1}^d{\lambda }_i{N}_i^2$, where ${\lambda }_i$s are positive real numbers and $N_i$s are independent standard normal random variables, then ${\int }_0^{\infty }\frac{\mathcal{G}(\sigma )-\mathcal{F}(\sigma )}{\sigma }d\sigma$ is convergent.

The proof of this lemma can be found in Appendix B.

According to Lemmas 2 and 3, the conditions of Theorem 1 hold by choosing X to be a CQF given in (1), and Y to be an arbitrary gamma random variable. Therefore, we can use (2) for calculating the expected logarithm of a CQF, and it is given by

$$𝔼[log({\displaystyle \sum _{i=1}^d}{\lambda }_i{N}_i^2\left)\right]=\Psi (k)+log(\theta )+{\int }_0^{\infty }\frac{{(1+\theta \sigma )}^{-k}-{\prod }_{i=1}^d{(1+2{\lambda }_i\sigma )}^{-\frac{1}{2}}}{\sigma }d\sigma .$$

(13)

The above equation holds for any choice of positive scalars k and θ. To the best of our knowledge, the above integral does not have a closed-form solution, so it must be computed numerically. This integral can be computed numerically using the variety of techniques available for one-dimensional integrals (see for example [39]).

2.2. Fast Methods

The integral method explained in the previous part can be computationally expensive for some applications. To this end, we derive three approximations that can be calculated analytically and, therefore, are much faster.

Using the first or higher order Taylor expansion around $𝔼\left[U\right]$ to approximate the expected logarithm of U has been already proposed in the literature [6,40]. However, we observed that lower order Taylor expansion does not give a very accurate approximation. Therefore, we use two different approximations, for which we can show that they provide a lower and an upper bound for the true expected logarithm. Finally, a convex combination of these two is used to get the final approximation.

Two different gamma distributions have been used in [15,41,42] to approximate a CQF. Since the expected logarithm of a gamma random variable has a closed-form solution, we use the expected logarithm of these gamma random variables to approximate the expected logarithm of a CQF. A further justification for this idea can be given based on (13) by choosing the shape and the scale parameters of gamma distribution such that the magnitude of the integral part in (13) becomes smaller.

Since the weights of CQF in the KL-divergence formulas in Section 3 are eigenvalues of a positive definite matrix Σ, we express the approximations based on this matrix. This way of expressing the approximations also elucidates the fact that the eigenvalues do not need to be calculated, which shows further computational benefits of these approximations. The shape and scale parameters of the first approximating gamma random variable are $d/2$ and $2\text{tr}(\Sigma )/d$, respectively. Therefore, for the first fast approximation we have

$$𝔼[log({\displaystyle \sum _{i=1}^d}{\lambda }_i{N}_i^2\left)\right]\approx \Psi (\frac{d}{2})+log\left(\frac{2\text{tr}(\Sigma )}{d}\right).$$

(14)

The shape and scale parameters of the gamma random variable for the second approximation are $\text{tr}{(\Sigma )}^2/2\text{tr}({\Sigma }^2)$ and $2\text{tr}({\Sigma }^2)/\text{tr}(\Sigma )$, respectively. Then, we obtain the following formula for the second fast approximation:

$$𝔼[log({\displaystyle \sum _{i=1}^d}{\lambda }_i{N}_i^2\left)\right]\approx \Psi \left(\frac{\text{tr}{(\Sigma )}^2}{2\text{tr}({\Sigma }^2)}\right)+log\left(\frac{2\text{tr}({\Sigma }^2)}{\text{tr}(\Sigma )}\right).$$

(15)

The following theorem shows that these approximations are lower and upper bounds to the true expected logarithm.

Theorem 2. 

If $U={\sum }_{i=1}^d{\lambda }_i{N}_i^2$, where ${\lambda }_i$s are eigenvalues of positive definite matrix ${\Sigma }_{d\times d}$ and $N_i$s are independent standard normal random variables, then

$$\Psi \left(\frac{\text{tr}{(\Sigma )}^2}{2\text{tr}({\Sigma }^2)}\right)+log\left(\frac{2\text{tr}({\Sigma }^2)}{\text{tr}(\Sigma )}\right)\le 𝔼[log(U)]\le \Psi (\frac{d}{2})+log\left(\frac{2\text{tr}(\Sigma )}{d}\right).$$

(16)

The proof of this theorem can be found in Appendix C.

From this theorem, we can conclude that there exist some convex combinations of the two previously mentioned approximations which perform equal or better than each of them, in the sense that they are closer to the true expected logarithm. Therefore, we define the third fast approximation to be

$$𝔼[log({\displaystyle \sum _{i=1}^d}{\lambda }_i{N}_i^2\left)\right]\approx (1-l)[\Psi (\frac{d}{2})+log(\frac{2\text{tr}(\Sigma )}{d}\left)\right]+l[\Psi (\frac{\text{tr}{(\Sigma )}^2}{2\text{tr}({\Sigma }^2)})+log(\frac{2\text{tr}({\Sigma }^2)}{\text{tr}(\Sigma )}\left)\right].$$

(17)

To determine parameter $l\in [0,1]$ in the above equation, we used the least squares fitting on thousands of positive definite matrices with different dimensions and unit trace sampled uniformly according to an algorithm given in [43]. We observed that the fitted value is roughly equal to $l=0.7$ and dimensionality has a negligible effect on the best value of l. For the case of $d=20$, the mean squared error for various values of l can be seen in Figure 1.

2.3. Series Method

One can represent the probability density function of a CQF given by (1) as an infinite weighted sum of gamma densities [13,14],

$$\begin{array}{c}\hfill f_U(u)={\displaystyle \sum _{j=0}^{\infty }}{c}_{j}g(u;\frac{d}{2}+j,2\beta ),\end{array}$$

(18)

where $g(u;d/2+j,2\beta )$ is the probability density function of a gamma random variable with parameters $d/2+j$ and $2\beta$, and

$$\begin{array}{cc}\hfill c_k& =\frac{1}{k}{\displaystyle \sum _{r=0}^{k-1}}{v}_{k-r}{c}_r,\hfill \end{array}$$

(19)

$$\begin{array}{cc}\hfill v_k& =\frac{1}{2}{\displaystyle \sum _{j=1}^d}(1-\beta {\lambda }_j^{-1}{)}^k,\hfill \end{array}$$

(20)

$$\begin{array}{cc}\hfill c_0& ={\displaystyle \prod _{j=1}^d}\sqrt{\beta {\lambda }_j^{-1}}.\hfill \end{array}$$

(21)

This result can be used for deriving a series formula for the expected logarithm of U. Thus,

$$\begin{array}{c}\hfill 𝔼[logU]={\displaystyle \sum _{j=0}^{\infty }}{c}_j\left(\Psi (\frac{d}{2}+j)+log(2\beta )\right).\end{array}$$

(22)

Ruben [13] analyzed the effect of various βs on the behavior of the series expansion and proposed the following β as an appropriate one:

$$\begin{array}{c}\hfill \beta =\frac{2{\lambda }_{max}{\lambda }_{min}}{{\lambda }_{max}+{\lambda }_{min}}.\end{array}$$

(23)

By using this β, ${\sum }_{j=0}^{\infty }{c}_j=1$ holds [13] and also knowing that the following relation holds for the digamma function:

$$\begin{array}{c}\hfill \Psi (x+1)=\frac{1}{x}+\Psi (x),\end{array}$$

then (22) can be simplified

$$\begin{array}{ccc}\hfill 𝔼[logU]& =& log(2\beta )+c_0\Psi (\frac{d}{2})+{\displaystyle \sum _{j=1}^{\infty }}{c}_j\left(\Psi (\frac{d}{2})+{\displaystyle \sum _{l=0}^{j-1}}\frac{1}{\frac{d}{2}+l}\right)\hfill \\ & =& log(2\beta )+\Psi (\frac{d}{2})+{\displaystyle \sum _{j=1}^{\infty }}{c}_j{\displaystyle \sum _{l=0}^{j-1}}\frac{1}{\frac{d}{2}+l}\hfill \\ & =& log(2\beta )+\Psi (\frac{d}{2})+{\displaystyle \sum _{i=0}^{\infty }}\left(\frac{1}{\frac{d}{2}+i}{\displaystyle \sum _{k=i+1}^{\infty }}{c}_k\right)\hfill \\ & =& log(2\beta )+\Psi (\frac{d}{2})+{\displaystyle \sum _{i=0}^{\infty }}(\frac{1}{\frac{d}{2}+i}(1-{\displaystyle \sum _{k=0}^i}{c}_k)).\hfill \end{array}$$

(24)

To approximate this formula, we cut the series coefficients, which means we only use a finite number of terms to evaluate the expectation:

$$\begin{array}{c}\hfill \widehat{𝔼}[logU]=log(2\beta )+\Psi (\frac{d}{2})+{\displaystyle \sum _{i=0}^{L-1}}\left(\frac{1}{\frac{d}{2}+i}(1-{\displaystyle \sum _{k=0}^i}{c}_k)\right).\end{array}$$

(25)

For this approximation, it is possible to compute an error bound which is expressed by the following lemma.

Lemma 4. 

The bound for error of the approximation (25) for $L>dϵ/(2-2ϵ)$ is as follows:

$$𝔼[logU]-\widehat{𝔼}[logU]\le c_0\frac{{ϵ}^{L+1}}{{\left(1-\frac{d/2+L}{L}ϵ\right)}^2}\frac{\Gamma (\frac{d}{2}+L)}{\Gamma (\frac{d}{2}+1)(L+1)!},$$

(26)

where $\Gamma (·)$ is gamma function, and $ϵ=({\lambda }_{max}-{\lambda }_{min})/({\lambda }_{max}+{\lambda }_{min})$.

The proof of this lemma is in Appendix D.

By using this bound, it is possible to calculate the expected logarithm with a given accuracy by selecting an appropriate L. Note that the upper bound given by (26) is growing with respect to ϵ, and ϵ is also increasing with ${\lambda }_{max}/{\lambda }_{min}$. As we will see in the simulation studies, when the ratio ${\lambda }_{max}/{\lambda }_{min}$ as well as the dimensionality d are small, this method performs better than the integral method.

3. KL-Divergence of Two Generalized Elliptical Distributions

In this section, we derive expressions for the KL-divergence and the entropy of two subclasses of generalized elliptical distributions, namely ZEG and ACG distributions [44]. We first start by reviewing some related materials.

3.1. Some Background on the Elliptical Distributions

Suppose the d-dimensional random vector X is distributed according to a zero-mean elliptical contoured (ZEC) distribution with a positive definite scatter matrix ${\Sigma }_{d\times d}$, that is $\mathbf{X}\sim \mathcal{Z}\mathcal{E}\mathcal{C}(\Sigma ,\phi )$. The probability density function of X is given by

$$f_{\mathbf{x}}(\mathbf{x};\Sigma ,\phi )={|\Sigma |}^{-\frac{1}{2}}\phi \left({\mathbf{x}}^{\top }{\Sigma }^{-1}\mathbf{x}\right),$$

(27)

for some density generator functions $\phi :{ℝ}^{+}\to ℝ$. We know that we can decompose the vector X into a uniform hyper-spherical component and a scaled-radial component so that, $\mathbf{X}={\Sigma }^{1/2}R\mathbf{U}$, where U is uniformly distributed over the unit sphere ${\mathbb{S}}^{d-1}$ and R is a univariate random variable given by $R={\parallel {\Sigma }^{-1/2}\mathbf{X}\parallel }_2$ [45]. Then, the random variable R has the density

$$f_R(r)=2{\pi }^{\frac{d}{2}}\phi (r^2)r^{d-1}/\Gamma (\frac{d}{2}).$$

(28)

Therefore, square radial component ${\rm Y}=R^2$ has the following density:

$$f_{{\rm Y}}(\upsilon )={\pi }^{\frac{d}{2}}\phi (\upsilon ){\upsilon }^{\frac{d}{2}-1}/\Gamma (\frac{d}{2}).$$

(29)

A ZEG is a ZEC whose square radial component is distributed according to a gamma distribution ${\rm Y}\sim Gamma(a,b)$. A gamma-distributed random variable has the density

$$f_{{\rm Y}}(\upsilon )=\Gamma {(a)}^{-1}{b}^{-a}{\upsilon }^{a-1}exp\left(-\upsilon /b\right),$$

(30)

where a is a shape parameter and b is a scale parameter. So the probability density function of a d-dimensional random variable $\mathbf{X}\sim \mathcal{Z}\mathcal{E}\mathcal{G}(\Sigma ,a,b)$ is given by

$$f_{\mathbf{x}}(\mathbf{x};\Sigma ,a,b)=\frac{\Gamma (\frac{d}{2})}{{\pi }^{\frac{d}{2}}\Gamma (a)b^a{|\Sigma |}^{\frac{1}{2}}}{\left({\mathbf{x}}^{\top }{\Sigma }^{-1}\mathbf{x}\right)}^{a-\frac{d}{2}}exp\left(-b^{-1}{\mathbf{x}}^{\top }{\Sigma }^{-1}\mathbf{x}\right),$$

(31)

where $\mathbf{x}\in {ℝ}^d$ and $\Sigma \succ 0$ is its scatter matrix, also $a,b>0$ are certain scale and shape parameters [31].

When ZEC random variable is projected onto a unit sphere, the resulting random variable is called ACG and denoted by $\mathbf{X}\sim \mathcal{A}\mathcal{C}\mathcal{G}(\Sigma )$. This distribution unlike many other distributions on the unit sphere has a nice closed-form density given by

$$f_{\mathbf{x}}(\mathbf{x};\Sigma )=\frac{\Gamma (\frac{d}{2})}{2{\pi }^{\frac{d}{2}}{|\Sigma |}^{\frac{1}{2}}}{\left({\mathbf{x}}^{\top }{\Sigma }^{-1}\mathbf{x}\right)}^{-\frac{d}{2}},$$

(32)

where $\mathbf{x}\in {\mathbb{S}}^{d-1}$ and $\Sigma \succ 0$ is its scatter matrix.

3.2. KL-Divergence between ZEG Distributions

Suppose we have two probability distributions P and Q with probability density functions p and q, KL-divergence between these two distributions is defined by

$$\text{KL}(P||Q)=\int log(p(\mathbf{x}))p(\mathbf{x})d\mathbf{x}-\int log(q(\mathbf{x}))p(\mathbf{x})d\mathbf{x}.$$

(33)

The negative of the first part, $\mathrm{H}(\mathbf{X})=-\int log(p(\mathbf{x}))p(\mathbf{x})d\mathbf{x}$, is the entropy and the second part, $𝔼[-log(q(\mathbf{X}))]=-\int log(q(\mathbf{x}))p(\mathbf{x})d\mathbf{x}$, is the averaged log-loss term, where X is a random variable distributed according to P.

Following lemma gives a general expression for the KL-divergence between two ZEC distributions. It is then used for deriving the KL-divergence between two ZEG distributions.

Lemma 5. 

Suppose we have two probability distributions on random variable Y, $P_{\mathbf{Y}}=\mathcal{Z}\mathcal{E}\mathcal{C}({\Sigma }_1,\phi )$ and $Q_{\mathbf{Y}}=\mathcal{Z}\mathcal{E}\mathcal{C}({\Sigma }_2,{\phi }^{\prime })$, then the KL-divergence between these two distributions is given by the following expression:

$$\begin{array}{rr}\text{KL}(P||Q)=& -\frac{1}{2}log(|\Sigma |)+{\int }_0^{\infty }log({\upsilon }^{1-\frac{d}{2}}{f}_{{\rm Y}}(\upsilon ))f_{{\rm Y}}(\upsilon )d\upsilon \\ & -\stackrel{\infty }{\underset{0}{\int \int }}\frac{1}{r}{f}_{wd}(\frac{\upsilon }{r})f_{{\rm Y}}(r)log({\upsilon }^{1-\frac{d}{2}}{f}_{{{\rm Y}}^{\prime }}(\upsilon ))d\upsilon dr,\end{array}$$

(34)

where $f_{{\rm Y}}$ and $f_{{{\rm Y}}^{\prime }}$ are the square radial components of distributions P and Q, respectively. Also, $f_{wd}$ is the density of ${\sum }_{i=1}^d{\lambda }_i{N}_i^2/{\sum }_{i=1}^d{N}_i^2$, where $N_i$s are independent standard normal random variables, and ${\lambda }_1,\dots ,{\lambda }_d$ are eigenvalues of matrix $\Sigma ={\Sigma }_2^{-1/2}{\Sigma }_1{\Sigma }_2^{-1/2}$.

Proof. 

KL-divergence is known to be invariant against invertible transformations of random variable Y [46]. To simplify the derivations, we apply a linear transformation $\mathbf{X}={\Sigma }_2^{-1/2}\mathbf{Y}$ that makes the scatter matrix of the second distribution identity. By using this change of variable, the problem becomes that of KL-divergence computation between $P_{\mathbf{X}}=\mathcal{Z}\mathcal{E}\mathcal{C}(\Sigma ,\phi )$ and $Q_{\mathbf{X}}=\mathcal{Z}\mathcal{E}\mathcal{C}(\mathbf{I},{\phi }^{\prime })$, where $\Sigma ={\Sigma }_2^{-1/2}{\Sigma }_1{\Sigma }_2^{-1/2}$.

As expressed in (33), the KL-divergence is the subtraction of the entropy from the averaged log-loss. Firstly, let us derive the entropy of X having distribution $P_{\mathbf{X}}$, that is $\mathrm{H}(\mathbf{X})=-\int log(p(\mathbf{x}))p(\mathbf{x})d\mathbf{x}$.

Assume the change of integration variable $\mathbf{y}={\Sigma }^{-1/2}\mathbf{x}$ and use (27), then we obtain the following expression for $\mathrm{H}(\mathbf{X})$:

$$\mathrm{H}(\mathbf{X})=\frac{1}{2}log(|\Sigma |)-\int log\left(\phi ({\mathbf{y}}^{\top }\mathbf{y})\right)\phi ({\mathbf{y}}^{\top }\mathbf{y})d\mathbf{y}.$$

Let $r={\parallel \mathbf{y}\parallel }_2$ and recall that the area of a sphere in dimension d with radius r equals $2r^{d-1}{\pi }^{d/2}/\Gamma (d/2)$, thus

$$\mathrm{H}(\mathbf{X})=\frac{1}{2}log(|\Sigma |)-{\int }_0^{\infty }log\left(\phi (r^2)\right)\frac{2{\pi }^{\frac{d}{2}}}{\Gamma (\frac{d}{2})}{r}^{d-1}\phi (r^2)dr.$$

Using the change of variable $\upsilon =r^2$ and replacing ϕ by square radial density $f_{{\rm Y}}$ as expressed in (29), we obtain

$$\mathrm{H}(\mathbf{X})=\frac{1}{2}log(|\Sigma |)-{\int }_0^{\infty }log\left(\frac{\Gamma (\frac{d}{2})}{{\pi }^{\frac{d}{2}}}{\upsilon }^{1-\frac{d}{2}}{f}_{{\rm Y}}(\upsilon )\right)f_{{\rm Y}}(\upsilon )d\upsilon .$$

(35)

Now, we are deriving an expression for the averaged log-loss given by $𝔼[-log(q(\mathbf{X}))]=-𝔼\left[log\left({\phi }^{\prime }({\mathbf{X}}^{\top }\mathbf{X})\right)\right]$. The argument of function ${\phi }^{\prime }$ is ${\mathbf{X}}^{\top }\mathbf{X}$; therefore, it is enough to compute the expectation of the function over the new random variable $Z={\parallel \mathbf{X}\parallel }_2^2$:

$$\begin{array}{ccc}\hfill 𝔼[-log(q(\mathbf{X}))]& =& -𝔼{}_Z\left[log\left({\phi }^{\prime }(Z)\right)\right]\hfill \\ & =& -{\int }_0^{\infty }log\left({\phi }^{\prime }(Z)\right)f_Z(\upsilon )d\upsilon ,\hfill \end{array}$$

(36)

where $f_Z$ is the density of $Z={\parallel \mathbf{X}\parallel }_2^2$, wherein $\mathbf{X}\sim \mathcal{Z}\mathcal{E}\mathcal{C}(\Sigma ,\phi )$. It is easy to see that the random variable Z can equally be written as $Z={\overline{\mathbf{X}}}^{\top }\Sigma \overline{\mathbf{X}}$ where $\overline{\mathbf{X}}\sim \mathcal{Z}\mathcal{E}\mathcal{C}(\mathbf{I},\phi )$. The density of Z with this representation has already been reported in [47]:

$$f_Z(z)={\int }_0^{\infty }\frac{1}{r}{f}_{\text{wd}}(z/r)f_{{\rm Y}}(r)dr,$$

(37)

where $f_{{\rm Y}}$ is the square radial density of $p_{\mathbf{Y}}$, and $f_{\text{wd}}$ is the density of a linear combination of Dirichlet random variable components,

$$\Lambda ={\displaystyle \sum _{j=1}^s}{l}_j{D}_j,$$

(38)

where $\mathbf{D}={(D_1,\cdots ,D_s)}^{\top }$ is a Dirichlet random variable with parameters $(r_1/2,\cdots ,r_s/2)$, and $l_j$s are s distinct eigenvalues of the positive definite matrix Σ with respective multiplicities $r_j$, for $j=1,\cdots ,s$.

It is known that if random variables $C_1,\cdots ,C_s$ are independent chi-square random variables having $r_1,\cdots ,r_s$ degrees of freedom, and $C={\sum }_{j=1}^s{C}_j$, then ${(C_1/C,\cdots ,C_s/C)}^{\top }$ is a Dirichlet random variable with the parameters $(r_1/2,\cdots ,r_s/2)$ [48]. Hence, the random variable Λ in (38) can be expressed as $\Lambda ={\sum }_{i=1}^s{l}_j{C}_j/C$. Equivalently, if $N_1,\cdots ,N_d$ are independent standard normal random variables, then Λ can be written as

$$\Lambda =\frac{{\sum }_{i=1}^d{\lambda }_i{N}_i^2}{{\sum }_{i=1}^d{N}_i^2}.$$

(39)

Using (37) in (36) and replacing ${\phi }^{\prime }$ by square radial density $f_{{{\rm Y}}^{\prime }}$ as expressed in (29), we obtain the following expression for the averaged log-loss:

$$𝔼[-log(q(\mathbf{X}))]=-\stackrel{\infty }{\underset{0}{\int \int }}\frac{1}{r}{f}_{\text{wd}}(\upsilon /r)f_{{\rm Y}}(r)log\left(\frac{\Gamma (\frac{d}{2})}{{\pi }^{\frac{d}{2}}}{\upsilon }^{1-\frac{d}{2}}{f}_{{{\rm Y}}^{\prime }}(\upsilon )\right)d\upsilon dr.$$

(40)

Subtracting (35) from (40), we obtain (34). ☐

Until now, we derived an expression for the KL-divergence between two ZEC distributions. We can further simplify the KL-divergence for the case of ZEG distributions to avoid computing double-integration, and the following theorem proves it.

Theorem 3. 

Suppose we have two distributions $P_{\mathbf{Y}}=\mathcal{Z}\mathcal{E}\mathcal{G}({\Sigma }_1,a_p,b_p)$, and $Q_{\mathbf{Y}}=\mathcal{Z}\mathcal{E}\mathcal{G}({\Sigma }_2,a_q,b_q)$, then the entropy of random variable Y distributed according to P_Y and the KL-divergence between these two distributions are given by the following expressions:

$$\mathrm{H}(\mathbf{Y})=\frac{1}{2}log(|{\Sigma }_1|)-log\left(\frac{\Gamma (\frac{d}{2})}{{\pi }^{\frac{d}{2}}\Gamma (a_p)b_p^{a_p}}\right)-(a_p-\frac{d}{2})\left(\Psi (a_p)+log(b_p)\right)+a_p,$$

(41)

$$\begin{array}{cc}\hfill \text{KL}(P||Q)=& -\frac{1}{2}log(|\Sigma |)+log\left(\frac{\Gamma (a_q)b_q^{a_q}}{\Gamma (a_p)b_p^{a_q}}\right)+(a_p-a_q)\Psi (a_p)-a_p\hfill \\ & +\frac{a_p{b}_p}{d{b}_q}\text{tr}(\Sigma )+(\frac{d}{2}-a_q)(𝔼[log({\displaystyle \sum _{i=1}^d}{\lambda }_i{N}_i^2)]-\Psi (\frac{d}{2})-log(2)),\hfill \end{array}$$

(42)

where $\Psi (·)$ is digamma function, and $tr(·)$ is the trace of a matrix. Also $N_i$s are independent standard normal random variables, and ${\lambda }_1,\dots ,{\lambda }_d$ are eigenvalues of matrix $\Sigma ={\Sigma }_2^{-1/2}{\Sigma }_1{\Sigma }_2^{-1/2}$.

Proof. 

Like the previous lemma, we apply the change of variable $\mathbf{X}={\Sigma }_2^{-1/2}\mathbf{Y}$ and compute the KL-divergence between the transformed distributions. The expression for entropy (35) in the case of ZEG distributions becomes

$$\mathrm{H}(\mathbf{X})=\frac{1}{2}log(|\Sigma |)-{\int }_0^{\infty }log\left(\frac{\Gamma (\frac{d}{2})}{{\pi }^{\frac{d}{2}}\Gamma (a_p)b_p^{a_p}}{\upsilon }^{a_p-\frac{d}{2}}exp\left(-\frac{\upsilon }{b_p}\right)\right)\frac{1}{\Gamma (a_p)b_p^{a_p}}{\upsilon }^{a_p-1}exp\left(-\frac{\upsilon }{b_p}\right)d\upsilon .$$

(43)

Next, recall the following gamma function identities [49]:

$$\frac{1}{\Gamma (a+1)b^{a+1}}{\int }_0^{\infty }{r}^{a}exp\left(-\frac{r}{b}\right)dr=1,$$

(44)

$$\frac{1}{\Gamma (a+1)b^{a+1}}{\int }_0^{\infty }log(r)r^{a}exp\left(-\frac{r}{b}\right)dr=\Psi (a+1)+log(b).$$

(45)

Using (44) and (45), we can simplify (43) to obtain

$$\mathrm{H}(\mathbf{X})=\frac{1}{2}log(|\Sigma |)-log\left(\frac{\Gamma (\frac{d}{2})}{{\pi }^{\frac{d}{2}}\Gamma (a_p)b_p^{a_p}}\right)+(\frac{d}{2}-a_p)\left(\Psi (a_p)+log(b_p)\right)+a_p.$$

Since $\mathbf{Y}={\Sigma }_2^{1/2}\mathbf{X}$, we can trivially derive the expression of $\mathrm{H}(\mathbf{Y})$ given in (41).

For deriving the averaged log-loss term, we obtain the following expression by putting the gamma square radial component (30) into (40):

$$𝔼[-log(q(\mathbf{X}))]=-\stackrel{\infty }{\underset{0}{\int \int }}\frac{1}{r}{f}_{\text{wd}}(\upsilon /r)\frac{r^{a_p-1}}{\Gamma (a_p)b_p^{a_p}}exp\left(-\frac{r}{b_p}\right)log\left(\frac{\Gamma (\frac{d}{2})}{{\pi }^{\frac{d}{2}}\Gamma (a_q)b_q^{a_q}}{\upsilon }^{a_q-\frac{d}{2}}exp\left(-\frac{\upsilon }{b_q}\right)\right)d\upsilon dr.$$

We apply the change of variable $\mu =\upsilon /r$ and express the integrals in terms of new variables μ and r,

$$\begin{array}{cc}\hfill 𝔼[-log(q(\mathbf{X}))]=& -log(\Gamma (\frac{d}{2}))+log({\pi }^{\frac{d}{2}}\Gamma (a_q)b_q^{a_q})\hfill \\ & +\frac{\frac{d}{2}-a_q}{\Gamma (a_p)b_p^{a_p}}{\int }_0^{\infty }{f}_{\text{wd}}(\mu )log(\mu )d\mu {\int }_0^{\infty }{r}^{a_p-1}exp\left(-\frac{r}{b_p}\right)dr\hfill \\ & +\frac{\frac{d}{2}-a_q}{\Gamma (a_p)b_p^{a_p}}{\int }_0^{\infty }{f}_{\text{wd}}(\mu )d\mu {\int }_0^{\infty }log(r)r^{a_p-1}exp\left(-\frac{r}{b_p}\right)dr\hfill \\ & +\frac{1}{b_q\Gamma (a_p)b_p^{a_p}}{\int }_0^{\infty }{f}_{\text{wd}}(\mu )\mu d\mu {\int }_0^{\infty }{r}^{a_p}exp\left(-\frac{r}{b_p}\right)dr.\hfill \end{array}$$

Using the equalities (44) and (45), we obtain

$$\begin{array}{cc}\hfill 𝔼[-log(q(\mathbf{X}))]=& -log(\Gamma (\frac{d}{2}))+log({\pi }^{\frac{d}{2}}\Gamma (a_q)b_q^{a_q})+(\frac{d}{2}-a_q)\left(\Psi (a_p)+log(b_p)\right)\hfill \\ & +\frac{a_p{b}_p}{d{b}_q}{\int }_0^{\infty }{f}_{\text{wd}}(\mu )\mu d\mu +(\frac{d}{2}-a_q){\int }_0^{\infty }{f}_{\text{wd}}(\mu )log(\mu )d\mu ,\hfill \end{array}$$

where similar to the previous lemma, $f_{\text{wd}}$ is the density of the random variable $\Lambda ={\sum }_{i=1}^d{\lambda }_i{N}_i^2/{\sum }_{i=1}^d{N}_i^2$, where $N_i$s are independent standard normal random variables, and ${\lambda }_1,\dots ,{\lambda }_d$ are the eigenvalues of matrix Σ. Subtracting the entropy from the averaged log-loss and knowing that ${\int }_0^{\infty }{f}_{\text{wd}}(\mu )\mu d\mu =𝔼[\Lambda ]$ and ${\int }_0^{\infty }{f}_{\text{wd}}(\mu )log(\mu )d\mu =𝔼[log(\Lambda )]$, we obtain

$$\text{KL}(P||Q)=-\frac{1}{2}log(|\Sigma |)+log\left(\frac{\Gamma (a_q)b_q^{a_q}}{\Gamma (a_p)b_p^{a_q}}\right)+(a_p-a_q)\Psi (a_p)-a_p+\frac{a_p{b}_p}{d{b}_q}𝔼[\Lambda ]+(\frac{d}{2}-a_q)𝔼[log(\Lambda )].$$

(46)

The moments of Λ were computed in [47], but we are giving a simple derivation of the first moment below. It is known that the random variable $V_i=N_i^2/{\sum }_{j=1}^d{N}_j^2$ has the following beta distribution:

$$V_i\sim \text{Beta}\left(\frac{1}{2},\frac{d-1}{2}\right).$$

(47)

Since $𝔼\left[V_i\right]=1/d$, then

$$\begin{array}{ccc}\hfill 𝔼[\Lambda ]& =& 𝔼\left[{\displaystyle \sum _{i=1}^d}{\lambda }_i{V}_i\right]\hfill \\ & =& \frac{\text{tr}(\Sigma )}{d}.\hfill \end{array}$$

(48)

Expected logarithm of Λ can be expressed as a difference of two expectations:

$$\begin{array}{c}\hfill 𝔼[log(\Lambda )]=𝔼[log({\displaystyle \sum _{i=1}^d}{\lambda }_i{N}_i^2\left)\right]-𝔼[log({\displaystyle \sum _{i=1}^d}{N}_i^2\left)\right].\end{array}$$

(49)

Using the fact that the expected logarithm of a chi-square random variable with d degrees of freedom is equal to $\Psi (d/2)+log(2)$, $𝔼[log(\Lambda )]$ can be computed by the following equation:

$$\begin{array}{c}\hfill 𝔼[log(\Lambda )]=𝔼[log({\displaystyle \sum _{i=1}^d}{\lambda }_i{N}_i^2\left)\right]-\Psi (\frac{d}{2})-log(2).\end{array}$$

(50)

With substitution (48) and (50) into (46), we get (42). ☐

3.3. KL-Divergence between ACG Distributions

The following theorem gives expressions for the KL-divergence between ACG distributions and the entropy of a single ACG distribution.

Theorem 4. 

Suppose we have two probability distributions $G_{\mathbf{Y}}=\mathcal{A}\mathcal{C}\mathcal{G}({\Sigma }_1)$ and $J_{\mathbf{Y}}=\mathcal{A}\mathcal{C}\mathcal{G}({\Sigma }_2)$, then the entropy of random variable Y distributed according to G_Y and the KL-divergence between these two distributions are given by the following expressions:

$$\mathrm{H}(\mathbf{Y})=log\left(\frac{2{\pi }^{\frac{d}{2}}{|{\Sigma }_1|}^{1/2}}{\Gamma (\frac{d}{2})}\right)-\frac{d}{2}(𝔼[log({\displaystyle \sum _{i=1}^d}{\sigma }_i{N}_i^2\left)\right]-\Psi (\frac{d}{2})-log(2)),$$

(51)

$$\text{KL}(G||J)=-\frac{1}{2}log(|\Sigma |)+\frac{d}{2}(𝔼[log({\displaystyle \sum _{i=1}^d}{\lambda }_i{N}_i^2\left)\right]-\Psi (\frac{d}{2})-log(2)),$$

(52)

where $N_i$s are independent standard normal random variables, ${\lambda }_1,\dots ,{\lambda }_d$ are eigenvalues of matrix $\Sigma ={\Sigma }_2^{-1/2}{\Sigma }_1{\Sigma }_2^{-1/2}$, and ${\sigma }_1,\dots ,{\sigma }_d$ are eigenvalues of matrix ${\Sigma }_1$.

Proof. 

Due to the invariance property of KL-divergence under invertible change of variables, we use the change of variable $\mathbf{\Omega }={\Sigma }_1^{-1/2}\mathbf{Y}$. It is easy to verify that $\mathbf{\Omega }$ is distributed according to a zero-mean generalized elliptical distribution with identity covariance [44]. From the definition of KL-divergence given by (33), we have

$$\text{KL}(G||J)=𝔼\left[log\left(\frac{\Gamma (\frac{d}{2})}{2{\pi }^{\frac{d}{2}}}{\left({\mathbf{\Omega }}^{\top }\mathbf{\Omega }\right)}^{-\frac{d}{2}}\right)\right]-𝔼\left[log\left(\frac{\Gamma (\frac{d}{2})}{2{\pi }^{\frac{d}{2}}{|\tilde{\Sigma }|}^{1/2}}{\left({\mathbf{\Omega }}^{\top }{\tilde{\Sigma }}^{-1}\mathbf{\Omega }\right)}^{-\frac{d}{2}}\right)\right],$$

where $\tilde{\Sigma }={\Sigma }_1^{-1/2}{\Sigma }_2{\Sigma }_1^{-1/2}$. By some simplifications, it is immediate that

$$\text{KL}(G||J)=\frac{1}{2}log(|\tilde{\Sigma }|)+\frac{d}{2}𝔼\left[log\left(\frac{{\mathbf{\Omega }}^{\top }{\tilde{\Sigma }}^{-1}\mathbf{\Omega }}{{\mathbf{\Omega }}^{\top }\mathbf{\Omega }}\right)\right].$$

(53)

Since projecting any zero-mean generalized elliptical distribution (with identity covariance) on the unit sphere gives an ACG random variable (with identity covariance) [50], we can substitute $𝔼[log({\mathbf{\Omega }}^{\top }{\tilde{\Sigma }}^{-1}\mathbf{\Omega }/{\mathbf{\Omega }}^{\top }\mathbf{\Omega })]$ with $𝔼[log{(}^{\top }{\tilde{\Sigma }}^{-1}\mathbf{X}/{\mathbf{X}}^{\top }\mathbf{X})]$, where the random vector X is distributed according to a multivariate normal distribution with identity covariance and zero mean. Because ${\mathbf{X}}^{\top }{\tilde{\Sigma }}^{-1}\mathbf{X}$ is a CQF and ${\mathbf{X}}^{\top }\mathbf{X}$ is a chi-square random variable, we have

$$\text{KL}(G||J)=\frac{1}{2}log(|\tilde{\Sigma }|)+\frac{d}{2}(𝔼[log({\displaystyle \sum _{i=1}^d}{\tilde{\lambda }}_i^{-1}{N}_i^2\left)\right]-\Psi (\frac{d}{2})-log(2)),$$

(54)

where ${\tilde{\lambda }}_i$s are the eigenvalues of $\tilde{\Sigma }$. Additionally, it is easy to verify that $|\Sigma |={|\tilde{\Sigma }|}^{-1}$ and ${\lambda }_i={\tilde{\lambda }}_i^{-1}$, therefore (52) holds.

Since one of the terms in the KL-divergence is equal to the minus entropy, we use our derived expression for the KL-divergence between ACG distributions to find a formula for the entropy of an ACG distribution. Define $S_{\mathbf{Y}}=\mathcal{A}\mathcal{C}\mathcal{G}(\mathbf{I})$, then the KL-divergence between $G_{\mathbf{Y}}$ and $S_{\mathbf{Y}}$ can be easily derived from the main definition (33):

$$\text{KL}(G||S)=-\mathrm{H}(\mathbf{Y})-log\left(\frac{\Gamma (\frac{d}{2})}{2{\pi }^{\frac{d}{2}}}\right),$$

(55)

where $\mathrm{H}(\mathbf{Y})$ is the entropy of the random variable Y. Now, we compute the above KL-divergence using (52) which is

$$\text{KL}(G||S)=-\frac{1}{2}log(|{\Sigma }_1|)+\frac{d}{2}(𝔼[log({\displaystyle \sum _{i=1}^d}{\sigma }_i{N}_i^2\left)\right]-\Psi (\frac{d}{2})-log(2)).$$

(56)

Equating the right-hand sides of (55) and (56) gives (51). ☐

The following corollary shows a relation between the KL-divergence of ACG distributions and the KL-divergence of ZEG distributions. It is an immediate consequence of Theorems 3 and 4.

Corollary 1. 

The KL-divergence between two ACG distributions $\mathcal{A}\mathcal{C}\mathcal{G}({\Sigma }_1)$ and $\mathcal{A}\mathcal{C}\mathcal{G}({\Sigma }_2)$ is equal to the KL-divergence between two ZEG distributions $\mathcal{Z}\mathcal{E}\mathcal{G}({\Sigma }_1,\mathit{a},\mathit{b})$ and $\mathcal{Z}\mathcal{E}\mathcal{G}({\Sigma }_2,\mathit{a},\mathit{b})$, when $a\to 0$.

4. Simulation Study

In Section 2, we proposed three different methods for computing the expected logarithm of the CQF given in (1). We assume the weights of CQFs that are used in the simulations are eigenvalues of some random positive definite matrices. These random matrices are generated uniformly from the space of positive definite matrices with unit trace according to the procedure proposed in [43]. In this section, we numerically investigate the running time and accuracy of these approaches. All methods were implemented in Matlab (Version R2014a) (64-bit), and the simulations were run on a personal laptop with an Intel Core i5 (2.5Ghz) processor under the OS X Yosemite 10.10.3 operating system. Since the series method depends heavily on loops that are slow in Matlab, we implemented this method in a Matlab Mex-file. For the integral method, the integral is numerically evaluated using Gauss–Kronrod 7-15 rule [51,52]. The absolute error tolerance is given as an input parameter of the numerical integration. In the integral method, the value can be computed with any given accuracy by choosing the absolute error tolerance; therefore, we do not analyze the integral method in the sense of the calculation error.

Figure 2 investigates the effects of dimensionality on the average running time of different methods for computing the expected logarithm of the CQF explained in Section 2. For the integral method (upper-left plot), two curves for two different absolute error tolerances are shown. The integral formula (13) has the parameters k and θ that can be chosen freely, and we choose those given in (14). Different curves for the series method (upper-right plot) correspond to different values of L, which is the truncation length of the series. The curve of the fast method (lower plot) corresponds to the computation time of the third fast method explained in Section 2. One reason of lower computation time of the fast method is its lack of need for any eigenvalue computation. There is a curve in upper-right plot showing the computational time of eigenvalue computation.

The approximation error of all three fast methods for different dimensions can be seen in Figure 3. The plot on the right-hand side of this figure magnifies the curve for the mean error of the third fast method (the blue curve with dots). As it can be observed in Figure 3, changing the dimensionality has a negligible effect on the mean and the standard deviation (SD) of the absolute approximation error for the fast methods. Small mean error and SD of the third method indicate the distinct advantage of the third fast method over the other two methods. This method uses a convex combination of the values of the other two approximations as explained in Section 2.

Approximating the expected logarithm of the CQF using the fast methods induces an error on the KL-divergence between ACG distributions given by (52). Figure 4 shows the mean percentage of relative error and its standard deviation as a function of dimensionality. It can be observed that the relative error decreases as the dimensionality increases. The third fast method is clearly superior to the other two fast methods. The reason for such a small percentage of relative error is the observation that whenever the error is large, then the KL-divergence is large too. We are not showing the results for the dimensions less than ten because the error percentage is quite large in that regime.

The amount of error in the series method for a given truncation length L depends on the parameters d and ϵ, as it can be observed in the upper bound error formula (26). In Figure 5, this error together with its upper bound are shown for $ϵ=0.6$, $ϵ=0.8$ and $ϵ=0.95$ with $d=2$. The curves are plotted for $L>dϵ/(2-2ϵ)$, because the error upper bound is valid only for this region. It can be observed that by increasing ϵ, the slopes of the error and the error upper bound increase and the distance between these errors slightly increases too. This behavior indicates that the series method is more suitable for smaller ϵ values. For the case of $ϵ=0.6$, we can see that the real error remains constant after about $L=60$, whereas the error upper bound still increases. It happens because the error is computed by the integral method, which has an error itself (in this case, the error is about ${10}^{-15}$).

Figure 6 shows how increasing dimension affects the performance of the series method. The parameter ϵ is set to 0.9 by choosing maximum and minimum weights in the CQF to be 1 and $1/19$, respectively. The other weights of CQF are sampled uniformly between the maximum and minimum weights. It can be seen that the dimensionality has a negligible effect on the slope of the curves. This can be predicted from the formula of upper bound in (26), because the exponential term ${ϵ}^L$ dominates other terms in the equation and the slopes of the curves are determined mainly by the parameter ϵ. In this figure, the standard deviations are due to the different distribution of the weights between the maximum and minimum weights. Figure 5 and Figure 6 demonstrate that the error upper bound is a relatively tight bound for the actual error.

In Figure 7, we investigate the effect of ϵ and d on the averaged L to achieve an acceptable upper bound error (here ${10}^{-8}$). We can see that as the amount of ϵ increases, the slopes of the curves increase and in the limit of $ϵ\to 1$, it goes to infinity. This figure justifies our previous claim that when ϵ and the dimensionality are small, the series method is very efficient due to relatively small L needed to achieve an acceptable error.

5. Conclusions

In this paper, we developed three methods for calculating the expected logarithm of a central quadratic form. The integral method was a direct application of a more general result applicable for positive random variables. We then introduced three fast methods for approximating the expected logarithm. Finally, using an infinite series representation of central quadratic forms, we proposed a series method for computing the expected logarithm. By proving a bound for the approximation error, we investigated the performance of this method.

We also derived expressions for the entropy and the KL-divergence of zero-mean elliptical gamma and angular central Gaussian distributions. The expected logarithm of the central quadratic form appeared in the form of KL-divergences and entropy of the angular central Gaussian distribution.

By conducting multiple experiments, we observed that the three methods for computing the expected logarithm of a central quadratic form differ in running time and accuracy. The possible user can choose the most appropriate method based on his/her requirements.

The methodologies developed in this paper can be used in many applications. For example, one can use the result of Theorem 1 for computing the expected logarithm of other positive random variables like a non-central quadratic form. Another line of research would be to use the KL-divergence between angular central Gaussian distributions with the fast approximations in learning problems that have a divergence measure in their cost functions.