On the Distribution of the Information Density of Gaussian Random Vectors: Explicit Formulas and Tight Approximations

Abstract

Based on the canonical correlation analysis, we derive series representations of the probability density function (PDF) and the cumulative distribution function (CDF) of the information density of arbitrary Gaussian random vectors as well as a general formula to calculate the central moments. Using the general results, we give closed-form expressions of the PDF and CDF and explicit formulas of the central moments for important special cases. Furthermore, we derive recurrence formulas and tight approximations of the general series representations, which allow efficient numerical calculations with an arbitrarily high accuracy as demonstrated with an implementation in Python publicly available on GitLab. Finally, we discuss the (in)validity of Gaussian approximations of the information density.

1. Introduction and Main Theorems

Let $\xi$ and $\eta$ be arbitrary random variables on an abstract probability space $(\mathrm{\Omega },\mathcal{F},\mathrm{P})$ such that the joint distribution ${\mathrm{P}}_{\xi \eta }$ is absolutely continuous w. r. t. the product ${\mathrm{P}}_{\xi }\otimes {\mathrm{P}}_{\eta }$ of the marginal distributions ${\mathrm{P}}_{\xi }$ and ${\mathrm{P}}_{\eta }$. If $\frac{\mathrm{d}{\mathrm{P}}_{\xi \eta }}{\mathrm{d}{\mathrm{P}}_{\xi }\otimes \mathrm{P}\eta }$ denotes the Radon–Nikodym derivative of ${\mathrm{P}}_{\xi \eta }$ w. r. t. ${\mathrm{P}}_{\xi }\otimes {\mathrm{P}}_{\eta }$, then

$$\begin{array}{c}\hfill \mathrm{i}(\xi ;\eta )=log\left(\frac{\mathrm{d}{\mathrm{P}}_{\xi \eta }}{\mathrm{d}{\mathrm{P}}_{\xi }\otimes {\mathrm{P}}_{\eta }}(\xi ,\eta )\right)\end{array}$$

is called the information density of $\xi$ and $\eta$. The expectation $\mathrm{E}\left(\mathrm{i}\right(\xi ;\eta \left)\right)=I(\xi ;\eta )$ of the information density, called mutual information, plays a key role in characterizing the asymptotic channel coding performance in terms of channel capacity. The non-asymptotic performance, however, is determined by the higher-order moments of the information density and its probability distribution. Achievability and converse bounds that allow a finite blocklength analysis of the optimum channel coding rate are closely related to the distribution function of the information density, also called information spectrum by Han and Verdú [1,2]. Moreover, based on the variance of the information density tight second-order finite blocklength approximations of the optimum code rate can be derived for various important channel models. First work on a non-asymptotic information theoretic analysis was already published in the early years of information theory by Shannon [3], Dobrushin [4], and Strassen [5], among others. Due to the seminal work of Polyanskiy et al. [6], considerable progress has been made in this area. The results of [6] on the one hand and the requirements of current and future wireless networks regarding latency and reliability on the other hand stimulated a significant new interest in this type of analysis (Durisi et al. [7]).

The information density $\mathrm{i}(\xi ;\eta )$ in the case when $\xi$ and $\eta$ are jointly Gaussian is of special interest due to the prominent role of the Gaussian distribution. Let $\xi =({\xi }_1,{\xi }_2,\dots ,{\xi }_p)$ and $\eta =({\eta }_1,{\eta }_2,\dots ,{\eta }_q)$ be real-valued random vectors with nonsingular covariance matrices $R_{\xi }$ and $R_{\eta }$ and cross-covariance matrix $R_{\xi rvY}$ with rank $r=rank\left(R_{\xi \eta }\right)$. (For notational convenience, we write vectors as row vectors. However, in expressions where matrix or vector multiplications occur, we consider all vectors as column vectors.) Without loss of generality for the subsequent results, we assume the expectation of all random variables to be zero. If $({\xi }_1,{\xi }_2,\dots ,{\xi }_p,{\eta }_1,{\eta }_2,\dots ,{\eta }_q)$ is a Gaussian random vector, then Pinsker [8], Ch. 9.6 has shown that the distribution of the information density $\mathrm{i}(\xi ;\eta )$ coincides with the distribution of the random variable

$$\begin{array}{cc}\hfill \nu & =\frac{1}{2}\sum _{i=1}^r{\varrho }_i\left({\tilde{\xi }}_i^2-{\tilde{\eta }}_i^2\right)+I(\xi ;\eta ).\hfill \end{array}$$

(1)

In this representation ${\tilde{\xi }}_1,{\tilde{\xi }}_2,\dots ,{\tilde{\xi }}_r,{\tilde{\eta }}_1,{\tilde{\eta }}_2,$…$,{\tilde{\eta }}_r$ are independent and identically distributed (i.i.d.) Gaussian random variables with zero mean and unit variance and the mutual information $I(\xi ;\eta )$ in (1) has the form

$$\begin{array}{cc}\hfill I(\xi ;\eta )& =\frac{1}{2}\sum _{i=1}^{r}log\left(\frac{1}{1-{\varrho }_i^2}\right).\hfill \end{array}$$

(2)

Moreover, ${\varrho }_1\ge {\varrho }_2\ge \dots \ge {\varrho }_r>0$ denote the positive canonical correlations of $\xi$ and $\eta$ in descending order, which are obtained by a linear method called canonical correlation analysis that yields the maximum correlations between two sets of random variables (see Section 3). The rank r of the cross-covariance matrix $R_{\xi \eta }$ satisfies $0\le r\le min\{p,q\}$, and for $r=0$ we have $\mathrm{i}(\xi ;\eta )\equiv 0$ almost surely and $I(\xi ;\eta )=0$. This corresponds to ${\mathrm{P}}_{\xi \eta }={\mathrm{P}}_{\xi }\otimes {\mathrm{P}}_{\eta }$ and the independence of $\xi$ and $\eta$ such that the resulting information density is deterministic. Throughout the rest of the paper, we exclude this degenerated case when the information density is considered and assume subsequently the setting and notation introduced above with $r\ge 1$. As customary notation, we further write $\mathbb{R}$, ${\mathbb{N}}_0$, and $\mathbb{N}$ to denote the set of real numbers, non-negative integers, and positive integers.

Main contributions. Based on (1), we derive in Section 4 series representations of the probability density function (PDF) and the cumulative distribution function (CDF) as well as explicit general formulas for the central moments of the information density $\mathrm{i}(\xi ;\eta )$ given subsequently in Theorems 1 to 3. The series representations are useful as they allow tight approximations with errors as low as desired by finite sums as shown in Section 5.2. Moreover, we derive recurrence formulas in Section 5.1 that allow efficient numerical calculations of the series representations in Theorems 1 and 2.

Theorem 1

(PDF of information density). The PDF $f_{\mathrm{i}(\xi ;\eta )}$ of the information density $\mathrm{i}(\xi ;\eta )$ is given by

$$\begin{array}{c}{f}_{\mathrm{i}(\xi ;\eta )}\left(x\right)=\frac{1}{{\varrho }_r\sqrt{\pi }}\sum _{k_1=0}^{\infty }\sum _{k_2=0}^{\infty }\cdots \sum _{k_{r-1}=0}^{\infty }\left[\prod _{i=1}^{r-1}\frac{{\varrho }_r}{{\varrho }_i}\frac{\left(2k_i\right)!}{{(k_i!)}^2{4}^{k_i}}{\left(1-\frac{{\varrho }_r^2}{{\varrho }_i^2}\right)}^{k_i}\right]\times \hfill \\ \hfill \frac{{\mathrm{K}}_{\frac{r-1}{2}+k_1+k_2+\cdots +k_{r-1}}\left(\frac{\left|x-I(\xi ;\eta )\right|}{{\varrho }_r}\right)}{\Gamma \left(\frac{r}{2}+k_1+k_2+\cdots +k_{r-1}\right)}{\left(\frac{\left|x-I(\xi ;\eta )\right|}{2{\varrho }_r}\right)}^{\left(\frac{r-1}{2}+k_1+k_2+\cdots +k_{r-1}\right)},x\in \mathbb{R}\backslash \left\{I(\xi ;\eta )\right\},\end{array}$$

(3)

where $\Gamma (·)$ denotes the gamma function [9], Sec. 5.2.1 and ${\mathrm{K}}_{\alpha }(·)$ denotes the modified Bessel function of second kind and order α [9], Sec. 10.25(ii). If $r\ge 2$, then $f_{\mathrm{i}(\xi ;\eta )}\left(x\right)$ is also well defined for $x=I(\xi ;\eta )$.

Theorem 2

(CDF of information density). The CDF $F_{\mathrm{i}(\xi ;\eta )}$ of the information density $\mathrm{i}(\xi ;\eta )$ is given by

$$F_{\mathrm{i}(\xi ;\eta )}\left(x\right)=\left\{\begin{array}{cc}\frac{1}{2}-\mathrm{V}\left(I(\xi ;\eta )-x\right)& ifx\le I(\xi ;\eta )\\ \hfill \frac{1}{2}+\mathrm{V}\left(x-I(\xi ;\eta )\right)& ifx>I(\xi ;\eta )\end{array}\right.,$$

with $\mathrm{V}\left(z\right)$ defined by

$$\begin{array}{cc}\hfill \mathrm{V}\left(z\right)=& \sum _{k_1=0}^{\infty }\sum _{k_2=0}^{\infty }\cdots \sum _{k_{r-1}=0}^{\infty }\left[\prod _{i=1}^{r-1}\frac{{\varrho }_r}{{\varrho }_i}\frac{\left(2k_i\right)!}{{(k_i!)}^2{4}^{k_i}}{\left(1-\frac{{\varrho }_r^2}{{\varrho }_i^2}\right)}^{k_i}\right]\frac{z}{2{\varrho }_r}\times \hfill \\ \hfill & [{\mathrm{K}}_{\frac{r-1}{2}+k_1+k_2+\cdots +k_{r-1}}\left(\frac{z}{{\varrho }_r}\right){\mathrm{L}}_{\frac{r-3}{2}+k_1+k_2+\cdots +k_{r-1}}\left(\frac{z}{{\varrho }_r}\right)\hfill \\ \hfill & +{\mathrm{K}}_{\frac{r-3}{2}+k_1+k_2+\cdots +k_{r-1}}\left(\frac{z}{{\varrho }_r}\right){\mathrm{L}}_{\frac{r-1}{2}+k_1+k_2+\cdots +k_{r-1}}\left(\frac{z}{{\varrho }_r}\right)],z\ge 0,\hfill \end{array}$$

(4)

where ${\mathrm{L}}_{\alpha }(·)$ denotes the modified Struve $\mathrm{L}$ function of order α [9], Sec. 11.2.

The method to obtain the result in Theorem 1 is adopted from Mathai [10], where a series representation of the PDF of the sum of independent gamma distributed random variables is derived. Previous work of Grad and Solomon [11] and Kotz et al. [12] goes in a similar direction as Mathai [10]; however, it is not directly applicable since only the restriction to positive series coefficients is considered there. Using Theorem 1, the series representation of the CDF of the information density in Theorem 2 is obtained. The details of the derivations of Theorems 1 and 2 are provided in Section 4.

Theorem 3

(Central moments of information density). The m-th central moment $\mathrm{E}\left({[\mathrm{i}(\xi ;\eta )-I(\xi ;\eta )]}^m\right)$ of the information density $\mathrm{i}(\xi ;\eta )$ is given by

$$\begin{array}{cc}\hfill \mathrm{E}\left(\right[\mathrm{i}(\xi ;\eta )& {-I(\xi ;\eta )]}^m)\hfill \end{array}$$

$$=\left\{\begin{array}{cc}\hfill \sum _{(m_1,m_2,\cdots ,m_r)\in {\mathcal{K}}_{m,r}^{\left[2\right]}}m!\prod _{i=1}^r\frac{\left(2m_i\right)!}{4^{m_i}{(m_i!)}^2}{\varrho }_i^{2m_i}& ifm=2\tilde{m}\hfill \\ 0\hfill & ifm=2\tilde{m}-1\hfill \end{array}\right.,$$

(5)

for all $\tilde{m}\in \mathbb{N}$, where ${\mathcal{K}}_{m,r}^{\left[2\right]}=\left\{(m_1,m_2,\cdots ,m_r)\in {\mathbb{N}}_0^r:2m_1+2m_2+\cdots +2m_r=m\right\}$.

Pinsker [8], Eq. (9.6.17) provided a formula for ${\sum }_{i=1}^r\mathrm{E}\left({\left[\frac{{\varrho }_i}{2}({\tilde{\xi }}_i^2-{\tilde{\eta }}_i^2)\right]}^m\right)$, which he called “derived m-th central moment” of the information density, where ${\tilde{\xi }}_i$ and ${\tilde{\eta }}_i$ are given as in (1). These special moments coincide for $m=2$ with the usual central moments considered in Theorem 3.

The rest of the paper is organized as follows: In Section 2, we discuss important special cases which allow simplified and explicit formulas. In Section 3, we provide some background on the canonical correlation analysis and its application to the calculation of the information density and mutual information for Gaussian random vectors. The proofs of the main Theorems 1 to 3 are given in Section 4. Recurrence formulas, finite sum approximations, and uniform bounds of the approximation error are derived in Section 5, which allow efficient and accurate numerical calculations of the PDF and CDF of the information density. Some examples and illustrations are provided in Section 6, where also the (in)validity of Gaussian approximations is discussed. Finally, Section 7 summarizes the paper. Note that a first version of this paper was published on arXiv as preprint [13].

2. Special Cases

2.1. Equal Canonical Correlations

A simple but important special case for which the series representations in Theorems 1 and 2 simplify to a single summand and the sum of products in Theorem 3 simplifies to a single product is considered in the following corollary.

Corollary 1

(PDF, CDF, and central moments of information density for equal canonical correlations). If all canonical correlations are equal, i.e.,

$${\varrho }_1={\varrho }_2=\dots ={\varrho }_r,$$

then we have the following simplifications.

(i) The PDF $f_{\mathrm{i}(\xi ;\eta )}$ of the information density $\mathrm{i}(\xi ;\eta )$ simplifies to

$$f_{\mathrm{i}(\xi ;\eta )}\left(x\right)=\frac{1}{{\varrho }_r\sqrt{\pi }\Gamma \left(\frac{r}{2}\right)}{\mathrm{K}}_{\frac{r-1}{2}}\left(\frac{\left|x-I(\xi ;\eta )\right|}{{\varrho }_r}\right){\left(\frac{\left|x-I(\xi ;\eta )\right|}{2{\varrho }_r}\right)}^{\frac{r-1}{2}},x\in \mathbb{R}\backslash \left\{I(\xi ;\eta )\right\},$$

(6)

where $I(\xi ;\eta )$ is given by

$$I(\xi ;\eta )=-\frac{r}{2}log\left(1-{\varrho }_r^2\right).$$

If $r\ge 2$, then $f_{\mathrm{i}(\xi ;\eta )}\left(x\right)$ is also well defined for $x=I(\xi ;\eta )$.

(ii) The CDF $F_{\mathrm{i}(\xi ;\eta )}$ of the information density $\mathrm{i}(\xi ;\eta )$ is given by

$$F_{\mathrm{i}(\xi ;\eta )}\left(x\right)=\left\{\begin{array}{cc}\hfill \frac{1}{2}-\mathrm{V}\left(I(\xi ;\eta )-x\right)& ifx\le I(\xi ;\eta )\\ \hfill \frac{1}{2}+\mathrm{V}\left(x-I(\xi ;\eta )\right)& ifx>I(\xi ;\eta )\end{array}\right.,$$

(7)

with $\mathrm{V}\left(z\right)$ defined by

$$\mathrm{V}\left(z\right)=\frac{z}{2{\varrho }_r}\left[{\mathrm{K}}_{\frac{r-1}{2}}\left(\frac{z}{{\varrho }_r}\right){\mathrm{L}}_{\frac{r-3}{2}}\left(\frac{z}{{\varrho }_r}\right)+{\mathrm{K}}_{\frac{r-3}{2}}\left(\frac{z}{{\varrho }_r}\right){\mathrm{L}}_{\frac{r-1}{2}}\left(\frac{z}{{\varrho }_r}\right)\right],z\ge 0.$$

(8)

(iii) The m-th central moment $\mathrm{E}\left({[\mathrm{i}(\xi ;\eta )-I(\xi ;\eta )]}^m\right)$ of the information density $\mathrm{i}(\xi ;\eta )$ has the form

$$\begin{array}{cc}\hfill \mathrm{E}\left({[\mathrm{i}(\xi ;\eta )-I(\xi ;\eta )]}^m\right)=& \left\{\begin{array}{cc}\hfill \frac{m!}{\left(m/2\right)!}\left({\displaystyle \prod _{j=1}^{m/2}}\left(\frac{r}{2}+j-1\right)\right){\varrho }_r^m& ifm=2\tilde{m}\hfill \\ 0\hfill & ifm=2\tilde{m}-1\hfill \end{array},\right.\hfill \end{array}$$

(9)

for all $\tilde{m}\in \mathbb{N}$.

Clearly, if all canonical correlations are equal, then the only nonzero term in the series (3) and (4) occur for $k_1=k_2=\dots =k_{r-1}=0$. For this single summand, the product in squared brackets in (3) and (4) is equal to 1 by applying $0^0=1$, which yields the results of part (i) and (ii) in Corollary 1. Details of the derivation of part (iii) of the corollary are provided in Section 4.

Note, if all canonical correlations are equal, then we can rewrite (1) as follows:

$$\begin{array}{cc}\hfill \nu & =\frac{{\varrho }_r}{2}\left(\sum _{i=1}^r{\tilde{\xi }}_i^2-\sum _{i=1}^r{\tilde{\eta }}_i^2\right)+I(\xi ;\eta ).\hfill \end{array}$$

This implies that $\nu$ coincides with the distribution of the random variable

$$\begin{array}{cc}\hfill {\nu }_{*}& =\frac{{\varrho }_r}{2}\left({\zeta }_1-{\zeta }_2\right)+I(\xi ;\eta ),\hfill \end{array}$$

where ${\zeta }_1$ and ${\zeta }_2$ are i.i.d. ${\chi }^2$-distributed random variables with r degrees of freedom. With this representation, we can obtain the expression of the PDF given in (6) also from [14], Sec. 4.A.4.

Special cases of Corollary 1. The case when all canonical correlations are equal is important because it occurs in various situations. The subsequent cases follow from the properties of canonical correlations given in Section 3.

(i) Assume that the random variables ${\xi }_1,{\xi }_2,\dots ,{\xi }_p,{\eta }_1,{\eta }_2,\dots ,{\eta }_q$ are pairwise uncorrelated with the exception of the pairs $({\xi }_i,{\eta }_i),i=1,2,\dots ,k\le min\{p,q\}$ for which we have $\mathrm{cor}({\xi }_i,{\eta }_i)=\rho \ne 0$, where $\mathrm{cor}(·,·)$ denotes the Pearson correlation coefficient. Then, $r=k$ and ${\varrho }_i=\left|\rho \right|$ for all $i=1,2,\dots ,r$. Note, if $p=q=k$, then for the previous conditions to hold, it is sufficient that the two-dimensional random vectors $({\xi }_i,{\eta }_i)$ are i.i.d. However, the identical distribution of the $({\xi }_i,{\eta }_i)$’s is not necessary. In Laneman [15], the distribution of the information density for an additive white Gaussian noise channel with i.i.d. Gaussian input is determined. This is a special case of the case with i.i.d. random vectors $({\xi }_i,{\eta }_i)$ just mentioned. In Wu and Jindal [16] and in Buckingham and Valenti [17], an approximation of the information density by a Gaussian random variable is considered for the setting in [15]. A special case very similar to that in [15] is also considered in Polyanskiy et al. [6], Sec. III.J. To the best of the authors’ knowledge, explicit formulas for the general case as considered in this paper are not available yet in the literature.

(ii) Assume that the conditions of part (i) are satisfied. Furthermore, assume that $\widehat{A}$ is a real nonsingular matrix of dimension $p\times p$ and $\widehat{B}$ is a real nonsingular matrix of dimension $q\times q$. Then, the random vectors

$$\begin{array}{c}\hfill \widehat{\xi }=\widehat{A}\xi \mathrm{and}\widehat{\eta }=\widehat{B}\eta \end{array}$$

have the same canonical correlations as the random vectors $\xi$ and $\eta$, i.e., ${\varrho }_i=\left|\rho \right|$ for all $i=1,2,\dots ,k\le min\{p,q\}$.

(iii) If $r=1$, i.e., if the cross-covariance matrix $R_{\xi ,\eta }$ has rank 1, then Corollary 1 obviously applies. Clearly, the most simple special case with $r=1$ occurs for $p=q=1$, where ${\varrho }_1=\left|\mathrm{cor}({\xi }_1,{\eta }_1)\right|$.

As a simple multivariate example, let the covariance matrix of the random vector $({\xi }_1,{\xi }_2,\dots ,{\xi }_p,{\eta }_1,{\eta }_2,\dots ,{\eta }_q)$ be given by the Kac-Murdock–Szegö matrix

$$\begin{array}{c}\hfill \left(\begin{array}{cc}{R}_{\xi }& R_{\xi \eta }\\ R_{\xi \eta }& R_{\eta }\end{array}\right)={\left({\rho }^{|i-j|}\right)}_{i,j=1}^{p+q}\end{array}$$

which is related to the covariance function of a first-order autoregressive process, where $0<\left|\rho \right|<1$. Then, $r=rank\left(R_{\xi \eta }\right)=1$ and ${\varrho }_1=\left|\rho \right|$.

(iv) As yet another example, assume $p=q$ and $R_{\xi \eta }=\rho R_{\xi }^{1/2}{R}_{\eta }^{1/2}$ for some $0<\left|\rho \right|<1$. Then, ${\varrho }_i=\left|\rho \right|$ for $i=1,2,\dots ,r=q$. Here, $A^{1/2}$ denotes the square root of the real-valued positive semidefinite matrix A, i.e., the unique positive semidefinite matrix B such that $BB=A$.

2.2. More on Special Cases with Simplified Formulas

Let us further evaluate the formulas given in Corollary 1 and Theorem 3 for some relevant parameter values.

(i) Single canonical correlation coefficient. In the most simple case, there is only a single non-zero canonical correlation coefficient, i.e., $r=1$. (Recall, in the beginning of the paper, we have excluded the degenerated case when all canonical correlations are zero.) Then, the formulas of the PDF and the m-th central moment in Corollary 1 simplify to the form

$$f_{\mathrm{i}(\xi ;\eta )}\left(x\right)=\frac{1}{{\varrho }_1\pi }{\mathrm{K}}_0\left(\frac{\left|x-I(\xi ;\eta \right|)}{{\varrho }_1}\right),x\in \mathbb{R}\backslash \left\{I(\xi ;\eta )\right\},$$

and

$$\begin{array}{cc}\hfill \mathrm{E}\left({[\mathrm{i}(\xi ;\eta )-I(\xi ;\eta )]}^m\right)=& \left\{\begin{array}{cc}\hfill {\left(\frac{m!}{\left(m/2\right)!}\right)}^2{\left(\frac{{\varrho }_1}{2}\right)}^m& ifm=2\tilde{m}\hfill \\ 0\hfill & ifm=2\tilde{m}-1\hfill \end{array}\right.\hfill \end{array}$$

(10)

for all $\tilde{m}\in \mathbb{N}$. A formula equivalent to (10) is also provided by Pinsker [8], Lemma 9.6.1 who considered the special case $p=q=1$, which implies $r=1$.

(ii) Second and fourth central moment. To demonstrate how the general formula given in Theorem 3 is used, we first consider $m=2$. In this case, the summation indices $m_1,m_2,\dots ,m_r$ have to satisfy $m_i=1$ for a single $i\in \{1,2,\dots ,r\}$, whereas the remaining $m_i$’s have to be zero. Thus, (5) evaluates for $m=2$ to

$$\begin{array}{c}\hfill \mathrm{E}\left({[\mathrm{i}(\xi ;\eta )-I(\xi ;\eta )]}^2\right)=\mathrm{var}\left(\mathrm{i}(\xi ;\eta )\right)=\sum _{i=1}^r{\varrho }_i^2.\end{array}$$

(11)

As a slightly more complex example, let $m=4$. In this case, either we have $m_i=2$ for a single $i\in \{1,2,\dots ,r\}$, whereas the remaining $m_i$’s are zero or we have $m_{i_1}=m_{i_2}=1$ for two $i_1\ne i_2\in \{1,2,\dots ,r\}$, whereas the remaining $m_i$’s have to be zero. Thus, (5) evaluates for $m=4$ to

$$\begin{array}{c}\hfill \mathrm{E}\left({[\mathrm{i}(\xi ;\eta )-I(\xi ;\eta )]}^4\right)=9\sum _{i=1}^r{\varrho }_i^4+6\sum _{i=2}^r\sum _{j=1}^{i-1}{\varrho }_i^2{\varrho }_j^2.\end{array}$$

(iii) Even number of equal canonical correlations. As in Corollary 1, assume that all canonical correlations are equal and additionally assume that the number r of canonical correlations is even, i.e., $r=2\tilde{r}$ for some $\tilde{r}\in \mathbb{N}$. Then, we can use [9], Secs. 10.47.9, 10.49.1, and 10.49.12 to obtain the following relation for the modified Bessel function ${\mathrm{K}}_{\alpha }(·)$ of a second kind and order $\alpha$

$$\begin{array}{c}\hfill {\mathrm{K}}_{\frac{r-1}{2}}\left(y\right)=\sqrt{\frac{\pi }{2}}exp\left(-y\right)\sum _{i=0}^{r/2-1}\frac{\left(r/2-1+i\right)!}{\left(r/2-1-i\right)!i!2^i}{y}^{-(i+\frac{1}{2})},y\in (0,\infty ).\end{array}$$

(12)

Plugging (12) into (6) and rearranging terms yields the following expression for the PDF of the information density:

$$\begin{array}{cc}\hfill f_{\mathrm{i}(\xi ;\eta )}\left(x\right)=\frac{1}{{\varrho }_r{2}^{r-1}\left(r/2-1\right)!}& exp\left(-\frac{\left|x-I(\xi ;\eta )\right|}{{\varrho }_r}\right)\times \hfill \\ \hfill & \sum _{i=0}^{r/2-1}\frac{\left(2(r/2-1)-i\right)!2^i}{\left(r/2-1-i\right)!i!}{\left(\frac{\left|x-I(\xi ;\eta )\right|}{{\varrho }_r}\right)}^i,x\in \mathbb{R}.\hfill \end{array}$$

By integration, we obtain for the function $\mathrm{V}(·)$ in (8) the expression

$$\begin{array}{cc}\hfill \mathrm{V}\left(z\right)=\frac{1}{2}-\frac{1}{2^{r-1}\left(r/2-1\right)!}& exp\left(-\frac{z}{{\varrho }_r}\right)\times \hfill \\ \hfill & \sum _{i=0}^{r/2-1}\frac{\left(2(r/2-1)-i\right)!2^i}{\left(r/2-1-i\right)!}\sum _{j=0}^i\frac{1}{(i-j)!}{\left(\frac{z}{{\varrho }_r}\right)}^{i-j},z\ge 0.\hfill \end{array}$$

Note that these special formulas can also be obtained directly from the results given in [14], Sec. 4.A.3.

To illustrate the principal behavior of the PDF and CDF of the information density for equal canonical correlations, it is instructive to consider the specific value $r=2$ in the above formulas, which yields

$$\begin{array}{cccc}\hfill f_{\mathrm{i}(\xi ;\eta )}\left(x\right)& =\frac{1}{2{\varrho }_r}exp\left(-\frac{\left|x-I(\xi ;\eta )\right|}{{\varrho }_r}\right),\hfill & & x\in \mathbb{R},\hfill \\ \hfill \mathrm{V}\left(z\right)& =\frac{1}{2}\left(1-exp\left(-\frac{z}{{\varrho }_r}\right)\right),\hfill & & z\ge 0\hfill \end{array}$$

and $r=4$, for which we obtain

$$\begin{array}{cccc}\hfill f_{\mathrm{i}(\xi ;\eta )}\left(x\right)& =\frac{1}{4{\varrho }_r}exp\left(-\frac{\left|x-I(\xi ;\eta )\right|}{{\varrho }_r}\right)\left(1+\frac{\left|x-I(\xi ;\eta )\right|}{{\varrho }_r}\right),\hfill & & x\in \mathbb{R},\hfill \\ \hfill \mathrm{V}\left(z\right)& =\frac{1}{2}\left(1-exp\left(-\frac{z}{{\varrho }_r}\right)\left(1+\frac{z}{2{\varrho }_r}\right)\right),\hfill & & z\ge 0.\hfill \end{array}$$

3. Mutual Information and Information Density in Terms of Canonical Correlations

First introduced by Hotelling [18], the canonical correlation analysis is a widely used linear method in multivariate statistics to determine the maximum correlations between two sets of random variables. It allows a particularly simple and useful representation of the mutual information and the information density of Gaussian random vectors in terms of the so-called canonical correlations. This representation was first obtained by Gelfand and Yaglom [19] and further extended by Pinsker [8], Ch. 9. For the convenience of the reader, we summarize in this section the essence of the canonical correlation analysis and demonstrate how it is applied to derive the representations in (1) and (2).

The formulation of the canonical correlation analysis given below is particularly suitable for implementations. The corresponding results are given without proof. Details and thorough discussions can be found, e.g., in Härdle and Simar [20], Koch [21], or Timm [22].

Based on the nonsingular covariance matrices $R_{\xi }$ and $R_{\eta }$ of the random vectors $\xi =({\xi }_1,{\xi }_2,\dots ,{\xi }_p)$ and $\eta =({\eta }_1,{\eta }_2,\dots ,{\eta }_q)$, and the cross-covariance matrix $R_{\xi \eta }$ with rank $r=rank\left(R_{\xi \eta }\right)$ satisfying $0\le r\le min\{p,q\}$, define the matrix

$$M=R_{\xi }^{-\frac{1}{2}}{R}_{\xi \eta }{R}_{\eta }^{-\frac{1}{2}},$$

where the inverse matrices $R_{\xi }^{-1/2}={\left(R_{\xi }^{1/2}\right)}^{-1}$ and $R_{\eta }^{-1/2}={\left(R_{\eta }^{1/2}\right)}^{-1}$ can be obtained from diagonalizing $R_{\xi }$ and $R_{\eta }$. Then, the matrix M has a singular value decomposition

$$M=UD{V}^{\mathrm{T}},$$

where $V^{\mathrm{T}}$ denotes the transpose of V. The only non-zero entries $d_{1,1},d_{2,2},\dots ,d_{r,r}>0$ of the matrix $D={\left(d_{i,j}\right)}_{i,j=1}^{p,q}$ are called canonical correlations of $\xi$ and $\eta$, denoted by ${\varrho }_i=d_{i,i},i=1,2,\dots ,r$. The singular value decomposition can be chosen such that ${\varrho }_1\ge {\varrho }_2\ge \dots \ge {\varrho }_r$ holds, which is assumed throughout the paper.

Define the random vectors

$$\begin{array}{c}\hfill \widehat{\xi }=({\widehat{\xi }}_1,{\widehat{\xi }}_2,\dots ,{\widehat{\xi }}_p)=A\xi \mathrm{and}\widehat{\eta }=({\widehat{\eta }}_1,{\widehat{\eta }}_2,\dots ,{\widehat{\eta }}_q)=B\eta ,\end{array}$$

where the nonsingular matrices A and B are given by

$$\begin{array}{c}\hfill A=U^{\mathrm{T}}{R}_{\xi }^{-\frac{1}{2}}\mathrm{and}B=V^{\mathrm{T}}{R}_{\eta }^{-\frac{1}{2}}.\end{array}$$

Then, the random variables ${\widehat{\xi }}_1,{\widehat{\xi }}_2,\dots ,{\widehat{\xi }}_p,{\widehat{\eta }}_1,{\widehat{\eta }}_2,\dots ,{\widehat{\eta }}_q$ have unit variance and they are pairwise uncorrelated with the exception of the pairs $({\widehat{\xi }}_i,{\widehat{\eta }}_i),i=1,2,\dots ,r$ for which we have $\mathrm{cor}({\widehat{\xi }}_i,{\widehat{\eta }}_i)={\varrho }_i$.

Using these results, we obtain for the mutual information and the information density

$$\begin{array}{cccccc}\hfill I(\xi ;\eta )& =I(A\xi ;B\eta )\hfill & & =I(\widehat{\xi };\widehat{\eta })\hfill & & =\sum _{i=1}^{r}I({\widehat{\xi }}_i;{\widehat{\eta }}_i)\hfill \end{array}$$

(13)

$$\begin{array}{cccccc}\hfill \mathrm{i}(\xi ;\eta )& =\mathrm{i}(A\xi ;B\eta )\hfill & & =\mathrm{i}(\widehat{\xi };\widehat{\eta })\hfill & & =\sum _{i=1}^r\mathrm{i}({\widehat{\xi }}_i;{\widehat{\eta }}_i)(\mathrm{P-almost}\mathrm{surely}).\hfill \end{array}$$

(14)

The first equality in (13) and (14) holds because A and B are nonsingular matrices, which follows, e.g., from Pinsker [8], Th. 3.7.1. Since we consider the case where $\xi$ and $\eta$ are jointly Gaussian, $\widehat{\xi }$ and $\widehat{\eta }$ are jointly Gaussian as well. Therefore, the correlation properties of $\widehat{\xi }$ and $\widehat{\eta }$ imply that all random variables ${\widehat{\xi }}_i,{\widehat{\eta }}_j$ are independent except for the pairs $({\widehat{\xi }}_i,{\widehat{\eta }}_i)$, $i=1,2,\dots ,r$. This implies the last equality in (13) and (14), where $\mathrm{i}({\widehat{\xi }}_1;{\widehat{\eta }}_1),\mathrm{i}({\widehat{\xi }}_2;{\widehat{\eta }}_2),\dots ,\mathrm{i}({\widehat{\xi }}_r;{\widehat{\eta }}_r)$ are independent. The sum representations follow from the chain rules of mutual information and information density and the equivalence between independence and vanishing mutual information and information density.

Since ${\widehat{\xi }}_i$ and ${\widehat{\eta }}_i$ are jointly Gaussian with correlation $\mathrm{cor}({\widehat{\xi }}_i,{\widehat{\eta }}_i)={\varrho }_i$, we obtain from (13) and the formula of mutual information for the bivariate Gaussian case the identity (2). Additionally, with ${\widehat{\xi }}_i$ and ${\widehat{\eta }}_i$ having zero mean and unit variance, the information density $\mathrm{i}({\widehat{\xi }}_i;{\widehat{\eta }}_i)$ is further given by

$$\begin{array}{c}\hfill \mathrm{i}({\widehat{\xi }}_i;{\widehat{\eta }}_i)=-\frac{1}{2}log(1-{\varrho }_i^2)-\frac{{\varrho }_i^2}{2(1-{\varrho }_i^2)}\left({\widehat{\xi }}_i^2-\frac{2{\widehat{\xi }}_i{\widehat{\eta }}_i}{{\varrho }_i}+{\widehat{\eta }}_i^2\right),i=1,2,\dots ,r.\end{array}$$

(15)

Now assume ${\tilde{\xi }}_1,{\tilde{\xi }}_2,\dots ,{\tilde{\xi }}_r,{\tilde{\eta }}_1,{\tilde{\eta }}_2,$…$,{\tilde{\eta }}_r$ are i.i.d. Gaussian random variables with zero mean and unit variance. Then, the distribution of the random vector

$$\begin{array}{c}\hfill \frac{1}{\sqrt{2}}\left(\sqrt{1+{\varrho }_i}{\tilde{\xi }}_i+\sqrt{1-{\varrho }_i}{\tilde{\eta }}_i,\sqrt{1+{\varrho }_i}{\tilde{\xi }}_i-\sqrt{1-{\varrho }_i}{\tilde{\eta }}_i\right)\end{array}$$

coincides with the distribution of the random vector $({\widehat{\xi }}_i,{\widehat{\eta }}_i)$ for all $i=1,2,\dots ,r$. Plugging this into (15), we obtain together with (14) that the distribution of the information density $\mathrm{i}(\xi ;\eta )$ coincides with the distribution of (1).

4. Proof of Main Results

4.1. Auxiliary Results

To prove Theorem 1, the following lemma regarding the characteristic function of the information density is utilized. The results of the lemma are also used in Ibragimov and Rozanov [23] but without proof. Therefore, the proof is given below for completeness.

Lemma 1 

(Characteristic function of (shifted) information density). The characteristic function of the shifted information density $\mathrm{i}(\xi ;\eta )-I(\xi ;\eta )$ is equal to the characteristic function of the random variable

$$\begin{array}{cc}\hfill \tilde{\nu }& =\frac{1}{2}\sum _{i=1}^r{\varrho }_i\left({\tilde{\xi }}_i^2-{\tilde{\eta }}_i^2\right),\hfill \end{array}$$

(16)

where ${\tilde{\xi }}_1,{\tilde{\xi }}_2,\dots ,{\tilde{\xi }}_r,{\tilde{\eta }}_1,{\tilde{\eta }}_2,$…$,{\tilde{\eta }}_r$ are i.i.d. Gaussian random variables with zero mean and unit variance, and ${\varrho }_1,{\varrho }_2,\dots ,{\varrho }_r$ are the canonical correlations of ξ and η. The characteristic function of $\tilde{\nu }$ is given by

$${\phi }_{\tilde{\nu }}\left(t\right)=\prod _{i=1}^r\frac{1}{\sqrt{1+{\varrho }_i^2{t}^2}},t\in \mathbb{R}.$$

(17)

Proof. 

Due to (1), the distribution of the shifted information density $\mathrm{i}(\xi ;\eta )-I(\xi ;\eta )$ coincides with the distribution of the random variable $\tilde{\nu }$ in (16) such that the characteristic functions of $\mathrm{i}(\xi ;\eta )-I(\xi ;\eta )$ and $\tilde{\nu }$ are equal.

It is a well known fact that ${\tilde{\xi }}_i^2$ and ${\tilde{\eta }}_i^2$ in (16) are chi-squared distributed random variables with one degree of freedom from which we obtain that the weighted random variables ${\varrho }_i{\tilde{\xi }}_i^2/2$ and ${\varrho }_i{\tilde{\eta }}_i^2/2$ are gamma distributed with a scale parameter of $1/{\varrho }_i$ and shape parameter of $1/2$. The characteristic function of these random variables therefore admits the form

$${\phi }_{\frac{{\varrho }_i}{2}{\tilde{\xi }}_i^2}\left(t\right)={\left(1-{\varrho }_{i}jt\right)}^{-\frac{1}{2}}.$$

Further, from the identity ${\phi }_{-{\varrho }_i{\tilde{\xi }}_i^2/2}\left(t\right)={\phi }_{{\varrho }_i{\tilde{\xi }}_i^2/2}(-t)$ for the characteristic function and from the independence of ${\tilde{\xi }}_i$ and ${\tilde{\eta }}_i$, we obtain the characteristic function of ${\tilde{\nu }}_i={\varrho }_i({\tilde{\xi }}_i^2-{\tilde{\eta }}_i^2)/2$ to be given by

$${\phi }_{{\tilde{\nu }}_i}\left(t\right)={\left(1-{\varrho }_{i}jt\right)}^{-\frac{1}{2}}{\left(1+{\varrho }_{i}jt\right)}^{-\frac{1}{2}}={\left(1+{\varrho }_i^2{t}^2\right)}^{-\frac{1}{2}}.$$

Finally, because $\tilde{\nu }$ in (16) is given by the sum of the independent random variables ${\tilde{\nu }}_i$, the characteristic function of $\tilde{\nu }$ results from multiplying the individual characteristic functions of the random variables ${\tilde{\nu }}_i$. By doing so, we obtain (17). □

As further auxiliary result, the subsequent proposition providing properties of the modified Bessel function ${\mathrm{K}}_{\alpha }$ of second kind and order $\alpha$ will be used to prove the main results.

Proposition 1

(Properties related to the function ${\mathrm{K}}_{\alpha }$). For all $\alpha \in \mathbb{R}$, the function

$$\begin{array}{c}\hfill y\mapsto y^{\alpha }{\mathrm{K}}_{\alpha }\left(y\right),y\in (0,\infty ),\end{array}$$

where ${\mathrm{K}}_{\alpha }(·)$ denotes the modified Bessel function of second kind and order α [9], Sec. 10.25(ii), is strictly positive and strictly monotonically decreasing. Furthermore, if $\alpha >0$, then we have

$$\begin{array}{c}\hfill \underset{y\to +0}{lim}{y}^{\alpha }{\mathrm{K}}_{\alpha }\left(y\right)=\underset{y\in (0,\infty )}{sup}{y}^{\alpha }{\mathrm{K}}_{\alpha }\left(y\right)=\Gamma \left(\alpha \right)2^{\alpha -1}.\end{array}$$

(18)

Proof. 

If $\alpha \in \mathbb{R}$ is fixed, then ${\mathrm{K}}_{\alpha }\left(y\right)$ is strictly positive and strictly monotonically decreasing w. r. t. $y\in (0,\infty )$ due to [9], Secs. 10.27.3 and 10.37. Furthermore, we obtain

$$\begin{array}{c}\hfill \frac{\mathrm{d}{y}^{\alpha }{\mathrm{K}}_{\alpha }\left(y\right)}{\mathrm{d}y}=-y^{\alpha }{\mathrm{K}}_{\alpha -1}\left(y\right),y\in (0,\infty )\end{array}$$

by applying the rules to calculate derivatives of Bessel functions given in [9], Sec. 10.29(ii). It follows that $y^{\alpha }{\mathrm{K}}_{\alpha }\left(y\right)$ is strictly positive and strictly monotonically decreasing w. r. t. $y\in (0,\infty )$ for all fixed $\alpha \in \mathbb{R}$.

Consider now the Basset integral formula as given in [9], Sec. 10.32.11

$$\begin{array}{c}{\mathrm{K}}_{\alpha }\left(yz\right)=\frac{\Gamma \left(\alpha +\frac{1}{2}\right){\left(2z\right)}^{\alpha }}{y^{\alpha }\sqrt{\pi }}\underset{u=0}{\overset{\infty }{\int }}\frac{cos\left(uy\right)}{{\left(u^2+z^2\right)}^{\alpha +\frac{1}{2}}}\mathrm{d}u\end{array}$$

(19)

for $|arg\left(z\right)|<\pi /2,y>0,\alpha >-\frac{1}{2}$ and the integral

$$\begin{array}{c}\hfill \underset{u=0}{\overset{\infty }{\int }}\frac{1}{{\left(u^2+1\right)}^{\alpha +\frac{1}{2}}}\mathrm{d}u=\frac{\sqrt{\pi }\Gamma \left(\alpha \right)}{2\Gamma \left(\alpha +\frac{1}{2}\right)}\end{array}$$

(20)

for $\alpha >0$, where the equality holds due to [24], Secs. 3.251.2 and 8.384.1. Using (19) and (20), we obtain

$$\begin{array}{cc}\hfill \underset{y\to +0}{lim}{y}^{\alpha }{\mathrm{K}}_{\alpha }\left(y\right)& =\underset{y\to +0}{lim}\frac{\Gamma \left(\alpha +\frac{1}{2}\right)2^{\alpha }}{\sqrt{\pi }}\underset{u=0}{\overset{\infty }{\int }}\frac{cos\left(uy\right)}{{\left(u^2+1\right)}^{\alpha +\frac{1}{2}}}\mathrm{d}u\hfill \\ \hfill & =\frac{\Gamma \left(\alpha +\frac{1}{2}\right)2^{\alpha }}{\sqrt{\pi }}\underset{u=0}{\overset{\infty }{\int }}\frac{1}{{\left(u^2+1\right)}^{\alpha +\frac{1}{2}}}\mathrm{d}u=\Gamma \left(\alpha \right)2^{\alpha -1},\hfill \end{array}$$

for all $\alpha >0$, where we also applied the dominated convergence theorem, which is possible due to $cos\left(uy\right)/{\left(u^2+1\right)}^{\alpha +1/2}\le 1/{\left(u^2+1\right)}^{\alpha +1/2}$. Using the previously derived monotonicity, we obtain (18). □

4.2. Proof of Theorem 1

To prove Theorem 1, we calculate the PDF $f_{\tilde{\nu }}$ of the random variable $\tilde{\nu }$ introduced in Lemma 1 by inverting the characteristic function ${\phi }_{\tilde{\nu }}$ given in (17) via the integral

$$\begin{array}{c}\hfill f_{\tilde{\nu }}\left(v\right)=\frac{1}{2\pi }{\int }_{-\infty }^{\infty }{\phi }_{\tilde{\nu }}\left(t\right)exp\left(-Jtv\right)\mathrm{d}t,v\in \mathbb{R}.\end{array}$$

(21)

Shifting the PDF of $\tilde{\nu }$ by $I(\xi ;\eta )$, we obtain the PDF $f_{\mathrm{i}(\xi ;\eta )}=f_{\tilde{\nu }}(x-I(\xi ;\eta ))$, $x\in \mathbb{R}$, of the information density $\mathrm{i}(\xi ;\eta )$.

The method used subsequently is based on the work of Mathai [10]. To invert the characteristic function ${\phi }_{\tilde{\nu }}$, we expand the factors in (17) as

$$\begin{array}{cc}\hfill {\left(1+{\varrho }_i^2{t}^2\right)}^{-\frac{1}{2}}& ={\left(1+{\varrho }_r^2{t}^2\right)}^{-\frac{1}{2}}\frac{{\varrho }_r}{{\varrho }_i}{\left(1+\left(\frac{{\varrho }_r^2}{{\varrho }_i^2}-1\right){\left(1+{\varrho }_r^2{t}^2\right)}^{-1}\right)}^{-\frac{1}{2}}\hfill \end{array}$$

(22)

$$\begin{array}{cc}\hfill & ={\left(1+{\varrho }_r^2{t}^2\right)}^{-\frac{1}{2}}\sum _{k=0}^{\infty }{(-1)}^k\left(\genfrac{}{}{0pt}{}{-1/2}{k}\right)\frac{{\varrho }_r}{{\varrho }_i}{\left(1-\frac{{\varrho }_r^2}{{\varrho }_i^2}\right)}^k{\left(1+{\varrho }_r^2{t}^2\right)}^{-k}.\hfill \end{array}$$

(23)

In (23), we have used the binomial series

$$\begin{array}{c}\hfill {(1+y)}^a=\sum _{k=0}^{\infty }\left(\genfrac{}{}{0pt}{}{a}{k}\right)y^k\end{array}$$

(24)

where $a\in \mathbb{R}$. The series is absolutely convergent for $\left|y\right|<1$ and

$$\begin{array}{c}\hfill \left(\genfrac{}{}{0pt}{}{a}{k}\right)=\prod _{\ell =1}^k\frac{a-\ell +1}{\ell },k\in \mathbb{N},\end{array}$$

(25)

denotes the generalized binomial coefficient with $\left(\genfrac{}{}{0pt}{}{a}{0}\right)=1$. Since

$$\begin{array}{c}\hfill \left|\left(1-\frac{{\varrho }_r^2}{{\varrho }_i^2}\right){\left(1+{\varrho }_r^2{t}^2\right)}^{-1}\right|<1\end{array}$$

(26)

holds for all $t\in \mathbb{R}$, the series in (23) is absolutely convergent for all $t\in \mathbb{R}$. Using the expansion in (23) and the absolute convergence together with the identity

$$\begin{array}{c}\hfill \left(\genfrac{}{}{0pt}{}{-1/2}{k}\right)=\frac{{(-1)}^k\left(2k\right)!}{{(k!)}^2{4}^k}\end{array}$$

(27)

we can rewrite the characteristic function ${\phi }_{\tilde{\nu }}$ as

$$\begin{array}{cc}\hfill {\phi }_{\tilde{\nu }}\left(t\right)=\sum _{k_1=0}^{\infty }\sum _{k_2=0}^{\infty }\cdots \sum _{k_{r-1}=0}^{\infty }& \left[\prod _{i=1}^{r-1}\frac{{\varrho }_r}{{\varrho }_i}\frac{\left(2k_i\right)!}{{(k_i!)}^2{4}^{k_i}}{\left(1-\frac{{\varrho }_r^2}{{\varrho }_i^2}\right)}^{k_i}\right]\times \hfill \\ \hfill & {\left(1+{\varrho }_r^2{t}^2\right)}^{-\left(\frac{r}{2}+k_1+k_2+\cdots +k_{r-1}\right)},t\in \mathbb{R}.\hfill \end{array}$$

(28)

To obtain the PDF $f_{\tilde{\nu }}$, we evaluate the inversion integral (21) based on the series representation in (28). Since every series in (28) is absolutely convergent, we can exchange summation and integration. Let $\beta =\frac{r}{2}+k_1+k_2+\cdots +k_{r-1}$. Then, by symmetry, we have for the integral of a summand

$$\underset{t=-\infty }{\overset{\infty }{\int }}\frac{exp\left(-Jtv\right)}{{(1+{\varrho }_r^2{t}^2)}^{\beta }}\mathrm{d}t=2\underset{t=0}{\overset{\infty }{\int }}\frac{cos\left(tv\right)}{{(1+{\varrho }_r^2{t}^2)}^{\beta }}\mathrm{d}t=\frac{2}{{\varrho }_r}\underset{u=0}{\overset{\infty }{\int }}\frac{cos\left(uv/{\varrho }_r\right)}{{(1+u^2)}^{\beta }}\mathrm{d}u,$$

(29)

where the second equality is a result of the substitution $t=u/{\varrho }_r$. By setting $z=1$, $\alpha =\beta -\frac{1}{2}\ge 0$ and $y=v/{\varrho }_r$ in the Basset integral formula given in (19) in the proof of Proposition 1 and using the symmetry with respect to v, we can evaluate (29) to the following form:

$$\underset{t=-\infty }{\overset{\infty }{\int }}\frac{exp\left(-Jtv\right)}{{(1+{\varrho }_r^2{t}^2)}^{\beta }}\mathrm{d}t=\frac{\sqrt{\pi }}{\Gamma \left(\beta \right)2^{\beta -\frac{3}{2}}{\varrho }_r^{\beta +\frac{1}{2}}}{\mathrm{K}}_{\beta -\frac{1}{2}}\left(\frac{\left|v\right|}{{\varrho }_r}\right){\left|v\right|}^{\beta -\frac{1}{2}},v\in \mathbb{R}\backslash \left\{0\right\}.$$

(30)

Combining (21), (28), and (30) yields

$$\begin{array}{c}{f}_{\tilde{\nu }}\left(v\right)=\frac{1}{2\sqrt{\pi }}\sum _{k_1=0}^{\infty }\sum _{k_2=0}^{\infty }\cdots \sum _{k_{r-1}=0}^{\infty }\left[\prod _{i=1}^{r-1}\frac{{\varrho }_r}{{\varrho }_i}\frac{\left(2k_i\right)!}{{(k_i!)}^2{4}^{k_i}}{\left(1-\frac{{\varrho }_r^2}{{\varrho }_i^2}\right)}^{k_i}\right]\times \hfill \\ \hfill \frac{{\mathrm{K}}_{\frac{r-1}{2}+k_1+k_2+\cdots +k_{r-1}}\left(\frac{\left|v\right|}{{\varrho }_r}\right){\left|v\right|}^{\left(\frac{r-1}{2}+k_1+k_2+\cdots +k_{r-1}\right)}}{\Gamma \left(\frac{r}{2}+k_1+k_2+\cdots +k_{r-1}\right)2^{\left(\frac{r-3}{2}+k_1+k_2+\cdots +k_{r-1}\right)}{\varrho }_r^{\left(\frac{r+1}{2}+k_1+k_2+\cdots +k_{r-1}\right)}},v\in \mathbb{R}\backslash \left\{0\right\}.\end{array}$$

(31)

Slightly rearranging terms and shifting $f_{\tilde{\nu }}(·)$ by $I(\xi ;\eta )$ yields (3).

It remains to show that $f_{\mathrm{i}(\xi ;\eta )}\left(x\right)$ is also well defined for $x=I(\xi ;\eta )$ if $r\ge 2$. Indeed, if $r\ge 2$, then we can use Proposition 1 to obtain

$$\begin{array}{c}\hfill \underset{x\to I(\xi ;\eta )}{lim}{f}_{\mathrm{i}(\xi ;\eta )}\left(x\right)=\frac{1}{2{\varrho }_r\sqrt{\pi }}\sum _{k_1=0}^{\infty }\sum _{k_2=0}^{\infty }\cdots \sum _{k_{r-1}=0}^{\infty }\left[\prod _{i=1}^{r-1}\frac{{\varrho }_r}{{\varrho }_i}\frac{\left(2k_i\right)!}{{(k_i!)}^2{4}^{k_i}}{\left(1-\frac{{\varrho }_r^2}{{\varrho }_i^2}\right)}^{k_i}\right]\times \\ \hfill \frac{\Gamma \left(\frac{r-1}{2}+k_1+k_2+\cdots +k_{r-1}\right)}{\Gamma \left(\frac{r-1}{2}+k_1+k_2+\cdots +k_{r-1}+\frac{1}{2}\right)}\end{array}$$

where we used the exchangeability of the limit and the summation due to the absolute convergence of the series. Since $\Gamma \left(\alpha \right)/\Gamma (\alpha +\frac{1}{2})$ is decreasing w. r. t. $\alpha \ge \frac{1}{2}$, we have

$$\begin{array}{c}\hfill \frac{\Gamma \left(\frac{r-1}{2}+k_1+k_2+\cdots +k_{r-1}\right)}{\Gamma \left(\frac{r-1}{2}+k_1+k_2+\cdots +k_{r-1}+\frac{1}{2}\right)}\le \frac{\Gamma \left(\frac{r-1}{2}\right)}{\Gamma \left(\frac{r-1}{2}+\frac{1}{2}\right)}\le \sqrt{\pi }.\end{array}$$

Then, with (69) in the proof of Theorem 4, it follows that ${lim}_{x\to I(\xi ;\eta )}{f}_{\mathrm{i}(\xi ;\eta )}\left(x\right)$ exists and is finite. □

4.3. Proof of Theorem 2

To prove Theorem 2, we calculate the CDF $F_{\tilde{\nu }}$ of the random variable $\tilde{\nu }$ introduced in Lemma 1 by integrating the PDF $f_{\tilde{\nu }}$ given in (31). Shifting the CDF of $\tilde{\nu }$ by $I(\xi ;\eta )$, we obtain the CDF $F_{\mathrm{i}(\xi ;\eta )}\left(x\right)=F_{\tilde{\nu }}(x-I(\xi ;\eta )),x\in \mathbb{R}$, of the information density $\mathrm{i}(\xi ;\eta )$. Using the symmetry of $f_{\tilde{\nu }}$, we can write

$$F_{\tilde{\nu }}\left(z\right)=\mathrm{P}(\tilde{\nu }\le z)=\left\{\begin{array}{cc}\hfill \frac{1}{2}-{\int }_{v=0}^{-z}{f}_{\tilde{\nu }}\left(v\right)\mathrm{d}v& \mathrm{for}z\le 0\\ \hfill \frac{1}{2}+{\int }_{v=0}^z{f}_{\tilde{\nu }}\left(v\right)\mathrm{d}v& \mathrm{for}z>0\end{array}\right..$$

It is therefore sufficient to evaluate the integral

$$\begin{array}{c}\hfill \mathrm{V}\left(z\right):={\int }_{v=0}^z{f}_{\tilde{\nu }}\left(v\right)\mathrm{d}v\end{array}$$

(32)

for $z\ge 0$. To calculate the integral (32), we plug (31) into (32) and exchange integration and summation, which is justified by the monotone convergence theorem. To evaluate the integral of a summand, consider the following identity

$$\begin{array}{c}\hfill {\int }_{x=0}^z{x}^{\alpha }{\mathrm{K}}_{\alpha }\left(x\right)\mathrm{d}x=2^{\alpha -1}\sqrt{\pi }\Gamma \left(\alpha +\frac{1}{2}\right)z\left[{\mathrm{K}}_{\alpha }\left(z\right){\mathrm{L}}_{\alpha -1}\left(z\right)+{\mathrm{K}}_{\alpha -1}\left(z\right){\mathrm{L}}_{\alpha }\left(z\right)\right]\end{array}$$

(33)

for $\alpha >-1/2$ given in [25], Sec. 1.12.1.3, where ${\mathrm{L}}_{\alpha }(·)$ denotes the modified Struve $\mathrm{L}$ function of order $\alpha$ [9], Sec. 11.2. Using (33) with $\alpha =\frac{r-1}{2}+k_1+k_2+\cdots +k_{r-1}\ge 0$, we obtain (4). □

4.4. Proof of Theorem 3

Using the random variable

$$\begin{array}{c}\hfill \tilde{\nu }=\sum _{i=1}^r{\tilde{\nu }}_i\mathrm{with}{\tilde{\nu }}_i=\frac{{\varrho }_i}{2}({\tilde{\xi }}_i-{\tilde{\eta }}_i)\end{array}$$

introduced in Lemma 1 and the well-known multinomial theorem [9], Sec. 26.4.9

$$\begin{array}{c}\hfill {\left(y_1+y_2+\dots y_r\right)}^m=\sum _{({\ell }_1,{\ell }_2,\dots ,{\ell }_r)\in {\mathcal{K}}_{m,r}}m!\prod _{i=1}^r\frac{y_i^{{\ell }_i}}{{\ell }_i!},\end{array}$$

where ${\mathcal{K}}_{m,r}=\left\{({\ell }_1,{\ell }_2,\cdots ,{\ell }_r)\in {\mathbb{N}}_0^r:{\ell }_1+{\ell }_2+\cdots +{\ell }_r=m\right\}$, we can write the m-th central moment of the information density $\mathrm{i}(\xi ;\eta )$ as

$$\begin{array}{cc}\hfill \mathrm{E}\left({[\mathrm{i}(\xi ;\eta )-I(\xi ;\eta )]}^m\right)& =\mathrm{E}\left({\left[\sum _{i=1}^r{\tilde{\nu }}_i\right]}^m\right)\hfill \\ \hfill & =\sum _{({\ell }_1,{\ell }_2,\dots ,{\ell }_r)\in {\mathcal{K}}_{m,r}}m!\prod _{i=1}^r\frac{\mathrm{E}\left({\tilde{\nu }}_i^{{\ell }_i}\right)}{{\ell }_i!}.\hfill \end{array}$$

(34)

To obtain the second equality in (34), we have exchanged expectation and summation and additionally used the identity $\mathrm{E}\left({\prod }_{i=1}^r{\tilde{\nu }}_i^{{\ell }_i}\right)={\prod }_{i=1}^r\mathrm{E}\left({\tilde{\nu }}_i^{{\ell }_i}\right)$, which holds due to the independence of the random variables ${\tilde{\nu }}_1,{\tilde{\nu }}_2,\dots ,{\tilde{\nu }}_r$.

Based on the relation between the ℓ-th central moment of a random variable and the ℓ-th derivative of its characteristic function at 0, we further have

$$\begin{array}{c}\hfill \mathrm{E}\left({\tilde{\nu }}_i^{{\ell }_i}\right)={(-j)}^{{\ell }_i}\frac{{\mathrm{d}}^{{\ell }_i}}{\mathrm{d}{t}^{{\ell }_i}}{\phi }_{{\tilde{\nu }}_i}\left(t\right){|}_{t=0},\end{array}$$

(35)

where ${\phi }_{{\tilde{\nu }}_i}\left(t\right)={\left(1+{\varrho }_i^2{t}^2\right)}^{-1/2},t\in \mathbb{R},$ is the characteristic function of the random variable ${\tilde{\nu }}_i$ derived in the proof of Lemma 1. As in the proof of Theorem 1, consider now the binomial series expansion using (24)

$$\begin{array}{c}\hfill {\phi }_{{\tilde{\nu }}_i}\left(t\right)={\left(1+{\varrho }_i^2{t}^2\right)}^{-\frac{1}{2}}=\sum _{m_i=0}^{\infty }\left(\genfrac{}{}{0pt}{}{-1/2}{m_i}\right){\left({\varrho }_{i}t\right)}^{2m_i}.\end{array}$$

The series is absolutely convergent for all $t<{\varrho }_i^{-1}$. Furthermore, consider the Taylor series expansion of the characteristic function ${\phi }_{{\tilde{\nu }}_i}$ at the point 0

$$\begin{array}{c}\hfill {\phi }_{{\tilde{\nu }}_i}\left(t\right)=\sum _{{\ell }_i=0}^{\infty }\left(\frac{{\mathrm{d}}^{{\ell }_i}}{\mathrm{d}{t}^{{\ell }_i}}{\phi }_{{\tilde{\nu }}_i}\left(t\right){|}_{t=0}\right)\frac{t^{{\ell }_i}}{{\ell }_i!}.\end{array}$$

Both series expansions must be identical in an open interval around 0 such that we obtain by comparing the series coefficients

$$\begin{array}{cc}\hfill \frac{{\mathrm{d}}^{{\ell }_i}}{\mathrm{d}{t}^{{\ell }_i}}{\phi }_{{\tilde{\nu }}_i}\left(t\right){|}_{t=0}=& \left\{\begin{array}{cc}\hfill {\ell }_i!\left(\genfrac{}{}{0pt}{}{-1/2}{{\ell }_i/2}\right){\varrho }_i^{{\ell }_i}& \mathrm{if}{\ell }_i=2m_i\hfill \\ 0\hfill & \mathrm{if}{\ell }_i=2m_i-1\hfill \end{array}\right.\hfill \end{array}$$

for all $m_i\in \mathbb{N}$. With this result, (35) evaluates to

$$\begin{array}{cc}\hfill \mathrm{E}\left({\tilde{\nu }}_i^{{\ell }_i}\right)=& \left\{\begin{array}{cc}\hfill \frac{{\left({\ell }_i!\right)}^2}{{\left(({\ell }_i/2)!\right)}^2{4}^{\frac{{\ell }_i}{2}}}{\varrho }_i^{{\ell }_i}& \mathrm{if}{\ell }_i=2m_i\hfill \\ 0\hfill & \mathrm{if}{\ell }_i=2m_i-1\hfill \end{array}\right.\hfill \end{array}$$

(36)

for all $m_i\in \mathbb{N}$, where we have additionally used the identity (27).

From (34) and (36) we now obtain $\mathrm{E}\left({[\mathrm{i}(\xi ;\eta )-I(\xi ;\eta )]}^m\right)=0$ for all $m=2\tilde{m}-1$ with $\tilde{m}\in \mathbb{N}$ because, if m is odd, then for all $({\ell }_1,{\ell }_2,\cdots ,{\ell }_r)\in {\mathcal{K}}_{m,r}$ at least one of the ${\ell }_i$’s has to be odd. If $m=2\tilde{m}$ with $\tilde{m}\in \mathbb{N}$, we obtain from (34) and (36)

$$\begin{array}{cc}\hfill \mathrm{E}\left({[\mathrm{i}(\xi ;\eta )-I(\xi ;\eta )]}^m\right)& =\sum _{({\ell }_1,{\ell }_2,\dots ,{\ell }_r)\in {\mathcal{K}}_{m,r}}m!\prod _{i=1}^r\frac{1}{{\ell }_i!}\frac{{\left({\ell }_i!\right)}^2}{{\left(({\ell }_i/2)!\right)}^2{4}^{\frac{{\ell }_i}{2}}}{\varrho }_i^{{\ell }_i}\hfill \\ \hfill & =\sum _{(m_1,m_2,\dots ,m_r)\in {\mathcal{K}}_{m,r}^{\left[2\right]}}m!\prod _{i=1}^r\frac{\left(2m_i\right)!}{{\left(m_i!\right)}^2{4}^{m_i}}{\varrho }_i^{2m_i}.\hfill \end{array}$$

 □

4.5. Proof of Part (iii) of Corollary 1

Using the random variable $\tilde{\nu }$ as in the proof of Theorem 3, we can write the m-th central moment of the information density $\mathrm{i}(\xi ;\eta )$ as

$$\begin{array}{c}\hfill \mathrm{E}\left({[\mathrm{i}(\xi ;\eta )-I(\xi ;\eta )]}^m\right)=\mathrm{E}\left({\tilde{\nu }}^m\right)={(-j)}^m\frac{{\mathrm{d}}^m}{\mathrm{d}{t}^m}{\phi }_{\tilde{\nu }}\left(t\right){|}_{t=0},\end{array}$$

where the characteristic function ${\phi }_{\tilde{\nu }}$ of $\tilde{\nu }$ is given by ${\phi }_{\tilde{\nu }}\left(t\right)={\left(1+{\varrho }_r^2{t}^2\right)}^{-r/2},t\in \mathbb{R}$, due to Lemma 1 and the equality of all canonical correlations. Using the binomial series and the Taylor series expansion as in the proof of Theorem 3, we obtain

$$\begin{array}{cc}\hfill \frac{{\mathrm{d}}^m}{\mathrm{d}{t}^m}{\phi }_{\tilde{\nu }}\left(t\right){|}_{t=0}=& \left\{\begin{array}{cc}\hfill m!\left(\genfrac{}{}{0pt}{}{-r/2}{m/2}\right){\varrho }_r^m& \mathrm{if}m=2\tilde{m}\hfill \\ 0\hfill & \mathrm{if}m=2\tilde{m}-1\hfill \end{array}\right.\hfill \end{array}$$

for all $\tilde{m}\in \mathbb{N}$. Collecting terms and additionally using the definition of the generalized binomial coefficient given in (25) in the proof of Theorem 1 yields (9). □

5. Recurrence Formulas and Finite Sum Approximations

If there are at least two distinct canonical correlations, then the PDF $f_{\mathrm{i}(\xi ;\eta )}$ and CDF $F_{\mathrm{i}(\xi ;\eta )}$ of the information density $\mathrm{i}(\xi ;\eta )$ are given by the infinite series in Theorems 1 and 2. If we consider only a finite number of summands in these representations, then we obtain approximations amenable in particular for numerical calculations. However, a direct finite sum approximation of the series in (3) and (4) is rather inefficient since modified Bessel and Struve $\mathrm{L}$ functions have to be evaluated for every summand. Therefore, we derive in this section recursive representations, which allow efficient numerical calculations. Furthermore, we derive uniform bounds of the approximation error. Based on the recurrence relations and the error bounds, an implementation in the programming language Python has been developed, which provides an efficient tool to numerically calculate the PDF and CDF of the information density with a predefined accuracy as high as desired. The developed source code as well as illustrating examples are made publicly available in an open access repository on GitLab [26].

Subsequently, we adopt all the previous notation and assume $r\ge 2$ and at least two distinct canonical correlations (since otherwise we have the case of Corollary 1, where the series reduce to a single summand).

5.1. Recurrence Formulas

The recursive approach developed below is based on the work of Moschopoulos [27], which extended the work of Mathai [10]. First, we rewrite the series representations of the PDF and CDF of the information density given in Theorem 1 and Theorem 2 in a form, which is suitable for recursive calculations. To begin with, we define two functions appearing in the series representations (3) and (4), which involve the modified Bessel function ${\mathrm{K}}_{\alpha }$ of second kind and order $\alpha$ and the modified Struve $\mathrm{L}$ function ${\mathrm{L}}_{\alpha }$ of order $\alpha$. Let us define for all $k\in {\mathbb{N}}_0$ the functions ${\mathrm{U}}_k$ and ${\mathrm{D}}_k$ by

$$\begin{array}{c}\hfill {\mathrm{U}}_k\left(z\right)=\frac{{\mathrm{K}}_{\frac{r-1}{2}+k}\left(z\right)}{\Gamma \left(\frac{r}{2}+k\right)}{\left(\frac{z}{2}\right)}^{\frac{r-1}{2}+k},z\ge 0\end{array}$$

(37)

and

$$\begin{array}{c}\hfill {\mathrm{D}}_k\left(z\right)=\frac{z}{2{\varrho }_r}\left[{\mathrm{K}}_{\frac{r-1}{2}+k}\left(\frac{z}{{\varrho }_r}\right){\mathrm{L}}_{\frac{r-3}{2}+k}\left(\frac{z}{{\varrho }_r}\right)+{\mathrm{K}}_{\frac{r-3}{2}+k}\left(\frac{z}{{\varrho }_r}\right){\mathrm{L}}_{\frac{r-1}{2}+k}\left(\frac{z}{{\varrho }_r}\right)\right],z\ge 0.\end{array}$$

(38)

Furthermore, we define for all $k\in {\mathbb{N}}_0$ the coefficient ${\delta }_k$ by

$$\begin{array}{c}\hfill {\delta }_k=\sum _{(k_1,k_2,\cdots ,k_{r-1})\in {\mathcal{K}}_{k,r-1}}\left[\prod _{i=1}^{r-1}\frac{\left(2k_i\right)!}{{(k_i!)}^2{4}^{k_i}}{\left(1-\frac{{\varrho }_r^2}{{\varrho }_i^2}\right)}^{k_i}\right],\end{array}$$

(39)

where ${\mathcal{K}}_{k,r-1}=\left\{(k_1,k_2,\cdots ,k_{r-1})\in {\mathbb{N}}_0^{r-1}:k_1+k_2+\cdots +k_{r-1}=k\right\}$. With these definitions, we obtain the following alternative series representations of (3) and (4) by observing that the multiple summations over the indices $k_1,k_2,\dots ,k_{r-1}$ can be shortened to one summation over the index $k=k_1+k_2+\dots +k_{r-1}$.

Proposition 2

(Alternative representation of PDF and CDF of the information density). The PDF $f_{\mathrm{i}(\xi ;\eta )}$ of the information density $\mathrm{i}(\xi ;\eta )$ given in Theorem 1 has the alternative series representation

$$\begin{array}{c}\hfill f_{\mathrm{i}(\xi ;\eta )}\left(x\right)=\frac{1}{{\varrho }_r\sqrt{\pi }}\left(\prod _{i=1}^{r-1}\frac{{\varrho }_r}{{\varrho }_i}\right)\sum _{k=0}^{\infty }{\delta }_k{\mathrm{U}}_k\left(\frac{\left|x-I(\xi ;\eta )\right|}{{\varrho }_r}\right),x\in \mathbb{R}.\end{array}$$

(40)

The function $\mathrm{V}(·)$ specifying the CDF $F_{\mathrm{i}(\xi ;\eta )}$ of the information density $\mathrm{i}(\xi ;\eta )$ as given in Theorem 2 has the alternative series representation

$$\begin{array}{c}\hfill \mathrm{V}\left(z\right)=\left(\prod _{i=1}^{r-1}\frac{{\varrho }_r}{{\varrho }_i}\right)\sum _{k=0}^{\infty }{\delta }_k{\mathrm{D}}_k\left(z\right),z\ge 0.\end{array}$$

(41)

Based on the representations in Proposition 2 and with recursive formulas for ${\mathrm{U}}_k(·)$, ${\mathrm{D}}_k(·)$ and ${\delta }_k$, we are in the position to calculate the PDF and CDF of the information density by a single summation over completely recursively defined terms. In the following, we will derive recurrence relations for ${\mathrm{U}}_k(·)$, ${\mathrm{D}}_k(·)$ and ${\delta }_k$, which allow the desired efficient calculations.

Lemma 2

(Recurrence formula of the function ${\mathrm{U}}_k$). If for all $k\in {\mathbb{N}}_0$ the function ${\mathrm{U}}_k$ is defined by (37), then ${\mathrm{U}}_k\left(z\right)$ satisfies for all $k\ge 2$ and $z\ge 0$ the recurrence formula

$$\begin{array}{c}\hfill {\mathrm{U}}_k\left(z\right)=\frac{z^2}{(r+2k-2)(r+2k-4)}{\mathrm{U}}_{k-2}\left(z\right)+\frac{r+2k-3}{r+2k-2}{\mathrm{U}}_{k-1}\left(z\right).\end{array}$$

(42)

Proof. 

First, assume $z=0$. Based on Proposition 1, we obtain for all $k\in {\mathbb{N}}_0$

$$\begin{array}{c}\hfill \underset{z\to +0}{lim}{\mathrm{U}}_k\left(z\right)=\frac{\Gamma \left(\frac{r-1}{2}+k\right)}{2\Gamma \left(\frac{r}{2}+k\right)},\end{array}$$

(43)

such that ${\mathrm{U}}_k\left(0\right)$ is well defined and finite. Using the recurrence relation $\Gamma (y+1)=y\Gamma \left(y\right)$ for the Gamma function [24], Sec. 8.331.1 we have

$$\begin{array}{c}\hfill \frac{\Gamma \left(\frac{r-1}{2}+k\right)}{2\Gamma \left(\frac{r}{2}+k\right)}=\frac{\left(\frac{r-1}{2}+k-1\right)}{\left(\frac{r}{2}+k-1\right)}·\frac{\Gamma \left(\frac{r-1}{2}+k-1\right)}{2\Gamma \left(\frac{r}{2}+k-1\right)}.\end{array}$$

This shows together with (43) that the recurrence formula (42) holds for ${\mathrm{U}}_k\left(0\right)$ and $k\ge 2$.

Now, assume $z>0$ and consider the recurrence formula

$$\begin{array}{c}\hfill z{\mathrm{K}}_{\alpha }\left(z\right)=z{\mathrm{K}}_{\alpha -2}\left(z\right)+2(\alpha -1){\mathrm{K}}_{\alpha -1}\left(z\right)\end{array}$$

(44)

for the modified Bessel function of the second kind and order $\alpha$ [24], Sec. 8.486.10. Plugging (44) into (37) for $\alpha =\frac{r-1}{2}+k$ yields for $k\ge 2$

$$\begin{array}{cc}\hfill {\mathrm{U}}_k\left(z\right)=\frac{{\mathrm{K}}_{\frac{r-1}{2}+k-2}\left(z\right)}{\Gamma \left(\frac{r}{2}+k\right)}& {\left(\frac{z}{2}\right)}^{\frac{r-1}{2}+k-2}{\left(\frac{z}{2}\right)}^2\hfill \\ \hfill & +\frac{\left(\frac{r-1}{2}+k-1\right){\mathrm{K}}_{\frac{r-1}{2}+k-1}\left(z\right)}{\Gamma \left(\frac{r}{2}+k\right)}{\left(\frac{z}{2}\right)}^{\frac{r-1}{2}+k-1}.\hfill \end{array}$$

(45)

Using again the relation $\Gamma (y+1)=y\Gamma \left(y\right)$, we obtain

$$\begin{array}{cc}\hfill \Gamma \left(\frac{r}{2}+k\right)& =\left(\frac{r}{2}+k-1\right)\Gamma \left(\frac{r}{2}+k-1\right)=\left(\frac{r}{2}+k-1\right)\left(\frac{r}{2}+k-2\right)\Gamma \left(\frac{r}{2}+k-2\right),\hfill \end{array}$$

which yields together with (45) and (37) the recurrence formula (42) for ${\mathrm{U}}_k\left(z\right)$ if $z>0$ and $k\ge 2$. □

Lemma 3

(Recurrence formula of the function ${\mathrm{D}}_k$). If, for all $k\in {\mathbb{N}}_0$, the function ${\mathrm{D}}_k$ is defined by (38), then ${\mathrm{D}}_k\left(z\right)$ satisfies for all $k\ge 1$ and $z\ge 0$ the recurrence formula

$$\begin{array}{c}\hfill {\mathrm{D}}_k\left(z\right)={\mathrm{D}}_{k-1}\left(z\right)-\frac{1}{2\sqrt{\pi }\left(\frac{r}{2}+k-1\right)}\left(\frac{z}{{\varrho }_r}\right){\mathrm{U}}_{k-1}\left(\frac{z}{{\varrho }_r}\right),\end{array}$$

(46)

with ${\mathrm{U}}_k(·)$ as defined in (37).

Proof. 

First, assume $z=0$. We have ${\mathrm{D}}_k\left(0\right)=0$ for all $k\in {\mathbb{N}}_0$ and from the proof of Lemma 2 we have ${\mathrm{U}}_k\left(0\right)=\Gamma \left(\frac{r-1}{2}+k\right)/2\Gamma \left(\frac{r}{2}+k\right)$ for all $k\in {\mathbb{N}}_0$. Thus, the left-hand side and the right-hand side of (46) are both zero, which shows that (46) holds for $z=0$ and $k\ge 1$.

Now, assume $z>0$ and consider the recurrence formula

$$\begin{array}{c}\hfill z{\mathrm{L}}_{\alpha }\left(z\right)=z{\mathrm{L}}_{\alpha -2}\left(z\right)-2(\alpha -1){\mathrm{L}}_{\alpha -1}\left(z\right)-\frac{2^{1-\alpha }{z}^{\alpha }}{\sqrt{\pi }\Gamma \left(\alpha +\frac{1}{2}\right)}\end{array}$$

for the modified Struve $\mathrm{L}$ function of order $\alpha$ [9], Sec. 11.4.25. Together with the recurrence formula (44) for the modified Bessel function of the second kind and order $\alpha$, we obtain

$$\begin{array}{cc}\hfill z{\mathrm{L}}_{\alpha }\left(z\right){\mathrm{K}}_{\alpha -1}\left(z\right)& =z{\mathrm{L}}_{\alpha -2}\left(z\right){\mathrm{K}}_{\alpha -1}\left(z\right)-2(\alpha -1){\mathrm{L}}_{\alpha -1}\left(z\right){\mathrm{K}}_{\alpha -1}\left(z\right)\hfill \end{array}$$

$$\begin{array}{cccc}\hfill & & & -\frac{2^{1-\alpha }{z}^{\alpha }}{\sqrt{\pi }\Gamma \left(\alpha +\frac{1}{2}\right)}{\mathrm{K}}_{\alpha -1}\left(z\right),\hfill \end{array}$$

(47)

$$\begin{array}{cccc}\hfill z{\mathrm{K}}_{\alpha }\left(z\right){\mathrm{L}}_{\alpha -1}\left(z\right)& =z{\mathrm{K}}_{\alpha -2}\left(z\right){\mathrm{L}}_{\alpha -1}\left(z\right)\hfill & & +2(\alpha -1){\mathrm{K}}_{\alpha -1}\left(z\right){\mathrm{L}}_{\alpha -1}\left(z\right).\hfill \end{array}$$

(48)

Plugging (47) and (48) into (38) for $\alpha =\frac{r-1}{2}+k$ yields for $k\ge 1$

$$\begin{array}{cc}\hfill {\mathrm{D}}_k\left(z\right)=\frac{z}{2{\varrho }_r}[& {\mathrm{K}}_{\frac{r-1}{2}+k-1}\left(\frac{z}{{\varrho }_r}\right){\mathrm{L}}_{\frac{r-3}{2}+k-1}\left(\frac{z}{{\varrho }_r}\right)+{\mathrm{K}}_{\frac{r-3}{2}+k-1}\left(\frac{z}{{\varrho }_r}\right){\mathrm{L}}_{\frac{r-1}{2}+k-1}\left(\frac{z}{{\varrho }_r}\right)]\hfill \\ \hfill & -\frac{1}{\sqrt{\pi }\Gamma \left(\frac{r}{2}+k\right)}{\left(\frac{z}{2{\varrho }_r}\right)}^{\frac{r-1}{2}+k}{\mathrm{K}}_{\frac{r-1}{2}+k-1}\left(\frac{z}{{\varrho }_r}\right).\hfill \end{array}$$

Together with (38), the identity $\Gamma \left(\frac{r}{2}+k\right)=\left(\frac{r}{2}+k-1\right)\Gamma \left(\frac{r}{2}+k-1\right)$, and the definition of the function ${\mathrm{U}}_k(·)$ in (37), we obtain the recurrence formula (46) for ${\mathrm{D}}_k\left(z\right)$ if $z>0$ and $k\ge 1$. □

Lemma 4

(Recursive formula of the coefficient ${\delta }_k$). The coefficient ${\delta }_k$ defined by (39) satisfies for all $k\in {\mathbb{N}}_0$ the recurrence formula

$$\begin{array}{c}\hfill {\delta }_{k+1}=\frac{1}{k+1}\sum _{j=1}^{k+1}j{\gamma }_j{\delta }_{k+1-j},\end{array}$$

(49)

where ${\delta }_0=1$ and

$$\begin{array}{c}\hfill {\gamma }_j=\sum _{i=1}^{r-1}\frac{1}{2j}{\left(1-\frac{{\varrho }_r^2}{{\varrho }_i^2}\right)}^j.\end{array}$$

(50)

For the derivation of Lemma 4, we use an adapted version of the method of Moschopoulos [27] and the following auxiliary result.

Lemma 5.

For $k\in {\mathbb{N}}_0$, let g be a real univariate $(k+1)$-times differentiable function. Then, we have the following recurrence relation for the $(k+1)$-th derivative of the composite function $h=exp\left(g\right)$

$$h^{(k+1)}=\sum _{j=1}^{k+1}\left(\genfrac{}{}{0pt}{}{k}{j-1}\right)g^{\left(j\right)}{h}^{(k-j+1)},$$

(51)

where $f^{\left(i\right)}$ denotes the i-th derivative of the function f with $f^{\left(0\right)}=f$.

Proof. 

We prove the assertion of Lemma 5 by induction over k. First, consider the base case for $k=0$. In this case, formula (51) gives

$$h^{\left(1\right)}=g^{\left(1\right)}h,$$

which is easily seen to be true.

Assuming formula (51) holds for $h^{\left(k\right)}$, we continue with the case $k+1$. Application of the product rule leads to

$$\begin{array}{cc}\hfill h^{(k+1)}=& {\left(h^{\left(k\right)}\right)}^{\left(1\right)}={\left(\sum _{j=1}^k\left(\genfrac{}{}{0pt}{}{k-1}{j-1}\right)g^{\left(j\right)}{h}^{(k-j)}\right)}^{\left(1\right)}\hfill \\ \hfill =& \sum _{j=1}^k\left(\genfrac{}{}{0pt}{}{k-1}{j-1}\right)g^{(j+1)}{h}^{(k-j)}+\sum _{j=1}^k\left(\genfrac{}{}{0pt}{}{k-1}{j-1}\right)g^{\left(j\right)}{h}^{(k-j+1)}.\hfill \end{array}$$

Substitution of $j^{\prime }=j+1$ in the first term gives

$$\begin{array}{cc}\hfill h^{(k+1)}=& \sum _{j^{\prime }=2}^{k+1}\left(\genfrac{}{}{0pt}{}{k-1}{j^{\prime }-2}\right)g^{\left(j^{\prime }\right)}{h}^{(k-j^{\prime }+1)}+\sum _{j=1}^k\left(\genfrac{}{}{0pt}{}{k-1}{j-1}\right)g^{\left(j\right)}{h}^{(k-j+1)}.\hfill \end{array}$$

With this representation and the identity,

$$\left(\genfrac{}{}{0pt}{}{k-1}{j-2}\right)+\left(\genfrac{}{}{0pt}{}{k-1}{j-1}\right)=\left(\genfrac{}{}{0pt}{}{k}{j-1}\right)$$

We finally have

$$\begin{array}{cc}\hfill h^{(k+1)}=& g^{\left(1\right)}{h}^{\left(k\right)}+\sum _{j=2}^k\left[\left(\genfrac{}{}{0pt}{}{k-1}{j-1}\right)+\left(\genfrac{}{}{0pt}{}{k-1}{j-2}\right)\right]g^{\left(j\right)}{h}^{(k-j+1)}+g^{(k+1)}h\hfill \\ \hfill =& \left(\genfrac{}{}{0pt}{}{k}{0}\right)g^{\left(1\right)}{h}^{\left(k\right)}+\sum _{j=2}^k\left(\genfrac{}{}{0pt}{}{k}{j-1}\right)g^{\left(j\right)}{h}^{(k-j+1)}+\left(\genfrac{}{}{0pt}{}{k}{k}\right)g^{(k+1)}h\hfill \\ \hfill =& \sum _{j=1}^{k+1}\left(\genfrac{}{}{0pt}{}{k}{j-1}\right)g^{\left(j\right)}{h}^{(k-j+1)}.\hfill \end{array}$$

This completes the proof of Lemma 5. □

Proof of Lemma 4.

To prove the recurrence formula (49), we consider the characteristic function

$${\phi }_{\tilde{\nu }}\left(t\right)=\prod _{i=1}^r{\left(1+{\varrho }_i^2{t}^2\right)}^{-\frac{1}{2}},t\in \mathbb{R}$$

(52)

of the random variable $\tilde{\nu }$ introduced in Lemma 1. On the one hand, the series representation of ${\phi }_{\tilde{\nu }}$ given in (28) in the proof of Theorem 1 can be rewritten as follows using the coefficient ${\delta }_k$ defined in (39):

$$\begin{array}{c}\hfill {\phi }_{\tilde{\nu }}\left(t\right)={\left(1+{\varrho }_r^2{t}^2\right)}^{-\frac{r}{2}}\left(\prod _{i=1}^{r-1}\frac{{\varrho }_r}{{\varrho }_i}\right)\sum _{\ell =0}^{\infty }{\delta }_{\ell }{\left(1+{\varrho }_r^2{t}^2\right)}^{-\ell },t\in \mathbb{R}.\end{array}$$

(53)

On the other hand, recall the expansion of ${\left(1+{\varrho }_i^2{t}^2\right)}^{-\frac{1}{2}}$ given in (22), which yields together with (52) and the application of the natural logarithm the identity

$$\begin{array}{cc}\hfill log\left({\phi }_{\tilde{\nu }}\left(t\right)\right)=& log\left({\left(1+{\varrho }_r^2{t}^2\right)}^{-\frac{r}{2}}\left(\prod _{i=1}^{r-1}\frac{{\varrho }_r}{{\varrho }_i}\right)\right)\hfill \\ \hfill & +\sum _{i=1}^{r-1}log\left({\left(1+\left(\frac{{\varrho }_r^2}{{\varrho }_i^2}-1\right){\left(1+{\varrho }_r^2{t}^2\right)}^{-1}\right)}^{-\frac{1}{2}}\right).\hfill \end{array}$$

(54)

Now consider the power series

$$log(1+y)=\sum _{\ell =1}^{\infty }\frac{{(-1)}^{\ell +1}}{\ell }{y}^{\ell },$$

(55)

which is absolutely convergent for $\left|y\right|<1$. With the same arguments as in the proof of Theorem 1, in particular due to (26), we can apply the series expansion (55) to the second term on the right-hand side of (54) to obtain the absolutely convergent series representation

$$\begin{array}{c}\hfill \begin{array}{cc}\hfill log\left({\phi }_{\tilde{\nu }}\left(t\right)\right)=& log\left({\left(1+{\varrho }_r^2{t}^2\right)}^{-\frac{r}{2}}\left(\prod _{i=1}^{r-1}\frac{{\varrho }_r}{{\varrho }_i}\right)\right)+\sum _{\ell =1}^{\infty }{\gamma }_{\ell }{\left(1+{\varrho }_r^2{t}^2\right)}^{-\ell },\hfill \end{array}\end{array}$$

(56)

where we have further used the definition of ${\gamma }_{\ell }$ given in (50). Applying the exponential function to both sides of (56) then yields the following expression for the characteristic function ${\phi }_{\tilde{\nu }}$.

$$\begin{array}{cc}\hfill {\phi }_{\tilde{\nu }}\left(t\right)=& {\left(1+{\varrho }_r^2{t}^2\right)}^{-\frac{r}{2}}\left(\prod _{i=1}^{r-1}\frac{{\varrho }_r}{{\varrho }_i}\right)exp\left(\sum _{\ell =1}^{\infty }{\gamma }_{\ell }{\left(1+{\varrho }_r^2{t}^2\right)}^{-\ell }\right)\hfill \end{array}$$

(57)

Comparing (53) and (57) yields the identity

$$\sum _{\ell =0}^{\infty }{\delta }_{\ell }{\left(1+{\varrho }_r^2{t}^2\right)}^{-\ell }=exp\left(\sum _{\ell =1}^{\infty }{\gamma }_{\ell }{\left(1+{\varrho }_r^2{t}^2\right)}^{-\ell }\right).$$

(58)

We now define $x={\left(1+{\varrho }_r^2{t}^2\right)}^{-1}$ and take the $(k+1)$-th derivative w. r. t. x on both sides of (58) using the identity

$$\frac{{\mathrm{d}}^m}{\mathrm{d}{x}^m}\left(\sum _{\ell =0}^{\infty }{a}_{\ell }{x}^{\ell }\right)=\frac{{\mathrm{d}}^m}{\mathrm{d}{x}^m}\left(\sum _{\ell =1}^{\infty }{a}_{\ell }{x}^{\ell }\right)=\sum _{\ell =m}^{\infty }\frac{\ell !}{(\ell -m)!}{a}_{\ell }{x}^{\ell -m}$$

(59)

for the m-th derivative of a power series ${\sum }_{\ell =0}^{\infty }{a}_{\ell }{x}^{\ell }$. For the left-hand side of (58), we obtain

$$\begin{array}{c}\hfill \frac{{\mathrm{d}}^{k+1}}{\mathrm{d}{x}^{k+1}}\left(\sum _{\ell =0}^{\infty }{\delta }_{\ell }{x}^{\ell }\right)=\sum _{\ell =k+1}^{\infty }\frac{\ell !}{(\ell -k-1)!}{\delta }_{\ell }{x}^{\ell -k-1}.\end{array}$$

(60)

For the right-hand side of (58), we obtain

$$\begin{array}{cc}\hfill \frac{{\mathrm{d}}^{k+1}}{\mathrm{d}{x}^{k+1}}\left(exp\left(\sum _{\ell =1}^{\infty }{\gamma }_{\ell }{x}^{\ell }\right)\right)& =\sum _{j=1}^{k+1}\left(\genfrac{}{}{0pt}{}{k}{j-1}\right)\frac{{\mathrm{d}}^j}{\mathrm{d}{x}^j}\left(\sum _{\ell =1}^{\infty }{\gamma }_{\ell }{x}^{\ell }\right)\frac{{\mathrm{d}}^{k-j+1}}{\mathrm{d}{x}^{k-j+1}}\left(exp\left(\sum _{\ell =1}^{\infty }{\gamma }_{\ell }{x}^{\ell }\right)\right)\hfill \\ \hfill & =\sum _{j=1}^{k+1}\left(\genfrac{}{}{0pt}{}{k}{j-1}\right)\frac{{\mathrm{d}}^j}{\mathrm{d}{x}^j}\left(\sum _{\ell =1}^{\infty }{\gamma }_{\ell }{x}^{\ell }\right)\frac{{\mathrm{d}}^{k-j+1}}{\mathrm{d}{x}^{k-j+1}}\left(\sum _{\ell =0}^{\infty }{\delta }_{\ell }{x}^{\ell }\right)\hfill \\ \hfill & =\sum _{j=1}^{k+1}\left(\genfrac{}{}{0pt}{}{k}{j-1}\right)\left(\sum _{\ell =j}^{\infty }\frac{\ell !{\gamma }_{\ell }}{(\ell -j)!}{x}^{\ell -j}\right)\times \hfill \\ \hfill & \left(\sum _{\ell =k+1-j}^{\infty }\frac{\ell !{\delta }_{\ell }}{(\ell -k+j-1)!}{x}^{\ell -k+j-1}\right),\hfill \end{array}$$

(61)

where we used Lemma 5 and the identities (58) and (59). From the equality

$$\begin{array}{c}\hfill \frac{{\mathrm{d}}^{k+1}}{\mathrm{d}{x}^{k+1}}\left(\sum _{\ell =0}^{\infty }{\delta }_{\ell }{x}^{\ell }\right)=\frac{{\mathrm{d}}^{k+1}}{\mathrm{d}{x}^{k+1}}\left(exp\left(\sum _{\ell =1}^{\infty }{\gamma }_{\ell }{x}^{\ell }\right)\right)\end{array}$$

and the evaluation of the right-hand side of (60) and (61), we obtain

$$\begin{array}{cccc}\hfill (k+1)!{\delta }_{k+1}& x^0+\left(\dots \right)\hfill & x^1+\left(\dots \right)\hfill & x^2\dots \hfill \\ \hfill =\left(\sum _{j=1}^{k+1}\left(\genfrac{}{}{0pt}{}{k}{j-1}\right)j!{\gamma }_j(k+1-j)!{\delta }_{k+1-j}\right)& x^0+\left(\dots \right)\hfill & x^1+\left(\dots \right)\hfill & x^2\dots \hfill \end{array}$$

Comparing the coefficients for $x^0$ finally yields

$$\begin{array}{cc}\hfill {\delta }_{k+1}& =\frac{1}{(k+1)!}\sum _{j=1}^{k+1}\left(\genfrac{}{}{0pt}{}{k}{j-1}\right)j!{\gamma }_j(k+1-j)!{\delta }_{k+1-j}\hfill \\ \hfill & =\frac{1}{(k+1)!}\sum _{j=1}^{k+1}\frac{k!}{(j-1)!(k+1-j)!}j!{\gamma }_j(k+1-j)!{\delta }_{k+1-j}\hfill \\ \hfill & =\frac{1}{(k+1)}\sum _{j=1}^{k+1}j{\gamma }_j{\delta }_{k+1-j}.\hfill \end{array}$$

This completes the proof of Lemma 4. □

5.2. Finite Sum Approximations

The results in the previous Section 5.1 can be used in the following way for efficient numerical calculations. Consider

$$\begin{array}{c}\hfill {\widehat{f}}_{\mathrm{i}(\xi ;\eta )}(x,n)=\frac{1}{{\varrho }_r\sqrt{\pi }}\left(\prod _{i=1}^{r-1}\frac{{\varrho }_r}{{\varrho }_i}\right)\sum _{k=0}^n{\delta }_k{\mathrm{U}}_k\left(\frac{\left|x-I(\xi ;\eta )\right|}{{\varrho }_r}\right),x\in \mathbb{R}\end{array}$$

(62)

for $n\in {\mathbb{N}}_0$, i.e., the finite sum approximation of the PDF given in (40). To calculate ${\widehat{f}}_{\mathrm{i}(\xi ;\eta )}(x,n)$, first calculate ${\mathrm{U}}_0\left(|x-I(\xi ;\eta )|/{\varrho }_r\right)$ and ${\mathrm{U}}_1\left(|x-I(\xi ;\eta )|/{\varrho }_r\right)$ using (37). Then, use the recurrence formulas (42) and (49) to calculate the remaining summands in (62). The great advantage of this approach is that only two evaluations of the modified Bessel function are required and for the rest of the calculations efficient recursive formulas are employed making the numerical computations efficient.

Similarly, consider

$$\begin{array}{cc}\hfill {\widehat{F}}_{\mathrm{i}(\xi ;\eta )}(x,n)=& \left\{\begin{array}{cc}\hfill \begin{array}{c}\hfill \frac{1}{2}-\widehat{\mathrm{V}}\left(I(\xi ;\eta )-x,n\right)\end{array}& \mathrm{if}x\le I(\xi ;\eta )\hfill \\ \hfill \frac{1}{2}+\widehat{\mathrm{V}}\left(x-I(\xi ;\eta ),n\right)& \mathrm{if}x>I(\xi ;\eta )\hfill \end{array}\right.,\hfill \end{array}$$

(63)

$$\begin{array}{c}\hfill \mathrm{with} \widehat{V}(z,n)=\left(\prod _{i=1}^{r-1}\frac{{\varrho }_r}{{\varrho }_i}\right)\sum _{k=0}^n{\delta }_k{\mathrm{D}}_k\left(z\right),z\ge 0,\end{array}$$

(64)

for $n\in {\mathbb{N}}_0$, i.e., the finite sum approximation of the alternative representation of the CDF of the information density, where $\widehat{\mathrm{V}}(z,n)$ is the finite sum approximation of the function $\mathrm{V}(·)$ given in (41). To calculate ${\widehat{F}}_{\mathrm{i}(\xi ;\eta )}(x,n)$, first calculate ${\mathrm{D}}_0\left(z\right)$, ${\mathrm{U}}_0\left(z/{\varrho }_r\right)$, and ${\mathrm{U}}_1\left(z/{\varrho }_r\right)$ for $z=I(\xi ;\eta )-x$ or $z=x-I(\xi ;\eta )$ using (37) and (38). Then, use the recurrence formulas (42), (46), and (49) to calculate the remaining summands in (64). This approach requires only three evaluations of the modified Bessel and Struve $\mathrm{L}$ function resulting in efficient numerical calculations also for the CDF of the information density.

The following theorem provides suitable bounds to evaluate and control the error related to the introduced finite sum approximations.

Theorem 4

(Bounds of the approximation error for the alternative representation of PDF and CDF). For the finite sum approximations in (62)–(64) of the alternative representation of the PDF and CDF of the information density as given in Proposition 2, we have for $n\in \mathbb{N}$ summands the error bounds

$$\begin{array}{c}\hfill |f_{\mathrm{i}(\xi ;\eta )}\left(x\right)-{\widehat{f}}_{\mathrm{i}(\xi ;\eta )}(x,n)|\le \frac{\Gamma \left(\frac{r-1}{2}+n\right)}{2{\varrho }_r\sqrt{\pi }\Gamma \left(\frac{r}{2}+n\right)}\left(1-\left(\prod _{i=1}^{r-1}\frac{{\varrho }_r}{{\varrho }_i}\right)\sum _{k=0}^n{\delta }_k\right),x\in \mathbb{R}\end{array}$$

(65)

and

$$\begin{array}{c}\hfill |\mathrm{V}\left(z\right)-\widehat{\mathrm{V}}(z,n)|\le \frac{1}{2}\left(1-\left(\prod _{i=1}^{r-1}\frac{{\varrho }_r}{{\varrho }_i}\right)\sum _{k=0}^n{\delta }_k\right),z\ge 0.\end{array}$$

(66)

Proof. 

From the special case where all canonical correlations are equal, we can conclude from the CDF given in Corollary 1 that the function

$$\begin{array}{c}\hfill z\mapsto z\left[{\mathrm{K}}_{\alpha }\left(z\right){\mathrm{L}}_{\alpha -1}\left(z\right)+{\mathrm{K}}_{\alpha -1}\left(z\right){\mathrm{L}}_{\alpha }\left(z\right)\right],z\ge 0,\end{array}$$

(67)

is monotonically increasing for all $\alpha =(j-1)/2$, $j\in \mathbb{N}$ and that further

$$\begin{array}{c}\hfill \underset{z\to \infty }{lim}z\left[{\mathrm{K}}_{\alpha }\left(z\right){\mathrm{L}}_{\alpha -1}\left(z\right)+{\mathrm{K}}_{\alpha -1}\left(z\right){\mathrm{L}}_{\alpha }\left(z\right)\right]=1\end{array}$$

(68)

holds. Using (68), we obtain from (4)

$$\begin{array}{c}\hfill \underset{z\to \infty }{lim}2\mathrm{V}\left(z\right)=\sum _{k_1=0}^{\infty }\sum _{k_2=0}^{\infty }\cdots \sum _{k_{r-1}=0}^{\infty }\left[\prod _{i=1}^{r-1}\frac{{\varrho }_r}{{\varrho }_i}\frac{\left(2k_i\right)!}{{(k_i!)}^2{4}^{k_i}}{\left(1-\frac{{\varrho }_r^2}{{\varrho }_i^2}\right)}^{k_i}\right]\end{array}$$

by exchanging the limit and the summation, which is justified by the monotone convergence theorem. Due to the properties of the CDF, we have ${lim}_{z\to \infty }2\mathrm{V}\left(z\right)=1$, which implies

$$\begin{array}{c}\hfill \left(\prod _{i=1}^{r-1}\frac{{\varrho }_r}{{\varrho }_i}\right)\sum _{k=0}^{\infty }{\delta }_k=\sum _{k_2=0}^{\infty }\cdots \sum _{k_{r-1}=0}^{\infty }\left[\prod _{i=1}^{r-1}\frac{{\varrho }_r}{{\varrho }_i}\frac{\left(2k_i\right)!}{{(k_i!)}^2{4}^{k_i}}{\left(1-\frac{{\varrho }_r^2}{{\varrho }_i^2}\right)}^{k_i}\right]=1,\end{array}$$

(69)

where the first equality follows from the definition of the coefficient ${\delta }_k$ in (39).

We now obtain with (41) and (64)

$$\begin{array}{cc}\hfill |\mathrm{V}\left(z\right)-\widehat{\mathrm{V}}(z,n)|=\left(\prod _{i=1}^{r-1}\frac{{\varrho }_r}{{\varrho }_i}\right)\sum _{k=n+1}^{\infty }{\delta }_k{\mathrm{D}}_k\left(z\right)\le & \left(\prod _{i=1}^{r-1}\frac{{\varrho }_r}{{\varrho }_i}\right)\sum _{k=n+1}^{\infty }\frac{1}{2}{\delta }_k.\hfill \end{array}$$

The inequality follows from the definition of the function ${\mathrm{D}}_k(·)$ in (38), the monotonicity of the function in (67), and from (68). Then, (66) follows from (69).

Similarly, we obtain with (40) and (62)

$$\begin{array}{cc}\hfill |f_{\mathrm{i}(\xi ;\eta )}\left(x\right)-{\widehat{f}}_{\mathrm{i}(\xi ;\eta )}(x,n)|=& \frac{1}{{\varrho }_r\sqrt{\pi }}\left(\prod _{i=1}^{r-1}\frac{{\varrho }_r}{{\varrho }_i}\right)\sum _{k=n+1}^{\infty }{\delta }_k{\mathrm{U}}_k\left(\frac{\left|x-I(\xi ;\eta )\right|}{{\varrho }_r}\right)\hfill \\ \hfill \le & \frac{1}{{\varrho }_r\sqrt{\pi }}\left(\prod _{i=1}^{r-1}\frac{{\varrho }_r}{{\varrho }_i}\right)\sum _{k=n+1}^{\infty }{\delta }_k\frac{\Gamma \left(\frac{r-1}{2}+n\right)}{2\Gamma \left(\frac{r}{2}+n\right)}.\hfill \end{array}$$

The inequality follows from the definition of the function ${\mathrm{U}}_k(·)$, Proposition 1, and the decreasing monotonicity of $\Gamma (\frac{r-1}{2}+k)/\Gamma (\frac{r}{2}+k)$ w. r. t. $k\in {\mathbb{N}}_0$. Then, (65) follows from (69). □

Remark 1.

Note that the bound in (65) can be further simplified using the inequality $\Gamma \left(\alpha \right)/\Gamma \left(\alpha +1/2\right)\le \sqrt{\pi }.$ Further note that the derived error bounds are uniform in the sense that they only depend on the parameters of the given Gaussian distribution and the number of summands considered. As can be seen from (69), the bounds converge to zero as the number of summands jointly increase.

Remark 2

(Relation to Bell polynomials). Interestingly, the coefficient ${\delta }_k$ can be expressed for all $k\in \mathbb{N}$ in the following form

$$\begin{array}{c}\hfill {\delta }_k=\frac{B_k\left({\gamma }_1,2{\gamma }_2,6{\gamma }_3,\dots ,k!{\gamma }_k\right)}{k!},\end{array}$$

where ${\gamma }_j$ is defined in (50), and $B_k$ denotes the complete Bell polynomial of order k [28], Sec. 3.3. Even though this is an interesting connection to the Bell polynomials, which provides an explicit formula of ${\delta }_k$, the recursive formula given in Lemma 4 is more efficient for numerical calculations.

6. Numerical Examples and Illustrations

We illustrate the results of this paper with some examples, which all can be verified with the Python implementation publicly available on GITLAB [26].

Equal canonical correlations. First, we consider the special case of Corollary 1 when all canonical correlations are equal. The PDF and CDF given by (6) and (7) are illustrated in Figure 1 and Figure 2 in centered form, i.e., shifted by $I(\xi ;\eta )$, for $r\in \{1,2,3,4,5\}$ and equal canonical correlations ${\varrho }_i=0.9,i=1,\dots ,r$. In Figure 3 and Figure 4, a fixed number of $r=5$ equal canonical correlations ${\varrho }_i\in \{0.1,0.2,0.5,0.7,0.9\},i=1,\dots ,r$ is considered. When all canonical correlations are equal, then, due to the central limit theorem, the distribution of the information density $\mathrm{i}(\xi ;\eta )$ converges to a Gaussian distribution as $r\to \infty$. Figure 5 and Figure 6 show for $r\in \{5,10,20,40\}$ and equal canonical correlations ${\varrho }_i=0.2,i=1,2,\dots ,r$ the PDF and CDF of the information density together with corresponding Gaussian approximations. The approximations are obtained by considering Gaussian distributions, which have the same variance as the information density $\mathrm{i}(\xi ;\eta )$. Recall that the variance of the information density is given by (11), i.e., by the sum of the squared canonical correlations. The illustrations show that only for a high number of equal canonical correlations the distribution of the information density becomes approximately Gaussian.

Figure 1. PDF $f_{\mathrm{i}(\xi ;\eta )-I(\xi ;\eta )}$ for $r\in \{1,2,3,4,5\}$ equal canonical correlations ${\varrho }_i=0.9$.

Figure 2. CDF $F_{\mathrm{i}(\xi ;\eta )-I(\xi ;\eta )}$ for $r\in \{1,2,3,4,5\}$ equal canonical correlations ${\varrho }_i=0.9$.

Figure 3. PDF $f_{\mathrm{i}(\xi ;\eta )-I(\xi ;\eta )}$ for $r=5$ equal canonical correlations ${\varrho }_i\in \{0.1,0.2,0.5,0.7,0.9\}$.

Figure 4. CDF $F_{\mathrm{i}(\xi ;\eta )-I(\xi ;\eta )}$ for $r=5$ equal canonical correlations ${\varrho }_i\in \{0.1,0.2,0.5,0.7,0.9\}$.

Figure 5. PDF $f_{\mathrm{i}(\xi ;\eta )-I(\xi ;\eta )}$ for $r\in \{5,10,20,40\}$ equal canonical correlations ${\varrho }_i=0.2$ vs. Gaussian approximation.

Figure 6. CDF $F_{\mathrm{i}(\xi ;\eta )-I(\xi ;\eta )}$ for $r\in \{5,10,20,40\}$ equal canonical correlations ${\varrho }_i=0.2$ vs. Gaussian approximation.

Different canonical correlations. To illustrate the case with different canonical correlations, let us consider two more examples.

(i) First, assume that the random vectors $\xi =({\xi }_1,{\xi }_2,\dots ,{\xi }_p)$ and $\eta =({\eta }_1,{\eta }_2,\dots ,{\eta }_q)$ have equal dimensions, i.e., $p=q$, and are related by

$$\begin{array}{c}\hfill ({\eta }_1,{\eta }_2,\dots ,{\eta }_p)=({\xi }_1+{\zeta }_1,{\xi }_2+{\zeta }_2,\dots ,{\xi }_p+{\zeta }_p),\end{array}$$

where $\xi =({\xi }_1,{\xi }_2,\dots ,{\xi }_p)$ and $\zeta =({\zeta }_1,{\zeta }_2,\dots ,{\zeta }_p)$ are zero mean Gaussian random vectors, independent of each other and with covariance matrices

$$\begin{array}{c}\hfill R_{\xi }={\left({\rho }^{|i-j|}\right)}_{i,j=1}^p\mathrm{and}{R}_{\zeta }={\sigma }_z^2{\mathbb{I}}_p,\end{array}$$

for parameters $0<\left|\rho \right|<1$ and ${\sigma }_z^2>0$, where ${\mathbb{I}}_p$ denotes the identity matrix of dimension $p\times p$. The covariance matrix of the Gaussian random vector $({\xi }_1,{\xi }_2,\dots ,{\xi }_p,{\eta }_1,{\eta }_2,\dots ,{\eta }_p)$ is the basis of the canonical correlation analysis and is given by

$$\left(\begin{array}{cc}{R}_{\xi }& R_{\xi \eta }\\ R_{\xi \eta }& R_{\eta }\end{array}\right)=\left(\begin{array}{cc}{R}_{\xi }& R_{\xi }\\ R_{\xi }& R_{\xi }+R_{\zeta }\end{array}\right).$$

The specified situation corresponds to a discrete-time additive noise channel, where a stationary first-order Markov-Gaussian input process is corrupted by a stationary additive white Gaussian noise process. In this setting, a block of p consecutive input and output symbols is considered.

For given parameter values $\rho$ and ${\sigma }_z^2$, the canonical correlations can be calculated numerically with the method described in Section 3. However, the example at hand even allows the derivation of explicit formulas for the canonical correlations. Evaluating the approach in Section 3 analytically yields

$$\begin{array}{c}\hfill {\varrho }_i(\rho ,{\sigma }_z^2)=\sqrt{\frac{{\lambda }_i}{{\lambda }_i+{\sigma }_z^2}}\mathrm{with}{\lambda }_i=\frac{1-{\rho }^2}{1-2\rho cos\left({\theta }_i\right)+{\rho }^2},i=1,2,\dots ,r=p,\end{array}$$

(70)

where ${\theta }_1,{\theta }_2,\dots ,{\theta }_r$ are the zeros of the function

$$\begin{array}{c}\hfill g\left(\theta \right)=sin\left((r+1)\theta \right)-2\rho sin\left(r\theta \right)+{\rho }^{2}sin\left((r-1)\theta \right),\theta \in (0,\pi ).\end{array}$$

In this representation, ${\lambda }_1,{\lambda }_2,\dots ,{\lambda }_r$ denote the eigenvalues of the covariance matrix $R_{\xi }={\left({\rho }^{|i-j|}\right)}_{i,j=1}^p$ derived in [29], Sec. 5.3.

As numerical examples Figure 7 and Figure 8 show, the approximated PDF ${\widehat{f}}_{\mathrm{i}(\xi ;\eta )-I(\xi ;\eta )}(·,n)$ and CDF ${\widehat{F}}_{\mathrm{i}(\xi ;\eta )-I(\xi ;\eta )}(·,n)$ for $p=r\in \{5,10,20,40\}$ and the parameter values $\rho =0.9$ and ${\sigma }_z^2=10$ using the finite sums (62) and (64). The bounds of the approximation error given in Theorem 4 are chosen $<1\times {10}^{-3}$ to obtain a high precision of the plotted curves. The number n of summands required in (62) and (64) to achieve these error bounds for $r\in \{5,10,20,40\}$ is equal to $n\in \{217,333,462,649\}$ for the PDF and $n\in \{282,444,618,847\}$ for the CDF. For this example, the distribution of the information density $\mathrm{i}(\xi ;\eta )$ converges to a Gaussian distribution as $r\to \infty$. However, Figure 7 and Figure 8 show that, even for $r=40$, there is still a significant gap between the exact distribution and the corresponding Gaussian approximation.

Figure 7. Approximated PDF ${\widehat{f}}_{\mathrm{i}(\xi ;\eta )-I(\xi ;\eta )}(·,n)$ for $r\in \{5,10,20,40\}$ canonical correlations ${\varrho }_i(\rho ,{\sigma }_z^2)$ given in (70) for $\rho =0.9$ and ${\sigma }_z^2=10$ (approximation error $<1\times {10}^{-3}$) vs. Gaussian approximation ($r\in \{20,40\}$).

Figure 8. Approximated CDF ${\widehat{F}}_{\mathrm{i}(\xi ;\eta )-I(\xi ;\eta )}(·,n)$ for $r\in \{5,10,20,40\}$ canonical correlations ${\varrho }_i(\rho ,{\sigma }_z^2)$ given in (70) for $\rho =0.9$ and ${\sigma }_z^2=10$ (approximation error $<1\times {10}^{-3}$) vs. Gaussian approximation ($r\in \{20,40\}$).

(ii) As a second example with different canonical correlations, let us consider the sequence $\{{\varrho }_1\left(T\right),{\varrho }_2\left(T\right),$$\dots ,{\varrho }_r\left(T\right)\}$ with

$$\begin{array}{c}\hfill {\varrho }_i\left(T\right)=\sqrt{\frac{T^2}{T^2+\pi {\left(i-\frac{1}{2}\right)}^2}},i=1,2,\dots ,r.\end{array}$$

(71)

These canonical correlations are related to the information density of a continuous-time additive white Gaussian noise channel confined to a finite time interval $[0,T]$ with a Brownian motion as input signal (see, e.g., Huffmann [30], Sec. 8.1 for more details). Figure 9 and Figure 10 show the approximated PDF ${\widehat{f}}_{\mathrm{i}(\xi ;\eta )-I(\xi ;\eta )}(·,n)$ and CDF ${\widehat{F}}_{\mathrm{i}(\xi ;\eta )-I(\xi ;\eta )}(·,n)$ for $r\in \{2,5,10,15\}$ and $T=1$ using the finite sums (62) and (64). The bounds of the approximation error given in Theorem 4 are chosen $<1\times {10}^{-2}$ such that there are no differences visible in the plotted curves by further lowering the approximation error. The number n of summands required in (62) and (64) to achieve these error bounds for $r\in \{2,5,10,15\}$ is equal to $n\in \{15,141,638,1688\}$ for the PDF and $n\in \{20,196,886,2071\}$ for the CDF. Choosing r larger than 15 for the canonical correlations (71) with $T=1$ does not result in visible changes of the PDF and CDF compared to $r=15$. This demonstrates, together with Figure 9 and Figure 10, that a Gaussian approximation is not valid for this example, even if $r\to \infty$.

Figure 9. Approximated PDF ${\widehat{f}}_{\mathrm{i}(\xi ;\eta )-I(\xi ;\eta )}(·,n)$ for $r\in \{2,5,10,15\}$ canonical correlations ${\varrho }_i\left(T\right)$ given in (71) for $T=1$ (approximation error $<1\times {10}^{-2}$) vs. Gaussian approximation ($r=15$).

Figure 10. Approximated CDF ${\widehat{F}}_{\mathrm{i}(\xi ;\eta )-I(\xi ;\eta )}(·,n)$ for $r\in \{2,5,10,15\}$ canonical correlations ${\varrho }_i\left(T\right)$ given in (71) for $T=1$ (approximation error $<1\times {10}^{-2}$) vs. Gaussian approximation ($r=15$).

Indeed, from [8], Th. 9.6.1 and the comment above Eq. (9.6.45) in [8], one can conclude that, whenever the canonical correlations satisfy

$$\begin{array}{c}\hfill \underset{r\to \infty }{lim}\sum _{i=1}^r{\varrho }_i^2<\infty ,\end{array}$$

then the distribution of the information density is not Gaussian.

7. Summary of Contributions

We derived series representations of the PDF and CDF of the information density for arbitrary Gaussian random vectors as well as a general formula for the central moments using canonical correlation analysis. We provided simplified and closed-form expressions for important special cases, in particular when all canonical correlations are equal, and derived recurrence formulas and uniform error bounds for finite sum approximations of the general series representations. These approximations and recurrence formulas are suitable for efficient and arbitrarily accurate numerical calculations, where the approximation error can be easily controlled with the derived error bounds. Moreover, we provided examples showing the (in)validity of approximating the information density with a Gaussian random variable.