A Two-Moment Inequality with Applications to Rényi Entropy and Mutual Information

Abstract

This paper explores some applications of a two-moment inequality for the integral of the rth power of a function, where $0<r<1$. The first contribution is an upper bound on the Rényi entropy of a random vector in terms of the two different moments. When one of the moments is the zeroth moment, these bounds recover previous results based on maximum entropy distributions under a single moment constraint. More generally, evaluation of the bound with two carefully chosen nonzero moments can lead to significant improvements with a modest increase in complexity. The second contribution is a method for upper bounding mutual information in terms of certain integrals with respect to the variance of the conditional density. The bounds have a number of useful properties arising from the connection with variance decompositions.

1. Introduction

The interplay between inequalities and information theory has a rich history, with notable examples including the relationship between the Brunn–Minkowski inequality and the entropy power inequality as well as the matrix determinant inequalities obtained from differential entropy [1]. In this paper, the focus is on a “two-moment” inequality that provides an upper bound on the integral of the rth power of a function. Specifically, if f is a nonnegative function defined on ${\mathbb{R}}^n$ and $p,q,r$ are real numbers satisfying $0<r<1$ and $p<1/r-1<q$, then

$$\begin{array}{c}\hfill {\left(\int f{\left(x\right)}^r\mathrm{d}x\right)}^{\frac{1}{r}}\le C_{n,p,q,r}{\left(\int {\parallel x\parallel }^{np}f\left(x\right)\mathrm{d}x\right)}^{\frac{qr+r-1}{(q-p)r}}{\left(\int {\parallel x\parallel }^{nq}f\left(x\right)\mathrm{d}x\right)}^{\frac{1-r-pr}{(q-p)r}},\end{array}$$

(1)

where the best possible constant $C_{n,p,q,r}$ is given exactly; see Propositions 2 and 3 ahead. The one-dimensional version of this inequality is a special case of the classical Carlson–Levin inequality [2,3,4], and the multidimensional version is a special case of a result presented by Barza et al. [5]. The particular formulation of the inequality used in this paper was derived independently in [6], where the proof follows from a direct application of Hölder’s inequality and Jensen’s inequality.

In the context of information theory and statistics, a useful property of the two-moment inequality is that it provides a bound on a nonlinear functional, namely the r-quasi-norm ${\parallel ·\parallel }_r$, in terms of integrals that are linear in f. Consequently, this inequality is well suited to settings where f is a mixture of simple functions whose moments can be evaluated. We note that this reliance on moments to bound a nonlinear functional is closely related to bounds obtained from variational characterizations such as the Donsker–Varadhan representation of Kullback divergence [7] and its generalizations to Rényi divergence [8,9].

The first application considered in this paper concerns the relationship between the entropy of a probability measure and its moments. This relationship is fundamental to the principle of maximum entropy, which originated in statistical physics and has since been applied to statistical inference problems [10]. It also plays a prominent role in information theory and estimation theory where the fact that the Gaussian distribution maximizes differential entropy under second moment constraints ([11], [Theorem 8.6.5]) plays a prominent role. Moment–entropy inequalities for Rényi entropy were studied in a series of works by Lutwak et al. [12,13,14], as well as related works by Costa et al. [15,16] and Johonson and Vignat [17], in which it is shown that, under a single moment constraint, Rényi entropy is maximized by a family of generalized Gaussian distributions. The connection between these moment–entropy inequalities and the Carlson–Levin inequality was noted recently by Nguyen [18].

In this direction, one of the contributions of this paper is a new family of moment–entropy inequalities. This family of inequalities follows from applying Inequality (1) in the setting where f is a probability density function, and thus there is a one-to-one correspondence between the integral of the rth power and the Rényi entropy of order r. In the special case where one of the moments is the zeroth moment, this approach recovers the moment–entropy inequalities given in previous work. More generally, the additional flexibility provided by considering two different moments can lead to stronger results. For example, in Proposition 6, it is shown that if f is the standard Gaussian density function defined on ${\mathbb{R}}^n$, then the difference between the Rényi entropy and the upper bound given by the two-moment inequality (equivalently, the ratio between the left- and right-hand sides of (1)) is bounded uniformly with respect to n under the following specification of the moments:

$$\begin{array}{cc}\hfill p_n& =\frac{1-r}{r}-\frac{1}{r}\sqrt{\frac{2(1-r)}{n+1}},q_n=\frac{1-r}{r}+\frac{1}{r}\sqrt{\frac{2(1-r)}{n+1}}.\hfill \end{array}$$

(2)

Conversely, if one of the moments is restricted to be equal to zero, as is the case in the usual moment–entropy inequalities, then the difference between the Rényi entropy and the upper bound diverges with n.

The second application considered in this paper is the problem of bounding mutual information. In conjunction with Fano’s inequality and its extensions, bounds on mutual information play a prominent role in establishing minimax rates of statistical estimation [19] as well as the information-theoretic limits of detection in high-dimensional settings [20]. In many cases, one of the technical challenges is to provide conditions under which the dependence between the observations and an underlying signal or model parameters converges to zero in the limit of high dimension.

This paper introduces a new method for bounding mutual information, which can be described as follows. Let $P_{X,Y}$ be a probability measure on $\mathcal{X}\times \mathcal{Y}$ such that $P_{Y\mid X=x}$ and $P_Y$ have densities $f(y\mid x)$ and $f\left(y\right)$ with respect to the Lebesgue measure on ${\mathbb{R}}^n$. We begin by showing that the mutual information between X and Y satisfies the upper bound

$$\begin{array}{c}\hfill I(X;Y)\le \int \sqrt{Var\left(f\right(y\mid X\left)\right)}\mathrm{d}y,\end{array}$$

(3)

where $Var\left(p(y\mid X)\right)=\int {\left(f(y\mid x)-f\left(y\right)\right)}^2\mathrm{d}{P}_X\left(x\right)$ is the variance of $f(y\mid X)$; see Proposition 8 ahead. In view of (3), an application of the two-moment Inequality (1) with $r=1/2$ leads to an upper bound with respect to the moments of the variance of the density:

$$\begin{array}{c}\hfill \int {\parallel y\parallel }^{ns}Var\left(f(y\mid X)\right)\mathrm{d}y\end{array}$$

(4)

where this expression is evaluated at $s\in \{p,q\}$ with $p<1<q$. A useful property of this bound is that the integrated variance is quadratic in $P_X$, and thus Expression (4) can be evaluated by swapping the integration over y and with the expectation of over two independent copies of X. For example, when $P_{X,Y}$ is a Gaussian scale mixture, this approach provides closed-form upper bounds in terms of the moments of the Gaussian density. An early version of this technique is used to prove Gaussian approximations for random projections [21] arising in the analysis of a random linear estimation problem appearing in wireless communications and compressed sensing [22,23].

2. Moment Inequalities

Let $L^p\left(S\right)$ be the space of Lebesgue measurable functions from S to $\mathbb{R}$ whose pth power is absolutely integrable, and for $p\ne 0$, define

$${\parallel f\parallel }_p:={\left({\int }_S{\left|f\left(x\right)\right|}^p\mathrm{d}x\right)}^{1/p}.$$

Recall that ${\parallel ·\parallel }_p$ is a norm for $p\ge 1$ but only a quasi-norm for $0<p<1$ because it does not satisfy the triangle inequality. The sth moment of f is defined as

$${\mathcal{M}}_s\left(f\right):={\int }_S{\parallel x\parallel }^s\left|f\left(x\right)\right|\mathrm{d}x,$$

where $\parallel ·\parallel$ denotes the standard Euclidean norm on vectors.

The two-moment Inequality (1) can be derived straightforwardly using the following argument. For $r\in (0,1)$, the mapping $f\mapsto {\parallel f\parallel }_r$ is concave on the subset of nonnegative functions and admits the variational representation

$$\begin{array}{c}\hfill {\parallel f\parallel }_r=inf\left\{\frac{{\parallel fg\parallel }_1}{{\parallel g\parallel }_{r^{*}}}:g\in L^{r^{*}}\right\},\end{array}$$

(5)

where $r^{*}=r/(r-1)\in (-\infty ,0)$ is the Hölder conjugate of r. Consequently, each $g\in L^{r^{*}}$ leads to an upper bound on ${\parallel f\parallel }_r$. For example, if f has bounded support S, choosing g to be the indicator function of S leads to the basic inequality ${\parallel f\parallel }_r\le {(Vol\left(S\right))}^{(1-r)/r}{\parallel f\parallel }_1$. The upper bound on ${\parallel f\parallel }_r$ given in Inequality (1) can be obtained by restricting the minimum in Expression (5) to the parametric class of functions of the form $g\left(x\right)={\nu }_1{\parallel x\parallel }^{np}+{\nu }_2{\parallel x\parallel }^{nq}$ with ${\nu }_1,{\nu }_2>0$ and then optimizing over the parameters $({\nu }_1,{\nu }_2)$. Here, the constraints on $p,q$ are necessary and sufficient to ensure that $g\in L^{r^{*}}\left({\mathbb{R}}^n\right)$.

In the following sections, we provide a more detailed derivation, starting with the problem of maximizing ${\parallel f\parallel }_r$ under multiple moment constraints and then specializing to the case of two moments. For a detailed account of the history of the Carlson type inequalities as well as some further extensions, see [4].

2.1. Multiple Moments

Consider the following optimization problem:

$$\begin{array}{cc}\hfill \mathrm{maximize}& {\parallel f\parallel }_r\hfill \\ \hfill \mathrm{subject}\mathrm{to}& f\left(x\right)\ge 0\hfill & \hfill & \mathrm{for}\mathrm{all}x\in S\hfill \\ & {\mathcal{M}}_{s_i}\left(f\right)\le m_i\hfill & & \mathrm{for}1\le i\le k.\hfill \end{array}$$

For $r\in (0,1)$, this is a convex optimization problem because ${\parallel ·\parallel }_r^r$ is concave and the moment constraints are linear. By standard theory in convex optimization (e.g., [24]), it can be shown that if the problem is feasible and the maximum is finite, then the maximizer has the form

$$f^{*}\left(x\right)={\left(\sum _{i=1}^k{\nu }_i^{*}{\parallel x\parallel }^{s_i}\right)}^{\frac{1}{r-1}},\mathrm{for}\mathrm{all}x\in S.$$

The parameters ${\nu }_1^{*},\cdots ,{\nu }_k^{*}$ are nonnegative and the ith moment constraint holds with equality for all i such that ${\nu }_i^{*}$ is strictly positive—that is, ${\nu }_i^{*}>0\Rightarrow {\mu }_{s_i}\left(f^{*}\right)=m_i$. Consequently, the maximum can be expressed in terms of a linear combination of the moments:

$$\parallel f^{*}{\parallel }_r^r=\parallel {\left(f^{*}\right)}^r{\parallel }_1={\parallel f^{*}{\left(f^{*}\right)}^{r-1}\parallel }_1=\sum _{i=1}^k{\nu }_i^{*}{m}_i.$$

For the purposes of this paper, it is useful to consider a relative inequality in terms of the moments of the function itself. Given a number $0<r<1$ and vectors $s\in {\mathbb{R}}^k$ and $\nu \in {\mathbb{R}}_{+}^k$, the function $c_r(\nu ,s)$ is defined according to

$$c_r(\nu ,s)={\left({\int }_0^{\infty }{\left(\sum _{i=1}^k{\nu }_i{x}^{s_i}\right)}^{-\frac{r}{1-r}}\mathrm{d}x\right)}^{\frac{1-r}{r}},$$

if the integral exists. Otherwise, $c_r(\nu ,s)$ is defined to be positive infinity. It can be verified that $c_r(\nu ,s)$ is finite provided that there exists $i,j$ such that ${\nu }_i$ and ${\nu }_j$ are strictly positive and $s_i<(1-r)/r<s_j$.

The following result can be viewed as a consequence of the constrained optimization problem described above. We provide a different and very simple proof that depends only on Hölder’s inequality.

Proposition 1.

Let f be a nonnegative Lebesgue measurable function defined on the positive reals ${\mathbb{R}}_{+}$. For any number $0<r<1$ and vectors $s\in {\mathbb{R}}^k$ and $\nu \in {\mathbb{R}}_{+}^k$, we have

$${\parallel f\parallel }_r\le c_r(\nu ,s)\sum _{i=1}^k{\nu }_i{\mathcal{M}}_{s_i}\left(f\right).$$

Proof. 

Let $g\left(x\right)={\sum }_{i=1}^k{\nu }_i{x}^{s_i}$. Then, we have

$$\begin{array}{cc}\hfill {\parallel f\parallel }_r^r& =\parallel g^{-r}{\left(fg\right)}^r{\parallel }_1\le \parallel g^{-r}{\parallel }_{\frac{1}{1-r}}{\parallel \left(gf\right)}^r{\parallel }_{\frac{1}{r}}=\parallel g^{\frac{-r}{1-r}}{\parallel }_1^{1-r}{\parallel gf\parallel }_1^r={\left(c_r(\nu ,s)\sum _{i=1}^k{\nu }_i{\mathcal{M}}_{s_i}\left(f\right)\right)}^r,\hfill \end{array}$$

where the second step is Hölder’s inequality with conjugate exponents $1/(1-r)$ and $1/r$. □

2.2. Two Moments

For $a,b>0$, the beta function $B(a,b)$ and gamma function $\mathsf{\Gamma }\left(a\right)$ are given by

$$\begin{array}{cc}\hfill B(a,b)& ={\int }_0^1{t}^{a-1}{(1-t)}^{b-1}\mathrm{d}t\hfill \\ \hfill \mathsf{\Gamma }\left(a\right)& ={\int }_0^{\infty }{t}^{a-1}{e}^{-t}dt,\hfill \end{array}$$

and satisfy the relation $B(a,b)=\mathsf{\Gamma }\left(a\right)\mathsf{\Gamma }\left(b\right)/\mathsf{\Gamma }(a+b)$, $a,b>0$. To lighten the notation, we define the normalized beta function

$$\begin{array}{c}\hfill \tilde{B}(a,b)=B(a,b){(a+b)}^{a+b}{a}^{-a}{b}^{-b}.\end{array}$$

(6)

Properties of these functions are provided in Appendix A.

The next result follows from Proposition 1 for the case of two moments.

Proposition 2.

Let f be a nonnegative Lebesgue measurable function defined on $[0,\infty )$. For any numbers $p,q,r$ with $0<r<1$ and $p<1/r-1<q$,

$${\parallel f\parallel }_r\le {\left[{\psi }_r(p,q)\right]}^{\frac{1-r}{r}}{\left[{\mathcal{M}}_p\left(f\right)\right]}^{\lambda }{\left[{\mathcal{M}}_q\left(f\right)\right]}^{1-\lambda },$$

where $\lambda =(q+1-1/r)/(q-p)$ and

$$\begin{array}{c}\hfill {\psi }_r(p,q)=\frac{1}{(q-p)}\tilde{B}\left(\frac{r\lambda }{1-r},\frac{r(1-\lambda )}{1-r}\right),\end{array}$$

(7)

where $\tilde{B}(·,·)$ is defined in Equation (6).

Proof. 

Letting $s=(p,q)$ and $\nu =({\gamma }^{1-\lambda },{\gamma }^{-\lambda })$ with $\lambda >0$, we have

$${\left[c_r(\nu ,s)\right]}^{\frac{r}{1-r}}={\int }_0^{\infty }{\left({\gamma }^{1-\lambda }{x}^p+{\gamma }^{-\lambda }{x}^q\right)}^{-\frac{r}{1-r}}\mathrm{d}x.$$

Making the change of variable $x\mapsto {\left(\gamma u\right)}^{\frac{1}{q-p}}$ leads to

$${\left[c_r(\nu ,s)\right]}^{\frac{r}{1-r}}=\frac{1}{q-p}{\int }_0^{\infty }\frac{u^{b-1}}{{(1+u)}^{a+b}}\mathrm{d}u=\frac{B\left(a,b\right)}{q-p},$$

where $a=\frac{r}{1-r}\lambda$ and $b=\frac{r}{1-r}(1-\lambda )$ and the second step follows from recognizing the integral representation of the beta function given in Equation (A3). Therefore, by Proposition 1, the inequality

$${\parallel f\parallel }_r\le {\left(\frac{B\left(a,b\right)}{q-p}\right)}^{\frac{1-r}{r}}\left({\gamma }^{1-\lambda }{\mathcal{M}}_p\left(f\right)+{\gamma }^{-\lambda }{\mathcal{M}}_q\left(f\right)\right),$$

holds for all $\gamma >0$. Evaluating this inequality with

$$\gamma =\frac{\lambda {\mathcal{M}}_q\left(f\right)}{(1-\lambda ){\mathcal{M}}_p\left(f\right)},$$

leads to the stated result. □

The special case $r=1/2$ admits the simplified expression

$${\psi }_{1/2}(p,q)=\frac{\pi {\lambda }^{-\lambda }{(1-\lambda )}^{-(1-\lambda )}}{(q-p)sin\left(\pi \lambda \right)},$$

(8)

where we have used Euler’s reflection formula for the beta function ([25], [Theorem 1.2.1]).

Next, we consider an extension of Proposition 2 for functions defined on ${\mathbb{R}}^n$. Given any measurable subset S of ${\mathbb{R}}^n$, we define

$$\omega \left(S\right)=Vol(B^n\cap \mathrm{cone}\left(S\right)),$$

(9)

where $B^n=\{u\in {\mathbb{R}}^n:\parallel u\parallel \le 1\}$ is the n-dimensional Euclidean ball of radius one and

$$cone\left(S\right)=\{x\in {\mathbb{R}}^n:tx\in S\mathrm{for}\mathrm{some}t>0\}.$$

The function $\omega \left(S\right)$ is proportional to the surface measure of the projection of S on the Euclidean sphere and satisfies

$$\omega \left(S\right)\le \omega \left({\mathbb{R}}^n\right)=\frac{{\pi }^{\frac{n}{2}}}{\mathsf{\Gamma }(\frac{n}{2}+1)},$$

(10)

for all $S\subseteq {\mathbb{R}}^n$. Note that $\omega \left({\mathbb{R}}_{+}\right)=1$ and $\omega \left(\mathbb{R}\right)=2$.

Proposition 3.

Let f be a nonnegative Lebesgue measurable function defined on a subset S of ${\mathbb{R}}^n$. For any numbers $p,q,r$ with $0<r<1$ and $p<1/r-1<q$,

$${\parallel f\parallel }_r\le {\left[\omega \left(S\right){\psi }_r(p,q)\right]}^{\frac{1-r}{r}}{\left[{\mathcal{M}}_{np}\left(f\right)\right]}^{\lambda }{\left[{\mathcal{M}}_{nq}\left(f\right)\right]}^{1-\lambda },$$

where $\lambda =(q+1-1/r)/(q-p)$ and ${\psi }_r(p,q)$ is given by Equation (7).

Proof. 

Let f be extended to ${\mathbb{R}}^n$ using the rule $f\left(x\right)=0$ for all x outside of S and let $g:{\mathbb{R}}_{+}\to {\mathbb{R}}_{+}$ be defined according to

$$g\left(y\right)=\frac{1}{n}{\int }_{{\mathbb{S}}^{n-1}}f\left(y^{1/n}u\right)\mathrm{d}\sigma \left(u\right),$$

where ${\mathbb{S}}^{n-1}=\{u\in {\mathbb{R}}^n:\parallel u\parallel =1\}$ is the Euclidean sphere of radius one and $\sigma \left(u\right)$ is the surface measure of the sphere. In the following, we will show that

$$\begin{array}{cc}\hfill {\parallel f\parallel }_r& \le {\left(\omega \left(S\right)\right)}^{\frac{1-r}{r}}{\parallel g\parallel }_r\hfill \end{array}$$

(11)

$$\begin{array}{cc}\hfill {\mathcal{M}}_{ns}\left(f\right)& ={\mathcal{M}}_s\left(g\right).\hfill \end{array}$$

(12)

Then, the stated inequality then follows from applying Proposition 2 to the function g.

To prove Inequality (11), we begin with a transformation into polar coordinates:

$${\parallel f\parallel }_r^r={\int }_0^{\infty }{\int }_{{\mathbb{S}}^{n-1}}{\left|f\left(tu\right)\right|}^r{t}^{n-1}\mathrm{d}\sigma \left(u\right)\mathrm{d}t.$$

(13)

Letting $1_{cone\left(S\right)}\left(x\right)$ denote the indicator function of the set $cone\left(S\right)$, the integral over the sphere can be bounded using:

$$\begin{array}{cc}\hfill {\int }_{{\mathbb{S}}^{n-1}}{\left|f\left(tu\right)\right|}^r\mathrm{d}\sigma \left(u\right)& ={\int }_{{\mathbb{S}}^{n-1}}{1}_{cone\left(S\right)}\left(u\right){\left|f\left(tu\right)\right|}^r\mathrm{d}\sigma \left(u\right)\hfill \\ \hfill & \stackrel{\left(a\right)}{\le }{\left({\int }_{{\mathbb{S}}^{n-1}}{1}_{cone\left(S\right)}\left(u\right)\mathrm{d}\sigma \left(u\right)\right)}^{1-r}{\left({\int }_{{\mathbb{S}}^{n-1}}\left|f\left(tu\right)\right|\mathrm{d}\sigma \left(u\right)\right)}^r\hfill \\ \hfill & \stackrel{\left(b\right)}{=}n{\left(\omega \left(S\right)\right)}^{1-r}{g}^r\left(t^n\right),\hfill \end{array}$$

(14)

where: (a) follows from Hölder’s inequality with conjugate exponents $\frac{1}{1-r}$ and $\frac{1}{r}$, and (b) follows from the definition of g and the fact that

$$\begin{array}{cc}\hfill \omega \left(S\right)& ={\int }_0^1{\int }_{{\mathbb{S}}^{n-1}}{1}_{cone\left(S\right)}\left(u\right)t^{n-1}\mathrm{d}\sigma \left(u\right)\mathrm{d}t\hfill \\ \hfill & =\frac{1}{n}{\int }_{{\mathbb{S}}^{n-1}}{1}_{cone\left(S\right)}\left(u\right)\mathrm{d}\sigma \left(u\right).\hfill \end{array}$$

Plugging Inequality (14) back into Equation (13) and then making the change of variable $t\to y^{\frac{1}{n}}$ yields

$${\parallel f\parallel }_r^r\le n{\left(\omega \left(S\right)\right)}^{1-r}{\int }_0^{\infty }{g}^r\left(t^n\right)t^{n-1}\mathrm{d}t={\left(\omega \left(S\right)\right)}^{1-r}{\parallel g\parallel }_r^r.$$

The proof of Equation (12) follows along similar lines. We have

$$\begin{array}{cc}\hfill {\mathcal{M}}_{ns}\left(f\right)& \stackrel{\left(a\right)}{=}{\int }_0^{\infty }{\int }_{{\mathbb{S}}^{n-1}}{t}^{ns}f\left(tu\right)t^{n-1}\mathrm{d}\sigma \left(u\right)\mathrm{d}t\hfill \\ \hfill & \stackrel{\left(b\right)}{=}\frac{1}{n}{\int }_0^{\infty }{\int }_{{\mathbb{S}}^{n-1}}{y}^{s}f\left(y^{\frac{1}{n}}u\right)\mathrm{d}\sigma \left(u\right)\mathrm{d}y\hfill \\ \hfill & ={\mathcal{M}}_s\left(g\right)\hfill \end{array}$$

where (a) follows from a transformation into polar coordinates and (b) follows from the change of variable $t\mapsto y^{\frac{1}{n}}$.

Having established Inequality (11) and Equation (12), an application of Proposition 2 completes the proof. □

3. Rényi Entropy Bounds

Let X be a random vector that has a density $f\left(x\right)$ with respect to the Lebesgue measure on ${\mathbb{R}}^n$. The differential Rényi entropy of order $r\in (0,1)\cup (1,\infty )$ is defined according to [11]:

$$h_r\left(X\right)=\frac{1}{1-r}log\left({\int }_{{\mathbb{R}}^n}{f}^r\left(x\right)\mathrm{d}x\right).$$

Throughout this paper, it is assumed that the logarithm is defined with respect to the natural base and entropy is measured in nats. The Rényi entropy is continuous and nonincreasing in r. If the support set $S=\{x\in {\mathbb{R}}^n:f\left(x\right)>0\}$ has finite measure, then the limit as r converges to zero is given by $h_0\left(X\right)=logVol\left(S\right)$. If the support does not have finite measure, then $h_r\left(X\right)$ increases to infinity as r decreases to zero. The case $r=1$ is given by the Shannon differential entropy:

$$h_1\left(X\right)=-{\int }_{S}f\left(x\right)logf\left(x\right)\mathrm{d}x.$$

Given a random variable X that is not identical to zero and numbers $p,q,r$ with $0<r<1$ and $p<1/r-1<q$, we define the function

$$L_r(X;p,q)=\frac{r\lambda }{1-r}log\mathbb{E}\left[{\left|X\right|}^p\right]+\frac{r(1-\lambda )}{1-r}log\mathbb{E}\left[{\left|X\right|}^q\right],$$

where $\lambda =(q+1-1/r)/(q-p)$.

The next result, which follows directly from Proposition 3, provides an upper bound on the Rényi entropy.

Proposition 4.

Let X be a random vector with a density on ${\mathbb{R}}^n$. For any numbers $p,q,r$ with $0<r<1$ and $p<1/r-1<q$, the Rényi entropy satisfies

$$h_r\left(X\right)\le log\omega \left(S\right)+log{\psi }_r(p,q)+L_r{(\parallel X\parallel }^n;p,q),$$

(15)

where $\omega \left(S\right)$ is defined in Equation (9) and ${\psi }_r(p,q)$ is defined in Equation (7).

Proof. 

This result follows immediately from Proposition 3 and the definition of Rényi entropy. □

The relationship between Proposition 4 and previous results depends on whether the moment p is equal to zero:

• One-moment inequalities: If $p=0$, then there exists a distribution such that Inequality (15) holds with equality. This is because the zero-moment constraint ensures that the function that maximizes the Rényi entropy integrates to one. In this case, Proposition 4 is equivalent to previous results that focused on distributions that maximize Rényi entropy subject to a single moment constraint [12,13,15]. With some abuse of terminology, we refer to these bounds as one-moment inequalities. (A more accurate name would be two-moment inequalities under the constraint that one of the moments is the zeroth moment.)
• Two-moment inequalities: If $p\ne 0$, then the right-hand side of Inequality (15) corresponds to the Rényi entropy of a nonnegative function that might not integrate to one. Nevertheless, the expression provides an upper bound on the Rényi entropy for any density with the same moments. We refer to the bounds obtained using a general pair $(p,q)$ as two-moment inequalities.

The contribution of two-moment inequalities is that they lead to tighter bounds. To quantify the tightness, we define ${\mathsf{\Delta }}_r(X;p,q)$ to be the gap between the right-hand side and left-hand side of Inequality (15) corresponding to the pair $(p,q)$—that is,

$$\begin{array}{cc}\hfill {\mathsf{\Delta }}_r(X;p,q)& =log\omega \left(S\right)+log{\psi }_r(p,q)+L_r{(\parallel X\parallel }^n;p,q)-h_r\left(X\right).\hfill \end{array}$$

The gaps corresponding to the optimal two-moment and one-moment inequalities are defined according to

$$\begin{array}{cc}\hfill {\mathsf{\Delta }}_r\left(X\right)& =\underset{p,q}{inf}{\mathsf{\Delta }}_r(X;p,q)\hfill \\ \hfill {\tilde{\mathsf{\Delta }}}_r\left(X\right)& =\underset{q}{inf}{\mathsf{\Delta }}_r(X;0,q).\hfill \end{array}$$

3.1. Some Consequences of These Bounds

By Lyapunov’s inequality, the mapping $s\mapsto \frac{1}{s}log\mathbb{E}\left[{\left|X\right|}^s\right]$ is nondecreasing on $[0,\infty )$, and thus

$$L_r(X;p,q)\le L_r(X;0,q)=\frac{1}{q}log\mathbb{E}\left[{\left|X\right|}^q\right],p\ge 0.$$

(16)

In other words, the case $p=0$ provides an upper bound on $L_r(X;p,q)$ for nonnegative p. Alternatively, we also have the lower bound

$$L_r(X;p,q)\ge \frac{r}{1-r}log\mathbb{E}\left[{\left|X\right|}^{\frac{1-r}{r}}\right],$$

(17)

which follows from the convexity of $log\mathbb{E}\left[{\left|X\right|}^s\right]$.

A useful property of $L_r(X;p,q)$ is that it is additive with respect to the product of independent random variables. Specifically, if X and Y are independent, then

$$\begin{array}{c}\hfill L_r(XY;p,q)=L_r(X;p,q)+L_r(Y;p,q).\end{array}$$

(18)

One consequence is that multiplication by a bounded random variable cannot increase the Rényi entropy by an amount that exceeds the gap of the two-moment inequality with nonnegative moments.

Proposition 5.

Let Y be a random vector on ${\mathbb{R}}^n$ with finite Rényi entropy of order $0<r<1$, and let X be an independent random variable that satisfies $0<X\le t$. Then,

$$h_r\left(XY\right)\le h_r\left(tY\right)+{\mathsf{\Delta }}_r(Y;p,q),$$

for all $0<p<1/r-1<q$.

Proof. 

Let $Z=XY$ and let $S_Z$ and $S_Y$ denote the support sets of Z and Y, respectively. The assumption that X is nonnegative means that $cone\left(S_Z\right)=cone\left(S_Y\right)$. We have

$$\begin{array}{cc}\hfill h_r\left(Z\right)& \stackrel{\left(a\right)}{\le }log\omega \left(S_Z\right)+log{\psi }_r(p,q)+L_r{(\parallel Z\parallel }^n;p,q)\hfill \\ \hfill & \stackrel{\left(b\right)}{=}{h}_r\left(Y\right)+L_r{\left(\right|X|}^n;p;q)+{\mathsf{\Delta }}_r(Y;p,q)\hfill \\ \hfill & \stackrel{\left(c\right)}{\le }{h}_r\left(Y\right)+nlogt+{\mathsf{\Delta }}_r(Y;p,q),\hfill \end{array}$$

where (a) follows from Proposition 4, (b) follows from Equation (18) and the definition of ${\mathsf{\Delta }}_r(Y;p,q)$, and (c) follows from Inequality (16) and the assumption $\left|X\right|\le t$. Finally, recalling that $h_r\left(tY\right)=h_r\left(Y\right)+nlogt$ completes the proof. □

3.2. Example with Log-Normal Distribution

If $W\sim \mathcal{N}(\mu ,{\sigma }^2)$, then the random variable $X=exp\left(W\right)$ has a log-normal distribution with parameters $(\mu ,{\sigma }^2)$. The Rényi entropy is given by

$$h_r\left(X\right)=\mu +\frac{1}{2}\left(\frac{1-r}{r}\right){\sigma }^2+\frac{1}{2}log\left(2\pi r^{\frac{1}{r-1}}{\sigma }^2\right),$$

and the logarithm of the sth moment is given by

$$log\mathbb{E}\left[{\left|X\right|}^s\right]=\mu s+\frac{1}{2}{\sigma }^2{s}^2.$$

With a bit of work, it can be shown that the gap of the optimal two-moment inequality does not depend on the parameters $(\mu ,{\sigma }^2)$ and is given by

$$\begin{array}{cc}\hfill {\mathsf{\Delta }}_r\left(X\right)& =log\left(\tilde{B}\left(\frac{r}{2(1-r)},\frac{r}{2(1-r)}\right)\sqrt{\frac{r}{4(1-r)}}\right)+\frac{1}{2}-\frac{1}{2}log\left(2\pi r^{\frac{1}{r-1}}\right).\hfill \end{array}$$

(19)

The details of this derivation are given in Appendix B.1. Meanwhile, the gap of the optimal one-moment inequality is given by

$$\begin{array}{cc}\hfill {\tilde{\mathsf{\Delta }}}_r\left(X\right)& =\underset{q}{inf}\left[log\left(\tilde{B}\left(\frac{r}{1-r}-\frac{1}{q},\frac{1}{q}\right)\frac{1}{q}\right)+\frac{1}{2}q{\sigma }^2\right]-\frac{1}{2}\left(\frac{1-r}{r}\right){\sigma }^2-\frac{1}{2}log\left(2\pi r^{\frac{1}{r-1}}{\sigma }^2\right).\hfill \end{array}$$

(20)

The functions ${\mathsf{\Delta }}_r\left(X\right)$ and ${\tilde{\mathsf{\Delta }}}_r\left(X\right)$ are illustrated in Figure 1 as a function of r for various ${\sigma }^2$. The function ${\mathsf{\Delta }}_r\left(X\right)$ is bounded uniformly with respect to r and converges to zero as r increases to one. The tightness of the two-moment inequality in this regime follows from the fact that the log-normal distribution maximizes Shannon entropy subject to a constraint on $\mathbb{E}\left[logX\right]$. By contrast, the function ${\tilde{\mathsf{\Delta }}}_r\left(X\right)$ varies with the parameter ${\sigma }^2$. For any fixed $r\in (0,1)$, it can be shown that ${\tilde{\mathsf{\Delta }}}_r\left(X\right)$ increases to infinity if ${\sigma }^2$ converges to zero or infinity.

3.3. Example with Multivariate Gaussian Distribution

Next, we consider the case where $Y\sim \mathcal{N}(0,I_n)$ is an n-dimensional Gaussian vector with mean zero and identity covariance. The Rényi entropy is given by

$$h_r\left(Y\right)=\frac{n}{2}log\left(2\pi r^{\frac{1}{r-1}}\right),$$

and the sth moment of the magnitude $\parallel Y\parallel$ is given by

$$\mathbb{E}\left[{\parallel Y\parallel }^s\right]=\frac{2^{\frac{s}{2}}\mathsf{\Gamma }\left(\frac{n+s}{2}\right)}{\mathsf{\Gamma }\left(\frac{n}{2}\right)}.$$

The next result shows that as the dimension n increases, the gap of the optimal two-moment inequality converges to the gap for the log-normal distribution. Moreover, for each $r\in (0,1)$, the following choice of moments is optimal in the large-n limit:

$$\begin{array}{cc}\hfill p_n& =\frac{1-r}{r}-\frac{1}{r}\sqrt{\frac{2(1-r)}{n+1}},q_n=\frac{1-r}{r}+\frac{1}{r}\sqrt{\frac{2(1-r)}{n+1}}.\hfill \end{array}$$

(21)

The proof is given in Appendix B.3.

Proposition 6.

If $Y\sim \mathcal{N}(0,I_n)$, then, for each $r\in (0,1)$,

$$\underset{n\to \infty }{lim}{\mathsf{\Delta }}_r\left(Y\right)=\underset{n\to \infty }{lim}{\mathsf{\Delta }}_r(Y;p_n,q_n)={\mathsf{\Delta }}_r\left(X\right),$$

where X has a log-normal distribution and $(p_n,q_n)$ are given by (21).

Figure 2 provides a comparison of ${\mathsf{\Delta }}_r\left(Y\right)$, ${\mathsf{\Delta }}_r(Y;p_n,q_n)$, and ${\tilde{\mathsf{\Delta }}}_r\left(Y\right)$ as a function of n for $r=0.1$. Here, we see that both ${\mathsf{\Delta }}_r\left(Y\right)$ and ${\mathsf{\Delta }}_r(Y;p_n,q_n)$ converge rapidly to the asymptotic limit given by the gap of the log-normal distribution. By contrast, the gap of the optimal one-moment inequality ${\tilde{\mathsf{\Delta }}}_r\left(Y\right)$ increases without bound.

3.4. Inequalities for Differential Entropy

Proposition 4 can also be used to recover some known inequalities for differential entropy by considering the limiting behavior as r converges to one. For example, it is well known that the differential entropy of an n-dimensional random vector X with finite second moment satisfies

$$\begin{array}{c}\hfill h\left(X\right)\le \frac{1}{2}log\left(2\pi e\mathbb{E}\left[\frac{1}{n}{\parallel X\parallel }^2\right]\right),\end{array}$$

(22)

with equality if and only if the entries of X are i.i.d. zero-mean Gaussian. A generalization of this result in terms of an arbitrary positive moment is given by

$$\begin{array}{cc}\hfill h\left(X\right)& \le log\frac{\mathsf{\Gamma }\left(\frac{n}{s}+1\right)}{\mathsf{\Gamma }\left(\frac{n}{2}+1\right)}+\frac{n}{2}log\pi +\frac{n}{s}log\left(es\mathbb{E}\left[\frac{1}{n}{\parallel X\parallel }^s\right]\right),\hfill \end{array}$$

(23)

for all $s>0$. Note that Inequality (22) corresponds to the case $s=2$.

Inequality (23) can be proved as an immediate consequence of Proposition 4 and the fact that $h_r\left(X\right)$ is nonincreasing in r. Using properties of the beta function given in Appendix A, it is straightforward to verify that

$$\underset{r\to 1}{lim}{\psi }_r(0,q)={\left(eq\right)}^{\frac{1}{q}}\mathsf{\Gamma }\left(\frac{1}{q}+1\right),\mathrm{for}\mathrm{all}q>0.$$

Combining this result with Proposition 4 and Inequality (16) leads to

$$\begin{array}{cc}\hfill h\left(X\right)& \le log\omega \left(S\right)+log\mathsf{\Gamma }\left(\frac{1}{q}+1\right)+\frac{1}{q}log\left(eq\mathbb{E}\left[{\parallel X\parallel }^{nq}\right]\right).\hfill \end{array}$$

Using Inequality (10) and making the substitution $s=nq$ leads to Inequality (23).

Another example follows from the fact that the log-normal distribution maximizes the differential entropy of a positive random variable X subject to constraints on the mean and variance of $log\left(X\right)$, and hence

$$\begin{array}{cc}\hfill h\left(X\right)& \le \mathbb{E}\left[log\left(X\right)\right]+\frac{1}{2}log\left(2\pi eVar(log(X\left)\right)\right),\hfill \end{array}$$

(24)

with equality if and only if X is log-normal. In Appendix B.4, it is shown how this inequality can be proved using our two-moment inequalities by studying the behavior as both p and q converge to zero as r increases to one.

4. Bounds on Mutual Information

4.1. Relative Entropy and Chi-Squared Divergence

Let P and Q be distributions defined on a common probability space that have densities p and q with respect to a dominating measure $\mu$. The relative entropy (or Kullback–Leibler divergence) is defined according to

$$D\left(P\parallel Q\right)=\int plog\left(\frac{p}{q}\right)\mathrm{d}\mu ,$$

and the chi-squared divergence is defined as

$${\chi }^2(P\parallel Q)=\int \frac{{\left(p-q\right)}^2}{q}\mathrm{d}\mu .$$

Both of these divergences can be seen as special cases of the general class of f-divergence measures and there exists a rich literature on comparisons between different divergences [8,26,27,28,29,30,31,32]. The chi-squared divergence can also be viewed as the squared $L_2$ distance between $p/\sqrt{q}$ and $\sqrt{q}$. The chi-square can also be interpreted as the first non-zero term in the power series expansion of the relative entropy ([26], [Lemma 4]). More generally, the chi-squared divergence provides an upper bound on the relative entropy via

$$\begin{array}{cc}\hfill D\left(P\parallel Q\right)& \le log(1+{\chi }^2(P\parallel Q)).\hfill \end{array}$$

(25)

The proof of this inequality follows straightforwardly from Jensen’s inequality and the concavity of the logarithm; see [27,31,32] for further refinements.

Given a random pair $(X,Y)$, the mutual information between X and Y is defined according to

$$I(X;Y)=D\left(P_{X,Y}\parallel P_X{P}_Y\right).$$

From Inequality (25), we see that the mutual information can always be upper bounded using

$$\begin{array}{c}\hfill I(X;Y)\le log(1+{\chi }^2(P_{X,Y}\parallel P_X{P}_Y)).\end{array}$$

(26)

The next section provides bounds on the mutual information that can improve upon this inequality.

4.2. Mutual Information and Variance of Conditional Density

Let $(X,Y)$ be a random pair such that the conditional distribution of Y given X has a density $f_{Y|X}\left(y\right|x)$ with respect to the Lebesgue measure on ${\mathbb{R}}^n$. Note that the marginal density of Y is given by $f_Y\left(y\right)=\mathbb{E}\left[f_{Y|X}\left(y\right|X)\right]$. To simplify notation, we will write $f\left(y\right|x)$ and $f\left(y\right)$ where the subscripts are implicit. The support set of Y is denoted by $S_Y$.

The measure of the dependence between X and Y that is used in our bounds can be understood in terms of the variance of the conditional density. For each y, the conditional density $f\left(y\right|X)$ evaluated with a random realization of X is a random variable. The variance of this random variable is given by

$$\begin{array}{c}\hfill Var\left(f\left(y\right|X)\right)=\mathbb{E}\left[{\left(f\left(y\right|X)-f(y)\right)}^2\right],\end{array}$$

(27)

where we have used the fact that the marginal density $f\left(y\right)$ is the expectation of $f\left(y\right|X)$. The sth moment of the variance of the conditional density is defined according to

$$\begin{array}{c}\hfill V_s\left(Y\right|X)={\int }_{S_Y}{\parallel y\parallel }^{s}Var\left(f\left(y\right|X)\right)\mathrm{d}y.\end{array}$$

(28)

The variance moment $V_s\left(Y\right|X)$ is nonnegative and equal to zero if and only if X and Y are independent.

The function $\kappa \left(t\right)$ is defined according to

$$\begin{array}{c}\hfill \kappa \left(t\right)=\underset{u\in (0,\infty )}{sup}\frac{log(1+u)}{u^t},t\in (0,1].\end{array}$$

(29)

The proof of the following result is given in Appendix C. The behavior of $\kappa \left(t\right)$ is illustrated in Figure 3.

Proposition 7.

The function $\kappa \left(t\right)$ defined in Equation (29) can be expressed as

$$\kappa \left(t\right)=\frac{log(1+u)}{u^t},t\in (0,1]$$

where

$$u=exp\left(W\left(-\frac{1}{t}exp\left(-\frac{1}{t}\right)\right)+\frac{1}{t}\right)-1,$$

and $W(·)$ denotes Lambert’s W- function, i.e., $W\left(z\right)$ is the unique solution to the equation $z=wexp\left(w\right)$ on the interval $[-1,\infty )$. Furthermore, the function $g\left(t\right)=t\kappa \left(t\right)$ is strictly increasing on $(0,1]$ with ${lim}_{t\to 0}g\left(t\right)=1/e$ and $g\left(1\right)=1$, and thus

$$\frac{1}{et}\le \kappa \left(t\right)\le \frac{1}{t},t\in (0,1],$$

where the lower bound $1/\left(et\right)$ is tight for small values of $t\in (0,1)$ and the upper bound $1/t$ is tight for values of t close to 1.

We are now ready to give the main results of this section, which are bounds on the mutual information. We begin with a general upper bound in terms of the variance of the conditional density.

Proposition 8.

For any $0<t\le 1$, the mutual information satisfies

$$I(X;Y)\le \kappa \left(t\right){\int }_{S_Y}{\left[f\left(y\right)\right]}^{1-2t}{\left[Var\left(f\right(y\mid X\left)\right)\right]}^t\mathrm{d}y.$$

Proof. 

We use the following series of inequalities:

$$\begin{array}{cc}\hfill I(X;Y)& \stackrel{\left(a\right)}{=}\int f\left(y\right)D\left(P_{X|Y=y}\parallel P_X\right)\mathrm{d}y\hfill \\ \hfill & \stackrel{\left(b\right)}{\le }\int f\left(y\right)log\left(1+{\chi }^2(P_{X|Y=y}\parallel P_X)\right)\mathrm{d}y\hfill \\ \hfill & \stackrel{\left(c\right)}{=}\int f\left(y\right)log\left(1+\frac{Var\left(f\right(y\mid X\left)\right)}{f^2\left(y\right)}\right)\mathrm{d}y\hfill \\ \hfill & \stackrel{\left(d\right)}{\le }\kappa \left(t\right)\int f\left(y\right){\left(\frac{Var\left(f\right(y\mid X\left)\right)}{f^2\left(y\right)}\right)}^t\mathrm{d}y,\hfill \end{array}$$

where (a) follows from the definition of mutual information, (b) follows from Inequality (25), and (c) follows from Bayes’ rule, which allows us to write the chi-square in terms of the variance of the conditional density:

$${\chi }^2(P_{X|Y=y}\parallel P_X)=\mathbb{E}\left[{\left(\frac{f\left(y\right|X)}{f\left(y\right)}-1\right)}^2\right]=\frac{Var\left(f\right(y\left|X\right))}{f^2\left(y\right)}.$$

Inequality (d) follows from the nonnegativity of the variance and the definition of $\kappa \left(t\right)$. □

Evaluating Proposition 8 with $t=1$ recovers the well-known inequality $I(X;Y)\le {\chi }^2(P_{X,Y}\parallel P_X{P}_Y)$. The next two results follow from the cases $0<t<\frac{1}{2}$ and $t=\frac{1}{2}$, respectively.

Proposition 9.

For any $0<r<1$, the mutual information satisfies

$$I(X;Y)\le \kappa \left(t\right){\left(e^{h_r\left(Y\right)}{V}_0\left(Y\right|X)\right)}^t,$$

where $t=(1-r)/(2-r)$.

Proof. 

Starting with Proposition 8 and applying Hölder’s inequality with conjugate exponents $1/(1-t)$ and $1/t$ leads to

$$\begin{array}{cc}\hfill I(X;Y)& \le \kappa \left(t\right){\left(\int f^r\left(y\right)\mathrm{d}y\right)}^{1-t}{\left(\int Var\left(f\right(y\mid X\left)\right)\mathrm{d}y\right)}^t=\kappa \left(t\right)e^{t{h}_r\left(Y\right)}{V}_0^t\left(Y\right|X),\hfill \end{array}$$

where we have used the fact that $r=(1-2t)/(1-t)$. □

Proposition 10.

For any $p<1<q$, the mutual information satisfies

$$I(X;Y)\le C\left(\lambda \right)\sqrt{\frac{\omega \left(S_Y\right)V_{np}^{\lambda }\left(Y\right|X)V_{nq}^{1-\lambda }\left(Y\right|X)}{(q-p)}},$$

where $\lambda =(q-1)/(q-p)$ and

$$C\left(\lambda \right)=\kappa \left(\frac{1}{2}\right)\sqrt{\frac{\pi {\lambda }^{-\lambda }{(1-\lambda )}^{-(1-\lambda )}}{sin\left(\pi \lambda \right)}},$$

with $\kappa \left(\frac{1}{2}\right)=0.80477\cdots$.

Proof. 

Evaluating Proposition 8 with $t=1/2$ gives

$$I(X;Y)\le \kappa \left(\frac{1}{2}\right){\int }_{S_Y}\sqrt{Var\left(f\right(y\mid X\left)\right)}\mathrm{d}y.$$

Evaluating Proposition 3 with $r=\frac{1}{2}$ leads to

$$\begin{array}{cc}\hfill {\left({\int }_{S_Y}\sqrt{Var\left(f\right(y\mid X\left)\right)}\mathrm{d}y\right)}^2& \le \omega \left(S_Y\right){\psi }_{1/2}(p,q)V_{np}^{\lambda }\left(Y\right|X)V_{nq}^{1-\lambda }\left(Y\right|X).\hfill \end{array}$$

Combining these inequalities with the expression for ${\psi }_{1/2}(p,q)$ given in Equation (8) completes the proof. □

The contribution of Propositions 9 and 10 is that they provide bounds on the mutual information in terms of quantities that can be easy to characterize. One application of these bounds is to establish conditions under which the mutual information corresponding to a sequence of random pairs $(X_k,Y_k)$ converges to zero. In this case, Proposition 9 provides a sufficient condition in terms of the Rényi entropy of $Y_n$ and the function $V_0\left(Y_n\right|X_n)$, while Proposition 10 provides a sufficient condition in terms of $V_s\left(Y_n\right|X_n)$ evaluated with two difference values of s. These conditions are summarized in the following result.

Proposition 11.

Let $(X_k,Y_k)$ be a sequence of random pairs such that the conditional distribution of $Y_k$ given $X_k$ has a density on ${\mathbb{R}}^n$. The following are sufficient conditions under which the mutual information of $I(X_k;Y_k)$ converges to zero as k increases to infinity:

• There exists $0<r<1$ such that

  $$\begin{array}{cc}\hfill \underset{k\to \infty }{lim}{e}^{h_r\left(Y_k\right)}{V}_0\left(Y_k\right|X_k)& =0.\hfill \end{array}$$
• There exists $p<1<q$ such that

  $$\begin{array}{c}\hfill \underset{k\to \infty }{lim}{V}_{np}^{q-1}\left(Y_k\right|X_k)V_{nq}^{1-p}\left(Y_k\right|X_k)=0.\end{array}$$

4.3. Properties of the Bounds

The variance moment $V_s\left(Y\right|X)$ has a number of interesting properties. The variance of the conditional density can be expressed in terms of an expectation with respect to two independent random variables $X_1$ and $X_2$ with the same distribution as X via the decomposition:

$$Var\left(f\left(y\right|X)\right)=\mathbb{E}\left[f\left(y\right|X)f\left(y\right|X)-f\left(y\right|X_1)f\left(y\right|X_2)\right].$$

Consequently, by swapping the order of the integration and expectation, we obtain

$$V_s\left(Y\right|X)=\mathbb{E}\left[K_s(X,X)-K_s(X_1,X_2)\right],$$

(30)

where

$$K_s(x_1,x_2)=\int {\parallel y\parallel }^{s}f\left(y\right|x_1)f\left(y\right|x_2)\mathrm{d}y.$$

The function $K_s(x_1,x_2)$ is a positive definite kernel that does not depend on the distribution of X. For $s=0$, this kernel has been studied previously in the machine learning literature [33], where it is referred to as the expected likelihood kernel.

The variance of the conditional density also satisfies a data processing inequality. Suppose that $U\to X\to Y$ forms a Markov chain. Then, the square of the conditional density of Y given U can be expressed as

$$f_{Y|U}^2\left(y\right|u)=\mathbb{E}\left[f_{Y|X}\left(y\right|X_1^{\prime })f_{Y|X}\left(y\right|X_2^{\prime })\mid U=u\right],$$

where $(U,X_1^{\prime },X_2^{\prime })\sim P_U{P}_{X_1|U}{P}_{X_2|U}$. Combining this expression with Equation (30) yields

$$\begin{array}{c}\hfill V_s\left(Y\right|U)=\mathbb{E}\left[K_s(X_1^{\prime },X_2^{\prime })-K_s(X_1,X_2)\right],\end{array}$$

(31)

where we recall that $(X_1,X_2)$ are independent copies of X

Finally, it is easy to verify that the function $V_s\left(Y\right)$ satisfies

$$V_s\left(aY\right|X)={\left|a\right|}^{s-n}{V}_s\left(Y\right|X),\mathrm{for}\mathrm{all}a\ne 0.$$

Using this scaling relationship, we see that the sufficient conditions in Proposition 11 are invariant to scaling of Y.

4.4. Example with Additive Gaussian Noise

We now provide a specific example of our bounds on the mutual information. Let $X\in {\mathbb{R}}^n$ be a random vector with distribution $P_X$ and let Y be the output of a Gaussian noise channel

$$Y=X+W,$$

(32)

where $W\sim \mathcal{N}(0,I_n)$ is independent of X. If $\parallel X\parallel$ has finite second moment, then the mutual information satisfies

$$I(X;Y)\le \frac{n}{2}log\left(1+\frac{1}{n}\mathbb{E}\left[{\parallel X\parallel }^2\right]\right),$$

(33)

where equality is attained if and only if X has zero-mean isotropic Gaussian distribution. This inequality follows straightforwardly from the fact that the Gaussian distribution maximizes differential entropy subject to a second moment constraint [11]. One of the limitations of this bound is that it can be loose when the second moment is dominated by events that have small probability. In fact, it is easy to construct examples for which $\parallel X\parallel$ does not have a finite second moment, and yet $I(X;Y)$ is arbitrarily close to zero.

Our results provide bounds on $I(X;Y)$ that are less sensitive to the effects of rare events. Let ${\varphi }_n\left(x\right)={\left(2\pi \right)}^{-n/2}{exp(-\parallel x\parallel }^2/2)$ denote the density of the standard Gaussian distribution on ${\mathbb{R}}^n$. The product of the conditional densities can be factored according to

$$\begin{array}{cc}\hfill f(y\mid x_1)f(y\mid x_2)& ={\varphi }_{2n}\left(\left[\begin{array}{c}y-x_1\\ y-x_2\end{array}\right]\right)={\varphi }_{2n}\left(\left[\begin{array}{c}\sqrt{2}y-(x_1+x_2)/\sqrt{2}\\ (x_1-x_2)/\sqrt{2}\end{array}\right]\right)\hfill \\ \hfill & ={\varphi }_n\left(\sqrt{2}y-\frac{x_1+x_2}{\sqrt{2}}\right){\varphi }_n\left(\frac{x_1-x_2}{\sqrt{2}}\right),\hfill \end{array}$$

where the second step follows because ${\varphi }_{2n}(·)$ is invariant to orthogonal transformations. Integrating with respect to y leads to

$$\begin{array}{cc}\hfill K_s(x_1,x_2)& =2^{-\frac{n+s}{2}}\mathbb{E}\left[{∥W+\frac{x_1+x_2}{\sqrt{2}}∥}^s\right]{\varphi }_n\left(\frac{x_1-x_2}{\sqrt{2}}\right),\hfill \end{array}$$

where we recall that $W\sim \mathcal{N}(0,I_n)$. For the case $s=0$, we see that $K_0(x_1,x_2)$ is a Gaussian kernel, thus

$$\begin{array}{c}\hfill V_0\left(Y\right|X)={\left(4\pi \right)}^{-\frac{n}{2}}\left[1-\mathbb{E}\left[e^{-\frac{1}{4}{∥X_1-X_2∥}^2}\right]\right].\end{array}$$

(34)

A useful property of $V_0\left(Y\right|X)$ is that the conditions under which it converges to zero are weaker than the conditions needed for other measures of dependence. Observe that the expectation in Equation (34) is bounded uniformly with respect to $(X_1,X_2)$. In particular, for every $ϵ>0$ and $x\in \mathbb{R}$, we have

$$\begin{array}{cc}\hfill 1-\mathbb{E}\left[e^{-\frac{1}{4}{\left(X_1-X_2\right)}^2}\right]& \le {ϵ}^2+2\mathbb{P}\left[|X-x|\ge ϵ\right],\hfill \end{array}$$

where we have used the inequality $1-e^{-x}\le x$ and the fact that $\mathbb{P}\left[|X_1-X_2|\ge 2ϵ\right]\le 2\mathbb{P}\left[|X-x|\ge ϵ\right]$. Consequently, $V_0\left(Y\right|X)$ converges to zero whenever X converges to a constant value x in probability.

To study some further properties of these bounds, we now focus on the case where X is a Gaussian scalar mixture generated according to

$$X=A\sqrt{U},A\sim \mathcal{N}(0,1),U\ge 0,$$

(35)

with A and U independent. In this case, the expectations with respect to the kernel $K_s(x_1,x_2)$ can be computed explicitly, leading to

$$\begin{array}{c}\hfill V_s\left(Y\right|X)=\frac{\mathsf{\Gamma }\left(\frac{1+s}{2}\right)}{2\pi }\mathbb{E}\left[{\left(1+2U\right)}^{\frac{s}{2}}-\frac{{(1+U_1)}^{\frac{s}{2}}{(1+U_2)}^{\frac{s}{2}}}{{(1+\frac{1}{2}(U_1+U_2))}^{\frac{s+1}{2}}}\right],\end{array}$$

(36)

where $(U_1,U_2)$ are independent copies of U. It can be shown that this expression depends primarily on the magnitude of U. This is not surprising given that X converges to a constant if and only if U converges to zero.

Our results can also be used to bound the mutual information $I(U;Y)$ by noting that $U\to X\to Y$ forms a Markov chain, and taking advantage of the characterization provided in Equation (31). Letting $X_1^{\prime }=A_1\sqrt{U}$ and $X_2^{\prime }=A_2\sqrt{U}$ with $(A_1,A_2,U)$ be mutually independent leads to

$$\begin{array}{c}\hfill V_s\left(Y\right|U)=\frac{\mathsf{\Gamma }\left(\frac{1+s}{2}\right)}{2\pi }\mathbb{E}\left[{\left(1+U\right)}^{\frac{s-1}{2}}-\frac{{(1+U_1)}^{\frac{s}{2}}{(1+U_2)}^{\frac{s}{2}}}{{(1+\frac{1}{2}(U_1+U_2))}^{\frac{s+1}{2}}}\right],\end{array}$$

(37)

In this case, $V_s\left(Y\right|U)$ is a measure of the variation in U. To study its behavior, we consider the simple upper bound

$$\begin{array}{c}\hfill V_s\left(Y\right|U)\le \frac{\mathsf{\Gamma }\left(\frac{1+s}{2}\right)}{2\pi }\mathbb{P}\left[U_1\ne U_2\right]\mathbb{E}\left[{\left(1+U\right)}^{\frac{s-1}{2}}\right],\end{array}$$

(38)

which follows from noting that the term inside the expectation in Equation (37) is zero on the event $U_1=U_2$. This bound shows that if $s\le 1$ then $V_s\left(Y\right|U)$ is bounded uniformly with respect to distributions on U, and if $s>1$, then $V_s\left(Y\right|U)$ is bounded in terms of the $\left(\frac{s-1}{2}\right)$th moment of U.

In conjunction with Propositions 9 and 10, the function $V_s\left(Y\right|U)$ provides bounds on the mutual information $I(U;Y)$ that can be expressed in terms of simple expectations involving two independent copies of U. Figure 4 provides an illustration of the upper bound in Proposition 10 for the case where U is a discrete random variable supported on two points, and X and Y are generated according to Equations (32) and (35). This example shows that there exist sequences of distributions for which our upper bounds on the mutual information converge to zero while the chi-squared divergence between $P_{XY}$ and $P_X{P}_Y$ is bounded away from zero.

5. Conclusions

This paper provides bounds on Rényi entropy and mutual information that are based on a relatively simple two-moment inequality. Extensions to inequalities with more moments are worth exploring. Another potential application is to provide a refined characterization of the “all-or-nothing” behavior seen in a sparse linear regression problem [34,35], where the current methods of analysis depend on a complicated conditional second moment method.