A Lower Bound on the Differential Entropy of Log-Concave Random Vectors with Applications

Abstract

We derive a lower bound on the differential entropy of a log-concave random variable X in terms of the p-th absolute moment of X. The new bound leads to a reverse entropy power inequality with an explicit constant, and to new bounds on the rate-distortion function and the channel capacity. Specifically, we study the rate-distortion function for log-concave sources and distortion measure $d(x,\widehat{x})={|x-\widehat{x}|}^r$, with $r\ge 1$, and we establish that the difference between the rate-distortion function and the Shannon lower bound is at most $log\left(\sqrt{\pi e}\right)\approx 1.5$ bits, independently of r and the target distortion d. For mean-square error distortion, the difference is at most $log\left(\sqrt{\frac{\pi e}{2}}\right)\approx 1$ bit, regardless of d. We also provide bounds on the capacity of memoryless additive noise channels when the noise is log-concave. We show that the difference between the capacity of such channels and the capacity of the Gaussian channel with the same noise power is at most $log\left(\sqrt{\frac{\pi e}{2}}\right)\approx 1$ bit. Our results generalize to the case of a random vector X with possibly dependent coordinates. Our proof technique leverages tools from convex geometry.

1. Introduction

It is well known that the differential entropy among all zero-mean random variables with the same second moment is maximized by the Gaussian distribution:

$$h\left(X\right)\le log\left(\sqrt{2\pi e\mathbb{E}\left[\right|X{|}^2]}\right).$$

(1)

More generally, the differential entropy under p-th moment constraint is upper bounded as (see e.g., [1] (Appendix 2)), for $p>0$,

$$\begin{array}{c}\hfill h\left(X\right)\le log\left({\alpha }_p{\parallel X\parallel }_p\right),\end{array}$$

(2)

where

$${\alpha }_p\triangleq 2e^{\frac{1}{p}}\mathrm{\Gamma }\left(1+\frac{1}{p}\right)p^{\frac{1}{p}},{\parallel X\parallel }_p\triangleq {\left(\mathbb{E}\left[\right|X{|}^p]\right)}^{\frac{1}{p}}.$$

(3)

Here, $\mathrm{\Gamma }$ denotes the Gamma function. Of course, if $p=2$, ${\alpha }_p=\sqrt{2\pi e}$, and Equation (2) reduces to Equation (1). A natural question to ask is whether a matching lower bound on $h\left(X\right)$ can be found in terms of p-norm of X, ${\parallel X\parallel }_p$. The quest is meaningless without additional assumptions on the density of X, as $h\left(X\right)=-\infty$ is possible even if ${\parallel X\parallel }_p$ is finite. In this paper, we show that if the density of X, $f_X\left(x\right)$, is log-concave (that is, $log{f}_X\left(x\right)$ is concave), then $h\left(X\right)$ stays within a constant from the upper bound in Equation (2) (see Theorem 3 in Section 2 below):

$$\begin{array}{c}\hfill h\left(X\right)\ge log\frac{{2\parallel X-\mathbb{E}\left[X\right]\parallel }_p}{\mathrm{\Gamma }{(p+1)}^{\frac{1}{p}}},\end{array}$$

(4)

where $p\ge 1$. Moreover, the bound (4) tightens for $p=2$, where we have

$$\begin{array}{c}\hfill h\left(X\right)\ge \frac{1}{2}log\left(4\mathrm{Var}\left[X\right]\right).\end{array}$$

(5)

The bound (4) actually holds for $p>-1$ if, in addition to being log-concave, X is symmetric (that is, $f_X\left(x\right)=f_X(-x)$), (see Theorem 1 in Section 2 below).

The class of log-concave distributions is rich and contains important distributions in probability, statistics and analysis. Gaussian distribution, Laplace distribution, uniform distribution on a convex set, chi distribution are all log-concave. The class of log-concave random vectors has good behavior under natural probabilistic operations: namely, a famous result of Prékopa [2] states that sums of independent log-concave random vectors, as well as marginals of log-concave random vectors, are log-concave. Furthermore, log-concave distributions have moments of all orders.

Together with the classical bound in Equation (2), the bound in (4) tells us that entropy and moments of log-concave random variables are comparable.

Using a different proof technique, Bobkov and Madiman [3] recently showed that the differential entropy of a log-concave X satisfies

$$h\left(X\right)\ge \frac{1}{2}log\left(\frac{1}{2}\mathrm{Var}\left[X\right]\right).$$

(6)

Our results in (4) and (5) tighten (6), in addition to providing a comparison with other moments.

Furthermore, this paper generalizes the lower bound on differential entropy in (4) to random vectors. If the random vector $X=(X_1,\dots ,X_n)$ consists of independent random variables, then the differential entropy of X is equal to the sum of differential entropies of the component random variables, and one can trivially apply (4) component-wise to obtain a lower bound on $h\left(X\right)$. In this paper, we show that, even for nonindependent components, as long as the density of the random vector X is log-concave and satisfies a symmetry condition, its differential entropy is bounded from below in terms of covariance matrix of X (see Theorem 4 in Section 2 below). As noted in [4], such a generalization is related to the famous hyperplane conjecture in convex geometry. We also extend our results to a more general class of random variables, namely, the class of $\gamma$-concave random variables, with $\gamma <0$.

The bound (4) on the differential entropy allows us to derive reverse entropy power inequalities with explicit constants. The fundamental entropy power inequality of Shannon [5] and Stam [6] states that for all independent continuous random vectors X and Y in ${\mathbb{R}}^n$,

$$N(X+Y)\ge N\left(X\right)+N\left(Y\right),$$

(7)

where

$$N\left(X\right)=e^{\frac{2}{n}h\left(X\right)}$$

(8)

denotes the entropy power of X. It is of interest to characterize distributions for which a reverse form of (7) holds. In this direction, it was shown by Bobkov and Madiman [7] that, given any continuous log-concave random vectors X and Y in ${\mathbb{R}}^n$, there exist affine volume-preserving maps $u_1,u_2$ such that a reverse entropy power inequality holds for $u_1\left(X\right)$ and $u_2\left(Y\right)$:

$$N(u_1\left(X\right)+u_2\left(Y\right))\le c(N\left(u_1\left(X\right)\right)+N\left(u_2\left(Y\right)\right))=c(N\left(X\right)+N\left(Y\right)),$$

(9)

for some universal constant $c\ge 1$ (independent of the dimension).

In applications, it is important to know the precise value of the constant c that appears in (9). It was shown by Cover and Zhang [8] that, if X and Y are identically distributed (possibly dependent) log-concave random variables, then

$$N(X+Y)\le 4N\left(X\right).$$

(10)

Inequality (10) easily extends to random vectors (see [9]). A similar bound for the difference of i.i.d. log-concave random vectors was obtained in [10], and reads as

$$N(X-Y)\le e^{2}N\left(X\right).$$

(11)

Recently, a new form of reverse entropy power inequality was investigated in [11], and a general reverse entropy power-type inequality was developed in [12]. For further details, we refer to the survey paper [13]. In Section 5, we provide explicit constants for non-identically distributed and uncorrelated log-concave random vectors (possibly dependent). In particular, we prove that as long as log-concave random variables X and Y are uncorrelated,

$$N(X+Y)\le \frac{\pi e}{2}(N\left(X\right)+N\left(Y\right)).$$

(12)

A generalization of (12) to arbitrary dimension is stated in Theorem 8 in Section 2 below.

The bound (4) on the differential entropy is essential in the study of the difference between the rate-distortion function and the Shannon lower bound that we describe next. Given a nonnegative number d, the rate-distortion function ${\mathbb{R}}_X\left(d\right)$ under r-th moment distortion measure is given by

$$\begin{array}{c}\hfill {\mathbb{R}}_X\left(d\right)=\underset{\begin{array}{c}{P}_{\widehat{X}|X}:\\ \mathbb{E}\left[\right|X-\widehat{X}{|}^r]\le d\end{array}}{inf}I(X;\widehat{X}),\end{array}$$

(13)

where the infimum is over all transition probability kernels $\mathbb{R}\mapsto \mathbb{R}$ satisfying the moment constraint. The celebrated Shannon lower bound [14] states that the rate-distortion function is lower bounded by

$${\mathbb{R}}_X\left(d\right)\ge {\mathbb{R}}_X\left(d\right)\triangleq h\left(X\right)-log\left({\alpha }_r{d}^{\frac{1}{r}}\right),$$

(14)

where ${\alpha }_r$ is defined in (3). For mean-square distortion ($r=2$), (14) simplifies to

$$\begin{array}{c}\hfill {\mathbb{R}}_X\left(d\right)\ge h\left(X\right)-log\sqrt{2\pi ed}.\end{array}$$

(15)

The Shannon lower bound states that the rate-distortion function is lower bounded by the difference between the differential entropy of the source and the term that increases with target distortion d, explicitly linking the storage requirements for X to the information content of X (measured by $h\left(X\right)$) and the desired reproduction distortion d. As shown in [15,16,17] under progressively less stringent assumptions (Koch [17] showed that (16) holds as long as $H(\lfloor X\rfloor )<\infty$), the Shannon lower bound is tight in the limit of low distortion,

$$\begin{array}{c}\hfill 0\le {\mathbb{R}}_X\left(d\right)-{\mathbb{R}}_X\left(d\right)\underset{d\to 0}{\to }0.\end{array}$$

(16)

The speed of convergence in (16) and its finite blocklength refinement were recently explored in [18]. Due to its simplicity and tightness in the high resolution/low distortion limit, the Shannon lower bound can serve as a proxy for the rate-distortion function ${\mathbb{R}}_X\left(d\right)$, which rarely has an explicit representation. Furthermore, the tightness of the Shannon lower bound at low d is linked to the optimality of simple lattice quantizers [18], an insight which has evident practical significance. Gish and Pierce [19] showed that, for mean-square error distortion, the difference between the entropy rate of a scalar quantizer, $H_1$, and the rate-distortion function ${\mathbb{R}}_X\left(d\right)$ converges to $\frac{1}{2}log\frac{2\pi e}{12}\approx 0.254$ bit/sample in the limit $d\downarrow 0$. Ziv [20] proved that $\widetilde{H_1}-{\mathbb{R}}_X\left(d\right)$ is bounded by $\frac{1}{2}log\frac{2\pi e}{6}\approx 0.754$ bit/sample, universally in d, where $\widetilde{H_1}$ is the entropy rate of a dithered scalar quantizer.

In this paper, we show that the gap between ${\mathbb{R}}_X\left(d\right)$ and ${\mathbb{R}}_X\left(d\right)$ is bounded universally in d, provided that the source density is log-concave: for mean-square error distortion ($r=2$ in (13)), we have

$${\mathbb{R}}_X\left(d\right)-\underset{-}{\mathbb{R}}{}_X\left(d\right)\le log\sqrt{\frac{\pi e}{2}}\approx 1.05\mathrm{bits}.$$

(17)

Besides leading to the reverse entropy power inequality and the reverse Shannon lower bound, the new bounds on the differential entropy allow us to bound the capacity of additive noise memoryless channels, provided that the noise follows a log-concave distribution.

The capacity of a channel that adds a memoryless noise Z is given by (see e.g., [21] (Chapter 9)),

$${\mathbb{C}}_Z\left(P\right)=\underset{X:\mathbb{E}\left[\right|X{|}^2]\le P}{sup}I(X;X+Z),$$

(18)

where P is the power allotted for the transmission. As a consequence of the entropy power inequality (7) (or more elementary as a consequence of the worst additive noise lemma, see [22,23]), it holds that

$${\mathbb{C}}_Z\left(P\right)\ge \underset{-}{\mathbb{C}}{}_Z\left(P\right)=\frac{1}{2}log\left(1+\frac{P}{\mathrm{Var}\left[Z\right]}\right),$$

(19)

for arbitrary noise Z, where $\underset{-}{\mathbb{C}}{}_Z\left(P\right)$ denotes the capacity of the additive white Gaussian noise channel with noise variance $\mathrm{Var}\left[Z\right]$. This fact is well known (see e.g., [21] (Chapter 9)), and is referred to as the saddle-point condition.

In this paper, we show that, whenever the noise Z is log-concave, the difference between the capacity ${\mathbb{C}}_Z\left(P\right)$ and the capacity of a Gaussian channel with the same noise power satisfies

$${\mathbb{C}}_Z\left(P\right)-{\mathbb{C}}_Z\left(P\right)\le log\sqrt{\frac{\pi e}{2}}\approx 1.05\mathrm{bits}.$$

(20)

Let us mention a similar result by Zamir and Erez [24], who showed that the capacity of an arbitrary memoryless additive noise channel is well approximated by the mutual information between the Gaussian input and the output of the channel:

$${\mathbb{C}}_Z\left(P\right)-I(X^{*};X^{*}+Z)\le \frac{1}{2}\mathrm{bits},$$

(21)

where $X^{*}$ is a Gaussian input satisfying the power constraint. The bounds (20) and (21) are not directly comparable.

The rest of the paper is organized as follows. Section 2 presents and discusses our main results: the lower bounds on differential entropy in Theorems 1, 3 and 4, the reverse entropy power inequalities with explicit constants in Theorems 7 and 8, the upper bounds on ${\mathbb{R}}_X\left(d\right)-{\underset{\_}{\mathbb{R}}}_X\left(d\right)$ in Theorems 9 and 10, and the bounds on the capacity of memoryless additive channels in Theorems 12 and 13. The convex geometry tools served to prove the bounds on differential entropy and the bounds in Theorems 1, 3 and 4 are presented in Section 3. In Section 4, we extend our results to the class of $\gamma$-concave random variables. The reverse entropy power inequalities in Theorems 7 and 8 are proven in Section 5. The bounds on the rate-distortion function in Theorems 9 and 10 are proven in Section 6. The bounds on the channel capacity in Theorems 12 and 13 are proven in Section 7.

2. Main Results

2.1. Lower Bounds on the Differential Entropy

A function $f:{\mathbb{R}}^n\to [0,+\infty )$ is log-concave if $logf:{\mathbb{R}}^n\to [-\infty ,\infty )$ is a concave function. Equivalently, f is log-concave if for every $\lambda \in [0,1]$ and for every $x,y\in {\mathbb{R}}^n$, one has

$$\begin{array}{c}\hfill f((1-\lambda )x+\lambda y)\ge f{\left(x\right)}^{1-\lambda }f{\left(y\right)}^{\lambda }.\end{array}$$

(22)

We say that a random vector X in ${\mathbb{R}}^n$ is log-concave if it has a probability density function $f_X$ with respect to Lebesgue measure in ${\mathbb{R}}^n$ such that $f_X$ is log-concave.

Our first result is a lower bound on the differential entropy of symmetric log-concave random variable in terms of its moments.

Theorem 1.

Let X be a symmetric log-concave random variable. Then, for every $p>-1$,

$$\begin{array}{c}\hfill h\left(X\right)\ge log\frac{{2\parallel X\parallel }_p}{\mathrm{\Gamma }{(p+1)}^{\frac{1}{p}}}.\end{array}$$

(23)

Moreover, (23) holds with equality for uniform distribution in the limit $p\downarrow -1$.

As we will see in Theorem 3, for $p=2$, the bound (23) tightens as

$$\begin{array}{c}\hfill h\left(X\right)\ge log(2\parallel X{\parallel }_2).\end{array}$$

(24)

The difference between the upper bound in (2) and the lower bound in (23) grows as log(p) as $p\to +\infty$, as $\frac{1}{\sqrt{p}}$ as $p\to 0^{+}$, and reaches its minimum value of log(e) ≈ 1.4 bits at $p=1$.

The next theorem, due to Karlin, Proschan and Barlow [25], shows that the moments of a symmetric log-concave random variable are comparable, and demonstrates that the bound in Theorem 1 tightens as $p\downarrow -1$.

Theorem 2.

Let X be a symmetric log-concave random variable. Then, for every $-1<p\le q$,

$$\begin{array}{c}\hfill \frac{{\parallel X\parallel }_q}{\mathrm{\Gamma }{(q+1)}^{\frac{1}{q}}}\le \frac{{\parallel X\parallel }_p}{\mathrm{\Gamma }{(p+1)}^{\frac{1}{p}}}.\end{array}$$

(25)

Moreover, the Laplace distribution satisfies (25) with equality [25].

Combining Theorem 2 with the well-known fact that ${\parallel X\parallel }_p$ is non-decreasing in p, we deduce that for every symmetric log-concave random variable X, for every $-1<p<q$,

$$\begin{array}{c}\hfill {\parallel X\parallel }_p\le {\parallel X\parallel }_q\le \frac{\mathrm{\Gamma }{(q+1)}^{\frac{1}{q}}}{\mathrm{\Gamma }{(p+1)}^{\frac{1}{p}}}{\parallel X\parallel }_p.\end{array}$$

(26)

Using Theorem 1 and (24), we immediately obtain the following upper bound for the relative entropy $D\left(X\right|\left|G_X\right)$ between a symmetric log-concave random variable X and a Gaussian $G_X$ with same variance as that of X.

Corollary 1.

Let X be a symmetric log-concave random variable. Then, for every $p>-1$,

$$\begin{array}{c}\hfill D\left(X\right||G_X)\le log\sqrt{\pi e}+{\Delta }_p,\end{array}$$

(27)

where $G_X\sim {\mathcal{N}(0,\parallel X\parallel }_2^2)$, and

$${\Delta }_p\triangleq \left\{\begin{array}{cc}log\left(\frac{\mathrm{\Gamma }{(p+1)}^{\frac{1}{p}}}{\sqrt{2}}\frac{{\parallel X\parallel }_2}{{\parallel X\parallel }_p}\right),\hfill & p\ne 2,\hfill \\ -log\sqrt{2},\hfill & p=2.\hfill \end{array}\right.$$

(28)

Remark 1.

The uniform distribution achieves equality in (27) in the limit $p\downarrow -1$. Indeed, if U is uniformly distributed on a symmetric interval, then

$${\Delta }_p=log\frac{\mathrm{\Gamma }{(p+2)}^{\frac{1}{p}}}{\sqrt{6}}\underset{p\to -1}{\to }\frac{1}{2}log\frac{1}{6},$$

(29)

and so, in the limit $p\downarrow -1$, the upper bound in Corollary 1 coincides with the true value of $D\left(U\right|\left|G_U\right)$:

$$\begin{array}{c}\hfill D\left(U\right||G_U)=\frac{1}{2}log\frac{\pi e}{6}.\end{array}$$

(30)

We next provide a lower bound for the differential entropy of log-concave random variables that are not necessarily symmetric.

Theorem 3.

Let X be a log-concave random variable. Then, for every $p\ge 1$,

$$\begin{array}{c}\hfill h\left(X\right)\ge log\frac{{2\parallel X-\mathbb{E}\left[X\right]\parallel }_p}{\mathrm{\Gamma }{(p+1)}^{\frac{1}{p}}}.\end{array}$$

(31)

Moreover, for $p=2$, the bound (31) tightens as

$$h\left(X\right)\ge log\left(2\sqrt{\mathrm{Var}\left[X\right]}\right).$$

(32)

The next proposition is an analog of Theorem 2 for log-concave random variables that are not necessarily symmetric.

Proposition 1.

Let X be a log-concave random variable. Then, for every $1\le p\le q$,

$$\frac{{\parallel X-\mathbb{E}\left[X\right]\parallel }_q}{\mathrm{\Gamma }{(q+1)}^{\frac{1}{q}}}\le 2\frac{{\parallel X-\mathbb{E}\left[X\right]\parallel }_p}{\mathrm{\Gamma }{(p+1)}^{\frac{1}{p}}}.$$

(33)

Remark 2.

Contrary to Theorem 2, we do not know whether there exists a distribution that realizes equality in (33).

Using Theorem 3, we immediately obtain the following upper bound for the relative entropy $D\left(X\right|\left|G_X\right)$ between an arbitrary log-concave random variable X and a Gaussian $G_X$ with same variance as that of X. Recall the definition of ${\Delta }_p$ in (28).

Corollary 2.

Let X be a zero-mean, log-concave random variable. Then, for every $p\ge 1$,

$$\begin{array}{c}\hfill D\left(X\right||G_X)\le log\sqrt{\pi e}+{\Delta }_p,\end{array}$$

(34)

where $G_X\sim {\mathcal{N}(0,\parallel X\parallel }_2^2)$. In particular, by taking $p=2$, we necessarily have

$$\begin{array}{c}\hfill D\left(X\right||G_X)\le log\sqrt{\frac{\pi e}{2}}.\end{array}$$

(35)

For a given distribution of X, one can optimize over p to further tighten (35), as seen in (29) for the uniform distribution.

We now present a generalization of the bound in Theorem 1 to random vectors satisfying a symmetry condition. A function $f:{\mathbb{R}}^n\to \mathbb{R}$ is called unconditional if, for every $(x_1,\cdots ,x_n)\in {\mathbb{R}}^n$ and every $({\epsilon }_1,\cdots ,{\epsilon }_n)\in {\{-1,1\}}^n$, one has

$$\begin{array}{c}\hfill f({\epsilon }_1{x}_1,\cdots ,{\epsilon }_n{x}_n)=f(x_1,\cdots ,x_n).\end{array}$$

(36)

For example, the probability density function of the standard Gaussian distribution is unconditional. We say that a random vector X in ${\mathbb{R}}^n$ is unconditional if it has a probability density function $f_X$ with respect to Lebesgue measure in ${\mathbb{R}}^n$ such that $f_X$ is unconditional.

Theorem 4.

Let X be a symmetric log-concave random vector in ${\mathbb{R}}^n$, $n\ge 2$. Then,

$$\begin{array}{c}\hfill h\left(X\right)\ge \frac{n}{2}log\frac{|K_X{|}^{\frac{1}{n}}}{c\left(n\right)},\end{array}$$

(37)

where $|K_X|$ denotes the determinant of the covariance matrix of X, and $c\left(n\right)=\frac{e^2{n}^2}{4\sqrt{2}(n+2)}$. If, in addition, X is unconditional, then $c\left(n\right)=\frac{e^2}{2}$.

By combining Theorem 4 with the well-known upper bound on the differential entropy, we deduce that, for every symmetric log-concave random vector X in ${\mathbb{R}}^n$,

$$\frac{n}{2}log\left(\frac{|K_X{|}^{\frac{1}{n}}}{c\left(n\right)}\right)\le h\left(X\right)\le \frac{n}{2}log\left(2\pi e|K_X{|}^{\frac{1}{n}}\right),$$

(38)

where $c\left(n\right)=\frac{e^2{n}^2}{4\sqrt{2}(n+2)}$ in general, and $c\left(n\right)=\frac{e^2}{2}$ if, in addition, X is unconditional.

Using Theorem 4, we immediately obtain the following upper bound for the relative entropy $D\left(X\right|\left|G_X\right)$ between a symmetric log-concave random vector X and a Gaussian $G_X$ with the same covariance matrix as that of X.

Corollary 3.

Let X be a symmetric log-concave random vector in ${\mathbb{R}}^n$. Then,

$$\begin{array}{c}\hfill D\left(X\right||G_X)\le \frac{n}{2}log\left(2\pi ec\left(n\right)\right),\end{array}$$

(39)

where $G_X\sim \mathcal{N}(0,K_X)$, with $c\left(n\right)=\frac{n^2{e}^2}{(n+2)4\sqrt{2}}$ in general, and $c\left(n\right)=\frac{e^2}{2}$ when X is unconditional.

For isotropic unconditional log-concave random vectors (whose definition we recall in Section 3.3 below), we extend Theorem 4 to other moments.

Theorem 5.

Let $X=(X_1,\cdots ,X_n)$ be an isotropic unconditional log-concave random vector. Then, for every $p>-1$,

$$h\left(X\right)\ge \underset{i\in \{1,\cdots ,n\}}{max}nlog\left(\frac{2\parallel X_i{\parallel }_p}{\mathrm{\Gamma }{(p+1)}^{\frac{1}{p}}}\frac{1}{c}\right),$$

(40)

where $c=e\sqrt{6}$. If, in addition, $f_X$ is invariant under permutations of coordinates, then $c=e$.

2.2. Extension to $\gamma$-Concave Random Variables

The bound in Theorem 1 can be extended to a larger class of random variables than log-concave, namely the class of $\gamma$-concave random variables that we describe next.

Let $\gamma <0$. We say that a probability density function $f:{\mathbb{R}}^n\to [0,+\infty )$ is γ-concave if $f^{\gamma }$ is convex. Equivalently, f is $\gamma$-concave if for every $\lambda \in [0,1]$ and every $x,y\in {\mathbb{R}}^n$, one has

$$f((1-\lambda )x+\lambda y)\ge {((1-\lambda )f{\left(x\right)}^{\gamma }+\lambda f{\left(y\right)}^{\gamma })}^{\frac{1}{\gamma }}.$$

(41)

As $\gamma \to 0$, (41) agrees with (22), and thus 0-concave distributions corresponds to log-concave distributions. The class of $\gamma$-concave distributions has been deeply studied in [26,27].

Since for fixed $a,b\ge 0$ the function ${((1-\lambda )a^{\gamma }+\lambda b^{\gamma })}^{\frac{1}{\gamma }}$ is non-decreasing in $\gamma$, we deduce that any log-concave distribution is $\gamma$-concave, for any $\gamma <0$.

For example, extended Cauchy distributions, that is, distributions of the form

$$f_X\left(x\right)=\frac{C_{\gamma }}{1+{\left|x\right|}^{n-\frac{1}{\gamma }}},x\in {\mathbb{R}}^n,$$

(42)

where $C_{\gamma }$ is the normalization constant, are $\gamma$-concave distributions (but are not log-concave).

We say that a random vector X in ${\mathbb{R}}^n$ is $\gamma$-concave if it has a probability density function $f_X$ with respect to Lebesgue measure in ${\mathbb{R}}^n$ such that $f_X$ is $\gamma$-concave.

We derive the following lower bound on the differential entropy for one-dimensional symmetric $\gamma$-concave random variables, with $\gamma \in (-1,0)$.

Theorem 6.

Let $\gamma \in (-1,0)$. Let X be a symmetric γ-concave random variable. Then, for every $p\in \left(-1,-1-\frac{1}{\gamma }\right)$,

$$h\left(X\right)\ge log\left(\frac{{2\parallel X\parallel }_p}{\mathrm{\Gamma }{(p+1)}^{\frac{1}{p}}}\frac{\mathrm{\Gamma }{(-1-\frac{1}{\gamma })}^{1+\frac{1}{p}}}{\mathrm{\Gamma }(-\frac{1}{\gamma })\mathrm{\Gamma }{(-\frac{1}{\gamma }-(p+1))}^{\frac{1}{p}}}\right).$$

(43)

Notice that (43) reduces to (23) as $\gamma \to 0$. Theorem 6 implies the following relation between entropy and second moment, for any $\gamma \in (-\frac{1}{3},0)$.

Corollary 4.

Let $\gamma \in (-\frac{1}{3},0)$. Let X be a symmetric γ-concave random variable. Then,

$$h\left(X\right)\ge \frac{1}{2}log\left({2\parallel X\parallel }_2^2\frac{\mathrm{\Gamma }{(-1-\frac{1}{\gamma })}^3}{\mathrm{\Gamma }{(-\frac{1}{\gamma })}^2\mathrm{\Gamma }(-\frac{1}{\gamma }-3)}\right)=\frac{1}{2}log\left({2\parallel X\parallel }_2^2\frac{(2\gamma +1)(3\gamma +1)}{{(\gamma +1)}^2}\right).$$

(44)

2.3. Reverse Entropy Power Inequality with an Explicit Constant

As an application of Theorems 3 and 4, we establish in Theorems 7 and 8 below a reverse form of the entropy power inequality (7) with explicit constants, for uncorrelated log-concave random vectors. Recall the definition of the entropy power (8).

Theorem 7.

Let X and Y be uncorrelated log-concave random variables. Then,

$$N(X+Y)\le \frac{\pi e}{2}(N\left(X\right)+N\left(Y\right)).$$

(45)

As a consequence of Corollary 4, reverse entropy power inequalities for more general distributions can be obtained. In particular, for any uncorrelated symmetric $\gamma$-concave random variables X and Y, with $\gamma \in (-\frac{1}{3},0)$,

$$N(X+Y)\le \pi e\frac{{(\gamma +1)}^2}{(2\gamma +1)(3\gamma +1)}(N\left(X\right)+N\left(Y\right)).$$

(46)

One cannot have a reverse entropy power inequality in higher dimensions for arbitrary log-concave random vectors. Indeed, just consider X uniformly distributed on $[-\frac{\epsilon }{2},\frac{\epsilon }{2}]\times [-\frac{1}{2},\frac{1}{2}]$ and Y uniformly distributed on $[-\frac{1}{2},\frac{1}{2}]\times [-\frac{\epsilon }{2},\frac{\epsilon }{2}]$ in ${\mathbb{R}}^2$, with $\epsilon >0$ small enough so that $N\left(X\right)$ and $N\left(Y\right)$ are arbitrarily small compared to $N(X+Y)$. Hence, we need to put X and Y in a certain position so that a reverse form of (7) is possible. While the isotropic position (discussed in Section 3) will work, it can be relaxed to the weaker condition that the covariance matrices are proportionals. Recall that we denote by $K_X$ the covariance matrix of X.

Theorem 8.

Let X and Y be uncorrelated symmetric log-concave random vectors in ${\mathbb{R}}^n$ such that $K_X$ and $K_Y$ are proportionals. Then,

$$N(X+Y)\le \frac{\pi e^3{n}^2}{2\sqrt{2}(n+2)}(N\left(X\right)+N\left(Y\right)).$$

(47)

If, in addition, X and Y are unconditional, then

$$N(X+Y)\le \pi e^3(N\left(X\right)+N\left(Y\right)).$$

(48)

2.4. New Bounds on the Rate-distortion Function

As an application of Theorems 1 and 3, we show in Corollary 5 below that in the class of one-dimensional log-concave distributions, the rate-distortion function does not exceed the Shannon lower bound by more than $log\left(\sqrt{\pi e}\right)\approx 1.55$ bits (which can be refined to $log\left(e\right)\approx 1.44$ bits when the source is symmetric), independently of d and $r\ge 1$. Denote for brevity

$$\begin{array}{cc}\hfill {\beta }_r& \triangleq \sqrt{1+\frac{r^{\frac{2}{r}}\mathrm{\Gamma }\left(\frac{3}{r}\right)}{\mathrm{\Gamma }\left(\frac{1}{r}\right)}},\hfill \end{array}$$

(49)

and recall the definition of ${\alpha }_r$ in (3).

We start by giving a bound on the difference between the rate-distortion function and the Shannon lower bound, which applies to general, not necessarily log-concave, random variables.

Theorem 9.

Let $d\ge 0$ and $r\ge 1$. Let X be an arbitrary random variable.

$\left(1\right)$ Let $r\in [1,2]$. If ${\parallel X\parallel }_2>d^{\frac{1}{r}}$, then

$$\begin{array}{c}\hfill {\mathbb{R}}_X\left(d\right)-{\underset{\_}{\mathbb{R}}}_X\left(d\right)\le D\left(X\right||G_X)+log\frac{{\alpha }_r}{\sqrt{2\pi e}}.\end{array}$$

(50)

If ${\parallel X\parallel }_2\le d^{\frac{1}{r}}$, then ${\mathbb{R}}_X\left(d\right)=0$.

$\left(2\right)$ Let $r>2$. If ${\parallel X\parallel }_2\ge d^{\frac{1}{r}}$, then

$$\begin{array}{c}\hfill {\mathbb{R}}_X\left(d\right)-{\underset{\_}{\mathbb{R}}}_X\left(d\right)\le D\left(X\right||G_X)+log{\beta }_r.\end{array}$$

(51)

If ${\parallel X\parallel }_r\le d^{\frac{1}{r}}$, then ${\mathbb{R}}_X\left(d\right)=0$. If ${\parallel X\parallel }_r>d^{\frac{1}{r}}$ and ${\parallel X\parallel }_2<d^{\frac{1}{r}}$, then ${\mathbb{R}}_X\left(d\right)\le log\frac{\sqrt{2\pi e}{\beta }_r}{{\alpha }_r}$.

Remark 3.

For Gaussian X and $r=2$, the upper bound in (50) is 0, as expected.

The next result refines the bounds in Theorem 9 for symmetric log-concave random variables when $r>2$.

Theorem 10.

Let $d\ge 0$ and $r>2$. Let X be a symmetric log-concave random variable.

If ${\parallel X\parallel }_2\ge d^{\frac{1}{r}}$, then

$$\begin{array}{c}\hfill {\mathbb{R}}_X\left(d\right)-{\underset{\_}{\mathbb{R}}}_X\left(d\right)\le D\left(X\right||G_X)+min\left\{log\left({\beta }_r\right),log\frac{{\alpha }_r\mathrm{\Gamma }{(r+1)}^{\frac{1}{r}}}{2\sqrt{\pi e}}\right\}.\end{array}$$

(52)

If ${\parallel X\parallel }_r\le d^{\frac{1}{r}}$ or ${\parallel X\parallel }_2\le \frac{\sqrt{2}}{\mathrm{\Gamma }{(r+1)}^{\frac{1}{r}}}{d}^{\frac{1}{r}}$, then ${\mathbb{R}}_X\left(d\right)=0$. If ${\parallel X\parallel }_r>d^{\frac{1}{r}}$ and ${\parallel X\parallel }_2\in \left(\frac{\sqrt{2}}{\mathrm{\Gamma }{(r+1)}^{\frac{1}{r}}}{d}^{\frac{1}{r}},d^{\frac{1}{r}}\right)$, then ${\mathbb{R}}_X\left(d\right)\le min\left\{log\frac{\sqrt{2\pi e}{\beta }_r}{{\alpha }_r},log\frac{\mathrm{\Gamma }{(r+1)}^{\frac{1}{r}}}{\sqrt{2}}\right\}$.

To bound ${\mathbb{R}}_X\left(d\right)-{\underset{\_}{\mathbb{R}}}_X\left(d\right)$ independently of the distribution of X, we apply the bound (35) on $D\left(X\right|\left|G_X\right)$ to Theorems 9 and 10:

Corollary 5.

Let X be a log-concave random variable. For $r\in [1,2]$, we have

$$\begin{array}{c}\hfill {\mathbb{R}}_X\left(d\right)-{\underset{\_}{\mathbb{R}}}_X\left(d\right)\le log\frac{{\alpha }_r}{2}.\end{array}$$

(53)

For $r>2$, we have

$$\begin{array}{c}\hfill {\mathbb{R}}_X\left(d\right)-{\underset{\_}{\mathbb{R}}}_X\left(d\right)\le log\left(\sqrt{\frac{\pi e}{2}}{\beta }_r\right).\end{array}$$

(54)

If, in addition, X is symmetric, then, for $r>2,$ we have

$$\begin{array}{c}\hfill {\mathbb{R}}_X\left(d\right)-{\underset{\_}{\mathbb{R}}}_X\left(d\right)\le min\left\{log\frac{{\alpha }_r\mathrm{\Gamma }{(r+1)}^{\frac{1}{r}}}{2\sqrt{2}},log\left(\sqrt{\frac{\pi e}{2}}{\beta }_r\right)\right\}.\end{array}$$

(55)

Figure 1a presents our bound for different values of r. Regardless of r and d,

$${\mathbb{R}}_X\left(d\right)-{\underset{\_}{\mathbb{R}}}_X\left(d\right)\le log\left(\sqrt{\pi e}\right)\approx 1.55\mathrm{bits}.$$

(56)

The bounds in Figure 1a tighten for symmetric log-concave sources when $r\in (2,4.3)$. Figure 1b presents this tighter bound for different values of r. Regardless of r and d,

$${\mathbb{R}}_X\left(d\right)-{\underset{\_}{\mathbb{R}}}_X\left(d\right)\le log\left(e\right)\approx 1.44\mathrm{bits}.$$

(57)

One can see that the graph in Figure 1b is continuous at $r=2$, contrary to the graph in Figure 1a. This is because Theorem 2, which applies to symmetric log-concave random variables, is strong enough to imply the tightening of (51) given in (52), while Proposition 1, which provides a counterpart of Theorem 2 applicable to all log-concave random variables, is insufficient to derive a similar tightening in that more general setting.

Remark 4.

While Corollary 5 bounds the difference ${\mathbb{R}}_X\left(d\right)-\underset{-}{\mathbb{R}}{}_X\left(d\right)$ by a universal constant independent of the distribution of X, tighter bounds can be obtained if one is willing to relinquish such universality. For example, for mean-square distortion ($r=2$) and a uniformly distributed source U, using Remark 1, we obtain

$$\begin{array}{c}\hfill {\mathbb{R}}_U\left(d\right)-\underset{-}{\mathbb{R}}{}_U\left(d\right)\le \frac{1}{2}log\frac{2\pi e}{12}\approx 0.254bits.\end{array}$$

(58)

Theorem 9 easily extends to random vector X in ${\mathbb{R}}^n$, $n\ge 2$, with a similar proof. The only difference being an extra term of $\frac{n}{2}log\left(\frac{1}{n}{\parallel X\parallel }_2^2/{\left|K_X\right|}^{\frac{1}{n}}\right)$ that will appear on the right-hand side of (50) and (51), and will come from the upper bound on the differential entropy (38). Here,

$${\parallel X\parallel }_p\triangleq {\left(\mathbb{E}\left[\sum _{i=1}^n{\left|X_i\right|}^p\right]\right)}^{\frac{1}{p}}.$$

As a result, the bound ${\mathbb{R}}_X\left(d\right)-\underset{-}{\mathbb{R}}{}_X\left(d\right)$ can be arbitrarily large in higher dimensions because of the term $\frac{1}{n}{\parallel X\parallel }_2^2/{\left|K_X\right|}^{\frac{1}{n}}$. However, for isotropic random vectors (whose definition we recall in Section 3.3 below), one has $\frac{1}{n}{\parallel X\parallel }_2^2={\left|K_X\right|}^{\frac{1}{n}}$. Hence, using the bound (39) on $D\left(X\right|\left|G_X\right)$, we can bound ${\mathbb{R}}_X\left(d\right)-{\underset{\_}{\mathbb{R}}}_X\left(d\right)$ independently of the distribution of isotropic log-concave random vector X in ${\mathbb{R}}^n$, $n\ge 2$.

Corollary 6.

Let X be an isotropic log-concave random vector in ${\mathbb{R}}^n$, $n\ge 2$. Then,

$${\mathbb{R}}_X\left(d\right)-{\underset{\_}{\mathbb{R}}}_X\left(d\right)\le \frac{n}{2}log\left(2\pi ec\left(n\right)\right),$$

(59)

where $c\left(n\right)=\frac{n^2{e}^2}{(n+2)4\sqrt{2}}$ in general, and $c\left(n\right)=\frac{e^2}{2}$ if, in addition, X is unconditional.

Let us consider the rate-distortion function under the determinant constraint for random vectors in ${\mathbb{R}}^n$, $n\ge 2$:

$${\mathbb{R}}_X^{cov}\left(d\right)=\underset{P_{\widehat{X}|X}:{\left|K_{X-\widehat{X}}\right|}^{\frac{1}{n}}\le d}{inf}I(X;\widehat{X}),$$

(60)

where the infimum is taken over all joint distributions satisfying the determinant constraint $|K_{X-\widehat{X}}{|}^{\frac{1}{n}}\le d$. For this distortion measure, we have the following bound.

Theorem 11.

Let X be a symmetric log-concave random vector in ${\mathbb{R}}^n$. If $|K_X{|}^{\frac{1}{n}}>d$, then

$$0\le {\mathbb{R}}_X^{cov}\left(d\right)-{\underset{\_}{\mathbb{R}}}_X\left(d\right)\le D\left(X\right||G_X)\le \frac{n}{2}log\left(2\pi ec\left(n\right)\right),$$

(61)

with $c\left(n\right)=\frac{n^2{e}^2}{(n+2)4\sqrt{2}}$. If, in addition, X is unconditional, then $c\left(n\right)=\frac{e^2}{2}$. If $|K_X{|}^{\frac{1}{n}}\le d$, then ${\mathbb{R}}_X^{cov}\left(d\right)=0$.

2.5. New Bounds on the Capacity of Memoryless Additive Channels

As another application of Theorem 3, we compare the capacity ${\mathbb{C}}_Z$ of a channel with log-concave additive noise Z with the capacity of the Gaussian channel. Recall that the capacity of the Gaussian channel is

$$\underset{-}{\mathbb{C}}{}_Z\left(P\right)=\frac{1}{2}log\left(1+\frac{P}{\mathrm{Var}\left[Z\right]}\right).$$

(62)

Theorem 12.

Let Z be a log-concave random variable. Then,

$$0\le {\mathbb{C}}_Z\left(P\right)-\underset{-}{\mathbb{C}}{}_Z\left(P\right)\le log\sqrt{\frac{\pi e}{2}}\approx 1.05\mathrm{bits}.$$

(63)

Remark 5.

Theorem 12 tells us that the capacity of a channel with log-concave additive noise exceeds the capacity a Gaussian channel by no more than $1.05$ bits.

As an application of Theorem 4, we can provide bounds for the capacity of a channel with log-concave additive noise Z in ${\mathbb{R}}^n$, $n\ge 1$. The formula for capacity (18) generalizes to dimension n as

$${\mathbb{C}}_Z\left(P\right)=\underset{X:\frac{1}{n}{\parallel X\parallel }_2^2\le P}{sup}I(X;X+Z).$$

(64)

Theorem 13.

Let Z be a symmetric log-concave random vector in ${\mathbb{R}}^n$. Then,

$$0\le {\mathbb{C}}_Z\left(P\right)-\frac{n}{2}log\left(1+\frac{P}{|K_Z{|}^{\frac{1}{n}}}\right)\le \frac{n}{2}log\left(2\pi ec\left(n\right)\left(\frac{\frac{1}{n}{\parallel Z\parallel }_2^2+P}{|K_Z{|}^{\frac{1}{n}}+P}\right)\right),$$

(65)

where $c\left(n\right)=\frac{n^2{e}^2}{(n+2)4\sqrt{2}}$. If, in addition, Z is unconditional, then $c\left(n\right)=\frac{e^2}{2}$.

The upper bound in Theorem 13 can be arbitrarily large by inflating the ratio $\frac{1}{n}{\parallel X\parallel }_2^2/{\left|K_X\right|}^{\frac{1}{n}}$. For isotropic random vectors (whose definition is recalled in Section 3.3 below), one has $\frac{1}{n}{\parallel Z\parallel }_2^2={\left|K_Z\right|}^{\frac{1}{n}}$, and the following corollary follows.

Corollary 7.

Let Z be an isotropic log-concave random vector in ${\mathbb{R}}^n$. Then,

$$0\le {\mathbb{C}}_Z\left(P\right)-\frac{n}{2}log\left(1+\frac{P}{|K_Z{|}^{\frac{1}{n}}}\right)\le \frac{n}{2}log\left(2\pi ec\left(n\right)\right),$$

(66)

where $c\left(n\right)=\frac{n^2{e}^2}{(n+2)4\sqrt{2}}$. If, in addition, Z is unconditional, then $c\left(n\right)=\frac{e^2}{2}$.

3. New Lower Bounds on the Differential Entropy

3.1. Proof of Theorem 1

The key to our development is the following result for one-dimensional log-concave distributions, well-known in convex geometry. It can be found in [28], in a slightly different form.

Lemma 1.

The function

$$\begin{array}{c}\hfill F\left(r\right)=\frac{1}{\mathrm{\Gamma }(r+1)}{\int }_0^{+\infty }{x}^{r}f\left(x\right)\mathrm{d}x\end{array}$$

(67)

is log-concave on $[-1,+\infty )$, whenever $f:[0;+\infty )\to [0;+\infty )$ is log-concave [28].

Proof of Theorem 1.

Let $p>0$. Applying Lemma 1 to the values $-1,0,p$, we have

$$\begin{array}{c}\hfill F\left(0\right)=F\left(\frac{p}{p+1}(-1)+\frac{1}{p+1}p\right)\ge F{(-1)}^{\frac{p}{p+1}}F{\left(p\right)}^{\frac{1}{p+1}}.\end{array}$$

(68)

The bound in Theorem 1 follows by computing the values $F(-1)$, $F\left(0\right)$ and $F\left(p\right)$ for $f=f_X$.

One has

$$\begin{array}{c}\hfill F\left(0\right)=\frac{1}{2},F\left(p\right)=\frac{{\parallel X\parallel }_p^p}{2\mathrm{\Gamma }(p+1)}.\end{array}$$

(69)

To compute $F(-1)$, we first provide a different expression for $F\left(r\right)$. Notice that

$$F\left(r\right)=\frac{1}{\mathrm{\Gamma }(r+1)}{\int }_0^{+\infty }{x}^r{\int }_0^{f_X\left(x\right)}\mathrm{d}t\mathrm{d}x=\frac{r+1}{\mathrm{\Gamma }(r+2)}{\int }_0^{max{f}_X}{\int }_{\{x\ge 0:f_X\left(x\right)\ge t\}}{x}^r\mathrm{d}x\mathrm{d}t.$$

(70)

Denote the generalized inverse of $f_X$ by $f_X^{-1}\left(t\right)\triangleq sup\{x\ge 0:f_X\left(x\right)\ge t\}$, $t\ge 0$. Since $f_X$ is log-concave and

$$f_X\left(x\right)\le f_X\left(0\right)=max{f}_X,$$

(71)

it follows that $f_X$ is non-increasing on $[0,+\infty )$. Therefore, $\{x\ge 0:f_X\left(x\right)\ge t\}=[0,f_X^{-1}\left(t\right)]$. Hence,

$$\begin{array}{c}\hfill F\left(r\right)=\frac{r+1}{\mathrm{\Gamma }(r+2)}{\int }_0^{f_X\left(0\right)}{\int }_0^{f_X^{-1}\left(t\right)}{x}^r\mathrm{d}x\mathrm{d}t=\frac{1}{\mathrm{\Gamma }(r+2)}{\int }_0^{f_X\left(0\right)}{\left(f_X^{-1}\left(t\right)\right)}^{r+1}\mathrm{d}t.\end{array}$$

(72)

We deduce that

$$\begin{array}{c}\hfill F(-1)=f_X\left(0\right).\end{array}$$

(73)

Plugging (69) and (73) into (68), we obtain

$$\begin{array}{c}\hfill f_X\left(0\right)\le \frac{\mathrm{\Gamma }{(p+1)}^{\frac{1}{p}}}{{2\parallel X\parallel }_p}.\end{array}$$

(74)

It follows immediately that

$$\begin{array}{c}\hfill h\left(X\right)=\int f_X\left(x\right)log\frac{1}{f_X\left(x\right)}\mathrm{d}x\ge log\frac{1}{f_X\left(0\right)}\ge log\frac{{2\parallel X\parallel }_p}{\mathrm{\Gamma }{(p+1)}^{\frac{1}{p}}}.\end{array}$$

(75)

For $p\in (-1,0)$, the bound is obtained similarly by applying Lemma 1 to the values $-1,p,0$.

We now show that equality is attained, by letting $p\downarrow -1$, by U uniformly distributed on a symmetric interval $[-\frac{a}{2},\frac{a}{2}]$, for some $a>0$. In this case, we have

$${\parallel U\parallel }_p^p={\left(\frac{a}{2}\right)}^p\frac{1}{p+1}.$$

(76)

Hence,

$$\frac{1}{p}log\frac{2^p{\parallel U\parallel }_p^p}{\mathrm{\Gamma }(p+1)}=log\frac{a}{\mathrm{\Gamma }{(p+2)}^{\frac{1}{p}}}\underset{p\to -1}{\to }log\left(a\right)=h\left(U\right).$$

(77)

☐

Remark 6.

From (71) and (74), we see that the following statement holds: For every symmetric log-concave random variable $X\sim f_X$, for every $p>-1$, and for every $x\in \mathbb{R}$,

$$\begin{array}{c}\hfill f_X\left(x\right)\le \frac{\mathrm{\Gamma }{(p+1)}^{\frac{1}{p}}}{{2\parallel X\parallel }_p}.\end{array}$$

(78)

Inequality (78) is the main ingredient in the proof of Theorem 1. It is instructive to provide a direct proof of inequality (78) without appealing to Lemma 1, the ideas going back to [25]:

Proof of inequality (78)

By considering $X|X\ge 0$, where X is symmetric log-concave, it is enough to show that for every log-concave density f supported on $[0,+\infty )$, one has

$$f\left(0\right){\left({\int }_0^{+\infty }{x}^{p}f\left(x\right)\mathrm{d}x\right)}^{\frac{1}{p}}\le \mathrm{\Gamma }{(p+1)}^{\frac{1}{p}}.$$

(79)

By a scaling argument, one may assume that $f\left(0\right)=1$. Take $g\left(x\right)=e^{-x}$. If $f=g$, then the result follows by a straightforward computation. Assume that $f\ne g$. Since $f\ne g$ and $\int f=\int g$, the function $f-g$ changes sign at least one time. However, since $f\left(0\right)=g\left(0\right)$, f is log-concave and g is log-affine, the function $f-g$ changes sign exactly once. It follows that there exists a unique point $x_0>0$ such that for every $0<x<x_0$, $f\left(x\right)\ge g\left(x\right)$, and for every $x>x_0$, $f\left(x\right)\le g\left(x\right)$. We deduce that for every $x>0$, and $p\ne 0$,

$$\frac{1}{p}(f\left(x\right)-g\left(x\right))(x^p-x_0^p)\le 0.$$

(80)

Integrating over $x>0$, we arrive at

$$\frac{1}{p}\left({\int }_0^{+\infty }{x}^{p}f\left(x\right)\mathrm{d}x-\mathrm{\Gamma }(p+1)\right)=\frac{1}{p}{\int }_0^{+\infty }(x^p-x_0^p)(f\left(x\right)-g\left(x\right))\mathrm{d}x\le 0,$$

(81)

which yields the desired result. ☐

Actually, the powerful and versatile result of Lemma 1, which implies (78), is also proved using the technique in (79)–(81). In the context of information theory, Lemma 1 has been previously applied to obtain reverse entropy power inequalities [7], as well as to establish optimal concentration of the information content [29]. In this paper, we make use of Lemma 1 to prove Theorem 1. Moreover, Lemma 1 immediately implies Theorem 2. Below, we recall the argument for completeness.

Proof of Theorem 2.

The result follows by applying Lemma 1 to the values $0,p,q$. If $0<p<q$, then

$$F\left(p\right)=F\left(0·\left(1-\frac{p}{q}\right)+q·\left(\frac{p}{q}\right)\right)\ge F{\left(0\right)}^{1-\frac{p}{q}}F{\left(q\right)}^{\frac{p}{q}}.$$

(82)

Hence,

$$\frac{{\parallel X\parallel }_p^p}{\mathrm{\Gamma }(p+1)}\ge {\left(\frac{{\parallel X\parallel }_q^q}{\mathrm{\Gamma }(q+1)}\right)}^{\frac{p}{q}},$$

(83)

which yields the desired result. The bound is obtained similarly if $p<q<0$ or if $p<0<q$. ☐

3.2. Proof of Theorem 3 and Proposition 1

The proof leverages the ideas from [10].

Proof of Theorem 3.

Let Y be an independent copy of X. Jensen’s inequality yields

$$h\left(X\right)=-\int f_{X}log\left(f_X\right)\ge -log\left(\int f_X^2\right)=-log\left(f_{X-Y}\left(0\right)\right).$$

(84)

Since $X-Y$ is symmetric and log-concave, we can apply inequality (74) to $X-Y$ to obtain

$$\frac{1}{f_{X-Y}\left(0\right)}\ge \frac{{2\parallel X-Y\parallel }_p}{\mathrm{\Gamma }{(p+1)}^{\frac{1}{p}}}\ge \frac{{2\parallel X-\mathbb{E}\left[X\right]\parallel }_p}{\mathrm{\Gamma }{(p+1)}^{\frac{1}{p}}},$$

(85)

where the last inequality again follows from Jensen’s inequality. Combining (84) and (85) leads to the desired result:

$$h\left(X\right)\ge log\left(\frac{1}{f_{X-Y}\left(0\right)}\right)\ge log\left(\frac{{2\parallel X-\mathbb{E}\left[X\right]\parallel }_p}{\mathrm{\Gamma }{(p+1)}^{\frac{1}{p}}}\right).$$

(86)

For $p=2$, one may tighten (85) by noticing that

$${\parallel X-Y\parallel }_2^2=2\mathrm{Var}\left[X\right].$$

(87)

Hence,

$$h\left(X\right)\ge log\left(\frac{1}{f_{X-Y}\left(0\right)}\right)\ge log\left(\sqrt{2}{\parallel X-Y\parallel }_2\right)=log\left(2\sqrt{\mathrm{Var}\left[X\right]}\right).$$

(88)

☐

Proof of Proposition 1.

Let Y be an independent copy of X. Since $X-Y$ is symmetric and log-concave, we can apply Theorem 2 to $X-Y$. Jensen’s inequality and triangle inequality yield:

$${\parallel X-\mathbb{E}\left[X\right]\parallel }_q\le {\parallel X-Y\parallel }_q\le \frac{\mathrm{\Gamma }{(q+1)}^{\frac{1}{q}}}{\mathrm{\Gamma }{(p+1)}^{\frac{1}{p}}}{\parallel X-Y\parallel }_p\le 2\frac{\mathrm{\Gamma }{(q+1)}^{\frac{1}{q}}}{\mathrm{\Gamma }{(p+1)}^{\frac{1}{p}}}{\parallel X-\mathbb{E}\left[X\right]\parallel }_p.$$

(89)

☐

3.3. Proof of Theorem 4

We say that a random vector $X\sim f_X$ is isotropic if X is symmetric and for all unit vectors $\theta$, one has

$$\begin{array}{c}\hfill \mathbb{E}\left[{\langle X,\theta \rangle }^2\right]=m_X^2,\end{array}$$

(90)

for some constant $m_X>0$. Equivalently, X is isotropic if its covariance matrix $K_X$ is a multiple of the identity matrix $I_n$,

$$K_X=m_X^2{I}_n,$$

(91)

for some constant $m_X>0$. The constant

$$\begin{array}{c}\hfill {\ell }_X\triangleq f_X{\left(0\right)}^{\frac{1}{n}}{m}_X\end{array}$$

(92)

is called the isotropic constant of X.

It is well known that ${\ell }_X$ is bounded from below by a positive constant independent of the dimension [30]. A long-standing conjecture in convex geometry, the hyperplane conjecture, asks whether the isotropic constant of an isotropic log-concave random vector is also bounded from above by a universal constant (independent of the dimension). This conjecture holds under additional assumptions, but, in full generality, ${\ell }_X$ is known to be bounded only by a constant that depends on the dimension. For further details, we refer the reader to [31]. We will use the following upper bounds on ${\ell }_X$ (see [32] for the best dependence on the dimension up to date).

Lemma 2.

Let X be an isotropic log-concave random vector in ${\mathbb{R}}^n$, with $n\ge 2$. Then, ${\ell }_X^2\le \frac{n^2{e}^2}{(n+2)4\sqrt{2}}$. If, in addition, X is unconditional, then ${\ell }_X^2\le \frac{e^2}{2}$.

If X is uniformly distributed on a convex set, these bounds hold without factor $e^2$.

Even though the bounds in Lemma 2 are well known, we could not find a reference in the literature. We thus include a short proof for completeness.

Proof. 

It was shown by Ball [30] (Lemma 8) that if X is uniformly distributed on a convex set, then ${\ell }_X^2\le \frac{n^2}{(n+2)4\sqrt{2}}$. If X is uniformly distributed on a convex set and is unconditional, then it is known that ${\ell }_X^2\le \frac{1}{2}$ (see e.g., [33] (Proposition 2.1)). Now, one can pass from uniform distributions on a convex set to log-concave distributions at the expense of an extra factor $e^2$, as shown by Ball [30] (Theorem 7). ☐

We are now ready to prove Theorem 4.

Proof of Theorem 4.

Let $\tilde{X}\sim f_{\tilde{X}}$ be an isotropic log-concave random vector. Notice that $f_{\tilde{X}}{\left(0\right)}^{\frac{2}{n}}{\left|K_{\tilde{X}}\right|}^{\frac{1}{n}}={\ell }_{\tilde{X}}^2$, hence, using Lemma 2, we have

$$\begin{array}{c}\hfill h\left(\tilde{X}\right)=\int f_{\tilde{X}}\left(x\right)log\frac{1}{f_{\tilde{X}}\left(x\right)}\mathrm{d}x\ge log\frac{1}{f_{\tilde{X}}\left(0\right)}\ge \frac{n}{2}log\frac{|K_{\tilde{X}}{|}^{\frac{1}{n}}}{c\left(n\right)},\end{array}$$

(93)

with $c\left(n\right)=\frac{n^2{e}^2}{(n+2)4\sqrt{2}}$. If, in addition, $\tilde{X}$ is unconditional, then again by Lemma 2, $c\left(n\right)=\frac{e^2}{2}$.

Now consider an arbitrary symmetric log-concave random vector X. One can apply a change of variable to put X in isotropic position. Indeed, by defining $\tilde{X}=K_X^{-\frac{1}{2}}X$, one has for every unit vector $\theta$,

$$\begin{array}{c}\hfill \mathbb{E}\left[{\langle \tilde{X},\theta \rangle }^2\right]=\mathbb{E}\left[{\langle X,K_X^{-\frac{1}{2}}\theta \rangle }^2\right]=\langle K_X\left(K_X^{-\frac{1}{2}}\theta \right),K_X^{-\frac{1}{2}}\theta \rangle =1.\end{array}$$

(94)

It follows that $\tilde{X}$ is an isotropic log-concave random vector with isotropic constant 1. Therefore, we can use (93) to obtain

$$\begin{array}{c}\hfill h\left(\tilde{X}\right)\ge \frac{n}{2}log\frac{1}{c\left(n\right)},\end{array}$$

(95)

where $c\left(n\right)=\frac{n^2{e}^2}{(n+2)4\sqrt{2}}$ in general, and $c\left(n\right)=\frac{e^2}{2}$ when X is unconditional. We deduce that

$$\begin{array}{c}\hfill h\left(X\right)=h\left(\tilde{X}\right)+\frac{n}{2}log{\left|K_X\right|}^{\frac{1}{n}}\ge \frac{n}{2}log\frac{|K_X{|}^{\frac{1}{n}}}{c\left(n\right)}.\end{array}$$

(96)

☐

3.4. Proof of Theorem 5

First, we need the following lemma.

Lemma 3.

Let $X\sim f_X$ be an isotropic unconditional log-concave random vector. Then, for every $i\in \{1,\cdots ,n\}$,

$$f_{X_i}\left(0\right)\ge \frac{f_X{\left(0\right)}^{\frac{1}{n}}}{c},$$

(97)

where $f_{X_i}$ is the marginal distribution of the i-th component of X, i.e., for every $t\in \mathbb{R}$,

$$f_{X_i}\left(t\right)={\int }_{{\mathbb{R}}^{n-1}}{f}_X(x_1,\cdots ,x_{i-1},t,x_{i+1},\cdots ,x_n)\mathrm{d}{x}_1\cdots \mathrm{d}{x}_{i-1}\mathrm{d}{x}_{i+1}\cdots \mathrm{d}{x}_n.$$

(98)

Here, $c=e\sqrt{6}$. If, in addition, $f_X$ is invariant under permutations of coordinates, then $c=e$ [33] (Proposition 3.2).

Proof of Theorem 5.

Let $i\in \{1,\cdots ,n\}$. We have

$$\parallel X_i{\parallel }_p^p={\int }_{\mathbb{R}}{\left|t\right|}^p{f}_{X_i}\left(t\right)\mathrm{d}t.$$

(99)

Since $f_X$ is unconditional and log-concave, it follows that $f_{X_i}$ is symmetric and log-concave, so inequality (74) applies to $f_{X_i}$:

$${\int }_{\mathbb{R}}{\left|t\right|}^p{f}_{X_i}\left(t\right)\mathrm{d}t\le \frac{\mathrm{\Gamma }(p+1)}{2^p{f}_{X_i}{\left(0\right)}^p}.$$

(100)

We apply Lemma 3 to pass from $f_{X_i}$ to $f_X$ in the right side of (100):

$$f_X{\left(0\right)}^{\frac{1}{n}}{\parallel X_i\parallel }_p\le \frac{\mathrm{\Gamma }{(p+1)}^{\frac{1}{p}}c}{2}.$$

(101)

Thus,

$$h\left(X\right)\ge log\frac{1}{f_X\left(0\right)}\ge nlog\frac{2\parallel X_i{\parallel }_p}{\mathrm{\Gamma }{(p+1)}^{\frac{1}{p}}c}.$$

(102)

☐

4. Extension to $\gamma$-Concave Random Variables

In this section, we prove Theorem 6, which extends Theorem 1 to the class of $\gamma$-concave random variables, with $\gamma <0$. First, we need the following key lemma, which extends Lemma 1.

Lemma 4.

Let $f:[0,+\infty )\to [0,+\infty )$ be a γ-concave function, with $\gamma <0$. Then, the function

$$F\left(r\right)=\frac{\mathrm{\Gamma }(-\frac{1}{\gamma })}{\mathrm{\Gamma }(-\frac{1}{\gamma }-(r+1))}\frac{1}{\mathrm{\Gamma }(r+1)}{\int }_0^{+\infty }{t}^{r}f\left(t\right)\mathrm{d}t$$

(103)

is log-concave on $\left[-1,-1-\frac{1}{\gamma }\right)$ [34] (Theorem 7).

One can recover Lemma 1 from Lemma 4 by letting $\gamma$ tend to 0 from below.

Proof of Theorem 6.

Let us first consider the case $p\in (-1,0)$. Let us denote by $f_X$ the probability density function of X. By applying Lemma 4 to the values $-1,p,0$, we have

$$F\left(p\right)=F(-1·(-p)+0·(p+1))\ge F{(-1)}^{-p}F{\left(0\right)}^{p+1}.$$

From the proof of Theorem 1, we deduce that $F(-1)=f_X\left(0\right)$. In addition, notice that, for $\gamma \in (-1,0)$,

$$F\left(0\right)=\frac{1}{2}\frac{\mathrm{\Gamma }(-\frac{1}{\gamma })}{\mathrm{\Gamma }(-\frac{1}{\gamma }-1)}.$$

(104)

Hence,

$$f_X{\left(0\right)}^{-p}\le \frac{2^p{\parallel X\parallel }_p^p}{\mathrm{\Gamma }(p+1)}\frac{\mathrm{\Gamma }{(-1-\frac{1}{\gamma })}^{p+1}}{\mathrm{\Gamma }{(-\frac{1}{\gamma })}^p\mathrm{\Gamma }(-\frac{1}{\gamma }-(p+1))},$$

(105)

and the bound on differential entropy follows:

$$h\left(X\right)\ge log\frac{1}{f_X\left(0\right)}\ge \frac{1}{p}log\left(\frac{2^p{\parallel X\parallel }_p^p}{\mathrm{\Gamma }(p+1)}\frac{\mathrm{\Gamma }{(-1-\frac{1}{\gamma })}^{p+1}}{\mathrm{\Gamma }{(-\frac{1}{\gamma })}^p\mathrm{\Gamma }(-\frac{1}{\gamma }-(p+1))}\right).$$

(106)

For the case $p\in \left(0,-1-\frac{1}{\gamma }\right)$, the bound is obtained similarly by applying Lemma 4 to the values $-1,0,p$. ☐

5. Reverse Entropy Power Inequality with Explicit Constant

5.1. Proof of Theorem 7

Proof. 

Using the upper bound on the differential entropy (1), we have

$$h(X+Y)\le \frac{1}{2}log\left(2\pi e\mathrm{Var}[X+Y]\right)=\frac{1}{2}log\left(2\pi e(\mathrm{Var}\left[X\right]+\mathrm{Var}\left[Y\right])\right),$$

(107)

the last equality being valid since X and Y are uncorrelated. Hence,

$$N(X+Y)\le 2\pi e\left(\mathrm{Var}\right[X]+\mathrm{Var}[Y\left]\right).$$

(108)

Using inequality (32), we conclude that

$$N(X+Y)\le \frac{\pi e}{2}(N\left(X\right)+N\left(Y\right)).$$

(109)

☐

5.2. Proof of Theorem 8

Proof. 

Since X and Y are uncorrelated and $K_X$ and $K_Y$ are proportionals,

$$|K_{X+Y}{|}^{\frac{1}{n}}=|K_X+K_Y{|}^{\frac{1}{n}}=|K_X{|}^{\frac{1}{n}}+{\left|K_Y\right|}^{\frac{1}{n}}.$$

(110)

Using (110) and the upper bound on the differential entropy (38), we obtain

$$h(X+Y)\le \frac{n}{2}log\left(2\pi e|K_{X+Y}{|}^{\frac{1}{n}}\right)=\frac{n}{2}log\left(2\pi e\left(|K_X{|}^{\frac{1}{n}}+{\left|K_Y\right|}^{\frac{1}{n}}\right)\right).$$

(111)

Using Theorem 4, we conclude that

$$N(X+Y)\le 2\pi e\left(|K_X{|}^{\frac{1}{n}}+{\left|K_Y\right|}^{\frac{1}{n}}\right)\le 2\pi ec\left(n\right)(N\left(X\right)+N\left(Y\right)),$$

(112)

where $c\left(n\right)=\frac{e^2{n}^2}{4\sqrt{2}(n+2)}$ in general, and $c\left(n\right)=\frac{e^2}{2}$ if X and Y are unconditional. ☐

6. New Bounds on the Rate-Distortion Function

6.1. Proof of Theorem 9

Proof. 

Under mean-square error distortion ($r=2$), the result is implicit in [21] (Chapter 10). Denote for brevity $\sigma ={\parallel X\parallel }_2$.

(1) Let $r\in [1,2]$. Assume that $\sigma >d^{\frac{1}{r}}$. We take

$$\widehat{X}=\left(1-d^{\frac{2}{r}}/{\sigma }^2\right)\left(X+Z\right),$$

(113)

where $Z\sim \mathcal{N}\left(0,\frac{{\sigma }^2{d}^{\frac{2}{r}}}{{\sigma }^2-d^{\frac{2}{r}}}\right)$ is independent of X. This choice of $\widehat{X}$ is admissible since

$$\begin{array}{c}\hfill \parallel X-\widehat{X}{\parallel }_r^r\le {\parallel X-\widehat{X}\parallel }_2^r={\left[{\left(\frac{d^{\frac{2}{r}}}{{\sigma }^2}\right)}^2{\sigma }^2+{\left(1-\frac{d^{\frac{2}{r}}}{{\sigma }^2}\right)}^2{\parallel Z\parallel }_2^2\right]}^{\frac{r}{2}}=d,\end{array}$$

(114)

where we used $r\le 2$ and the left-hand side of inequality (26). Upper-bounding the rate-distortion function by the mutual information between X and $\widehat{X}$, we obtain

$$\begin{array}{c}\hfill {\mathbb{R}}_X\left(d\right)\le I(X;\widehat{X})=h(X+Z)-h\left(Z\right),\end{array}$$

(115)

where we used homogeneity of differential entropy for the last equality. Invoking the upper bound on the differential entropy (1), we have

$$h(X+Z)-h\left(Z\right)\le \frac{1}{2}log\left(2\pi e\left({\sigma }^2+\frac{{\sigma }^2{d}^{\frac{2}{r}}}{{\sigma }^2-d^{\frac{2}{r}}}\right)\right)-h\left(Z\right)={\underset{\_}{\mathbb{R}}}_X\left(d\right)+D\left(X\right||G_X)+log\frac{{\alpha }_r}{\sqrt{2\pi e}},$$

(116)

and (50) follows.

If ${\parallel X\parallel }_2\le d^{\frac{1}{r}}$, then ${\parallel X\parallel }_r\le {\parallel X\parallel }_2\le d^{\frac{1}{r}}$, and setting $\widehat{X}\equiv 0$ leads to ${\mathbb{R}}_X\left(d\right)=0$.

(2) Let $r>2$. The argument presented here works for every $r\ge 1$. However, for $r\in [1,2]$, the argument in part (1) provides a tighter bound. Assume that $\sigma \ge d^{\frac{1}{r}}$. We take

$$\widehat{X}=X+Z,$$

(117)

where Z is independent of X and realizes the maximum differential entropy under the r-th moment constraint, ${\parallel Z\parallel }_r^r=d$. The probability density function of Z is given by

$$f_Z\left(x\right)=\frac{r^{1-\frac{1}{r}}}{2\mathrm{\Gamma }\left(\frac{1}{r}\right)d^{\frac{1}{r}}}{e}^{-\frac{{\left|x\right|}^r}{rd}},x\in \mathbb{R}.$$

(118)

Notice that

$${\parallel Z\parallel }_2^2=d^{\frac{2}{r}}\frac{r^{\frac{2}{r}}\mathrm{\Gamma }\left(\frac{3}{r}\right)}{\mathrm{\Gamma }\left(\frac{1}{r}\right)}.$$

(119)

We have

$$\begin{array}{cc}\hfill h(X+Z)-h\left(Z\right)\le & \frac{1}{2}log\left(2\pi e\right({\sigma }^2+{\parallel Z\parallel }_2^2\left)\right)-log\left({\alpha }_r{d}^{\frac{1}{r}}\right)\hfill \end{array}$$

(120)

$$\begin{array}{cc}\hfill \le & {\underset{\_}{\mathbb{R}}}_X\left(d\right)+log\left(\sqrt{2\pi e}{\beta }_r\sigma \right)-h\left(X\right),\hfill \end{array}$$

(121)

where ${\beta }_r$ is defined in (49). Hence,

$${\mathbb{R}}_X\left(d\right)-{\underset{\_}{\mathbb{R}}}_X\left(d\right)\le D\left(X\right||G_X)+log{\beta }_r.$$

(122)

If ${\parallel X\parallel }_r^r\le d$, then setting $\widehat{X}\equiv 0$ leads to ${\mathbb{R}}_X\left(d\right)=0$. Finally, if ${\parallel X\parallel }_r^r>d$ and $\sigma <d^{\frac{1}{r}}$, then, from (120), we obtain

$${\mathbb{R}}_X\left(d\right)\le log\left(\sqrt{2\pi e}{\beta }_r{d}^{\frac{1}{r}}\right)-log\left({\alpha }_r{d}^{\frac{1}{r}}\right)=log\frac{\sqrt{2\pi e}{\beta }_r}{{\alpha }_r}.$$

(123)

☐

6.2. Proof of Theorem 10

Proof. 

Denote for brevity $\sigma ={\parallel X\parallel }_2$, and recall that X is a symmetric log-concave random variable.

Assume that $\sigma \ge d^{\frac{1}{r}}$. We take

$$\begin{array}{c}\hfill \widehat{X}=\left(1-\frac{\delta }{{\sigma }^2}\right)(X+Z),\delta \triangleq \frac{2}{\mathrm{\Gamma }{(r+1)}^{\frac{2}{r}}}{d}^{\frac{2}{r}},\end{array}$$

(124)

where $Z\sim \mathcal{N}\left(0,\frac{{\sigma }^2\delta }{{\sigma }^2-\delta }\right)$ is independent of X. This choice of $\widehat{X}$ is admissible since

$$\begin{array}{c}\hfill \parallel X-\widehat{X}{\parallel }_r^r\le {\parallel X-\widehat{X}\parallel }_2^r\frac{\mathrm{\Gamma }(r+1)}{2^{\frac{r}{2}}}={\delta }^{\frac{r}{2}}\frac{\mathrm{\Gamma }(r+1)}{2^{\frac{r}{2}}}=d,\end{array}$$

(125)

where we used $r>2$ and Theorem 2. Using the upper bound on the differential entropy (1), we have

$$\begin{array}{c}\hfill h(X+Z)-h\left(Z\right)\le \frac{1}{2}log\left(2\pi e\left({\sigma }^2+\frac{{\sigma }^2\delta }{{\sigma }^2-\delta }\right)\right)-h\left(Z\right)=\frac{1}{2}log\frac{{\sigma }^2}{\delta }.\end{array}$$

(126)

Hence,

$${\mathbb{R}}_X\left(d\right)-\underset{-}{\mathbb{R}}{}_X\left(d\right)\le D\left(X\right||G_X)+log\frac{{\alpha }_r\mathrm{\Gamma }{(r+1)}^{\frac{1}{r}}}{2\sqrt{\pi e}}.$$

(127)

If ${\sigma }^2\le \delta$, then from Theorem 2 ${\parallel X\parallel }_r^r\le d$, hence ${\mathbb{R}}_X\left(d\right)=0$. Finally, if ${\parallel X\parallel }_r^r>d$ and ${\sigma }^2\in (\delta ,d^{\frac{2}{r}})$, then, from (126), we obtain

$$\begin{array}{c}\hfill {\mathbb{R}}_X\left(d\right)\le \frac{1}{2}log\frac{{\sigma }^2}{\delta }\le \frac{1}{2}log\frac{\mathrm{\Gamma }{(r+1)}^{\frac{2}{r}}}{2}.\end{array}$$

(128)

☐

Remark 7.

1) Let us explain the strategy in the proof of Theorems 9 and 10. By definition, ${\mathbb{R}}_X\left(d\right)\le I(X;\widehat{X})$ for any $\widehat{X}$ satisfying the constraint. In our study, we chose $\widehat{X}$ of the form $\lambda (X+Z)$, with $\lambda \in [0,1]$, where Z is independent of X. To find the best bounds possible with this choice of $\widehat{X}$, we need to minimize $\parallel X-\widehat{X}{\parallel }_r^r$ over λ. Notice that if $\widehat{X}=\lambda (X+Z)$ and Z symmetric, then $\parallel X-\widehat{X}{\parallel }_r^r={\parallel (1-\lambda )X+\lambda Z\parallel }_r^r$.

To estimate ${\parallel (1-\lambda )X+\lambda Z\parallel }_r^r$ in terms of ${\parallel X\parallel }_r$ and ${\parallel Z\parallel }_r$, one can use triangle inequality and the convexity of ${\parallel ·\parallel }^r$ to get the bound

$${\parallel (1-\lambda )X+\lambda Z\parallel }_r^r\le 2^{r-1}{((1-\lambda )}^r{\parallel X\parallel }_r^r+{\lambda }^r{\parallel Z\parallel }_r^r),$$

(129)

or one can apply Jensen’s inequality directly to get the bound

$${\parallel (1-\lambda )X+\lambda Z\parallel }_r^r\le {(1-\lambda )\parallel X\parallel }_r^r+\lambda {\parallel Z\parallel }_r^r.$$

(130)

A simple study shows that (130) provides a tighter bound over (129). This justifies choosing $\widehat{X}$ as in (117) in the proof of (51).

To justify the choice of $\widehat{X}$ in (113) (also in (124)), which leads the tightening of (51) for $r\in [1,2]$ in (50) (also in (52)), we bound r-th norm by second norm, and we note that by the independence of X and Z,

$${\parallel (1-\lambda )X+\lambda Z\parallel }_2^2\le {(1-\lambda )}^2{\parallel X\parallel }_2^2+{\lambda }^2{\parallel Z\parallel }_2^2.$$

(131)

A simple study shows that (131) provides a tighter bound over (130).

2) Using Corollary 2, if $r=2$, one may rewrite our bound in terms of the rate-distortion function of a Gaussian source as follows:

$$\begin{array}{c}\hfill {\mathbb{R}}_X\left(d\right)\ge {\mathbb{R}}_{G_X}\left(d\right)-log\sqrt{\pi e}-{\Delta }_p,\end{array}$$

(132)

where ${\Delta }_p$ is defined in (28), and where

$${\mathbb{R}}_{G_X}\left(d\right)=\frac{1}{2}log\frac{{\sigma }^2}{d}$$

(133)

is the rate-distortion function of a Gaussian source with the same variance ${\sigma }^2$ as X. It is well known that for arbitrary source and mean-square distortion (see e.g., [21] (Chapter 10))

$${\mathbb{R}}_X\left(d\right)\le {\mathbb{R}}_{G_X}\left(d\right).$$

(134)

By taking $p=2$ in (132), we obtain

$$0\le {\mathbb{R}}_{G_X}\left(d\right)-{\mathbb{R}}_X\left(d\right)\le \frac{1}{2}log\left(\frac{\pi e}{2}\right).$$

(135)

The bounds in (134) and (135) tell us that the rate-distortion function of any log-concave source is approximated by that of a Gaussian source. In particular, approximating ${\mathbb{R}}_X\left(d\right)$ of an arbitrary log-concave source by

$${\widehat{\mathbb{R}}}_X\left(d\right)=\frac{1}{2}log\frac{{\sigma }^2}{d}-\frac{1}{4}log\left(\frac{\pi e}{2}\right),$$

(136)

we guarantee the approximation error $|{\mathbb{R}}_X\left(d\right)-{\widehat{\mathbb{R}}}_X\left(d\right)|$ of at most $\frac{1}{4}log\left(\frac{\pi e}{2}\right)\approx \frac{1}{2}$ bits.

6.3. Proof of Theorem 11

Proof. 

If $|K_X{|}^{\frac{1}{n}}>d$, then we choose $\widehat{X}=\left(1-\frac{d}{|K_X{|}^{\frac{1}{n}}}\right)(X+Z)$, where $Z\sim \mathcal{N}\left(0,\frac{d}{|K_X{|}^{\frac{1}{n}}-d}·K_X\right)$ is independent of X. This choice is admissible by independence of X and Z and the fact that $K_X$ and $K_Z$ are proportionals. Upper-bounding the rate-distortion function by the mutual information between X and $\widehat{X}$, we have

$$\begin{array}{c}\hfill {\mathbb{R}}_X^{cov}\left(d\right)\le h(X+Z)-h\left(Z\right)\le \frac{n}{2}log\frac{|K_X{|}^{\frac{1}{n}}}{d}.\end{array}$$

(137)

Since the Shannon lower bound for determinant constraint coincides with that for the mean-square error constraint,

$${\mathbb{R}}_X^{cov}\left(d\right)\ge \underset{-}{\mathbb{R}}{}_X\left(d\right)=h\left(X\right)-\frac{n}{2}log\left(2\pi ed\right).$$

(138)

On the other hand, using (137), we have

$$\begin{array}{c}\hfill {\mathbb{R}}_X^{cov}\left(d\right)-\underset{-}{\mathbb{R}}{}_X\left(d\right)\le D\left(X\right||G_X)\le \frac{n}{2}log\left(2\pi ec\left(n\right)\right),\end{array}$$

(139)

where (139) follows from Corollary 3.

If $|K_X{|}^{\frac{1}{n}}\le d$, then we put $\widehat{X}\equiv 0$, which leads to ${\mathbb{R}}_X^{cov}\left(d\right)=0$. ☐

7. New Bounds on the Capacity of Memoryless Additive Channels

Recall that the capacity of such a channel is

$${\mathbb{C}}_Z\left(P\right)=\underset{X:\frac{1}{n}{\parallel X\parallel }_2^2\le P}{sup}I(X;X+Z)=\underset{X:\frac{1}{n}{\parallel X\parallel }_2^2\le P}{sup}h(X+Z)-h\left(Z\right).$$

(140)

We compare the capacity ${\mathbb{C}}_Z$ of a channel with log-concave additive noise with the capacity of the Gaussian channel.

7.1. Proof of Theorem 12

Proof. 

The lower bound is well known, as mentioned in (19). To obtain the upper bound, we first use the upper bound on the differential entropy (1) to conclude that

$$h(X+Z)\le \frac{1}{2}log\left(2\pi e(P+\mathrm{Var}\left[Z\right])\right),$$

(141)

for every random variable X such that ${\parallel X\parallel }_2^2\le P$. By combining (140), (141) and (32), we deduce that

$${\mathbb{C}}_Z\left(P\right)\le \frac{1}{2}log\left(2\pi e(P+\mathrm{Var}\left[Z\right])\right)-\frac{1}{2}log\left(4\mathrm{Var}\left[Z\right]\right)=\frac{1}{2}log\left(\frac{\pi e}{2}\left(1+\frac{P}{\mathrm{Var}\left[Z\right]}\right)\right),$$

(142)

which is the desired result. ☐

7.2. Proof of Theorem 13

Proof. 

The lower bound is well known, as mentioned in (19). To obtain the upper bound, we write

$$h(X+Z)-h\left(Z\right)\le \frac{n}{2}log\left(2\pi e|K_{X+Z}{|}^{\frac{1}{n}}\right)-h\left(Z\right)\le \frac{n}{2}log\left(2\pi ec\left(n\right)\left(\frac{\frac{1}{n}{\parallel Z\parallel }_2^2}{|K_Z{|}^{\frac{1}{n}}}+\frac{P}{|K_Z{|}^{\frac{1}{n}}}\right)\right),$$

(143)

where $c\left(n\right)=\frac{n^2{e}^2}{(n+2)4\sqrt{2}}$ in general, and $c\left(n\right)=\frac{e^2}{2}$ if Z is unconditional. The first inequality in (143) is obtained from the upper bound on the differential entropy (38). The last inequality in (143) is obtained by applying the arithmetic-geometric mean inequality and Theorem 4. ☐

8. Conclusions

Several recent results show that the entropy of log-concave probability densities have nice properties. For example, reverse, strengthened and stable versions of the entropy power inequality were recently obtained for log-concave random vectors (see e.g., [3,11,35,36,37,38]). This line of developments suggest that, in some sense, log-concave random vectors behave like Gaussians.

Our work follows this line of results, by establishing a new lower bound on differential entropy for log-concave random variables in (4), for log-concave random vectors with possibly dependent coordinates in (37), and for $\gamma$-concave random variables in (43). We made use of the new lower bounds in several applications. First, we derived reverse entropy power inequalities with explicit constants for uncorrelated, possibly dependent log-concave random vectors in (12) and (47). We also showed a universal bound on the difference between the rate-distortion function and the Shannon lower bound for log-concave random variables in Figure 1a and Figure 1b, and for log-concave random vectors in (59). Finally, we established an upper bound on the capacity of memoryless additive noise channels when the noise is a log-concave random vector in (20) and (66).

Under the Gaussian assumption, information-theoretic limits in many communication scenarios admit simple closed-form expressions. Our work demonstrates that, at least in three such scenarios (source coding, channel coding and joint source-channel coding), the information-theoretic limits admit a closed-form approximation with at most 1 bit of error if the Gaussian assumption is relaxed to the log-concave one. We hope that the approach will be useful in gaining insights into those communication and data processing scenarios in which the Gaussianity of the observed distributions is violated but the log-concavity is preserved.