An Integral Representation of the Relative Entropy

Abstract

Recently the identity of de Bruijn type between the relative entropy and the relative Fisher information with the reference moving has been unveiled by Verdú via MMSE in estimation theory. In this paper, we shall give another proof of this identity in more direct way that the derivative is calculated by applying integrations by part with the heat equation. We shall also derive an integral representation of the relative entropy, as one of the applications of which the logarithmic Sobolev inequality for centered Gaussian measures will be given.

1. Introduction

Probability measures on ${\mathbb{R}}^n$ treated in this paper are absolutely continuous with respect to the standard Lebesgue measure and we shall identify them with their densities.

For a probability measure f, the entropy $H\left(f\right)$ and the Fisher information $J\left(f\right)$ can be introduced, which play important roles in information theory, probability, and statistics. For more details on these subjects see the famous book [1].

Hereafter, for an n-variables function $\varphi \left(\mathit{x}\right)=\varphi (x_1,x_2,\dots ,x_n)$ on ${\mathbb{R}}^n$, the integral of ϕ over the whole ${\mathbb{R}}^n$ by the standard Lebesgue measure $d\mathit{x}=d{x}_{1}d{x}_2\cdots d{x}_n$ is abbreviated as

$${\int }_{{\mathbb{R}}^n}\varphi d\mathit{x}=\int \int \cdots {\int }_{{\mathbb{R}}^n}\varphi (x_1,x_2,\dots ,x_n)d{x}_{1}d{x}_2\cdots d{x}_n.$$

that is, we shall leave out $(x_1,x_2,\dots ,x_n)$ in the integrand in order to simplify the expressions.

Definition 1.1. Let f be a probability measure on ${\mathbb{R}}^n$. Then the (differential) entropy of f is defined by

$$H\left(f\right)=-{\int }_{{\mathbb{R}}^n}flogfd\mathit{x}.$$

For a random variable $\mathit{X}$ on ${\mathbb{R}}^n$ with the density f, we write the entropy of $\mathit{X}$ by $H\left(\mathit{X}\right)=H\left(f\right)$.

The Fisher information for a differentiable density f is defined by

$$J\left(f\right)={\int }_{{\mathbb{R}}^n}\frac{\parallel \nabla f{\parallel }^2}{f}d\mathit{x}={\int }_{{\mathbb{R}}^n}f\parallel \nabla (logf){\parallel }^{2}d\mathit{x}.$$

When the random variable $\mathit{X}$ on ${\mathbb{R}}^n$ has the differentiable density f, we also write as $J\left(\mathit{X}\right)=J\left(f\right)$.

The important result for a behavior of the Fisher information on convolution (sum of independent random variables) is the Stam inequality, which was first stated by Stam in [2] and subsequently proved by Blachman [3],

$$\frac{1}{J\left(f*g\right)}\ge \frac{1}{J\left(f\right)}+\frac{1}{J\left(g\right)}$$

(1)

where we have the equality if and only if f and g are Gaussian.

The importance of the Stam inequality can be found in its applications, for instance, the entropy power inequality [2]; the logarithmic Sobolev inequality [4]; Cercignani conjecture [5]; the Shannon conjecture on entropy and the central limit theorem [6,7].

For $t\ge 0$, we denote by $P_{t}f$ the convolution of f with the n-dimensional Gaussian density with mean vector $\mathbf{0}$ and covariance matrix $t{I}_n$, where $I_n$ is the identity matrix. Namely, ${\left(P_t\right)}_{t\ge 0}$ is the heat semigroup acting on f and satisfies the partial differential equation

$$\frac{\partial }{\partial t}{P}_{t}f=\frac{1}{2}\Delta \left(P_{t}f\right)$$

(2)

which is called the heat equation. In this paper, we simply denote $P_{t}f$ by $f_t$ and call it the Gaussian perturbation of f. Namely, letting $\mathit{X}$ be the random variable on ${\mathbb{R}}^n$ with the density f and $\mathit{Z}$ be an n-dimensional Gaussian random variable independent of $\mathit{X}$ with mean vector $\mathbf{0}$ and covariance matrix $I_n$, the Gaussian perturbation $f_t$ stands the density function $f(\mathit{x},t)$ of the independent sum $\mathit{X}+\sqrt{t}\mathit{Z}$.

The remarkable relation between the entropy and the Fisher information can be established by a Gaussian perturbation (see, for instance, [1], [2] or [8]);

$${\displaystyle \frac{d}{dt}}H\left(f_t\right)=\frac{1}{2}J\left(f_t\right)\text{for}t>0$$

(3)

which is known as the de Bruijn identity.

Let f and g be probability measures on ${\mathbb{R}}^n$ such that $f\ll g$ (f is absolutely continuous with respect to g). Setting the probability measure g as a reference, the relative entropy and the relative Fisher information can be introduced as follows:

Definition 1.2. The relative entropy of f with respect to g, $D(f\parallel g)$ is defined by

$$D(f\parallel g)={\int }_{{\mathbb{R}}^n}f\left(log{\displaystyle \frac{f}{g}}\right)d\mathit{x}={\int }_{{\mathbb{R}}^n}flogfd\mathit{x}-{\int }_{{\mathbb{R}}^n}floggd\mathit{x},$$

which takes always a non-negative value.

We also define the relative Fisher information of f with respect to g by

$$J(f\parallel g)={\int }_{{\mathbb{R}}^n}f\parallel \nabla \left(log\frac{f}{g}\right){\parallel }^{2}d\mathit{x}={\int }_{{\mathbb{R}}^n}f\parallel \nabla (logf)-\nabla (logg){\parallel }^{2}d\mathit{x},$$

which is also non-negative. When random variables $\mathit{X}$ and $\mathit{Y}$ have the densities f and g, respectively, the relative entropy and the relative Fisher information of $\mathit{X}$ with respect to $\mathit{Y}$ are defined by $D(\mathit{X}\parallel \mathit{Y})=D(f\parallel g)$ and $J(\mathit{X}\parallel \mathit{Y})=J(f\parallel g)$, respectively.

In view of the de Bruijn identity, one might expect that there is a similar connection between the relative entropy and the relative Fisher information. Indeed, the gradient formulas for the relative entropy functionals were obtained in [9,10,11], where the reference measures would not be changed in their cases.

Recently Verdú in [12], however, investigated the derivative in t of $D(f_t\parallel g_t)$ for two Gaussian perturbations $f_t$ and $g_t$. Here we should note that the reference measure does move by the same time parameter in this case. The following identity of de Bruijn type

$$\frac{d}{dt}D(f_t\parallel g_t)=-\frac{1}{2}J(f_t\parallel g_t)$$

has been derived via MMSE in estimation theory (see also [13], for general perturbations).

The main aim in this paper is that we shall give an alternative proof of this identity by direct calculation with integrations by part, the method of which is similar to ones in [11,14]. Moreover, it will be easily found that the above identity yields an integral representation of the relative entropy. We shall also see the simple proof of the logarithmic Sobolev inequality for centered Gaussian in univariate ($n=1$) case as an application of the integral representation.

2. An Integral Representation of the Relative Entropy

We shall make the Gaussian perturbations $f_t$ and $g_t$, respectively, and consider the relative entropy $D(f_t\parallel g_t)$, where the absolute continuity $f_t\ll g_t$ remains true for $t>0$.

Here, we regard $D(f_t\parallel g_t)$ as a function of t and calculate the derivative,

$$\frac{d}{dt}D(f_t\parallel g_t)=\frac{d}{dt}{\int }_{{\mathbb{R}}^n}{f}_{t}log\frac{f_t}{g_t}d\mathit{x}=\frac{d}{dt}{\int }_{{\mathbb{R}}^n}{f}_{t}log{f}_{t}d\mathit{x}-\frac{d}{dt}{\int }_{{\mathbb{R}}^n}{f}_{t}log{g}_{t}d\mathit{x}$$

(4)

by integrations by part with help of the heat equation.

Proposition 2.1. Let $f\ll g$ be probability measures on ${\mathbb{R}}^n$ with finite Fisher informations $J\left(f\right)<\infty$ and $J\left(g\right)<\infty$, and finite relative entropy $D(f\parallel g)<\infty$. Then we obtain

$$\frac{d}{dt}D(f_t\parallel g_t)=-\frac{1}{2}J(f_t\parallel g_t)fort>0.$$

Proof. First we should notice that the Fisher informations $J\left(f_t\right)$ and $J\left(g_t\right)$ are finite at any $t>0$. Because, for instance, if an n-dimensional random variable $\mathit{X}$ has the density f and $\mathit{Z}$ is an n-dimensional Gaussian random variable independent of $\mathit{X}$ with mean vector $\mathbf{0}$ and covariance matrix $I_n$, then by applying the Stam inequality (1) to independent random variables $\mathit{X}$ and $\sqrt{t}\mathit{Z}$, we have that

$$J\left(f_t\right)=J\left(\mathit{X}+\sqrt{t}\mathit{Z}\right)\le {\left({\displaystyle \frac{1}{J\left(\mathit{X}\right)}+\frac{1}{J\left(\sqrt{t}\mathit{Z}\right)}}\right)}^{-1}={\displaystyle \frac{J\left(\mathit{X}\right)}{1+\frac{t}{n}J\left(\mathit{X}\right)}}\le J\left(f\right)<\infty$$

(5)

where $J\left(\mathit{Z}\right)=n$ is by simple calculation. We shall also notice that the function $D(f_t\parallel g_t)$ is non-increasing in t, that is, for $t>0$,

$$0\le D(f_t\parallel g_t)\le D(f\parallel g)<\infty ,$$

which can be found in [15] (p. 101). Therefore, $D(f_t\parallel g_t)$ is finite for $t>0$. But by a nonlinear approximation argument in [11], we can impose a stronger assumption without loss of generality that

$$“\text{the relative density }{\displaystyle {\displaystyle \frac{f_t}{g_t}}}\text{ is bounded away from }0\text{ and }\infty \text{ on }{\mathbb{R}}^n”$$

(6)

Concerning the first term in the most right hand side of (4), it follows immediately that

$$\frac{d}{dt}{\int }_{{\mathbb{R}}^n}{f}_{t}log{f}_{t}d\mathit{x}=-\frac{1}{2}{\int }_{{\mathbb{R}}^n}\frac{\parallel \nabla f_t{\parallel }^2}{f_t}d\mathit{x}$$

(7)

by the de Bruijn identity (3), hence, we shall concentrate our attention upon the second term.

Since the densities $f_t$ and $g_t$ satisfy the heat equation (2), the second term can be reformulated as follows:

$$\begin{array}{ccc}\hfill {\displaystyle \frac{d}{dt}}{\int }_{{\mathbb{R}}^n}{f}_{t}log{g}_{t}d\mathit{x}& =& {\int }_{{\mathbb{R}}^n}{f}_t\left({\partial }_{t}log{g}_t\right)d\mathit{x}+{\int }_{{\mathbb{R}}^n}log{g}_t\left({\partial }_t{f}_t\right)d\mathit{x}\hfill \end{array}$$

$$\begin{array}{ccc}& =& {\int }_{{\mathbb{R}}^n}{f}_t\frac{{\partial }_t{g}_t}{g_t}d\mathit{x}+{\int }_{{\mathbb{R}}^n}log{g}_t\left(\frac{1}{2}\Delta f_t\right)d\mathit{x}\hfill \end{array}$$

$$\begin{array}{ccc}& =& {\int }_{{\mathbb{R}}^n}\frac{f_t}{g_t}\left(\frac{1}{2}\Delta g_t\right)d\mathit{x}+{\int }_{{\mathbb{R}}^n}log{g}_t\left(\frac{1}{2}\Delta f_t\right)d\mathit{x}\hfill \end{array}$$

(8)

In this reformulation, we have changed integration and differentiation at the first equality, which is justified by a routine argument with the bounded convergence theorem (see, for instance, [16]).

Applying integration by part to the first term in the last expression of (8), it becomes

$${\int }_{{\mathbb{R}}^n}\frac{f_t}{g_t}\left(\frac{1}{2}\Delta g_t\right)d\mathit{x}=-\frac{1}{2}{\int }_{{\mathbb{R}}^n}\nabla \left(\frac{f_t}{g_t}\right)·\nabla g_{t}d\mathit{x}$$

(9)

which can be asserted by the observation below. As $g_t$ has finite Fisher information $J\left(g_t\right)<\infty$, ${\displaystyle \frac{\nabla g_t}{\sqrt{g_t}}}$ has finite 2-norm in $L^2\left({\mathbb{R}}^n\right)$ and must be bounded at infinity. Furthermore, from our technical assumption (6), ${\displaystyle \sqrt{\frac{f_t}{g_t}}}$ is also bounded. Hence if we factorize as

$$\frac{f_t}{g_t}\left(\nabla g_t\right)=\sqrt{f_t}{\displaystyle \sqrt{\frac{f_t}{g_t}}}\frac{\nabla g_t}{\sqrt{g_t}},$$

then it can be found that ${\displaystyle \frac{f_t}{g_t}\left(\nabla g_t\right)}$ will vanish at infinity.

Applying integration by part to the second term in the last expression of (8), it becomes

$${\int }_{{\mathbb{R}}^n}log{g}_t\left(\frac{1}{2}\Delta f_t\right)d\mathit{x}=-\frac{1}{2}{\int }_{{\mathbb{R}}^n}\frac{\nabla g_t}{g_t}·\nabla f_{t}d\mathit{x}$$

(10)

Here it should be noted that $log{g}_t(\nabla f_t)$ will vanish at infinity by the following observation. Similarly, we factorize it as

$$log{g}_t(\nabla f_t)=2\left(\sqrt{g_t}log\sqrt{g_t}\right){\displaystyle \sqrt{\frac{f_t}{g_t}}}\left({\displaystyle \frac{\nabla f_t}{\sqrt{f_t}}}\right).$$

Then the boundedness of ${\displaystyle \frac{\nabla f_t}{\sqrt{f_t}}}$ comes from that $J\left(f_t\right)<\infty$, and one of ${\displaystyle \sqrt{\frac{f_t}{g_t}}}$ is by the assumption (6) same as before. Furthermore, the limit formula ${\displaystyle \underset{\xi \to 0}{lim}\xi log\xi =0}$ ensures that $\left(\sqrt{g_t}log\sqrt{g_t}\right)$ will vanish at infinity.

Substitute the Equation (9) and Equation (10) into (8), it follows that

$$\begin{array}{ccc}\hfill {\displaystyle \frac{d}{dt}}{\int }_{{\mathbb{R}}^n}{f}_{t}log{g}_{t}d\mathit{x}& =& -\frac{1}{2}{\int }_{{\mathbb{R}}^n}\nabla \left(\frac{f_t}{g_t}\right)·\nabla g_{t}d\mathit{x}-\frac{1}{2}{\int }_{{\mathbb{R}}^n}\frac{\nabla g_t}{g_t}·\nabla f_{t}d\mathit{x}\hfill \end{array}$$

$$\begin{array}{ccc}& =& -\frac{1}{2}{\int }_{{\mathbb{R}}^n}\left(\frac{\nabla f_t}{g_t}-f_t\frac{\nabla g_t}{g_t^2}\right)·\nabla g_{t}d\mathit{x}-\frac{1}{2}{\int }_{{\mathbb{R}}^n}{f}_t\frac{\nabla g_t}{g_t}·\frac{\nabla f_t}{f_t}d\mathit{x}\hfill \end{array}$$

$$\begin{array}{ccc}& =& -{\int }_{{\mathbb{R}}^n}{f}_t\frac{\nabla g_t}{g_t}·\frac{\nabla f_t}{f_t}d\mathit{x}+\frac{1}{2}{\int }_{{\mathbb{R}}^n}{f}_t\frac{\nabla g_t}{g_t}·\frac{\nabla g_t}{g_t}d\mathit{x}\hfill \end{array}$$

(11)

Combining the Equation (7) and Equation (11), we have that

$$\begin{array}{cc}\hfill {\displaystyle \frac{d}{dt}}& {\int }_{{\mathbb{R}}^n}{f}_{t}log{f}_{t}d\mathit{x}-{\displaystyle \frac{d}{dt}}{\int }_{{\mathbb{R}}^n}{f}_{t}log{g}_{t}d\mathit{x}\hfill \\ & =-\frac{1}{2}{\int }_{{\mathbb{R}}^n}{f}_t\frac{\nabla f_t}{f_t}·\frac{\nabla f_t}{f_t}d\mathit{x}+{\int }_{{\mathbb{R}}^n}{f}_t\frac{\nabla g_t}{g_t}·\frac{\nabla f_t}{f_t}d\mathit{x}-\frac{1}{2}{\int }_{{\mathbb{R}}^n}{f}_t\frac{\nabla g_t}{g_t}·\frac{\nabla g_t}{g_t}d\mathit{x}\hfill \\ & =-\frac{1}{2}{\int }_{{\mathbb{R}}^n}{f}_t\parallel \frac{\nabla f_t}{f_t}-\frac{\nabla g_t}{g_t}{\parallel }^{2}d\mathit{x}\hfill \end{array}$$

which means

$$\frac{d}{dt}D(f_t\parallel g_t)=-\frac{1}{2}{\int }_{{\mathbb{R}}^n}{f}_t\parallel \nabla (log{f}_t)-\nabla (log{g}_t){\parallel }^{2}d\mathit{x}=-\frac{1}{2}J(f_t\parallel g_t).$$

Let $\mathit{X}$ and $\mathit{Y}$ be n-dimensional random variables with the densities f and g, respectively, and $\mathit{Z}$ be an n-dimensional Gaussian random variable independent of $\mathit{X}$ and $\mathit{Y}$ with mean vector $\mathbf{0}$ and covariance matrix $I_n$.

Since the relative entropy is scale invariant, it follows that

$$D(\mathit{X}+\sqrt{t}\mathit{Z}\parallel \mathit{Y}+\sqrt{t}\mathit{Z})=D\left(\frac{1}{\sqrt{t}}\mathit{X}+\mathit{Z}\parallel \frac{1}{\sqrt{t}}\mathit{Y}+\mathit{Z}\right).$$

We know that both of $\frac{1}{\sqrt{t}}X+Z$ and $\frac{1}{\sqrt{t}}Y+Z$, as $t\to \infty$ converge to Z in distribution. Thus, we have

$$\underset{t\to \infty }{lim}D(f_t\parallel g_t)=0,$$

and the following integral representation for the relative entropy can be obtained:

Theorem 2.2. Let $f\ll g$ be probability measures with finite Fisher informations and finite relative entropy $D(f\parallel g)$. Then we have the integral representation,

$$D(f\parallel g)=\frac{1}{2}{\int }_0^{\infty }J(f_t\parallel g_t)dt.$$

3. An Application to the Logarithmic Sobolev Inequality

In this section, we shall give a proof of the logarithmic Sobolev inequality for a centered Gaussian measure in case of $n=1$. Although several proofs of the logarithmic Sobolev inequality have already been given in many literatures (see, for instance, [10,17]), we shall give it here again as an application of the integral representation in Theorem 2.2.

Theorem 3.1. Let g be the centered Gaussian measure of variance ${\sigma }^2$. Then for any probability measure f on $\mathbb{R}$ of finite moment of order 2 with finite Fisher information $J\left(f\right)<\infty$, the following inequality holds:

$$D(f\parallel g)\le \frac{{\sigma }^2}{2}J(f\parallel g).$$

Proof. It is clear that the perturbed measure $g_t$ is the centered Gaussian of variance ${\sigma }^2+t$ and the score of which is given by

$$\left({\partial }_{x}log{g}_t\right)=-\frac{x}{{\sigma }^2+t}.$$

Then using the Stein relation (see, for instance, [15]), the relative Fisher information $J(f_t\parallel g_t)$ can be expanded as follows:

$$\begin{array}{ccc}\hfill J(f_t\parallel g_t)& =& {\int }_{\mathbb{R}}{\left\{\left({\partial }_{x}log{f}_t\right)-\left({\partial }_{x}log{g}_t\right)\right\}}^2{f}_{t}dx\hfill \end{array}$$

$$\begin{array}{ccc}& =& J\left(f_t\right)+2{\int }_{\mathbb{R}}\left\{{\partial }_x\left(-\frac{x}{{\sigma }^2+t}\right)\right\}{f}_{t}dx+{\int }_{\mathbb{R}}{\left(-\frac{x}{{\sigma }^2+t}\right)}^2{f}_{t}dx\hfill \end{array}$$

$$\begin{array}{ccc}& =& J\left(f_t\right)-\frac{2}{{\sigma }^2+t}{\int }_{\mathbb{R}}{f}_{t}dx+\frac{1}{{({\sigma }^2+t)}^2}{\int }_{\mathbb{R}}{x}^2{f}_{t}dx\hfill \end{array}$$

(12)

As it was seen in (5), by Stam inequality, we have that

$$J\left(f_t\right)\le {\left({\displaystyle \frac{1}{J\left(f\right)}+t}\right)}^{-1}=\frac{1}{(1/\alpha )+t}$$

(13)

where we put $\alpha =J\left(f\right)<\infty$.

Since f has finite moment of order 2, if we put the second moment of f as $\beta =m_2\left(f\right)<\infty$, then it is easy to see that the second moment of $f_t$ is given by

$$m_2\left(f_t\right)=\int x^2{f}_{t}dx=\beta +t$$

(14)

Substitute (13) and (14) into (12) and we obtain that

$$J(f_t\parallel g_t)\le \frac{1}{(1/\alpha )+t}-\frac{2}{{\sigma }^2+t}+\frac{\beta +t}{{({\sigma }^2+t)}^2}=\frac{1}{(1/\alpha )+t}-\frac{1}{{\sigma }^2+t}+\frac{\beta -{\sigma }^2}{{({\sigma }^2+t)}^2}.$$

Integrating for $t\ge 0$, we have

$$\begin{array}{cc}\hfill \frac{1}{2}{\int }_0^{\infty }J(f_t\parallel g_t)dt& \le \frac{1}{2}{\int }_0^{\infty }\left(\frac{1}{(1/\alpha )+t}-\frac{1}{{\sigma }^2+t}+\frac{\beta -{\sigma }^2}{{({\sigma }^2+t)}^2}\right)dt\hfill \\ & =\frac{1}{2}{\left[log\left(\frac{(1/\alpha )+t}{{\sigma }^2+t}\right)-\frac{\beta -{\sigma }^2}{{\sigma }^2+t}\right]}_0^{\infty }\hfill \\ & =\frac{1}{2}\left(log\left({\sigma }^2\alpha \right)+\frac{\beta }{{\sigma }^2}-1\right).\hfill \end{array}$$

Since $logy$ is dominated as $logy\le y-1$ for $y>0$, it follows that

$$\frac{1}{2}{\int }_0^{\infty }J(f_t\parallel g_t)dt\le \frac{1}{2}\left({\sigma }^2\alpha -2+\frac{\beta }{{\sigma }^2}\right)$$

(15)

On the other hand, the relative Fisher information $J(f\parallel g)$ can be given as

$$\begin{array}{ccc}\hfill J(f\parallel g)& =& {\int }_{\mathbb{R}}{\left({\partial }_{x}logf-\left(-\frac{x}{{\sigma }^2}\right)\right)}^{2}fdx\hfill \end{array}$$

$$\begin{array}{ccc}& =& {\int }_{\mathbb{R}}{\left({\partial }_{x}logf\right)}^{2}fdx-\frac{2}{{\sigma }^2}{\int }_{\mathbb{R}}fdx+\frac{1}{{\left({\sigma }^2\right)}^2}{\int }_{\mathbb{R}}{x}^{2}fdx\hfill \end{array}$$

$$\begin{array}{ccc}& =& J\left(f\right)-\frac{2}{{\sigma }^2}+\frac{m_2\left(f\right)}{{\left({\sigma }^2\right)}^2}=\alpha -\frac{2}{{\sigma }^2}+\frac{\beta }{{\left({\sigma }^2\right)}^2}\hfill \end{array}$$

(16)

Combining (15) and (16), we have

$$\frac{1}{2}{\int }_0^{\infty }J(f_t\parallel g_t)dt\le \frac{{\sigma }^2}{2}J(f\parallel g),$$

which means our desired inequality by Theorem 2.2.

Remark 3.2. Similar way to the proof of Theorem 3.1 can be found in the paper by Stam [2], where it is not for relative case. Namely, based on convolution inequalities and the de Bruijn identity, the isoperimetric inequality on entropy for a standardized random variable X on $\mathbb{R}$,

$$\left(2\pi e\right)e^{-2H\left(X\right)}\le J\left(X\right)$$

(17)

was shown. This inequality is essentially the same as the logarithmic Sobolev inequality for the standard Gaussian measure, where the left hand side in (17) is the reciprocal of the entropy power.