Article An Integral Representation of the Relative Entropy

Recently the identity of de Bruijn type between the relative entropy and the relative Fisher information with the reference moving has been unveiled by Verdu via MMSE in estimation theory. In this paper, we shall give another proof of this identity in more direct way that the derivative is calculated by applying integrations by part with the heat equation. We shall also derive an integral representation of the relative entropy, as one of the applications of which the logarithmic Sobolev inequality for centered Gaussian measures will be given.


Introduction
Probability measures on R n treated in this paper are absolutely continuous with respect to the standard Lebesgue measure and we shall identify them with their densities.
For a probability measure f , the entropy H(f ) and the Fisher information J(f ) can be introduced, which play important roles in information theory, probability, and statistics.For more details on these subjects see the famous book [1].
Hereafter, for an n-variables function ϕ(x) = ϕ(x 1 , x 2 , . . ., x n ) on R n , the integral of ϕ over the whole R n by the standard Lebesgue measure dx = dx that is, we shall leave out (x 1 , x 2 , . . ., x n ) in the integrand in order to simplify the expressions.
Definition 1.1.Let f be a probability measure on R n .Then the (differential) entropy of f is defined by For a random variable X on R n with the density f , we write the entropy of X by H(X) = H(f ).
The Fisher information for a differentiable density f is defined by When the random variable X on R n has the differentiable density f , we also write as J(X) = J(f ).
The important result for a behavior of the Fisher information on convolution (sum of independent random variables) is the Stam inequality, which was first stated by Stam in [2] and subsequently proved by Blachman [3], 1 where we have the equality if and only if f and g are Gaussian.
The importance of the Stam inequality can be found in its applications, for instance, the entropy power inequality [2]; the logarithmic Sobolev inequality [4]; Cercignani conjecture [5]; the Shannon conjecture on entropy and the central limit theorem [6,7].
For t ≥ 0, we denote by P t f the convolution of f with the n-dimensional Gaussian density with mean vector 0 and covariance matrix t I n , where I n is the identity matrix.Namely, ( P t ) t≥0 is the heat semigroup acting on f and satisfies the partial differential equation which is called the heat equation.In this paper, we simply denote P t f by f t and call it the Gaussian perturbation of f .Namely, letting X be the random variable on R n with the density f and Z be an n-dimensional Gaussian random variable independent of X with mean vector 0 and covariance matrix I n , the Gaussian perturbation f t stands the density function f (x, t) of the independent sum X + √ tZ.The remarkable relation between the entropy and the Fisher information can be established by a Gaussian perturbation (see, for instance, [1], [2] or [8]); which is known as the de Bruijn identity.
Let f and g be probability measures on R n such that f ≪ g (f is absolutely continuous with respect to g).Setting the probability measure g as a reference, the relative entropy and the relative Fisher information can be introduced as follows: Definition 1.2.The relative entropy of f with respect to g, D(f ∥ g) is defined by which takes always a non-negative value.
We also define the relative Fisher information of f with respect to g by which is also non-negative.When random variables X and Y have the densities f and g, respectively, the relative entropy and the relative Fisher information of X with respect to Y are defined by In view of the de Bruijn identity, one might expect that there is a similar connection between the relative entropy and the relative Fisher information.Indeed, the gradient formulas for the relative entropy functionals were obtained in [9][10][11], where the reference measures would not be changed in their cases.
Recently Verdú in [12], however, investigated the derivative in t of D(f t ∥ g t ) for two Gaussian perturbations f t and g t .Here we should note that the reference measure does move by the same time parameter in this case.The following identity of de Bruijn type has been derived via MMSE in estimation theory (see also [13], for general perturbations).
The main aim in this paper is that we shall give an alternative proof of this identity by direct calculation with integrations by part, the method of which is similar to ones in [11,14].Moreover, it will be easily found that the above identity yields an integral representation of the relative entropy.We shall also see the simple proof of the logarithmic Sobolev inequality for centered Gaussian in univariate (n = 1) case as an application of the integral representation.

An Integral Representation of the Relative Entropy
We shall make the Gaussian perturbations f t and g t , respectively, and consider the relative entropy D(f t ∥ g t ), where the absolute continuity f t ≪ g t remains true for t > 0.
Here, we regard D(f t ∥ g t ) as a function of t and calculate the derivative, by integrations by part with help of the heat equation.
Proposition 2.1.Let f ≪ g be probability measures on R n with finite Fisher informations J(f ) < ∞ and J(g) < ∞, and finite relative entropy D(f ∥ g) < ∞.Then we obtain Proof.First we should notice that the Fisher informations J(f t ) and J(g t ) are finite at any t > 0. Because, for instance, if an n-dimensional random variable X has the density f and Z is an n-dimensional Gaussian random variable independent of X with mean vector 0 and covariance matrix I n , then by applying the Stam inequality (1) to independent random variables X and √ tZ, we have that where J(Z) = n is by simple calculation.We shall also notice that the function which can be found in [15] (p.101).Therefore, D(f t ∥ g t ) is finite for t > 0. But by a nonlinear approximation argument in [11], we can impose a stronger assumption without loss of generality that "the relative density f t g t is bounded away from 0 and ∞ on R n " Concerning the first term in the most right hand side of (4), it follows immediately that by the de Bruijn identity (3), hence, we shall concentrate our attention upon the second term.Since the densities f t and g t satisfy the heat equation ( 2), the second term can be reformulated as follows: In this reformulation, we have changed integration and differentiation at the first equality, which is justified by a routine argument with the bounded convergence theorem (see, for instance, [16]).Applying integration by part to the first term in the last expression of (8), it becomes ∫ which can be asserted by the observation below.As g t has finite Fisher information J(g t ) < ∞, and must be bounded at infinity.Furthermore, from our technical assumption (6), √ f t g t is also bounded.Hence if we factorize as then it can be found that will vanish at infinity.
Applying integration by part to the second term in the last expression of ( 8), it becomes ∫ Here it should be noted that log g t (∇f t ) will vanish at infinity by the following observation.Similarly, we factorize it as Then the boundedness of is by the assumption ( 6) same as before.Furthermore, the limit formula lim ξ→0 ξ log ξ = 0 ensures that ( √ g t log √ g t ) will vanish at infinity.Substitute the Equations ( 9) and ( 10) into (8), it follows that Combining the Equations ( 7) and ( 11), we have that which means Let X and Y be n-dimensional random variables with the densities f and g, respectively, and Z be an n-dimensional Gaussian random variable independent of X and Y with mean vector 0 and covariance matrix I n .
Since the relative entropy is scale invariant, it follows that We know that both of and the following integral representation for the relative entropy can be obtained: Theorem 2.2.Let f ≪ g be probability measures with finite Fisher informations and finite relative entropy D(f ∥ g).Then we have the integral representation,

An Application to the Logarithmic Sobolev Inequality
In this section, we shall give a proof of the logarithmic Sobolev inequality for a centered Gaussian measure in case of n = 1.Although several proofs of the logarithmic Sobolev inequality have already been given in many literatures (see, for instance, [10,17]), we shall give it here again as an application of the integral representation in Theorem 2.2.Theorem 3.1.Let g be the centered Gaussian measure of variance σ 2 .Then for any probability measure f on R of finite moment of order 2 with finite Fisher information J(f ) < ∞, the following inequality holds: Proof.It is clear that the perturbed measure g t is the centered Gaussian of variance σ 2 + t and the score of which is given by Then using the Stein relation (see, for instance, [15]), the relative Fisher information J(f t ∥ g t ) can be expanded as follows: As it was seen in ( 5), by Stam inequality, we have that where we put α = J(f ) < ∞.Since f has finite moment of order 2, if we put the second moment of f as β = m 2 (f ) < ∞, then it is easy to see that the second moment of f t is given by Substitute ( 13) and ( 14) into (12) and we obtain that Integrating for t ≥ 0, we have ) .
Since log y is dominated as log y ≤ y − 1 for y > 0, it follows that 1 2 On the other hand, the relative Fisher information J(f ∥ g) can be given as Combining ( 15) and ( 16), we have which means our desired inequality by Theorem 2.2.
Remark 3.2.Similar way to the proof of Theorem 3.1 can be found in the paper by Stam [2], where it is not for relative case.Namely, based on convolution inequalities and the de Bruijn identity, the isoperimetric inequality on entropy for a standardized random variable X on R, (2πe)e −2H(X) ≤ J(X) (17) was shown.This inequality is essentially the same as the logarithmic Sobolev inequality for the standard Gaussian measure, where the left hand side in (17) is the reciprocal of the entropy power.