Onicescu’s Informational Energy and Correlation Coefficient in Exponential Families

Abstract

The informational energy of Onicescu is a positive quantity that measures the amount of uncertainty of a random variable. However, contrary to Shannon’s entropy, the informational energy is strictly convex and increases when randomness decreases. We report a closed-form formula for Onicescu’s informational energy and its associated correlation coefficient when the probability distributions belong to an exponential family. We show how to instantiate the generic formula for several common exponential families. Finally, we discuss the characterization of valid thermodynamic process trajectories on a statistical manifold by enforcing that the entropy and the informational energy shall vary in opposite directions.

1. Introduction

1.1. Onicescu’s Informational Energy

Let $(\mathcal{X},\mathcal{F},\mu )$ be a probability space [1] with $\sigma$-algebra $\mathcal{F}$ on the sample space $\mathcal{X}$, and $\mu$ a base measure often chosen as the Lebesgue measure or as the counting measure. Let $\mathcal{M}$ denote the set of Radon-Nikodym densities of probability measures dominated by $\mu$. Two probability densities p and q are said equal (i.e., $p=q$) if and only if $p\left(x\right)=q\left(x\right)$ $\mu$-almost everywhere, and different (i.e., $p\ne q$) when $\mu \left(\{x\in \mathcal{X} : p(x)\ne q(x\left)\right\}\right)\ne 0$.

Octav Onicescu [2,3] (1892–1983) was a renowned Romanian mathematician who founded the school of probability theory and statistics [4] in Romania. Onicescu introduced the informational energy [5] (also termed information energy [6] in the literature) of a probability measure $P\ll \mu$ with Radon-Nikodym density $p=\frac{\mathrm{d}P}{\mathrm{d}\mu }$ as

$$I\left(p\right):=\int p^2\left(x\right)\mathrm{d}\mu \left(x\right)>0.$$

(1)

The Rényi entropy [7,8] of order-2 can be written using the informational energy:

$$R_2\left(p\right):=-log\int p^2\left(x\right)\mathrm{d}\mu \left(x\right)=-logI\left(p\right),$$

(2)

as well as Vajda’s quadratic entropy [9]:

$$V_2\left(p\right):=1-\int p^2\left(x\right)\mathrm{d}\mu \left(x\right)=1-I\left(p\right).$$

(3)

Notice that it follows from Cauchy-Schwarz’s inequality that we have $I\left(p\right)\le 1$ when $p\in L^2\left(\mu \right)$, the Lebesgue space of square integrable functions [1]. The informational energy for continuous distributions p is $I\left(p\right)={\int }_{\mathcal{X}}{p}^2\left(x\right)\mathrm{d}x$ and the informational energy for discrete distributions q, $I\left(q\right)={\sum }_{x\in \mathcal{X}}{q}^2\left(x\right)$. Notice that the informational energy in the continuous case is not a limit of the informational energy in the discrete case [6,10].

The informational energy is an important concept in statistics which fruitfully interplays with Shannon’s entropy [11,12] $H\left(p\right)$:

$$H\left(p\right):=\int p\left(x\right)log\frac{1}{p\left(x\right)}\mathrm{d}\mu \left(x\right)=\int -p\left(x\right)logp\left(x\right)\mathrm{d}\mu \left(x\right).$$

(4)

Notice that the informational energy is always positive but the Shannon’s entropy may be negative for continuous distributions (e.g., differential entropy of the normal distributions for small standard deviation). For the Dirac’s distribution ${\delta }_e$ (with ${\delta }_e\left(x\right)=1$ when $x=e$ and 0 otherwise), the informational energy is $I\left({\delta }_e\right)=+\infty$ but Shannon’s entropy is $H\left({\delta }_e\right)=-\infty$.

The informational energy measures the amount of uncertainty of a random variable like Shannon’s entropy but augments when randomness decreases.

Onicescu’s informational energy finds also applications in various other fields since Onicescu’s informational energy corresponds to the Herfindahl–Hirschman index [13] $\mathrm{HHI}\left(p\right)=I\left(p\right)$ in economics, to the index coincidence [14] $\mathrm{IC}\left(p\right)=I\left(p\right)$ in information theory (originally developed for cryptanalysis [15]), and is related to Simpson’s diversity index [16,17] $S\left(p\right)=I\left(p\right)$ in ecology (also called the Gini-Simpson index of diversity [18]). Section 4 will consider the joint variations of the Shannon entropy and the informational energy to characterize valid thermodynamic process paths on a statistical manifold.

Another key difference with Shannon’s entropy is that Shannon’s entropy is always strictly concave [12] but the informational energy is always strictly convex:

Property 1. 

Onicescu’s informational energy$I(·)$is a strictly convex functional.

Proof. 

A function F is strictly convex if and only if for any $\alpha \in (0,1)$ and $p\ne q$ two densities of a convex domain $\mathcal{M}$, we have $F\left(\right(1-\alpha )p+\alpha q)<(1-\alpha )F\left(p\right)+\alpha F\left(q\right)$. Let us check this strict inequality for the informational energy I:

$$\begin{array}{ccc}\hfill I\left(\right(1-\alpha )p+\alpha q)& =& {(1-\alpha )}^{2}I\left(p\right)+{\alpha }^{2}I\left(q\right)+2\alpha (1-\alpha )\int p\left(x\right)q\left(x\right)\mathrm{d}\mu \left(x\right),\hfill \\ & =& (1-\alpha )I(p)+\alpha I(q)+2\alpha (1-\alpha )\int p(x)q(x)\mathrm{d}\mu (x)\hfill \\ & & -\alpha (1-\alpha )\left(I\right(p)+I(q\left)\right),\hfill \end{array}$$

(5)

$$\begin{array}{ccc}& =& (1-\alpha )I\left(p\right)+\alpha I\left(q\right)-\underset{>0}{\underbrace{\alpha (1-\alpha )\int {(p\left(x\right)-q\left(x\right))}^2\mathrm{d}\mu \left(x\right)}},\hfill \end{array}$$

(6)

$$\begin{array}{ccc}& <& (1-\alpha )I\left(p\right)+\alpha I\left(q\right),\hfill \end{array}$$

(7)

when $p\ne q$ and $\alpha \in (0,1)$. □

Table 1. Comparison between Shannon’s entropy and Onicescu’s informational energy.

	Entropy $\mathit{H}\left(\mathit{p}\right)$	Informational Energy $\mathit{I}\left(\mathit{p}\right)$
	$H\left(p\right)=-\int p\left(x\right)logp\left(x\right)\mathrm{d}\mu \left(x\right)$	$I\left(p\right)=\int p^2\left(x\right)\mathrm{d}\mu \left(x\right)$
convexity	strictly concave	strictly convex
range	can be negative	always positive
uncertainty measure	augments with disorder	decreases with disorder
uniform discrete distribution u	$H\left(u\right)=logd$	$I\left(u\right)=\frac{1}{d}$
(with alphabet size $\left|\mathcal{X}\right|=d$)
bound	$H\left(p\right)\ge 1-I\left(p\right)$	$I\left(p\right)\ge 1-H\left(p\right)$
Inequality: $H\left(p\right)+\frac{1}{2} I\left(p\right)\ge 1-log2$

summarizes the comparison between Shannon’s entropy and Onicescu’s informational energy.

Since $I\left(p\right)$ is strictly convex, we can define the informational energy divergence as the following Jensen divergence [19] measuring the convexity gap:

$$\begin{array}{ccc}\hfill J_I(p,q)& :=& \frac{I\left(p\right)+I\left(q\right)}{2}-I\left(\frac{p+q}{2}\right),\hfill \end{array}$$

(8)

$$\begin{array}{ccc}& =& \frac{1}{4}{\int }_{\mathcal{X}}{(p\left(x\right)-q\left(x\right))}^2\mathrm{d}\mu \left(x\right).\hfill \end{array}$$

(9)

For a uniform discrete distribution p on an alphabet $\mathcal{X}$ of d letters, we have $I\left(p\right)=\frac{1}{d}$, and for any probability mass function p on $\mathcal{X}$, we have $I\left(p\right)\ge \frac{1}{d}$. Recall that $H\left(p\right)\le logd$ with equality when p is the discrete uniform distribution. More generally, for a continuous density on an interval $\mathcal{X}=[a,b]$, we have $I\left(p\right)\ge \frac{1}{b-a}$.

For discrete or continuous distributions, we have the following inequality [20] (Proposition 5.8.5):

$$H\left(p\right)+\frac{1}{2}I\left(p\right)\ge 1-log2>0.69897,$$

(10)

and Shannon’s cross-entropy

$$H^{\times }(p:q)=\int -p\left(x\right)logq\left(x\right)\mathrm{d}\mu \left(x\right)$$

(11)

can be lower bounded using the informational energy as follows (Problem 5.8 in [20]): For any $x>0$, we have $logx\le x-1$. Thus we get $logq\left(x\right)\le q\left(x\right)-1$ and $-p\left(x\right)logq\left(x\right)\ge p\left(x\right)-p\left(x\right)q\left(x\right)$. Therefore we have

$$H^{\times }(p:q)\ge 1-\int p\left(x\right)q\left(x\right)\mathrm{d}\mu \left(x\right).$$

Using Cauchy–Schwarz’s inequality

$$\int p\left(x\right)q\left(x\right)\mathrm{d}\mu \left(x\right)\le \sqrt{\int p{\left(x\right)}^2\mathrm{d}\mu \left(x\right)} \sqrt{\int q{\left(x\right)}^2\mathrm{d}\mu \left(x\right)},$$

we get:

$$H^{\times }(p:q)\ge 1-\sqrt{I\left(p\right) I\left(q\right)}.$$

(12)

In particular, we get a lower bound on Shannon’s entropy:

$$H\left(p\right)=H^{\times }(p:q)\ge 1-I\left(p\right).$$

For an in-depth treatment of Onicescu’s informational energy, we refer to the paper [6] (77 pages, with main properties listed in pp. 167–169 as statistical applications). Onicescu’s informational energy has been used in physics [21], information theory in electronic structure theory of atomic and molecular systems [22,23,24], machine learning [17,25], and complex systems [26], among others.

1.2. Onicescu’s Correlation Coefficient

Onicescu also defined a correlation coefficient (see [20], Chapter 5):

$$\rho (p,q):=\frac{I(p,q)}{\sqrt{I\left(p\right) I\left(q\right)}},$$

(13)

where $I(p,q)$ denotes the cross-informational energy:

$$I(p,q):=\int p(x)q(x)\mathrm{d}\mu (x),$$

(14)

with $I\left(p\right)=I(p,p)$. Notice that it follows from the Cauchy-Schwarz inequality that $I(p,q)\le \sqrt{I\left(p\right)I\left(q\right)}$, and therefore we have:

$$0<\rho (p,q)\le 1,$$

assuming both densities p and q belong to the Lebesgue space $L^2\left(\mu \right)$.

Notice that the informational energy of a statistical mixture $m\left(x\right)={\sum }_{i=1}^k{w}_i{p}_i\left(x\right)$ with k weighted components $p_1\left(x\right),\dots ,p_k\left(x\right)$ (with $w\in {\Delta }_k$ the $(k-1)$-dimensional standard simplex) can be expressed as follows:

$$I\left(m\right)=\int {\left(\sum _{i=1}^k{w}_i{p}_i\left(x\right)\right)}^2\mathrm{d}\mu \left(x\right)=\sum _{i=1}^k\sum _{j=1}^k{w}_i{w}_j I(p_i,p_j).$$

(15)

The Cauchy-Schwarz divergence [27,28] is defined by

$$D_{\mathrm{CS}}(p,q):=-log\left(\frac{{\int }_{\mathcal{X}}p\left(x\right)q\left(x\right)\mathrm{d}\mu \left(x\right)}{\sqrt{\left({\int }_{\mathcal{X}}p{\left(x\right)}^2\mathrm{d}\mu \left(x\right)\right)\left({\int }_{\mathcal{X}}q{\left(x\right)}^2\mathrm{d}\mu \left(x\right)\right)}}\right)\ge 0.$$

(16)

Thus, the Cauchy-Schwarz divergence is a projective divergence (that is, we have $D_{\mathrm{CS}}(p,q)=D_{\mathrm{CS}}(\lambda p,{\lambda }^{\prime }q)$ for any $\lambda >0$ and ${\lambda }^{\prime }>0$) which can be rewritten using the Onicescu’s correlation coefficient as:

$$D_{\mathrm{CS}}(p,q)=-log\left(\rho (p,q)\right).$$

(17)

1.3. Exponential Families

Consider a natural exponential family [29,30] (NEF)

$$\mathcal{E}=\left\{p_{\theta }\left(x\right)=exp\left({\theta }^{\top }t\left(x\right)-F\left(\theta \right)+k\left(x\right)\right) : \theta \in \Theta \right\},$$

(18)

where $t\left(x\right)$ denotes the minimal sufficient statistics, $k\left(x\right)$ an auxiliary measure carrier term, and

$$F\left(\theta \right):=log\left({\int }_{\mathcal{X}}exp\left({\theta }^{\top }t\left(x\right)\right)\mathrm{d}\mu \left(x\right)\right),$$

(19)

the cumulant function which is commonly called the log-normalizer (or log-partition function in statistical physics). Parameter $\theta$ is called the natural parameter and is defined on the open convex natural parameter space $\Theta$.

Many familiar families of distributions $\{p_{\lambda }\left(x\right) \lambda \in \Lambda \}$ are exponential families in disguise after reparameterization: $p_{\lambda }\left(x\right)=p_{\theta \left(\lambda \right)}\left(x\right)$ (e.g., normal family or Poisson family). Those families are called exponential families (omitting the leading adjective ‘natural’), and their densities are canonically factorized as follows:

$$p_{\lambda }\left(x\right)=exp\left(\theta {\left(\lambda \right)}^{\top }t\left(x\right)-F\left(\theta \left(\lambda \right)\right)+k\left(x\right)\right).$$

(20)

We call parameter $\lambda \in \Lambda$ the source parameter, and parameter $\theta \left(\lambda \right)\in \Theta$ is called the corresponding natural parameter. Densities of an exponential family have all the same support $\mathcal{X}$.

2. Onicescu’s Informational Energy and Correlation Coefficient in Exponential Families

In this section, we first report a closed-form formula for the informational energy and its associated correlation coefficient in Section 2.1. We then describe some statistical divergences related to Onicescu’s correlation coefficient in Section 2.2.

2.1. Closed-Form Formula

We report closed-form formulas for Onicescu’s informational energy and correlation coefficient when densities belong to a prescribed exponential family, and then illustrate those formula on common families of probability distributions.

Theorem 1 

(Onicescu’s informational energy and correlation coefficient in exponential families). In an exponential family $\mathcal{E}={\left\{p_{\theta }\right\}}_{\theta \in \Theta }$, Onicescu’s informational energy of a probability density $p_{\theta }$ is given by:

$$I\left(p_{\theta }\right)=exp\left(F\left(2\theta \right)-2F\left(\theta \right)\right) E_{p_{2\theta }}\left[exp\left(k\right(x\left)\right)\right],$$

(21)

provided that$2\theta \in \Theta$so that$p_{2\theta }\in \mathcal{E}$. When the auxiliary carrier term$k\left(x\right)$vanishes, we have$E_{p_{2\theta }}\left[exp\left(k\right(x\left)\right)\right]=1$.

The Onicescu’s correlation coefficient$\rho (p_{{\theta }_1},p_{{\theta }_2})$between densities$p_1=p_{{\theta }_1}$and$p_2=p_{{\theta }_2}$is

$$\rho (p_{{\theta }_1},p_{{\theta }_2})=exp(-J_F(2{\theta }_1:2{\theta }_2))\times \frac{E_{p_{{\theta }_1+{\theta }_2}}\left[exp\left(k\right(x\left)\right)\right]}{\sqrt{E_{p_{2{\theta }_1}}\left[exp\left(k\right(x\left)\right)\right] E_{p_{2{\theta }_2}}\left[exp\left(k\right(x\left)\right)\right]}},$$

(22)

provided that${\theta }_1+{\theta }_2\in \Theta$,$2{\theta }_1\in \Theta$,$2{\theta }_2\in \Theta$, where

$$J_F({\theta }_1,{\theta }_2):=\frac{F\left({\theta }_1\right)+F\left({\theta }_2\right)}{2}-F\left(\frac{{\theta }_1+{\theta }_2}{2}\right)\ge 0,$$

(23)

is a Jensen divergence [19] induced by the cumulant function of the exponential family.

Proof. 

The proof follows the same line of arguments as in [31]. Consider the term $I(p_{{\theta }_1},p_{{\theta }_2})$:

$$\begin{array}{ccc}\hfill I(p_{{\theta }_1},p_{{\theta }_2})& :=& \int exp(t{\left(x\right)}^{\top }{\theta }_1-F\left({\theta }_1\right)+k\left(x\right))exp(t{\left(x\right)}^{\top }{\theta }_2-F\left({\theta }_2\right)+k\left(x\right)) \mathrm{d}\mu \left(x\right),\hfill \\ & =& \int exp\left(t{\left(x\right)}^{\top }({\theta }_1+{\theta }_2)-F({\theta }_1+{\theta }_2)+k\left(x\right)\right.\hfill \\ & & \left.+F({\theta }_1+{\theta }_2)-F\left({\theta }_1\right)-F\left({\theta }_2\right)+k\left(x\right)\right)\mathrm{d}\mu \left(x\right),\hfill \end{array}$$

(24)

$$\begin{array}{ccc}& =& exp\left(F({\theta }_1+{\theta }_2)-F\left({\theta }_1\right)-F\left({\theta }_2\right)\right) \int p_{{\theta }_1+{\theta }_2}\left(x\right)exp\left(k\left(x\right)\right)\mathrm{d}\mu \left(x\right),\hfill \end{array}$$

(25)

$$\begin{array}{ccc}& =& exp\left(F({\theta }_1+{\theta }_2)-F\left({\theta }_1\right)-F\left({\theta }_2\right)\right) E_{p_{{\theta }_1+{\theta }_2}}\left[exp\left(k\right(x\left)\right)\right],\hfill \end{array}$$

(26)

provided that ${\theta }_1+{\theta }_2\in \Theta$. This condition is always satisfied when the natural parameter space is either a cone [31] (e.g., Gaussian family, Wishart family, etc.) or an affine space [32] (e.g., Poisson family, isotropic Gaussian family, etc). Since $I\left(p\right)=I(p,p)$ and $\rho (p,q)=\frac{I(p,q)}{\sqrt{I\left(p\right)I\left(q\right)}}$, we deduce formula of Equations (21) and (22).  □

2.2. Divergences Related to Onicescu’s Correlation Coefficient

Since $D_{\mathrm{CS}}(p,q)=-log\left(\rho (p,q)\right)$, we get the following closed-form for the Cauchy-Schwarz divergence:

$$D_{\mathrm{CS}}(p_{{\theta }_1},p_{{\theta }_2})=J_F(2{\theta }_1:2{\theta }_2)+log\left(\frac{\sqrt{E_{p_{2{\theta }_1}}\left[exp\left(k\right(x\left)\right)\right] E_{p_{2{\theta }_2}}\left[exp\left(k\right(x\left)\right)\right]}}{E_{p_{{\theta }_1+{\theta }_2}}\left[exp\left(k\right(x\left)\right)\right]}\right).$$

(27)

We check that when ${\theta }_1={\theta }_2$, we have $D_{\mathrm{CS}}(p_{{\theta }_1},p_{{\theta }_2})=0$. Closed-form formula were also reported for the Cauchy-Schwarz divergence between densities of an exponential family in [33].

Table 2. Comparisons between Shannon’s entropy and Onicescu’s informational energy for common distributions of exponential families. $|·|$ denotes the matrix determinant, $\mathsf{\Gamma }(·)$ the gamma function, $\psi (·)$ the digamma function, and $\mathrm{B}(\alpha ,\beta )=\frac{\mathsf{\Gamma }\left(\alpha \right)\mathsf{\Gamma }\left(\beta \right)}{\mathsf{\Gamma }(\alpha +\beta )}$.

Family	Entropy	Informational Energy
Generic $\mathcal{E}$	$F\left(\theta \right)-{\theta }^{\top }\nabla F\left(\theta \right)-E_{p_{\theta }}\left[k\left(x\right)\right]$	$e^{F\left(2\theta \right)-2F\left(\theta \right)}{E}_{p_{2\theta }}\left[exp\left(k\right(x\left)\right)\right]$
Univar. normal $N(\mu ,\sigma )$	$\frac{1}{2}log\left(2\pi e{\sigma }^2\right)$	$\frac{1}{2\sigma \sqrt{\pi }}$
Multivar. normal $N(\mu ,\Sigma )$	$\frac{1}{2}log|2\pi e\Sigma |$	${\pi }^{-\frac{d}{2}}{2}^{-d}{|\Sigma |}^{-\frac{1}{2}}$
$\mathrm{LogNormal}(\mu ,\sigma )$	$log\left(\sigma e^{\mu +\frac{1}{2}}\sqrt{2\pi }\right)$	$\frac{1}{2\sigma \sqrt{\pi }}exp(\frac{{\sigma }^2}{4}-\mu )$
$\mathrm{Exponential}\left(\lambda \right)$	$1-log\lambda$	$\frac{\lambda }{2}$
${\mathrm{Pareto}}_k\left(a\right)$	$1+\frac{1}{a}+log\frac{k}{a}$	$\frac{a^2}{k(2a+1)}$
$\mathrm{Gamma}(\alpha ,\beta )$	$\alpha +log\frac{\mathsf{\Gamma }\left(\alpha \right)}{\beta }+(1-\alpha )\psi \left(\alpha \right)$	$\frac{1}{\beta (2\alpha -1)B\left(\alpha ,\frac{1}{2}\right)}$
$\mathrm{Beta}(\alpha ,\beta )$	$log\mathrm{B}(\alpha ,\beta )-(\alpha -1)\psi \left(\alpha \right)$	$B^2(\alpha ,\beta )\frac{\mathsf{\Gamma }(2\alpha -1)\mathsf{\Gamma }(2\beta -1)}{\mathsf{\Gamma }(2\alpha +2\beta -2)}$
	$-(\beta -1)\psi \left(\beta \right)$
	$+(\alpha +\beta -2)\psi (\alpha +\beta )$
$\mathrm{Poisson}\left(\lambda \right)$	$\lambda (1-log\lambda )+e^{-\lambda }{\sum }_{i=0}^{\infty }\frac{{\lambda }^{i}logi!}{i!}$	$exp(-2\lambda ){\sum }_{i=0}^{\infty }\frac{{\lambda }^{2i}}{{(i!)}^2}$

reports the formula for Onicescu’s informational energy and the Shannon’s entropy [34] for densities belonging to some common exponential families. These formula can be recovered easily from the generic formula using the canonical decompositions of exponential families reported in [29]. Shannon’s entropy [34] of a density $p_{\theta }$ of $\mathcal{E}$ is

$$\begin{array}{ccc}\hfill H\left(p_{\theta }\right)& =& F\left(\theta \right)-{\theta }^{\top }\nabla F\left(\theta \right)-E_{p_{\theta }}\left[k\left(x\right)\right],\hfill \end{array}$$

(28)

$$\begin{array}{ccc}\hfill H\left(p_{\theta }\right)& =& -F^{*}\left(\eta \right)-E_{p_{\theta }}\left[k\left(x\right)\right],\hfill \end{array}$$

(29)

where $F^{*}$ denotes the Legendre-Fenchel convex conjugate and $\eta =\nabla F\left(\theta \right)=E_{p_{\theta }}\left[t\left(x\right)\right]$ the moment parameter.

Notice that when $k\left(x\right)=0$ (no auxiliary carrier term, e.g., Gaussian family), we have $E_p\left[e^{k\left(x\right)}\right]=E_p\left[1\right]=\int p\left(x\right)\mathrm{d}\mu \left(x\right)=1$ for any density $p\in \mathcal{M}$. In that case, the above formula simplify as follows:

$$\begin{array}{ccc}\hfill I\left(p_{\theta }\right)& =& exp\left(F\left(2\theta \right)-2F\left(\theta \right)\right),\hfill \end{array}$$

(30)

$$\begin{array}{ccc}\hfill \rho (p_{{\theta }_1},p_{{\theta }_2})& =& exp(-J_F(2{\theta }_1:2{\theta }_2)),\hfill \end{array}$$

(31)

$$\begin{array}{ccc}\hfill D_{\mathrm{CS}}(p_{{\theta }_1},p_{{\theta }_2})& =& J_F(2{\theta }_1:2{\theta }_2).\hfill \end{array}$$

(32)

The Cauchy-Schwarz divergence between mixtures of Gaussians has been reported in [35], and extended to mixtures of exponential families with conic natural parameter spaces in [33].

Furthermore, since the Jensen divergence $J_F$ is defined for a strictly convex generator F modulo an affine term, we may choose the representative $F\left(\theta \right)=-log{p}_{\theta }\left(\omega \right)=:-l_{\theta }\left(\omega \right)$ for the equivalence class $\left[F\right]$ of strictly convex functions, where $\omega$ is any point belonging to the the support $\mathcal{X}$ of the exponential family $\mathcal{X}$ and $l_{\theta }(·)$ the (concave) log-likelihood function, see [36] for details.

It follows that, we can rewrite the Onicescu’s informational energy, correlation coefficient and the Cauchy-Schwarz divergence when $k\left(x\right)=0$ as follows:

$$\begin{array}{ccc}\hfill I\left(p_{\theta }\right)& =& \frac{p_{\theta }^2\left(\omega \right)}{p_{2\theta }\left(\omega \right)}, \forall \omega \in \mathcal{X}\hfill \end{array}$$

(33)

$$\begin{array}{ccc}\hfill \rho (p_{{\theta }_1},p_{{\theta }_2})& =& \frac{\sqrt{p_{2{\theta }_1}\left(\omega \right)}\sqrt{p_{2{\theta }_2}\left(\omega \right)}}{p_{{\theta }_1+{\theta }_2}\left(\omega \right)}, \forall \omega \in \mathcal{X},\hfill \end{array}$$

(34)

$$\begin{array}{ccc}\hfill D_{\mathrm{CS}}(p_{{\theta }_1},p_{{\theta }_2})& =& -log\left(\frac{\sqrt{p_{2{\theta }_1}\left(\omega \right)}\sqrt{p_{2{\theta }_2}\left(\omega \right)}}{p_{{\theta }_1+{\theta }_2}\left(\omega \right)}\right),\hfill \end{array}$$

(35)

$$\begin{array}{ccc}& =& l_{{\theta }_1+{\theta }_2}\left(\omega \right)-\frac{l_{2{\theta }_1}\left(\omega \right)+l_{2{\theta }_2}\left(\omega \right)}{2}, \forall \omega \in \mathcal{X}.\hfill \end{array}$$

(36)

Moreover, the Cauchy-Schwarz divergence can be generalized to the broader class of Hölder divergences [28] for conjugate exponents $\frac{1}{\alpha }+\frac{1}{\beta }=1$ with $\alpha >1$, $\beta >1$, and $\gamma >0$ as follows:

$$\begin{array}{ccc}\hfill D_{\mathrm{H}\ddot{\mathrm{o}}\mathrm{lder}}^{\alpha ,\gamma }(p_{{\theta }_1},p_{{\theta }_2})& :=& -log\left(\frac{{\int }_{\mathcal{X}}p{\left(x\right)}^{\gamma /\alpha }q{\left(x\right)}^{\gamma /\beta }\mathrm{d}\mu \left(x\right)}{{\left({\int }_{\mathcal{X}}p{\left(x\right)}^{\gamma }\mathrm{d}\mu \left(x\right)\right)}^{1/\alpha }{\left({\int }_{\mathcal{X}}q{\left(x\right)}^{\gamma }\mathrm{d}\mu \left(x\right)\right)}^{1/\beta }}\right),\hfill \end{array}$$

(37)

$$\begin{array}{ccc}& =& log\left(\frac{p_{\frac{\gamma }{\alpha }{\theta }_1+\frac{\gamma }{\beta }{\theta }_2}\left(\omega \right)}{p_{\gamma {\theta }_1}^{\frac{1}{\alpha }}\left(\omega \right)p_{\gamma {\theta }_2}^{\frac{1}{\beta }}\left(\omega \right)}\right), \forall \omega \in \mathcal{X},\hfill \end{array}$$

(38)

$$\begin{array}{ccc}\hfill D_{\mathrm{H}\ddot{\mathrm{o}}\mathrm{lder}}^{\alpha ,\gamma }(p_{{\lambda }_1},p_{{\lambda }_2})& =& log\left(\frac{p_{\frac{\gamma }{\alpha }\theta \left({\lambda }_1\right)+\frac{\gamma }{\beta }\theta \left({\lambda }_2\right)}\left(\omega \right)}{p_{\gamma \theta \left({\lambda }_1\right)}^{\frac{1}{\alpha }}\left(\omega \right)p_{\gamma \theta \left({\lambda }_2\right)}^{\frac{1}{\beta }}\left(\omega \right)}\right), \forall \omega \in \mathcal{X}.\hfill \end{array}$$

(39)

The latter two formulae hold when $k\left(x\right)=0$ (no auxiliary carrier term like the Gaussian family). When $\alpha =\beta =\gamma =2$, we recover the Cauchy-Schwarz divergence: $D_{\mathrm{H}\ddot{\mathrm{o}}\mathrm{lder}}^{2,2}(p_{{\theta }_1},p_{{\theta }_2})=D_{\mathrm{CS}}(p_{{\theta }_1},p_{{\theta }_2})$.

3. Some Illustrating Examples

Let us illustrate how to instantiate the generic formula with some examples of exponential families.

3.1. Exponential Family of Exponential Distributions

Consider the family of exponential distributions with rate parameter $\lambda >0$. The densities of this exponential family writes as $p_{\lambda }\left(x\right)=\lambda exp(-\lambda x)$ with support $\mathcal{X}=[0,\infty )$. We use the canonical decomposition of the exponential family to get $t\left(x\right)=-x$, $\theta =\lambda$, $F\left(\theta \right)=-log\theta$ and $k\left(x\right)=0$. It follows that

$$\begin{array}{ccc}\hfill I\left(p_{\theta }\right)& =& exp\left(F\left(2\theta \right)-2F\left(\theta \right)\right),\hfill \end{array}$$

(40)

$$\begin{array}{ccc}& =& exp\left(-log2\theta +2log\theta \right),\hfill \end{array}$$

(41)

$$\begin{array}{ccc}& =& exp\left(-log2+log\theta \right),\hfill \end{array}$$

(42)

$$\begin{array}{ccc}& =& \frac{\theta }{2}.\hfill \end{array}$$

(43)

Thus $I\left(p_{\lambda }\right)=\frac{\lambda }{2}$. Similarly, we find that

$$\begin{array}{ccc}\hfill I(p_{{\lambda }_1},p_{{\lambda }_2})& =& exp\left(F\left({\theta }_1+{\theta }_2\right)-F\left({\theta }_1\right)-F\left({\theta }_2\right)\right)E_{p_{{\theta }_1+{\theta }_2}}[exp\left(k\left(x\right)\right)],\hfill \end{array}$$

(44)

$$\begin{array}{ccc}& =& \frac{{\lambda }_1{\lambda }_2}{{\lambda }_1+{\lambda }_2}.\hfill \end{array}$$

(45)

Thus $\rho (p_{{\lambda }_1},p_{{\lambda }_2})=\frac{2\sqrt{{\lambda }_1{\lambda }_2}}{{\lambda }_1+{\lambda }_2}$, and $D_{\mathrm{CS}}(p_{{\lambda }_1},p_{{\lambda }_2})=log\left(\frac{{\lambda }_1+{\lambda }_2}{2}\right)-\frac{1}{2}log\left({\lambda }_1{\lambda }_2\right)$. We check that $D_{\mathrm{CS}}(p_{{\lambda }_1},p_{{\lambda }_2})\ge 0$ since the arithmetic mean $A({\lambda }_1,{\lambda }_2)=\frac{{\lambda }_1+{\lambda }_2}{2}$ is greater or equal than the geometric mean $G({\lambda }_1,{\lambda }_2)=\sqrt{{\lambda }_1{\lambda }_2}$, and $D_{\mathrm{CS}}(p_{{\lambda }_1},p_{{\lambda }_2})=log\frac{A({\lambda }_1,{\lambda }_2)}{G({\lambda }_1,{\lambda }_2)}$.

Choose $\omega =0$ so that $p_{\lambda }\left(\omega \right)=\lambda$ and $l_{\lambda }\left(\omega \right)=log\lambda$.

$$\begin{array}{ccc}\hfill \rho (p_{{\theta }_1},p_{{\theta }_2})& =& \frac{\sqrt{p_{2{\theta }_1}\left(\omega \right)}\sqrt{p_{2{\theta }_2}\left(\omega \right)}}{p_{{\theta }_1+{\theta }_2}\left(\omega \right)},\hfill \end{array}$$

(46)

$$\begin{array}{ccc}& =& \frac{2\sqrt{{\lambda }_1{\lambda }_2}}{{\lambda }_1+{\lambda }_2},\hfill \end{array}$$

(47)

$$\begin{array}{ccc}\hfill D_{\mathrm{CS}}(p_{{\theta }_1},p_{{\theta }_2})& =& l_{{\theta }_1+{\theta }_2}\left(\omega \right)-\frac{l_{2{\theta }_1}\left(\omega \right)+l_{2{\theta }_2}\left(\omega \right)}{2},\hfill \end{array}$$

(48)

$$\begin{array}{ccc}& =& log({\lambda }_1+{\lambda }_2)-\frac{1}{2}(log\left(2{\lambda }_1\right)+log\left(2{\lambda }_2\right))=log\left(\frac{{\lambda }_1+{\lambda }_2}{2\sqrt{{\lambda }_1{\lambda }_2}}\right).\hfill \end{array}$$

(49)

To illustrate the fact that the formula Equation() is independent of the choice of $\omega$, let us consider $\omega =1$ so that $p_{\lambda }\left(\omega \right)=\lambda exp(-\lambda )$ with log-likelihood $l_{\lambda }\left(\omega \right)=log\left(\lambda \right)-\lambda$. The correlation coefficient is then calculated as

$$\begin{array}{ccc}\hfill \rho (p_{{\theta }_1},p_{{\theta }_2})& =& \frac{\sqrt{p_{2{\theta }_1}\left(\omega \right)}\sqrt{p_{2{\theta }_2}\left(\omega \right)}}{p_{{\theta }_1+{\theta }_2}\left(\omega \right)},\hfill \end{array}$$

(50)

$$\begin{array}{ccc}& =& \frac{\sqrt{2{\lambda }_{1}exp(-2{\lambda }_1)}\sqrt{2{\lambda }_{2}exp(-2{\lambda }_2)}}{({\lambda }_1+{\lambda }_2)exp(-({\lambda }_1+{\lambda }_2))},\hfill \end{array}$$

(51)

$$\begin{array}{ccc}& =& \frac{2\sqrt{{\lambda }_1{\lambda }_2}}{{\lambda }_1+{\lambda }_2}\frac{exp(-({\lambda }_1+{\lambda }_2))}{exp(-({\lambda }_1+{\lambda }_2))},\hfill \end{array}$$

(52)

$$\begin{array}{ccc}& =& \frac{2\sqrt{{\lambda }_1{\lambda }_2}}{{\lambda }_1+{\lambda }_2}.\hfill \end{array}$$

(53)

3.2. Exponential Family of Poisson Distributions

The Poisson family of probability mass functions (PMFs) $p_{\lambda }\left(x\right)=\frac{{\lambda }^{x}exp(-\lambda )}{x!}$ where $\lambda >0$ denotes the intensity parameter and $x\in \mathcal{X}=\{0,1,\dots ,\}$ is a discrete exponential family with sufficient statistic $t\left(x\right)=x$, natural parameter $\theta \left(\lambda \right)=log\lambda$ (affine natural parameter space), cumulant function $F\left(\theta \right)=exp\left(\theta \right)$, and auxiliary carrier term $k\left(x\right)=-logx!$. The informational energy is

$$\begin{array}{ccc}\hfill I\left(p_{\lambda }\right)& =& \sum _{x=0}^{\infty }{p}_{\lambda }^2\left(x\right),\hfill \end{array}$$

(54)

$$\begin{array}{ccc}& =& I\left(p_{\theta }\right)=exp\left(F\left(2\theta \right)-2F\left(\theta \right)\right)E_{p_{2\theta }}\left[exp\left(k\right(x\left)\right)\right],\hfill \end{array}$$

(55)

$$\begin{array}{ccc}& =& exp(e^{2\theta }-2e^{\theta })E_{p_{2\theta }}\left[\frac{1}{x!}\right],\hfill \end{array}$$

(56)

$$\begin{array}{ccc}& =& e^{{\lambda }^2-2\lambda }{E}_{p_{{\lambda }^2}}\left[\frac{1}{x!}\right].\hfill \end{array}$$

(57)

3.3. Exponential Family of Univariate Normal Distributions

Consider the set of univariate normal probability density function:

$$\mathcal{N}:=\left\{p_{\lambda }\left(x\right)=\frac{1}{\sigma \sqrt{2\pi }}exp\left(-\frac{1}{2}{\left(\frac{x-\mu }{\sigma }\right)}^2\right), \lambda =(\mu ,{\sigma }^2)\in \mathbb{R}\times {\mathbb{R}}_{++}\right\},$$

(58)

where ${\mathbb{R}}_{++}=\{x\in \mathbb{R} : x>0\}$ denote the set of positive reals. Family $\mathcal{N}$ is interpreted as an exponential family indexed by the source parameter $\lambda =(\mu ,{\sigma }^2)\in \Lambda$ with $\Lambda =\mathbb{R}\times {\mathbb{R}}_{++}$. The corresponding natural parameter is $\theta \left(\lambda \right)=\left(\frac{\mu }{{\sigma }^2},-\frac{1}{2{\sigma }^2}\right)$ with the sufficient statistic $t\left(x\right)=(x,x^2)$ on the support $\mathcal{X}=(-\infty ,\infty )$ (and no additional carrier term, i.e., $k\left(x\right)=0$). The cumulant function for the normal family is $F\left(\theta \right)=-\frac{{\theta }_1^2}{4{\theta }_2}+\frac{1}{2}log\left(-\frac{\pi }{{\theta }_2}\right)$.

We have

$$\begin{array}{ccc}\hfill I\left(p_{\theta }\right)& =& exp\left(F\left(2\theta \right)-2F\left(\theta \right)\right),\hfill \end{array}$$

(59)

$$\begin{array}{ccc}& =& exp\left(-\frac{{\left(2{\theta }_1\right)}^2}{4\left(2{\theta }_2\right)}+\frac{1}{2}log\frac{\pi }{(-2{\theta }_2)}+\frac{1}{2}\frac{{\theta }_1^2}{{\theta }_2}-log\frac{\pi }{(-{\theta }_2)}\right),\hfill \end{array}$$

(60)

$$\begin{array}{ccc}& =& exp\left(\frac{1}{2}log\frac{\pi }{(-2{\theta }_2)}-log\frac{\pi }{(-{\theta }_2)}\right),\hfill \end{array}$$

(61)

$$\begin{array}{ccc}& =& exp\left(-\frac{1}{2}log\left(2\pi \right)-\frac{1}{2}log\left(2{\sigma }^2\right)\right),\hfill \end{array}$$

(62)

$$\begin{array}{ccc}& =& \frac{1}{2\sigma \sqrt{\pi }}.\hfill \end{array}$$

(63)

Similar calculations for ${\theta }_1=({\mu }_1,{\sigma }_1)$ and ${\theta }_2=({\mu }_2,{\sigma }_2)$ yield

$$I(p_{{\mu }_1,{\sigma }_1},p_{{\mu }_2,{\sigma }_2})=\frac{1}{\sqrt{2\pi }}\frac{exp\left(-\frac{{({\mu }_1-{\mu }_2)}^2}{2{\sigma }_1^2+2{\sigma }_2^2}\right)}{\sqrt{{\sigma }_1^2+{\sigma }_2^2}}.$$

(64)

We check that $I\left(p_{\theta }\right)=I(p_{\theta },p_{\theta })$.

It follows that Onicescu’s correlation coefficient between two normal densities is:

$$\rho (p_{{\mu }_1,{\sigma }_1},p_{{\mu }_2,{\sigma }_2})=\sqrt{\frac{2{\sigma }_1{\sigma }_2}{{\sigma }_1^2+{\sigma }_2^2}}exp\left(-\frac{{({\mu }_1-{\mu }_2)}^2}{2{\sigma }_1^2+2{\sigma }_2^2}\right),$$

(65)

and the Cauchy-Schwarz divergence between two univariate Gaussians is:

$$D_{\mathrm{CS}}(p_{{\mu }_1,{\sigma }_1},p_{{\mu }_2,{\sigma }_2})=-log\rho (p_{{\mu }_1,{\sigma }_1},p_{{\mu }_2,{\sigma }_2})=\frac{{({\mu }_1-{\mu }_2)}^2}{2{\sigma }_1^2+2{\sigma }_2^2}+\frac{1}{2}log\left(\frac{1}{2}\left(\frac{{\sigma }_1}{{\sigma }_2}+\frac{{\sigma }_2}{{\sigma }_1}\right)\right).$$

(66)

3.4. Exponential Family of Multivariate Normal Distributions

Consider the example of the multivariate normal (MVN) family: The parameter $\lambda =({\lambda }_v,{\lambda }_M)$ of a MVN consists of a vector part ${\lambda }_v=\mu$ and a $d\times d$ positive-definite matrix part ${\lambda }_M=\Sigma \succ 0$. The density is given by

$$p_{\lambda }(x;\lambda )=\frac{1}{{\left(2\pi \right)}^{\frac{d}{2}}\sqrt{|{\lambda }_M|}}exp\left(-\frac{1}{2}{(x-{\lambda }_v)}^{\top }{\lambda }_M^{-1}(x-{\lambda }_v)\right),$$

(67)

where $|·|$ denotes the matrix determinant. Choose the sufficient statistic $t\left(x\right)=(x,-\frac{1}{2}x{x}^{\top })$ so that $\theta =({\theta }_v={\Sigma }^{-1}\mu ,{\theta }_M={\Sigma }^{-1})$. Since $k\left(x\right)=0$, let $\omega =0$,

$$p_{\lambda }\left(0\right)=\frac{1}{{\left(2\pi \right)}^{\frac{d}{2}}\sqrt{|\Sigma |}}exp\left(-\frac{1}{2}{\mu }^{\top }{\Sigma }^{-1}\mu \right),$$

and apply the formula of Equation (33) with $2{\theta }_M=2{\Sigma }^{-1}={\left(\frac{1}{2}\Sigma \right)}^{-1}$:

$$\begin{array}{ccc}\hfill I\left(p_{\theta }\right)& =& \frac{p_{\theta }^2\left(0\right)}{p_{2\theta }\left(0\right)},\hfill \end{array}$$

(68)

$$\begin{array}{ccc}& =& \frac{{\left(2\pi \right)}^{\frac{d}{2}}{\left|\frac{1}{2}\Sigma \right|}^{\frac{1}{2}}}{{\left(2\pi \right)}^d|\Sigma |}=\frac{1}{2^d{\pi }^{\frac{d}{2}}{|\Sigma |}^{\frac{1}{2}}}.\hfill \end{array}$$

(69)

Let us calculate the formula for the Cauchy-Schwarz divergence between two multivariate Gaussian distributions. We have $\theta \left(\lambda \right)=({\theta }_b,{\theta }_M)=({\Sigma }^{-1}\mu ,{\Sigma }^{-1})$ for $\lambda =(\mu ,\Sigma )$. Conversely, we have $\lambda \left(\theta \right)=({\theta }_M^{-1}{\theta }_v,{\theta }_M^{-1})$. It follows that

$$\lambda ({\theta }_1+{\theta }_2)=\left({({\Sigma }_1^{-1}+{\Sigma }_2^{-1})}^{-1}({\Sigma }_1^{-1}{\mu }_1+{\Sigma }_2^{-1}{\mu }_2),{({\Sigma }_1^{-1}+{\Sigma }_2^{-1})}^{-1}\right).$$

(70)

In particular, we have $\lambda \left(2\theta \right)=(\mu ,\frac{1}{2}\Sigma )$. Let $\omega =0$ so that

$$\begin{array}{ccc}\hfill p_{\lambda }\left(0\right)& =& \frac{1}{{\left(2\pi \right)}^{\frac{d}{2}}\sqrt{|\Sigma |}}exp\left(-\frac{1}{2}{\mu }^{\top }{\Sigma }^{-1}\mu \right),\hfill \end{array}$$

(71)

$$\begin{array}{ccc}\hfill l_{\lambda }\left(0\right)& =& -\frac{d}{2}log\left(2\pi \right)-\frac{1}{2}log|\Sigma |-\frac{1}{2}{\mu }^{\top }{\Sigma }^{-1}\mu .\hfill \end{array}$$

(72)

Thus we get

$$\begin{array}{ccc}\hfill D_{\mathrm{CS}}(p_{{\lambda }_1},p_{{\lambda }_2})& =& l_{\lambda ({\theta }_1+{\theta }_2)}\left(\omega \right)-\frac{l_{\lambda \left(2{\theta }_1\right)}\left(\omega \right)+l_{\lambda 2{\theta }_2}\left(\omega \right)}{2},\hfill \\ \hfill D_{\mathrm{CS}}(p_{{\mu }_1,{\Sigma }_1},p_{{\mu }_2,{\Sigma }_2})& =& \frac{1}{2}log\left(\frac{1}{2^d}\frac{\sqrt{|{\Sigma }_1| {\Sigma }_2\left|\right|}}{|{({\Sigma }_1^{-1}+{\Sigma }_2^{-1})}^{-1}|}\right)\hfill \\ & & +\frac{1}{2}{\mu }_1^{\top }{\Sigma }_1^{-1}{\mu }_1+\frac{1}{2}{\mu }_2^{\top }{\Sigma }_2^{-1}{\mu }_2\hfill \end{array}$$

(73)

$$\begin{array}{ccc}& & -\frac{1}{2}{({\Sigma }_1^{-1}{\mu }_1+{\Sigma }_2^{-1}{\mu }_2)}^{\top }{({\Sigma }_1^{-1}+{\Sigma }_2^{-1})}^{-1}({\Sigma }_1^{-1}{\mu }_1+{\Sigma }_2^{-1}{\mu }_2).\hfill \end{array}$$

(74)

The formula coincides with the formula of Equation (66) when ${\Sigma }_1={\sigma }_1^2$ and ${\Sigma }_2={\sigma }_2^2$.

3.5. Exponential Family of Pareto Distributions

Consider the family of Pareto densities defined by a shape parameter $a>0$ and a prescribed scale parameter $k>0$ as follows:

$$\left\{p_a\left(x\right)=\frac{a{k}^a}{x^{a+1}}, x\in [k,\infty )\right\}.$$

(75)

Writing the density as $p_a\left(x\right)=exp(alogk+loga-(a+1)logx)=p_{\theta \left(a\right)}\left(x\right)$, we deduce that the Pareto densities form an exponential family with natural parameter $\theta =a+1$, sufficient statistic $t\left(x\right)=logx$ and $k\left(x\right)=0$. Let us choose $\omega =k$, and apply the generic formula for the informational energy with $\theta \left(a\right)=a+1$ and ${\theta }^{-1}\left(b\right)=b-1$ (and $2\theta \left(a\right)={\theta }^{-1}(2a+2)=2a+2-1=2a+1$):

$$\begin{array}{ccc}\hfill I\left(p_{\theta }\right)& =& \frac{p_{\theta }^2\left(\omega \right)}{p_{2\theta }\left(\omega \right)},\hfill \end{array}$$

(76)

$$\begin{array}{ccc}\hfill I\left(p_a\right)& =& \frac{p_{\theta \left(a\right)}^2\left(k\right)}{p_{2\theta \left(a\right)}\left(k\right)}=\frac{p_a^2\left(k\right)}{p_{2a+1}\left(k\right)}\hfill \end{array}$$

(77)

$$\begin{array}{ccc}& =& {\left(\frac{a{k}^a}{x^{a+1}}\right)}^2\frac{k^{2a+2}}{(2a+1)k^{2a+1}},\hfill \end{array}$$

(78)

$$\begin{array}{ccc}& =& \frac{a^2}{k(2a+1)}.\hfill \end{array}$$

(79)

3.6. Instantiating Formula with a Computer Algebra System

In general, we can automate the calculations of closed-form formula for conic exponential families using a computer algebra system (CAS) by defining the source-to-natural parameter conversion function $\theta \left(\lambda \right)$, and then apply the formula

$$D_{\mathrm{CS}}(p_{{\lambda }_1},p_{{\lambda }_2})=log\left(\frac{p_{\theta \left({\lambda }_1\right)+\theta \left({\lambda }_2\right)}\left(\omega \right)}{\sqrt{p_{2\theta \left({\lambda }_1\right)}\left(\omega \right)}\sqrt{p_{2\theta \left({\lambda }_2\right)}\left(\omega \right)}}\right), \forall \omega \in \mathcal{X}.$$

(80)

For example, using the CAS Maxima (http://maxima.sourceforge.net/ accessed on 14 April 2022), we can calculate the formula of the information energy of Pareto densities as follows:

• /* Pareto densities form an exponential family */
• assume(k>0);
• assume(a>0);

• Pareto(x,a):=a*(k**a)/(x**(a+1));

• /* check that it is a density (=1) */
• integrate(Pareto(x,a),x,k,inf);

• /* calculate Onicescu’s informational energy */
• integrate(Pareto(x,a)**2,x,k,inf);

• /* method bypassing the integral calculation */
• omega:k;
• (Pareto(omega,a)**2)/Pareto(omega,2*a+1);

4. Informational Energy and the Laws of Thermodynamics

The informational energy was originally motivated by an analogy to kinetic energy in physics, and proves useful when investigating the thermodynamics laws on a statistical manifold [20] where thermodynamic processes [37] can be viewed as oriented trajectories on the manifold as depicted in Figure 1: Indeed, the third law of thermodynamics states that the entropy and the kinetic energy (i.e., informational energy) of an isolated thermodynamical system should vary with opposite directions. Thus when viewing a parametric family $\mathcal{M}=\{p_{\theta } : \theta \in \Theta \}$ of distributions as a statistical manifold [20,38], a valid oriented thermodynamic path $\{p_{\theta \left(t\right)},t\in I\subset \mathbb{R}\}$ (with time t increasing) should satisfy the following condition:

$$\left(H\left(p_{\theta (t+\mathrm{d}t)}\right)-H\left(p_{\theta \left(t\right)}\right)\right) \times \left(I\left(p_{\theta (t+\mathrm{d}t)}\right)-I\left(p_{\theta \left(t\right)}\right)\right)<0.$$

This condition can be written equivalently as the following variational inequality:

$$\frac{d}{\mathrm{d}t}H\left(p_{\theta \left(t\right)}\right) \times \frac{d}{\mathrm{d}t}I\left(p_{\theta \left(t\right)}\right)<0.$$

(81)

We consider the thermodynamic process paths on exponential family manifolds in Section 4.1 and on location-scale manifolds in Section 4.2.

4.1. Exponential Family Manifolds

Consider statistical manifolds induced by exponential families [20], and let us report the variations of entropy and informational energy on those manifolds.

Lemma 1. 

The variation of entropy of a density$p_{\theta \left(t\right)}$of a D-dimensional exponential family with log-normalized $F\left(\theta \right)$for$\theta =({\theta }_1,\dots ,{\theta }_D)$is

$$\frac{d}{\mathrm{d}t}H\left(p_{\theta }\left(t\right)\right)=-\dot{\theta }{\left(t\right)}^{\top } {\nabla }^{2}F\left(\theta \left(t\right)\right) \theta \left(t\right),$$

where$\dot{\theta }\left(t\right)=\left(\frac{d}{\mathrm{d}t}{\theta }_1\left(t\right),\dots ,\frac{d}{\mathrm{d}t}{\theta }_D\left(t\right)\right)$.

Proof. 

The entropy $H\left(p_{\theta }\right)$ of a density $p_{\theta }$ of a (potentially reparametrized) natural exponential family with log-normalizer $F\left(\theta \right)$ is given by [34]:

$$H\left(p_{\theta }\right)=-F^{*}\left(\eta \right)=F\left(\theta \right)-\langle \theta ,\nabla F\left(\theta \right)\rangle ,$$

where $\langle ·,·\rangle$ denotes the scalar product (Euclidean inner product). It follows that we have

$$\begin{array}{ccc}\hfill \frac{d}{\mathrm{d}t}H\left(p_{\theta }\left(t\right)\right)& =& \frac{d}{\mathrm{d}t} \left(F\left(\theta \right(t\left)\right)-\langle \theta \left(t\right),\nabla F\left(\theta \right(t\left)\right)\rangle \right),\hfill \\ & =& \langle \dot{\theta }\left(t\right),\nabla F\left(\theta \left(t\right)\right)\rangle -\langle \dot{\theta }\left(t\right),\nabla F\left(\theta \left(t\right)\right)\rangle -\langle \theta \left(t\right),\langle {\nabla }^{2}F\left(\theta \left(t\right)\right),\dot{\theta }\left(t\right)\rangle \rangle ,\hfill \\ & =& -\dot{\theta }{\left(t\right)}^{\top } {\nabla }^{2}F\left(\theta \left(t\right)\right) \theta \left(t\right).\hfill \end{array}$$

Notice that the Hessian matrix ${\nabla }^{2}F\left(\theta \right)$ is positive-definite since the log-normalizer $F\left(\theta \right)$ is strictly convex for a minimal regular exponential family [20]. □

Next, we report the variation of the informational energy on any thermodynamic process trajectory:

Lemma 2. 

The variation of informational energy of a density$p_{\theta \left(t\right)}$of an exponential family of order D with log-normalizer$F\left(\theta \right)$and zero auxiliary carrier term$k\left(x\right)$is

$$\frac{d}{\mathrm{d}t}I\left(p_{\theta }\left(t\right)\right)=2 \langle \dot{\theta }\left(t\right),\nabla F\left(2\theta \left(t\right)\right)-\nabla F\left(\theta \left(t\right)\right)\rangle I\left(p_{\theta }\left(t\right)\right),$$

provided that$2\theta \in \Theta$.

Proof. 

Assume no extra auxiliary carrier term (i.e., $k\left(x\right)=0$) so that by Theorem 1 we have $I\left(p_{\theta }\right)=exp(F\left(2\theta \right)-2F\left(\theta \right))$. Therefore, we get

$$\frac{d}{\mathrm{d}t}I\left(p_{\theta }\left(t\right)\right)=2\langle \dot{\theta }\left(t\right),\nabla F\left(2\theta \left(t\right)\right)-\nabla F\left(\theta \left(t\right)\right)\rangle I\left(p_{\theta }\left(t\right)\right).$$

□

Thus we can check that whether the statistical normal manifold [20] satisfies the third thermodynamic law everywhere or not as follows: We have $H\left(p_{\mu \left(t\right),\sigma \left(t\right)}\right)=log\left(\sqrt{2\pi e}\sigma \left(t\right)\right)$ and $I\left(p_{\mu \left(t\right),\sigma \left(t\right)}\right)=\frac{1}{2\sqrt{\pi }\sigma \left(t\right)}$. Therefore we get:

$$\begin{array}{ccc}\hfill \frac{d}{\mathrm{d}t}H\left(p_{\theta \left(t\right)}\right)\times \frac{d}{\mathrm{d}t}I\left(p_{\theta \left(t\right)}\right)& =& \frac{\dot{\sigma }\left(t\right)}{\sigma \left(t\right)}\times \left(-\dot{\sigma }\left(t\right)\frac{1}{2\sqrt{\pi }}{\sigma }^2\left(t\right)\right),\hfill \\ & =& -\frac{1}{2\sqrt{\pi }}\frac{\dot{\sigma }{\left(t\right)}^2}{{\sigma }^3\left(t\right)}<0,\hfill \end{array}$$

since $\sigma \left(t\right)>0$ for all t. Any smooth curve on a statistical normal manifold can be a thermodynamic process path satisfying the third law of thermodynamics.

4.2. Location-Scale Manifolds

Consider the setting where $\mathcal{M}$ is a statistical manifold modeling a location-scale family:

$$\mathcal{M}=\left\{p_{l,s}\left(x\right)=\frac{1}{s} p_{\mathrm{std}}\left(\frac{x-l}{s}\right), (l,s)\in (\mathbb{R},{\mathbb{R}}_{++})\right\},$$

where $p_{\mathrm{std}}\left(x\right)$ denotes the standard probability density function of the family (i.e., $p_{\mathrm{std}}\left(x\right)=p_{0,1}\left(x\right)$). The family of normal distributions and the family of Cauchy distributions are two examples of location-scale families with standard density $p_{\mathrm{std}}\left(x\right)=\frac{1}{\sqrt{2\pi }} exp\left(-\frac{1}{2}{x}^2\right)$ and $p_{\mathrm{std}}\left(x\right)=\frac{1}{\pi (1+x^2)}$, respectively.

By a change of variable $y=\frac{x-l}{s}$ in the integral definitions of the entropy (Equation (4)) and the informational energy (Equation (1)) of a location-scale probability density function $p_{l,s}\left(x\right)$, we find that

$$\begin{array}{ccc}\hfill H\left(p_{l,s}\right)& =& H\left(p_{\mathrm{std}}\right)+logs,\hfill \end{array}$$

(82)

$$\begin{array}{ccc}\hfill I\left(p_{l,s}\right)& =& \frac{1}{s} I\left(p_{\mathrm{std}}\right).\hfill \end{array}$$

(83)

The informational energy of the standard normal and Cauchy distributions are $\frac{1}{2\sqrt{\pi }}$ (since ${\int }_{-\infty }^{\infty }{e}^{-x^2}\mathrm{d}x=\sqrt{\pi }$) and $\frac{1}{2}$ (since ${\int }_{-\infty }^{\infty }\frac{1}{{(1+x^2)}^2}\mathrm{d}x=\frac{\pi }{2}$), respectively. Thus the informational energy of the normal distributions $N(\mu ,\sigma )$ and the Cauchy distributions $C(l,s)$ are $\frac{1}{2\sigma \sqrt{\pi }}$ (see also Table 2) and $\frac{1}{2s}$, respectively.

Using a first-order Taylor expansion on the scale parameter

$$s(t+\mathrm{d}t)\approx s\left(t\right)+\dot{s}\left(t\right)\mathrm{d}t,$$

we get

$$\frac{d}{\mathrm{d}t}H\left(p_{l\left(t\right),s\left(t\right)}\right)=\dot{s}\left(t\right),$$

and

$$\frac{d}{\mathrm{d}t}I\left(p_{l\left(t\right),s\left(t\right)}\right)=-\frac{\dot{s}\left(t\right)}{s^2\left(t\right)} I\left(p_{\mathrm{std}}\right).$$

Therefore, we have

$$\frac{d}{\mathrm{d}t}H\left(p_{\theta \left(t\right)}\right)\times \frac{d}{\mathrm{d}t}I\left(p_{\theta \left(t\right)}\right)=-\frac{\dot{s}{\left(t\right)}^2}{s^3\left(t\right)} I\left(p_{\mathrm{std}}\right)<0,$$

since $s\left(t\right)>0$ and $I\left(p_{\mathrm{std}}\right)>0$ for any (standard) probability density function. Thus all smooth paths on a location-scale manifold are compatible with the third law of thermodynamics.

Note that the second law of thermodynamics states that the entropy of an isolated system increases over time. That is, for a thermodynamic process $\{p_{\theta \left(t\right)},t\in I\subset \mathbb{R}\}$, we shall have $H\left(p_{\theta \left(t\right)}\right)<H\left(p_{\theta (t+\mathrm{d}t)}\right)$. On a statistical location-scale manifold with differential entropy $H\left(p_{l\left(t\right),s\left(t\right)}\right)=H\left(p_{\mathrm{std}}\right)+logs\left(t\right)$, we have

$$\frac{d}{\mathrm{d}t} H\left(p_{l\left(t\right),s\left(t\right)}\right)=\frac{\dot{s}\left(t\right)}{s\left(t\right)}.$$

Thus the entropy of a thermodynamic process increases whenever the scale $s\left(t\right)$ increases with time (i.e., $\dot{s}\left(t\right)>0$) since $s\left(t\right)>0$. Therefore not all oriented thermodynamic process paths on a location-scale manifold satisfies the second law of thermodynamics but only paths with increasing scales.

5. Summary and Discussion

Shannon’s entropy [11] and Onicescu’s informational energy [5] are two complementary measures of uncertainty of a random variable: When randomness increases, Shannon’s entropy increases but Onicescu’s informational energy decreases. Table 1 compares the properties of these randomness measures: Interestingly, the Onicescu’s informational energy is strictly convex and is always positive while Shannon’s entropy is strictly concave and can be negative for continuous random variables (e.g., differential entropy of a normal distribution with small variance). The fundamental quantity of the informational energy of Equation (1) has been studied in various other fields under different names: Simpson’s diversity index [16] or Gini-Simpson index [18] in ecology, Herfindahl–Hirschman index [13] in economics, or the index of coincidence [14] in cryptanalysis. In this work, we show how to compute the informational energy for a distribution belonging to an exponential family in Theorem 1, and reported closed-form formula for its associated correlation coefficient and related statistical divergences. We then show how to characterize thermodynamic processes satisfying the third law of thermodynamics by interpreting them as smooth oriented trajectory paths on a statistical manifold and asserting that the entropy and the informational energy (interpreted as the kinetic energy) shall vary in opposite directions.