Divergences Induced by the Cumulant and Partition Functions of Exponential Families and Their Deformations Induced by Comparative Convexity

Abstract

Exponential families are statistical models which are the workhorses in statistics, information theory, and machine learning, among others. An exponential family can either be normalized subtractively by its cumulant or free energy function, or equivalently normalized divisively by its partition function. Both the cumulant and partition functions are strictly convex and smooth functions inducing corresponding pairs of Bregman and Jensen divergences. It is well known that skewed Bhattacharyya distances between the probability densities of an exponential family amount to skewed Jensen divergences induced by the cumulant function between their corresponding natural parameters, and that in limit cases the sided Kullback–Leibler divergences amount to reverse-sided Bregman divergences. In this work, we first show that the $\alpha$-divergences between non-normalized densities of an exponential family amount to scaled $\alpha$-skewed Jensen divergences induced by the partition function. We then show how comparative convexity with respect to a pair of quasi-arithmetical means allows both convex functions and their arguments to be deformed, thereby defining dually flat spaces with corresponding divergences when ordinary convexity is preserved.

1. Introduction

In information geometry [1], any strictly convex and smooth function induces a dually flat space (DFS) with a canonical divergence which can be expressed in charts either as dual Bregman divergences [2] or equivalently as dual Fenchel–Young divergences [3]. For example, the cumulant function of an exponential family [4] (also called the free energy) generates a DFS, that is, an exponential family manifold [5] with the canonical divergence yielding the reverse Kullback–Leibler divergence. Another typical example of a strictly convex and smooth function generating a DFS is the negative entropy of a mixture family, that is, a mixture family manifold with the canonical divergence yielding the (forward) Kullback–Leibler divergence [3]. In addition, any strictly convex and smooth function induces a family of scaled skewed Jensen divergences [6,7], which in limit cases includes the sided forward and reverse Bregman divergences.

In Section 2, we present two equivalent approaches to normalizing an exponential family: first by its cumulant function, and second by its partition function. Because both the cumulant and partition functions are strictly convex and smooth, they induce corresponding families of scaled skewed Jensen divergences and Bregman divergences, with corresponding dually flat spaces and related statistical divergences.

In Section 3, we recall the well-known result that the statistical $\alpha$-skewed Bhattacharyya distances between the probability densities of an exponential family amount to a scaled $\alpha$-skewed Jensen divergence between their natural parameters. In Section 4, we prove that the $\alpha$-divergences [8] between the unnormalized densities of a exponential family amount to scaled $\alpha$-skewed Jensen divergence between their natural parameters (Proposition 5). More generally, we explain in Section 5 how to deform a convex function using comparative convexity [9]: When the ordinary convexity of the deformed convex function is preserved, we obtain new skewed Jensen divergences and Bregman divergences with corresponding dually flat spaces. Finally, Section 6 concludes this work with a discussion.

2. Dual Subtractive and Divisive Normalizations of Exponential Families

2.1. Natural Exponential Families

Let $(\mathcal{X},\mathcal{A},\mu )$ be a measure space [10], where $\mathcal{X}$ denotes the sample set (e.g., finite alphabet, $\mathbb{N}$, ${\mathbb{R}}^d$, space of positive-definite matrices ${\mathrm{Sym}}_{++}\left(d\right)$, etc.), $\mathcal{A}$ a $\sigma$-algebra on $\mathcal{X}$ (e.g., power set $2^{\mathcal{X}}$, Borel $\sigma$-algebra $\mathcal{B}\left(\mathcal{X}\right)$, etc.), and $\mu$ a positive measure (e.g., counting measure or Lebesgue measure) on the measurable space $(\mathcal{X},\mathcal{A})$.

A natural exponential family [4,11] (commonly abbreviated as NEF [12]) is a set of probability distributions $\mathcal{P}=\{P_{\theta }:\theta \in \mathrm{\Theta }\}$ all dominated by $\mu$ such that their Radon–Nikodym densities $p_{\theta }\left(x\right)=\frac{\mathrm{d}{P}_{\theta }}{\mathrm{d}\mu }\left(x\right)$ can be expressed canonically as

$$p_{\theta }\left(x\right)\propto {\tilde{p}}_{\theta }\left(x\right)=exp\left(\sum _{i=1}^m{\theta }_i{x}_i\right),$$

(1)

where $\theta$ is called the natural parameter and $x=(x_1,\dots ,x_m)$ denotes the linear sufficient statistic vector [11]. The order of the NEF [13] is m. When the parameter $\theta$ ranges in the full natural parameter space

$$\mathrm{\Theta }=\left\{\theta :{\int }_{\mathcal{X}}{\tilde{p}}_{\theta }\left(x\right)\mathrm{d}\mu \left(x\right)<\infty \right\}\subset {\mathbb{R}}^m,$$

the family is called full. The NEF is said to be regular when $\mathrm{\Theta }$ is topologically open.

The unnormalized positive density ${\tilde{p}}_{\theta }\left(x\right)$ is indicated with a tilde notation and the corresponding normalized probability density is obtained as $p_{\theta }\left(x\right)=\frac{1}{Z\left(\theta \right)}{\tilde{p}}_{\theta }\left(x\right)$, where $Z\left(\theta \right)=\int {\tilde{p}}_{\theta }\left(x\right)\mathrm{d}\mu \left(x\right)$ is the Laplace transform of $\mu$ (the density normalizer). For example, the family of exponential distributions $\mathcal{E}=\{\lambda e^{-\lambda x}:\lambda >0\}$ is an NEF with densities defined on the support $\mathcal{X}={\mathbb{R}}_{\ge 0}=\{x\in \mathbb{R}:x\ge 0\}$, natural parameter $\theta =-\lambda$ in $\mathrm{\Theta }={\mathbb{R}}_{<0}=\{\theta \in \mathbb{R}:\theta <0\}$, sufficient linear statistic x, and normalizer $Z\left(\theta \right)=-\frac{1}{\theta }$.

2.2. Exponential Families

More generally, exponential families include many well known distributions after reparameterization [4] of their ordinary parameter $\lambda$ by $\theta \left(\lambda \right)$. The general canonical form of the densities of an exponential family is

$$p_{\lambda }\left(x\right)\propto {\tilde{p}}_{\lambda }\left(x\right)=exp\left(〈\theta \left(\lambda \right),t\left(x\right)〉\right)h\left(x\right),$$

(2)

where $t\left(x\right)=(t_1\left(x\right),\dots ,t_m\left(x\right))$ are the sufficient statistic vector (such that $1,t_1\left(x\right),\dots ,t_m\left(x\right)$ are linearly independent), $h\left(x\right)$ is an auxiliary term used to define the base measure with respect to $\mu$, and $〈·,·〉$ is an inner product (e.g., scalar product of ${\mathbb{R}}^m$, trace product of symmetric matrices, etc.). By defining a new measure $\nu$ such that $\frac{\mathrm{d}\mu }{\mathrm{d}\nu }\left(x\right)=h\left(x\right)$, we may consider without loss of generality the densities ${\overline{p}}_{\lambda }\left(x\right)=\frac{\mathrm{d}{P}_{\lambda }}{\mathrm{d}\nu }\left(x\right)$ with $h\left(x\right)=1$.

For example, the Bernoulli distributions, Gaussian or normal distributions, Gamma and Beta distributions, Poisson distributions, Rayleigh distributions, and Weibull distributions with prescribed shape parameter are just a few examples of exponential families with the inner product on ${\mathbb{R}}^m$ defined as the scalar product. The categorical distributions (i.e., discrete distributions on a finite alphabet sample space) form an exponential family as well [1]. Zero-centered Gaussian distributions and Wishart distributions are examples of exponential families parameterized by positive-definite matrices with inner products defined by the matrix trace product, which is $〈A,B〉=\mathrm{tr}\left(AB\right)$.

Exponential families abound in statistics and machine learning. Any two probability measures Q and R with densities q and r with respect to a dominating measure, say, $\mu =\frac{Q+R}{2}$, define an exponential family

$${\mathcal{P}}_{Q,R}=\left\{p_{\lambda }\left(x\right)\propto q^{\lambda }\left(x\right)r^{1-\lambda }\left(x\right):\lambda \in (0,1)\right\},$$

which is called the likelihood ratio exponential family [14], as the sufficient statistic is $t\left(x\right)=log\frac{q\left(x\right)}{r\left(x\right)}$ (with auxiliary carrier term $h\left(x\right)=r\left(x\right)$), or the Bhattacharyya arc, as the cumulant function of ${\mathcal{P}}_{Q,R}$ is expressed as the negative of the skewed Bhattacharyya distances [7,15].

In machine learning, undirected graphical models [16] and energy-based models [17], including Markov random fields [18] and conditional random fields, are exponential families [19]. Exponential families are universal approximators of smooth densities [20].

From a theoretical standpoint, it is often enough to consider (without loss of generality) natural exponential families with densities expressed as in Equation (1). However, here we consider generic exponential families with the densities expressed in Equation (2) in order to report common examples encountered in practice, such as the multivariate Gaussian family [21].

When the natural parameter space $\mathrm{\Theta }$ is not full but rather parameterized by $\lambda =c\left({\lambda }^{\prime }\right)$ for ${\lambda }^{\prime }\in {\Lambda }^{\prime }$ with $\dim \left({\Lambda }^{\prime }\right)<m$ and a smooth function $c\left(u\right)$, the exponential family is called a curved exponential family [1]. For example, the special family of normal distributions $\{p_{\mu ,{\sigma }^2={\mu }^2}:\mu \in \mathbb{R}\}$ is a curved exponential family with $u=\mu$ and $c\left(u\right)=(u,u^2)$ [1].

2.3. Normalizations of Exponential Families

Recall that ${\tilde{p}}_{\theta }\left(x\right)=exp\left(〈\theta ,t\left(x\right)〉\right)h\left(x\right)$ denotes the unnormalized density expressed using the natural parameter $\theta =\theta \left(\lambda \right)$. We can normalize ${\tilde{p}}_{\theta }\left(x\right)$ using either the partition function $Z\left(\theta \right)$ or equivalently using the cumulant function $F\left(\theta \right)$, as follows:

$$\begin{array}{ccc}\hfill p_{\theta }\left(x\right)& =& \frac{exp\left(〈\theta ,t\left(x\right)〉\right)}{Z\left(\theta \right)}h\left(x\right),\hfill \end{array}$$

(3)

$$\begin{array}{ccc}& =& exp\left(〈\theta ,t\left(x\right)〉-F\left(\theta \right)+k\left(x\right)\right),\hfill \end{array}$$

(4)

where $h\left(x\right)=exp\left(k\right(x\left)\right)$, $Z\left(\theta \right)=\int {\tilde{p}}_{\theta }\left(x\right)\mathrm{d}\mu \left(x\right)$, and $F\left(\theta \right)=logZ\left(\theta \right)=log\int {\tilde{p}}_{\theta }\left(x\right)\mathrm{d}\mu \left(x\right)$. Thus, the logarithm and exponential functions allow conversion to and from the dual normalizers Z and F:

$$Z\left(\theta \right)=exp\left(F\right(\theta \left)\right)\iff F\left(\theta \right)=logZ\left(\theta \right).$$

We may view Equation (3) as an exponential tilting [13] of density $h\left(x\right)\mathrm{d}\mu \left(x\right)$.

In the context of $\lambda$-deformed exponential families [22] which generalize exponential families, the function $Z\left(\theta \right)$ is called the divisive normalization factor (Equation (3)) and the function $F\left(\theta \right)$ is called the subtractive normalization factor (Equation (4)). Notice that $F\left(\theta \right)$ is called the cumulant function because when $X\sim p_{\theta }\left(x\right)$ is a random variable following a probability distribution of an exponential family, the function $F\left(\theta \right)$ appears in the cumulant generating function of X: $K_X\left(t\right)=log{E}_X\left[e^{〈t,X〉}\right]=F(\theta +t)-F\left(\theta \right)$. In statistical physics, the cumulant function is called the log-normalizer or log-partition function. Because $Z>0$ and $F=logZ$, we can deduce that $F\ge Z$, as $logx\le x$ for $x>0$.

It is well known that the cumulant function $F\left(\theta \right)$ is a strictly convex function and that the partition function $Z\left(\theta \right)$ is strictly log-convex [11].

Proposition 1

([11]). The natural parameter space Θ of an exponential family is convex.

Proposition 2

([11]). The cumulant function $F\left(\theta \right)$ is strictly convex and the partition function $Z\left(\theta \right)$ is positive and strictly log-convex.

It can be shown that the cumulant and partition functions are smooth $C^{\infty }$ analytic functions [4]. A remarkable property is that strictly log-convex functions are also strictly convex.

Proposition 3

([23], Section 3.5). A strictly log-convex function $Z:\mathrm{\Theta }\subset {\mathbb{R}}^m\to \mathbb{R}$ is strictly convex.

The converse of Proposition 3 is not necessarily true, however; certain convex functions are not log-convex, and as such the class of strictly log-convex functions is a proper subclass of strictly convex functions. For example, ${\theta }^2$ is convex but log-concave, as ${(log{\theta }^2)}^{″}=-\frac{2}{{\theta }^2}<0$ (Figure 1).

Remark 1.

Because $Z=exp\left(F\right)$ is strictly convex (Proposition 3), F is exponentially convex.

Definition 1.

The cumulant function F and partition function Z of a regular exponential family are both strictly convex and smooth functions inducing a pair of dually flat spaces with corresponding Bregman divergences [2] $B_F$ (i.e., $B_{logZ}$) and $B_Z$ (i.e., $B_{expF}$):

$$\begin{array}{ccc}\hfill B_Z({\theta }_1:{\theta }_2)& =& Z\left({\theta }_1\right)-Z\left({\theta }_2\right)-〈{\theta }_1-{\theta }_2,\nabla Z\left({\theta }_2\right)〉\ge 0,\hfill \end{array}$$

(5)

$$\begin{array}{ccc}\hfill B_{logZ}({\theta }_1:{\theta }_2)& =& log\left(\frac{Z\left({\theta }_1\right)}{Z\left({\theta }_2\right)}\right)-〈{\theta }_1-{\theta }_2,\frac{\nabla Z\left({\theta }_2\right)}{Z\left({\theta }_2\right)}〉\ge 0,\hfill \end{array}$$

(6)

along with a pair of families of skewed Jensen divergences $J_{F,\alpha }$ and $J_{Z,\alpha }$:

$$\begin{array}{ccc}\hfill J_{Z,\alpha }({\theta }_1:{\theta }_2)& =& \alpha Z\left({\theta }_1\right)+(1-\alpha )Z\left({\theta }_2\right)-Z(\alpha {\theta }_1+(1-\alpha ){\theta }_2)\ge 0,\hfill \end{array}$$

(7)

$$\begin{array}{ccc}\hfill J_{logZ,\alpha }({\theta }_1:{\theta }_2)& =& log\frac{Z{\left({\theta }_1\right)}^{\alpha }Z{\left({\theta }_2\right)}^{1-\alpha }}{Z(\alpha {\theta }_1+(1-\alpha ){\theta }_2)}\ge 0.\hfill \end{array}$$

(8)

For a strictly convex function $F\left(\theta \right)$, we define the symmetric Jensen divergence as follows:

$$J_F({\theta }_1,{\theta }_2)=J_{F,\frac{1}{2}}({\theta }_1:{\theta }_2)=\frac{F\left({\theta }_1\right)+F\left({\theta }_2\right)}{2}-F\left(\frac{{\theta }_1+{\theta }_2}{2}\right).$$

Let ${\mathcal{B}}_{\mathrm{\Theta }}$ denote the set of real-valued strictly convex and differentiable functions defined on an open set $\mathrm{\Theta }$, called Bregman generators. We may equivalently consider the set of strictly concave and differentiable functions $G\left(\theta \right)$ and let $F\left(\theta \right)=-G\left(\theta \right)$; see [24] (Equation (1)).

Remark 2.

The non-negativeness of the Bregman divergences for the cumulant and partition functions define the criteria for checking the strict convexity or log-convexity of a $C^1$ function:

$$\begin{array}{ccc}\hfill F\left(\theta \right) is strictly convex& \iff & \forall {\theta }_1\ne {\theta }_2,B_F({\theta }_1:{\theta }_2)>0,\hfill \\ & \iff & \forall {\theta }_1\ne {\theta }_2,F\left({\theta }_1\right)>F\left({\theta }_2\right)+〈{\theta }_1-{\theta }_2,\nabla F\left(\theta \right)〉,\hfill \end{array}$$

and

$$\begin{array}{ccc}\hfill Z\left(\theta \right) is strictly log-convex& \iff & \forall {\theta }_1\ne {\theta }_2,B_{logZ}({\theta }_1:{\theta }_2)>0,\hfill \\ & \iff & \forall {\theta }_1\ne {\theta }_2,logZ\left({\theta }_1\right)>logZ\left({\theta }_2\right)+〈{\theta }_1-{\theta }_2,\frac{\nabla Z\left({\theta }_2\right)}{Z\left({\theta }_2\right)}〉.\hfill \end{array}$$

The forward Bregman divergence $B_F({\theta }_1:{\theta }_2)$ and reverse Bregman divergence $B_F({\theta }_2:{\theta }_1)$ can be unified with the $\alpha$-skewed Jensen divergences by rescaling $J_{F,\alpha }$ and allowing $\alpha$ to range in $\mathbb{R}$ [6,7]:

$$J_{F,\alpha }^s({\theta }_1:{\theta }_2)=\left\{\begin{array}{cc}\frac{1}{\alpha (1-\alpha )}{J}_{F,\alpha }({\theta }_1:{\theta }_2),\hfill & \alpha \in \mathbb{R}\setminus \{0,1\},\hfill \\ B_F({\theta }_1:{\theta }_2),\hfill & \alpha =0,\hfill \\ 4J_F({\theta }_1,{\theta }_2),\hfill & \alpha =\frac{1}{2},\hfill \\ B_F^{*}({\theta }_1:{\theta }_2)=B_F({\theta }_2:{\theta }_1),\hfill & \alpha =1.\hfill \end{array}\right.,$$

(9)

where ${B_F}^{*}$ denotes the reverse Bregman divergence obtained by swapping the parameter order (reference duality [6]): ${B_F}^{*}({\theta }_1:{\theta }_2)=B_F({\theta }_2:{\theta }_1)$.

Remark 3.

Alternatively, we may rescale $J_F$ by a factor $\kappa \left(\alpha \right)=\frac{1}{\alpha (1-\alpha )4^{4\alpha (1-\alpha )}}$, i.e., $J_{F,\alpha }^{\overline{s}}({\theta }_1:{\theta }_2)=\kappa \left(\alpha \right)J_{F,\alpha }({\theta }_1:{\theta }_2)$ such that $\kappa \left(\frac{1}{2}\right)=1$ and $J_{F,\frac{1}{2}}^{\overline{s}}({\theta }_1:{\theta }_2)=J_F({\theta }_1,{\theta }_2)$.

Next, in Section 3 we first recall the connections between these Jensen and Bregman divergences, which are divergences between parameters, and the statistical divergence counterparts between probability densities. Then, in Section 4 we introduce the novel connections between these parameter divergences and $\alpha$-divergences between unnormalized densities.

3. Divergences Related to the Cumulant Function

Consider the scaled $\alpha$-skewed Bhattacharyya distances [7,15] between two probability densities $p\left(x\right)$ and $q\left(x\right)$:

$$D_{B,\alpha }^s(p:q)=-\frac{1}{\alpha (1-\alpha )}log\int p^{\alpha }{q}^{1-\alpha }\mathrm{d}\mu ,\alpha \in \mathbb{R}\setminus \{0,1\}.$$

The scaled $\alpha$-skewed Bhattacharyya distances can additionally be interpreted as Rényi divergences [25] scaled by $\frac{1}{\alpha }$: $D_{B,\alpha }^s(p:q)=\frac{1}{\alpha }{D}_{R,\alpha }(p:q)$, where the Rényi $\alpha$-divergences are defined by

$$D_{R,\alpha }(p:q)=\frac{1}{\alpha -1}log\int p^{\alpha }{q}^{1-\alpha }\mathrm{d}\mu .$$

The Bhattacharyya distance $D_B(p,q)=-log\int \sqrt{pq}\mathrm{d}\mu$ corresponds to one-fourth of $D_{B,\frac{1}{2}}^s(p:q)$: $D_B(p,q)=\frac{1}{4}{D}_{B,\frac{1}{2}}^s(p:q)$. Because $D_{B,\alpha }^s$ tends to the Kullback–Leibler divergence $D_{\mathrm{KL}}$ when $\alpha \to 1$ and to the reverse Kullback–Leibler divergence ${D_{\mathrm{KL}}}^{*}$ when $\alpha \to 0$, we have

$$D_{B,\alpha }^s(p:q)=\left\{\begin{array}{cc}-\frac{1}{\alpha (1-\alpha )}log\int p^{\alpha }{q}^{1-\alpha }\mathrm{d}\mu ,\hfill & \alpha \in \mathbb{R}\setminus \{0,1\},\hfill \\ D_{\mathrm{KL}}(p:q),\hfill & \alpha =1,\hfill \\ 4D_B(p,q)\hfill & \alpha =\frac{1}{2},\hfill \\ {D_{\mathrm{KL}}}^{*}(p:q)=D_{\mathrm{KL}}(q:p)\hfill & \alpha =0.\hfill \end{array}\right.$$

When both probability densities belong to the same exponential family $\mathcal{E}=\{p_{\theta }\left(x\right):\theta \in \mathrm{\Theta }\}$ with cumulant $F\left(\theta \right)$, we have the following proposition.

Proposition 4

([7]). The scaled α-skewed Bhattacharyya distances between two probability densities $p_{{\theta }_1}$ and $p_{{\theta }_2}$ of an exponential family amount to the scaled α-skewed Jensen divergence between their natural parameters:

$$D_{B,\alpha }^s(p_{{\theta }_1}:p_{{\theta }_2})=J_{F,\alpha }^s({\theta }_1,{\theta }_2).$$

(10)

Proof. 

The proof follows by first considering the $\alpha$-skewed Bhattacharyya similarity coefficient ${\rho }_{\alpha }(p,q)=\int p^{\alpha }{q}^{1-\alpha }\mathrm{d}\mu$.

$$\begin{array}{ccc}\hfill {\rho }_{\alpha }(p_{{\theta }_1}:p_{{\theta }_2})& =& \int exp{\left(〈{\theta }_1,x〉-F\left({\theta }_1\right)\right)}^{\alpha }exp{\left(〈{\theta }_2,x〉-F\left({\theta }_2\right)\right)}^{1-\alpha }\mathrm{d}\mu ,\hfill \\ & =& \int exp\left(〈\alpha {\theta }_1+(1-\alpha ){\theta }_2),x〉\right)exp\left(-(\alpha F\left({\theta }_1\right)+(1-\alpha )F\left({\theta }_2\right))\right)\mathrm{d}\mu .\hfill \end{array}$$

Multiplying the last equation by $exp\left(F(\alpha {\theta }_1+(1-\alpha ){\theta }_2)\right)exp(-F(\alpha {\theta }_1+(1-\alpha ){\theta }_2))=exp\left(0\right)=1$ with $\overline{\theta }=\alpha {\theta }_1+(1-\alpha ){\theta }_2$, we obtain

$${\rho }_{\alpha }(p_{{\theta }_1}:p_{{\theta }_2})=exp(-(\alpha F\left({\theta }_1\right)+(1-\alpha )F\left({\theta }_2\right))exp\left(F\left(\overline{\theta }\right)\right)\int exp(〈\overline{\theta },x〉-F\left(\overline{\theta }\right))\mathrm{d}\mu .$$

Because $\overline{\theta }\in \mathrm{\Theta }$, we have $\int exp(〈\overline{\theta },x〉-F\left(\overline{\theta }\right))\mathrm{d}\mu =1$; therefore, we obtain

$${\rho }_{\alpha }(p_{{\theta }_1}:p_{{\theta }_2})=exp(-J_{F,\alpha }({\theta }_1:{\theta }_2)).$$

□

For practitioners in machine learning, it is well known that the Kullback–Leibler divergence between two probability densities $p_{{\theta }_1}$ and $p_{{\theta }_2}$ of an exponential family amounts to a Bregman divergence for the cumulant generator on a swapped parameter order (e.g., [26,27]):

$$D_{\mathrm{KL}}(p_{{\theta }_1}:p_{{\theta }_2})=B_F({\theta }_2:{\theta }_1).$$

This is a particular instance of Equation (10) obtained for $\alpha =1$:

$$D_{B,1}^s(p_{{\theta }_1}:p_{{\theta }_2})=J_{F,1}^s({\theta }_1,{\theta }_2).$$

This formula has been further generalized in [28] by considering truncations of exponential family densities. Let ${\mathcal{X}}_1\subseteq {\mathcal{X}}_2\subseteq \mathcal{X}$ and ${\mathcal{E}}_1=\left\{1_{{\mathcal{X}}_1}\left(x\right)p_{\theta }\left(x\right)\right\}$, ${\mathcal{E}}_2=\left\{1_{{\mathcal{X}}_2}\left(x\right)q_{{\theta }^{\prime }}\left(x\right)\right\}$ be two truncated families of $\mathcal{X}$ with corresponding cumulant functions

$$F_1\left(\theta \right)=log{\int }_{{\mathcal{X}}_1}exp\left(〈t\left(x\right),\theta 〉\right)\mathrm{d}\mu$$

and

$$F_2\left({\theta }^{\prime }\right)=log{\int }_{{\mathcal{X}}_2}exp\left(〈t\left(x\right),\theta 〉\right)\mathrm{d}\mu \ge F_1\left({\theta }^{\prime }\right).$$

Then, we have

$$\begin{array}{ccc}\hfill D_{\mathrm{KL}}(p_{{\theta }_1}:q_{{\theta }_2^{\prime }})& =& B_{F_2,F_1}({\theta }_2^{\prime }:{\theta }_1),\hfill \\ & =& F_2\left({\theta }_2^{\prime }\right)-F_1\left({\theta }_1\right)-〈{\theta }_2^{\prime }-{\theta }_1,\nabla F_1\left({\theta }_1\right)〉.\hfill \end{array}$$

Truncated exponential families are normalized exponential families which may not be regular [29], i.e., the parameter space $\mathrm{\Theta }$ may not be open.

4. Divergences Related to the Partition Function

Certain exponential families have intractable cumulant/partition functions (e.g., exponential families with sufficient statistics $t\left(x\right)=(x,x^2,\dots ,x^m)$ for high degrees m [20]) or cumulant/partition functions which require exponential time to compute [30] (e.g., graphical models [16], high-dimensional grid sample spaces, energy-based models [17] in deep learning, etc.). In such cases, the maximum likelihood estimator (MLE) cannot be used to infer the natural parameter of exponential densities. Many alternative methods have been proposed to handle such exponential families with untractable partition functions, e.g., score matching [31] or divergence-based inference [32,33]). Thus, it is important to consider dissimilarities between non-normalized statistical models.

The squared Hellinger distance [1] between two positive potentially unnormalized densities $\tilde{p}$ and $\tilde{q}$ is defined by

$$\begin{array}{ccc}\hfill D_H^2(\tilde{p},\tilde{q})& =& \frac{1}{2}\int {(\sqrt{\tilde{p}}-\sqrt{\tilde{q}})}^2\mathrm{d}\mu ,\hfill \\ & =& \frac{\int \tilde{p}\mathrm{d}\mu +\int \tilde{q}\mathrm{d}\mu }{2}-\int \sqrt{\tilde{p}\tilde{q}}\mathrm{d}\mu .\hfill \end{array}$$

Notice that the Hellinger divergence can be interpreted as the integral of the difference between the arithmetical mean $A(\tilde{p},\tilde{q})=\frac{\tilde{p}+\tilde{q}}{2}$ minus the geometrical mean $G(\tilde{p},\tilde{q})=\sqrt{\tilde{p}\tilde{q}}$ of the densities: $D_H^2(\tilde{p},\tilde{q})=\int (A(\tilde{p},\tilde{q})-G(\tilde{p},\tilde{q}))\mathrm{d}\mu$. This further proves that $D_H(\tilde{p},\tilde{q})\ge 0$, as $A\ge G$. The Hellinger distance $D_H$ satisfies the metric axioms of distances.

When considering unnormalized densities ${\tilde{p}}_{{\theta }_1}=exp\left(〈t\left(x\right),{\theta }_1〉\right)$ and ${\tilde{p}}_{{\theta }_2}=exp\left(〈t\left(x\right),{\theta }_2〉\right)$ of an exponential family $\mathcal{E}$ with a partition function $Z\left(\theta \right)=\int {\tilde{p}}_{\theta }\mathrm{d}\mu$, we obtain

$$D_H^2({\tilde{p}}_{{\theta }_1},{\tilde{p}}_{{\theta }_2})=\frac{Z\left({\theta }_1\right)+Z\left({\theta }_2\right)}{2}-Z\left(\frac{{\theta }_1+{\theta }_2}{2}\right)=J_Z({\theta }_1,{\theta }_2),$$

(11)

as $\sqrt{{\tilde{p}}_{{\theta }_1}{\tilde{p}}_{{\theta }_2}}={\tilde{p}}_{\frac{{\theta }_1+{\theta }_2}{2}}$.

The Kullback–Leibler divergence [1] as extended to two positive densities $\tilde{p}$ and $\tilde{q}$ is defined by

$$D_{\mathrm{KL}}(\tilde{p}:\tilde{q})=\int \left(\tilde{p}log\frac{\tilde{p}}{\tilde{q}}+\tilde{q}-\tilde{p}\right)\mathrm{d}\mu .$$

(12)

When considering unnormalized densities ${\tilde{p}}_{{\theta }_1}$ and ${\tilde{p}}_{{\theta }_2}$ of $\mathcal{E}$, we obtain

$$\begin{array}{ccc}\hfill D_{\mathrm{KL}}({\tilde{p}}_{{\theta }_1}:{\tilde{p}}_{{\theta }_2})& =& \int \left({\tilde{p}}_{{\theta }_1}\left(x\right)log\frac{{\tilde{p}}_{{\theta }_1}\left(x\right)}{{\tilde{p}}_{{\theta }_2}\left(x\right)}+{\tilde{p}}_{{\theta }_2}\left(x\right)-{\tilde{p}}_{{\theta }_1}\left(x\right)\right)\mathrm{d}\mu \left(x\right),\hfill \end{array}$$

(13)

$$\begin{array}{ccc}& =& \int \left(e^{〈t\left(x\right),{\theta }_1〉}〈{\theta }_1-{\theta }_2,t\left(x\right)〉+e^{〈t\left(x\right),{\theta }_2〉}-e^{〈t\left(x\right),{\theta }_1〉}\right)\mathrm{d}\mu \left(x\right),\hfill \end{array}$$

(14)

$$\begin{array}{ccc}& =& 〈\int t\left(x\right)e^{〈t\left(x\right),{\theta }_1〉}\mathrm{d}\mu \left(x\right),{\theta }_1-{\theta }_2〉+Z\left({\theta }_2\right)-Z\left({\theta }_1\right),\hfill \end{array}$$

(15)

$$\begin{array}{ccc}& =& 〈{\theta }_1-{\theta }_2,\nabla Z\left({\theta }_1\right)〉+Z\left({\theta }_2\right)-Z\left({\theta }_1\right)=B_Z({\theta }_2:{\theta }_1),\hfill \end{array}$$

(16)

as $\nabla Z\left(\theta \right)=\int t\left(x\right){\tilde{p}}_{\theta }\left(x\right)\mathrm{d}\mu \left(x\right)$. Let $D_{\mathrm{KL}}^{*}(\tilde{p}:\tilde{q})=D_{\mathrm{KL}}(\tilde{q}:\tilde{p})$ denote the reverse KLD.

More generally, the family of $\alpha$-divergences [1] between the unnormalized densities $\tilde{p}$ and $\tilde{q}$ is defined for $\alpha \in \mathbb{R}$ by

$$D_{\alpha }(\tilde{p}:\tilde{q})=\left\{\begin{array}{cc}\frac{1}{\alpha (1-\alpha )}\int \left(\alpha \tilde{p}+(1-\alpha )\tilde{q}-{\tilde{p}}^{\alpha }{\tilde{q}}^{1-\alpha }\right)\mathrm{d}\mu ,\hfill & \alpha \notin \{0,1\}\hfill \\ D_{\mathrm{KL}}^{*}(\tilde{p}:\tilde{q})=D_{\mathrm{KL}}(\tilde{q}:\tilde{p})\hfill & \alpha =0,\hfill \\ 4D_H^2(\tilde{p},\tilde{q})\hfill & \alpha =\frac{1}{2},\hfill \\ D_{\mathrm{KL}}(\tilde{p}:\tilde{q})\hfill & \alpha =1.\hfill \end{array}\right.$$

We now have $D_{\alpha }^{*}(\tilde{p}:\tilde{q})=D_{\alpha }(\tilde{q}:\tilde{p})=D_{1-\alpha }(\tilde{p}:\tilde{q})$, and the $\alpha$-divergences are homogeneous divergences of degree 1. For all $\lambda >0$, we have $D_{\alpha }(\lambda \tilde{q}:\lambda \tilde{p})=\lambda D_{\alpha }(\tilde{q}:\tilde{p})$. Moreoever, because $\alpha \tilde{p}+(1-\alpha )\tilde{q}-{\tilde{p}}^{\alpha }{\tilde{q}}^{1-\alpha }$ can be expressed as the difference of the weighted arithmetic mean minus the weighted geometric mean $A(\tilde{p},\tilde{q};\alpha ;1-\alpha )-G(\tilde{p},\tilde{q};\alpha ;1-\alpha )$, it follows from the arithmetical–geometrical mean inequality that we have $D_{\alpha }(\tilde{p}:\tilde{q})\ge 0$.

When considering unnormalized densities ${\tilde{p}}_{{\theta }_1}$ and ${\tilde{p}}_{{\theta }_2}$ of $\mathcal{E}$, we obtain

$$D_{\alpha }({\tilde{p}}_{{\theta }_1}:{\tilde{p}}_{{\theta }_2})=\left\{\begin{array}{cc}\frac{1}{\alpha (1-\alpha )}{J}_{Z,\alpha }({\theta }_1:{\theta }_2),\hfill & \alpha \notin \{0,1\}\hfill \\ B_Z({\theta }_1:{\theta }_2)\hfill & \alpha =0,\hfill \\ 4J_Z({\theta }_1,{\theta }_2)\hfill & \alpha =\frac{1}{2},\hfill \\ B_Z^{*}({\theta }_1:{\theta }_2)=B_Z({\theta }_2:{\theta }_1)\hfill & \alpha =1\hfill \end{array}.\right.$$

Proposition 5.

The α-divergences between the unnormalized densities of an exponential family amount to scaled α-Jensen divergences between their natural parameters for the partition function

$$D_{\alpha }({\tilde{p}}_{{\theta }_1}:{\tilde{p}}_{{\theta }_2})=J_{Z,\alpha }^s({\theta }_1:{\theta }_2).$$

When $\alpha \in \{0,1\}$, the oriented Kullback–Leibler divergences between unnormalized exponential family densities amount to reverse Bregman divergences on their corresponding natural parameters for the partition function

$$D_{\mathrm{KL}}(\tilde{p}:\tilde{q})=B_Z({\theta }_2:{\theta }_1).$$

Proof. 

For $\alpha \notin \{0,1\}$, consider

$$D_{\alpha }({\tilde{p}}_{{\theta }_1}:{\tilde{p}}_{{\theta }_2})=\frac{1}{\alpha (1-\alpha )}\int \left(\alpha {\tilde{p}}_{{\theta }_1}+(1-\alpha ){\tilde{p}}_{{\theta }_2}-{\tilde{p}}_{{\theta }_1}^{\alpha }{\tilde{p}}_{{\theta }_2}^{1-\alpha }\right)\mathrm{d}\mu .$$

Here, we have $\int \alpha {\tilde{p}}_{{\theta }_1}\mathrm{d}\mu =\alpha Z\left({\theta }_1\right)$, $\int (1-\alpha ){\tilde{p}}_{{\theta }_2}\mathrm{d}\mu =(1-\alpha )Z\left({\theta }_2\right)$ and $\int {\tilde{p}}_{{\theta }_1}^{\alpha }{\tilde{p}}_{{\theta }_2}^{1-\alpha }\mathrm{d}\mu =\int {\tilde{p}}_{\alpha {\theta }_1+(1-\alpha ){\theta }_2}\mathrm{d}\mu =Z(\alpha {\theta }_1+(1-\alpha ){\theta }_2)$. It follows that

$$D_{\alpha }({\tilde{p}}_{{\theta }_1}:{\tilde{p}}_{{\theta }_2})=\frac{1}{\alpha (1-\alpha )}{J}_{Z,\alpha }({\theta }_1:{\theta }_2)=J_{Z,\alpha }^s({\theta }_1:{\theta }_2).$$

□

Notice that the KLD extended to unnormalized densities can be written as a generalized relative entropy, i.e., it can be obtained as the difference of the extended cross-entropy minus the extended entropy (self cross-entropy):

$$\begin{array}{ccc}\hfill D_{\mathrm{KL}}(\tilde{p}:\tilde{q})& =& H^{\times }(\tilde{p}:\tilde{q})-H\left(\tilde{p}\right),\hfill \\ & =& \int \left(\tilde{p}log\frac{\tilde{p}}{\tilde{q}}+\tilde{q}-\tilde{p}\right)\mathrm{d}\mu \hfill \end{array}$$

with

$$H^{\times }(\tilde{p}:\tilde{q})=\int \left(\tilde{p}\left(x\right)log\frac{1}{\tilde{q}\left(x\right)}+\tilde{q}\left(x\right)\right)\mathrm{d}\mu \left(x\right)-1$$

and

$$H\left(\tilde{p}\right)=H^{\times }(\tilde{p}:\tilde{p})=\int \left(\tilde{p}\left(x\right)log\frac{1}{\tilde{p}\left(x\right)}+\tilde{p}\left(x\right)\right)\mathrm{d}\mu \left(x\right)-1.$$

Remark 4.

In general, we can consider two unnormalized positive densities $\tilde{p}\left(x\right)$ and $\tilde{q}\left(x\right)$. Let $p\left(x\right)=\frac{\tilde{p}\left(x\right)}{Z_p}$ and $q\left(x\right)=\frac{\tilde{q}\left(x\right)}{Z_q}$ denote their corresponding normalized densities (with normalizing factors $Z_p=\int \tilde{p}\mathrm{d}\mu$ and $Z_q=\int \tilde{q}\mathrm{d}\mu$); then, the KLD between $\tilde{p}$ and $\tilde{q}$ can be expressed using the KLD between their normalized densities and normalizing factors, as follows:

$$D_{\mathrm{KL}}(\tilde{p}:\tilde{q})=Z_p\left(D_{\mathrm{KL}}(p:q)+log\frac{Z_p}{Z_q}\right)+Z_q-Z_p.$$

(17)

Similarly, we have

$$\begin{array}{ccc}\hfill H^{\times }(\tilde{p}:\tilde{q})& =& Z_p{H}^{\times }(p:q)-Z_{p}log{Z}_q+Z_q-1,\hfill \end{array}$$

(18)

$$\begin{array}{ccc}\hfill H\left(\tilde{p}\right)& =& Z_{p}H\left(p\right)-Z_{p}log{Z}_p+Z_p-1,\hfill \end{array}$$

(19)

and $D_{\mathrm{KL}}(\tilde{p}:\tilde{q})=H^{\times }(\tilde{p}:\tilde{q})-H\left(\tilde{p}\right)$.

Notice that Equation (17) allows us to derive the following identity between $B_Z$ and $B_F$:

$$\begin{array}{ccc}\hfill B_Z({\theta }_2:{\theta }_1)& =& Z\left({\theta }_1\right)B_F({\theta }_2:{\theta }_1)+Z\left({\theta }_1\right)log\frac{Z\left({\theta }_1\right)}{Z\left({\theta }_2\right)}+Z\left({\theta }_2\right)-Z\left({\theta }_1\right),\hfill \end{array}$$

(20)

$$\begin{array}{ccc}& =& exp\left(F\left({\theta }_1\right)\right)B_F({\theta }_2:{\theta }_1)+(expF\left({\theta }_1\right))(F\left({\theta }_1\right)-F\left({\theta }_2\right))+exp\left(F\left({\theta }_2\right)\right)-exp\left(F\left({\theta }_1\right)\right).\hfill \end{array}$$

(21)

Let $D_{\mathrm{skl}}(a:b)=alog\frac{a}{b}+b-a$ be the scalar KLD for $a>0$ and $b>0$. Then, we can rewrite Equation (17) as

$$D_{\mathrm{KL}}(\tilde{p}:\tilde{q})=Z_p{D}_{\mathrm{KL}}(p:q)+D_{\mathrm{skl}}(Z_p:Z_q),$$

and we have

$$B_Z({\theta }_2:{\theta }_1)=Z\left({\theta }_1\right)B_F({\theta }_2:{\theta }_1)+D_{\mathrm{skl}}(Z\left({\theta }_1\right):Z\left({\theta }_2\right)).$$

In addition, the KLD between the unnormalized densities $\tilde{p}$ and $\tilde{q}$ with support $\mathcal{X}$ can be written as a definite integral of a scalar Bregman divergence:

$$D_{\mathrm{KL}}(\tilde{p}:\tilde{q})={\int }_{\mathcal{X}}{D}_{\mathrm{skl}}(\tilde{p}\left(x\right):\tilde{q}\left(x\right))\mathrm{d}\mu \left(x\right)={\int }_{\mathcal{X}}{B}_{f_{\mathrm{skl}}}(\tilde{p}\left(x\right):\tilde{q}\left(x\right))\mathrm{d}\mu \left(x\right),$$

where $f_{\mathrm{skl}}\left(x\right)=xlogx-x$. Because $B_{f_{\mathrm{skl}}}(a:b)\ge 0\forall a>0,b>0$, we can deduce that $D_{\mathrm{KL}}(\tilde{p}:\tilde{q})\ge 0$ with equality iff $\tilde{p}\left(x\right)=\tilde{q}\left(x\right)$$\mu$ almost everywhere.

Notice that $B_Z({\theta }_2:{\theta }_1)=Z\left({\theta }_1\right)B_F({\theta }_2:{\theta }_1)+D_{\mathrm{skl}}(Z\left({\theta }_1\right):Z\left({\theta }_2\right))$ can be interpreted as the sum of two divergences, that is, a conformal Bregman divergence with a scalar Bregman divergence.

Remark 5.

Consider the KLD between the normalized $p_{{\theta }_1}$ and unnormalized ${\tilde{p}}_{{\theta }_2}$ densities of the same exponential family. In this case, we have

$$\begin{array}{ccc}\hfill D_{\mathrm{KL}}(p_{{\theta }_1}:{\tilde{p}}_{{\theta }_2})& =& B_F({\theta }_2:{\theta }_1)-logZ\left({\theta }_2\right)+Z\left({\theta }_2\right)-1,\hfill \\ & =& Z\left({\theta }_2\right)-1-F\left({\theta }_1\right)-〈{\theta }_2-{\theta }_1,\nabla F\left({\theta }_2\right)〉,\hfill \end{array}$$

(22)

$$\begin{array}{ccc}& =& B_{Z-1,F}({\theta }_2:{\theta }_1).\hfill \end{array}$$

(23)

The divergence $B_{Z-1,F}$ is a dual Bregman pseudo-divergence [28]:

$$B_{F_1,F_2}({\theta }_1:{\theta }_2)=F_1\left({\theta }_1\right)-F_2\left({\theta }_2\right)-〈{\theta }_1-{\theta }_2,\nabla F_2\left({\theta }_2\right)〉,$$

for $F_1$ and $F_2$ that are two strictly convex and smooth functions such that $F_1\ge F_2$. Indeed, we can check that generators $F_1\left(\theta \right)=Z\left(\theta \right)-1$ and $F_2\left(\theta \right)=F\left(\theta \right)$ are both Bregman generators; then, we have $F_1\left(\theta \right)\ge F_2\left(\theta \right)$, as $e^x\ge x+1$ for all x (with equality when $x=0$), i.e., $Z\left(\theta \right)-1\ge F\left(\theta \right)$.

More generally, the α-divergences between $p_{{\theta }_1}$ and ${\tilde{p}}_{{\theta }_2}$ can be written as

$$D_{\alpha }(p_{{\theta }_1}:{\tilde{p}}_{{\theta }_2})=\frac{1}{\alpha (1-\alpha )}\left(\alpha Z\left({\theta }_1\right)+(1-\alpha )-\frac{Z(\alpha {\theta }_1+(1-\alpha ){\theta }_2)}{Z\left({\theta }_2\right)}\right),$$

(24)

with the (signed) α-skewed Bhattacharyya distances provided by

$$D_{B,\alpha }(p_{{\theta }_1}:{\tilde{p}}_{{\theta }_2})=logZ\left({\theta }_2\right)-logZ(\alpha {\theta }_1+(1-\alpha ){\theta }_2).$$

Let us illustrate Proposition 5 with some examples.

Example 1.

Consider the family of exponential distributions $\mathcal{E}=\{p_{\lambda }\left(x\right)=1_{x\ge 0}\lambda exp(-\lambda x)\}$, where $\mathcal{E}$ is an exponential family with a natural parameter $\theta =\lambda$, parameter space $\mathrm{\Theta }=\mathbb{R}>0$, sufficient statistic $t\left(x\right)=-x$. The partition function is $Z\left(\theta \right)=\frac{1}{\theta }$, with $Z^{\prime }\left(\theta \right)=-\frac{1}{{\theta }^2}$ and $Z^{″}\left(\theta \right)=\frac{2}{{\lambda }^3}>0$, while the cumulant function is $F\left(\theta \right)=logZ\left(\theta \right)=-log\theta$ with moment parameter $\eta =E_{p_{\lambda }}\left[t\left(x\right)\right]=F^{\prime }\left(\theta \right)=-\frac{1}{\theta }$. The α-divergences between two unnormalized exponential distributions are

$$D_{\alpha }({\tilde{p}}_{{\lambda }_1}:{\tilde{p}}_{{\lambda }_2})=\left\{\begin{array}{cc}\frac{1}{\alpha (1-\alpha )}{J}_{Z,\alpha }({\theta }_1:{\theta }_2)=\frac{{({\lambda }_1-{\lambda }_2)}^2)}{\alpha {\lambda }_1^2{\lambda }_2+(1-\alpha ){\lambda }_1{\lambda }_2^2}\hfill & \alpha \notin \{0,1\}\hfill \\ D_{\mathrm{KL}}({\tilde{p}}_{{\lambda }_2}:{\tilde{p}}_{{\lambda }_1})=B_Z({\theta }_1:{\theta }_2)=\frac{{({\lambda }_1-{\lambda }_2)}^2}{{\lambda }_1{\lambda }_2^2}\hfill & \alpha =0,\hfill \\ 4J_Z({\theta }_1,{\theta }_2)=\frac{{({\lambda }_1-{\lambda }_2)}^2}{2({\lambda }_1{\lambda }_2^2+{\lambda }_1^2{\lambda }_2)}\hfill & \alpha =\frac{1}{2},\hfill \\ D_{\mathrm{KL}}({\tilde{p}}_{{\lambda }_1}:{\tilde{p}}_{{\lambda }_2})=B_Z({\theta }_2:{\theta }_1)=\frac{{({\lambda }_1-{\lambda }_2)}^2}{{\lambda }_2{\lambda }_1^2}\hfill & \alpha =1\hfill \end{array}.\right.$$

(25)

Example 2.

Consider the family of univariate centered normal distributions with ${\tilde{p}}_{{\sigma }^2}\left(x\right)\propto exp(-\frac{x^2}{2{\sigma }^2})$ and partition function $Z\left({\sigma }^2\right)=\sqrt{2\pi {\sigma }^2}$ such that $p_{{\sigma }^2}\left(x\right)=\frac{1}{Z\left({\sigma }^2\right)}{\tilde{p}}_{{\sigma }^2}\left(x\right)=\frac{1}{\sqrt{2\pi {\sigma }^2}}exp(-\frac{x^2}{2{\sigma }^2})$. Here, we have a natural parameter $\theta =\frac{1}{{\sigma }^2}\in \mathrm{\Theta }={\mathbb{R}}_{>0}$ and sufficient statistic $t\left(x\right)=-\frac{x^2}{2}$. The partition function expressed with the natural parameter is $Z\left(\theta \right)=\sqrt{\frac{2\pi }{\theta }}$, with $Z^{\prime }\left(\theta \right)=-\sqrt{\frac{\pi }{2}}{\theta }^{-\frac{3}{2}}$ and $Z^{″}\left(\theta \right)=\frac{3\sqrt{\pi }}{2^{\frac{3}{2}}}{\theta }^{-\frac{5}{2}}>0$ (strictly convex on Θ). The unnormalized KLD between ${\tilde{p}}_{{\sigma }_1^2}$ and ${\tilde{p}}_{{\sigma }_2^2}$ is

$$D_{\mathrm{KL}}({\tilde{p}}_{{\sigma }_1^2}:{\tilde{p}}_{{\sigma }_2^2})=B_Z({\theta }_2:{\theta }_1)=\sqrt{\frac{\pi }{2}}\left(2{\sigma }_2-3{\sigma }_1+\frac{{\sigma }_1^3}{{\sigma }_2^2}\right).$$

We can check that we have $D_{\mathrm{KL}}({\tilde{p}}_{{\sigma }^2}:{\tilde{p}}_{{\sigma }^2})=0$.

For the Hellinger divergence, we have

$$D_H^2({\tilde{p}}_{{\sigma }_1^2}:{\tilde{p}}_{{\sigma }_2^2})=J_Z({\theta }_1,{\theta }_2)=\sqrt{\frac{\pi }{2}}({\sigma }_1+{\sigma }_2)-2\sqrt{\pi }\frac{{\sigma }_1{\sigma }_2}{\sqrt{{\sigma }_1^2+{\sigma }_2^2}},$$

and we can check that $D_H({\tilde{p}}_{{\sigma }^2}:{\tilde{p}}_{{\sigma }^2})=0$.

Consider the family of the d-variate case of centered normal distributions with unnormalized density

$${\tilde{p}}_{\Sigma }\left(x\right)\propto exp(-\frac{1}{2}{x}^{\top }{\Sigma }^{-1}x)=exp\left(-\frac{1}{2}\mathrm{tr}\left(x^{\top }{\Sigma }^{-1}x\right)\right)=exp\left(-\frac{1}{2}\mathrm{tr}\left(x{x}^{\top }{\Sigma }^{-1}\right)\right)$$

obtained using the matrix trace cyclic property, where Σ is the covariance matrix. Here, we have $\theta ={\Sigma }^{-1}$ (precision matrix) and $\mathrm{\Theta }={\mathrm{Sym}}_{++}\left(d\right)$ for $t\left(x\right)=-\frac{1}{2}x{x}^{\top }$, with the matrix inner product $〈A,B〉=\mathrm{tr}\left(A^{\top }B\right)$. The partition function $Z(\Sigma )={\left(2\pi \right)}^{\frac{d}{2}}\sqrt{\det (\Sigma )}$ expressed with the natural parameter is $Z\left(\theta \right)={\left(2\pi \right)}^{\frac{d}{2}}\sqrt{\frac{1}{\det \left(\theta \right)}}$. This is a convex function with

$$\nabla Z\left(\theta \right)=-\frac{1}{2}{\left(2\pi \right)}^{\frac{d}{2}}\frac{{\nabla }_{\theta }\det \left(\theta \right)}{\det {\left(\theta \right)}^{\frac{3}{2}}}=-\frac{1}{2}{\left(2\pi \right)}^{\frac{d}{2}}\frac{{\theta }^{-1}}{\det {\left(\theta \right)}^{\frac{1}{2}}},$$

as ${\nabla }_{\theta }\det \left(\theta \right)=\det \left(\theta \right){\theta }^{-\top }$ using matrix calculus.

Now, consider the family of univariate normal distributions

$$\mathcal{E}=\left\{p_{\mu ,{\sigma }^2}\left(x\right)=\frac{1}{\sqrt{2\pi {\sigma }^2}}exp\left(-\frac{1}{2}{\left(\frac{x-\mu }{\sigma }\right)}^2\right)\right\}.$$

Let $\theta =\left({\theta }_1=\frac{1}{{\sigma }_2},{\theta }_2=\frac{\mu }{{\sigma }^2}\right)$ and

$$Z({\theta }_1,{\theta }_2)=\sqrt{\frac{2\pi }{{\theta }_1}}exp\left(\frac{1}{2}\frac{{\theta }_2^2}{{\theta }_1}\right).$$

The unnormalized densities are ${\tilde{p}}_{\theta }\left(x\right)=exp\left(-\frac{{\theta }_1{x}^2}{2}+x{\theta }_2\right)$, and we have

$$\nabla Z\left(\theta \right)=\left[\begin{array}{c}\sqrt{\frac{\pi }{2}}\frac{({\theta }_1+{\theta }_2^2)exp\left(\frac{{\theta }_2^2}{2{\theta }_1}\right)}{{\theta }_1^{\frac{5}{2}}}\hfill \\ \sqrt{2\pi }\frac{{\theta }_{2}exp\left(\frac{{\theta }_2^2}{2{\theta }_1}\right)}{{\theta }_1^{\frac{3}{2}}}\hfill \end{array}\right].$$

It follows that $D_{\mathrm{KL}}[{\tilde{p}}_{\theta }:{\tilde{p}}_{{\theta }^{\prime }}]=B_Z({\theta }^{\prime }:\theta )$.

5. Deforming Convex Functions and Their Induced Dually Flat Spaces

5.1. Comparative Convexity

The log-convexity can be interpreted as a special case of comparative convexity with respect to a pair $(M,N)$ of comparable weighted means [9], as follows.

A function Z is $(M,N)$-convex if and only if for $\alpha \in [0,1]$ we have

$$Z\left(M\right(x,y;\alpha ,1-\alpha \left)\right)\le N\left(Z\right(x),Z(y);\alpha ,1-\alpha ),$$

(26)

and is strictly $(M,N)$-convex iff we have strict inequality for $\alpha \in (0,1)$ and $x\ne y$. Furthermore, a function Z is (strictly) $(M,N)$-concave if $-Z$ is (strictly) $(M,N)$-convex.

Log-convexity corresponds to $(A,G)$-convexity, i.e., convexity with respect to the weighted arithmetical and geometrical means defined respectively by $A(x,y;\alpha ,1-\alpha )=\alpha x+(1-\alpha )y$ and $G(x,y;\alpha ,1-\alpha )=x^{\alpha }{y}^{1-\alpha }$. Ordinary convexity is $(A,A)$-convexity.

A weighted quasi-arithmetical mean [34] (also called a Kolmogorov–Nagumo mean [35]) is defined for a continuous and strictly increasing function h by

$$M_h(x,y;\alpha ,1-\alpha )=h^{-1}(\alpha h\left(x\right)+(1-\alpha )h\left(x\right)).$$

We let $M_h(x,y)=M_h\left(x,y;\frac{1}{2},\frac{1}{2}\right)$. Quasi-arithmetical means include the arithmetical mean obtained for $h\left(u\right)=\mathrm{id}\left(u\right)=u$ and the geometrical mean for $h\left(u\right)=log\left(u\right)$, and more generally power means

$$M_p(x,y;\alpha ,1-\alpha )={\left(\alpha x^p+(1-\alpha )y^p\right)}^{\frac{1}{p}}=M_{h_p}(x,y;\alpha ,1-\alpha ),p\ne 0,$$

which are quasi-arithmetical means obtained for the family of generators $h_p\left(u\right)=\frac{u^p-1}{p}$ with inverse $h_p^{-1}\left(u\right)={(1+up)}^{\frac{1}{p}}$. In the limit $p\to 0$, we have $M_0(x,y)=G(x,y)$ for the generator ${lim}_{p\to 0}{h}_p\left(u\right)=h_0\left(u\right)=logu$.

Proposition 6

([36,37]). A function $Z\left(\theta \right)$ is strictly $(M_{\rho },M_{\tau })$-convex with respect to two strictly increasing smooth functions ρ and τ if and only if the function $F=\tau \circ Z\circ {\rho }^{-1}$ is strictly convex.

Notice that the set of strictly increasing smooth functions form a non-Abelian group, with the group operation as the function composition, the neutral element as the identity function, and the inverse element as the functional inverse function.

Because log-convexity is $(A=M_{\mathrm{id}},G=M_{log})$-convexity, a function Z is strictly log-convex iff $log\circ Z\circ {\mathrm{id}}^{-1}=log\circ Z$ is strictly convex. We have

$$Z={\tau }^{-1}\circ F\circ \rho \iff F=\tau \circ Z\circ {\rho }^{-1}.$$

Starting from a given convex function $F\left(\theta \right)$, we can deform the function $F\left(\theta \right)$ to obtain a function $Z\left(\theta \right)$ using two strictly monotone functions $\tau$ and $\rho$: $Z\left(\theta \right)={\tau }^{-1}\left(F\left(\rho \left(\theta \right)\right)\right)$.

For a $(M_{\rho },M_{\tau })$-convex function $Z\left(\theta \right)$ which is also strictly convex, we can define a pair of Bregman divergences $B_Z$ and $B_F$ with $F\left(\theta \right)=\tau \left(Z\left({\rho }^{-1}\left(\theta \right)\right)\right)$ and a corresponding pair of skewed Jensen divergences.

Thus, we have the following generic deformation scheme.

$$\underset{(M_{{\rho }^{-1}},M_{{\tau }^{-1}})\text{-}\mathrm{c}\mathrm{o}\mathrm{n}\mathrm{v}\mathrm{e}\mathrm{x} \mathrm{w}\mathrm{h}\mathrm{e}\mathrm{n} \mathrm{Z} \mathrm{i}\mathrm{s} \mathrm{c}\mathrm{o}\mathrm{n}\mathrm{v}\mathrm{e}\mathrm{x}}{\underbrace{F=\tau \circ Z\circ {\rho }^{-1}}} \underset{({\rho }^{-1},{\tau }^{-1})\text{-deformation}}{\underset{\rightleftharpoons }{(\rho ,\tau )\text{-deformation}}} \underset{(M_{\rho },M_{\tau })\text{-}\mathrm{c}\mathrm{o}\mathrm{n}\mathrm{v}\mathrm{e}\mathrm{x} \mathrm{w}\mathrm{h}\mathrm{e}\mathrm{n} F \mathrm{i}\mathrm{s} \mathrm{c}\mathrm{o}\mathrm{n}\mathrm{v}\mathrm{e}\mathrm{x}}{\underbrace{Z={\tau }^{-1}\circ F\circ \rho }}$$

In particular, when the function Z is deformed by strictly increasing the power functions $h_{p_1}$ and $h_{p_2}$ for $p_1$ and $p_2$ in $\mathbb{R}$ as

$$Z_{p_1,p_2}=h_{p_2}\circ Z\circ h_{p_1}^{-1},$$

then $Z_{p_1,p_2}$ is strictly convex when it is strictly $(M_{p_1},M_{p_2})$-convex, and as such induces corresponding Bregman and Jensen divergences.

Example 3.

Consider the partition function $Z\left(\theta \right)=\frac{1}{\theta }$ of the exponential distribution family ($\theta >0$ with $\mathrm{\Theta }={\mathbb{R}}_{>0}$). Let $Z_p\left(\theta \right)=(h_p\circ Z)\left(\theta \right)=\frac{{\theta }^{-p}-1}{p}$; then, we have $Z_p^{″}\left(\theta \right)=(1+p)\frac{1}{{\theta }^{2+p}}>0$ when $p>-1$. Thus, we can deform Z smoothly by $Z_p$ while preserving the convexity by ranging p from $-1$ to $+\infty$. In this way, we obtain a corresponding family of Bregman and Jensen divergences.

The proposed convex deformation using quasi-arithmetical mean generators differs from the interpolation of convex functions using the technique of proximal averaging [38].

Note that in [37] the comparative convexity with respect to a pair of quasi-arithmetical means $(M_{\rho },M_{\tau })$ is used to define a $(M_{\rho },M_{\tau })$-Bregman divergence, which turns out to be equivalent to a conformal Bregman divergence on the $\rho$-embedding of the parameters.

5.2. Dually Flat Spaces

We start with a refinement of the class of convex functions used to generate dually flat spaces.

Definition 2

(Legendre type function [39]). $(\mathrm{\Theta },F)$ is of Legendre type if the function $F:\mathrm{\Theta }\to \mathbb{R}$ is strictly convex and differentiable with $\mathrm{\Theta }\ne \varnothing$ and

$$\underset{\lambda \to 0}{lim}\frac{d}{\mathrm{d}\lambda }F(\lambda \theta +(1-\lambda )\overline{\theta })=-\infty ,\forall \theta \in \mathrm{\Theta },\forall \overline{\theta }\in \partial \mathrm{\Theta }.$$

(27)

Legendre-type functions $F\left(\mathrm{\Theta }\right)$ admit a convex conjugate $F^{*}\left(\eta \right)$ via the Legendre transform $F^{*}\left(\eta \right)={sup}_{\theta \in \mathrm{\Theta }}〈\theta ,\eta 〉-F\left(\theta \right)$:

$$F^{*}\left(\eta \right)=〈\nabla F^{-1}\left(\eta \right),\eta 〉-F(\nabla F^{-1}\left(\eta \right)).$$

A smooth and strictly convex function $(\mathrm{\Theta },F(\theta \left)\right)$ of Legendre type induces a dually flat space [1] $\mathcal{M}$, i.e., a smooth Hessian manifold [40] with a single global chart $(\mathrm{\Theta },\theta (·\left)\right)$ [1]. A canonical divergence $D(p:q)$ between two points p and q of $\mathcal{M}$ is viewed as a single-parameter contrast function [41] $\mathcal{D}\left(r_{pq}\right)$ on the product manifold $\mathcal{M}\times \mathcal{M}$. The canonical divergence and its dual canonical divergence ${\mathcal{D}}^{*}\left(r_{qp}\right)=\mathcal{D}\left(r_{pq}\right)$ can be expressed equivalently as either dual Bregman divergences or dual Fenchel–Young divergences (Figure 2):

$$\begin{array}{ccc}\hfill \mathcal{D}\left(r_{pq}\right)& =& B_F(\theta \left(p\right):\theta \left(q\right))=Y_{F,F^{*}}(\theta \left(p\right):\eta \left(q\right)),\hfill \\ & =& {\mathcal{D}}^{*}\left(r_{qp}\right)=B_{F^{*}}(\eta \left(q\right):\eta \left(p\right))=Y_{F^{*},F}(\eta \left(q\right):\theta \left(p\right)),\hfill \end{array}$$

where $Y_{F,F^{*}}$ is the Fenchel–Young divergence:

$$Y_{F,F^{*}}(\theta \left(p\right):\eta \left(q\right))=F\left(\theta \left(p\right)\right)+F^{*}\left(\eta \left(q\right)\right)-〈\theta \left(p\right),\eta \left(q\right)〉.$$

We have the dual global coordinate system $\eta =\nabla F\left(\theta \right)$ and the domain $H=\{\nabla F(\theta ):\theta \in \mathrm{\Theta }\}$ which defines the dual Legendre-type potential function $(H,F^{*}\left(\eta \right))$. The Legendre-type function ensures that ${F^{*}}^{*}=F$ (a sufficient condition is to have F be convex and lower semi-continuous [42]).

A manifold $\mathcal{M}$ is called dually flat, as the torsion-free affine connections ∇ and ${\nabla }^{*}$ induced by the potential functions $F\left(\theta \right)$ and $F^{*}\left(\eta \right)$ linked with the Legendre–Fenchel transformation are flat [1], that is, their Christoffel symbols vanishes in the dual coordinate system: $\mathrm{\Gamma }\left(\theta \right)=0$ and ${\mathrm{\Gamma }}^{*}\left(\eta \right)=0$.

The Legendre-type function $(\mathrm{\Theta },F(\theta \left)\right)$ is not defined uniquely; the function $\overline{F}\left(\overline{\theta }\right)=F(A\theta +b)+C\theta +d$ with $\overline{\theta }=A\theta +b$ for A and C invertible matrices and b and d vectors defines the same dually flat space with the same canonical divergence $D(p,q)$:

$$D(p:q)=B_F(\theta \left(p\right):\theta \left(q\right))=B_{\overline{F}}(\overline{\theta }\left(p\right):\overline{\theta }\left(q\right)).$$

Thus, a log-convex Legendre-type function $Z\left(\theta \right)$ induces two dually flat spaces by considering the DFSs induced by $Z\left(\theta \right)$ and $F\left(\theta \right)=logZ\left(\theta \right)$. Let the gradient maps be $\eta =\nabla Z\left(\theta \right)$ and $\tilde{\eta }=\nabla F\left(\theta \right)=\frac{\eta }{Z\left(\theta \right)}$.

When $F\left(\theta \right)$ is chosen as the cumulant function of an exponential family, the Bregman divergence $B_F({\theta }_1:{\theta }_2)$ can be interpreted as a statistical divergence between corresponding probability densities, meaning that the Bregman divergence amounts to the reverse Kullback–Leibler divergence: $B_F({\theta }_1:{\theta }_2)=D_{\mathrm{KL}}^{*}(p_{{\theta }_1}:p_{{\theta }_2})$, where $D_{\mathrm{KL}}^{*}$ is the reverse KLD.

Notice that deforming a convex function $F\left(\theta \right)$ into $F\left(\rho \right(\theta \left)\right)$ such that $F\circ \rho$ remains strictly convex has been considered by Yoshizawa and Tanabe [43] to build a two-parameter deformation ${\rho }_{\alpha ,\beta }$ of the dually flat space induced by the cumulant function $F\left(\theta \right)$ of the multivariate normal family. Additionally, see the method of Hougaard [44] for obtaining other exponential families from a given exponential family.

Thus, in general, there are many more dually flat spaces with corresponding divergences and statistical divergences than the usually considered exponential family manifold [5] induced by the cumulant function. It is interesting to consider their use in information sciences.

6. Conclusions and Discussion

For machine learning practioners, it is well known that the Kullback–Leibler divergence (KLD) between two probability densities $p_{{\theta }_1}$ and $p_{{\theta }_2}$ of an exponential family with cumulant function F (free energy in thermodynamics) amounts to a reverse Bregman divergence [26] induced by F, or equivalently to a reverse Fenchel–Young divergence [27]

$$D_{\mathrm{KL}}(p_{{\theta }_1}:p_{{\theta }_2})=B_F({\theta }_2:{\theta }_1)=Y_{F,F^{*}}({\theta }_2:{\eta }_1),$$

where $\eta =\nabla F\left(\theta \right)$ is the dual moment or expectation parameter.

In this paper, we have shown that the KLD as extended to positive unnormalized densities ${\tilde{p}}_{{\theta }_1}$ and ${\tilde{p}}_{{\theta }_2}$ of an exponential family with a convex partition function $Z\left(\theta \right)$ (Laplace transform) amounts to a reverse Bregman divergence induced by Z, or equivalently to a reverse Fenchel–Young divergence

$$D_{\mathrm{KL}}({\tilde{p}}_{{\theta }_1}:{\tilde{p}}_{{\theta }_2})=B_Z({\theta }_2:{\theta }_1)=Y_{Z,Z^{*}}({\theta }_2:{\tilde{\eta }}_1),$$

where $\tilde{\eta }=\nabla Z\left(\theta \right)$.

More generally, we have shown that the scaled $\alpha$-skewed Jensen divergences induced by the cumulant and partition functions between natural parameters coincide with the scaled $\alpha$-skewed Bhattacharyya distances between probability densities and the $\alpha$-divergences between unnormalized densities, respectively:

$$\begin{array}{ccc}\hfill D_{B,\alpha }^s(p_{{\theta }_1}:p_{{\theta }_2})& =& J_{F,\alpha }^s({\theta }_1:{\theta }_2),\hfill \\ \hfill D_{\alpha }({\tilde{p}}_{{\theta }_1}:{\tilde{p}}_{{\theta }_2})& =& J_{Z,\alpha }^s({\theta }_1:{\theta }_2).\hfill \end{array}$$

We have noted that the partition functions Z of exponential families are both convex and log-convex, and that the corresponding cumulant functions are both convex and exponentially convex.

Figure 3 summarizes the relationships between statistical divergences and between the normalized and unnormalized densities of an exponential family, as well as the corresponding divergences between their natural parameters. Notice that Brekelmans and Nielsen [45] considered deformed uni-order likelihood ratio exponential families (LREFs) for annealing paths and obtained an identity for the $\alpha$-divergences between unnormalized densities and Bregman divergences induced by multiplicatively scaled partition functions.

Because the log-convex partition function is also convex, we have generalized the principle of building pairs of convex generators using the comparative convexity with respect to a pair of quasi-arithmetical means, and have further discussed the induced dually flat spaces and divergences. In particular, by considering the convexity-preserving deformations obtained by power mean generators, we have shown how to obtain a family of convex generators and dually flat spaces. Notice that some parametric families of Bregman divergences, such as the $\alpha$-divergences [46], $\beta$-divergences [47], and V-geometry [48] of symmetric positive-definite matrices, yield families of dually flat spaces.

Banerjee et al. [49] proved a duality between regular exponential families and a subclass of Bregman divergences, which they accordingly termed regular Bregman divergences. In particular, this duality allows the Maximum Likelihood Estimator (MLE) of an exponential family with a cumulant function F to be viewed as a right-sided Bregman centroid with respect to the Legendre–Fenchel dual $F^{*}$. In [50], the scope of this duality was further extended for arbitrary Bregman divergences by introducing a class of generalized exponential families.

Concave deformations have been recently studied in [51], where the authors introduced the ${log}_{\varphi }$-concavity induced by a positive continuous function $\varphi$ generating a deformed logarithm ${log}_{\varphi }$ as the $(A,{log}_{\varphi })$-comparative concavity (Definition 1.2 in [51]), as well as the weaker notion of F-concavity which corresponds to the $(A,F)$-concavity (Definition 2.1 in [51], requiring strictly increasing functions F). Our deformation framework $Z={\tau }^{-1}\circ F\circ \rho$ is more general, as it is double-sided. We jointly deform the function F by $F_{\tau }={\tau }^{-1}\circ F$ and its argument $\theta$ by ${\theta }_{\rho }=\rho \left(\theta \right)$.

Exponentially concave functions have been considered as generators of L-divergences in [24]; $\alpha$-exponentially concave functions G such that $exp\left(\alpha G\right)$ are concave for $\alpha >0$ generalize the L-divergences to $L_{\alpha }$-divergences, which can be expressed equivalently using a generalization of the Fenchel–Young divergence based on the c-transforms [24]. When $\alpha <0$, exponentially convex functions are considered instead of exponentially concave functions. The information geometry induced by $L_{\alpha }$-divergences are dually projectively flat with constant curvature, and reciprocally possess a dually projectively flat structure with constant curvature, inducing (locally) a canonical $L_{-\alpha }$-divergence. Wong and Zhang [52] investigated a one-parameter deformation of convex duality, called $\lambda$-duality, by considering functions f such that $\frac{1}{\lambda }(e^{\lambda f}-1)$ are convex for $\lambda \ne 0$. They defined the $\lambda$-conjugate transform as a particular case of the c-transform [24] and studie the information geometry of the induced $\lambda$-logarithmic divergences. The $\lambda$-duality yields a generalization of exponential and mixture families to $\lambda$-exponential and $\lambda$-mixture families related to the Rényi divergence.

Finally, certain statistical divergences, called projective divergences, are invariant under rescaling, and as such can define dissimilarities between non-normalized densities. For example, the $\gamma$-divergences [32] $D_{\gamma }$ are such that $D_{\gamma }(p:q)=D_{\gamma }(\tilde{p}:\tilde{q})$ (with $\gamma$-divergences tending to the KLD when $\gamma \to 0$) or the Cauchy–Schwarz divergence [53].