Statistical Divergences between Densities of Truncated Exponential Families with Nested Supports: Duo Bregman and Duo Jensen Divergences

Abstract

By calculating the Kullback–Leibler divergence between two probability measures belonging to different exponential families dominated by the same measure, we obtain a formula that generalizes the ordinary Fenchel–Young divergence. Inspired by this formula, we define the duo Fenchel–Young divergence and report a majorization condition on its pair of strictly convex generators, which guarantees that this divergence is always non-negative. The duo Fenchel–Young divergence is also equivalent to a duo Bregman divergence. We show how to use these duo divergences by calculating the Kullback–Leibler divergence between densities of truncated exponential families with nested supports, and report a formula for the Kullback–Leibler divergence between truncated normal distributions. Finally, we prove that the skewed Bhattacharyya distances between truncated exponential families amount to equivalent skewed duo Jensen divergences.

1. Introduction

1.1. Exponential Families

Let $(\mathcal{X},\Sigma )$ be a measurable space, and consider a regular minimal exponential family [1] $\mathcal{E}$ of probability measures $P_{\theta }$ all dominated by a base measure $\mu$ ($P_{\theta }\ll \mu$):

$$\mathcal{E}=\{P_{\theta }:\theta \in \Theta \}.$$

(1)

The Radon–Nikodym derivatives or densities of the probability measures $P_{\theta }$ with respect to $\mu$ can be written canonically as

$$p_{\theta }\left(x\right)=\frac{\mathrm{d}{P}_{\theta }}{\mathrm{d}\mu }\left(x\right)=exp\left({\theta }^{\top }t\left(x\right)-F\left(\theta \right)+k\left(x\right)\right),$$

(2)

where $\theta$ denotes the natural parameter, $t\left(x\right)$ the sufficient statistic [1,2,3,4], and $F\left(\theta \right)$ the log-normalizer [1] (or cumulant function). The optional auxiliary term $k\left(x\right)$ allows us to change the base measure $\mu$ into the measure $\nu$ such that $\frac{\mathrm{d}\nu }{\mathrm{d}\mu }\left(x\right)=e^{k\left(x\right)}$. The order D of the family is the dimension of the natural parameter space $\Theta$:

$$\Theta =\left\{\theta \in {\mathbb{R}}^D:{\int }_{\mathcal{X}}exp\left({\theta }^{\top }t\left(x\right)+k\left(x\right)\right)\mathrm{d}\mu \left(x\right)<\infty \right\},$$

(3)

where $\mathbb{R}$ denotes the set of reals. The sufficient statistic $t\left(x\right)=(t_1\left(x\right),\dots ,t_D\left(x\right))$ is a vector of D functions. The sufficient statistic $t\left(x\right)$ is said to be minimal when the $D+1$ functions 1, $t_1\left(x\right)$, …, $t_D\left(x\right)$ are linearly independent [1]. The sufficient statistics $t\left(x\right)$ are such that the probability $\mathrm{Pr}\left[X\right|\theta ]=\mathrm{Pr}[X\left|t\right(X\left)\right]$. That is, all information necessary for the statistical inference of parameter $\theta$ is contained in $t\left(X\right)$. Exponential families are characterized as families of parametric distributions with finite-dimensional sufficient statistics [1]. Exponential families $\left\{p_{\lambda }\right\}$ include among others the exponential, normal, gamma/beta, inverse gamma, inverse Gaussian, and Wishart distributions once a reparameterization $\theta =\theta \left(\lambda \right)$ of the parametric distributions $\left\{p_{\lambda }\right\}$ is performed to reveal their natural parameters [1].

When the sufficient statistic $t\left(x\right)$ is x, these exponential families [1] are called natural exponential families or tilted exponential families [5] in the literature. Indeed, the distributions $P_{\theta }$ of the exponential family $\mathcal{E}$ can be interpreted as distributions obtained by tilting the base measure $\mu$ [6]. In this paper, we consider either discrete exponential families like the family of Poisson distributions (univariate distributions of order $D=1$ with respect to the counting measure) or continuous exponential families like the family of normal distributions (univariate distributions of order $D=2$ with respect to the Lebesgue measure). The Radon–Nikodym derivative of a discrete exponential family is a probability mass function (pmf), and the Radon–Nikodym derivative of a continuous exponential family is a probability density function (pdf). The support of a pmf $p\left(x\right)$ is $\mathrm{supp}\left(p\right)=\{x\in \mathbb{Z}:p(x)>0\}$ (where $\mathbb{Z}$ denotes the set of integers) and the support of a d-variate pdf $p\left(x\right)$ is $\mathrm{supp}\left(p\right)=\{x\in {\mathbb{R}}^d:p\left(x\right)>0\}$. The Poisson distributions have support $\mathbb{N}\cup \left\{0\right\}$ where $\mathbb{N}$ denotes the set of natural numbers $\{1,2,\dots ,\}$. Densities of an exponential family all have coinciding support [1].

1.2. Truncated Exponential Families with Nested Supports

In this paper, we shall consider truncated exponential families [7] with nested supports. A truncated exponential family is a set of parametric probability distributions obtained by truncation of the support of an exponential family. Truncated exponential families are exponential families but their statistical inference is more subtle [8,9]. Let ${\mathcal{E}}_{\mathrm{Trunc}}=\left\{q_{\theta }\right\}$ be a truncated exponential family of $\mathcal{E}=\left\{p_{\theta }\right\}$ with nested supports $\mathrm{supp}\left(q_{\theta }\right)\subset \mathrm{supp}\left(p_{\theta }\right)$. The canonical decompositions of densities $p_{\theta }$ and $q_{\theta }$ have the following expressions:

$$\begin{array}{ccc}\hfill p_{\theta }\left(x\right)& =& exp\left({\theta }^{\top }t\left(x\right)+k\left(x\right)-F\left(\theta \right)\right),\hfill \end{array}$$

(4)

$$\begin{array}{ccc}\hfill q_{\theta }\left(x\right)& =& \frac{p_{\theta }\left(x\right)}{Z^{{\mathcal{X}}_{\mathrm{Trunc}}}\left(\theta \right)}=exp\left({\theta }^{\top }t\left(x\right)+k\left(x\right)-F_{\mathrm{Trunc}}\left(\theta \right)\right),\hfill \end{array}$$

(5)

where the log-normalizer of the truncated exponential family is:

$$F_{\mathrm{Trunc}}\left(\theta \right)=F\left(\theta \right)+log{Z}^{{\mathcal{X}}_{\mathrm{Trunc}}}\left(\theta \right),$$

(6)

where $Z^{{\mathcal{X}}_{\mathrm{Trunc}}}\left(\theta \right)$ is a normalizing term that takes into account the truncated support ${\mathcal{X}}_{\mathrm{Trunc}}$. These equations show that densities of truncated exponential families only differ by their log-normalizer functions. Let ${\mathcal{X}}_{\mathrm{Trunc}}$ denote the support of the distributions of ${\mathcal{E}}_{\mathrm{Trunc}}=\mathrm{supp}\left(q_{\theta }\right)$ and $\mathcal{X}=\mathrm{supp}\left(p_{\theta }\right)$ the support of $\mathcal{E}$. Family ${\mathcal{E}}_{\mathrm{Trunc}}$ is a truncated exponential family of $\mathcal{E}$ that can be notationally written as ${\mathcal{E}}_{{\mathcal{X}}_{\mathrm{Trunc}}}$. Family $\mathcal{E}$ can also be interpreted as the (un)truncated exponential family ${\mathcal{E}}_{\mathcal{X}}$ with densities $p_{\theta }^{\mathcal{X}}=p_{\theta }$. A truncated exponential family ${\mathcal{E}}_{{\mathcal{X}}_{\mathrm{Trunc}}}$ of $\mathcal{E}$ is said to have nested support when ${\mathcal{X}}_{\mathrm{Trunc}}\subset \mathcal{X}$. For example, the family of half-normal distributions defined on the support ${\mathcal{X}}_{\mathrm{Trunc}}=[0,\infty )$ is a nested truncated exponential family of the family of normal distributions defined on the support $\mathcal{X}=(-\infty ,\infty )$.

1.3. Kullback–Leibler Divergence between Exponential Family Distributions

For two $\sigma$-finite probability measures P and Q on $(\mathcal{X},\Sigma )$ such that P is dominated by Q ($P\ll Q$), the Kullback–Leibler divergence between P and Q is defined by

$$D_{\mathrm{KL}}[P:Q]={\int }_{\mathcal{X}}log\frac{\mathrm{d}P}{\mathrm{d}Q}\mathrm{d}P=E_P\left[log\frac{\mathrm{d}P}{\mathrm{d}Q}\right],$$

(7)

where $E_P\left[X\right]$ denotes the expectation of a random variable $X\sim P$ [10]. When $P\cancel{\ll }Q$, we set $D_{\mathrm{KL}}[P:Q]=+\infty$. Gibbs’ inequality [11] $D_{\mathrm{KL}}[P:Q]\ge 0$ shows that the Kullback–Leibler divergence (KLD for short) is always non-negative. The proof of Gibbs’ inequality relies on Jensen’s inequality and holds for the wide class of f-divergences [12] induced by convex generators $f\left(u\right)$:

$$I_f[P:Q]={\int }_{\mathcal{X}}f\left(\frac{\mathrm{d}Q}{\mathrm{d}P}\right)\mathrm{d}P\ge f\left({\int }_{\mathcal{X}}\frac{\mathrm{d}Q}{\mathrm{d}P}\mathrm{d}P\right)\ge f\left(1\right).$$

(8)

The KLD is an f-divergence obtained for the convex generator $f\left(u\right)=-logu$.

1.4. Kullback–Leibler Divergence between Exponential Family Densities

It is well-known that the KLD between two distributions $P_{{\theta }_1}$ and $P_{{\theta }_2}$ of $\mathcal{E}$ amounts to computing an equivalent Fenchel–Young divergence [13]:

$$D_{\mathrm{KL}}[P_{{\theta }_1}:P_{{\theta }_2}]={\int }_{\mathcal{X}}{p}_{{\theta }_1}\left(x\right)log\frac{p_{{\theta }_1}\left(x\right)}{p_{{\theta }_2}\left(x\right)}\mathrm{d}\mu \left(x\right)=Y_{F,F^{*}}({\theta }_2,{\eta }_1),$$

(9)

where $\eta =\nabla F\left(\theta \right)=E_{P_{\theta }}\left[t\left(x\right)\right]$ is the moment parameter [1] and

$$\nabla F\left(\theta \right)={\left[\frac{\partial }{\partial {\theta }_1}F\left(\theta \right),\dots ,\frac{\partial }{\partial {\theta }_D}F\left(\theta \right)\right]}^{\top },$$

(10)

is the gradient of F with respect to $\theta ={[{\theta }_1,\dots ,{\theta }_D]}^{\top }$. The Fenchel–Young divergence is defined for a pair of strictly convex conjugate functions [14] $F\left(\theta \right)$ and $F^{*}\left(\eta \right)$ related by the Legendre–Fenchel transform by

$$Y_{F,F^{*}}({\theta }_1,{\eta }_2):=F\left({\theta }_1\right)+F^{*}\left({\eta }_2\right)-{\theta }_1^{\top }{\eta }_2.$$

(11)

Amari (1985) first introduced this formula as the canonical divergence of dually flat spaces in information geometry [15] (Equation 3.21), and proved that the Fenchel–Young divergence is obtained as the KLD between densities belonging to the same exponential family [15] (Theorem 3.7). Azoury and Warmuth expressed the KLD $D_{\mathrm{KL}}[P_{{\theta }_1}:P_{{\theta }_2}]$ using dual Bregman divergences in [13] (2001):

$$D_{\mathrm{KL}}[P_{{\theta }_1}:P_{{\theta }_2}]=B_F({\theta }_2:{\theta }_1)=B_{F^{*}}({\eta }_1:{\eta }_2),$$

(12)

where a Bregman divergence [16] $B_F({\theta }_1:{\theta }_2)$ is defined for a strictly convex and differentiable generator $F\left(\theta \right)$ by:

$$B_F({\theta }_1:{\theta }_2):=F\left({\theta }_1\right)-F\left({\theta }_2\right)-{({\theta }_1-{\theta }_2)}^{\top }\nabla F\left({\theta }_2\right).$$

(13)

Acharyya termed the divergence $Y_{F,F^{*}}$ the Fenchel–Young divergence in his PhD thesis [17] (2013), and Blondel et al. called such divergences Fenchel–Young losses (2020) in the context of machine learning [18] (Equation (9) in Definition 2). This term was also used by the author the Legendre–Fenchel divergence in [19]. The Fenchel–Young divergence stems from the Fenchel–Young inequality [14,20]:

$$F\left({\theta }_1\right)+F^{*}\left({\eta }_2\right)\ge {\theta }_1^{\top }{\eta }_2,$$

(14)

with equality if and only if ${\eta }_2=\nabla F\left({\theta }_1\right)$.

Figure 1 visualizes the 1D Fenchel–Young divergence and gives a geometric proof that $Y_{F,F^{*}}({\theta }_1,{\eta }_2)\ge 0$ with equality if and only if ${\eta }_2=F^{\prime }\left({\theta }_1\right)$. Indeed, by considering the behavior of the Legendre–Fenchel transformation under translations:

• if $F_t\left(\theta \right)=F(\theta +t)$ then $F_t^{*}\left(\eta \right)=F^{*}\left(\eta \right)-{\eta }^{\top }t$ for all $t\in \mathbb{R}$, and
• if $F_{\lambda }\left(\theta \right)=F\left(\theta \right)+\lambda$ then $F_{\lambda }^{*}\left(\eta \right)=F^{*}\left(\eta \right)-\lambda$ for all $\lambda \in \mathbb{R}$,

we may assume without loss of generality that $F\left(0\right)=0$. The function $F^{\prime }\left(\theta \right)$ is strictly increasing and continuous since $F\left(\theta \right)$ is a strictly convex and differentiable convex function. Thus we have $F\left(\theta \right)={\int }_0^{\theta }{F}^{\prime }\left(\theta \right)\mathrm{d}\theta$ and $F^{*}\left(\eta \right)={\int }_0^{\eta }{F^{*}}^{\prime }\left(\eta \right)\mathrm{d}\eta ={\int }_0^{\eta }{F^{\prime }}^{-1}\left(\eta \right)\mathrm{d}\eta$.

Figure 1. Visualizing the Fenchel–Young divergence.

Figure 1. Visualizing the Fenchel–Young divergence.

The Bregman divergence $B_F({\theta }_1:{\theta }_2)$ amounts to a dual Bregman divergence [13] between the dual parameters with swapped order: $B_F({\theta }_1:{\theta }_2)=B_{F^{*}}({\eta }_2:{\eta }_1)$ where ${\eta }_i=\nabla F\left({\theta }_i\right)$ for $i\in \{1,2\}$. Thus the KLD between two distributions $P_{{\theta }_1}$ and $P_{{\theta }_2}$ of $\mathcal{E}$ can be expressed equivalently as follows:

$$D_{\mathrm{KL}}[P_{{\theta }_1}:P_{{\theta }_2}]=Y_{F,F^{*}}({\theta }_2:{\eta }_1)=B_F({\theta }_2:{\theta }_1)=B_{F^{*}}({\eta }_1:{\eta }_2)=Y_{F^{*},F}({\eta }_1:{\eta }_2).$$

(15)

The symmetrized Kullback–Leibler divergence $D_J[P_{{\theta }_1}:P_{{\theta }_2}]$ between two distributions $P_{{\theta }_1}$ and $P_{{\theta }_2}$ of $\mathcal{E}$ is called Jeffreys’ divergence [21] and amounts to a symmetrized Bregman divergence [22]:

$$\begin{array}{ccc}\hfill D_J[P_{{\theta }_1}:P_{{\theta }_2}]& =& D_{\mathrm{KL}}[P_{{\theta }_1}:P_{{\theta }_2}]+D_{\mathrm{KL}}[P_{{\theta }_2}:P_{{\theta }_1}],\hfill \end{array}$$

(16)

$$\begin{array}{ccc}& =& B_F({\theta }_2:{\theta }_1)+B_F({\theta }_1:{\theta }_2),\hfill \end{array}$$

(17)

$$\begin{array}{ccc}& =& {({\theta }_2-{\theta }_1)}^{\top }({\eta }_2-{\eta }_1):=S_F({\theta }_1,{\theta }_2).\hfill \end{array}$$

(18)

Note that the Bregman divergence $B_F({\theta }_1:{\theta }_2)$ can also be interpreted as a surface area:

$$B_F({\theta }_1:{\theta }_2)={\int }_{{\theta }_2}^{{\theta }_1}(F^{\prime }\left(\theta \right)-F^{\prime }\left({\theta }_2\right))\mathrm{d}\theta .$$

(19)

Figure 2 illustrates the sided and symmetrized Bregman divergences.

Figure 2. Visualizing the sided and symmetrized Bregman divergences.

1.5. Contributions and Paper Outline

We recall in Section 2 the formula obtained for the Kullback–Leibler divergence between two exponential family densities equivalent to each other [23] (Equation (29)). Inspired by this formula, we give a definition of the duo Fenchel–Young divergence induced by a pair of strictly convex functions $F_1$ and $F_2$ (Definition 1) in Section 3, and prove that the divergence is always non-negative provided that $F_1$ upper bounds $F_2$. We then define the duo Bregman divergence (Definition 2) corresponding to the duo Fenchel–Young divergence. In Section 4, we show that the Kullback–Leibler divergence between a truncated density and a density of a same parametric exponential family amounts to a duo Fenchel–Young divergence or equivalently to a duo Bregman divergence on swapped parameters (Theorem 1). That is, we consider a truncated exponential family [7] ${\mathcal{E}}_1$ of an exponential family ${\mathcal{E}}_1$ such that the common support of the distributions of ${\mathcal{E}}_1$ is contained in the common support of the distributions of ${\mathcal{E}}_2$ and both canonical decompositions of the families coincide (see Equation (2)). In particular, when ${\mathcal{E}}_2$ is also a truncated exponential family of $\mathcal{E}$, then we express the KLD between two truncated distributions as a duo Bregman divergence. As examples, we report the formula for the Kullback–Leibler divergence between two densities of truncated exponential families (Corollary 1), and illustrate the formula for the Kullback–Leibler divergence between truncated exponential distributions (Example 6) and for the Kullback–Leibler divergence between truncated normal distributions (Example 7).

In Section 5, we further consider the skewed Bhattacharyya distance between densities of truncated exponential families and prove that it amounts to a duo Jensen divergence (Theorem 2). Finally, we conclude in Section 6.

2. Kullback–Leibler Divergence between Different Exponential Families

Consider now two exponential families [1] $\mathcal{P}$ and $\mathcal{Q}$ defined by their Radon–Nikodym derivatives with respect to two positive measures ${\mu }_{\mathcal{P}}$ and ${\mu }_{\mathcal{Q}}$ on $(\mathcal{X},\Sigma )$:

$$\begin{array}{ccc}\hfill \mathcal{P}& =& \left\{P_{\theta }:\theta \in \Theta \right\},\hfill \end{array}$$

(20)

$$\begin{array}{ccc}\hfill \mathcal{Q}& =& \left\{Q_{{\theta }^{\prime }}:{\theta }^{\prime }\in {\Theta }^{\prime }\right\}.\hfill \end{array}$$

(21)

The corresponding natural parameter spaces are

$$\begin{array}{ccc}\hfill \Theta & =& \left\{\theta \in {\mathbb{R}}^D:{\int }_{\mathcal{X}}exp({\theta }^{\top }{t}_{\mathcal{P}}\left(x\right)+k_{\mathcal{P}}\left(x\right))\mathrm{d}{\mu }_{\mathcal{P}}\left(x\right)<\infty \right\},\hfill \end{array}$$

(22)

$$\begin{array}{ccc}\hfill {\Theta }^{\prime }& =& \left\{{\theta }^{\prime }\in {\mathbb{R}}^{D^{\prime }}:{\int }_{\mathcal{X}}exp({{\theta }^{\prime }}^{\top }{t}_{\mathcal{Q}}\left(x\right)+k_{\mathcal{Q}}\left(x\right))\mathrm{d}{\mu }_{\mathcal{Q}}\left(x\right)<\infty \right\},\hfill \end{array}$$

(23)

The order of $\mathcal{P}$ is D, $t_{\mathcal{P}}\left(x\right)$ denotes the sufficient statistics of $P_{\theta }$, and $k_{\mathcal{P}}\left(x\right)$ is a term to adjust/tilt the base measure ${\mu }_{\mathcal{P}}$. Similarly, the order of $\mathcal{Q}$ is $D^{\prime }$, $t_{\mathcal{Q}}\left(x\right)$ denotes the sufficient statistics of $Q_{{\theta }^{\prime }}$, and $k_{\mathcal{Q}}\left(x\right)$ is an optional term to adjust the base measure ${\mu }_{\mathcal{Q}}$. Let $p_{\theta }$ and $q_{{\theta }^{\prime }}$ denote the Radon–Nikodym derivatives with respect to the measures ${\mu }_{\mathcal{P}}$ and ${\mu }_{\mathcal{Q}}$, respectively:

$$\begin{array}{ccc}\hfill p_{\theta }& =& \frac{\mathrm{d}{P}_{\theta }}{\mathrm{d}{\mu }_{\mathcal{P}}}=exp({\theta }^{\top }{t}_{\mathcal{P}}\left(x\right)-F_{\mathcal{P}}\left(\theta \right)+k_{\mathcal{P}}\left(x\right)),\hfill \end{array}$$

(24)

$$\begin{array}{ccc}\hfill q_{{\theta }^{\prime }}& =& \frac{\mathrm{d}{Q}_{{\theta }^{\prime }}}{\mathrm{d}{\mu }_{\mathcal{Q}}}=exp({{\theta }^{\prime }}^{\top }{t}_{\mathcal{Q}}\left(x\right)-F_{\mathcal{Q}}\left({\theta }^{\prime }\right)+k_{\mathcal{Q}}\left(x\right)),\hfill \end{array}$$

(25)

where $F_{\mathcal{P}}\left(\theta \right)$ and $F_{\mathcal{Q}}\left({\theta }^{\prime }\right)$ denote the corresponding log-normalizers of $\mathcal{P}$ and $\mathcal{Q}$, respectively.

$$\begin{array}{ccc}\hfill F_{\mathcal{P}}\left(\theta \right)& =& log\left(\int exp({\theta }^{\top }{t}_{\mathcal{P}}\left(x\right)+k_{\mathcal{P}}\left(x\right))\mathrm{d}{\mu }_{\mathcal{P}}\left(x\right)\right),\hfill \end{array}$$

(26)

$$\begin{array}{ccc}\hfill F_{\mathcal{Q}}\left(\theta \right)& =& log\left(\int exp({\theta }^{\top }{t}_{\mathcal{Q}}\left(x\right)+k_{\mathcal{Q}}\left(x\right))\mathrm{d}{\mu }_{\mathcal{Q}}\left(x\right)\right).\hfill \end{array}$$

(27)

The functions $F_{\mathcal{P}}$ and $F_{\mathcal{Q}}$ are strictly convex and real analytic [1]. Hence, those functions are infinitely many times differentiable on their open natural parameter spaces.

Consider the KLD between $P_{\theta }\in \mathcal{P}$ and $Q_{{\theta }^{\prime }}\in \mathcal{Q}$ such that ${\mu }_{\mathcal{P}}={\mu }_{\mathcal{Q}}$ (and hence $P_{\theta }\ll Q_{{\theta }^{\prime }}$). Then the KLD between $P_{\theta }$ and $Q_{{\theta }^{\prime }}$ was first considered in [23]:

$$\begin{array}{ccc}\hfill D_{\mathrm{KL}}[P_{\theta }:Q_{{\theta }^{\prime }}]& =& E_P\left[log\left(\frac{\mathrm{d}{P}_{\theta }}{\mathrm{d}{Q}_{{\theta }^{\prime }}}\right)\right],\hfill \\ & =& E_{P_{\theta }}\left[\left({\theta }^{\top }{t}_{\mathcal{P}}\left(x\right)-{{\theta }^{\prime }}^{\top }{t}_{\mathcal{Q}}\left(x\right)-F_{\mathcal{P}}\left(\theta \right)+F_{\mathcal{Q}}\left({\theta }^{\prime }\right)+k_{\mathcal{P}}\left(x\right)-k_{\mathcal{Q}}\left(x\right)\right)\underset{=1}{\underbrace{\frac{\mathrm{d}{\mu }_{\mathcal{P}}}{\mathrm{d}{\mu }_{\mathcal{Q}}}}}\right],\hfill \\ & =& F_{\mathcal{Q}}\left({\theta }^{\prime }\right)-F_{\mathcal{P}}\left(\theta \right)+{\theta }^{\top }{E}_{P_{\theta }}\left[t_{\mathcal{P}}\left(x\right)\right]-{{\theta }^{\prime }}^{\top }{E}_{P_{\theta }}\left[t_{\mathcal{Q}}\left(x\right)\right]+E_{P_{\theta }}\left[k_{\mathcal{P}}\left(x\right)-k_{\mathcal{Q}}\left(x\right)\right].\hfill \end{array}$$

(28)

Recall that the dual parameterization of an exponential family density $P_{\theta }$ is $P^{\eta }$ with $\eta =E_{P_{\theta }}\left[t_{\mathcal{P}}\left(x\right)\right]=\nabla F_{\mathcal{P}}\left(\theta \right)$ [1], and that the Fenchel–Young equality is $F\left(\theta \right)+F^{*}\left(\eta \right)={\theta }^{\top }\eta$ for $\eta =\nabla F\left(\theta \right)$. Thus the KLD between $P_{\theta }$ and $Q_{{\theta }^{\prime }}$ can be rewritten as

$$D_{\mathrm{KL}}[P_{\theta }:Q_{{\theta }^{\prime }}]=F_{\mathcal{Q}}\left({\theta }^{\prime }\right)+F_{\mathcal{P}}^{*}\left(\eta \right)-{{\theta }^{\prime }}^{\top }{E}_{P_{\theta }}\left[t_{\mathcal{Q}}\left(x\right)\right]+E_{P_{\theta }}\left[k_{\mathcal{P}}\left(x\right)-k_{\mathcal{Q}}\left(x\right)\right].$$

(29)

This formula was reported in [23] and generalizes the Fenchel–Young divergence [17] obtained when $\mathcal{P}=\mathcal{Q}$ (with $t_{\mathcal{P}}\left(x\right)=t_{\mathcal{Q}}\left(x\right)$, $k_{\mathcal{P}}\left(x\right)=k_{\mathcal{Q}}\left(x\right)$, and $F\left(\theta \right)=F_{\mathcal{P}}\left(\theta \right)=F_{\mathcal{Q}}\left(\theta \right)$ and $F^{*}\left(\eta \right)=F_{\mathcal{P}}^{*}\left(\eta \right)=F_{\mathcal{Q}}^{*}\left(\eta \right)$).

The formula of Equation (29) was illustrated in [23] with two examples: the KLD between Laplacian distributions and zero-centered Gaussian distributions, and the KLD between two Weibull distributions. Both these examples use the Lebesgue base measure for ${\mu }_{\mathcal{P}}$ and ${\mu }_{\mathcal{Q}}$.

Let us report another example that uses the counting measure as the base measure for ${\mu }_{\mathcal{P}}$ and ${\mu }_{\mathcal{Q}}$.

Example 1.

Consider the KLD between a Poisson probability mass function (pmf) and a geometric pmf. The canonical decompositions of the Poisson and geometric pmfs are summarized in Table 1. The KLD between a Poisson pmf $p_{\lambda }$ and a geometric pmf $q_p$ is equal to

$$\begin{array}{ccc}\hfill D_{\mathrm{KL}}[P_{\lambda }:Q_p]& =& F_{\mathcal{Q}}\left({\theta }^{\prime }\right)+F_{\mathcal{P}}^{*}\left(\eta \right)-E_{P_{\theta }}\left[t_{\mathcal{Q}}\left(x\right)\right]·{\theta }^{\prime }+E_{P_{\theta }}[k_{\mathcal{P}}\left(x\right)-k_{\mathcal{Q}}\left(x\right)],\hfill \end{array}$$

(30)

$$\begin{array}{ccc}& =& -logp+\lambda log\lambda -\lambda -\lambda log(1-p)-E_{P_{\lambda }}[logx!]\hfill \end{array}$$

(31)

Table 1. Canonical decomposition of the Poisson and the geometric discrete exponential families.

Quantity	Poisson Family $\mathcal{P}$	Geometric Family $\mathcal{Q}$
support	$\mathbb{N}\cup \left\{0\right\}$	$\mathbb{N}\cup \left\{0\right\}$
base measure	counting measure	counting measure
ordinary parameter	rate $\lambda >0$	success probability $p\in (0,1)$
pmf	$\frac{{\lambda }^x}{x!}exp(-\lambda )$	${(1-p)}^{x}p$
sufficient statistic	$t_{\mathcal{P}}\left(x\right)=x$	$t_{\mathcal{Q}}\left(x\right)=x$
natural parameter	$\theta \left(\lambda \right)=log\lambda$	$\theta \left(p\right)=log(1-p)$
cumulant function	$F_{\mathcal{P}}\left(\theta \right)=exp\left(\theta \right)$	$F_{\mathcal{Q}}\left(\theta \right)=-log(1-exp\left(\theta \right))$
	$F_{\mathcal{P}}\left(\lambda \right)=\lambda$	$F_{\mathcal{Q}}\left(p\right)=-log\left(p\right)$
auxiliary term	$k_{\mathcal{P}}\left(x\right)=-logx!$	$k_{\mathcal{Q}}\left(x\right)=0$
moment $\eta =E\left[t\right(x\left)\right]$	$\eta =\lambda$	$\eta =\frac{e^{\theta }}{1-e^{\theta }}=\frac{1}{p}-1$
negentropy	$F_{\mathcal{P}}^{*}\left(\eta \left(\lambda \right)\right)=\lambda log\lambda -\lambda$	$F_{\mathcal{Q}}^{*}\left(\eta \left(p\right)\right)=\left(1-\frac{1}{p}\right)log(1-p)+logp$
($F^{*}\left(\eta \right)=\theta ·\eta -F\left(\theta \right)$)

Since $E_{p_{\lambda }}[-logx!]=-{\sum }_{k=0}^{\infty }{e}^{-\lambda }\frac{{\lambda }^{k}log(k!)}{k!}$, we have

$$D_{\mathrm{KL}}[P_{\lambda }:Q_p]=-logp+\lambda log\frac{\lambda }{1-p}-\lambda -\sum _{k=0}^{\infty }{e}^{-\lambda }\frac{{\lambda }^{k}log(k!)}{k!}.$$

(32)

Note that we can calculate the KLD between two geometric distributions $Q_{p_1}$ and $Q_{p_2}$ as

$$\begin{array}{ccc}\hfill D_{\mathrm{KL}}[Q_{p_1}:Q_{p_2}]& =& B_{F_{\mathcal{Q}}}(\theta \left(p_2\right):\theta \left(p_1\right)),\hfill \end{array}$$

(33)

$$\begin{array}{ccc}& =& F_{\mathcal{Q}}\left(\theta \left(p_2\right)\right)-F_{\mathcal{Q}}\left(\theta \left(p_1\right)\right)-(\theta \left(p_2\right)-\theta \left(p_1\right))\eta \left(p_1\right),\hfill \end{array}$$

(34)

We obtain:

$$D_{\mathrm{KL}}[Q_{p_1}:Q_{p_2}]=log\left(\frac{p_1}{p_2}\right)-\left(1-\frac{1}{p_1}\right)log\frac{1-p_1}{1-p_2}.$$

3. The Duo Fenchel–Young Divergence and Its Corresponding Duo Bregman Divergence

Inspired by formula of Equation (29), we shall define the duo Fenchel–Young divergence using a dominance condition on a pair $(F_1\left(\theta \right),F_2\left(\theta \right))$ of strictly convex generators.

Definition 1

(duo Fenchel–Young divergence). Let $F_1\left(\theta \right)$ and $F_2\left(\theta \right)$ be two strictly convex functions such that $F_1\left(\theta \right)\ge F_2\left(\theta \right)$ for any $\theta \in {\Theta }_{12}=\mathrm{dom}\left(F_1\right)\cap \mathrm{dom}\left(F_2\right)$. Then the duo Fenchel–Young divergence $Y_{F_1,F_2^{*}}(\theta ,{\eta }^{\prime })$ is defined by

$$Y_{F_1,F_2^{*}}(\theta ,{\eta }^{\prime }):=F_1\left(\theta \right)+F_2^{*}\left({\eta }^{\prime }\right)-{\theta }^{\top }{\eta }^{\prime }.$$

(35)

When $F_1\left(\theta \right)=F_2\left(\theta \right)=:F\left(\theta \right)$, we have $F_1^{*}\left(\eta \right)=F_2^{*}\left(\eta \right)=:F^{*}\left(\eta \right)$, and we retrieve the ordinary Fenchel–Young divergence [17]:

$$Y_{F,F^{*}}(\theta ,{\eta }^{\prime }):=F\left(\theta \right)+F^{*}\left({\eta }^{\prime }\right)-{\theta }^{\top }{\eta }^{\prime }\ge 0.$$

(36)

Note that in Equation (35), we have ${\eta }^{\prime }=\nabla F_2\left({\theta }^{\prime }\right)$.

Property 1

(Non-negative duo Fenchel–Young divergence). The duo Fenchel–Young divergence is always non-negative.

Proof. 

The proof relies on the reverse dominance property of strictly convex and differentiable conjugate functions:

Lemma 1

(Reverse majorization order of functions by the Legendre–Fenchel transform). Let $F_1\left(\theta \right)$ and $F_2\left(\theta \right)$ be two Legendre-type convex functions [14]. Then if $F_1\left(\theta \right)\ge F_2\left(\theta \right)$ then we have $F_2^{*}\left(\eta \right)\ge F_1^{*}\left(\eta \right)$.

Proof .

This property is graphically illustrated in Figure 3. The reverse dominance property of the Legendre–Fenchel transformation can be checked algebraically as follows:

$$\begin{array}{ccc}\hfill F_1^{*}\left(\eta \right)& =& \underset{\theta \in \Theta }{sup}\{{\eta }^{\top }\theta -F_1\left(\theta \right)\},\hfill \end{array}$$

(37)

$$\begin{array}{ccc}& =& {\eta }^{\top }{\theta }_1-F_1\left({\theta }_1\right)\left(\mathrm{with}\eta =\nabla F_1\left({\theta }_1\right)\right),\hfill \end{array}$$

(38)

$$\begin{array}{ccc}& \le & {\eta }^{\top }{\theta }_1-F_2\left({\theta }_1\right),\hfill \end{array}$$

(39)

$$\begin{array}{ccc}& \le & \underset{\theta \in \Theta }{sup}\{{\eta }^{\top }\theta -F_2\left(\theta \right)\}=F_2^{*}\left(\eta \right).\hfill \end{array}$$

(40)

 □

Thus we have $F_1^{*}\left(\eta \right)\le F_2^{*}\left(\eta \right)$ when $F_1\left(\theta \right)\ge F_2\left(\theta \right)$. Therefore it follows that $Y_{F_1,F_2^{*}}(\theta ,{\eta }^{\prime })$$\ge 0$ since we have

$$\begin{array}{ccc}\hfill Y_{F_1,F_2^{*}}(\theta ,{\eta }^{\prime })& :=& F_1\left(\theta \right)+F_2^{*}\left({\eta }^{\prime }\right)-{\theta }^{\top }{\eta }^{\prime },\hfill \end{array}$$

(41)

$$\begin{array}{ccc}& \ge & F_1\left(\theta \right)+F_1^{*}\left({\eta }^{\prime }\right)-{\theta }^{\top }{\eta }^{\prime }=Y_{F_1,F_1^{*}}(\theta ,{\eta }^{\prime })\ge 0,\hfill \end{array}$$

(42)

where $Y_{F_1,F_1^{*}}$ is the ordinary Fenchel–Young divergence, which is guaranteed to be non-negative from the Fenchel–Young inequality. □

Figure 3. (a) Visual illustration of the Legendre–Fenchel transformation: $F^{*}\left(\eta \right)$ is measured as the vertical gap (left long black line with both arrows) between the origin and the hyperplane of the “slope” $\eta$ tangent at $F\left(\theta \right)$ evaluated at $\theta =0$. (b) The Legendre transforms $F_1^{*}\left(\eta \right)$ and $F_1^{*}\left(\eta \right)$ of two functions $F_1\left(\theta \right)$ and $F_2\left(\theta \right)$ such that $F_1\left(\theta \right)\ge F_2\left(\theta \right)$ reverse the dominance order: $F_2^{*}\left(\eta \right)\ge F_1^{*}\left(\eta \right)$.

Figure 3. (a) Visual illustration of the Legendre–Fenchel transformation: $F^{*}\left(\eta \right)$ is measured as the vertical gap (left long black line with both arrows) between the origin and the hyperplane of the “slope” $\eta$ tangent at $F\left(\theta \right)$ evaluated at $\theta =0$. (b) The Legendre transforms $F_1^{*}\left(\eta \right)$ and $F_1^{*}\left(\eta \right)$ of two functions $F_1\left(\theta \right)$ and $F_2\left(\theta \right)$ such that $F_1\left(\theta \right)\ge F_2\left(\theta \right)$ reverse the dominance order: $F_2^{*}\left(\eta \right)\ge F_1^{*}\left(\eta \right)$.

We can express the duo Fenchel–Young divergence using the primal coordinate systems as a generalization of the Bregman divergence to two generators that we term the duo Bregman divergence (see Figure 4):

$$B_{F_1,F_2}(\theta :{\theta }^{\prime }):=Y_{F_1,F_2^{*}}(\theta ,{\eta }^{\prime })=F_1\left(\theta \right)-F_2\left({\theta }^{\prime }\right)-{(\theta -{\theta }^{\prime })}^{\top }\nabla F_2\left({\theta }^{\prime }\right),$$

(43)

with ${\eta }^{\prime }=\nabla F_2\left({\theta }^{\prime }\right)$.

Figure 4. The duo Bregman divergence induced by two strictly convex and differentiable functions $F_1$ and $F_2$ such that $F_1\left(\theta \right)\ge F_2\left(\theta \right)$. We check graphically that $B_{F_1,F_2}(\theta :{\theta }^{\prime })\ge B_{F_2}(\theta :{\theta }^{\prime })$ (vertical gaps).

This generalized Bregman divergence is non-negative when $F_1\left(\theta \right)\ge F_2\left(\theta \right)$. Indeed, we check that

$$\begin{array}{ccc}\hfill B_{F_1,F_2}(\theta :{\theta }^{\prime })& =& F_1\left(\theta \right)-F_2\left({\theta }^{\prime }\right)-{(\theta -{\theta }^{\prime })}^{\top }\nabla F_2\left({\theta }^{\prime }\right),\hfill \end{array}$$

(44)

$$\begin{array}{ccc}& \ge & F_2\left(\theta \right)-F_2\left({\theta }^{\prime }\right)-{(\theta -{\theta }^{\prime })}^{\top }\nabla F_2\left({\theta }^{\prime }\right)=B_{F_2}(\theta :{\theta }^{\prime })\ge 0.\hfill \end{array}$$

(45)

Definition 2

(duo Bregman divergence). Let $F_1\left(\theta \right)$ and $F_2\left(\theta \right)$ be two strictly convex functions such that $F_1\left(\theta \right)\ge F_2\left(\theta \right)$ for any $\theta \in {\Theta }_{12}=\mathrm{dom}\left(F_1\right)\cap \mathrm{dom}\left(F_2\right)$. Then the generalized Bregman divergence is defined by

$$B_{F_1,F_2}(\theta :{\theta }^{\prime })=F_1\left(\theta \right)-F_2\left({\theta }^{\prime }\right)-{(\theta -{\theta }^{\prime })}^{\top }\nabla F_2\left({\theta }^{\prime }\right)\ge 0.$$

(46)

Example 2.

Consider $F_1\left(\theta \right)=\frac{a}{2}{\theta }^2$ for $a>0$. We have $\eta =a\theta$, $\theta =\frac{\eta }{a}$, and

$$F_1^{*}\left(\eta \right)=\frac{{\eta }^2}{a}-\frac{a}{2}\frac{{\eta }^2}{a^2}=\frac{{\eta }^2}{2a}.$$

(47)

Let $F_2\left(\theta \right)=\frac{1}{2}{\theta }^2$ so that $F_1\left(\theta \right)\ge F_2\left(\theta \right)$ for $a\ge 1$. We check that $F_1^{*}\left(\eta \right)=\frac{{\eta }^2}{2a}\le F_2^{*}\left(\eta \right)$ when $a\ge 1$. The duo Fenchel–Young divergence is

$$Y_{F_1,F_2^{*}}(\theta ,{\eta }^{\prime })=\frac{a}{2}{\theta }^2+\frac{1}{2}{{\eta }^{\prime }}^2-\theta {\eta }^{\prime }\ge 0,$$

(48)

when $a\ge 1$. We can express the duo Fenchel–Young divergence in the primal coordinate systems as

$$B_{F_1,F_2}(\theta ,{\theta }^{\prime })=\frac{a}{2}{\theta }^2+\frac{1}{2}{{\theta }^{\prime }}^2-\theta {\theta }^{\prime }.$$

(49)

When $a=1$, $F_1\left(\theta \right)=F_2\left(\theta \right)=\frac{1}{2}{\theta }^2:=F\left(\theta \right)$, and we obtain $B_F(\theta ,{\theta }^{\prime })=\frac{1}{2}{\parallel \theta -{\theta }^{\prime }\parallel }_2^2$, half the squared Euclidean distance as expected. Figure 5 displays the graph plot of the duo Bregman divergence for several values of a.

Figure 5. The duo half squared Euclidean distance $D_a^2(\theta :{\theta }^{\prime }):=\frac{a}{2}{\theta }^2+\frac{1}{2}{{\theta }^{\prime }}^2-\theta {\theta }^{\prime }$ is non-negative when $a\ge 1$: (a) half squared Euclidean distance ($a=1$), (b) $a=2$, (c) $a=\frac{1}{2}$, which shows that the divergence can be negative then since $a<1$.

Example 3.

Consider $F_1\left(\theta \right)={\theta }^2$ and $F_2\left(\theta \right)={\theta }^4$ on the domain $\Theta =[0,1]$. We have $F_1\left(\theta \right)\ge F_2\left(\theta \right)$ for $\theta \in \Theta$. The convex conjugate of $F_1\left(\eta \right)$ is $F_1^{*}\left(\eta \right)=\frac{1}{4}{\eta }^2$. We have

$$F_2^{*}\left(\eta \right)={\eta }^{\frac{4}{3}}\left({\left(\frac{1}{4}\right)}^{\frac{1}{3}}-{\left(\frac{1}{4}\right)}^{\frac{4}{3}}\right)=\frac{3}{4^{\frac{4}{3}}}{\eta }^{\frac{4}{3}}$$

(50)

with ${\eta }_2\left(\theta \right)=4{\theta }^3$. Figure 6 plots the convex functions $F_1\left(\theta \right)$ and $F_2\left(\theta \right)$, and their convex conjugates $F_1^{*}\left(\eta \right)$ and $F_2^{*}\left(\eta \right)$. We observe that $F_1\left(\theta \right)\ge F_2\left(\theta \right)$ on $\theta \in [0,1]$ and that $F_1^{*}\left(\eta \right)\le F_2^{*}\left(\eta \right)$ on $H=[0,2]$.

Figure 6. The Legendre transform reverses the dominance ordering: $F_1\left(\theta \right)={\theta }^2\ge F_2\left(\theta \right)={\theta }^4\iff F_1^{*}\left(\eta \right)\le F_2^{*}\left(\eta \right)$ for $\theta \in [0,1]$.

We now state a property between dual duo Bregman divergences:

Property 2

(Dual duo Fenchel–Young and Bregman divergences). We have

$$Y_{F_1,F_2^{*}}(\theta :{\eta }^{\prime })=B_{F_1,F_2}(\theta :{\theta }^{\prime })=B_{F_2^{*},F_1^{*}}({\eta }^{\prime }:\eta )=Y_{F_2^{*},F_1}({\eta }^{\prime }:\theta )$$

(51)

Proof. 

From the Fenchel–Young equalities of the inequalities, we have $F_1\left(\theta \right)={\theta }^{\top }\eta -F_1^{*}\left(\eta \right)$ for $\eta =\nabla F_1\left(\theta \right)$ and $F_2\left({\theta }^{\prime }\right)={{\theta }^{\prime }}^{\top }{\eta }^{\prime }-F_2^{*}\left({\eta }^{\prime }\right)$ with ${\eta }^{\prime }=\nabla F_2\left({\theta }^{\prime }\right)$. Thus we have

$$\begin{array}{ccc}\hfill B_{F_1,F_2}(\theta :{\theta }^{\prime })& =& F_1\left(\theta \right)-F_2\left({\theta }^{\prime }\right)-{(\theta -{\theta }^{\prime })}^{\top }\nabla F_2\left({\theta }^{\prime }\right),\hfill \end{array}$$

(52)

$$\begin{array}{ccc}& =& {\theta }^{\top }\eta -F_1^{*}\left(\eta \right)-{{\theta }^{\prime }}^{\top }{\eta }^{\prime }+F_2^{*}\left({\eta }^{\prime }\right)-{(\theta -{\theta }^{\prime })}^{\top }{\eta }^{\prime },\hfill \end{array}$$

(53)

$$\begin{array}{ccc}& =& F_2^{*}\left({\eta }^{\prime }\right)-F_1^{*}\left(\eta \right)-{({\eta }^{\prime }-\eta )}^{\top }\theta ,\hfill \end{array}$$

(54)

$$\begin{array}{ccc}& =& B_{F_2^{*},F_1^{*}}({\eta }^{\prime }:\eta ).\hfill \end{array}$$

(55)

Recall that $F_1\left(\theta \right)\ge F_2\left(\theta \right)$ implies that $F_1^{*}\left(\eta \right)\le F_2^{*}\left(\eta \right)$ (Lemma 1), $\theta =\nabla F_1^{*}\left(\eta \right)$, and therefore the dual duo Bregman divergence is non-negative:

$$\begin{array}{ccc}\hfill B_{F_2^{*},F_1^{*}}({\eta }^{\prime }:\eta )& =& F_2^{*}\left({\eta }^{\prime }\right)-F_1^{*}\left(\eta \right)-{({\eta }^{\prime }-\eta )}^{\top }\theta ,\hfill \\ & \ge & \underset{B_{F_1^{*}}({\eta }^{\prime }:\eta )\ge 0}{\underbrace{F_1^{*}\left({\eta }^{\prime }\right)-F_1^{*}\left(\eta \right)-{({\eta }^{\prime }-\eta )}^{\top }\nabla F_1^{*}\left(\eta \right)}}.\hfill \end{array}$$

 □

4. Kullback–Leibler Divergence between Distributions of Truncated Exponential Families

Let ${\mathcal{E}}_1=\{P_{\theta }:\theta \in {\Theta }_1\}$ be an exponential family of distributions all dominated by $\mu$ with Radon–Nikodym density $p_{\theta }\left(x\right)=exp({\theta }^{\top }t\left(x\right)-F_1\left(\theta \right)+k\left(x\right))\mathrm{d}\mu \left(x\right)$ defined on the support ${\mathcal{X}}_1$. Let ${\mathcal{E}}_2=\{Q_{\theta }:\theta \in {\Theta }_2\}$ be another exponential family of distributions all dominated by $\mu$ with Radon–Nikodym density $q_{\theta }\left(x\right)=exp({\theta }^{\top }t\left(x\right)-F_2\left(\theta \right)+k\left(x\right))\mathrm{d}\mu \left(x\right)$ defined on the support ${\mathcal{X}}_2$ such that ${\mathcal{X}}_1\subseteq {\mathcal{X}}_2$. Let ${\tilde{p}}_{\theta }\left(x\right)=exp({\theta }^{\top }t\left(x\right)+k\left(x\right))\mathrm{d}\mu \left(x\right)$ be the common unnormalized density so that

$$p_{\theta }\left(x\right)=\frac{{\tilde{p}}_{\theta }\left(x\right)}{Z_1\left(\theta \right)}$$

(56)

and

$$q_{\theta }\left(x\right)=\frac{{\tilde{p}}_{\theta }\left(x\right)}{Z_2\left(\theta \right)}=\frac{Z_1\left(\theta \right)}{Z_2\left(\theta \right)}{p}_{\theta }\left(x\right),$$

(57)

with $Z_1\left(\theta \right)=exp\left(F_1\left(\theta \right)\right)$ and $Z_2\left(\theta \right)=exp\left(F_2\left(\theta \right)\right)$ being the log-normalizer functions of ${\mathcal{E}}_1$ and ${\mathcal{E}}_2$, respectively.

We have

$$\begin{array}{ccc}\hfill D_{\mathrm{KL}}[p_{{\theta }_1}:q_{{\theta }_2}]& =& {\int }_{{\mathcal{X}}_1}{p}_{{\theta }_1}\left(x\right)log\frac{p_{{\theta }_1}\left(x\right)}{q_{{\theta }_2}\left(x\right)}\mathrm{d}\mu \left(x\right),\hfill \end{array}$$

(58)

$$\begin{array}{ccc}& =& {\int }_{{\mathcal{X}}_1}{p}_{{\theta }_1}\left(x\right)log\frac{p_{{\theta }_1}\left(x\right)}{p_{{\theta }_2}\left(x\right)}\mathrm{d}\mu \left(x\right)+{\int }_{{\mathcal{X}}_1}{p}_{{\theta }_1}\left(x\right)log\left(\frac{Z_2\left({\theta }_2\right)}{Z_1\left({\theta }_2\right)}\right)\mathrm{d}\mu \left(x\right),\hfill \end{array}$$

(59)

$$\begin{array}{ccc}& =& D_{\mathrm{KL}}[p_{{\theta }_1}:p_{{\theta }_2}]+log{Z}_2\left({\theta }_2\right)-log{Z}_1\left({\theta }_2\right).\hfill \end{array}$$

(60)

Since $D_{\mathrm{KL}}[p_{{\theta }_1}:p_{{\theta }_2}]=B_{F_1}({\theta }_2:{\theta }_1)$ and $log{Z}_i\left(\theta \right)=F_i\left(\theta \right)$, we obtain

$$\begin{array}{ccc}\hfill D_{\mathrm{KL}}[p_{{\theta }_1}:q_{{\theta }_2}]& =& B_{F_1}({\theta }_2:{\theta }_1)+F_2\left({\theta }_2\right)-F_1\left({\theta }_2\right),\hfill \end{array}$$

(61)

$$\begin{array}{ccc}& =& F_1\left({\theta }_2\right)-F_1\left({\theta }_1\right)-{({\theta }_2-{\theta }_1)}^{\top }\nabla F_1\left({\theta }_1\right)+F_2\left({\theta }_2\right)-F_1\left({\theta }_2\right),\hfill \end{array}$$

(62)

$$\begin{array}{ccc}& =& F_2\left({\theta }_2\right)-F_1\left({\theta }_1\right)-{({\theta }_2-{\theta }_1)}^{\top }\nabla F_1\left({\theta }_1\right)=:B_{F_2,F_1}({\theta }_2:{\theta }_1).\hfill \end{array}$$

(63)

Observe that since ${\mathcal{X}}_1\subseteq \mathcal{X}2$, we have:

$$F_2\left(\theta \right)=log{\int }_{{\mathcal{X}}_2}{\tilde{p}}_{\theta }\left(x\right)\mathrm{d}\mu \left(x\right)\ge log{\int }_{{\mathcal{X}}_1}{\tilde{p}}_{\theta }\left(x\right)\mathrm{d}\mu \left(x\right):=F_1\left(\theta \right).$$

(64)

Therefore ${\Theta }_2\subseteq {\Theta }_1$, and the common natural parameter space is ${\Theta }_{12}={\Theta }_1\cap {\Theta }_2={\Theta }_2$.

Notice that the reverse Kullback–Leibler divergence $D_{\mathrm{KL}}^{*}[p_{{\theta }_1}:q_{{\theta }_2}]=D_{\mathrm{KL}}[q_{{\theta }_2}:p_{{\theta }_1}]=+\infty$ since $Q_{{\theta }_2}\cancel{\ll }{P}_{{\theta }_1}$.

Theorem 1

(Kullback–Leibler divergence between truncated exponential family densities). Let ${\mathcal{E}}_2=\left\{q_{{\theta }_2}\right\}$ be an exponential family with support ${\mathcal{X}}_2$, and ${\mathcal{E}}_1=\left\{p_{{\theta }_1}\right\}$ a truncated exponential family of ${\mathcal{E}}_2$ with support ${\mathcal{X}}_1\subset {\mathcal{X}}_2$. Let $F_1$ and $F_2$ denote the log-normalizers of ${\mathcal{E}}_1$ and ${\mathcal{E}}_2$ and ${\eta }_1$ and ${\eta }_2$ the moment parameters corresponding to the natural parameters ${\theta }_1$ and ${\theta }_2$. Then the Kullback–Leibler divergence between a truncated density of ${\mathcal{E}}_1$ and a density of ${\mathcal{E}}_2$ is

$$D_{\mathrm{KL}}[p_{{\theta }_1}:q_{{\theta }_2}]=Y_{F_2,F_1^{*}}({\theta }_2:{\eta }_1)=B_{F_2,F_1}({\theta }_2:{\theta }_1)=B_{F_1^{*},F_2^{*}}({\eta }_1:{\eta }_2)=Y_{F_1^{*},F_2}({\eta }_1:{\theta }_2).$$

(65)

For example, consider the calculation of the KLD between an exponential distribution (view as half a Laplacian distribution, i.e., a truncated Laplacian distribution on the positive real support) and a Laplacian distribution defined on the real line support.

Example 4.

Let ${\mathbb{R}}_{++}=\{x\in \mathbb{R}:x>0\}$ denote the set of positive reals. Let ${\mathcal{E}}_1=\{p_{\lambda }\left(x\right)=\lambda exp(-\lambda x),\lambda \in {\mathbb{R}}_{++},x>0\}$ and ${\mathcal{E}}_2=\{q_{\lambda }\left(x\right)=\lambda exp(-\lambda |x\left|\right),\lambda \in {\mathbb{R}}_{++},x\in \mathbb{R}\}$ denote the exponential families of exponential distributions and Laplacian distributions, respectively. We have the sufficient statistic $t\left(x\right)=-\left|x\right|$ and natural parameter $\theta =\lambda$ so that ${\tilde{p}}_{\theta }\left(x\right)=exp(-|x\left|\theta \right)$. The log-normalizers are $F_1\left(\theta \right)=-log\theta$ and $F_2\left(\theta \right)=-log\theta +log2$ (hence $F_2\left(\theta \right)\ge F_1\left(\theta \right)$). The moment parameter $\eta =\nabla F_1\left(\theta \right)=\nabla F_2\left(\theta \right)=-\frac{1}{\theta }=-\frac{1}{\lambda }$. Thus using the duo Bregman divergence, we have:

$$\begin{array}{ccc}\hfill D_{\mathrm{KL}}[p_{{\theta }_1}:q_{{\theta }_2}]& =& B_{F_2,F_1}({\theta }_2:{\theta }_1),\hfill \end{array}$$

(66)

$$\begin{array}{ccc}& =& F_2\left({\theta }_2\right)-F_1\left({\theta }_1\right)-{({\theta }_2-{\theta }_1)}^{\top }\nabla F_1\left({\theta }_1\right),\hfill \end{array}$$

(67)

$$\begin{array}{ccc}& =& log2+log\frac{{\lambda }_1}{{\lambda }_2}+\frac{{\lambda }_2}{{\lambda }_1}-1.\hfill \end{array}$$

(68)

Moreover, we can interpret that divergence using the Itakura–Saito divergence [24]:

$$D_{\mathrm{IS}}[{\lambda }_1:{\lambda }_2]:=\frac{{\lambda }_1}{{\lambda }_2}-log\frac{{\lambda }_1}{{\lambda }_2}-1\ge 0.$$

(69)

we have

$$D_{\mathrm{KL}}[p_{{\theta }_1}:q_{{\theta }_2}]=D_{\mathrm{IS}}[{\lambda }_2:{\lambda }_1]+log2\ge 0.$$

(70)

We check the result using the duo Fenchel–Young divergence:

$$D_{\mathrm{KL}}[p_{{\theta }_1}:q_{{\theta }_2}]=Y_{F_2,F_1^{*}}({\theta }_2:{\eta }_1),$$

(71)

with $F_1^{*}\left(\eta \right)=-1+log\left(-\frac{1}{\eta }\right)$:

$$\begin{array}{ccc}\hfill D_{\mathrm{KL}}[p_{{\theta }_1}:q_{{\theta }_2}]& =& Y_{F_2,F_1^{*}}({\theta }_2:{\eta }_1),\hfill \end{array}$$

(72)

$$\begin{array}{ccc}& =& -log{\lambda }_2+log2-1+log{\lambda }_1+\frac{{\lambda }_2}{{\lambda }_1},\hfill \end{array}$$

(73)

$$\begin{array}{ccc}& =& log\frac{{\lambda }_1}{{\lambda }_2}+\frac{{\lambda }_2}{{\lambda }_1}+{log}_2-1.\hfill \end{array}$$

(74)

Next, consider the calculation of the KLD between a half-normal distribution and a (full) normal distribution:

Example 5.

Consider ${\mathcal{E}}_1$ and ${\mathcal{E}}_2$ to be the scale family of the half standard normal distributions and the scale family of the standard normal distribution, respectively. We have ${\tilde{p}}_{\theta }\left(x\right)=exp\left(-\frac{x^2}{2{\sigma }^2}\right)$ with $Z_1\left(\theta \right)=\sigma \sqrt{\frac{\pi }{2}}$ and $Z_2\left(\theta \right)=\sigma \sqrt{2\pi }$. Let the sufficient statistic be $t\left(x\right)=-\frac{x^2}{2}$ so that the natural parameter is $\theta =\frac{1}{{\sigma }^2}\in {\mathbb{R}}_{++}$. Here, we have both ${\Theta }_1={\Theta }_2={\mathbb{R}}_{++}$. For this example, we check that $Z_1\left(\theta \right)=\frac{1}{2}{Z}_2\left(\theta \right)$. We have $F_1\left(\theta \right)=-\frac{1}{2}log\theta +\frac{1}{2}log\frac{\pi }{2}$ and $F_2\left(\theta \right)=-\frac{1}{2}log\theta +\frac{1}{2}log\left(2\pi \right)$ (with $F_2\left(\theta \right)\ge F_1\left(\theta \right)$). We have $\eta =-\frac{1}{2\theta }=-\frac{1}{2}{\sigma }^2$. The KLD between two half scale normal distributions is

$$\begin{array}{ccc}\hfill D_{\mathrm{KL}}[p_{{\theta }_1}:p_{{\theta }_2}]& =& B_{F_1}({\theta }_2:{\theta }_1),\hfill \end{array}$$

(75)

$$\begin{array}{ccc}& =& \frac{1}{2}\left(log\frac{{\sigma }_2^2}{{\sigma }_1^2}+\frac{{\sigma }_1^2}{{\sigma }_2^2}-1\right).\hfill \end{array}$$

(76)

Since $F_1\left(\theta \right)$ and $F_2\left(\theta \right)$ differ only by a constant and the Bregman divergence is invariant under an affine term of its generator, we have

$$\begin{array}{ccc}\hfill D_{\mathrm{KL}}[q_{{\theta }_1}:q_{{\theta }_2}]& =& B_{F_2}({\theta }_2:{\theta }_1),\hfill \end{array}$$

(77)

$$\begin{array}{ccc}& =& B_{F_1}({\theta }_2:{\theta }_1)=\frac{1}{2}\left(log\frac{{\sigma }_2^2}{{\sigma }_1^2}+\frac{{\sigma }_1^2}{{\sigma }_2^2}-1\right).\hfill \end{array}$$

(78)

Moreover, we can interpret those Bregman divergences as half of the Itakura–Saito divergence:

$$D_{\mathrm{KL}}[p_{{\theta }_1}:p_{{\theta }_2}]=D_{\mathrm{KL}}[q_{{\theta }_1}:q_{{\theta }_2}]=B_{F_2}({\theta }_2:{\theta }_1)=\frac{1}{2}{D}_{\mathrm{IS}}[{\sigma }_1^2:{\sigma }_2^2].$$

(79)

It follows that

$$\begin{array}{ccc}\hfill D_{\mathrm{KL}}[p_{{\theta }_1}:q_{{\theta }_2}]& =& B_{F_2,F_1}({\theta }_2:{\theta }_1)=F_2\left({\theta }_2\right)-F_1\left({\theta }_1\right)-{({\theta }_2-{\theta }_1)}^{\top }\nabla F_1\left({\theta }_1\right),\hfill \end{array}$$

(80)

$$\begin{array}{ccc}& =& \frac{1}{2}\left(log\frac{{\sigma }_2^2}{{\sigma }_1^2}+\frac{{\sigma }_1^2}{{\sigma }_2^2}+log4-1\right),\hfill \end{array}$$

(81)

$$\begin{array}{ccc}& =& D_{\mathrm{KL}}[q_{{\theta }_1}:q_{{\theta }_2}]+log2.\hfill \end{array}$$

(82)

Since $log2>0$, we have $D_{\mathrm{KL}}[p_{{\theta }_1}:q_{{\theta }_2}]\ge D_{\mathrm{KL}}[q_{{\theta }_1}:q_{{\theta }_2}]$.

Thus the Kullback–Leibler divergence between a truncated density and another density of the same exponential family amounts to calculate a duo Bregman divergence on the reverse parameter order: $D_{\mathrm{KL}}[p_{{\theta }_1}:q_{{\theta }_2}]=B_{F_2,F_1}({\theta }_2:{\theta }_1)$. Let $D_{\mathrm{KL}}^{*}[p:q]:=D_{\mathrm{KL}}[q:p]$ be the reverse Kullback–Leibler divergence. Then $D_{\mathrm{KL}}^{*}[q_{{\theta }_2}:p_{{\theta }_1}]=B_{F_2,F_1}({\theta }_2:{\theta }_1)$.

Notice that truncated exponential families are also exponential families but those exponential families may be non-steep [25].

Let ${\mathcal{E}}_1=\left\{p_{\theta }^{a_1,b_1}\right\}$ and ${\mathcal{E}}_2=\left\{p_{\theta }^{a_2,b_2}\right\}$ be two truncated exponential families of the exponential family $\mathcal{E}=\{p_{\theta }=\frac{\mathrm{d}{P}_{\theta }}{\mathrm{d}\mu }\}$ with log-normalizer $F\left(\theta \right)$ such that

$$p_{\theta }^{a_i,b_i}\left(x\right)=\frac{p_{\theta }\left(x\right)}{Z_{a_i,b_i}\left(\theta \right)},$$

(83)

with $Z_{a_i,b_i}\left(\theta \right)={\Phi }_{\theta }\left(b_i\right)-{\Phi }_{\theta }\left(a_i\right)$, where ${\Phi }_{\theta }\left(x\right)$ denotes the CDF of $p_{\theta }\left(x\right)$. Then the log-normalizer of ${\mathcal{E}}_i$ is $F_i\left(\theta \right)=F\left(\theta \right)+log({\Phi }_{\theta }\left(b_i\right)-{\Phi }_{\theta }\left(a_i\right))$ for $i\in \{1,2\}$.

Corollary 1

(Kullback–Leibler divergence between densities of truncated exponential families). Let ${\mathcal{E}}_i=\left\{p_{\theta }^{a_i,b_i}\right\}$ be truncated exponential families of the exponential family $\mathcal{E}=\left\{p_{\theta }\right\}$ with support ${\mathcal{X}}_i=[a_i,b_i]\subset \mathcal{X}$ (where $\mathcal{X}$ denotes the support of $\mathcal{E}$) for $i\in \{1,2\}$. Then the Kullback–Leibler divergence between $p_{{\theta }_1}^{a_1,b_1}$ and $p_{{\theta }_2}^{a_2,b_2}$ is infinite if $[a_1,b_1]\cancel{\subset }[a_2,b_2]$ and has the following formula when $[a_1,b_1]\subset [a_2,b_2]$:

$$D_{\mathrm{KL}}[p_{{\theta }_1}^{a_1,b_1}:p_{{\theta }_2}^{a_2,b_2}]=D_{\mathrm{KL}}[p_{{\theta }_1}^{a_1,b_1}:p_{{\theta }_2}^{a_1,b_1}]+log\frac{Z_{a_2,b_2}\left({\theta }_2\right)}{Z_{a_1,b_1}\left({\theta }_2\right)}.$$

(84)

Proof. 

We have $p_{\theta }^{a_1,b_1}=\frac{p_{\theta }}{Z_{a_1,b_1}\left(\theta \right)}$ and $p_{\theta }^{a_2,b_2}=\frac{p_{\theta }}{Z_{a_2,b_2}\left(\theta \right)}$. Therefore $p_{\theta }^{a_2,b_2}=p_{\theta }^{a_1,b_1}\frac{Z_{a_1,b_1}\left(\theta \right)}{Z_{a_2,b_2}\left(\theta \right)}$. Thus we have

$$\begin{array}{ccc}\hfill D_{\mathrm{KL}}[p_{{\theta }_1}^{a_1,b_1}:p_{{\theta }_2}^{a_2,b_2}]& =& {\int }_{{\mathcal{X}}_1}{p}_{{\theta }_1}^{a_1,b_1}\left(x\right)log\frac{p_{{\theta }_1}^{a_1,b_1}\left(x\right)}{p_{{\theta }_2}^{a_2,b_2}}\mathrm{d}\mu \left(x\right),\hfill \end{array}$$

(85)

$$\begin{array}{ccc}& =& {\int }_{{\mathcal{X}}_1}{p}_{{\theta }_1}^{a_1,b_1}\left(x\right)log\frac{p_{{\theta }_1}^{a_1,b_1}\left(x\right)}{p_{{\theta }_2}^{a_1,b_1}}\mathrm{d}\mu \left(x\right)+log\frac{Z_{a_2,b_2}\left({\theta }_2\right)}{Z_{a_1,b_1}\left({\theta }_2\right)},\hfill \end{array}$$

(86)

$$\begin{array}{ccc}& =& D_{\mathrm{KL}}[p_{{\theta }_1}^{a_1,b_1}:p_{{\theta }_2}^{a_1,b_1}]+log\frac{Z_{a_2,b_2}\left({\theta }_2\right)}{Z_{a_1,b_1}\left({\theta }_2\right)}.\hfill \end{array}$$

(87)

 □

Thus the KLD between truncated exponential family densities $p_{{\theta }_1}^{a_1,b_1}$ and $p_{{\theta }_2}^{a_2,b_2}$ amounts to the KLD between the densities with the same truncation parameter with an additive term depending on the log ratio of the mass with respect to the truncated supports evaluated at ${\theta }_2$. We shall illustrate with two examples the calculation of the KLD between truncated exponential families.

Example 6.

Consider the calculation of the KLD between a truncated exponential distribution $p_{{\lambda }_1}^{a_1,b_1}$ with support ${\mathcal{X}}_1=[a_1,b_1]$ ($b_1>a_1\ge 0$) and another truncated exponential distribution $p_{{\lambda }_2}^{a_2,b_2}$ with support ${\mathcal{X}}_2=[a_2,b_2]$ ($b_2>a_2\ge 0$). We have $p_{\lambda }\left(x\right)=\lambda exp(-\lambda x)$ (density of the untruncated exponential family with natural parameter $\theta =\lambda$, sufficient statistic $t\left(x\right)=-x$ and log-normalizer $F\left(\theta \right)=-log\theta$), $p_{{\lambda }_1}^{a_1,b_1}=\frac{1}{Z_{a_1,b_1}\left(\lambda \right)}{p}_{{\lambda }_1}\left(x\right)$, and $p_{{\lambda }_2}^{a_2,b_2}=\frac{1}{Z_{a_2,b_2}\left(\lambda \right)}{p}_{{\lambda }_2}\left(x\right)$. Let ${\Phi }_{\lambda }\left(x\right)=1-exp(-\lambda x)$ denote the cumulative distribution function of the exponential distribution. We have $Z_{a,b}\left(\lambda \right)={\Phi }_b\left(\lambda \right)-{\Phi }_a\left(\lambda \right)$ and

$$F_{a,b}\left(\lambda \right)=F\left(\lambda \right)+log({\Phi }_b\left(\lambda \right)-{\Phi }_a\left(\lambda \right))=-log\lambda +log(e^{-\lambda a}-e^{-\lambda b}).$$

(88)

If $[a_1,b_1]\notin [a_2,b_2]$ then $D_{\mathrm{KL}}[p_{{\lambda }_1}:q_{{\lambda }_2}]=+\infty$. Otherwise, $[a_1,b_1]\in [a_2,b_2]$, and the exponential family $\left\{p_{\lambda }\right\}$ is the truncated exponential family $\left\{q_{\lambda }\right\}$. Using the computer algebra system Maxima (https://maxima.sourceforge.io/ accessed on 15 March 2022), we find that

$$-E_{p_{\lambda }}\left[x\right]=\frac{(1+\lambda b)e^{\lambda a}-(1+\lambda a)e^{\lambda b}}{\lambda (e^{\lambda b}-e^{\lambda a})}=F_{a,b}^{\prime }\left(\lambda \right).$$

(89)

Thus we have:

$$\begin{array}{ccc}\hfill D_{\mathrm{KL}}[p_{{\lambda }_1}^{a_1,b_1}:q_{{\lambda }_2}^{a_2,b_2}]& =& B_{F_2,F_1}({\theta }_2:{\theta }_1),\hfill \\ & =& F_{a_2,b_2}\left({\lambda }_2\right)-F_{a_1,b_1}\left({\lambda }_1\right)-({\lambda }_2-{\lambda }_1)F_{a_1,b_1}^{\prime }\left({\lambda }_1\right),\hfill \end{array}$$

(90)

$$\begin{array}{ccc}& =& log\frac{{\lambda }_1}{{\lambda }_2}+({\lambda }_2-{\lambda }_1)E_{p_{{\lambda }_1}}\left[x\right]+log\frac{e^{-{\lambda }_2{a}_2}-e^{-{\lambda }_2{b}_2}}{e^{-{\lambda }_1{a}_1}-e^{-{\lambda }_1{b}_1}}.\hfill \end{array}$$

(91)

When $a_1=a_2=0$ and $b_1=b_2=+\infty$, we recover the KLD between two exponential distributions $p_{{\lambda }_1}$ and $p_{{\lambda }_2}$:

$$\begin{array}{ccc}\hfill D_{\mathrm{KL}}[p_{{\lambda }_1}:p_{{\lambda }_2}]& =& B_F({\lambda }_2:{\lambda }_1),\hfill \end{array}$$

(92)

$$\begin{array}{ccc}& =& F\left({\theta }_2\right)-F\left({\theta }_1\right)-({\theta }_2-{\theta }_1)F^{\prime }\left({\theta }_1\right),\hfill \end{array}$$

(93)

$$\begin{array}{ccc}& =& \frac{{\lambda }_2}{{\lambda }_1}-log\frac{{\lambda }_2}{{\lambda }_1}-1=D_{\mathrm{IS}}[{\lambda }_2:{\lambda }_1].\hfill \end{array}$$

(94)

Note that the KLD between two truncated exponential distributions with the same truncation support $\mathcal{X}=[a,b]$ is

$$D_{\mathrm{KL}}[p_{{\lambda }_1}^{a,b}:p_{{\lambda }_2}^{a,b}]=log\frac{{\lambda }_2}{{\lambda }_1}+log\frac{{\Phi }_{{\lambda }_2}\left(b\right)-{\Phi }_{{\lambda }_2}\left(a\right)}{{\Phi }_{{\lambda }_1}\left(b\right)-{\Phi }_{{\lambda }_1}\left(a\right)}+({\lambda }_2-{\lambda }_1)E_{p_1^{a,b}}\left[x\right].$$

(95)

We also check Corollary 1:

$$D_{\mathrm{KL}}[p_{{\lambda }_1}^{a_1,b_1}:p_{{\lambda }_2}^{a_2,b_2}]=D_{\mathrm{KL}}[p_{{\lambda }_1}^{a_1,b_1}:p_{{\lambda }_2}^{a_1,b_1}]+log\frac{Z_{a_2,b_2}\left({\lambda }_2\right)}{Z_{a_1,b_1}\left({\lambda }_2\right)},$$

(96)

The next example shows how to compute the Kullback–Leibler divergence between two truncated normal distributions:

Example 7.

Let $N_{a,b}(m,s)$ denote a truncated normal distribution with support the open interval $(a,b)$ ($a<b$) and probability density function defined by:

$$p_{m,s}^{a,b}\left(x\right)=\frac{1}{Z_{a,b}(m,s)}exp\left(-\frac{{(x-m)}^2}{2s^2}\right),$$

(97)

where $Z_{a,b}(m,s)$ is related to the partition function [26] expressed using the cumulative distribution function (CDF) ${\Phi }_{m,s}\left(x\right)$:

$$Z_{a,b}(m,s)=\sqrt{2\pi }s\left({\Phi }_{m,s}\left(b\right)-{\Phi }_{m,s}\left(a\right)\right),$$

(98)

with

$${\Phi }_{m,s}\left(x\right)=\frac{1}{2}\left(1+\mathrm{erf}\left(\frac{x-m}{\sqrt{2}s}\right)\right),$$

(99)

where $\mathrm{erf}\left(x\right)$ is the error function:

$$\mathrm{erf}\left(x\right):=\frac{2}{\sqrt{\pi }}{\int }_0^x{e}^{-t^2}\mathrm{d}t.$$

(100)

Thus we have $\mathrm{erf}\left(x\right)=2\Phi \left(\sqrt{2}x\right)-1$ where $\Phi \left(x\right)={\Phi }_{0,1}\left(x\right)$.

The pdf can also be written as

$$p_{m,s}^{a,b}\left(x\right)=\frac{1}{s}\frac{\varphi \left(\frac{x-m}{s}\right)}{\Phi \left(\frac{b-m}{s}\right)-\Phi \left(\frac{a-m}{s}\right)},$$

(101)

where $\varphi \left(x\right)$ denotes the standard normal pdf ($\varphi \left(x\right)=p_{0,1}^{-\infty ,+\infty }\left(x\right)$):

$$\varphi \left(x\right):=\frac{1}{\sqrt{2\pi }}exp\left(-\frac{x^2}{2}\right),$$

(102)

and $\Phi \left(x\right)={\Phi }_{0,1}\left(x\right)={\int }_{-\infty }^x\varphi \left(t\right)\mathrm{d}t$ is the standard normal CDF. When $a=-\infty$ and $b=+\infty$, we have $Z_{-\infty ,\infty }(m,s)=\sqrt{2\pi }s$ since $\Phi (-\infty )=0$ and $\Phi (+\infty )=1$.

The density $p_{m,s}^{a,b}\left(x\right)$ belongs to an exponential family ${\mathcal{E}}_{a,b}$ with natural parameter $\theta =\left(\frac{m}{s^2},-\frac{1}{2s^2}\right)$, sufficient statistics $t\left(x\right)=(x,x^2)$, and log-normalizer:

$$F_{a,b}\left(\theta \right)=-\frac{{\theta }_1^2}{4{\theta }_2}+log{Z}_{a,b}\left(\theta \right)$$

(103)

The natural parameter space is $\Theta =\mathbb{R}\times {\mathbb{R}}_{--}$ where ${\mathbb{R}}_{--}=\{x\in \mathbb{R}:x<0\}$ denotes the set of negative real numbers.

The log-normalizer can be expressed using the source parameters $(m,s)$ (which are not the mean and standard deviation when the support is truncated, hence the notation m and s):

$$\begin{array}{ccc}\hfill F_{a,b}(m,s)& =& \frac{m^2}{2s^2}+log{Z}_{a,b}(m,s),\hfill \end{array}$$

(104)

$$\begin{array}{ccc}& =& \frac{m^2}{2s^2}+\frac{1}{2}log2\pi s^2+log\left({\Phi }_{m,s}\left(b\right)-{\Phi }_{m,s}\left(a\right)\right).\hfill \end{array}$$

(105)

We shall use the fact that the gradient of the log-normalizer of any exponential family distribution amounts to the expectation of the sufficient statistics [1]:

$$\nabla F_{a,b}\left(\theta \right)=E_{p_{m,s}^{a,b}}\left[t\left(x\right)\right]=\eta .$$

(106)

Parameter η is called the moment or expectation parameter [1].

The mean $\mu (m,s;a,b)=E_{p_{m,s}^{a,b}}\left[x\right]=\frac{\partial }{\partial {\theta }_1}{F}_{a,b}\left(\theta \right)$ and the variance ${\sigma }^2(m,s;a,b)=E_{p_{m,s}^{a,b}}\left[x^2\right]-{\mu }^2$ (with $E_{p_{m,s}^{a,b}}\left[x^2\right]=\frac{\partial }{\partial {\theta }_2}{F}_{a,b}\left(\theta \right)$) of the truncated normal $p_{m,s}^{a,b}$ can be expressed using the following formula [26,27] (page 25):

$$\begin{array}{ccc}\hfill \mu (m,s;a,b)& =& m-s\frac{\varphi \left(\beta \right)-\varphi \left(\alpha \right)}{\Phi \left(\beta \right)-\Phi \left(\alpha \right)},\hfill \end{array}$$

(107)

$$\begin{array}{ccc}\hfill {\sigma }^2(m,s;a,b)& =& s^2\left(1-\frac{\beta \varphi \left(\beta \right)-\alpha \varphi \left(\alpha \right)}{\Phi \left(\beta \right)-\Phi \left(\alpha \right)}-{\left(\frac{\varphi \left(\beta \right)-\varphi \left(\alpha \right)}{\Phi \left(\beta \right)-\Phi \left(\alpha \right)}\right)}^2\right),\hfill \end{array}$$

(108)

where $\alpha :=\frac{a-m}{s}$ and $\beta :=\frac{b-m}{s}$. Thus we have the following moment parameter $\eta =({\eta }_1,{\eta }_2)$ with

$$\begin{array}{ccc}\hfill {\eta }_1(m,s;a,b)& =& E_{p_{m,s}^{a,b}}\left[x\right]=\mu (m,s;a,b),\hfill \end{array}$$

(109)

$$\begin{array}{ccc}\hfill {\eta }_2(m,s;a,b)& =& E_{p_{m,s}^{a,b}}\left[x^2\right]={\sigma }^2(m,s;a,b)+{\mu }^2(m,s;a,b).\hfill \end{array}$$

(110)

Now consider two truncated normal distributions $p_{m_1,s_1}^{a_1,b_1}$ and $p_{m_2,s_2}^{a_2,b_2}$ with $[a_1,b_1]\subseteq [a_2,b_2]$ (otherwise, we have $D_{\mathrm{KL}}[p_{m_1,s_1}^{a_1,b_1}:p_{m_2,s_2}^{a_2,b_2}]=+\infty$). Then the KLD between $p_{m_1,s_1}^{a_1,b_1}$ and $p_{m_2,s_2}^{a_2,b_2}$ is equivalent to a duo Bregman divergence:

$$\begin{array}{ccc}\hfill D_{\mathrm{KL}}[p_{m_1,s_1}^{a_1,b_1}:p_{m_2,s_2}^{a_2,b_2}]& =& F_{m_2,s_2}\left({\theta }_2\right)-F_{m_1,s_1}\left({\theta }_1\right)-{({\theta }_2-{\theta }_1)}^{\top }\nabla F_{m_1,s_1}\left({\theta }_1\right),\hfill \\ & =& \frac{m_2}{2s_2^2}-\frac{m_1}{2s_1^2}+log\frac{Z_{a_2,b_2}(m_2,s_2)}{Z_{a_1,b_1}(m_1,s_1)}-\left(\frac{m_2}{s_2^2}-\frac{m_1}{s_1^2}\right){\eta }_1(m_1,s_1;a_1,b_1)\hfill \\ & & -\left(\frac{1}{2s_1^2}-\frac{1}{2s_2^2}\right){\eta }_2(m_1,s_1;a_1,b_1).\hfill \end{array}$$

(111)

Note that $F_{m_2,s_2}\left(\theta \right)\ge F_{m_1,s_1}\left(\theta \right)$.

This formula is valid for (1) the KLD between two truncated normal distributions, or for (2) the KLD between a truncated normal distribution and a (full support) normal distribution. Note that the formula depends on the erf function used in function Φ. Furthermore, when $a_1=a_2=-\infty$ and $b_1=b_2=+\infty$, we recover (3) the KLD between two univariate normal distributions, since $log\frac{Z_{a_2,b_2}(m_2,s_2)}{Z_{a_1,b_1}(m_1,s_1)}=log\frac{{\sigma }_2}{{\sigma }_1}=\frac{1}{2}log\frac{{\sigma }_2^2}{{\sigma }_1^2}$:

$$D_{\mathrm{KL}}[p_{m_1,s_1}:p_{m_2,s_2}]=\frac{1}{2}\left(log\frac{s_2^2}{s_1^2}+\frac{{\sigma }_1^2}{{\sigma }_2^2}+\frac{{(m_2-m_1)}^2}{s_2^2}-1.\right).$$

(112)

Note that for full support normal distributions, we have $\mu (m,s;-\infty ;+\infty )=m$ and ${\sigma }^2(m,s;-\infty ;+\infty )=s^2$.

The entropy of a truncated normal distribution (an exponential family [28]) is $h\left[p_{m,s}^{a,b}\right]=-{\int }_a^b{p}_{m,s}^{a,b}\left(x\right)log{p}_{m,s}^{a,b}\mathrm{d}x=-F^{*}\left(\eta \right)=F\left(\theta \right)-{\theta }^{\top }\eta$. We find that

$$h\left[p_{m,s}^{a,b}\right]=log\left(\sqrt{2\pi e}s\left(\Phi \left(\beta \right)-\Phi \left(\alpha \right)\right)\right)+\frac{\alpha \varphi \left(\alpha \right)-\beta \varphi \left(\beta \right)}{2\left(\Phi \left(\beta \right)-\Phi \left(\alpha \right)\right)}.$$

(113)

When $(a,b)=(-\infty ,\infty )$, we have $\Phi \left(\beta \right)-\Phi \left(\alpha \right)=1$ and $\alpha \varphi \left(\alpha \right)-\beta \varphi \left(\beta \right)=0$ since $\beta =-\alpha$, $\varphi (-x)=\varphi \left(x\right)$ (an even function), and ${lim}_{\beta \to +\infty }\beta \varphi \left(\beta \right)=0$. Thus we recover the differential entropy of a normal distribution: $h\left[p_{\mu ,\sigma }\right]=log\left(\sqrt{2\pi e}\sigma \right)$.

5. Bhattacharyya Skewed Divergence between Truncated Densities of an Exponential Family

The Bhattacharyya $\alpha$-skewed divergence [29,30] between two densities $p\left(x\right)$ and $q\left(x\right)$ with respect to $\mu$ is defined for a skewing scalar parameter $\alpha \in (0,1)$ as:

$$D_{\mathrm{Bhat},\alpha }[p:q]:=-log{\int }_{\mathcal{X}}p{\left(x\right)}^{\alpha }q{\left(x\right)}^{1-\alpha }\mathrm{d}\mu \left(x\right),$$

(114)

where $\mathcal{X}$ denotes the support of the distributions. The Bhattacharyya distance is

$$D_{\mathrm{Bhat}}[p,q]=D_{\mathrm{Bhat},\frac{1}{2}}[p:q]=-log{\int }_{\mathcal{X}}\sqrt{p\left(x\right)q\left(x\right)}\mathrm{d}\mu \left(x\right).$$

(115)

The Bhattacharyya distance is not a metric distance since it does not satisfy the triangle inequality. The Bhattacharyya distance is related to the Hellinger distance [31] as follows:

$$D_H[p,q]=\sqrt{\frac{1}{2}{\int }_{\mathcal{X}}{\left(\sqrt{p\left(x\right)}-\sqrt{q\left(x\right)}\right)}^2\mathrm{d}\mu \left(x\right)}=\sqrt{1-exp(-D_{\mathrm{Bhat}}[p,q])}.$$

(116)

The Hellinger distance is a metric distance.

Let $I_{\alpha }[p:q]:={\int }_{\mathcal{X}}p{\left(x\right)}^{\alpha }q{\left(x\right)}^{1-\alpha }\mathrm{d}\mu \left(x\right)$ denote the skewed affinity coefficient so that $D_{\mathrm{Bhat},\alpha }[p:q]=-log{I}_{\alpha }[p:q]$. Since $I_{\alpha }[p:q]=I_{1-\alpha }[q:p]$, we have $D_{\mathrm{Bhat},\alpha }[p:q]=D_{\mathrm{Bhat},1-\alpha }[q:p]$.

Consider an exponential family $\mathcal{E}=\left\{p_{\theta }\right\}$ with log-normalizer $F\left(\theta \right)$. Then it is well-known that the $\alpha$-skewed Bhattacharyya divergence between two densities of an exponential family amounts to a skewed Jensen divergence [30] (originally called Jensen difference in [32]):

$$D_{\mathrm{Bhat},\alpha }[p_{{\theta }_1}:p_{{\theta }_2}]=J_{F,\alpha }({\theta }_1:{\theta }_2),$$

(117)

where the skewed Jensen divergence is defined by

$$J_{F,\alpha }({\theta }_1:{\theta }_2)=\alpha F\left({\theta }_1\right)+(1-\alpha )F\left({\theta }_2\right)-F(\alpha {\theta }_1+(1-\alpha ){\theta }_2).$$

(118)

The convexity of the log-normalizer $F\left(\theta \right)$ ensures that $J_{F,\alpha }({\theta }_1:{\theta }_2)\ge 0$. The Jensen divergence can be extended to full real $\alpha$ by rescaling it by $\frac{1}{\alpha (1-\alpha )}$, see [33].

Remark 1.

The Bhattacharyya skewed divergence $D_{\mathrm{Bhat},\alpha }[p:q]$ appears naturally as the negative of the log-normalizer of the exponential family induced by the exponential arc $\{r_{\alpha }\left(x\right)\alpha \in (0,1)\}$ linking two densities p and q with $r_{\alpha }\left(x\right)\propto p{\left(x\right)}^{\alpha }q{\left(x\right)}^{1-\alpha }$. This arc is an exponential family of order 1:

$$\begin{array}{ccc}\hfill r_{\alpha }\left(x\right)& =& exp\left(\alpha logp\left(x\right)+(1-\alpha )logq\left(x\right)-log{Z}_{\alpha }(p:q)\right),\hfill \end{array}$$

(119)

$$\begin{array}{ccc}& =& exp\left(\alpha log\frac{p\left(x\right)}{q\left(x\right)}-F_{pq}\left(\alpha \right)\right)q\left(x\right).\hfill \end{array}$$

(120)

The sufficient statistic is $t\left(x\right)=\frac{p\left(x\right)}{q\left(x\right)}$, the natural parameter $\alpha \in (0,1)$, and the log-normalizer $F_{pq}\left(\alpha \right)=log{Z}_{\alpha }(p:q)=log\int p{\left(x\right)}^{\alpha }q{\left(x\right)}^{1-\alpha }\mathrm{d}\mu \left(x\right)=-D_{\mathrm{Bhat},\alpha }[p:q]$. This shows that $D_{\mathrm{Bhat},\alpha }[p:q]$ is concave with respect to α since log-normalizers $F_{pq}\left(\alpha \right)$ are always convex. Grünwald called those exponential families the likelihood ratio exponential families [34].

Now, consider calculating $D_{\mathrm{Bhat},\alpha }[p_{{\theta }_1}:q_{{\theta }_2}]$ where $p_{{\theta }_1}\in {\mathcal{E}}_1$ with ${\mathcal{E}}_1$ a truncated exponential family of ${\mathcal{E}}_2$ and $q_{{\theta }_2}\in {\mathcal{E}}_2=\left\{q_{\theta }\right\}$. We have $q_{\theta }\left(x\right)=\frac{Z_1\left(\theta \right)}{Z_2\left(\theta \right)}{p}_{\theta }\left(x\right)$, where $Z_1\left(\theta \right)$ and $Z_2\left(\theta \right)$ are the partition functions of ${\mathcal{E}}_1$ and ${\mathcal{E}}_2$, respectively. Thus we have

$$I_{\alpha }[p_{{\theta }_1}:q_{{\theta }_2}]={\left(\frac{Z_1\left({\theta }_2\right)}{Z_2\left({\theta }_2\right)}\right)}^{1-\alpha }{I}_{\alpha }[p_{{\theta }_1}:p_{{\theta }_2}],$$

(121)

and the $\alpha$-skewed Bhattacharyya divergence is

$$D_{\mathrm{Bhat},\alpha }[p_{{\theta }_1}:q_{{\theta }_2}]=D_{\mathrm{Bhat},\alpha }[p_{{\theta }_1}:p_{{\theta }_2}]-(1-\alpha )(F_1\left({\theta }_2\right)-F_2\left({\theta }_2\right)).$$

(122)

Therefore we obtain

$$\begin{array}{ccc}\hfill D_{\mathrm{Bhat},\alpha }[p_{{\theta }_1}:q_{{\theta }_2}]& =& J_{F_1,\alpha }({\theta }_1:{\theta }_2)-(1-\alpha )(F_1\left({\theta }_2\right)-F_2\left({\theta }_2\right)),\hfill \end{array}$$

(123)

$$\begin{array}{ccc}& =& \alpha F_1\left({\theta }_1\right)+(1-\alpha )F_2\left({\theta }_2\right)-F_1(\alpha {\theta }_1+(1-\alpha ){\theta }_2),\hfill \end{array}$$

(124)

$$\begin{array}{ccc}& =:& J_{F_1,F_2,\alpha }({\theta }_1:{\theta }_2).\hfill \end{array}$$

(125)

We call $J_{F_1,F_2,\alpha }({\theta }_1:{\theta }_2)$ the duo Jensen divergence. Since $F_2\left(\theta \right)\ge F_1\left(\theta \right)$, we check that

$$J_{F_1,F_2,\alpha }({\theta }_1:{\theta }_2)\ge J_{F_1,\alpha }({\theta }_1:{\theta }_2)\ge 0.$$

(126)

Figure 7 illustrates graphically the duo Jensen divergence $J_{F_1,F_2,\alpha }({\theta }_1:{\theta }_2)$.

Figure 7. The duo Jensen divergence $J_{F_1,F_2,\alpha }({\theta }_1:{\theta }_2)$ is greater than the Jensen divergence $J_{F_1,\alpha }({\theta }_1:{\theta }_2)$ for $F_2\left(\theta \right)\ge F_1\left(\theta \right)$.

Theorem 2.

The α-skewed Bhattacharyya divergence for $\alpha \in (0,1)$ between a truncated density of ${\mathcal{E}}_1$ with log-normalizer $F_1\left(\theta \right)$ and another density of an exponential family ${\mathcal{E}}_2$ with log-normalizer $F_2\left(\theta \right)$ amounts to a duo Jensen divergence:

$$D_{\mathrm{Bhat},\alpha }[p_{{\theta }_1}:q_{{\theta }_2}]=J_{F_1,F_2,\alpha }({\theta }_1:{\theta }_2),$$

(127)

where $J_{F_1,F_2,\alpha }({\theta }_1:{\theta }_2)$ is the duo skewed Jensen divergence induced by two strictly convex functions $F_1\left(\theta \right)$ and $F_2\left(\theta \right)$ such that $F_2\left(\theta \right)\ge F_1\left(\theta \right)$:

$$J_{F_1,F_2,\alpha }({\theta }_1:{\theta }_2)=\alpha F_1\left({\theta }_1\right)+(1-\alpha )F_2\left({\theta }_2\right)-F_1(\alpha {\theta }_1+(1-\alpha ){\theta }_2).$$

(128)

In [30], it is reported that

$$\begin{array}{ccc}\hfill D_{\mathrm{KL}}[p_{{\theta }_1}:p_{{\theta }_2}]& =& B_F({\theta }_2:{\theta }_1),\hfill \end{array}$$

(129)

$$\begin{array}{ccc}& =& \underset{\alpha \to 0}{lim}\frac{1}{\alpha }{J}_{F,\alpha }({\theta }_2:{\theta }_1)=\underset{\alpha \to 0}{lim}\frac{1}{\alpha }{J}_{F,1-\alpha }({\theta }_1:{\theta }_2),\hfill \end{array}$$

(130)

$$\begin{array}{ccc}& =& \underset{\alpha \to 0}{lim}\frac{1}{\alpha }{D}_{\mathrm{Bhat},\alpha }[p_{{\theta }_2}:p_{{\theta }_1}]=\underset{\alpha \to 0}{lim}\frac{1}{\alpha }{D}_{\mathrm{Bhat},1-\alpha }[p_{{\theta }_1}:p_{{\theta }_2}].\hfill \end{array}$$

(131)

Indeed, using the first-order Taylor expansion of

$$F({\theta }_1+\alpha ({\theta }_2-{\theta }_1))\underset{\alpha \to 0}{\approx }F\left({\theta }_1\right)+\alpha {({\theta }_2-{\theta }_1)}^{\top }\nabla F\left({\theta }_1\right)$$

(132)

when $\alpha \to 0$, we check that we have

$$\begin{array}{ccc}\hfill \frac{1}{\alpha }{J}_{F,\alpha }({\theta }_2:{\theta }_1)& :=& \frac{F\left({\theta }_1\right)+\alpha (F\left({\theta }_2\right)-F\left({\theta }_1\right))-F({\theta }_1+\alpha ({\theta }_2-{\theta }_1))}{\alpha },\hfill \end{array}$$

(133)

$$\begin{array}{ccc}& \underset{\alpha \to 0}{\overset{Equation\left(132\right)}{\approx }}& \frac{\bcancel{\cancel{F\left({\theta }_1\right)}}+\alpha (F\left({\theta }_2\right)-F\left({\theta }_1\right))-\bcancel{\cancel{F\left({\theta }_1\right)}}-\alpha {({\theta }_2-{\theta }_1)}^{\top }\nabla F\left({\theta }_1\right)}{\alpha },\hfill \end{array}$$

(134)

$$\begin{array}{ccc}& =& F\left({\theta }_2\right)-F\left({\theta }_1\right)-{({\theta }_2-{\theta }_1)}^{\top }\nabla F\left({\theta }_1\right),\hfill \end{array}$$

(135)

$$\begin{array}{ccc}& =:& B_F({\theta }_2:{\theta }_1).\hfill \end{array}$$

(136)

Thus we have ${lim}_{\alpha \to 0}\frac{1}{\alpha }{J}_{F,\alpha }({\theta }_2:{\theta }_1)=B_F({\theta }_2:{\theta }_1)$.

Moreover, we have

$$\underset{\alpha \to 0}{lim}\frac{1}{\alpha }{D}_{\mathrm{Bhat},1-\alpha }[p:q]=D_{\mathrm{KL}}[p:q].$$

(137)

Similarly, we can prove that

$$\underset{\alpha \to 1}{lim}\frac{1}{1-\alpha }{J}_{F_1,F_2,\alpha }({\theta }_1:{\theta }_2)=B_{F_2,F_1}({\theta }_2:{\theta }_1),$$

(138)

which can be reinterpreted as

$$\underset{\alpha \to 1}{lim}\frac{1}{1-\alpha }{D}_{\mathrm{Bhat},\alpha }[p_{{\theta }_1}:q_{{\theta }_2}]=D_{\mathrm{KL}}[p_{{\theta }_1}:q_{{\theta }_2}].$$

(139)

6. Concluding Remarks

We considered the Kullback–Leibler divergence between two parametric densities $p_{\theta }\in {\mathcal{E}}_1$ and $q_{{\theta }^{\prime }}\in {\mathcal{E}}_2$ belonging to truncated exponential families [7] ${\mathcal{E}}_1$ and ${\mathcal{E}}_2$, and we showed that their KLD is equivalent to a duo Bregman divergence on swapped parameter order (Theorem 1). This result generalizes the study of Azoury and Warmuth [13]. The duo Bregman divergence can be rewritten as a duo Fenchel–Young divergence using mixed natural/moment parameterizations of the exponential family densities (Definition 1). This second result generalizes the approach taken in information geometry [15,35]. We showed how to calculate the Kullback–Leibler divergence between two truncated normal distributions as a duo Bregman divergence. More generally, we proved that the skewed Bhattacharyya distance between two parametric densities of truncated exponential families amounts to a duo Jensen divergence (Theorem 2). We showed asymptotically that scaled duo Jensen divergences tend to duo Bregman divergences generalizing a result of [30,33]. This study of duo divergences induced by pair of generators was motivated by the formula obtained for the Kullback–Leibler divergence between two densities of two different exponential families originally reported in [23] (Equation (29)).

It is interesting to find applications of the duo Fenchel–Young, Bregman, and Jensen divergences beyond the scope of calculating statistical distances between truncated exponential family densities. Note that in [36], the authors exhibit a relationship between densities with nested supports and quasi-convex Bregman divergences. However, those considered parametric densities are not exponential families since their supports depend on the parameter. Recently, Khan and Swaroop [37] used this duo Fenchel–Young divergence in machine learning for knowledge-adaptation priors in the so-called change regularizer task.