On a Variational Definition for the Jensen-Shannon Symmetrization of Distances Based on the Information Radius

Abstract

We generalize the Jensen-Shannon divergence and the Jensen-Shannon diversity index by considering a variational definition with respect to a generic mean, thereby extending the notion of Sibson’s information radius. The variational definition applies to any arbitrary distance and yields a new way to define a Jensen-Shannon symmetrization of distances. When the variational optimization is further constrained to belong to prescribed families of probability measures, we get relative Jensen-Shannon divergences and their equivalent Jensen-Shannon symmetrizations of distances that generalize the concept of information projections. Finally, we touch upon applications of these variational Jensen-Shannon divergences and diversity indices to clustering and quantization tasks of probability measures, including statistical mixtures.

1. Introduction: Background and Motivations

The goal of the author is to methodologically contribute to an extension of the Sibson’s information radius [1] and also concentrate on analysis of the specified families of distributions called exponential families [2].

Let $(\mathcal{X},\mathcal{F})$ denote a measurable space [3] with sample space $\mathcal{X}$ and $\sigma$-algebra $\mathcal{F}$ on the set $\mathcal{X}$. The Jensen-Shannon divergence [4] (JSD) between two probability measures P and Q (or probability distributions) on $(\mathcal{X},\mathcal{F})$ is defined by:

$$D_{\mathrm{JS}}[P,Q]:=\frac{1}{2}\left(D_{\mathrm{KL}}\left[P:\frac{P+Q}{2}\right]+D_{\mathrm{KL}}\left[Q:\frac{P+Q}{2}\right]\right),$$

(1)

where $D_{\mathrm{KL}}$ denotes the Kullback–Leibler divergence [5,6] (KLD):

$$D_{\mathrm{KL}}[P:Q]:=\left\{\begin{array}{cc}{\int }_{\mathcal{X}}log\left(\frac{\mathrm{d}P\left(x\right)}{\mathrm{d}Q\left(x\right)}\right)\mathrm{d}P,\hfill & P\ll Q\hfill \\ +\infty ,\hfill & P\ll Q\hfill \end{array}\right.$$

(2)

where $P\ll Q$ means that P is absolutely continuous with respect to Q [3], and $\frac{\mathrm{d}P}{\mathrm{d}Q}$ is the Radon–Nikodym derivative of P with respect to Q. Equation (2) can be rewritten using the chain rule as:

$$D_{\mathrm{KL}}[P:Q]:=\left\{\begin{array}{cc}{\int }_{\mathcal{X}}\frac{\mathrm{d}P\left(x\right)}{\mathrm{d}Q\left(x\right)}log\left(\frac{\mathrm{d}P\left(x\right)}{\mathrm{d}Q\left(x\right)}\right)\mathrm{d}Q,\hfill & P\ll Q\hfill \\ +\infty ,\hfill & P\ll Q\hfill \end{array}\right.$$

(3)

Consider a measure $\mu$ for which both the Radon–Nikodym derivatives $p:=\frac{\mathrm{d}P}{\mathrm{d}\mu }$ and $q:=\frac{\mathrm{d}P}{\mathrm{d}\mu }$ exist (e.g., $\mu =\frac{P+Q}{2}$). Subsequently the Kullback–Leibler divergence can be rewritten as (see Equation (2.5) page 5 of [5] and page 251 of the Cover & Thomas’ textbook [6]):

$$D_{\mathrm{KL}}[p:q]:={\int }_{\mathcal{X}}p\left(x\right)log\left(\frac{p\left(x\right)}{q\left(x\right)}\right)\mathrm{d}\mu \left(x\right).$$

(4)

Denote by $\mathcal{D}=\mathcal{D}\left(\mathcal{X}\right)$ the set of all densities with full support $\mathcal{X}$ (Radon–Nikodym derivatives of probability measures with respect to $\mu$):

$$\mathcal{D}\left(\mathcal{X}\right):=\left\{p:\mathcal{X}\to \mathbb{R}:p\left(x\right)>0\mu -\mathrm{a}\mathrm{l}\mathrm{m}\mathrm{o}\mathrm{s}\mathrm{t} \mathrm{e}\mathrm{v}\mathrm{e}\mathrm{r}\mathrm{y}\mathrm{w}\mathrm{h}\mathrm{e}\mathrm{r}\mathrm{e},{\int }_{\mathcal{X}}p\left(x\right)\mathrm{d}\mu \left(x\right)=1\right\}.$$

Subsequently, the Jensen-Shannon divergence [4] between two densities p and q of $\mathcal{D}$ is defined by:

$$D_{\mathrm{JS}}[p,q]:=\frac{1}{2}\left(D_{\mathrm{KL}}\left[p:\frac{p+q}{2}\right]+D_{\mathrm{KL}}\left[q:\frac{p+q}{2}\right]\right).$$

(5)

Often, one considers the Lebesgue measure [3] $\mu ={\mu }_{\mathcal{L}}$ on $({\mathbb{R}}^d,\mathcal{B}\left({\mathbb{R}}^d\right))$, where $\mathcal{B}\left({\mathbb{R}}^d\right)$ is the Borel $\sigma$-algebra, or the counting measure [3]$\mu ={\mu }_{\#}$ on $(\mathcal{X},2^{\mathcal{X}})$ where $\mathcal{X}$ is a countable set, for defining the measure space $(\mathcal{X},\mathcal{F},\mu )$.

The JSD belongs to the class of f-divergences [7,8,9] which are known as the invariant decomposable divergences of information geometry (see [10], pp. 52–57). Although the KLD is asymmetric (i.e., $D_{\mathrm{KL}}[p:q]\ne D_{\mathrm{KL}}[q:p]$), the JSD is symmetric (i.e., $D_{\mathrm{JS}}[p,q]=D_{\mathrm{JS}}[q,p]$). The notation ‘:’ is used as a parameter separator to indicate that the parameters are not permutation invariant, and that the order of parameters is important.

In this work, a distance $D(O_1:O_2)$ is a measure of dissimilarity between two objects $O_1$ and $O_2$, which do not need to be symmetric or satisfy the triangle inequality of metric distances. A distance only satisfies the identity of indiscernibles: $D(O_1:O_2)=0$ if and only if $O_1=O_2$. When the objects $O_1$ and $O_2$ are probability densities with respect to $\mu$, we call this distance a statistical distance, use the brackets to enclose the arguments of the statistical distance (i.e., $D[O_1:O_2]$), and we have $D[O_1:O_2]=0$ if and only if $O_1\left(x\right)=O_2\left(x\right)$$\mu$-almost everywhere.

The 2-point JSD of Equation (4) can be extended to a weighted set of n densities $\mathcal{P}:=\{(w_1,p_1),\dots ,(w_n,p_n)\}$ (with positive $w_i$’s normalized to sum up to unity, i.e., ${\sum }_{i=1}^n{w}_i=1$) thus providing a diversity index, i.e., a n-point JSD for $\mathcal{P}$:

$$D_{\mathrm{JS}}\left(\mathcal{P}\right):=\sum _{i=1}^n{w}_i{D}_{\mathrm{KL}}\left[p_i:\overline{p}\right],$$

(6)

where $\overline{p}:={\sum }_{i=1}^n{w}_i{p}_i$ denotes the statistical mixture [11] of the densities of $\mathcal{P}$. We have $D_{\mathrm{JS}}[p:q]=D_{\mathrm{JS}}\left(\{(\frac{1}{2},p),(\frac{1}{2},q)\}\right)$. We call $D_{\mathrm{JS}}\left(\mathcal{P}\right)$ the Jensen-Shannon diversity index.

The KLD is also called the relative entropy since it can be expressed as the difference between the cross entropy $h[p:q]$ and the entropy $h\left[p\right]$:

$$\begin{array}{ccc}\hfill D_{\mathrm{KL}}[p:q]& :=& {\int }_{\mathcal{X}}p\left(x\right)log\left(\frac{p\left(x\right)}{q\left(x\right)}\right)\mathrm{d}\mu \left(x\right)\hfill \end{array}$$

(7)

$$\begin{array}{ccc}& =& {\int }_{\mathcal{X}}p\left(x\right)logp\left(x\right)\mathrm{d}\mu \left(x\right)-{\int }_{\mathcal{X}}p\left(x\right)logq\left(x\right)\mathrm{d}\mu \left(x\right),\hfill \end{array}$$

(8)

$$\begin{array}{ccc}& =& h[p:q]-h\left[p\right],\hfill \end{array}$$

(9)

with the cross-entropy and entropy defined, respectively, by

$$\begin{array}{ccc}\hfill h[p:q]& :=& -{\int }_{\mathcal{X}}p\left(x\right)logq\left(x\right)\mathrm{d}\mu \left(x\right),\hfill \end{array}$$

(10)

$$\begin{array}{ccc}\hfill h\left[p\right]& :=& -{\int }_{\mathcal{X}}p\left(x\right)logp\left(x\right)\mathrm{d}\mu \left(x\right).\hfill \end{array}$$

(11)

Because $h\left[p\right]=h[p:p]$, we may say that the entropy is the self-cross-entropy.

When $\mu$ is the Lebesgue measure, the Shannon entropy is also called the differential entropy [6]. Although the discrete entropy $H\left[p\right]=-{\sum }_i{p}_{i}log{p}_i$ (i.e., entropy with respect to the counting measure) is always positive and bounded by $log\left|\mathcal{X}\right|$, the differential entropy may be negative (e.g., entropy of a Gaussian distribution with small variance).

The Jensen-Shannon divergence of Equation (6) can be rewritten as:

$$D_{\mathrm{JS}}[p,q]=h\left[\overline{p}\right]-\sum _{i=1}^n{w}_{i}h\left[p_i\right]:=J_{-h}[p,q].$$

(12)

The JSD representation of Equation (12) is a Jensen divergence [12] for the strictly convex negentropy $F\left(p\right)=-h\left[p\right]$, since the entropy function $h[.]$ is strictly concave. Therefore, it is appropriate to call this divergence the Jensen-Shannon divergence.

Because $\frac{p_i\left(x\right)}{\overline{p}\left(x\right)}\le \frac{p_i\left(x\right)}{w_i{p}_i\left(x\right)}=\frac{1}{w_i}$, it can be shown that the Jensen-Shannon diversity index is upper bounded by $H\left(w\right):=-{\sum }_{i=1}^n{w}_{i}log{w}_i$, the discrete Shannon entropy. Thus, the Jensen-Shannon diversity index is bounded by $logn$, and the 2-point JSD is bounded by $log2$, although the KLD is unbounded and it may even be equal to $+\infty$ when the definite integral diverges (e.g., KLD between the standard Cauchy distribution and the standard Gaussian distribution). Another nice property of the JSD is that its square root yields a metric distance [13,14]. This property further holds for the quantum JSD [15]. The JSD has gained interest in machine learning. See, for example, the Generative Adversarial Networks [16] (GANs) in deep learning [17], where it was proven that minimizing the GAN objective function by adversarial training is equivalent to minimizing a JSD.

To delineate the different roles that are played by the factor $\frac{1}{2}$ in the ordinary Jensen-Shannon divergence (i.e., in weighting the two KLDs and in weighting the two densities), let us introduce two scalars $\alpha ,\beta \in (0,1)$, and define a generic $(\alpha ,\beta )$-skewed Jensen-Shannon divergence, as follows:

$$\begin{array}{ccc}\hfill D_{\mathrm{JS},\alpha ,\beta }[p:q]& :=& (1-\beta )D_{\mathrm{KL}}[p:m_{\alpha }]+\beta D_{\mathrm{KL}}[q:m_{\alpha }],\hfill \end{array}$$

(13)

$$\begin{array}{ccc}& =& (1-\beta )h[p:m_{\alpha }]+\beta h[q:m_{\alpha }]-(1-\beta )h\left[p\right]-\beta h\left[q\right],\hfill \end{array}$$

(14)

$$\begin{array}{ccc}& =& h[m_{\beta }:m_{\alpha }]-\left((1-\beta )h\left[p\right]+\beta h\left[q\right]\right),\hfill \end{array}$$

(15)

where $m_{\alpha }:=(1-\alpha )p+\alpha q$ and $m_{\beta }:=(1-\beta )p+\beta q$. This identity holds, because $D_{\mathrm{JS},\alpha ,\beta }$ is bounded by $(1-\beta )log\frac{1}{1-\alpha }+\beta log\frac{1}{\alpha }$, see [18]. Thus, when $\beta =\alpha$, we have $D_{\mathrm{JS},\alpha }[p,q]=D_{\mathrm{JS},\alpha ,\alpha }[p,q]=h\left[m_{\alpha }\right]-((1-\alpha )h\left[p\right]+\alpha h\left[q\right])$, since the self-cross entropy corresponds to the entropy: $h[m_{\alpha }:m_{\alpha }]=h\left[m_{\alpha }\right]$.

A f-divergence [9,19,20] is defined for a convex generator f, which is strictly convex at 1 (to satisfy the identity of the indiscernibles) and that satisfies $f\left(1\right)=0$, by

$$I_f[p:q]:=\int p\left(x\right)f\left(\frac{q\left(x\right)}{p\left(x\right)}\right)\mathrm{d}\mu \left(x\right)\ge f\left(1\right)=0,$$

(16)

where the right-hand-side follows from Jensen’s inequality [20]. For example, the total variation distance $D_{\mathrm{TV}}[p:q]=\frac{1}{2}{\int }_{\mathcal{X}}|p\left(x\right)-q\left(x\right)|\mathrm{d}\mu \left(x\right)$ is a f-divergence for the generator $f_{\mathrm{TV}}\left(u\right)=|u-1|$: $D_{\mathrm{TV}}[p:q]=I_{f_{\mathrm{TV}}}[p:q]$. The generator $f_{\mathrm{TV}}\left(u\right)$ is convex on $\mathbb{R}$, strictly convex at 1, and it satisfies $f\left(u\right)=1$.

The $D_{\mathrm{JS},\alpha ,\beta }$ divergence is a f-divergence

$$D_{\mathrm{JS},\alpha ,\beta }[p:q]=I_{f_{\mathrm{JS},\alpha ,\beta }}[p:q],$$

(17)

for the generator:

$$f_{\mathrm{JS},\alpha ,\beta }\left(u\right)=-\left((1-\beta )log\left(\alpha u+(1-\alpha )\right)+\beta ulog\left(\frac{1-\alpha }{u}+\alpha \right)\right).$$

(18)

We check that the generator $f_{\mathrm{JS},\alpha ,\beta }$ is strictly convex, since, for any $a\in (0,1)$ and $b\in (0,1)$, we have

$$f_{\mathrm{JS},\alpha ,\beta }^{″}\left(u\right)=\frac{a^2(1-b)u+{(a-1)}^{2}b}{a^2{u}^3+2a(1-a)u^2+{(a-1)}^{2}u}>0,$$

(19)

when $u>0$.

The Jensen-Shannon principle of taking the average of the (Kullback–Leibler) divergences between the source parameters to the mid-parameter can be applied to other distances. For example, the Jensen–Bregman divergence is a Jensen-Shannon symmetrization of the Bregman divergence $B_F$ [12]:

$$B_F^{\mathrm{JS}}({\theta }_1:{\theta }_2):=\frac{1}{2}\left(B_F\left({\theta }_1:\frac{{\theta }_1+{\theta }_2}{2}\right)+B_F\left({\theta }_2:\frac{{\theta }_1+{\theta }_2}{2}\right)\right),$$

(20)

where the Bregman divergence [21] $B_F$ is defined by

$$B_F(\theta :{\theta }^{\prime }):=F\left(\theta \right)-F\left({\theta }^{\prime }\right)-{(\theta -{\theta }^{\prime })}^{\top }\nabla F\left({\theta }^{\prime }\right).$$

(21)

The Jensen–Bregman divergence $B_F^{\mathrm{JS}}$ can also be written as an equivalent Jensen divergence $J_F$:

$$B_F^{\mathrm{JS}}({\theta }_1:{\theta }_2)=J_F({\theta }_1:{\theta }_2):=\frac{F\left({\theta }_1\right)+F\left({\theta }_2\right)}{2}-F\left(\frac{{\theta }_1+{\theta }_2}{2}\right),$$

(22)

where F is a strictly convex function ensuring $J_F({\theta }_1:{\theta }_2)\ge 0$ with equality if ${\theta }_1={\theta }_2$.

Because of its use in various fields of information sciences [22], various generalizations of the JSD have been proposed: These generalizations are either based on Equation (5) [23] or Equation (12) [18,24,25]. For example, the (arithmetic) mixture $\overline{p}={\sum }_i{w}_i{p}_i$ in Equation (6) was replaced by an abstract statistical mixture with respect to a generic mean M in [23] (e.g., the geometric mixture induced by the geometric mean), and the two KLDS defining the JSD in Equation (5) was further averaged using another abstract mean N, thus yielding the following generic $(M,N)$-Jensen-Shannon divergence [23] (abbreviated as $(M,N)$-JSD):

$$D_{\mathrm{JS}}^{M,N}[p:q]:=N\left(D_{\mathrm{KL}}\left[p:{\left(pq\right)}_{\frac{1}{2}}^M\right],D_{\mathrm{KL}}\left[q:{\left(pq\right)}_{\frac{1}{2}}^M\right]\right),$$

(23)

where ${\left(pq\right)}_{\alpha }^M$ denotes the statistical weighted M-mixture:

$${\left(pq\right)}_{\alpha }^M:=\frac{M_{\alpha }(p\left(x\right),q\left(x\right))}{{\int }_{\mathcal{X}}{M}_{\alpha }(p\left(x\right),q\left(x\right))\mathrm{d}\mu \left(x\right)}.$$

(24)

Notice that, when $M=N=A$ (the arithmetic mean), Equation (23) of the $(A,A)$-JSD reduces to the ordinary JSD of Equation (5). When the means M and N are symmetric, the $(M,N)$-JSD is symmetric.

In general, a weighted mean $M_{\alpha }(a,b)$ for any $\alpha \in [0,1]$ shall satisfy the in-betweeness property [26] (i.e., a mean should be contained inside its extrema):

$$min\{a,b\}\le M_{\alpha }(a,b)\le max\{a,b\}.$$

(25)

The three Pythagorean means defined for positive scalars $a>0$ and $b>0$ are classic examples of means:

• The arithmetic mean $A(a,b)=\frac{a+b}{2}$,
• the geometric mean $G(a,b)=\sqrt{ab}$, and
• the harmonic mean $H(a,b)=\frac{2ab}{a+b}$.

These Pythagorean means may be interpreted as special instances of another parametric family of means: The power means

$$P_{\alpha }(a,b):={\left(\frac{a^{\alpha }+b^{\alpha }}{2}\right)}^{\frac{1}{\alpha }},$$

(26)

defined for $\alpha \in \mathbb{R}\backslash \left\{0\right\}$ (also called Hölder means). The power means can be extended to the full range $\alpha \in \mathbb{R}$ by using the property that ${lim}_{\alpha \to 0}{P}_{\alpha }(a,b)=G(a,b)$. The power means are homogeneous means: $P_{\alpha }(\lambda a,\lambda b)=\lambda P_{\alpha }(a,b)$ for any $\lambda >0$. We refer to the handbook of means [27] to obtain definitions and principles of other means beyond these power means.

A weighted mean (also called barycenter) can be built from a non-weighted mean $M(a,b)$ (i.e., $\alpha =\frac{1}{2}$) by using the dyadic expansion of the real weight $\alpha \in [0,1]$, see [28]. That is, we can define the weighted mean $M(p,q;w,1-w)$ for $w=\frac{i}{2^k}$ with $i\in \{0,\dots ,2^k\}$ and k an integer. For example, consider a symmetric mean $M(p,q)=M(q,p)$. Subsequently, we get the following weighted means when $k=3$:

$$\begin{array}{ccc}\hfill M\left(p,q;\frac{0}{8}=0,\frac{8}{8}=1\right)& =& q\hfill \\ \hfill M\left(p,q;\frac{1}{8},\frac{7}{8}\right)& =& M\left(M\right(M(p,q),q),q)\hfill \\ \hfill M\left(p,q;\frac{2}{8}=\frac{1}{4},\frac{6}{8}=\frac{3}{4}\right)& =& M\left(M\right(p,q),q)\hfill \\ \hfill M\left(p,q;\frac{3}{8},\frac{5}{8}\right)& =& M\left(M\right(M(p,q),p),q)\hfill \\ \hfill M\left(p,q;\frac{4}{8}=\frac{1}{2},\frac{4}{8}=\frac{1}{2}\right)& =& M(p,q)\hfill \\ \hfill M\left(p,q;\frac{5}{8},\frac{3}{8}\right)& =& M\left(M\right(M(p,q),q),p)\hfill \\ \hfill M\left(p,q;\frac{6}{8}=\frac{3}{4},\frac{2}{8}=\frac{1}{4}\right)& =& M\left(M\right(p,q),p)\hfill \\ \hfill M\left(p,q;\frac{7}{8},\frac{1}{8}\right)& =& M\left(M\right(M(p,q),p),p)\hfill \\ \hfill M\left(p,q;\frac{8}{8}=1,\frac{0}{8}=0\right)& =& p\hfill \end{array}$$

Let $w={\sum }_{i=1}^{\infty }\frac{d_i}{2^i}$ be the unique dyadic expansion of the real number $w\in (0,1)$, where the $d_i$’s are binary digits (i.e., $d_i\in \{0,1\}$). We define the weighted mean $M(x,y;w,1-w)$ of two positive reals p and q for a real weight $w\in (0,1)$ as

$$M(x,y;w,1-w):=\underset{n\to \infty }{lim}M\left(x,y;\sum _{i=1}^n\frac{d_i}{2^i},1-\sum _{i=1}^n\frac{d_i}{2^i}\right).$$

(27)

Choosing the abstract mean M in accordance with the family $\mathcal{R}=\{p_{\theta }:\theta \in \Theta \}$ of the densities allows one to obtain closed-form formula for the $(M,N)$-JSDs that rely on definite integral calculations [23]. For example, the JSD between two Gaussian densities does not admit a closed-form formula because of the log-sum integral, but the $(G,N)$-JSD admits a closed-form formula when using geometric statistical mixtures (i.e., when $M=G$). The calculus trick is to find a weighted mean $M_{\alpha }$, such that, for two densities $p_{{\theta }_1}$ and $p_{{\theta }_2}$, the weighted mean distribution $M_{\alpha }(p_{{\theta }_1}\left(x\right),p_{{\theta }_2}\left(x\right))=\frac{p_{{\theta }_{1,2,\alpha }}\left(x\right)}{Z_{M_{\alpha }}({\theta }_1,{\theta }_2)}$, where $Z_{M_{\alpha }}({\theta }_1,{\theta }_2)$ is the normalizing coefficient and $p_{{\theta }_{1,2,\alpha }}\in \mathcal{R}$. Thus, the integral calculation can be simply calculated as $\int M_{\alpha }(p_{{\theta }_1}\left(x\right),p_{{\theta }_2}\left(x\right))\mathrm{d}\mu \left(x\right)=\frac{1}{Z_{M_{\alpha }}({\theta }_1,{\theta }_2)}$ since $p_{{\theta }_{1,2,\alpha }}\left(x\right)$, and, therefore, $\int p_{{\theta }_{1,2,\alpha }}\left(x\right)\mathrm{d}\mu \left(x\right)=1$. This trick has also been used in Bayesian hypothesis testing for upper bounding the probability of error between two densities of a parametric family of distributions by replacing the usual geometric mean (Section 11.7 of [6], page 375) by a more general quasi-arithmetic mean [29]. For example, the harmonic mean is well-suited to Cauchy distributions, and the power means to Student t-distributions [29].

As an application of these generalized JSDs, Deasy et al. [30] used the skewed geometric JSD (namely, the $(G_{\alpha },A_{1-\alpha })$-JSD for $\alpha \in (0,1)$), which admits a closed-form formula between normal densities [23], and showed how regularizing an optimization task with this G-JSD divergence improved reconstruction and generation of Variational AutoEncoders (VAEs).

More generally, instead of using the KLD, one can also use any arbitrary distance D to define its JS-symmetrization, as follows:

$$D_{M,N}^{\mathrm{JS}}[p:q]:=N\left(D\left[p:{\left(pq\right)}_{\frac{1}{2}}^M\right],D\left[q:{\left(pq\right)}_{\frac{1}{2}}^M\right]\right).$$

(28)

These symmetrizations may further be skewed by using $M_{\alpha }$ and/or $N_{\beta }$ for $\alpha \in (0,1)$ and $\beta \in (0,1)$, yielding the definition [23]:

$$D_{M_{\alpha },N_{\beta }}^{\mathrm{JS}}[p:q]:=N_{\beta }\left(D\left[p:{\left(pq\right)}_{\alpha }^M\right],D\left[q:{\left(pq\right)}_{\alpha }^M\right]\right).$$

(29)

With these notations, the ordinary JSD is $D_{\mathrm{JS}}={D_{\mathrm{KL}}}_{A,A}^{\mathrm{JS}}$, the $(A,A)$ JS-symmetrization of the KLD with respect to the arithmetic means $M=A$ and $N=A$.

The JS-symmetrization can be interpreted as the $N_{\beta }$-Jeffreys’ symmetrization of a generalization of Lin’s $\alpha$-skewed K-divergence [4]$D_{M_{\alpha }}^K[p:q]$:

$$\begin{array}{ccc}\hfill D_{M_{\alpha },N_{\beta }}^{\mathrm{JS}}[p:q]& =& N_{\beta }(D_{M_{\alpha }}^K[p:q],D_{M_{\alpha }}^K[p:q]),\hfill \end{array}$$

(30)

$$\begin{array}{ccc}\hfill D_{M_{\alpha }}^K[p:q]& :=& D\left[p:{\left(pq\right)}_{\alpha }^{M_{\alpha }}\right].\hfill \end{array}$$

(31)

In this work, we consider symmetrizing an arbitrary distance D (including the KLD), generalizing the Jensen-Shannon divergence by using a variational formula for the JSD. Namely, we observe that the Jensen-Shannon divergence can also be defined as the following minimization problem:

$$D_{\mathrm{JS}}[p,q]:=\underset{c\in \mathcal{D}}{min}\frac{1}{2}\left(D_{\mathrm{KL}}[p:c]+D_{\mathrm{KL}}[q:c]\right),$$

(32)

since the optimal density c is proven unique using the calculus of variation [1,31,32] and it corresponds to the mid density $\frac{p+q}{2}$, a statistical (arithmetic) mixture.

Proof. 

Let $S\left(c\right)=D_{\mathrm{KL}}[p:c]+D_{\mathrm{KL}}[q:c]\ge 0$. We use the method of the Lagrange multipliers for the constrained optimization problem ${min}_{c}S\left(c\right)$ such that $\int c\left(x\right)\mathrm{d}\mu \left(x\right)=1$. Let us minimize $S\left(c\right)+\lambda \left(\int c\left(x\right)\mathrm{d}\mu \left(x\right)-1\right)$. The density c realizing the minimum $S\left(c\right)$ satisfies the Euler–Lagrange equation $\frac{\partial L}{\partial c}=0$, where $L\left(c\right):=plog\frac{p}{c}+qlog\frac{q}{c}+\lambda c$ is the Lagrangian. That is, $-\frac{p}{c}-\frac{q}{c}+\lambda =0$ or, equivalently, $c=\frac{1}{\lambda }(p+q)$. Parameter $\lambda$ is then evaluated from the constraint ${\int }_{\mathcal{X}}c\left(x\right)\mathrm{d}\mu \left(x\right)=1$: we get $\lambda =2$ since ${\int }_{\mathcal{X}}(p\left(x\right)+q\left(x\right))\mathrm{d}\mu \left(x\right)=2$. Therefore, we find that $c\left(x\right)=\frac{p\left(x\right)+q\left(x\right)}{2}$, the mid density of $p\left(x\right)$ and $q\left(x\right)$. □

Considering Equation (32) instead of Equation (5) for defining the Jensen-Shannon divergence is interesting, because it allows one to consider a novel approach for generalizing the Jensen-Shannon divergence. This variational approach was first considered by Sibson [1] to define the $\alpha$-information radius of a set of weighted distributions while using Rényi $\alpha$-entropies that are based on Rényi principled $\alpha$-means [33]. The $\alpha$-information radius includes the Jensen-Shannon diversity index when $\alpha =1$. Sibson’s work is our point of departure for generalizing the Jensen-Shannon divergence and proposing the Jensen-Shannon symmetrizations of arbitrary distances.

The paper is organized, as follows: in Section 2, we recall the rationale and definitions of the Rényi $\alpha$-entropy and the Rényi $\alpha$-divergence [33], and explain the information radius of Sibson [1], which includes, as a special case, the ordinary Jensen-Shannon divergence and that can be interpreted as generalized skew Bhattacharyya distances. We report, in Theorem 2, a closed-form formula for calculating the information radius of order $\alpha$ between two densities of an exponential family when $\frac{1}{\alpha }$ is an integer. It is noteworthy to point out that Sibson’s work (1969) includes, as a particular case of the information radius, a definition of the JSD, prior to the well-known reference paper of Lin [4] (1991). In Section 3, we present the JS-symmetrization variational definition that is based on a generalization of the information radius with a generic mean (Equation (88) and Definition 3). In Section 4, we constrain the mixture density to belong to a prescribed class of (parametric) probability densities, like an exponential family [2], and obtain a relative information radius generalizing information radius and related to the concept of information projections. Our Definition 5 generalizes the (relative) normal information radius of Sibson [1], who considered the multivariate normal family (Proposition 4). We illustrate this notion of relative information radius by calculating the density of an exponential family minimizing the reverse Kullback–Leibler divergence between a mixture of densities of that exponential family (Proposition 6). Moreover, we get a semi-closed-form formula for the Kullback–Leibler divergence between the densities of two different exponential families (Proposition 5), generalizing the Fenchel–Young divergence [34]. As an application of these relative variational JSDs, we touch upon the problems of clustering and quantization of probability densities in Section 4.2. Finally, we conclude by summarizing our contributions and discussing related works in Section 5.

2. Rényi Entropy and Divergence, and Sibson Information Radius

Rényi [33] investigated a generalization of the four axioms of Fadeev [35], yielding the unique Shannon entropy [20]. In doing so, Rényi replaced the ordinary weighted arithmetic mean by a more general class of averaging schemes. Namely, Rényi considered the weighted quasi-arithmetic means [36]. A weighted quasi-arithmetic mean can be induced by a strictly monotonous and continuous function g, as follows:

$$M_g(x_1,\dots ,x_n;w_1,\dots ,w_n):=g^{-1}\left(\sum _{i=1}^n{w}_{i}g\left(x_i\right)\right),$$

(33)

where the $x_i$’s and the $w_i$’s are positive (the weights are normalized, so that ${\sum }_{i=1}^n{w}_i=1$). Because $M_g=M_{-g}$, we may assume without loss of generality that g is a strictly increasing and continuous function. The quasi-arithmetic means were investigated independently by Kolmogorov [36], Nagumo [37], and de Finetti [38].

For example, the power means $P_{\alpha }(a,b)={\left(\frac{a^{\alpha }+b^{\alpha }}{2}\right)}^{\frac{1}{\alpha }}$ introduced earlier are quasi-arithmetic means for the generator $g_{\alpha }^P\left(u\right):=u^{\alpha }$:

$$P_{\alpha }(a,b)=M_{g_{\alpha }^P}\left(a,b;\frac{1}{2},\frac{1}{2}\right).$$

(34)

Rényi proved that, among the class of weighted quasi-arithmetic means, only the means induced by the family of functions

$$\begin{array}{ccc}\hfill g_{\alpha }\left(u\right)& :=& 2^{(\alpha -1)u},\hfill \end{array}$$

(35)

$$\begin{array}{ccc}\hfill g_{\alpha }^{-1}\left(v\right)& :=& \frac{1}{\alpha -1}{log}_{2}v,\hfill \end{array}$$

(36)

for $\alpha >0$ and $\alpha \ne 1$ yield a proper generalization of Shannon entropy, nowadays called the Rényi $\alpha$-entropy. The Rényi $\alpha$-mean is

$$\begin{array}{ccc}\hfill M_{\alpha }^R(x_1,\dots ,x_n;w_1,\dots ,w_n)& =& M_{g_{\alpha }}\left(x_1,\dots ,x_n;w_1,\dots ,w_n\right),\hfill \end{array}$$

(37)

$$\begin{array}{ccc}& =& \frac{1}{\alpha -1}{log}_2\left(\sum _{i=1}^n{w}_i{2}^{(\alpha -1)x_i}\right).\hfill \end{array}$$

(38)

The Rényi $\alpha$-means $M_{\alpha }^R$ are not power means: They are not homogeneous means [31]. Let $M_{\alpha }^R(p,q)=M_{\alpha }^R\left(p,q;\frac{1}{2},\frac{1}{2}\right)=\frac{1}{\alpha -1}{log}_2\frac{2^{(\alpha -1)p}+2^{(\alpha -1)q}}{2}$. Subsequently, we have ${lim}_{\alpha \to \infty }$$M_{\alpha }^R(p,q)=max\{p,q\}$ and ${lim}_{\alpha \to 1}{M}_{\alpha }^R(p,q)=A(p,q)=\frac{p+q}{2}$. Indeed, we have

$$\begin{array}{ccc}\hfill M_{\alpha }^R(p,q)& =& \frac{1}{\alpha -1}{log}_2\frac{2^{(\alpha -1)p}+2^{(\alpha -1)q}}{2},\hfill \\ & =& \frac{1}{\alpha -1}{log}_2\frac{e^{(\alpha -1)plog2}+e^{(\alpha -1)qlog2}}{2},\hfill \\ & {\approx }_{\alpha \to 1}& \frac{1}{\alpha -1}{log}_2\left(1+(\alpha -1)\frac{p+q}{2}log2\right),\hfill \\ & {\approx }_{\alpha \to 1}& \frac{1}{\alpha -1}\frac{1}{log2}(\alpha -1)\frac{p+q}{2}log2,\hfill \\ & {\approx }_{\alpha \to 1}& \frac{p+q}{2}=A(p,q),\hfill \end{array}$$

using the following first-order approximations: $e^x{\approx }_{x\to 0}=1+x$ and $log(1+x){\approx }_{x\to 0}=x$.

To obtain an intuition of the Rényi entropy, we may consider generalized entropies derived from quasi-arithmetic means, as follows:

$$h_g\left[p\right]:=-M_g({log}_2{p}_1,\dots ,{log}_2{p}_n;p_1,\dots ,p_n).$$

(39)

When $g\left(u\right)=u$, we recover Shannon entropy. When $g_2\left(u\right)=2^u$, we get $h_{g_2}\left[p\right]=-{log}_2{\sum }_i{p}_i^2$, called the collision entropy, since $-log\mathrm{Pr}[X_1=X_2]=h_{g_2}\left[p\right]$, when $X_1$ and $X_2$ are independent and identically distributed random variables with $X_1\sim p$ and $X_2\sim p$. When $g\left(u\right)=g_{\alpha }\left(u\right)=2^{(\alpha -1)u}$, we get

$$\begin{array}{ccc}\hfill h_{g_{\alpha }}\left[p\right]& =& -\frac{1}{\alpha -1}{log}_2\left(\sum _i{p}_i{2}^{(\alpha -1){log}_2{p}_i}\right),\hfill \end{array}$$

(40)

$$\begin{array}{ccc}& =& \frac{1}{1-\alpha }{log}_2\sum _i{p}_i{p}_i^{\alpha -1}=\frac{1}{1-\alpha }{log}_2\sum _i{p}_i^{\alpha }.\hfill \end{array}$$

(41)

The formula of Equation (41) is the discrete Rényi $\alpha$-entropy [33], which can be defined more generally on a measure space $(\mathcal{X},\mathcal{F},\mu )$, as follows:

$$h_{\alpha }^R\left[p\right]:=\frac{1}{1-\alpha }log\left({\int }_{\mathcal{X}}{p}^{\alpha }\left(x\right)\mathrm{d}\mu \left(x\right)\right),\alpha \in (0,1)\cup (1,\infty ).$$

(42)

In the limit case $\alpha \to 1$, the Rényi $\alpha$-entropy converges to Shannon entropy: ${lim}_{\alpha \to 1}{h}_{\alpha }^R\left[p\right]=h\left[p\right]$. Rényi $\alpha$-entropies are non-increasing with respect to increasing $\alpha$: $h_{\alpha }^R\left[p\right]\ge h_{{\alpha }^{\prime }}^R\left[p\right]$ for $\alpha <{\alpha }^{\prime }$. In the discrete case (i.e., counting measure $\mu$ on a finite alphabet $\mathcal{X}$), we can further define $h_0\left[p\right]=log\left|\mathcal{X}\right|$ for $\alpha =0$ (also called max-entropy or Hartley entropy). The Rényi $+\infty$-entropy

$$h_{+\infty }\left[p\right]=-log\underset{x\in \mathcal{X}}{max}p\left(x\right)$$

is also called the min-entropy, since the sequence $h_{\alpha }$ is non-increasing with respect to increasing $\alpha$.

Similarly, Rényi obtained the $\alpha$-divergences for $\alpha >0$ and $\alpha \ne 1$ (originally called information gain of order $\alpha$):

$$D_{\alpha }^R[p:q]:=\frac{1}{\alpha -1}{log}_2\left({\int }_{\mathcal{X}}p{\left(x\right)}^{\alpha }q{\left(x\right)}^{1-\alpha }\mathrm{d}\mu \left(x\right)\right),$$

(43)

generalizing the Kullback–Leibler divergence, since ${lim}_{\alpha \to 1}{D}_{\alpha }^R[p:q]=D_{\mathrm{KL}}[p:q]$. Rényi $\alpha$-divergences are non-decreasing with respect to increasing $\alpha$ [39]: $D_{\alpha }^R[p:q]\le D_{{\alpha }^{\prime }}^R[p:q]$ for ${\alpha }^{\prime }\ge \alpha$.

Sibson (Robin Sibson (1944–2017) is also renown for inventing the natural neighbour interpolation [40]) [1] considered both the Rényi $\alpha$-divergence [33]$D_{\alpha }^R$ and the Rényi $\alpha$-weighted mean $M_{\alpha }^R:=M_{g_{\alpha }}$ to define the information radius $R_{\alpha }$ of order $\alpha$ of a weighted set $\mathcal{P}={\left\{(w_i,p_i)\right\}}_{i=1}^n$ of densities $p_i$’s as the following minimization problem:

$$R_{\alpha }\left(\mathcal{P}\right):=\underset{c\in \mathcal{D}}{min}{R}_{\alpha }(\mathcal{P},c),$$

(44)

where

$$R_{\alpha }(\mathcal{P},c):=M_{\alpha }^R\left(D_{\alpha }^R[p_1:c],\dots ,D_{\alpha }^R[p_n:c];w_1,\dots ,w_n\right).$$

(45)

The Rényi $\alpha$-weighted mean $M_{\alpha }^R$ can be rewritten as

$$\begin{array}{ccc}\hfill M_{\alpha }^R(x_1,\dots ,x_n;w_1,\dots ,w_n)& =& \frac{1}{\alpha -1}\mathrm{LSE}\left((\alpha -1)x_{1}log2+log{w}_1,\dots ,(\alpha -1)x_{i}log2+log{w}_i\right),\hfill \end{array}$$

(46)

where function $\mathrm{LSE}(a_1,\dots ,a_n):=log\left({\sum }_{i=1}^n{e}^{a_i}\right)$ denotes the log-sum-exp (convex) function [41,42].

Notice that $2^{(\alpha -1)D_{\alpha }^R[p:q]}={\int }_{\mathcal{X}}p{\left(x\right)}^{\alpha }q{\left(x\right)}^{1-\alpha }\mathrm{d}\mu \left(x\right)$, the Bhattacharyya $\alpha$-coefficient [12] (also called Chernoff $\alpha$-coefficient [43,44]):

$$C_{\mathrm{Bhat},\alpha }[p:q]:={\int }_{\mathcal{X}}p{\left(x\right)}^{\alpha }q{\left(x\right)}^{1-\alpha }\mathrm{d}\mu \left(x\right).$$

(47)

Thus, we have

$$R_{\alpha }(\mathcal{P},c)=\frac{1}{\alpha -1}{log}_2\left(\sum w_i{C}_{\mathrm{Bhat},\alpha }[p_i:c]\right).$$

(48)

The ordinary Bhattacharyya coefficient is obtained for $\alpha =\frac{1}{2}$: $C_{\mathrm{Bhat}}[p:q]:={\int }_{\mathcal{X}}\sqrt{p\left(x\right)}$$\sqrt{q\left(x\right)}\mathrm{d}\mu \left(x\right)$.

Sibson [1] also considered the limit case $\alpha \to \infty$ when defining the information radius:

$$D_{\infty }^R[p:q]:={log}_2\underset{x\in \mathcal{X}}{sup}\frac{p\left(x\right)}{q\left(x\right)}.$$

(49)

Sibson reported the following theorem in his information radius study [1]:

Theorem 1

(Theorem 2.2 and Corollary 2.3 of [1]). The optimal density $c_{\alpha }^{*}=arg{min}_{c\in \mathcal{D}}{R}_{\alpha }$$(\mathcal{P},c)$ is unique, and we have:

$$\begin{array}{ccc}{c}_1^{*}\left(x\right)=\sum _i{w}_i{p}_i\left(x\right),\hfill & R_1\left(\mathcal{P}\right)=R_1(\mathcal{P},c_1^{*})={\int }_{\mathcal{X}}\sum _i{w}_i{p}_i{log}_2\frac{p_i}{{\sum }_j{w}_j{p}_j\left(x\right)}\mathrm{d}\mu \left(x\right),\hfill & \\ c_{\alpha }^{*}\left(x\right)=\frac{{\left({\sum }_i{w}_i{p}_i{\left(x\right)}^{\alpha }\right)}^{\frac{1}{\alpha }}}{{\int }_{\mathcal{X}}{\left({\sum }_i{w}_i{p}_i{\left(x\right)}^{\alpha }\right)}^{\frac{1}{\alpha }}\mathrm{d}\mu \left(x\right)},\hfill & R_{\alpha }\left(\mathcal{P}\right)=R_{\alpha }(\mathcal{P},c_{\alpha }^{*})=\frac{1}{\alpha -1}{log}_2{\left({\int }_{\mathcal{X}}{\left(\sum _i{w}_i{p}_i{\left(x\right)}^{\alpha }\right)}^{\frac{1}{\alpha }}\mathrm{d}\mu \left(x\right)\right)}^{\alpha },\hfill & \\ & \alpha \in (0,1)\cup (1,\infty )\hfill \\ c_{\infty }^{*}\left(x\right)=\frac{{max}_i{p}_i\left(x\right)}{{\int }_{\mathcal{X}}\left({max}_i{p}_i\left(x\right)\right)\mathrm{d}\mu \left(x\right)},\hfill & R_{\infty }\left(\mathcal{P}\right)=R_{\infty }(\mathcal{P},c_{\infty }^{*})={log}_2{\int }_{\mathcal{X}}\left(\underset{i}{max}{p}_i\left(x\right)\right)\mathrm{d}\mu \left(x\right),\hfill \end{array}$$

Observe that $R_{\infty }\left(\mathcal{P}\right)$ does not depend on the (positive) weights.

The proof follows from the following decomposition of the information radius:

Proposition 1.

We have:

$$R_{\alpha }(\mathcal{P},c)-R_{\alpha }(\mathcal{P},c_{\alpha }^{*})=D_{\alpha }^R(c_{\alpha }^{*},c)\ge 0.$$

(50)

Because the proof is omitted in [1], we report it here:

Proof. 

Let $\Delta (c,c_{\alpha }^{*}):=R_{\alpha }(\mathcal{P},c)-R_{\alpha }(\mathcal{P},c_{\alpha }^{*})$. We handle the three cases, depending on the $\alpha$ values:

• Case $\alpha \in (0,1)\cup (1,\infty )$: Let $P_{\alpha }\left(\mathcal{P}\right)\left(x\right):={\left({\sum }_i{w}_i{p}_i{\left(x\right)}^{\alpha }\right)}^{\frac{1}{\alpha }}$. We have ${\left(c_{\alpha }^{*}\left(x\right)\right)}^{\alpha }=\frac{{\sum }_i{w}_i{p}_i{\left(x\right)}^{\alpha }}{{\left(\int P_{\alpha }\left(\mathcal{P}\right)\left(x\right)\mathrm{d}\mu \left(x\right)\right)}^{\alpha }}$. We obtain

  $$\begin{array}{ccc}\hfill \Delta (c,c_{\alpha }^{*})& =& \frac{1}{\alpha -1}{log}_2\left(\sum _i{w}_i\int p_i{\left(x\right)}^{\alpha }c{\left(x\right)}^{1-\alpha }\mathrm{d}\mu \left(x\right)\right)-\frac{1}{\alpha -1}{log}_2{\left(\int P_{\alpha }\left(\mathcal{P}\right)\left(x\right)\mathrm{d}\mu \left(x\right)\right)}^{\alpha },\hfill \end{array}$$

  (51)

  $$\begin{array}{ccc}& =& \frac{1}{\alpha -1}{log}_2\frac{{\sum }_i{w}_i\int p_i{\left(x\right)}^{\alpha }c{\left(x\right)}^{1-\alpha }\mathrm{d}\mu }{{(\int P_{\alpha }\left(\mathcal{P}\right)\left(x\right)\mathrm{d}\mu \left(x\right))}^{\alpha }},\hfill \end{array}$$

  (52)

  $$\begin{array}{ccc}& =& \frac{1}{\alpha -1}{log}_2\frac{\int \left({\sum }_i{w}_i{p}_i{\left(x\right)}^{\alpha }\right)c{\left(x\right)}^{1-\alpha }}{{(\int P_{\alpha }\left(\mathcal{P}\right)\left(x\right)\mathrm{d}\mu \left(x\right))}^{\alpha }}\mathrm{d}\mu \left(x\right),\hfill \end{array}$$

  (53)

  $$\begin{array}{ccc}& =& \frac{1}{\alpha -1}{log}_2\int {\left(c_{\alpha }^{*}\left(x\right)\right)}^{\alpha }c{\left(x\right)}^{1-\alpha }\mathrm{d}\mu \left(x\right),\hfill \end{array}$$

  (54)

  $$\begin{array}{ccc}& :=& D_{\alpha }^R(c_{\alpha }^{*},c).\hfill \end{array}$$

  (55)
• Case $\alpha =1$: we have $\Delta (c,c_1^{*}):=R_1(\mathcal{P},c)-R_1(\mathcal{P},c_1^{*})$ with $c_1^{*}={\sum }_i{w}_i{p}_i$. Because $R_1(\mathcal{P},c)={\sum }_i{w}_i{D}_{\mathrm{KL}}[p_i:c]$, we have

  $$\begin{array}{ccc}\hfill R_1(\mathcal{P},c)& =& \sum _i{w}_{i}h[p_i:c]-w_{i}h\left[p_i\right],\hfill \end{array}$$

  (56)

  $$\begin{array}{ccc}& =& h[\sum _i{w}_i{p}_i:c]-\sum _i{w}_{i}h\left[p_i\right],\hfill \end{array}$$

  (57)

  $$\begin{array}{ccc}& =& h[c_1^{*}:c]-\sum _i{w}_{i}h\left[p_i\right].\hfill \end{array}$$

  (58)
  It follows that

  $$\begin{array}{ccc}\hfill \Delta (c,c_1^{*})& =& h[c_1^{*}:c]-\sum _i{w}_{i}h\left[p_i\right]-\left(h[c_1^{*}:c_1^{*}]-\sum _i{w}_{i}h\left[p_i\right]\right),\hfill \end{array}$$

  (59)

  $$\begin{array}{ccc}& =& h^{[}{c}_1^{*}:c]-h\left[c_1^{*}\right],\hfill \end{array}$$

  (60)

  $$\begin{array}{ccc}& =& D_{\mathrm{KL}}[c_1^{*}:c]=D_1^R[c_1^{*}:c].\hfill \end{array}$$

  (61)
• Case $\alpha =\infty$: we have $c_{\infty }^{*}=\frac{{max}_i{p}_i\left(x\right)}{\int \left({max}_i{p}_i\left(x\right)\right)\mathrm{d}\mu \left(x\right)}$, $R_{\infty }(\mathcal{P},c_{\infty }^{*})={log}_2\int \left({max}_i{p}_i\left(x\right)\right)\mathrm{d}\mu \left(x\right)$, and $D_{\infty }^R[p:q]={log}_2{sup}_x\frac{p\left(x\right)}{q\left(x\right)}$. We have $R_{\infty }(\mathcal{P},c)={log}_2{sup}_x\frac{p_i\left(x\right)}{c\left(x\right)}$ Thus, $\Delta (c,c_{\alpha }^{*}):=R_{\infty }(\mathcal{P},c)-R_{\infty }(\mathcal{P},c_{\infty }^{*})={log}_2{sup}_x\frac{c_{\infty }^{*}\left(x\right)}{c\left(x\right)}=D_{\infty }^R[c_{\infty }^{*}:c]$.

 □

It follows that

$$\underset{c}{min}{R}_{\alpha }(\mathcal{P},c)=\underset{c}{min}{R}_{\alpha }(\mathcal{P},c_{\alpha }^{*})+D_{\alpha }^R(c_{\alpha }^{*},c)\equiv \underset{c}{min}{D}_{\alpha }^R(c_{\alpha }^{*},c)\ge 0.$$

Thus we have $c=c_{\alpha }^{*}$ since $D_{\alpha }^R(c_{\alpha }^{*},c)$ is minimized for $c=c_{\alpha }^{*}$.

Notice that $c_{\infty }^{*}\left(x\right)=\frac{max\{p_1\left(x\right),\dots ,p_n\left(x\right)\}}{{\int }_{\mathcal{X}}\left({max}_i{p}_i\left(x\right)\right)\mathrm{d}\mu \left(x\right)}$ is the upper envelope of the densities $p_i\left(x\right)$’s normalized to be a density. Provided that the densities $p_i$’s intersect pairwise in at most s locations (i.e., $\left|\right\{p_i\left(x\right)\cap p_j\left(x\right)\left\}\right|\le s$ for $i\ne j$), we can efficiently compute this upper envelope using an output-sensitive algorithm [45] of computational geometry.

When the point set is $\mathcal{P}=\left\{\left(\frac{1}{2},p\right),\left(\frac{1}{2},q\right)\right\}$ with $w_1=w_2=\frac{1}{2}$, the information radius defines a (2-point) symmetric distance, as follows:

$$\begin{array}{cc}{R}_1(p,q)=\frac{1}{2}{\int }_{\mathcal{X}}p\left(x\right){log}_2\frac{2p}{p\left(x\right)+q\left(x\right)}\mathrm{d}\mu \left(x\right)+\frac{1}{2}{\int }_{\mathcal{X}}q\left(x\right){log}_2\frac{2q\left(x\right)}{p\left(x\right)+q\left(x\right)}\mathrm{d}\mu \left(x\right),\hfill & \alpha =1\hfill \\ R_{\alpha }(p,q)=\frac{\alpha }{\alpha -1}{log}_2{\int }_{\mathcal{X}}{\left(\frac{p{\left(x\right)}^{\alpha }+q{\left(x\right)}^{\alpha }}{2}\right)}^{\frac{1}{\alpha }}\mathrm{d}\mu \left(x\right)=\frac{\alpha }{\alpha -1}{log}_2{\int }_{\mathcal{X}}{P}_{\alpha }(p\left(x\right),q\left(x\right))\mathrm{d}\mu \left(x\right),\hfill & \alpha \in (0,1)\cup (1,\infty )\hfill \\ R_{\infty }(p,q)={log}_2{\int }_{\mathcal{X}}max\{p\left(x\right),q\left(x\right)\}\mathrm{d}\mu \left(x\right),\hfill & \alpha =\infty .\hfill \end{array}$$

This family of symmetric divergences may be called the Sibson’s $\alpha$-divergences, and the Jensen-Shannon divergence is interpreted as a limit case when $\alpha \to 1$. Notice that, since we have ${lim}_{\alpha \to \infty }{P}_{\alpha }(p,q)=max\{p,q\}$ and ${lim}_{\alpha \to \infty }\frac{\alpha }{\alpha -1}=1$, we have ${lim}_{\alpha \to \infty }{R}_{\alpha }(p,q)=R_{\infty }(p,q)$. Notice that, for $\alpha =1$, the integral and logarithm operations are swapped as compared to $R_{\alpha }$ for $\alpha \in (0,1)\cup (1,\infty )$.

Theorem 2.

When $\alpha =\frac{1}{k}$ for an integer $k\ge 2$, the Sibson α-divergences between two densities $p_{{\theta }_1}$ and $p_{{\theta }_2}$ of an exponential family $\{p_{\theta }:\theta \in \Theta \}$ with cumulant function $F\left(\theta \right)$ is available in closed form:

$$R_{\alpha }(p_{{\theta }_1},p_{{\theta }_2})=-\frac{1}{k-1}{log}_2\left(\frac{1}{2^k}\sum _{i=0}^k\left(\genfrac{}{}{0pt}{}{k}{i}\right)exp\left(F\left(\frac{i}{k}{\theta }_1+\left(1-\frac{i}{k}\right){\theta }_2\right)-\left(\frac{i}{k}F\left({\theta }_1\right)+\left(1-\frac{i}{k}\right)F\left({\theta }_2\right)\right)\right)\right).$$

Proof. 

Let $p=p_{{\theta }_1}$ and $q=p_{{\theta }_2}$ be two densities of an exponential family [2] with cumulant function $F\left(\theta \right)$ and natural parameter space $\Theta$. Without a loss of generality, we may consider a natural exponential family [2] with densities written canonically as $p_{\theta }\left(x\right)=exp(x^{\top }\theta -F\left(\theta \right))$ for $\theta \in \Theta$. It can be shown that the cumulant function $F\left(\theta \right)=log{\int }_{\mathcal{X}}exp\left(x^{\top }\theta \right)\mathrm{d}\mu \left(x\right)$ is strictly convex and analytic on the open convex natural parameter space $\Theta$ [2].

When $\alpha =\frac{1}{2}$ (i.e., $k=2$), we have:

$$\begin{array}{ccc}\hfill R_{\frac{1}{2}}(p,q)& =& -{log}_2{\int }_{\mathcal{X}}{\left(\frac{\sqrt{p\left(x\right)}+\sqrt{q\left(x\right)}}{2}\right)}^2\mathrm{d}\mu \left(x\right),\hfill \end{array}$$

(62)

$$\begin{array}{ccc}& =& -{log}_2\left(\frac{1}{2}+\frac{1}{2}{\int }_{\mathcal{X}}\sqrt{p\left(x\right)}\sqrt{q\left(x\right)}\mathrm{d}\mu \left(x\right)\right),\hfill \end{array}$$

(63)

$$\begin{array}{ccc}& =& -{log}_2\left(\frac{1}{2}+\frac{1}{2}{C}_{\mathrm{Bhat}}[p:q]\right)\ge 0,\hfill \end{array}$$

(64)

where $C_{\mathrm{Bhat}}[p:q]:={\int }_{\mathcal{X}}\sqrt{p\left(x\right)}\sqrt{q\left(x\right)}\mathrm{d}\mu \left(x\right)$ is the Bhattacharyya coefficient (with $0\le C_{\mathrm{Bhat}}[p:q]\le 1$). Using Theorem 3 of [12], we have

$$C_{\mathrm{Bhat}}[p_{{\theta }_1},p_{{\theta }_2}]=exp\left(F\left(\frac{{\theta }_p+{\theta }_q}{2}\right)-\frac{F\left({\theta }_p\right)+F\left({\theta }_q\right)}{2}\right),$$

so that we obtain the following closed-form formula:

$$R_{\frac{1}{2}}(p_{{\theta }_1},p_{{\theta }_2})=-{log}_2\left(\frac{1}{2}+\frac{1}{2}exp\left(F\left(\frac{{\theta }_p+{\theta }_q}{2}\right)-\frac{F\left({\theta }_p\right)+F\left({\theta }_q\right)}{2}\right)\right)\ge 0,$$

Now, assume that $k=\frac{1}{\alpha }\ge 2$ is an arbitrary integer, and let us apply the binomial expansion for $P_{\alpha }(p_{{\theta }_1},p_{{\theta }_2})$ in the spirit of [46,47]:

$$\begin{array}{ccc}\hfill {\int }_{\mathcal{X}}{P}_{\alpha }(p_{{\theta }_1}\left(x\right),p_{{\theta }_2}\left(x\right))\mathrm{d}\mu \left(x\right)& =& {\int }_{\mathcal{X}}{\left(\frac{p_{{\theta }_1}{\left(x\right)}^{\frac{1}{k}}+p_{{\theta }_2}{\left(x\right)}^{\frac{1}{k}}}{2}\right)}^k\mathrm{d}\mu \left(x\right),\hfill \end{array}$$

(65)

$$\begin{array}{ccc}& =& \frac{1}{2^k}\sum _{i=0}^k\left(\genfrac{}{}{0pt}{}{k}{i}\right){\int }_{\mathcal{X}}{\left(p_{{\theta }_1}{\left(x\right)}^{\frac{1}{k}}\right)}^i{\left(p_{{\theta }_2}{\left(x\right)}^{\frac{1}{k}}\right)}^{k-i}\mathrm{d}\mu \left(x\right).\hfill \end{array}$$

(66)

Let $I_{k,i}({\theta }_1,{\theta }_2):={\int }_{\mathcal{X}}{\left(p_{{\theta }_1}{\left(x\right)}^{\frac{1}{k}}\right)}^i{\left(p_{{\theta }_2}{\left(x\right)}^{\frac{1}{k}}\right)}^{k-i}\mathrm{d}\mu \left(x\right)$. Because $\frac{i}{k}{\theta }_1+\frac{k-i}{k}{\theta }_2={\theta }_2+\frac{i}{k}({\theta }_1-{\theta }_2)\in \Theta$ for $i\in \{0,\dots ,k\}$, we get by following the calculation steps in [12]:

$$I_{k,i}({\theta }_1,{\theta }_2):=exp\left(F\left(\frac{i}{k}{\theta }_1+\left(1-\frac{i}{k}\right){\theta }_2\right)-\left(\frac{i}{k}F\left({\theta }_1\right)+\left(1-\frac{i}{k}\right)F\left({\theta }_2\right)\right)\right)<\infty .$$

Notice that $I_{2,1}=C_{\mathrm{Bhat}}[p_{{\theta }_1},p_{{\theta }_2}]$, and $I_{k,0}=I_{k,k}=1$.

Thus, we get the following closed-form formula:

$$\begin{array}{ccc}\hfill R_{\alpha }(p_{{\theta }_1},p_{{\theta }_2})& =& -\frac{1}{k-1}{log}_2\left(\frac{1}{2^k}\sum _{i=0}^k\left(\genfrac{}{}{0pt}{}{k}{i}\right)exp\left(F\left(\frac{i}{k}{\theta }_1+\left(1-\frac{i}{k}\right){\theta }_2\right)-\left(\frac{i}{k}F\left({\theta }_1\right)+\left(1-\frac{i}{k}\right)F\left({\theta }_2\right)\right)\right)\right).\hfill \end{array}$$

(67)

 □

This closed-form formula applies, in particular, to the family $\left\{\mathcal{N}\right(\mu ,\Sigma \left)\right\}$ of (multivariate) normal distributions: In this case, the natural parameters $\theta$ are expressed using both a vector parameter component v and a matrix parameter component M:

$$\theta =(v,M)=\left({\Sigma }^{-1}m,-\frac{1}{2}{\Sigma }^{-1}\right),$$

(68)

and the cumulant function is:

$$F_{\mathcal{N}}\left(\theta \right)=\frac{d}{2}log\pi -\frac{1}{2}log|-2M|-\frac{1}{4}{v}^{\top }{M}^{-1}v,$$

(69)

where $|·|$ denotes the matrix determinant.

In general, the optimal density $c_{\alpha }^{*}=arg{min}_{c\in \mathcal{D}}{R}_{\alpha }(\mathcal{P},c)$ yielding the information radius $R_{\alpha }\left(\mathcal{P}\right)$ can be interpreted as a generalized centroid (extending the notion of Fréchet means [48]) with respect to $(M_{\alpha }^R,D_{\alpha }^R)$, where a $(M,D)$-centroid is defined by:

Definition 1

($(M,D)$-centroid). Let $\mathcal{P}=\{(w_1,p_1),\dots ,(w_n,p_n)\}$ be a normalized weighted parameter set, M a mean, and D a distance. Subsequently, the $(M,D)$-centroid is defined as

$$c_{M,D}\left(\mathcal{P}\right)=arg\underset{c}{min}M(D(p_1:c),\dots ,D(p_n:c);w_1,\dots ,w_n).$$

Here, we give a general definition of the $(M,D)$-centroid for an arbitrary distance (not necessarily a symmetric nor metric distance). The parameter set can either be probability measures having densities with respect to a given measure $\mu$ or a set of vectors. In the first case, the distance D is called a statistical distance. When the densities belong to a parametric family of densities $\mathcal{P}=\{p_{\theta }:\theta \in \Theta \}$, the statistical distance $D[p_{{\theta }_1}:p_{{\theta }_2}]$ amounts to a parameter distance: $D_{\mathcal{P}}({\theta }_1:{\theta }_2):=D[p_{{\theta }_1}:p_{{\theta }_2}]$. For example, when all of the densities $p_i$’s belong to a same natural exponential family [2]

$$\mathcal{P}=\{p_{\theta }\left(x\right)=exp({\theta }^{\top }t\left(x\right)-F\left(\theta \right)):\theta \in \Theta \}$$

with cumulant function $F\left(\theta \right)=log\int exp\left({\theta }^{\top }t\left(x\right)\right)\mathrm{d}\mu \left(x\right)$ (i.e., $p_i=p_{{\theta }_i}$) and sufficient statistic vector $t\left(x\right)$, we have $D_{\mathrm{KL}}[p_{\theta }:p_{{\theta }_i}]=B_F^{*}(\theta :{\theta }_i):=B_F({\theta }_i:\theta )$, where $B_F^{*}$ denotes the reverse Bregman divergence (by parameter order swapping) the Bregman divergence [21]$B_F$ defined by

$$B_F(\theta :{\theta }^{\prime }):=F\left(\theta \right)-F\left({\theta }^{\prime }\right)-{(\theta -{\theta }^{\prime })}^{\top }\nabla F\left({\theta }^{\prime }\right).$$

(70)

Thus, we have $D_{\mathcal{P}}({\theta }_1:{\theta }_2):=B_F^{*}({\theta }_1:{\theta }_2)=D_{\mathrm{KL}}[p_{{\theta }_1}:p_{{\theta }_2}]$.

Let $\mathcal{V}=\{(w_1,{\theta }_1),\dots ,(w_n,{\theta }_n)\}$ be the parameter set corresponding to $\mathcal{P}$. Define

$$R_F(\mathcal{V},\theta ):=\sum _{i=1}^n{w}_i{B}_F({\theta }_i:\theta ).$$

(71)

Subsequently, we have the equivalent decomposition of Proposition 1:

$$R_F(\mathcal{V},\theta )-R_F(\mathcal{V},{\theta }^{*})=B_F({\theta }^{*}:\theta ),$$

(72)

with ${\theta }^{*}=\overline{\theta }:={\sum }_{i=1}^n{w}_i{\theta }_i$. (this decomposition is used to prove Proposition 1 of [21]). The quantity $R_F\left(\mathcal{V}\right)=R_F(\mathcal{V},{\theta }^{*})$ was termed the Bregman information [21,49]. The Bregman information generalizes the variance that was obtained when the Bregman divergence is the squared Euclidean distance. $R_F\left(\mathcal{V}\right)$ could also be called Bregman information radius according to Sibson. Because $R_F\left(\mathcal{V}\right)={\sum }_{i=1}^n{w}_i{D}_{\mathrm{KL}}[p_{\overline{\theta }}:p_{{\theta }_i}]$, we can interpret the Bregman information as a Sibson’s information radius for densities of an exponential family with respect to the arithmetic mean $M_1^R=A$ and the reverse Kullback–Leibler divergence: $D_{\mathrm{KL}}^{*}[p:q]:=D_{\mathrm{KL}}[q:p]$. This observation yields us the JS-symmetrization of distances based on generalized information radii in Section 3.

More generally, we may consider the densities belonging to a deformed q-exponential family (see [10], page 85–89 and the monograph [50]). Deformed q-exponential families generalize the exponential families, and include the q-Gaussians [10]. A common way to measure the statistical distance between two densities of a q-exponential family is the q-divergence [10], which is related to Tsallis’ entropy [51]. We may also define another statistical divergence between two densities of a q-exponential family which amounts to Bregman divergence. For example, we refer to [52] for details concerning the family of Cauchy distributions, which are q-Gaussians for $q=2$.

Sibson proved that the information radii of any order are all upper bounded (Theorem 2.8 and Theorem 2.9 of [1]) as follows:

$$\begin{array}{ccc}\hfill R_1\left(\mathcal{P}\right)& \le & \sum _i{w}_i{log}_2\frac{1}{w_j}\le {log}_{2}n<\infty ,\hfill \end{array}$$

(73)

$$\begin{array}{ccc}\hfill R_{\alpha }\left(\mathcal{P}\right)& \le & \frac{\alpha }{\alpha -1}{log}_2\left(\sum _i{w}_i^{\frac{1}{\alpha }}\right)\le {log}_{2}n<\infty ,\alpha \in (0,1)\cup (1,\infty )\hfill \end{array}$$

(74)

$$\begin{array}{ccc}\hfill R_{\infty }\left(\mathcal{P}\right)& \le & {log}_{2}n<\infty .\hfill \end{array}$$

(75)

We interpret Sibson’s upper bounds of Equations (73)–(75), as follows:

Proposition 2

(Information radius upper bound). The information radius of order α of a weighted set of distributions is upper bounded by the discrete Rényi entropy of order $\frac{1}{\alpha }$ of the weight distribution: $R_{\alpha }\left(\mathcal{P}\right)\le H_{\frac{1}{\alpha }}^R\left[w\right]$, where $H_{\alpha }^R\left[w\right]:=\frac{1}{1-\alpha }log\left({\sum }_i{w}_i^{\alpha }\right)$.

3. JS-Symmetrization of Distances Based on Generalized Information Radius

Let us give the following definitions generalizing the information radius (i.e., Jensen-Shannon symmetrization of the distance when $\left|\mathcal{P}\right|=2$) and the ordinary Jensen-Shannon divergence:

Definition 2

($(M,D)$-information radius). Let M be a weighted mean and D a distance. Subsequently, the generalized information radius for a weighted set of points (e.g., vectors or densities) $(w_1,p_1),\dots ,(w_n,p_n)$ is:

$$R_{M,D}\left(\mathcal{P}\right):=\underset{c\in \mathcal{D}}{min}M\left(D(p_1:c),\dots ,D(p_n:c);w_1,\dots ,w_n\right).$$

Recall that we also defined the $(M,D)$-centroid in Definition 1 as follows:

$$c_{M,D}\left(\mathcal{P}\right):=arg\underset{c\in \mathcal{D}}{min}M\left(D(p_1:c),\dots ,D(p_n:c);w_1,\dots ,w_n\right).$$

When $M=A$, we recover the notion of Fréchet mean [48]. Notice that, although the minimum $R_{M,D}\left(\mathcal{P}\right)$ is unique, several generalized centroids $c_{M,D}\left(\mathcal{P}\right)$ may potentially exist, depending on $(M,D)$. In particular, Definition 2 and Definition 1 apply when D is a statistical distance, i.e., a distance between densities (Radon–Nikodym derivatives of corresponding probability measures with respect to a dominating measure $\mu$).

The generalized information radius can be interpreted as a diversity index or an n-point distance. When $n=2$, we get the following (2-point) distances, which are considered as a generalization of the Jensen-Shannon divergence or Jensen-Shannon symmetrization:

Definition 3

(M-vJS symmetrization of D). Let M be a mean and D a statistical distance. Subsequently, the variational Jensen-Shannon symmetrization of D is defined by the formula of a generalized information radius:

$$D_M^{\mathrm{vJS}}[p:q]:=\underset{c\in \mathcal{D}}{min}M\left(D[p:c],D[q:c]\right).$$

We use the acronym $\mathrm{vJS}$ to distinguish it with the JS-symmetrization reported in [23]:

$$D_M^{\mathrm{JS}}[p:q]=D_{M,A}^{\mathrm{JS}}[p:q]:=\frac{1}{2}\left(D\left[p:{\left(pq\right)}_{\frac{1}{2}}^M\right]+D\left[q:{\left(pq\right)}_{\frac{1}{2}}^M\right]\right).$$

We recover Sibson’s information radius $R_{\alpha }[p:q]$ induced by two densities p and q from Definition 3 as the $M_{\alpha }^R$-vJS symmetrization of the Rényi divergence $D_{\alpha }^R$. We have ${B_F}_A^{\mathrm{vJS}}$, which is the Bregman information [21]. Notice that we may skew these generalized JSDs by taking weighted mean $M_{\beta }$ instead of M for $\beta \in (0,1)$, yielding the general definition:

Definition 4

(Skew $M_{\beta }$-vJS symmetrization of D). Let $M_{\beta }$ be a weighted mean and D a statistical distance. Subsequently, the variational skewed Jensen-Shannon symmetrization of D is defined by the formula of a generalized information radius:

$$\begin{array}{|c|}\hline D_{M_{\beta }}^{\mathrm{vJS}}[p:q]:=\underset{c\in \mathcal{D}}{min}{M}_{\beta }\left(D[p:c],D[q:c]\right)\\ \hline\end{array}$$

Example 1.

For example, the skewed Jensen–Bregman divergence of Equation (20) can be interpreted as a Jensen-Shannon symmetrization of the Bregman divergence $B_F$ [12] since we have:

$$\begin{array}{ccc}\hfill {B_F}_{A_{\beta }}^{\mathrm{vJS}}({\theta }_1:{\theta }_2)& =& \underset{\theta \in \Theta }{min}{A}_{\beta }\left(B_F({\theta }_1:\theta ),B_F({\theta }_2:\theta )\right),\hfill \end{array}$$

(76)

$$\begin{array}{ccc}& =& \underset{\theta \in \Theta }{min}(1-\beta )B_F({\theta }_1:\theta )+\beta B_F({\theta }_2:\theta ),\hfill \end{array}$$

(77)

$$\begin{array}{ccc}& =& (1-\beta )B_F({\theta }_1:(1-\beta ){\theta }_1+\beta {\theta }_2)+\beta B_F({\theta }_2:(1-\beta ){\theta }_1+\beta {\theta }_2),\hfill \end{array}$$

(78)

$$\begin{array}{ccc}& =:& {\mathrm{JB}}_{F,\beta }({\theta }_1:{\theta }_2).\hfill \end{array}$$

(79)

Indeed, the Bregman barycenter $arg{min}_{\theta \in \Theta }(1-\beta )B_F({\theta }_1:\theta )+B_F({\theta }_2:\theta )$ is unique and it corresponds to $\theta =(1-\beta ){\theta }_1+\beta {\theta }_2$, see [21]. The skewed Jensen–Bregman divergence ${\mathrm{JB}}_{F,\beta }({\theta }_1:{\theta }_2)$ can also be rewritten as an equivalent skewed Jensen divergence (see Equation (22)):

$$\begin{array}{ccc}\hfill {\mathrm{JB}}_{F,\beta }({\theta }_1:{\theta }_2)& =& (1-\beta )B_F({\theta }_1:(1-\beta ){\theta }_1+\beta {\theta }_2)+\beta B_F({\theta }_2:(1-\beta ){\theta }_1+\beta {\theta }_2),\hfill \end{array}$$

(80)

$$\begin{array}{ccc}& =& (1-\beta )F\left({\theta }_1\right)+\beta F\left({\theta }_2\right)-F((1-\beta ){\theta }_1+\beta {\theta }_2),\hfill \end{array}$$

(81)

$$\begin{array}{ccc}& =:& J_{F,\beta }({\theta }_1:{\theta }_2).\hfill \end{array}$$

(82)

Example 2.

Consider a conformal Bregman divergence [53] that is defined by

$$B_{F,\rho }({\theta }_1:{\theta }_2)=\rho \left({\theta }_1\right)B_F({\theta }_1:{\theta }_2),$$

(83)

where $\rho \left(\theta \right)>0$ is a conformal factor. Subsequently, we have

$$\begin{array}{ccc}\hfill {B_{F,\rho }}_{A_{\beta }}^{\mathrm{vJS}}({\theta }_1:{\theta }_2)& =& \underset{\theta \in \Theta }{min}{A}_{\beta }\left(B_{F,\rho }({\theta }_1:\theta ),B_{F,\rho }({\theta }_2:\theta )\right),\hfill \end{array}$$

(84)

$$\begin{array}{ccc}& =& \underset{\theta \in \Theta }{min}(1-\beta )B_{F,\rho }({\theta }_1:\theta )+B_{F,\rho }({\theta }_2:\theta ),\hfill \end{array}$$

(85)

$$\begin{array}{ccc}& =& (1-\beta )B_F({\theta }_1:{\gamma }_1{\theta }_1+{\gamma }_2{\theta }_2)+\beta B_F({\theta }_2:{\gamma }_1{\theta }_1+{\gamma }_2{\theta }_2),\hfill \end{array}$$

(86)

where ${\gamma }_1=\frac{(1-\beta )\rho \left({\theta }_1\right)}{(1-\beta )\rho \left({\theta }_1\right)+\beta \rho \left({\theta }_2\right)}$ and ${\gamma }_2=\frac{\beta \rho \left({\theta }_2\right)}{(1-\beta )\rho \left({\theta }_1\right)+\beta \rho \left({\theta }_2\right)}=1-{\gamma }_1$.

Notice that this definition is implicit and it can be made explicit when the centroid $c^{*}(p,q)$ is unique:

$$D_{M_{\beta }}^{\mathrm{vJS}}[p:q]=M_{\beta }\left(D[p:c^{*}(p,q)],D[q:c^{*}(p,q)]\right.$$

(87)

In particular, when $D=D_{\mathrm{KL}}$, the KLD, we obtain generalized skewed Jensen-Shannon divergences for $M_{\beta }$ a weighted mean with $\beta \in (0,1)$:

$$D_{\mathrm{vJS}}^{M_{\beta }}[p:q]:=\underset{c\in \mathcal{D}}{min}{M}_{\beta }\left(D_{\mathrm{KL}}[p:c],D_{\mathrm{KL}}[q:c]\right).$$

(88)

Example 3.

Amari [31] obtained the $(A,D_{\alpha })$-information radius and its corresponding unique centroid for $D_{\alpha }$, the α-divergence of information geometry [10] (page 67).

Example 4.

Brekelmans et al. [54] studied the geometric path ${\left(p_1{p}_2\right)}_{\beta }^G\left(x\right)\propto p_1^{1-\beta }\left(x\right)p_2^{\beta }\left(x\right)$ between two distributions $p_1$ and $p_2$ of $\mathcal{D}$, where $G_{\beta }(a,b)=a^{1-\beta }{b}^{\beta }$ (with $a,b>0$) is the weighted geometric mean. They proved the variational formula:

$${\left(p_1{p}_2\right)}_{\beta }^G=\underset{c\in \mathcal{D}}{min}(1-\beta )D_{\mathrm{KL}}[c:p_1]+\beta D_{\mathrm{KL}}[c:p_2].$$

(89)

That is, ${\left(p_1{p}_2\right)}_{\beta }^G$ is a $G_{\beta }$-$D_{\mathrm{KL}}^{*}$ centroid, where $D_{\mathrm{KL}}^{*}$ is the reverse KLD. The corresponding $(G_{\beta },D_{\mathrm{KL}}^{*})$-vJSD is studied is [23] and it is used in deep learning in [30].

It is interesting to study the link between $(M_{\beta },D)$-variational Jensen-Shannon symmetrization of D and the $(M_{\alpha }^{\prime },N_{\beta }^{\prime })$-JS symmetrization of [23]. In particular, the link between $M_{\beta }$ for averaging in the minimization and $M_{\alpha }^{\prime }$ the mean for generating abstract mixtures.

More generally, Brekelmans et al. [55] considered the α-divergences extended to positive measures (i.e., a separable divergence built as the different between a weighted arithmetic mean and a geometric mean [56]):

$$D_{\alpha }^e[p:q]:=\frac{4}{1-{\alpha }^2}{\int }_{\mathcal{X}}\left(\frac{1-\alpha }{2}p\left(x\right)+\frac{1+\alpha }{2}q\left(x\right)-p^{\frac{1-\alpha }{2}}\left(x\right)q^{\frac{1+\alpha }{2}}\left(x\right)\right)\mathrm{d}\mu \left(x\right)$$

(90)

and proved that

$$c_{\beta }^{*}=arg\underset{c\in \mathcal{D}}{min}\{(1-\beta )D_{\alpha }^e[p_1:c]+\beta D_{\alpha }^e[p_2:c]\}$$

(91)

is a density of a likelihood ratio q-exponential family: $c_{\beta }^{*}=\frac{p_1\left(x\right)}{Z_{\beta ,q}}{exp}_q\left(\beta {log}_q\frac{p_2\left(x\right)}{p_1\left(x\right)}\right)$ for $q=\frac{1+\alpha }{2}$. That is, $c_{\beta }^{*}$ is the $(A_{\beta },D_{\alpha }^e)$-generalized centroid, and the corresponding information radius is the variational JS symmetrization:

$${D_{\alpha }^e}^{\mathrm{vJS}}[p_1:p_2]=(1-\beta )D_{\alpha }^e[p_1:c_{\beta }^{*}]+\beta D_{\alpha }^e[p_2:c_{\beta }^{*}]$$

(92)

Example 5.

The q-divergence [57]$D_q$ between two densities of a q-exponential family amounts to a Bregman divergence [10,57]. Thus, $D_q^{\mathrm{vJS}}$ for $M=A$ is a generalized information radius that amounts to a Bregman information.

For the case $\alpha =\infty$ in Sibson’s information radius, we find that the information radius is related to the total variation:

Proposition 3

(Lemma 2.4 [1]). :

$$D_{\infty }^{\mathrm{vJS},R}[p:q]={log}_2(1+D_{\mathrm{TV}}[p:q]),$$

(93)

where $D_{\mathrm{TV}}$ denotes the total variation

$$D_{\mathrm{TV}}[p:q]=\frac{1}{2}{\int }_{\mathcal{X}}|p\left(x\right)-q\left(x\right)|\mathrm{d}\mu \left(x\right).$$

(94)

Proof. 

Because $max\{p\left(x\right),q\left(x\right)\}=\frac{p\left(x\right)+q\left(x\right)}{2}+\frac{1}{2}|q\left(x\right)-p\left(x\right)|$, it follows that we have:

$${\int }_{\mathcal{X}}max\{p\left(x\right),q\left(x\right)\}\mathrm{d}\mu \left(x\right)=1+D_{\mathrm{TV}}[p:q].$$

From Theorem 1, we have $R_{\infty }(\{(\frac{1}{2},p),(\frac{1}{2},q))={log}_2{\int }_{\mathcal{X}}max\{p\left(x\right),q\left(x\right)\}\mathrm{d}\mu \left(x\right)$ and, therefore, $R_{\infty }(\{(\frac{1}{2},p),(\frac{1}{2},q))={log}_2\left(1+D_{\mathrm{TV}}[p:q]\right)$. □

Notice that, when $M=M_g$ is a quasi-arithmetic mean, we may consider the divergence $D_g[p:q]=g^{-1}\left(D[p:q)\right)$, so that the centroid of the $(M_g,D_g)$-JS symmetrization is:

$$arg\underset{c}{min}{g}^{-1}\left(\sum _{i=1}^n{w}_{i}D[p_i:c]\right)\equiv arg\underset{c}{min}\sum _{i=1}^n{w}_{i}D[p_i:c].$$

(95)

The generalized $\alpha$-skewed Bhattacharyya divergence [29] can also be considered with respect to a weighted mean $M_{\alpha }$:

$$D_{\mathrm{Bhat},M_{\alpha }}[p:q]=-log{\int }_{\mathcal{X}}{M}_{\alpha }(p\left(x\right),q\left(x\right))\mathrm{d}\mu \left(x\right).$$

In particular, when $M_{\alpha }$ is a quasi-arithmetic weighted mean that is induced by a strictly continuous and monotone function g, we have

$$D_{\mathrm{Bhat},g,\alpha }[p:q]:=-log{\int }_{\mathcal{X}}{M}_g(p\left(x\right),q\left(x\right);\alpha )\mathrm{d}\mu \left(x\right)=:D_{\mathrm{Bhat},{\left(M_g\right)}_{\alpha }}[p:q].$$

Because $min\{p\left(x\right),q\left(x\right)\}\le M_g(p\left(x\right),q\left(x\right);\alpha )\le max\{p\left(x\right),q\left(x\right)\}$, $min\{a,b\}=\frac{a+b}{2}-\frac{|b-a|}{2}$ and $max\{a,b\}=\frac{a+b}{2}+\frac{|b-a|}{2}$, we deduce that we have:

$$0\le 1-D_{\mathrm{TV}}[p,q]\le {\int }_{\mathcal{X}}{M}_g(p\left(x\right),q\left(x\right);\alpha )\mathrm{d}\mu \left(x\right)\le 1+D_{\mathrm{TV}}[p,q]\le 2.$$

(96)

The information radius of Sibson for $\alpha \in (0,1)\cup (1,\infty )$ may be interpreted as generalized scaled $\alpha$-skewed Bhattacharyya divergences with respect to the power means $P_{\alpha }$, since we have $R_{\alpha }(p,q)=\frac{\alpha }{\alpha -1}{log}_2{\int }_{\mathcal{X}}{P}_{\alpha }(p\left(x\right),q\left(x\right);\alpha )\mathrm{d}\mu \left(x\right)=\frac{\alpha }{1-\alpha }{D}_{\mathrm{Bhat},P_{\alpha }}[p:q]$.

4. Relative Information Radius and Relative Jensen-Shannon Symmetrizations of Distances

4.1. Relative Information Radius

In this section, instead of considering the full space of densities $\mathcal{D}$ on $(\mathcal{X},\mathcal{F},\mu )$ for performing the variational optimization of the information radius, we rather consider a subfamily of (parametric) densities $\mathcal{R}\subset \mathcal{D}$. Subsequently, we define accordingly the $\mathcal{R}$-relative Jensen-Shannon divergence ($\mathcal{R}$-JSD for short) as

$$D_{\mathrm{vJS}}^{\mathcal{R}}[p:q]:=\underset{c\in \mathcal{R}}{min}\left\{\frac{1}{2}{D}_{\mathrm{KL}}[p:c]+\frac{1}{2}{D}_{\mathrm{KL}}[q:c]\right\}.$$

(97)

In particular, Sibson [1] considered the normal information radius, i.e., the $\mathcal{R}$-relative Jensen-Shannon divergence with $\mathcal{R}=\{\mathcal{N}(\mu ,\Sigma ):(\mu ,\Sigma )\in {\mathbb{R}}^d\times {\mathbb{P}}_{++}^d\}$, where ${\mathbb{P}}_{++}^d$ denotes the open cone of $d\times d$ positive-definite matrices (positive-definite covariance matrices of Gaussian distributions). More generally, we may consider any exponential family $\mathcal{E}$ [2].

Definition 5

(Relative $(\mathcal{R},M)$-JS symmetrization of D). Let M be a mean and D a statistical distance. Subsequently, the relative $(\mathcal{R},M)$-JS symmetrization of D is:

$$D_{M,\mathcal{R}}^{\mathrm{vJS}}[p:q]:=\underset{c\in \mathcal{R}}{min}M\left(D[p:c],D[q:c]\right).$$

We obtain the relative Jensen-Shannon divergences when $D=D_{\mathrm{KL}}$.

Example 6.

Grosse et al. [58] considered geometric and moment average paths for annealing. They proved that, when $p_1=p_{{\theta }_1}$ and $p_2=p_{{\theta }_2}$ belong to an exponential family [2]${\mathcal{E}}_F$ with cumulant function F, we have

$${\left(p_1{p}_2\right)}_{\beta }^G=\frac{p_1{\left(x\right)}^{1-\beta }{p}_2{\left(x\right)}^{\beta }}{\int p_1{\left(x\right)}^{1-\beta }{p}_2{\left(x\right)}^{\beta }\mathrm{d}\mu \left(x\right)}=arg\underset{c\in {\mathcal{E}}_F}{min}\left\{(1-\beta )D_{\mathrm{KL}}[c:p_1]+\beta D_{\mathrm{KL}}[c:p_2]\right\},$$

(98)

and

$$p_{\overline{\eta }}=arg\underset{c\in {\mathcal{E}}_F}{min}\left\{(1-\beta )D_{\mathrm{KL}}[p_1:c]+\beta D_{\mathrm{KL}}[c:p_2]\right\},$$

(99)

where $\overline{\eta }=(1-\beta ){\eta }_1+\beta {\eta }_2$, ${\eta }_i=E_{p_{{\theta }_i}}\left[t\left(x\right)\right]$ (this is not an arithmetic mixture, but an exponential family density moment parameter that is a mixture of the parameters).

The corresponding minima can be interpreted as relative skewed Jensen-Shannon symmetrization for the reverse KLD $D_{\mathrm{KL}}^{*}$ (Equation (98)) and the relative skewed Jensen-Shannon divergence (Equation (99)):

$$\begin{array}{ccc}\hfill {D_{\mathrm{KL}}^{*}}_{A_{\beta },{\mathcal{E}}_F}^{\mathrm{vJS}}[p_1:p_2]& =& \underset{c\in {\mathcal{E}}_F}{min}\left\{(1-\beta )D_{\mathrm{KL}}^{*}[p_1:c]+\beta D_{\mathrm{KL}}^{*}[p_2:c]\right\},\hfill \end{array}$$

(100)

$$\begin{array}{ccc}\hfill D_{A_{\beta },{\mathcal{E}}_F}^{\mathrm{vJS}}[p_1:p_2]& =& \underset{c\in {\mathcal{E}}_F}{min}\left\{(1-\beta )D_{\mathrm{KL}}[c:p_1]+\beta D_{\mathrm{KL}}[c:p_2]\right\},\hfill \end{array}$$

(101)

where $A_{\beta }(a,b):=(1-\beta )a+\beta b$ is the weighted arithmetic mean for $\beta \in (0,1)$.

Notice that, when $p=q$, we have $D_{M,\mathcal{R}}^{\mathrm{vJS}}[p:p]={min}_{c\in \mathcal{R}}D[p:c]$, which is the information projection [59] with respect to D of density p to the submanifold $\mathcal{R}$. Thus, when $p\notin \mathcal{R}$, we have $D_{M,\mathcal{R}}^{\mathrm{vJS}}[p:p]>0$, i.e., the relative JSDs are not proper divergences, since a proper divergence ensures that $D[p:q]\ge 0$ with equality if $p=q$. Figure 1 illustrates the main cases of the relative Jensen-Shannon divergenc between p and q: Either p and q are both inside or outside $\mathcal{R}$, or one point is inside $\mathcal{R}$, while the other point is outside $\mathcal{R}$. When $p=q$, we get an information projection when both of the points are outside $\mathcal{R}$, and $D_{\mathrm{vJS}}^{\mathcal{R}}[p:p]=0$ when $p\in \mathcal{R}$. When $p,q\in \mathcal{R}$ with $p\ne q$, the value $D_{\mathrm{vJS}}^{\mathcal{R}}[p:q]$ corresponds to the information radius (and the arg min to the right-sided Kullback–Leibler centroid).

Figure 1. Illustrating several cases of the relative Jensen-Shannon divergence based on whether $p\in \mathcal{R}$ and $q\in \mathcal{R}$ or not.

4.2. Relative Jensen-Shannon Divergences: Applications to Density Clustering and Quantization

Let $D_{\mathrm{KL}}[p:q_{\theta }]$ be the Kullback–Leibler divergence between an arbitrary density p and a density $q_{\theta }$ of an exponential family $\mathcal{Q}=\{q_{\theta }:\theta \in \Theta \}$. Let us canonically express [2,60] the density $q_{\theta }\left(x\right)$, as

$$q_{\theta }\left(x\right)=exp\left({\theta }^{\top }{t}_{\mathcal{Q}}\left(x\right)-F_{\mathcal{Q}}\left(\theta \right)+k_{\mathcal{Q}}\left(x\right)\right),$$

where $t_{\mathcal{Q}}\left(x\right)$ denotes the sufficient statistics, $k_{\mathcal{Q}}\left(x\right)$ is an auxiliary carrier measure term (e.g., $k\left(x\right)=0$ for the Gaussian family and $k\left(x\right)=log\left(x\right)$ for the Rayleigh family [60]), and $F_{\mathcal{Q}}\left(\theta \right)$ the cumulant function. Assume that we know in closed-form the following quantities:

• $m_p:=E_p\left[t_{\mathcal{Q}}\left(x\right)\right]=\int p\left(x\right)t_{\mathcal{Q}}\left(x\right)\mathrm{d}\mu \left(x\right)$ and
• the Shannon entropy $h\left[p\right]=-\int p\left(x\right)logp\left(x\right)\mathrm{d}\mu \left(x\right)$ of p.

Subsequently, we can express the KLD using a semi-closed-form formula.

Proposition 4.

Let $q_{\theta }\in \mathcal{Q}$ be a density of an exponential family and p an arbitrary density with $m_p=E_p\left[t_{\mathcal{Q}}\left(x\right)\right]$. Subsequently, the Kullback–Leibler divergence between p and $q_{\theta }$ is expressed as:

$$D_{\mathrm{KL}}[p:q_{\theta }]=F_{\mathcal{Q}}\left(\theta \right)-m_p^{\top }\theta -E_p\left[k_{\mathcal{Q}}\left(x\right)\right]-h\left[p\right],$$

(102)

where $h[p:q_{\theta }]=F_{\mathcal{Q}}\left(\theta \right)-m_p^{\top }\theta -E_p\left[k_{\mathcal{Q}}\left(x\right)\right]$ is the cross-entropy between p and $q_{\theta }$.

Proof. 

The proof is straightforward since $log{q}_{\theta }\left(x\right)={\theta }^{\top }{t}_{\mathcal{Q}}\left(x\right)-F_{\mathcal{Q}}\left(\theta \right)+k_{\mathcal{Q}}\left(x\right)$. Therefore, we have:

$$\begin{array}{ccc}\hfill D_{\mathrm{KL}}[p:q_{\theta }]& =& h[p:q_{\theta }]-h\left[p\right],\hfill \end{array}$$

(103)

$$\begin{array}{ccc}& =& -{\int }_{\mathcal{X}}p\left(x\right)log{q}_{\theta }\left(x\right)\mathrm{d}\mu \left(x\right)-h\left[p\right],\hfill \end{array}$$

(104)

$$\begin{array}{ccc}& =& F_{\mathcal{Q}}\left(\theta \right)-m_p^{\top }\theta -E_p\left[k_{\mathcal{Q}}\left(x\right)\right]-h\left[p\right].\hfill \end{array}$$

(105)

 □

Example 7.

For example, when $q_{\theta }=q_{\mu ,\Sigma }$ is the density of a multivariate Gaussian distribution $\mathcal{N}(\mu ,\Sigma )$ (with $k_{\mathcal{N}}\left(x\right)=0$), we have

$$D_{\mathrm{KL}}[p:q_{\mu ,\Sigma }]=\frac{1}{2}\left(log\left|2\pi \Sigma \right|+{(\mu -m)}^{\top }{\Sigma }^{-1}(\mu -m)+\mathrm{tr}\left({\Sigma }^{-1}S\right)\right)-h\left[p\right],$$

(106)

where $m=\mu \left(p\right)=E_p\left[X\right]$ and $S=\mathrm{Cov}\left(p\right):=E_p\left[X{X}^{\top }\right]-E_p\left[X\right]E_p{\left[X\right]}^{\top }$.

The formula of Proposition 4 is said in semi-closed-form, because it relies on knowing both the entropy h of p and the sufficient statistic moments $E_p\left[t_{\mathcal{Q}}\left(x\right)\right]$. Yet, this semi-closed formula may prove to be useful in practice: For example, we can answer the comparison predicate

“Is $D_{\mathrm{KL}}[p:q_{{\theta }_1}]\ge D_{\mathrm{KL}}[p:q_{{\theta }_2}]$ or not?”

by checking whether $F_{\mathcal{Q}}\left({\theta }_1\right)-F_{\mathcal{Q}}\left({\theta }_2\right)-m_p^{\top }({\theta }_1-{\theta }_2)\ge 0$ or not (i.e., the terms $-E_p\left[k_{\mathcal{Q}}\left(x\right)\right]-h\left[p\right]$ in Equation (102) cancel out). Thus, we get a closed-form predicate, although $D_{\mathrm{KL}}$ is only known in semi-closed-form. This KLD comparison predicate shall be used later on when clustering densities with respect to centroids in Section 4.2.

Remark 1.

Note that when $Y=f\left(X\right)$ for an invertible and differentiable transformation f then we have $h\left[Y\right]=h\left[X\right]+E_X[log|J_f\left(X\right)\left|\right]$ where $J_f$ denotes the Jacobian matrix. For example, when $Y=f\left(X\right)=AX$, we have $h\left[Y\right]=h\left[X\right]+log\left|A\right|$.

When p belongs to an exponential family $\mathcal{P}$ ($\mathcal{P}$ may be different from $\mathcal{Q}$) with cumulant function $F_{\mathcal{P}}$, sufficient statistics $t_{\mathcal{P}}\left(x\right)$, auxiliary carrier term $k_{\mathcal{P}}\left(x\right)$, and natural parameter $\theta$, we have the entropy [61] expressed, as follows:

$$\begin{array}{ccc}\hfill h\left[p\right]& =& F_{\mathcal{P}}\left(\theta \right)-{\theta }^{\top }\nabla F_{\mathcal{P}}\left(\theta \right)-E_p\left[k_{\mathcal{P}}\left(x\right)\right],\hfill \end{array}$$

(107)

$$\begin{array}{ccc}& =& -F_{\mathcal{P}}^{*}\left(\eta \right)-E_p\left[k_{\mathcal{P}}\left(x\right)\right],\hfill \end{array}$$

(108)

where $F_{\mathcal{P}}^{*}\left(\eta \right)={\theta }^{\top }\nabla F\left(\theta \right)-F\left(\theta \right)$ is the Legendre transform of $F\left(\theta \right)$ and $\eta =\eta \left(\theta \right)=\nabla F\left(\theta \right)$ is called the moment parameter since we have $\eta \left(\theta \right)=E_p\left[t_{\mathcal{P}}\left(x\right)\right]$ [2,60].

It follows the following proposition refining Proposition 4 when $p=p_{\theta }\in \mathcal{P}$:

Proposition 5.

Let $p_{\theta }$ be a density of an exponential family $\mathcal{P}$ and $q_{{\theta }^{\prime }}$ be a density of an exponential family $\mathcal{Q}$. Subsequently, the Kullback–Leibler divergence between $p_{\theta }$ and $q_{{\theta }^{\prime }}$ is expressed as:

$$D_{\mathrm{KL}}[p_{\theta }:q_{{\theta }^{\prime }}]=F_{\mathcal{Q}}\left({\theta }^{\prime }\right)+F_{\mathcal{P}}^{*}\left(\eta \right)-E_{p_{\theta }}{\left[t_{\mathcal{Q}}\left(x\right)\right]}^{\top }{\theta }^{\prime }+E_{p_{\theta }}[k_{\mathcal{P}}\left(x\right)-k_{\mathcal{Q}}\left(x\right)].$$

(109)

Proof. 

We have

$$\begin{array}{ccc}\hfill D_{\mathrm{KL}}[p_{\theta }:q_{{\theta }^{\prime }}]& =& h[p_{\theta }:q_{{\theta }^{\prime }}]-h\left[p_{\theta }\right],\hfill \end{array}$$

(110)

$$\begin{array}{ccc}& =& F_{\mathcal{Q}}\left({\theta }^{\prime }\right)-m_{p_{\theta }}^{\top }{\theta }^{\prime }-E_{p_{\theta }}\left[k_{\mathcal{Q}}\left(x\right)\right]+F_{\mathcal{P}}^{*}\left(\eta \right)+E_{p_{\theta }}\left[k_{\mathcal{P}}\left(x\right)\right],\hfill \end{array}$$

(111)

$$\begin{array}{ccc}& =& F_{\mathcal{Q}}\left({\theta }^{\prime }\right)+F_{\mathcal{P}}^{*}\left(\eta \right)-E_{p_{\theta }}{\left[t_{\mathcal{Q}}\left(x\right)\right]}^{\top }{\theta }^{\prime }+E_{p_{\theta }}[k_{\mathcal{P}}\left(x\right)-k_{\mathcal{Q}}\left(x\right)].\hfill \end{array}$$

(112)

 □

In particular, when p and q belong both to the same exponential family (i.e., $\mathcal{P}=\mathcal{Q}$ with $k_{\mathcal{P}}\left(x\right)=k_{\mathcal{Q}}\left(x\right)$), we have $F\left(\theta \right):=F_{\mathcal{P}}\left(\theta \right):=F_{\mathcal{Q}}\left(\theta \right)$ and $E_{p_{\theta }}\left[t_{\mathcal{Q}}\left(x\right)\right]=\nabla F\left(\theta \right)=:\eta$, and

$$D_{\mathrm{KL}}[p_{\theta }:q_{{\theta }^{\prime }}]=F\left({\theta }^{\prime }\right)+F^{*}\left(\eta \right)-{\theta }^{\prime \top }\eta .$$

This last equation is the Fenchel–Young divergence in Bregman manifolds [34,62] (called dually flat spaces in information geometry [10]). Thus the divergence can be rewritten as equivalent dual Bregman divergences:

$$\begin{array}{ccc}\hfill D_{\mathrm{KL}}[p_{\theta }:q_{{\theta }^{\prime }}]& =& F\left({\theta }^{\prime }\right)+F^{*}\left(\eta \right)-{\eta }^{\top }{\theta }^{\prime },\hfill \end{array}$$

(113)

$$\begin{array}{ccc}& =& B_F({\theta }^{\prime }:\theta ),\hfill \end{array}$$

(114)

$$\begin{array}{ccc}& =& B_{F^{*}}(\eta :{\eta }^{\prime }),\hfill \end{array}$$

(115)

where ${\eta }^{\prime }=\nabla F\left({\theta }^{\prime }\right)$.

Notice that $D_{\mathrm{KL}}[p_{\theta }:\mathcal{Q}]:={min}_{{\theta }^{\prime }\in {\Theta }^{\prime }}{D}_{\mathrm{KL}}[p_{\theta }:q_{{\theta }^{\prime }}]$ is unique and can be calculated as ${\eta }^{\prime }=\nabla F_{\mathcal{Q}}\left({\theta }^{\prime }\right)=E_{p_{\theta }}\left[t_{\mathcal{Q}}\left(x\right)\right]$.

Let us report two examples of calculations of the KLD between two densities of two exponential families.

Example 8.

For the first exponential family, consider the family of Laplacian distributions:

$$\mathcal{P}=\mathcal{L}=\left\{p_{\sigma }\left(x\right):=\frac{1}{2\sigma }exp\left(-\frac{\left|x\right|}{\sigma }\right):\sigma >0\right\}.$$

The canonical decomposition of the density yields $t_{\mathcal{L}}\left(x\right)=\left|x\right|$, $\theta =-\frac{1}{\sigma }$, $k_{\mathcal{L}}\left(x\right)=0$, and $F_{\mathcal{L}}\left(\theta \right)=log\frac{2}{-\theta }$. (i.e., $F_{\mathcal{L}}\left(\theta \left(\sigma \right)\right)=log2\sigma$). It follows that $\eta \left(\theta \right)=F_{\mathcal{L}}^{\prime }\left(\theta \right)=-\frac{1}{\theta }$ ($\eta \left(\sigma \right)=\sigma =E\left[\right|x\left|\right]$), $\theta \left(\eta \right)=-\frac{1}{\eta }$, and $F_{\mathcal{L}}^{*}\left(\eta \right)=-1-log\left(2\eta \right)$ and, therefore, $F_{\mathcal{L}}^{*}\left(\eta \left(\sigma \right)\right)=-1-log\left(2\sigma \right)$.

For the second family, consider the exponential family of zero-centered Gaussian distributions:

$$\mathcal{Q}={\mathcal{N}}_0=\left\{q_{{\sigma }^{\prime }}\left(x\right)=\frac{1}{\sqrt{2\pi {\left({\sigma }^{\prime }\right)}^2}}exp\left(-\frac{x^2}{2{\left({\sigma }^{\prime }\right)}^2}\right)\right\}.$$

We have $t_{{\mathcal{N}}_0}\left(x\right)=x^2$, $k_{{\mathcal{N}}_0}\left(x\right)=0$, ${\theta }^{\prime }=-\frac{1}{2{\left({\sigma }^{\prime }\right)}^2}$, and $F_{{\mathcal{N}}_0}\left({\sigma }^{\prime }\right)=\frac{1}{2}log\left(2\pi {\left({\sigma }^{\prime }\right)}^2\right)$.

Moreover, let us calculate $E_{p_{\sigma }}\left[t_{{\mathcal{N}}_0}\left(x\right)\right]=E_{p_{\sigma }}\left[x^2\right]=2{\sigma }^2$. Subsequently, we can calculate the Kullback–Leibler divergence between $p_{\sigma }\sim \mathcal{L}\left(\sigma \right)$ and $q_{{\sigma }^{\prime }}\sim {\mathcal{N}}_0\left({\sigma }^{\prime }\right)$, as follows:

$$\begin{array}{ccc}\hfill D_{\mathrm{KL}}[p_{\sigma }:q_{{\sigma }^{\prime }}]& =& F_{\mathcal{Q}}\left({\theta }^{\prime }\left({\sigma }^{\prime }\right)\right)+F_{\mathcal{P}}^{*}\left(\eta \left(\sigma \right)\right)-E_{p_{\sigma }}{\left[t_{\mathcal{Q}}\left(x\right)\right]}^{\top }{\theta }^{\prime }\left({\sigma }^{\prime }\right)+E_{p_{\sigma }}[k_{\mathcal{P}}\left(x\right)-k_{\mathcal{Q}}\left(x\right)],\hfill \end{array}$$

(116)

$$\begin{array}{ccc}& =& \frac{1}{2}log\left(2\pi {\left({\sigma }^{\prime }\right)}^2\right)-1-log\left(2\sigma \right)-2{\sigma }^2\left(-\frac{1}{2{\left({\sigma }^{\prime }\right)}^2}\right),\hfill \end{array}$$

(117)

$$\begin{array}{ccc}& =& log\left(\frac{{\sigma }^{\prime }}{\sigma }\right)+{\left(\frac{\sigma }{{\sigma }^{\prime }}\right)}^2+\frac{1}{2}log\left(\frac{\pi }{2}\right)-1.\hfill \end{array}$$

(118)

Notice that $D_{\mathrm{KL}}[p_{\sigma }:q_{{\sigma }^{\prime }}]\ge 0$, but never 0 since the $\mathcal{P}\cap \mathcal{Q}=\varnothing$.

Let us now compute the reverse Kullback–Leibler divergence $D_{\mathrm{KL}}[q_{{\sigma }^{\prime }}:p_{\sigma }]$. We first calculate $E_{q_{{\sigma }^{\prime }}}\left[t_{\mathcal{L}}\left(x\right)\right]=E_{q_{{\sigma }^{\prime }}\left({\sigma }^{\prime }\right)}\left[\right|x\left|\right]=\sqrt{\frac{2}{\pi }}{\sigma }^{\prime }$. Since $F_{\mathcal{Q}}\left({\theta }^{\prime }\right)=\frac{1}{2}log\left(\frac{\pi }{-{\theta }^{\prime }}\right)$, we have ${\eta }^{\prime }\left({\theta }^{\prime }\right)=F_{\mathcal{Q}}^{\prime }\left({\theta }^{\prime }\right)=-\frac{1}{2{\theta }^{\prime }}$. Thus ${\eta }^{\prime }\left({\sigma }^{\prime }\right)={\left({\sigma }^{\prime }\right)}^2$ and $F_{\mathcal{Q}}^{*}\left({\eta }^{\prime }\right)=-\frac{1}{2}-\frac{1}{2}log\left(2\pi \eta \right)$. Therefore, we get $F_{\mathcal{Q}}^{*}\left({\eta }^{\prime }\left({\sigma }^{\prime }\right)\right)=-h\left[q_{{\sigma }^{\prime }}\right]=-\frac{1}{2}log\left(2\pi e{\left({\sigma }^{\prime }\right)}^2\right)$.

It follows that

$$\begin{array}{ccc}\hfill D_{\mathrm{KL}}[q_{{\sigma }^{\prime }}:p_{\sigma }]& =& F_{\mathcal{P}}\left(\theta \left(\sigma \right)\right)+F_{\mathcal{Q}}^{*}\left({\eta }^{\prime }\left({\sigma }^{\prime }\right)\right)-E_{q_{{\theta }^{\prime }}}{\left[t_{\mathcal{P}}\left(x\right)\right]}^{\top }\theta \left(\sigma \right)+E_{q_{{\theta }^{\prime }}}[k_{\mathcal{P}}\left(x\right)-k_{\mathcal{Q}}\left(x\right)],\hfill \end{array}$$

(119)

$$\begin{array}{ccc}& =& log\left(2\sigma \right)-\frac{1}{2}log\left(2\pi e{\left({\sigma }^{\prime }\right)}^2\right)-\sqrt{\frac{2}{\pi }}{\sigma }^{\prime }\times \left(-\frac{1}{\sigma }\right),\hfill \end{array}$$

(120)

$$\begin{array}{ccc}& =& \sqrt{\frac{2}{\pi }}\frac{{\sigma }^{\prime }}{\sigma }+log\left(\frac{\sigma }{{\sigma }^{\prime }}\right)-\frac{1}{2}log\left(\frac{\pi }{2}e\right).\hfill \end{array}$$

(121)

Again, we have $D_{\mathrm{KL}}[q_{{\sigma }^{\prime }}:p_{\sigma }]\ge 0$, but never 0, because $\mathcal{P}\cap \mathcal{Q}=\varnothing$.

Example 9.

Let us use the formula of Equation (109) to calculate the KLD between two Weibull distributions [63]. A Weibull distribution of shape $\kappa >0$ and scale $\sigma >0$ has a density defined on $\mathcal{X}=[0,\infty )$, as follows:

$$p_{\kappa ,\sigma }^{\mathrm{Wei}}\left(x\right):=\frac{\kappa }{\sigma }{\left(\frac{x}{\sigma }\right)}^{\kappa -1}exp\left(-{\left(\frac{x}{\sigma }\right)}^{\kappa }\right).$$

For a fixed shape κ, the set of Weibull distributions ${\mathcal{W}}_{\kappa }:=\{p_{\kappa ,\sigma }^{\mathrm{Wei}}:\sigma >0\}$ form an exponential family with natural parameter $\theta =-\frac{1}{{\sigma }^{\kappa }}$, sufficient statistic $t_{\kappa }\left(x\right)=x^{\kappa }$, auxiliary carrier term $k_{\kappa }\left(x\right)=(\kappa -1)logx+log\kappa$, and cumulant function $F_{\kappa }\left(\theta \right)=-log(-\theta )$ (so that $F_{\kappa }\left(\theta \left(\sigma \right)\right)=F_{\kappa }\left(\sigma \right)=\kappa log\sigma$):

$$p_{\kappa ,\sigma }^{\mathrm{Wei}}\left(x\right):=exp\left(-\frac{1}{{\sigma }^{\kappa }}{x}^k+log\frac{1}{{\sigma }^{\kappa }}+k\left(x\right)\right).$$

We recover the exponential family of exponential distributions of rate parameter $\lambda =\frac{1}{\sigma }$ when $\kappa =1$:

$$\begin{array}{ccc}\hfill p_{\lambda }^{\mathrm{Exp}}\left(x\right)& =& p_{1,\sigma }^{\mathrm{Wei}}\left(x\right)=\frac{1}{\sigma }exp\left(-\frac{x}{\sigma }\right),\hfill \\ & =& \lambda exp\left(-\lambda x\right),\hfill \end{array}$$

and the exponential family of Rayleigh distributions when $\kappa =2$ with scale parameter ${\sigma }_{\mathrm{Ray}}=\frac{\sigma }{\sqrt{2}}$:

$$\begin{array}{ccc}\hfill p_{{\sigma }_{\mathrm{Ray}}}^{\mathrm{Ray}}\left(x\right)& =& p_{2,\sigma }^{\mathrm{Wei}}\left(x\right)=\frac{2x}{{\sigma }^2}exp\left(-\frac{x^2}{{\sigma }^2}\right),\hfill \\ & =& \frac{x}{{\sigma }_{\mathrm{Ray}}^2}exp\left(-\frac{x^2}{2{\sigma }_{\mathrm{Ray}}^2}\right).\hfill \end{array}$$

Now, assume that we are given the differential entropy of the Weibull distributions [64] (pp. 155–156):

$$h\left[p_{{\kappa }_1,{\sigma }_1}^{\mathrm{Wei}}\right]=\gamma \left(1-\frac{1}{{\kappa }_1}\right)+log\frac{{\sigma }_1}{{\kappa }_1}+1,$$

where $\gamma \approx 0.5772156649$ is the Euler–Mascheroni constant, and the Weibull raw moments [64] (p. 155):

$$m=E_{p_{{\kappa }_1,\sigma }^{\mathrm{Wei}}}\left[x^{{\kappa }_2}\right]={\sigma }_1^{{\kappa }_2}\Gamma \left(1+\frac{{\kappa }_2}{{\kappa }_1}\right),$$

where $\Gamma \left(x\right)={\int }_0^{\infty }{t}^{x-1}{e}^{-t}\mathrm{d}t$ is the gamma function (with $\Gamma \left(n\right)=(n-1)!$ for integers n). Because $h\left[p_{\kappa ,\sigma }^{\mathrm{Wei}}\right]=F_{\kappa }\left(\theta \right)-{\theta }^{\top }\nabla F_{\kappa }\left(\theta \right)-E_{p_{\kappa ,\sigma }^{\mathrm{Wei}}}\left[k_{\kappa }\left(x\right)\right]=-F_{\kappa }^{*}\left(\eta \right)-E_{p_{\kappa ,\sigma }^{\mathrm{Wei}}}\left[k_{\kappa }\left(x\right)\right]$, we deduce that

$$E_{p_{\kappa ,\sigma }^{\mathrm{Wei}}}\left[k_{\kappa }\left(x\right)\right]=-F_{\kappa }^{*}\left(\eta \right)-h\left[p_{\kappa ,\sigma }^{\mathrm{Wei}}\right],$$

where $F_{\kappa }^{*}\left(\eta \right)$ is the Legendre transform of $F_{\kappa }\left(\theta \right)$ and $\eta \left(\theta \right)=\nabla F_{\kappa }\left(\theta \right)=-\frac{1}{\theta }=E\left[t\left(x\right)\right]=E\left[x^{\kappa }\right]$. We have $\theta \left(\eta \right)=\nabla F_{\kappa }^{*}\left(\eta \right)=-\frac{1}{\eta }$ and $F_{\kappa }^{*}\left(\eta \right)={\eta }^{\top }\nabla F_{\kappa }^{*}\left(\eta \right)-F_{\kappa }(\nabla F_{\kappa }^{*}\left(\eta \right))=-1-log\eta$. It follows that

$$E_{p_{\kappa ,\sigma }^{\mathrm{Wei}}}\left[k_{\kappa }\left(x\right)\right]=1+log\left(\sigma \Gamma \left(1+\frac{1}{\kappa }\right)\right)-\gamma \left(1-\frac{1}{\kappa }\right)-log\frac{\sigma }{\kappa }+1.$$

Therefore, we deduce that the logarithmic moment of $p_{{\kappa }_1,\sigma }^{\mathrm{Wei}}$ is:

$$E_{p_{{\kappa }_1,\sigma }^{\mathrm{Wei}}}[logx]=-\frac{\gamma }{{\kappa }_1}+log{\sigma }_1.$$

This coincides with the explicit definite integral calculation reported in [63].

Subsequently, we calculate the KLD between two Weibull distributions using Equation (109), as follows:

$$\begin{array}{ccc}\hfill D_{\mathrm{KL}}\left[p_{{\kappa }_1,{\sigma }_1}^{\mathrm{Wei}}:p_{{\kappa }_2,{\sigma }_2}^{\mathrm{Wei}}\right]& =& F_{{\kappa }_2}\left({\theta }^{\prime }\right)+F_{{\kappa }_1}^{*}\left(\eta \right)-E_{p_{{\kappa }_1,{\sigma }_1}}{\left[x^{{\kappa }_2}\right]}^{\top }{\theta }^{\prime }+E_{p_{{\kappa }_1,{\sigma }_1}}[k_{{\kappa }_1}\left(x\right)-k_{{\kappa }_2}\left(x\right)]\hfill \end{array}$$

(122)

$$\begin{array}{ccc}& =& log\frac{{\kappa }_1}{{\sigma }_1^{{\kappa }_1}}-log\frac{{\kappa }_2}{{\sigma }_2^{{\kappa }_2}}+\left({\kappa }_1-{\kappa }_2\right)\left[log{\sigma }_1-\frac{\gamma }{{\kappa }_1}\right]+{\left(\frac{{\sigma }_1}{{\sigma }_2}\right)}^{{\kappa }_2}\Gamma \left(\frac{{\kappa }_2}{{\kappa }_1}+1\right)-1\hfill \\ \hfill \end{array}$$

(123)

since we have the following terms:

$$\begin{array}{ccc}\hfill F_{{\kappa }_2}\left({\theta }^{\prime }\right)& =& log{\sigma }_2^{{\kappa }_2},\hfill \\ \hfill F_{{\kappa }_1}^{*}\left(\eta \right)& =& -1-log{\sigma }_1^{{\kappa }_1},\hfill \\ \hfill -E_{p_{{\kappa }_1,{\sigma }_1}}{\left[x^{{\kappa }_2}\right]}^{\top }{\theta }^{\prime }& =& \frac{1}{{\sigma }_2^{{\kappa }_2}}{\sigma }_1^{{\kappa }_2}\Gamma \left(1+\frac{{\kappa }_2}{{\kappa }_1}\right)\hfill \\ \hfill E_{p_{{\kappa }_1,{\sigma }_1}}[k_{{\kappa }_1}\left(x\right)-k_{{\kappa }_2}\left(x\right)]& =& ({\kappa }_1-{\kappa }_2)E_{p_{{\kappa }_1,{\sigma }_1}}[logx]+log\frac{{\kappa }_1}{{\kappa }_2},\hfill \\ & =& log\frac{{\kappa }_1}{{\kappa }_2}+({\kappa }_1-{\kappa }_2)\left(log{\sigma }_1-\frac{\gamma }{{\kappa }_1}\right).\hfill \end{array}$$

This formula matches the formula reported in [63].

When ${\kappa }_1={\kappa }_2=1$, we recover the ordinary KLD formula between two exponential distributions [60] with ${\lambda }_i=\frac{1}{{\sigma }_i}$ since $\Gamma \left(2\right)=(2-1)!=1$:

$$\begin{array}{ccc}\hfill D_{\mathrm{KL}}\left[p_{1,{\sigma }_1}^{\mathrm{Wei}}:p_{1,{\sigma }_2}^{\mathrm{Wei}}\right]& =& log\frac{{\sigma }_2}{{\sigma }_1}+\frac{{\sigma }_1}{{\sigma }_2}-1,\hfill \end{array}$$

(124)

$$\begin{array}{ccc}& =& \frac{{\lambda }_2}{{\lambda }_1}-log\frac{{\lambda }_2}{{\lambda }_1}-1.\hfill \end{array}$$

(125)

When ${\kappa }_1={\kappa }_2=2$, we recover the ordinary KLD formula between two Rayleigh distributions [60], with ${\sigma }_{\mathrm{Ray}}=\frac{\sigma }{\sqrt{2}}$:

$$\begin{array}{ccc}\hfill D_{\mathrm{KL}}\left[p_{2,{\sigma }_1}^{\mathrm{Wei}}:p_{2,{\sigma }_2}^{\mathrm{Wei}}\right]& =& log\left(\frac{{\sigma }_2^2}{{\sigma }_1^2}\right)+\frac{{\sigma }_1^2}{{\sigma }_2^2}-1,\hfill \end{array}$$

(126)

$$\begin{array}{ccc}& =& log\left(\frac{{{\sigma }_{\mathrm{Ray}}}_2^2}{{{\sigma }_{\mathrm{Ray}}}_1^2}\right)+\frac{{{\sigma }_{\mathrm{Ray}}}_1^2}{{{\sigma }_{\mathrm{Ray}}}_2^2}-1.\hfill \end{array}$$

(127)

The formulae of Equations (127) and (126) are linked by the fact that if $X\sim \mathrm{Exp}\left(\lambda \right)$ and $Y=\sqrt{X}$ then $Y\sim \mathrm{Ray}\left(\frac{1}{\sqrt{2\lambda }}\right)$, and f-divergences [65], including the Kullback–Leibler divergence are invariant by a differentiable transformation [66].

Jeffreys’ divergence symmetrizes the KLD divergence, as follows:

$$D_J[p:q]:=D_{\mathrm{KL}}[p:q]+D_{\mathrm{KL}}[q:p]=2A(D_{\mathrm{KL}}[p:q],D_{\mathrm{KL}}[q:p]).$$

(128)

The Jeffreys divergence between two densities of different exponential families $\mathcal{P}$ and $\mathcal{Q}$ is

$$D_J[p_{\theta }:q_{{\theta }^{\prime }}]={\theta }^{\prime \top }({\eta }^{\prime }-E_{p_{\theta }}\left[t_{\mathcal{Q}}\left(x\right)\right])+{\theta }^{\top }(\eta -E_{q_{{\theta }^{\prime }}}\left[t_{\mathcal{P}}\left(x\right)\right])+E_{p_{\theta }}[k_{\mathcal{P}}\left(x\right)-k_{\mathcal{Q}}\left(x\right)]+E_{q_{{\theta }^{\prime }}}[k_{\mathcal{Q}}\left(x\right)-k_{\mathcal{P}}\left(x\right)].$$

(129)

When $\mathcal{P}=\mathcal{Q}$, we have $E_{p_{\theta }}\left[t_{\mathcal{Q}}\left(x\right)\right]=\eta$ and $E_{q_{{\theta }^{\prime }}}\left[t_{\mathcal{P}}\left(x\right)\right])={\eta }^{\prime }$, so that we find the usual expression of the Jeffreys divergence between two densities of an exponential family:

$$D_J[p_{\theta }:p_{{\theta }^{\prime }}]={({\theta }^{\prime }-\theta )}^{\top }({\eta }^{\prime }-\eta ).$$

(130)

To find the best density $q_{\theta }$ approximating p by minimizing ${min}_{\theta }{D}_{\mathrm{KL}}[p:q_{\theta }]$, we solve $\nabla F\left(\theta \right)=\eta =m$ and, therefore, $\theta =\nabla F^{*}\left(m\right)={(\nabla F)}^{-1}\left(m\right)$, where $F^{*}\left(\eta \right)=E_{q_{\eta }}[log{q}_{\eta }\left(m\right)]$, with $F^{*}$ denoting the Legendre–Fenchel convex conjugate [2]. In particular, when $p=\sum w_i{p}_{{\theta }_i}$ is a mixture of EFs (with $m=E_p\left[t\left(x\right)\right]=\sum w_i{\eta }_i$ with ${\eta }_i=E_{p_{{\theta }_i}}\left[t\left(x\right)\right]$ thanks to the linearity of the expectation), then the best density of the EF simplifying p is

$$\begin{array}{ccc}\hfill \underset{\theta }{min}{D}_{\mathrm{KL}}[p:q_{\theta }]& =& \underset{\theta }{min}F\left(\theta \right)-m^{\top }\theta ,\hfill \end{array}$$

(131)

$$\begin{array}{ccc}& =& \underset{\theta }{min}F\left(\theta \right)-\sum w_i{\eta }_i^{\top }\theta .\hfill \end{array}$$

(132)

Taking the gradient with respect to $\theta$, we have $\nabla F\left(\theta \right)=\eta =\sum w_i{\eta }_i$. This yields another proof without the Pythagoras theorem [67,68].

Proposition 6.

Let $m\left(x\right)=\sum w_i{p}_{{\theta }_i}\left(x\right)$ be a mixture with components that belong to an exponential family with cumulant function F. Subsequently, ${\theta }^{*}={arg}_{\theta }{min}_{\theta }{D}_{\mathrm{KL}}[p:q_{\theta }]$ is $\nabla F^{*}\left({\sum }_{i=1}^n{w}_i{\eta }_i\right)$, where the ${\eta }_i=\nabla F\left({\theta }_i\right)$ are the moment parameters of the mixture components.

Consider the following two problems:

Problem 1

(Density clustering). Given a set of n weighted densities $(w_1,p_1),\dots ,(w_n,p_n)$, partition them into k clusters ${\mathcal{C}}_1,\dots ,{\mathcal{C}}_k$ in order to minimize the k-centroid objective function with respect to a statistical divergence D: ${\sum }_{i=1}^n{w}_i{min}_{l\in \{1,\dots ,k\}}D[p_i:c_l]$, where $c_l$ denotes the centroid of cluster ${\mathcal{C}}_l$ for $l\in \{1,\dots ,k\}$.

For example, when all the densities $p_i$’s are isotropic Gaussians, we recover the k-means objective function [69].

Problem 2

(Mixture component quantization). Given a statistical mixture $m\left(x\right)={\sum }_{i=1}^n{w}_i$$p_i\left(x\right)$, quantize the mixture components into k densities $q_1,\dots ,q_k$ in order to minimize ${\sum }_i{w}_i$${min}_{l\in \{1,\dots ,k\}}D[p_i:q_l]$.

Notice that, in Problem 1, the input densities $p_i$’s may be mixtures, i.e., $p_i\left(x\right)={\sum }_{j=1}^{n_i}{w}_{i,j}{p}_{i,j}\left(x\right)$. Using the relative information radius, we can cluster a set of distributions (potentially mixtures) into an exponential family mixture, or quantize an exponential family mixture. Indeed, we can implement an extension of k-means [69] with k-centers $q_{{\theta }_i}$, to assign density $p_i$ to cluster $C_j$ (with center $q_j$), we need to perform basic comparison tests $D_{\mathrm{KL}}[p_i:q_{{\theta }_l}]\ge D_{\mathrm{KL}}[p_i:q_{{\theta }_j}]$. Provided that the cumulant F of the exponential family is in closed-form, we do not need formula for the entropies $h\left(p_i\right)$.

Clustering and quantization of densities/mixtures have been widely studied in the literature, see, for example, [70,71,72,73,74,75,76].

5. Conclusions

To summarize, the ordinary Jensen-Shannon divergence has been defined in three equivalent ways in the literature:

$$\begin{array}{ccc}\hfill D_{\mathrm{JS}}[p,q]& :=& \underset{c\in \mathcal{D}}{min}\frac{1}{2}\left(D_{\mathrm{KL}}[p:c]+D_{\mathrm{KL}}[q:c]\right),\hfill \end{array}$$

(133)

$$\begin{array}{ccc}& =& \frac{1}{2}\left(D_{\mathrm{KL}}\left[p:\frac{p+q}{2}\right]+D_{\mathrm{KL}}\left[q:\frac{p+q}{2}\right]\right),\hfill \end{array}$$

(134)

$$\begin{array}{ccc}& =& h\left[\frac{p+q}{2}\right]-\frac{h\left[p\right]+h\left[q\right]}{2}.\hfill \end{array}$$

(135)

The JSD Equation (133) was studied by Sibson in 1969 within the wider scope of information radius [1]: Sibson relied on the Rényi $\alpha$-divergences (relative Rényi $\alpha$-entropies [77]) and recovered the ordinary Jensen-Shannon divergence as a particular case of the $\alpha$-information radius when $\alpha =1$ and $n=2$ points. The $\alpha$-information radii are related to generalized Bhattacharyya distances with respect to power means and the total variation distance in the limit case of $\alpha =\infty$.

Lin [4] investigated the JSD Equation (134) in 1991 with its connection to the JSD defined in Equation (134)). In Lin [4], the JSD is interpreted as the arithmetic symmetrization of the K-divergence [24]. Generalizations of the JSD based on Equation (134) were proposed in [23] using a generic mean instead of the arithmetic mean. One motivation was to obtain a closed-form formula for the geometric JSD between multivariate Gaussian distributions, which relies on the geometric mixture (see [30] for a use case of that formula in deep learning). Indeed, the ordinary JSD between Gaussians is not available in closed-form (not analytic). However, the JSD between Cauchy distributions admit a closed-form formula [78], despite the calculation of a definite integral of a log-sum term. Instead of using an abstract mean to define a mid-distribution of two densities, one may also consider the mid-point of a geodesic linking these two densities (the arithmetic means $\frac{p+q}{2}$ is interpreted as a geodesic midpoint). Recently, Li [79] investigated the transport Jensen-Shannon divergence as a symmetrization of the Kullback–Leibler divergence in the $L^2$-Wasserstein space. See Section 5.4 of [79] and the closed-form formula of Equation (18) obtained for the transport Jensen-Shannon divergence between two multivariate Gaussian distributions.

The generalization of the identity between the JSD of Equation (134) and the JSD of Equation (135) was studied while using a skewing vector in [18]. Although the JSD is a f-divergence [8,18], the Sibson-M Jensen-Shannon symmetrization of a distance does not belong, in general, to the class of f-divergences. The variational JSD definition of Equation (133) is implicit, while the definitions of Equations (134) and (135) are explicit because the unique optimal centroid $c^{*}=\frac{p+q}{2}$ has been plugged into the objective function that was minimized by Equation (133).

In this paper, we proposed a generalization of the Jensen-Shannon divergence based on the variational definition of the ordinary Jensen-Shannon divergence based on the variational JSD definition of Equation (133): $D_{\mathrm{vJS}}[p:q]={min}_c\frac{1}{2}(D_{\mathrm{KL}}[p:c]+D_{\mathrm{KL}}[q:c])$. We introduced the Jensen-Shannon symmetrization of an arbitrary divergence D by considering a generalization of the information radius with respect to an abstract weighted mean $M_{\beta }$: $D_M^{\mathrm{vJS}}[p:q]:={min}_c{M}_{\beta }(D[p:c],D[q:c])$. Notice that, in the variational JSD, the mean $M_{\beta }$ is used for averaging divergence values, while the mean $M_{\alpha }$ in the $(M_{\alpha },N_{\beta })$ JSD is used to define generic statistical mixtures. We also consider relative variational JS symmetrization when the centroid has to belong to a prescribed family of densities. For the case of exponential family, we showed how to compute the relative centroid in closed form, thus extending the pioneering work of Sibson, who considered the relative normal centroid used to calculate the relative normal information radius. Figure 2 illustrates the three generalizations of the ordinary skewed Jensen-Shannon divergence. Notice that, in general, the $(M,N)$-JSDs and the variational JDSs are not f-divergences (except in the ordinary case).

Figure 2. Three equivalent expressions of the ordinary (skewed) Jensen-Shannon divergence which yield three different generalizations.

In a similar vein, Chen et al. [80] considered the following minimax symmetrization of the scalar Bregman divergence [81]:

$$\begin{array}{ccc}\hfill B_f^{\mathrm{minmax}}(p,q)& :=& \underset{c}{min}\underset{\lambda \in [0,1]}{max}\lambda B_f(p:c)+(1-\lambda )B_f(q:c),\hfill \end{array}$$

(136)

$$\begin{array}{ccc}& =& \underset{\lambda \in [0,1]}{max}\lambda B_f(p:\lambda p+(1-\lambda )q)+(1-\lambda )B_f(q:\lambda p+(1-\lambda )),\hfill \end{array}$$

(137)

$$\begin{array}{ccc}& =& \lambda f\left(p\right)+(1-\lambda )f\left(q\right)-f(\lambda p+(1-\lambda \left)\right)\hfill \end{array}$$

(138)

where $B_f$ denotes the scalar Bregman divergence induced by a strictly convex and smooth function f:

$$B_f(p:q)=f\left(p\right)-f\left(q\right)-(p-q)f^{\prime }\left(q\right).$$

(139)

They proved that $\sqrt{B_f^{\mathrm{minmax}}(p,q)}$ yields a metric when $3{(log{f}^{\prime \prime })}^{″}\ge {\left({(log{f}^{\prime \prime })}^{\prime }\right)}^2$, and extend the definition to the vector case and conjecture that the square-root metrization still holds in the multivariate case. In a sense, this definition geometrically highlights the notion of radius, since the minmax optimization amount to find a smallest enclosing ball enclosing [82] the source distributions. The circumcenter, also called the Chebyshev center [83], is then the mid-distribution instead of the centroid for the information radius. The term "information radius” is well-suited to measure the distance between two points for an arbitrary distance D. Indeed, the JS-symmetrization of D is defined by $D^{\mathrm{JS}}[p:q]:={min}_c\{\frac{1}{2}D[p:c]+\frac{1}{2}D[q:c]\}$. When $D[p:q]=D_E[p:q]=\parallel p-q\parallel$ is the Euclidean distance, we have $c=\frac{p+q}{2}$, and $D[p:c]=D[q:c]=\frac{1}{2}\parallel p-q\parallel =:r$ (i.e., the radius being half of the diameter $\parallel p-q\parallel$). Thus, $D_E^{\mathrm{JS}}[p:q]=r$; hence, the term chosen by Sibson [1] for $D^{\mathrm{JS}}$: information radius. Besides providing another viewpoint, variational definitions of divergences have proven to be useful in practice (e.g., for estimation). For example, a variational definition of the Rényi divergence generalizing the Donsker–Varadhan variational formula of the KLD is given in [84], which is used to estimate the Rényi Divergences.