Two Types of Geometric Jensen–Shannon Divergences

Abstract

The geometric Jensen–Shannon divergence (G-JSD) has gained popularity in machine learning and information sciences thanks to its closed-form expression between Gaussian distributions. In this work, we introduce an alternative definition of the geometric Jensen–Shannon divergence tailored to positive densities which does not normalize geometric mixtures. This novel divergence is termed the extended G-JSD, as it applies to the more general case of positive measures. We explicitly report the gap between the extended G-JSD and the G-JSD when considering probability densities, and show how to express the G-JSD and extended G-JSD using the Jeffreys divergence and the Bhattacharyya distance or Bhattacharyya coefficient. The extended G-JSD is proven to be an f-divergence, which is a separable divergence satisfying information monotonicity and invariance in information geometry. We derive a corresponding closed-form formula for the two types of G-JSDs when considering the case of multivariate Gaussian distributions that is often met in applications. We consider Monte Carlo stochastic estimations and approximations of the two types of G-JSD using the projective $\gamma$-divergences. Although the square root of the JSD yields a metric distance, we show that this is no longer the case for the two types of G-JSD. Finally, we explain how these two types of geometric JSDs can be interpreted as regularizations of the ordinary JSD.

1. Introduction

1.1. Kullback–Leibler and Jensen–Shannon Divergences

Let $(\mathcal{X},\mathcal{E},\mu )$ be a measure space on the sample space $\mathcal{X}$, $\sigma$-algebra of events $\mathcal{E}$, with $\mu$ a prescribed positive measure on the measurable space $(\mathcal{X},\mathcal{E})$ (e.g., counting measure or Lebesgue measure). Let $M_{+}\left(\mathcal{X}\right)=\left\{Q\right\}$ be the set of positive distributions Q and $M_{+}^1\left(\mathcal{X}\right)=\left\{P\right\}$ be the subset of probability measures P. We denote by $M_{\mu }=\{{\displaystyle \frac{\mathrm{d}Q}{\mathrm{d}\mu }}:Q\in M_{+}\left(\mathcal{X}\right)\}$ and $M_{\mu }^1=\{{\displaystyle \frac{\mathrm{d}P}{\mathrm{d}\mu }}:P\in M_{+}^1\left(\mathcal{X}\right)\}$ the corresponding sets of Radon–Nikodym positive and probability densities, respectively.

Consider two probability measures $P_1$ and $P_2$ of $M_{+}^1\left(\mathcal{X}\right)$ with Radon–Nikodym densities with respect to $\mu$ $p_1:={\displaystyle \frac{\mathrm{d}{P}_1}{\mathrm{d}\mu }}\in M_{\mu }^1$ and $p_2:={\displaystyle \frac{\mathrm{d}{P}_2}{\mathrm{d}\mu }}\in M_{\mu }^1$, respectively. The deviation of $P_1$ to $P_2$ (also called distortion, dissimilarity, or deviance) is commonly measured in information theory [1] by the Kullback–Leibler divergence (KLD):

$$\mathrm{KL}(p_1,p_2):=\int p_{1}log{\displaystyle \frac{p_1}{p_2}}\mathrm{d}\mu =E_{p_1}\left[log{\displaystyle \frac{p_1}{p_2}}\right].$$

(1)

Informally, the KLD quantifies the information lost when $p_2$ is used to approximate $p_1$ by measuring, on average, the surprise when outcomes sampled from $p_1$ are assumed to emanate from $p_2$: Shannon entropy $H\left(p\right)=\int plog{\displaystyle \frac{1}{p}}\mathrm{d}\mu$ is the expected surprise $H\left(p\right)=E_p[-logp]$, where $-logp\left(x\right)$ measures the surprise of the outcome x. Logarithms are taken to base 2 when information is measured in bits, and to base e when it is measured in nats. Gibbs’ inequality asserts that $\mathrm{KL}(P_1,P_2)\ge 0$ with equality if and only if $P_1=P_2$ $\mu$-almost everywhere. Since $\mathrm{KL}(p_1,p_2)\ne \mathrm{KL}(p_2,p_1)$, various symmetrization schemes of the KLD have been proposed in the literature [1] (e.g., Jeffreys divergence [1,2], resistor average divergence [3] (harmonic KLD symmetrization), Chernoff information [1], etc.)

An important symmetrization technique of the KLD is the Jensen–Shannon divergence [4,5] (JSD):

$$\mathrm{JS}(p_1,p_2):={\displaystyle \frac{1}{2}}\left(\mathrm{KL}(p_1,a)+\mathrm{KL}(p_2,a)\right),$$

(2)

where $a={\displaystyle \frac{1}{2}}{p}_1+{\displaystyle \frac{1}{2}}{p}_2$ denotes the statistical mixture of $p_1$ and $p_2$. The JSD is guaranteed to be upper-bounded by $log2$ even when the support of $p_1$ and $p_2$ differ, making it attractive in applications. Furthermore, its square root $\sqrt{\mathrm{JS}}$ yields a metric distance [6,7].

The JSD can be extended to a set of densities to measure the diversity of the set as an information radius [8]. In information theory, the JSD can also be interpreted as an information gain [6] since it can be equivalently written as

$$\mathrm{JS}(p_1,p_2)=H\left({\displaystyle \frac{1}{2}}{p}_1+{\displaystyle \frac{1}{2}}{p}_2\right)-{\displaystyle \frac{H\left(p_1\right)+H\left(p_2\right)}{2}},$$

where $H\left(p\right)=-\int plogp\mathrm{d}\mu$ is Shannon entropy (Shannon entropy for discrete measures and differential entropy for continuous measures). The JSD has also been defined in the setting of quantum information [9], where it has also been proven that its square root yields a metric distance [10].

Remark 1. 

Both the KLD and the JSD belong to the family of f-divergences [11,12] defined for a convex generator $f\left(u\right)$ (strictly convex at 1) by

$$I_f(p_1,p_2):=\int p_{1}f\left({\displaystyle \frac{p_2}{p_1}}\right)\mathrm{d}\mu .$$

Indeed, we have $\mathrm{KL}(p_1,p_2)=I_{f_{\mathrm{KL}}}(p_1,p_2)$ and $\mathrm{JS}(p_1,p_2)=I_{f_{\mathrm{JS}}}(p_1,p_2)$ for the following generators:

$$\begin{array}{ccc}\hfill f_{\mathrm{KL}}\left(u\right)& :=& -logu,\hfill \\ \hfill f_{\mathrm{JS}}\left(u\right)& :=& -(1+u)log{\displaystyle \frac{1+u}{2}}+ulogu.\hfill \end{array}$$

The family of f-divergences is the invariant divergences in information geometry [13]. The f-divergences guarantee information monotonicity by coarse graining [13] (also called lumping in information theory [14]). Using Jensen inequality, we get $I_f(p_1,p_2)\ge f\left(1\right)$.

Remark 2. 

The metrization of f-divergences was studied in [15]. Once a metric distance $D(p_1,p_2)$ is given, we may use the following metric transform [16] to obtain another metric which is guaranteed to be bounded by 1:

$$0\le d(p_1,p_2)={\displaystyle \frac{D(p_1,p_2)}{1+D(p_1,p_2)}}\le 1.$$

1.2. Jensen–Shannon Symmetrization of Dissimilarities with Generalized Mixtures

In [17], a generalization of the KLD Jensen–Shannon symmetrization scheme was studied for arbitrary statistical dissimilarity $D(·,·)$ using an arbitrary weighted mean [18] $M_{\alpha }$. A generic weighted mean $M_{\alpha }(a,b)=M_{1-\alpha }(b,a)$ for $a,b\in {\mathbb{R}}_{>0}$ is a continuous symmetric monotonic map $\alpha \in [0,1]\mapsto M_{\alpha }(a,b)$ such that $M_0(a,b)=b$ and $M_1(a,b)=1$. For example, the quasi-arithmetic means [18] are defined according to a monotonous continuous function $\varphi$ as follows:

$$M_{\alpha }^{\varphi }(a,b):={\varphi }^{-1}\left(\alpha \varphi \left(a\right)+(1-\alpha )\varphi \left(b\right)\right).$$

When ${\varphi }_p\left(u\right)=u^p$, we get the p-power mean $M_{\alpha }^{{\varphi }_p}(a,b)={(\alpha a^p+(1-\alpha )b^p)}^{{\displaystyle \frac{1}{p}}}$ for $p\in \mathbb{R}\setminus \left\{0\right\}$. We extend ${\varphi }_p$ for $p=0$ by defining ${\varphi }_0\left(u\right)=logu$, and get $M_{\alpha }^{{\varphi }_0}(a,b)=a^{\alpha }{b}^{1-\alpha }$, the weighted geometric mean $G_{\alpha }$.

Let us recall the generalization of the Jensen–Shannon symmetrization scheme of a dissimilarity measure presented in [17]:

Definition 1 

($(\alpha ,\beta )$ M-JS dissimilarity [17]). The Jensen–Shannon skew symmetrization of a statistical dissimilarity $D(·,·)$ with respect to an arbitrary weighted bivariate mean $M_{\alpha }(·,·)$ is given by

$$D_{M_{\alpha },\beta }^{\mathrm{JS}}(p_1,p_2):=\beta D\left(p_1,{\left(p_1{p}_2\right)}_{M_{\alpha }}\right)+(1-\beta )D\left(p_2,{\left(p_1{p}_2\right)}_{M_{\alpha }}\right),(\alpha ,\beta )\in {(0,1)}^2,$$

(3)

where ${\left(p_1{p}_2\right)}_{M_{\alpha }}$ is the statistical normalized weighted M-mixture of $p_1$ and $p_2$:

$${\left(p_1{p}_2\right)}_{M_{\alpha }}\left(x\right):={\displaystyle \frac{M_{\alpha }(p_1\left(x\right),p_2\left(x\right))}{\int M_{\alpha }(p_1\left(x\right),p_2\left(x\right))\mathrm{d}\mu \left(x\right)}}.$$

(4)

Remark 3. 

A more general definition is given in [17] by using another arbitrary weighted mean $N_{\beta }$ to average the two dissimilarities in Equation (3):

$$D_{M_{\alpha },N_{\beta }}^{\mathrm{JS}}(p_1,p_2):=N_{\beta }\left(D\left(p_1,{\left(p_1{p}_2\right)}_{M_{\alpha }}\right),D\left(p_2,{\left(p_1{p}_2\right)}_{M_{\alpha }}\right)\right),(\alpha ,\beta )\in {(0,1)}^2.$$

(5)

When $N_{\beta }=A_{\alpha }$, the weighted arithmetic mean $A_{\alpha }(a,b)=\alpha a+(1-\alpha )b$, Equation (5) amounts to Equation (3).

When $\alpha ={\displaystyle \frac{1}{2}}$, we write, for short, ${\left(p_1{p}_2\right)}_M$ instead of ${\left(p_1{p}_2\right)}_{M_{\frac{1}{2}}}$ in the reminder.

When $D=\mathrm{KL}$, $M=N=A_{\frac{1}{2}}$, Equation (5) yields the Jensen–Shannon divergence of Equation (2): $\mathrm{JS}(p_1,p_2)={\mathrm{KL}}_{A_{\frac{1}{2}},A_{\frac{1}{2}}}^{\mathrm{JS}}(p_1,p_2)={\mathrm{KL}}_{A,A}^{\mathrm{JS}}(p_1,p_2)$.

Lower and upper bounds for the skewed $\alpha$-Jensen–Shannon divergence were reported in [19].

The abstract mixture normalizer of ${\left(p_1{p}_2\right)}_{M_{\alpha }}$ shall be denoted by

$$Z_{M_{\alpha }}(p_1,p_2):=\int M_{\alpha }(p_1\left(x\right),p_2\left(x\right))\mathrm{d}\mu \left(x\right),$$

so that the normalized M-mixture is written as ${\left(p_1{p}_2\right)}_{M_{\alpha }}\left(x\right)={\displaystyle \frac{M_{\alpha }(p_1\left(x\right),p_2\left(x\right))}{Z_{M_{\alpha }}(p_1,p_2)}}$. The normalizer $Z_{M_{\alpha }}(p_1,p_2)$ is always finite, and thus the weighted M-mixtures ${\left(p_1{p}_2\right)}_{M_{\alpha }}$ are well-defined:

Proposition 1. 

For any generic weighted mean $M_{\alpha }$, we have the normalizer of the weighted M-mixture bounded by 2:

$$0\le Z_{M_{\alpha }}(p_1,p_2)\le 2.$$

Proof. 

Since $M_{\alpha }$ is a scalar weighted mean, it satisfies the following in-betweenness property:

$$\min \{p_1\left(x\right),p_2\left(x\right)\}\le M_{\alpha }(p_1\left(x\right),p_2\left(x\right))\le \max \{p_1\left(x\right),p_2\left(x\right)\}.$$

(6)

Hence, by using the following two identities for $a\ge 0$ and $b\ge 0$,

$$\begin{array}{ccc}\hfill \min \{a,b\}& =& {\displaystyle \frac{a+b}{2}}-{\displaystyle \frac{1}{2}}|a-b|,\hfill \\ \hfill \max \{a,b\}& =& {\displaystyle \frac{a+b}{2}}+{\displaystyle \frac{1}{2}}|a-b|,\hfill \end{array}$$

we get

$$\begin{array}{ccc}\hfill \int \min \{p_1\left(x\right),p_2\left(x\right)\}\mathrm{d}\mu \left(x\right)\le & \int M_{\alpha }(p_1\left(x\right),p_2\left(x\right))\mathrm{d}\mu \left(x\right)& \le \int \max \{p_1\left(x\right),p_2\left(x\right)\}\mathrm{d}\mu \left(x\right),\hfill \\ \hfill 0\le 1-\mathrm{TV}(p_1,p_2)\le & Z_{M_{\alpha }}(p_1,p_2)& \le 1+\mathrm{TV}(p_1,p_2)\le 2,\hfill \end{array}$$

(7)

where

$$\mathrm{TV}(p_1,p_2):={\displaystyle \frac{1}{2}}\int |p_1-p_2|\mathrm{d}\mu ,$$

is the total variation distance, upper-bounded by 1. When the support of the densities $p_1$ and $p_2$ intersect (i.e., non-singular probability measures $P_1$ and $P_2$), we have $Z_{M_{\alpha }}(p_1,p_2)>0$, and therefore the weighted M-mixtures ${\left(p_1{p}_2\right)}_{M_{\alpha }}$ are well-defined. □

The generic Jensen–Shannon symmetrization of dissimilarities given in Definition 1 allows us to re-interpret some well-known statistical dissimilarities:

For example, the Chernoff information [1,20] is defined by

$$C(p_1,p_2):=\underset{\alpha \in (0,1)}{\max }{B}_{\alpha }(p_1,p_2),$$

(8)

where $B_{\alpha }(p_1,p_2)$ denotes the $\alpha$-skewed Bhattacharrya distance:

$$B_{\alpha }(p_1,p_2):=-log\int p_1^{\alpha }{p}_2^{1-\alpha }\mathrm{d}\mu$$

(9)

When $\alpha ={\displaystyle \frac{1}{2}}$, we note $B(p_1,p_2)=B_{\frac{1}{2}}(p_1,p_2)$, the Bhattacharrya distance. Notice that the Bhattacharrya distance is not a metric distance as it violates the triangle inequality of metrics.

Using the framework of JS-symmetrization of dissimilarities, we can reinterpret the Chernoff information as

$$C(p_1,p_2)={\left({\mathrm{KL}}^{*}\right)}_{G_{{\alpha }^{*}},A_{\frac{1}{2}}}^{\mathrm{JS}}(p_1,p_2),$$

where ${\alpha }^{*}$ is provably the unique optimal skewing factor in Equation (8), such that we have [20]:

$$\begin{array}{ccc}\hfill C(p_1,p_2)& =& {\mathrm{KL}}^{*}(p_1,{\left(p_1{p}_2\right)}_{G_{{\alpha }^{*}}})={\mathrm{KL}}^{*}(p_2,{\left(p_1{p}_2\right)}_{G_{{\alpha }^{*}}}),\hfill \\ & =& {\displaystyle \frac{1}{2}}\left({\mathrm{KL}}^{*}(p_1,{\left(p_1{p}_2\right)}_{G_{{\alpha }^{*}}})+{\mathrm{KL}}^{*}(p_2,{\left(p_1{p}_2\right)}_{G_{{\alpha }^{*}}})\right),\hfill \end{array}$$

where ${\mathrm{KL}}^{*}$ denotes the reverse KLD:

$${\mathrm{KL}}^{*}(p_1,p_2):=\mathrm{KL}(p_2,p_1).$$

Note that the KLD is sometimes called the forward KLD (e.g., [21]), and we have ${{\mathrm{KL}}^{*}}^{*}(p_1,p_2)=\mathrm{KL}(p_1,p_2)$.

Although arithmetic mixtures are most often used in statistics, geometric mixtures are also encountered, for example in Bayesian statistics [22] or in Markov chain Monte Carlo annealing [23], just to give two examples. In information geometry, statistical power mixtures based on the homogeneous power means are used to perform stochastic integration of statistical models [24].

Proposition 2 (Bhattacharyya distance as G-JSD).

The Bhattacharyya distance [25] and the α-skewed Bhattacharyya distances can be interpreted as JS-symmetrizations of the reverse KLD with respect to the geometric mean G:

$$\begin{array}{ccc}\hfill B(p_1,p_2)& :=& -log\int \sqrt{p_1{p}_2}\mathrm{d}\mu ={\left({\mathrm{KL}}^{*}\right)}_G^{\mathrm{JS}}(p_1,p_2),\hfill \\ \hfill B_{\alpha }(p_1,p_2)& :=& -log\int p_1^{\alpha }{p}_2^{1-\alpha }\mathrm{d}\mu ={\left({\mathrm{KL}}^{*}\right)}_{G_{\alpha }}^{\mathrm{JS}}(p_1,p_2).\hfill \end{array}$$

Proof. 

Let $m={\left(p_1{p}_2\right)}_G={\displaystyle \frac{\sqrt{p_1{p}_2}}{Z(p_1,p_2)}}$ denote the weighted geometric mixture with the normalizer $Z_G(p_1,p_2)=\int \sqrt{p_1{p}_2}\mathrm{d}\mu$. By definition of the JS-symmetrization of the reverse KLD, we have

$$\begin{array}{ccc}\hfill {\left({\mathrm{KL}}^{*}\right)}_G^{\mathrm{JS}}(p_1,p_2)& :=& {\displaystyle \frac{1}{2}}\left({\mathrm{KL}}^{*}(p_1,{\left(p_1{p}_2\right)}_G)+{\mathrm{KL}}^{*}(p_2,{\left(p_1{p}_2\right)}_G)\right),\hfill \\ & =& {\displaystyle \frac{1}{2}}\left(\mathrm{KL}({\left(p_1{p}_2\right)}_G,p_1)+\mathrm{KL}({\left(p_1{p}_2\right)}_G,p_2)\right),\hfill \\ & =& {\displaystyle \frac{1}{2}}\left(\int \left(mlog{\displaystyle \frac{\sqrt{p_1{p}_2}}{p_1{Z}_G(p_1,p_2)}}+mlog{\displaystyle \frac{\sqrt{p_1{p}_2}}{p_2{Z}_G(p_1,p_2)}}\right)\mathrm{d}\mu \right),\hfill \\ & =& {\displaystyle \frac{1}{2}}\left(\int {\displaystyle \frac{1}{2}}mlog{\displaystyle \frac{p_2}{p_1}}{\displaystyle \frac{p_1}{p_2}}\mathrm{d}\mu -2log{Z}_G(p_1,p_2)\int m\mathrm{d}\mu \right),\hfill \\ & =& -log{Z}_G(p_1,p_2)=:B(p_1,p_2).\hfill \end{array}$$

The proof carries on similarly for the $\alpha$-skewed JS-symmetrization of the reverse KLD: we now let $m_{\alpha }={\left(p_1{p}_2\right)}_{G_{\alpha }}={\displaystyle \frac{p_1^{\alpha }{p}_2^{1-\alpha }}{Z_{G_{\alpha }}(p_1,p_2)}}$ be the $\alpha$-weighted geometric mixture with the normalizer $Z_{G_{\alpha }}(p_1,p_2)=\int p_1^{\alpha }{p}_2^{1-\alpha }\mathrm{d}\mu$, written as $Z_{G_{\alpha }}$ for short below:

$$\begin{array}{ccc}\hfill {{\mathrm{KL}}^{*}}_{G_{\alpha },\alpha }^{\mathrm{JS}}(p_1,p_2)& :=& \alpha {\mathrm{KL}}^{*}(p_1,{\left(p_1{p}_2\right)}_{G_{\alpha }})+(1-\alpha ){\mathrm{KL}}^{*}(p_2,{\left(p_1{p}_2\right)}_{G_{\alpha }}),\hfill \\ & =& \alpha \mathrm{KL}(m_{\alpha },p_1)+(1-\alpha )\mathrm{KL}(m_{\alpha },p_2),\hfill \\ & =& \int \left(\alpha m_{\alpha }log{\displaystyle \frac{p_1^{\alpha }{p}_2^{1-\alpha }}{Z_{G_{\alpha }}{p}_1}}+(1-\alpha )m_{\alpha }log{\displaystyle \frac{p_1^{\alpha }{p}_2^{1-\alpha }}{Z_{G_{\alpha }}{p}_2}}\right)\mathrm{d}\mu ,\hfill \\ & =& -(\alpha +1-\alpha )log{Z}_{G_{\alpha }}\int m_{\alpha }\mathrm{d}\mu +\int m_{\alpha }log{\left({\displaystyle \frac{p_2}{p_1}}\right)}^{\alpha (1-\alpha )}{\left({\displaystyle \frac{p_1}{p_2}}\right)}^{\alpha (1-\alpha )}\mathrm{d}\mu ,\hfill \\ & =& -log{Z}_{G_{\alpha }}(p_1,p_2)=:B_{\alpha }(p_1,p_2).\hfill \end{array}$$

□

Besides information theory [1], the JSD also plays an important role in machine learning [26,27,28]. However, one drawback that refrains its use in practice is that the JSD between two Gaussian distributions (normal distributions) is not known in closed form, since no analytic formula is known for the differential entropy of a two-component Gaussian mixture [29], and thus the JSD needs to be numerically approximated in practice by various methods.

To circumvent this problem, the geometric G-JSD was defined in [17] as follows:

Definition 2 

(G-JSD [17]). The geometric Jensen–Shannon divergence (G-JSD) between two probability densities, $p_1$ and $p_2$, is defined by

$${\mathrm{JS}}_G(p_1,p_2):={\displaystyle \frac{1}{2}}\left(\mathrm{KL}(p_1,{\left(p_1{p}_2\right)}_G)+\mathrm{KL}(p_2,{\left(p_1{p}_2\right)}_G)\right),$$

where ${\left(p_1{p}_2\right)}_G\left(x\right)={\displaystyle \frac{\sqrt{p_1\left(x\right)p_2\left(x\right)}}{\int \sqrt{p_1\left(x\right)p_2\left(x\right)}\mathrm{d}\mu }}$ is the (normalized) geometric mixture of $p_1$ and $p_2$.

We have ${\mathrm{JS}}_G(p_1,p_2)={\mathrm{KL}}_G^{\mathrm{JS}}(p_1,p_2)$. Since, by default, the M- mixture JS-symmetrization of dissimilarities D is performed on the right argument (i.e., $D_M^{\mathrm{JS}}$), we may also consider a dual JS-symmetrization by setting the M-mixtures on the left argument. We denote this left mixture JS-symmetrization with $D_M^{{\mathrm{JS}}^{*}}$. We have $D_M^{{\mathrm{JS}}^{*}}(p_1,p_2)={\left(D^{*}\right)}_M^{\mathrm{JS}}(p_1,p_2)$, i.e., the left-sided JS-symmetrization of D amounts to a right-sided JS-symmetrization of the dual dissimilarity $D^{*}(p_1,p_2):=D(p_2,p_1)$.

Thus, a left-sided G-JSD divergence ${\mathrm{JS}}_G^{*}$ was also defined in [17]:

Definition 3. 

The left-sided geometric Jensen–Shannon divergence (G-JSD) between two probability densities $p_1$ and $p_2$ is defined by

$$\begin{array}{ccc}\hfill {\mathrm{JS}}_G^{*}(p_1,p_2)& :=& {\displaystyle \frac{1}{2}}\left(\mathrm{KL}({\left(p_1{p}_2\right)}_G,p_1)+\mathrm{KL}({\left(p_1{p}_2\right)}_G,p_2)\right),\hfill \\ & =& {\displaystyle \frac{1}{2}}\left({\mathrm{KL}}^{*}(p_1,{\left(p_1{p}_2\right)}_G)+{\mathrm{KL}}^{*}(p_2,{\left(p_1{p}_2\right)}_G)\right),\hfill \end{array}$$

where ${\left(p_1{p}_2\right)}_G\left(x\right)={\displaystyle \frac{\sqrt{p_1\left(x\right)p_2\left(x\right)}}{\int \sqrt{p_1\left(x\right)p_2\left(x\right)}\mathrm{d}\mu }}$ is the (normalized) geometric mixture of $p_1$ and $p_2$.

To contrast with the numerical approximation limitation of the JSD between Gaussians, one advantage of the geometric Jensen–Shannon divergence (G-JSD) is that it admits a closed-form expression between Gaussian distributions [17]. However, the G-JSD is no longer bounded. The G-JSD formula between Gaussian distributions has been used in several scenarios. See [30,31,32,33,34,35,36,37,38] for a few use cases.

Let us express the G-JSD divergence using other familiar divergences.

Proposition 3. 

We have the following expression of the geometric Jensen–Shannon divergence:

$${\mathrm{JS}}_G(p_1,p_2)={\displaystyle \frac{1}{4}}J(p_1,p_2)-B(p_1,p_2),$$

where $J(p_1,p_2):=\int (p_1-p_2)log{\displaystyle \frac{p_1}{p_2}}\mathrm{d}\mu$ is Jeffreys’ divergence [2], and

$$B(p_1,p_2)=-log\int \sqrt{p_1{p}_2}\mathrm{d}\mu =-log{Z}_G(p_1,p_2),$$

is the Bhattacharrya distance.

Proof. 

We have the following:

$$\begin{array}{ccc}\hfill {\mathrm{JS}}_G(p_1,p_2)& :=& {\displaystyle \frac{1}{2}}\left(\mathrm{KL}(p_1,{\left(p_1{p}_2\right)}_G)+\mathrm{KL}(p_2,{\left(p_1{p}_2\right)}_G)\right),\hfill \\ & =& {\displaystyle \frac{1}{2}}\left(\int \left(p_1\left(x\right)log{\displaystyle \frac{p_1\left(x\right)Z_G(p_1,p_2)}{\sqrt{p_1\left(x\right)p_2\left(x\right)}}}+p_2\left(x\right)log{\displaystyle \frac{p_2\left(x\right)Z_G(p_1,p_2)}{\sqrt{p_1\left(x\right)p_2\left(x\right)}}}\right)\mathrm{d}\mu \left(x\right)\right),\hfill \\ & =& {\displaystyle \frac{1}{2}}\left(\int \left(p_1\left(x\right)+p_2\left(x\right)\right)log{Z}_G(p_1,p_2)\mathrm{d}\mu \left(x\right)+{\displaystyle \frac{1}{2}}\mathrm{KL}(p_1,p_2)+{\displaystyle \frac{1}{2}}\mathrm{KL}(p_2,p_1)\right),\hfill \\ & =& log{Z}_G(p_1,p_2)+{\displaystyle \frac{1}{4}}J(p_1,p_2),\hfill \\ & =& {\displaystyle \frac{1}{4}}J(p_1,p_2)-B(p_1,p_2).\hfill \end{array}$$

□

Corollary 1 (G-JSD upper bound).

We have the upper bound ${\mathrm{JS}}_G(p,q)\le {\displaystyle \frac{1}{4}}J(p,q)$.

Proof. 

Since $B(p_1,p_2)\ge 0$ and ${\mathrm{JS}}_G(p_1,p_2)={\displaystyle \frac{1}{4}}J(p_1,p_2)-B(p_1,p_2)$, we have ${\mathrm{JS}}_G(p,q)\le {\displaystyle \frac{1}{4}}J(p,q)$. □

Remark 4. 

Although the KLD and JSD are separable divergences (i.e., f-divergences expressed as integrals of scalar divergences), the M-JSD divergence is, in general, not separable, because it requires mixtures to be normalized inside the log terms. Notice that the Bhattacharyya distance is, similarly, not a separable divergence, but the Bhattacharyya similarity coefficient $\mathrm{BC}(p_1,p_2)=exp(-B(p_1,p_2))=\int \sqrt{p_1{p}_2}\mathrm{d}\mu$ is a separable “f-divergence”/f-coefficient for $f_{\mathrm{BC}}\left(u\right)=\sqrt{u}$ (here, a concave generator): $\mathrm{BC}(p_1,p_2)=I_{f_{\mathrm{BC}}}(p_1,p_2)$. Notice that $f_{\mathrm{BC}}\left(1\right)=1$, and because of the concavity of $f_{\mathrm{BC}}$, we have $I_{f_{\mathrm{BC}}}(p_1,p_2)\le f_{\mathrm{BC}}\left(1\right)=1$ (hence, the term f-coefficient to reflect the notion of a similarity measure).

1.3. Paper Outline

The paper is organized as follows: We first give an alternative definition of the M-JSD in Section 2 (Definition 4) which extends to positive measures and does not require normalization of geometric mixtures. We call this new divergence the extended M-JSD, and we compare the two types of geometric JSDs when dealing with probability measures. In Section 4, we show that these normalized/extended M-JSD divergences can be interpreted as regularizations of the Jensen–Shannon divergence, and exhibit several bounds. We discuss Monte Carlo stochastic approximations and approximations using $\gamma$-divergences [39] in Section 5. For the case of geometric mixtures, although the G-JSD is not an f-divergence, we show that the extended G-JSD is an f-divergence (Proposition 5), and we express both the G-JSD and the extended G-JSD using both the Jeffreys divergence and the Bhattacharyya divergence or coefficient. We report a related closed-form formula for the G-JSD and extended G-JSD between two Gaussian distributions in Section 3. Finally, we summarize the main results in the concluding section, Section 6.

A list of notations is provided in Nomenclature.

2. A Novel Definition: The G-JSD, Extended to Positive Measures

2.1. Definition and Properties

We may consider the following two modifications of the G-JSD:

• First, we replace the KLD with the extended KLD between positive densities $q_1\in M_{\mu }^{+}$ and $q_2\in M_{\mu }^{+}$ instead of normalized densities:

  $${\mathrm{KL}}^{+}(q_1,q_2):=\int \left(q_{1}log{\displaystyle \frac{q_1}{q_2}}+q_2-q_1\right)\mathrm{d}\mu ,$$

  (10)
  (with ${\mathrm{KL}}^{+}(p_1,p_2)=\mathrm{KL}(p_1,p_2)$);
• Second, we consider unnormalized M-mixture densities:

  $${\left(q_1{q}_2\right)}_{{\tilde{M}}_{\alpha }}\left(x\right):=M_{\alpha }(q_1\left(x\right),q_2\left(x\right)),$$

  where we use the $\tilde{M}$ tilde notation to indicate that the M-mixture is not normalized, instead of normalized densities ${\left(q_1{q}_2\right)}_{M_{\alpha }}\left(x\right)$.

The extended KLD can be interpreted as a pointwise integral of a scalar Bregman divergence obtained for the negative Shannon entropy generator [40]. This proves that ${\mathrm{KL}}^{+}(q_1,q_2)\ge 0$ with equality if and only if $q_1=q_2$ $\mu$-almost everywhere. Notice that $\mathrm{KL}(q_1,q_2)$ may be negative when $q_1$ and/or $q_2$ are not normalized to probability densities, but we always have ${\mathrm{KL}}^{+}(q_1,q_2)\ge 0$.

The extended KLD is an extended f-divergence [41]: ${\mathrm{KL}}^{+}(q_1,q_2)=I_{f_{{\mathrm{KL}}^{+}}}^{+}(q_1,q_2)$ for $f_{{\mathrm{KL}}^{+}}\left(u\right)=-log\left(u\right)+u-1$, where $I_f^{+}(q_1,q_2)$ denotes the f-divergence extended to positive densities $q_1$ and $q_2$:

$$I_f^{+}(q_1,q_2)=\int q_{1}f\left({\displaystyle \frac{q_2}{q_1}}\right)\mathrm{d}\mu .$$

Remark 5. 

As a side remark, it is preferable in practice to estimate the KLD between $p_1$ and $p_2$ by Monte Carlo methods using Equation (10) instead of Equation (1) in order to guarantee the non-negativeness of the KLD (Gibbs’ inequality). Indeed, the sampling of s samples $x_1,\dots ,x_s$, defining two unnormalized distributions $q_1\left(x\right)={\displaystyle \frac{1}{s}}{\sum }_{i=1}^s{p}_1\left(x\right){\delta }_{x_i}\left(x\right)$ and $q_2\left(x\right)={\displaystyle \frac{1}{s}}{\sum }_{i=1}^s{p}_2\left(x\right){\delta }_{x_i}\left(x\right)$, where

$${\delta }_{x_i}\left(x\right)=\left\{\begin{array}{cc}1,\hfill & ifx=x_i\hfill \\ 0,\hfill & otherwise\hfill \end{array}\right..$$

Remark 6. 

For an arbitrary distortion measure $D^{+}(q_1,q_2)$ between positive measures $q_1$ and $q_2$, we can build a corresponding projective divergence $\tilde{D}(q_1,q_2)$ as follows:

$$\tilde{D}(q_1,q_2):=D^{+}\left({\displaystyle \frac{q_1}{Z\left(q_1\right)}},{\displaystyle \frac{q_1}{Z\left(q_2\right)}}\right),$$

where $Z\left(q\right):=\int q\mathrm{d}\mu$ is the normalization factor of the positive density q. The divergence $\tilde{D}$ is said to be projective because we have, for all ${\lambda }_1>0, {\lambda }_2>0$, the property that $\tilde{D}({\lambda }_1{q}_1,{\lambda }_2{q}_2)=\tilde{D}(q_1,q_2)=D^{+}(p_1,p_2)$, where $p_i={\displaystyle \frac{q_i}{Z\left(q_i\right)}}$ are the normalized densities. The projective Kullback–Leibler divergence $\widetilde{\mathrm{KL}}$ is thus another projective extension of the KLD to non-normalized densities which coincide with the KLD for probability densities. But the projective KLD is different from the extended KLD of Equation (10), and furthermore, we have $\widetilde{\mathrm{KL}}(q_1,q_2)=0$ if and only if $q_1=\lambda q_2$ μ-almost everywhere for some $\lambda >0$.

Let us now define the Jensen–Shannon symmetrization of an extended statistical divergence $D^{+}$ with respect to an arbitrary weighted mean $M_{\alpha }$ as follows:

Definition 4 (Extended M-JSD).

A Jensen–Shannon skew symmetrization of a statistical divergence $D^{+}(·,·)$ between two positive measures $q_1$ and $q_2$ with respect to a weighted mean $M_{\alpha }$ is defined by

$$D_{{\tilde{M}}_{\alpha },\beta }^{{\mathrm{JS}}^{+}}(q_1,q_2):=\beta D^{+}\left(q_1,{\left(q_1{q}_2\right)}_{{\tilde{M}}_{\alpha }}\right)+(1-\beta )D^{+}\left(q_1,{\left(q_1{q}_2\right)}_{{\tilde{M}}_{\alpha }}\right),$$

(11)

When $\beta ={\displaystyle \frac{1}{2}}$, we write, for short, $D_{{\tilde{M}}_{\alpha }}^{{\mathrm{JS}}^{+}}(q_1,q_2)$, and furthermore, when $\alpha ={\displaystyle \frac{1}{2}}$, we simplify the notation to $D_{\tilde{M}}^{{\mathrm{JS}}^{+}}(q_1,q_2)$.

When $D^{+}={\mathrm{KL}}^{+}$, we obtain the extended geometric Jensen–Shannon divergence, ${\mathrm{JS}}_{\tilde{G}}^{+}(q_1,q_2)={\mathrm{KL}}_{\tilde{G}}^{{\mathrm{JS}}^{+}}(q_1,q_2)$:

Definition 5 (Extended G-JSD).

The extended geometric Jensen–Shannon divergence between two positive densities $q_1$ and $q_2$ is

$$\begin{array}{ccc}\hfill {\mathrm{JS}}_{\tilde{G}}^{+}(q_1,q_2)& =& {\displaystyle \frac{1}{2}}\left({\mathrm{KL}}^{+}(q_1,{\left(q_1{q}_2\right)}_{\tilde{G}})+{\mathrm{KL}}^{+}(q_2,{\left(q_1{q}_2\right)}_{\tilde{G}}))\right),\hfill \end{array}$$

(12)

The extended G-JSD between two normalized densities $p_1$ and $p_2$ is thus

$$\begin{array}{ccc}\hfill {\mathrm{JS}}_{\tilde{G}}^{+}(p_1,p_2)& =& {\displaystyle \frac{1}{2}}\left(\int \left(p_{1}log{\displaystyle \frac{p_1}{\sqrt{p_1{p}_2}}}+p_{2}log{\displaystyle \frac{p_2}{\sqrt{p_1{p}_2}}}\right)\mathrm{d}\mu +\int \sqrt{p_1{p}_2}\mathrm{d}\mu )\right)-1,\hfill \end{array}$$

(13)

$$\begin{array}{ccc}& =& {\displaystyle \frac{1}{2}}\left(\int \left(p_{1}log\sqrt{{\displaystyle \frac{p_1}{p_2}}}+p_{2}log\sqrt{{\displaystyle \frac{p_2}{p_1}}}\right)\mathrm{d}\mu +Z_G(p_1,p_2)\right)-1,\hfill \end{array}$$

(14)

with $Z_G(p_1,p_2)=exp(-B(p_1,p_2))$.

Thus, we get the following propositions:

Proposition 4. 

The extended geometric Jensen–Shannon divergence (G-JSD) can be expressed as follows:

$${\mathrm{JS}}_{\tilde{G}}^{+}(p_1,p_2)={\displaystyle \frac{1}{4}}J(p_1,p_2)+exp(-B(p_1,p_2))-1.$$

Proof. 

We have

$$\begin{array}{ccc}\hfill {\mathrm{JS}}_{\tilde{G}}^{+}(p_1,p_2)& =& {\displaystyle \frac{1}{2}}\left({\mathrm{KL}}^{+}(p_1,{\left(p_1{p}_2\right)}_{\tilde{G}})+{\mathrm{KL}}^{+}(p_2,{\left(p_1{p}_2\right)}_{\tilde{G}})\right),\hfill \\ & =& {\displaystyle \frac{1}{2}}\left(\int \left(p_{1}log\sqrt{{\displaystyle \frac{p_1}{p_2}}}+p_{2}log\sqrt{{\displaystyle \frac{p_2}{p_1}}}+2\sqrt{p_1{p}_2}-(p_1+p_2)\right)\mathrm{d}\mu \right),\hfill \\ & =& \int {\displaystyle \frac{1}{4}}(p_1-p_2)log{\displaystyle \frac{p_1}{p_2}}\mathrm{d}\mu +\int \sqrt{p_1{p}_2}\mathrm{d}\mu -1,\hfill \\ & =& {\displaystyle \frac{1}{4}}J(p_1,p_2)+exp(-B(p_1,p_2))-1.\hfill \end{array}$$

□

Thus, we can express the gap between ${\mathrm{JS}}_{\tilde{G}}^{+}(p_1,p_2)$ and ${\mathrm{JS}}_G(p_1,p_2)$:

$${\Delta }_G(p_1,p_2)={\mathrm{JS}}_{\tilde{G}}^{+}(p_1,p_2)-{\mathrm{JS}}_G(p_1,p_2)=exp(-B(p_1,p_2))+B(p_1,p_2)-1.$$

Since $Z_G(p_1,p_2)=exp(-B(p_1,p_2))$, we have:

$${\Delta }_G(p_1,p_2)=Z_G(p_1,p_2)-log{Z}_G(p_1,p_2)-1.$$

Proposition 5. 

The extended G-JSD is an f-divergence for the generator

$$f_{\tilde{G}}\left(u\right)={\displaystyle \frac{1}{4}}\left(u-1\right)logu+\sqrt{u}-1.$$

That is, we have ${\mathrm{JS}}_{\tilde{G}}^{+}(p_1,p_2)=I_{f_{\tilde{G}}}(p_1,p_2)$.

Proof. 

We proved that ${\mathrm{JS}}_{\tilde{G}}^{+}(p_1,p_2)={\displaystyle \frac{1}{4}}J(p_1,p_2)+\mathrm{BC}(p_1,p_2)-1$. The Jeffreys divergence is an f-divergence for the generator $f_J\left(u\right)=(u-1)logu$, and the Bhattacharyya coefficient is an f-coefficient for $f_{\mathrm{BC}}\left(u\right)=\sqrt{u}$ (a “f-divergence” for a concave generator). Thus, we have

$$f_{\tilde{G}}\left(u\right)={\displaystyle \frac{1}{4}}\left(u-1\right)logu+\sqrt{u}-1,$$

such that ${\mathrm{JS}}_{\tilde{G}}^{+}(p_1,p_2)=I_{f_{\tilde{G}}}(p_1,p_2)$. We check that $f_{\tilde{G}}\left(u\right)$ is convex, since $f_{\tilde{G}}^{″}\left(u\right)={\displaystyle \frac{\sqrt{u}(u+1)-u}{4u^{\frac{5}{2}}}}$ (and by a change of variable $t=\sqrt{u}$, the numerator $t(t^2-t+1)$ is shown to be positive, since the discriminant of $t^2-t+1$ is negative), and we have $f_{\tilde{G}}\left(1\right)=0$. Thus, the extended G-JSD is a proper f-divergence. □

It follows that the extended G-JSD satisfies the information monotonicity of invariant divergences in information geometry [13].

By abuse of notations, we have

$${\mathrm{KL}}^{+}(q_1,q_2):=\mathrm{KL}(q_1,q_2)+\int \left(q_2-q_1\right)\mathrm{d}\mu ,$$

although $q_1$ and $q_2$ may not need to be normalized in the $\mathrm{KL}$ term (which can then yield a potentially negative value). Letting $Z\left(q_i\right):=\int q_i\mathrm{d}\mu$ be the total mass of positive density $q_i$, we have

$${\mathrm{KL}}^{+}(q_1,q_2)=\mathrm{KL}(q_1,q_2)+Z\left(q_2\right)-Z\left(q_1\right).$$

(15)

Let ${\tilde{m}}_{\alpha }=M_{\alpha }(q_1,q_2)$ be the unnormalized M-mixture of positive densities $q_1$ and $q_2$, and set $Z_{M_{\alpha }}=\int {\tilde{m}}_{\alpha }\mathrm{d}\mu$ be the normalization term so that we have $m_{\alpha }={\displaystyle \frac{{\tilde{m}}_{\alpha }}{Z_{M_{\alpha }}}}$ and ${\tilde{m}}_{\alpha }=Z_{M_{\alpha }}{m}_{\alpha }$. When clear from context, we write $Z_{\alpha }$ instead of $Z_{M_{\alpha }}$.

We get, after elementary calculus, the following identity:

$${\mathrm{JS}}_{{\tilde{M}}_{\alpha },\beta }^{+}(q_1,q_2)={\mathrm{JS}}_{M_{\alpha },\beta }(q_1,q_2)-(\beta Z\left(q_1\right)+(1-\beta )Z\left(q_2\right))log{Z}_{\alpha }+Z_{\alpha }-(\beta Z\left(q_1\right)+(1-\beta )Z\left(q_2\right)).$$

(16)

Therefore, the difference gap ${\Delta }_{M_{\alpha },\beta }(p_1,p_2)$ (written for short as $\Delta (p_1,p_2)$) between the normalized JSD and the unnormalized M-JSD between two normalized densities $p_1$ and $p_2$ (i.e., with $Z_1=Z\left(p_1\right)=1$ and $Z_2=Z\left(p_2\right)=1$) is

$$\Delta (p_1,p_2):={{\mathrm{JS}}^{+}}_{{\tilde{M}}_{\alpha },\beta }(p_1,p_2)-{\mathrm{JS}}^{M_{\alpha },\beta }(p_1,p_2)=Z_{\alpha }-log\left(Z_{\alpha }\right)-1.$$

(17)

Proposition 6 (Extended/normalized M-JSD Gap).

The following identity holds:

$${{\mathrm{JS}}^{+}}_{{\tilde{M}}_{\alpha },\beta }(p_1,p_2)={\mathrm{JS}}_{M_{\alpha },\beta }(p_1,p_2)+Z_{\alpha }-log\left(Z_{\alpha }\right)-1.$$

Thus, ${{\mathrm{JS}}^{+}}_{{\tilde{M}}_{\alpha },\beta }(p_1,p_2)\ge {\mathrm{JS}}_{M_{\alpha },\beta }(p_1,p_2)$ when $\Delta (p_1,p_2)\ge 0$, and ${{\mathrm{JS}}^{+}}_{{\tilde{M}}_{\alpha },\beta }(p_1,p_2)\le {\mathrm{JS}}_{M_{\alpha },\beta }(p_1,p_2)$ when $\Delta (p_1,p_2)\le 0$.

When we consider the weighted arithmetic mean $A_{\alpha }$, we always have $Z_{\alpha }=1$ for $\alpha \in (0,1)$, and thus the two definitions (Definition 1 and Definition 4) of the A-JSD coincide (i.e., $Z_{\alpha }^A-log\left(Z_{\alpha }^A\right)-1=0$):

$${\mathrm{JS}}_A(p_1,p_2)={\mathrm{JS}}_{\tilde{A}}(p_1,p_2).$$

However, when the weighted mean $M_{\alpha }$ differs from the weighted arithmetic mean (i.e., $M_{\alpha }\ne A_{\alpha }$), the two definitions of the M-JSD ${\mathrm{JS}}_M$ and extended M-JSD ${\mathrm{JS}}_{\tilde{M}}$ differ by the gap expressed in Equation (17).

Remark 7. 

When information is measured in bits, logarithms are taken to base 2, and when information is measured in nats, base e is considered. Thus, we shall generally consider the gap ${\Delta }_b=Z_{\alpha }-{log}_b\left(Z_{\alpha }\right)-1$, where b denotes the base of the logarithm. When $b=e$, we have ${\Delta }_e\ge 0$ for all $Z_{\alpha }>0$. When $b=2$, we have ${\Delta }_2=Z_{\alpha }-{log}_2\left(Z_{\alpha }\right)-1\ge 0$ when $0<Z_{\alpha }\le 1$ or $Z_{\alpha }\ge 2$. But since $Z_{\alpha }\le 2$ (see Equation (7)), the condition simplifies to ${\Delta }_2\ge 0$ if and only if $Z_{\alpha }\le 1$.

Remark 8. 

Although $\sqrt{\mathrm{JS}}$ is a metric distance [5], $\sqrt{{\mathrm{JS}}_G}$ is not a metric distance, as the triangle inequality is not satisfied. It suffices to report a counterexample of the triangle inequality for a triple of points $p_1$, $p_2$, and $p_3$: Consider $p_1=(0.55,0.45)$, $p_2=(0.002,0.998)$, and $p_3=(0.045,0.955)$. Then we have $\sqrt{{\mathrm{JS}}_G}(p_1,p_2)\approx 1.0263227\dots$, $\sqrt{{\mathrm{JS}}_G}(p_1,p_3)\approx 0.63852342\dots$, and $\sqrt{{\mathrm{JS}}_G}(p_3,p_2)\approx 0.19794622\dots$. The triangle inequality fails with an error of

$$\sqrt{{\mathrm{JS}}_G}(p_1,p_2)-(\sqrt{{\mathrm{JS}}_G}(p_1,p_3)+\sqrt{{\mathrm{JS}}_G}(p_3,p_2))\approx 0.1898531\dots .$$

Similarly, the triangle inequality also fails for the extended G-JSD: we have $\sqrt{{\mathrm{JS}}_G^{+}(p_1,p_2)}\approx 1.0788275\dots$, $\sqrt{{\mathrm{JS}}_G^{+}(p_1,p_3)}\approx 0.6691922\dots$, and $\sqrt{{\mathrm{JS}}_G^{+}(p_3,p_2)}\approx 0.1984633\dots$ with a triangle inequality defect value of

$$\sqrt{{\mathrm{JS}}_G^{+}(p_1,p_2)}-(\sqrt{{\mathrm{JS}}_G^{+}(p_1,p_3)}+\sqrt{{\mathrm{JS}}_G^{+}(p_3,p_2)})\approx 0.2111719\dots .$$

2.2. Power JSDs and (Extended) Min-JSD and Max-JSD

Let $P_{\gamma ,\alpha }(a,b):={(\alpha a^{\gamma }+(1-\alpha )b^{\gamma })}^{{\displaystyle \frac{1}{\gamma }}}$ be the $\gamma$-power mean for $\gamma \ne 0$ (with $A_{\alpha }=P_{1,\alpha }$). Further define $P_{0,\alpha }(a,b)=G_{\alpha }(a,b)$ so that $P_{\gamma ,\alpha }$ defines the weighted power means for $\gamma \in \mathbb{R}$ and $\alpha \in (0,1)$ in the reminder. Since $P_{\gamma ,\alpha }(a,b)\le P_{{\gamma }^{\prime },\alpha }(a,b)$ when ${\gamma }^{\prime }\ge \gamma$ for any $a,b>0$, we have

$$Z_{\alpha }^{P_{\gamma }}(p_1,p_2)=\int P_{\gamma ,\alpha }(p_1\left(x\right),p_2\left(x\right))\mathrm{d}\mu \le Z_{\alpha }^{P_{{\gamma }^{\prime }}}(p_1,p_2)=\int P_{{\gamma }^{\prime },\alpha }(p_1\left(x\right),p_2\left(x\right))\mathrm{d}\mu .$$

(18)

Let $P_{\gamma }(a,b)=P_{\gamma ,\frac{1}{2}}(a,b)$. We have ${lim}_{\gamma \to -\infty }{P}_{\gamma }(a,b)=\min (a,b)$ and ${lim}_{\gamma \to +\infty }{P}_{\gamma }(a,b)=\max (a,b)$. Thus, we can define both (extended) min-JSD and (extended) max-JSD. Using the fact that $\min (a,b)={\displaystyle \frac{a+b}{2}}-{\displaystyle \frac{1}{2}}|a-b|$ and $\max (a,b)={\displaystyle \frac{a+b}{2}}+{\displaystyle \frac{1}{2}}|a-b|$, we obtain the extremal mixture normalization terms as follows:

$$\begin{array}{ccc}\hfill Z_{\min }(p_1,p_2)& =& \int \min (p_1,p_2)\mathrm{d}\mu =1-\mathrm{TV}(p_1,p_2),\hfill \end{array}$$

(19)

$$\begin{array}{ccc}\hfill Z_{\max }(p_1,p_2)& =& \int \max (p_1,p_2)\mathrm{d}\mu =1+\mathrm{TV}(p_1,p_2),\hfill \end{array}$$

(20)

where $\mathrm{TV}(p_1,p_2)={\displaystyle \frac{1}{2}}\int |p_1-p_2|\mathrm{d}\mu$ is the total variation distance.

Proposition 7 (max-JSD).

The following upper bound holds for max-JSD:

$$0\le {{\mathrm{JS}}^{+}}_{\tilde{\max }}(p_1,p_2)\le \mathrm{TV}(p_1,p_2).$$

(21)

Furthermore, the following identity relates the two types of max-JSDs:

$${{\mathrm{JS}}^{+}}_{\tilde{\max }}(p_1,p_2)={\mathrm{JS}}_{\tilde{\max }}(p_1,p_2)+\mathrm{TV}(p_1,p_2)-log\left(1+\mathrm{TV}(p_1,p_2)\right).$$

(22)

Proof. 

We have

$${{\mathrm{JS}}^{+}}_{\tilde{\max }}(p_1,p_2):={\displaystyle \frac{1}{2}}\int \left(p_{1}log{\displaystyle \frac{p_1}{\max (p_1,p_2)}}+p_{2}log{\displaystyle \frac{p_2}{\max (p_1,p_2)}}+2\max (p_1,p_2)-(p_1+p_2)\right)\mathrm{d}\mu .$$

Since both $log{\displaystyle \frac{p_1}{\max (p_1,p_2)}}\le 0$ and $log{\displaystyle \frac{p_2}{\max (p_1,p_2)}}\le 0$, and $\max (a,b)={\displaystyle \frac{a+b}{2}}+{\displaystyle \frac{1}{2}}|b-a|$, we have

$${{\mathrm{JS}}^{+}}_{\tilde{\max }}(p_1,p_2)\le \int \left({\displaystyle \frac{p_1+p_2}{2}}+{\displaystyle \frac{1}{2}}|p_2-p_1|-{\displaystyle \frac{p_1+p_2}{2}}\right)\mathrm{d}\mu .$$

That is, ${{\mathrm{JS}}^{+}}_{\tilde{\max }}(p_1,p_2)\le \mathrm{TV}(p_1,p_2)$.

We characterize the gap as follows:

$$\begin{array}{ccc}\hfill {\Delta }_{\max }(p_1,p_2)& =& Z_{\max }(p_1,p_2)-log{Z}_{\max }(p_1,p_2)-1,\hfill \\ & =& \mathrm{TV}(p_1,p_2)-log(1+\mathrm{TV}(p_1,p_2))\ge 0,\hfill \end{array}$$

since $0\le \mathrm{TV}\le 1$. Thus ${{\mathrm{JS}}^{+}}_{\tilde{\max }}(p_1,p_2)\ge {\mathrm{JS}}_{\max }(p_1,p_2)$. □

Proposition 8 (min-JSD).

We have the following lower bound on the extended min-JSD:

$${{\mathrm{JS}}^{+}}_{\tilde{\min }}(p_1,p_2)\ge {\displaystyle \frac{1}{4}}J(p_1,p_2)-\mathrm{TV}(p_1,p_2),$$

where $J(p_1,p_2):=\mathrm{KL}(p_1,p_2)+\mathrm{KL}(p_2,p_1)=\int (p_1-p_2)log{\displaystyle \frac{p_1}{p_2}}\mathrm{d}\mu$ is the Jeffreys’ divergence [2] and

$${{\mathrm{JS}}^{+}}_{\tilde{\min }}(p_1,p_2)={\mathrm{JS}}_{\min }(p_1,p_2)-\mathrm{TV}(p_1,p_2)+log(1-\mathrm{TV}(p_1,p_2)).$$

Proof. 

We have $Z_{\min }(p_1,p_2)=\int \min \{p_1,p_2\}\mathrm{d}\mu =1-\mathrm{TV}(p_1,p_2)\le 1$ and

$$\begin{array}{ccc}\hfill {\Delta }_{\min }(p_1,p_2)& =& Z_{\min }(p_1,p_2)-log{Z}_{\min }(p_1,p_2)-1,\hfill \\ & =& -\mathrm{TV}(p_1,p_2)-log(1-\mathrm{TV}(p_1,p_2))\ge 0,\hfill \end{array}$$

since $-x-log(1-x)\ge 0$ for $x\le 1$. Note that the gap can be arbitrarily large when $\mathrm{TV}(p_1,p_2)\to 1^{-}$.

Thus, we have ${{\mathrm{JS}}^{+}}_{\tilde{\min }}(p_1,p_2)\ge {\mathrm{JS}}_{\min }(p_1,p_2)$.

To get the lower bound, we use the fact that $\min (p_1,p_2)\le \sqrt{p_1{p}_2}$. Indeed, we have

$$\begin{array}{ccc}\hfill {{\mathrm{JS}}^{+}}_{\tilde{\min }}(p_1,p_2)& =& {\displaystyle \frac{1}{2}}\left(\int (p_{1}log{\displaystyle \frac{p_1}{\min (p_1,p_2)}}+p_{2}log{\displaystyle \frac{p_2}{\min (p_1,p_2)}}+2\min (p_1,p_2)-(p_1+p_2)\right)\mathrm{d}\mu ,\hfill \\ & \ge & {\displaystyle \frac{1}{2}}\int \left({\displaystyle \frac{1}{2}}{p}_{1}log{\displaystyle \frac{p_1}{p_2}}+{\displaystyle \frac{1}{2}}{p}_{2}log{\displaystyle \frac{p_2}{p_1}}+2\min (p_1,p_2)-(p_1+p_2)\right)\mathrm{d}\mu ,\hfill \\ & =& {\displaystyle \frac{1}{4}}J(p_1,p_2)-\mathrm{TV}(p_1,p_2).\hfill \end{array}$$

□

Remark 9. 

Let us report the total variation distance between two univariate Gaussian distributions $p_{{\mu }_1,{\sigma }_1}$ and $p_{{\mu }_2,{\sigma }_2}$ in closed form using the error function [42] $\mathrm{erf}\left(x\right)={\displaystyle \frac{1}{\sqrt{\pi }}}{\int }_{-x}^x{e}^{-t^2}\mathrm{d}t$.

• When ${\sigma }_1={\sigma }_2=\sigma$, we have

  $$\mathrm{TV}(p_1,p_2)={\displaystyle \frac{1}{2}}\left|\Phi (x^{*};{\mu }_2,\sigma )-\Phi (x^{*};{\mu }_1,\sigma )\right|,$$

  (23)

  where $\Phi (x;\mu ,\sigma )={\displaystyle \frac{1}{2}}(1+\mathrm{erf}\left({\displaystyle \frac{x-\mu }{\sigma \sqrt{2}}}\right))$ is the cumulative distribution, and

  $$x^{*}={\displaystyle \frac{{\mu }_1^2-{\mu }_2^2}{2({\mu }_1-{\mu }_2)}}.$$

  (24)
• When ${\sigma }_1\ne {\sigma }_2$, we let $x_1={\displaystyle \frac{-b-\sqrt{\Delta }}{2a}}$ and $x_2={\displaystyle \frac{-b+\sqrt{\Delta }}{2a}}$, where $\Delta =b^2-4ac\ge 0$ and

  $$\begin{array}{ccc}\hfill a& =& {\displaystyle \frac{1}{{\sigma }_1^2}}-{\displaystyle \frac{1}{{\sigma }_2^2}},\hfill \end{array}$$

  (25)

  $$\begin{array}{ccc}\hfill b& =& 2\left({\displaystyle \frac{{\mu }_2}{{\sigma }_2}}-{\displaystyle \frac{{\mu }_1}{{\sigma }_1}}\right),\hfill \end{array}$$

  (26)

  $$\begin{array}{ccc}\hfill c& =& {\left({\displaystyle \frac{{\mu }_1}{{\sigma }_1}}\right)}^2-{\left({\displaystyle \frac{{\mu }_2}{{\sigma }_2}}\right)}^2-2log{\displaystyle \frac{{\sigma }_2}{{\sigma }_1}}.\hfill \end{array}$$

  (27)
  The total variation is given by

  $$\begin{array}{c}\hfill {\displaystyle \mathrm{TV}(p_1,p_2)=}\\ & & {\displaystyle \frac{1}{2}}\left(\left|\mathrm{erf}\left({\displaystyle \frac{x_1-{\mu }_1}{{\sigma }_1\sqrt{2}}}\right)-\mathrm{erf}\left({\displaystyle \frac{x_1-{\mu }_2}{{\sigma }_2\sqrt{2}}}\right)\right|+\left|\mathrm{erf}\left({\displaystyle \frac{x_2-{\mu }_1}{{\sigma }_1\sqrt{2}}}\right)-\mathrm{erf}\left({\displaystyle \frac{x_2-{\mu }_2}{{\sigma }_2\sqrt{2}}}\right)\right|\right)\hfill \end{array}$$

  (28)

Next, we shall consider the important case of $p_1$ and $p_2$ belonging to the family of multivariate normal distributions, commonly called Gaussian distributions.

3. Geometric JSDs Between Gaussian Distributions

3.1. Exponential Families

The formula for the G-JSD between two Gaussian distributions was reported in [17] using the more general framework of exponential families. An exponential family [43] is a family of probability measures $\left\{P_{\lambda }\right\}$ with Radon–Nikodym densities $p_{\lambda }$ with respect to $\mu$ expressed canonically as

$$\begin{array}{ccc}\hfill p_{\lambda }\left(x\right)& :=& exp\left(〈\theta \left(\lambda \right),t\left(x\right)〉-F\left(\theta \right)+k\left(x\right)\right),\hfill \\ & =& {\displaystyle \frac{1}{Z\left(\theta \right)}}exp\left(〈\theta \left(\lambda \right),t\left(x\right)〉+k\left(x\right)\right),\hfill \end{array}$$

where $\theta \left(\lambda \right)$ is the natural parameter, $t\left(x\right)$ the sufficient statistic, $k\left(x\right)$ an auxiliary carrier term with respect to $\mu$, and $F\left(\theta \right)$ the cumulant function. The partition function $Z\left(\theta \right)$ is the normalizer denominator: $Z\left(\theta \right)=exp\left(F\right(\theta \left)\right)$. The cumulant function $F\left(\theta \right)=logZ\left(\theta \right)$ is strictly convex and analytic [43], and the partition function $Z\left(\theta \right)=exp\left(F\right(\theta \left)\right)$ is strictly log-convex (and hence also strictly convex).

We consider the exponential family of multivariate Gaussian distributions

$$\mathcal{N}=\{N(\mu ,\Sigma ):\mu \in {\mathbb{R}}^d,\Sigma \in \mathrm{PD}\left(d\right)\},$$

where $\mathrm{PD}\left(d\right)$ denotes the set of symmetric positive–definite matrices of size $d\times d$. Let $\lambda :=({\lambda }_v,{\lambda }_M)=(\mu ,\Sigma )$ denote the compound (vector, matrix) parameter of a Gaussian. The d-variate Gaussian density is given by

$$\begin{array}{ccc}\hfill p_{\lambda }(x;\lambda )& :=& {\displaystyle \frac{1}{{\left(2\pi \right)}^{\frac{d}{2}}\sqrt{|{\lambda }_M|}}}exp\left(-{\displaystyle \frac{1}{2}}{(x-{\lambda }_v)}^{\top }{\lambda }_M^{-1}(x-{\lambda }_v)\right),\hfill \end{array}$$

(29)

where $|·|$ denotes the matrix determinant. The natural parameters $\theta$ are expressed using both a vector parameter ${\theta }_v$ and a matrix parameter ${\theta }_M$ in a compound parameter $\theta =({\theta }_v,{\theta }_M)$. By defining the following compound inner product on a compound (vector, matrix) parameter

$$〈\theta ,{\theta }^{\prime }〉:={\theta }_v^{\top }{\theta }_v^{\prime }+\mathrm{tr}\left({{\theta }_M^{\prime }}^{\top }{\theta }_M\right),$$

(30)

where $\mathrm{tr}(·)$ denotes the matrix trace, we rewrite the Gaussian density of Equation (29) in the canonical form of an exponential family:

$$\begin{array}{ccc}\hfill p_{\theta }(x;\theta )& :=& exp\left(〈t\left(x\right),\theta 〉-F_{\theta }\left(\theta \right)\right)=p_{\lambda }\left(x\right),\hfill \end{array}$$

(31)

where $\theta =\theta \left(\lambda \right)$ with

$$\theta =({\theta }_v,{\theta }_M)=\left({\Sigma }^{-1}\mu ,-{\displaystyle \frac{1}{2}}{\Sigma }^{-1}\right)=\theta \left(\lambda \right)=\left({\lambda }_M^{-1}{\lambda }_v,-{\displaystyle \frac{1}{2}}{\lambda }_M^{-1}\right),$$

(32)

is the compound vector-matrix natural parameter and

$$t\left(x\right)=(x,-x{x}^{\top }),$$

(33)

is the compound vector-matrix sufficient statistic. There is no auxiliary carrier term (i.e., $k\left(x\right)=0$). The function $F_{\theta }$ is given by

$$F_{\theta }\left(\theta \right):={\displaystyle \frac{1}{2}}\left(dlog\pi -log|{\theta }_M|+{\displaystyle \frac{1}{2}}{\theta }_v^{\top }{\theta }_M^{-1}{\theta }_v\right),$$

(34)

Remark 10. 

Beware that when the cumulant function is expressed using the ordinary parameter $\lambda =(\mu ,\Sigma )$, the cumulant function $F_{\theta }\left(\theta \left(\lambda \right)\right)$ is no longer convex:

$$\begin{array}{ccc}\hfill F_{\lambda }\left(\lambda \right)& =& {\displaystyle \frac{1}{2}}\left({\lambda }_v^{\top }{\lambda }_M^{-1}{\lambda }_v+log\left|{\lambda }_M\right|+dlog2\pi \right),\hfill \end{array}$$

(35)

$$\begin{array}{ccc}& =& {\displaystyle \frac{1}{2}}\left({\mu }^{\top }{\Sigma }^{-1}\mu +log|\Sigma |+dlog2\pi \right).\hfill \end{array}$$

(36)

We convert between the ordinary parameterization $\lambda =(\mu ,\Sigma )$ and the natural parameterization $\theta$ using these formulas:

$$\begin{array}{ccc}\hfill \theta =({\theta }_v,{\theta }_M)=\left\{\begin{array}{c}{\theta }_v\left(\lambda \right)={\lambda }_M^{-1}{\lambda }_v={\Sigma }^{-1}\mu \hfill \\ {\theta }_M\left(\lambda \right)={\displaystyle \frac{1}{2}}{\lambda }_M^{-1}={\displaystyle \frac{1}{2}}{\Sigma }^{-1}\hfill \end{array}\right.& \iff & \lambda =({\lambda }_v,{\lambda }_M)=\left\{\begin{array}{c}{\lambda }_v\left(\theta \right)={\displaystyle \frac{1}{2}}{\theta }_M^{-1}{\theta }_v=\mu \hfill \\ {\lambda }_M\left(\theta \right)={\displaystyle \frac{1}{2}}{\theta }_M^{-1}=\Sigma \hfill \end{array}\right.\hfill \end{array}$$

The geometric mixture $p_{{\theta }_1}^{\alpha }{p}_{{\theta }_2}^{1-\alpha }$ of two densities of an exponential family is a density $p_{\alpha {\theta }_1+(1-\alpha ){\theta }_2}$ of the exponential family with the partition function $Z_{\alpha }({\theta }_1,{\theta }_2)=exp(-J_{F,\alpha }({\theta }_1,{\theta }_2))$, where $J_{F,\alpha }({\theta }_1,{\theta }_2)$ denotes the skew Jensen divergence [44,45]:

$$J_{F,\alpha }({\theta }_1,{\theta }_2):=\alpha F\left({\theta }_1\right)+(1-\alpha )F\left({\theta }_2\right)-F(\alpha {\theta }_1+(1-\alpha ){\theta }_2).$$

Therefore, the difference gap of Equation (17) between the G-JSD and the extended G-JSD between exponential family densities is given by

$$\begin{array}{ccc}\hfill \Delta ({\theta }_1,{\theta }_2)& =& exp\left(-J_{F,\alpha }({\theta }_1,{\theta }_2)\right)+J_{F,\alpha }({\theta }_1,{\theta }_2)-1,\hfill \end{array}$$

(37)

$$\begin{array}{ccc}& =& Z_{\alpha }({\theta }_1,{\theta }_2)-log{Z}_{\alpha }({\theta }_1,{\theta }_2)-1,\hfill \end{array}$$

(38)

$$\begin{array}{ccc}& =& Z_{\alpha }({\theta }_1,{\theta }_2)-F(\alpha {\theta }_1+(1-\alpha ){\theta }_2)-1.\hfill \end{array}$$

(39)

Since $Z_{\alpha }=exp(-J_{F,\alpha }({\theta }_1,{\theta }_2))\le 1$, the gap $\Delta$ is negative, and we have

$${{\mathrm{JS}}^{+}}_{{\tilde{G}}_{\alpha },\beta }(p_{{\mu }_1,{\Sigma }_1},p_{{\mu }_2,{\Sigma }_2})\le {\mathrm{JS}}_{G_{\alpha },\beta }(p_{{\mu }_1,{\Sigma }_1},p_{{\mu }_2,{\Sigma }_2}).$$

Corollary 2. 

When $p_1=p_{{\theta }_1}$ and $p_2=p_{{\theta }_2}$ belongs to a same exponential family with the cumulant function $F\left(\theta \right)$, we have

$${\mathrm{JS}}_G(p_{{\theta }_1},p_{{\theta }_2})={\displaystyle \frac{1}{4}}{({\theta }_2-{\theta }_1)}^{\top }(\nabla F\left({\theta }_2\right)-\nabla F\left({\theta }_1\right))-\left({\displaystyle \frac{F\left({\theta }_1\right)+F\left({\theta }_2\right)}{2}}-F\left({\displaystyle \frac{{\theta }_1+{\theta }_2}{2}}\right)\right),$$

(40)

since $J(p_{{\theta }_1},p_{{\theta }_2})=〈{\theta }_2-{\theta }_1,\nabla F\left({\theta }_2\right)-\nabla F\left({\theta }_1\right)〉$ amounts to a symmetrized Bregman divergence.

Proof. 

We have $J(p_{{\theta }_1},p_{{\theta }_2})={({\theta }_2-{\theta }_1)}^{\top }(\nabla F\left({\theta }_2\right)-\nabla F\left({\theta }_1\right))$ and $J(p_{{\theta }_1},p_{{\theta }_2})=J_F({\theta }_1,{\theta }_2)$. □

The extended geometric Jensen–Shannon divergence and geometric Jensen–Shannon divergence between two densities of an exponential family are given by

$$\begin{array}{ccc}\hfill {\mathrm{JS}}_G(p_{{\theta }_1},p_{{\theta }_2})& =& {\displaystyle \frac{1}{4}}{({\theta }_2-{\theta }_1)}^{\top }(\nabla F\left({\theta }_2\right)-\nabla F\left({\theta }_1\right))-\left({\displaystyle \frac{F\left({\theta }_1\right)+F\left({\theta }_2\right)}{2}}-F\left({\displaystyle \frac{{\theta }_1+{\theta }_2}{2}}\right)\right),\hfill \\ \hfill {\mathrm{JS}}_{\tilde{G}}(p_{{\theta }_1},p_{{\theta }_2})& =& {\displaystyle \frac{1}{4}}〈{\theta }_2-{\theta }_1,\nabla F\left({\theta }_2\right)-\nabla F\left({\theta }_1\right)〉-exp(-J_F({\theta }_1,{\theta }_2))-1,\hfill \\ \hfill {{\mathrm{JS}}^{*}}_G(p_{{\theta }_1},p_{{\theta }_2})& =& J_F({\theta }_1,{\theta }_2)\hfill \end{array}$$

Remark 11. 

Given two densities $p_1$ and $p_2$, the family $\mathcal{G}$ of geometric mixtures $\{{\left(p_1{p}_2\right)}_{G_{\alpha }}\propto p_1^{\alpha }{p}_2^{1-\alpha }:\alpha \in (0,1)\}$ forms a 1D exponential family that has been termed the likelihood ratio exponential family [46] (LREF). The cumulant function of this LREF is $F\left(\alpha \right)=-B_{\alpha }(p_1,p_2)$. Hence, $\mathcal{G}$ has also been called a Bhattacharyya arc or Hellinger arc in the literature [47]. However, notice that $\mathrm{KL}(p_i:{\left(p_1{p}_2\right)}_{G_{\alpha }})$ does not necessarily amount to a Bregman divergence, because neither $p_1$ nor $p_2$ belongs to $\mathcal{G}$.

3.2. Closed-Form Formula for Gaussian Distributions

Let us report the corresponding closed-form formula for d-variate Gaussian distributions.

When $\alpha ={\displaystyle \frac{1}{2}}$, we proved that ${\mathrm{JS}}_G(p_1,p_2)={\displaystyle \frac{1}{4}}J(p_1,p_2)-B(p_1,p_2)$ and ${\mathrm{JS}}_{\tilde{G}}^{+}(p_1,p_2)={\displaystyle \frac{1}{4}}J(p_1,p_2)+exp(-B(p_1,p_2))-1$ where $\mathrm{BC}(p_1,p_2)=exp(-B(p_1,p_2))$. Thus, for the case of balanced geometric mixtures, we need to report the closed form for the Jeffreys and Bhattacharyya distances:

$$\begin{array}{ccc}\hfill J(p_{{\mu }_1,{\Sigma }_1},p_{{\mu }_2,{\Sigma }_2})& =& {\displaystyle \frac{1}{2}}\left(\mathrm{tr}\left({\Sigma }_1{\Sigma }_2^{-1}+{\Sigma }_2{\Sigma }_1^{-1}\right)+{({\mu }_1-{\mu }_2)}^{\top }({\Sigma }_1^{-1}+{\Sigma }_2^{-1})({\mu }_1-{\mu }_2)-2d\right),\hfill \\ \hfill B(p_{{\mu }_1,{\Sigma }_1},p_{{\mu }_2,{\Sigma }_2})& =& {\displaystyle \frac{1}{8}}{({\mu }_1-{\mu }_2)}^{\top }{\overline{\Sigma }}^{-1}({\mu }_1-{\mu }_2)+{\displaystyle \frac{1}{2}}log\left({\displaystyle \frac{det\overline{\Sigma }}{\sqrt{det{\Sigma }_{1}det{\Sigma }_2}}}\right),\hfill \end{array}$$

where $\overline{\Sigma }={\displaystyle \frac{1}{2}}\left({\Sigma }_1+{\Sigma }_2\right)$.

Otherwise, for the arbitrary weighted geometric mixture $G_{\alpha }$, define ${\left({\theta }_1{\theta }_2\right)}_{\alpha }=\alpha {\theta }_1+(1-\alpha ){\theta }_2$, the weighted linear interpolation of the natural parameters ${\theta }_1$ and ${\theta }_2$.

Corollary 3. 

The skew G-Jensen–Shannon divergence ${\mathrm{JS}}_{\alpha }^G$ and the dual skew G-Jensen–Shannon divergence ${{\mathrm{JS}}^{*}}_{\alpha }^G$ between two d-variate Gaussian distributions $N({\mu }_1,{\Sigma }_1)$ and $N({\mu }_2,{\Sigma }_2)$ is

$$\begin{array}{ccc}\hfill {\mathrm{JS}}_{G_{\alpha }}(p_{({\mu }_1,{\Sigma }_1)},p_{({\mu }_2,{\Sigma }_2)})& =& \alpha \mathrm{KL}(p_{({\mu }_1,{\Sigma }_1)},p_{({\mu }_{\alpha },{\Sigma }_{\alpha })})+(1-\alpha )\mathrm{KL}(p_{({\mu }_2,{\Sigma }_2)},p_{({\mu }_{\alpha },{\Sigma }_{\alpha })}),\hfill \\ & =& \alpha B_F({\left({\theta }_1{\theta }_2\right)}_{\alpha },{\theta }_1)+(1-\alpha )B_F({\left({\theta }_1{\theta }_2\right)}_{\alpha },{\theta }_2),\hfill \\ & =& {\displaystyle \frac{1}{2}}\left(\mathrm{tr}\left({\Sigma }_{\alpha }^{-1}(\alpha {\Sigma }_1+(1-\alpha ){\Sigma }_2)\right)+log\left({\displaystyle \frac{|{\Sigma }_{\alpha }|}{|{\Sigma }_1{|}^{\alpha }{\left|{\Sigma }_2\right|}^{1-\alpha }}}\right)\right.\hfill \\ & =& \left.+\alpha {({\mu }_{\alpha }-{\mu }_1)}^{\top }{\Sigma }_{\alpha }^{-1}({\mu }_{\alpha }-{\mu }_1)+(1-\alpha ){({\mu }_{\alpha }-{\mu }_2)}^{\top }{\Sigma }_{\alpha }^{-1}({\mu }_{\alpha }-{\mu }_2)-d\right)\hfill \\ \hfill {\mathrm{JS}}_{G_{\alpha }}^{*}(p_{({\mu }_1,{\Sigma }_1)},p_{({\mu }_2,{\Sigma }_2)})& =& (1-\alpha )\mathrm{KL}(p_{({\mu }_{\alpha },{\Sigma }_{\alpha })},p_{({\mu }_1,{\Sigma }_1)})+\alpha \mathrm{KL}(p_{({\mu }_{\alpha },{\Sigma }_{\alpha })},p_{({\mu }_2,{\Sigma }_2)}),\hfill \\ & =& \alpha B_F({\theta }_1,{\left({\theta }_1{\theta }_2\right)}_{\alpha })+(1-\alpha )B_F({\theta }_2,{\left({\theta }_1{\theta }_2\right)}_{\alpha }),\hfill \\ & =& J_{F,\alpha }({\theta }_1,{\theta }_2)=:B_{\alpha }(p_{({\mu }_1,{\Sigma }_1)},p_{({\mu }_2,{\Sigma }_2)}),\hfill \\ & =& {\displaystyle \frac{1}{2}}\left(\alpha {\mu }_1^{\top }{\Sigma }_1^{-1}{\mu }_1+(1-\alpha ){\mu }_2^{\top }{\Sigma }_2^{-1}{\mu }_2-{\mu }_{\alpha }^{\top }{\Sigma }_{\alpha }^{-1}{\mu }_{\alpha }+log{\displaystyle \frac{|{\Sigma }_1{|}^{\alpha }{\left|{\Sigma }_2\right|}^{1-\alpha }}{|{\Sigma }_{\alpha }|}}\right),\hfill \\ \hfill F(\mu ,\Sigma )& =& {\displaystyle \frac{1}{2}}\left({\mu }^{\top }{\Sigma }^{-1}\mu +log|\Sigma |+dlog2\pi \right),\hfill \\ \hfill F({\theta }_v,{\theta }_M)& =& {\displaystyle \frac{1}{2}}\left(dlog\pi -log|{\theta }_M|+{\displaystyle \frac{1}{2}}{\theta }_v^{\top }{\theta }_M^{-1}{\theta }_v\right),\hfill \\ \hfill \Delta ({\theta }_1,{\theta }_2)& =& exp\left(-J_{F,\alpha }({\theta }_1,{\theta }_2)\right)+J_{F,\alpha }({\theta }_1,{\theta }_2)-1,\hfill \end{array}$$

where ${\Sigma }_{\alpha }$ is the matrix harmonic barycenter:

$${\Sigma }_{\alpha }={\left(\alpha {\Sigma }_1^{-1}+(1-\alpha ){\Sigma }_2^{-1}\right)}^{-1},$$

(41)

and

$${\mu }_{\alpha }={\Sigma }_{\alpha }\left(\alpha {\Sigma }_1^{-1}{\mu }_1+(1-\alpha ){\Sigma }_2^{-1}{\mu }_2\right).$$

(42)

4. The Extended and Normalized G-JSDs as Regularizations of the Ordinary JSD

The M-Jensen–Shannon divergence ${\mathrm{JS}}_M(p,q)$ can be interpreted as a regularization of the ordinary JSD:

Proposition 9 (JSD regularization).

For any arbitrary mean M, the following identity holds:

$${\mathrm{JS}}_M(p_1,p_2)=\mathrm{JS}(p_1,p_2)+\mathrm{KL}\left({\displaystyle \frac{p_1+p_2}{2}},{\left(p_1{p}_2\right)}_M\right).$$

(43)

Notice that ${\left(p_1{p}_2\right)}_A={\displaystyle \frac{p_1+p_2}{2}}$.

Proof. 

We have

$$\begin{array}{ccc}\hfill {\mathrm{JS}}_M(p_1,p_2)& :=& {\displaystyle \frac{1}{2}}\left(\mathrm{KL}(p_1,{\left(p_1{p}_2\right)}_M)+\mathrm{KL}(p_2,{\left(p_1{p}_2\right)}_M)\right),\hfill \\ & =& {\displaystyle \frac{1}{2}}\int \left(p_{1}log{\displaystyle \frac{p_1{\left(p_1{p}_2\right)}_A}{{\left(p_1{p}_2\right)}_M{\left(p_1{p}_2\right)}_A}}+p_{2}log{\displaystyle \frac{p_2{\left(p_1{p}_2\right)}_A}{{\left(p_1{p}_2\right)}_M{\left(p_1{p}_2\right)}_A}}\right)\mathrm{d}\mu ,\hfill \\ & =& {\displaystyle \frac{1}{2}}\int \left(p_{1}log{\displaystyle \frac{p_1}{{\left(p_1{p}_2\right)}_A}}+p_{1}log{\displaystyle \frac{{\left(p_1{p}_2\right)}_A}{{\left(p_1{p}_2\right)}_M}}+p_{2}log{\displaystyle \frac{p_2}{{\left(p_1{p}_2\right)}_A}}+p_{2}log{\displaystyle \frac{{\left(p_1{p}_2\right)}_A}{{\left(p_1{p}_2\right)}_M}}\right)\mathrm{d}\mu ,\hfill \\ & =& {\displaystyle \frac{1}{2}}\int \left(p_{1}log{\displaystyle \frac{p_1}{{\left(p_1{p}_2\right)}_A}}+p_{2}log{\displaystyle \frac{p_2}{{\left(p_1{p}_2\right)}_A}}\right)\mathrm{d}\mu +\int {\displaystyle \frac{1}{2}}(p_1+p_2)log{\displaystyle \frac{{\left(p_1{p}_2\right)}_A}{{\left(p_1{p}_2\right)}_M}}\mathrm{d}\mu ,\hfill \\ & =& \mathrm{JS}(p_1,p_2)+\int {\left(p_1{p}_2\right)}_{A}log{\displaystyle \frac{{\left(p_1{p}_2\right)}_A}{{\left(p_1{p}_2\right)}_M}}\mathrm{d}\mu ,\hfill \\ & =& \mathrm{JS}(p_1,p_2)+\mathrm{KL}({\left(p_1{p}_2\right)}_A,{\left(p_1{p}_2\right)}_M).\hfill \end{array}$$

□

Remark 12. 

One way to symmetrize the KLD is to consider two distinct symmetric means $M_1(a,b)=M_1(b,a)$ and $M_2(a,b)=M_2(b,a)$ and define

$${\mathrm{KL}}_{M_1,M_2}(p_1,p_2)=\mathrm{KL}({\left(p_1{p}_2\right)}_{M_1},{\left(p_1{p}_2\right)}_{M_2})={\mathrm{KL}}_{M_1,M_2}(p_2,p_1).$$

We notice that $\sqrt{{\mathrm{KL}}^{A,G}}$ is not a metric distance by reporting a triple of points $(p_1,p_2,p_3)$ that fails the triangle inequality. Consider $p_1=(0.55,0.45)$, $p_2=(0.002,0.998)$, and $p_3=(0.045,0.955)$. We have $\sqrt{{\mathrm{KL}}_{M_1,M_2}(p_1,p_2)}=0.5374165\dots$, $\sqrt{{\mathrm{KL}}_{M_1,M_2}(p_1,p_3)}=0.1759400\dots$, and $\sqrt{{\mathrm{KL}}_{M_1,M_2}(p_3,p_2)}=0.08485931\dots$. The triangle inequality defect is

$$\sqrt{{\mathrm{KL}}_{M_1,M_2}(p_1,p_2)}-(\sqrt{{\mathrm{KL}}_{M_1,M_2}(p_1,p_3)}+\sqrt{{\mathrm{KL}}_{M_1,M_2}(p_3,p_2)})=0.2766171\dots .$$

We can also similarly symmetrize the extended KLD as follows:

$${\mathrm{KL}}_{{\tilde{M}}_1,{\tilde{M}}_2}^{+}(q_1,q_2)={\mathrm{KL}}^{+}({\left(q_1{q}_2\right)}_{{\tilde{M}}_1},{\left(q_1{q}_2\right)}_{{\tilde{M}}_2})={\mathrm{KL}}_{{\tilde{M}}_1,{\tilde{M}}_2}(q_2,q_1).$$

In particular, when $M_1=A$ and $M_2=G$, we get the ${\mathrm{KL}}_{A,M}$ divergence:

$${\mathrm{KL}}_{A,M}(p_1,p_2)={\displaystyle \frac{p_1+p_2}{2}}log{\displaystyle \frac{p_1+p_2}{2\sqrt{p_1{p}_2}}}+log{Z}_G(p_1,p_2),$$

which is related to Taneja T-divergence [48]:

$$T(p_1,p_2)=\int {\displaystyle \frac{p_1+p_2}{2}}log{\displaystyle \frac{p_1+p_2}{2\sqrt{p_1{p}_2}}}.$$

(44)

The T-divergence is an f-divergence [11,12] obtained for the generator $f_T\left(u\right)={\displaystyle \frac{1+u}{2}}log{\displaystyle \frac{1+u}{2\sqrt{u}}}$.

Corollary 4 (JSD lower bound on M-JSD).

We have ${\mathrm{JS}}_M(p,q)\ge \mathrm{JS}(p,q)$.

Proof. 

Since ${\mathrm{JS}}_M(p,q)=\mathrm{JS}(p,q)+\mathrm{KL}\left({\displaystyle \frac{p+q}{2}},{\left(pq\right)}_M\right)$ and $\mathrm{KL}\ge 0$ by Gibbs’ inequality, we have ${\mathrm{JS}}_M(p,q)\ge \mathrm{JS}(p,q)$. □

Since the extended M-JSD is ${\mathrm{JS}}_{{\tilde{M}}_{\alpha },\beta }^{+}(p_1,p_2)={\mathrm{JS}}_{M_{\alpha },\beta }(p_1,p_2)+Z_{\alpha }-log\left(Z_{\alpha }\right)-1$, the extended M-JSD ${\mathrm{JS}}_{{\tilde{M}}_{\alpha },\beta }^{+}$ can also be interpreted as another regularization of the Jensen–Shannon divergence when dealing with probability densities:

$${\mathrm{JS}}_{{\tilde{M}}_{\alpha },\beta }^{+}(p_1,p_2)=\mathrm{JS}(p_1,p_2)+\mathrm{KL}\left({\displaystyle \frac{p_1+p_2}{2}},{\left(p_1{p}_2\right)}_M\right)+Z_{M_{\alpha }}(p_1,p_2)-log\left(Z_{M_{\alpha }}(p_1,p_2)\right)-1.$$

(45)

It is well known that the JSD can be rewritten as a diversity index [4] using concave entropy:

$$\mathrm{JS}(p_1,p_2)=H\left({\displaystyle \frac{p_1+p_2}{2}}\right)-{\displaystyle \frac{H\left(p_1\right)+H\left(p_2\right)}{2}}.$$

(46)

We generalize this decomposition as the difference of a cross-entropy term minus entropies, as follows:

Proposition 10 (M-JSD cross-entropy decomposition).

We have

$${\mathrm{JS}}_M(p_1,p_2)=H^{\times }({\left(p_1{p}_2\right)}_A,{\left(p_1{p}_2\right)}_M)-{\displaystyle \frac{H\left(p_1\right)+H\left(p_2\right)}{2}}.$$

Proof. 

From Proposition 9, we have

$${\mathrm{JS}}_M(p_1,p_2)=\mathrm{JS}(p_1,p_2)+\mathrm{KL}\left({\displaystyle \frac{p_1+p_2}{2}},{\left(p_1{p}_2\right)}_M\right).$$

Since $\mathrm{KL}(p_1,p_2)=H^{\times }(p_1,p_2)-H\left(p_1\right)$, where $H^{\times }(p_1,p_2)=-\int p_{1}log{p}_2\mathrm{d}\mu$ is the cross-entropy and $H\left(p\right)=-\int plogp\mathrm{d}\mu =H^{\times }(p,p)$ is the entropy. Plugging Equation (46) into Equation (43), we get

$$\begin{array}{ccc}\hfill {\mathrm{JS}}_M(p_1,p_2)& =& H\left({\displaystyle \frac{p_1+p_2}{2}}\right)-{\displaystyle \frac{H\left(p_1\right)+H\left(p_2\right)}{2}}+H^{\times }\left({\displaystyle \frac{p_1+p_2}{2}},{\left(p_1{p}_2\right)}_M\right)-H\left({\displaystyle \frac{p_1+p_2}{2}}\right),\hfill \\ & =& H^{\times }\left({\displaystyle \frac{p_1+p_2}{2}},{\left(p_1{p}_2\right)}_M\right)-{\displaystyle \frac{H\left(p_1\right)+H\left(p_2\right)}{2}}.\hfill \end{array}$$

Note that when $M=A$, the arithmetic mean, we have $H^{\times }\left({\displaystyle \frac{p_1+p_2}{2}},{\left(p_1{p}_2\right)}_M\right)=H\left({\displaystyle \frac{p_1+p_2}{2}}\right)$ and we recover the fact that ${\mathrm{JS}}_M(p_1,p_2)=\mathrm{JS}(p_1,p_2)$. □

5. Estimation and Approximation of the Extended and Normalized M-JSDs

Let us recall the two definitions of the extended M-JSD and the normalized M-JSD (for the case of $\alpha =\beta ={\displaystyle \frac{1}{2}}$) between two normalized densities $p_1$ and $p_2$:

$$\begin{array}{ccc}\hfill {\mathrm{JS}}_M(p_1,p_2)& =& {\displaystyle \frac{1}{2}}\left(\mathrm{KL}\left(p_1,{\left(p_1{p}_2\right)}_M\right)+\mathrm{KL}\left(p_2,{\left(p_1{p}_2\right)}_M\right)\right),\hfill \\ \hfill {\mathrm{JS}}_M^{+}(p_1,p_2)& =& {\displaystyle \frac{1}{2}}\left({\mathrm{KL}}^{+}\left(p_1,{\left(p_1{p}_2\right)}_{\tilde{M}}\right)+{\mathrm{KL}}^{+}\left(p_2,{\left(p_1{p}_2\right)}_{\tilde{M}}\right)\right),\hfill \end{array}$$

where ${\left(p_1{p}_2\right)}_M\left(x\right)={\displaystyle \frac{M(p_1\left(x\right),p_2\left(x\right))}{Z_M(p_1,p_2)}}$ (with $Z_M(p_1,p_2)=\int M(p_1\left(x\right),p_2\left(x\right))\mathrm{d}\mu \left(x\right)$) and ${\left(p_1{p}_2\right)}_{\tilde{M}}\left(x\right)=M(p_1\left(x\right),p_2\left(x\right))$.

In practice, one needs to estimate the extended and normalized G-JSDs when they do not admit a closed-form formula.

5.1. Monte Carlo Estimators

To estimate ${\mathrm{JS}}_M(p_1,p_2)$, we can use Monte Carlo samplings to estimate both KLD integrals and mixture normalizers $Z_M$; for example, the normalizer $Z_M(p_1,p_2)$ is estimated by

$${\widehat{Z}}_M(p_1,p_2)={\displaystyle \frac{1}{s}}\sum _{i=1}^s{\displaystyle \frac{1}{r\left(x_i\right)}}M(p_1\left(x_i\right),p_2\left(x_i\right)),$$

where $r\left(x\right)$ is the proposal distribution which can be chosen according to the mean M and the types of probability distributions $p_1$ and $p_2$, and $x_1,\dots ,x_s$ are s identically and independently sampled (iid.) from $r\left(x\right)$. However, since ${\left(p_1{p}_2\right)}_M\left(x\right)$ is now estimated as ${\left(p_1{p}_2\right)}_{\widehat{M}\left(x\right)}$, it is no longer a normalized M-mixture, and thus we consider estimating

$${\mathrm{JS}}_{\widehat{M}}^{+}(p_1,p_2)={\displaystyle \frac{1}{2}}\left({\mathrm{KL}}^{+}\left(p_1,{\left(p_1{p}_2\right)}_{\widehat{M}}\right)+{\mathrm{KL}}^{+}\left(p_2,{\left(p_1{p}_2\right)}_{\widehat{M}}\right)\right)$$

to ensure the non-negativity of the divergence ${\mathrm{JSD}}_{\widehat{M}}$.

Let us consider the estimation of the term

$${\mathrm{KL}}^{+}\left(p_1,{\left(p_1{p}_2\right)}_{\tilde{M}}\right)=\int \left(p_{1}log{\displaystyle \frac{p_1}{M(p_1,p_2)}}+M(p_1,p_2)-p_1\right)\mathrm{d}\mu .$$

By choosing the proposal distribution $r\left(x\right)=p_1\left(x\right)$, we have ${\mathrm{KL}}^{+}\left(p_1,{\left(p_1{p}_2\right)}_{\widehat{M}}\right)\approx \widehat{{\mathrm{KL}}^{+}}\left(p_1,{\left(p_1{p}_2\right)}_{\tilde{M}}\right)$ (for large enough s), where

$${\widehat{\mathrm{KL}}}^{+}\left(p_1,{\left(p_1{p}_2\right)}_{\tilde{M}}\right)={\displaystyle \frac{1}{s}}\sum _{i=1}^s\left(log{\displaystyle \frac{p_1\left(x_i\right)}{M(p_1\left(x_i\right),p_2\left(x_i\right))}}+{\displaystyle \frac{1}{p_1\left(x_i\right)}}M(p_1\left(x_i\right),p_2\left(x_i\right))-1\right).$$

Monte Carlo (MC) stochastic integration [49] is a well-studied topic in statistics, with many results available regarding the consistency and variance of MC estimators.

Note that even if we have a generic formula for the G-JSD between two densities of an exponential family given by Corollary 2, the cumulant function $F\left(\theta \right)$ may not be in closed form [50,51]. This is the case when the sufficient statistic vector of the exponential family is $t\left(x\right)=(x,x^2,\dots ,x^m)$ (for $m\ge 5$), yielding the polynomial exponential family (also called exponential–polynomial family [51]).

5.2. Approximations via $\gamma$-Divergences

One way to circumvent the lack of computational tractable density normalizers is to consider the family of $\gamma$-divergences [39] instead of the KLD:

$${\tilde{D}}_{\gamma }(q_1,q_2)={\displaystyle \frac{1}{\gamma (1+\gamma )}}log{I}_{\gamma }(q_1,q_2)-{\displaystyle \frac{1}{\gamma }}log{I}_{\gamma }(q_1,q_2)+{\displaystyle \frac{1}{1+\gamma }}log{I}_{\gamma }(q_1,q_2),\gamma >0,$$

where

$$I_{\gamma }(q_1,q_2)=\int q_1\left(x\right)q_2^{\gamma }\left(x\right)\mathrm{d}\mu \left(x\right).$$

The $\gamma$-divergences are projective divergences, i.e., they enjoy the property that

$${\tilde{D}}_{\gamma }({\lambda }_1{q}_1,{\lambda }_2{q}_2)={\tilde{D}}_{\gamma }(q_1,q_2),\forall {\lambda }_1>0, {\lambda }_2>0.$$

Furthermore, we have ${lim}_{\gamma \to 0}{\tilde{D}}_{\gamma }(p_1,p_2)=\mathrm{KL}(p_1,p_2)$. (Note that the KLD is not projective.)

Let us define the projective M-JSD:

$${\mathrm{JS}}_{\tilde{M},\gamma }(p_1,p_2)={\displaystyle \frac{1}{2}}\left({\tilde{D}}_{\gamma }\left(p_1,{\left(p_1{p}_2\right)}_{\tilde{M}}\right)+{\tilde{D}}_{\gamma }\left(p_2,{\left(p_1{p}_2\right)}_{\tilde{M}}\right)\right).$$

(47)

We have, for $\gamma =ϵ$, a small enough value (e.g., $ϵ\le {10}^{-3}$), ${\mathrm{JS}}_M(p_1,p_2)\approx {\mathrm{JS}}_{\tilde{M},\gamma }(p_1,p_2)$, since

$$\mathrm{KL}(p_1,{\left(p_1{p}_2\right)}_M){\approx }_{\gamma =ϵ}{\tilde{D}}_{\gamma }(p_1,{\left(p_1{p}_2\right)}_{\tilde{M}}).$$

In particular, for exponential family densities $p_{{\theta }_1}\left(x\right)={\displaystyle \frac{q_{{\theta }_1}\left(x\right)}{Z\left({\theta }_1\right)}}$ and $p_{{\theta }_2}\left(x\right)={\displaystyle \frac{q_{{\theta }_2}\left(x\right)}{Z\left({\theta }_2\right)}}$, we have

$$I_{\gamma }(p_{{\theta }_1},p_{{\theta }_2})=exp\left(F({\theta }_1+\gamma {\theta }_2)-F\left({\theta }_1\right)-\gamma F\left({\theta }_2\right)\right),$$

provided that ${\theta }_1+\gamma {\theta }_2$ belongs to the natural parameter space (otherwise, the integral $I_{\gamma }$ diverges).

Even when $F\left(\theta \right)$ is not known in closed form, we may estimate the $\gamma$-divergence by estimating the $I_{\gamma }$ integrals as follows:

$${\widehat{I}}_{\gamma }(q_{{\theta }_1},q_{{\theta }_2})\approx {\displaystyle \frac{1}{s}}\sum _{i=1}^s{q}_2\left(x_i\right),$$

where $x_1,\dots ,x_s$ are iid. sampled from $p_1\left(x\right)$. For example, we may use Monte Carlo importance sampling methods [52] or exponential family Langevin dynamics [53] to sample densities of exponential family densities with computationally intractable normalizers (e.g., polynomial exponential families).

6. Summary and Concluding Remarks

In this paper, we first recalled the Jensen–Shannon symmetrization (JS-symmetrization) scheme of [17] for an arbitrary statistical dissimilarity $D(·,·)$ using an arbitrary weighted scalar mean $M_{\alpha }$ as follows:

$$D_{M_{\alpha },\beta }^{\mathrm{JS}}(p_1,p_2):=\beta D\left(p_1,{\left(p_1{p}_2\right)}_{M_{\alpha }}\right)+(1-\beta )D\left(p_2,{\left(p_1{p}_2\right)}_{M_{\alpha }}\right),(\alpha ,\beta )\in {(0,1)}^2,$$

In particular, we showed that the skewed Bhattacharyya distance and the Chernoff information can both be interpreted as JS-symmetrizations of the reverse Kullback–Leibler divergence.

Then we defined two types of geometric Jensen–Shannon divergence between probability densities. The first type ${\mathrm{JS}}_M$ requires normalization of M-mixtures and relies on the Kullback–Leibler divergence: ${\mathrm{JS}}_M={\mathrm{KL}}_{M_{\frac{1}{2}},\frac{1}{2}}^{\mathrm{JS}}$. The second type ${\mathrm{JS}}_{\tilde{M}}^{+}$ does not normalize M-mixtures and uses the extended Kullback–Leibler divergence ${\mathrm{KL}}^{+}$ to take into account unnormalized mixtures: ${\mathrm{JS}}_{\tilde{M}}^{+}={\mathrm{KL}}_{{\tilde{M}}_{\frac{1}{2}},\frac{1}{2}}^{{\mathrm{JS}}^{+}}$. When M is the arithmetic mean A, both M-JSD types coincide with the ordinary Jensen–Shannon divergence of Equation (2).

We have shown that both M-JSD types can be interpreted as regularized Jensen–Shannon divergences $\mathrm{JS}$ with additive terms. Namely, we have

$$\begin{array}{ccc}\hfill {\mathrm{JS}}_M(p_1,p_2)& =& \mathrm{JS}(p_1,p_2)+\mathrm{KL}({\left(p_1{p}_2\right)}_A,{\left(p_1{p}_2\right)}_M),\hfill \\ \hfill {\mathrm{JS}}_{\tilde{M}}^{+}(p_1,p_2)& =& {\mathrm{JS}}_M(p_1,p_2)+Z_M(p_1,p_2)-log{Z}_M(p_1,p_2)-1,\hfill \\ & =& \mathrm{JS}(p_1,p_2)+\mathrm{KL}({\left(p_1{p}_2\right)}_A,{\left(p_1{p}_2\right)}_M)+Z_M(p_1,p_2)-log{Z}_M(p_1,p_2)-1,\hfill \end{array}$$

where $Z_M(p_1,p_2)=\int M(p_1,p_2)\mathrm{d}\mu$ is the M-mixture normalizer. The gap between these two types of M-JSD is

$$\begin{array}{ccc}\hfill {\Delta }_M(p_1,p_2)& =& {\mathrm{JS}}_{\tilde{M}}^{+}(p_1,p_2)-{\mathrm{JS}}_M(p_1,p_2),\hfill \\ & =& Z_M(p_1,p_2)-log{Z}_M(p_1,p_2)-1.\hfill \end{array}$$

When taking the geometric mean $M=G$, we showed that both G-JSD types can be expressed using the Jeffreys divergence and the Bhattacharyya divergence (or Bhattacharyya coefficient):

$$\begin{array}{ccc}\hfill {\mathrm{JS}}_G(p_1,p_2)& =& {\displaystyle \frac{1}{4}}J(p_1,p_2)-B(p_1,p_2),\hfill \\ \hfill {\mathrm{JS}}_{\tilde{G}}^{+}(p_1,p_2)& =& {\displaystyle \frac{1}{4}}J(p_1,p_2)+exp(-B(p_1,p_2))-1,\hfill \\ & =& {\displaystyle \frac{1}{4}}J(p_1,p_2)+\mathrm{BC}(p_1,p_2)-1.\hfill \end{array}$$

Thus, the gap between these two types of G-JSD is

$$\begin{array}{ccc}\hfill {\Delta }_G(p_1,p_2)& :=& {\mathrm{JS}}_{\tilde{G}}^{+}(p_1,p_2)-{\mathrm{JS}}_G(p_1,p_2),\hfill \\ & =& \mathrm{BC}(p_1,p_2)+B(p_1,p_2)-1,\hfill \\ & =& Z_G(p_1,p_2)-log{Z}_G(p_1,p_2)-1,\hfill \end{array}$$

since $Z_G(p_1,p_2)=\int \sqrt{p_1{p}_2}\mathrm{d}\mu =\mathrm{BC}(p_1,p_2)$.

Although the square root of the Jensen–Shannon divergence yields a metric distance, this is no longer the case for the geometric-JSD and the extended geometric-JSD: we reported counterexamples in Remark 8. Moreover, we have shown that the KL symmetrization $\sqrt{\mathrm{KL}({\left(p_1{p}_2\right)}_A,{\left(p_1{p}_2\right)}_G)}$ is not a metric distance (Remark 12).

We discussed the merit of the extended G-JSD, which does not require normalization of the geometric mixture, in Section 5, and we showed how to approximate the G-JSD using the projective $\gamma$-divergences [39] for $\gamma =ϵ$, a small enough value (i.e., $\gamma =ϵ={10}^{-3}$). From the viewpoint of information geometry, the extended G-JSD has been shown to be an f-divergence [13] (separable divergence), while the G-JSD is not separable in general because of the normalization of mixtures (with the exception of the ordinary JSD, which is an f-divergence because the arithmetic mixtures do not require normalization).

We studied power JSDs by considering the power means and studied the $\pm \infty$ limits, the extended max-JSD, and the min-JSD: We proved that the extended max-JSD is upper-bounded by the total variation distance $\mathrm{TV}(p_1,p_2)={\displaystyle \frac{1}{2}}\int |p_1-p_2|\mathrm{d}\mu$:

$$0\le {\mathrm{JS}}_{\tilde{\max }}^{+}(p_1,p_2)\le \mathrm{TV}(p_1,p_2),$$

and that the extended min-JSD is lower-bounded as follows:

$${\mathrm{JS}}_{\tilde{\min }}^{+}(p_1,p_2)\ge {\displaystyle \frac{1}{4}}J(p_1,p_2)-\mathrm{TV}(p_1,p_2),$$

where J denotes the Jeffreys divergence: $J(p_1,p_2)=\mathrm{KL}(p_1,p_2)+\mathrm{KL}(p_2,p_1)$.

The advantage of using the extended G-JSD is that we do not need to normalize geometric mixtures, while this novel divergence is proven to be an f-divergence [13] and retains the property that it amounts to a regularization of the ordinary Jensen–Shannon divergence by an extra additive gap term.

Finally, we expressed ${\mathrm{JS}}_G$ (Equation (41)) and ${\mathrm{JS}}_{\tilde{G}}^{+}$ (Equation (41)) for exponential families, characterized the gap between these two types of divergences as a function of the cumulant and partition functions, and reported a corresponding explicit formula for the multivariate Gaussian (exponential) family. The G-JSD between Gaussian distributions has already been used successfully in many applications [30,32,33,34,35,36,37,38].