On the Jensen–Shannon Symmetrization of Distances Relying on Abstract Means

Abstract

The Jensen–Shannon divergence is a renowned bounded symmetrization of the unbounded Kullback–Leibler divergence which measures the total Kullback–Leibler divergence to the average mixture distribution. However, the Jensen–Shannon divergence between Gaussian distributions is not available in closed form. To bypass this problem, we present a generalization of the Jensen–Shannon (JS) divergence using abstract means which yields closed-form expressions when the mean is chosen according to the parametric family of distributions. More generally, we define the JS-symmetrizations of any distance using parameter mixtures derived from abstract means. In particular, we first show that the geometric mean is well-suited for exponential families, and report two closed-form formula for (i) the geometric Jensen–Shannon divergence between probability densities of the same exponential family; and (ii) the geometric JS-symmetrization of the reverse Kullback–Leibler divergence between probability densities of the same exponential family. As a second illustrating example, we show that the harmonic mean is well-suited for the scale Cauchy distributions, and report a closed-form formula for the harmonic Jensen–Shannon divergence between scale Cauchy distributions. Applications to clustering with respect to these novel Jensen–Shannon divergences are touched upon.

1. Introduction and Motivations

1.1. Kullback–Leibler Divergence and Its Symmetrizations

Let $(\mathcal{X},\mathcal{A})$ be a measurable space [1] where $\mathcal{X}$ denotes the sample space and $\mathcal{A}$ the $\sigma$-algebra of measurable events. Consider a positive measure $\mu$ (usually the Lebesgue measure ${\mu }_L$ with Borel $\sigma$-algebra $\mathcal{B}\left({\mathbb{R}}^d\right)$ or the counting measure ${\mu }_c$ with power set $\sigma$-algebra $2^{\mathcal{X}}$). Denote by $\mathcal{P}$ the set of probability distributions.

The Kullback–Leibler Divergence [2] (KLD) $\mathrm{KL}:\mathcal{P}\times \mathcal{P}\to [0,\infty ]$ is the most fundamental distance [2] between probability distributions, defined by:

$$\mathrm{KL}(P:Q):=\int plog\frac{p}{q}\mathrm{d}\mu ,$$

(1)

where p and q denote the Radon–Nikodym derivatives of probability measures P and Q with respect to $\mu$ (with $P,Q\ll \mu$). The KLD expression between P and Q in Equation (1) is independent of the dominating measure $\mu$.

Table A1. Summary of Distances and Their Notations.

Weighted mean	$M_{\alpha }$, $\alpha \in (0,1)$
Arithmetic mean	$A_{\alpha }(x,y)=(1-\alpha )x+\alpha y$
Geometric mean	$G_{\alpha }(x,y)=x^{1-\alpha }{y}^{\alpha }$
Harmonic mean	$H_{\alpha }(x,y)=\frac{xy}{(1-\alpha )y+\alpha x}$
Power mean	$P_{\alpha }^p(x,y)={((1-\alpha )x^p+\alpha y^p)}^{\frac{1}{p}},p\in \mathbb{R}\setminus \left\{0\right\}$, ${lim}_{p\to 0}{P}_{\alpha }^p=G$
Quasi-arithmetic mean	$M_{\alpha }^f(x,y)=f^{-1}((1-\alpha )f\left(x\right)+\alpha f\left(y\right))$, f strictly monotonous
M-mixture	$Z_{\alpha }^M(p,q)={\int }_{t\in \mathcal{X}}{M}_{\alpha }(p\left(t\right),q\left(t\right))\mathrm{d}\mu \left(t\right)$ with $Z_{\alpha }^M(p,q)={\int }_{t\in \mathcal{X}}{M}_{\alpha }(p\left(t\right),q\left(t\right))\mathrm{d}\mu \left(t\right)$
Statistical distance	$D(p:q)$
Dual/reverse distance $D^{*}$	$D^{*}(p:q):=D(q:p)$
Kullback-Leibler divergence	$\mathrm{KL}(p:q)=\int p\left(x\right)log\frac{p\left(x\right)}{q\left(x\right)}\mathrm{d}\mu \left(x\right)$
reverse Kullback-Leibler divergence	${\mathrm{KL}}^{*}(p:q)=\mathrm{KL}(q:p)=\int q\left(x\right)log\frac{q\left(x\right)}{p\left(x\right)}\mathrm{d}\mu \left(x\right)$
Jeffreys divergence	$J(p;q)=\mathrm{KL}(p:q)+\mathrm{KL}(q:p)=\int (p\left(x\right)-q\left(x\right))log\frac{p\left(x\right)}{q\left(x\right)}\mathrm{d}\mu \left(x\right)$
Resistor divergence	$\frac{1}{R(p;q)}=\frac{1}{2}\left(\frac{1}{\mathrm{KL}(p:q)}+\frac{1}{\mathrm{KL}(q:p)}\right)$. $R(p;q)=\frac{2J(p;q)}{\mathrm{KL}(p:q)\mathrm{KL}(q:p)}$
skew K-divergence	$K_{\alpha }(p:q)=\int p\left(x\right)log\frac{p\left(x\right)}{(1-\alpha )p\left(x\right)+\alpha q\left(x\right)}\mathrm{d}\mu \left(x\right)$
Jensen-Shannon divergence	$\mathrm{JS}(p,q)=\frac{1}{2}\left(\mathrm{KL}\left(p:\frac{p+q}{2}\right)+\mathrm{KL}\left(q:\frac{p+q}{2}\right)\right)$
skew Bhattacharrya divergence	$B_{\alpha }(p:q)=-log{\int }_{\mathcal{X}}p{\left(x\right)}^{1-\alpha }q{\left(x\right)}^{\alpha }\mathrm{d}\mu \left(x\right)$
Hellinger distance	$D_H(p,q)=\sqrt{1-{\int }_{\mathcal{X}}\sqrt{p\left(x\right)}\sqrt{q\left(x\right)}\mathrm{d}\mu \left(x\right)}$
$\alpha$-divergences	$I_{\alpha }(p:q)=\int \left(\alpha p\left(x\right)+(1-\alpha )q\left(x\right)-p{\left(x\right)}^{\alpha }q{\left(x\right)}^{1-\alpha }\right)\mathrm{d}\mu \left(x\right),\alpha \notin \{0,1\}$ $I_{\alpha }(p:q)=A_{\alpha }(q:p)-G_{\alpha }(q:p)$
Mahalanobis distance	$D_Q(p:q)=\sqrt{{(p-q)}^{\top }Q(p-q)}$ for a positive-definite matrix $Q\succ 0$
f-divergence	$I_f(p:q)=\int p\left(x\right)f\left(\frac{q\left(x\right)}{p\left(x\right)}\right)\mathrm{d}\mu \left(x\right)$, with $f\left(1\right)=f^{\prime }\left(1\right)=0$ f strictly convex at 1
reverse f-divergence	$I_f^{*}(p:q)=\int q\left(x\right)f\left(\frac{p\left(x\right)}{q\left(x\right)}\right)\mathrm{d}\mu \left(x\right)=I_{f^{\diamond }}(p:q)$ for $f^{\diamond }\left(u\right)=uf\left(\frac{1}{u}\right)$
J-symmetrized f-divergence	$J_f(p;q)=\frac{1}{2}(I_f(p:q)+I_f(q:p))$
JS-symmetrized f-divergence	$I_f^{\alpha }(p;q):=(1-\alpha )I_f(p:{\left(pq\right)}_{\alpha })+\alpha I_f(q:{\left(pq\right)}_{\alpha })=I_{f_{\alpha }^{\mathrm{JS}}}(p:q)$ for $f_{\alpha }^{\mathrm{JS}}\left(u\right):=(1-\alpha )f(\alpha u+1-\alpha )+\alpha f\left(\alpha +\frac{1-\alpha }{u}\right)$
Parameter distance
Bregman divergence	$B_F(\theta :{\theta }^{\prime }):=F\left(\theta \right)-F\left({\theta }^{\prime }\right)-\langle \theta -{\theta }^{\prime },\nabla F\left({\theta }^{\prime }\right)\rangle$
skew Jeffreys-Bregman divergence	$S_F^{\alpha }=(1-\alpha )B_F(\theta :{\theta }^{\prime })+\alpha B_F({\theta }^{\prime }:\theta )$
skew Jensen divergence	$J_F^{\alpha }(\theta :{\theta }^{\prime }):={\left(F\left(\theta \right)F\left({\theta }^{\prime }\right)\right)}_{\alpha }-F\left({\left(\theta {\theta }^{\prime }\right)}_{\alpha }\right)$
Jensen-Bregman divergence	${\mathrm{JB}}_F(\theta ;{\theta }^{\prime })=\frac{1}{2}\left(B_F\left(\theta :\frac{\theta +{\theta }^{\prime }}{2}\right)+B_F\left({\theta }^{\prime }:\frac{\theta +{\theta }^{\prime }}{2}\right)\right)=J_F(\theta ;{\theta }^{\prime })$.
Generalized Jensen-Shannon divergences
skew J-symmetrization	$J_D^{\alpha }(p:q):=(1-\alpha )D\left(p:q\right)+\alpha D\left(q:p\right)$
skew $\mathrm{JS}$-symmetrization	${\mathrm{JS}}_D^{\alpha }(p:q):=(1-\alpha )D\left(p:(1-\alpha )p+\alpha q\right)+\alpha D\left(q:(1-\alpha )p+\alpha q\right)$
skew M-Jensen-Shannon divergence	${\mathrm{JS}}^{M_{\alpha }}(p:q):=(1-\alpha )\mathrm{KL}\left(p:{\left(pq\right)}_{\alpha }^M\right)+\alpha \mathrm{KL}\left(q:{\left(pq\right)}_{\alpha }^M\right)$
skew M-$\mathrm{JS}$-symmetrization	${\mathrm{JS}}_D^{M_{\alpha }}(p:q):=(1-\alpha )D\left(p:{\left(pq\right)}_{\alpha }^M\right)+\alpha D\left(q:{\left(pq\right)}_{\alpha }^M\right)$
N-Jeffreys divergence	$J^{N_{\beta }}(p:q):=N_{\beta }(\mathrm{KL}\left(p:q\right),\mathrm{KL}\left(q:p\right))$
N-J D divergence	$J_D^{N_{\beta }}(p:q)=N_{\beta }(D(p:q),D(q:p))$
skew $(M,N)$-D JS divergence	${\mathrm{JS}}_D^{M_{\alpha },N_{\beta }}(p:q):=N_{\beta }\left(D\left(p:{\left(pq\right)}_{\alpha }^M\right),D\left(q:{\left(pq\right)}_{\alpha }^M\right)\right)$

summarizes the various distances and their notations used in this paper.

The KLD is also called the relative entropy [2] because it can be written as the difference of the cross-entropy minus the entropy:

$$\mathrm{KL}(p:q)=h_{\times }(p:q)-h\left(p\right),$$

(2)

where $h_{\times }$ denotes the cross-entropy [2]:

$$h_{\times }(p:q):=\int plog\frac{1}{q}\mathrm{d}\mu ,$$

(3)

and

$$h\left(p\right):=\int plog\frac{1}{p}\mathrm{d}\mu =h_{\times }(p:p),$$

(4)

denotes the Shannon entropy [2]. Although the formula of the Shannon entropy in Equation (4) unifies both the discrete case and the continuous case of probability distributions, the behavior of entropy in the discrete case and the continuous case is very different: When $\mu ={\mu }_c$, Equation (4) yields the discrete Shannon entropy which is always positive and upper bounded by $log\left|\mathcal{X}\right|$. When $\mu ={\mu }_L$, Equation (4) defines the Shannon differential entropy which may be negative and unbounded [2] (e.g., the differential entropy of the Gaussian distribution $N(m,\sigma )$ is $\frac{1}{2}log\left(2\pi e{\sigma }^2\right)$). See also [3] for further important differences between the discrete case and the continuous case.

In general, the KLD is an asymmetric distance (i.e., $\mathrm{KL}(p:q)\ne \mathrm{KL}(q:p)$, hence the argument separator notation using the delimiter ‘:’) In information theory [2], it is customary to use the double bar notation ‘‖’ instead of the comma ‘,’ notation to avoid confusion with joint random variables. The reverse KL divergence or dual KL divergence is:

$${\mathrm{KL}}^{*}(P:Q):=\mathrm{KL}(Q:P)=\int qlog\frac{q}{p}\mathrm{d}\mu .$$

(5)

In general, the reverse distance or dual distance for a distance D is written as:

$$D^{*}(p:q):=D(q:p).$$

(6)

One way to symmetrize the KLD is to consider the Jeffreys Divergence [4] (JD, Sir Harold Jeffreys (1891–1989) was a British statistician.):

$$J(p;q):=\mathrm{KL}(p:q)+\mathrm{KL}(q:p)=\int (p-q)log\frac{p}{q}\mathrm{d}\mu =J(q;p).$$

(7)

However, this symmetric distance is not upper bounded, and its sensitivity can raise numerical issues in applications. Here, we used the optional argument separator notation ‘;’ to emphasize that the distance is symmetric but not necessarily a metric distance. This notation matches the notational convention of the mutual information if two joint random variables in information theory [2].

The symmetrization of the KLD may also be obtained using the harmonic mean instead of the arithmetic mean, yielding the resistor average distance [5] $R(p;q)$:

$$\begin{array}{ccc}\hfill \frac{1}{R(p;q)}& =& \frac{1}{2}\left(\frac{1}{\mathrm{KL}(p:q)}+\frac{1}{\mathrm{KL}(q:p)}\right),\hfill \end{array}$$

(8)

$$\begin{array}{ccc}\hfill R(p;q)& =& \frac{2\left(\mathrm{KL}(p:q)+\mathrm{KL}(q:p)\right)}{\mathrm{KL}(p:q)\mathrm{KL}(q:p)}=\frac{2J(p;q)}{\mathrm{KL}(p:q)\mathrm{KL}(q:p)}.\hfill \end{array}$$

(9)

Another famous symmetrization of the KLD is the Jensen–Shannon Divergence [6] (JSD) defined by:

$$\begin{array}{ccc}\hfill \mathrm{JS}(p;q)& :=& \frac{1}{2}\left(\mathrm{KL}\left(p:\frac{p+q}{2}\right)+\mathrm{KL}\left(q:\frac{p+q}{2}\right)\right),\hfill \end{array}$$

(10)

$$\begin{array}{cc}=& \frac{1}{2}\int \left(plog\frac{2p}{p+q}+qlog\frac{2q}{p+q}\right)\mathrm{d}\mu .\hfill \end{array}$$

(11)

This distance can be interpreted as the total divergence to the average distribution (see Equation (10)). The JSD can be rewritten as a Jensen divergence (or Burbea–Rao divergence [7]) for the negentropy generator $-h$ (called Shannon information):

$$\mathrm{JS}(p;q)=h\left(\frac{p+q}{2}\right)-\frac{h\left(p\right)+h\left(q\right)}{2}.$$

(12)

An important property of the Jensen–Shannon divergence compared to the Jeffreys divergence is that this distance is always bounded:

$$0\le \mathrm{JS}(p:q)\le log2.$$

(13)

This follows from the fact that

$$\mathrm{KL}\left(p:\frac{p+q}{2}\right)=\int plog\frac{2p}{p+q}\mathrm{d}\mu \le \int plog\frac{2p}{p}\mathrm{d}\mu =log2.$$

(14)

Finally, the square root of the JSD (i.e., $\sqrt{\mathrm{JS}}$) yields a metric distance satisfying the triangular inequality [8,9]. The JSD has found applications in many fields such as bioinformatics [10] and social sciences [11], just to name a few. Recently, the JSD has gained attention in the deep learning community with the Generative Adversarial Networks (GANs) [12]. In computer vision and pattern recognition, one often relies on information-theoretic techniques to perform registration and recognition tasks. For example, in [13], the authors use a mixture of Principal Axes Registrations (mPAR) whose parameters are estimated by minimizing the KLD between the considered two-point distributions. In [14], the authors parameterize both shapes and deformations using Gaussian Mixture Models (GMMs) to perform non-rigid shape registration. The lack of closed-form formula for the KLD between GMMs [15] spurred the use of other statistical distances which admit a closed-form expression for GMMs. For example, in [16], shape registration is performed by using the Jensen-Rényi divergence between GMMs. See also [17] for other information-theoretic divergences that admit closed-form formula for some statistical mixtures extending GMMs.

In information geometry [18], the KLD, JD and JSD are invariant divergences which satisfy the property of information monotonicity [18]. The class of (separable) distances satisfying the information monotonicity are exhaustively characterized as Csiszár’s f-divergences [19]. A f-divergence is defined for a convex generator function f strictly convex at 1 (with $f\left(1\right)=f^{\prime }\left(1\right)=0$) by:

$$I_f(p:q)=\int pf\left(\frac{q}{p}\right)\mathrm{d}\mu .$$

(15)

The Jeffreys and Jensen–Shannon f-generators are:

$$\begin{array}{ccc}\hfill f_J\left(u\right)& :=& (u-1)logu,\hfill \end{array}$$

(16)

$$\begin{array}{ccc}\hfill f_{\mathrm{JS}}\left(u\right)& :=& -(u+1)log\frac{1+u}{2}+ulogu.\hfill \end{array}$$

(17)

1.2. Statistical Distances and Parameter Divergences

In information and probability theory, the term “divergence” informally means a statistical distance [2]. However in information geometry [18], a divergence has a stricter meaning of being a smooth parametric distance (called a contrast function in [20]) from which a dual geometric structure can be derived [21,22].

Consider parametric distributions $p_{\theta }$ belonging to a parametric family of distributions $\{p_{\theta }:\theta \in \mathsf{\Theta }\}$ (e.g., Gaussian family or Cauchy family), where $\mathsf{\Theta }$ denotes the parameter space. Then a statistical distance D between distributions $p_{\theta }$ and $p_{{\theta }^{\prime }}$ amount to an equivalent parameter distance:

$$P(\theta :{\theta }^{\prime }):=D(p_{\theta }:p_{{\theta }^{\prime }}).$$

(18)

For example, the KLD between two distributions belonging to the same exponential family (e.g., Gaussian family) amount to a reverse Bregman divergence for the cumulant generator F of the exponential family [23]:

$$\mathrm{KL}(p_{\theta }:p_{{\theta }^{\prime }})=B_F^{*}(\theta :{\theta }^{\prime })=B_F({\theta }^{\prime }:\theta ).$$

(19)

A Bregman divergence $B_F$ is defined for a strictly convex and differentiable generator F as:

$$B_F(\theta :{\theta }^{\prime }):=F\left(\theta \right)-F\left({\theta }^{\prime }\right)-\langle \theta -{\theta }^{\prime },\nabla F\left({\theta }^{\prime }\right)\rangle ,$$

(20)

where $\langle ·,·\rangle$ is an inner product (usually the Euclidean dot product for vector parameters).

Similar to the interpretation of the Jensen–Shannon divergence (statistical divergence) as a Jensen divergence for the negentropy generator, the Jensen–Bregman divergence [7] ${\mathrm{JB}}_F$ (parametric divergence JBD) amounts to a Jensen divergence $J_F$ for a strictly convex generator $F:\mathsf{\Theta }\to \mathbb{R}$:

$$\begin{array}{ccc}\hfill {\mathrm{JB}}_F(\theta :{\theta }^{\prime })& :=& \frac{1}{2}\left(B_F\left(\theta :\frac{\theta +{\theta }^{\prime }}{2}\right)+B_F\left({\theta }^{\prime }:\frac{\theta +{\theta }^{\prime }}{2}\right)\right),\hfill \end{array}$$

(21)

$$\begin{array}{cc}=& \frac{F\left(\theta \right)+F\left({\theta }^{\prime }\right)}{2}-F\left(\frac{\theta +{\theta }^{\prime }}{2}\right)=:J_F(\theta :{\theta }^{\prime }),\hfill \end{array}$$

(22)

Let us introduce the notation ${\left({\theta }_p{\theta }_q\right)}_{\alpha }:=(1-\alpha ){\theta }_p+\alpha {\theta }_q$ to denote the linear interpolation (LERP) of the parameters. Then we have more generally that the skew Jensen–Bregman divergence ${\mathrm{JB}}_F^{\alpha }(\theta :{\theta }^{\prime })$ amounts to a skew Jensen divergence $J_F^{\alpha }(\theta :{\theta }^{\prime })$:

$$\begin{array}{ccc}\hfill {\mathrm{JB}}_F^{\alpha }(\theta :{\theta }^{\prime })& :=& (1-\alpha )B_F\left(\theta :{\left(\theta {\theta }^{\prime }\right)}_{\alpha }\right)+\alpha B_F\left({\theta }^{\prime }:{\left(\theta {\theta }^{\prime }\right)}_{\alpha })\right),\hfill \end{array}$$

(23)

$$\begin{array}{ccc}& =& {\left(F\left(\theta \right)F\left({\theta }^{\prime }\right)\right)}_{\alpha }-F\left({\left(\theta {\theta }^{\prime }\right)}_{\alpha }\right)=:J_F^{\alpha }(\theta :{\theta }^{\prime }),\hfill \end{array}$$

(24)

1.3. J-Symmetrization and $JS$-Symmetrization of Distances

For any arbitrary distance $D(p:q)$, we can define its skew J-symmetrization for $\alpha \in [0,1]$ by:

$$J_D^{\alpha }(p:q):=(1-\alpha )D\left(p:q\right)+\alpha D\left(q:p\right),$$

(25)

and its JS-symmetrization by:

$$\begin{array}{ccc}\hfill {\mathrm{JS}}_D^{\alpha }(p:q)& :=& (1-\alpha )D\left(p:(1-\alpha )p+\alpha q\right)+\alpha D\left(q:(1-\alpha )p+\alpha q\right),\hfill \end{array}$$

(26)

$$\begin{array}{cc}=& (1-\alpha )D\left(p:{\left(pq\right)}_{\alpha }\right)+\alpha D\left(q:{\left(pq\right)}_{\alpha }\right).\hfill \end{array}$$

(27)

Usually, $\alpha =\frac{1}{2}$, and for notational brevity, we drop the superscript: ${\mathrm{JS}}_D(p:q):={\mathrm{JS}}_D^{\frac{1}{2}}(p:q)$. The Jeffreys divergence is twice the J-symmetrization of the KLD, and the Jensen–Shannon divergence is the $\mathrm{JS}$-symmetrization of the KLD.

The J-symmetrization of a f-divergence $I_f$ is obtained by taking the generator

$$f_{\alpha }^J\left(u\right)=(1-\alpha )f\left(u\right)+\alpha f^{\diamond }\left(u\right),$$

(28)

where $f^{\diamond }\left(u\right)=uf\left(\frac{1}{u}\right)$ is the conjugate generator:

$$I_{f^{\diamond }}(p:q)=I_f^{*}(p:q)=I_f(q:p).$$

(29)

The $\mathrm{JS}$-symmetrization of a f-divergence

$$I_f^{\alpha }(p:q):=(1-\alpha )I_f(p:{\left(pq\right)}_{\alpha })+\alpha I_f(q:{\left(pq\right)}_{\alpha }),$$

(30)

with ${\left(pq\right)}_{\alpha }=(1-\alpha )p+\alpha q$ is obtained by taking the generator

$$f_{\alpha }^{\mathrm{JS}}\left(u\right):=(1-\alpha )f(\alpha u+1-\alpha )+\alpha f\left(\alpha +\frac{1-\alpha }{u}\right).$$

(31)

We check that we have:

$$I_f^{\alpha }(p:q)=(1-\alpha )I_f(p:{\left(pq\right)}_{\alpha })+\alpha I_f(q:{\left(pq\right)}_{\alpha })=I_f^{1-\alpha }(q:p)=I_{f_{\alpha }^{\mathrm{JS}}}(p:q).$$

(32)

A family of symmetric distances unifying the Jeffreys divergence with the Jensen–Shannon divergence was proposed in [24]. Finally, let us mention that once we have symmetrized a distance D, we may also metrize this symmetric distance by choosing (when it exists) the largest exponent $\delta >0$ such that $D^{\delta }$ becomes a metric distance [8,25,26,27,28].

1.4. Contributions and Paper Outline

The paper is organized as follows:

Section 2 reports the special case of mixture families in information geometry [18] for which the Jensen–Shannon divergence can be expressed as a Bregman divergence (Theorem 1), and highlight the lack of closed-form formula when considering exponential families. This fact precisely motivated this work.

Section 3 introduces the generalized Jensen–Shannon divergences using statistical mixtures derived from abstract weighted means (Definitions 2 and 5), presents the JS-symmetrization of statistical distances, and report a sufficient condition to get bounded JS-symmetrizations (Property 1).

In Section 4.1, we consider the calculation of the geometric JSD between members of the same exponential family (Theorem 2) and instantiate the formula for the multivariate Gaussian distributions (Corollary 1). We discuss about applications for k-means clustering in Section 4.1.2. In Section 4.2, we illustrate the method with another example that calculates in closed form the harmonic JSD between scale Cauchy distributions (Theorem 4).

Finally, we wrap up and conclude this work in Section 5.

2. Jensen–Shannon Divergence in Mixture and Exponential Families

We are interested to calculate the JSD between densities belonging to parametric families of distributions.

A trivial example is when $p=(p_0,\dots ,p_D)$ and $q=(q_0,\dots ,q_D)$ are categorical distributions: The average distribution $\frac{p+q}{2}$ is a again categorical distribution, and the JSD is expressed plainly as:

$$\mathrm{JS}(p,q)=\frac{1}{2}\sum _{i=0}^D\left(p_{i}log\frac{2p_i}{p_i+q_i}+q_{i}log\frac{2q_i}{p_i+q_i}\right).$$

(33)

Another example is when $p=m_{{\theta }_p}$ and $q=m_{{\theta }_q}$ both belong to the same mixture family [18] $\mathcal{M}$:

$$\mathcal{M}:=\left\{m_{\theta }\left(x\right)=\left(1-\sum _{i=1}^D{\theta }_i{p}_i\left(x\right)\right)p_0\left(x\right)+\sum _{i=1}^D{\theta }_i{p}_i\left(x\right):{\theta }_i>0,\sum _i{\theta }_i<1\right\},$$

(34)

for linearly independent component distributions $p_0,p_1,\dots ,p_D$. We have [29]:

$$\mathrm{KL}(m_{{\theta }_p}:m_{{\theta }_q})=B_F({\theta }_p:{\theta }_q),$$

(35)

where $B_F$ is a Bregman divergence defined in Equation (20) obtained for the convex negentropy generator [29] $F\left(\theta \right)=-h\left(m_{\theta }\right)$. The proof that $F\left(\theta \right)$ is a strictly convex function is not trivial [30].

The mixture families include the family of categorical distributions over a finite alphabet $\mathcal{X}=\{E_0,\dots ,E_D\}$ (the D-dimensional probability simplex) since those categorical distributions form a mixture family with $p_i\left(x\right):=Pr(X=E_i)={\delta }_{E_i}\left(x\right)$. Beware that mixture families impose to prescribe the component distributions. Therefore, a density of a mixture family is a special case of statistical mixtures (e.g., GMMs) with prescribed component distributions.

The mathematical identity of Equation (35) that does not yield a practical formula since $F\left(\theta \right)$ is usually not itself available in closed form. Worse, the Bregman generator can be non-analytic [31]. Nevertheless, this identity is useful for computing the right-sided Bregman centroid (left KL centroid of mixtures) since this centroid is equivalent to the center of mass, and independent of the Bregman generator [29].

Since the mixture of mixtures is also a mixture, specifically

$$\frac{m_{{\theta }_p}+m_{{\theta }_q}}{2}=m_{\frac{{\theta }_p+{\theta }_q}{2}}\in \mathcal{M},$$

(36)

it follows that we get a closed-form expression for the JSD between mixtures belonging to $\mathcal{M}$.

Theorem 1 (JSD between mixtures).

The Jensen–Shannon divergence between two distributions $p=m_{{\theta }_p}$ and $q=m_{{\theta }_q}$ belonging to the same mixture family $\mathcal{M}$ is expressed as a Jensen–Bregman divergence for the negentropy generator F:

$$\mathrm{JS}(m_{{\theta }_p},m_{{\theta }_q})=\frac{1}{2}\left(B_F\left({\theta }_p:\frac{{\theta }_p+{\theta }_q}{2}\right)+B_F\left({\theta }_q:\frac{{\theta }_p+{\theta }_q}{2}\right)\right).$$

(37)

This amounts to calculate the Jensen divergence:

$$\mathrm{JS}(m_{{\theta }_p},m_{{\theta }_q})=J_F({\theta }_1;{\theta }_2)={\left(F\left({\theta }_1\right)F\left({\theta }_2\right)\right)}_{\frac{1}{2}}-F\left({\left({\theta }_1{\theta }_2\right)}_{\frac{1}{2}}\right),$$

(38)

where ${\left(v_1{v}_2\right)}_{\alpha }:=(1-\alpha )v_1+\alpha v_2$.

Now, consider distributions $p=e_{{\theta }_p}$ and $q=e_{{\theta }_q}$ belonging to the same exponential family [18] $\mathcal{E}$:

$$\mathcal{E}:=\left\{e_{\theta }\left(x\right)=exp\left({\theta }^{\top }x-F\left(\theta \right)\right):\theta \in \mathsf{\Theta }\right\},$$

(39)

where

$$\mathsf{\Theta }:=\left\{\theta \in {\mathbb{R}}^D:\int exp\left({\theta }^{\top }x\right)\mathrm{d}\mu <\infty \right\},$$

(40)

denotes the natural parameter space. We have [18]:

$$\mathrm{KL}(e_{{\theta }_p}:e_{{\theta }_q})=B_F({\theta }_q:{\theta }_p),$$

(41)

where F denotes the log-normalizer or cumulant function of the exponential family [18].

However, $\frac{e_{{\theta }_p}+e_{{\theta }_q}}{2}$does not belong to $\mathcal{E}$ in general, except for the case of the categorical/multinomial family which is both an exponential family and a mixture family [18].

For example, the mixture of two Gaussian distributions with distinct components is not a Gaussian distribution. Thus, it is not obvious to get a closed-form expression for the JSD in that case. This limitation precisely motivated the introduction of generalized JSDs defined in the next section.

Notice that in [32,33], it is shown how to express or approximate the f-divergences using expansions of power $\chi$ pseudo-distances. These power chi distances can all be expressed in closed form when dealing with isotropic Gaussians. This result holds for the JSD since the JSD is a f-divergence [33].

3. Generalized Jensen–Shannon Divergences

We first define abstract means M, and then generic statistical M-mixtures from which generalized Jensen–Shannon divergences are built thereof.

Definitions

Consider an abstract mean [34] M. That is, a continuous bivariate function $M(·,·):I\times I\to I$ on an interval $I\subset \mathbb{R}$ that satisfies the following in-betweenness property:

$$inf\{x,y\}\le M(x,y)\le sup\{x,y\},\forall x,y\in I.$$

(42)

Using the unique dyadic expansion of real numbers, we can always build a corresponding weighted mean $M_{\alpha }(p,q)$ (with $\alpha \in [0,1]$) following the construction reported in [34] (page 3) such that $M_0(p,q)=p$ and $M_1(p,q)=q$. In the remainder, we consider $I=(0,\infty )$.

Examples of common weighted means are:

• the arithmetic mean $A_{\alpha }(x,y)=(1-\alpha )x+{\alpha }_y$,
• the geometric mean $G_{\alpha }(x,y)=x^{1-\alpha }{y}^{\alpha }$, and
• the harmonic mean $H_{\alpha }(x,y)=\frac{xy}{(1-\alpha )y+\alpha x}$.

These means can be unified using the concept of quasi-arithmetic means [34] (also called Kolmogorov–Nagumo means):

$$M_{\alpha }^h(x,y):=h^{-1}\left((1-\alpha )h\left(x\right)+\alpha h\left(y\right)\right),$$

(43)

where h is a strictly monotonous function. For example, the geometric mean $G_{\alpha }(x,y)$ is obtained as $M_{\alpha }^h(x,y)$ for the generator $h\left(u\right)=log\left(u\right)$. Rényi used the concept of quasi-arithmetic means instead of the arithmetic mean to define axiomatically the Rényi entropy [35] of order $\alpha$ in information theory [2].

For any abstract weighted mean, we can build a statistical mixture called a M-mixture as follows:

Definition 1 (M-mixture).

The$M_{\alpha }$-interpolation ${\left(pq\right)}_{\alpha }^M$ (with $\alpha \in [0,1]$) of densities p and q with respect to a mean M is a α-weighted M-mixture defined by:

$${\left(pq\right)}_{\alpha }^M\left(x\right):=\frac{M_{\alpha }(p\left(x\right),q\left(x\right))}{Z_{\alpha }^M(p:q)},$$

(44)

where

$$Z_{\alpha }^M(p:q)={\int }_{t\in \mathcal{X}}{M}_{\alpha }(p\left(t\right),q\left(t\right))\mathrm{d}\mu \left(t\right)=:〈M_{\alpha }(p,q)〉.$$

(45)

is the normalizer function (or scaling factor) ensuring that ${\left(pq\right)}_{\alpha }^M\in \mathcal{P}$. (The bracket notation $〈f〉$ denotes the integral of f over $\mathcal{X}$.)

The A-mixture ${\left(pq\right)}_{\alpha }^A\left(x\right)=(1-\alpha )p\left(x\right)+\alpha q\left(x\right)$ (‘A’ standing for the arithmetic mean) represents the usual statistical mixture [36] (with $Z_{\alpha }^A(p:q)=1$). The G-mixture ${\left(pq\right)}_{\alpha }^G\left(x\right)=\frac{p{\left(x\right)}^{1-\alpha }q{\left(x\right)}^{\alpha }}{Z_{\alpha }^G(p:q)}$ of two distributions $p\left(x\right)$ and $q\left(x\right)$ (’G’ standing for the geometric mean G) is an exponential family of order [37] 1:

$${\left(pq\right)}_{\alpha }^G\left(x\right)=exp\left((1-\alpha )p\left(x\right)+\alpha q\left(x\right)-log{Z}_{\alpha }^G(p:q)\right).$$

(46)

The two-component M-mixture can be generalized to a k-component M-mixture with $\alpha \in {\mathsf{\Delta }}_{k-1}$, the $(k-1)$-dimensional standard simplex:

$${(p_1\dots p_k)}_{\alpha }^M:=\frac{p_1{\left(x\right)}^{{\alpha }_1}\times \dots \times p_k{\left(x\right)}^{{\alpha }_k}}{Z_{\alpha }(p_1,\dots ,p_k)},$$

(47)

where $Z_{\alpha }(p_1,\dots ,p_k):={\int }_{\mathcal{X}}{p}_1{\left(x\right)}^{{\alpha }_1}\times \dots \times p_k{\left(x\right)}^{{\alpha }_k}\mathrm{d}\mu \left(x\right)$.

For a given pair of distributions p and q, the set $\{M_{\alpha }(p\left(x\right),q\left(x\right)):\alpha \in [0,1]\}$ describes a path in the space of probability density functions. This density interpolation scheme was investigated for quasi-arithmetic weighted means in [38,39,40]. In [41], the authors study the Fisher information matrix for the $\alpha$-mixture models (using $\alpha$-power means).

We call ${\left(pq\right)}_{\alpha }^M$ the $\alpha$-weighted M-mixture, thus extending the notion of $\alpha$-mixtures [42] obtained for power means $P_{\alpha }$. Notice that abstract means have also been used to generalize Bregman divergences using the concept of $(M,N)$-convexity [43].

Let us state a first generalization of the Jensen–Shannon divergence:

Definition 2 (M-Jensen–Shannon divergence).

For a mean M, the skew M-Jensen–Shannon divergence (for $\alpha \in [0,1]$) is defined by

$${\mathrm{JS}}^{M_{\alpha }}(p:q):=(1-\alpha )\mathrm{KL}\left(p:{\left(pq\right)}_{\alpha }^M\right)+\alpha \mathrm{KL}\left(q:{\left(pq\right)}_{\alpha }^M\right)$$

(48)

When $M_{\alpha }=A_{\alpha }$, we recover the ordinary Jensen–Shannon divergence since $A_{\alpha }(p:q)={\left(pq\right)}_{\alpha }$ (and $Z_{\alpha }^A(p:q)=1$).

We can extend the definition to the $\mathrm{JS}$-symmetrization of any distance:

Definition 3 (M-JS symmetrization).

For a mean M and a distance D, the skew M-$\mathrm{JS}$ symmetrization of D (for $\alpha \in [0,1]$) is defined by

$${\mathrm{JS}}_D^{M_{\alpha }}(p:q):=(1-\alpha )D\left(p:{\left(pq\right)}_{\alpha }^M\right)+\alpha D\left(q:{\left(pq\right)}_{\alpha }^M\right)$$

(49)

By notation, we have ${\mathrm{JS}}^{M_{\alpha }}(p:q)={\mathrm{JS}}_{\mathrm{KL}}^{M_{\alpha }}(p:q)$. That is, the arithmetic JS-symmetrization of the KLD is the JSD.

Let us define the $\alpha$-skew K-divergence [6,44] $K_{\alpha }(p:q)$ as

$$K_{\alpha }\left(p:q\right):=\mathrm{KL}(p:(1-\alpha )p+\alpha q)=\mathrm{KL}(p:{\left(pq\right)}_{\alpha }),$$

(50)

where ${\left(pq\right)}_{\alpha }\left(x\right):=(1-\alpha )p\left(x\right)+\alpha q\left(x\right)$. Then the Jensen–Shannon divergence and the Jeffreys divergence can be rewritten [24] as

$$\begin{array}{ccc}\hfill \mathrm{JS}\left(p;q\right)& =& \frac{1}{2}\left(K_{\frac{1}{2}}\left(p:q\right)+K_{\frac{1}{2}}\left(q:p\right)\right),\hfill \end{array}$$

(51)

$$\begin{array}{ccc}\hfill J\left(p;q\right)& =& K_1(p:q)+K_1(q:p),\hfill \end{array}$$

(52)

since $\mathrm{KL}(p:q)=K_1(p:q)$. Then ${\mathrm{JS}}_{\alpha }(p:q)=(1-\alpha )K_{\alpha }(p:q)+\alpha K_{1-\alpha }(q:p)$. Similarly, we can define the generalized skew K-divergence:

$$K_D^{M_{\alpha }}(p:q):=D\left(p:{\left(pq\right)}_{\alpha }^M\right).$$

(53)

The success of the JSD compared to the JD in applications is partially due to the fact that the JSD is upper bounded by $log2$. So, one question to ask is whether those generalized JSDs are upper bounded or not?

To report a sufficient condition, let us first introduce the dominance relationship between means: We say that a mean M dominates a mean N when $M(x,y)\ge N(x,y)$ for all $x,y\ge 0$, see [34]. In that case we write concisely $M\ge N$. For example, the Arithmetic-Geometric-Harmonic (AGH) inequality states that $A\ge G\ge H$.

Consider the term

$$\begin{array}{ccc}\hfill \mathrm{KL}(p:{\left(pq\right)}_{\alpha }^M)& =& \int p\left(x\right)log\frac{p\left(x\right)Z_{\alpha }^M(p,q)}{M_{\alpha }(p\left(x\right),q\left(x\right))}\mathrm{d}\mu \left(x\right),\hfill \end{array}$$

(54)

$$\begin{array}{cc}=& log{Z}_{\alpha }^M(p,q)+\int p\left(x\right)log\frac{p\left(x\right)}{M_{\alpha }(p\left(x\right),q\left(x\right))}\mathrm{d}\mu \left(x\right).\hfill \end{array}$$

(55)

When mean $M_{\alpha }$ dominates the arithmetic mean $A_{\alpha }$, we have

$$\int p\left(x\right)log\frac{p\left(x\right)}{M_{\alpha }(p\left(x\right),q\left(x\right))}\mathrm{d}\mu \left(x\right)\le \int p\left(x\right)log\frac{p\left(x\right)}{A_{\alpha }(p\left(x\right),q\left(x\right))}\mathrm{d}\mu \left(x\right),$$

and

$$\int p\left(x\right)log\frac{p\left(x\right)}{A_{\alpha }(p\left(x\right),q\left(x\right))}\mathrm{d}\mu \left(x\right)\le \int p\left(x\right)log\frac{p\left(x\right)}{(1-\alpha )p\left(x\right)}\mathrm{d}\mu \left(x\right)=-log(1-\alpha ).$$

Notice that $Z_{\alpha }^A(p:q)=1$ (when $M=A$ is the arithmetic mean), and we recover the fact that the $\alpha$-skew Jensen–Shannon divergence is upper bounded by $-log(1-\alpha )$ (e.g., $log2$ when $\alpha =\frac{1}{2}$).

We summarize the result in the following property:

Property 1 (Upper bound on M-JSD).

The M-JSD is upper bounded by $log\frac{Z_{\alpha }^M(p,q)}{1-\alpha }$ when $M\ge A$.

Let us observe that dominance of means can be used to define distances: For example, the celebrated $\alpha$-divergences

$$I_{\alpha }(p:q)=\int \left(\alpha p\left(x\right)+(1-\alpha )q\left(x\right)-p{\left(x\right)}^{\alpha }q{\left(x\right)}^{1-\alpha }\right)\mathrm{d}\mu \left(x\right),\alpha \notin \{0,1\}$$

(56)

can be interpreted as a difference of two means, the arithmetic mean and the geometry mean:

$$I_{\alpha }(p:q)=\int \left(A_{\alpha }(q\left(x\right):p\left(x\right))-G_{\alpha }(q\left(x\right):p\left(x\right))\right)\mathrm{d}\mu \left(x\right).$$

(57)

We can also define the generalized Jeffreys divergence as follows:

Definition 4 (N-Jeffreys divergence).

For a mean N, the skew N-Jeffreys divergence (for $\beta \in [0,1]$) is defined by

$$\begin{array}{ccc}\hfill J^{N_{\beta }}(p:q)& :=& N_{\beta }(\mathrm{KL}\left(p:q\right),\mathrm{KL}\left(q:p\right)).\hfill \end{array}$$

(58)

This definition includes the (scaled) resistor average distance [5] $R(p;q)$, obtained for the harmonic mean $N=H$ for the KLD with skew parameter $\beta =\frac{1}{2}$:

$$\begin{array}{ccc}\hfill \frac{1}{R(p;q)}& =& \frac{1}{2}\left(\frac{1}{\mathrm{KL}(p:q)}+\frac{1}{\mathrm{KL}(q:p)}\right),\hfill \end{array}$$

(59)

$$\begin{array}{ccc}\hfill R(p;q)& =& \frac{2J(p;q)}{\mathrm{KL}(p:q)\mathrm{KL}(q:p)}.\hfill \end{array}$$

(60)

In [5], the factor $\frac{1}{2}$ is omitted to keep the spirit of the original Jeffreys divergence.

We can further extend this definition for any arbitrary divergence D as follows:

Definition 5 (Skew $(M,N)$-D divergence).

The skew $(M,N)$-divergence with respect to weighted means $M_{\alpha }$ and $N_{\beta }$ as follows:

$${\mathrm{JS}}_D^{M_{\alpha },N_{\beta }}(p:q):=N_{\beta }\left(D\left(p:{\left(pq\right)}_{\alpha }^M\right),D\left(q:{\left(pq\right)}_{\alpha }^M\right)\right).$$

(61)

We now show how to choose the abstract mean according to the parametric family of distributions to obtain some closed-form formula for some statistical distances.

4. Some Closed-Form Formula for the M-Jensen–Shannon Divergences

Our motivation to introduce these novel families of M-Jensen–Shannon divergences is to obtain closed-form formula when probability densities belong to some given parametric families ${\mathcal{P}}_{\mathsf{\Theta }}$. We shall illustrate the principle of the method to choose the right abstract mean for the considered parametric family, and report corresponding formula for the following two case studies:

• The geometric G-Jensen–Shannon divergence for the exponential families (Section 4.1), and
• the harmonic H-Jensen–Shannon divergence for the family of Cauchy scale distributions (Section 4.2).

Recall that the arithmetic A-Jensen–Shannon divergence is well-suited for mixture families (Theorem 1).

4.1. The Geometric G-Jensen–Shannon Divergence

Consider an exponential family [37] ${\mathcal{E}}_F$ with log-normalizer F:

$${\mathcal{E}}_F=\left\{p_{\theta }\left(x\right)\mathrm{d}\mu =exp({\theta }^{\top }x-F\left(\theta \right))\mathrm{d}\mu :\theta \in \mathsf{\Theta }\right\},$$

(62)

and natural parameter space

$$\mathsf{\Theta }=\left\{\theta :{\int }_{\mathcal{X}}exp\left({\theta }^{\top }x\right)\mathrm{d}\mu <\infty \right\}.$$

(63)

The log-normalizer (a log-Laplace function also called log-partition or cumulant function) is a real analytic convex function.

We seek for a mean M such that the weighted M-mixture density ${\left(p_{{\theta }_1}{p}_{{\theta }_2}\right)}_{\alpha }^M$ of two densities $p_{{\theta }_1}$ and $p_{{\theta }_2}$ of the same exponential family yields another density of that exponential family (e.g., $p_{{\left({\theta }_1{\theta }_2\right)}_{\alpha }}$). When considering exponential families, choose the weighted geometric mean $G_{\alpha }$ for the abstract mean $M_{\alpha }(x,y)$: $M_{\alpha }(x,y)=G_{\alpha }(x,y)=x^{1-\alpha }{y}^{\alpha }$, for $x,y>0$. Indeed, it is well-known that the normalized weighted product of distributions belonging to the same exponential family also belongs to this exponential family [45]:

$$\begin{array}{ccc}\hfill \forall x\in \mathcal{X},{\left(p_{{\theta }_1}{p}_{{\theta }_2}\right)}_{\alpha }^G\left(x\right)& :=& \frac{G_{\alpha }(p_{{\theta }_1}\left(x\right),p_{{\theta }_2}\left(x\right))}{\int G_{\alpha }(p_{{\theta }_1}\left(t\right),p_{{\theta }_2}\left(t\right))\mathrm{d}\mu \left(t\right)}=\frac{p_{{\theta }_1}^{1-\alpha }\left(x\right)p_{{\theta }_2}^{\alpha }\left(x\right)}{Z_{\alpha }^G(p:q)},\hfill \end{array}$$

(64)

$$\begin{array}{cc}=& p_{{\left({\theta }_1{\theta }_2\right)}_{\alpha }}\left(x\right),\hfill \end{array}$$

(65)

where the normalization factor is

$$Z_{\alpha }^G(p:q)=exp(-J_F^{\alpha }({\theta }_1:{\theta }_2)),$$

(66)

for the skew Jensen divergence $J_F^{\alpha }$ defined by:

$$J_F^{\alpha }({\theta }_1:{\theta }_2):={\left(F\left({\theta }_1\right)F\left({\theta }_2\right)\right)}_{\alpha }-F\left({\left({\theta }_1{\theta }_2\right)}_{\alpha }\right).$$

(67)

Notice that since the natural parameter space $\mathsf{\Theta }$ is convex, the distribution $p_{{\left({\theta }_1{\theta }_2\right)}_{\alpha }}\in {\mathcal{E}}_F$ (since ${\left({\theta }_1{\theta }_2\right)}_{\alpha }\in \mathsf{\Theta }$).

Thus, it follows that we have:

$$\begin{array}{ccc}\hfill \mathrm{KL}\left(p_{\theta }:{\left(p_{{\theta }_1}{p}_{{\theta }_2}\right)}_{\alpha }^G\right)& =& \mathrm{KL}\left(p_{\theta }:p_{{\left({\theta }_1{\theta }_2\right)}_{\alpha }}\right),\hfill \end{array}$$

(68)

$$\begin{array}{cc}=& B_F({\left({\theta }_1{\theta }_2\right)}_{\alpha }:\theta ).\hfill \end{array}$$

(69)

This allows us to conclude that the G-Jensen–Shannon divergence admits the following closed-form expression between densities belonging to the same exponential family:

$$\begin{array}{ccc}\hfill {\mathrm{JS}}_{\alpha }^G(p_{{\theta }_1}:p_{{\theta }_2})& :=& (1-\alpha )\mathrm{KL}(p_{{\theta }_1}:{\left(p_{{\theta }_1}{p}_{{\theta }_2}\right)}_{\alpha }^G)+\alpha \mathrm{KL}(p_{{\theta }_2}:{\left(p_{{\theta }_1}{p}_{{\theta }_2}\right)}_{\alpha }^G),\hfill \end{array}$$

(70)

$$\begin{array}{cc}=& (1-\alpha )B_F({\left({\theta }_1{\theta }_2\right)}_{\alpha }:{\theta }_1)+\alpha B_F({\left({\theta }_1{\theta }_2\right)}_{\alpha }:{\theta }_2).\hfill \end{array}$$

(71)

Please note that since ${\left({\theta }_1{\theta }_2\right)}_{\alpha }-{\theta }_1=\alpha ({\theta }_2-{\theta }_1)$ and ${\left({\theta }_1{\theta }_2\right)}_{\alpha }-{\theta }_2=(1-\alpha )({\theta }_1-{\theta }_2)$, it follows that $(1-\alpha )B_F({\theta }_1:{\left({\theta }_1{\theta }_2\right)}_{\alpha })+\alpha B_F({\theta }_2:{\left({\theta }_1{\theta }_2\right)}_{\alpha })=J_F^{\alpha }({\theta }_1:{\theta }_2)$.

The dual divergence [46] $D^{*}$ (with respect to the reference argument) or reverse divergence of a divergence D is defined by swapping the calling arguments: $D^{*}(\theta :{\theta }^{\prime }):=D({\theta }^{\prime }:\theta )$.

Thus, if we defined the Jensen–Shannon divergence for the dual KL divergence ${\mathrm{KL}}^{*}(p:q):=\mathrm{KL}(q:p)$

$$\begin{array}{ccc}\hfill {\mathrm{JS}}_{{\mathrm{KL}}^{*}}(p:q)& :=& \frac{1}{2}\left({\mathrm{KL}}^{*}\left(p:\frac{p+q}{2}\right)+{\mathrm{KL}}^{*}\left(q:\frac{p+q}{2}\right)\right),\hfill \end{array}$$

(72)

$$\begin{array}{cc}=& \frac{1}{2}\left(\mathrm{KL}\left(\frac{p+q}{2}:p\right)+\mathrm{KL}\left(\frac{p+q}{2}:q\right)\right),\hfill \end{array}$$

(73)

then we obtain:

$$\begin{array}{ccc}\hfill {\mathrm{JS}}_{{\mathrm{KL}}^{*}}^{G_{\alpha }}(p_{{\theta }_1}:p_{{\theta }_2})& :=& (1-\alpha )\mathrm{KL}({\left(p_{{\theta }_1}{p}_{{\theta }_2}\right)}_{\alpha }^G:p_{{\theta }_1})+\alpha \mathrm{KL}({\left(p_{{\theta }_1}{p}_{{\theta }_2}\right)}_{\alpha }^G:p_{{\theta }_2}),\hfill \end{array}$$

(74)

$$\begin{array}{cc}=& (1-\alpha )B_F({\theta }_1:{\left({\theta }_1{\theta }_2\right)}_{\alpha })+\alpha B_F({\theta }_2:{\left({\theta }_1{\theta }_2\right)}_{\alpha })={\mathrm{JB}}_F^{\alpha }({\theta }_1:{\theta }_2),\hfill \end{array}$$

(75)

$$\begin{array}{cc}=& (1-\alpha )F\left({\theta }_1\right)+\alpha F\left({\theta }_2\right)-F\left({\left({\theta }_1{\theta }_2\right)}_{\alpha }\right),\hfill \end{array}$$

(76)

$$\begin{array}{cc}=& J_F^{\alpha }({\theta }_1:{\theta }_2).\hfill \end{array}$$

(77)

Please note that ${\mathrm{JS}}_{D^{*}}\ne {{\mathrm{JS}}_D}^{*}$.

In general, the JS-symmetrization for the reverse KL divergence is

$$\begin{array}{ccc}\hfill {\mathrm{JS}}_{{\mathrm{KL}}^{*}}(p;q)& =& \frac{1}{2}\left(\mathrm{KL}\left(\frac{p+q}{2}:p\right)+\mathrm{KL}\left(\frac{p+q}{2}:q\right)\right),\hfill \end{array}$$

(78)

$$\begin{array}{cc}=& \int mlog\frac{m}{\sqrt{pq}}\mathrm{d}\mu =\int A(p,q)log\frac{A(p,q)}{G(p,q)}\mathrm{d}\mu ,\hfill \end{array}$$

(79)

where $m=\frac{p+q}{2}=A(p,q)$ and $G(p,q)=\sqrt{pq}$. Since $A\ge G$ (arithmetic-geometric inequality), it follows that ${\mathrm{JS}}_{{\mathrm{KL}}^{*}}(p;q)\ge 0$.

Theorem 2 (G-JSD and its dual JS-symmetrization in exponential families).

The α-skew G-Jensen–Shannon divergence ${\mathrm{JS}}^{G_{\alpha }}$ between two distributions $p_{{\theta }_1}$ and $p_{{\theta }_2}$ of the same exponential family ${\mathcal{E}}_F$ is expressed in closed form for $\alpha \in (0,1)$ as:

$$\begin{array}{ccc}\hfill {\mathrm{JS}}^{G_{\alpha }}(p_{{\theta }_1}:p_{{\theta }_2})& =& (1-\alpha )B_F\left({\left({\theta }_1{\theta }_2\right)}_{\alpha }:{\theta }_1\right)+\alpha B_F\left({\left({\theta }_1{\theta }_2\right)}_{\alpha }:{\theta }_2\right),\hfill \end{array}$$

(80)

$$\begin{array}{ccc}\hfill {\mathrm{JS}}_{{\mathrm{KL}}^{*}}^{G_{\alpha }}(p_{{\theta }_1}:p_{{\theta }_2})& =& {\mathrm{JB}}_F^{\alpha }({\theta }_1:{\theta }_2)=J_F^{\alpha }({\theta }_1:{\theta }_2).\hfill \end{array}$$

(81)

4.1.1. Case Study: The Multivariate Gaussian Family

Consider the exponential family [18,37] of multivariate Gaussian distributions [47,48,49]

$$\{N(\mu ,\mathsf{\Sigma }):\mu \in {\mathbb{R}}^d,\mathsf{\Sigma }\succ 0\}.$$

(82)

The multivariate Gaussian family is also called the multivariate normal family in the literature, or MVN family for short.

Let $\lambda :=({\lambda }_v,{\lambda }_M)=(\mu ,\mathsf{\Sigma })$ denote the composite (vector,matrix) parameter of an MVN. The d-dimensional MVN density is given by

$$\begin{array}{ccc}\hfill p_{\lambda }(x;\lambda )& :=& \frac{1}{{\left(2\pi \right)}^{\frac{d}{2}}\sqrt{|{\lambda }_M|}}exp\left(-\frac{1}{2}{(x-{\lambda }_v)}^{\top }{\lambda }_M^{-1}(x-{\lambda }_v)\right),\hfill \end{array}$$

(83)

where $|·|$ denotes the matrix determinant. The natural parameters $\theta$ are also expressed using both a vector parameter ${\theta }_v$ and a matrix parameter ${\theta }_M$ in a compound object $\theta =({\theta }_v,{\theta }_M)$. By defining the following compound inner product on a composite (vector,matrix) object

$$\langle \theta ,{\theta }^{\prime }\rangle :={\theta }_v^{\top }{\theta }_v^{\prime }+\mathrm{tr}\left({{\theta }_M^{\prime }}^{\top }{\theta }_M\right),$$

(84)

where $\mathrm{tr}(·)$ denotes the matrix trace, we rewrite the MVN density of Equation (83) in the canonical form of an exponential family [37]:

$$\begin{array}{ccc}\hfill p_{\theta }(x;\theta )& :=& exp\left(\langle t\left(x\right),\theta \rangle -F_{\theta }\left(\theta \right)\right)=p_{\lambda }(x;\lambda \left(\theta \right)),\hfill \end{array}$$

(85)

where

$$\theta =({\theta }_v,{\theta }_M)=\left({\mathsf{\Sigma }}^{-1}\mu ,-\frac{1}{2}{\mathsf{\Sigma }}^{-1}\right)=\theta \left(\lambda \right)=\left({\lambda }_M^{-1}{\lambda }_v,-\frac{1}{2}{\lambda }_M^{-1}\right),$$

(86)

is the compound natural parameter and

$$t\left(x\right)=(x,-x{x}^{\top })$$

(87)

is the compound sufficient statistic. The function $F_{\theta }$ is the strictly convex and continuously differentiable log-normalizer defined by:

$$F_{\theta }\left(\theta \right)=\frac{1}{2}\left(dlog\pi -log|{\theta }_M|+\frac{1}{2}{\theta }_v^{\top }{\theta }_M^{-1}{\theta }_v\right),$$

(88)

The log-normalizer can be expressed using the ordinary parameters, $\lambda =(\mu ,\mathsf{\Sigma })$, as:

$$\begin{array}{ccc}\hfill F_{\lambda }\left(\lambda \right)& =& \frac{1}{2}\left({\lambda }_v^{\top }{\lambda }_M^{-1}{\lambda }_v+log\left|{\lambda }_M\right|+dlog2\pi \right),\hfill \end{array}$$

(89)

$$\begin{array}{cc}=& \frac{1}{2}\left({\mu }^{\top }{\mathsf{\Sigma }}^{-1}\mu +log\left|\mathsf{\Sigma }\right|+dlog2\pi \right).\hfill \end{array}$$

(90)

The moment/expectation parameters [18,49] are

$$\eta =({\eta }_v,{\eta }_M)=E\left[t\left(x\right)\right]=\nabla F\left(\theta \right).$$

(91)

We report the conversion formula between the three types of coordinate systems (namely the ordinary parameter $\lambda$, the natural parameter $\theta$ and the moment parameter $\eta$) as follows:

$$\begin{array}{ccc}\hfill \left\{\begin{array}{c}{\theta }_v\left(\lambda \right)={\lambda }_M^{-1}{\lambda }_v={\mathsf{\Sigma }}^{-1}\mu \hfill \\ {\theta }_M\left(\lambda \right)=\frac{1}{2}{\lambda }_M^{-1}=\frac{1}{2}{\mathsf{\Sigma }}^{-1}\hfill \end{array}\right.& \iff & \left\{\begin{array}{c}{\lambda }_v\left(\theta \right)=\frac{1}{2}{\theta }_M^{-1}{\theta }_v=\mu \hfill \\ {\lambda }_M\left(\theta \right)=\frac{1}{2}{\theta }_M^{-1}=\mathsf{\Sigma }\hfill \end{array}\right.\hfill \end{array}$$

(92)

$$\begin{array}{ccc}\hfill \left\{\begin{array}{c}{\eta }_v\left(\theta \right)=\frac{1}{2}{\theta }_M^{-1}{\theta }_v\hfill \\ {\eta }_M\left(\theta \right)=-\frac{1}{2}{\theta }_M^{-1}-\frac{1}{4}\left({\theta }_M^{-1}{\theta }_v\right){\left({\theta }_M^{-1}{\theta }_v\right)}^{\top }\hfill \end{array}\right.& \iff & \left\{\begin{array}{c}{\theta }_v\left(\eta \right)=-{({\eta }_M+{\eta }_v{\eta }_v^{\top })}^{-1}{\eta }_v\hfill \\ {\theta }_M\left(\eta \right)=-\frac{1}{2}{({\eta }_M+{\eta }_v{\eta }_v^{\top })}^{-1}\hfill \end{array}\right.\hfill \end{array}$$

(93)

$$\begin{array}{ccc}\hfill \left\{\begin{array}{c}{\lambda }_v\left(\eta \right)={\eta }_v=\mu \hfill \\ {\lambda }_M\left(\eta \right)=-{\eta }_M-{\eta }_v{\eta }_v^{\top }=\mathsf{\Sigma }\hfill \end{array}\right.& \iff & \left\{\begin{array}{c}{\eta }_v\left(\lambda \right)={\lambda }_v=\mu \hfill \\ {\eta }_M\left(\lambda \right)=-{\lambda }_M-{\lambda }_v{\lambda }_v^{\top }=-\mathsf{\Sigma }-\mu {\mu }^{\top }\hfill \end{array}\right.\hfill \end{array}$$

(94)

The dual Legendre convex conjugate [18,49] is

$$F_{\eta }^{*}\left(\eta \right)=-\frac{1}{2}\left(log(1+{\eta }_v^{\top }{\eta }_M^{-1}{\eta }_v)+log|-{\eta }_M|+d(1+log2\pi )\right),$$

(95)

and $\theta ={\nabla }_{\eta }{F}_{\eta }^{*}\left(\eta \right)$.

We check the Fenchel-Young equality when $\eta =\nabla F\left(\theta \right)$ and $\theta =\nabla F^{*}\left(\eta \right)$:

$$F_{\theta }\left(\theta \right)+F_{\eta }^{*}\left(\eta \right)-\langle \theta ,\eta \rangle =0.$$

(96)

The Kullback–Leibler divergence between two d-dimensional Gaussians distributions $p_{({\mu }_1,{\mathsf{\Sigma }}_1)}$ and $p_{({\mu }_2,{\mathsf{\Sigma }}_2)}$ (with ${\mathsf{\Delta }}_{\mu }={\mu }_2-{\mu }_1$) is

$$\begin{array}{ccc}\hfill \mathrm{KL}(p_{({\mu }_1,{\mathsf{\Sigma }}_1)}:p_{({\mu }_2,{\mathsf{\Sigma }}_2)})& =& \frac{1}{2}\left\{\mathrm{tr}\left({\mathsf{\Sigma }}_2^{-1}{\mathsf{\Sigma }}_1\right)+{\mathsf{\Delta }}_{\mu }^{\top }{\mathsf{\Sigma }}_2^{-1}{\mathsf{\Delta }}_{\mu }+log\frac{|{\mathsf{\Sigma }}_2|}{|{\mathsf{\Sigma }}_1|}-d\right\}=\mathrm{KL}(p_{{\lambda }_1}:p_{{\lambda }_2}).\hfill \end{array}$$

(97)

We check that $\mathrm{KL}(p_{(\mu ,\mathsf{\Sigma })}:p_{(\mu ,\mathsf{\Sigma })})=0$ since ${\mathsf{\Delta }}_{\mu }=0$ and $\mathrm{tr}\left({\mathsf{\Sigma }}^{-1}\mathsf{\Sigma }\right)=\mathrm{tr}\left(I\right)=d$. Notice that when ${\mathsf{\Sigma }}_1={\mathsf{\Sigma }}_2=\mathsf{\Sigma }$, we have

$$\mathrm{KL}(p_{({\mu }_1,\mathsf{\Sigma })}:p_{({\mu }_2,\mathsf{\Sigma })})=\frac{1}{2}{\mathsf{\Delta }}_{\mu }^{\top }{\mathsf{\Sigma }}^{-1}{\mathsf{\Delta }}_{\mu }=\frac{1}{2}{D}_{{\mathsf{\Sigma }}^{-1}}^2({\mu }_1,{\mu }_2),$$

(98)

that is half the squared Mahalanobis distance for the precision matrix ${\mathsf{\Sigma }}^{-1}$ (a positive-definite matrix: ${\mathsf{\Sigma }}^{-1}\succ 0$), where the Mahalanobis distance is defined for any positive matrix $Q\succ 0$ as follows:

$$D_Q(p_1:p_2)=\sqrt{{(p_1-p_2)}^{\top }Q(p_1-p_2)}.$$

(99)

The Kullback–Leibler divergence between two probability densities of the same exponential families amount to a Bregman divergence [18]:

$$\mathrm{KL}(p_{({\mu }_1,{\mathsf{\Sigma }}_1)}:p_{({\mu }_2,{\mathsf{\Sigma }}_2)})=\mathrm{KL}(p_{{\lambda }_1}:p_{{\lambda }_2})=B_F({\theta }_2:{\theta }_1)=B_{F^{*}}({\eta }_1:{\eta }_2),$$

(100)

where the Bregman divergence is defined by

$$B_F(\theta :{\theta }^{\prime }):=F\left(\theta \right)-F\left({\theta }^{\prime }\right)-\langle \theta -{\theta }^{\prime },\nabla F\left({\theta }^{\prime }\right)\rangle ,$$

(101)

with ${\eta }^{\prime }=\nabla F\left({\theta }^{\prime }\right)$. Define the canonical divergence [18]

$$A_F({\theta }_1:{\eta }_2)=F\left({\theta }_1\right)+F^{*}\left({\eta }_2\right)-\langle {\theta }_1,{\eta }_2\rangle =A_{F^{*}}({\eta }_2:{\theta }_1),$$

(102)

since ${F^{*}}^{*}=F$. We have $B_F({\theta }_1:{\theta }_2)=A_F({\theta }_1:{\eta }_2)$.

Now, observe that $p_{\theta }(0,\theta )=exp(-F\left(\theta \right))$ when $\langle t\left(0\right),\theta \rangle =0$. In particular, this holds for the multivariate normal family. Thus, we have the following proposition.

Proposition 1. 

For the MVN family, we have

$$p_{\theta }(x;{\left({\theta }_1{\theta }_2\right)}_{\alpha })=\frac{p_{\theta }{(x,{\theta }_1)}^{1-\alpha }{p}_{\theta }{(x,{\theta }_2)}^{\alpha }}{Z_{\alpha }^G(p_{{\theta }_1}:p_{{\theta }_2})},$$

(103)

with the scaling normalization factor:

$$Z_{\alpha }^G(p_{{\theta }_1}:p_{{\theta }_2})=exp(-J_F^{\alpha }({\theta }_1:{\theta }_2))=\frac{p_{\theta }{(0;{\theta }_1)}^{1-\alpha }{p}_{\theta }{(0;{\theta }_2)}^{\alpha }}{p_{\theta }(0;{\left({\theta }_1{\theta }_2\right)}_{\alpha })}.$$

(104)

More generally, we have for a k-dimensional weight vector $\alpha$ belonging to the $(k-1)$-dimensional standard simplex:

$$Z_{\alpha }^G(p_{{\theta }_1},\dots p_{{\theta }_k})=\frac{{\prod }_{i=1}^k{p}_{\theta }{(0,{\theta }_i)}^{{\alpha }_i}}{p_{\theta }(0;\overline{\theta })},$$

(105)

where $\overline{\theta }={\sum }_{i=1}^k{\alpha }_i{\theta }_i$.

Finally, we state the formulas for the G-JS divergence between MVNs for the KL and reverse KL, respectively:

Corollary 1 (G-JSD between Gaussians).

The skew G-Jensen–Shannon divergence ${\mathrm{JS}}_{\alpha }^G$ and the dual skew G-Jensen–Shannon divergence ${{\mathrm{JS}}^{*}}_{\alpha }^G$ between two multivariate Gaussians $N({\mu }_1,{\mathsf{\Sigma }}_1)$ and $N({\mu }_2,{\mathsf{\Sigma }}_2)$ is

$$\begin{array}{ccc}\hfill {\mathrm{JS}}^{G_{\alpha }}(p_{({\mu }_1,{\mathsf{\Sigma }}_1)}:p_{({\mu }_2,{\mathsf{\Sigma }}_2)})& =& (1-\alpha )\mathrm{KL}(p_{({\mu }_1,{\mathsf{\Sigma }}_1)}:p_{({\mu }_{\alpha },{\mathsf{\Sigma }}_{\alpha })})+\alpha \mathrm{KL}(p_{({\mu }_2,{\mathsf{\Sigma }}_2)}:p_{({\mu }_{\alpha },{\mathsf{\Sigma }}_{\alpha })}),\hfill \end{array}$$

(106)

$$\begin{array}{ccc}& =& (1-\alpha )B_F({\left({\theta }_1{\theta }_2\right)}_{\alpha }:{\theta }_1)+\alpha B_F({\left({\theta }_1{\theta }_2\right)}_{\alpha }:{\theta }_2),\hfill \\ & =& \frac{1}{2}\left(\mathrm{tr}\left({\mathsf{\Sigma }}_{\alpha }^{-1}((1-\alpha ){\mathsf{\Sigma }}_1+\alpha {\mathsf{\Sigma }}_2)\right)+log\frac{|{\mathsf{\Sigma }}_{\alpha }|}{|{\mathsf{\Sigma }}_1{|}^{1-\alpha }{\left|{\mathsf{\Sigma }}_2\right|}^{\alpha }}+\right.\hfill \end{array}$$

(107)

$$\begin{array}{ccc}& & \left.(1-\alpha ){({\mu }_{\alpha }-{\mu }_1)}^{\top }{\mathsf{\Sigma }}_{\alpha }^{-1}({\mu }_{\alpha }-{\mu }_1)+\alpha {({\mu }_{\alpha }-{\mu }_2)}^{\top }{\mathsf{\Sigma }}_{\alpha }^{-1}({\mu }_{\alpha }-{\mu }_2)-d\right)\hfill \end{array}$$

(108)

$$\begin{array}{ccc}\hfill {\mathrm{JS}}_{*}^{G_{\alpha }}(p_{({\mu }_1,{\mathsf{\Sigma }}_1)}:p_{({\mu }_2,{\mathsf{\Sigma }}_2)})& =& (1-\alpha )\mathrm{KL}(p_{({\mu }_{\alpha },{\mathsf{\Sigma }}_{\alpha })}:p_{({\mu }_1,{\mathsf{\Sigma }}_1)})+\alpha \mathrm{KL}(p_{({\mu }_{\alpha },{\mathsf{\Sigma }}_{\alpha })}:p_{({\mu }_2,{\mathsf{\Sigma }}_2)}),\hfill \end{array}$$

(109)

$$\begin{array}{ccc}& =& (1-\alpha )B_F({\theta }_1:{\left({\theta }_1{\theta }_2\right)}_{\alpha })+\alpha B_F({\theta }_2:{\left({\theta }_1{\theta }_2\right)}_{\alpha }),\hfill \end{array}$$

(110)

$$\begin{array}{ccc}& =& J_F({\theta }_1:{\theta }_2),\hfill \end{array}$$

(111)

$$\begin{array}{ccc}& =& \frac{1}{2}\left((1-\alpha ){\mu }_1^{\top }{\mathsf{\Sigma }}_1^{-1}{\mu }_1+\alpha {\mu }_2^{\top }{\mathsf{\Sigma }}_2^{-1}{\mu }_2-{\mu }_{\alpha }^{\top }{\mathsf{\Sigma }}_{\alpha }^{-1}{\mu }_{\alpha }+log\frac{|{\mathsf{\Sigma }}_1{|}^{1-\alpha }{\left|{\mathsf{\Sigma }}_2\right|}^{\alpha }}{|{\mathsf{\Sigma }}_{\alpha }|}\right),\hfill \end{array}$$

(112)

where

$${\mathsf{\Sigma }}_{\alpha }={\left({\mathsf{\Sigma }}_1{\mathsf{\Sigma }}_2\right)}_{\alpha }^{\mathsf{\Sigma }}={\left((1-\alpha ){\mathsf{\Sigma }}_1^{-1}+\alpha {\mathsf{\Sigma }}_2^{-1}\right)}^{-1},$$

(113)

(matrix harmonic barycenter) and

$${\mu }_{\alpha }={\left({\mu }_1{\mu }_2\right)}_{\alpha }^{\mu }={\mathsf{\Sigma }}_{\alpha }\left((1-\alpha ){\mathsf{\Sigma }}_1^{-1}{\mu }_1+\alpha {\mathsf{\Sigma }}_2^{-1}{\mu }_2\right).$$

(114)

Notice that the α-skew Bhattacharyya distance [7]:

$$B_{\alpha }(p:q)=-log{\int }_{\mathcal{X}}{p}^{1-\alpha }{q}^{\alpha }\mathrm{d}\mu$$

(115)

between two members of the same exponential family amounts to a $\alpha$-skew Jensen divergence between the corresponding natural parameters:

$$B_{\alpha }(p_{{\theta }_1}:p_{{\theta }_2})=J_F^{\alpha }({\theta }_1:{\theta }_2).$$

(116)

A simple proof follows from the fact that

$$\int p_{{\left({\theta }_1{\theta }_2\right)}_{\alpha }}\left(x\right)\mathrm{d}\mu \left(x\right)=1=\int \frac{p_{{\theta }_1}^{1-\alpha }\left(x\right)p_{{\theta }_2}^{\alpha }\left(x\right)}{Z_{\alpha }^G(p_{{\theta }_1}:p_{{\theta }_2})}\mathrm{d}\mu \left(x\right).$$

(117)

Therefore, we have

$$log1=0=log\int p_{{\theta }_1}^{1-\alpha }\left(x\right)p_{{\theta }_2}^{\alpha }\left(x\right)\mathrm{d}\mu \left(x\right)-log{Z}_{\alpha }^G(p_{{\theta }_1}:p_{{\theta }_2}),$$

(118)

with $Z_{\alpha }^G(p_{{\theta }_1}:p_{{\theta }_2})=exp(-J_F(p_{{\theta }_1}:p_{{\theta }_2}))$. Thus, it follows that

$$\begin{array}{ccc}\hfill B_{\alpha }(p_{{\theta }_1}:p_{{\theta }_2})& =& -log\int p_{{\theta }_1}^{1-\alpha }\left(x\right)p_{{\theta }_2}^{\alpha }\left(x\right)\mathrm{d}\mu \left(x\right),\hfill \end{array}$$

(119)

$$\begin{array}{cc}=& -log{Z}_{\alpha }^G(p_{{\theta }_1}:p_{{\theta }_2}),\hfill \end{array}$$

(120)

$$\begin{array}{cc}=& J_F(p_{{\theta }_1}:p_{{\theta }_2}).\hfill \end{array}$$

(121)

Corollary 2. 

The JS-symmetrization of the reverse Kullback–Leibler divergence between densities of the same exponential family amount to calculate a Jensen/Burbea–Rao divergence between the corresponding natural parameters.

4.1.2. Applications to k-Means Clustering

Let $P=\{p_1,\dots ,p_n\}$ denote a point set, and $C=\{c_1,\dots ,c_k\}$ denote a set of k (cluster) centers. The generalized k-means objective [23] with respect to a distance D is defined by:

$$E_D(P,C)=\frac{1}{n}\sum _{i=1}^n\underset{j\in \{1,\dots ,k\}}{min}D(p_i:c_j).$$

(122)

By defining the distance $D(p,C)={min}_{j\in \{1,\dots ,k\}}D(p:c_j)$ of a point to a set of points, we can rewrite compactly the objective function as $E_D(P,C)=\frac{1}{n}{\sum }_{i=1}^{n}D(p_i,C)$. Denote by $E_D^{*}(P,k)$ the minimum objective loss for a set of $k=\left|C\right|$ clusters: $E_D^{*}(P,k)={min}_{\left|C\right|=k}{E}_D(P,C)$. It is NP-hard [50] to compute $E_D^{*}(P,k)$ when $k>1$ and the dimension $d>1$. The most common heuristic is Lloyd’s batched k-means [23] that yields a local minimum.

The performance of the probabilistic k-means++ initialization [51] has been extended to arbitrary distances in [52] as follows:

Theorem 3 

(Generalized k-means++ performance, [53]). Let ${\kappa }_1$ and ${\kappa }_2$ be two constants such that ${\kappa }_1$ defines the quasi-triangular inequality property:

$$D(x:z)\le {\kappa }_1\left(D(x:y)+D(y:z)\right),\forall x,y,z\in {\mathsf{\Delta }}^d,$$

(123)

and ${\kappa }_2$ handles the symmetry inequality:

$$D(x:y)\le {\kappa }_{2}D(y:x),\forall x,y\in {\mathsf{\Delta }}^d.$$

(124)

Then the generalized k-means++ seeding guarantees with high probability a configuration C of cluster centers such that:

$$E_D(P,C)\le 2{\kappa }_1^2(1+{\kappa }_2)(2+logk)E_D^{*}(P,k).$$

(125)

To bound the constants ${\kappa }_1$ and ${\kappa }_2$, we rewrite the generalized Jensen–Shannon divergences using quadratic form expressions: That is, using a squared Mahalanobis distance:

$$D_Q(p:q)=\sqrt{{(p-q)}^{\top }Q(p-q)},$$

(126)

for a positive-definite matrix $Q\succ 0$. Since the Bregman divergence can be interpreted as the tail of a first-order Taylor expansion, we have:

$$B_F({\theta }_1:{\theta }_2)=\frac{1}{2}{({\theta }_1-{\theta }_2)}^{\top }{\nabla }^{2}F\left(\xi \right)({\theta }_1-{\theta }_2),$$

(127)

for $\xi \in \mathsf{\Theta }$ (open convex). Similarly, the Jensen divergence can be interpreted as a Jensen–Bregman divergence, and thus we have

$$J_F({\theta }_1:{\theta }_2)\frac{1}{2}{({\theta }_1-{\theta }_2)}^{\top }{\nabla }^{2}F\left({\xi }^{\prime }\right)({\theta }_1-{\theta }_2),$$

(128)

for ${\xi }^{\prime }\in \mathsf{\Theta }$. More precisely, for a prescribed point set $\{{\theta }_1,\dots ,{\theta }_n\}$, we have $\xi ,{\xi }^{\prime }\in \mathrm{CH}\left(\{{\theta }_1,\dots ,{\theta }_n\}\right)$, where $\mathrm{CH}$ denotes the closed convex hull. We can therefore upper bound ${\kappa }_1$ and ${\kappa }_2$ using the ratio $\frac{{max}_{\theta \in \mathrm{CH}\left(\{{\theta }_1,\dots ,{\theta }_n\}\right)}\parallel {\nabla }^{2}F\left(\theta \right)\parallel }{{max}_{\theta \in \mathrm{CH}\left(\{{\theta }_1,\dots ,{\theta }_n\}\right)}\parallel {\nabla }^{2}F\left(\theta \right)\parallel }$. See [54] for further details.

A centroid for a set of parameters ${\theta }_1,\dots ,{\theta }_n$ is defined as the minimizer of the functional

$$E_D\left(\theta \right)=\frac{1}{n}\sum _{i}D({\theta }_i:\theta ).$$

(129)

In particular, the symmetrized Bregman centroids have been studied in [55] (for ${\mathrm{JS}}^{G_{\alpha }}$), and the Jensen centroids (for ${\mathrm{JS}}_{*}^{G_{\alpha }}$) have been investigated in [7] using the convex-concave iterative procedure.

4.2. The Harmonic Jensen–Shannon Divergence (H-$JS$)

The principle to get closed-form formula for generalized Jensen–Shannon divergences between distributions belonging to a parametric family ${\mathcal{P}}_{\mathsf{\Theta }}=\{p_{\theta }:\theta \in \mathsf{\Theta }\}$ consists of finding an abstract mean M such that the M-mixture ${\left(p_{{\theta }_1}{p}_{{\theta }_2}\right)}_{\alpha }^M$ belongs to the family ${\mathcal{P}}_{\mathsf{\Theta }}$. In particular, when $\mathsf{\Theta }$ is a convex domain, we seek a mean M such that ${\left(p_{{\theta }_1}{p}_{{\theta }_2}\right)}_{\alpha }^M=p_{{\left({\theta }_1{\theta }_2\right)}_{\alpha }}$ with ${\left({\theta }_1{\theta }_2\right)}_{\alpha }\in \mathsf{\Theta }$.

Let us consider the weighted harmonic mean [34] (induced by the harmonic mean) H:

$$H_{\alpha }(x,y):=\frac{1}{(1-\alpha )\frac{1}{x}+\alpha \frac{1}{y}}=\frac{xy}{(1-\alpha )y+\alpha x}=\frac{xy}{{\left(xy\right)}_{1-\alpha }},\alpha \in [0,1].$$

(130)

The harmonic mean is a quasi-arithmetic mean $H_{\alpha }(x,y)=M_{\alpha }^h(x,y)$ obtained for the monotone (decreasing) function $h\left(u\right)=\frac{1}{u}$ (or equivalently for the increasing monotone function $h\left(u\right)=-\frac{1}{u}$).

This harmonic mean is well-suited for the scale family $\mathcal{C}$ of Cauchy probability distributions (also called Lorentzian distributions):

$${\mathcal{C}}_{\Gamma }:=\left\{p_{\gamma }\left(x\right)=\frac{1}{\gamma }{p}_{\mathrm{std}}\left(\frac{x}{\gamma }\right)=\frac{\gamma }{\pi ({\gamma }^2+x^2)}:\gamma \in \mathsf{\Gamma }=(0,\infty )\right\},$$

(131)

where $\gamma$ denotes the scale and $p_{\mathrm{std}}\left(x\right)=\frac{1}{\pi (1+x^2)}$ the standard Cauchy distribution.

Using the computer algebra system Maxima (http://maxima.sourceforge.net/) we find that (see Appendix B)

$${\left(p_{{\gamma }_1}{p}_{{\gamma }_2}\right)}_{\frac{1}{2}}^H\left(x\right)=\frac{H_{\alpha }(p_{{\gamma }_1}\left(x\right):p_{{\gamma }_2}\left(x\right))}{Z_{\alpha }^H({\gamma }_1,{\gamma }_2)}=p_{{\left({\gamma }_1{\gamma }_2\right)}_{\alpha }}$$

(132)

where the normalizing coefficient is

$$Z_{\alpha }^H({\gamma }_1,{\gamma }_2):=\sqrt{\frac{{\gamma }_1{\gamma }_2}{{\left({\gamma }_1{\gamma }_2\right)}_{\alpha }{\left({\gamma }_1{\gamma }_2\right)}_{1-\alpha }}}=\sqrt{\frac{{\gamma }_1{\gamma }_2}{{\left({\gamma }_1{\gamma }_2\right)}_{\alpha }{\left({\gamma }_2{\gamma }_1\right)}_{\alpha }}},$$

(133)

since we have ${\left({\gamma }_1{\gamma }_2\right)}_{1-\alpha }={\left({\gamma }_2{\gamma }_1\right)}_{\alpha }$.

The H-Jensen–Shannon symmetrization of a distance D between distributions writes as:

$${\mathrm{JS}}_D^{H_{\alpha }}(p:q)=(1-\alpha )D(p:{\left(pq\right)}_{\alpha }^H)+\alpha D(q:{\left(pq\right)}_{\alpha }^H),$$

(134)

where $H_{\alpha }$ denote the weighted harmonic mean. When D is available in closed form for distributions belonging to the scale Cauchy distributions, so is ${\mathrm{JS}}_D^{H_{\alpha }}(p:q)$.

For example, consider the KL divergence formula between two scale Cauchy distributions:

$$\mathrm{KL}(p_{{\gamma }_1}:p_{{\gamma }_2})=2log\frac{A({\gamma }_1,{\gamma }_2)}{G({\gamma }_1,{\gamma }_2)}=2log\frac{{\gamma }_1+{\gamma }_2}{2\sqrt{{\gamma }_1{\gamma }_2}},$$

(135)

where A and G denote the arithmetic and geometric means, respectively. The formula initially reported in [56] has been corrected by the authors. Since $A\ge G$ (and $\frac{A}{G}\ge 1$), it follows that $\mathrm{KL}(p_{{\gamma }_1}:p_{{\gamma }_2})\ge 0$. Notice that the KL divergence is symmetric for Cauchy scale distributions. We note in passing that for exponential families, the KL divergence is symmetric only for the location Gaussian family (since the only symmetric Bregman divergences are the squared Mahalanobis distances [57]). The cross-entropy between scale Cauchy distributions is $h^{\times }(p_{{\gamma }_1}:p_{{\gamma }_2})=log\pi \frac{{({\gamma }_1+{\gamma }_2)}^2}{{\gamma }_2}$, and the differential entropy is $h\left(p_{\gamma }\right)=h^{\times }(p_{\gamma }:p_{\gamma })=log4\pi \gamma$.

Then the H-JS divergence between $p=p_{{\gamma }_1}$ and $q=p_{{\gamma }_2}$ is:

$$\begin{array}{ccc}\hfill {\mathrm{JS}}^H(p:q)& =& \frac{1}{2}\left(\mathrm{KL}\left(p:{\left(pq\right)}_{\frac{1}{2}}^H\right)+\mathrm{KL}\left(q:{\left(pq\right)}_{\frac{1}{2}}^H\right)\right),\hfill \end{array}$$

(136)

$$\begin{array}{ccc}\hfill {\mathrm{JS}}^H(p_{{\gamma }_1}:p_{{\gamma }_2})& =& \frac{1}{2}\left(\mathrm{KL}\left(p_{{\gamma }_1}:p_{\frac{{\gamma }_1+{\gamma }_2}{2}}\right)+\mathrm{KL}\left(p_{{\gamma }_2}:p_{\frac{{\gamma }_1+{\gamma }_2}{2}}\right)\right),\hfill \end{array}$$

(137)

$$\begin{array}{cc}=& log\frac{(3{\gamma }_1+{\gamma }_2)(3{\gamma }_2+{\gamma }_1)}{8\sqrt{{\gamma }_1{\gamma }_2}({\gamma }_1+{\gamma }_2)}.\hfill \end{array}$$

(138)

We check that when ${\gamma }_1={\gamma }_2=\gamma$, we have ${\mathrm{JS}}^{H_{\alpha }}(p_{\gamma }:p_{\gamma })=0$.

Theorem 4 (Harmonic JSD between scale Cauchy distributions).

The harmonic Jensen–Shannon divergence between two scale Cauchy distributions $p_{{\gamma }_1}$ and $p_{{\gamma }_2}$ is ${\mathrm{JS}}^H(p_{{\gamma }_1}:p_{{\gamma }_2})=log\frac{(3{\gamma }_1+{\gamma }_2)(3{\gamma }_2+{\gamma }_1)}{8\sqrt{{\gamma }_1{\gamma }_2}({\gamma }_1+{\gamma }_2)}$.

Let us report some numerical examples: Consider $p_{{\gamma }_1}=0.1$ and $p_{{\gamma }_1}=0.5$, we find that ${\mathrm{JS}}^H(p_{{\gamma }_1}:p_{{\gamma }_2})\simeq 0.176$. When $p_{{\gamma }_1}=0.2$ and $p_{{\gamma }_1}=0.8$, we find that ${\mathrm{JS}}^H(p_{{\gamma }_1}:p_{{\gamma }_2})\simeq 0.129$.

Notice that KL formula is scale-invariant, and this property holds for any scale family:

Lemma 1. 

The Kullback–Leibler divergence between two distributions $p_{s_1}$ and $p_{s_2}$ belonging to the same scale family ${\{p_s\left(x\right)=\frac{1}{s}p\left(\frac{x}{s}\right)\}}_{s\in (0,\infty )}$ with standard density p is scale-invariant: $\mathrm{KL}(p_{\lambda s_1}:p_{\lambda s_2})=\mathrm{KL}(p_{s_1}:p_{s_2})=\mathrm{KL}(p:p_{\frac{s_2}{s_1}})=\mathrm{KL}(p_{\frac{s_1}{s_2}}:p)$ for any $\lambda >0$.

A direct proof follows from a change of variable in the KL integral with $y=\frac{x}{\lambda }$ and $\mathrm{d}x=\lambda \mathrm{d}y$. Please note that although the KLD between scale Cauchy distributions is symmetric, it is not the case for all scale families: For example, the Rayleigh distributions form a scale family with the KLD amounting to compute a Bregman asymmetric Itakura–Saito divergence between parameters [37].

Instead of the KLD, we can choose the total variation distance for which a formula has been reported in [38] between two Cauchy distributions. Notice that the Cauchy distributions are alpha-stable distributions for $\alpha =1$ and q Gaussian distributions for $q=2$ ([58], p. 104). A closed-form formula for the divergence between two q-Gaussians is given in [58] when $q<2$. The definite integral $h_q\left(p\right)={\int }_{-\infty }^{+\infty }p{\left(x\right)}^q\mathrm{d}\mu$ is available in closed form for Cauchy distributions. When $q=2$, we have $h_2\left(p_{\gamma }\right)=\frac{1}{2\pi \gamma }$.

We refer to [38] for yet other illustrative examples considering the family of Pearson type VII distributions and central multivariate t-distributions which use the power means (quasi-arithmetic means $M^h$ induced by $h\left(u\right)=u^{\alpha }$ for $\alpha >0$) for defining mixtures.

Table 1. Summary of the weighted means M chosen according to the parametric family in order to ensure that the family is closed under M-mixturing: ${\left(p_{{\theta }_1}{p}_{{\theta }_2}\right)}_{\alpha }^M=p_{{\left({\theta }_1{\theta }_2\right)}_{\alpha }}$.

${\mathit{JS}}^{{\mathit{M}}_{\mathit{\alpha }}}$	Mean M	Parametric Family	${\mathit{Z}}_{\mathit{\alpha }}^{\mathit{M}}(\mathit{p}:\mathit{q})$
${\mathrm{JS}}^{A_{\alpha }}$	arithmetic A	mixture family	$Z_{\alpha }^M({\theta }_1:{\theta }_2)=1$
${\mathrm{JS}}^{G_{\alpha }}$	geometric G	exponential family	$Z_{\alpha }^G({\theta }_1:{\theta }_2)=exp(-J_F^{\alpha }({\theta }_1:{\theta }_2))$
${\mathrm{JS}}^{H_{\alpha }}$	harmonic H	Cauchy scale family	$Z_{\alpha }^H({\theta }_1:{\theta }_2)=\sqrt{\frac{{\theta }_1{\theta }_2}{{\left({\theta }_1{\theta }_2\right)}_{\alpha }{\left({\theta }_1{\theta }_2\right)}_{1-\alpha }}}$

summarizes the various examples introduced in the paper.

4.3. The M-Jensen–Shannon Matrix Distances

In this section, we consider distances between matrices which play an important role in quantum computing [59,60]. We refer to [61] for the matrix Jensen–Bregman logdet divergence. The Hellinger distance can be interpreted as the difference of an arithmetic mean A and a geometric mean G:

$$D_H(p,q)=\sqrt{1-{\int }_{\mathcal{X}}\sqrt{p\left(x\right)}\sqrt{q\left(x\right)}\mathrm{d}\mu \left(x\right)}=\sqrt{{\int }_{\mathcal{X}}(A(p\left(x\right),q\left(x\right))-G(p\left(x\right),q\left(x\right)))\mathrm{d}\mu \left(x\right)}.$$

(139)

Notice that since $A\ge G$, we have $D_H(p,q)\ge 0$. The scaled and squared Hellinger distance is an $\alpha$-divergence $I_{\alpha }$ for $\alpha =0$. Recall that the $\alpha$-divergence can be interpreted as the difference of a weighted arithmetic minus a weighted geometry mean.

In general, if a mean $M_1$ dominates a mean $M_2$, we may define the distance as

$$D_{M_1,M_2}(p,q)={\int }_{\mathcal{X}}\left(M_1(p,q)-M_2(p,q)\right)\mathrm{d}\mu \left(x\right).$$

(140)

When considering matrices [62], there is not a unique definition of a geometric matrix mean, and thus we have different notions of matrix Hellinger distances [62], some of them are divergences (smooth distances defining a dualistic structure in information geometry).

We define the matrix M-Jensen–Shannon divergence for a matrix divergence [63,64] D as follows:

$${\mathrm{JS}}_D^M(X_1,X_2)=\frac{1}{2}\left(D(X_1:M(X_1,X_2))+D(X_2:M(X_1,X_2))\right)={\mathrm{JS}}_D^M(X_2,X_1).$$

(141)

For example, we can choose the von Neumann matrix divergence [63]:

$$D_{\mathrm{vN}}(X_1:X_2):=\mathrm{tr}\left(X_{1}log{X}_1-X_{1}log{X}_2-X_1+X_2\right),$$

(142)

or the LogDet matrix divergence [63]:

$$D_{\mathrm{ld}}(X_1:X_2):=\mathrm{tr}\left(X_1{X}_2^{-1}\right)-log\left|X_1{X}_2^{-1}\right|-d,$$

(143)

where square matrices $X_1$ and $X_2$ have dimension d.

5. Conclusions and Perspectives

We introduced a generalization of the celebrated Jensen–Shannon divergence [6], termed the $(M,N)$-Jensen–Shannon divergences, based on M-mixtures derived from abstract means M. This new family of divergences includes the ordinary Jensen–Shannon divergence when both M and N are set to the arithmetic mean. We reported closed-form expressions of the M Jensen–Shannon divergences for mixture families and exponential families in information geometry by choosing the arithmetic and geometric weighted mean, respectively. The $\alpha$-skewed geometric Jensen–Shannon divergence (G-Jensen–Shannon divergence) between densities $p_{{\theta }_1}$ and $p_{{\theta }_2}$ of the same exponential family with cumulant function F is

$${\mathrm{JS}}_{\mathrm{KL}}^{G_{\alpha }}[p_{{\theta }_1}:p_{{\theta }_2}]={\mathrm{JS}}_{{B_F}^{*}}^{A_{\alpha }}({\theta }_1:{\theta }_2).$$

Here, we used the bracket notation to emphasize that the statistical distance ${\mathrm{JS}}_{\mathrm{KL}}^{G_{\alpha }}$ is between densities, and the parenthesis notation to emphasize that the distance ${\mathrm{JS}}_{{B_F}^{*}}^{A_{\alpha }}$ is between parameters. We also have ${\mathrm{JS}}_{{\mathrm{KL}}^{*}}^{G_{\alpha }}[p_{{\theta }_1}:p_{{\theta }_2}]=J_F^{\alpha }({\theta }_1:{\theta }_2)$. We also show how to get a closed-form formula for the harmonic Jensen–Shannon divergence of Cauchy scale distributions by taking harmonic mixtures.

For an arbitrary distance D, we define the skew N-Jeffreys symmetrization:

$$J_D^{N_{\beta }}(p_1:p_2)=N_{\beta }(D(p_1:p_2),D(p_{{}_2}:p_1)),$$

(144)

and the skew $(M,N)$-JS-symmetrization:

$${\mathrm{JS}}_D^{M_{\alpha },N_{\beta }}(p_1:p_2)=N_{\beta }(D(p_1:{\left(p_1{p}_2\right)}_{\alpha }^M),D(p_2:{\left(p_1{p}_2\right)}_{\alpha }^M)).$$

(145)

A Java™ source code for computing the geometric Jensen–Shannon divergence between multivariate Gaussian distributions is available online at https://franknielsen.github.io/M-JS/.