On the Jensen–Shannon Symmetrization of Distances Relying on Abstract Means

The Jensen–Shannon divergence is a renowned bounded symmetrization of the unbounded Kullback–Leibler divergence which measures the total Kullback–Leibler divergence to the average mixture distribution. However, the Jensen–Shannon divergence between Gaussian distributions is not available in closed form. To bypass this problem, we present a generalization of the Jensen–Shannon (JS) divergence using abstract means which yields closed-form expressions when the mean is chosen according to the parametric family of distributions. More generally, we define the JS-symmetrizations of any distance using parameter mixtures derived from abstract means. In particular, we first show that the geometric mean is well-suited for exponential families, and report two closed-form formula for (i) the geometric Jensen–Shannon divergence between probability densities of the same exponential family; and (ii) the geometric JS-symmetrization of the reverse Kullback–Leibler divergence between probability densities of the same exponential family. As a second illustrating example, we show that the harmonic mean is well-suited for the scale Cauchy distributions, and report a closed-form formula for the harmonic Jensen–Shannon divergence between scale Cauchy distributions. Applications to clustering with respect to these novel Jensen–Shannon divergences are touched upon.


Kullback-Leibler Divergence and Its Symmetrizations
Let (X , A) be a measurable space [1] where X denotes the sample space and A the σ-algebra of measurable events. Consider a positive measure µ (usually the Lebesgue measure µ L with Borel σ-algebra B(R d ) or the counting measure µ c with power set σ-algebra 2 X ). Denote by P the set of probability distributions.
The Kullback-Leibler Divergence [2] (KLD) KL : P × P → [0, ∞] is the most fundamental distance [2] between probability distributions, defined by: where p and q denote the Radon-Nikodym derivatives of probability measures P and Q with respect to µ (with P, Q µ). The KLD expression between P and Q in Equation (1) is independent of the dominating measure µ. Table A1 summarizes the various distances and their notations used in this paper.
The KLD is also called the relative entropy [2] because it can be written as the difference of the cross-entropy minus the entropy: where h × denotes the cross-entropy [2]: and h(p):= p log denotes the Shannon entropy [2]. Although the formula of the Shannon entropy in Equation (4) unifies both the discrete case and the continuous case of probability distributions, the behavior of entropy in the discrete case and the continuous case is very different: When µ = µ c , Equation (4) yields the discrete Shannon entropy which is always positive and upper bounded by log |X |. When µ = µ L , Equation (4) defines the Shannon differential entropy which may be negative and unbounded [2] (e.g., the differential entropy of the Gaussian distribution N(m, σ) is 1 2 log(2πeσ 2 )). See also [3] for further important differences between the discrete case and the continuous case.
In general, the KLD is an asymmetric distance (i.e., KL(p : q) = KL(q : p), hence the argument separator notation using the delimiter ':') In information theory [2], it is customary to use the double bar notation ' ' instead of the comma ',' notation to avoid confusion with joint random variables. The reverse KL divergence or dual KL divergence is: KL * (P : Q):=KL(Q : P) = q log q p dµ.
In general, the reverse distance or dual distance for a distance D is written as: D * (p : q):=D(q : p).
However, this symmetric distance is not upper bounded, and its sensitivity can raise numerical issues in applications. Here, we used the optional argument separator notation ';' to emphasize that the distance is symmetric but not necessarily a metric distance. This notation matches the notational convention of the mutual information if two joint random variables in information theory [2].
The symmetrization of the KLD may also be obtained using the harmonic mean instead of the arithmetic mean, yielding the resistor average distance [5] R(p; q): R(p; q) = 2 (KL(p : q) + KL(q : p)) KL(p : q)KL(q : p) = 2J(p; q) KL(p : q)KL(q : p) . (9) Another famous symmetrization of the KLD is the Jensen-Shannon Divergence [6] (JSD) defined by: JS(p; q) := 1 2 KL p : p + q 2 + KL q : p + q 2 , (10) This distance can be interpreted as the total divergence to the average distribution (see Equation (10)). The JSD can be rewritten as a Jensen divergence (or Burbea-Rao divergence [7]) for the negentropy generator −h (called Shannon information): An important property of the Jensen-Shannon divergence compared to the Jeffreys divergence is that this distance is always bounded: This follows from the fact that KL p : Finally, the square root of the JSD (i.e., √ JS) yields a metric distance satisfying the triangular inequality [8,9]. The JSD has found applications in many fields such as bioinformatics [10] and social sciences [11], just to name a few. Recently, the JSD has gained attention in the deep learning community with the Generative Adversarial Networks (GANs) [12]. In computer vision and pattern recognition, one often relies on information-theoretic techniques to perform registration and recognition tasks. For example, in [13], the authors use a mixture of Principal Axes Registrations (mPAR) whose parameters are estimated by minimizing the KLD between the considered two-point distributions. In [14], the authors parameterize both shapes and deformations using Gaussian Mixture Models (GMMs) to perform non-rigid shape registration. The lack of closed-form formula for the KLD between GMMs [15] spurred the use of other statistical distances which admit a closed-form expression for GMMs. For example, in [16], shape registration is performed by using the Jensen-Rényi divergence between GMMs. See also [17] for other information-theoretic divergences that admit closed-form formula for some statistical mixtures extending GMMs.
In information geometry [18], the KLD, JD and JSD are invariant divergences which satisfy the property of information monotonicity [18]. The class of (separable) distances satisfying the information monotonicity are exhaustively characterized as Csiszár's f -divergences [19]. A f -divergence is defined for a convex generator function f strictly convex at 1 (with f (1) = f (1) = 0) by: The Jeffreys and Jensen-Shannon f -generators are:

Statistical Distances and Parameter Divergences
In information and probability theory, the term "divergence" informally means a statistical distance [2]. However in information geometry [18], a divergence has a stricter meaning of being a smooth parametric distance (called a contrast function in [20]) from which a dual geometric structure can be derived [21,22].
Consider parametric distributions p θ belonging to a parametric family of distributions {p θ : θ ∈ Θ} (e.g., Gaussian family or Cauchy family), where Θ denotes the parameter space. Then a statistical distance D between distributions p θ and p θ amount to an equivalent parameter distance: For example, the KLD between two distributions belonging to the same exponential family (e.g., Gaussian family) amount to a reverse Bregman divergence for the cumulant generator F of the exponential family [23]: A Bregman divergence B F is defined for a strictly convex and differentiable generator F as: where ·, · is an inner product (usually the Euclidean dot product for vector parameters). Similar to the interpretation of the Jensen-Shannon divergence (statistical divergence) as a Jensen divergence for the negentropy generator, the Jensen-Bregman divergence [7] JB F (parametric divergence JBD) amounts to a Jensen divergence J F for a strictly convex generator F : Θ → R: Let us introduce the notation (θ p θ q ) α :=(1 − α)θ p + αθ q to denote the linear interpolation (LERP) of the parameters. Then we have more generally that the skew Jensen-Bregman divergence JB α F (θ : θ ) amounts to a skew Jensen divergence J α F (θ : θ ):

J-Symmetrization and JS-Symmetrization of Distances
For any arbitrary distance D(p : q), we can define its skew J-symmetrization for α ∈ [0, 1] by: and its JS-symmetrization by: Usually, α = 1 2 , and for notational brevity, we drop the superscript: JS D (p : q) := JS 1 2 D (p : q). The Jeffreys divergence is twice the J-symmetrization of the KLD, and the Jensen-Shannon divergence is the JS-symmetrization of the KLD.
The J-symmetrization of a f -divergence I f is obtained by taking the generator where f (u) = u f ( 1 u ) is the conjugate generator: The JS-symmetrization of a f -divergence with (pq) α = (1 − α)p + αq is obtained by taking the generator We check that we have: A family of symmetric distances unifying the Jeffreys divergence with the Jensen-Shannon divergence was proposed in [24]. Finally, let us mention that once we have symmetrized a distance D, we may also metrize this symmetric distance by choosing (when it exists) the largest exponent δ > 0 such that D δ becomes a metric distance [8,[25][26][27][28].

Contributions and Paper Outline
The paper is organized as follows: Section 2 reports the special case of mixture families in information geometry [18] for which the Jensen-Shannon divergence can be expressed as a Bregman divergence (Theorem 1), and highlight the lack of closed-form formula when considering exponential families. This fact precisely motivated this work. Section 3 introduces the generalized Jensen-Shannon divergences using statistical mixtures derived from abstract weighted means (Definitions 2 and 5), presents the JS-symmetrization of statistical distances, and report a sufficient condition to get bounded JS-symmetrizations (Property 1).
In Section 4.1, we consider the calculation of the geometric JSD between members of the same exponential family (Theorem 2) and instantiate the formula for the multivariate Gaussian distributions (Corollary 1). We discuss about applications for k-means clustering in Section 4.1.2. In Section 4.2, we illustrate the method with another example that calculates in closed form the harmonic JSD between scale Cauchy distributions (Theorem 4).
Finally, we wrap up and conclude this work in Section 5.

Jensen-Shannon Divergence in Mixture and Exponential Families
We are interested to calculate the JSD between densities belonging to parametric families of distributions.
A trivial example is when p = (p 0 , . . . , p D ) and q = (q 0 , . . . , q D ) are categorical distributions: The average distribution p+q 2 is a again categorical distribution, and the JSD is expressed plainly as: Another example is when p = m θ p and q = m θ q both belong to the same mixture family [18] M: for linearly independent component distributions p 0 , p 1 , . . . , p D . We have [29]: where B F is a Bregman divergence defined in Equation (20) obtained for the convex negentropy generator [29] F(θ) = −h(m θ ). The proof that F(θ) is a strictly convex function is not trivial [30]. The mixture families include the family of categorical distributions over a finite alphabet Beware that mixture families impose to prescribe the component distributions. Therefore, a density of a mixture family is a special case of statistical mixtures (e.g., GMMs) with prescribed component distributions.
The mathematical identity of Equation (35) that does not yield a practical formula since F(θ) is usually not itself available in closed form. Worse, the Bregman generator can be non-analytic [31]. Nevertheless, this identity is useful for computing the right-sided Bregman centroid (left KL centroid of mixtures) since this centroid is equivalent to the center of mass, and independent of the Bregman generator [29].
Since the mixture of mixtures is also a mixture, specifically it follows that we get a closed-form expression for the JSD between mixtures belonging to M.
Theorem 1 (JSD between mixtures). The Jensen-Shannon divergence between two distributions p = m θ p and q = m θ q belonging to the same mixture family M is expressed as a Jensen-Bregman divergence for the negentropy generator F: This amounts to calculate the Jensen divergence: Now, consider distributions p = e θ p and q = e θ q belonging to the same exponential family [18] E : where denotes the natural parameter space. We have [18]: where F denotes the log-normalizer or cumulant function of the exponential family [18].
However, e θp +e θq 2 does not belong to E in general, except for the case of the categorical/multinomial family which is both an exponential family and a mixture family [18].
For example, the mixture of two Gaussian distributions with distinct components is not a Gaussian distribution. Thus, it is not obvious to get a closed-form expression for the JSD in that case. This limitation precisely motivated the introduction of generalized JSDs defined in the next section.
Notice that in [32,33], it is shown how to express or approximate the f -divergences using expansions of power χ pseudo-distances. These power chi distances can all be expressed in closed form when dealing with isotropic Gaussians. This result holds for the JSD since the JSD is a f -divergence [33].

Generalized Jensen-Shannon Divergences
We first define abstract means M, and then generic statistical M-mixtures from which generalized Jensen-Shannon divergences are built thereof.

Definitions
Consider an abstract mean [34] M. That is, a continuous bivariate function M(·, ·) : I × I → I on an interval I ⊂ R that satisfies the following in-betweenness property: Using the unique dyadic expansion of real numbers, we can always build a corresponding weighted mean M α (p, q) (with α ∈ [0, 1]) following the construction reported in [34] (page 3) such that M 0 (p, q) = p and M 1 (p, q) = q. In the remainder, we consider I = (0, ∞).
Examples of common weighted means are: the geometric mean G α (x, y) = x 1−α y α , and • the harmonic mean H α (x, y) = xy (1−α)y+αx . These means can be unified using the concept of quasi-arithmetic means [34] (also called Kolmogorov-Nagumo means): where h is a strictly monotonous function. For example, the geometric mean G α (x, y) is obtained as M h α (x, y) for the generator h(u) = log(u). Rényi used the concept of quasi-arithmetic means instead of the arithmetic mean to define axiomatically the Rényi entropy [35] of order α in information theory [2].
For any abstract weighted mean, we can build a statistical mixture called a M-mixture as follows: ) of densities p and q with respect to a mean M is a α-weighted M-mixture defined by: where is the normalizer function (or scaling factor) ensuring that (pq) M α ∈ P. (The bracket notation f denotes the integral of f over X .) standing for the arithmetic mean) represents the usual statistical mixture [36] of two distributions p(x) and q(x) ('G' standing for the geometric mean G) is an exponential family of order [37] 1: The two-component M-mixture can be generalized to a k-component M-mixture with α ∈ ∆ k−1 , the (k − 1)-dimensional standard simplex: where For a given pair of distributions p and q, the set {M α (p(x), q(x)) : α ∈ [0, 1]} describes a path in the space of probability density functions. This density interpolation scheme was investigated for quasi-arithmetic weighted means in [38][39][40]. In [41], the authors study the Fisher information matrix for the α-mixture models (using α-power means).
We call (pq) M α the α-weighted M-mixture, thus extending the notion of α-mixtures [42] obtained for power means P α . Notice that abstract means have also been used to generalize Bregman divergences using the concept of (M, N)-convexity [43].
Let us state a first generalization of the Jensen-Shannon divergence: When M α = A α , we recover the ordinary Jensen-Shannon divergence since A α (p : q) = (pq) α (and Z A α (p : q) = 1). We can extend the definition to the JS-symmetrization of any distance: By notation, we have JS M α (p : q) = JS M α KL (p : q). That is, the arithmetic JS-symmetrization of the KLD is the JSD.
Let us define the α-skew K-divergence [6,44] K α (p : q) as Then the Jensen-Shannon divergence and the Jeffreys divergence can be rewritten [24] as since KL(p : Similarly, we can define the generalized skew K-divergence: The success of the JSD compared to the JD in applications is partially due to the fact that the JSD is upper bounded by log 2. So, one question to ask is whether those generalized JSDs are upper bounded or not? To report a sufficient condition, let us first introduce the dominance relationship between means: We say that a mean M dominates a mean N when M(x, y) ≥ N(x, y) for all x, y ≥ 0, see [34]. In that case we write concisely M ≥ N. For example, the Arithmetic-Geometric-Harmonic (AGH) inequality states that A ≥ G ≥ H.
Consider the term When mean M α dominates the arithmetic mean A α , we have Notice that Z A α (p : q) = 1 (when M = A is the arithmetic mean), and we recover the fact that the α-skew Jensen-Shannon divergence is upper bounded by − log(1 − α) (e.g., log 2 when α = 1 2 ). We summarize the result in the following property: Let us observe that dominance of means can be used to define distances: For example, the celebrated α-divergences can be interpreted as a difference of two means, the arithmetic mean and the geometry mean: We can also define the generalized Jeffreys divergence as follows: Definition 4 (N-Jeffreys divergence). For a mean N, the skew N-Jeffreys divergence (for β ∈ [0, 1]) is defined by This definition includes the (scaled) resistor average distance [5] R(p; q), obtained for the harmonic mean N = H for the KLD with skew parameter β = 1 2 : R(p; q) = 2J(p; q) KL(p : q)KL(q : p) .
In [5], the factor 1 2 is omitted to keep the spirit of the original Jeffreys divergence. We can further extend this definition for any arbitrary divergence D as follows: We now show how to choose the abstract mean according to the parametric family of distributions to obtain some closed-form formula for some statistical distances.

Some Closed-Form Formula for the M-Jensen-Shannon Divergences
Our motivation to introduce these novel families of M-Jensen-Shannon divergences is to obtain closed-form formula when probability densities belong to some given parametric families P Θ . We shall illustrate the principle of the method to choose the right abstract mean for the considered parametric family, and report corresponding formula for the following two case studies: 1.
The geometric G-Jensen-Shannon divergence for the exponential families (Section 4.1), and 2.
the harmonic H-Jensen-Shannon divergence for the family of Cauchy scale distributions (Section 4.2).
Recall that the arithmetic A-Jensen-Shannon divergence is well-suited for mixture families (Theorem 1).

The Geometric G-Jensen-Shannon Divergence
Consider an exponential family [37] E F with log-normalizer F: and natural parameter space The log-normalizer (a log-Laplace function also called log-partition or cumulant function) is a real analytic convex function.
We seek for a mean M such that the weighted M-mixture density (p θ 1 p θ 2 ) M α of two densities p θ 1 and p θ 2 of the same exponential family yields another density of that exponential family (e.g., p (θ 1 θ 2 ) α ). When considering exponential families, choose the weighted geometric mean G α for the abstract mean M α (x, y): M α (x, y) = G α (x, y) = x 1−α y α , for x, y > 0. Indeed, it is well-known that the normalized weighted product of distributions belonging to the same exponential family also belongs to this exponential family [45]: where the normalization factor is for the skew Jensen divergence J α F defined by: Notice that since the natural parameter space Θ is convex, the distribution p (θ 1 θ 2 ) α ∈ E F (since (θ 1 θ 2 ) α ∈ Θ).
Thus, it follows that we have: This allows us to conclude that the G-Jensen-Shannon divergence admits the following closed-form expression between densities belonging to the same exponential family: Please note that since ( . The dual divergence [46] D * (with respect to the reference argument) or reverse divergence of a divergence D is defined by swapping the calling arguments: D * (θ : θ ):=D(θ : θ).
Thus, if we defined the Jensen-Shannon divergence for the dual KL divergence KL * (p : q):=KL(q : p) then we obtain: Please note that JS D * = JS D * .
In general, the JS-symmetrization for the reverse KL divergence is where m = p+q 2 = A(p, q) and G(p, q) = √ pq. Since A ≥ G (arithmetic-geometric inequality), it follows that JS KL * (p; q) ≥ 0.
Theorem 2 (G-JSD and its dual JS-symmetrization in exponential families). The α-skew G-Jensen-Shannon divergence JS G α between two distributions p θ 1 and p θ 2 of the same exponential family E F is expressed in closed form for α ∈ (0, 1) as:
The multivariate Gaussian family is also called the multivariate normal family in the literature, or MVN family for short.
Let λ:=(λ v , λ M ) = (µ, Σ) denote the composite (vector,matrix) parameter of an MVN. The d-dimensional MVN density is given by where | · | denotes the matrix determinant. The natural parameters θ are also expressed using both a vector parameter θ v and a matrix parameter θ M in a compound object θ = (θ v , θ M ). By defining the following compound inner product on a composite (vector,matrix) object where tr(·) denotes the matrix trace, we rewrite the MVN density of Equation (83) in the canonical form of an exponential family [37]: where is the compound natural parameter and is the compound sufficient statistic. The function F θ is the strictly convex and continuously differentiable log-normalizer defined by: The log-normalizer can be expressed using the ordinary parameters, λ = (µ, Σ), as: The moment/expectation parameters [18,49] are We report the conversion formula between the three types of coordinate systems (namely the ordinary parameter λ, the natural parameter θ and the moment parameter η) as follows: The dual Legendre convex conjugate [18,49] is and θ = ∇ η F * η (η). We check the Fenchel-Young equality when η = ∇F(θ) and θ = ∇F * (η): The Kullback-Leibler divergence between two d-dimensional Gaussians distributions p (µ 1 ,Σ 1 ) and We check that KL(p (µ,Σ) : p (µ,Σ) ) = 0 since ∆ µ = 0 and tr(Σ −1 Σ) = tr(I) = d. Notice that when that is half the squared Mahalanobis distance for the precision matrix Σ −1 (a positive-definite matrix: Σ −1 0), where the Mahalanobis distance is defined for any positive matrix Q 0 as follows: The Kullback-Leibler divergence between two probability densities of the same exponential families amount to a Bregman divergence [18]: where the Bregman divergence is defined by with η = ∇F(θ ). Define the canonical divergence [18] A F (θ 1 : since F * * = F. We have B F (θ 1 : θ 2 ) = A F (θ 1 : η 2 ). Now, observe that p θ (0, θ) = exp(−F(θ)) when t(0), θ = 0. In particular, this holds for the multivariate normal family. Thus, we have the following proposition.

Applications to k-Means Clustering
Let P = {p 1 , . . . , p n } denote a point set, and C = {c 1 , . . . , c k } denote a set of k (cluster) centers. The generalized k-means objective [23] with respect to a distance D is defined by: By defining the distance D(p, C) = min j∈{1,...,k} D(p : c j ) of a point to a set of points, we can rewrite compactly the objective function as E D (P, C) = 1 n ∑ n i=1 D(p i , C). Denote by E * D (P, k) the minimum objective loss for a set of k = |C| clusters: E * D (P, k) = min |C|=k E D (P, C). It is NP-hard [50] to compute E * D (P, k) when k > 1 and the dimension d > 1. The most common heuristic is Lloyd's batched k-means [23] that yields a local minimum.
A centroid for a set of parameters θ 1 , . . . , θ n is defined as the minimizer of the functional In particular, the symmetrized Bregman centroids have been studied in [55] (for JS G α ), and the Jensen centroids (for JS G α * ) have been investigated in [7] using the convex-concave iterative procedure.

The Harmonic Jensen-Shannon Divergence (H-JS)
The principle to get closed-form formula for generalized Jensen-Shannon divergences between distributions belonging to a parametric family P Θ = {p θ : θ ∈ Θ} consists of finding an abstract mean M such that the M-mixture (p θ 1 p θ 2 ) M α belongs to the family P Θ . In particular, when Θ is a convex domain, we seek a mean M such that (p θ 1 p θ 2 ) M α = p (θ 1 θ 2 ) α with (θ 1 θ 2 ) α ∈ Θ. Let us consider the weighted harmonic mean [34] (induced by the harmonic mean) H: The harmonic mean is a quasi-arithmetic mean H α (x, y) = M h α (x, y) obtained for the monotone (decreasing) function h(u) = 1 u (or equivalently for the increasing monotone function h(u) = − 1 u ). This harmonic mean is well-suited for the scale family C of Cauchy probability distributions (also called Lorentzian distributions): where γ denotes the scale and p std (x) = 1 π(1+x 2 ) the standard Cauchy distribution. Using the computer algebra system Maxima (http://maxima.sourceforge.net/) we find that (see Appendix B) where the normalizing coefficient is since we have (γ 1 γ 2 ) 1−α = (γ 2 γ 1 ) α .
The H-Jensen-Shannon symmetrization of a distance D between distributions writes as: where H α denote the weighted harmonic mean. When D is available in closed form for distributions belonging to the scale Cauchy distributions, so is JS H α D (p : q). For example, consider the KL divergence formula between two scale Cauchy distributions: where A and G denote the arithmetic and geometric means, respectively. The formula initially reported in [56] has been corrected by the authors. Since A ≥ G (and A G ≥ 1), it follows that KL(p γ 1 : p γ 2 ) ≥ 0. Notice that the KL divergence is symmetric for Cauchy scale distributions. We note in passing that for exponential families, the KL divergence is symmetric only for the location Gaussian family (since the only symmetric Bregman divergences are the squared Mahalanobis distances [57]). The cross-entropy between scale Cauchy distributions is h × (p γ 1 : p γ 2 ) = log π (γ 1 +γ 2 ) 2 γ 2 , and the differential entropy is h(p γ ) = h × (p γ : p γ ) = log 4πγ.
Notice that KL formula is scale-invariant, and this property holds for any scale family: A direct proof follows from a change of variable in the KL integral with y = x λ and dx = λdy. Please note that although the KLD between scale Cauchy distributions is symmetric, it is not the case for all scale families: For example, the Rayleigh distributions form a scale family with the KLD amounting to compute a Bregman asymmetric Itakura-Saito divergence between parameters [37].
Instead of the KLD, we can choose the total variation distance for which a formula has been reported in [38] between two Cauchy distributions. Notice that the Cauchy distributions are alpha-stable distributions for α = 1 and q Gaussian distributions for q = 2 ([58], p. 104). A closed-form formula for the divergence between two q-Gaussians is given in [58] when q < 2. The definite integral h q (p) = +∞ −∞ p(x) q dµ is available in closed form for Cauchy distributions. When q = 2, we have h 2 (p γ ) = 1 2πγ .
We refer to [38] for yet other illustrative examples considering the family of Pearson type VII distributions and central multivariate t-distributions which use the power means (quasi-arithmetic means M h induced by h(u) = u α for α > 0) for defining mixtures. Table 1 summarizes the various examples introduced in the paper.

The M-Jensen-Shannon Matrix Distances
In this section, we consider distances between matrices which play an important role in quantum computing [59,60]. We refer to [61] for the matrix Jensen-Bregman logdet divergence. The Hellinger distance can be interpreted as the difference of an arithmetic mean A and a geometric mean G: Notice that since A ≥ G, we have D H (p, q) ≥ 0. The scaled and squared Hellinger distance is an α-divergence I α for α = 0. Recall that the α-divergence can be interpreted as the difference of a weighted arithmetic minus a weighted geometry mean.
In general, if a mean M 1 dominates a mean M 2 , we may define the distance as When considering matrices [62], there is not a unique definition of a geometric matrix mean, and thus we have different notions of matrix Hellinger distances [62], some of them are divergences (smooth distances defining a dualistic structure in information geometry).
For example, we can choose the von Neumann matrix divergence [63]: or the LogDet matrix divergence [63]: where square matrices X 1 and X 2 have dimension d.

Conclusions and Perspectives
We introduced a generalization of the celebrated Jensen-Shannon divergence [6], termed the (M, N)-Jensen-Shannon divergences, based on M-mixtures derived from abstract means M. This new family of divergences includes the ordinary Jensen-Shannon divergence when both M and N are set to the arithmetic mean. We reported closed-form expressions of the M Jensen-Shannon divergences for mixture families and exponential families in information geometry by choosing the arithmetic and geometric weighted mean, respectively. The α-skewed geometric Jensen-Shannon divergence (G-Jensen-Shannon divergence) between densities p θ 1 and p θ 2 of the same exponential family with cumulant function F is JS G α KL [p θ 1 : p θ 2 ] = JS A α B F * (θ 1 : θ 2 ).
Here, we used the bracket notation to emphasize that the statistical distance JS G α KL is between densities, and the parenthesis notation to emphasize that the distance JS A α B F * is between parameters. We also have JS G α KL * [p θ 1 : p θ 2 ] = J α F (θ 1 : θ 2 ). We also show how to get a closed-form formula for the harmonic Jensen-Shannon divergence of Cauchy scale distributions by taking harmonic mixtures.