Fast Proxy Centers for the Jeffreys Centroid: The Jeffreys–Fisher–Rao Center and the Gauss–Bregman Inductive Center

Abstract

The symmetric Kullback–Leibler centroid, also called the Jeffreys centroid, of a set of mutually absolutely continuous probability distributions on a measure space provides a notion of centrality which has proven useful in many tasks, including information retrieval, information fusion, and clustering. However, the Jeffreys centroid is not available in closed form for sets of categorical or multivariate normal distributions, two widely used statistical models, and thus needs to be approximated numerically in practice. In this paper, we first propose the new Jeffreys–Fisher–Rao center defined as the Fisher–Rao midpoint of the sided Kullback–Leibler centroids as a plug-in replacement of the Jeffreys centroid. This Jeffreys–Fisher–Rao center admits a generic formula for uni-parameter exponential family distributions and a closed-form formula for categorical and multivariate normal distributions; it matches exactly the Jeffreys centroid for same-mean normal distributions and is experimentally observed in practice to be close to the Jeffreys centroid. Second, we define a new type of inductive center generalizing the principle of the Gauss arithmetic–geometric double sequence mean for pairs of densities of any given exponential family. This new Gauss–Bregman center is shown experimentally to approximate very well the Jeffreys centroid and is suggested to be used as a replacement for the Jeffreys centroid when the Jeffreys–Fisher–Rao center is not available in closed form. Furthermore, this inductive center always converges and matches the Jeffreys centroid for sets of same-mean normal distributions. We report on our experiments, which first demonstrate how well the closed-form formula of the Jeffreys–Fisher–Rao center for categorical distributions approximates the costly numerical Jeffreys centroid, which relies on the Lambert W function, and second show the fast convergence of the Gauss–Bregman double sequences, which can approximate closely the Jeffreys centroid when truncated to a first few iterations. Finally, we conclude this work by reinterpreting these fast proxy Jeffreys–Fisher–Rao and Gauss–Bregman centers of Jeffreys centroids under the lens of dually flat spaces in information geometry.

1. Introduction

Let $(\mathcal{X},\mathcal{F})$ be a measurable space with sample space $\mathcal{X}$ and $\sigma$-algebra of events $\mathcal{F}$, and $\mu$ a positive measure. We consider a finite set $\{P_1,\dots ,P_n\}$ of n probability distributions all dominated by $\mu$ and weighted by a vector w belonging to the open standard simplex ${\Delta }_n=\left\{x=(x_1,\dots ,x_n):x_1>0,\dots ,x_n>0,{\sum }_{i=1}^n{x}_i=1\right\}\subset {\mathbb{R}}^n$. Let $\mathcal{P}=\{p_1,\dots ,p_n\}$ be the Radon–Nikodym densities of $P_1,\dots ,P_n$ with respect to $\mu$, i.e., $p_i={\displaystyle \frac{\mathrm{d}{P}_i}{\mathrm{d}\mu }}$.

The Kullback–Leibler divergence (KLD) between two densities $p\left(x\right)$ and $q\left(x\right)$ is defined by $D_{\mathrm{KL}}(p:q)=\int p\left(x\right)\log {\displaystyle \frac{p\left(x\right)}{q\left(x\right)}}\mathrm{d}\mu \left(x\right)$. The KLD is asymmetric: $D_{\mathrm{KL}}(p:q)\ne D_{\mathrm{KL}}(q:p)$. We use the argument delimiter ‘:’ as a notation to indicate this asymmetry. The Jeffreys divergence [1] symmetrizes the KLD as follows:

$$\begin{array}{ccc}\hfill D_J(p,q)& =& D_{\mathrm{KL}}(p:q)+D_{\mathrm{KL}}(q:p),\hfill \\ & =& {\int }_{\mathcal{X}}(p\left(x\right)-q\left(x\right))\log {\displaystyle \frac{p\left(x\right)}{q\left(x\right)}}\mathrm{d}\mu \left(x\right).\hfill \end{array}$$

In general, the D-barycenter $C_D$ of $\mathcal{P}$ with respect to a statistical dissimilarity measure $D(·:·)$ yields a notion of centrality $C_R$ defined by the following optimization problem:

$$c_R=\arg \underset{p}{\min }\sum _{i=1}^n{w}_{i}D(p_i:p).$$

(1)

Here, the upper case letter ‘R’ indicates that the optimization defining the D-barycenter is carried on the right argument. When $w=({\displaystyle \frac{1}{n}},\dots ,{\displaystyle \frac{1}{n}})$ is the uniform weight vector, the D-barycenter is called the D-centroid. We shall loosely call centroids barycenters in the remainder even when the weight vector is not uniform. Centroids with respect to information-theoretic measures have been studied in the literature.

Let us mention some examples of centroids: The entropic centroids [2] (i.e., Bregman centroids and f-divergences centroids), the Burbea–Rao and Bhattacharyya centroids [3], the $\alpha$-centroids with respect to $\alpha$-divergences [4], the Jensen–Shannon centroids [5], etc.

The $D_J$-centroid is also called the symmetric Kullback–Leibler (SKL) divergence centroid [6] in the literature. However, since there are many possible symmetrizations of the KLD [7] like the Jensen–Shannon divergence [8] or the resistor KLD [9], we prefer to use the term Jeffreys centroid instead of SKL centroid to avoid any possible ambiguity on the underlying divergence. Notice that the square root of the Jensen–Shannon divergence is a metric distance [10,11] but all powers $D_J^{\alpha }$ of Jeffreys divergence $D_J$ for $\alpha >0$ do not yield metric distances [12].

This paper considers the Jeffreys centroids of a finite weighted set of densities $\mathcal{P}=\{p_{{\theta }_1},\dots ,p_{{\theta }_n}\}$ belonging to some prescribed exponential family [13] $\mathcal{E}$:

$$c=\arg \underset{p}{\min }\sum _{i=1}^n{w}_i{D}_J(p_{{\theta }_i},p).$$

(2)

In particular, we are interested in computing the Jeffreys centroids for sets of categorical distributions or sets of multivariate normal distributions [14].

In general, centroids are used in k means [15,16]-type clustering or hierarchical clustering (e.g., Ward criterion [17]) and information fusion tasks [18] (related to distributed model estimation [19]) among others. See Figure 1. The choice of the dissimilarity measure depends on the application at hand [20]. Clustering with respect to Jeffreys divergence/-Jeffreys centroid has proven useful in many scenarios: for example, it was shown to perform experimentally better than Euclidean or square Euclidean distances for compressed histograms of gradient descriptors [21] or in fuzzy clustering [22]. Jeffreys divergence has also been used for image processing [23], including image segmentation [24], speech processing [25], and computer vision [26], just to name a few. In particular, finding weighted means of centered 3D normal distributions plays an important role in diffusion tensor imaging (DTI) for smoothing and filtering DT images [27] which consist of sets of normal distributions centered at 3D grid locations.

In general, the Jeffreys centroid is not known in closed form for exponential families [28] like the family of categorical distributions or the family of normal distributions often met in applications and thus needs to be numerically approximated in practice. The main contribution of this paper is to present and study two proxy centers as drop-in replacements of the Jeffreys centroids in applications and report the generic structural formula for generic exponential families with an explicit closed-form formula for the families of categorical and multivariate normal distributions. Namely, we define the Jeffreys–Fisher–Rao (JFR) center (Definition 2) and the Gauss–Bregman (GB) inductive center (Definition 3) in Section 2.

This paper is organized as follows: By interpreting in two different ways the closed-form formula of the Jeffreys centroids for the particular case of sets of centered multivariate normal distributions [29] (proof reported in Appendix B), we define the Gauss–Bregman (GB) centers and the Jeffreys–Fisher–Rao (JFR) centers for sets of densities belonging to an exponential family in Section 2. The Jeffreys centroid coincides with both the Gauss–Bregman inductive center and the Jeffreys–Fisher–Rao center for centered multivariate normal distributions but differ from each other in general. In Section 2.4, we study the Gauss–Bregman inductive center [30] induced by the cumulant function of an exponential family and prove the convergence under the separability condition of the generalized Gauss double sequences in the limit (Theorem 3). This Gauss–Bregman center can be easily approximated by limiting the number of iterations of a double sequence inducing it. In Section 4, we report the generic formula for Jeffreys–Fisher–Rao centers for sets of uni-order exponential families [13] and explicitly give the closed-form formula for the categorical family and the multivariate normal family. A comparison of those proxy centers with the numerical Jeffreys centroids is experimentally studied and visually illustrated with some examples. Thus, we propose to use in applications (e.g., clustering) either the fast Jeffreys–Fisher–Rao center when a closed-form formula is available for the family of distributions at hand or the Gauss–Bregman center approximation with a prescribed number of iterations as a drop-in replacement of the numerical Jeffreys centroids while keeping the Jeffreys divergence. Some experiments of the JFR and GB centers are reported for the Jeffreys centroid of categorical distributions in Section 5. Finally, we conclude this paper in Section 6 with a discussion and a generalization of our results to the more general setting of dually flat spaces of information geometry [14].

The core of this paper is followed by an Appendix section as follows: In Appendix A, we explicitly give the algorithm outlined in [31] for numerically computing the Jeffreys centroid of sets of categorical distributions. In Appendix B, we report a proof on the closed-form formula of the Jeffreys centroid for centered normal distributions [29] that motivated this paper. In Appendix C, we explain how to calculate in practice the elaborated closed-form formula for the Fisher–Rao geodesic midpoint between two multivariate normal distributions [32].

2. Proxy Centers for Jeffreys Centroids

2.1. Background on Jeffreys Centroids

A density $p_{\theta }$ belonging to an exponential family [13] $\mathcal{E}$ can be expressed canonically as $p_{\theta }\left(x\right)=\exp (\langle \theta ,t\left(x\right)\rangle -F\left(\theta \right))\mathrm{d}\mu \left(x\right)$, where $t\left(x\right)$ is a sufficient statistic vector, $F\left(\theta \right)=\log \int \exp \left(\langle \theta ,t\left(x\right)\rangle \right)\mathrm{d}\mu \left(x\right)$ is the log-normalizer, and $\theta$ is the natural parameter belonging to the natural parameter space $\Theta$. We consider minimal regular exponential families [13] like the discrete family of categorical distributions (i.e., $\mu$ is the counting measure) or the continuous family of multivariate normal distributions (i.e., $\mu$ is the Lebesgue measure).

The Jeffreys centroid of categorical distributions was first studied by Veldhuis [6], who designed a numerical two-nested loops Newton-like algorithm [6]. A random variable X following a categorical distribution $\mathrm{Cat}\left(p\right)$ for a parameter $p\in {\Delta }_d$ in sample space $\mathcal{X}=\{{\omega }_1,\dots ,{\omega }_d\}$ is such that $\mathrm{Pr}(X={\omega }_i)=p_i$. Categorical distributions are often used in image processing to statistically model normalized histograms with non-empty bins. The exact characterization of the Jeffreys centroid was given in [31].

We summarize the results regarding the categorical Jeffreys centroid [31] in the following theorem:

Theorem 1

(Categorical Jeffreys centroid [31]). The Jeffreys centroid of a set of n categorical distributions parameterized by $\mathcal{P}=\{p_1,\dots ,p_n\}\in {\Delta }_d$ arranged in a matrix $P=\left[p_{i,j}\right]\in {\mathbb{R}}^{n\times d}$ and weighted by a vector $w=(w_1,\dots ,w_n)\in {\Delta }^n$ is $c\left(\lambda \right)=(c_1\left(\lambda \right),\dots ,c_d\left(\lambda \right))$ with

$$c_j\left(\lambda \right)={\displaystyle \frac{a_j}{W_0\left(\frac{a_j}{g_j}{e}^{1+\lambda }\right)}},\forall j\in \{1,\dots ,d\},$$

where $a_j={\sum }_{i=1}^n{w}_i{p}_{i,j}$ and $g_j={\displaystyle \frac{{\prod }_{i=1}^n{p}_{i,j}^{w_i}}{{\sum }_{j=1}^d{\prod }_{i=1}^n{p}_{i,j}^{w_i}}}$ are the j-th components of the weighted arithmetic and normalized geometric means, respectively; $W_0$ is the principal branch of the Lambert W function [33]; and $\lambda \le 0$ is the unique real value such that $\lambda =-D_{\mathrm{KL}}(c\left(\lambda \right):g)$.

Furthermore, a simple bisection search is reported in [31] §III.B that we convert into Algorithm A1 in Appendix A, which allows one to numerically approximate the Jeffreys centroid to arbitrary fine precision.

2.2. Jeffreys Centroids on Exponential Family Densities: Symmetrized Bregman Centroids

The Jeffreys divergence between two densities of an exponential family $\mathcal{E}=\{p_{\theta }\left(x\right)=\exp (\langle t\left(x\right),\theta \rangle -F\left(\theta \right)):\theta \in \Theta \}$ with cumulant function $F\left(\theta \right)$ amounts to a symmetrized Bregman divergence [28] (SBD):

$$D_J(p_{\theta },p_{{\theta }^{\prime }})=S_F(\theta ,{\theta }^{\prime }):=\langle {\theta }_1-{\theta }_2,\nabla F\left({\theta }_1\right)-\nabla F\left({\theta }_2\right)\rangle .$$

Using convex duality, we have $S_F(\theta ,{\theta }^{\prime })=S_{F^{*}}(\eta ,{\eta }^{\prime })$, where $\eta =\nabla F\left(\theta \right)$ and $F^{*}\left(\eta \right)=\langle \eta ,{(\nabla F)}^{-1}\left(\eta \right)\rangle -F\left({(\nabla F)}^{-1}\left(\eta \right)\right)$ is the Legendre–Fenchel convex conjugate. Thus, the Jeffreys barycenter of $\mathcal{P}=\{p_{{\theta }_1},\dots ,p_{{\theta }_n}\}$ amounts to either a symmetrized Bregman barycenter on the natural parameters ${\mathcal{P}}_{\theta }=\{{\theta }_1,\dots ,{\theta }_n\}$ with respect to $S_F$ or a symmetrized Bregman barycenter on the dual moment parameters ${\mathcal{P}}_{\eta }=\{{\eta }_1,\dots ,{\eta }_n\}$ with respect to $S_{F^{*}}$.

It was shown in [28] that the symmetrized Bregman barycenter ${\theta }_S$ of n weighted points amounts to the following minimization problem involving only the sided Bregman centroids:

$$\begin{array}{ccc}\hfill {\theta }_S& :=& {\displaystyle \arg \underset{\theta \in \Theta }{\min }\sum _i{w}_i{S}_F(\theta ,{\theta }_i),}\hfill \\ \hfill & \equiv & {\displaystyle \arg \underset{\theta \in \Theta }{\min }{B}_F(\overline{\theta }:\theta )+B_F(\theta :\underline{\theta }),}\hfill \end{array}$$

(3)

where $\overline{\theta }={\sum }_i{w}_i{\theta }_i$ (right Bregman centroid) and $\underline{\theta }={(\nabla F)}^{-1}({\sum }_i{w}_i\nabla F\left({\theta }_i\right))$ (left Bregman centroid). Those $\overline{\theta }$ and $\underline{\theta }$ centers are centroids [28] with respect to the Bregman divergence $B_F({\theta }_1:{\theta }_2)=F\left({\theta }_1\right)-F\left({\theta }_2\right)-\langle {\theta }_1-{\theta }_2,\nabla F\left({\theta }_2\right)\rangle$ and reverse Bregman divergence: ${B_F}^{*}({\theta }_1:{\theta }_2):=B_F({\theta }_2:{\theta }_1)$:

$$\begin{array}{ccc}\hfill \overline{\theta }& =& \arg \underset{\theta }{\min }\sum _i{w}_i{B}_F({\theta }_i:\theta ),\hfill \\ \hfill \underline{\theta }& =& \arg \underset{\theta }{\min }\sum _i{w}_i{B}_F(\theta :{\theta }_i)=\arg \underset{\theta }{\min }\sum _i{w}_i{B_F}^{*}({\theta }_i:\theta ).\hfill \end{array}$$

In general, when $H:{\mathbb{R}}^m\to \mathbb{R}$ is a strictly convex differentiable real-valued function of Legendre type [34], the gradient $\nabla H$ is globally invertible (in general, the implicit inverse function theorem only locally guarantees the inverse function) and we can define a quasi-arithmetic center of a point set $\mathcal{P}=\{{\theta }_1,\dots ,{\theta }_n\}$ weighted by w as follows:

Definition 1

(Quasi-arithmetic center). Let $H=\nabla F$ be the gradient of a strictly convex or concave differentiable real-valued function F of Legendre type. The quasi-arithmetic center $c_H({\theta }_1,\dots ,{\theta }_n;w)$ is defined by

$$c_H({\theta }_1,\dots ,{\theta }_n;w)=H^{-1}\left(\sum _{i=1}^n{w}_{i}H\left({\theta }_i\right)\right).$$

This definition generalizes the scalar quasi-arithmetic means [35] for univariate functions h, which are continuous and strictly monotone. Quasi-arithmetic means (QAMs) are also called f means or Kolmogorov–Nagumo means. Let $m_{\nabla F}({\theta }_1,{\theta }_2)=c_{\nabla F}({\theta }_1,{\theta }_2;{\displaystyle \frac{1}{2}},{\displaystyle \frac{1}{2}})$. Notice that $A({\theta }_1,{\theta }_2)=\nabla F^{*}\left(m_{\nabla F^{*}}({\eta }_1,{\eta }_2)\right)$ and $A({\eta }_1,{\eta }_2)=\nabla F\left(m_{\nabla F}({\theta }_1,{\theta }_2)\right)$. That is, the arithmetic mean in a primal representation amounts to a QAM in the dual representation.

Thus, we can solve for ${\theta }_S$ by setting the gradient of $L\left(\theta \right)=B_F(\overline{\theta }:\theta )+B_F(\theta :\underline{\theta })$ to zero. In general, no closed-form formula is known for the symmetrized Bregman centroids, and a numerical approximation method was reported in [28]. To circumvent the lack of a closed-form formula of symmetrized Bregman centroids for clustering, Nock et al. [36] proposed a mixed Bregman clustering where each cluster has two representative dual Bregman centroids $\overline{\theta }={\sum }_i{w}_i{\theta }_i$ (right Bregman centroid) and $\underline{\theta }={(\nabla F)}^{-1}({\sum }_i{w}_i\nabla F\left({\theta }_i\right))$ (left Bregman centroid), and the dissimilarity measure is a mixed Bregman divergence defined by

$${\Delta }_F({\theta }_1:\theta :{\theta }_2):={\displaystyle \frac{1}{2}}{B}_F({\theta }_1:\theta )+{\displaystyle \frac{1}{2}}{B}_F(\theta :{\theta }_2).$$

Notice that minimizing Equation (3) amounts to minimizing the mixed Bregman divergence:

$$\underset{\theta }{\min }{\Delta }_F(\overline{\theta }:\theta :\underline{\theta }).$$

By using the dual parameterization $\eta =\nabla F\left(\theta \right)$ (with dual domain $H=\{\nabla F(\theta ):\theta \in \Theta \}$) and the dual Bregman divergence $B_{F^{*}}({\eta }_1:{\eta }_2)=F^{*}\left({\eta }_1\right)-F^{*}\left({\eta }_2\right)-{\langle {\eta }_1-{\eta }_2,\nabla F\rangle }^{*}\left({\eta }_1\right)=B_F({\theta }_2:{\eta }_1)$, we have

$$\begin{array}{ccc}\hfill {\theta }_S& :=& {\displaystyle \arg \underset{\theta \in \Theta }{\min }\sum _i{w}_i{S}_F(\theta ,{\theta }_i),}\hfill \\ \hfill {\eta }_S& =& {\displaystyle \arg \underset{\eta \in H}{\min }\sum _i{w}_i{S}_{F^{*}}(\eta ,{\eta }_i),}\hfill \\ \hfill & \equiv & {\displaystyle \arg \underset{\eta \in H}{\min }{B}_{F^{*}}(\eta :\nabla F\left(\overline{\theta }\right))+B_{F^{*}}(\nabla F\left(\underline{\theta }\right):\eta ).}\hfill \end{array}$$

(4)

Since $\nabla F\left(\overline{\theta }\right)={(\nabla F^{*})}^{-1}({\sum }_i{w}_i\nabla F^{*}\left({\eta }_i\right))=\underline{\eta }$ and $\nabla F\left(\underline{\theta }\right)=\nabla F\left({(\nabla F)}^{-1}{\sum }_i{w}_i{\eta }_i\right)=\overline{\eta }$, we obtain the dual equivalent optimization problem:

$${\theta }_S=\nabla F^{*}\left({\eta }_S\right)=\arg \underset{\theta \in \Theta }{\min }{B}_F(\overline{\theta }:\theta )+B_F(\theta :\underline{\theta }),$$

or

$${\eta }_S=\nabla F\left({\theta }_S\right)=\arg \underset{\eta \in H}{\min }{B}_{F^{*}}(\overline{\eta }:\theta )+B_{F^{*}}(\eta :\underline{\eta }).$$

However, a remarkable special case is the family of multivariate normal distributions centered at the origin for which the Jeffreys centroid was reported in closed form in [29]. Let ${\mathcal{N}}_0=\{p_{\sum }:\sum \in {\mathrm{Sym}}^{++}(\mathbb{R},d)\}$ be the exponential family with sufficient statistics $t\left(x\right)=-{\displaystyle \frac{1}{2}}(x,x{x}^{\top })$, natural parameter $\theta ={\sum }^{-1}$ (the precision matrix) where the covariance matrix belongs to the cone ${\mathrm{Sym}}^{++}(\mathbb{R},d)$ of symmetric positive-definite matrices, inner product $\langle X,Y\rangle =\mathrm{tr}\left(XY\right)$, and $F\left(\theta \right)=-{\displaystyle \frac{1}{2}}\log \det \left(\theta \right)$. In that case, the Jeffreys divergence amounts to a symmetrized Bregman log-det (ld) divergence between the corresponding natural parameters:

$$D_J(p_{\sum },p_{{\sum }^{\prime }})={\displaystyle \frac{1}{2}}\mathrm{tr}(({{\sum }^{\prime }}^{-1}-{\sum }^{-1})(\sum -{\sum }^{\prime }))=:{\displaystyle \frac{1}{2}}{S}_{\mathrm{ld}}({\sum }^{-1},{{\sum }^{\prime }}^{-1}).$$

Using the standard covariance matrix parameterization $\sum$, we can further express the Jeffreys divergence between two multivariate normal distributions $p_{\sum }$ and $p_{{\sum }^{\prime }}$ as

$$D_J(p_{\sum },p_{{\sum }^{\prime }})=\sum _{i=1}^d{\left(\sqrt{{\lambda }_i}-{\displaystyle \frac{1}{\sqrt{{\lambda }_i}}}\right)}^2,$$

where ${\lambda }_i$s are the eigenvalues of ${\sum }^{-1}{\sum }^{\prime }$. The symmetrized log-det divergence $S_{\mathrm{ld}}$ is also called the symmetrized Stein loss [37,38]. When $d=1$, this divergence is the symmetrized Itakura–Saito divergence also called the COSH distance [28]. The Jeffreys centroid can be characterized using the Fisher–Rao geometry [39] of ${\mathcal{N}}_0$ as the Fisher–Rao geodesic midpoint of the sided Kullback–Leibler centroids as follows:

Theorem 2

([29]). The Jeffreys centroid C of a set of n centered multivariate normal distributions $\mathcal{P}=\{p_{{\sum }_1},\dots ,p_{{\sum }_n}\}$ weighted with $w_i\in {\Delta }_n$ amounts to the symmetrized log-det Bregman centroid for the corresponding weighted set of positive-definite precision matrices ${\mathcal{P}}_{\theta }=\{P_1={\sum }_1^{-1},\dots ,P_n={\sum }_n^{-1}\}$. The symmetrized log-det Bregman barycenter C is the Riemannian geodesic midpoint $A\#H$ of the arithmetic barycenter $A={\sum }_{i=1}^n{w}_i{P}_i$ and harmonic barycenter $H={\left({\sum }_{i=1}^n{w}_i{P}_i^{-1}\right)}^{-1}$ where $X\#Y:=X^{{\displaystyle \frac{1}{2}}}{\left(X^{-{\displaystyle \frac{1}{2}}}Y{X}^{-{\displaystyle \frac{1}{2}}}\right)}^{{\displaystyle \frac{1}{2}}}{X}^{{\displaystyle \frac{1}{2}}}$ is the matrix geometric mean [40] $G(X,Y)=X\#Y$:

$$C=(\sum _{i=1}^n{w}_i{P}_i)\#{\left(\sum _{i=1}^n{w}_i{P}_i^{-1}\right)}^{-1}.$$

(5)

Since the proof of this result mentioned in [29] was omitted in [29], we report a proof involving matrix analysis in full detail in Appendix B.

Next, we shall define two types of centers for sets of densities of a prescribed exponential family based on two different interpretations of Equation (5). We call them centers and not centroids because those points are defined by a generic structural formula instead of solutions of minimization problems of average divergences of Equation (1).

2.3. The Jeffreys–Fisher–Rao Center

Since an exponential family $\mathcal{E}=\left\{p_{\theta }\left(x\right)\right\}$ induces the Riemannian manifold $(\mathcal{M},g)$ with the Fisher metric g expressed in the $\theta$-parameterization by the Fisher information matrix ${\nabla }^{2}F\left(\theta \right)$ and Fisher–Rao geodesics $\gamma (p,q,t)$ defined with respect to the Levi-Civita connection $\overline{\nabla }$ (induced by g), we shall define the Jeffreys–Fisher–Rao center on $\mathcal{M}$ using the Fisher–Rao geodesics as follows:

Definition 2

(Jeffreys–Fisher–Rao (JFR) center). The Jeffreys–Fisher–Rao center ${\theta }_{\mathrm{JFR}}$ of a set $\{p_{{\theta }_1},\dots ,p_{{\theta }_n}\}$ of weighted densities by $w\in {\Delta }_n$ is defined as the Fisher–Rao midpoint of the sided Kullback–Leibler centroids $\overline{\theta }={\sum }_i{w}_i{\theta }_i$ and $\underline{\theta }={(\nabla F)}^{-1}({\sum }_i{w}_i\nabla F\left({\theta }_i\right))$:

$${\theta }_{\mathrm{JFR}}=\overline{\theta }\#\underline{\theta },$$

(6)

where $p\#q=\gamma \left(p,q,{\displaystyle \frac{1}{2}}\right)$.

Equation (6) is a generalization of Equation (5); therefore, the JFR center matches the Jeffreys centroid for same-mean multivariate normal distributions (Theorem 2).

Let ${\mathcal{P}}_{\theta }=\{{\theta }_1,\dots ,{\theta }_n\}$ and ${\mathcal{P}}_{\eta }=\{{\eta }_1,\dots ,{\eta }_n\}$, where $\eta =\nabla F\left(\theta \right)$ and $\theta =\nabla F^{*}\left(\eta \right)$. Denote by ${\mathrm{JFR}}_F({\mathcal{P}}_{\theta };w)$ the JFR center of $\theta$-coordinate ${\theta }_{\mathrm{JFR}}$. Then, ${\mathrm{JFR}}_{F^{*}}({\mathcal{P}}_{\eta };w)$ = $\nabla F\left({\theta }_{\mathrm{JFR}}\right):={\eta }_{\mathrm{JFR}}$.

2.4. Gauss–Bregman Inductive Center

Another remarkable property of the Jeffreys centroid for a set $\{p_{\mu ,{\sum }_1},\dots ,p_{\mu ,{\sum }_n}\}$ of same-mean multivariate normal distributions weighted by $w\in {\Delta }_n$ with arithmetic and harmonic means $A={\sum }_{i=1}^n{w}_i{\sum }_i^{-1}$ and $H={\left({\sum }_{i=1}^n{w}_i{\sum }_i\right)}^{-1}$ on the precision matrices ${\sum }_1^{-1},\dots ,{\sum }_n^{-1}$, respectively, is that we have the following invariance of the Jeffreys centroid (see Lemma 17.4.4 of [29]):

$$G(A,H)=G\left({\displaystyle \frac{A+H}{2}},2{(A^{-1}+H^{-1})}^{-1}\right).$$

(7)

Nakamura [41] defined the following double sequence scheme converging to the matrix geometry mean $G(P,Q)$ for any two symmetric positive-definite matrices P and Q:

$$\begin{array}{ccc}\hfill P_{t+1}& =& A(P_t,Q_t):={\displaystyle \frac{P_t+Q_t}{2}},\hfill \\ \hfill Q_{t+1}& =& H(P_t,Q_t):=2{(P_t^{-1}+Q_t^{-1})}^{-1},\hfill \end{array}$$

initialized with $P_0=P$ and $Q_0=Q$. We have ${\lim }_{t\to \infty }{P}_t={\lim }_{t\to \infty }{Q}_t=P\#Q=G(P,Q)$. Let $P_{\infty }={\lim }_{t\to \infty }{P}_t$ and $Q_{\infty }={\lim }_{t\to \infty }{Q}_t$. That is, the geometric matrix mean can be obtained as the limits of a double sequence of means. We can thus approximate $G(P,Q)$ by stopping the double sequence after T iterations to obtain

$$G^{\left(T\right)}(P,Q)=A(P_T,Q_T)\approx G(P,Q).$$

Notice that we can recover those iterations from the invariance property of Equation (7): Indeed, we have

$$G(P_0,Q_0)=G(\underset{=:P_1}{\underbrace{A(P_0,Q_0)}},\underset{=:Q_1}{\underbrace{H(P_0,Q_0)}})=G(\underset{=:P_2}{\underbrace{A(P_1,Q_1)}},\underset{=:Q_2}{\underbrace{H(P_1,Q_1)}})=\dots ,$$

(8)

and $\parallel P_t-Q_t\parallel =\sqrt{\mathrm{tr}\left((P_t-Q_t)(P_t-Q_t)\right)}$ decreases [41] as the number of iterations t increases. Thus, by induction, $G(P_0,Q_0)=G(P_{\infty },Q_{\infty })$ with $P_{\infty }=Q_{\infty }$. Since $G(X,X)=X$ (means are reflexive), it follows that $G(P_0,Q_0)=P_{\infty }=Q_{\infty }$. It is proved in [41] that the convergence rate of the sequence of double iterations is quadratic. This type of mean has been called an inductive mean [30,42] (or compound mean [43]) and originated from the Gauss arithmetic–geometric mean [44].

Our second interpretation of the geometric matrix mean of Equation (5) is to consider it as an inductive mean [30] and to generalize this double sequence process to pairs/sets of densities of an exponential family as follows:

Definition 3

(Gauss–Bregman $(A,\nabla F)$ center). Let $\mathcal{P}=\{p_{{\theta }_1},\dots ,p_{{\theta }_n}\}$ be a set of n distributions of an exponential family with the cumulant function $F\left(\theta \right)$ weighted by a vector $w\in {\Delta }_n$. Then, the Gauss–Bregman inductive center ${\theta }_{\mathrm{GB}}$ is defined as the limit of the double sequence:

$$\begin{array}{ccc}\hfill {\overline{\theta }}_{t+1}& =& A({\overline{\theta }}_t,{\underline{\theta }}_t):={\displaystyle \frac{{\overline{\theta }}_t+{\underline{\theta }}_t}{2}},\hfill \\ \hfill {\underline{\theta }}_{t+1}& =& m_{\nabla F}({\overline{\theta }}_t,{\underline{\theta }}_t):={(\nabla F)}^{-1}\left({\displaystyle \frac{\nabla F\left({\overline{\theta }}_t\right)+\nabla F\left({\underline{\theta }}_t\right)}{2}}\right),\hfill \end{array}$$

initialized with ${\overline{\theta }}_0=\overline{\theta }={\sum }_{i=1}^n{w}_i{\theta }_i$ (right Bregman centroid) and ${\underline{\theta }}_0=\underline{\theta }=\nabla F^{-1}\left({\sum }_{i=1}^n{w}_i\nabla F\left({\theta }_i\right)\right)$ (left Bregman centroid). That is, we have

$${\theta }_{\mathrm{GB}}=\underset{t\to \infty }{\lim }{\overline{\theta }}_t=\underset{t\to \infty }{\lim }{\underline{\theta }}_t.$$

(9)

Let ${\theta }_{\mathrm{GB}}={\mathrm{GB}}_F(\overline{\theta },\underline{\theta })$. Then, we have ${\eta }_{\mathrm{GB}}={\mathrm{GB}}_{F^{*}}(\overline{\eta },\overline{\theta })=\nabla F\left({\theta }_{\mathrm{GB}}\right)$. The Gauss–Bregman center $c_{\mathrm{GB}}$ has $\theta$-coordinates ${\theta }_{G}B$ and $\eta$-coordinates ${\eta }_{\mathrm{GB}}$.

Algorithm 1 describes the approximation of the Gauss–Bregman inductive center by stopping the double sequence when the iterated centers are close enough to each other. We shall prove the matching convergence of those ${\overline{\theta }}_t$ and ${\underline{\theta }}_t$ sequences under separability conditions in Section 2.4.

Algorithm 1: Gauss–Bregman inductive center.

For example, the Gauss–Bregman center of two categorical distributions $p=(p_1,\dots ,p_d)$ and $p^{\prime }=(p_1^{\prime },\dots ,p_d^{\prime })$ on a sample space $\mathcal{X}$ of d elements is obtained for the cumulant function $F\left(\theta \right)=\log (1+{\sum }_{i=1}^{d-1}{e}^{{\theta }_i})$ with gradient $\nabla F\left(\theta \right)={\left[{\eta }_i={\displaystyle \frac{e^{{\theta }_i}}{1+{\sum }_{j=1}^{d-1}{e}^{{\theta }_j}}}\right]}_i$ where $\theta =({\theta }_1=\log {\displaystyle \frac{p_1}{p_d}},\dots ,{\theta }_{d-1}=\log {\displaystyle \frac{p_{d-1}}{p_d}})$ is the natural parameter. The reciprocal gradient is ${(\nabla F)}^{-1}\left(\eta \right)={\left[\log {\displaystyle \frac{{\eta }_i}{1-{\sum }_{j=1}^{d-1}{\eta }_j}}\right]}_i$.

We may also compute the Gauss–Bregman center of two categorical distributions $\mathrm{Cat}\left(p\right)$ and $\mathrm{Cat}\left(p^{\prime }\right)$ using iterations of arithmetic means $a_t$ and geometric normalized means $g_t$:

$$\begin{array}{ccc}\hfill a_{t+1}^i& =& A(a_t^i,g_t^i):={\displaystyle \frac{a_t^i+g_t^i}{2}},\forall i\in \{1,\dots ,d\}\hfill \\ \hfill u_{t+1}^i& =& \sqrt{a_t^i{g}_t^i},\forall i\in \{1,\dots ,d\},\hfill \\ \hfill g_{t+1}^i& =& {\displaystyle \frac{u_{t+1}^i}{{\sum }_{j=1}^d{u}_{t+1}^j}},\forall i\in \{1,\dots ,d\},\hfill \end{array}$$

where the $u_t$s are unnormalized geometric means and the $g_t$ represents normalized geometric means. We initialize the sequence with $a_0=p$ and $g_0=p^{\prime }$, and the Gauss–Bregman center is obtained in the limit $m_{\mathrm{GB}}^{\mathrm{Cat}}(p,p^{\prime })={\lim }_{t\to \infty }{a}_t={\lim }_{t\to \infty }{g}_t$. See Algorithm 2.

The Jeffreys centroid of a set of centered multivariate normal distributions is the Gauss–Bregman center obtained for the generator $F\left(\theta \right)=-{\displaystyle \frac{1}{2}}\log \det \left(\theta \right)$, the cumulant function of the exponential family of centered normal distributions.

Algorithm 2: Gauss–Bregman inductive center for sets of categorical distributions.

Figure 2 displays the arithmetic and normalized geometric and numerical Jeffreys, Jeffreys–Fisher–Rao, and Gauss–Bregman centroids/centers for a set of 32 trinomial distributions. We may consider normalized intensity histograms of images (modeled as multinomials with one trial) quantized with $d=256$ bins; that is, a normalized histogram with d bins is interpreted as a point in ${\Delta }_d$ and visualized as a polyline with $d-1$ line segments. Figure 3 (left) displays the various centroids and centers obtained for an input set consisting of two histograms (the commonly used Barbara and Lena images, which have been used in [31]). Notice that the JFR center (purple) and GB center (yellow) are close to the numerical Jeffreys centroid (green). We also provide a close-up window in Figure 3 (right).

Notice that we can experimentally check the quality of the approximation of the Gauss–Bregman center to the Jeffreys centroid by defining the symmetrized Bregman centroid energy:

$$E_F\left(\theta \right):=\langle \theta -\overline{\theta },\nabla F\left(\theta \right)\rangle -\langle \theta ,\nabla F\left(\overline{\theta }\right)\rangle ,$$

and checking that $\nabla E_F\left(\theta \right)$:

$$\begin{array}{ccc}\hfill \forall i,& & {\partial }_i\left(\sum _{i=1}^d({\theta }_i-{\overline{\theta }}_i){\partial }_{i}F\left(\theta \right)-{\theta }_i{\partial }_{i}F\left(\overline{\theta }\right)\right)=0,\hfill \end{array}$$

(10)

$$\begin{array}{ccc}& & {\partial }_{i}F\left(\theta \right)+({\theta }_i-{\overline{\theta }}_i){\partial }_i^{2}F\left(\theta \right)-{\partial }_{i}F\left(\overline{\theta }\right)+\left(\sum _{j\ne i}({\theta }_j-{\overline{\theta }}_j){\partial }_i{\partial }_{j}F\left(\theta \right)-{\partial }_i{\theta }_j{\partial }_{j}F\left(\overline{\theta }\right)\right)=0\hfill \end{array}$$

(11)

is close to zero, where ${\partial }_l:={\displaystyle \frac{\partial }{\partial {\theta }_l}}$.

Next, we study these two new types of centers and how well they approximate the Jeffreys centroid.

3. Gauss–Bregman Inductive Centers: Convergence Analysis and Properties

Let $F\left(\theta \right)$ be a strictly convex and differentiable real-valued function of Legendre type [45] defined on an open parameter space $\Theta$. Then, the gradient map $\theta \mapsto \eta \left(\theta \right)=\nabla F\left(\theta \right)$ is a bijection with the reciprocal inverse function $\eta \mapsto \theta \left(\eta \right)=\nabla F^{*}\left(\eta \right)={(\nabla F)}^{-1}\left(\eta \right)$ where $F^{*}\left(\eta \right)=\langle \eta ,\nabla F^{-1}\left(\eta \right)\rangle -F(\nabla F^{-1}\left(\eta \right))$ is the Legendre–Fenchel convex conjugate. For example, we may consider the cumulant functions of regular exponential families.

We define the Gauss–Bregman center ${\theta }_{\mathrm{GB}}$ of a set $\{{\theta }_1,\dots ,{\theta }_n\}$ weighted by $w\in {\Delta }_n$ as the limits of the sequences ${\overline{\theta }}_1,\dots$ and ${\underline{\theta }}_1,\dots$ defined by

$$\begin{array}{ccc}\hfill {\overline{\theta }}_{t+1}& =& A({\overline{\theta }}_t,{\underline{\theta }}_t):={\displaystyle \frac{{\overline{\theta }}_t+{\underline{\theta }}_t}{2}},\hfill \end{array}$$

(12)

$$\begin{array}{ccc}\hfill {\underline{\theta }}_{t+1}& =& m_{\nabla F}({\overline{\theta }}_t,{\underline{\theta }}_t):={(\nabla F)}^{-1}\left({\displaystyle \frac{\nabla F\left({\overline{\theta }}_t\right)+\nabla F\left({\underline{\theta }}_t\right)}{2}}\right),\hfill \end{array}$$

(13)

initialized with ${\overline{\theta }}_0=\overline{\theta }={\sum }_{i=1}^n{w}_i{\theta }_i$ and ${\underline{\theta }}_0=\underline{\theta }=\nabla F^{-1}\left({\sum }_{i=1}^n{w}_i\nabla F\left({\theta }_i\right)\right)$. That is, we have

$${\theta }_{\mathrm{GB}}=\underset{n\to \infty }{\lim }{\overline{\theta }}_t=\underset{n\to \infty }{\lim }{\underline{\theta }}_t.$$

Such a center has been called an inductive mean by Sturm [30]. See [42] for an overview of inductive means. Figure 4 geometrically illustrates the double sequence iterations converging to the Gauss–Bregman mean.

Theorem 3.

The Gauss–Bregman $(A,\nabla F)$ center with respect to a Legendre type function $F\left(\theta \right)$ is well defined (i.e., the double sequence converges) for separable Bregman generators.

Proof. 

We need to prove the convergence of $\left\{{\overline{\theta }}_t\right\}$ and $\left\{{\underline{\theta }}_t\right\}$ to the same finite limit. When $F\left(\theta \right)$ is univariate, the convergence of the inductive centers was reported in [43]. We need to prove that the double iterations of Equations (12) and (13) converge.

Let us consider the following cases:

• When the dimension is one, the quasi-arithmetic mean $m_{f^{\prime }}$ for f, a strictly convex and differentiable function, lies between the minimal and maximal argument (i.e., this is the definition of a strict mean):

  $$\min \{{\theta }_1,{\theta }_2\}\le m_{f^{\prime }}({\theta }_1,{\theta }_2)\le \max \{{\theta }_1,{\theta }_2\}.$$
  Thus, we have

  $$|{\overline{\theta }}_{t+1}-{\underline{\theta }}_{t+1}|\le {\displaystyle \frac{1}{2}}|{\overline{\theta }}_t-{\underline{\theta }}_t|,$$

  and it follows that $|{\overline{\theta }}_{t+1}-{\underline{\theta }}_{t+1}|\le {\displaystyle \frac{1}{2^t}}|{\overline{\theta }}_0-{\underline{\theta }}_0|$. Thus, we have quadratic convergence of scalar $(A,f^{\prime })$ means. See Figure 5.
• When $F\left(\theta \right)$ is multivariate and separable, i.e., $F\left(\theta \right)={\sum }_{i=1}^d{f}_i\left({\theta }^i\right)$ where $\theta =({\theta }^1,\dots ,{\theta }^d)$ are the components of $\theta \in {\mathbb{R}}^d$ and the $f_i$s are scalar strictly convex and differentiable functions, we can apply case 1 dimension-wise to obtain the quadratic convergence.
• Otherwise, we consider the multivariate quasi-arithmetic center $m_{\nabla F}(\theta ,{\theta }^{\prime })$ with the uniform weight vector $w=({\displaystyle \frac{1}{2}},{\displaystyle \frac{1}{2}})$. One problem we face is that the quasi-arithmetic center $m_{\nabla F}(\theta ,{\theta }^{\prime })$ for $\theta \ne {\theta }^{\prime }$ may lie outside the open bounding box of ${\mathbb{R}}^d$ with diagonal corners $\theta$ and ${\theta }^{\prime }$:

  $${\theta }_m=(\min \{{\theta }^1,{{\theta }^{\prime }}^1\},\dots ,\min \{{\theta }^d,{{\theta }^{\prime }}^d\}),{\theta }_M=(\max \{{\theta }^1,{{\theta }^{\prime }}^1\},\dots ,\max \{{\theta }^d,{{\theta }^{\prime }}^d\}).$$
  Indeed, in the 2D case, we may consider $\theta =(x,y)$ and ${\theta }^{\prime }=(x^{\prime },y)$. Clearly, the open bounding box is empty, and the midpoint $m_{\nabla F}(\theta ,{\theta }^{\prime })$ lies outside this box. Yet, we are interested in the convergence rate when ${\theta }^{\prime }\approx \theta$.
  In general, we shall measure the difference between two iterations by the squared norm distance induced by the inner product:

  $$\parallel A(\theta ,{\theta }^{\prime })-m_{\nabla F}(\theta ,{\theta }^{\prime }){\parallel }^2=\langle A(\theta ,{\theta }^{\prime })-m_{\nabla F}(\theta ,{\theta }^{\prime }),A(\theta ,{\theta }^{\prime })-m_{\nabla F}(\theta ,{\theta }^{\prime })\rangle .$$

 □

Let $m_F^{\mathrm{GB}}({\theta }_1,{\theta }_2)$ denote the Gauss–Bregman center of ${\theta }_1$ and ${\theta }_2$, $A({\theta }_1,{\theta }_2)={\displaystyle \frac{{\theta }_1+{\theta }_2}{2}}$ the arithmetic mean, and $m_{\nabla F}({\theta }_1,{\theta }_2)={(\nabla F)}^{-1}\left({\displaystyle \frac{\nabla F\left({\theta }_1\right)+\nabla F\left({\theta }_2\right)}{2}}\right)$ the quasi-arithmetic center.

By construction, the Gauss–Bregman center enjoys the following invariance property generalizing Lemma 17.4.4 of [29] in the case of the log det generator:

Property 1.

We have $m_F^{\mathrm{GB}}({\theta }_1,{\theta }_2)=m_F^{\mathrm{GB}}\left(A({\theta }_1,{\theta }_2),m_{\nabla F}({\theta }_1,{\theta }_2)\right)$.

Proof. 

Similar to the cascaded inequalities of Equation (8), we have

$$m_F^{\mathrm{GB}}({\theta }_1,{\theta }_2)=m_{\mathrm{GB}}^F(\underset{=:{\theta }_1^{\left(1\right)}}{\underbrace{A({\theta }_1,{\theta }_2)}},\underset{={\theta }_2^{\left(1\right)}}{\underbrace{m_{\nabla F}({\theta }_1,{\theta }_2)}})=\dots$$

(14)

In the limit $t\to \infty$, we have $m_F^{\mathrm{GB}}({\theta }_1,{\theta }_2)=m_F^{\mathrm{GB}}({\theta }_1^{(\infty )},{\theta }_2^{(\infty )})=m_F^{\mathrm{GB}}({\theta }_1^{(\infty -1)},{\theta }_2^{(\infty -1)})=\dots$ Since $\infty -1=\infty$, we obtain the desired invariance property:

$$m_F^{\mathrm{GB}}({\theta }_1,{\theta }_2)=m_F^{\mathrm{GB}}\left(A({\theta }_1,{\theta }_2),m_{\nabla F}({\theta }_1,{\theta }_2)\right).$$

 □

Note that when $F\left(\theta \right)$ is univariate, the Gauss–Bregman mean $m_F^{\mathrm{GB}}({\theta }_1,{\theta }_2)$ converges at a quadratic rate [43]. In particular, when $F\left(\theta \right)=-\log \theta$ (Burg negentropy), we have $F^{\prime }\left(\theta \right)=-{\displaystyle \frac{1}{\theta }}$ ($m_{F^{\prime }}$ is the harmonic mean) and the Gauss–Bregman mean is the arithmetic–harmonic mean (AHM) which converges to the geometric mean, a simple closed-form formula. Notice that the geometric mean $g=\sqrt{xy}$ of two scalars $x>0$ and $y>0$ can be expressed using the arithmetic mean $a={\displaystyle \frac{x+y}{2}}$ and the harmonic mean $h={\displaystyle \frac{2xy}{x+y}}$: $g=\sqrt{ah}$. But when $F\left(\theta \right)=\theta \log \theta -\theta$ (Shannon negentropy), the Gauss–Bregman mean $m_F^{\mathrm{GB}}({\theta }_1,{\theta }_2)$ coincides with the Gauss arithmetic–geometric mean [44] (AGM) since $F^{\prime }\left(\theta \right)=\log \theta$ and $m_{F^{\prime }}({\theta }_1,{\theta }_2)=\sqrt{{\theta }_1{\theta }_2}$, the geometric mean. Thus, $m_F^{\mathrm{GB}}({\theta }_1,{\theta }_2)$ is related to the elliptic integral K of the first type [44]: there is no closed-form formula for the AGM in terms of elementary functions as this induced mean is related to the complete elliptic integral of the first kind $K(·)$:

$$\mathrm{AGM}(x,y)={\displaystyle \frac{\pi }{4}}{\displaystyle \frac{x+y}{K\left(\frac{x-y}{x+y}\right)}},$$

(15)

where $K\left(u\right)={\int }_0^{{\displaystyle \frac{\pi }{2}}}{\displaystyle \frac{\mathrm{d}\theta }{\sqrt{1-u^2{\sin }^2\left(\theta \right)}}}$ is the elliptic integral. Thus, it is difficult, in general, to report a closed-form formula for the inductive Gauss–Bregman means even for univariate generators $F\left(\theta \right)$.

The Jeffreys centroid of $x>0$ and $y>0$ with respect to the scalar Jeffreys divergence $D_J(p,q)=(p-q)\log {\displaystyle \frac{p}{q}}$ admits a closed-form solution [31]:

$$c={\displaystyle \frac{a}{W_0\left(\frac{a}{g}e\right)}}$$

(16)

where $a={\displaystyle \frac{x+y}{2}}$ and $g=\sqrt{xy}$ and $W_0$ is the principal branch of the Lambert W function [33]. This example shows that the Gauss–Bregman center does not coincide with the Jeffreys centroid in general (e.g., compare Equation (15) with Equation (16)).

4. Jeffreys–Fisher–Rao Centers: Generic Structural Formula and Some Closed-Form Formula

4.1. Jeffreys–Fisher–Rao Center for Uni-Parametric Statistical Models

Consider a set $\mathcal{P}=\{p_{{\theta }_1},\dots ,p_{{\theta }_n}\}$ of n parametric distributions where $\theta \in \Theta \subset \mathbb{R}$ is a scalar parameter. Let $w=(w_1,\dots ,w_n)\in {\Delta }_n$ be a weight vector on $\mathcal{P}$ such that the weight of $p_{{\theta }_i}$ is $w_i$. The distributions $p_{\theta }$s may not necessarily belong to an exponential family (e.g., the Cauchy scale family). The Fisher–Rao geometry [46,47] of the parametric family of distributions $\mathcal{F}=\{p_{\theta }:\theta \in \Theta \}$ (the statistical model) can be modeled as a Riemannian manifold with the Fisher metric $g\left(\theta \right)=I\left(\theta \right)$ defined by the Fisher information $I\left(\theta \right)=E_{\theta }\left[{\left(\log p_{\theta }\left(x\right)\right)}^2\right]=-E_{\theta }\left[{\nabla }^2\log p_{\theta }\left(x\right)\right]$. When $\mathcal{F}$ is an exponential family with the cumulant function $F\left(\theta \right)$, we have $I\left(\theta \right)=F^{\prime \prime }\left(\theta \right)$.

The underlying geometry of $(\mathcal{F},g(\theta )=I(\theta \left)\right)$ is Euclidean after a change in variable $\eta \left(\theta \right)=\sqrt{I\left(\theta \right)}$ since we can write the metric tensor as follows:

$$g\left(\theta \right)=\sqrt{I\left(\theta \right)}\times \underset{=g_{\mathrm{Euclidean}}}{\underbrace{1}},\times \sqrt{I\left(\theta \right)}.$$

Thus, the Riemannian Fisher–Rao distance is the Euclidean distance expressed in the $h\left(\theta \right)$-coordinate system with $h\left(\theta \right)={\int }_{{\theta }_0}^{\theta }\sqrt{I\left(u\right)}\mathrm{d}u$, and we have the Fisher–Rao distance given by

$$\rho (p_{{\theta }_1},p_{{\theta }_2})=|h\left({\theta }_1\right)-h\left({\theta }_2\right)|.$$

When $\mathcal{F}$ is an exponential family with the cumulant function $f\left(\theta \right)$, we have $I\left(u\right)=f^{″}\left(u\right)$.

We summarize the result on the JFR center in the following theorem:

Theorem 4

(Jeffreys–Fisher–Rao centroid in uni-order exponential families). The Jeffreys–Fisher–Rao centroid ${\theta }_S$ of n densities $p_{{\theta }_1},\dots ,p_{{\theta }_n}$ of an exponential family of order one with the log-normalizer $f\left(\theta \right)$ for $\theta \in \Theta$, the natural parameter space, and weight vector $w\in {\Delta }_n$ is

$${\theta }_S=m_h(\overline{\theta },\underline{\theta }),$$

(17)

where $m_h(\overline{\theta },\underline{\theta })=h^{-1}\left({\displaystyle \frac{h\left(\overline{\theta }\right)+h\left(\underline{\theta }\right)}{2}}\right)$ is the quasi-arithmetic mean [35] of the dual left and right KL centroids $\overline{\theta }={\sum }_{i=1}^n{w}_i{\theta }_i={\theta }_R$ and $\underline{\theta }={\left(f^{\prime }\right)}^{-1}\left({\sum }_{i=1}^n{w}_i{f}^{\prime }\left({\theta }_i\right)\right)$ with respect to the scalar monotone function $h={\int }_{{\theta }_0}^{\theta }\sqrt{f^{″}\left(u\right)}\mathrm{d}u$ for any $\theta \in \Theta$.

Proof. 

Since the Fisher information is $I\left(\theta \right)=f^{″}\left(\theta \right)$, we have $h\left(\theta \right)={\int }_{{\theta }_0}^{\theta }\sqrt{f^{″}\left(u\right)}\mathrm{d}u$. The Riemannian center of mass [48] minimizes

$${\theta }_S=\arg \underset{\theta }{\min }\sum _{i=1}^n{w}_i{\rho }^2({\theta }_i,\theta ).$$

But in the h-parameterization, the Riemannian centroid, amounts to a Euclidean center of mass/centroid in the h-Cartesian coordinate system:

$$h\left({\theta }_S\right)=\sum _{i=1}^n{w}_{i}h\left({\theta }_i\right).$$

Therefore, we have ${\theta }_S=h^{-1}\left({\sum }_i{w}_{i}h\left({\theta }_i\right)\right)=:m_h({\theta }_1,\dots ,{\theta }_n;w_1,\dots ,w_n)$, a weighted quasi-arithmetic mean. Since the Jeffreys centroid amounts to a symmetrized Bregman centroid of the left and right Bregman centroids [28], $\underline{\theta }=m_{f^{\prime }}({\theta }_1,\dots ,{\theta }_n;w_1,\dots ,w_n)$ and $\overline{\theta }={\sum }_i{w}_i{\theta }_i$. It follows that the Jeffreys–Fisher–Rao center is ${\theta }_{\mathrm{JFR}}=m_h(\overline{\theta },\underline{\theta })$ after using Property 3. □

4.2. Jeffreys–Fisher–Rao Center for Categorical Distributions

Recall from Theorem 1 that the Jeffreys centroid $c=(c_1,\dots ,c_j,\dots ,c_d)$ of a set of n categorical distributions with parameters arranged in the matrix $\left[p_{i,j}\right]$ is given by

$$c_j\left(\lambda \right)={\displaystyle \frac{a_j}{W_0\left(\frac{a_j}{g_j}{e}^{1+\lambda }\right)}},\forall j\in \{1,\dots ,d\},$$

where $a_j={\sum }_{i=1}^n{w}_i{p}_{i,j}$ and $g_j={\displaystyle \frac{{\prod }_{i=1}^n{p}_{i,j}^{w_i}}{{\sum }_{j=1}^d{\prod }_{i=1}^n{p}_{i,j}^{w_i}}}$ are the components of the weighted arithmetic and normalized geometric means, respectively, and $W_0$ is the principal branch of the Lambert W function [33]. The optimal $\lambda \le 0$ is unique and satisfies $\lambda =-D_{\mathrm{KL}}(c_j\left(\lambda \right):g)$.

Let $c\left(\lambda \right)=(c_1\left(\lambda \right),\dots ,c_d\left(\lambda \right))$. Let $L_J\left(p\right)$ denote the Jeffreys loss function to minimize to find the optimal Jeffreys centroid:

$$L_J\left(p\right)=\sum _{i=1}^n{w}_i{D}_J(p_i,p)$$

(18)

We say that p is a $(1+ϵ)$ approximation of the exact Jeffreys centroid c when we have

$$L_J\left(c\right)\le L_J\left(p\right)\le (1+ϵ)L_J\left(c\right).$$

It was shown in [31] that $\tilde{c}=c\left(0\right)$, called the unnormalized Jeffreys center, yields a $s\left(\lambda \right)-1$ approximation on c where $s\left(\lambda \right)={\sum }_j{c}_j\left(\lambda \right)\le 1$.

Since the Fisher–Rao geodesic midpoints on the categorical Fisher–Rao manifold are known in closed form [49], we give the mathematical expression of the JFR center as follows:

Theorem 5

(JFR centroid of categorical distributions). Let ${\mathcal{P}}_w=\{p_1,\dots ,p_n\}$ be a set of n probability mass functions weighted by $w\in {\Delta }_n$ with $p_i=(p_{i,1},\dots ,p_{i,d})\in {\Delta }_d$ for $i\in \{1,\dots ,n\}$ and $w\in {\Delta }_n$. Then, the JFR barycenter c minimizing is unique and given by the following formula:

$$c_j={\displaystyle \frac{{(\sqrt{a_j}+\sqrt{g_j})}^2}{2(1+{\sum }_{l=1}^d\sqrt{a_j}\sqrt{g_j})}},\forall j\in \{1,\dots ,d\},$$

(19)

where $a=(a_1,\dots ,a_d)={\sum }_{i=1}^n{w}_i{p}_i$ is the weighted arithmetic mean and $g=(g_1,\dots ,g_d)$ is the normalized weighted geometric mean with components $g_j={\displaystyle \frac{{\prod }_{i=1}^n{p}_{i,j}^{w_i}}{{\sum }_{j=1}^d{\prod }_{i=1}^n{p}_{i,j}^{w_i}}}$ for $i\in \{1,\dots ,d\}$.

Notice that the JFR center differs from the Jeffreys centroid, which requires the use of the Lambert W function [33]. However, we noticed that for practical applications, the JFR centroid approximates the Jeffreys centroid well and is much faster to compute (see the experiments in Section 5).

4.3. Jeffreys–Fisher–Rao Center for Multivariate Normal Distributions

Let $\mathcal{P}=\{p_{{\mu }_1,{\sum }_1},\dots ,p_{{\mu }_n,{\sum }_n}\}$ be a set of n probability density functions (PDFs) of d-variate normal distributions weighted by $w\in {\Delta }_n$, where the PDF of a multivariate normal distribution of mean $\mu$ and the covariance matrix $\sum$ is given by

$$p_{\mu ,\sum }={\displaystyle \frac{1}{{\left(2\pi \right)}^{\frac{d}{2}}\sqrt{\det (\sum )}}}\exp \left(-{\displaystyle \frac{1}{2}}{(x-\mu )}^{\top }{\sum }^{-1}(x-\mu )\right).$$

Let ${\lambda }_i=({\mu }_i,{\sum }_i)$ be the ordinary parameterization of normal distributions $p_{{\mu }_i,{\sum }_i}$. The family $\mathcal{F}=\{p_{\mu ,\sum }\left(x\right):\mu \in {\mathbb{R}}^d,\sum \in {\mathrm{Sym}}^{++}(\mathbb{R},d)\}$ of multivariate normal distributions forms an exponential family with the dual natural $\theta$- and moment $\eta$-parameterizations [7] given by

$$\begin{array}{ccc}\hfill \theta \left(\lambda \right)& =& ({\theta }_v,{\theta }_M)=\left({\sum }^{-1}\mu ,{\displaystyle \frac{1}{2}}{\sum }^{-1}\right),\hfill \\ \hfill \eta \left(\lambda \right)& =& \left(\mu ,\mu {\mu }^{\top }+\sum \right),\hfill \end{array}$$

when choosing the sufficient statistic $t\left(x\right)=(x,x{x}^{\top })$. The Jeffreys divergence between two d-variate normal distributions $N({\mu }_1,{\sum }_1)$ and $N({\mu }_2,{\sum }_2)$ is given by the formula

$$D_J(p_{{\mu }_1,{\sum }_1},p_{{\mu }_2,{\sum }_2})=({\mu }_2-{\mu }_1)\top ({\sum }_1^{-1}+{\sum }_2^{-1})({\mu }_2-{\mu }_1)+\mathrm{tr}\left({\sum }_1^{-1}{\sum }_2+{\sum }_2^{-1}{\sum }_1\right)-2d.$$

The left and right Kullback–Leibler barycenters amount to the corresponding right and left Bregman barycenters [28] induced by the cumulant function

$$F\left(\theta \right)=F({\theta }_v,{\theta }_M)={\displaystyle \frac{1}{2}}\left(d\log \pi -\log \det \left({\theta }_M\right)+{\displaystyle \frac{1}{2}}{\theta }_v^{\top }{\theta }_M^{-1}{\theta }_v\right),$$

and the gradient of $F\left(\theta \right)$ defines the dual moment parameter with

$$\eta \left(\theta \right)=\nabla F\left(\theta \right)=\left({\displaystyle \frac{1}{2}}{\theta }_M^{-1}{\theta }_v,{\displaystyle \frac{1}{2}}{\theta }_M^{-1}-{\displaystyle \frac{1}{4}}\left({\theta }_M^{-1}{\theta }_v\right){\left({\theta }_M^{-1}{\theta }_v\right)}^{\top }\right).$$

The reciprocal gradient is given by

$$\theta \left(\eta \right)=\theta ({\eta }_v,{\eta }_M)={(\nabla F)}^{-1}\left(\eta \right)=\left({\theta }_v=-{({\eta }_M+{\eta }_v{\eta }_v^{\top })}^{-1}{\eta }_v,{\theta }_M=-{\displaystyle \frac{1}{2}}{({\eta }_M+{\eta }_v{\eta }_v^{\top })}^{-1}\right).$$

The Gauss–Bregman center is a $(A,m_{\nabla F})$-inductive center, which can be approximated by carrying a prescribed number T of iterations of the Gauss–Bregman double sequence.

Although the Rao distance between two d-variate normal distributions is not available in closed form when $d>1$ [50,51], the Jeffreys–Fisher–Rao center can be computed in closed form. Indeed, the sided Kullback–Leibler centroids of multivariate normal distributions amount to reverse-sided Bregman centroids [28], and the Fisher–Rao geodesic midpoint between two multivariate normal distributions was recently reported in [32]. Appendix C concisely describes the method of Kobayashi [32], which allows one to obtain the Fisher–Rao midpoints of multivariate normal distributions. An implementation of that algorithm is available in the Python software library pyBregMan [52].

Thus, the Jeffreys–Fisher–Rao center is available in closed form:

Theorem 6

(JFR center of MVNs). The Jeffreys–Fisher–Rao center of a finite set of weighted multivariate normal distributions is available in closed form.

Note that the Fisher–Rao distance between normal distributions is invariant under the action of the positive affine group [50], as are the Jeffreys centroid, the JFR center, and the GB center. Figure 6 shows several examples of the JFR and GB centers of two univariate normal distributions. We can observe that those centers are close to each other although they are distinct when the normal distributions do not share the same means and covariance matrices.

Figure 7 shows the various centroids/centers between two bivariate normal distributions displayed as ellipsoids centered as their means. Observe that the inductive Gauss–Bregman center is visually closer than the Jeffreys–Fisher–Rao center to the Jeffreys centroid.

Figure 8 displays the various centroids and centers for pairs of bivariate normal distributions centered at the same mean. Figure 9 shows the centroids and centers for pairs of bivariate normal distributions with the same covariance matrix.

Remark 1.

In general, an exponential family may be characterized equivalently by two convex functions: (1) its log-normalizer $F\left(\theta \right)$ or (2) its partition function $Z\left(\theta \right)=\exp \left(F\right(\theta \left)\right)$, which is log-convex and hence also convex [53]. It has been shown that the Bregman divergence $B_Z$ for $Z=\sqrt{\det \left(\theta \right)}$ (convex) corresponds to the reverse extended Kullback–Leibler divergence between unnormalized PDFs of normal distributions:

$$B_Z({\theta }_1:{\theta }_2)=D_{\mathrm{KL}}^{+}({\tilde{p}}_{\lambda \left({\theta }_2\right)}:{\tilde{p}}_{\lambda \left({\theta }_1\right)}),$$

where ${\tilde{p}}_{\mu ,\sum }=\exp \left(-{\displaystyle \frac{1}{2}}{(x-\mu )}^{\top }{\sum }^{-1}(x-\mu )\right)$ and the extended KLD between two positive measures is given by

$$D_{\mathrm{KL}}^{+}(m_1:m_2)=\int \left(m_1\left(x\right)\log {\displaystyle \frac{m_1\left(x\right)}{m_2\left(x\right)}}+m_2\left(x\right)-m_1\left(x\right)\right)\mathrm{d}\mu \left(x\right).$$

Remark 2.

We may further define yet another center for multivariate normal distributions by considering the Fisher–Rao isometric embedding of the Fisher–Rao d-variate normal manifold $\mathcal{M}=\left\{p_{\mu ,\sum }\right\}$ into the Fisher–Rao $(d+1)$-variate centered manifold ${\mathcal{N}}_0^{+}=\{q_P\left(y\right)=p_{0,P}\left(y\right):P\in {\mathrm{Sym}}^{++}(\mathbb{R},d+1)\}$ using Calvo and Oller mapping [50]:

$$f(\mu ,\sum ):=\left[\begin{array}{cc}\sum +\mu {\mu }^{\top }\hfill & \mu \hfill \\ {\mu }^{\top }\hfill & 1\hfill \end{array}\right].$$

Let $\overline{\mathcal{M}}=\{f\left(p\right):p\in \mathcal{M}\}$ denote the embedded submanifold of codimension one in ${\mathcal{N}}_0^{+}$. The Calvo–Oller center is then defined by taking the Fisher–Rao midpoints $q_{\mathrm{CO}}$ of $q_{P_1}$ and $q_{P_2}$, projecting $q_{\mathrm{CO}}$ onto $\overline{\mathcal{M}}$ as $q_{\mathrm{CO}}^{\prime }$ and converting $q_{\mathrm{CO}}^{\prime }$ into $p_{\mathrm{CO}}\in \mathcal{M}$ using the inverse mapping $f^{-1}$ [51].

The Fisher orthogonal projection of a $(d+1)\times (d+1)$ matrix $P\in {\mathcal{N}}_0^{+}$ onto the submanifold $\overline{\mathcal{M}}$ is performed as follows: Let $\beta =P_{d+1,d+1}$ and write $P=\left[\begin{array}{cc}\sum +\beta \mu {\mu }^{\top }\hfill & \beta \mu \hfill \\ \beta {\mu }^{\top }\hfill & \beta \hfill \end{array}\right]$. Then, the orthogonal projection at $P\in \mathcal{P}$ onto $\overline{\mathcal{M}}$ is $\left[\begin{array}{cc}\sum +\mu {\mu }^{\top }\hfill & {\mu }^{\top }\hfill \\ \mu \hfill & 1\hfill \end{array}\right]$. See [51] for details of the Calvo and Oller embedding/projection method.

5. Experiments

We run all experiments on a Dell Inspiron 5502 Core i7-116567@2.8 GHz using compiled Java programs. For each experiment, we consider a set of $n=2$ uniformly randomized histograms with d bins (i.e., points in ${\Delta }_d$) and calculate the numerical Jeffreys centroid, which requires the time-consuming Lambert W function, the GB center, and the JFR center. For each prescribed value of d, we run 10,000 experiments to collect various statistics like the average and maximum approximations and running times. The approximations of the JFR and GB methods are calculated either as the approximation of the Jeffreys information (Equation (18)) or as the approximation of the centers with respect to the numerical Jeffreys centroids measured using the total variation distance.

Table 1. Experiments for JFR and GB centers approximating the numerical Jeffreys centroid.

dim.	Jeffreys–Fisher–Rao Center						Gauss–Bregman Center
	Avg Info $\mathit{ϵ}$	Max Info $\mathit{ϵ}$	Avg TV	Max TV	Avg Time	×Speed	Avg Info $\mathit{ϵ}$	Max Info $\mathit{ϵ}$	Avg TV	Max TV	Avg Time	×Speed
d = 2	5.662 × ${10}^{-6}$	6.386 × ${10}^{-3}$	8.735 × ${10}^{-5}$	5.005 × ${10}^{-2}$	1.614 × ${10}^{-7}$	82.541	1.507 × ${10}^{-4}$	9.745 × ${10}^{-2}$	6.304 × ${10}^{-4}$	5.005 × ${10}^{-2}$	5.072 × ${10}^{-7}$	26.258
d = 4	1.283 × ${10}^{-5}$	5.294 × ${10}^{-3}$	1.690 × ${10}^{-4}$	3.969 × ${10}^{-2}$	1.418 × ${10}^{-7}$	182.309	4.696 × ${10}^{-4}$	7.695 × ${10}^{-2}$	1.431 × ${10}^{-3}$	3.969 × ${10}^{-2}$	1.623 × ${10}^{-7}$	159.304
d = 8	2.766 × ${10}^{-5}$	6.970 × ${10}^{-3}$	2.210 × ${10}^{-4}$	3.470 × ${10}^{-2}$	1.772 × ${10}^{-7}$	292.125	1.011 × ${10}^{-3}$	9.677 × ${10}^{-2}$	2.033 × ${10}^{-3}$	3.470 × ${10}^{-2}$	1.955 × ${10}^{-7}$	264.680
d = 16	3.531 × ${10}^{-5}$	8.544 × ${10}^{-3}$	2.325 × ${10}^{-4}$	2.450 × ${10}^{-2}$	6.318 × ${10}^{-7}$	224.370	1.388 × ${10}^{-3}$	9.231 × ${10}^{-2}$	2.275 × ${10}^{-3}$	2.450 × ${10}^{-2}$	7.208 × ${10}^{-7}$	196.660
d = 32	4.123 × ${10}^{-5}$	5.242 × ${10}^{-3}$	2.457 × ${10}^{-4}$	1.230 × ${10}^{-2}$	4.811 × ${10}^{-7}$	462.754	1.674 × ${10}^{-3}$	5.398 × ${10}^{-2}$	2.449 × ${10}^{-3}$	1.230 × ${10}^{-2}$	5.457 × ${10}^{-7}$	408.007
d = 64	4.747 × ${10}^{-5}$	3.437 × ${10}^{-3}$	2.486 × ${10}^{-4}$	9.756 × ${10}^{-3}$	9.789 × ${10}^{-7}$	578.354	1.863 × ${10}^{-3}$	3.685 × ${10}^{-2}$	2.498 × ${10}^{-3}$	9.756 × ${10}^{-3}$	1.160 × ${10}^{-6}$	488.246
d = 128	5.020 × ${10}^{-5}$	2.540 × ${10}^{-3}$	2.491 × ${10}^{-4}$	6.580 × ${10}^{-3}$	5.874 × ${10}^{-6}$	477.412	1.937 × ${10}^{-3}$	2.374 × ${10}^{-2}$	2.522 × ${10}^{-3}$	6.580 × ${10}^{-3}$	6.605 × ${10}^{-6}$	424.609
d = 256	4.735 × ${10}^{-5}$	1.410 × ${10}^{-3}$	2.476 × ${10}^{-4}$	4.855 × ${10}^{-3}$	9.349 × ${10}^{-6}$	528.452	1.914 × ${10}^{-3}$	1.521 × ${10}^{-2}$	2.529 × ${10}^{-3}$	4.855 × ${10}^{-3}$	1.110 × ${10}^{-5}$	445.304

is a verbatim export of our experimental results when we range the dimension of histograms for $d=2$ to $d=256$ by doubling the dimension at each round. The inductive GB center is stopped when the total variation ${\displaystyle \frac{1}{2}}{\parallel a_t-g_t\parallel }_1\le {10}^{-8}$.

We observe that the JFR center is faster to compute than the GB center but the GB center is of higher quality (i.e., a better approximation with a lower $ϵ$) than the JFR center to approximate the numerical Jeffreys centroid.

Another test consists of choosing $d=3$ and the following two 3D normalized histograms: $({\displaystyle \frac{1}{3}},{\displaystyle \frac{1}{3}},{\displaystyle \frac{1}{3}})$ and $(1-\alpha ,\alpha /2,\alpha /2)$ for $\alpha \in \{{10}^{-1},{10}^{-2},\dots ,{10}^{-7},{10}^{-8}\}$.

Table 2. Experiments for JFR and GB centers approximating the numerical Jeffreys centroid for the following setting of two normalized histograms of 3 bins: $({\displaystyle \frac{1}{3}},{\displaystyle \frac{1}{3}},{\displaystyle \frac{1}{3}})$ and $(1-\alpha ,\alpha /2,\alpha /2)$.

$\mathit{\alpha }$	Info. $\mathit{ϵ}$	TV $\mathit{ϵ}$	Avg Time	×Speed	Info. $\mathit{ϵ}$	TV $\mathit{ϵ}$	Avg Time	×Speed
1.000 × ${10}^{-1}$	6.882 × ${10}^{-9}$	2.495 × ${10}^{-5}$	1.767 × ${10}^{-7}$	125.960	1.338 × ${10}^{-6}$	3.480 × ${10}^{-4}$	2.334 × ${10}^{-7}$	95.356
1.000 × ${10}^{-2}$	2.607 × ${10}^{-5}$	1.722 × ${10}^{-3}$	1.371 × ${10}^{-7}$	167.932	1.061 × ${10}^{-3}$	1.108 × ${10}^{-2}$	1.565 × ${10}^{-7}$	147.104
1.000 × ${10}^{-3}$	6.262 × ${10}^{-4}$	7.530 × ${10}^{-3}$	1.033 × ${10}^{-7}$	218.450	1.272 × ${10}^{-2}$	3.534 × ${10}^{-2}$	1.208 × ${10}^{-7}$	186.698
1.000 × ${10}^{-4}$	3.632 × ${10}^{-3}$	1.570 × ${10}^{-2}$	1.171 × ${10}^{-7}$	193.345	4.580 × ${10}^{-2}$	6.065 × ${10}^{-2}$	1.367 × ${10}^{-7}$	165.571
1.000 × ${10}^{-5}$	1.121 × ${10}^{-2}$	2.419 × ${10}^{-2}$	1.546 × ${10}^{-7}$	150.807	7.322 × ${10}^{-3}$	1.929 × ${10}^{-2}$	2.834 × ${10}^{-7}$	82.261
1.000 × ${10}^{-6}$	2.457 × ${10}^{-2}$	3.204 × ${10}^{-2}$	1.619 × ${10}^{-7}$	141.896	1.655 × ${10}^{-2}$	2.579 × ${10}^{-2}$	2.512 × ${10}^{-7}$	91.467
1.000 × ${10}^{-7}$	4.375 × ${10}^{-2}$	3.897 × ${10}^{-2}$	1.357 × ${10}^{-7}$	170.065	3.065 × ${10}^{-2}$	3.183 × ${10}^{-2}$	2.131 × ${10}^{-7}$	108.314
1.000 × ${10}^{-8}$	6.806 × ${10}^{-2}$	4.492 × ${10}^{-2}$	1.315 × ${10}^{-7}$	173.698	4.948 × ${10}^{-2}$	3.725 × ${10}^{-2}$	2.017 × ${10}^{-7}$	113.292
1.000 × ${10}^{-9}$	9.651 × ${10}^{-2}$	4.999 × ${10}^{-2}$	1.125 × ${10}^{-7}$	208.627	7.240 × ${10}^{-2}$	4.199 × ${10}^{-2}$	1.590 × ${10}^{-7}$	147.610
1.000 × ${10}^{-10}$	1.281 × ${10}^{-1}$	5.428 × ${10}^{-2}$	8.366 × ${10}^{-8}$	242.967	9.862 × ${10}^{-2}$	4.610 × ${10}^{-2}$	1.111 × ${10}^{-7}$	183.000
1.000 × ${10}^{-11}$	1.621 × ${10}^{-1}$	5.792 × ${10}^{-2}$	1.066 × ${10}^{-7}$	215.817	1.274 × ${10}^{-1}$	4.963 × ${10}^{-2}$	1.325 × ${10}^{-7}$	173.614
1.000 × ${10}^{-12}$	1.979 × ${10}^{-1}$	6.100 × ${10}^{-2}$	1.028 × ${10}^{-7}$	229.484	1.580 × ${10}^{-1}$	5.266 × ${10}^{-2}$	1.329 × ${10}^{-7}$	177.581
1.000 × ${10}^{-13}$	2.348 × ${10}^{-1}$	6.363 × ${10}^{-2}$	9.541 × ${10}^{-8}$	244.587	1.901 × ${10}^{-1}$	5.526 × ${10}^{-2}$	1.255 × ${10}^{-7}$	185.940
1.000 × ${10}^{-14}$	2.727 × ${10}^{-1}$	6.589 × ${10}^{-2}$	1.062 × ${10}^{-7}$	219.787	2.231 × ${10}^{-1}$	5.750 × ${10}^{-2}$	1.361 × ${10}^{-7}$	171.456
1.000 × ${10}^{-15}$	3.112 × ${10}^{-1}$	6.784 × ${10}^{-2}$	9.043 × ${10}^{-8}$	248.688	2.570 × ${10}^{-1}$	5.943 × ${10}^{-2}$	1.322 × ${10}^{-7}$	170.122
1.000 × ${10}^{-16}$	3.483 × ${10}^{-1}$	6.946 × ${10}^{-2}$	8.857 × ${10}^{-8}$	267.219	2.897 × ${10}^{-1}$	6.105 × ${10}^{-2}$	1.438 × ${10}^{-7}$	164.535

reports the experiments. The objective is to find a setting where both the JFR and GB centers are distinguished from the Jeffreys centroid. We see that as we decrease $\alpha$, the approximation factor $ϵ$ gets worse for both the JFR center and the GB center. The JFR center is often faster to compute than the GB inductive center, but the approximation of the GB center is better than the JFR approximation.

Finally, we implemented the Gauss–Bregman and Jeffreys–Fisher–Rao centers and Jeffreys centroid using multi-precision arithmetic. We report the following experiments using 200-digit precision arithmetic for the following input of two normalized histograms: $p=(0.1,0.9)$ and $q=(0.8,0.2)$. We report the various first 17-digit mantissas obtained with the corresponding Jeffreys information:

• Jeffreys center: $(0.42490383904214813,0.575096160957851866)$
  Jeffreys information: $1.2490723231955352$.
• Gauss–Bregman center: (0.42490383904276856, 0.575096160957231439)
  Jeffreys information: $1.2490723231955353$.
• Jeffreys–Fisher–Rao center: (0.42490390202906282, 0.575096097970937175)
  Jeffreys information: $1.2490723232068266$.

The total variation distance between the Jeffreys centroid and the Gauss–Bregman center is $6.204271148284087422350000686372{10}^{-13}$.

The total variation distance between the Jeffreys centroid and the Jeffreys–Fisher–Rao center is $6.298691469047911984039363762611{10}^{-8}$.

The total variation distance between the Gauss–Bregman center and the Jeffreys–Fisher–Rao center is $6.298629426336429143165140262604{10}^{-8}$.

Although all those points are close to each other, they are all distinct points (note that using the limited precision of the IEEE 754 floating point standard may yield a misleading interpretation of experiments).

6. Conclusions and Discussion

In this work, we considered the Jeffreys centroid of a finite weighted set of densities of a given exponential family $\mathcal{E}=\{p_{\theta }\left(x\right):\theta \in \Theta \}$. This Jeffreys centroid amounts to a symmetrized Bregman centroid on the corresponding weighted set of natural parameters of the densities [28]. In general, the Jeffreys centroids do not admit closed-form formulas [28,31] except for sets of same-mean normal distributions [29] (see Appendix B).

In this paper, we interpreted the closed-form formula for same-mean multivariate normal distributions in two different ways:

• First, as the Fisher–Rao geodesic midpoint of the sided Kullback–Leibler centroids. This interpretation lets us relax the midpoint definition to arbitrary exponential families to define the Jeffreys–Fisher–Rao center (the JFR center of Definition 2);
• Second, as an inductive $(A,m_{\nabla F})$ center using a multivariate Gauss-type double sequence, which converges to the Gauss–Bregman center (the GB center of Definition 3). The latter definition yields an extension of Nakamura’s arithmetic–harmonic $(A,H)$ mean [41] to an arbitrary $(A,m_{\nabla F})$ mean for which we proved convergence under a separability condition in Theorem 3. Convergence proof remains to be performed in the general case, although we noticed in practice convergence when $\nabla F\left(\theta \right)$ is the moment parameter of categorical or multivariate normal distributions.

In general, the Jeffreys, JFR, and GB centers differ from each other (e.g., the case of categorical distributions). But for sets of same-mean normal distributions, they remarkably coincide altogether: namely, this was the point of departure of this research. We reported generic or closed-form formulas for the JFR centers of (a) uni-order parametric families in Section 4.1 (Theorem 4), (b) categorical families in Section 4.2 (Theorem 5), and (c) multivariate normal families in Section 4.3 (Theorem 6).

Table 3. Summary of the results: Δ indicates a generic formula, √ a closed-form formula, and × no-known formula.

Family	Jeffreys	Jeffreys–Fisher–Rao	Gauss–Bregman
Exponential family	Equation (2)	Definition 2	Definition 3
One-dimensional exponential family	×	Δ	×
	Theorem 4		[43]
Categorical family	Δ	√	×
	[31]	Theorem 5	Theorem 3
Normal family	×	√	×
	[28]	Theorem 6	Theorem 3
Centered normal family	√	√	√
	[29]	[29]	[41]

summarizes the new results obtained in this paper and states references of prior work. Notice that in practice, we approximate the Gauss–Bregman center by prescribing a number of iterations $T\in \mathbb{N}$ for the Gauss–Bregman double sequence to obtain $m_{\mathrm{GB}}^{\left(T\right)}$. Prescribing the number of GB center iterations T allows us to tune the time complexity of computing $m_{\mathrm{GB}}^{\left(T\right)}$ while adjusting the quality of the approximation of the Jeffreys centroid.

In applications requiring the Jeffreys centroid, we thus propose to either use the fast Jeffreys–Fisher–Rao center when a closed-form formula is available for the family of distributions at hand or use the Gauss–Bregman center approximation with a prescribed number of iterations as a drop-in replacement of the numerical Jeffreys centroids while keeping the Jeffreys divergence (the centers we defined are not centroids as we do not exhibit distances from which they are population minimizers).

More generally, let us rephrase the results in a purely geometric setting using the framework of information geometry [14]: let $P_1,\dots ,P_n$ be a set of n points weighted by a vector $w\in {\Delta }_n$ on an m-dimensional dually flat space $(M,g,\nabla ,{\nabla }^{*})$ with the ∇-affine coordinate system $\theta (·)$ and dual ${\nabla }^{*}$-affine coordinate system $\eta (·)$, where ∇ and ${\nabla }^{*}$ are two torsion-free dual affine connections. The Riemannian metric g is a Hessian metric [54], which may be expressed in the $\theta$-coordinate system as $g\left(\theta \right)={\nabla }^{2}F\left(\theta \right)$ or in the dual coordinate system as $g\left(\eta \right)={\nabla }^2{F}^{*}\left(\eta \right)$, where $F\left(\theta \right)$ and $F^{*}\left(\eta \right)$ are dual convex potential functions related by the Legendre–Fenchel transform [14,54]. Let ${\eta }_i=\nabla F\left({\theta }_i\right)$ and ${\theta }_i=\nabla F^{*}\left({\eta }_i\right)$ be the coordinates of point $P_i$ in the $\eta$- and $\theta$-coordinate systems, respectively. An arbitrary point P can be either referenced in the $\theta$-coordinate system ($P=P_{\theta }$) or in the $\eta$-coordinate system ($P=P_{\eta }$). Then, the Jeffreys–Fisher–Rao center is defined as the midpoint with respect to the Levi-Civita connection $\overline{\nabla }={\displaystyle \frac{\nabla +{\nabla }^{*}}{2}}={\nabla }^g$ of g:

$$C_{\mathrm{JFR}}:={\gamma }_{\overline{\nabla }}(C_{\overline{\theta }},C_{\underline{\theta }},{\displaystyle \frac{1}{2}})=:C_{\overline{\theta }}\#C_{\underline{\theta }}.$$

(20)

The point $C_{\overline{\theta }}$ is the centroid with respect to the canonical flat divergence $\mathcal{D}(P:Q)=F\left(\theta \left(P\right)\right)+F^{*}\left(\eta \left(Q\right)\right)-{\sum }_{i=1}^m{\theta }_i\left(P\right){\eta }_i\left(Q\right)$, and the point $C_{\underline{\theta }}$ is the centroid with respect to the dual canonical flat divergence ${\mathcal{D}}^{*}(P:Q):=\mathcal{D}(Q:P)$. The canonical divergence is expressed using the mixed coordinates $\theta$/$\eta$ but can also be expressed using the $\theta$-coordinates as an equivalent Bregman divergence $\mathcal{D}(P:Q)=B_F(\theta \left(P\right):\theta \left(Q\right))$ or as a reverse dual Bregman divergence $\mathcal{D}(P:Q)=B_{F^{*}}(\eta \left(Q\right):\eta \left(P\right))$. This JFR center $C_{\mathrm{JFR}}$ approximates the symmetrized centroid with respect to the canonical symmetrized divergence $\mathcal{S}(P,Q)=\mathcal{D}(P:Q)+\mathcal{D}(Q:P)$ (i.e., Jeffreys divergence when written using the $\theta$-coordinate system). This symmetrized divergence $\mathcal{S}(P,Q)$ can be interpreted as the energy of the Riemannian length element $\mathrm{d}s$ along the primal geodesic $\gamma \left(t\right)$ and dual geodesic ${\gamma }^{*}\left(t\right)$ (with $\gamma \left(0\right)={\gamma }^{*}\left(0\right)=P$ and $\gamma \left(1\right)={\gamma }^{*}\left(1\right)=Q$), see [14]: $\mathcal{S}(P,Q)={\int }_0^1\mathrm{d}{s}^2\left(\gamma \left(t\right)\right)\mathrm{d}t={\int }_0^1\mathrm{d}{s}^2\left({\gamma }^{*}\left(t\right)\right)\mathrm{d}t$. The Riemannian distance $\rho (P,Q)$ corresponds to the Riemannian length element along the Riemannian geodesic $\overline{\gamma }\left(t\right)$ induced by the Levi-Civita connection $\overline{\nabla }={\displaystyle \frac{\nabla +{\nabla }^{*}}{2}}$: $\rho (P,Q)={\int }_0^1\mathrm{d}s\left(\overline{\gamma }\left(t\right)\right)\mathrm{d}t$.

The inductive Gauss–Bregman center $C_{\mathrm{GB}}$ is obtained as a limit sequence of taking iteratively the ∇ midpoints and ${\nabla }^{*}$ midpoints with respect to the ∇ and ${\nabla }^{*}$ connections. Those midpoints correspond to the right and left centroids $C_{t+1}$ and $C_{t+1}^{*}$ with respect to $\mathcal{D}(·:·)$:

$$\begin{array}{ccc}\hfill C_{t+1}& =& {\gamma }_{\nabla }\left(C_t,C_t^{*},{\displaystyle \frac{1}{2}}\right),\hfill \\ \hfill C_{t+1}^{*}& =& {\gamma }_{{\nabla }^{*}}\left(C_t,C_t^{*},{\displaystyle \frac{1}{2}}\right),\hfill \end{array}$$

initialized with $\theta \left(C_0\right)={\sum }_{i=1}^n{w}_i\theta \left(P_i\right)$ and $\eta \left(C_0^{*}\right)={\sum }_{i=1}^n{w}_i\eta \left(P_i\right)$. We have $C_0=\arg {\min }_{C\in M}$${\sum }_i{w}_i\mathcal{D}(P_i:C)$ and $C_0^{*}=\arg {\min }_{C\in M}{\sum }_i{w}_i\mathcal{D}(P_i:C)$. Figure 10 geometrically illustrates the double sequence of iteratively taking dual geodesic midpoints to converge toward the Gauss–Bregman center $C_{\mathrm{GB}}$. Thus, the GB double sequence can be interpreted as a geometric optimization technique. Figure 11 illustrates the JFR and GB centers on a dually flat space. Notice that $C_{\mathrm{JFR}}$ has coordinates ${\mathrm{JFR}}_F({\mathcal{P}}_{\theta };w)$ in the $\theta$-chart and coordinates ${\mathrm{JFR}}_{F^{*}}({\mathcal{P}}_{\eta };w)$ in the $\eta$-chart. Similarly, $C_{\mathrm{GB}}$ has coordinates ${\mathrm{GB}}_F(\overline{\theta },\underline{\theta })$ in the $\theta$-chart and coordinates ${\mathrm{GB}}_{F^{*}}(\overline{\eta },\underline{\eta })$ in the $\eta$-chart.

As a final remark, let us emphasize that choosing a proper mean or center depends on the application at hand [55,56]. For example, in Bayesian hypothesis testing, the Chernoff mean [57] is used to upper bound Bayes’ error and has been widely used in information fusion [18] for its empirical robustness [58] in practice. Jeffreys centroid has been successfully used in information retrieval tasks [6].