A Simple Approximation Method for the Fisher–Rao Distance between Multivariate Normal Distributions

Abstract

We present a simple method to approximate the Fisher–Rao distance between multivariate normal distributions based on discretizing curves joining normal distributions and approximating the Fisher–Rao distances between successive nearby normal distributions on the curves by the square roots of their Jeffreys divergences. We consider experimentally the linear interpolation curves in the ordinary, natural, and expectation parameterizations of the normal distributions, and compare these curves with a curve derived from the Calvo and Oller’s isometric embedding of the Fisher–Rao d-variate normal manifold into the cone of $(d+1)\times (d+1)$ symmetric positive–definite matrices. We report on our experiments and assess the quality of our approximation technique by comparing the numerical approximations with both lower and upper bounds. Finally, we present several information–geometric properties of Calvo and Oller’s isometric embedding.

1. Introduction

1.1. The Fisher–Rao Normal Manifold

Let $\mathrm{Sym}\left(d\right)$ be the set of $d\times d$ symmetric matrices with real entries and $\mathbb{P}\left(d\right)\subset \mathrm{Sym}\left(d\right)$ denote the set of symmetric positive–definite $d\times d$ matrices that forms a convex regular cone. Let us denote by $\mathcal{N}\left(d\right)=\{N(\mu ,\Sigma ):(\mu ,\Sigma )\in \Lambda \left(d\right)={\mathbb{R}}^d\times \mathbb{P}\left(d\right)\}$ the set of d-variate normal distributions, MultiVariate Normals or MVNs for short, also called Gaussian distributions. A MVN distribution $N(\mu ,\Sigma )$ has probability density function (pdf) on the support ${\mathbb{R}}^d$:

$$p_{\lambda =(\mu ,\Sigma )}\left(x\right)={\left(2\pi \right)}^{{}^{\frac{d}{2}}}{|\Sigma |}^{-\frac{1}{2}}exp\left(-\frac{1}{2}{(x-\mu )}^{\top }{\Sigma }^{-1}(x-\mu )\right),x\in {\mathbb{R}}^d,$$

where $\left|M\right|=\det \left(M\right)$ denotes the determinant of matrix M.

The statistical model $\mathcal{N}\left(d\right)$ is of dimension $m=\dim (\Lambda \left(d\right))=d+\frac{d(d+1)}{2}=\frac{d(d+3)}{2}$ since it is identifiable, i.e., there is a one-to-one correspondence $\lambda \leftrightarrow p_{\lambda }\left(x\right)$ between $\lambda \in \Lambda \left(d\right)$ and $N(\mu ,\Sigma )\in \mathcal{N}\left(d\right)$. The statistical model $\mathcal{N}\left(d\right)$ is said to be regular since the second-order derivatives $\frac{{\partial }^2{p}_{\lambda }}{\partial {\lambda }_i\partial {\lambda }_j}$ and third-order derivatives $\frac{{\partial }^3{p}_{\lambda }}{\partial {\lambda }_i\partial {\lambda }_j\partial {\lambda }_k}$ are smooth functions (defining the metric and cubic tensors in information geometry [1]), and the set of first-order partial derivatives $\left\{\frac{\partial p_{\lambda }}{\partial {\lambda }_1},\dots ,\frac{\partial p_{\lambda }}{\partial {\lambda }_1}\right\}$ are linearly independent.

Let $\mathrm{Cov}\left(X\right)$ denote the covariance of X (variance when X is scalar). A matrix M is a semi-positive–definite if and only if $\forall x\ne 0,x^{\top }Mx\ge 0$. The Fisher information matrix [1,2] (FIM) is the following symmetric semi-positive–definite matrix:

$$I\left(\lambda \right)=\mathrm{Cov}[\nabla log{p}_{\lambda }\left(x\right)]⪰0.$$

For regular statistical models $\left\{p_{\lambda }\right\}$, the FIM is positive–definite: $I\left(\lambda \right)\succ 0$, i.e., $\forall x\ne 0,x^{\top }I\left(\lambda \right)x>0$. $M_1\succ M_2$ denotes Löwner partial ordering, i.e., the fact that $M_1-M_2$ is positive–definite.

The FIM is covariant under the reparameterization of the statistical model [2]. That is, let $\theta \left(\lambda \right)$ be a new parameterization of the MVNs. Then we have:

$$I_{\theta }\left(\lambda \right)={\left[\frac{\partial \lambda }{\partial \theta }\right]}^{\top }\times I_{\lambda }\left(\lambda \left(\theta \right)\right)\times \left[\frac{\partial \lambda }{\partial \theta }\right].$$

For example, we may parameterize univariate normal distributions by $\lambda =(\mu ,{\sigma }^2)$ or $\theta =(\mu ,\sigma )$. We obtain the following Fisher information matrices for these parameterizations:

$$I_{\lambda }\left(\lambda (\mu ,\sigma )\right)=\left[\begin{array}{cc}\frac{1}{{\sigma }^2}\hfill & 0\hfill \\ 0\hfill & \frac{1}{2{\sigma }^4}\hfill \end{array}\right]\text{and}{I}_{\theta }\left(\theta (\mu ,\sigma )\right)=\left[\begin{array}{cc}\frac{1}{{\sigma }^2}\hfill & 0\hfill \\ 0\hfill & \frac{1}{2{\sigma }^2}\hfill \end{array}\right].$$

In higher dimensions, parameterization $\lambda =(\mu ,{\sigma }^2)$ corresponds to the parameterization $(\mu ,\Sigma )$ while parameterization $\theta =(\mu ,L)$ where $\Sigma =L{L}^{\top }$ is the unique Cholesky decomposition with $L\in \mathrm{GL}\left(d\right)$, the group of invertible $d\times d$ matrices. Another useful parameterization for optimization is the log–Cholesky parameterization [3] ($\eta =(\mu ,log{\sigma }^2)\in {\mathbb{R}}^2$ for univariate normal distributions) which ensures that a gradient descent always stays in the domain. The Fisher information matrix with respect to the log–Cholesky parameterization is $I_{\eta }\left(\eta (\mu ,\sigma )\right)=\left[\begin{array}{cc}\frac{1}{{\sigma }^2}\hfill & 0\hfill \\ 0\hfill & 2\hfill \end{array}\right]$ with $\eta (\mu ,\sigma )\in {\mathbb{R}}^2$.

Since the statistical model $\mathcal{N}\left(d\right)$ is identifiable and regular, the Fisher information matrix can be written equivalently as follows [2,4]:

$$\begin{array}{ccc}\hfill I(\mu ,\Sigma )=\mathrm{Cov}[\nabla log{p}_{(\mu ,\Sigma )}]& =& E_{p_{(\mu ,\Sigma )}}\left[\nabla log{p}_{(\mu ,\Sigma )}\nabla log{p}_{(\mu ,\Sigma )}^{\top }\right],\hfill \end{array}$$

(1)

$$=-E_{p_{(\mu ,\Sigma )}}\left[{\nabla }^{2}log{p}_{(\mu ,\Sigma )}\right].$$

(2)

For multivariate distributions parameterized by a m-dimensional vector (with $m=\frac{d(d+3)}{2}$)

$$\theta =({\theta }_1,\dots ,{\theta }_d,{\theta }_{d+1},\dots ,{\theta }_m)\in {\mathbb{R}}^m,$$

with $\mu =({\theta }_1,\dots ,{\theta }_d)$ and $\Sigma \left(\theta \right)=\mathrm{vech}({\theta }_{d+1},\dots ,{\theta }_m)$ (inverse half-vectorization of matrices [5]), we have [6,7,8,9]:

$$I\left(\theta \right)=\left[I_{ij}\left(\theta \right)\right],\text{with}{I}_{ij}\left(\theta \right)={\left[\frac{\partial \mu }{\partial {\theta }_i}\right]}^{\top }{\Sigma }^{-1}\frac{\partial \mu }{\partial {\theta }_j}+\frac{1}{2}\mathrm{tr}\left({\Sigma }^{-1}\frac{\partial \mu }{\partial {\theta }_i}{\Sigma }^{-1}\frac{\partial \mu }{\partial {\theta }_j}\right).$$

By equipping the regular statistical model $\mathcal{N}\left(d\right)$ with the Fisher information metric

$$g_{\mathcal{N}}^{\mathrm{Fisher}}(\mu ,\Sigma )=\mathrm{Cov}[\nabla log{p}_{(\mu ,\Sigma )}\left(x\right)]$$

we obtain a Riemannian manifold $\mathcal{M}={\mathcal{M}}_{\mathcal{N}}$ called the Fisher–Rao Gaussian or normal manifold [6,7]. The tangent space $T_N\mathcal{M}$ is identified with the product space ${\mathbb{R}}^d\times \mathrm{Sym}\left(d\right)$. Let $\{\partial \mu ,\partial \Sigma \}$ be a natural vector basis in $T_N\mathcal{M}$, and denote by $\left[v\right]$ and $\left[V\right]$ the vector components in that natural basis. We have

$$\begin{array}{ccc}\hfill g_{(\mu ,\Sigma )}^{\mathrm{Fisher}}((v_1,V_1),(v_2,V_2))& =& {⟨(v_1,V_1),(v_2,V_2)⟩}_{(\mu ,\Sigma )},\hfill \\ & =& {\left[v_1\right]}^{\top }{\Sigma }^{-1}\left[v_2\right]+\frac{1}{2}\mathrm{tr}\left({\Sigma }^{-1}\left[V_1\right]{\Sigma }^{-1}\left[V_2\right]\right).\hfill \end{array}$$

The induced Riemannian geodesic distance ${\rho }_{\mathcal{N}}(·,·)$ is called the Rao distance [10] or the Fisher–Rao distance [11,12]:

$${\rho }_{\mathcal{N}}(N\left({\lambda }_1\right),N\left({\lambda }_2\right))=\underset{\begin{array}{c}c\left(t\right)\\ c\left(0\right)=p_{{\lambda }_1}\\ c\left(1\right)=p_{{\lambda }_2}\end{array}}{inf}\left\{\mathrm{Length}\left(c\right)\right\},$$

(3)

where the Riemannian length of any smooth curve $c\left(t\right)\in \mathcal{M}$ is defined by

$$\mathrm{Length}\left(c\right)={\int }_0^1\sqrt{{⟨\dot{c}\left(t\right),\dot{c}\left(t\right)⟩}_{c\left(t\right)}}\mathrm{d}t={\int }_0^1\mathrm{d}{s}_{\mathcal{N}}\left(t\right)\mathrm{d}t={\int }_0^1{\parallel \dot{c}\left(t\right)\parallel }_{c\left(t\right)}\mathrm{d}t,$$

(4)

where $\dot{}=\frac{d}{\mathrm{d}t}$ denotes the derivative with respect to parameter t, $\mathrm{d}{s}_{\mathcal{N}}\left(t\right)$ is the Riemannian length element of $(\mathcal{M},g_{\mathcal{N}}^{\mathrm{Fisher}})$ and ${\parallel ·\parallel }_{c\left(t\right)}=\sqrt{{⟨·,·⟩}_{c\left(t\right)}}$. We also write ${\rho }_{\mathcal{N}}(p_{{\lambda }_1},p_{{\lambda }_2})$ for ${\rho }_{\mathcal{N}}(N\left({\lambda }_1\right),N\left({\lambda }_2\right))$.

The minimizing curve $c\left(t\right)={\gamma }_{\mathcal{N}}(p_{{\lambda }_1},p_{{\lambda }_2};t)$ of Equation (3) is called the Fisher–Rao geodesic. The Fisher–Rao geodesic is also an autoparallel curve [2] with respect to the Levi–Civita connection ${\nabla }_{\mathcal{N}}^{\mathrm{Fisher}}$ induced by the Fisher metric $g_{\mathcal{N}}^{\mathrm{Fisher}}$.

Remark 1. 

If we consider the Riemannian manifold $(\mathcal{M},\beta g)$ for $\beta >0$ instead of $(\mathcal{M},g)$ then the length element $\mathrm{d}s$ is scaled by $\sqrt{\beta }$: $\mathrm{d}{s}_{\beta g}=\beta \mathrm{d}{s}_g$. It follows that the length of a curve c becomes

$${\mathrm{Length}}_{\beta g}\left(c\right)=\sqrt{\beta }{\mathrm{Length}}_g\left(c\right).$$

However, the geodesics joining any two points $p_1$ and $p_2$ of $\mathcal{M}$ are the same: ${\gamma }_{\beta g}(p_1,p_2;t)={\gamma }_g(p_1,p_2;t)$ (with ${\gamma }_g(p_1,p_2;0)=p_1$ and ${\gamma }_g(p_1,p_2;1)=p_2$).

Historically, Hotelling [13] first used this Fisher Riemannian geodesic distance in the late 1920s. From the viewpoint of information geometry [1], the Fisher metric is the unique Markov invariant metric up to rescaling [14,15,16]. The counterpart to the Fisher metric on the compact manifold has been reported in [17], proving its uniqueness under the action of the diffeomorphism group. The Fisher–Rao distance has been used to design statistical hypothesis testing [18,19,20,21], to measure the distance between the prior and posterior distributions in Bayesian statistics [22], in clustering [23,24], in signal processing [25,26,27,28], and in deep learning [29], just to mention a few.

The squared line element induced by the Fisher metric of the multivariate normal family [6,7] is

$$\begin{array}{ccc}\hfill \mathrm{d}{s}_{\mathcal{N}}^2(\mu ,\Sigma )& =& {\left[\begin{array}{c}\mathrm{d}\mu \hfill \\ \mathrm{d}\Sigma \hfill \end{array}\right]}^{\top }I(\mu ,\Sigma )\left[\begin{array}{c}\mathrm{d}\mu \hfill \\ \mathrm{d}\Sigma \hfill \end{array}\right],\hfill \\ \hfill & =& \mathrm{d}{\mu }^{\top }{\Sigma }^{-1}\mathrm{d}\mu +\frac{1}{2}\mathrm{tr}\left({\left({\Sigma }^{-1}\mathrm{d}\Sigma \right)}^2\right).\hfill \end{array}$$

(5)

There are many ways to calculate the FIM/length element for multivariate normal distributions [7,9]. Let us give a simple approach based on the fact that the family $\mathcal{N}\left(d\right)$ of normal distributions forms a regular exponential family [30]:

$$\mathcal{N}\left(d\right)=\left\{p_{\theta \left(\lambda \right)}=exp\left(⟨{\theta }_v\left(\mu \right),x⟩+⟨{\theta }_M(\Sigma ),x{x}^{\top }⟩-F_{\mathcal{N}}({\theta }_v,{\theta }_M)\right)\right\},$$

with $\theta \left(\lambda \right)=({\theta }_v=({\Sigma }^{-1}\mu ,{\theta }_M=\frac{1}{2}{\Sigma }^{-1})$ the natural parameters and log-partition function (also called cumulant function)

$$F_{\mathcal{N}}\left(\theta \right)=\frac{1}{2}\left(dlog\pi -log|{\theta }_M|+\frac{1}{2}{\theta }_v^{\top }{\theta }_M^{-1}{\theta }_v\right).$$

The vector inner product is $⟨v_1,v_2⟩=v_1^{\top }{v}_2$, and the matrix inner product is $⟨M_1,M_2⟩=\mathrm{tr}\left(M_1{M}_2^{\top }\right)$. The exponential family is said to be regular when the natural parameter space is open. Using Equation (2), it follows that the MVN FIM is $I_{\theta }\left(\theta \right)=-E[{\nabla }^{2}log{p}_{\theta }]={\nabla }^{2}F\left(\theta \right)$. This proves that the FIM is well-defined, i.e., ${\left(I_{\theta }\left(\theta \right)\right)}_{ij}<\infty$. As an exponential family [1], we also have $I_{\theta }\left(\theta \right)=E\left[t\left(x\right)\right]$, where $t\left(x\right)=(x,x{x}^{\top })$ is the sufficient statistic. Thus, the Fisher metric is a Hessian metric [31]. Let $F_{\mathcal{N}}({\theta }_v,{\theta }_M)=F_v\left({\theta }_v\right)+F_M\left({\theta }_M\right)$ with $F_v\left({\theta }_v\right)=\frac{1}{2}\left(dlog\pi +\frac{1}{2}{\theta }_v^{\top }{\theta }_M^{-1}{\theta }_v\right)$ and $F_M\left({\theta }_M\right)=-\frac{1}{2}log\left|{\theta }_M\right|$. We obtain the following block-diagonal expression of the FIM:

$$I\left(\theta \left(\lambda \right)\right)={\nabla }^2{F}_{\mathcal{N}}\left(\theta (\mu ,\Sigma )\right)=\left[\begin{array}{cc}{\Sigma }^{-1}\hfill & 0\hfill \\ 0\hfill & \frac{1}{2}{\nabla }_{{\theta }_M}^{2}log\left|\frac{1}{2}{\Sigma }^{-1}\right|\hfill \end{array}\right].$$

Therefore $\mathrm{d}{s}_{\mathcal{N}}^2(\mu ,\Sigma )=\mathrm{d}{s}_v^2+\mathrm{d}{s}_M^2$ with $\mathrm{d}{s}_v^2\left(\mu \right)=\mathrm{d}{\mu }^{\top }{\Sigma }^{-1}\mathrm{d}\mu$ and $\mathrm{d}{s}_M^2(\Sigma )=\frac{1}{2}\mathrm{tr}\left({\left({\Sigma }^{-1}\mathrm{d}\Sigma \right)}^2\right)$. Let us note in passing that ${\nabla }_{{\theta }_M}^{2}log\left|{\theta }_M\right|$ is a fourth order tensor [4].

The family $\mathcal{N}\left(d\right)$ can also be considered to be an elliptical family [32], thus highlighting the affine-invariance property of the Fisher information metric. That is, the Fisher metric is invariant with respect to affine transformations [33]: Let $(a,A)$ be an element of the affine group $\mathrm{Aff}\left(d\right)$ with $a\in {\mathbb{R}}^d$ and $A\in \mathrm{GL}\left(d\right)$. The group identity element of $\mathrm{Aff}\left(d\right)$ is $e=(0,I)$ and the group operation is $(a_1,A_1).(a_2,A_2)=(a_1+A_1{a}_2,A_1{A}_2)$ with inverse ${(a,A)}^{-1}=(-A^{-1}a,A^{-1})$). Then we have

Property 1 

(Fisher–Rao affine invariance). For all $A\in \mathrm{GL}\left(d\right),a\in {\mathbb{R}}^d$, we have

$${\rho }_{\mathcal{N}}(N(A{\mu }_1+a,A{\Sigma }_1{A}^{\top }),N(A{\mu }_2+a,A{\Sigma }_2{A}^{\top }))={\rho }_{\mathcal{N}}(N({\mu }_1,{\Sigma }_1),N({\mu }_2,{\Sigma }_2)).$$

(6)

This can be proven by checking that $\mathrm{d}{s}_{\mathcal{N}}({\mu }^{\prime },{\Sigma }^{\prime })=\mathrm{d}{s}_{\mathcal{N}}(\mu ,\Sigma )$ where ${\mu }^{\prime }=A\mu +a$ and ${\Sigma }^{\prime }=A{\Sigma }_2{A}^{\top }$. It follows that we can reduce the calculation of the Fisher–Rao distance to a canonical case where one argument is $N_{\mathrm{std}}=N(0,I)$, the standard d-variate distribution:

$$\begin{array}{ccc}\hfill {\rho }_{\mathcal{N}}(N({\mu }_1,{\Sigma }_1),N({\mu }_2,{\Sigma }_2))& =& {\rho }_{\mathcal{N}}\left(N_{\mathrm{std}},N\left({\Sigma }_1^{-\frac{1}{2}}({\mu }_2-{\mu }_1),{\Sigma }_1^{-\frac{1}{2}}{\Sigma }_2{\Sigma }_1^{-\frac{1}{2}}\right)\right),\hfill \\ & =& {\rho }_{\mathcal{N}}\left(N\left({\Sigma }_2^{-\frac{1}{2}}({\mu }_1-{\mu }_2),{\Sigma }_2^{-\frac{1}{2}}{\Sigma }_1{\Sigma }_2^{-\frac{1}{2}}\right),N_{\mathrm{std}}\right),\hfill \end{array}$$

where ${\Sigma }^p$ is the fractional matrix power which can be calculated from the Singular Value Decomposition $OD{O}^{\top }$ of $\Sigma$ (where O is an orthogonal matrix and $D=\mathrm{diag}({\lambda }_1,\dots ,{\lambda }_d)$ a diagonal matrix): ${\Sigma }^p=O{D}^p{O}^{\top }$ with $D^p=\mathrm{diag}({\lambda }_1^p,\dots ,{\lambda }_d^p)$.

The family of normal elliptical distributions can be obtained from the standard normal distribution by the action of the affine group [12,32] $\mathrm{Aff}\left(d\right)$:

$$N(\mu ,\Sigma )=(\mu ,{\Sigma }^{\frac{1}{2}}).N_{\mathrm{std}}=N((\mu ,{\Sigma }^{\frac{1}{2}}).(0,I)).$$

1.2. Fisher–Rao Distance between Normal Distributions: Some Subfamilies with Closed-Form Formula

In general, the Fisher–Rao distance ${\rho }_{\mathcal{N}}(N_1,N_2)$ between two multivariate normal distributions $N_1$ and $N_2$ is not known in closed form [34,35,36,37], and several lower and upper bounds [38], and numerical techniques such as the geodesic shooting [39,40,41] have been investigated. See [42] for a recent review. Unfortunately, the geodesic shooting (GS) approach is time-consuming and numerically unstable for large Fisher–Rao distances [21,42]. In 3D Diffusion Tensor Imaging (DTI), $3\times 3$ covariance matrices ${\Sigma }_{i,j,k}$ are stored a 3D grid locations ${\mu }_{i,j,k}$ thus generating 3D MVNs $N_{i,j,k}=N({\mu }_{i,j,k},{\Sigma }_{i,j,k})$ with means ${\mu }_{i,j,k}$ regularly spaced to each others. The Fisher–Rao distances can be calculated between an MVN $N_{i,j,k}$ and another MVN $N_{i^{\prime },j^{\prime },k^{\prime }}$ in a neighborhood of $N_{i,j,k}$ (using 6- or 26-neighborhood) using geodesic shooting. For larger Fisher–Rao distances between non-neighbors MVNs, we can use the shortest path distance using Dijkstra’s algorithm [43] on the graph induced by the MVNs with edges between adjacent MVNs weighted by their Fisher–Rao distances.

The two main difficulties with calculating the Fisher–Rao distance are

• to know explicitly the expression of the Riemannian Fisher–Rao geodesic ${\gamma }_{\mathcal{N}}^{\mathrm{FR}}(p_{{\lambda }_1},p_{{\lambda }_2};t)$ and
• to integrate, in closed form, the length element $\mathrm{d}{s}_{\mathcal{N}}$ along this Riemannian geodesic.

Please note that the Fisher–Rao geodesics [1] ${\gamma }_{\mathcal{N}}^{\mathrm{FR}}(p_{{\lambda }_1},p_{{\lambda }_2};t)$ are parameterized by constant speed (i.e., $\dot{\mu }\left(t\right)=\dot{\mu }\left(0\right)$ and $\dot{\Sigma }\left(t\right)=\dot{\Sigma }\left(0\right)$), or equivalently parametrized using the arc length:

$${\rho }_{\mathcal{N}}\left({\gamma }_{\mathcal{N}}^{\mathrm{FR}}(p_{{\lambda }_1},p_{{\lambda }_2};s),{\gamma }_{\mathcal{N}}^{\mathrm{FR}}(p_{{\lambda }_1},p_{{\lambda }_2};t)\right)=|s-t|{\rho }_{\mathcal{N}}(p_{{\lambda }_1},p_{{\lambda }_2}),\forall s,t\in [0,1].$$

However, in several special cases, the Fisher–Rao distance between normal distributions belonging to restricted subsets of $\mathcal{N}$ is known.

Three such prominent cases are (see [42] for other cases)

• when the normal distributions are univariate ($d=1$),
• when we consider the set ${\mathcal{N}}_{\mu }=\{N(\mu ,\Sigma ):\Sigma \in \mathcal{P}\left(d\right)\}\subset {\mathcal{M}}_{\mathcal{N}}$ of normal distributions sharing the same mean $\mu$ (with the embedded submanifold ${\mathcal{S}}_{\mu }\in \mathcal{M}$), and
• when we consider the set ${\mathcal{N}}_{\Sigma }=\{N(\mu ,\Sigma ):\Sigma \in \mathcal{P}\left(d\right)\}\subset \mathcal{N}$ of normal distributions sharing the same covariance matrix $\Sigma$ (with the corresponding embedded submanifold ${\mathcal{S}}_{\Sigma }\in \mathcal{M}$).

Let us report the formula of the Fisher–Rao distance in these three cases:

• In the univariate case $\mathcal{N}\left(1\right)$, the Fisher–Rao distance between $N_1=N({\mu }_1,{\sigma }_1^2)$ and $N_2=N({\mu }_2,{\sigma }_2^2)$ can be derived from the hyperbolic distance [44] expressed in the Poincaré upper space since we have

  $$\mathrm{d}{s}_{\mathcal{N}}^2=g_{(\mu ,\sigma )}(\mathrm{d}\mu ,\mathrm{d}\sigma )=\frac{\mathrm{d}{\mu }^2+2\mathrm{d}{\sigma }^2}{{\sigma }^2}=2\frac{{\left(\frac{\mathrm{d}\mu }{\sqrt{2}}\right)}^2+\mathrm{d}{\sigma }^2}{{\sigma }^2}=2\frac{\mathrm{d}{x}^2+\mathrm{d}{y}^2}{y^2}=\mathrm{d}{s}_{\text{Poincaré}}^2,$$

  where $x=\frac{\mu }{\sqrt{2}}$ and $y=\sigma$. It follows that

  $${\rho }_{\mathcal{N}}(N_1,N_2)=\sqrt{2}{\rho }_{\text{Poincaré}}((x_1,y_1),(x_2,y_2))=\sqrt{2}{\rho }_{\text{Poincaré}}\left(\left(\frac{{\mu }_1}{\sqrt{2}},{\sigma }_1\right),\left(\frac{{\mu }_2}{\sqrt{2}},{\sigma }_2\right)\right).$$
  Thus, we have the following expression for the Fisher–Rao distance between univariate normal distributions:

  $${\rho }_{\mathcal{N}}(N_1,N_2)=\sqrt{2}log\left(\frac{1+\Delta ({\mu }_1,{\sigma }_1;{\mu }_2,{\sigma }_2)}{1-\Delta ({\mu }_1,{\sigma }_1;{\mu }_2,{\sigma }_2)}\right),$$

  (7)

  with

  $$\Delta (a,b;c,d)=\sqrt{\frac{{(c-a)}^2+2{(d-b)}^2}{{(c-a)}^2+2{(d+b)}^2}},(a,b,c,d)\in {\mathbb{R}}^4\backslash \left\{0\right\}.$$

  (8)
  In particular, we have

  –

  $\Delta (a,b;a,d)=\left|\frac{d-b}{d+b}\right|$ when $a=c$ (same mean),

  –

  $\Delta (a,b;c,b)=\sqrt{\frac{1}{1+8\frac{b^2}{{(c-a)}^2}}}$ when $b=d$ (same variance),

  –

  $\Delta (0,1;c,d)=\sqrt{\frac{c^2+2{(d-1)}^2}{c^2+2{(d+1)}^2}}$ when $a=0$ and $b=1$ (standard normal).

  In 1D, the affine-invariance property (Property 1) extends to function $\Delta$ as follows:

  $$\Delta ({\mu }_1,{\sigma }_1;{\mu }_2,{\sigma }_2)=\Delta \left(0,1;\frac{{\mu }_2-{\mu }_1}{{\sigma }_1},\frac{{\sigma }_2}{{\sigma }_1}\right)=\Delta \left(\frac{{\mu }_1-{\mu }_2}{{\sigma }_2},\frac{{\sigma }_1}{{\sigma }_2};0,1\right).$$
  Using one of the many identities between inverse hyperbolic functions (e.g., arctanh, arccosh, arcsinh), we can obtain an equivalent formula for Equation (7). For example, since $\mathrm{arctanh}\left(u\right):=\frac{1}{2}log\left(\frac{1+u}{1-u}\right)$ for $0<u<1$, we have equivalently:

  $${\rho }_{\mathcal{N}}(N_1,N_2)=2\sqrt{2}\mathrm{arctanh}(\Delta ({\mu }_1,{\sigma }_1;{\mu }_2,{\sigma }_2)).$$

  (9)
  The Fisher–Rao geodesics are semi-ellipses with centers located on the x-axis. See Appendix A.1 for the parametric equations of Fisher–Rao geodesics between univariate normal distributions. Figure 1 displays four univariate normal distributions with their pairwise geodesics and Fisher–Rao distances.
  

  Figure 1. Four univariate normal distributions $N_1=N(0,1)$, $N_2=N(3,1)$, $N_3=N(2,2.5)$, $N_4=N(0,2)$, and their pairwise full geodesics in gray and geodesics linking them in red. The Fisher–Rao distances are ${\rho }_{\mathcal{N}}(N_1,N_2)=2.6124\dots$, ${\rho }_{\mathcal{N}}(N_3,N_4)=0.9317\dots$, ${\rho }_{\mathcal{N}}(N_1,N_4)=0.9803\dots$, ${\rho }_{\mathcal{N}}(N_2,N_3)=1.4225\dots$${\rho }_{\mathcal{N}}(N_2,N_4)=2.1362\dots$, and ${\rho }_{\mathcal{N}}(N_1,N_3)=1.7334\dots$ The ellipses are Tissot indicatrices, which visualize the metric tensor $g_{\mathcal{N}}^{\mathrm{Fisher}}$ at grid positions.
  Using the identity $\mathrm{arctanh}\left(\frac{u^2-1}{u^2+1}\right)=\mathrm{arccosh}\left(\frac{1+u^2}{2u}\right)$ with $\mathrm{arccosh}\left(x\right):=log(x+\sqrt{x^2-1})$, we also have

  $${\rho }_{\mathcal{N}}(N_1,N_2)=2\sqrt{2}\mathrm{arccosh}\left(\frac{1}{\sqrt{(1-\Delta ({\mu }_1,{\sigma }_1;{\mu }_2,{\sigma }_2))(1+\Delta ({\mu }_1,{\sigma }_1;{\mu }_2,{\sigma }_2))}}\right),$$
  Since the inverse hyperbolic cosecant (CSC) function is defined by $\mathrm{arccsch}\left(u\right):=\mathrm{arccosh}(1/u)$, we further obtain

  $${\rho }_{\mathcal{N}}(N_1,N_2)=2\sqrt{2}\mathrm{arccsch}\left(\sqrt{(1-\Delta ({\mu }_1,{\sigma }_1;{\mu }_2,{\sigma }_2))(1+\Delta ({\mu }_1,{\sigma }_1;{\mu }_2,{\sigma }_2))}\right),$$
  We can also write

  $${\rho }_{\mathcal{N}}(N_1,N_2)=\sqrt{2}\mathrm{arccosh}\left(1+\frac{{({\mu }_2-{\mu }_1)}^2+2{({\sigma }_2-{\sigma }_1)}^2}{4{\sigma }_1{\sigma }_2}\right)$$
  Thus, using the many-conversions formula between inverse hyperbolic functions, we obtain many equivalent different formulas of the Fisher–Rao distance, which are used in the literature.
• In the second case, the Fisher–Rao distance between $N_1=N(\mu ,{\Sigma }_1)$ and $N_2=N(\mu ,{\Sigma }_2)$ has been reported in [6,7,45,46,47]:

  $$\begin{array}{ccc}\hfill {\rho }_{{\mathcal{N}}_{\mu }}(N_1,N_2)& =& \sqrt{\frac{1}{2}\sum _{i=1}^d{log}^2{\lambda }_i\left({\Sigma }_1^{-1}{\Sigma }_2\right)},\hfill \end{array}$$

  (10)

  $$\begin{array}{ccc}\hfill & =& {\rho }_{{\mathcal{N}}_{\mu }}({\Sigma }_1,{\Sigma }_2),\hfill \end{array}$$

  (11)

  where ${\lambda }_i\left(M\right)$ denotes the i-th generalized largest eigenvalue of matrix M, where the generalized eigenvalues are solutions of the equation $|{\Sigma }_1-\lambda {\Sigma }_2|=0$. Let us notice that ${\rho }_{{\mathcal{N}}_{\mu }}((\mu ,{\Sigma }_1),(\mu ,{\Sigma }_2))={\rho }_{{\mathcal{N}}_{\mu }}((\mu ,{\Sigma }_1^{-1}),(\mu ,{\Sigma }_2^{-1}))$ since ${\lambda }_i\left({\Sigma }_2^{-1}{\Sigma }_1\right)=\frac{1}{{\lambda }_i\left({\Sigma }_1^{-1}{\Sigma }_2\right)}$ and ${log}^2{\lambda }_i\left({\Sigma }_2^{-1}{\Sigma }_1\right)={(-log{\lambda }_i\left({\Sigma }_1^{-1}{\Sigma }_2\right))}^2={log}^2{\lambda }_i\left({\Sigma }_1^{-1}{\Sigma }_2\right)$. Matrix ${\Sigma }_1^{-1}{\Sigma }_2$ may not be SPD and thus the ${\lambda }_i$’s are generalized eigenvalues. We may consider instead the SPD matrix ${\Sigma }_1^{-\frac{1}{2}}{\Sigma }_2{\Sigma }_1^{-\frac{1}{2}}$ which is SPD and such that ${\lambda }_i\left({\Sigma }_1^{-1}{\Sigma }_2\right)={\lambda }_i\left({\Sigma }_1^{-\frac{1}{2}}{\Sigma }_2{\Sigma }_1^{-\frac{1}{2}}\right)$. The Fisher–Rao distance of Equation (11) can be equivalently written [48] as

  $${\rho }_{{\mathcal{N}}_{\mu }}(N_1,N_2)=\frac{1}{\sqrt{2}}{∥\mathrm{Log}\left({\Sigma }_1^{-\frac{1}{2}}{\Sigma }_2{\Sigma }_1^{-\frac{1}{2}}\right)∥}_F,$$

  where $\mathrm{Log}\left(M\right)$ is the matrix logarithm (unique when M is SPD) and ${\parallel M\parallel }_F=\sqrt{{\sum }_{i,j}{M}_{i,j}^2}=\sqrt{\mathrm{tr}\left(M{M}^{\top }\right)}$ is the matrix Fröbenius norm. This metric distance between SPD matrices although first studied by Siegel [45] in 1964 was rediscovered and analyzed recently in [49] (2003). Let ${\rho }_{\mathrm{SPD}}(P_1,P_2)=\sqrt{{\sum }_{i=1}^d{log}^2{\lambda }_i\left(P_1^{-1}{P}_2\right)}$ so that ${\rho }_{{\mathcal{N}}_{\mu }}(N(\mu ,P_1),N(\mu ,P_2))=\frac{1}{\sqrt{2}}{\rho }_{\mathrm{SPD}}(P_1,P_2)$.
  The Riemannian SPD distance ${\rho }_{\mathrm{SPD}}$ enjoys the following well-known invariance properties:

  –

  Invariance by congruence transformation:

  $$\forall X\in \mathrm{GL}\left(d\right),{\rho }_{\mathrm{SPD}}(X{P}_1{X}^{\top },X{P}_2{X}^{\top })={\rho }_{\mathrm{SPD}}(P_1,P_2),$$

  (12)

  –

  Invariance by inversion:

  $$\forall P_1,P_2\in \mathbb{P}\left(d\right),\rho (P_1^{-1},P_2^{-1})={\rho }_{\mathrm{SPD}}(P_1,P_2).$$

  Let $P_1=L_1{L}_1^{\top }$ be the Cholesky decomposition (unique when $P_1\succ 0$). Then apply the congruence invariance for $X=L_1^{-1}$:

  $${\rho }_{\mathrm{SPD}}(P_1,P_2)={\rho }_{\mathrm{SPD}}(L_1^{-1}{P}_1{(L_1^{-1})}^{\top },L_1^{-1}{P}_2{(L_1^{-1})}^{\top })={\rho }_{\mathrm{SPD}}(I,L_1^{-1}{P}_2{(L_1^{-1})}^{\top }).$$

  (13)

  We can also consider the factorization $P_1=S_1{S}_1$ where $S_1=P_1^{\frac{1}{2}}$ is the unique symmetric square root matrix [50]. Then we have

  $${\rho }_{\mathrm{SPD}}(P_1,P_2)={\rho }_{\mathrm{SPD}}(S_1^{-1}{P}_1{(S_1^{-1})}^{\top },S_1^{-1}{P}_2{(S_1^{-1})}^{\top })={\rho }_{\mathrm{SPD}}(I,S_1^{-1}{P}_2{(S_1^{-1})}^{\top }).$$

• The Fisher–Rao distance between $N_1=N({\mu }_1,\Sigma )$ and $N_2=N({\mu }_2,\Sigma )$ has been reported in closed form [42] (Proposition 3). The method is described with full details in Appendix B. We present a simpler scheme based on the inverse ${\Sigma }^{-\frac{1}{2}}$ of the symmetric square root factorization [50] of $\Sigma ={\Sigma }^{\frac{1}{2}}{\Sigma }^{\frac{1}{2}}$ (ith ${\left({\Sigma }^{-\frac{1}{2}}\right)}^{\top }={\Sigma }^{-\frac{1}{2}}$). Let us use the affine-invariance property of the Fisher–Rao distance under the affine transformation ${\Sigma }^{-\frac{1}{2}}$ and then apply affine invariance under translation as follows:

  $$\begin{array}{ccc}\hfill {\rho }_{\mathcal{N}}(N({\mu }_1,\Sigma ),N({\mu }_2,\Sigma ))& =& {\rho }_{\mathcal{N}}(N({\Sigma }^{-\frac{1}{2}}{\mu }_1,{\Sigma }^{-\frac{1}{2}}\Sigma {\Sigma }^{-\frac{1}{2}}),N({\Sigma }^{-\frac{1}{2}}{\mu }_2,{\Sigma }^{-\frac{1}{2}}\Sigma {\Sigma }^{-\frac{1}{2}})),\hfill \\ & =& {\rho }_{\mathcal{N}}(N(0,I),N({\Sigma }^{-\frac{1}{2}}({\mu }_2-{\mu }_1),I)),\hfill \\ & =& {\rho }_{\mathcal{N}}(N(0,1),N(\parallel {\Sigma }^{-\frac{1}{2}}({\mu }_2-{\mu }_1){\parallel }_2,1)).\hfill \end{array}$$
  The right-hand side Fisher–Rao distance is computed from Equation (7) and justified by the method [42] (Proposition 3) described in Appendix B using a rotation matrix R with $R{R}^{\top }=I$ so that

  $$\begin{array}{ccc}\hfill {\rho }_{\mathcal{N}}(N(0,I),N({\Sigma }^{-\frac{1}{2}}({\mu }_2-{\mu }_1),I))& =& {\rho }_{\mathcal{N}}(N(0,I),N(R{\Sigma }^{-\frac{1}{2}}({\mu }_2-{\mu }_1),RI{R}^{\top })),\hfill \\ & =& {\rho }_{\mathcal{N}}(N(0,I),\parallel {\Sigma }^{-\frac{1}{2}}({\mu }_2-{\mu }_1){\parallel }_2,I)).\hfill \end{array}$$
  Then we apply the formula of Equation (23) of [42]. Section 1.5 shall report a simpler closed-form formula by proving that the Fisher–Rao distance between $N({\mu }_1,\Sigma )$ and $N({\mu }_2,\Sigma )$ is a scalar function of their Mahalanobis distance [51] using the algebraic method of maximal invariants [52].

1.3. Fisher–Rao Distance: Totally versus Non-Totally Geodesic Submanifolds

Consider ${\mathcal{N}}^{\prime }=\{N\left(\lambda \right):{\lambda }^{\prime }\in {\Lambda }^{\prime }\}\subset \mathcal{N}$ a statistical submodel of the MVN statistical model $\mathcal{N}$. Using the Fisher information matrix $I_{{\lambda }^{\prime }}\left({\lambda }^{\prime }\right)$, we obtain the intrinsic Fisher–Rao manifold ${\mathcal{M}}^{\prime }={\mathcal{M}}_{{\mathcal{N}}^{\prime }}$. We may also consider ${\mathcal{M}}^{\prime }$ to be an embedded submanifold of $\mathcal{M}$. Let us write ${\mathcal{S}}^{\prime }={\mathcal{S}}_{{\mathcal{N}}^{\prime }}\subset \mathcal{M}$ the embedded submanifold.

A totally geodesic submanifold ${\mathcal{S}}^{\prime }\subset \mathcal{M}$ is such that the geodesics ${\gamma }_{{\mathcal{M}}^{\prime }}(N_1^{\prime },N_2^{\prime };t)$ fully stay in ${\mathcal{M}}^{\prime }$ for any pair of points $N_1^{\prime },N_2^{\prime }\in {\mathcal{N}}^{\prime }$. For example, the submanifold ${\mathcal{M}}_{\mu }=\{N(\mu ,\Sigma ):\Sigma \in \mathbb{P}\left(d\right)\}\subset \mathcal{M}$ of MVNs with fixed mean $\mu$ is a totally geodesic submanifold [53] of $\mathcal{M}$ but the submanifold ${\mathcal{M}}_{\Sigma }=\{N(\mu ,\Sigma ):\mu \in {\mathbb{R}}^d\}\subset \mathcal{M}$ of MVNs sharing the same covariance matrix $\Sigma$ is not totally geodesic. When an embedded submanifold $\mathcal{S}\subset \mathcal{M}$ is totally geodesic, we always have ${\rho }_{\mathcal{M}}(N_1,N_2)={\rho }_{\mathcal{S}}(N_1,N_2)$. Thus, we have ${\rho }_{\mathcal{N}}(N(\mu ,{\Sigma }_1),N(\mu ,{\Sigma }_2))={\rho }_{\mathrm{SPD}}({\Sigma }_1,{\Sigma }_2)$. However, when an embedded submanifold $\mathcal{S}\subset \mathcal{M}$ is not totally geodesic, we have ${\rho }_{\mathcal{M}}(N_1,N_2)\le {\rho }_{\mathcal{S}}(N_1,N_2)$ because the Riemannian geodesic length in $\mathcal{S}$ is necessarily longer or equal than the Riemannian geodesic length in $\mathcal{M}$. The merit to consider submanifolds is to be able to calculate in closed form the Fisher–Rao distance which may then provide an upper bound on the Fisher–Rao distance for the full statistical model. For example, consider $N_1=N({\mu }_1,\Sigma )$ and $N_2=N({\mu }_2,\Sigma )$ in ${\mathcal{M}}_{\Sigma }$, a non-totally geodesic submanifold. The Rao distance between $N_1$ and $N_2$ in $\mathcal{M}$ is upper bounded by the Riemannian distance in ${\mathcal{M}}_{\Sigma }$ (with line element $\mathrm{d}{s}_{\Sigma }^2=\mathrm{d}{\mu }^{\top }{\Sigma }^{-1}\mathrm{d}\mu$) which corresponds to the Mahalanobis distance [10,51]${\Delta }_{\Sigma }({\mu }_1,{\mu }_2)$:

$${\rho }_{{\mathcal{M}}_{\mu }}(N_1,N_2)\le {\Delta }_{\Sigma }({\mu }_1,{\mu }_2):=\sqrt{{({\mu }_2-{\mu }_1)}^{\top }{\Sigma }^{-1}({\mu }_2-{\mu }_1)}.$$

(14)

The Mahalanobis distance can be interpreted as the Euclidean distance $D_E(p,q)={\Delta }_I(p,q)=\sqrt{{(p-q)}^{\top }(p-q)}$ (where I denotes the identity matrix) after an affine transformation: Let $\Sigma =L{L}^{\top }=U^{\top }U$ be the Cholesky decomposition of $\Sigma \gg 0$ with L a lower triangular matrix or $U=L^{\top }$ an upper triangular matrix. Then we have

$$\begin{array}{ccc}\hfill {\Delta }_{\Sigma }({\mu }_1,{\mu }_2)& =& \sqrt{{({\mu }_2-{\mu }_1)}^{\top }{\left(L^{\top }\right)}^{-1}{L}^{-1}({\mu }_2-{\mu }_1)},\hfill \\ & =& \parallel {\Sigma }^{-\frac{1}{2}}({\mu }_2-{\mu }_1){\parallel }_2,\hfill \\ & =& {\Delta }_I(L^{-1}{\mu }_1,L^{-1}{\mu }_2)=D_E(L^{-1}{\mu }_1,L^{-1}{\mu }_2),\hfill \end{array}$$

where ${\parallel ·\parallel }_2$ denotes the vector ${\ell }_2$-norm.

The Rao distance ${\rho }_{\Sigma }$ of Equation (A1) between two MVNs with fixed covariance matrix emanates from the property that the submanifold ${\mathcal{M}}_{\left[v\right],\Sigma }=\{N(av,\Sigma ):a\in \mathbb{R}\}$ is totally geodesic [54].

Let us emphasize that for a submanifold $\mathcal{S}\subset \mathcal{M}$ to be totally geodesic or not depend on the underlying metric in $\mathcal{M}$. The same subset ${\mathcal{N}}^{\prime }\subset \mathcal{N}$ with $\mathcal{N}$ equipped with two different metrics $g_1$ and $g_2$ can be totally geodesic regarding $g_1$ and non-totally geodesic regarding $g_2$. See Remark 3 for such an example.

In general, using the triangle inequality of the Riemannian metric distance ${\rho }_{\mathcal{N}}$, we can upper bound ${\rho }_{\mathcal{N}}(N_1,N_2)$ with $N_1=({\mu }_1,{\Sigma }_1)$ and $N_1=({\mu }_2,{\Sigma }_2)$ as follows:

$$\begin{array}{ccc}\hfill {\rho }_{\mathcal{N}}(N_1,N_2)& \le & {\rho }_{{\mathcal{M}}_{{\mu }_1}}(N_1,N_{12})+{\rho }_{{\mathcal{M}}_{{\Sigma }_2}}(N_{12},N_2),\hfill \\ & \le & {\rho }_{{\mathcal{M}}_{{\Sigma }_1}}(N_1,N_{21})+{\rho }_{{\mathcal{M}}_{{\mu }_2}}(N_{21},N_2),\hfill \end{array}$$

where $N_{12}=({\mu }_1,{\Sigma }_2)$ and $N_{21}=N({\mu }_2,{\Sigma }_1)$. See Figure 2 for an illustration of the Fisher–Rao geodesic triangle $▵N_1,N_2,N_{12}$. Furthermore, since ${\rho }_{{\mathcal{N}}_{{\Sigma }_1}}(N_1,N_{21})\le {\Delta }_{{\Sigma }_1}({\mu }_1,{\mu }_2)$ and ${\rho }_{{\mathcal{N}}_{{\Sigma }_2}}(N_{12},N_2)\le {\Delta }_{{\Sigma }_2}({\mu }_1,{\mu }_2)$, we obtain the following upper bound on the Rao distance between MVNs:

$${\rho }_{\mathcal{N}}(N_1,N_2)\le {\rho }_{\mathcal{P}}({\Sigma }_1,{\Sigma }_2)+min\{{\Delta }_{{\Sigma }_1}({\mu }_1,{\mu }_2),{\Delta }_{{\Sigma }_2}({\mu }_1,{\mu }_2)\}.$$

(15)

See also [55].

Figure 2. The submanifolds ${\mathcal{N}}_{\Sigma }$ are not totally geodesic (i.e., ${\rho }_{\mathcal{N}}(N_1^{\prime },N_2^{\prime })$ is upper bounded by their Mahalanobis distance) but the submanifolds ${\mathcal{N}}_{\mu }$ are totally geodesic. Using the triangle inequality of the Riemannian metric distance ${\rho }_{\mathcal{N}}$, we can upper bound ${\rho }_{\mathcal{N}}(N_1,N_2)$.

In general, the difficulty with calculating the Fisher–Rao distance comes from the fact that

• we do not know the Fisher–Rao geodesics with boundary value conditions (BVP) in closed form but the geodesics with initial value conditions [48] (IVP) are known explicitly using the natural parameters $({\Sigma }^{-1}\mu ,{\Sigma }^{-1})$ of MVNs,
• we must integrate the line element $\mathrm{d}{s}_{\mathcal{N}}$ along the geodesic.

As we shall see in Section 3.1, the above first problem is much harder to solve than the second problem which can be easily approximated by discretizing the curve. The lack of a closed-form formula and fast and good approximations for ${\rho }_{\mathcal{N}}$ between MVNs is a current limiting factor for its use in applications. Indeed, many applications (e.g., [56,57]) consider the restricted case of the Rao distance between zero-centered MVNs which have closed form (distance of Equation (11) in the SPD cone). The SPD cone is a symmetric Hadamard manifold, and its isometries have been fully studied and classified in [58] (Section 4). The Fisher–Rao geometry of zero-centered generalized MVNs was recently studied in [59].

1.4. Contributions and Paper Outline

The main contribution of this paper is to propose an approximation of ${\rho }_{\mathcal{N}}$ based on Calvo and Oller’s embedding [19] (C&O for short) and report its experimental performance. First, we concisely recall C&O’s family of embeddings $f_{\beta }$ of $\mathcal{N}\left(d\right)$ as submanifolds ${\overline{\mathcal{N}}}_{\beta }$ of $\mathcal{P}(d+1)$ in Section 2. Next, we present our approximation technique in Section 3 which differs from the usual geodesic shooting approach [39], and report experimental results. Finally, we study some information–geometric properties [1] of the isometric embedding in Section 5 such as the fact that it preserves mixture geodesics (embedded C&O submanifold is autoparallel with respect to the mixture affine connection) but not exponential geodesics. Moreover, we prove that the Fisher–Rao distance between multivariate normal distributions sharing the same covariance matrix is a scalar function of their Mahalanobis distance in Section 1.5 using the framework of Eaton [52] of maximal invariants.

1.5. A Closed-Form Formula for the Fisher–Rao Distance between Normal Distributions Sharing the Same Covariance Matrix

Consider the Fisher–Rao distance between $N_1=({\mu }_1,\Sigma )$ and $N_1=({\mu }_2,\Sigma )$ for a fixed covariance matrix $\Sigma$ and the translation action $a.\mu :=\mu +a$ of the translation group ${\mathbb{R}}^d$ (a subgroup of the affine group). Both the Fisher–Rao distance and the Mahalanobis distance are invariant under translations:

$${\rho }_{\mathcal{N}}(({\mu }_1+a,\Sigma ),({\mu }_2+a,\Sigma ))={\rho }_{\mathcal{N}}(({\mu }_1,\Sigma ),({\mu }_2,\Sigma )),{\Delta }_{\Sigma }({\mu }_1+a,{\mu }_2+a)={\Delta }_{\Sigma }({\mu }_1,{\mu }_2).$$

To prove that ${\rho }_{\mathcal{N}}(({\mu }_1,\Sigma ),({\mu }_2,\Sigma ))=h_{\mathrm{FR}}\left({\Delta }_{\Sigma }({\mu }_1,{\mu }_2)\right)$ for a scalar function $h_{\mathrm{FR}}(·)$, we shall prove that the Mahalanobis distance is a maximal invariant, and use the framework of maximal invariants of Eaton [52] (Chapter 2) who proved that any other invariant function is necessarily a function of a maximal invariant, i.e., a function of the Mahalanobis distance in our case.

The Mahalanobis distance is a maximal invariant because we can write ${\Delta }_{\Sigma }({\mu }_1,{\mu }_2)={\Delta }_1(0,{\Delta }_{\Sigma }({\mu }_1,{\mu }_2))$ and when ${\Delta }_{\Sigma }({\mu }_1,{\mu }_2)={\Delta }_{\Sigma }({\mu }_1^{\prime },{\mu }_2^{\prime })$ in 1D there exists $a\in \mathbb{R}$ such that $({\mu }_1+a,{\mu }_2+a)=({\mu }_1^{\prime },{\mu }_2^{\prime })$. We must prove equivalently that when $|m_1-m_2|=|m_1^{\prime }-m_2^{\prime }|$ that there exists $a\in \mathbb{R}$ such that $(m_1+a,m_2+a)=(m_1^{\prime },m_2^{\prime })$. Assume without loss of generality that $m_1\ge m_2$. When $m_1-m_2=m_1^{\prime }-m_2^{\prime }$, there exists $a=m_1^{\prime }-m_1$ so that $m_1^{\prime }=a.m_1=m_1+a$ and $m_2^{\prime }=a.m_2=m_2+a$ with $m_1^{\prime }-m_2^{\prime }=m_1-m_2$. Thus, using Eaton’s theorem [52], there exists a scalar function $h_{\mathrm{FR}}$ such that ${\rho }_{\mathcal{N}}(({\mu }_1,\Sigma ),({\mu }_2,\Sigma ))=h_{\mathrm{FR}}\left({\Delta }_{\Sigma }({\mu }_1,{\mu }_2)\right)$.

To find explicitly the scalar function $h_{\mathrm{FR}}(·)$, let us consider the univariate case of normal distributions for which the Fisher–Rao distance is given in closed form in Equation (7). In that case, the univariate Mahalanobis distance is ${\Delta }_{{\sigma }^2}({\mu }_1,{\mu }_2)=\sqrt{({\mu }_2-{\mu }_1){\left({\sigma }^2\right)}^{-1}({\mu }_2-{\mu }_1)}=\frac{|{\mu }_2-{\mu }_1|}{\sigma }$ and we can write formula of Equation (7) as $h_{\mathrm{FR}}\left({\Delta }_{{\sigma }^2}({\mu }_1,{\mu }_2)\right)$ with

$$\begin{array}{ccc}\hfill h_{\mathrm{FR}}\left(u\right)& =& \sqrt{2}log\left(\frac{\sqrt{8+u^2}+u}{\sqrt{8+u^2}-u}\right),\hfill \end{array}$$

(16)

$$\begin{array}{ccc}\hspace{1em}\hspace{1em} & =& \sqrt{2}\mathrm{arccosh}\left(1+\frac{1}{4}{u}^2\right),\hfill \end{array}$$

(17)

using the identities

$$log\left(x\right)=\mathrm{arccosh}\left(\frac{1+x^2}{2x}\right)=\mathrm{arctanh}\left(\frac{x^2-1}{1+x^2}\right),x>1,$$

where $\mathrm{arctanh}\left(u\right)=\frac{1}{2}log\frac{1+u}{1-u}$.

Proposition 1. 

The Fisher–Rao distance ${\rho }_{\mathcal{N}}(({\mu }_1,\Sigma ),({\mu }_2,\Sigma ))$ between two MVNs with same covariance matrix is

$$\begin{array}{ccc}\hfill {\rho }_{\mathcal{N}}(({\mu }_1,\Sigma ),({\mu }_2,\Sigma ))& =& {\rho }_{\mathcal{N}}((0,1),({\Delta }_{\Sigma }({\mu }_1,{\mu }_2),1)),\hfill \end{array}$$

(18)

$$\begin{array}{ccc}& =& \sqrt{2}log\left(\frac{\sqrt{8+{\Delta }_{\Sigma }^2({\mu }_1,{\mu }_2)}+{\Delta }_{\Sigma }({\mu }_1,{\mu }_2)}{\sqrt{8+{\Delta }_{\Sigma }^2({\mu }_1,{\mu }_2)}-{\Delta }_{\Sigma }({\mu }_1,{\mu }_2)}\right),\hfill \end{array}$$

(19)

$$\begin{array}{ccc}& =& \sqrt{2}\mathrm{arccosh}\left(1+\frac{1}{4}{\Delta }_{\Sigma }^2({\mu }_1,{\mu }_2)\right),\hfill \end{array}$$

(20)

where ${\Delta }_{\Sigma }({\mu }_1,{\mu }_2)=\sqrt{{({\mu }_2-{\mu }_1)}^{\top }{\Sigma }^{-1}({\mu }_2-{\mu }_1)}$ is the Mahalanobis distance.

Indeed, notice that the d-variate Mahalanobis distance ${\Delta }_{\Sigma }({\mu }_1,{\mu }_2)$ can be interpreted as a univariate Mahalanobis distance between the standard normal distribution $N(0,1)$ and $N({\Delta }_{\Sigma }({\mu }_1,{\mu }_2),1)$:

$${\Delta }_{\Sigma }({\mu }_1,{\mu }_2)={\Delta }_1(0,{\Delta }_{\Sigma }({\mu }_1,{\mu }_2)).$$

Thus, we have ${\rho }_{\mathcal{N}}(({\mu }_1,\Sigma ),({\mu }_2,\Sigma ))={\rho }_{\mathcal{N}}((0,1),({\Delta }_{\Sigma }({\mu }_1,{\mu }_2),1))$, where the right-hand-side term is the univariate Fisher–Rao distance of Equation (7). Let us notice that the square length element on ${\mathcal{M}}_{\Sigma }$ is $\mathrm{d}{s}^2=\mathrm{d}{\mu }^{\top }{\Sigma }^{-1}\mathrm{d}\mu ={\Delta }_{\Sigma }^2(\mu ,\mu +\mathrm{d}\mu )$. This result can be extended to elliptical distributions [12] (Theorem 1).

Let us corroborate this result by checking the formula of Equation (1) with two examples in the literature: In [38] (Figure 4), we Fisher–Rao distance between $N_1=(0,I)$ and $N_2=\left(\left[\begin{array}{c}\frac{1}{2}\hfill \\ \frac{1}{2}\hfill \end{array}\right],I\right)$ is studied. We find ${\rho }_{\mathcal{N}}(N_1,N_2)=0.69994085$ in accordance with their result shown in Figure 4. The second example is Example 1 of [42] (p. 11) with $N_1=\left(\left[\begin{array}{c}-1\hfill \\ 0\hfill \end{array}\right],\Sigma \right)$ and $N_2=\left(\left[\begin{array}{c}6\hfill \\ 3\hfill \end{array}\right],\Sigma \right)$ for $\Sigma =\left[\begin{array}{cc}1.1\hfill & 0.9\hfill \\ 0.9\hfill & 1.1\hfill \end{array}\right]$. Formula of Equation (18) yields the Fisher–Rao distance $5.006483034546878$ in accordance with [42] which reports $5.00648$.

Similarly, the statistical Ali–Silvey–Csiszár f-divergences [60,61]

$$I_f[p_{({\mu }_1,\Sigma )}:p_{({\mu }_2,\Sigma )}]={\int }_{{\mathbb{R}}^d}{p}_{({\mu }_1,\Sigma )}\left(x\right)f\left(\frac{p_{({\mu }_2,\Sigma )}}{p_{({\mu }_1,\Sigma )}}\right)\mathrm{d}x,$$

between two MVNs sharing the same covariance matrix are increasing functions of the Mahalanobis distance because the f-divergences between two MVNs sharing the same covariance matrix are invariant under the action of the translation group [62]. Thus, we have $I_f[p_{({\mu }_1,\Sigma )}:p_{({\mu }_2,\Sigma )}]=h_f\left({\Delta }_{\Sigma }({\mu }_1,{\mu }_2)\right)$. Since ${\Delta }_{\Sigma }({\mu }_1,{\mu }_2)={\Delta }_1(0,{\Delta }_{\Sigma }({\mu }_1,{\mu }_2))$, we thus have

$$I_f[p_{({\mu }_1,\Sigma )}:p_{({\mu }_2,\Sigma )}]=h_f({\Delta }_1(0,{\Delta }_{\Sigma }({\mu }_1,{\mu }_2))=I_f[p_{(0,1)}:p_{({\Delta }_{\Sigma }({\mu }_1,{\mu }_2),1)}],$$

where the right-hand side f-divergence is between univariate normal distributions. See Table 2 of [62] for some explicit functions $h_f$.

2. Calvo and Oller’s Family of Diffeomorphic Embeddings

Calvo and Oller [19,32] noticed that we can embed the space of normal distributions in $\mathcal{P}(d+1)$ by using the following mapping:

$$f_{\beta }\left(N\right)=f_{\beta }(\mu ,\Sigma )=\left[\begin{array}{cc}\Sigma +\beta \mu {\mu }^{\top }\hfill & \beta \mu \hfill \\ \beta {\mu }^{\top }\hfill & \beta \hfill \end{array}\right]\in \mathcal{P}(d+1),$$

(21)

where $\beta \in {\mathbb{R}}_{>0}$ and $N=N(\mu ,\Sigma )$. Notice that since the dimension of $\mathcal{P}(d+1)$ is $\frac{(d+1)(d+2)}{2}$, we only use $\frac{(d+1)(d+2)}{2}-\frac{d(d+3)}{2}=1$ extra dimension for embedding $\mathcal{N}\left(d\right)$ into $\mathcal{P}(d+1)$. By foliating $\mathbb{P}={\mathbb{R}}_{>0}\times {\mathbb{P}}_c$ where ${\mathbb{P}}_c=\{P\in \mathbb{P}:|P|=c\}$ denotes the subsets of $\mathbb{P}$ with determinant c, we obtain the following Riemannian Calvo and Oller metric on the SPD cone:

$$\begin{array}{ccc}\hfill \mathrm{d}{s}_{\mathrm{CO}}^2& =& \frac{1}{2}\mathrm{tr}\left({\left(f^{-1}(\mu ,\Sigma )\mathrm{d}f(\mu ,\Sigma )\right)}^2\right),\hfill \\ & =& \frac{1}{2}{\left(\frac{\mathrm{d}\beta }{\beta }\right)}^2+\beta \mathrm{d}{\mu }^{\top }{\Sigma }^{-1}\mathrm{d}\mu +\frac{1}{2}\mathrm{tr}\left({\left({\Sigma }^{-1}\mathrm{d}\Sigma \right)}^2\right).\hfill \end{array}$$

Let

$${\overline{\mathcal{N}}}_{\beta }\left(d\right)=\left\{\overline{P}=f_{\beta }(\mu ,\Sigma ):(\mu ,\Sigma )\in \mathcal{N}\left(d\right)={\mathbb{R}}^d\times \mathcal{P}\left(d\right)\right\}$$

denote the submanifold of $\mathcal{P}(d+1)$ of codimension 1, and $\overline{\mathcal{N}}={\overline{\mathcal{N}}}_1$ (i.e., $\beta =1$). The family of mappings $f_{\beta }$ provides diffeomorphisms between $\mathcal{N}\left(d\right)$ and ${\overline{\mathcal{N}}}_{\beta }\left(d\right)$. Let $f_{\beta }^{-1}\left(\overline{P}\right)=({\mu }_{\overline{P}},{\Sigma }_{\overline{P}})$ denote the inverse mapping for $\overline{P}\in {\overline{\mathcal{N}}}_{\beta }\left(d\right)$, and let $f=f_1$ (i.e., $\beta =1$):

$$f\left(N\right)=f(\mu ,\Sigma )=\left[\begin{array}{cc}\Sigma +\mu {\mu }^{\top }\hfill & \mu \hfill \\ {\mu }^{\top }\hfill & 1\hfill \end{array}\right].$$

By equipping the cone $\mathcal{P}(d+1)$ by the trace metric [63,64] (also called the affine invariant Riemannian metric, AIRM) scaled by $\frac{1}{2}$:

$$g_P^{\mathrm{trace}}(P_1,P_2):=\mathrm{tr}\left(P^{-1}{P}_1{P}^{-1}{P}_2\right)$$

(yielding the squared line element $\mathrm{d}{s}_{\mathcal{P}}^2=\frac{1}{2}\mathrm{tr}({(P\mathrm{d}P)}^2)$), Calvo and Oller [19] proved that $\overline{\mathcal{N}}(d)$ is isometric to $\mathcal{N}(d)$ (i.e., the Riemannian metric of $\mathcal{P}(d+1)$ restricted to $\mathcal{N}(d)$ coincides with the Riemannian metric of $\mathcal{N}(d)$ induced by f) but $\overline{\mathcal{N}}(d)$ is not totally geodesic (i.e., the geodesics ${\gamma }_{\mathcal{P}}({\overline{P}}_1,{\overline{P}}_2;t)$ for ${\overline{P}}_1=f(N_1),{\overline{P}}_2=f(N_2)\in \overline{\mathcal{N}}(d)$ leaves the embedded normal submanifold $\overline{\mathcal{N}}(d))$. Please note that $g_P^{\mathrm{trace}}$ can be interpreted as the Fisher metric for the family ${\mathcal{N}}_0$ of 0-centered normal distributions. Thus, we have $(\mathcal{N}(d),g^{\mathrm{Fisher}})\hookrightarrow (\mathcal{P}(d+1),g^{\mathrm{trace}})$, and the following diagram between parameter spaces and corresponding distributions:

$$\begin{array}{ccc}\mathcal{N}(d)& \hookrightarrow & {\mathcal{N}}_0(d+1)\\ \updownarrow & & \updownarrow \\ \Lambda (d)& \hookrightarrow & \mathbb{P}(d+1)\end{array}$$

Remark 2. 

The trace metric was first studied by Siegel [45,65] using the wider scope of complex symmetric matrices with positive–definite imaginary parts generalizing the Poincaré upper half-plane (see Appendix D).

We omit to specify the dimensions and write for short $\mathcal{N}$, $\overline{\mathcal{N}}$, and $\mathcal{P}$ when clear from the context. Thus, C&O proposed to use the embedding $f=f_1$ to give a lower bound ${\rho }_{\mathrm{CO}}$ of the Fisher–Rao distance ${\rho }_{\mathcal{N}}$ between normals:

$${\mathrm{LC}}_{\mathrm{CO}}:{\rho }_{\mathcal{N}}(N_1,N_2)\ge {\rho }_{\mathrm{CO}}(\underset{{\overline{P}}_1}{\underbrace{f({\mu }_1,{\Sigma }_1)}},\underset{{\overline{P}}_2}{\underbrace{f({\mu }_2,{\Sigma }_2)}})=\sqrt{\frac{1}{2}\sum _{i=1}^{d+1}{log}^2{\lambda }_i\left({\overline{P}}_1^{-1}{\overline{P}}_2\right)}.$$

(22)

We let ${\rho }_{\mathrm{CO}}(N_1,N_2)={\rho }_{\mathrm{CO}}(f\left(N_1\right),f\left(N_2\right))$. The ${\rho }_{\mathrm{CO}}$ distance is invariant under affine transformations such as the Fisher–Rao distance of Property 1:

Property 2 

(affine invariance of C&O distance [19]). For all $A\in \mathrm{GL}\left(d\right),a\in {\mathbb{R}}^d$, we have ${\rho }_{\mathrm{CO}}((A{\mu }_1+a,A{\Sigma }_1{A}^{\top }),(A{\mu }_2+a,A{\Sigma }_2{A}^{\top }))={\rho }_{\mathrm{CO}}(N({\mu }_1,{\Sigma }_1),N({\mu }_2,{\Sigma }_2))$.

When ${\Sigma }_1={\Sigma }_2=\Sigma$, we have $|{\overline{P}}_1|=|{\overline{P}}_2|=|\Sigma |$. Since the Riemannian geodesics ${\gamma }_{\mathbb{P}}(P_1,P_2;t)$ in the SPD cone are given by ${\gamma }_{\mathbb{P}}(P_1,P_2;t)=P_1^{\frac{1}{2}}{\left(P_1^{-\frac{1}{2}}{P}_2{P}_1^{-\frac{1}{2}}\right)}^t{P}_1^{\frac{1}{2}}$ [66] (also written ${\gamma }_{\mathrm{SPD}}(P_1,P_2;t)$), we have $|{\gamma }_{\mathbb{P}}(P_1,P_2;t)|=|\Sigma |$. Although the submanifold ${\mathbb{P}}_c=\{P\in \mathbb{P}:|P|=c\}$ is totally geodesic with respect to the trace metric, it is not totally geodesic with respect to $\frac{1}{2}\mathrm{tr}({(\overline{P}\mathrm{d}\overline{P})}^2)$. Thus, although ${\gamma }_{\mathbb{P}}(P_1,P_2)\in \overline{\mathcal{N}}$, it does not correspond to the embedded MVN geodesics with respect to the Fisher metric. The C&O distance between two MVNs $N({\mu }_1,\Sigma )$ and $N({\mu }_2,\Sigma )$ sharing the same covariance matrix [19] is

$${\rho }_{\mathrm{CO}}(N({\mu }_1,\Sigma ),N({\mu }_2,\Sigma ))=\mathrm{arccosh}\left(1+\frac{1}{2}{\Delta }_{\Sigma }^2({\mu }_1,{\mu }_2)\right),$$

(23)

where $\mathrm{arccosh}\left(x\right):=log(x+\sqrt{x^2-1})$ for $x\ge 1$ and ${\Delta }_{\Sigma }({\mu }_1,{\mu }_2)$ is the Mahalanobis distance between $N({\mu }_1,\Sigma )$ and $N({\mu }_2,\Sigma )$. In that case, we thus have ${\rho }_{\mathrm{CO}}(N({\mu }_1,\Sigma ),N({\mu }_2,\Sigma ))=h_{\mathrm{CO}}\left({\Delta }_{\Sigma }({\mu }_1,{\mu }_2)\right)$ where $h_{\mathrm{CO}}\left(u\right)=\mathrm{arccosh}\left(1+\frac{1}{2}{u}^2\right)$ is a strictly monotone increasing function. Let us note in passing that in [19] (Corollary, page 230) there is a confusing or typographic error since the distance is reported as $\mathrm{arccosh}\left(1+\frac{1}{2}{d}_M({\mu }_1,{\mu }_2)\right)$ where $d_M$ denotes “Mahalanobis distance” [51]. Therefore, either $d_M={\Delta }_{\Sigma }^2$, Mahalanobis $D^2$-distance, or there is a missing square in the equation of the Corollary page 230. To obtain a flavor of how good is the approximation of the C&O distance, we may consider the same covariance case where we have both closed-form solutions for ${\rho }_{\mathcal{N}}$ (Equation (20)) and ${\rho }_{\mathrm{CO}}$ (Equation (23)). Figure 3 plots the two functions $h_{\mathrm{CO}}$ and $h_{\mathrm{FR}}$ (with $h_{\mathrm{CO}}\left(u\right)\le h_{\mathrm{FR}}\left(u\right)\le u$ for $u\in [0,\infty )$).

Figure 3. Quality of the C&O lower bound compared to the exact Fisher–Rao distance in the case of $N_1,N_2\in {\mathcal{M}}_{\Sigma }$ (MVNs sharing the same covariance matrix $\Sigma$). We have ${\rho }_{\mathrm{CO}}\le {\rho }_{\mathcal{N}}\le {\Delta }_{\Sigma }$.

Let us remark that similarly all f-divergences between $N_1=({\mu }_1,\Sigma )$ and $N_2=({\mu }_2,\Sigma )$ are scalar functions of their Mahalanobis distance ${\Delta }_{\Sigma }({\mu }_1,{\mu }_2)$ too, see [62].

The C&O distance ${\rho }_{\mathrm{CO}}$ is a metric distance that has been used in many applications ranging from computer vision [57,67,68,69] to signal/sensor processing, statistics [70,71], machine learning [29,72,73,74,75,76] and analogical reasoning [77].

Remark 3. 

In a second paper, Calvo and Oller [32] noticed that we can embed normal distributions in $\mathcal{P}(d+1)$ by the following more general mapping (Lemma 3.1 [32]):

$$g_{\alpha ,\beta ,\gamma }(\mu ,\Sigma )={|\Sigma |}^{\alpha }\left[\begin{array}{cc}\Sigma +\beta {\gamma }^2\mu {\mu }^{\top }\hfill & \beta \gamma \mu \hfill \\ \beta \gamma {\mu }^{\top }\hfill & \beta \hfill \end{array}\right]\in \mathcal{P}(d+1),$$

(24)

where $\alpha \in \mathbb{R}$, $\beta \in {\mathbb{R}}_{>0}$ and $\gamma \in \mathbb{R}$. It is show in [32] that the induced length element is

$$\begin{array}{ccc}\hfill \mathrm{d}{s}_{\alpha ,\beta ,\gamma }^2& =& \frac{1}{2}\left(\alpha ((d+1)+2\alpha ){\mathrm{tr}}^2({\Sigma }^{-1}\mathrm{d}\Sigma )+\mathrm{tr}\left({({\Sigma }^{-1}\mathrm{d}\Sigma )}^2\right)\right.\hfill \\ & & \left.+2\beta {\gamma }^2\mathrm{d}{\mu }^{\top }{\Sigma }^{-1}\mathrm{d}\mu +2\alpha \mathrm{tr}({\Sigma }^{-1}\mathrm{d}\Sigma )\frac{\mathrm{d}\beta }{\beta }+{\left(\frac{\mathrm{d}\beta }{\beta }\right)}^2\right).\hfill \end{array}$$

When $\gamma =\beta =1$, we have

$$\mathrm{d}{s}_{\alpha }^2=\frac{1}{2}\left(\alpha ((d+1)+2\alpha ){\mathrm{tr}}^2({\Sigma }^{-1}\mathrm{d}\Sigma )+\mathrm{tr}\left({({\Sigma }^{-1}\mathrm{d}\Sigma )}^2\right)+2\beta {\gamma }^2\mathrm{d}{\mu }^{\top }{\Sigma }^{-1}\mathrm{d}\mu \right).$$

Thus, to cancel the term ${\mathrm{tr}}^2({\Sigma }^{-1}\mathrm{d}\Sigma )$, we may either choose $\alpha =0$ or $\alpha =-\frac{2}{1+d}$.

In some applications [78], the embedding

$$g_{-\frac{1}{d+1},1,1}(\mu ,\Sigma )={|\Sigma |}^{-\frac{1}{d+1}}\left[\begin{array}{cc}\Sigma +\mu {\mu }^{\top }\hfill & \mu \hfill \\ {\mu }^{\top }\hfill & 1\hfill \end{array}\right]:=\widehat{f}(\mu ,\Sigma ),$$

(25)

is used to ensure that $\left|g_{-\frac{1}{d+1},1,1}(\mu ,\Sigma )\right|=1$. That is normal distributions are embedded diffeomorphically into the submanifold of positive–definite matrices with a unit determinant (also called SSPD, acronym of Special SPD). In [32], C&O showed that there exists a second isometric embedding of the Fisher–Rao Gaussian manifold $\mathcal{N}\left(d\right)$ into a submanifold of the cone $\mathcal{P}(d+1)$: $f_{\mathrm{SSPD}}(\mu ,\Sigma )={|\Sigma |}^{-\frac{2}{d+1}}\left[\begin{array}{cc}\Sigma +\mu {\mu }^{\top }\hfill & \mu \hfill \\ {\mu }^{\top }\hfill & 1\hfill \end{array}\right]$. Let $\widehat{P}=f_{\mathrm{SSPD}}(\mu ,\Sigma )$. This mapping can be understood as taking the elliptic isometry $P\mapsto {\left|P\right|}^{-\frac{2}{d+1}}P$ of $P\in \mathcal{P}(d+1)$ [64] since $|\Sigma |=|\overline{P}(\mu ,\Sigma )|$ (see proof in Proposition 3). It follows that

$${\rho }_{\mathrm{CO}}(N_1,N_2)={\rho }_{\mathcal{P}}({\overline{P}}_1,{\overline{P}}_2)={\rho }_{\mathcal{P}}({\widehat{P}}_1,{\widehat{P}}_2)\le {\rho }_{\mathcal{N}}(N_1,N_2).$$

Similarly, we could have mapped $P\mapsto P^{-1}$ to obtain another isometric embedding. See the four types of elliptic isometric of the SPD cone described in [64]. Finally, let us remark that the SSPD submanifold is totally geodesic with respect to the trace metric but not with respect to the C&O metric.

Interestingly, Calvo and Oller [48] (p. 131) proved that $(({\overline{\mu }}_1,\dots ,{\overline{\mu }}_d),\mathrm{diag}({\overline{\sigma }}_1^2,\dots ,{\overline{\sigma }}_d^2))$ is a maximal invariant for the action of the affine group $\mathrm{Aff}\left(d\right)$, where $\overline{\mu }=Q^{-1}({\mu }_2-{\mu }_1)$ and ${\Sigma }_2{\Sigma }_1^{-1}=Q\mathrm{diag}({\overline{\sigma }}_1^2,\dots ,{\overline{\sigma }}_d^2)Q^{-1}$ (in [48], the authors considered ${\Sigma }_1{\Sigma }_1^{-2}$). Thus, we consider the following dissimilarity

$$D_{\mathrm{CO}}(N({\mu }_1,{\Sigma }_1),N({\mu }_2,{\Sigma }_2))=\sqrt{2}\sqrt{\sum _{i=1}^d{log}^2\left(\frac{1+\Delta (0,1;{\overline{\mu }}_i,{\overline{\sigma }}_i)}{1-\Delta (0,1;{\overline{\mu }}_i,{\overline{\sigma }}_i)}\right)}.$$

(26)

Dissimilarity $D_{\mathrm{CO}}$ is symmetric (i.e., $D_{\mathrm{CO}}(N_1,N_2)=D_{\mathrm{CO}}(N_2,N_1)$) and $D_{\mathrm{CO}}(N_1,N_2)=0$ if and only if $N_1=N_2$. Please note that when $d=1$, $D_{\mathrm{CO}}$ is different from the Fisher–Rao distance of Equation (7).

3. Approximating the Fisher–Rao Distance

3.1. Approximating Length of Curves

Recall that the Fisher–Rao’s distance [79] is the Riemannian geodesic distance

$${\rho }_{\mathcal{N}}(N\left({\lambda }_1\right),N\left({\lambda }_2\right))=\underset{\begin{array}{c}c\left(t\right)\\ c\left(0\right)=p_{{\lambda }_1}\\ c\left(1\right)=p_{{\lambda }_2}\end{array}}{inf}\left\{\mathrm{Length}\left(c\right)\right\},$$

where

$$\mathrm{Length}\left(c\right)={\int }_0^1\underset{\mathrm{d}{s}_{\mathcal{N}}\left(t\right)}{\underbrace{{\sqrt{⟨\dot{c}\left(t\right),\dot{c}\left(t\right)⟩}}_{c\left(t\right)}}}\mathrm{d}t.$$

We can approximate the Rao distance ${\rho }_{\mathcal{N}}(N_1,N_2)$ by discretizing regularly any smooth curve $c\left(t\right)$ joining $N_1=c\left(0\right)$ to $N_2=c\left(1\right)$ (Figure 4):

$${\rho }_{\mathcal{N}}(N_1,N_2)\le \frac{1}{T}\sum _{i=1}^{T-1}{\rho }_{\mathcal{N}}\left(c\left(\frac{i}{T}\right),c\left(\frac{i+1}{T}\right)\right),$$

with equality holding iff $c\left(t\right)={\gamma }_{\mathcal{N}}(N_1,N_2;t)$ is the Riemannian geodesic defined by the Levi–Civita metric connection induced by the Fisher information metric.

Figure 4. Approximating the Fisher–Rao geodesic distance ${\rho }_{\mathcal{N}}(N_1,N_2)$: The Fisher–Rao geodesic ${\gamma }_{\mathcal{N}}^{\mathrm{FR}}$ is not known in closed form. We consider a tractable curve $c\left(t\right)$, discretize $c\left(t\right)$ at $T+1$ points $c\left(\frac{i}{T}\right)$ with $c\left(0\right)=N_1$ and $c\left(1\right)=N_2$, and approximate ${\rho }_{\mathcal{N}}\left(c\left(\frac{i}{T}\right),c\left(\frac{i+1}{T}\right)\right)$ by $\sqrt{D_J\left[c\left(\frac{i}{T}\right),c\left(\frac{i+1}{T}\right)\right]}$, considering that different tractable curves $c\left(t\right)$ yield different approximations.

When the number of discretization steps T is sufficiently large, the normal distributions $c\left(\frac{i}{T}\right)$ and $c\left(\frac{i+1}{T}\right)$ are close to each other, and we can approximate ${\rho }_{\mathcal{N}}\left(c\left(\frac{i}{T}\right),c\left(\frac{i+1}{T}\right)\right)$ by $\sqrt{D_J\left[c\left(\frac{i}{T}\right),c\left(\frac{i+1}{T}\right)\right]}$, where $D_J[N_1,N_2]=D_{\mathrm{KL}}[N_1,N_2]+D_{\mathrm{KL}}[N_2,N_1]$ is Jeffreys divergence, and $D_{\mathrm{KL}}$ is the Kullback–Leibler divergence:

$$D_{\mathrm{KL}}[p_{({\mu }_1,{\Sigma }_1)}:p_{({\mu }_2,{\Sigma }_2)}]=\frac{1}{2}\left(\mathrm{tr}\left({\Sigma }_2^{-1}{\Sigma }_1\right)+\Delta {\mu }^{\top }{\Sigma }_2^{-1}\Delta \mu -d+log\frac{|{\Sigma }_2|}{|{\Sigma }_1|}\right).$$

Thus, the costly determinant computations cancel each other in Jeffreys divergence (i.e., $log\frac{|{\Sigma }_2|}{|{\Sigma }_1|}+log\frac{|{\Sigma }_1|}{|{\Sigma }_2|}=0$) and we have:

$$D_J[p_{({\mu }_1,{\Sigma }_1)}:p_{({\mu }_2,{\Sigma }_2)}]=\mathrm{tr}\left(\frac{{\Sigma }_2^{-1}{\Sigma }_1+{\Sigma }_1^{-1}{\Sigma }_2}{2}-I\right)+\Delta {\mu }^{\top }\frac{{\Sigma }_1^{-1}+{\Sigma }_2^{-1}}{2}\Delta \mu .$$

Figure 4 summarizes our method to approximate the Fisher–Rao geodesic distance.

In general, it holds that

$$I_f[p:q]\approx \frac{f^{″}\left(1\right)}{2}\mathrm{d}{s}_{\mathrm{Fisher}}^2,$$

between infinitesimally close distributions p and q ($\mathrm{d}s\approx \sqrt{\frac{2I_f[p:q]}{f^{″}\left(1\right)}}$), where $I_f[·:·]$ denotes a f-divergence [1]. The Jeffreys divergence is a f-divergence obtained for $f_J\left(u\right)=-logu+ulogu$ with $f_J^{″}\left(1\right)=2$. It is thus interesting to find low computational cost f-divergences between multivariate normal distributions to approximate the infinitesimal length element $\mathrm{d}s$. Please note that f-divergences between MVNs are also invariant under the action of the affine group [62]. Thus, for infinitesimally close distributions p and q, this informally explains that $\mathrm{d}{s}_{\mathrm{Fisher}}$ is invariant under the action of the affine group (see Proposition 1).

Although the definite integral of the length element along the Fisher–Rao geodesic ${\gamma }_{\mathcal{N}}^{\mathrm{FR}}$ is not known in closed form (i.e., Fisher–Rao distance), the integral of the squared length element along the mixture geodesic ${\gamma }_{\mathcal{N}}^m(N_1,N_2)$ and exponential geodesic ${\gamma }_{\mathcal{N}}^e(N_1,N_2)$ coincide with Jeffreys divergence $D_J[N_1,N_2]$ between $N_1$ and $N_2$ [1]:

Property 3 

([1]). We have

$$D_J[p_{{\lambda }_1},p_{{\lambda }_2}]={\int }_0^1\mathrm{d}{s}_{\mathcal{N}}^2\left({\gamma }_{\mathcal{N}}^m(p_{{\lambda }_1},p_{{\lambda }_2};t)\right)\mathrm{d}t={\int }_0^1\mathrm{d}{s}_{\mathcal{N}}^2\left({\gamma }_{\mathcal{N}}^e(p_{{\lambda }_1},p_{{\lambda }_2};t)\right)\mathrm{d}t.$$

Proof. 

Let us report a proof of this remarkable fact in the general setting of Bregman manifolds. Indeed, since

$$D_J[p_{{\lambda }_1},p_{{\lambda }_2}]=D_{\mathrm{KL}}[p_{{\lambda }_1}:p_{{\lambda }_2}]+D_{\mathrm{KL}}[p_{{\lambda }_2}:p_{{\lambda }_1}],$$

and $D_{\mathrm{KL}}[p_{{\lambda }_1}:p_{{\lambda }_2}]=B_F(\theta \left({\lambda }_2\right):\theta \left({\lambda }_1\right))$, where $B_F$ denotes the Bregman divergence induced by the cumulant function of the multivariate normals and $\theta \left(\lambda \right)$ is the natural parameter corresponding to $\lambda$, we have

$$\begin{array}{ccc}\hfill D_J[p_{{\lambda }_1},p_{{\lambda }_2}]& =& B_F({\theta }_1:{\theta }_2)+B_F({\theta }_2:{\theta }_1),\hfill \\ & =& S_F({\theta }_1;{\theta }_2)={({\theta }_2-{\theta }_1)}^{\top }({\eta }_2-{\eta }_1)=S_{F^{*}}({\eta }_1;{\eta }_2),\hfill \end{array}$$

where $\eta =\nabla F\left(\theta \right)$ and $\theta =\nabla F^{*}\left(\eta \right)$ denote the dual parameterizations obtained by the Legendre–Fenchel convex conjugate $F^{*}\left(\eta \right)$ of $F\left(\theta \right)$. Moreover, we have $F^{*}\left(\eta \right)=-h\left(p_{\mu ,\Sigma }\right)$ [1], i.e., the convex conjugate function is Shannon negentropy.

Then we conclude using the fact that $S_F({\theta }_1;{\theta }_2)={\int }_0^1\mathrm{d}{s}^2\left(\gamma \left(t\right)\right)\mathrm{d}t={\int }_0^1\mathrm{d}{s}^2\left({\gamma }^{*}\left(t\right)\right)\mathrm{d}t$, i.e., the symmetrized Bregman divergence amounts to integral energies on dual geodesics on a Bregman manifold. The proof of this general property is reported in Appendix E. □

It follows the following upper bound on the Fisher–Rao distance:

Property 4 

(Fisher–Rao upper bound). The Fisher–Rao distance between normal distributions is upper bounded by the square root of the Jeffreys divergence: ${\rho }_{\mathcal{N}}(N_1,N_2)\le \sqrt{D_J(N_1,N_2)}$.

Proof. 

Consider the Cauchy–Schwarz inequality for positive functions $f\left(t\right)$ and $g\left(t\right)$: ${\int }_0^{1}f\left(t\right)g\left(t\right)\mathrm{d}t\le \sqrt{\left({\int }_0^{1}f{\left(t\right)}^2\mathrm{d}t\right)\left({\int }_0^{1}g{\left(t\right)}^2\mathrm{d}t\right)}$), and let $f\left(t\right)=\mathrm{d}{s}_{\mathcal{N}}({\gamma }_{\mathcal{N}}^c(p_{{\lambda }_1},p_{{\lambda }_2};t)$ and $g\left(t\right)=1$. Then we obtain:

$${\left({\int }_0^1\mathrm{d}{s}_{\mathcal{N}}({\gamma }_{\mathcal{N}}^c(p_{{\lambda }_1},p_{{\lambda }_2};t)\mathrm{d}t\right)}^2\le \left({\int }_0^1\mathrm{d}{s}_{\mathcal{N}}^2({\gamma }_{\mathcal{N}}^c(p_{{\lambda }_1},p_{{\lambda }_2};t)\mathrm{d}t\right)\left(\underset{=1}{\underbrace{{\int }_0^1{1}^2\mathrm{d}t}}\right).$$

Furthermore, since by definition of ${\gamma }_{\mathcal{N}}^{\mathrm{FR}}$, we have

$${\int }_0^1\mathrm{d}{s}_{\mathcal{N}}({\gamma }_{\mathcal{N}}^c(p_{{\lambda }_1},p_{{\lambda }_2};t)\mathrm{d}t\ge {\int }_0^1\mathrm{d}{s}_{\mathcal{N}}({\gamma }_{\mathcal{N}}^{\mathrm{FR}}(p_{{\lambda }_1},p_{{\lambda }_2};t)\mathrm{d}t=:{\rho }_{\mathcal{N}}(N_1,N_2).$$

It follows for $c={\gamma }_{\mathcal{N}}^e$ (i.e., e-geodesic) using Property 3 that we have:

$${\rho }_{\mathcal{N}}{(N_1,N_2)}^2\le {\int }_0^1\mathrm{d}{s}_{\mathcal{N}}^2({\gamma }_{\mathcal{N}}^e(p_{{\lambda }_1},p_{{\lambda }_2};t)\mathrm{d}t=D_J(N_1,N_2).$$

Thus, we conclude that ${\rho }_{\mathcal{N}}(N_1,N_2)\le \sqrt{D_J(N_1,N_2)}$.

Please note that in Riemannian geometry, a curve $\gamma$ minimizes the energy $E\left(\gamma \right)={\int }_0^1{\parallel \dot{\gamma }\left(t\right)\parallel }^2\mathrm{d}t$ if it minimizes the length $L\left(\gamma \right)={\int }_0^1\parallel \dot{\gamma }\left(t\right)\parallel \mathrm{d}t$ and $\parallel \dot{\gamma }\left(t\right)\parallel$ is constant. Using Cauchy-Schwartz inequality, we can show that $L\left(\gamma \right)\le E\left(\gamma \right)$. □

This upper bound is tight at infinitesimal scale (i.e., when $N_2=N_1+\mathrm{d}N$) since ${\rho }_{\mathcal{N}}(N_1,N_2)\approx \mathrm{d}{s}_{\mathcal{N}}\left(N_1\right)\approx \sqrt{\frac{2I_f[N_1:N_2]}{f^{″}\left(1\right)}}$ and the f-divergence in right-hand side of the identity can be chosen as Jeffreys divergence. To appreciate the quality of the square root of Jeffreys divergence upper bound of Property 4, consider the case where $N_1,N_2\in {\mathcal{M}}_{\Sigma }$. In that case, we have ${\rho }_{\mathcal{N}}(N({\mu }_1,\Sigma ),N({\mu }_2,\Sigma ))=\sqrt{2}\mathrm{arccosh}(1+\frac{1}{4}{\Delta }_{\Sigma }^2({\mu }_1,{\mu }_2))$ and $\sqrt{D_J[N({\mu }_1,\Sigma ),N({\mu }_2,\Sigma )]}={\Delta }_{\Sigma }({\mu }_1,{\mu }_2)$ (since $D_{\mathrm{KL}}[N({\mu }_1,\Sigma ),N({\mu }_2,\Sigma )]=\frac{1}{2}{\Delta }_{\Sigma }^2({\mu }_1,{\mu }_2)$). The upper bound can thus be checked since we have $\sqrt{2}\mathrm{arccosh}(1+\frac{1}{4}{x}^2)\le x$ for $x\ge 0$. The plots of Figure 5 shows visually the quality of the $\sqrt{D_J}$ upper bound.

Figure 5. Quality of the $\sqrt{D_J}$ upper bound on the Fisher–Rao distance ${\rho }_{\mathcal{N}}$ when normal distributions have the same covariance matrix.

For any smooth curve $c\left(t\right)$, we can thus approximate ${\rho }_{\mathcal{N}}$ for large T by

$${\tilde{\rho }}_{\mathcal{N}}^c(N_1,N_2):=\frac{1}{T}\sum _{i=1}^{T-1}\sqrt{D_J\left[c\left(\frac{i}{T}\right),c\left(\frac{i+1}{T}\right)\right].}$$

(27)

For example, we may consider the following curves on ${\mathcal{M}}_{\mathcal{N}}$ which admit closed-form parameterizations in $t\in [0,1]$:

• linear interpolation (LERP, Linear intERPolation) $c_{\lambda }\left(t\right)=t({\mu }_1,{\Sigma }_1)+(1-t)({\mu }_2,{\Sigma }_2)$ between $({\mu }_1,{\Sigma }_1)$ and $({\mu }_2,{\Sigma }_2)$,
• the mixture geodesic [80] $c_m\left(t\right)={\gamma }_{\mathcal{N}}^m(N_1,N_2;t)=({\mu }_t^m,{\Sigma }_t^m)$ with ${\mu }_t^m={\overline{\mu }}_t$ and ${\Sigma }_t^m={\overline{\Sigma }}_t+t{\mu }_1{\mu }_1^{\top }+(1-t){\mu }_2{\mu }_2^{\top }-{\overline{\mu }}_t{\overline{\mu }}_t^{\top }$ where ${\overline{\mu }}_t=t{\mu }_1+(1-t){\mu }_2$ and ${\overline{\Sigma }}_t=t{\Sigma }_1+(1-t){\Sigma }_2$,
• the exponential geodesic [80] $c_e\left(t\right)={\gamma }_{\mathcal{N}}^e(N_1,N_2;t)=({\mu }_t^e,{\Sigma }_t^e)$ with ${\mu }_t^e={\overline{\Sigma }}_t^H(t{\Sigma }_1^{-1}{\mu }_1+(1-t){\Sigma }_2^{-1}{\mu }_2)$ and ${\Sigma }_t^e={\overline{\Sigma }}_t^H$ where ${\overline{\Sigma }}_t^H={(t{\Sigma }_1^{-1}+(1-t){\Sigma }_2^{-1})}^{-1}$ is the matrix harmonic mean,
• the curve $c_{em}\left(t\right)=\frac{1}{2}\left({\gamma }_{\mathcal{N}}^e(N_1,N_2;t)+{\gamma }_{\mathcal{N}}^m(N_1,N_2;t)\right)$ which is obtained by averaging the mixture geodesic with the exponential geodesic.

Figure 6 visualizes the exponential and mixture geodesics between two bivariate normal distributions.

Figure 6. Visualizing the exponential and mixture geodesics between two bivariate normal distributions.

Let us denote by ${\tilde{\rho }}_{\mathcal{N}}^{\lambda }={\tilde{\rho }}_{\mathcal{N}}^{c_{\lambda }}$, ${\tilde{\rho }}_{\mathcal{N}}^m={\tilde{\rho }}_{\mathcal{N}}^{c_m}$, ${\tilde{\rho }}_{\mathcal{N}}^e={\tilde{\rho }}_{\mathcal{N}}^{c_e}$ and ${\tilde{\rho }}_{\mathcal{N}}^{em}={\tilde{\rho }}_{\mathcal{N}}^{c_{em}}$ the approximations obtained by these curves following from Equation (27). When T is sufficiently large, the approximated distances ${\tilde{\rho }}^x$ are close to the length of curve x, and we may thus consider a set of several curves ${\left\{c_i\right\}}_{i\in I}$ and report the smallest Fisher–Rao distance approximations obtained among these curves: ${\rho }_{\mathcal{N}}(N_1,N_2)\approx {min}_{i\in I}{\tilde{\rho }}_{\mathcal{N}}^{c_i}(N_1,N_2)$.

Please note that we consider the regular spacing for approximating a curve length and do not optimize the position of the sample points on the curve. Indeed, as $T\to \infty$, the curve length approximation tends to the Riemannian curve length. In other words, we can measure approximately finely the length of any curve available with closed-form reparameterization by increasing T. Thus, the key question of our method is how to best approximate the Fisher–Rao geodesic by a curve that can be parametrized by a closed-form formula and is close enough to the Fisher–Rao geodesic.

Next, we introduce our approximation curve $c_{\mathrm{CO}}\left(t\right)$ derived from Calvo and Oller isometric mapping f which experimentally behaves better when normals are not too far from each other.

3.2. A Curve Derived from Calvo and Oller’s Embedding

This approximation consists of leveraging the closed-form expression of the SPD geodesics [63,66]:

$${\gamma }_{\mathcal{P}}(P,Q;t)=P^{\frac{1}{2}}{\left(P^{-\frac{1}{2}}{Q}^{\frac{1}{2}}{P}^{-\frac{1}{2}}\right)}^t{P}^{\frac{1}{2}},t\in [0,1]$$

to approximate the Fisher–Rao normal geodesic ${\gamma }_{\mathcal{N}}^{\mathrm{Fisher}}(N_1,N_2;t)$ as follows: Let ${\overline{P}}_1=f\left(N_1\right),{\overline{P}}_2=f\left(N_2\right)\in \overline{\mathcal{N}}$, and consider the smooth curve

$${\overline{c}}_{\mathrm{CO}}({\overline{P}}_1,{\overline{P}}_2;t)={\mathrm{proj}}_{\overline{\mathcal{N}}}\left({\gamma }_{\mathcal{P}}({\overline{P}}_1,{\overline{P}}_2;t)\right),$$

(28)

where ${\mathrm{proj}}_{\overline{\mathcal{N}}}\left(P\right)$ denotes the orthogonal projection of $P\in \mathcal{P}(d+1)$ onto $\overline{\mathcal{N}}$ (Figure 7). Thus, curve $c_{\mathrm{CO}}\left(t\right)$ ($t\in [0,1]$) is then defined by taking the inverse mapping $f^{-1}\left({\overline{c}}_{\mathrm{CO}}\right)$ (Figure 8):

$$c_{\mathrm{CO}}\left(t\right)=f^{-1}\left({\mathrm{proj}}_{\overline{\mathcal{N}}}\left({\gamma }_{\mathcal{P}}({\overline{P}}_1,{\overline{P}}_2;t)\right)\right).$$

(29)

Figure 7. Projecting an SPD matrix $P\in \mathcal{P}$ onto $\overline{\mathcal{N}}=f\left(\mathcal{N}\right)$: ${\gamma }_{\mathcal{P}}(P,{\overline{P}}_{\perp })$ is orthogonal to $\overline{\mathcal{N}}$ with respect to the trace metric.

Figure 8. Illustration of the approximation of the Fisher–Rao distance between two multivariate normals $N_1$ and $N_2$ (red geodesic length ${\gamma }_{\mathcal{N}}(N_1,N_2)$ by discretizing curve ${\overline{c}}_{\mathrm{CO}}\in \overline{\mathcal{N}}$ or equivalently curve $c_{\mathrm{CO}}\in \mathcal{N}$.

Please note that the matrix power $P^t$ can be computed as $P^t=U\mathrm{diag}({\lambda }_1^t,\dots ,{\lambda }_d^t)V^{\top }$ where $P=U\mathrm{diag}({\lambda }_1^t,\dots ,{\lambda }_d^t)V^{\top }$ is the eigenvalue decomposition of P.

Let us now explain how to project $P=\left[P_{i,j}\right]\in \mathcal{P}(d+1)$ onto $\overline{\mathcal{N}}$ based on the analysis of the Appendix of [19] (p. 239):

Proposition 2 

(Projection of an SPD matrix onto the embedded normal submanifold $\overline{\mathcal{N}}$). Let $\beta =P_{d+1,d+1}$ and write $P=\left[\begin{array}{cc}\Sigma +\beta \mu {\mu }^{\top }\hfill & \beta \mu \hfill \\ \beta {\mu }^{\top }\hfill & \beta \hfill \end{array}\right]$. Then the orthogonal projection at $P\in \mathcal{P}$ onto $\overline{\mathcal{N}}$ is:

$${\overline{P}}_{\perp }:={\mathrm{proj}}_{\overline{\mathcal{N}}}\left(P\right)=\left[\begin{array}{cc}\Sigma +\mu {\mu }^{\top }\hfill & {\mu }^{\top }\hfill \\ \mu \hfill & 1\hfill \end{array}\right],$$

(30)

and the SPD distance between P and ${\overline{P}}_{\perp }$ is

$${\rho }_{\mathcal{P}}(P,{\overline{P}}_{\perp })=\frac{1}{\sqrt{2}}|log\beta |.$$

(31)

Notice that the projection of P is easily computed since $\beta =P_{d+1,d+1}$.

$${\mathrm{proj}}_{\overline{\mathcal{N}}}\left(\left[\begin{array}{cc}\Sigma +\beta \mu {\mu }^{\top }\hfill & \beta \mu \hfill \\ \beta {\mu }^{\top }\hfill & \beta \hfill \end{array}\right]\right)=\left[\begin{array}{cc}\Sigma +\mu {\mu }^{\top }\hfill & {\mu }^{\top }\hfill \\ \mu \hfill & 1\hfill \end{array}\right]$$

Remark 4. 

In Diffusion Tensor Imaging [39] (DTI), the Fisher–Rao distance can be used to evaluate the distance between three-dimensional normal distributions with means located at a 3D grid position. We may consider $3\times 3\times 3-1=26$ neighbor graphs induced by the grid, and for each normal N of the grid, calculate the approximations of the Fisher–Rao distance of N with its neighbors $N^{\prime }$ as depicted in Figure 9. Then the distance between two tensors $N_1$ and $N_2$ of the 3D grid is calculated as the shortest path on the weighted graph using Dijkstra’s algorithm [39].

Figure 9. Diffusion tensor imaging (DTI) on a 2D grid: (a) Ellipsoids shown at the $8\times 8$ grid locations with C&O curves in green, and (b) some interpolated ellipsoids are further shown along the C&O curves.

Please note that the Fisher–Rao projection of $N_1=({\mu }_1,{\Sigma }_1)$ onto a submanifold ${\mathcal{M}}_{{\mu }_2}$ with fixed mean ${\mu }_2$ was recently reported in closed form in [72] (Equation (21)):

$$N^{*}=N\left({\mu }_2,{\Sigma }_1+\frac{1}{2}({\mu }_2-{\mu }_1){({\mu }_2-{\mu }_1)}^{\top }\right),$$

with

$${\rho }_{\mathcal{N}}(N_1,N^{*})=\frac{1}{\sqrt{2}}\mathrm{arccosh}\left(d+{({\mu }_2-{\mu }_1)}^{\top }{\Sigma }_1^{-1}({\mu }_2-{\mu }_1)\right),$$

and the Fisher–Rao projection of $N_1=({\mu }_1,{\Sigma }_1)$ onto submanifold ${\mathcal{M}}_{{\Sigma }_2}$ is the “vertical projection” $N^{*}=({\mu }_1,{\Sigma }_2)$ (Figure 10) with

$${\rho }_{\mathcal{N}}(N_1,N^{*})={\rho }_{{\mathcal{N}}_{\mu }}({\Sigma }_1,{\Sigma }_2).$$

Figure 10. Examples of projection of $N(\mu ,\Sigma )$ onto the submanifolds ${\mathcal{M}}_{{\mu }_0}$ and ${\mathcal{M}}_{{\Sigma }_0}$. Tissot indicatrices are rendered in green at the projected normal distributions $\left({\mu }_0,\Sigma +\frac{1}{2}({\mu }_0-\mu ){({\mu }_0-\mu )}^{\top }\right)$ and $(\mu ,{\Sigma }_0)$, respectively.

We can upper bound the Fisher–Rao distance ${\rho }_{\mathcal{N}}(({\mu }_1,{\Sigma }_1),({\mu }_2,{\Sigma }_2))$ by projecting ${\Sigma }_1$ onto ${\mathcal{M}}_{{\mu }_2}$ and projecting ${\Sigma }_2$ onto ${\mathcal{M}}_{{\mu }_1}$. Let ${\Sigma }_{12}\in {\mathcal{M}}_{{\mu }_2}$ and ${\Sigma }_{21}\in {\mathcal{M}}_{{\mu }_1}$ denote those Fisher–Rao orthogonal projections. Using the triangular inequality property of the Fisher–Rao distance, we obtain the following upper bounds:

$$\begin{array}{ccc}\hfill {\rho }_{\mathcal{N}}(({\mu }_1,{\Sigma }_1),({\mu }_2,{\Sigma }_2))& \le & {\rho }_{\mathcal{N}}(({\mu }_1,{\Sigma }_1),({\mu }_2,{\Sigma }_{12}))+{\rho }_{\mathcal{N}}({\mu }_2,{\Sigma }_{12},({\mu }_2,{\Sigma }_2)),\hfill \end{array}$$

(32)

$$\begin{array}{ccc}& \le & {\rho }_{\mathcal{N}}(({\mu }_2,{\Sigma }_2),({\mu }_1,{\Sigma }_{21}))+{\rho }_{\mathcal{N}}(({\mu }_1,{\Sigma }_{21}),({\mu }_1,{\Sigma }_1)).\hfill \end{array}$$

(33)

See Figure 11 for an illustration.

Figure 11. Upper bounding the Fisher–Rao’s distance ${\rho }_{\mathcal{N}}(({\mu }_1,{\Sigma }_1),({\mu }_2,{\Sigma }_2))$ (red points) using projections (green points) onto submanifolds with fixed means.

Let ${\overline{c}}_{\mathrm{CO}}\left(t\right)={\overline{S}}_t$ and $c_{\mathrm{CO}}\left(t\right)=f^{-1}\left(c_{\mathrm{CO}}\left(t\right)\right)=:G_t$. The following proposition shows that we have $D_J[{\overline{S}}_t,{\overline{S}}_{t+1}]=D_J[G_t,G_{t+1}]$.

Proposition 3. 

The Kullback–Leibler divergence between $p_{{\mu }_1,{\Sigma }_1}$ and $p_{{\mu }_2,{\Sigma }_2}$ amounts to the KLD between $q_{{\overline{P}}_1}=p_{0,f({\mu }_1,{\Sigma }_1)}$ and $q_{{\overline{P}}_2}=p_{0,f({\mu }_2,{\Sigma }_2)}$ where ${\overline{P}}_i=f({\mu }_i,{\Sigma }_i)$:

$$D_{\mathrm{KL}}[p_{{\mu }_1,{\Sigma }_1}:p_{{\mu }_2,{\Sigma }_2}]=D_{\mathrm{KL}}[q_{{\overline{P}}_1}:q_{{\overline{P}}_2}].$$

The KLD between two centered $(d+1)$-variate normals $q_{P_1}=p_{0,P_1}$ and $q_{P_2}=p_{0,P_2}$ is

$$D_{\mathrm{KL}}[q_{P_1}:q_{P_2}]=\frac{1}{2}\left(\mathrm{tr}\left(P_2^{-1}{P}_1\right)-d-1+log\frac{|P_2|}{|P_1|}\right).$$

This divergence can be interpreted as the matrix version of the Itakura–Saito divergence [81]. The SPD cone equipped with $\frac{1}{2}$ of the trace metric can be interpreted as Fisher–Rao centered normal manifolds: $({\mathcal{N}}_{\mu },g_{{\mathcal{N}}_{\mu }}^{\mathrm{Fisher}})=(\mathcal{P},\frac{1}{2}{g}^{\mathrm{trace}})$.

Since the determinant of a block matrix is

$$\left|\left[\begin{array}{cc}A\hfill & B\hfill \\ C\hfill & D\hfill \end{array}\right]\right|=\left|A-B{D}^{-1}C\right|,$$

we obtain with $D=1$: $\left|f(\mu ,\Sigma )\right|=|\Sigma +\mu {\mu }^{\top }-\mu {\mu }^{\top }|=|\Sigma |$.

Let ${\overline{P}}_1=f({\mu }_1,{\Sigma }_1)$ and ${\overline{P}}_2=f({\mu }_2,{\Sigma }_2)$. Checking $D_{\mathrm{KL}}[p_{{\mu }_1,{\Sigma }_1}:p_{{\mu }_2,{\Sigma }_2}]=D_{\mathrm{KL}}[q_{{\overline{P}}_1}:q_{{\overline{P}}_2}]$ where $q_{\overline{P}}=p_{0,\overline{P}}$ amounts to verify that

$$\mathrm{tr}\left({\overline{P}}_2^{-1}{\overline{P}}_1\right)=1+\mathrm{tr}({\Sigma }_2^{-1}{\Sigma }_1+{\Delta }_{\mu }^{\top }{\Sigma }_2^{-1}{\Delta }_{\mu }).$$

Indeed, using the inverse matrix

$$f{(\mu ,\Sigma )}^{-1}=\left[\begin{array}{cc}{\Sigma }^{-1}\hfill & -{\Sigma }^{-1}\mu \hfill \\ -{\mu }^{\top }{\Sigma }^{-1}\hfill & 1+{\mu }^{\top }{\Sigma }^{-1}\mu \hfill \end{array}\right],$$

we have

$$\begin{array}{ccc}\hfill \mathrm{tr}\left({\overline{P}}_2^{-1}{\overline{P}}_1\right)& =& \mathrm{tr}\left(\left[\begin{array}{cc}{\Sigma }_2^{-1}\hfill & -{\Sigma }_2^{-1}{\mu }_2\hfill \\ -{\mu }_2^{\top }{\Sigma }_2^{-1}\hfill & 1+{\mu }_2^{\top }{\Sigma }_2^{-1}{\mu }_2\hfill \end{array}\right]\left[\begin{array}{cc}{\Sigma }_1+{\mu }_1{\mu }_1^{\top }\hfill & {\mu }_1\hfill \\ {\mu }_1^{\top }\hfill & 1\hfill \end{array}\right]\right),\hfill \\ & =& 1+\mathrm{tr}({\Sigma }_2^{-1}{\Sigma }_1+{\Delta }_{\mu }^{\top }{\Sigma }_2^{-1}{\Delta }_{\mu }).\hfill \end{array}$$

Thus, even if the dimension of the sample spaces of $p_{\mu ,\Sigma }$ and $q_{\overline{P}=f(\mu ,\Sigma )}$ differs by one, we obtain the same KLD by Calvo and Oller’s isometric mapping f.

This property holds for the KLD/Jeffreys divergence $D_J$ but not for all f-divergences [1]$I_f$ in general (e.g., it fails for the Hellinger divergence).

Figure 12 shows the various geodesics and curves used to approximate the Fisher–Rao distance with the Fisher metric shown using Tissot indicatrices.

Figure 12. Geodesics and curves used to approximate the Fisher–Rao distance with the Fisher metric shown using Tissot’s indicatrices: exponential geodesic (red), mixture geodesic (blue), mid-exponential-mixture curve (purple), projected CO curve (green), and target Fisher–Rao geodesic (black). (Visualization in the parameter space of normal distributions).

Please note that the introduction of parameter $\beta$ is related to the foliation of the SPD cone $\mathcal{P}$ by $\{f_{\beta }\left(\mathcal{N}\right):\beta >0\}$: $\mathcal{P}(d+1)={\mathbb{R}}_{>0}\times f_{\beta }\left(\mathcal{N}\right)$. See Figure 7. Thus, we may define how good the projected C&O curve is to the Fisher–Rao geodesic by measuring the average distance between points on ${\gamma }_{\mathcal{P}}({\overline{P}}_1,{\overline{P}}_2;t)$ and their projections ${\overline{{\gamma }_{\mathcal{P}}({\overline{P}}_1,{\overline{P}}_2;t)}}^{\perp }$ onto $\overline{\mathcal{N}}$:

$${\delta }^{\mathrm{CO}}(N_1,N_2)={\delta }^{\mathrm{CO}}({\overline{P}}_1,{\overline{P}}_2)={\int }_0^1{\rho }_{\mathcal{P}}({\gamma }_{\mathcal{P}}({\overline{P}}_1,{\overline{P}}_2;t),{\overline{{\gamma }_{\mathcal{P}}({\overline{P}}_1,{\overline{P}}_2;t)}}^{\perp })\mathrm{d}t.$$

In practice, we evaluate this integral at the sampling points $S_t$:

$${\delta }^{\mathrm{CO}}(P_1,P_2)\approx {\delta }_T^{\mathrm{CO}}(P_1,P_2):=\frac{1}{T}\sum _{i=1}^T{\rho }_{\mathcal{P}}(S_t,{\overline{S}}_t),$$

(34)

where $S_t={\gamma }_{\mathcal{P}}({\overline{P}}_1,{\overline{P}}_2;t)$ and ${\overline{S}}_t={\gamma }_{\mathcal{P}}{({\overline{P}}_1,{\overline{P}}_2;t)}^{\perp }$. We checked experimentally (see Section 3.3) that for close by normals $N_1$ and $N_1$, we have ${\delta }^{\mathrm{CO}}({\overline{N}}_1,{\overline{N}}_2)$ small, and that when $N_1$ becomes further separated from $N_2$, the average projection error ${\delta }^{\mathrm{CO}}({\overline{N}}_1,{\overline{N}}_2)$ increases. Thus, ${\delta }_T^{\mathrm{CO}}(P_1,P_2)$ is a good measure of the precision of our Fisher–Rao distance approximation.

Lemma 1. 

We have ${\rho }_{\overline{\mathcal{N}}}({\overline{S}}_t,{\overline{S}}_{t+1})\le {\rho }_{\mathcal{P}}({\overline{S}}_t,S_t)+{\rho }_{\mathcal{P}}(S_t,S_{t+1})+{\rho }_{\mathcal{P}}(S_{t+1},{\overline{S}}_{t+1})$.

Proof. 

The proof consists of applying twice the triangle inequality of metric distance ${\rho }_{\mathcal{P}}$:

$$\begin{array}{ccc}\hfill {\rho }_{\overline{\mathcal{N}}}({\overline{S}}_t,{\overline{S}}_{t+1})& \le & {\rho }_{\mathcal{P}}({\overline{S}}_t,S_{t+1})+{\rho }_{\mathcal{P}}(S_{t+1},{\overline{S}}_{t+1}),\hfill \\ & \le & {\rho }_{\mathcal{P}}({\overline{S}}_t,S_t)+{\rho }_{\mathcal{P}}(S_t,S_{t+1})+{\rho }_{\mathcal{P}}(S_{t+1},{\overline{S}}_{t+1}).\hfill \end{array}$$

See Figure 13 where the left-hand-side geodesic length is shown in blue and the right-hand-side upper bound is visualized in red. □

Figure 13. Bounding ${\rho }_{\mathcal{N}}({\overline{S}}_t,{\overline{S}}_{t+1})$ using the triangular inequality of ${\rho }_{\mathcal{P}}$ in the SPD cone $\mathcal{P}(d+1)$.

Property 5. 

We have ${\rho }_{\mathcal{N}}(N_1,N_2)\le {\rho }_{\mathcal{N}}^{\mathrm{CO}}(N_1,N_2)\le {\rho }_{\mathcal{N}}(N_1,N_2)+2{\delta }_T^{\mathrm{CO}}({\overline{P}}_1,{\overline{P}}_2)$.

Proof. 

At infinitesimal scale when $S_{t+1}\approx S_t$, using Lemma 1 and ${\rho }_{\mathcal{P}}(S_{t+1},{\overline{S}}_{t+1})\approx {\rho }_{\mathcal{P}}({\overline{S}}_t,S_t)$ we have

$$\mathrm{d}{s}_{\mathcal{N}}\left({\overline{S}}_t\right)\le \mathrm{d}{s}_{\mathcal{P}}\left(S_t\right)+2{\rho }_{\mathcal{P}}(S_t,{\overline{S}}_t).$$

Taking the integral along the curve $c^{\mathrm{CO}}\left(t\right)=\overline{{\gamma }^{\mathrm{CO}}({\overline{P}}_1,{\overline{P}}_2;t)}$, we obtain

$${\rho }_{\mathcal{N}}^{\mathrm{CO}}(N_1,N_2)\le {\rho }_{\mathcal{P}}({\overline{P}}_1,{\overline{P}}_2)+2{\delta }_T^{\mathrm{CO}}({\overline{P}}_1,{\overline{P}}_2)$$

Since ${\rho }_{\mathcal{P}}({\overline{P}}_1,{\overline{P}}_2)\le {\rho }_{\mathcal{N}}(N_1,N_2)$, we have

$${\rho }_{\mathcal{N}}(N_1,N_2)\le {\rho }_{\mathcal{N}}^{\mathrm{CO}}(N_1,N_2)\le {\rho }_{\mathcal{N}}(N_1,N_2)+2{\delta }_T^{\mathrm{CO}}({\overline{P}}_1,{\overline{P}}_2).$$

□

Notice that ${\sum }_{i=0}^{T-1}{\rho }_{\mathcal{P}}(S_t,S_{t+1})={\rho }_{\mathcal{P}}({\overline{P}}_1,{\overline{P}}_2)$.

Example 1. 

Let us consider Example 1 of [42] (p. 11):

$$N_1=\left(\left[\begin{array}{c}-1\hfill \\ 0\hfill \end{array}\right],\Sigma \right),N_2=\left(\left[\begin{array}{c}6\hfill \\ 3\hfill \end{array}\right],\Sigma \right),\Sigma =\left[\begin{array}{cc}1.1\hfill & 0.9\hfill \\ 0.9\hfill & 1.1\hfill \end{array}\right].$$

The Fisher–Rao distance is evaluated numerically in [42] as $5.00648$. We have the lower bound ${\rho }_{\mathcal{N}}^{\mathrm{CO}}(N_1,N_2)=4.20447$, and the Mahalanobis distance $8.06226$ upper bounds the Fisher–Rao distance (not totally geodesic submanifold ${\mathcal{N}}_{\Sigma }$). Our projected C&O curve discretized with $T=1000$ yields an approximation ${\tilde{\rho }}_{\mathcal{N}}^{\mathrm{CO}}(N_1,N_2)=5.31667$. The average projection distance ${\rho }_{\mathcal{P}}(S_t,{\overline{S}}_t)$ is ${\delta }_T^{\mathrm{CO}}(N_1,N_2)=0.61791$, and the maximum projected distance is $1.00685$. We check that

$$5.00648\approx {\rho }_{\mathcal{N}}(N_1,N_2)\le {\tilde{\rho }}_{\mathcal{N}}^{\mathrm{CO}}(N_1,N_2)\approx 5.31667\le {\rho }_{\mathcal{N}}(N_1,N_2)+2{\delta }_T^{\mathrm{CO}}({\overline{P}}_1,{\overline{P}}_2)\approx 5.44028.$$

The Killing distance [82] obtained for ${\kappa }_{\mathrm{Killing}}=2$ is ${\rho }_{\mathrm{Killing}}(N_1,N_2)\approx 6.82028$ (see Appendix C). Notice that geodesic shooting is time-consuming compared to our approximation technique.

3.3. Some Experiments

The KLD $D_{\mathrm{KL}}$ and Jeffreys divergence $D_J$, the Fisher–Rao distance ${\rho }_{\mathcal{N}}$ and the Calvo and Oller distance ${\rho }_{\mathrm{CO}}$ are all invariant under the congruence action of the affine group $\mathrm{Aff}\left(d\right)={\mathbb{R}}^d⋊\mathrm{GL}\left(d\right)$ with the group operation

$$(a_1,A_1)(a_2,A_2)=(a_1+A_1{a}_2,A_1{A}_2).$$

Let $(A,a)\in \mathrm{Aff}\left(d\right)$, and define the action on the normal space $\mathcal{N}$ as follows:

$$(A,a).N(\mu ,\Sigma )=N(A^{\top }\mu +a,A\Sigma A^{\top }).$$

Then we have:

$$\begin{array}{ccc}\hfill {\rho }_{\mathcal{N}}((A,a).N_1,(A,a).N_2)& =& {\rho }_{\mathcal{N}}(N_1,N_2),\hfill \\ \hfill {\rho }_{\mathrm{CO}}((A,a).N_1,(A,a).N_2)& =& {\rho }_{\mathrm{CO}}(N_1,N_2),\hfill \\ \hfill D_{\mathrm{KL}}[(A,a).N_1:(A,a).N_2]& =& D_{\mathrm{KL}}[N_1:N_2].\hfill \end{array}$$

This invariance extends to our approximations ${\tilde{\rho }}_{\mathcal{N}}^c$ (see Equation (27)).

Since we have

$${\tilde{\rho }}_{\mathcal{N}}^c(N_1,N_2)\approx {\rho }_{\mathcal{N}}(N_1,N_2)\ge {\rho }_{\mathrm{CO}}(N_1,N_2),$$

the ratio ${\kappa }_c=\frac{{\tilde{\rho }}_{\mathcal{N}}^c}{{\rho }_{\mathrm{CO}}}\ge \kappa =\frac{{\tilde{\rho }}_{\mathcal{N}}^c}{{\rho }_{\mathcal{N}}}$ gives an upper bound on the approximation factor of ${\tilde{\rho }}_{\mathcal{N}}^c$ compared to the true Fisher–Rao distance ${\rho }_{\mathcal{N}}$:

$${\kappa }_c{\rho }_{\mathcal{N}}(N_1,N_2)\ge \kappa {\rho }_{\mathcal{N}}(N_1,N_2)\ge {\tilde{\rho }}_{\mathcal{N}}^c(N_1,N_2)\approx {\rho }_{\mathcal{N}}(N_1,N_2)\ge {\rho }_{\mathrm{CO}}(N_1,N_2).$$

Let us now report some numerical experiments of our approximated Fisher–Rao distances ${\tilde{\rho }}_{\mathcal{N}}^x$ with $x\in \{l,m,e,\mathrm{em},\mathrm{CO}\}$. Although that dissimilarity ${\tilde{\rho }}_{\mathcal{N}}$ is positive–definite, it does not satisfy the triangular inequality of metric distances (e.g., Riemannian distances ${\rho }_{\mathcal{N}}$ and ${\rho }_{\mathrm{CO}}$).

First, we draw multivariate normals by sampling means $\mu \sim \mathrm{Unif}(0,1)$ and sample covariance matrices $\Sigma$ as follows: We draw a lower triangular matrix L with entries $L_{ij}$ iid sampled from $\mathrm{Unif}(0,1)$, and take $\Sigma =L{L}^{\top }$. We use $T=1000$ samples on curves and repeat the experiment 1000 times to gather average statistics on ${\kappa }_c$’s of curves. Results are summarized in Table 1.

Table 1. First set of experiments demonstrates the advantage of the $c_{\mathrm{CO}}\left(t\right)$ curve.

d	${\mathit{\kappa }}_{\mathbf{CO}}$	${\mathit{\kappa }}_{\mathit{l}}$	${\mathit{\kappa }}_{\mathit{e}}$	${\mathit{\kappa }}_{\mathit{m}}$	${\mathit{\kappa }}_{\mathbf{em}}$
1	1.0025	1.0414	1.1521	1.0236	1.0154
2	1.0167	1.0841	1.1923	1.0631	1.0416
3	1.0182	1.8997	2.6072	1.9965	1.07988
4	1.0207	2.0793	1.8080	2.1687	1.1873
5	1.0324	4.1207	12.3804	5.6170	4.2349

For that scenario that the C&O curve (either ${\overline{c}}_{\mathrm{CO}}\in \overline{\mathcal{N}}$ or $c_{\mathrm{CO}}\in \mathcal{N}$) performs best compared to the linear interpolation curves with respect to source parameter (l), mixture geodesic (m), exponential geodesic (e), or exponential-mixture mid-curve ($\mathrm{em}$). Let us point out that we sample ${\gamma }_{\mathcal{P}}({\overline{P}}_1,{\overline{P}}_2;\frac{i}{T})$ for $i\in \{0,\dots ,T\}$.

Strapasson, Porto, and Costa [38] (SPC)reported the following upper bound on the Fisher–Rao distance between multivariate normals

$${\rho }_{\mathrm{CO}}(N_1,N_2)\le {\rho }_{\mathcal{N}}(N_1,N_2)\le U_{\mathrm{SPC}}(N_1,N_2),$$

with:

$$U_{\mathrm{SPC}}(N_1,N_2)=\sqrt{2\sum _{i=1}^d{log}^2\left(\frac{\sqrt{{(1+D_{ii})}^2+{\mu }_i^2}+\sqrt{{(1-D_{ii})}^2+{\mu }_i^2}}{\sqrt{{(1+D_{ii})}^2+{\mu }_i^2}-\sqrt{{(1-D_{ii})}^2+{\mu }_i^2}}\right)},$$

(35)

where $\Sigma ={\Sigma }_1^{-\frac{1}{2}}{\Sigma }_2{\Sigma }_1^{-\frac{1}{2}}$, $\Sigma =\Omega D{\Omega }^{\top }$ is the eigen decomposition, and $\mu ={\Omega }^{\top }{\Sigma }_1^{-\frac{1}{2}}({\mu }_2-{\mu }_1)$. This upper bound performs better when the normals are well-separated and worse than the $\sqrt{D_J}$-upper bound when the normals are close to each other.

Let us compare ${\rho }_{\mathrm{CO}}(N_1,N_2)$ with ${\rho }_{\mathcal{N}}(N_1,N_2)\approx {\tilde{\rho }}^{c_{\mathrm{CO}}}(N_1,N_2)$ and the upper bound $U(N_1,N_2)$ by averaging over 1000 trials with $N_1$ and $N_2$ chosen randomly as before and $T=1000$. We have ${\rho }_{\mathrm{CO}}(N_1,N_2)\le {\rho }_{\mathcal{N}}(N_1,N_2)\approx {\tilde{\rho }}^{c_{\mathrm{CO}}}(N_1,N_2)\le U(N_1,N_2)$. Table 2 shows that our Fisher–Rao approximation is close to the lower bound (and hence to the underlying true Fisher–Rao distance) and that the upper bound is about twice the lower bound for that particular scenario.

Table 2. Comparing our Fisher–Rao approximation with the Calvo and Oller lower bound and the upper bound of [38].

d	${\mathit{\rho }}_{\mathbf{CO}}({\mathit{N}}_1,{\mathit{N}}_2)$	${\tilde{\mathit{\rho }}}^{{\mathit{c}}_{\mathbf{CO}}}({\mathit{N}}_1,{\mathit{N}}_2)$	$\mathit{U}({\mathit{N}}_1,{\mathit{N}}_2)$
1	1.7563	1.8020	3.1654
2	3.2213	3.3194	6.012
3	4.6022	4.7642	8.7204
4	5.9517	6.1927	11.3990
5	7.156	7.3866	13.8774

Second, since the distances are invariant under the action of the affine group, we can set wlog. $N_1=(0,I)$ (standard normal distribution) and let $N_2=\mathrm{diag}(u_1,\dots ,u_d)$ where $u_i\sim \mathrm{Unif}(0,a)$. As normals $N_1$ and $N_2$ separate from each other, we notice experimentally that the performance of the $c_{\mathrm{CO}}$ curve degrades in the second experiment with $a=5$ (see Table 3): Indeed, the mixture geodesic works experimentally better than the C&O curve when $d\ge 11$.

Table 3. Second set of experiments shows limitations of the $c_{\mathrm{CO}}\left(t\right)$ curve.

d	${\mathit{\kappa }}_{\mathbf{CO}}$	${\mathit{\kappa }}_{\mathit{l}}$	${\mathit{\kappa }}_{\mathit{e}}$	${\mathit{\kappa }}_{\mathit{m}}$
1	1.0569	1.1405	1.139	1.0734
5	1.1599	1.4696	1.5201	1.1819
10	1.2180	1.6963	1.7887	1.2184
11	1.2260	1.7333	1.8285	1.2235
12	1.2301	1.7568	1.8539	1.2282
15	1.2484	1.8403	1.9557	1.2367
20	1.2707	1.9519	2.0851	1.2466

Figure 14 display the various curves considered for approximating the Fisher–Rao distance between bivariate normal distributions: For a curve $c\left(t\right)$, we visualize its corresponding bivariate normal distributions $({\mu }_{c\left(t\right)},{\Sigma }_{c\left(t\right)})$ at several increment steps $t\in [0,1]$ by plotting the ellipsoid

$$E_{c\left(t\right)}={\mu }_{c\left(t\right)}+\left\{L^{\top }x,x=(cos\theta ,sin\theta ),\theta \in [0,2\pi )\right\},$$

where ${\Sigma }_{c\left(t\right)}=L_{c\left(t\right)}{L}_{c\left(t\right)}^{\top }$.

Figure 14. Visualizing at discrete positions (10 increment steps between 0 and 1) some curves used to approximate the Fisher–Rao distance between two bivariate normal distributions: (a) exponential geodesic $c^e={\gamma }_{\mathcal{N}}^e$ (red), (b) mixture geodesic $c^m={\gamma }_{\mathcal{N}}^m$ (blue), (c) mid-mixture-exponential curve $c^{\mathrm{em}}$ (purple), (d) projected Calvo and Oller curve $c^{\mathrm{CO}}$ (green), (e) $c^{\lambda }$: ordinary linear interpolation in $\lambda$ (yellow), and (f) All superposed curves at once.

Example 2. 

Let us report some numerical results for bivariate normals with $T=1000$:

• We use the following example of Han and Park [39] (Equation (26)):

  $$N_1=\left(\left[\begin{array}{c}0\hfill \\ 0\hfill \end{array}\right],\left[\begin{array}{cc}1\hfill & 0\hfill \\ 0\hfill & 0.1\hfill \end{array}\right]\right),N_2=\left(\left[\begin{array}{c}1\hfill \\ 1\hfill \end{array}\right],\left[\begin{array}{cc}0.1\hfill & 0\hfill \\ 0\hfill & 1\hfill \end{array}\right]\right).$$

  Their geodesic shooting algorithm [39] evaluates the Fisher–Rao distance to ${\rho }_{\mathcal{N}}(N_1,N_2)\approx \mathbf{3.1329}$ (precision ${10}^{-5}$).
  We obtain:

  –

  Calvo and Oller lower bound: ${\rho }_{\mathrm{CO}}(N_1,N_2)\approx \mathbf{3.0470}$,

  –

  Upper bound using Equation (15): $7.92179$,

  –

  SPC upper bound (Equation (35)): $U_{\mathrm{SPC}}(N_1,N_2)\approx 5.4302$,

  –

  $\sqrt{D_J}$ upper bound: $U_{\sqrt{J}}(N_1,N_2)\approx \mathbf{4.3704}$,

  –

  ${\tilde{\rho }}_{\mathcal{N}}^{\lambda }(N_1,N_2)\approx 3.4496$,

  –

  ${\tilde{\rho }}_{\mathcal{N}}^m(N_1,N_2)\approx 3.5775$,

  –

  ${\tilde{\rho }}_{\mathcal{N}}^e(N_1,N_2)\approx 3.7314$,

  –

  ${\tilde{\rho }}_{\mathcal{N}}^{\mathrm{em}}(N_1,N_2)\approx 3.1672$,

  –

  ${\tilde{\rho }}_{\mathcal{N}}^{\mathrm{CO}}(N_1,N_2)\approx \mathbf{3.1391}$.

  In that setting, the $\sqrt{D_J}$ upper bound is better than the upper bound of Equation (35), and the projected Calvo and Oller geodesic yields the best approximation of the Fisher–Rao distance (Figure 15) with an absolute error of $0.0062$ (about $0.2\%$ relative error). When $T=10$, we have ${\tilde{\rho }}_{\mathcal{N}}^{\mathrm{CO}}(N_1,N_2)\approx 3.1530$, when $T=100$, we obtain ${\tilde{\rho }}_{\mathcal{N}}^{\mathrm{CO}}(N_1,N_2)\approx 3.1136$, and when $T=500$ we obtain ${\tilde{\rho }}_{\mathcal{N}}^{\mathrm{CO}}(N_1,N_2)\approx 3.1362$ (which is better than the approximation obtained for $T=1000$). Figure 16 shows the fluctuations of the approximation of the Fisher–Rao distance by the projected C&O curve when T ranges from 3 to 100.
  

  Figure 15. Comparison of our approximation curves with the Fisher–Rao geodesic (f) obtained by geodesic shooting (Figure 5 of [39]). Exponential (a) and mixture (b) geodesics with the mid-exponential-mixture curve (c), and the projected C&O curve (d). Superposed curves (e) and comparison with geodesic shooting (Figure 5 of [39]). Beware that color coding is not related between (a) and (f), and scale for depicting ellipsoids are different.
  

  Figure 16. Approximation of the Fisher–Rao distance obtained using the projected C&O curve when T ranges from 3 to 100 [39].
• Bivariate normal $N_1=(0,I)$ and bivariate normal $N_2=({\mu }_2,{\Sigma }_2)$ with ${\mu }_2={\left[10\right]}^{\top }$ and ${\Sigma }_2=\left[\begin{array}{cc}1\hfill & -1\hfill \\ -1\hfill & 2\hfill \end{array}\right]$. We obtain

  –

  Calvo and Oller lower bound:$1.4498$

  –

  Upper bound of Equation (35): $2.6072$

  –

  $\sqrt{D_J}$ upper bound:$1.5811$

  –

  ${\tilde{\rho }}^{\lambda }$: $1.5068$

  –

  ${\tilde{\rho }}^m$: $1.5320$

  –

  ${\tilde{\rho }}^e$: $1.5456$

  –

  ${\tilde{\rho }}^{\mathrm{em}}$: $1.4681$

  –

  ${\tilde{\rho }}^{\mathrm{co}}$:$1.4673$

• Bivariate normal $N_1=(0,I)$ and bivariate normal $N_2=({\mu }_2,{\Sigma }_2)$ with ${\mu }_2={\left[50\right]}^{\top }$ and ${\Sigma }_2=\left[\begin{array}{cc}1\hfill & -1\hfill \\ -1\hfill & 2\hfill \end{array}\right]$. We get:

  –

  Calvo and Oller lower bound:$3.6852$

  –

  Upper bound of Equation (35):$6.0392$

  –

  $\sqrt{D_J}$ upper bound: $6.2048$

  –

  ${\tilde{\rho }}^{\lambda }$: $5.7319$

  –

  ${\tilde{\rho }}^m$: $4.4039$

  –

  ${\tilde{\rho }}^e$: $5.9205$

  –

  ${\tilde{\rho }}^{\mathrm{em}}$:$4.2901$

  –

  ${\tilde{\rho }}^{\mathrm{co}}$: $4.3786$

See Supplementary Materials for further experiments.

4. Approximating the Smallest Enclosing Fisher–Rao Ball of MVNs

We may use these closed-form distance ${\rho }_{\mathrm{CO}}(N,N^{\prime })$ between N and $N^{\prime }$ to compute an approximation (of the center) of the smallest enclosing Fisher–Rao ball $B^{*}=\mathrm{ball}(C^{*},r^{*})$ of a set $\mathcal{G}=\{N_1=({\mu }_1,{\Sigma }_1),\dots ,N_n=({\mu }_n,{\Sigma }_n)\}$ of nd-variate normal distributions:

$$C^{*}=arg\underset{C\in \mathcal{N}}{min}\underset{i\in \{1,\dots ,n\}}{max}{\rho }_{\mathcal{N}}(C,N_i)$$

where $\mathrm{ball}(C,r)=\{N\in \mathcal{N}:{\rho }_{\mathcal{N}}(C,N)\le r\}$.

The method proceeds as follows:

• First, we convert MVN set $\mathcal{G}$ into the equivalent set of $(d+1)$-dimensional SPD matrices $\overline{\mathcal{G}}=\{{\overline{P}}_i=f\left(N_i\right)\}$ using the C&O embedding. We relax the problem of approximating the circumcenter $C^{*}$ of the smallest enclosing Fisher–Rao ball by

  $$P^{*}=arg\underset{P\in \mathbb{P}(d+1)}{min}\underset{i\in \{1,\dots ,n\}}{max}{\rho }_{\mathrm{CO}}(P,{\overline{P}}_i).$$
• Second, we approximate the center of the smallest enclosing Riemannian ball of $\overline{\mathcal{G}}$ using the iterative smallest enclosing Riemannian ball algorithm in [66] with say $T=1000$ iterations. Let $\tilde{P}\in \mathbb{P}(d+1)$ denote this approximation center: $P_T={\mathrm{RieSEB}}_{\mathrm{SPD}}(\overline{\mathcal{G}},T)$.
• Finally, we project back $P_T$ onto $\overline{\mathcal{N}}$: ${\overline{P}}_T={\mathrm{proj}}_{\overline{\mathcal{N}}}\left(P_T\right)$. We return ${\overline{P}}_T$ as the approximation of $C^{*}$.

Algorithm [66] ${\mathrm{RieSEB}}_{\mathrm{SPD}}(\{P_1,\dots ,P_n\},T)$ is described for a set of SPD matrices $\{P_1,\dots ,P_n\}$ as follows:

• Let $C_1\leftarrow P_1$
• For $t=1$ to T

  –

  Compute the index of the SPD matrix which is farthest from the current circumcenter $C_t$:

  $$f_t=arg\underset{i\in \{1,\dots ,n\}}{max}{\rho }_{\mathrm{SPD}}(C_t,P_i)$$

  –

  Update the circumcenter by walking along the geodesic linking $C_t$ to $P_{f_t}$:

  $$C_{t+1}={\gamma }_{\mathrm{SPD}}\left(C_t,P_{f_t};\frac{1}{t+1}\right)=C_t^{\frac{1}{2}}{\left(C_t^{-\frac{1}{2}}{P}_{f_t}{C}_t^{-\frac{1}{2}}\right)}^{\frac{1}{t+1}}{C}_t^{\frac{1}{2}}$$

• Return $C_T$

The convergence of the algorithm ${\mathrm{RieSEB}}_{\mathrm{SPD}}$ follows from the fact that the SPD trace manifold is a Hadamard manifold (with negative sectional curvatures). See [66] for proof of convergence.

The SPD distance ${\rho }_{\mathcal{P}}(C_T,{\overline{C}}_T)$ indicates the quality of the approximation. Figure 17 shows the result of implementing this heuristic.

Figure 17. Approximation of the smallest enclosing Riemannian ball of a set of n bivariate normals $N_i=N({\mu }_i,{\Sigma }_i)$ with respect to C&O distance ${\rho }_{\mathrm{CO}}$ (the approximate circumcenter ${\overline{C}}_T$ is depicted as a red ellipse): (a) $n=8$ with different covariance matrices, (b) $n=8$ with identical covariance matrices amount to the smallest enclosing ball of a set of n points $\left\{{\mu }_i\right\}$, (c) $n=2$ displays the midpoint of the C&O geodesic visualized as an equivalent bivariate normal distribution in the sample space.

Let us notice that when all MVNs share the same covariance matrix $\Sigma$, we have from Equation (18) or Equation (23) that ${\rho }_{\mathcal{N}}({\mu }_1,\Sigma ),N({\mu }_2,\Sigma )$ and ${\rho }_{\mathrm{CO}}(N({\mu }_1,\Sigma ),N({\mu }_2,\Sigma ))$ are strictly increasing function of their Mahalanobis distance. Using the Cholesky decomposition ${\Sigma }^{-1}=L{L}^{\top }$, we deduce that the smallest Fisher–Rao enclosing ball coincides with the smallest Calvo and Oller enclosing ball, and the circumcenter of that ball can be found as an ordinary Euclidean circumcenter [83] (Figure 17b). Please note that in 1D, we can find the exact smallest enclosing Fisher–Rao ball as an equivalent smallest enclosing ball in hyperbolic geometry.

Furthermore, we may extend the computation of the approximated circumcenter to k-center clustering [84] of n multivariate normal distributions. Since the circumcenter of the clusters is approximated and not exact, we extend straightforwardly the variational approach of k-means described in [85] to k-center clustering. An application of k-center clustering of MVNs is to simplify a Gaussian mixture model [42] (GMM).

Similarly, we can consider other Riemannian distances with closed-form formulas between MVNs such as the Killing distance in the symmetric space [82] (see Appendix C) or the Siegel-based distance proposed in Appendix D.

5. Some Information–Geometric Properties of the C&O Embedding

In information geometry [1], the manifold $\mathcal{N}$ admits a dual structure denoted by the quadruple

$$(\mathcal{N},g_{\mathcal{N}}^{\mathrm{Fisher}},{\nabla }_{\mathcal{N}}^e,{\nabla }_{\mathcal{N}}^m),$$

when equipped with the exponential connection ${\nabla }_{\mathcal{N}}^e$ and the mixture connection ${\nabla }_{\mathcal{N}}^m$. The connections ${\nabla }_{\mathcal{N}}^e$ and ${\nabla }_{\mathcal{N}}^m$ are said to be dual since $\frac{{\nabla }_{\mathcal{N}}^e+{\nabla }_{\mathcal{N}}^m}{2}={\overline{\nabla }}_{\mathcal{N}}$, the Levi–Civita connection induced by $g_{\mathcal{N}}^{\mathrm{Fisher}}$. Furthermore, by viewing $\mathcal{N}$ as an exponential family $\left\{p_{\theta }\right\}$ with natural parameter $\theta =({\theta }_v,{\theta }_M)$ (using the sufficient statistics [80]$(x,-x{x}^{\top })$), and taking the convex log-normalizer function $F_{\mathcal{N}}\left(\theta \right)$ of the normals, we can build a dually flat space [1] where the canonical divergence amounts to a Bregman divergence which coincides with the reverse Kullback–Leibler divergence [30,86] (KLD). The Legendre duality

$$F^{*}\left(\eta \right)=⟨\nabla F\left(\theta \right),\eta ⟩-F(\nabla F\left(\theta \right))$$

(with $⟨(v_1,M_1),(v_2,M_2)⟩=\mathrm{tr}(v_1{v}_2^{\top }+M_1{M}_2^{\top })=v_1·v_2+\mathrm{tr}\left(M_1{M}_2^{\top }\right)$) yields: $\theta =({\theta }_v,{\theta }_M)=\left({\Sigma }^{-1}\mu ,\frac{1}{2}{\Sigma }^{-1}\right)$,

$$F_{\mathcal{N}}\left(\theta \right)=\frac{1}{2}\left(dlog\pi -log|{\theta }_M|+\frac{1}{2}{\theta }_v^{\top }{\theta }_M^{-1}{\theta }_v\right),$$

$\eta =({\eta }_v,{\eta }_M)=\nabla F_{\mathcal{N}}\left(\theta \right)=\left(\frac{1}{2}{\theta }_M^{-1}{\theta }_v,{\theta }_M^{-1}\right)$,

$$F_{\mathcal{N}}^{*}\left(\eta \right)=-\frac{1}{2}\left(log(1+{\eta }_v^{\top }{\eta }_M^{-1}{\eta }_v)+log|-{\eta }_M|+d(log2\pi e)\right),$$

and we have

$$B_{F_{\mathcal{N}}}({\theta }_1,{\theta }_2)=D_{\mathrm{KL}}^{*}(p_{{\lambda }_1}:p_{{\lambda }_2})=D_{\mathrm{KL}}(p_{{\lambda }_2}:p_{{\lambda }_1})=B_{F_{\mathcal{N}}^{*}}({\eta }_2:{\eta }_1),$$

where $D_{\mathrm{KL}}^{*}[p:q]=D_{\mathrm{KL}}[q:p]$ is the reverse KLD.

In a dually flat space, we can express the canonical divergence as a Fenchel–Young divergence using the mixed coordinate systems $B_{F_{\mathcal{N}}}({\theta }_1:{\theta }_2)=Y_{F_{\mathcal{N}}}({\theta }_1:{\eta }_2)$ where ${\eta }_i=\nabla F_{\mathcal{N}}\left({\theta }_i\right)$ and

$$Y_{F_{\mathcal{N}}}({\theta }_1:{\eta }_2):=F_{\mathcal{N}}\left({\theta }_1\right)+F_{\mathcal{N}}^{*}\left({\eta }_2\right)-⟨{\theta }_1,{\eta }_2⟩.$$

The moment $\eta$-parameterization of a normal is $(\eta =\mu ,H=-\Sigma -\mu {\mu }^{\top })$ with its reciprocal function $(\lambda =\eta ,\Lambda =-H-\eta {\eta }^{\top })$.

Let $F_{\mathcal{P}}\left(P\right)=F_{\mathcal{N}}(0,P)$, $\overline{\theta }=\frac{1}{2}{\overline{P}}^{-1}$, $\overline{\eta }=\nabla F_{\mathcal{P}}\left(\overline{\theta }\right)$. Then we have the following proposition which proves that the Fenchel–Young divergences in $\mathcal{N}$ and $\overline{\mathcal{N}}$ (as a submanifold of $\mathcal{P}$) coincide:

Proposition 4. 

We have

$$\begin{array}{ccc}\hfill D_{\mathrm{KL}}[p_{{\mu }_1,{\Sigma }_1}:p_{{\mu }_2,{\Sigma }_2}]& =& B_{F_{\mathcal{N}}}({\theta }_2:{\theta }_1)=Y_{F_{\mathcal{N}}}({\theta }_2:{\eta }_1)=Y_{F_{\mathcal{P}}}({\overline{\theta }}_2:{\overline{\eta }}_1)\hfill \\ & =& B_{F_{\mathcal{P}}}({\overline{\theta }}_2:{\overline{\theta }}_1)=D_{\mathrm{KL}}[p_{0,{\overline{P}}_1=f({\mu }_1,{\Sigma }_2)}:p_{0,{\overline{P}}_2=f({\mu }_2,{\Sigma }_2)}].\hfill \end{array}$$

Consider now the ${\nabla }^e$-geodesics and ${\nabla }^m$-geodesics on $\mathcal{N}$ (linear interpolation with respect to natural and dual moment parameterizations, respectively): ${\gamma }_{\mathcal{N}}^e(N_1,N_2;t)=({\mu }_t^e,{\Sigma }_t^e)$ and ${\gamma }_{\mathcal{N}}^m(N_1,N_2;t)=({\mu }_t^m,{\Sigma }_t^m)$.

Proposition 5 

(Mixture geodesics preserved). The mixture geodesics are preserved by the embedding f: $f\left({\gamma }_{\mathcal{N}}^m(N_1,N_2;t)\right)={\gamma }_{\mathcal{P}}^m(f\left(N_1\right),f\left(N_2\right);t)$. The exponential geodesics are preserved for the subspace of $\mathcal{N}$ with fixed mean μ: ${\mathcal{N}}_{\mu }$.

Proof. 

For the m-geodesics, let us check that

$$f({\mu }_t^m,{\Sigma }_t^m)=\left[\begin{array}{cc}{\Sigma }_t^m+{\mu }_t^m{{\mu }_t^m}^{\top }\hfill & {\mu }_t^m\hfill \\ {\left({\mu }_t^m\right)}^{\top }\hfill & 1\hfill \end{array}\right]=t\underset{{\overline{P}}_1}{\underbrace{f({\mu }_1,{\Sigma }_1)}}+(1-t)\underset{{\overline{P}}_2}{\underbrace{f({\mu }_2,{\Sigma }_2)}},$$

since ${\Sigma }_t^m+{\mu }_t{{\mu }_t^m}^{\top }={\overline{\Sigma }}_t+t{\mu }_1{\mu }_1^{\top }+(1-t){\mu }_2{\mu }_2^{\top }$$=t({\Sigma }_1+{\mu }_1{\mu }_1^{\top })+(1-t)({\Sigma }_2+{\mu }_2{\mu }_2^{\top })$. Thus, we have $f\left({\gamma }_{\mathcal{N}}^m(N_1,N_2;t)\right)={\gamma }_{\mathcal{P}}^m({\overline{P}}_1,{\overline{P}}_2;t)$. □

Therefore, all algorithms on $\mathcal{N}$ which only require m-geodesics or m-projections [1] by minimizing the right-hand side of the KLD can be implemented by algorithms on $\mathcal{P}$. See, for example, the minimum enclosing ball approximation algorithm called BBC in [87]. Notice that ${\overline{\mathcal{N}}}_{\mu }$ (fixed mean normal submanifolds) preserve both mixture and exponential geodesics: The submanifolds ${\overline{\mathcal{N}}}_{\mu }$ are said to be doubly autoparallel [88].

Remark 5. 

In [2] (p. 355), exercises 13.8 and 13.9 ask to prove the equivalence of the following statements for $\mathcal{S}$ a submanifold of $\mathcal{M}$:

• $\mathcal{S}$ is an exponential family ⇔$\mathcal{S}$ is ${\nabla }^1$-autoparallel in $\mathcal{M}$ (exercise 13.8),
• $\mathcal{S}$ is a mixture family ⇔$\mathcal{S}$ is ${\nabla }^{-1}$-autoparallel in $\mathcal{M}$ (exercise 13.9).

Let $\overline{P}=\left[\begin{array}{cc}\Sigma +\mu {\mu }^{\top }\hfill & \mu \hfill \\ {\mu }^{\top }\hfill & 1\hfill \end{array}\right]$ (with $|\overline{P}|=|\Sigma |$), ${\overline{P}}^{-1}=\left[\begin{array}{cc}{\Sigma }^{-1}\hfill & -{\Sigma }^{-1}\mu \hfill \\ -{\mu }^{\top }{\Sigma }^{-1}\hfill & 1+{\mu }^{\top }{\Sigma }^{-1}\mu \hfill \end{array}\right]$, and $y=(x,1)$. Then we have

$$\begin{array}{ccc}\hfill q_{\overline{P}}\left(y\right)& =& \frac{1}{{\left(2\pi \right)}^{\frac{d+1}{2}}\sqrt{|\overline{P}|}}exp\left(-\frac{1}{2}{y}^{\top }{\overline{P}}^{-1}y\right),\hfill \\ & =& \frac{1}{{\left(2\pi \right)}^{\frac{d+1}{2}}\sqrt{|\Sigma |}}exp\left(-\frac{1}{2}{y}^{\top }{\overline{P}}^{-1}y\right),\hfill \\ & =& \frac{1}{{\left(2\pi \right)}^{\frac{d+1}{2}}\sqrt{|\Sigma |}}exp\left(\left[x^{\top }1\right]\left[\begin{array}{cc}{\Sigma }^{-1}\hfill & -{\Sigma }^{-1}\mu \hfill \\ -{\mu }^{\top }{\Sigma }^{-1}\hfill & 1+{\mu }^{\top }{\Sigma }^{-1}\mu \hfill \end{array}\right]\left[\begin{array}{c}x\hfill \\ 1\hfill \end{array}\right]\right).\hfill \end{array}$$

Thus, $\overline{\mathcal{N}}=\left\{q_{\overline{P}}(x,1)\right\}$ is an exponential family. Therefore, we deduce that $\mathcal{P}$ is ${\nabla }^e$-autoparallel in $\mathcal{P}$. However, $\overline{\mathcal{N}}$ is not a mixture family and thus $\mathcal{P}$ is not ${\nabla }^m$-autoparallel in $\mathcal{P}$.

6. Conclusions and Discussion

In general, the Fisher–Rao distance between multivariate normals (MVNs) is not known in closed form. In practice, the Fisher–Rao distance is usually approximated by costly geodesic shooting techniques [39,40,41] which requires time-consuming computations of the Riemannian exponential map and are nevertheless limited to normals within a short range of each other. In this work, we consider a simple alternative approach for approximating the Fisher–Rao distance by approximating the Riemannian lengths of curves, which admits closed-form parameterizations. In particular, we considered the mixed exponential-mixture curved and the projected symmetric positive–definite matrix geodesic obtained from Calvo and Oller isometric submanifold embedding into the SPD cone [19]. We summarize our method to approximate ${\rho }_{\mathcal{N}}(N_1,N_2)$ between $N_1=N({\mu }_1,{\Sigma }_1)$ and $N_2=N({\mu }_2,{\Sigma }_2)$ as follows:

$${\tilde{\rho }}_T^{\mathrm{CO}}(N_1,N_2):=\frac{1}{T}\sum _{i=1}^{T-1}\sqrt{D_J\left[{\overline{S}}_t,{\overline{S}}_{t+1}\right]},$$

where

$${\overline{S}}_t={\mathrm{proj}}_{\overline{\mathcal{N}}}\left(S_t\right),{\mathrm{proj}}_{\overline{\mathcal{N}}}\left(\left[\begin{array}{cc}\Sigma +\beta \mu {\mu }^{\top }\hfill & \beta \mu \hfill \\ \beta {\mu }^{\top }\hfill & \beta \hfill \end{array}\right]\right)=\left[\begin{array}{cc}\Sigma +\mu {\mu }^{\top }\hfill & {\mu }^{\top }\hfill \\ \mu \hfill & 1\hfill \end{array}\right]$$

and

$$S_t={\overline{P}}_1^{\frac{1}{2}}{\left({\overline{P}}_1^{-\frac{1}{2}}{\overline{P}}_2^{\frac{1}{2}}{\overline{P}}_1^{-\frac{1}{2}}\right)}^{\frac{t}{T}}{\overline{P}}_1^{\frac{1}{2}}$$

with

$${\overline{P}}_1=f\left(N_1\right)=\left[\begin{array}{cc}{\Sigma }_1+{\mu }_1{\mu }_1^{\top }\hfill & {\mu }_1\hfill \\ {\mu }_1^{\top }\hfill & 1\hfill \end{array}\right],{\overline{P}}_2=f\left(N_2\right)=\left[\begin{array}{cc}{\Sigma }_2+{\mu }_2{\mu }_2^{\top }\hfill & {\mu }_2\hfill \\ {\mu }_2^{\top }\hfill & 1\hfill \end{array}\right].$$

We proved the following sandwich bounds of our approximation

$${\rho }_{\mathcal{N}}(N_1,N_2)\le {\tilde{\rho }}_T^{\mathrm{CO}}(N_1,N_2)\le {\rho }_{\mathcal{N}}(N_1,N_2)+2{\delta }_T^{\mathrm{CO}}({\overline{P}}_1,{\overline{P}}_2),$$

where

$${\delta }_T^{\mathrm{CO}}(P_1,P_2):=\frac{1}{T}\sum _{i=1}^T{\rho }_{\mathcal{P}}(S_t,{\overline{S}}_t).$$

Notice that we may calculate equivalently $D_J\left[{\overline{S}}_t,{\overline{S}}_{t+1}\right]$ as $D_J[G_t,G_{t+1}]$ where $G_i=f^{-1}\left({\overline{S}}_i\right)=N(m_i,C_i)$ for $i\in \{0,\dots ,T\}$ (see Proposition 3).

We also reported a fast way to upper bound the Fisher–Rao distance by the square root of Jeffreys’ divergence: ${\rho }_{\mathcal{N}}(N_1,N_2)\le \sqrt{D_J[N_1,N_2]}$ which is tight at infinitesimal scale. In practice, this upper bound beats the upper bound of [38] when normal distributions are not too far from each other. Finally, we show that not only is Calvo and Oller SPD submanifold embedding [19] isometric, but it also preserves the Kullback–Leibler divergence, the Fenchel–Young divergence, and the mixture geodesics. Our approximation technique extends to elliptical distribution, which generalizes multivariate normal distributions [32,55]. Moreover, we obtained a closed form for the Fisher–Rao distance between normals sharing the same covariance matrix using the technique of maximal invariance under the action of the affine group in Section 1.5. We may also consider other distances different from the Fisher–Rao distance, which admits a closed-form formula: For example, the Calvo and Oller metric distance [19] (a lower bound on the Fisher–Rao distance) or the metric distance proposed in [82] (see Appendix C) whose geodesics enjoys the asymptotic property of the Fisher–Rao geodesics [89]). The C&O distance is very well-suited for short Fisher–Rao distances while the symmetric space distance is well-tailored for large Fisher–Rao distances. The calculations of these closed-form distances rely on generalized eigenvalues. We also propose an embedding of normals into the Siegel upper space in Appendix D. To conclude, let us propose yet another alternative distance, The Hilbert projective distance on the SPD cone [90], which only needs to calculate the minimal and maximal eigenvalues (say, using the power iteration method [91]):

$${\rho }_{\mathrm{Hilbert}}(P_1,P_2)=log\left(\frac{{\lambda }_{\max }\left(P_1^{-1}{P}_2\right)}{{\lambda }_{\min }\left(P_1^{-1}{P}_2\right)}\right).$$

(36)

The dissimilarity is said projective on the SPD cone because ${\rho }_{\mathrm{Hilbert}}(P_1,P_2)=0$ if and only if $P_1=\lambda P_2$ for some $\lambda >0$. However, let us notice that it yields a proper metric distance on $\overline{\mathcal{N}}$:

$${\rho }_{\mathrm{Hilbert}}(N_1,N_2):={\rho }_{\mathrm{Hilbert}}({\overline{P}}_1,{\overline{P}}_2),$$

since ${\overline{P}}_1=\lambda {\overline{P}}_2$ if and only if $\lambda =1$ because the array element ${\left(P_1\right)}_{d+1,d+1}={\left(P_2\right)}_{d+1,d+1}=1$, i.e., ${\overline{P}}_1={\overline{P}}_2$ implying $P_1=P_2$ by the isometric diffeomorphism f.

Notice that since ${\lambda }_{\max }\left(P\right)={\lambda }_{\min }\left(P^{-1}\right)$, ${\lambda }_{\min }\left(P\right)={\lambda }_{\max }\left(P^{-1}\right)$,

${\lambda }_{\max }\left(P_1{P}_2\right)\le {\lambda }_{\max }\left(P_1\right){\lambda }_{\max }\left(P_2\right)$, and ${\lambda }_{\min }\left(P_1{P}_2\right)\ge {\lambda }_{\min }\left(P_1\right){\lambda }_{\min }\left(P_2\right)$, we have the following upper bound on Hilbert distance: ${\rho }_{\mathrm{Hilbert}}(P_1,P_2)\le log\frac{{\lambda }_{\max }\left(P_1\right)}{{\lambda }_{\min }\left(P_1\right)}+log\frac{{\lambda }_{\max }\left(P_2\right)}{{\lambda }_{\min }\left(P_2\right)}$.

Appendix A.1. Parametric Equations of the Fisher–Rao Geodesics between Univariate Normal Distributions

The Fisher–Rao geodesics ${\gamma }_{\mathcal{N}}^{\mathrm{FR}}(N_1,N_2)$ on the Fisher–Rao univariate normal manifolds are either vertical line segments when ${\mu }_1={\mu }_2$, or semi-circle with origin on the x-axis and x-axis stretched by $\sqrt{2}$ [92] (Figure A1):

$${\gamma }_{\mathcal{N}}^{\mathrm{FR}}({\mu }_1,{\sigma }_1;{\mu }_2,{\sigma }_2)=\left\{\begin{array}{cc}(\mu ,(1-t){\sigma }_1+t{\sigma }_2),\hfill & {\mu }_1={\mu }_2=\mu \hfill \\ (\sqrt{2}(c+rcost,rsint),t\in [min\{{\theta }_1,{\theta }_2\},max\{{\theta }_1,{\theta }_2\}],\hfill & {\mu }_1\ne {\mu }_2,\hfill \end{array}\right.,$$

where

$$c=\frac{\frac{1}{2}({\mu }_2^2-{\mu }_1^2)+{\sigma }_2^2-{\sigma }_1^2}{\sqrt{2}({\mu }_1-{\mu }_2)},r=\sqrt{{\left(\frac{{\mu }_i}{\sqrt{2}}-c\right)}^2+{\sigma }_i^2},i\in \{1,2\},$$

and

$${\theta }_i=arctan\left(\frac{{\sigma }_i}{\frac{{\mu }_i}{\sqrt{2}}-c}\right),i\in \{1,2\},$$

provided that ${\theta }_i\ge 0$ for $i\in \{1,2\}$ (otherwise, we let ${\theta }_i\leftarrow {\theta }_i+\pi$).

Figure A1. Visualizing some Fisher–Rao geodesics of univariate normal distributions on the stretched Poincaré upper plane (semi-circles with origin on the x-axis and stretched by $\sqrt{2}$ on the x-axis). Full geodesics are plotted with a thin gray style and geodesic arcs are plotted with a thick black style.

Notice that it is remarkable that the Fisher–Rao distance between normal distributions is available in closed form: Indeed, the Euclidean length (with respect to the Euclidean metric) of semi-ellipse curves (perimeters) is not known in closed form but can be expressed using the so-called complete elliptic integral of the second kind [93].

Appendix A.2. Geodesics with Initial Values on the Multivariate Fisher–Rao Normal Manifold

The geodesic equation is given by

$$\left\{\begin{array}{ccc}\ddot{\mu }-\dot{\Sigma }{\Sigma }^{-1}\dot{\mu }\hfill & =& 0,\hfill \\ \ddot{\Sigma }+\dot{\mu }{\dot{\mu }}^{\top }-\dot{\Sigma }{\Sigma }^{-1}\dot{\Sigma }\hfill & =& 0.\hfill \end{array}\right.$$

We concisely report the parametric geodesics using another variant of the natural parameters of the normal distributions (slightly differing from the $\theta$-coordinate system since natural parameters can be chosen up to a fixed affine transformation by changing accordingly the sufficient statistics by the inverse affine transformation) viewed as an exponential family:

$$\left(\xi ={\Sigma }^{-1}\mu ,\Xi ={\Sigma }^{-1}\right).$$

In general, the geodesics with boundary values ${\gamma }_{\mathcal{N}}^{\mathrm{Fisher}}(N_1,N_2;t)$ are not known in closed form. However, Calvo and Oller [48] (Theorem 3.1 and Corollary 1) reported the explicit equations of the geodesics when the initial values are given, i.e., ${\gamma }_{\mathcal{N}}^{\mathrm{Fisher}}(N_0,v_0;t)$ where $v_0={\dot{\gamma }}_{\mathcal{N}}^{\mathrm{Fisher}}(N_0,v_0;0)=(\dot{\xi }(0),\dot{\Xi }(0))$ is in $T_{N_0}\mathcal{M}$ and ${\gamma }_{\mathcal{N}}^{\mathrm{Fisher}}(N_0,v_0;0)=N_0$.

Let

$$\begin{array}{ccc}\hfill B& =& -\Xi {\left(0\right)}^{-\frac{1}{2}}\dot{\Xi }\left(0\right)\Xi {\left(0\right)}^{-\frac{1}{2}},\hfill \\ \hfill a& =& \Xi {\left(0\right)}^{-\frac{1}{2}}\dot{\xi }\left(0\right)+B{\Xi }_0^{-\frac{1}{2}}\xi \left(0\right),\hfill \\ \hfill G& =& {(B^2+2a{a}^{\top })}^{\frac{1}{2}},\hfill \end{array}$$

and $G^{†}$ be the Moore–Penrose generalized inverse matrix of G: $G^{†}={\left(G^{\top }G\right)}^{-1}{G}^{\top }$ or $G^{†}=G^{\top }{\left(G{G}^{\top }\right)}^{-1}$. The Moore–Penrose pseudo-inverse matrix can be replaced by any other pseudo-inverse matrix $G^{-}$ [48].

Then we have $(\xi \left(t\right),\Xi \left(t\right))={\gamma }_{\mathcal{N}}^{\mathrm{Fisher}}(N_0,v_0;t)$ with

$$\begin{array}{c}\hfill R\left(t\right)=\mathrm{Cosh}\left(\frac{1}{2}Gt\right)-B{G}^{†}\mathrm{Sinh}\left(\frac{1}{2}Gt\right),\\ \hfill \Xi \left(t\right)=\Xi {\left(0\right)}^{\frac{1}{2}}R\left(t\right)R{\left(t\right)}^{\top }\Xi {\left(0\right)}^{\frac{1}{2}},\\ \hfill \xi \left(t\right)=2\Xi {\left(0\right)}^{\frac{1}{2}}R\left(t\right)\mathrm{Sinh}\left(\frac{1}{2}Gt\right)G^{†}a+\Xi \left(t\right){\Xi }^{-1}\left(0\right)\xi \left(0\right),\end{array}$$

where the Cosh and Sinh functions of a matrix M are defined by the following absolutely convergent series [48] (Equation (9), p. 122):

$$\begin{array}{ccc}\hfill \mathrm{Sinh}\left(M\right)& =& M+\sum _{i=1}^{\infty }\frac{M^{2i+1}}{(2i+1)!},\hfill \\ \hfill \mathrm{Cosh}\left(M\right)& =& I+\sum _{i=1}^{\infty }\frac{M^{2i}}{\left(2i\right)!},\hfill \end{array}$$

and satisfies the identity ${\mathrm{Sinh}}^2\left(M\right)+{\mathrm{Cosh}}^2\left(M\right)=I$. The matrix Cosh and Sinh functions can be calculated from the eigendecomposition of $M=O\mathrm{diag}({\lambda }_1,\dots ,{\lambda }_d)O^{\top }$ as follows:

$$\begin{array}{ccc}\hfill \mathrm{Sinh}\left(M\right)& =& O\mathrm{diag}(\sinh \left({\lambda }_1\right),\dots ,\sinh \left({\lambda }_d\right))O^{\top },\sinh \left(u\right)=\frac{e^u-e^{-u}}{2}=\sum _{i=0}^{\infty }\frac{u^{2i+1}}{(2i+1)!},\hfill \\ \hfill \mathrm{Cosh}\left(M\right)& =& O\mathrm{diag}(\cosh \left({\lambda }_1\right),\dots ,\cosh \left({\lambda }_d\right))O^{\top },\cosh \left(u\right)=\frac{e^u+e^{-u}}{2}=\sum _{i=0}^{\infty }\frac{u^{2i}}{\left(2i\right)!}.\hfill \end{array}$$

When we restrict the manifold to a totally geodesic submanifold ${\mathcal{M}}_{\mu }=\{P\succ 0\}$, the geodesic equation becomes $\ddot{P}-\dot{P}{P}^{-1}\dot{P}=0$, and the geodesic with initial values $P\left(0\right)=P$ and $\dot{P}\left(0\right)=S\in \mathrm{Sym}$ is:

$$P\left(t\right)=P^{\frac{1}{2}}exp\left(t{P}^{-\frac{1}{2}}S{P}^{-\frac{1}{2}}\right)P^{\frac{1}{2}}.$$

The geodesic with boundary values $P\left(0\right)=P_1$ and $P\left(1\right)=P_2$ is

$$P\left(t\right)=P_1^{\frac{1}{2}}exp\left(t\mathrm{Log}(P_1^{-\frac{1}{2}}{P}_2{P}_1^{-\frac{1}{2}})\right)P_1^{\frac{1}{2}}.$$

Furthermore, we can convert a geodesic with boundary values ${\gamma }_{\mathbb{P}}(P_1,P_2;t)$ to an equivalent geodesic with initial values ${\gamma }_{\mathbb{P}}(P,S;t)$ by letting

$$S=P_1^{\frac{1}{2}}\mathrm{Log}\left(P_1^{-\frac{1}{2}}{P}_2{P}_1^{-\frac{1}{2}}\right)P_1^{\frac{1}{2}}.$$