Revisiting Chernoff Information with Likelihood Ratio Exponential Families

Abstract

The Chernoff information between two probability measures is a statistical divergence measuring their deviation defined as their maximally skewed Bhattacharyya distance. Although the Chernoff information was originally introduced for bounding the Bayes error in statistical hypothesis testing, the divergence found many other applications due to its empirical robustness property found in applications ranging from information fusion to quantum information. From the viewpoint of information theory, the Chernoff information can also be interpreted as a minmax symmetrization of the Kullback–Leibler divergence. In this paper, we first revisit the Chernoff information between two densities of a measurable Lebesgue space by considering the exponential families induced by their geometric mixtures: The so-called likelihood ratio exponential families. Second, we show how to (i) solve exactly the Chernoff information between any two univariate Gaussian distributions or get a closed-form formula using symbolic computing, (ii) report a closed-form formula of the Chernoff information of centered Gaussians with scaled covariance matrices and (iii) use a fast numerical scheme to approximate the Chernoff information between any two multivariate Gaussian distributions.

1. Introduction

1.1. Chernoff Information: Definition and Related Statistical Divergences

Let $(𝒳,\mathcal{A})$ denote a measurable space [1] with sample space $\mathcal{X}$ and finite $\sigma$-algebra $\mathcal{A}$ of events. A measure P is absolutely continuous with respect to another measure Q if $P\left(A\right)=0$ whenever $Q\left(A\right)=0$: P is said dominated by Q and written notationally for short as $P\ll Q$. We shall write $P\cancel{\ll }Q$ when P is not dominated by Q. When $P\ll Q$, we denote by $\frac{\mathrm{d}P}{\mathrm{d}Q}$ the Radon–Nikodym density [1] of P with respect to Q.

The Chernoff information [2], also called Chernoff information number [3,4] or the Chernoff divergence [5,6], is the following symmetric measure of dissimilarity (see Appendix A for some background on statistical divergences) between any two comparable probability measures P and Q dominated by $\mu$:

$$D_C[P,Q]:=\underset{\alpha \in (0,1)}{max}-\log {\rho }_{\alpha }[P:Q]=D_C[Q,P],$$

(1)

$${\rho }_{\alpha }[P:Q]:=\int p^{\alpha }{q}^{1-\alpha }\mathrm{d}\mu ={\rho }_{1-\alpha }[Q:P],$$

(2)

is the $\alpha$-skewed Bhattacharyya affinity coefficient [7] (a coefficient measuring the similarity of two densities). In the remainder, we shall use the following conventions: When a (dis)similarity is asymmetric (e.g., ${\rho }_{\alpha }[P:Q]$), we use the colon notation “:” to separate its arguments. When the (dis)similarity is symmetric (e.g., $D_C[P,Q]$), we use the comma notation “,” to separate its arguments.

The $\alpha$-skewed Bhattacharyya coefficients are always upper bounded by 1 and are strictly greater than zero for non-empty intersecting support (non-singular PMs):

$$0<{\rho }_{\alpha }[P:Q]\le 1.$$

A proof can be obtained by applying Hölder’s inequality (see also Appendix A for an alternative proof).

Since the affinity coefficient ${\rho }_{\alpha }[P:Q]$ does not depend on the underlying dominating measure $\mu$ [4], we shall write $D_C[p,q]$ instead of $D_C[P,Q]$ in the reminder.

Let $D_{B,\alpha }[p:q]$ denote the $\alpha$-skewed Bhattacharyya distance [7,8]:

$$D_{B,\alpha }[p:q]:=-\log {\rho }_{\alpha }[P:Q]=D_{B,1-\alpha }[q:p],$$

(3)

The $\alpha$-skewed Bhattacharyya distances are not metric distances since they can be asymmetric and do not satisfy the triangle inequality even when $\alpha =\frac{1}{2}$.

Thus, the Chernoff information is defined as the maximal skewed Bhattacharyya distance:

$$D_C[p,q]=\underset{\alpha \in (0,1)}{max}{D}_{B,\alpha }[p:q].$$

(4)

Grünwald [9,10] called the skewed Bhattacharyya coefficients and distances the $\alpha$-Rényi affinity and the unnormalized Rényi divergence, respectively, (see Section 19.6 of [9]) since the Rényi divergence [11,12] is defined by

$$D_{R,\alpha }[P:Q]=\frac{1}{\alpha -1}\log \int p^{\alpha }{q}^{1-\alpha }\mathrm{d}\mu =\frac{1}{1-\alpha }{D}_{B,\alpha }[P:Q].$$

(5)

Thus $D_{B,\alpha }[P:Q]=(1-\alpha )D_{R,\alpha }[P:Q]$ can be interpreted as the unnormalized Rényi divergence in [9]. However, let us notice that the Rényi $\alpha$-divergences are defined in general for a wider range $\alpha \in [0,\infty ]\setminus \left\{1\right\}$ with ${lim}_{\alpha \to 1}{D}_{R,\alpha }[P:Q]=D_{\mathrm{KL}}[P:Q]$ but the skew Bhattacharyya distances are defined for $\alpha \in (0,1)$ in general.

The Chernoff information was originally introduced to upper bound the probability error of misclassification in Bayesian binary hypothesis testing [2] where the optimal skewing parameter ${\alpha }^{*}$ such that $D_C[p,q]=D_{B,{\alpha }^{*}}[p:q]$ is referred to in the statistical literature as the Chernoff error exponent [13,14,15] or Chernoff exponent [16,17] for short. The Chernoff information has found many other fruitful applications beyond its original statistical hypothesis testing scope like in computer vision [18], information fusion [19], time-series clustering [20], and more generally in machine learning [21] (just to cite a few use cases). It has been observed empirically that the Chernoff information exhibits superior robustness [22] compared to the Kullback–Leibler divergence in distributed fusion of Gaussian Mixtures Models [19] (GMMs) or in target detection in radar sensor network [23]. The Chernoff information has also been used for analysis deepfake detection performance of Generative Adversarial Networks [22] (GANs).

1.2. Prior Work and Contributions

The Chernoff information between any two categorical distributions (multinomial distributions with one trial also called “multinoulli” since they are extensions of the Bernoulli distributions) has been very well-studied and described in many reference textbooks of information theory or statistics (e.g., see Section 12.9 of [13]). The Chernoff information between two probability distributions of an exponential family was considered from the viewpoint of information geometry in [24], and in the general case from the viewpoint of unnormalized Rényi divergences in [11] (Theorem 32). By replacing the weighted geometric mean in the definition of the Bhattacharyya coefficient ${\rho }_{\alpha }$ of Equation (2) by an arbitrary weighted mean, the generalized Bhattacharyya coefficient and its associated divergences including the Chernoff information was generalized in [25]. The geometry of the Chernoff error exponent was studied in [26,27] when dealing with a finite set of mutually absolutely probability distributions $P_1,\dots ,P_n$. In this case, the Chernoff information amounts to the minimum pairwise Chernoff information of the probability distributions [28]:

$$D_C[P_1,\dots ,P_n]:=\underset{i\in \{1,\dots ,n\}\ne j\in \{1,\dots ,n\}}{min}{D}_C[P_i,P_j].$$

We summarize our contributions as follows: In Section 2, we study the Chernoff information between two given mutually non-singular probability measures P and Q by considering their “exponential arc” [29] as a special 1D exponential family termed a Likelihood Ratio Exponential Family (LREF) in [10]. We show that the optimal skewing value (Chernoff exponent) defining their Chernoff information is unique (Proposition 1) and can be characterized geometrically on the Banach vector space $L^1\left(\mu \right)$ of equivalence classes of measurable functions (i.e., two functions $f_1$ and $f_2$ are said equivalent in $L^1\left(\mu \right)$ if they are equal $\mu$-almost everywhere, abbreviated as $\mu$-a.e. in the remainder) for which their absolute value is Lebesgue integrable (Proposition 4). This geometric characterization allows us to design a generic dichotomic search algorithm (Algorithm 1) to approximate the Chernoff optimal skewing parameter, generalizing the prior work [24]. When P and Q belong to a same exponential family, we recover in Section 3 the results of [24]. This geometric characterization also allows us to reinterpret the Chernoff information as a minmax symmetrization of the Kullback–Leibler divergence, and we define by analogy the forward and reverse Chernoff–Bregman divergences in Section 4 (Definition 2). In Section 5, we consider the Chernoff information between Gaussian distributions: We show that the optimality condition for the Chernoff information between univariate Gaussian distributions can be solved exactly and report a closed-form formula for the Chernoff information between any two univariate Gaussian distributions (Proposition 10). For multivariate Gaussian distributions, we show how to implement the dichotomic search algorithms to approximate the Chernoff information, and report a closed-form formula for the Chernoff information between two centered multivariate Gaussian distributions with scaled covariance matrices (Proposition 11). Finally, we conclude in Section 7.

2. Chernoff Information from the Viewpoint of Likelihood Ratio Exponential Families

2.1. LREFs and the Chernoff Information

Recall that $L^1\left(\mu \right)$ denotes the Lebesgue vector space of measurable functions f such that ${\int }_{\mathcal{X}}\left|f\right|\mathrm{d}\mu <\infty$. Given two prescribed densities p and q of $L^1\left(\mu \right)$, consider building a uniparametric exponential family [30] ${\mathcal{E}}_{pq}$ which consists of the weighted geometric mixtures of p and q:

$${\mathcal{E}}_{pq}:=\left\{{\left(pq\right)}_{\alpha }^G\left(x\right):=\frac{p{\left(x\right)}^{\alpha }q{\left(x\right)}^{1-\alpha }}{Z_{pq}\left(\alpha \right)}:\alpha \in \Theta \right\},$$

(6)

where

$$Z_{pq}\left(\alpha \right)={\int }_{\mathcal{X}}p{\left(x\right)}^{\alpha }q{\left(x\right)}^{1-\alpha }\mathrm{d}\mu \left(x\right)={\rho }_{\alpha }[p:q]$$

(7)

denotes the normalizer (or partition function) of the geometric mixture

$${\left(pq\right)}_{\alpha }^G\left(x\right)\propto p{\left(x\right)}^{\alpha }q{\left(x\right)}^{1-\alpha }$$

so that ${\int }_{\mathcal{X}}{\left(pq\right)}_{\alpha }^G\mathrm{d}\mu =1$. Parameter space $\Theta$ is defined as the set of $\alpha$ values which yieds convergence of the definite integral $Z_{pq}\left(\alpha \right)$:

$$\Theta :=\{\alpha \in \mathbb{R}:Z_{pq}\left(\alpha \right)<\infty \}.$$

(8)

Let us express the density ${\left(pq\right)}_{\alpha }^G$ in the canonical form (∗) of exponential families [30]:

$$\begin{array}{ccc}\hfill {\left(pq\right)}_{\alpha }^G\left(x\right)& =& \exp \left(\alpha \log \frac{p\left(x\right)}{q\left(x\right)}-\log Z_{pq}\left(\alpha \right)\right)q\left(x\right),\hfill \end{array}$$

(9)

$$\begin{array}{ccc}& \stackrel{*}{=:}& \exp \left(\alpha t\left(x\right)-F_{pq}\left(\alpha \right)+k\left(x\right)\right).\hfill \end{array}$$

(10)

It follows from this decomposition that $\alpha \in \Theta \subset \mathbb{R}$ is the scalar natural parameter, $t\left(x\right)=\log \frac{p\left(x\right)}{q\left(x\right)}$ denotes the sufficient statistic (minimal when $p\left(x\right)\ne q\left(x\right)$$\mu$-a.e.), $k\left(x\right)=\log q\left(x\right)$ is an auxiliary carrier term wrt. measure $\mu$ (i.e., measure $\mathrm{d}\nu \left(x\right)=q\left(x\right)\mathrm{d}\mu \left(x\right)$), and

$$F_{pq}\left(\alpha \right)=\log Z_{pq}\left(\alpha \right)=-D_{B,\alpha }[p:q]<0$$

(11)

is the log-normalizer (or log-partition or cumulant function). Since the sufficient statistic is the logarithm of the likelihood ratio of $p\left(x\right)$ and $q\left(x\right)$, Grünwald [9] (Section 19.6) termed ${\mathcal{E}}_{pq}$ a Likelihood Ratio Exponential Family (LREF). See also [31] for applications of LREFs to Markov chain Monte Carlo (McMC) methods.

We have $p={\left(pq\right)}_1^G$ and $q={\left(pq\right)}_0^G$. Thus, let ${\alpha }_p=1$ and ${\alpha }_q=0$, and let us interpret geometrically $\{{\left(pq\right)}_{\alpha }^G,\alpha \in \Theta \}$ as a maximal exponential arc [29,32,33] where $\Theta \subseteq \mathbb{R}$ is an interval. We denote by ${\mathcal{E}}_{\overline{pq}}$ the open exponential arc with extremities p and q.

Since the log-normalizers $F\left(\theta \right)$ of exponential families are always strictly convex and real analytic [30] (i.e., $F\left(\theta \right)\in C^{\omega }\left(\mathbb{R}\right)$), we deduce that $D_{B,\alpha }[p:q]=-F_{pq}\left(\alpha \right)$ is strictly concave and real analytic. Moreover, we have $D_{B,0}[p:q]=D_{B,1}[p:q]=0$. Hence, the Chernoff optimal skewing parameter ${\alpha }^{*}$ is unique when $p\ne q$$\mu$-a.e., and we get the Chernoff information calculated as

$$D_C[p:q]=D_{B,{\alpha }^{*}}(p,q).$$

See Figure 1 for a plot of the strictly concave function $D_{B,\alpha }[p:q]$ and the strictly convex function $F_{pq}\left(\alpha \right)=-D_{B,\alpha }[p:q]$ when $p=p_{0,1}$ is the standard normal density and $q=p_{1,2}$ is a normal density of mean 1 and variance 2.

Consider the full natural parameter space ${\Theta }_{pq}$ of ${\mathcal{E}}_{pq}$:

$${\Theta }_{pq}=\left\{\alpha \in \mathbb{R}:{\rho }_{\alpha }(p:q)<+\infty \right\}.$$

(12)

The natural parameter space ${\Theta }_{pq}$ is always convex [30] and since ${\rho }_0(p:q)={\rho }_1(p:q)=1$, we necessarily have $(0,1)\in {\Theta }_{pq}$ but not necessarily $[0,1]\in {\Theta }_{pq}$ as detailed in the following remark:

Remark 1.

In order to be an exponential family, the densities ${\left(pq\right)}_{\alpha }^G$ shall have the same coinciding support for all values of α belonging to the natural parameter space. The support of the geometric mixture density ${\left(pq\right)}_{\alpha }^G$ is

$$\mathrm{supp}\left({\left(pq\right)}_{\alpha }^G\right)=\left\{\begin{array}{cc}\mathrm{supp}\left(p\right)\cap \mathrm{supp}\left(q\right),\hfill & \alpha \in {\Theta }_{pq}\setminus \{0,1\}\hfill \\ \mathrm{supp}\left(p\right),\hfill & \alpha =1\hfill \\ \mathrm{supp}\left(q\right),\hfill & \alpha =0.\hfill \end{array}\right.$$

This condition is trivially satisfied when the supports of p and q coincide, and therefore $[0,1]\subset {\Theta }_{pq}$ in that case. Otherwise, we may consider the common support ${\mathcal{X}}_{pq}=\mathrm{supp}\left(p\right)\cap \mathrm{supp}\left(q\right)$ for $\alpha \in (0,1)$. In this latter case, we are poised to restrict the natural parameter space to ${\Theta }_{pq}=(0,1)$ even if ${\rho }_{\alpha }(p:q)<\infty$ for some α outside that range.

To emphasize that ${\alpha }^{*}$ depends on p and q, we shall use the notation ${\alpha }^{*}(p:q)$ whenever necessary. We have ${\alpha }^{*}(q:p)=1-{\alpha }^{*}(p:q)$, and since $D_{B,\alpha }(p:q)=D_{B,1-\alpha }(q:p)$, and we check that

$$D_C[p,q]=D_{B,{\alpha }^{*}(p:q)}(p:q)=D_{B,{\alpha }^{*}(q:p)}(q:p)=D_C[q,p].$$

Thus the skewing value ${\alpha }^{*}(q:p)$ may be called the conjugate Chernoff exponent (i.e., depends on the convention chosen for interpolating on the exponential arc).

However, since the Chernoff information does not satisfy the triangle inequality, it is not a metric distance and the Chernoff information is called a quasi-distance.

Proposition 1

(Uniqueness of the Chernoff information optimal skewing parameter [11,12]). Let P and Q be two probability measures dominated by a positive measure μ with corresponding Radon–Nikodym densities p and q, respectively. The Chernoff information optimal skewing parameter ${\alpha }^{*}(p:q)$ is unique when $p\ne q$ μ-almost everywhere, and

$$D_C[p,q]=D_{B,{\alpha }^{*}(p:q)}(p:q)=D_{B,{\alpha }^{*}(q:p)}(q:p)=D_C[q,p].$$

When $p=q$ μ-a.e., we have $D_C[p:q]=0$ and ${\alpha }^{*}$ is undefined since it can range in $[0,1]$.

Definition 1.

An exponential family is called regular [30] when the natural parameter space Θ is open, i.e., $\Theta ={\Theta }^{\circ }$ where ${\Theta }^{\circ }$ denotes the interior of Θ (i.e., an open interval).

Proposition 2

(Finite sided Kullback–Leibler divergences). When the LREF ${\mathcal{E}}_{pq}$ is a regular exponential family with natural parameter space $\Theta ⊋[0,1]$, both the forward Kullback–Leibler divergence $D_{\mathrm{KL}}[p:q]$ and the reverse Kullback–Leibler divergence $D_{\mathrm{KL}}[q:p]$ are finite.

Proof. 

A reverse parameter divergence $D^{*}({\theta }_1:{\theta }_2)$ is a parameter divergence on the swapped parameter order: $D^{*}({\theta }_1:{\theta }_2):=D({\theta }_2:{\theta }_1)$. Similarly, a reverse statistical divergence $D^{*}[p:q]$ is a statistical divergence on the swapped parameter order: $D^{*}[p:q]:=D[q:p]$. We shall use the result pioneered in [34,35] that the KLD between two densities $p_{{\theta }_1}$ and $p_{{\theta }_2}$ of a regular exponential family $\mathcal{E}=\{p_{\theta }:\theta \in \Theta \}$ amounts to a reverse Bregman divergence ${\left(B_F\right)}^{*}$ (i.e., a Bregman divergence on swapped parameter order) induced by the log-normalizer of the family:

$$D_{\mathrm{KL}}[p_{{\theta }_1}:p_{{\theta }_2}]={\left(B_F\right)}^{*}({\theta }_1:{\theta }_2)=B_F({\theta }_2:{\theta }_1),$$

(13)

where $B_F$ is the Bregman divergence defined on domain $D=\mathrm{dom}\left(F\right)$ (see Definition 1 of [36]):

$$\begin{array}{ccc}\hfill B_F:& & D\times \mathrm{ri}\left(D\right)\to [0,\infty )\hfill \\ & & ({\theta }_1,{\theta }_2)\mapsto B_F({\theta }_1:{\theta }_2)=F\left({\theta }_1\right)-F\left({\theta }_2\right)-{({\theta }_1-{\theta }_2)}^{\top }\nabla F\left({\theta }_2\right)<+\infty ,\hfill \end{array}$$

where $\mathrm{ri}\left(D\right)$ denotes the relative interior of domain D. Bregman divergences are always finite and the only symmetric Bregman divergences are squared Mahalanobis distances [37] (i.e., with corresponding Bregman generators defining quadratic forms).

For completeness, we recall the proof as follows: We have

$$\log \frac{p_{{\theta }_1}\left(x\right)}{p_{{\theta }_2}\left(x\right)}={({\theta }_1-{\theta }_2)}^{\top }t\left(x\right)-F\left({\theta }_1\right)+F\left({\theta }_2\right).$$

Thus we get

$$\begin{array}{ccc}\hfill D_{\mathrm{KL}}[p_{{\theta }_1}:p_{{\theta }_2}]& =& E_{p_{{\theta }_1}}\left[\log \frac{p_{{\theta }_1}}{p_{{\theta }_2}}\right],\hfill \\ & =& F\left({\theta }_2\right)-F\left({\theta }_1\right)-{({\theta }_1-{\theta }_2)}^{\top }{E}_{p_{{\theta }_1}}\left[t\left(x\right)\right],\hfill \end{array}$$

using the linearity property of the expectation operator. When $\mathcal{E}$ is regular, we also have $E_{p_{{\theta }_1}}\left[t\left(x\right)\right]=\nabla F\left({\theta }_1\right)$ (see [38]), and therefore we get

$$D_{\mathrm{KL}}[p_{{\theta }_1}:p_{{\theta }_2}]=F\left({\theta }_2\right)-F\left({\theta }_1\right)-{({\theta }_1-{\theta }_2)}^{\top }\nabla F\left({\theta }_1\right)=:B_F({\theta }_2:{\theta }_1)={\left(B_F\right)}^{*}({\theta }_1:{\theta }_2).$$

In our LREF setting, we thus have:

$$D_{\mathrm{KL}}[p:q]={\left(B_F\right)}^{*}({\alpha }_p:{\alpha }_q)=B_{F_{pq}}({\alpha }_q:{\alpha }_p)=B_{F_{pq}}(0:1),$$

and $D_{\mathrm{KL}}[q:p]=B_{F_{pq}}({\alpha }_p:{\alpha }_q)=B_{F_{pq}}(1:0)$ where $B_{F_{pq}}({\alpha }_1:{\alpha }_2)$ denotes the following scalar Bregman divergence:

$$B_{F_{pq}}({\alpha }_1:{\alpha }_2)=F_{pq}\left({\alpha }_1\right)-F_{pq}\left({\alpha }_2\right)-({\alpha }_1-{\alpha }_2)F_{pq}^{\prime }\left({\alpha }_2\right).$$

Since $F_{pq}\left(0\right)=F_{pq}\left(1\right)=0$ and $B_{F_{pq}}:\Theta \times \mathrm{ri}(\Theta )\to [0,\infty )$, we have

$$D_{\mathrm{KL}}[p:q]=B_{F_{pq}}({\alpha }_q:{\alpha }_p)=B_{F_{pq}}(0:1)=F_{pq}^{\prime }\left(1\right)<\infty .$$

Similarly

$$D_{\mathrm{KL}}[q:p]=B_{F_{pq}}({\alpha }_p:{\alpha }_q)=B_{F_{pq}}(1:0)=-F_{pq}^{\prime }\left(0\right)<\infty .$$

Notice that since $B_{F_{pq}}({\alpha }_1:{\alpha }_2)>0$, we have $F_{pq}^{\prime }\left(1\right)>0$ and $F_{pq}^{\prime }\left(0\right)<0$ when $p\ne q$$\mu$-almost everywhere (a.e.). Moreover, since $F_{pq}\left(\alpha \right)$ is strictly convex, $F_{pq}^{\prime }\left(\alpha \right)$ is strictly monotonically increasing, and therefore there exists a unique ${\alpha }^{*}\in (0,1)$ such that $F_{pq}^{\prime }\left({\alpha }^{*}\right)=0$.    □

Example 1.

When p and q belongs to a same regular exponential family $\mathcal{E}$ (e.g., p and q are two normal densities), their sided KLDs [37] are both finite. The LREF induced by two Cauchy distributions $p_{l_1,s_1}$ and $p_{l_2,s_2}$ is such that $[0,1]\subset \Theta$ since the skewed Bhattacharyya distance is defined and finite for $\alpha \in \mathbb{R}$ [39]. Therefore the KLDs between two Cauchy distributions are always finite [39], see the closed-form formula in [40].

Remark 2.

If $0\in {\Theta }^{\circ }$, then $B_{F_{pq}}(1:0)<\infty$ and therefore $D_{\mathrm{KL}}[q:p]<\infty$. Since the KLD between a standard Cauchy distribution p and a standard normal distribution q is $+\infty$, we deduce that $D_{\mathrm{KL}}[p:q]\ne B_{F_{pq}}(0:1)$, and therefore $1\notin {\Theta }^{\circ }$. Similarly, when $1\in {\Theta }^{\circ }$, we have $B_{F_{pq}}(0:1)<\infty$ and therefore $D_{\mathrm{KL}}[p:q]<\infty$.

Proposition 3

(Chernoff information expressed as KLDs). (see also Theorem 32 of [11]) We have at the Chernoff information optimal skewing value ${\alpha }^{*}\in (0,1)$ the following identities:

$$D_C[p:q]=D_{\mathrm{KL}}[{\left(pq\right)}_{{\alpha }^{*}}^G:p]=D_{\mathrm{KL}}[{\left(pq\right)}_{{\alpha }^{*}}^G:q].$$

Proof. 

Since the skewed Bhattacharyya distance between two densities $p_{{\theta }_1}$ and $p_{{\theta }_2}$ of an exponential family with log-normalizer F amounts to a skew Jensen divergence for the log-normalizer [8,41], we have:

$$D_{B,\alpha }(p_{{\theta }_1}:p_{{\theta }_2})=J_{F,\alpha }({\theta }_1:{\theta }_2),$$

where the skew Jensen divergence [42] is given by

$$J_{F,\alpha }({\theta }_1:{\theta }_2)=\alpha F\left({\theta }_1\right)+(1-\alpha )F\left({\theta }_2\right)-F(\alpha {\theta }_1+(1-\alpha ){\theta }_2).$$

In the setting of the LREF, we have

$$\begin{array}{ccc}\hfill D_{B,\alpha }({\left(pq\right)}_{{\alpha }_1}^G:{\left(pq\right)}_{{\alpha }_2}^G)& =& J_{F_{pq},\alpha }({\alpha }_1:{\alpha }_2),\hfill \\ & =& \alpha F_{pq}\left({\alpha }_1\right)+(1-\alpha )F_{pq}\left({\alpha }_2\right)-F_{pq}(\alpha {\alpha }_1+(1-\alpha ){\alpha }_2).\hfill \end{array}$$

At the optimal value ${\alpha }^{*}$, we have $F_{pq}^{\prime }\left({\alpha }^{*}\right)=0$. Since $D_{\mathrm{KL}}[{\left(pq\right)}_{{\alpha }^{*}}^G:p]=B_{F_{pq}}(1:{\alpha }^{*})=-F\left({\alpha }^{*}\right)$ and $D_{\mathrm{KL}}[{\left(pq\right)}_{{\alpha }^{*}}^G:q]=B_{F_{pq}}(0:{\alpha }^{*})=-F\left({\alpha }^{*}\right)$ and $D_C[p:q]=-\log {\rho }_{{\alpha }^{*}}(p:q)=J_{F_{pq},{\alpha }^{*}}(1:0)=-F_{pq}\left({\alpha }^{*}\right)$, we get

$$D_C[p:q]=D_{\mathrm{KL}}[{\left(pq\right)}_{{\alpha }^{*}}^G:p]=D_{\mathrm{KL}}[{\left(pq\right)}_{{\alpha }^{*}}^G:q].$$

Figure 2 illustrates the proposition on the plot of the scalar function $F_{pq}\left(\alpha \right)$.    □

Corollary 1.

The Chernoff information optimal skewing value ${\alpha }^{*}(p:q)\in (0,1)$ can be used to calculate the Chernoff information $D_C[p,q]$ as a Bregman divergence induced by the LREF:

$$D_C[p:q]=B_{F_{pq}}[1:{\alpha }^{*}]=B_{F_{pq}}[0:{\alpha }^{*}]=J_{F_{pq},{\alpha }^{*}}(1:0).$$

In general, the divergence $J_F^C({\theta }_1,{\theta }_2)={max}_{\alpha \in (0,1)}{J}_{F,\alpha }({\theta }_1:{\theta }_2)$ is called a Jensen–Chernoff divergence.

Proposition 3 let us interpret the Chernoff information as a special symmetrization of the Kullback–Leibler divergence [43], different from the Jeffreys divergence or the Jensen–Shannon divergence [44]. Indeed, the Chernoff information can be rewritten as

$$D_C[p:q]=\underset{r\in {\mathcal{E}}_{\overline{pq}}}{min}\{D_{\mathrm{KL}}[r:p],D_{\mathrm{KL}}[r:q]\}.$$

(14)

As such, we can interpret the Chernoff information as the radius of a minimum enclosing left-sided Kullback–Leibler ball on the space $L^1\left(\mu \right)$. A related concept is the radius [12] of two densities p and q with respect to Rényi divergences of order $\alpha$ (see Equation (2) of [12]):

$$r_{\alpha }(p,q):=\underset{c}{inf}max\{D_{R,\alpha }[p:c],D_{R,\alpha }[q:c]\}.$$

When $\alpha =1$, the radius is called the Shannon radius [12] since the Rényi divergences of order 1 corresponds to the Kullback–Leibler divergence (relative entropy).

2.2. Geometric Characterization of the Chernoff Information and the Chernoff Information Distribution

Let us term the probability distribution ${\left(PQ\right)}_{{\alpha }^{*}}^G\ll \mu$ with corresponding density ${\left(pq\right)}_{{\alpha }^{*}}^G$ the Chernoff information distribution to avoid confusion with another concept of Chernoff distributions [45] used in statistics. We can characterize geometrically the Chernoff information distribution ${\left(pq\right)}_{{\alpha }^{*}}^G$ on $L^1\left(\mu \right)$ as the intersection of a left-sided Kullback–Leibler divergence bisector:

$${\mathrm{Bi}}_{\mathrm{KL}}^{\mathrm{left}}(p,q):=\left\{r\in L^1\left(\mu \right):D_{\mathrm{KL}}[r:p]=D_{\mathrm{KL}}[r:q]\right\},$$

(15)

with an exponential arc [29]

$${\gamma }^G(p,q):=\left\{{\left(pq\right)}_{\alpha }^G:\alpha \in [0,1]\right\}.$$

(16)

We thus interpret Proposition 3 geometrically by the following proposition (see Figure 3):

Proposition 4

(Geometric characterization of the Chernoff information). On the vector space $L^1\left(\mu \right)$, the Chernoff information distribution is the unique distribution

$${\left(pq\right)}_{{\alpha }^{*}}^G={\gamma }^G(p,q)\cap {\mathrm{Bi}}_{\mathrm{KL}}^{\mathrm{left}}(p,q).$$

The point ${\left(pq\right)}_{{\alpha }^{*}}^G$ has been called the Chernoff point in [24].

Proposition 4 allows us to design a dichotomic search to numerically approximate ${\alpha }^{*}$ as reported in pseudo-code in Algorithm 1 (see also the illustration in Figure 4).

Algorithm 1 Dichotomic search for approximating the Chernoff information by approximating the optimal skewing parameter value $\tilde{\alpha }\approx {\alpha }^{*}$ and reporting $D_C[p:q]\approx D_{\mathrm{KL}}[{\left(pq\right)}_{\tilde{\alpha }}^G:p]$. The search requires $\lceil {\log }_2\frac{1}{ϵ}\rceil$ iterations to guarantee $|{\alpha }^{*}-\tilde{\alpha }|\le ϵ$.

Remark 3.

We do not need to necessarily handle normalized densities p and q since we have for $\alpha \in \mathbb{R}\setminus \{0,1\}$:

$${\left(pq\right)}_{\alpha }^G={\left(\tilde{p}\tilde{q}\right)}_{\alpha }^G,$$

where $p\left(x\right)=\frac{\tilde{p}\left(x\right)}{Z_p}$ and $q\left(x\right)=\frac{\tilde{q}\left(x\right)}{Z_q}$ with $\tilde{p}$ and $\tilde{q}$ denoting the computationally-friendly unnormalized positive densities. This property of geometric mixtures is used in Annealed Importance Sampling [46,47] (AIS), and for designing an asymptotically efficient estimator for computationally-intractable parametric densities [48] ${\tilde{q}}_{\theta }$ (e.g., distributions learned by Boltzmann machines).

2.3. Dual Parameterization of LREFs

The densities ${\left(pq\right)}_{\alpha }^G$ of a LREF can also be parameterized by their dual moment parameter [30] (or mean parameter):

$$\beta =\beta \left(\alpha \right):=E_{{\left(pq\right)}_{\alpha }^G}\left[t\left(x\right)\right]=E_{{\left(pq\right)}_{\alpha }^G}\left[\log \frac{p\left(x\right)}{q\left(x\right)}\right].$$

(17)

When the LREF is regular (and therefore steep [38]), we have $\beta \left(\alpha \right)=F_{pq}^{\prime }\left(\alpha \right)$ and $\alpha ={F_{pq}^{*}}^{\prime }\left(\beta \right)$, where $F_{pq}^{*}$ denotes the Legendre transform of $F_{pq}$. At the optimal value ${\alpha }^{*}$, we have $F_{pq}^{\prime }\left({\alpha }^{*}\right)=0$. Therefore an equivalent condition of optimality is

$$\beta \left({\alpha }^{*}\right)=F_{pq}^{\prime }\left({\alpha }^{*}\right)=0=E_{{\left(pq\right)}_{{\alpha }^{*}}^G}\left[\log \frac{p\left(x\right)}{q\left(x\right)}\right].$$

Notice that when $[0,1]\subset {\Theta }^{\circ }$, we have finite forward and reverse Kullback–Leibler divergences:

• $\alpha =1$, we have ${\left(pq\right)}_1^G=p$ and

  $$\beta \left(1\right)=E_p\left[\log \frac{p\left(x\right)}{q\left(x\right)}\right]=D_{\mathrm{KL}}[p:q]=F_{pq}^{\prime }\left(1\right)>0.$$
• $\alpha =0$, we have ${\left(pq\right)}_0^G=q$ and

  $$\beta \left(0\right)=E_q\left[\log \frac{p\left(x\right)}{q\left(x\right)}\right]=-D_{\mathrm{KL}}[q:p]=F_{pq}^{\prime }\left(0\right)<0.$$

Since $F_{pq}\left(\alpha \right)$ is strictly convex, we have $F_{pq}^{″}\left(\alpha \right)>0$ and $F_{pq}^{\prime }$ is strictly increasing with $F_{pq}^{\prime }\left(0\right)=-D_{\mathrm{KL}}[q:p]<0$ and $F_{pq}^{\prime }\left(1\right)=D_{\mathrm{KL}}[p:q]>0$. The value ${\alpha }^{*}$ is thus the unique value such that $F_{pq}^{\prime }\left({\alpha }^{*}\right)=0$.

Proposition 5

(Dual optimality condition for the Chernoff information). The unique Chernoff information optimal skewing parameter ${\alpha }^{*}$ is such that

$${\mathrm{OC}}_{\alpha }:D_{\mathrm{KL}}[{\left(pq\right)}_{{\alpha }^{*}}^G:p]=D_{\mathrm{KL}}[{\left(pq\right)}_{{\alpha }^{*}}^G:q]\iff {\mathrm{OC}}_{\beta }:\beta \left({\alpha }^{*}\right)=E_{{\left(pq\right)}_{{\alpha }^{*}}^G}\left[\log \frac{p\left(x\right)}{q\left(x\right)}\right]=0.$$

One can understand that the Chernoff information is more robust or stable than a skewed Bhattacharrya distance by considering the derivative of the corresponding skewed Bhattacharrya distance. Consider without loss of generality densities $p_{{\theta }_1}$ and $p_{{\theta }_2}$ of a 1D exponential family. Their skewed Bhattacharrya distances amounts to skew Jensen divergences, and we have:

$$J_{F,\alpha }^{\prime }({\theta }_1:{\theta }_2):=\frac{\mathrm{d}}{\mathrm{d}\alpha }{J}_{F,\alpha }({\theta }_1:{\theta }_2)=F\left({\theta }_1\right)-F\left({\theta }_2\right)-({\theta }_1-{\theta }_2)F^{\prime }(\alpha {\theta }_1+(1-\alpha ){\theta }_2).$$

Since $J_{F,{\alpha }^{*}}$ is by definition maximal, we have $F^{\prime }({\alpha }^{*}{\theta }_1+(1-{\alpha }^{*}){\theta }_2)=0$, and therefore $|J_{F,\alpha }^{\prime }({\theta }_1:{\theta }_2)-J_{F,{\alpha }^{*}}^{\prime }({\theta }_1:{\theta }_2)|=|({\theta }_1-{\theta }_2)F^{\prime }(\alpha {\theta }_1+(1-\alpha ){\theta }_2)|>0$. Further assuming without loss of generality that ${\theta }_2-{\theta }_1=1$, we get $|J_{F,\alpha }^{\prime }({\theta }_1:{\theta }_2)-J_{F,{\alpha }^{*}}^{\prime }({\theta }_1:{\theta }_2)|=|F^{\prime }(\alpha {\theta }_1+(1-\alpha ){\theta }_2)|>0=F^{\prime }({\alpha }^{*}{\theta }_1+(1-{\alpha }^{*}){\theta }_2)$.

As a side remark, let us notice that the Fisher information of a likelihood ratio exponential family ${\mathcal{E}}_{pq}$ is

$$I_{pq}\left(\alpha \right)=-E_{{\left(pq\right)}_{\alpha }^G}\left[{\left(\log {\left(pq\right)}_{\alpha }^G\right)}^{″}\right]=F_{pq}^{″}\left(\alpha \right)>0,$$

and $=F_{pq}^{″}\left(\alpha \right){F^{*}}_{pq}^{″}\left(\beta \right)=1$.

3. Chernoff Information between Densities of an Exponential Family

3.1. General Case

We shall now consider that the densities p and q (with respect to measure $\mu$) belong to a same exponential family [30]:

$$\mathcal{E}=\left\{P_{\lambda }:\frac{\mathrm{d}{P}_{\lambda }}{\mathrm{d}\mu }=p_{\lambda }\left(x\right)=\exp (\theta {\left(\lambda \right)}^{\top }t\left(x\right)-F\left(\theta \left(\lambda \right)\right)),\lambda \in \Lambda \right\},$$

where $\theta \left(\lambda \right)$ denotes the natural parameter associated with the ordinary parameter $\lambda$, $t\left(x\right)$ the sufficient statistic vector and $F\left(\theta \right(\lambda \left)\right)$ the log-normalizer. When $\theta \left(\lambda \right)=\lambda$ and $t\left(x\right)=x$, the exponential family is called a natural exponential family (NEF). The exponential family $\mathcal{E}$ is defined by $\mu$ and $t\left(x\right)$, hence we may write when necessary $\mathcal{E}={\mathcal{E}}_{\mu ,t}$.

Example 2.

The set of univariate Gaussian distributions

$$\mathcal{N}=\{p_{\mu ,{\sigma }^2}\left(x\right):\lambda =(\mu ,{\sigma }^2)\in \Lambda =\mathbb{R}\times {\mathbb{R}}_{++}\}$$

forms an exponential family with the following decomposition terms:

$$\begin{array}{ccc}\hfill \lambda & =& (\mu ,{\sigma }^2)\in \Lambda =\mathbb{R}\times {\mathbb{R}}_{++},\hfill \\ \hfill \theta \left(\lambda \right)& =& \left({\theta }_1=\frac{\mu }{{\sigma }^2},{\theta }_2=-\frac{1}{2{\sigma }^2}\right)\in \Theta =\mathbb{R}\times {\mathbb{R}}_{--},\hfill \\ \hfill t\left(x\right)& =& (x,x^2),\hfill \\ \hfill F\left(\theta \right)& =& -\frac{{\theta }_1^2}{4{\theta }_2}+\frac{1}{2}\log \left(-\frac{\pi }{{\theta }_2}\right),\hfill \end{array}$$

where ${\mathbb{R}}_{++}=\{x\in \mathbb{R}:x>0\}$ and ${\mathbb{R}}_{--}=\{x\in \mathbb{R}:x<0\}$ denotes the set of positive real numbers and negative real numbers, respectively. Letting $v={\sigma }^2$ be the variance parameter, we get the equivalent natural parameters $\left(\frac{\mu }{v},-\frac{1}{2v}\right)$. The log-normalizer can be written using the $(\mu ,v)$-parameterization as $F(\mu ,v)=\frac{1}{2}\log \left(2\pi v\right)+\frac{{\mu }^2}{2v}$ and $\theta =\left({\theta }_1=\frac{\mu }{v},-\frac{1}{2v}\right)$. See Appendix B for further details concerning this normal exponential family.

Notice that we can check easily that the LREF between two densities of an exponential family forms a 1D sub-exponential family of the exponential family:

$$\begin{array}{ccc}\hfill p_{{\theta }_1}{\left(x\right)}^{\alpha }{p}_{{\theta }_2}{\left(x\right)}^{1-\alpha }& \propto & \exp (\langle \alpha {\theta }_1+(1-\alpha ){\theta }_2,t\left(x\right)\rangle -\alpha F\left({\theta }_1\right)-(1-\alpha )F\left({\theta }_2\right)),\hfill \\ & =& p_{\alpha {\theta }_1+(1-\alpha ){\theta }_2}\left(x\right)\exp (F(\alpha {\theta }_1+(1-\alpha ){\theta }_2)-\alpha F\left({\theta }_1\right)-(1-\alpha )F\left({\theta }_2\right))),\hfill \\ & =& p_{\alpha {\theta }_1+(1-\alpha ){\theta }_2}\left(x\right)\exp (-J_{F,\alpha }({\theta }_1:{\theta }_2)),\hfill \end{array}$$

where $J_F$ denote the Jensen divergence induced by F.

The optimal skewing value condition of the Chernoff information between two categorical distributions [13] was extended to densities $p_{{\theta }_1}$ and $p_{{\theta }_2}$ of an exponential family in [24]. The family of categorical distributions with d choices forms an exponential family with natural parameter of dimension $d-1$. Thus, Proposition 7 generalizes the analysis in [13].

Let $p=p_{{\theta }_1}$ and $q=p_{{\theta }_2}$. Then we have the property that exponential families are closed under geometric mixtures:

$${\left(p_{{\theta }_1}{p}_{{\theta }_2}\right)}_{\alpha }^G=p_{\alpha {\theta }_1+(1-\alpha ){\theta }_2}.$$

Since the natural parameter space $\Theta$ is convex, we have $\alpha {\theta }_1+(1-\alpha ){\theta }_2\in \Theta$.

The KLD between two densities $p_{{\theta }_1}$ and $p_{{\theta }_2}$ of a regular exponential family $\mathcal{E}$ amounts to a reverse Bregman divergence for the log-normalizer of $\mathcal{E}$:

$$D_{\mathrm{KL}}[p_{{\theta }_1}:p_{{\theta }_2}]=B_F({\theta }_2:{\theta }_1),$$

where $B_F({\theta }_2:{\theta }_1)$ denotes the Bregman divergence:

$$B_F({\theta }_2:{\theta }_1)=F\left({\theta }_2\right)-F\left({\theta }_1\right)-{({\theta }_2-{\theta }_1)}^{\top }\nabla F\left({\theta }_1\right).$$

Thus, when the exponential family $\mathcal{E}$ is regular, both the forward and reverse KLD are finite, and we can rewrite Proposition 3 to characterize ${\alpha }^{*}$ as follows:

$${\mathrm{OC}}_{\mathrm{EF}}:B_F({\theta }_1:{\theta }_{{\alpha }^{*}})=B_F({\theta }_2:{\theta }_{{\alpha }^{*}}),$$

(18)

where ${\theta }_{{\alpha }^{*}}={\alpha }^{*}{\theta }_1+(1-{\alpha }^{*}){\theta }_2$.

The Legendre–Fenchel transform of $F\left(\theta \right)$ yields the convex conjugate

$$F^{*}\left(\eta \right)=\underset{\theta \in \Theta }{sup}\{{\theta }^{\top }\eta -F\left(\theta \right)\}$$

(19)

with $\eta \left(\theta \right)=\nabla F\left(\theta \right)$. Let $H=\left\{\eta \right(\theta ):\theta \in \Theta \}$ denote the dual moment parameter space also called domain of means. The Legendre transform associates to $(\Theta ,F(\theta \left)\right)$ the convex conjugate $(H,F^{*}\left(\eta \right))$. In order for $(H,F^{*}\left(\eta \right))$ to be of the same well-behaved type of $(\Theta ,F(\theta \left)\right)$, we shall consider convex functions $F\left(\theta \right)$ which are steep, meaning that their gradient diverges when nearing the boundary $\mathrm{bd}(\Theta )$ [49] and thus ensures that domain H is also convex. Steep convex functions are said of Legendre-type, and ${\left({(\Theta ,F\left(\theta \right))}^{*}\right)}^{*}=(\Theta ,F\left(\theta \right))$ (Moreau biconjugation theorem which shows that the Legendre transform is involutive). For Legendre-type functions, there is a one-to-one mapping between parameters $\theta \left(\eta \right)$ and parameters $\eta \left(\theta \right)$ as follows:

$$\theta \left(\eta \right)=\nabla F^{*}\left(\eta \right)={(\nabla F)}^{-1}\left(\eta \right),$$

(20)

and

$$\eta \left(\theta \right)=\nabla F\left(\theta \right)={(\nabla F^{*})}^{-1}\left(\theta \right).$$

(21)

Exponential families with log-normalizers of Legendre-type are called steep exponential families [30]. All regular exponential families are steep, and the maximum likelihood estimator in steep exponential families exists and is unique [38] (with the likelihood equations corresponding to the method of moments for the sufficient statistics). The set of inverse Gaussian distributions form a non-regular but steep exponential family, and the set of singly truncated normal distributions form a non-regular and non-steep exponential family [50] (but the exponential family of doubly truncated normal distributions is regular and hence steep).

For Legende-type convex generators $F\left(\theta \right)$, we can express the Bregman divergence $B_F({\theta }_1:{\theta }_2)$ using the dual Bregman divergence: $B_F({\theta }_1:{\theta }_2)=B_{F^{*}}({\eta }_2:{\eta }_1)$ since there is a one-to-one correspondence between $\eta =\nabla F\left(\theta \right)$ and $\theta =\nabla F^{*}\left(\eta \right)$.

For Legendre-type generators $F\left(\theta \right)$, the Bregman divergence $B_F({\theta }_1:{\theta }_2)$ can be rewritten as the following Fenchel–Young divergence:

$$B_F({\theta }_1:{\theta }_2)=F\left({\theta }_1\right)+F^{*}\left({\eta }_2\right)-{\theta }_1^{\top }{\eta }_2:=Y_{F,F^{*}}({\theta }_1:{\eta }_2).$$

Proposition 6.

(KLD between densities of a regular (and steep) exponential family). The KLD between two densities $p_{{\theta }_1}$ and $p_{{\theta }_2}$ of a regular and steep exponential family can be obtained equivalently as

$$D_{\mathrm{KL}}[p_{{\theta }_1}:p_{{\theta }_2}]=B_F({\theta }_2:{\theta }_1)=Y_{F,F^{*}}({\theta }_2:{\eta }_1)=Y_{F^{*},F}({\eta }_1:{\theta }_2)=B_{F^{*}}({\eta }_1:{\eta }_2),$$

where $F\left(\theta \right)$ and its convex conjugate $F^{*}\left(\eta \right)$ are Legendre-type functions.

Figure 5 illustrates the taxonomy of regularity and steepness of exponential families by a Venn diagram.

It follows that the optimal condition of Equation (18) can be restated as

$${\mathrm{OC}}_{\mathrm{YF}}:Y_{F,F^{*}}({\theta }_1:{\eta }_{{\alpha }^{*}})=Y_{F,F^{*}}({\theta }_2:{\eta }_{{\alpha }^{*}}),$$

(22)

where ${\eta }_{{\alpha }^{*}}=\nabla F^{-1}({\alpha }^{*}{\theta }_1+(1-{\alpha }^{*}){\theta }_2)$. From the equality of Equation (22), we get the following simplified optimality condition:

$${\mathrm{OC}}_{\mathrm{SEF}}:{({\theta }_2-{\theta }_1)}^{\top }{\eta }_{{\alpha }^{*}}=F\left({\theta }_2\right)-F\left({\theta }_1\right),$$

(23)

where ${\eta }_{{\alpha }^{*}}=\nabla F({\alpha }^{*}{\theta }_1+(1-{\alpha }^{*}){\theta }_2)$.

Remark 4.

We can recover (${\mathrm{OC}}_{\mathrm{SEF}}$) by instantiating the equivalent condition $E_{p_{{\overline{\theta }}_{{\alpha }^{*}}}}\left[\log \frac{p_{{\theta }_1}}{p_{{\theta }_2}}\right]=0$. Indeed, since $\log \frac{p_{{\theta }_1}}{p_{{\theta }_2}}={({\theta }_1-{\theta }_2)}^{\top }t\left(x\right)-F\left({\theta }_1\right)+F\left({\theta }_2\right)$, we get

$$\begin{array}{ccc}\hfill E_{p_{{\overline{\theta }}_{{\alpha }^{*}}}}[{({\theta }_1-{\theta }_2)}^{\top }t\left(x\right)-F\left({\theta }_1\right)+F\left({\theta }_2\right)]& =& 0,\hfill \\ \hfill {({\theta }_1-{\theta }_2)}^{\top }{\overline{\eta }}_{{\alpha }^{*}}=F\left({\theta }_1\right)-F\left({\theta }_2\right).\end{array}$$

Since the $\alpha$-skewed Bhattacharyya distance amounts to a $\alpha$-skewed Jensen divergence [8], we get the Chernoff information as

$$\begin{array}{ccc}\hfill D_C[p_{{\lambda }_1}:p_{{\lambda }_2}]& =& J_{F,{\alpha }^{*}}(\theta \left({\lambda }_1\right):\theta \left({\lambda }_2\right)),\hfill \\ & =& B_F({\theta }_1:{\theta }_{{\alpha }^{*}})=B_F({\theta }_2:{\theta }_{{\alpha }^{*}}),\hfill \end{array}$$

where $J_{F,\alpha }({\theta }_1:{\theta }_2)$ is the Jensen divergence:

$$J_{F,\alpha }({\theta }_1:{\theta }_2)=\alpha F\left({\theta }_1\right)+(1-\alpha )F\left({\theta }_2\right)-F(\alpha {\theta }_1+(1-\alpha ){\theta }_2).$$

Notice that we have the induced LREF with log-normalizer expressed as the negative Jensen divergence induced the log-normalizer of $\mathcal{E}$:

$$F_{p_{{\theta }_1}{p}_{{\theta }_2}}\left(\alpha \right)=-\log {\rho }_{\alpha }[p_{{\theta }_1}:p_{{\theta }_2}]=-J_{F,\alpha }({\theta }_1:{\theta }_2).$$

We summarize the result in the following proposition:

Proposition 7.

Let $p_{{\lambda }_1}$ and $p_{{\lambda }_2}$ be two densities of a regular exponential family $\mathcal{E}$ with natural parameter $\theta \left(\lambda \right)$ and log-normalizer $F\left(\theta \right)$. Then the Chernoff information is

$$D_C[p_{{\lambda }_1}:p_{{\lambda }_2}]=J_{F,{\alpha }^{*}}(\theta \left({\lambda }_1\right):\theta \left({\lambda }_2\right))=B_F({\theta }_1:{\theta }_{{\alpha }^{*}})=B_F({\theta }_2:{\theta }_{{\alpha }^{*}}),$$

where ${\theta }_1=\theta \left({\lambda }_1\right)$, ${\theta }_2=\theta \left({\lambda }_2\right)$, and the optimal skewing parameter ${\alpha }^{*}$ is unique and satisfies the following optimality condition:

$${\mathrm{OC}}_{\mathrm{EF}}:{({\theta }_2-{\theta }_1)}^{\top }{\eta }_{{\alpha }^{*}}=F\left({\theta }_2\right)-F\left({\theta }_1\right),$$

(24)

where ${\eta }_{{\alpha }^{*}}=\nabla F({\alpha }^{*}{\theta }_1+(1-{\alpha }^{*}){\theta }_2)=E_{p_{{\alpha }^{*}{\theta }_1+(1-{\alpha }^{*}){\theta }_2}}\left[t\left(x\right)\right]$.

Figure 6 illustrates geometrically the Chernoff point [24] which is the geometric mixture ${\left(p_{{\theta }_1}{p}_{{\theta }_2}\right)}_{{\alpha }^{*}}$ induced by two comparable probability measures $P_{{\theta }_1},P_{{\theta }_2}\ll \mu$.

In information geometry [51], the manifold of densities $M=\{p_{\theta }:\theta \in \Theta \}$ of this exponential family is a dually flat space [51] $\mathcal{M}=(\left\{p_{\theta }\right\},g_F\left(\theta \right)={\nabla }^{2}F\left(\theta \right),{\nabla }^m,{\nabla }^e)$ with respect to the exponential connection ${\nabla }^e$ and the mixture connection ${\nabla }^m$, where $g_F\left(\theta \right)$ is the Fisher information metric expressed in the $\theta$-coordinate system as ${\nabla }^{2}F\left(\theta \right)$ (and in the dual moment parameter $\eta$ as $g_F\left(\eta \right)={\nabla }^2{F}^{*}\left(\eta \right)$). Then the exponential geodesic ${\nabla }^e$ is flat and corresponds to the exponential arc of geometric mixtures when parameterized with the ${\nabla }^e$-affine coordinate system $\theta$.

The left-sided Kullback–Voronoi bisector:

$${\mathrm{Bi}}_{\mathrm{KL}}^{\mathrm{left}}(p_{{\theta }_1},p_{{\theta }_2})=\{p_{\theta }:D_{\mathrm{KL}}[p_{\theta }:p_{{\theta }_1}]=D_{\mathrm{KL}}[p_{\theta }:p_{{\theta }_1}]\}$$

corresponds to a Bregman right-sided bisector [52] and is ${\nabla }^m$ flat (i.e., an affine subspace in the $\eta$-coordinate system):

$${\mathrm{Bi}}_F^{\mathrm{right}}({\theta }_1,{\theta }_2)=\{\theta \in \Theta :B_F({\theta }_1,\theta )=B_F({\theta }_2,\theta )\}.$$

The Chernoff information distribution ${\left(p_{{\theta }_1}{p}_{{\theta }_2}\right)}_{{\alpha }^{*}}^G$ is called the Chernoff point on this exponential family manifold (see Figure 6). Since the Chernoff point is unique and since in general statistical manifolds $(\mathcal{M},g,\nabla ,{\nabla }^{*})$ can be realized by statistical models [53], we deduce the following proposition of interest for information geometry [51]:

Proposition 8.

Let $(\mathcal{M},g,\nabla ,{\nabla }^{*})$ be a dually flat space with corresponding canonical divergence a Bregman divergence $B_F$. Let ${\gamma }_{pq}^e\left(\alpha \right)$ and ${\gamma }_{pq}^m\left(\alpha \right)$ be a e-geodesic and m-geodesic passing through the points p and q of $\mathcal{M}$, respectively. Let ${\mathrm{Bi}}^m(p,q)$ and ${\mathrm{Bi}}^e(p,q)$ be the right-sided ${\nabla }^m$-flat and left-sided ${\nabla }^e$-flat Bregman bisectors, respectively. Then the intersection of ${\gamma }_{pq}^e\left(\alpha \right)$ with ${\mathrm{Bi}}^m(p,q)$ and the intersection of ${\gamma }_{pq}^m\left(\alpha \right)$ with ${\mathrm{Bi}}^e(p,q)$ are unique. The point ${\gamma }_{pq}^e\left(\alpha \right)\cap {\mathrm{Bi}}^m(p,q)$ is called the Chernoff point and the point ${\gamma }_{pq}^m\left(\alpha \right)\cap {\mathrm{Bi}}^e(p,q)$ is termed the reverse or dual Chernoff point.

3.2. Case of One-Dimensional Parameters

When the exponential family has one-dimensional natural parameter $\alpha \in \Theta \subset \mathbb{R}$, we thus get from ${\mathrm{OC}}_{\mathrm{SEF}}$:

$${\eta }_{{\alpha }^{*}}=\frac{F\left({\alpha }_2\right)-F\left({\alpha }_1\right)}{{\alpha }_2-{\alpha }_1}.$$

That is, ${\alpha }^{*}$ can be obtained as the following closed-form formula:

$${\alpha }^{*}=\frac{{F^{\prime }}^{-1}\left(\frac{F\left({\alpha }_2\right)-F\left({\alpha }_1\right)}{{\alpha }_2-{\alpha }_1}\right)-{\alpha }_2}{{\alpha }_1-{\alpha }_2}.$$

(25)

Example 3.

Consider the exponential family $\{p_v\left(x\right):v>0\}$ of 0-centered scale univariate normal distributions with variance $v={\sigma }^2$ and density

$$p_v\left(x\right)=\frac{1}{\sqrt{2\pi v}}\exp \left(-\frac{1}{2}\frac{x^2}{v}\right).$$

The natural parameter corresponding to the sufficient statistic $t\left(x\right)=x^2$ is $\theta =-\frac{1}{2v}$. The log-normalizer is $F\left(\theta \right)=\frac{1}{2}\log \frac{\pi }{-\theta }$. We have $\eta =F^{\prime }\left(\theta \right)=-\frac{1}{2\theta }$ and ${F^{\prime }}^{-1}\left(\eta \right)=-\frac{1}{2\eta }$. It follows that

$${\alpha }^{*}(p_{v_1}:p_{v_2})=\frac{v_1\log \frac{v_1}{v_2}-v_1+v_2}{(v_2-v_1)\log \frac{v_2}{v_1}}.$$

Let $s=\frac{v_2}{v_1}$. Then we can rewrite ${\alpha }^{*}$ as

$${\alpha }^{*}(p_{v_1}:p_{v_2})=\frac{s-1-\log s}{(s-1)\log s}.$$

The Chernoff information is $D_C[p_{v_1},p_{v_2}]=-\log {\rho }_{{\alpha }^{*}}[p_{v_1},p_{v_2}]$, with

$${\rho }_{\alpha }[p_{v_1},p_{v_2}]=\frac{{\sigma }_1^{1-\alpha }{\sigma }_2^{\alpha }}{\sqrt{(1-\alpha ){\sigma }_1^2+\alpha {\sigma }_2^2}}.$$

This result will be generalized in Proposition 11 to multivariate centered Gaussians with scaled covariance matrices.

For multi-dimensional parameters $\theta$, we may consider the one-dimensional LREF ${\mathcal{E}}_{p_{{\theta }_1}{p}_{{\theta }_2}}$ induced by $p_{{\theta }_1}$ and $p_{{\theta }_2}$ with $F_{{\theta }_1,{\theta }_2}\left(\alpha \right)=F((1-\alpha ){\theta }_1+\alpha {\theta }_2)$, and write $F_{pq}^{\prime }\left(\alpha \right)$ as the following directional derivative:

$$\begin{array}{ccc}\hfill {\nabla }_{{\theta }_2-{\theta }_1}{F}_{{\theta }_1,{\theta }_2}\left(\alpha \right)& :=& \underset{ϵ\to 0}{lim}\frac{F({\theta }_1+(ϵ+\alpha )({\theta }_2-{\theta }_1))-F({\theta }_1+\alpha ({\theta }_2-{\theta }_1))}{ϵ},\hfill \end{array}$$

(26)

$$\begin{array}{ccc}& =& {({\theta }_2-{\theta }_1)}^{\top }\nabla F({\theta }_1+\alpha ({\theta }_2-{\theta }_1)),\hfill \end{array}$$

(27)

using a first-order Taylor expansion. Thus, the optimality condition

$${\mathrm{OC}}_{{\mathrm{SEF}}^{\prime }}:F_{{\theta }_1,{\theta }_2}^{\prime }\left(\alpha \right)=0$$

amounts to

$${\mathrm{OC}}_{\mathrm{SEF}}:{({\theta }_2-{\theta }_1)}^{\top }\nabla F({\theta }_1+{\alpha }^{*}({\theta }_2-{\theta }_1))=F\left({\theta }_2\right)-F\left({\theta }_1\right).$$

(28)

This is equivalent to Equation (8) of [24].

Remark 5.

In general, we may consider multivariate Bregman divergences as univariate Bregman divergences: We have

$$B_F({\theta }_1:{\theta }_2)=B_{F_{{\theta }_1,{\theta }_2}}(0:1),\forall {\theta }_1,{\theta }_2\in \Theta$$

(29)

where

$$F_{{\theta }_1,{\theta }_2}\left(u\right):=F({\theta }_1+u({\theta }_2-{\theta }_1)).$$

(30)

The functions $F_{{\theta }_1,{\theta }_2}$ are 1D Bregman generators (i.e., strictly convex and $C^1$), and we have the directional derivative

$$\begin{array}{ccc}\hfill {\nabla }_{{\theta }_2-{\theta }_1}{F}_{{\theta }_1,{\theta }_2}\left(u\right)& =& \underset{ϵ\to 0}{lim}\frac{F({\theta }_1+(ϵ+u)({\theta }_2-{\theta }_1))-F({\theta }_1+u({\theta }_2-{\theta }_1))}{ϵ},\hfill \\ & =& {({\theta }_2-{\theta }_1)}^{\top }\nabla F({\theta }_1+u({\theta }_2-{\theta }_1)),\hfill \end{array}$$

Since $F_{{\theta }_1,{\theta }_2}\left(0\right)=F\left({\theta }_1\right)$, $F_{{\theta }_1,{\theta }_2}\left(1\right)=F\left({\theta }_2\right)$, and $F_{{\theta }_1,{\theta }_2}^{\prime }\left(u\right)={\nabla }_{{\theta }_2-{\theta }_1}{F}_{{\theta }_1,{\theta }_2}\left(u\right)$, it follows that

$$\begin{array}{ccc}\hfill B_{F_{{\theta }_1,{\theta }_2}}(0:1)& =& F_{{\theta }_1,{\theta }_2}\left(0\right)-F_{{\theta }_1,{\theta }_2}\left(1\right)-(0-1){\nabla }_{{\theta }_2-{\theta }_1}{F}_{{\theta }_1,{\theta }_2}\left(1\right),\hfill \\ & =& F\left({\theta }_1\right)-F\left({\theta }_2\right)+{({\theta }_2-{\theta }_1)}^{\top }\nabla F\left({\theta }_2\right)=B_F({\theta }_1:{\theta }_2).\hfill \end{array}$$

Similarly, we can reparameterize Bregman divergences on a k-dimensional simplex by k-dimensional Bregman divergences.

Remark 6.

Closing the loop: The Chernoff information although obtained from the one-dimensional likelihood ratio exponential family yields as a corollary the general multi-parametric exponential families which as a special instance includes the one-dimensional exponential families (e.g., LREFs!).

4. Forward and Reverse Chernoff–Bregman Divergences

In this section, we shall define Chernoff-type symmetrizations of Bregman divergences inspired by the study of Chernoff information, and briefly mention applications of these Chernoff–Bregman divergences in information theory.

4.1. Chernoff–Bregman Divergence

Let us define a Chernoff-like symmetrization of Bregman divergences [43] different from the traditional Jeffreys–Bregman symmetrization:

$$\begin{array}{ccc}\hfill B_F^J({\theta }_1:{\theta }_2)& =& B_F({\theta }_1:{\theta }_2)+B_F({\theta }_2:{\theta }_1),\hfill \\ & =& {({\theta }_1-{\theta }_2)}^{\top }(\nabla F\left({\theta }_1\right)-\nabla F\left({\theta }_2\right)),\hfill \end{array}$$

or Jensen–Shannon-type symmetrization [44,54] which yields a Jensen divergence [42]:

$$\begin{array}{ccc}\hfill B_F^{\mathrm{JS}}({\theta }_1:{\theta }_2)& =& \frac{1}{2}\left(B_F\left({\theta }_1:\frac{{\theta }_1+{\theta }_2}{2}\right)+B_F\left({\theta }_2:\frac{{\theta }_1+{\theta }_2}{2}\right)\right),\hfill \\ & =& \frac{F\left({\theta }_1\right)+F\left({\theta }_2\right)}{2}-F\left(\frac{{\theta }_1+{\theta }_2}{2}\right)=:J_F({\theta }_1,{\theta }_2).\hfill \end{array}$$

Definition 2

(Chernoff–Bregman divergence). Let the Chernoff symmetrization of Bregman divergence $B_F({\theta }_1;{\theta }_2)$ be the forward Chernoff–Bregman divergence $C_F({\theta }_1,{\theta }_2)$ defined by

$$C_F({\theta }_1,{\theta }_2)=\underset{\alpha \in (0,1)}{max}{J}_{F,\alpha }({\theta }_1:{\theta }_2),$$

(31)

where $J_{F,\alpha }$ is the α-skewed Jensen divergence.

The optimization problem in Equation (31) may be equivalently rewritten [43] as ${min}_{\theta }R$ such that both $B_F({\theta }_1:\theta )\le R$ and $B_F({\theta }_2:\theta )\le R$. Thus, the optimal value of $\alpha$ defines the circumcenter ${\theta }^{*}=\alpha {\theta }_1+(1-\alpha ){\theta }_2$ of the minimum enclosing right-sided Bregman sphere [55,56] and the Chernoff–Bregman divergence:

$$C_F({\theta }_1,{\theta }_2)=\underset{\theta }{min}\{B_F({\theta }_1:\theta ),B_F({\theta }_2:\theta )\},$$

(32)

corresponds to the radius of a minimum enclosing Bregman ball. To summarize, this Chernoff symmetrization is a min-max symmetrization, and we have the following identities:

$$\begin{array}{ccc}\hfill C_F({\theta }_1,{\theta }_2)& =& \underset{\theta }{min}\{B_F({\theta }_1:\theta ),B_F({\theta }_2:\theta )\},\hfill \\ & =& \underset{\theta \in \Theta }{min}\{\alpha B_F({\theta }_1:\theta )+(1-\alpha )B_F({\theta }_2:\theta )\},\hfill \\ & =& \underset{\alpha \in (0,1)}{max}\{\alpha B_F({\theta }_1:\alpha {\theta }_1+(1-\alpha ){\theta }_2)+(1-\alpha )B_F({\theta }_2:\alpha {\theta }_1+(1-\alpha ){\theta }_2)\},\hfill \\ & =& \underset{\alpha \in (0,1)}{max}{J}_{F,\alpha }({\theta }_1:{\theta }_2).\hfill \end{array}$$

The second identity shows that the Chernoff symmetrization can be interpreted as a variational Jensen–Shannon-type divergence [54].

Notice that in general $C_F({\theta }_1,{\theta }_2)\ne C_{F^{*}}({\eta }_1,{\eta }_2)$ because the primal and dual geodesics do not coincide. Those geodesics coincide only for symmetric Bregman divergences which are squared Mahalanobis divergences [52].

When $F\left(\theta \right)=F_{\mathrm{Shannon}}\left(\theta \right)={\sum }_{i=1}^D{\theta }_i\log {\theta }_i$ (discrete Shannon negentropy), the Chernoff–Bregman divergence is related to the capacity of a discrete memoryless channel in information theory [13,43].

Conditions for which $C_F{({\theta }_1,{\theta }_2)}^a$ (with $a>0$) becomes a metric have been studied in [43]: For example, $C_{F_{\mathrm{Shannon}}}^{\frac{1}{e}}$ is a metric distance [43] (i.e., $a=\frac{1}{e}\simeq 0.36787944117$). It is also known that the square root of the Chernoff distance between two univariate normal distributions is a metric distance [57].

We can thus use the Bregman generalization of the Badoiu–Clarkson (BC) algorithm [55] (Algorithm 2) to compute an approximation of the smallest enclosing Bregman ball which in turn yields an approximation of the Chernoff–Bregman divergence:    

Algorithm 2 Approximating the circumcenter of the Bregman smallest enclosing ball of two parameters ${\theta }_1$ and ${\theta }_2$.

    Notice that when there are only two points to compute their smallest enclosing Bregman ball, all the arcs ${\left({\theta }^{(i-1)}{\theta }_{f_i}\right)}^G$ are sub-arcs of the exponential arc ${\left({\theta }_1{\theta }_2\right)}^G$. See [55] for convergence results of this iterative algorithm. Let us notice that Algorithm 1 approximates ${\alpha }^{*}$ while the Bregman BC Algorithm 2 approximates in spirit $D_C({\theta }_1,{\theta }_2)$ (and as a byproduct ${\alpha }^{*}$).

Remark 7.

To compute the farthest point to the current circumcenter with respect to Bregman divergence, we need to find the sign of

$$B_F({\theta }_2:\theta )-B_F({\theta }_1:\theta )=F\left({\theta }_2\right)-F\left({\theta }_1\right)-({\theta }_2-{\theta }_1)\nabla F\left(\theta \right).$$

Thus we need to pre-calculate only once $F\left({\theta }_1\right)$ and $F\left({\theta }_2\right)$ which can be costly (e.g., $-\log \det \left(\Sigma \right)$ functions need to be calculated only once when approximating the Chernoff information between Gaussians).

4.2. Reverse Chernoff–Bregman Divergence and Universal Coding

Similarly, we may define the reverse Chernoff–Bregman divergence by considering the minimum enclosing left-sided Bregman ball:

$$C_F^R({\theta }_1,{\theta }_2)=\underset{\theta }{min}\{B_F(\theta :{\theta }_1),B_F(\theta :{\theta }_2)\}.$$

(33)

Thus the reverse Bregman Chernoff divergence $D_C^R[{\theta }_1,{\theta }_2]=R^{*}$ is the radius of a minimum enclosing left-sided Bregman ball.

This reverse Chernoff–Bregman divergence finds application in universal coding in information theory (chapter 13 of [13], pp. 428–433): Let $\mathcal{X}=\{A_1,\dots ,A_d\}$ be a finite discrete alphabet of d letters, and X be a random variable with probability mass function p on $\mathcal{X}$. Let $p_{\lambda }\left(x\right)$ denote the categorical distribution corresponding to X so that $Pr(X=A_i)=p_{\lambda }\left(A_i\right)$ with $\lambda =({\lambda }^1,\dots ,{\lambda }^d)\in {\mathbb{R}}_{++}^d$ and ${\sum }_{i=1}^d{\lambda }^i=1$. The Huffman codeword for $x\in \mathcal{X}$ is of length $l\left(x\right)=-\log p\left(x\right)$ (ignoring integer ceil rounding), and the expected codeword length of X is thus given by Shannon’s entropy $H\left(X\right)=-{\sum }_{x}p\left(x\right)\log p\left(x\right)$.

If we code according to a distribution $p_{{\lambda }^{\prime }}$ instead of the true distribution $p_{\lambda }$, the code is not optimal, and the redundancy $R(p_{\lambda },p_{{\lambda }^{\prime }})$ is defined as the difference between the expected lengths of the codewords for $p_{{\lambda }^{\prime }}$ and $p_{\lambda }$:

$$R(p_{\lambda },p_{{\lambda }^{\prime }})=(-E_{p_{\lambda }}\left[\log p_{{\lambda }^{\prime }}\left(x\right)\right]-(-E_{p_{\lambda }}\left[\log p_{\lambda }\left(x\right)\right])=D_{\mathrm{KL}}[p_{\lambda }:p_{{\lambda }^{\prime }}]\ge 0,$$

where $D_{\mathrm{KL}}$ is the Kullback–Leibler divergence.

Now, suppose that the true distribution $p_{\lambda }$ belong to one of two prescribed distributions that we do not know: $p_{\lambda }\in \mathcal{P}=\{p_{{\lambda }_1},p_{{\lambda }_2}\}$. Then we seek for the minimax redundancy:

$$R^{*}=\underset{p_{\lambda }}{min}\underset{i\in \{1,2\}}{max}{D}_{\mathrm{KL}}[p_{{\lambda }_i}:p_{\lambda }].$$

(34)

The distribution $p_{{\lambda }^{*}}$ achieving the minimax redundancy is the circumcenter of the right-centered KL ball enclosing the distributions $\mathcal{P}$.

Using the natural coordinates $\theta =({\theta }_1,\dots ,{\theta }_D)\in {\mathbb{R}}^D$ with ${\theta }_i=\log \frac{{\lambda }^i}{{\lambda }^d}$ of the log-normalizer of the categorical distributions (an exponential family of order $D=d-1$), we end up with calculating the smallest left-sided Bregman enclosing ball for the Bregman generator [58]: $F_{\mathrm{Categorical}}\left(\theta \right)=\log (1+{\sum }_{i=1}^D\exp {\theta }_i)$:

$$R^{*}=\underset{\theta \in \in {\mathbb{R}}^D}{min}\underset{i\in \{1,2\}}{max}{B}_{F_{\mathrm{categorical}}}(\theta :{\theta }_i).$$

This latter minimax problem is unconstrained since $\theta \in {\mathbb{R}}^D={\mathbb{R}}^{d-1}$.

5. Chernoff Information between Gaussian Distributions

5.1. Invariance of Chernoff Information under the Action of the Affine Group

The d-variate Gaussian density $p_{\lambda }\left(x\right)$ with parameter $\lambda =({\lambda }_v=\mu ,{\lambda }_M=\Sigma )$ where $\mu \in {\mathbb{R}}^d$ denotes the mean ($\mu =E_{p_{\lambda }}\left[x\right]$) and $\Sigma$ is a positive-definite covariance matrix ($\Sigma ={\mathrm{Cov}}_{p_{\lambda }}\left[X\right]$ for $X\sim p_{\lambda }$) is given by

$$p_{\lambda }(x;\lambda )=\frac{1}{{\left(2\pi \right)}^{\frac{d}{2}}\sqrt{|{\lambda }_M|}}\exp \left(-\frac{1}{2}{(x-{\lambda }_v)}^{\top }{\lambda }_M^{-1}(x-{\lambda }_v)\right),$$

where $|·|$ denotes the matrix determinant. The set of d-variate Gaussian distributions form a regular (and hence steep) exponential family with natural parameters $\theta \left(\lambda \right)=\left({\lambda }_M^{-1}{\lambda }_v,\frac{1}{2}{\lambda }_M^{-1}\right)$ and sufficient statistics $t\left(x\right)=(x,x{x}^{\top })$.

The Bhattacharrya distance between two multivariate Gaussians distributions $p_{{\mu }_1,{\Sigma }_1}$ and $p_{{\mu }_2,{\Sigma }_2}$ is

$$D_{B,\alpha }[p_{{\mu }_1,{\Sigma }_1},p_{{\mu }_2,{\Sigma }_2}]=\frac{1}{2}\left(\alpha {\mu }_1^{\top }{\Sigma }_1^{-1}{\mu }_1+(1-\alpha ){\mu }_2^{\top }{\Sigma }_2^{-1}{\mu }_2-{\mu }_{\alpha }^{\top }{\Sigma }_{\alpha }^{-1}{\mu }_{\alpha }+\log \frac{|{\Sigma }_1{|}^{\alpha }{\left|{\Sigma }_2\right|}^{1-\alpha }}{|{\Sigma }_{\alpha }|}\right),$$

where

$$\begin{array}{ccc}\hfill {\Sigma }_{\alpha }& =& {(\alpha {\Sigma }_1^{-1}+(1-\alpha ){\Sigma }_2^{-1})}^{-1},\hfill \\ \hfill {\mu }_{\alpha }& =& {\Sigma }_{\alpha }(\alpha {\Sigma }_1^{-1}{\mu }_1+(1-\alpha ){\Sigma }_2^{-1}).\hfill \end{array}$$

The Gaussian density can be rewritten as a multivariate location-scale family:

$$p_{\lambda }(x;\lambda )={\left|{\lambda }_M\right|}^{-\frac{1}{2}}{p}_{\mathrm{std}}\left({\lambda }_M^{\frac{1}{2}}(x-{\lambda }_v)\right),$$

where

$$p_{\mathrm{std}}\left(x\right)=\frac{1}{{\left(2\pi \right)}^{\frac{d}{2}}}\exp \left(-\frac{1}{2}{x}^{\top }x\right)=p_{(0,I)}$$

denotes the standard multivariate Gaussian distribution. The matrix ${\lambda }_M^{\frac{1}{2}}$ is the unique symmetric square-root matrix which is positive-definite when ${\lambda }_M$ is positive-definite.

Remark 8.

Notice that the product of two symmetric positive-definite matrices $P_1$ and $P_2$ may not be symmetric but $P_1^{\frac{1}{2}}{P}_2{P}_1^{\frac{1}{2}}$ is always symmetric positive-definite, and the eigenvalues of $P_1^{\frac{1}{2}}{P}_2{P}_1^{\frac{1}{2}}$ coincides with the eigenvalues of $P_1{P}_2$. Hence, we have ${\lambda }_{\mathrm{sp}}\left(P_1^{-\frac{1}{2}}{P}_2{P}_1^{-\frac{1}{2}}\right)={\lambda }_{\mathrm{sp}}\left(P_1^{-1}{P}_2\right)$ where ${\lambda }_{\mathrm{sp}}\left(M\right)$ denotes the eigenspectrum of matrix M.

We may interpret the Gaussian family as obtained by the action of the affine group $\mathrm{Aff}\left({\mathbb{R}}^d\right)={\mathbb{R}}^d⋊{\mathrm{GL}}_d\left(\mathbb{R}\right)$ on the standard density $p_{\mathrm{std}}$: Let the dot symbol “.” denotes the group action. The affine group is equipped with the following (outer) semidirect product:

$$(l_1,A_1).(l_2,A_2)=(l_1+A_1{l}_2,A_1{A}_2),$$

(35)

and this group can be handled as a matrix group with the following mapping of its elements to matrices:

$$(l,A)\equiv \left[\begin{array}{cc}A& l\\ 0& 1\end{array}\right].$$

Then we have

$$p_{(\mu ,\Sigma )}\left(x\right)=(\mu ,{\Sigma }^{-\frac{1}{2}}).p_{\mathrm{std}}\left(x\right)=(\mu ,{\Sigma }^{-\frac{1}{2}}).p_{0,I}\left(x\right)=p_{(\mu ,{\Sigma }^{-\frac{1}{2}}).(0,I)}\left(x\right).$$

We can show the following invariance of the skewed Bhattacharyya divergences:

Proposition 9

(Invariance of the Bhattacharyya divergence and f-divergences under the action of the affine group (Equation (35))). We have

$$\begin{array}{ccc}\hfill D_{B,\alpha }[(\mu ,{\Sigma }^{-\frac{1}{2}}).p_{{\mu }_1,{\Sigma }_1}:(\mu ,{\Sigma }^{-\frac{1}{2}}).p_{{\mu }_2,{\Sigma }_2}]& :=& D_{B,\alpha }[p_{(\mu ,{\Sigma }^{-\frac{1}{2}}).({\mu }_1,{\Sigma }_1)}:p_{(\mu ,{\Sigma }^{-\frac{1}{2}}).({\mu }_2,{\Sigma }_2})],\hfill \\ & =& D_{B,\alpha }\left[p_{{\Sigma }^{-\frac{1}{2}}({\mu }_1-\mu ):{\Sigma }^{-\frac{1}{2}}{\Sigma }_1{\Sigma }^{-\frac{1}{2}}},p_{{\Sigma }^{-\frac{1}{2}}({\mu }_2-\mu ),{\Sigma }^{-\frac{1}{2}}{\Sigma }_2{\Sigma }^{-\frac{1}{2}}}\right],\hfill \\ & =& D_{B,\alpha }[p_{{\mu }_1,{\Sigma }_1}:p_{{\mu }_2,{\Sigma }_2}].\hfill \end{array}$$

Proof. 

The proof follows from the $(f,g)$-form of Ali and Silvey’s divergences [59]. We can express $D_{B,\alpha }[p:q]=g\left(I_{h_{\alpha }}[p:q]\right)$ where $h_{\alpha }\left(u\right)=-u^{\alpha }$ (convex for $\alpha \in (0,1)$) and $g\left(v\right)=-\log -v$. Then we rely on the proof of invariance of f-divergences under the action of the affine group (see Proposition 3 of [60] relying on a change of variable in the integral):

$$I_f[p_{{\mu }_1,{\Sigma }_1}:p_{{\mu }_2,{\Sigma }_2}]=I_f\left[p_{0,I},p_{{\Sigma }_1^{-\frac{1}{2}}({\mu }_2-{\mu }_1):{\Sigma }_1^{-\frac{1}{2}}{\Sigma }_2{\Sigma }_1^{-\frac{1}{2}}}\right]=I_f\left[p_{{\Sigma }_2^{-\frac{1}{2}}({\mu }_1-{\mu }_2),{\Sigma }_2^{-\frac{1}{2}}{\Sigma }_1{\Sigma }_2^{-\frac{1}{2}}}:p_{0,I}\right],$$

where I denotes the identity matrix.    □

Thus, by choosing $(\mu ,\Sigma )=({\mu }_1,{\Sigma }_1)$ and $(\mu ,\Sigma )=({\mu }_2,{\Sigma }_2)$, we obtain the following corollary:

Corollary 2

(Bhattacharyya divergence from canonical Bhattacharyya divergences). We have

$$D_{B,\alpha }[p_{{\mu }_1,{\Sigma }_1}:p_{{\mu }_2,{\Sigma }_2}]=D_{B,\alpha }\left[p_{0,I},p_{{\Sigma }_1^{-\frac{1}{2}}({\mu }_2-{\mu }_1):{\Sigma }_1^{-\frac{1}{2}}{\Sigma }_2{\Sigma }_1^{-\frac{1}{2}}}\right]=D_{B,\alpha }\left[p_{{\Sigma }_2^{-\frac{1}{2}}({\mu }_1-{\mu }_2):{\Sigma }_2^{-\frac{1}{2}}{\Sigma }_1{\Sigma }_2^{-\frac{1}{2}}},p_{0,I}\right].$$

It follows that the Chernoff optimal skewing parameter enjoys the same invariance property:

$${\alpha }^{*}(p_{{\mu }_1,{\Sigma }_1}:p_{{\mu }_2,{\Sigma }_2})={\alpha }^{*}\left(p_{0,I},p_{{\Sigma }_1^{-\frac{1}{2}}({\mu }_2-{\mu }_1),{\Sigma }_1^{-\frac{1}{2}}{\Sigma }_2{\Sigma }_1^{-\frac{1}{2}}}\right)={\alpha }^{*}\left(p_{{\Sigma }_2^{-\frac{1}{2}}({\mu }_1-{\mu }_2):{\Sigma }_2^{-\frac{1}{2}}{\Sigma }_1{\Sigma }_2^{-\frac{1}{2}}},p_{0,I}\right).$$

As a byproduct, we get the invariance of the Chernoff information under the action of the affine group:

Corollary 3

(Invariance of the Chernoff information under the action of the affine group). We have:

$$D_C[p_{{\mu }_1,{\Sigma }_1},p_{{\mu }_2,{\Sigma }_2}]=D_C\left[p_{0,I},p_{{\Sigma }_1^{-\frac{1}{2}}({\mu }_2-{\mu }_1),{\Sigma }_1^{-\frac{1}{2}}{\Sigma }_2{\Sigma }_1^{-\frac{1}{2}}}\right]=D_C\left[p_{{\Sigma }_2^{-\frac{1}{2}}({\mu }_1-{\mu }_2),{\Sigma }_2^{-\frac{1}{2}}{\Sigma }_1{\Sigma }_2^{-\frac{1}{2}}},p_{0,I}\right].$$

Thus, the formula for the Chernoff information between two Gaussians

$$D_C({\mu }_1,{\Sigma }_1,{\mu }_2,{\Sigma }_2):=D_C[p_{{\mu }_1,{\Sigma }_1},p_{{\mu }_2,{\Sigma }_2}]=D_C({\mu }_{12},{\Sigma }_{12})$$

can be written as a function of two terms ${\mu }_{12}={\Sigma }_1^{-\frac{1}{2}}({\mu }_2-{\mu }_1)$ and ${\Sigma }_{12}={\Sigma }_1^{-\frac{1}{2}}{\Sigma }_2{\Sigma }_1^{-\frac{1}{2}}$.

5.2. Closed-Form Formula for the Chernoff Information between Univariate Gaussian Distributions

We shall report the exact solution for the Chernoff information between univariate Gaussian distributions by solving a quadratic equation. We can also report a complex closed-form formula by using symbolic computing because the calculations are lengthy and thus prone to human error.

Instantiating Equation (24) for the case of univariate Gaussian distributions paramterized by $(\mu ,{\sigma }^2)$, we get the following equation for the optimality condition of ${\alpha }^{*}$:

$$\begin{array}{ccc}\hfill \langle {\theta }_2-{\theta }_1,{\eta }_{{\alpha }^{*}}\rangle & =& F\left({\theta }_2\right)-F\left({\theta }_1\right),\hfill \end{array}$$

(36)

$$\begin{array}{ccc}\hfill 〈\left(\frac{{\mu }_2}{{\sigma }_2^2}-\frac{{\mu }_1}{{\sigma }_1^2},\frac{1}{2{\sigma }_1^2}-\frac{1}{2{\sigma }_2^2}\right),(m_{\alpha },v_{\alpha })〉& =& \frac{1}{2}\log \frac{{\sigma }_2^2}{{\sigma }_1^2}+\frac{{\mu }_2^2}{2{\sigma }_2^2}-\frac{{\mu }_1^2}{2{\sigma }_1^2},\hfill \end{array}$$

(37)

where $\langle ·,·\rangle$ denotes the scalar product and with the interpolated mean and variance along an exponential arc ${\left\{(m_{\alpha },v_{\alpha })\right\}}_{\alpha \in (0,1)}$ passing through $({\mu }_1,{\sigma }_1^2)$ when $\alpha =1$ and $({\mu }_2,{\sigma }_2^2)$ when $\alpha =0$ given by

$$\begin{array}{ccc}\hfill m_{\alpha }& =& \frac{\alpha {\mu }_1{\sigma }_2^2+(1-\alpha ){\mu }_2{\sigma }_1^2}{(1-\alpha ){\sigma }_1^2+\alpha {\sigma }_2^2}=\frac{\alpha ({\mu }_1{\sigma }_2^2-{\mu }_2{\sigma }_1^2)+{\mu }_2{\sigma }_1^2}{{\sigma }_1^2+\alpha ({\sigma }_2^2-{\sigma }_1^2)},\hfill \end{array}$$

(38)

$$\begin{array}{ccc}\hfill v_{\alpha }& =& \frac{{\sigma }_1^2{\sigma }_2^2}{(1-\alpha ){\sigma }_1^2+\alpha {\sigma }_2^2}=\frac{{\sigma }_1^2{\sigma }_2^2}{{\sigma }_1^2+\alpha ({\sigma }_2^2-{\sigma }_1^2)}.\hfill \end{array}$$

(39)

That is, for $p=p_{{\mu }_1,{\sigma }_1^2}$ and $q=p_{{\mu }_2,{\sigma }_2^2}$, we have the weighted geometric mixture ${\left(pq\right)}_{\alpha }^G=p_{m_{\alpha },v_{\alpha }}$.

Thus, the optimality condition of the Chernoff optimal skewing parameter is given by:

$${\mathrm{OC}}_{\mathrm{Gaussian}}:\left(\frac{{\mu }_2}{{\sigma }_2^2}-\frac{{\mu }_1}{{\sigma }_1^2}\right)m_{\alpha }-\left(\frac{1}{2{\sigma }_2^2}-\frac{1}{2{\sigma }_1^2}\right)v_{\alpha }=\frac{1}{2}\log \frac{{\sigma }_2^2}{{\sigma }_1^2}+\frac{{\mu }_2^2}{2{\sigma }_2^2}-\frac{{\mu }_1^2}{2{\sigma }_1^2}.$$

(40)

Let us rewrite compactly Equation (40) as

$${\mathrm{OC}}_{\mathrm{Gaussian}}:a_{12}{m}_{\alpha }+b_{12}{v}_{\alpha }+c_{12}=0,$$

(41)

with the following coefficients:

$$\begin{array}{ccc}\hfill a_{12}& =& \frac{{\mu }_2}{{\sigma }_2^2}-\frac{{\mu }_1}{{\sigma }_1^2},\hfill \end{array}$$

(42)

$$\begin{array}{ccc}\hfill b_{12}& =& \frac{1}{2{\sigma }_1^2}-\frac{1}{2{\sigma }_2^2},\hfill \end{array}$$

(43)

$$\begin{array}{ccc}\hfill c_{12}& =& \frac{1}{2}\log \frac{{\sigma }_1^2}{{\sigma }_2^2}+\frac{{\mu }_1^2}{2{\sigma }_1^2}-\frac{{\mu }_2^2}{2{\sigma }_2^2}.\hfill \end{array}$$

(44)

By multiplying both sides of Equation (41) by ${\sigma }_1^2+\alpha {\Delta }_v$ where ${\Delta }_v:={\sigma }_2^2-{\sigma }_1^2$ and rearranging terms, we get a quadratic equation with positive root being ${\alpha }^{*}$.

Using the computer algebra system (CAS) Maxima, we can also solve exactly this quadratic equation in $\alpha$ as a function of ${\mu }_1$, ${\sigma }_1^2$, ${\mu }_2$, and ${\sigma }_2^2$: See listing in Appendix C.

Once we get the optimal value of ${\alpha }^{*}={\alpha }^{*}({\mu }_1,{\sigma }_1^2,{\mu }_2,{\sigma }_2^2)$, we get the Chernoff information as

$$D_C[p_{{\mu }_1,{\sigma }_1^2},p_{{\mu }_2,{\sigma }_2^2}]=D_{\mathrm{KL}}[p_{m_{{\alpha }^{*}},v_{{\alpha }^{*}}}:p_{{\mu }_1,{\sigma }_1^2}]$$

with the Kullback–Leibler divergence between two univariate Gaussians distributions $p_{{\mu }_1,{\sigma }_1^2}$ and $p_{{\mu }_2,{\sigma }_2^2}$ given by

$$D_{\mathrm{KL}}[p_{{\mu }_1,{\sigma }_1^2}:p_{{\mu }_2,{\sigma }_2^2}]=\frac{1}{2}\left(\frac{{({\mu }_2-{\mu }_1)}^2}{{\sigma }_2^2}+\frac{{\sigma }_1^2}{{\sigma }_2^2}-\log \frac{{\sigma }_1^2}{{\sigma }_2^2}-1\right).$$

Notice that from the invariance of Proposition 9, we have for any $(\mu ,{\sigma }^2)\in \mathbb{R}\times {\mathbb{R}}_{++}$:

$$D_{\mathrm{KL}}[p_{{\mu }_1,{\sigma }_1^2}:p_{{\mu }_2,{\sigma }_2^2}]=D_{\mathrm{KL}}\left[p_{\frac{{\mu }_1-\mu }{\sigma },\frac{{\sigma }_1^2}{{\sigma }^2}}:p_{\frac{{\mu }_2-\mu }{\sigma },\frac{{\sigma }_2^2}{{\sigma }^2}}\right],$$

and therefore by choosing $(\mu ,{\sigma }^2)=({\mu }_1,{\sigma }_1^2)$, we have

$$D_{\mathrm{KL}}[p_{{\mu }_1,{\sigma }_1^2}:p_{{\mu }_2,{\sigma }_2^2}]=D_{\mathrm{KL}}\left[p_{0,1},p_{\frac{{\mu }_2-{\mu }_1}{{\sigma }_1}:\frac{{\sigma }_2^2}{{\sigma }_1^2}}\right].$$

Proposition 10.

The Chernoff information between two univariate Gaussian distributions can be calculated exactly in closed form.

One can also program these closed-form solutions in Python using the SymPy package (https://www.sympy.org/en/index.html (accessed on 30 July 2022)) for performing symbolic computations.

Let us report special cases with some illustrating examples.

• First, let us consider the Gaussian subfamily with prescribed variance. When ${\sigma }_1^2={\sigma }_2^2={\sigma }^2$, we always have ${\alpha }^{*}=\frac{1}{2}$, and the Chernoff information is

  $$D_C[p_{{\mu }_1,{\sigma }^2}:p_{{\mu }_2,{\sigma }^2}]=\frac{{({\mu }_2-{\mu }_1)}^2}{8{\sigma }^2}.$$

  (45)
  Notice that it amounts to one eight of the squared Mahalanobis distance (see [60] for a detailed explanation).
• Second, let us consier Gaussian subfamily with prescribed mean. When ${\mu }_1={\mu }_2=\mu$, we get the optimal skewing value independent of the mean $\mu$:

  $${\alpha }^{*}=-\frac{v_1\log \left(2v_2\right)-v_2-v_1\log \left(2v_1\right)+v_1}{\left(v_2-v_1\right)\log \left(2v_2\right)-\log \left(2v_1\right)v_2+v_1\log \left(2v_1\right)}$$

  where $v_1={\sigma }_1^2$ and $v_2={\sigma }_2^2$. The Chernoff information is

  $$D_C[p_{{\mu }_1,v_1}:p_{{\mu }_2,v_2}]-\frac{\left(v_2-v_1\right)\log \left(\frac{v_2\log \left(2v_2\right)-\log \left(2v_1\right)v_2}{v_2-v_1}\right)-v_2\log \left(2v_2\right)+\left(\log \left(2v_1\right)+1\right)v_2-v_1}{2v_2-2v_1}.$$

  (46)
• Third, consider the Chernoff information between the standard normal distribution and another normal distribution. When $({\mu }_1,{\sigma }_1^2)=(0,1)$ and $({\mu }_2,{\sigma }_2^2)=(\mu ,v)$, we get    ${\alpha }^{*}=\frac{\sqrt{\left(4{\mu }^2{v}^2-4{\mu }^{2}v\right)\log \left(2v\right)+v^4-4v^3+\left(6-4\log 2{\mu }^2\right)v^2+\left(4{\mu }^4+4\log 2{\mu }^2-4\right)v+1}+\left(2-2v\right)\log \left(2v\right)+v^2+\left(2\log 2-2\right)v-2{\mu }^2-2\log 2+1}{\left(2v^2-4v+2\right)\log \left(2v\right)-2\log 2v^2+\left(2{\mu }^2+4\log 2\right)v-2{\mu }^2-2\log 2}$

Example 4.

Let us consider $N({\mu }_1=0,{\sigma }_1^2=1)$ and $N({\mu }_2=1,{\sigma }_2^2=2)$. The Chernoff exponent is

$${\alpha }^{*}=\frac{\sqrt{8\log 4-8\log 2+9}-2\log 4+2\log 2-1}{2\log 4-2\log 2+2}\approx 0.4215580558605244,$$

and the Chernoff information is (zoom in for the formula):

  $-\frac{\sqrt{8\log 4-8\log 2+9}\left(\left(2\log 4-2\log 2+3\right)\log \left(\frac{4\log 4-4\log 2+4}{\sqrt{8\log 4-8\log 2+9}+1}\right)-4{\left(\log 4\right)}^2+\left(8\log 2-6\right)\log 4-4{\left(\log 2\right)}^2+6\log 2-2\right)+\left(6\log 4-6\log 2+7\right)\log \left(\frac{4\log 4-4\log 2+4}{\sqrt{8\log 4-8\log 2+9}+1}\right)+4{\left(\log 4\right)}^2+\left(10-8\log 2\right)\log 4+4{\left(\log 2\right)}^2-10\log 2+6}{\left(4\log 4-4\log 2+6\right)\sqrt{8\log 4-8\log 2+9}+12\log 4-12\log 2+14}$

$$\approx 0.1155433222682347$$

Using the bisection search of [24] with $ϵ={10}^{-8}$ takes 28 iterations, and we get

$${\alpha }^{*}\approx 0.42155805602669716,$$

and the Chernoff information is approximately $0.11554332226823472$. Now, if we swap $p_{{\mu }_1,{\sigma }_1^2}\leftrightarrow p_{{\mu }_2,{\sigma }_2^2}$, we find ${\alpha }^{*}=0.5784419439733028$ (and $0.5784419439733028+0.42155805602669716\approx 1$).

Notice that in general, we may evaluate how good is the approximation $\tilde{\alpha }$ of ${\alpha }^{*}$ by evaluating the deficiency of the optimal condition:

$$\left|{({\theta }_2-{\theta }_1)}^{\top }{\eta }_{\tilde{\alpha }}-F\left({\theta }_2\right)+F\left({\theta }_1\right)\right|.$$

Example 5.

Let us consider ${\mu }_1=1$, ${\sigma }_1^2=3$ and ${\mu }_2=5$ and ${\sigma }_2^2=5$. We get

$${\alpha }^{*}=\frac{\sqrt{120\log 10-120\log 6+961}-3\log 10+3\log 6-23}{2\log 10-2\log 6+16}\approx 0.4371453168322306$$

and the Chernoff information is reported in closed form and evaluated numerically as

$$0.5242883659200144.$$

In comparison, the bisection algorithm of [24] with $ϵ={10}^{-8}$ takes 28 iterations, and reports ${\alpha }^{*}\approx 0.43714531883597374$ and the Chernoff information about

$$0.5242883659200137.$$

Corollary 4.

The smallest enclosing left-sided Kullback–Leibler disk of n univariate Gaussian distributions can be calculated exactly in randomized linear time [56].

5.3. Fast Approximation of the Chernoff Information of Multivariate Gaussian Distributions

In general, the Chernoff information between d-variate Gaussians distributions is not known in closed-form formula when $d>1$, see for example [61,62,63]. We shall consider below some special cases:

• When the Gaussians have the same covariance matrix $\Sigma$, the Chernoff information optimal skewing parameter is $\alpha =\frac{1}{2}$ and the Chernoff information is

  $$D_C[p_{{\mu }_1,\Sigma },p_{{\mu }_2,\Sigma }]=\frac{1}{8}{\Delta }_{\Sigma }^2({\mu }_1,{\mu }_2),$$

  where ${\Delta }_{\Sigma }^2({\mu }_1,{\mu }_2)={({\mu }_2-{\mu }_1)}^{\top }{\Sigma }^{-1}({\mu }_2-{\mu }_1)$ is the squared Mahalanobis distance. The Mahalanobis distance enjoys the following property by congruence transformation:

  $${\Delta }_{\Sigma }({\mu }_1,{\mu }_2)={\Delta }_{A\Sigma A^{\top }}(A{\mu }_1,A{\mu }_2),\forall A\in \mathrm{GL}\left(d\right).$$

  (47)
  Notice that we can rewrite the (squared) Mahalanobis distance as

  $${\Delta }_{\Sigma }^2({\mu }_1,{\mu }_2)=\mathrm{tr}\left({\Sigma }^{-1}({\mu }_2-{\mu }_1){({\mu }_2-{\mu }_1)}^{\top }\right)$$

  using the matrix trace cyclic property. Then we check that

  $$\begin{array}{ccc}\hfill {\Delta }_{A\Sigma A^{\top }}^2(A{\mu }_1,A{\mu }_2)& =& \mathrm{tr}\left(A^{-\top }{\Sigma }^{-1}{A}^{-1}A({\mu }_2-{\mu }_1){({\mu }_2-{\mu }_1)}^{\top }{A}^{\top }\right),\hfill \\ & =& \mathrm{tr}\left({\Sigma }^{-1}({\mu }_2-{\mu }_1){({\mu }_2-{\mu }_1)}^{\top }\right)={\Delta }_{\Sigma }^2({\mu }_1,{\mu }_2).\hfill \end{array}$$
• The Chernoff information for the special case of centered multivariate Gaussians distributions was studied in [62]. The KLD between two centered Gaussians $p_{\mu ,{\Sigma }_1}$ and $p_{{\mu }_{,}{\Sigma }_2}$ is half of the matrix Burg distance:

  $$D_{\mathrm{KL}}[p_{\mu ,{\Sigma }_1}:p_{\mu ,{\Sigma }_2}]=\frac{1}{2}\left(\log \frac{\det \left({\Sigma }_2\right)}{\det \left({\Sigma }_1\right)}+\mathrm{tr}\left({\Sigma }_2^{-1}{\Sigma }_1\right)-d\right)=:\frac{1}{2}{D}_{\mathrm{Burg}}[{\Sigma }_1:{\Sigma }_2].$$

  (48)
  When $d=1$, the Burg distance corresponds to the well-known Itakura–Saito divergence. The matrix Burg distance is a matrix spectral distance [62]:

  $$D_{\mathrm{Burg}}[{\Sigma }_1:{\Sigma }_2]=\left(\sum _{i=1}^d{\lambda }_i-\log {\lambda }_i-1\right),$$

  where the ${\lambda }_i$’s are the eigenvalues of ${\Sigma }_2{\Sigma }_1^{-1}$. The reverse KLD divergence $D_{\mathrm{KL}}[p_{\mu ,{\Sigma }_2}:p_{\mu ,{\Sigma }_1}]=\frac{1}{2}{D}_{\mathrm{Burg}}[{\Sigma }_2:{\Sigma }_1]$ is obtained by replacing ${\lambda }_i\leftrightarrow \frac{1}{{\lambda }_i}$:

  $$D_{\mathrm{KL}}[p_{\mu ,{\Sigma }_2}:p_{{\mu }_{,}{\Sigma }_1}]=\frac{1}{2}\left(\sum _{i=1}^d\frac{1}{{\lambda }_i}+\log {\lambda }_i-1\right).$$
  More generally, the f-divergences between centered Gaussian distributions are always matrix spectral divergences [60].

Otherwise, for the general multivariate case, we implement the dichotomic search of Algorithm 1 in Algorithm 3 with the KLD between two multivariate Gaussian distributions expressed as

$$\begin{array}{ccc}\hfill D_{\mathrm{KL}}[p_{{\mu }_1,{\Sigma }_1}:p_{{\mu }_2,{\Sigma }_2}]& =& \frac{1}{2}{\Delta }_{\Sigma }^2({\mu }_1,{\mu }_2)+\frac{1}{2}{D}_{\mathrm{Burg}}[{\Sigma }_1:{\Sigma }_2],\hfill \end{array}$$

(49)

$$\begin{array}{ccc}& =& \frac{1}{2}\left(\mathrm{tr}\left({\Sigma }_2^{-1}{\Sigma }_1\right)-\log \frac{\det \left({\Sigma }_2\right)}{\det \left({\Sigma }_1\right)}-d+{({\mu }_2-{\mu }_1)}^{\top }{\Sigma }_2^{-1}({\mu }_2-{\mu }_1)\right).\hfill \end{array}$$

(50)

Algorithm 3 Dichotomic search for approximating the Chernoff information between two multivariate normal distributions $p_{{\mu }_1,{\Sigma }_1}$ and $p_{{\mu }_2,{\Sigma }_2}$ by approximating the optimal skewing parameter value $\alpha \approx {\alpha }^{*}$.

Example 6.

Let $d=2$, $p_{{\mu }_1,{\Sigma }_1}=p_{0,I}$ be the standard bivariate Gaussian distribution and $p_{{\mu }_2,{\Sigma }_2}$ be the bivariate Gaussian distribution with mean ${\mu }_2={\left[12\right]}^{\top }$ and covariance matrix ${\Sigma }_2=\left[\begin{array}{cc}1& -1\\ -1& 2\end{array}\right]$. Setting the numerical precision threshold ϵ to $ϵ={10}^{-8}$, the dichotomic search performs 28 split iterations, and approximate ${\alpha }^{*}$ by

$${\alpha }^{*}\approx 0.5825489424169064.$$

The Chernoff information $D_C[p_{0,I},p_{{\mu }_2,{\Sigma }_2}]$ is approximated by $0.8827640697808525$.

The m-interpolation of multivariate Gaussian distributions $p_{{\mu }_1,{\Sigma }_1}$ and $p_{{\mu }_2,{\Sigma }_2}$ with respect to the mixture connection ${\nabla }^m$ is given by

$${\gamma }_{p_{{\mu }_1,{\Sigma }_1},p_{{\mu }_2,{\Sigma }_2}}^m\left(\alpha \right)=p_{{\mu }_{\alpha }^m,{\Sigma }_{\alpha }^m},$$

where

$$\begin{array}{ccc}\hfill {\mu }_{\alpha }^m& =& (1-\alpha ){\mu }_1+\alpha {\mu }_2=:{\overline{\mu }}_{\alpha },\hfill \\ \hfill {\Sigma }_{\alpha }^m& =& (1-\alpha ){\Sigma }_1+\alpha {\Sigma }_2+(1-\alpha ){\mu }_1{\mu }_1^{\top }+\alpha {\mu }_2{\mu }_2^{\top }-{\overline{\mu }}_{\alpha }{\overline{\mu }}_{\alpha }^{\top }.\hfill \end{array}$$

To e-interpolation of multivariate Gaussian distributions $p_{{\mu }_1,{\Sigma }_1}$ and $p_{{\mu }_2,{\Sigma }_2}$ with respect to the exponential connection ${\nabla }^e$ is given by

$${\gamma }_{p_{{\mu }_1,{\Sigma }_1},p_{{\mu }_2,{\Sigma }_2}}^e\left(\alpha \right)=p_{{\mu }_{\alpha }^e,{\Sigma }_{\alpha }^e},$$

where

$$\begin{array}{ccc}\hfill {\mu }_{\alpha }^e& =& {\Sigma }_{\alpha }^e\left((1-\alpha ){\Sigma }_1^{-1}{\mu }_1+\alpha {\Sigma }_2^{-1}{\mu }_2\right),\hfill \\ \hfill {\Sigma }_{\alpha }^e& =& {\left((1-\alpha ){\Sigma }_1^{-1}+\alpha {\Sigma }_2^{-1}\right)}^{-1}.\hfill \end{array}$$

In information geometry, both these e- and m-connections defined with respect to an exponential family are shown to be flat. These geodesics correspond to linear interpolations in the ${\nabla }^e$-affine coordinate system $\theta$ and in the dual ${\nabla }^m$ coordinate system $\eta$, respectively.

Figure 7 displays these two e-geodesic and m-geodesic between two multivariate normal distributions. Notice that the Riemannian geodesic with the Levi–Civita metric connection $\frac{{\nabla }^e+{\nabla }^m}{2}$ is not known in closed form for boundary value conditions. The expression of the Riemannian geodesic is known only for initial value conditions [64] (i.e., starting point with a given vector direction).

5.4. Chernoff Information between Centered Multivariate Normal Distributions

The set

$${\mathcal{N}}_0=\left\{p_{\Sigma }\left(x\right)=\frac{1}{\sqrt{\det \left(2\pi \Sigma \right)}}\exp \left(-\frac{1}{2}{x}^{\top }{\Sigma }^{-1}x\right):\Sigma \succ 0\right\}$$

of centered multivariate normal distributions is a regular exponential family with natural parameter $\theta ={\Sigma }^{-1}$, sufficient statistic $t\left(x\right)=-\frac{1}{2}x{x}^{\top }$, log-normalizer $F\left(\theta \right)=-\frac{1}{2}\log \det \left(\theta \right)$ and auxiliary carrier term $k\left(x\right)=-\frac{d}{2}\log \left(2\pi \right)$. Family ${\mathcal{N}}_0$ is also a multivariate scale family with scale matrices ${\Sigma }^{\frac{1}{2}}$ (standard deviation $\sigma$ in 1D).

Let $\langle A,B\rangle =\mathrm{tr}\left(A^{\top }B\right)$ defines the inner product between two symmetric matrices A and B. Then we can write the centered Gaussian distribution $p_{\Sigma }\left(x\right)$ in the canonical form of exponential families:

$$p_{\theta }\left(x\right)=\exp \left(\langle \theta ,t\left(x\right)\rangle -F\left(\theta \right)+k\left(x\right)\right).$$

The function log det of a positive-definite matrix is strictly concave [65], and hence we check that $F\left(\theta \right)$ is strictly convex. Furthermore, we have ${\nabla }_X\log \det \left(X\right)=X^{-\top }$ so that ${\nabla }_{\theta }F\left(\theta \right)=-\frac{1}{2}{\theta }^{-\top }$.

The optimality condition equation of Chernoff best skewing parameter ${\alpha }^{*}$ becomes:

$$\begin{array}{ccc}\hfill \langle {\theta }_2-{\theta }_1,\nabla F({\theta }_1+{\alpha }^{*}({\theta }_2-{\theta }_1))\rangle & =& F\left({\theta }_2\right)-F\left({\theta }_1\right),\hfill \end{array}$$

(51)

$$\begin{array}{ccc}\hfill -\frac{1}{2}\mathrm{tr}\left({({\theta }_2-{\theta }_1)}^{\top }{({\theta }_1+{\alpha }^{*}({\theta }_2-{\theta }_1))}^{-1}\right)& =& -\frac{1}{2}\log \frac{\det \left({\theta }_2\right)}{\det \left({\theta }_1\right)},\hfill \end{array}$$

(52)

$$\begin{array}{ccc}\hfill \mathrm{tr}\left({({\theta }_2-{\theta }_1)}^{\top }{({\theta }_1+{\alpha }^{*}({\theta }_2-{\theta }_1))}^{-1}\right)& =& \log \frac{\det \left({\theta }_2\right)}{\det \left({\theta }_1\right)},\hfill \end{array}$$

(53)

$$\begin{array}{ccc}\hfill \mathrm{tr}\left(({\Sigma }_2^{-1}-{\Sigma }_1^{-1}){({\Sigma }_1^{-1}+{\alpha }^{*}({\Sigma }_2^{-1}-{\Sigma }_1^{-1}))}^{-1}\right)& =& \log \frac{\det \left({\Sigma }_1\right)}{\det \left({\Sigma }_2\right)}=\log \det \left({\Sigma }_1{\Sigma }_2^{-1}\right).\hfill \end{array}$$

(54)

When ${\Sigma }_2=s{\Sigma }_1$ (and ${\Sigma }_2^{-1}=\frac{1}{s}{\Sigma }_1^{-1}$) for $s>0$ and $s\ne 1$, we get a closed-form for ${\alpha }^{*}$ using the fact that $\det \left(\frac{I}{s}\right)=\frac{1}{s^d}$ and $\mathrm{tr}\left(I\right)=d$ for d-dimensional identity matrix I. Solving Equation (54) yields

$${\alpha }^{*}\left(s\right)=\frac{s-1-\log s}{(s-1)\log s}\in (0,1).$$

(55)

Therefore the Chernoff information between two scaled centered Gaussian distributions $p_{\mu ,\Sigma }$ and $p_{\mu ,s\Sigma }$ is available in closed form.

Proposition 11.

The Chernoff information between two scaled d-dimensional centered Gaussian distributions $p_{\mu ,\Sigma }$ and $p_{\mu ,s\Sigma }$ of ${\mathcal{N}}_{\mu }$ (for $s>0$) is available in closed form:

$$D_C[p_{\mu ,\Sigma },p_{\mu ,s\Sigma }]=D_{B,{\alpha }^{*}}[p_{\mu ,\Sigma },p_{\mu ,s\Sigma }]=d\frac{(s-1)\log \left(\frac{s}{s-1}\log s\right)-s\log s+s-1}{2(1-s)},$$

(56)

where ${\alpha }^{*}=\frac{s-1-\log s}{(s-1)\log s}\in (0,1)$.

Notice that ${\alpha }^{*}(p_{\mu ,\Sigma }:p_{\mu ,s\Sigma })={\alpha }^{*}(p_{\mu ,\Sigma },p_{\mu ,\frac{1}{s}\Sigma })$ and $D_C[p_{\mu ,\Sigma },p_{\mu ,s\Sigma }]=D_C[p_{\mu ,\Sigma },p_{\mu ,\frac{1}{s}\Sigma }]$.

Example 7.

Consider ${\mu }_1={\mu }_2=0$ and ${\Sigma }_1=I$, ${\Sigma }_2=\frac{1}{2}I$. We find that ${\alpha }^{*}=\frac{2\log 2-1}{\log 2}$, which is independent of the dimension of the matrices. The Chernoff information depends on the dimension:

$$D_C[p_{0,I},p_{0,\frac{1}{2}I}]=d\frac{\log 2-\log \log 2-1}{2}.$$

Notice that when $d=1$, we have $s=\frac{{\sigma }_2^2}{{\sigma }_1^2}$, and we recover a special case of the closed-form formula for the Chernoff information between univariate Gaussians.

In [62], the following equation is reported for finding ${\alpha }^{*}$ based on Equation (54):

$${\mathrm{OC}}_{\mathrm{Centered}\mathrm{Gaussians}}:\sum _{i=1}^d\frac{1-{\lambda }_i}{{\alpha }^{*}+(1-{\alpha }^{*}){\lambda }_i}+\log {\lambda }_i=0$$

(57)

where the ${\lambda }_i$’s are generalized eigenvalues of ${\Sigma }_1{\Sigma }_2^{-1}$ (this excludes the case of all ${\lambda }_i$’s equal to one). The value of ${\alpha }^{*}$ satisfying Equation (57) is unique. Let us notice that the product of two symmetric positive-definite matrices is not necessarily symmetric anymore. We can derive Equation (57) by expressing Equation (54) using the identity matrix I and matrix ${\Sigma }_2^{-\frac{1}{2}}{\Sigma }_1{\Sigma }_2^{-\frac{1}{2}}$.

Remark 9.

We can get closed-form solutions for ${\alpha }^{*}$ and the corresponding Chernoff information in some particular cases. For example, when the dimension $d=2$, we need to solve a quadratic equation to get ${\alpha }^{*}$. Thus, for $d\le 4$, we get a closed-form solution for ${\alpha }^{*}$ by solving a polynomial equation characterizing the optimal condition, and obtain the Chernoff information in closed-form as a byproduct.

Example 8.

Consider the Chernoff information between $p_{0,I}$ and $p_{0,\Lambda }$ with $\Lambda =\mathrm{diag}(1,2,3,4)$. We get the exact Chernoff exponent value ${\alpha }^{*}$ by taking the root of a quartic polynomial equation falling in $(0,1)$. By evaluating numerically this root, we find that ${\alpha }^{*}\simeq 0.59694$ and that the Chernoff information is $D_C[p_{0,I},p_{0,\Lambda }]\simeq 0.22076$. See Appendix C for some symbolic computation code.

6. Chernoff Information between Densities of Different Exponential Families

Let

$${\mathcal{E}}_1=\{p_{\theta }=\exp (\langle \theta ,t_1\left(x\right)\rangle -F_1\left(\theta \right)):\theta \in \Theta \},$$

and

$${\mathcal{E}}_2=\{q_{{\theta }^{\prime }}=\exp (\langle {\theta }^{\prime },t_2\left(x\right)\rangle -F_2\left({\theta }^{\prime }\right):{\theta }^{\prime }\in {\Theta }^{\prime }\},$$

be two distinct exponential families, and consider the Chernoff information between the densities $p_{{\theta }_1}$ and $q_{{\theta }_2^{\prime }}$. The exponential arc induced by $p_{{\theta }_1}$ and $q_{{\theta }_2^{\prime }}$ is

$$\{{\left(p_{{\theta }_1}{q}_{{\theta }_2^{\prime }}\right)}_{\alpha }^G\propto p_{{\theta }_1}^{\alpha }{q}_{{\theta }_2^{\prime }}^{1-\alpha }:\alpha \in (0,1)\}.$$

Let ${\mathcal{E}}_{12}$ denote the exponential family with sufficient statistics $(t_1\left(x\right),t_2\left(x\right))$, log-normalizer $F_{12}(\theta ,{\theta }^{\prime })$, and denote by ${\Theta }_{12}$ its natural parameter space. Family ${\mathcal{E}}_{12}$ can be interpreted as a product exponential family which yields an exponential family. We have

$${\left(p_{{\theta }_1}{q}_{{\theta }_2^{\prime }}\right)}_{\alpha }^G=\exp \left(\langle (t_1\left(x\right),t_2\left(x\right)),(\alpha {\theta }_1,(1-\alpha ){\theta }_2^{\prime })\rangle -F_{12}(\alpha {\theta }_1,(1-\alpha ){\theta }_2^{\prime })\right).$$

Thus the induced LREF ${\mathcal{E}}_{p_{{\theta }_1}{q}_{{\theta }_2^{\prime }}}$ with natural parameter space ${\Theta }_{p_{{\theta }_1}{q}_{{\theta }_2^{\prime }}}$ can be interpreted as a 1D curved exponential family of the product exponential family ${\mathcal{E}}_{12}$.

The optimal skewing parameter ${\alpha }^{*}$ is found by setting the derivative of $F_{12}(\alpha {\theta }_1,(1-\alpha ){\theta }_2^{\prime })$ with respect $\alpha$ to zero:

$$\frac{d}{\mathrm{d}\alpha }{F}_{12}(\alpha {\theta }_1,(1-\alpha ){\theta }_2^{\prime })=0.$$

Example 9.

Let ${\mathcal{E}}_1$ can be chosen as the exponential family of exponential distributions

$${\mathcal{E}}_1=\left\{e_{\lambda }\left(x\right)=\lambda \exp (-\lambda x),\lambda \in (0,+\infty )\right\}$$

defined on the support ${\mathcal{X}}_1=(0,\infty )$ and ${\mathcal{E}}_2$ can be chosen as the exponential family of half-normal distributions

$${\mathcal{E}}_2=\left\{h_{\sigma }\left(x\right)=\sqrt{\frac{2}{\pi {\sigma }^2}}\exp (-\frac{x^2}{2{\sigma }^2}):{\sigma }^2>0\right\}$$

with support ${\mathcal{X}}_2=(0,\infty )$.

The product exponential family corresponds to the singly truncated normal family [50] which is a non-regular (i.e., parameter space is not topologically an open set):

$${\Theta }_{12}=(\mathbb{R}\times {\mathbb{R}}_{++})\cup {\Theta }_0,$$

with ${\Theta }_0=\{(\theta ,0):\theta <0\}$ (the part corresponding to the exponential family of exponential distributions). This exponential family ${\mathcal{E}}_{12}=\left\{p_{{\theta }_1,{\theta }_2}\right\}$ of singly truncated normal distributions is also non-steep [50]. The log-normalizer is

$$F_{12}({\theta }_1,{\theta }_2)=\frac{1}{2}\log \frac{\pi }{{\theta }_2}+\log \Phi \left(\frac{{\theta }_1}{\sqrt{2{\theta }_2}}\right)+\frac{{\theta }_1^2}{4{\theta }_2},$$

where ${\theta }_1=\frac{\mu }{{\sigma }^2}$ and ${\theta }_2=\frac{1}{2{\sigma }^2}$, and Φ denotes the cumulative distribution function of the standard normal. Function $F_{12}$ is proven of class $C^1$ on ${\Theta }_{12}$ (see Proposition 3.1 of [50]) with $F_{12}(\theta ,0)=-\log (-\theta )$ for $\theta <0$.

Notice that the KLD between an exponential distribution and a half-normal distribution is $+\infty$ since the definite integral diverges (hence $D_{\mathrm{KL}}[e_{\lambda }:h_{\sigma }]$ is not equivalent to a Bregman divergence, and ${\Theta }_{e_{{\theta }_1}{h}_{{\theta }_2^{\prime }}}$ is not open at 1) but the reverse KLD between a half-normal distribution and an exponential distribution is available in closed-form (using symbolic computing):

$$D_{\mathrm{KL}}[h_{\sigma }:e_{\lambda }]=\sqrt{\sqrt{8}\sigma \lambda -\sqrt{\pi }(1+\log \frac{\pi {\lambda }^2{\sigma }^2}{2})}2\sqrt{\pi }.$$

Figure 8 illustrate the domain of the singly truncated normal distributions and displays an exponential arc between an exponential distribution and a half-normal distribution. Notice that we could have also considered a similar but different example by taking the exponential family of Rayleigh distributions which exhibit an additional extra carrier term $k\left(x\right)$.

The Bhattacharyya α-skewed coefficient calculated using symbolic computing (see Appendix C) is

$${\rho }_{\alpha }[h_{\sigma }:e_{\lambda }]={\rho }_{1-\alpha }[e_{\lambda }:h_{\sigma }]=\frac{{\pi }^{\frac{1}{2}-\frac{\alpha }{2}}{e}^{-{\sigma }^2{\lambda }^2}\left(2^{\frac{\alpha }{2}+\frac{1}{2}}\sigma \lambda e^{\frac{\alpha {\sigma }^2{\lambda }^2}{2}+\frac{{\sigma }^2{\lambda }^2}{2\alpha }}\mathrm{erf}\left(\frac{\left(\sqrt{2}\alpha -\sqrt{2}\right)\sigma \lambda }{2\sqrt{\alpha }}\right)+2^{\frac{\alpha }{2}+\frac{1}{2}}\sigma \lambda e^{\frac{\alpha {\sigma }^2{\lambda }^2}{2}+\frac{{\sigma }^2{\lambda }^2}{2\alpha }}\right)}{2\sqrt{\alpha }{\sigma }^{\alpha }{\lambda }^{\alpha }},$$

where $\mathrm{erf}$ denotes the error function.

7. Conclusions

In this work, we revisited the Chernoff information [2] (1952) which was originally introduced to upper bound Bayes’ error in binary hypothesis testing. A general characterization of Chernoff information between two arbitrary probability measures was given in [11] (Theorem 32) by considering Rényi divergences which can be interpreted as scaled skewed Bhattacharyya divergences. Since its inception, the Chernoff information has proven useful as a statistical divergence (Chernoff divergence) in many applications ranging from information fusion to quantum metrology due to its empirical robustness property [19]. Informally, we may observe empirically that in practice the skewed Bhattacharyya divergence is more stable around the Chernoff exponent ${\alpha }^{*}$ than in other part of the range $(0,1)$. By considering the maximal extension of the exponential arc joining two densities p and q on a Lebesgue space $L^1\left(\mu \right)$, we built full likelihood ratio exponential families [10]${\mathcal{E}}_{pq}$ (LREFs) in Section 2. When the LREF ${\mathcal{E}}_{pq}$ is a regular exponential family (with coinciding support of p and q), both the forward and reverse Kullback–Leibler divergence are finite and can be rewritten as finite Bregman divergences induced by the log-normalizer $F_{pq}$ of ${\mathcal{E}}_{pq}$ which amounts to minus skewed Bhattacharyya divergences. Since log-normalizers of exponential families are strictly convex, we deduced that the skewed Bhattacharyya divergences are strictly concave and their maximization yielding the Chernoff information is hence proven unique. As a byproduct, this geometric characterization in $L^1\left(\mu \right)$ allowed us to prove that the intersection of a e-geodesic with a m-bisector is unique in dually flat subspaces of $L^1\left(\mu \right)$, and similarly that the intersection of a m-geodesic with a e-bisector is unique (Proposition 8). We then considered the exponential families of univariate and multivariate normal distributions: We reported closed-form solutions for the Chernoff information of univariate normal distribution and centered normal distributions with scaled covariance matrices, and show how to implement efficiently a dichotomic search for approximating the Chernoff information between two multivariate normal distributions (Algorithm 3).

Table 1. Summary of the optimal conditions characterizing the Chernoff exponent.

Generic case
Primal LREF	${\mathrm{OC}}_{\alpha }:D_{\mathrm{KL}}[{\left(pq\right)}_{{\alpha }^{*}}^G:p]=D_{\mathrm{KL}}[{\left(pq\right)}_{{\alpha }^{*}}^G:q]$
Dual LREF	${\mathrm{OC}}_{\beta }:\beta \left({\alpha }^{*}\right)=E_{{\left(pq\right)}_{{\alpha }^{*}}^G}\left[\log \frac{p\left(x\right)}{q\left(x\right)}\right]=0$
Geometric OC	${\left(pq\right)}_{{\alpha }^{*}}^G={\gamma }^G(p,q)\cap {\mathrm{Bi}}_{\mathrm{KL}}^{\mathrm{left}}(p,q]$
Case of exponential families
Bregman	${\mathrm{OC}}_{\mathrm{EF}}:B_F({\theta }_1:{\theta }_{{\alpha }^{*}})=B_F({\theta }_2:{\theta }_{{\alpha }^{*}})$
Fenchel–Young	${\mathrm{OC}}_{\mathrm{YF}}:Y_{F,F^{*}}({\theta }_1:{\eta }_{{\alpha }^{*}})=Y_{F,F^{*}}({\theta }_2:{\eta }_{{\alpha }^{*}})$
Simplified	${\mathrm{OC}}_{{\mathrm{SEF}}^{\prime }}:F_{{\theta }_1,{\theta }_2}^{\prime }\left(\alpha \right)=0$
	${\mathrm{OC}}_{\mathrm{SEF}}:{({\theta }_2-{\theta }_1)}^{\top }\nabla F({\theta }_1+{\alpha }^{*}({\theta }_2-{\theta }_1))=F\left({\theta }_2\right)-F\left({\theta }_1\right)$
Geometric OC	${\gamma }_{pq}^e\left(\alpha \right)\cap {\mathrm{Bi}}^m(p,q)$
1D EF	${\alpha }^{*}=\frac{{F^{\prime }}^{-1}\left(\frac{F\left({\theta }_2\right)-F\left({\theta }_1\right)}{{\theta }_2-{\theta }_1}\right)-{\theta }_2}{{\theta }_1-{\theta }_2}$
Gaussian case
1D Gaussians	${\mathrm{OC}}_{\mathrm{Gaussian}}:\left(\frac{{\mu }_2}{{\sigma }_2^2}-\frac{{\mu }_1}{{\sigma }_1^2}\right)m_{\alpha }-\left(\frac{1}{2{\sigma }_2^2}-\frac{1}{2{\sigma }_1^2}\right)v_{\alpha }=\frac{1}{2}\log \frac{{\sigma }_2^2}{{\sigma }_1^2}+\frac{{\mu }_2^2}{2{\sigma }_2^2}-\frac{{\mu }_1^2}{2{\sigma }_1^2}$
	${\alpha }^{*}$ is root of quadratic polynomial in $(0,1)$
Centered Gaussians	${\mathrm{OC}}_{\mathrm{Centered}\mathrm{Gaussians}}:{\sum }_{i=1}^d\frac{1-{\lambda }_i}{{\alpha }^{*}+(1-{\alpha }^{*}){\lambda }_i}+\log {\lambda }_i=0$
	where ${\lambda }_i$ is the i-th eigenvalue of ${\Sigma }_1{\Sigma }_2^{-1}$
Centered Gaussians	${\alpha }^{*}=\frac{s-1-\log s}{(s-1)\log s}\in (0,1)$
scaled covariances	when ${\Sigma }_2=s{\Sigma }_1$

summarizes the various optimal condition studied characterizing the Chernoff exponent. Finally, inspired by the Chernoff information study, we defined in Section 4, the forward and reverse Bregman–Chernoff divergences [66], and show how these divergences are related to the capacity of a discrete memoryless channel and the minimax redundancy of universal coding in information theory [13].

Additional material including Maxima and Java^® snippet codes is available online at https://franknielsen.github.io/ChernoffInformation/index.html (accessed on 30 July 2022).