Tight Bounds Between the Jensen–Shannon Divergence and the Minmax Divergence

Abstract

Motivated by questions arising at the intersection of information theory and geometry, we compare two dissimilarity measures between finite categorical distributions. One is the well-known Jensen–Shannon divergence, which is easy to compute and whose square root is a proper metric. The other is what we call the minmax divergence, which is harder to compute. Just like the Jensen–Shannon divergence, it arises naturally from the Kullback–Leibler divergence. The main contribution of this paper is a proof showing that the minmax divergence can be tightly approximated by the Jensen–Shannon divergence. The bounds suggest that the square root of the minmax divergence is a metric, and we prove that this is indeed true in the one-dimensional case. The general case remains open. Finally, we consider analogous questions in the context of another Bregman divergence and the corresponding Burbea–Rao (Jensen–Bregman) divergence.

1. Introduction

The starting point of this work is the introduction of the minmax divergence between two finite categorical distributions. It is a special case of a natural measurement arising in certain constructions from computational geometry and topology. Specifically, it coincides with the smallest radius at which two appropriately defined balls intersect. However, since our results promise to be useful also outside the field of geometry and topology, we will tailor the exposition accordingly.

The main result are tight bounds between the minmax divergence and the standard Jensen–Shannon divergence (as well as its generalizations). Given these bounds and the well-known fact the square root of the Jensen–Shannon divergence is a metric, we ask if the same can be proven about the minmax divergence. A supplementary result is a proof that in one dimension its square root is a metric. The general case is left as an open question.

The basic definition of the minmax divergence is based on the classic notion of Shannon entropy and has an intuitive information theoretic interpretation, as will be explained shortly. The definition coincides with an interpretation of the Chernoff information recently provided by Nielsen in [1]. In Section 2, we generalize this definition slightly, allowing us to work with the entire positive orthant of ${\mathbb{R}}^n$. Later in Section 5, we further generalize using the language of Bregman divergences, which allows us to work with other notions of entropy, such as the Burg entropy. Before we proceed, we mention that the following exposition is tailored towards this generalization. In particular, we prefer to work with negative Shannon entropy. Being a strictly convex function it allows for a smoother transition to the case of Bregman divergences, which are defined via such functions.

Shannon entropy and related concepts. We start from basic definitions and review their information-theoretical interpretations. Consider a random variable that takes one of n values. Letting $x=(x_1,x_2,\dots ,x_n)$ be the vector of probabilities, we can encode the outcome with expected efficiency $-E\left(x\right)$, in which

$$\begin{array}{cc}\hfill E\left(x\right)& ={\sum }_{i=1}^n{x}_{i}ln{x}_i\hfill \end{array}$$

(1)

is the (negative) Shannon entropy of x. We remark that if a binary logarithm was used, this quantity would be expressed in bits. We use natural logarithms to simplify calculations. Suppose we mistakenly assume that the underlying probability vector is $y=(y_1,y_2,\dots ,y_n)$, and we encode the random variable based on this faulty assumption. The expected efficiency is then $-E\left(y\right)-\langle \nabla E\left(y\right),x-y\rangle$. Comparing this with $-E\left(x\right)$, we get

$$\begin{array}{cc}\hfill D(x\parallel y)& =E\left(x\right)-E\left(y\right)-\langle \nabla E\left(y\right),x-y\rangle \hfill \end{array}$$

(2)

$$\begin{array}{cc}\hfill & ={\sum }_{i=1}^n{x}_{i}ln\frac{x_i}{y_i}\hfill \end{array}$$

(3)

as a measure of the efficiency loss due to the faulty assumption. This quantity is often referred to as the relative entropy, which we note is not symmetric. Next assume we know that the random variable is either chosen according to the distribution x or according to the distribution y, each with $50\%$ likelihood. Our best bet is to encode the result as if the underlying probability distribution were the average, $\mu =\frac{1}{2}(x+y)$. The expected efficiency is $-E\left(\mu \right)$, which we compare with $-\frac{1}{2}E\left(x\right)-\frac{1}{2}E\left(y\right)$, the expected efficiency assuming we know from which distribution the random variable is chosen. The difference,

$$\begin{array}{cc}\hfill \mathrm{JS}(x,y)& =\frac{1}{2}E\left(x\right)+\frac{1}{2}E\left(y\right)-E\left(\mu \right)\hfill \end{array}$$

(4)

$$\begin{array}{cc}\hfill & =\frac{1}{2}{\sum }_{i=1}^n[x_{i}ln\frac{2x_i}{x_i+y_i}+y_{i}ln\frac{2y_i}{x_i+y_i}]\hfill \end{array}$$

(5)

is the Jensen–Shannon divergence. This can be generalized to non-negative likelihoods $\xi +\eta =1$, for which our best bet is to encode using $\xi x+\eta y$, with expected divergence $\xi E\left(x\right)+\eta E\left(y\right)-E(\xi x+\eta y)$. There are unique likelihoods, ${\xi }_0+{\eta }_0=1$, for which the expected divergence is maximized. Because E is convex, this is also the situation in which the maximum divergence is minimized. We therefore consider this solution our best bet if we do not know how x and y split the likelihood. Setting $z={\xi }_{0}x+{\eta }_{0}y$, the expected efficiency is $-E\left(z\right)$, which we compare with $-{\xi }_{0}E\left(x\right)-{\eta }_{0}E\left(y\right)$. The difference,

$$\begin{array}{cc}\hfill \mathrm{Mx}(x,y)& ={\xi }_{0}E\left(x\right)+{\eta }_{0}E\left(y\right)-E\left(z\right)\hfill \end{array}$$

(6)

$$\begin{array}{cc}\hfill & ={\sum }_{i=1}^n[{\xi }_0{x}_{i}ln\frac{x_i}{z_i}+{\eta }_0{y}_{i}ln\frac{y_i}{z_i}],\hfill \end{array}$$

(7)

minimizes the maximum divergence over all possible choices of likelihoods $\xi +\eta =1$. We therefore call $\mathrm{Mx}(x,y)$ the minmax divergence between the two probability distributions. In the next section, we will provide an easier to work with alternative definition.

Main result. Our main result is a comparison of the minmax divergence with the standard Jensen–Shannon divergence. Specifically, we prove

$$\begin{array}{c}\hfill \frac{eln2}{2}\mathrm{Mx}(x,y)\le \mathrm{JS}(x,y)\le \mathrm{Mx}(x,y)\end{array}$$

(8)

as tight bounds. We remark that the final result uses a generalized form of the above concepts, as defined in the next section.

Given that $\frac{eln2}{2}\approx 0.94208$, the two divergences are very close. (As we will argue shortly, it makes sense to consider the square roots of these divergences, in which case the constant is approximately $0.9706$, which is even closer to 1.) Additionally, the Jensen–Shannon divergence and the minmax divergence also yield the same intrinsic metric, which is $\frac{1}{4}$ times the intrinsic metric defined by the relative entropy. In the literature, the relative entropy is also known as the Kullback–Leibler divergence [2], and the corresponding intrinsic metric is known as the Fisher information metric [3]. Both are however not length metrics, which means that integrating infinitesimal steps gives a corresponding intrinsic metric, which is different from the original metric.

In light of these similarities, we ask if they share more properties. In particular Endres and Schindelin proved that $\sqrt{\mathrm{JS}(x,y)}$ is a metric [4]. We were able to prove a similar result in dimension one (specifically, in a slightly generalized setting in which $x,y\in {\mathbb{R}}_{+}$). Namely, we prove that $\sqrt{\mathrm{Mx}(x,y)}$ is a metric. The result in higher dimensions turns out to be challenging, and remains open. We do believe that the geometric proof techniques we introduce are worth sharing, and are likely to be useful in the high-dimensional case (in combination with some other techniques).

Applications in computational geometry. We briefly explain how the above concepts can be used in computational geometry and topology and, in particular, why the main result is useful. Our main motivation comes from the field of topological data analysis, a subfield of computational geometry and topology. In short, the idea is to characterize the shape of data or, in other words, its geometric-topological structure. We briefly describe a simple case to which our main result is immediately applicable. For a comprehensive treatment of topological data analysis in the context of arbitrary Bregman divergences, which includes our setup with relative entropy, see [5].

In the simplest case—which is also most relevant here—the input data is a finite collection of points. We consider the union of the balls centered at these points and increase their common radius from zero to infinity. As the radius changes, so does the connectivity (or topology) of the union of balls. One basic topological property is the structure of the connected components. At radius zero, each point constitutes its own connected component, and these components may merge as the radius increases. Specifically, two components may merge when two balls develop a nonempty intersection for the first time. (We remark that higher-degree topological features can also be considered, but this requires tools from algebraic topology, which are beyond the scope of this paper.).

This setup can be easily implemented in the Euclidean space. However, in the more interesting situation in which each point is a finite categorical distribution, the balls are better defined using the relative entropy (and not the Euclidean distance), as it allows the outcome to have an information-theoretic interpretation, as outlined above. The radius at which two balls intersect coincides with the minmax divergence [6], whose direct computation (especially for many pairs of points) can be slow. The proposed inequality suggests we compute the approximating Jensen–Shannon divergence instead. It remains fairly accurate while being significantly faster, simpler, and more robust. In particular, this allows for a simple computation of a weighted undirected graph that describes the connected structure of data measured with relative entropy. In topological data analysis language, it would be called the 1-skeleton of the Čech complex (the full complex would be a weighted simplicial complex that encodes intersections between arbitrary tuples of balls and captures higher-degree topological information.).

In summary, our results simplify a fundamental computation in topological data analysis for an important kind of inputs. This is often the first step towards computing a topological descriptor, which has been proven useful in a variety of fields [7] ranging from biology, to astronomy, to materials science. Moreover, if the square root of the minmax divergence is indeed a metric (a fact we were unable to prove beyond dimension one), it would allow one to use existing theoretic in computational tools. Here we mention stability results in topological data analysis that exploit the metric structure of data [8], and classical nearest neighbour search tools that focus on metric spaces [9].

Outline. Section 2 introduces the main concepts in their generalized forms and proves some of their fundamental properties. Section 3 shows that the Jensen–Shannon divergence approximates the minmax divergence within a factor $1.061\dots$. Section 4 proves that in one dimension the square root of the minmax divergence is a metric. Section 5 extends the results beyond the Shannon entropy using the framework of Bregman divergences. Section 6 concludes the paper.

Related work. Many concepts used in this paper lead back to the seminal work of Claude Shannon [10] on information theory and, in particular, the notion of Shannon entropy. This notion was extended to a dissimilarity measure between two probability distributions by Kullback and Leibler [2], often referred to as the relative (Shannon) entropy or the Kullback–Leibler divergence. The best known metric derived from relative Shannon entropy is based on the Jensen–Shannon divergence, defined by Lin in 1991 [11]. Interestingly, its more general form was introduced a decade earlier by Burbea and Rao [12]. Lew Bregman introduced a notion of Bregman divergence [13], which generalizes the relative entropy. Various other metrics derived from Bregman divergences were studied in [14,15]. More recently, Bregman divergences were further generalized by Nielsen in various ways [16,17,18]. Our work is motivated by results at the intersection of Bregman geometry and computational geometry. The starting point for this direction is the work of Boissonnat, Nielsen and Nock [19,20,21] on computational geometry in the Bregman context.

2. Generalized Definitions

In the Introduction, we described basic concepts (such as the relative entropy) along with their information-theoretical interpretation. In this section, we introduce more general versions of these concepts, allowing us to work in the entire positive orthant ${\mathbb{R}}_{+}^n$. We remark that the definitions change in subtle ways, and that applying the usual definitions in this extended setup may on occasion be non-sensical.

For the reminder of this paper, we are exclusively concerned with two spaces: the n-dimensional positive orthant, denoted ${\mathbb{R}}_{+}^n$, which consists of all points $x=(x_1,x_2,\dots ,x_n)$ with $x_i>0$ for $1\le i\le n$, and the open $(n-1)$-dimensional standard simplex, denoted $\Delta ={\Delta }^{n-1}$, which consists of the points $x\in {\mathbb{R}}_{+}^n$ that satisfy ${\sum }_{i=1}^n{x}_i=1$. A point $x\in \Delta$ is really a finite probability distribution, which leads us to believe that $\Delta$ is the more important setting. However, ${\mathbb{R}}_{+}^n$ is often easier to work with, and we can restrict results to $\Delta \subseteq {\mathbb{R}}_{+}^n$.

Shannon entropy and relative entropy. Writing $lnt$ for the natural logarithm of $t>0$, the (negative) Shannon entropy is $E:{\mathbb{R}}_{+}^n\to \mathbb{R}$ defined by $E\left(x\right)={\sum }_{i=1}^n[x_{i}ln{x}_i-x_i]$. (Subtracting the extra term is a standard trick to simplify computations while not affecting the interpretation of the resulting relative entropy. However, the interpretation of the Shannon entropy mentioned in the Introduction holds only up to a constant.) We write ${E|}_{\Delta }:\Delta \to \mathbb{R}$ for its restriction to the standard simplex.

As mentioned in Section 1, $-E\left(x\right)$ pertains to the expected efficiency of encoding a random variable that distributes according to $x\in \Delta$. If we encode the values assuming the random variable distributes according to $y\in \Delta$, the expected efficiency is the negative of the best linear approximation of E at y evaluated at x, which is $-E\left(y\right)-\langle \nabla E\left(y\right),x-y\rangle$. The relative entropy can be viewed as the non-negative difference between the above approximation and the Shannon entropy of x; see Figure 1.

Going back to $x,y\in {\mathbb{R}}_{+}^n$, we consider a generalized form of the relative entropy from x to y:

$$\begin{array}{cc}\hfill D(x\parallel y)& =E\left(x\right)-E\left(y\right)-\langle \nabla E\left(y\right),x-y\rangle \hfill \end{array}$$

(9)

$$\begin{array}{cc}\hfill & ={\sum }_{i=1}^n[x_{i}ln\frac{x_i}{y_i}-x_i+y_i]\hfill \end{array}$$

(10)

$$\begin{array}{cc}\hfill & ={\sum }_{i=1}^{n}D(x_i\parallel y_i).\hfill \end{array}$$

(11)

Note the extra additive terms, which are absent in the usual form and ensure that the result is nonnegative. When restricted to the standard simplex, it simplifies to its more standard form, namely ${\sum }_{i=1}^n{x}_{i}ln\frac{x_i}{y_i}$.

We say the relative entropy is decomposable because it satisfies (11). Observe that the notation separates the points x and y by a double bar as opposed to a comma to remind us that the measure is generally not symmetric. The relative entropy is also known as the Kullback divergence, the Kullback–Leibler divergence, or simply the divergence; see [3] (page 57). It measures the divergence in encoding efficiency due to assuming a different distribution.

Jensen–Shannon divergence. Similar to the relative entropy, the Jensen–Shannon divergence generalized to the positive orthant is a function $\mathrm{JS}:{\mathbb{R}}_{+}^n\times {\mathbb{R}}_{+}^n\to \mathbb{R}$, but it is symmetric by taking the average relative entropy from x to $\mu =\frac{1}{2}(x+y)$ and from y to $\mu$; see Figure 2:

$$\begin{array}{cc}\hfill \mathrm{JS}(x,y)& =\frac{1}{2}[D(x\parallel \mu )+D(y\parallel \mu )]\hfill \end{array}$$

(12)

$$\begin{array}{cc}\hfill & =\frac{1}{2}E\left(x\right)+\frac{1}{2}E\left(y\right)-E\left(\mu \right)\hfill \end{array}$$

(13)

$$\begin{array}{cc}\hfill & =\frac{1}{2}{\sum }_{i=1}^n[x_{i}ln\frac{2x_i}{x_i+y_i}+y_{i}ln\frac{2y_i}{x_i+y_i}].\hfill \end{array}$$

(14)

$$\begin{array}{cc}\hfill & ={\sum }_{i=1}^n\mathrm{JS}(x_i,y_i),\hfill \end{array}$$

(15)

in which we get (13) by noting $(x-\mu )+(y-\mu )=0$. Similar to the relative entropy, the Jensen–Shannon divergence is decomposable (15). As pointed out in [4], $\mathrm{JS}(x,y)$ measures the divergence of expected efficiency when we encode a random variable that distributes half of the time according to $x\in \Delta$ and the other half of the time according to $y\in \Delta$ using the average of x and y.

It is not difficult to prove that if we substitute any other point for $\mu$, then the average relative entropy and therefore the divergence in efficiency increases. For reasons that will become obvious shortly, we prove this optimality result for a weighted version of this divergence. Let $\xi +\eta =1$ and set $w=\xi x+\eta y$. The corresponding weighted Jensen–Shannon divergence is

$$\begin{array}{cc}\hfill {\mathrm{JS}}_{\xi }(x\parallel y)& =\xi D(x\parallel w)+\eta D(y\parallel w)\hfill \end{array}$$

(16)

$$\begin{array}{cc}\hfill & =\xi E\left(x\right)+\eta E\left(y\right)-E\left(w\right)\hfill \end{array}$$

(17)

$$\begin{array}{cc}\hfill & ={\sum }_{i=1}^n[\xi x_{i}ln\frac{x_i}{w_i}+\eta y_{i}ln\frac{y_i}{w_i}]\hfill \end{array}$$

(18)

$$\begin{array}{cc}\hfill & ={\sum }_{i=1}^n{\mathrm{JS}}_{\xi }(x_i\parallel y_i).\hfill \end{array}$$

(19)

To prove optimality, we set $f_{\xi }\left(u\right)=\xi D(x\parallel u)+\eta D(y\parallel u)$, noting that $f_{\xi }\left(w\right)={\mathrm{JS}}_{\xi }(x\parallel y)$. The following lemma and proof can also be found in [22].

Lemma 1 

(Optimality of Weighted JS). Let $x,y\in {\mathbb{R}}_{+}^n$, $\xi +\eta =1$, and $w=\xi x+\eta y$. Then $f_{\xi }\left(w\right)\le f_{\xi }\left(u\right)$ for every $u\in {\mathbb{R}}_{+}^n$, with equality iff $u=w$.

Proof. 

Computing the difference, $X=f_{\xi }\left(u\right)-f_{\xi }\left(w\right)$, most terms cancel and we get

$$\begin{array}{cc}\hfill X& =\xi [D(x\parallel u)-D(x\parallel w)]+\eta [D(y\parallel u)-D(y\parallel w)]\hfill \end{array}$$

(20)

$$\begin{array}{cc}\hfill & =E\left(w\right)-E\left(u\right)-\langle \nabla E\left(u\right),w-u\rangle ,\hfill \end{array}$$

(21)

in which we use $\xi (x-w)+\eta (y-w)=0$. In other words, the difference is equal to $D(w\parallel u)$, which is non-negative and zero iff $u=w$ by the strict convexity of E. □

Remark 1. 

To get an information theoretic interpretation of the result, we assume $x,y\in \Delta$ and suppose a random variable that distributes according to x with likelihood $0\le \xi \le 1$ and according to y with likelihood $\eta =1-\xi$. Lemma 1 says that our best bet is to encode with $w=\xi x+\eta y$. In words, w minimizes the weighted Jensen–Shannon divergence.

Minmax divergence. Similar to the Jensen–Shannon divergence, the minmax divergence is a symmetric function $\mathrm{Mx}:{\mathbb{R}}_{+}^n\times {\mathbb{R}}_{+}^n\to \mathbb{R}$. It is defined by mapping $x,y$ to the larger relative entropy to a third point, $z\in {\mathbb{R}}_{+}^n$, in which z is selected so as to minimize this maximum:

$$\begin{array}{cc}\hfill \mathrm{Mx}(x,y)& =\underset{z\in {\mathbb{R}}_{+}^n}{inf}max\{D(x\parallel z),D(y\parallel z)\}.\hfill \end{array}$$

(22)

We call the point z that gives the infimum the minmax divergence center of the pair. We remark that the general form of minmax divergence introduced in Section 5 generalizes the Chernoff information [1] popular in statistics.

As proved for example in [6], z is a convex combination of x and y. We strengthen this result by proving that it is the particular convex combination that maximizes the weighted Jensen–Shannon divergence, and that this weighted Jensen–Shannon divergence equals the minmax divergence.

Lemma 2 

(Minmax-Maxweight). Let $x,y\in {\mathbb{R}}_{+}^n$ and ${\xi }_0+{\eta }_0=1$ such that $z_0={\xi }_{0}x+{\eta }_{0}y$ is the minmax divergence center. Then $\mathrm{Mx}(x,y)={\mathrm{JS}}_{{\xi }_0}(x\parallel y)\ge {\mathrm{JS}}_{\xi }(x\parallel y)$ for all ξ.

Proof. 

Let $\xi +\eta =1$ and write $z=\xi x+\eta y$ for a general affine combination of x and y. The restriction of E to the line of such points is a strictly convex function. It follows that there is a unique affine combination, $z_0={\xi }_{0}x+{\eta }_{0}y$, such that $D(x\parallel z_0)=D(y\parallel z_0)$. The weighted Jensen–Shannon divergence at $z_0$ is ${\mathrm{JS}}_{{\xi }_0}(x\parallel y)={\xi }_{0}D(x\parallel z_0)+{\eta }_{0}D(y\parallel z_0)$, which is equal to $D(x\parallel z_0)=D(y\parallel z_0)$, as claimed.

By convexity of the restriction of E, the maximum relative entropy is larger than this shared value for every affine combination $z\ne z_0$. Similarly, the weighted Jensen–Shannon divergence is smaller than ${\mathrm{JS}}_{{\xi }_0}(x\parallel y)$ at every affine combination $z\ne z_0$. □

Remark 2. 

In contrast to the other measures discussed so far, the minmax divergence is not decomposable.

Remark 3. 

Since the minmax divergence center of x and y lies between these two points, it is constrained to a compact set so we can replace the infimum in (22) by a minimum. This justifies the name of corresponding divergence. Lemma 2 says that the minimum of the maximum relative entropy is equal to the maximum weighted Jensen–Shannon divergence, which justifies the name of the lemma.

Explicit formula. While the minmax divergence is defined as an infimum, it is possible to compute it explicitly. To state the formula, we introduce $G:{\mathbb{R}}_{+}\to \mathbb{R}$ defined by

$$\begin{array}{cc}\hfill G\left(t\right)& =\frac{t^{\frac{t}{t-1}}}{e}-\frac{tlnt}{t-1}.\hfill \end{array}$$

(23)

It is well defined at all positive $t\ne 1$, and we get $G\left(1\right)=0$ in the limit because ${lim}_{t\to 1}{t}^{{\displaystyle \frac{t}{t-1}}}=e$ and ${lim}_{t\to 1}\frac{tlnt}{t-1}=1$ by the rule de l’Hôpital.

Lemma 3 

(Minmax divergence Formula). For $x,y\in {\mathbb{R}}_{+}$, we have $\mathrm{Mx}(x,y)=xG\left(\frac{y}{x}\right)$.

Proof. 

For $x=y$ both sides vanish, so the relation holds. We therefore assume $x\ne y$ for the remainder of the proof. Letting z be the minmax divergence center of x and y, we recall that the derivative at z is the slope of the line that passes through $(x,E(x\left)\right)$ and $(y,E(y\left)\right)$: $E^{\prime }\left(z\right)=lnz=[E\left(y\right)-E\left(x\right)]/[y-x]$. We express this equation in terms of $\frac{y}{x}$:

$$\begin{array}{cc}\hfill lnz& ={\displaystyle \frac{ylny-xlnx-(y-x)}{y-x}}\hfill \end{array}$$

(24)

$$\begin{array}{cc}\hfill & ={\displaystyle \frac{\frac{y}{x}(ln\frac{y}{x}+lnx)-lnx}{\frac{y}{x}-1}}-1\hfill \end{array}$$

(25)

$$\begin{array}{cc}\hfill & ={\displaystyle \frac{\frac{y}{x}ln\frac{y}{x}}{\frac{y}{x}-1}}+lnx-1.\hfill \end{array}$$

(26)

Write $A=({\displaystyle \frac{y}{x}}ln{\displaystyle \frac{y}{x}})/({\displaystyle \frac{y}{x}}-1)$ for the first term on the right-hand side of (26). Recall that the minmax divergence of x and y is the vertical distance between the point $(z,E(z\left)\right)$ and the line that passes through $(x,E(x\left)\right)$ and $(y,E(y\left)\right)$. By construction, the slope of this line is $lnz$. We express the vertical distance as the sum of two vertical distances, which we then express in terms of A and ${\displaystyle \frac{y}{x}}$:

$$\begin{array}{cc}\hfill \mathrm{Mx}(x,y)& =\left[\right(z-x)lnz]+\left[E\right(x)-E(z\left)\right]\hfill \end{array}$$

(27)

$$\begin{array}{cc}\hfill & =z-x+xlnx-xlnz\hfill \end{array}$$

(28)

$$\begin{array}{cc}\hfill & =z-x+xlnx-x[A+lnx-1]\hfill \end{array}$$

(29)

$$\begin{array}{cc}\hfill & =x[e^{A-1}-A]\hfill \end{array}$$

(30)

$$\begin{array}{cc}\hfill & =xG\left(\frac{y}{x}\right),\hfill \end{array}$$

(31)

in which we use (26) to get (29), we cancel $-x+xlnx$ and replace z to get (30), and finally use ${\displaystyle \frac{y}{x}}=e^{ln{\displaystyle \frac{y}{x}}}$ to get (31). □

Bregman divergences. The relative entropy can be viewed from the perspective of Bregman divergences [13]. Indeed, it is an instance of a Bregman divergence generated by the negative Shannon entropy. We briefly introduce the setup for general Bregman divergences, generated by arbitrary convex functions, or more technically functions of Legendre type.

Given an open convex set $\Omega \subseteq {\mathbb{R}}^d$, a function $F:\Omega \to \mathbb{R}$ is of Legendre type if it is (1) differentiable, (2) strictly convex and (3) the magnitude of its gradient diverges to positive infinity when evaluated at points converging to the boundary of the domain. Given a function F of Legendre type, the Bregman divergence [13] generated by F is defined as

$$\begin{array}{c}\hfill D_F:\Omega \times \Omega \to \mathbb{R},D_F(x\parallel y)=F\left(x\right)-(F\left(y\right)+\langle \nabla F\left(y\right),x-y\rangle ).\end{array}$$

We will use this concept to generalize our main result in Section 5.

3. Comparison of Divergences

We think of the Jensen–Shannon divergence as a readily computed approximation of the minmax divergence. The approximation is very close, and we prove in this section that the Jensen–Shannon divergence is always between $\frac{eln2}{2}=0.942\dots$ and 1 times the minmax divergence. We prove this first in one dimension and then generalize the result to n dimensions.

Approximation with ellipse. The main tool in proving the relation between the minmax divergence and the Jensen–Shannon divergence is the—surprisingly close—approximation of the graph of the Shannon entropy defined over ${\mathbb{R}}_{+}$ with an arc of an ellipse. We are interested in the interval $[0,e]$, so we choose the ellipse to

• pass through the points $(0,0)$ and $(e,0)$;
• have its minimum at the point $(1,-1)$;
• have the same curvature at $(1,-1)$ as the graph of the Shannon entropy.

Writing the ellipse as the zero-set of a function, we introduce $\Gamma :{\mathbb{R}}^2\to \mathbb{R}$ defined by

$$\begin{array}{cc}\hfill \Gamma (x_1,x_2)& =x_1^2+(2-e)x_1{x}_2+(3-e)x_2^2\hfill \\ \hfill & -e{x}_1+(2-e)x_2.\hfill \end{array}$$

(32)

It is not difficult to check that ${\Gamma }^{-1}\left(0\right)$ satisfies the above three properties. We also note that $\Gamma (x_1,x_2)$ is negative for points inside the ellipse and positive for points outside the ellipse. Within $[0,e]$, the approximation of E by the lower portion of the ellipse is astonishingly close, and we exaggerate the difference in Figure 3 to make it visible. We prove that from 0 to 1 the graph of E is below the ellipse, and from 1 to e it is above the ellipse.

Lemma 4 

(Below-Above). Letting $E:{\mathbb{R}}_{+}\to \mathbb{R}$ be the 1-dimensional Shannon entropy and $\Gamma :{\mathbb{R}}^2\to \mathbb{R}$ the function defined in (32). Then

$$\begin{array}{c}\hfill \Gamma (x,E\left(x\right))\left\{\begin{array}{cc}>0& \mathrm{for}0<x<1,\hfill \\ <0& \mathrm{for}1<x<e.\hfill \end{array}\right.\end{array}$$

(33)

Proof. 

Write $\Gamma (x,E(x\left)\right)=xf\left(x\right)$, in which

$$\begin{array}{cc}\hfill f\left(x\right)& =(2x-2)+(2-e+ex-4x)lnx\hfill \\ \hfill & +(3-e)x{ln}^{2}x.\hfill \end{array}$$

(34)

We have $f\left(1\right)=f\left(e\right)=0$ by construction of $\Gamma$, but not necessarily $f\left(0\right)=0$ because we divided by x. It suffices to prove that $f\left(x\right)$ is positive for $0<x<1$ and negative for $1<x<e$. Next we compute the first two derivatives, again after removing a monotonic factor to simplify the computations. Specifically, we write $f^{\prime }\left(x\right)=\frac{1}{x}g\left(x\right)$, in which

$$\begin{array}{cc}\hfill g\left(x\right)& =(e-2)(x-1-xlnx)+(3-e)x{ln}^{2}x,\hfill \end{array}$$

(35)

$$\begin{array}{cc}\hfill g^{\prime }\left(x\right)& =(3-e){ln}^{2}x-(3e-8)lnx.\hfill \end{array}$$

(36)

The derivative of g is quadratic in $lnx$, with zeros at $x=1$ and $x=u_0=exp\left(\frac{3e-8}{3-e}\right)=1.732\dots$. Hence, $g^{\prime }\left(x\right)$ is negative for $1<x<u_0$ and positive outside the corresponding closed interval. Returning to g, we note that g is zero, negative, positive at $1<u_0<e$:

$$\begin{array}{cc}\hfill g\left(1\right)& =0,g\left(u_0\right)=-0.010\dots ,g\left(e\right)=0.047\dots .\hfill \end{array}$$

(37)

Our analysis of $g^{\prime }$ implies that g increases from 0 to 1, it decreases from 1 to $u_0$, and finally it increases again from $u_0$ to e, with a zero at some value $u_1$ between $u_0$ and e. Hence, f decreases from 0 to $u_1$, with $1<u_1<e$, and it increases from $u_1$ to e. Since $f\left(1\right)=f\left(e\right)=0$, this implies that f is positive from 0 to 1 and negative from 1 to e. The claimed inequalities for $\Gamma$ follow. □

Midpoint lines. We are interested in the relative position of two midpoint lines. The first is defined by the Shannon entropy, E, and the second by the function $G:[0,e]\to \mathbb{R}$ whose graph is the portion of the ellipse on and below the horizontal coordinate axis. Both lines consist of points $(\frac{1}{2}(x+y),t)$ with $0\le x\le 1\le y\le e$, in which the points of the first line satisfy $E\left(x\right)=E\left(y\right)=t$ and the points of the second line satisfy $G\left(x\right)=G\left(y\right)=t$.

The ellipse can be obtained by shearing a circle, and since this operation takes straight lines to straight lines, it follows that the midpoint line of G is a straight line segment. Its endpoints are $(1,-1)$ and $(\frac{e}{2},0)$. By Lemma 4, the midpoint line of E is a curve that connects the same two endpoints but lies otherwise to the left of the line segment; see Figure 3.

Inequalities. Recall the definition of the weighted Jensen–Shannon divergence for a real parameter and points:

$$\begin{array}{cc}\hfill {\mathrm{JS}}_{\xi }(x\parallel y)& =\xi E\left(x\right)+\eta E\left(y\right)-E\left(z\right),\hfill \end{array}$$

(38)

in which $\eta =1-\xi$ and $z=\xi x+\eta y$. The (unweighted) Jensen–Shannon divergence is $\mathrm{JS}(x,y)={\mathrm{JS}}_{1/2}(x\parallel y)$, and from Lemma 2 we know that the minmax divergence can be written as $\mathrm{Mx}(x,y)={max}_{\xi }{\mathrm{JS}}_{\xi }(x\parallel y)$. We prove that the two measures of information divergence are good approximations of each other.

Theorem 1 

(Loss Comparison). Letting $x,y\in {\mathbb{R}}_{+}^n$, we have $\frac{eln2}{2}\mathrm{Mx}(x,y)\le \mathrm{JS}(x,y)\le \mathrm{Mx}(x,y)$.

Proof. 

The upper bound on $\mathrm{JS}(x,y)$ follows from Lemma 2, so we focus on proving the lower bound. We begin with the 1-dimensional case, when $n=1$. Recall that $D(Cx\parallel Cy)=CD(x\parallel y)$ for every $C\ge 0$. Hence,

$$\begin{array}{cc}\hfill \mathrm{JS}(Cx,Cy)& =C\mathrm{JS}(x,y),\hfill \end{array}$$

(39)

$$\begin{array}{cc}\hfill \mathrm{Mx}(Cx,Cy)& =C\mathrm{Mx}(x,y),\hfill \end{array}$$

(40)

which implies that the ratio is independent of C. Given $x<y$, we can find $C>0$ such that $E\left(Cx\right)=E\left(Cy\right)$, so we assume $E\left(x\right)=E\left(y\right)$ for the remainder of the 1-dimensional argument. This implies that the minmax divergence center is $z=1$. Setting $x=0$ and $y=e$, we have $E\left(0\right)=E\left(e\right)=0$, and the ratio is

$$\begin{array}{cc}\hfill {\displaystyle \frac{\mathrm{Mx}(0,e)}{\mathrm{JS}(0,e)}}& ={\displaystyle \frac{-1}{E\left(\frac{e}{2}\right)}}={\displaystyle \frac{-1}{\frac{e}{2}ln\frac{e}{2}-\frac{e}{2}}}={\displaystyle \frac{2}{eln2}}.\hfill \end{array}$$

(41)

Observe that this ratio is the length of the left edge divided by the length of the right edge of the trapezoid $T_0$ in Figure 3. For a general pair $0<x<y$ with $E\left(x\right)=E\left(y\right)$, we represent the ratio by the left and right edges of a similar trapezoid, $T_x$. Importantly, the two trapezoids share $(1,-1)$ as their common lower left corner, and the bottom edge of $T_x$ has smaller positive slope than the bottom edge of $T_0$. The height of $T_0$ is 1 and that of $T_x$ is $t<1$. To compare the two ratios, we thus consider $T_x/t+(1-t,t-1)$, which is the scaled version of $T_x$ whose lower left corner is $(1,-1)$ and whose height is 1. The upper right corner of $T_x$ lies on the midpoint line of E, which by Lemma 4 implies that the upper right corner of the scaled trapezoid lies to the left of $\frac{e}{2}$ on the horizontal coordinate axis. It follows that the width of the scaled trapezoid is smaller than the width of $T_0$. Hence,

$$\begin{array}{c}\hfill {\displaystyle \frac{\mathrm{Mx}(x,y)}{\mathrm{JS}(x,y)}}<{\displaystyle \frac{\mathrm{Mx}(0,e)}{\mathrm{JS}(0,e)}}={\displaystyle \frac{2}{eln2}}.\end{array}$$

(42)

We get equality for $x=0$ and for $x=1$, which implies the claimed lower bound for $n=1$ dimension. Moving on to $n\ge 1$ dimensions, we recall that the minmax divergence center is a convex combination of the two points [6]. Specifically, the center satisfies $z=\xi x+\eta y$ with $\xi ,\eta \ge 0$ and $\xi +\eta =1$. Using Lemma 2, the decomposability of the weighted Jensen–Shannon divergence (19), and the claimed inequality in $n=1$ dimension—in this sequence—we get

$$\begin{array}{cc}\hfill \mathrm{Mx}(x,y)& ={\mathrm{JS}}_{\xi }(x\parallel y)\hfill \end{array}$$

(43)

$$\begin{array}{cc}\hfill & ={\sum }_{i=1}^n{\mathrm{JS}}_{\xi }(x_i\parallel y_i)\hfill \end{array}$$

(44)

$$\begin{array}{cc}\hfill & \le {\displaystyle \frac{2}{eln2}}{\sum }_{i=1}^n\mathrm{JS}(x_i,y_i)\hfill \end{array}$$

(45)

$$\begin{array}{cc}\hfill & ={\displaystyle \frac{2}{eln2}}\mathrm{JS}(x,y),\hfill \end{array}$$

(46)

as claimed. □

Remark 4. 

The bounds in Theorem 1 are tight. To see this for the lower bound, we note that $\mathrm{Mx}(0,e)=1$ and $\mathrm{JS}(0,e)=-E\left(\frac{e}{2}\right)=\frac{eln2}{2}$. While 0 is formally not part of the domain, we can take points arbitrarily close to 0 and thus get the bound in the limit. To see that the upper bound is tight, we let $\epsilon >0$ be small and set $x=1-\epsilon$ and $y=1+\epsilon$. We can see the two entropies geometrically as the vertical distance of two points on the graph of E below the line that passes through $(x,E(x\left)\right)$ and $(y,E(y\left)\right)$; see Figure 2. For $\mathrm{JS}(x,y)$ this point is $(\mu ,E(\mu \left)\right)=(1,-1)$, and for $\mathrm{Mx}(x,y)$ this point is $(z,E(z\left)\right)$, in which z is the minmax divergence center of x and y. To determine z, we note that $D(x\parallel z)=D(y\parallel z)$. After some computations, including the Taylor expansions of $lns$ around $s=1$ and of $e^s$ around $s=0$, we find that z is $1-\frac{1}{3}{\epsilon }^2$ plus a fourth-order term in $\epsilon$. In words, z approaches $\mu =1$ much faster than a and b. It follows that in the limit, the two entropies are the same, as required.

4. Proof of Metric in Dimension One

As proved in [4], the square root of the Jensen–Shannon divergence is a metric in ${\mathbb{R}}_{+}$. Using the decomposability of the Jensen–Shannon divergence together with the Minkowski inequality, it is then easy to prove that this square root is also a metric in ${\mathbb{R}}_{+}^n$. We prove that the square root of the minmax divergence is a metric in ${\mathbb{R}}_{+}$. Since the minmax divergence is not decomposable, we do not know yet whether its square root is a metric in ${\mathbb{R}}_{+}^n$.

The ratio method. Suppose $A:\mathbb{R}\times \mathbb{R}\to \mathbb{R}$ satisfies the triangle inequality and $B:\mathbb{R}\times \mathbb{R}\to \mathbb{R}$ is another function on the product. To prove that B also satisfies the triangle inequality, we may consider the ratio, $f(x,y)=B(x,y)/A(x,y)$ and prove its monotonicity, that is:

$$\begin{array}{cc}\hfill f(a,b)& \le f(x,y)\hfill \end{array}$$

(47)

whenever $a\le x\le y\le b$.

Lemma 5 

(Triangle Inequality). Let $A,B,f:\mathbb{R}\times \mathbb{R}\to \mathbb{R}$ in which A satisfies the triangle inequality and $f=B/A$ is monotonic. Then B satisfies the triangle inequality.

Proof. 

Let $x\le y\le z$. Then

$$\begin{array}{cc}\hfill B(x,y)+B(y,z)& =f(x,y)A(x,y)+f(y,z)A(y,z)\hfill \end{array}$$

(48)

$$\begin{array}{cc}\hfill & \ge f(x,z)A(x,z)\hfill \end{array}$$

(49)

$$\begin{array}{cc}\hfill & =B(x,z),\hfill \end{array}$$

(50)

in which we get (49) using the monotonicity of f and the triangle inequality for A. □

To apply the lemma, we set $B(x,y)=\sqrt{\mathrm{Mx}(x,y)}$ and $A(x,y)=|\sqrt{y}-\sqrt{x}|$. Clearly, A is a metric in ${\mathbb{R}}_{+}$, so it will suffice to show that the ratio is monotonic.

A first application. As a warm up exercise, we use the ratio method expressed in Lemma 5 to re-prove the main result of [4].

Theorem 2 

(JS Revisited). Let $E:{\mathbb{R}}_{+}^n\to \mathbb{R}$ be the Shannon entropy and $\mathrm{JS}:{\mathbb{R}}_{+}^n\times {\mathbb{R}}_{+}^n\to \mathbb{R}$ map every pair to the Jensen–Shannon divergence. Then $\sqrt{\mathrm{JS}}$ is a metric in ${\mathbb{R}}_{+}^n$.

Proof. 

We begin with the one-dimensional case, $n=1$. It is clear that $\mathrm{JS}(x,y)$ is non-negative, zero iff $x=y$, and symmetric. It remains to prove that its square root satisfies the triangle inequality. Setting

$$\begin{array}{cc}\hfill f(x,y)& ={\displaystyle \frac{\sqrt{\mathrm{JS}(x,y)}}{|\sqrt{y}-\sqrt{x}|}},\hfill \end{array}$$

(51)

we will prove that f is monotonic as defined in (47). Recall that $\mathrm{JS}(Cx,Cy)=C\mathrm{JS}(x,y)$ for every $C>0$. It follows that $f(Cx,Cy)=f(x,y)$, which allows us to assume $x=1$. To simplify the notation, we set $t^2=y$ assuming $t^2\ge 1$. Writing $g\left(t\right)=\mathrm{JS}(1,t^2)/{(t-1)}^2=f{(1,y)}^2$, we aim at proving that g is monotonically decreasing, that is: $g^{\prime }\left(t\right)<0$ for all $t\ge 1$. Recall from (5) that

$$\begin{array}{cc}\hfill \mathrm{JS}(1,t^2)& =\frac{1}{2}\left[ln\frac{2}{1+t^2}+t^{2}ln\frac{2t^2}{1+t^2}\right],\hfill \end{array}$$

(52)

$$\begin{array}{cc}\hfill {\displaystyle \frac{\partial \mathrm{JS}(1,t^2)}{\partial t}}& =tln\frac{2t^2}{t^2+1}.\hfill \end{array}$$

(53)

The derivative of g is $g^{\prime }\left(t\right)=N\left(t\right)/{(t-1)}^3$, in which

$$\begin{array}{cc}\hfill N\left(t\right)& =(t-1){\displaystyle \frac{\partial \mathrm{JS}(1,t^2)}{\partial t}}-2\mathrm{JS}(1,t^2)\hfill \end{array}$$

(54)

$$\begin{array}{cc}\hfill & =-tln{t}^2+(t^2+1)ln\frac{t^2+1}{2}.\hfill \end{array}$$

(55)

For $t=1$, both the numerator and the denominator vanish: $N\left(1\right)=0$ and $D\left(1\right)=0$ in which $D\left(t\right)={(t-1)}^3$. Applying the rule de l’Hôpital three times, we get

$$\begin{array}{cc}\hfill N^{\prime }\left(t\right)& =ln\frac{t^2+1}{2}-ln{t}^2+\frac{2t-2}{t^2+1},\hfill \end{array}$$

(56)

$$\begin{array}{cc}\hfill N^{\prime \prime }\left(t\right)& =-\frac{2(t+1){(t-1)}^2}{{(t^2+1)}^{2}t},\hfill \end{array}$$

(57)

$$\begin{array}{cc}\hfill N^{\prime \prime \prime }\left(t\right)& =\frac{4t^7-6t^6-8t^5+6t^4-12t^3+14t^2+1}{{(t^5+2t^3+t)}^2},\hfill \end{array}$$

(58)

with $N^{\prime }\left(1\right)=N^{\prime \prime }\left(1\right)=N^{\prime \prime }\left(1\right)=0$. However, $D^{\prime }\left(1\right)=D^{\prime \prime }\left(1\right)=0$ and $D^{\prime \prime \prime }\left(1\right)=6$. Hence, $g^{\prime }\left(1\right)=0$ and it suffices to prove $g^{\prime \prime }\left(t\right)<0$ for $t>1$. Since the denominator of $g^{\prime }$ is positive, this is equivalent to $N^{\prime }\left(t\right)<0$ for $t>1$. But this follows from $N^{\prime }\left(1\right)=0$ and $N^{\prime \prime }\left(t\right)<0$ for $t>1$, which can be seen from (57). Hence f is monotonic and since the denominator in (51) satisfies the triangle inequality, Lemma 5 implies that $\sqrt{\mathrm{JS}}$ also satisfies the triangle inequality and therefore is a metric.

Having established the claim in one dimension, we get the n-dimensional result using the Minkowski inequality, which for non-negative real numbers $a_i$ and $b_i$ implies

$$\begin{array}{cc}\hfill \sqrt{{\sum }_{i=1}^n{(a_i+b_i)}^2}& \le \sqrt{{\sum }_{i=1}^n{a}_i^2}+\sqrt{{\sum }_{i=1}^n{b}_i^2}.\hfill \end{array}$$

(59)

Recall that the Jensen–Shannon divergence is decomposable: $\mathrm{JS}(x,y)={\sum }_{i=1}^n\mathrm{JS}(x_i,y_i)$ for $x,y\in {\mathbb{R}}_{+}^n$. Letting $z\in {\mathbb{R}}_{+}^n$ be a third point, we set $a_i^2=\mathrm{JS}(x_i,y_i)$ and $b_i^2=\mathrm{JS}(y_i,z_i)$ for all i. Since $\sqrt{\mathrm{JS}}$ satisfies the triangle inequality in one dimension, we have ${(a_i+b_i)}^2\ge \mathrm{JS}(x_i,z_i)$ for $1\le i\le n$. It follows that the left-hand side of (59) is larger than or equal to $\sqrt{\mathrm{JS}(x,z)}$. The right-hand side of (59) is equal to $\sqrt{\mathrm{JS}(x,y)}+\sqrt{\mathrm{JS}(y,z)}$, which implies the triangle inequality for the square root of the Jensen–Shannon divergence in n dimensions. □

Further preparations. Recall that the Shannon entropy is defined by $E\left(t\right)=tlnt-t$. The related function, $F:{\mathbb{R}}_{+}\to \mathbb{R}$, defined by

$$\begin{array}{cc}\hfill F\left(t\right)& =\sqrt{E\left(t^2\right)+1}\hfill \end{array}$$

(60)

$$\begin{array}{cc}\hfill & =\sqrt{2t^{2}lnt-t^2+1},\hfill \end{array}$$

(61)

will play a crucial role in the proof of our next theorem; see Figure 4. The derivative has a discontinuity at $t=1$, but if we reflect the preceding branch to get $\overline{F}:{\mathbb{R}}_{+}\to \mathbb{R}$ defined by $\overline{F}\left(t\right)=-F\left(t\right)$ for $0<t\le 1$ and $\overline{F}\left(t\right)=F\left(t\right)$ for $1\le t$, we obtain a convex function; see again Figure 4 on the left. Appendix A proves that $\overline{F}$ is convex and everywhere differentiable, and that its derivative, ${\overline{F}}^{\prime }$, is concave; see again Figure 4.

One dimension. We are now ready to prove that the square root of the minmax divergence is a metric in one dimension.

Theorem 3 

(1D Metric). Let $E:{\mathbb{R}}_{+}\to \mathbb{R}$ be the Shannon entropy and $\mathrm{Mx}:{\mathbb{R}}_{+}\times {\mathbb{R}}_{+}\to \mathbb{R}$ map every pair to its minmax divergence. Then $\sqrt{\mathrm{Mx}}$ is a metric in ${\mathbb{R}}_{+}$.

Proof. 

It is clear that $\sqrt{\mathrm{Mx}(x,y)}$ is non-negative, zero iff $x=y$, and symmetric. It thus remains to prove that it satisfies the triangle inequality. Setting

$$\begin{array}{cc}\hfill f(x,y)& ={\displaystyle \frac{\sqrt{\mathrm{Mx}(x,y)}}{|\sqrt{y}-\sqrt{x}|}},\hfill \end{array}$$

(62)

we will prove shortly that f is monotonic as defined in (47). Lemma 5 then implies that $\sqrt{\mathrm{Mx}}$ satisfies the triangle inequality. It thus remains to prove $f(a,b)\le f(x,y)$ whenever $a\le x\le y\le b$. We begin by noting that we may assume these intervals are canonical, by which we mean that $E\left(a\right)=E\left(b\right)$ and $E\left(x\right)=E\left(y\right)$. Indeed, if $E\left(x\right)\ne E\left(y\right)$, then we can find $C>0$ such that $E\left(Cx\right)=E\left(Cy\right)$, which then implies $f(Cx,Cy)=f(x,y)$.

We now use the function $f:{\mathbb{R}}_{+}\to \mathbb{R}$ defined by $F\left(t\right)=\sqrt{E\left(t^2\right)+1}$ to draw a geometric picture of the situation; see Figure 5.

To prove monotonicity, that is: $f(a,b)\le f(x,y)$, we consider the linear functions $L,R:\mathbb{R}\to \mathbb{R}$ that satisfy $L\left(t\right)=F\left(t\right)$, for $t=\sqrt{a},\sqrt{x}$, and $R\left(t\right)=F\left(t\right)$, for $t=\sqrt{y},\sqrt{b}$. The goal is to show that the aspect ratio of the rectangle defined by $a,b$ is less than that defined by $x,y$. Equivalently, we show that the point at which the lines L and R meet has positive second coordinate. By the differentiability of F and the transitivity of order along the real line, it suffices to show this in the limit case, when $a=x$ and $y=b$. In this case, L and R are the tangent lines of F at $t=\sqrt{x}$ and $t=\sqrt{y}$, as shown in Figure 5. Let $\mathsf{\ell }=L\left(1\right)$ and $r=R\left(1\right)$ be the coordinates of the points at which L and R intersect the vertical line $t=1$, and note that $r\le 0\le \mathsf{\ell }$ by convexity of $\overline{F}$. The convexity of this function furthermore implies

$$\begin{array}{cc}\hfill 1-\sqrt{x}& >\sqrt{y}-1,\hfill \end{array}$$

(63)

$$\begin{array}{cc}\hfill |L^{\prime }\left(\sqrt{x}\right)|& <|R^{\prime }\left(\sqrt{y}\right)|.\hfill \end{array}$$

(64)

To compare the absolute values of ℓ and r, we express both as integrals:

$$\begin{array}{cc}\hfill \mathsf{\ell }& ={\int }_{t=\sqrt{x}}^1[F^{\prime }\left(\sqrt{x}\right)-F^{\prime }\left(t\right)]\mathrm{d}t,\hfill \end{array}$$

(65)

$$\begin{array}{cc}\hfill r& ={\int }_{t=1}^{\sqrt{y}}[F^{\prime }\left(t\right)-F^{\prime }\left(\sqrt{y}\right)]\mathrm{d}t;\hfill \end{array}$$

(66)

see Figure 4 on the right. The concavity of ${\overline{F}}^{\prime }$ together with (63) implies $|\mathsf{\ell }|\ge \left|r\right|$, and since L has smaller absolute slope than R (64), it follows that the two lines intersect above the zero line, as required. □

This concludes the proof in dimension one. The main obstacle to extending the proof to arbitrary dimension is the lack of decomposability (separability) of the minmax loss. Still, we decided to present the partial results and techniques, as they may help other researchers complete the proof in the future. In particular, techniques for extending results from one to arbitrary dimensions are present in the information theory literature; see for example the work on Pinsker’s inequality [23], which compares the relative entropy with another distance. This gives us hope that researchers in information theory may be well-equipped to extend the result to arbitrary dimension.

5. Extensions to Other Bregman Divergences

In this section, we provide a perspective for our results by extending them beyond the Shannon entropy. Specifically, we weaken the bounds while keeping them tight to generalize Theorem 1 to Bregman divergences, and we prove that the square root of the minmax divergence for the Burg entropy is a metric in ${\mathbb{R}}_{+}$.

This way the inequality can be used in other applied contexts. For example, the Burg entropy (and the Itakura–Saito divergence it induces) are used in speech recognition [24].

Burbea–Rao divergence. The Burbea–Rao divergence, also called the Jensen–Bregman [25], is a straightforward generalization of the Jensen–Shannon divergence. Specifically, the underlying divergence is generalized from the relative entropy to a Bregman divergence. Letting $F:\Omega \to \mathbb{R}$ be a function of Legendre type generating a Bregman divergence and $x,y\in \Omega$, we define

$$\begin{array}{cc}\hfill {\mathrm{BR}}_F\left(x,y\right)& =\frac{1}{2}[D_F(x\parallel \mu )+D_F(y\parallel \mu )]\hfill \end{array}$$

(67)

$$\begin{array}{cc}\hfill & =\frac{1}{2}[F\left(x\right)+F\left(y\right)-2F\left(\mu \right)],\hfill \end{array}$$

(68)

in which $\mu =(x+y)/2$. Lemma 1 generalizes with a verbatim proof in which we substitute F for E. As done in [1], we also generalize the minmax divergence from the Kullback–Leibler divergence to a general underlying Bregman divergence $D_F$:

$$\begin{array}{cc}\hfill {\mathrm{Mx}}_F(x,y)& =\underset{z\in \Omega }{min}max\{D_F(x\parallel z),D_F(y\parallel z)\}.\hfill \end{array}$$

(69)

When we compare the two, we get bounds that are considerably worse than for the special case of the Shannon entropy.

Theorem 4 

(Burbea–Rao divergence Comparison). Let $F:\Omega \to \mathbb{R}$ be a Legendre type function, and let $x,y$ be points in $\Omega$. Then $\frac{1}{2}{\mathrm{Mx}}_F(x,y)\le {\mathrm{BR}}_F\left(x,y\right)\le {\mathrm{Mx}}_F(x,y)$.

Proof. 

Let $z\in \Omega$ be the point that minimizes the right-hand side of (69). Using the generalization of Lemma 1 to the Burbea–Rao divergence, we get

$$\begin{array}{cc}\hfill {\mathrm{BR}}_F\left(x,y\right)& \le \frac{1}{2}\left[D_F(x\parallel z)+D_F(y\parallel z)\right].\hfill \end{array}$$

(70)

Since $D_F(x\parallel z)=D_F(y\parallel z)$, the right-hand side of (70) is equal to ${\mathrm{Mx}}_F(x,y)$, which implies the claimed upper bound on the Burbea–Rao divergence. To prove the lower bound, we note that the larger of the two divergences to $\mu$ is at least as large as ${\mathrm{Mx}}_F(x,y)$. Hence, ${\mathrm{Mx}}_F(x,y)$ is at most the sum:

$$\begin{array}{cc}\hfill {\mathrm{Mx}}_F(x,y)& \le D_F(x\parallel \mu )+D_F(y\parallel \mu ),\hfill \end{array}$$

(71)

and the claimed lower bound follows because the right-hand side of (71) evaluates to $2{\mathrm{BR}}_F\left(x,y\right)$. □

Remark 5. 

The bounds in Theorem 4 are tight. To see this for the lower bound, we consider $F\left(t\right)=t+\frac{1}{t}$, which is strictly convex, differentiable, and with minimum at $t=1$. Setting $x=\epsilon$, $y=\frac{1}{\epsilon }$ for $0<\epsilon <1$, we have $F\left(x\right)=F\left(y\right)=\epsilon +\frac{1}{\epsilon }$, which implies that $t=1$ minimizes the maximum divergence from x and y. Some computations show that the divergences are

$$\begin{array}{cc}\hfill {\mathrm{Mx}}_F(x,y)& =F\left(x\right)-F\left(1\right)=\epsilon -2+\frac{1}{\epsilon },\hfill \end{array}$$

(72)

$$\begin{array}{cc}\hfill {\mathrm{BR}}_F\left(x,y\right)& =\frac{F\left(x\right)+F\left(y\right)}{2}-F\left(\mu \right)=\frac{{\epsilon }^2+1}{2\epsilon }-\frac{2\epsilon }{{\epsilon }^2+1}.\hfill \end{array}$$

(73)

For $\epsilon \to 0$, the ratio of ${\mathrm{BR}}_F\left(x,y\right)$ over ${\mathrm{Mx}}_F(x,y)$ goes to $\frac{1}{2}$, as required. To see the upper bound, we choose $F\left(t\right)=t^2$ for which ${\mathrm{BR}}_F\left(x,y\right)={\mathrm{Mx}}_F(x,y)$ for all $x,y\in \mathbb{R}$.

Minmax divergence for Burg entropy. Another significant entropy appearing in this context is the Burg entropy: $B\left(x\right)=x-ln\left(x\right)+1$, for $x>0$. Let us denote the corresponding minmax divergence my $M_B$.

Theorem 5 

(Metric for Burg Entropy). Let $F:{\mathbb{R}}_{+}\to \mathbb{R}$ be the Burg entropy. Then $\sqrt{M_B}$ is a metric in ${\mathbb{R}}_{+}$.

Proof. 

The proof follows a similar structure as the proof of Theorem 3. Setting up the ratio method, we define

$$\begin{array}{cc}\hfill f(x,y)& ={\displaystyle \frac{\sqrt{M_B(x,y)}}{|lnx-lny|}},\hfill \end{array}$$

(74)

with the denominator being the induced path metric. By Lemma 5, we have to prove that f is monotonic.

Next we simplify. Specifically, because $D_B(x\parallel y)=D_B(kx\parallel ky)$, we have $f(kx,ky)=f(x,y)$ for all $k>0$. We may therefore assume all intervals considered by the Ratio Method to be canonical, i.e., $B\left(x\right)=B\left(y\right)$. In such case, we have $M_B(x,y)=B\left(x\right)=B\left(y\right)$.

To prepare the geometric picture sketched in Figure 6, we define $H\left(x\right)=\sqrt{e^x-x-1}$ for $x\in \mathbb{R}$. Note that $f(x,y)$ is the aspect ratio of the rectangle in this picture. As in the proof of Theorem 3, the monotonicity of f is established by proving that the tangent lines to H at $lnx$ and $lny$ intersect above the horizontal axis for all canonical intervals $[x,y]$. To this end, it suffices to show that $\lambda >\rho$, in which

• $\lambda$ is the coordinate of the intersection of the horizontal axis with the tangent on H at $lnx$;
• $\rho$ is the coordinate of the intersection of the horizontal axis with the tangent on H at $lny$.

Reflecting the left part of H, we obtain an injective function $\overline{H}:\mathbb{R}\to \mathbb{R}$ defined by

$$\begin{array}{cc}\hfill \overline{H}\left(x\right)& =\left\{\begin{array}{cc}\hfill -H\left(x\right)& \mathrm{for}x<0,\hfill \\ \hfill H\left(x\right)& \mathrm{for}0\le x.\hfill \end{array}\right.\hfill \end{array}$$

(75)

Note that this does not change the intersections mentioned above. Letting K be the inverse of $\overline{H}$, We consider the graph of K; see Figure 7.

Expressing this with integrals, and setting $t=H(lnx)=H(lny)$, we can compare the two coordinates:

$$\begin{array}{cc}\hfill \lambda & =K(-t)+t{K}^{\prime }(-t)=\underset{0}{\overset{-t}{\int }}[K^{\prime }\left(s\right)-K^{\prime }(-t)]\mathrm{d}s,\hfill \end{array}$$

(76)

$$\begin{array}{cc}\hfill \rho & =K\left(t\right)-t{K}^{\prime }\left(t\right)=\underset{0}{\overset{t}{\int }}[K^{\prime }\left(s\right)-K^{\prime }\left(t\right)]\mathrm{d}s.\hfill \end{array}$$

(77)

By Lemmas A3 and A4, $K^{\prime }$ is convex and decreasing. Using the integral representation above, we interpret $\lambda$ and $\rho$ as areas; see Figure 3. It follows that $\lambda >\rho$ and consequently $\sqrt{M_B}$ is a metric. □

6. Discussion

The main result of our work are the tight bounds between the minmax information divergence and the Jensen–Shannon divergence (as well as analogous results for their generalizations). The former arises naturally in the context of computational geometry and topology. Specifically, it coincides with the smallest radius of the nonempty intersection between two balls in the geometry induced by the relative entropy (which generalizes to other Bregman divergences). In this setting, this quantity is calculated repeatedly, for example to compute the one-skeleton of the Čech complex, which is one of the standard constructions in topological data analysis. In this case, our bounds are best presented as

$$\begin{array}{c}\hfill \sqrt{\mathrm{JS}(x,y)}\le \sqrt{\mathrm{Mx}(x,y)}\le \sqrt{\frac{2}{eln2}\mathrm{JS}(x,y)}\approx 1.030\sqrt{\mathrm{JS}(x,y)}.\end{array}$$

(78)

One can therefore closely approximate the relatively costly computations of the minmax divergence, with the straightforward and efficient computations of the Jensen–Shannon divergence. Indeed, computing the minmax information requires performing a binary search or another numerical algorithm. On the other hand the formula for the Jensen–Shannon divergences is not more complex then computing the Euclidean distance.

The tightness of the bounds makes it believable that the square root of the minmax divergence may be a metric, similar to the Jensen–Shannon divergence. This turned out to be true in the one dimensional case, but the general case of n-dimensional spaces appears to be significantly more difficult. One key reason is that—unlike the Jensen–Shannon divergence—the minmax divergence is not decomposable (separable). We leave this case open and hope that a combinations of the proposed geometric proof techniques with additional techniques may eventually lead to a successful resolution.