A Class of New Metrics Based on Triangular Discrimination

Abstract

In the field of information theory, statistics and other application areas, the information-theoretic divergences are used widely. To meet the requirement of metric properties, we introduce a class of new metrics based on triangular discrimination which are bounded. Moreover, we obtain some sharp inequalities for the triangular discrimination and other information-theoretic divergences. Their asymptotic approximation properties are also involved.

1. Introduction

In many applications such as pattern recognition, machine learning, statistics, optimization and other applied branches of mathematics, it is beneficial to use the information-theoretic divergences rather than the squared Euclidean distance to estimate the (dis)similarity of two probability distributions or positive arrays [1,2,3,4,5,6,7,8,9]. Among them the Kullback–Leibler divergence (relative entropy), triangular discrimination, variation distance, Hellinger distance, Jensen–Shannon divergence, symmetric Chi-square divergence, J-divergence and other important measures often play a critical role. Unfortunately, most of these divergences do not satisfy the metric properties and unboundedness [10]. As we know, metric properties are the preconditions for numerous convergence properties of iterative algorithms [11]. Moreover, boundedness is also highly concerned in numerical computations and simulations. In paper [12], Endres and Schindelin have proved that the square root of twice Jensen–Shannon divergence is a metric. Triangular discrimination presented by Topsøe in [13] is a non-logarithmic measure and is simple in complex computation. Inspired by [12], we discuss the triangular discrimination. In this paper, the main result is that a class of new metrics derived from the triangular discrimination are introduced. Finally, some new relationships among triangular discrimination, Jensen–Shannon divergence, square of Hellinger distance, variation distance are also obtained.

2. Definition and Auxiliary Results

Definition 1. Let

$${\Gamma }_n=\left\{P=(p_1,p_2,\cdots ,p_n)|p_i\ge 0,\sum _{i=1}^n{p}_i=1\right\},n\ge 2$$

be the set of all complete finite discrete probability distributions. For all $P,Q\in {\Gamma }_n$, the triangular discrimination is defined by

$$\Delta (P,Q)=\sum _{i=1}^n\frac{{(p_i-q_i)}^2}{p_i+q_i}.$$

(1)

In the above definition, we use convention based on limitation property that $\frac{0}{0}=0$.

The triangular discrimination is obviously symmetric, nonnegative and vanishes for $P=Q$, but it does not fulfill the triangle inequality. In the view of the foregoing, the concept of triangular discrimination should be generalized. If $P,Q\in {\Gamma }_n$, the function ${\Delta }_{\alpha }(P,Q)$ is studied:

$${\Delta }_{\alpha }(P,Q)={\left(\sum _{i=1}^n\frac{{(p_i-q_i)}^2}{p_i+q_i}\right)}^{\alpha },$$

(2)

where $\alpha \in (0,+\infty )$.

In the following, the $\alpha -$power of the summand in $\Delta (P,Q)$ with all $\alpha \in (0,+\infty )$ are discussed.

Definition 2. Let the function $L(p,q):[0,+\infty )\times [0,+\infty )\to [0,+\infty )$ be defined by

$$L(p,q)=\frac{{(p-q)}^2}{p+q}.$$

(3)

It is easy to see that $L(p,q)\ge 0$ and $L(p,q)=L(q,p)$. To all $\alpha \in (0,+\infty )$, the issue of whether ${\left(L(p,q)\right)}^{\alpha }$ satisfies the triangle inequality is considered in the following.

Lemma 1. If the function $g:[0,a)\cup (a,+\infty )\to (-\infty ,+\infty )$ is defined by

$$g\left(x\right)=\frac{\frac{(x-a)(x+3a)}{{(x+a)}^2}}{\sqrt{L(a,x)}}$$

with $a>0$, then

$$\underset{x\to a^{+}}{lim}g\left(x\right)=\sqrt{\frac{2}{a}},\underset{x\to a^{-}}{lim}g\left(x\right)=-\sqrt{\frac{2}{a}}.$$

Proof. As

$$g\left(x\right)=\left\{\begin{array}{c}\frac{x+3a}{{(x+a)}^{\frac{3}{2}}},x>a\hfill \\ -\frac{x+3a}{{(x+a)}^{\frac{3}{2}}},0\le x<a\hfill \end{array}\right.$$

we can get

$$\underset{x\to a^{+}}{lim}g\left(x\right)=\frac{a+3a}{{(a+a)}^{\frac{3}{2}}}=\sqrt{\frac{2}{a}},\underset{x\to a^{-}}{lim}g\left(x\right)=-\frac{a+3a}{{(a+a)}^{\frac{3}{2}}}=-\sqrt{\frac{2}{a}}.$$

Lemma 2. If the function $h:[0,+\infty )\to (0,+\infty )$ is defined by $h\left(x\right)=\frac{3x+a}{{(x+a)}^{\frac{3}{2}}}$ with $a>0$, then h is monotonic increasing in $[0,a)$ and monotonic decreasing in $(a,+\infty )$.

Proof. Straightforward derivative shows

$$h^{\prime }\left(x\right)=\frac{3(a-x)}{2{(x+a)}^{\frac{5}{2}}},$$

$h^{\prime }\left(x\right)>0$ in $[0,a)$ and $h^{\prime }\left(x\right)<0$ in $(a,+\infty )$. Thus the lemma holds.

Assuming $0<p<q$, we introduce function $R_{pq}:[0,+\infty )\to [0,+\infty )$ defined by

$$R_{pq}\left(r\right)=\sqrt{L(p,r)}+\sqrt{L(q,r)}.$$

Lemma 3. The function $R_{pq}\left(r\right)$ has two minima, one at $r=p$ and the other at $r=q$.

Proof. The derivative of the function $R_{pq}\left(r\right)$ is

$${R_{pq}}^{\prime }\left(r\right)=\frac{1}{2}\left(\frac{\frac{(r-p)(r+3p)}{{(r+p)}^2}}{\sqrt{L(p,r)}}+\frac{\frac{(r-q)(r+3q)}{{(r+q)}^2}}{\sqrt{L(q,r)}}\right).$$

(4)

So ${R_{pq}}^{\prime }\left(r\right)<0$ for $r\in [0,p)$ and ${R_{pq}}^{\prime }\left(r\right)>0$ for $r\in (q,+\infty )$. It shows $R_{pq}\left(r\right)$ is monotonic decreasing in $[0,p)$ and monotonic increasing in $[q,+\infty )$.

Next consider the monotonicity of $R_{pq}\left(r\right)$ in the open interval $(p,q)$.

From Lemma 3, we have

$$\begin{array}{cc}& \underset{r\to p^{+}}{lim}\frac{\frac{(r-p)(r+3p)}{{(r+p)}^2}}{\sqrt{L(p,r)}}=\sqrt{\frac{2}{p}},\hfill \\ & \underset{r\to q^{-}}{lim}\frac{\frac{(r-q)(r+3q)}{{(r+q)}^2}}{\sqrt{L(q,r)}}=-\sqrt{\frac{2}{q}}.\hfill \end{array}$$

(5)

From Lemma 2, we have

$$\begin{array}{cc}& \frac{\frac{(p-q)(p+3q)}{{(p+q)}^2}}{\sqrt{L(p,q)}}=-\frac{p+3q}{{(p+q)}^{\frac{3}{2}}}>-\frac{p+3p}{{(p+p)}^{\frac{3}{2}}}=-\sqrt{\frac{2}{p}},\hfill \\ & \frac{\frac{(q-p)(q+3p)}{{(p+q)}^2}}{\sqrt{L(p,q)}}=\frac{q+3p}{{(p+q)}^{\frac{3}{2}}}<\frac{q+3q}{{(q+q)}^{\frac{3}{2}}}=\sqrt{\frac{2}{q}}.\hfill \end{array}$$

(6)

Using (5) and (6),

$$\begin{array}{cc}& \underset{r\to p^{+}}{lim}{R_{pq}}^{\prime }\left(r\right)=\frac{1}{2}\left(\underset{r\to p^{+}}{lim}\frac{\frac{(r-p)(r+3p)}{{(r+p)}^2}}{\sqrt{L(p,r)}}+\frac{\frac{(p-q)(p+3q)}{{(p+q)}^2}}{\sqrt{L(p,q)}}\right)=\frac{1}{2}\left(\sqrt{\frac{2}{p}}-\frac{p+3q}{{(p+q)}^{\frac{3}{2}}}\right)>0,\hfill \\ & \underset{r\to q^{-}}{lim}{R_{pq}}^{\prime }\left(r\right)=\frac{1}{2}\left(\frac{\frac{(q-p)(q+3p)}{{(q+p)}^2}}{\sqrt{L(p,q)}}+\underset{r\to q^{-}}{lim}\frac{\frac{(r-q)(r+3q)}{{(r+q)}^2}}{\sqrt{L(r,q)}}\right)=\frac{1}{2}\left(\frac{q+3p}{{(p+q)}^{\frac{3}{2}}}-\sqrt{\frac{2}{q}}\right)<0.\hfill \end{array}$$

Let

$$A(y,r)=\frac{\frac{(r-y)(r+3y)}{{(r+y)}^2}}{\sqrt{L(y,r)}}=\frac{\frac{(r-y)(r+3y)}{{(r+y)}^2}}{\sqrt{r}\sqrt{L(\frac{y}{r},1)}}=\frac{1}{\sqrt{r}}B(y,r),y>0,$$

then

$$\frac{\partial B(y,r)}{\partial r}=-\frac{3y\sqrt{\frac{{(r-y)}^2}{r+y}}}{2\sqrt{r}{(r+y)}^2}\le 0.$$

The equality holds if and only if $r=y$. So with respect to variable r in the open interval $(p,q)$, $B(p,r)$ and $B(q,r)$ are both monotonic decreasing, $B(p,r)+B(q,r)$ is also monotonic decreasing. Using (4),

$${R_{pq}}^{\prime }\left(r\right)=\frac{1}{2}\left(A(p,r)+A(q,r)\right)=\frac{1}{2\sqrt{r}}\left(B(p,r)+B(q,r)\right),$$

this shows $\underset{r\to p^{+}}{lim}B(p,r)+B(q,r)>0,\underset{r\to q^{-}}{lim}B(p,r)+B(q,r)<0$. So we can see $B(p,r)+B(q,r)$ has only one zero point in the open interval $(p,q)$ with respect to variable r. As a consequence, ${R_{pq}}^{\prime }\left(r\right)$ has only one zero point $x_0$ in the open interval $(p,q)$ with respect to variable r. This means ${R_{pq}}^{\prime }\left(r\right)>0$ in the interval $(p,x_0)$, ${R_{pq}}^{\prime }\left(r\right)<0$ in the interval $(x_0,q)$. From the above we know ${R_{pq}}^{\prime }\left(r\right)$ has only one maximum and no minimum in the open interval $(p,q)$.

As a result, the conclusion in the lemma is obtained.   ☐

Theorem 1. Let $p,q,r\in [0,+\infty )$, then

$${\left(L(p,q)\right)}^{\frac{1}{2}}\le {\left(L(p,r)\right)}^{\frac{1}{2}}+{\left(L(q,r)\right)}^{\frac{1}{2}}.$$

(7)

Proof. If $p=q$, then $L(p,q)=0$. The triangle inequality (7) obviously holds.

If $p\ne q$ and one of $p,q$ is equal to 0, it is easy to obtain that (7) holds.

Next we assume $0<p<q$ without loss of generality. Note that the formula is valid:

$${\left(L(p,q)\right)}^{\frac{1}{2}}=\underset{r\to p}{lim}\left({\left(L(p,r)\right)}^{\frac{1}{2}}+{\left(L(q,r)\right)}^{\frac{1}{2}}\right)=\underset{r\to q}{lim}\left({\left(L(p,r)\right)}^{\frac{1}{2}}+{\left(L(q,r)\right)}^{\frac{1}{2}}\right).$$

From Lemma 3 the triangle inequality (7) can be easily proved for any number $r\in [0,+\infty )$.   ☐

Corollary 1. Let $p,q,r\in [0,+\infty )$. If $0<\alpha <\frac{1}{2}$, then

$${\left(L(p,q)\right)}^{\alpha }\le {\left(L(p,r)\right)}^{\alpha }+{\left(L(q,r)\right)}^{\alpha }.$$

(8)

Proof. Let $a,b>0$ and $0<\gamma <1$, then $a^{\gamma }+b^{\gamma }>{(a+b)}^{\gamma }$ which follows from the concavity of $x^{\gamma }$. Now a γ which satisfies $\alpha =\frac{1}{2}\gamma$ can be found. Thus from Theorem 1,

$$\begin{array}{cc}\hfill {\left(L(p,r)\right)}^{\alpha }+{\left(L(q,r)\right)}^{\alpha }=& {\left(L(p,r)\right)}^{\frac{1}{2}\gamma }+{\left(L(q,r)\right)}^{\frac{1}{2}\gamma }\hfill \\ \hfill \ge & {\left({\left(L(p,r)\right)}^{\frac{1}{2}}+{\left(L(q,r)\right)}^{\frac{1}{2}}\right)}^{\gamma }\ge {\left(L(p,q)\right)}^{\frac{1}{2}\gamma }={\left(L(p,q)\right)}^{\alpha }.\hfill \end{array}$$

This is the triangle inequality (8) for the function ${\left(L(p,q)\right)}^{\alpha }$.   ☐

Theorem 2. Let $p,q,r\in [0,+\infty )$. If $\alpha >\frac{1}{2}$, then the triangle inequality (8) does not hold.

Proof. Assuming $0<p<q$, let $l\left(r\right)={\left(L(p,r)\right)}^{\alpha }+{\left(L(q,r)\right)}^{\alpha }$. Firstly the formula is valid:

$${\left(L(p,q)\right)}^{\alpha }=\underset{r\to p}{lim}\left({\left(L(p,r)\right)}^{\alpha }+{\left(L(q,r)\right)}^{\alpha }\right)=\underset{r\to q}{lim}\left({\left(L(p,r)\right)}^{\alpha }+{\left(L(q,r)\right)}^{\alpha }\right).$$

The derivative of the function l is

$$l^{\prime }\left(r\right)=\alpha \left(\frac{(r-p)(3p+r)}{{(p+r)}^2}{\left(L(p,r)\right)}^{\alpha -1}+\frac{(r-q)(3q+r)}{{(q+r)}^2}{\left(L(q,r)\right)}^{\alpha -1}\right).$$

When $r\in (p,q)$, let

$$m\left(r\right)={\left(\frac{(r-p)(3p+r)}{{(p+r)}^2}{\left(L(p,r)\right)}^{\alpha -1}\right)}^{\frac{1}{1-\alpha }}.$$

Using l’Hôspital’s rule,

$$\underset{r\to p^{+}}{lim}m\left(r\right)=\frac{8p^2}{(1-\alpha ){(p+r)}^3}{\left(\frac{(r-p)(3p+r)}{{(p+r)}^2}\right)}^{\frac{2\alpha -1}{1-\alpha }}=0.$$

So

$$\underset{r\to p^{+}}{lim}{l}^{\prime }\left(r\right)=\frac{(p-q)(3q+p)}{{(q+p)}^2}{\left(L(p,q)\right)}^{\alpha -1}<0.$$

According to the definition of derivative, there exists a $\delta >0$ such that for any $s\in (p,p+\delta )$,

$${\left(L(p,q)\right)}^{\alpha }=\underset{r\to p^{+}}{lim}\left({\left(L(p,r)\right)}^{\alpha }+{\left(L(q,r)\right)}^{\alpha }\right)>{\left(L(p,s)\right)}^{\alpha }+{\left(L(q,s)\right)}^{\alpha }.$$

This shows the triangle inequality (8) does not hold.   ☐

To sum up the theorems and corollary above, we can obtain the main theorem:

Theorem 3. The function ${\left(L(p,q)\right)}^{\alpha }$ satisfies the triangle inequality (8) if and only if $0<\alpha \le \frac{1}{2}$.

3. Metric Properties of ${\Delta }_{\alpha }(P,Q)$

In this section, we mainly prove the following theorem:

Theorem 4. The function ${\Delta }_{\alpha }(P,Q)$ is a metric on the space ${\Gamma }_n$ if and only if $0<\alpha \le \frac{1}{2}$.

Proof. From (2) we can get ${\Delta }_{\alpha }(P,Q)={\left({\displaystyle \sum _{i=1}^n}L(p_i,q_i)\right)}^{\alpha }$. It is easy to see that ${\Delta }_{\alpha }(P,Q)\ge 0$ with equality only for $P=Q$ and ${\Delta }_{\alpha }(P,Q)={\Delta }_{\alpha }(Q,P)$. So what we concern is whether the triangle inequality

$${\Delta }_{\alpha }(P,Q)\le {\Delta }_{\alpha }(P,R)+{\Delta }_{\alpha }(Q,R)$$

(9)

holds for any $P,Q,R\in {\Gamma }_n$.

☐

When $P=Q$, ${\Delta }_{\alpha }(P,Q)=0$, the triangle inequality (9) holds apparently. So we assume $P\ne Q$ in the following.

Next we consider the value of α in two cases respectively:

(i) $0<\alpha \le \frac{1}{2}$:

From Theorem 3, the inequality ${\left(L(p_i,q_i)\right)}^{\alpha }\le {\left(L(p_i,r_i)\right)}^{\alpha }+{\left(L(q_i,r_i)\right)}^{\alpha }$ holds. Applying Minkowski’s inequality we have

$$\begin{array}{cc}\hfill {\left(\sum _{i=1}^{n}L\left(p_i,q_i\right)\right)}^{\alpha }& ={\left\{\sum _{i=1}^n{\left({\left(L(p_i,q_i)\right)}^{\alpha }\right)}^{\frac{1}{\alpha }}\right\}}^{\alpha }\hfill \\ & \le {\left\{\sum _{i=1}^n{\left({\left(L(p_i,r_i)\right)}^{\alpha }+{\left(L(q_i,r_i)\right)}^{\alpha }\right)}^{\frac{1}{\alpha }}\right\}}^{\alpha }\hfill \\ & \le {\left\{\sum _{i=1}^n{\left({\left(L(p_i,r_i)\right)}^{\alpha }\right)}^{\frac{1}{\alpha }}\right\}}^{\alpha }+{\left\{\sum _{i=1}^n{\left({\left(L(q_i,r_i)\right)}^{\alpha }\right)}^{\frac{1}{\alpha }}\right\}}^{\alpha }\hfill \\ & ={\left(\sum _{i=1}^{n}L\left(p_i,r_i\right)\right)}^{\alpha }+{\left(\sum _{i=1}^{n}L\left(q_i,r_i\right)\right)}^{\alpha }.\hfill \end{array}$$

So the triangle inequality (9) holds.

(ii) $\alpha >\frac{1}{2}$:

Let

$$F(x_1,\cdots ,x_n)=F_1(x_1,\cdots ,x_n)+F_2(x_1,\cdots ,x_n),$$

where

$$\begin{array}{cc}\hfill F_1(x_1,\cdots ,x_n)& ={\left(\sum _{i=1}^n\frac{{(p_i-x_i)}^2}{p_i+x_i}\right)}^{\alpha },\hfill \\ \hfill F_2(x_1,\cdots ,x_n)& ={\left(\sum _{i=1}^n\frac{{(q_i-x_i)}^2}{q_i+x_i}\right)}^{\alpha }.\hfill \end{array}$$

Then $F(p_1,\cdots ,p_n)=F(q_1,\cdots ,q_n)={\Delta }_{\alpha }(P,Q)$.

Next we prove $(p_1,\cdots ,p_n)$ and $(q_1,\cdots ,q_n)$ are not the extreme points of the function $F(x_1,\cdots ,x_n)$. By the symmetry we only need to prove $(p_1,\cdots ,p_n)$ is not the extreme point.

By partial derivative,

$$\begin{array}{c}\hfill \frac{\partial F}{\partial x_i}{|}_{(x_1,\cdots ,x_n)=(p_1,\cdots ,p_n)}=\frac{\partial F_1}{\partial x_i}{|}_{(x_1,\cdots ,x_n)=(p_1,\cdots ,p_n)}+\frac{\partial F_2}{\partial x_i}{|}_{(x_1,\cdots ,x_n)=(p_1,\cdots ,p_n)}.\end{array}$$

(10)

Since $P\ne Q$, we might as well assume $p_1\ne q_1$ and $p_1>0$.

$$\begin{array}{cc}\hfill \frac{\partial F_2}{\partial x_1}{|}_{(x_1,\cdots ,x_n)=(p_1,\cdots ,p_n)}=& \frac{\alpha (p_1-q_1)(p_1+3q_1)}{{(p_1+q_1)}^2}·{\left(\sum _{i=1}^n\frac{{(p_i-q_i)}^2}{p_i+q_i}\right)}^{\alpha -1}\hfill \\ \hfill \ne & 0.\hfill \end{array}$$

(11)

$$\begin{array}{cc}\hfill \frac{\partial F_1}{\partial x_1}{|}_{(x_1,\cdots ,x_n)=(p_1,\cdots ,p_n)}=& \underset{\Delta x_1\to 0}{lim}\frac{1}{\Delta x_1}\left(F_1(p_1+\Delta x_1,\cdots ,p_n)-F_1(p_1,\cdots ,p_n)\right)\hfill \\ \hfill =& \underset{\Delta x_1\to 0}{lim}\frac{1}{\Delta x_1}{\left(\frac{\Delta x_1^2}{2p_1+x_1}\right)}^{\alpha }\hfill \\ \hfill =& \underset{\Delta x_1\to 0}{lim}\frac{\Delta x_1^{2\alpha -1}}{{(2p_1+x_1)}^{\alpha }}\hfill \\ \hfill =& 0.\hfill \end{array}$$

(12)

Then taking (11) and (12) into (10), we have

$$\frac{\partial F}{\partial x_1}{|}_{(x_1,\cdots ,x_n)=(p_1,\cdots ,p_n)}\ne 0.$$

Therefore, $(p_1,\cdots ,p_n)$ is not the extreme point of the function $F(x_1,\cdots ,x_n)$. For the same reason, $(q_1,\cdots ,q_n)$ is also not the extreme point.

Using the definition of extreme point, there exists a point $R=(r_1,\cdots ,r_n)$ such that $F(r_1,\cdots ,r_n)<F(p_1,\cdots ,p_n)={\Delta }_{\alpha }(P,Q)$. As $F_1(r_1,\cdots ,r_n)={\Delta }_{\alpha }(P,R),F_2(r_1,\cdots ,r_n)={\Delta }_{\alpha }(Q,R)$, then ${\Delta }_{\alpha }(P,R)+{\Delta }_{\alpha }(Q,R)<{\Delta }_{\alpha }(P,Q)$. The inequality is not consistent with the triangle inequality (9).

From what has been discussed above, the conclusion in the theorem is obtained.   ☐s

The generalization of this result to continuous probability distributions is straightforward. Consider a measurable space $(\mathcal{X},\mathcal{A})$, and P, Q are probability distributions with Radon-Nykodym densities $p=\frac{\mathrm{d}P}{\mathrm{d}\mu }$, $q=\frac{\mathrm{d}Q}{\mathrm{d}\mu }$ w.r.t. a dominating σ-finite measure μ. Then

$${\Delta }_{\alpha }(P,Q)={\left({\int }_{\mathcal{X}}\frac{{(p-q)}^2}{p+q}\mathrm{d}\mu \right)}^{\alpha }$$

(13)

is a metric if and only if $0<\alpha \le \frac{1}{2}$.

Next we will discuss the maxima and minima of ${\Delta }_{\alpha }(P,Q)$. It is obvious that ${\Delta }_{\alpha }(P,Q)=0$ is the minima, if and only if $P=Q$. Because $\Delta (P,Q)$ can rewrite in the form

$$\begin{array}{cc}\hfill \Delta (P,Q)& =\sum _{i=1}^n\frac{{(p_i-q_i)}^2}{p_i+q_i}\hfill \\ & =\sum _{i=1}^n\left(p_i+q_i-\frac{4p_i{q}_i}{p_i+q_i}\right)\hfill \\ & =2-\sum _{i=1}^n\frac{4p_i{q}_i}{p_i+q_i}\le 2.\hfill \end{array}$$

(14)

$\Delta (P,Q)$ obtains the maxima 2 when $P,Q$ are two distinct deterministic distributions, namely $p_i{q}_i=0$. Then the metric ${\Delta }_{\alpha }(P,Q)$ achieves its maximum value $2^{\alpha }$.

4. Some Inequalities among the Information-Theoretic Divergences

Definition 3. For all $P,Q\in {\Gamma }_n$, the Jensen–Shannon divergence is defined by

$$JS(P,Q)=\frac{1}{2}\sum _{i=1}^n\left[p_{i}ln\left(\frac{2p_i}{p_i+q_i}\right)+q_{i}ln\left(\frac{2q_i}{p_i+q_i}\right)\right].$$

The square of the Hellinger distance is defined by

$$H^2(P,Q)=\frac{1}{2}\sum _{i=1}^n{(\sqrt{p_i}-\sqrt{q_i})}^2.$$

The variance distance is defined by

$$V(P,Q)=\sum _{i=1}^n|p_i-q_i|.$$

Next we introduce the Csiszár’s f-divergence [14].

Definition 4. Let $f:[0,+\infty )\to (-\infty ,+\infty )$ be a convex function satisfying $f\left(1\right)=0$, the f-divergence measure introduced by Csiszár is defined as

$$C_f(P,Q)=\sum _{i=1}^n{q}_{i}f\left(\frac{p_i}{q_i}\right)$$

(15)

for all $P,Q\in {\Gamma }_n.$

The triangular discrimination, Jensen–Shannon divergence, the square of the Hellinger distance, variance distance are all f-divergence.

Example 1. (Triangular Discrimination) Let us consider

$$f_{\Delta }\left(x\right)=\frac{{(x-1)}^2}{x+1},x\in [0,+\infty )$$

in (15). Then we can verify $f_{\Delta }\left(x\right)$ is convex because $f_{\Delta }^{″}\left(x\right)=\frac{8}{{(x+1)}^3}\ge 0$, $f_{\Delta }\left(1\right)=0$, $f_{\Delta }\left(x\right)\ge 0$ and $C_{f_{\Delta }}(P,Q)=\Delta (P,Q)$.

Example 2. (Jensen–Shannon divergence) Let us consider

$$f_{JS}\left(x\right)=\frac{x}{2}ln\frac{2x}{x+1}+\frac{1}{2}ln\frac{2}{x+1},x\in [0,+\infty )$$

in (15). Then we can verify $f_{JS}\left(x\right)$ is convex because $f_{JS}^{″}\left(x\right)=\frac{1}{2x^2+2x}\ge 0$, $f_{JS}\left(1\right)=0$ and $C_{f_{JS}}(P,Q)=JS(P,Q)$. By standard inequality $lnx\ge 1-\frac{1}{x}$, $f_{JS}\left(x\right)\ge \frac{x}{2}(1-\frac{x+1}{2x})+\frac{1}{2}(1-\frac{x+1}{2})=0$ holds.

Example 3. (Square of Hellinger distance) Let us consider

$$f_h\left(x\right)=\frac{1}{2}{(\sqrt{x}-1)}^2,x\in [0,+\infty )$$

in (15). Then we can verify $f_h\left(x\right)$ is convex because $f_h^{″}\left(x\right)=\frac{1}{4x\sqrt{x}}\ge 0$, $f_h\left(1\right)=0$, $f_h\left(x\right)\ge 0$ and $C_{f_h}(P,Q)=H^2(P,Q)$.

Example 4. (Variation distance) Let us consider

$$f_V\left(x\right)=|x-1|,x\in [0,+\infty )$$

in (15). Then we can easily get $f_V\left(x\right)$ is convex, $f_V\left(1\right)=0$, $f_V\left(x\right)\ge 0$ and $C_{f_V}(P,Q)=V(P,Q)$.

Theorem 5. Let $f_1,f_2$ be two nonnegative generating functions and there exists the real constants $k,K$ such that $k<K$ and if $f_2\left(x\right)\ne 0$ then

$$k\le \frac{f_1\left(x\right)}{f_2\left(x\right)}\le K,$$

if $f_2\left(x\right)=0$, then $f_1\left(x\right)=0$. We have the inequalities:

$$k{C}_{f_2}(P,Q)\le C_{f_1}(P,Q)\le K{C}_{f_2}(P,Q).$$

Proof. The conditions can be rewritten as $k{f}_2\left(x\right)\le f_1\left(x\right)\le K{f}_2\left(x\right)$. So from the formula (15),

$$\begin{array}{cc}\hfill C_{f_1}(P,Q)& =\sum _{i=1}^n{q}_i{f}_1\left(\frac{p_i}{q_i}\right)\ge \sum _{i=1}^n{q}_i\left(k{f}_2\left(\frac{p_i}{q_i}\right)\right)\hfill \\ & =k\sum _{i=1}^n{q}_i{f}_2\left(\frac{p_i}{q_i}\right)=k{C}_{f_2}(P,Q).\hfill \end{array}$$

and

$$\begin{array}{cc}\hfill C_{f_1}(P,Q)& =\sum _{i=1}^n{q}_i{f}_1\left(\frac{p_i}{q_i}\right)\le \sum _{i=1}^n{q}_i\left(K{f}_2\left(\frac{p_i}{q_i}\right)\right)\hfill \\ & =K\sum _{i=1}^n{q}_i{f}_2\left(\frac{p_i}{q_i}\right)=K{C}_{f_2}(P,Q).\hfill \end{array}$$

☐

We have shown that $f_{\Delta }$, $f_{JS}$, $f_h$, $f_V$ are all nonnegative. In the following we will have some inequalities.

Theorem 6.

$$\frac{1}{4}\Delta (P,Q)\le JS(P,Q)\le \frac{ln2}{2}\Delta (P,Q).$$

Proof. When $x\ne 1$, both $f_{\Delta }\left(1\right)$ and $f_{JS}\left(1\right)$ are not equal to 0. We consider the function:

$$\varphi \left(x\right)=\frac{f_{JS}\left(x\right)}{f_{\Delta }\left(x\right)}=\frac{\frac{x}{2}ln\frac{2x}{x+1}+\frac{1}{2}ln\frac{2}{x+1}}{\frac{{(x-1)}^2}{x+1}}.$$

The derivative of the function $\varphi \left(x\right)$ is

$${\varphi }^{\prime }\left(x\right)=\frac{(1+3x)lnx+4(1+x)ln\frac{2}{x+1}}{2{(1-x)}^3}.$$

Let

$$\psi \left(x\right)=(1+3x)lnx+4(1+x)ln\frac{2}{x+1}.$$

(16)

Straightforward derivative shows

$$\begin{array}{cc}\hfill {\psi }^{\prime }\left(x\right)& =3lnx+4ln\frac{2}{1+x}+\frac{1}{x}-1,\hfill \\ \hfill {\psi }^{\prime \prime }\left(x\right)& =-\frac{{(x-1)}^2}{x^2(x+1)}<0.\hfill \end{array}$$

So $\psi \left(x\right)$ is concave function when $x\in [0,+\infty )$ and ${\psi }^{\prime }\left(1\right)=\psi \left(1\right)=0$. This means $\psi \left(x\right)$ gets the maximum 0 at the point $x=1$. Accordingly $\psi \left(x\right)<0$ when $x\ne 1$. From (16), we find

$$\left\{\begin{array}{c}{\varphi }^{\prime }\left(x\right)<0,0<x<1\hfill \\ {\varphi }^{\prime }\left(x\right)>0,x>1\hfill \end{array}\right.$$

and

$$\underset{x\to 0^{+}}{lim}\varphi \left(x\right)=\frac{\frac{1}{2}ln2}{1}=\frac{ln2}{2}.$$

Using l’Hôspital’s rule (differentiate twice),

$$\underset{x\to 1}{lim}\varphi \left(x\right)=\underset{x\to 1}{lim}\frac{\frac{1}{2}\left(\frac{1}{x}-\frac{1}{x+1}\right)}{\frac{8}{{(x+1)}^3}}=\frac{1}{4}.$$

Using l’Hôspital’s rule (differentiate once),

$$\underset{x\to +\infty }{lim}\varphi \left(x\right)=\frac{\frac{1}{2}ln\frac{2x}{x+1}}{\frac{(x-1)(x+3)}{{(x+1)}^2}}=\frac{ln2}{2}.$$

Thus

$$\frac{1}{4}\le \varphi \left(x\right)=\frac{f_{JS}\left(x\right)}{f_{\Delta }\left(x\right)}\le \frac{ln2}{2}.$$

When $x=1$, $f_{\Delta }\left(1\right)=f_{JS}\left(1\right)=0$. As a consequence of Theorem 5, we obtain the result

$$\frac{1}{4}{C}_{f_{\Delta }}(P,Q)\le C_{f_{JS}}(P,Q)\le \frac{ln2}{2}{C}_{f_{\Delta }}(P,Q).$$

Thus the theorem is proved.   ☐

Theorem 7.

$$JS(P,Q)\le H^2(P,Q)\le \frac{1}{ln2}JS(P,Q).$$

Proof. When $x\ne 1$, both $f_h\left(1\right)$ and $f_{JS}\left(1\right)$ are not equal to 0. We consider the function:

$$\xi \left(x\right)=\frac{f_{JS}\left(x\right)}{f_h\left(x\right)}=\frac{\frac{x}{2}ln\frac{2x}{x+1}+\frac{1}{2}ln\frac{2}{x+1}}{\frac{1}{2}{(\sqrt{x}-1)}^2}.$$

The derivative of the function $\varphi \left(x\right)$ is

$${\xi }^{\prime }\left(x\right)=\frac{ln\frac{2}{x+1}+\sqrt{x}ln\frac{2x}{x+1}}{\sqrt{x}{(1-\sqrt{x})}^3}.$$

By standard inequality ${\displaystyle lnx\ge 1-\frac{1}{x}}$,

$$\begin{array}{cc}\hfill ln\frac{2}{x+1}+\sqrt{x}ln\frac{2x}{x+1}& \ge 1-\frac{x+1}{2}+\sqrt{x}\left(1-\frac{x+1}{2x}\right)\hfill \\ & =\frac{{(\sqrt{x}-1)}^2(\sqrt{x}+1)}{2\sqrt{x}}>0\hfill \end{array}$$

So

$$\left\{\begin{array}{c}{\xi }^{\prime }\left(x\right)>0,0<x<1\hfill \\ {\xi }^{\prime }\left(x\right)<0,x>1\hfill \end{array}\right.$$

and

$$\underset{x\to 0^{+}}{lim}\xi \left(x\right)=\frac{\frac{1}{2}ln2}{\frac{1}{2}}=ln2.$$

Using l’Hôspital’s rule (differentiate twice),

$$\underset{x\to 1}{lim}\xi \left(x\right)=\underset{x\to 1}{lim}\frac{\frac{1}{2}\left(\frac{1}{x}-\frac{1}{x+1}\right)}{\frac{1}{4\sqrt{x^3}}}=1.$$

Using l’Hôspital’s rule (differentiate once),

$$\underset{x\to +\infty }{lim}\xi \left(x\right)=\frac{\frac{1}{2}ln\frac{2x}{x+1}}{\frac{\sqrt{x}-1}{2\sqrt{x}}}=ln2.$$

Thus

$$ln2\le \varphi \left(x\right)=\frac{f_{JS}\left(x\right)}{f_h\left(x\right)}\le 1,$$

or

$$1\le \frac{1}{\varphi \left(x\right)}=\frac{f_h\left(x\right)}{f_{JS}\left(x\right)}\le \frac{1}{ln2}.$$

When $x=1$, $f_h\left(1\right)=f_{JS}\left(1\right)=0$. As a consequence of Theorem 5, we obtain the result

$$C_{f_{JS}}(P,Q)\le C_{f_h}(P,Q)\le \frac{1}{ln2}{C}_{f_{JS}}(P,Q).$$

Thus the theorem is proved.   ☐

Theorem 8.

$$\frac{1}{2}{V}^2(P,Q)\le \Delta (P,Q)\le V(P,Q).$$

Proof. When $x\ne 1$, both $f_{\Delta }\left(1\right)$ and $f_V\left(1\right)$ are not equal to 0. We consider the function:

$$\frac{f_{\Delta }\left(x\right)}{f_V\left(x\right)}=\frac{\frac{{(x-1)}^2}{x+1}}{|x-1|}=\frac{|x-1|}{x+1}\le 1.$$

When $x=1$, $f_{\Delta }\left(1\right)=f_V\left(1\right)=0$. As a consequence of Theorem 5, we obtain the result $C_{f_{\Delta }}(P,Q)\le C_{f_V}(P,Q)$. This means $\Delta (P,Q)\le V(P,Q)$. Next,

$$\begin{array}{cc}\hfill \frac{1}{2}{V}^2(P,Q)& =\frac{1}{2}{\left(\sum _{i=1}^n|p_i-q_i|\right)}^2\hfill \\ & \le \frac{1}{2}\left(\sum _{i=1}^n(p_i+q_i)\right)\left(\sum _{i=1}^n\frac{{(p_i-q_i)}^2}{p_i+q_i}\right)\text{(Cauchy--Schwarz}\text{inequality)}\hfill \\ & =\frac{1}{2}·2·\left(\sum _{i=1}^n\frac{{(p_i-q_i)}^2}{p_i+q_i}\right)\hfill \\ & =\sum _{i=1}^n\frac{{(p_i-q_i)}^2}{p_i+q_i}=\Delta (P,Q)\hfill \end{array}$$

Thus the theorem is proved.   ☐

From the above theorems, inequalities among these measures are given by

$$\begin{array}{cc}& \frac{1}{8}{V}^2(P,Q)\le \frac{1}{4}\Delta (P,Q)\le JS(P,Q)\le H^2(P,Q)\le \frac{1}{ln2}JS(P,Q)\hfill \\ & \le \frac{1}{2}\Delta (P,Q)\le \frac{1}{2}V(P,Q)\hfill \end{array}$$

(17)

These inequalities are sharper than the inequalities in [13] Theorem 2 and [15] (Section 3.1).

5. Asymptotic Approximation

Definition 5. For all $P,Q\in {\Gamma }_n$, the Chi-square divergence is defined by

$${\chi }^2(P,Q)=\sum _{i=1}^n\frac{{(p_i-q_i)}^2}{q_i}.$$

In [12],

$$JS(P,Q)=\frac{1}{2}{D}_{PQ}^2\approx \frac{1}{2}\sum _{i=1}^n\frac{1}{4q_i}{(p_i-q_i)}^2=\frac{1}{8}{\chi }^2(P,Q).$$

In this section, we will discuss the asymptotic approximation of $\Delta (P,Q)$ and $H^2(P,Q)$ when $P\to Q$ in $L^2$ norm.

Theorem 9. If ${\parallel P-Q\parallel }_2\to 0$, then

$$\Delta (P,Q)\to \frac{1}{2}{\chi }^2(P,Q),H^2(P,Q)\to \frac{1}{8}{\chi }^2(P,Q).$$

Proof. From Taylor’s series expansion at q, we have

$$\begin{array}{cc}\hfill \frac{{(x-q)}^2}{x+q}& =\frac{{(x-q)}^2}{2q}+o\left({(x-q)}^2\right)\hfill \\ \hfill \frac{1}{2}{(\sqrt{x}-\sqrt{q})}^2& =\frac{{(x-q)}^2}{8q}+o\left({(x-q)}^2\right)\hfill \end{array}$$

Hence

$$\begin{array}{cc}\hfill \Delta (P,Q)& =\sum _{i=1}^n\frac{{(p_i-q_i)}^2}{p_i+q_i}=\sum _{i=1}^n\frac{{(p_i-q_i)}^2}{2q_i}+o\left({\parallel P-Q\parallel }_2^2\right)\hfill \\ & =\frac{1}{2}{\chi }^2(P,Q)+o\left({\parallel P-Q\parallel }_2^2\right)\hfill \end{array}$$

$$\begin{array}{cc}\hfill H^2(P,Q)& =\sum _{i=1}^n\frac{1}{2}{(\sqrt{p_i}-\sqrt{q_i})}^2=\sum _{i=1}^n\frac{{(p_i-q_i)}^2}{8q_i}+o\left({\parallel P-Q\parallel }_2^2\right)\hfill \\ & =\frac{1}{8}{\chi }^2(P,Q)+o\left({\parallel P-Q\parallel }_2^2\right)\hfill \end{array}$$

☐

Equivalently, $JS(P,Q)\approx H^2(P,Q)\approx \frac{1}{4}\Delta (P,Q)\approx \frac{1}{8}{\chi }^2(P,Q)$ when $P\to Q$. So in some cases, one of the information-theoretic divergences can be substituted for another. The asymptotic property can also interpret the boundedness of triangular discrimination and, on the other hand, the new metrics.