Strongly Convex Divergences

Abstract

We consider a sub-class of the f-divergences satisfying a stronger convexity property, which we refer to as strongly convex, or $\kappa$-convex divergences. We derive new and old relationships, based on convexity arguments, between popular f-divergences.

1. Introduction

The concept of an f-divergence, introduced independently by Ali-Silvey [1], Morimoto [2], and Csisizár [3], unifies several important information measures between probability distributions, as integrals of a convex function f, composed with the Radon–Nikodym of the two probability distributions. (An additional assumption can be made that f is strictly convex at 1, to ensure that $D_f(\mu ||\nu )>0$ for $\mu \ne \nu$. This obviously holds for any $f^{″}(1)>0$, and can hold for some f-divergences without classical derivatives at 0, for instance the total variation is strictly convex at 1. An example of an f-divergence not strictly convex is provided by the so-called “hockey-stick” divergence, where $f(x)={(x-\gamma )}_{+}$, see [4,5,6].) For a convex function $f:(0,\infty )\to \mathbb{R}$ such that $f(1)=0$, and measures P and Q such that $P\ll Q$, the f-divergence from P to Q is given by $D_f(P||Q):=\int f\left(\frac{dP}{dQ}\right)dQ.$ The canonical example of an f-divergence, realized by taking $f(x)=x\log x$, is the relative entropy (often called the KL-divergence), which we denote with the subscript f omitted. f-divergences inherit many properties enjoyed by this special case; non-negativity, joint convexity of arguments, and a data processing inequality. Other important examples include the total variation, the ${\chi }^2$-divergence, and the squared Hellinger distance. The reader is directed to Chapter 6 and 7 of [7] for more background.

We are interested in how stronger convexity properties of f give improvements of classical f-divergence inequalities. More explicitly, we consider consequences of f being $\kappa$-convex, in the sense that the map $x\mapsto f(x)-\kappa x^2/2$ is convex. This is in part inspired by the work of Sason [8], who demonstrated that divergences that are $\kappa$-convex satisfy “stronger than ${\chi }^2$” data-processing inequalities.

Perhaps the most well known example of an f-divergence inequality is Pinsker’s inequality, which bounds the square of the total variation above by a constant multiple of the relative entropy. That is for probability measures P and Q, ${|P-Q|}_{TV}^2\le cD(P||Q)$. The optimal constant is achieved for Bernoulli measures, and under our conventions for total variation, $c=1/2\log e$. Many extensions and sharpenings of Pinsker’s inequality exist (for examples, see [9,10,11]). Building on the work of Guntuboyina [9] and Topsøe [11], we achieve a further sharpening of Pinsker’s inequality in Theorem 9.

Aside from the total variation, most divergences of interest have stronger than affine convexity, at least when f is restricted to a sub-interval of the real line. This observation is especially relevant to the situation in which one wishes to study $D_f(P||Q)$ in the existence of a bounded Radon–Nikodym derivative $\frac{dP}{dQ}\in (a,b)⊊(0,\infty )$. One naturally obtains such bounds for skew divergences. That is divergences of the form $(P,Q)\mapsto D_f((1-t)P+tQ||(1-s)P+sQ)$ for $t,s\in [0,1]$, as in this case, $\frac{(1-t)P+tQ}{(1-s)P+sQ}\le \max \left\{\frac{1-t}{1-s},\frac{t}{s}\right\}$. Important examples of skew-divergences include the skew divergence [12] based on the relative entropy and the Vincze–Le Cam divergence [13,14], called the triangular discrimination in [11] and its generalization due to Györfi and Vajda [15] based on the ${\chi }^2$-divergence. The Jensen–Shannon divergence [16] and its recent generalization [17] give examples of f-divergences realized as linear combinations of skewed divergences.

Let us outline the paper. In Section 2, we derive elementary results of $\kappa$-convex divergences and give a table of examples of $\kappa$-convex divergences. We demonstrate that $\kappa$-convex divergences can be lower bounded by the ${\chi }^2$-divergence, and that the joint convexity of the map $(P,Q)\mapsto D_f(P||Q)$ can be sharpened under $\kappa$-convexity conditions on f. As a consequence, we obtain bounds between the mean square total variation distance of a set of distributions from its barycenter, and the average f-divergence from the set to the barycenter.

In Section 3, we investigate general skewing of f-divergences. In particular, we introduce the skew-symmetrization of an f-divergence, which recovers the Jensen–Shannon divergence and the Vincze–Le Cam divergences as special cases. We also show that a scaling of the Vincze–Le Cam divergence is minimal among skew-symmetrizations of $\kappa$-convex divergences on $(0,2)$. We then consider linear combinations of skew divergences and show that a generalized Vincze–Le Cam divergence (based on skewing the ${\chi }^2$-divergence) can be upper bounded by the generalized Jensen–Shannon divergence introduced recently by Nielsen [17] (based on skewing the relative entropy), reversing the classical convexity bounds $D(P||Q)\le \log (1+{\chi }^2(P||Q))\le \log e{\chi }^2(P||Q)$. We also derive upper and lower total variation bounds for Nielsen’s generalized Jensen–Shannon divergence.

In Section 4, we consider a family of densities $\left\{p_i\right\}$ weighted by ${\lambda }_i$, and a density q. We use the Bayes estimator $T(x)=\arg {\max }_i{\lambda }_i{p}_i(x)$ to derive a convex decomposition of the barycenter $p={\sum }_i{\lambda }_i{p}_i$ and of q, each into two auxiliary densities. (Recall, a Bayes estimator is one that minimizes the expected value of a loss function. By the assumptions of our model, that $\mathbb{P}(\theta =i)={\lambda }_i$, and $\mathbb{P}(X\in A|\theta =i)={\int }_A{p}_i(x)dx$, we have $\mathbb{E}\ell (\theta ,\widehat{\theta })=1-\int {\lambda }_{\widehat{\theta }(x)}{p}_{\widehat{\theta }(x)}(x)dx$ for the loss function $\ell (i,j)=1-{\delta }_i(j)$ and any estimator $\widehat{\theta }$. It follows that $\mathbb{E}\ell (\theta ,\widehat{\theta })\ge \mathbb{E}\ell (\theta ,T)$ by ${\lambda }_{\widehat{\theta }(x)}{p}_{\widehat{\theta }(x)}(x)\le {\lambda }_{T(x)}{p}_{T(x)}(x)$. Thus, T is a Bayes estimator associated to ℓ. ) We use this decomposition to sharpen, for $\kappa$-convex divergences, an elegant theorem of Guntuboyina [9] that generalizes Fano and Pinsker’s inequality to f-divergences. We then demonstrate explicitly, using an argument of Topsøe, how our sharpening of Guntuboyina’s inequality gives a new sharpening of Pinsker’s inequality in terms of the convex decomposition induced by the Bayes estimator.

Notation

Throughout, f denotes a convex function $f:(0,\infty )\to \mathbb{R}\cup \{\infty \}$, such that $f(1)=0$. For a convex function defined on $(0,\infty )$, we define $f(0):={\lim }_{x\to 0}f(x)$. We denote by $f^{\ast }$, the convex function $f^{\ast }:(0,\infty )\to \mathbb{R}\cup \{\infty \}$ defined by $f^{\ast }(x)=xf(x^{-1})$. We consider Borel probability measures P and Q on a Polish space $\mathcal{X}$ and define the f-divergence from P to Q, via densities p for P and q for Q with respect to a common reference measure $\mu$ as

$$\begin{array}{cc}\hfill D_f(p||q)& ={\int }_{\mathcal{X}}f\left(\frac{p}{q}\right)qd\mu \hfill \\ \hfill & ={\int }_{\{pq>0\}}qf\left(\frac{p}{q}\right)d\mu +f(0)Q(\{p=0\})+f^{\ast }(0)P(\{q=0\}).\hfill \end{array}$$

(1)

We note that this representation is independent of $\mu$, and such a reference measure always exists, take $\mu =P+Q$ for example.

For $t,s\in [0,1]$, define the binary f-divergence

$$D_f(t||s):=sf\left(\frac{t}{s}\right)+(1-s)f\left(\frac{1-t}{1-s}\right)$$

(2)

with the conventions, $f(0)={\lim }_{t\to 0^{+}}f(t)$, $0f(0/0)=0$, and $0f(a/0)=a{\lim }_{t\to \infty }f(t)/t$. For a random variable X and a set A, we denote the probability that X takes a value in A by $\mathbb{P}(X\in A)$, the expectation of the random variable by $\mathbb{E}X$, and the variance by ${\mathrm{Var}(X):=\mathbb{E}|X-\mathbb{E}X|}^2$. For a probability measure $\mu$ satisfying $\mu (A)=\mathbb{P}(X\in A)$ for all Borel A, we write $X\sim \mu$, and, when there exists a probability density function such that $\mathbb{P}(X\in A)={\int }_{A}f(x)d\gamma (x)$ for a reference measure $\gamma$, we write $X\sim f$. For a probability measure $\mu$ on $\mathcal{X}$, and an $L^2$ function $f:\mathcal{X}\to \mathbb{R}$, we denote ${\mathrm{Var}}_{\mu }(f):=\mathrm{Var}(f(X))$ for $X\sim \mu$.

2. Strongly Convex Divergences

Definition 1.

A $\mathbb{R}\cup \{\infty \}$-valued function f on a convex set $K\subseteq \mathbb{R}$ is κ-convex when $x,y\in K$ and $t\in [0,1]$ implies

$$f((1-t)x+ty)\le (1-t)f(x)+tf(y)-\kappa t(1-t){(x-y)}^2/2.$$

(3)

For example, when f is twice differentiable, (3) is equivalent to $f^{″}(x)\ge \kappa$ for $x\in K$. Note that the case $\kappa =0$ is just usual convexity.

Proposition 1.

For $f:K\to \mathbb{R}\cup \{\infty \}$ and $\kappa \in [0,\infty )$, the following are equivalent:

• f is κ-convex.
• The function $f-\kappa {(t-a)}^2/2$ is convex for any $a\in \mathbb{R}$.
• The right handed derivative, defined as $f_{+}^{\prime }(t):={\lim }_{h\downarrow 0}\frac{f(t+h)-f(t)}{h}$ satisfies,

  $$f_{+}^{\prime }(t)\ge f_{+}^{\prime }(s)+\kappa (t-s)$$
  for $t\ge s$.

Proof. 

Observe that it is enough to prove the result when $\kappa =0$, where the proposition is reduced to the classical result for convex functions. □

Definition 2.

An f-divergence $D_f$ is κ-convex on an interval K for $\kappa \ge 0$ when the function f is κ-convex on K.

Table 1. Examples of Strongly Convex Divergences.

Divergence	f	$\mathit{\kappa }$	Domain
relative entropy (KL)	$t\log t$	$\frac{1}{M}$	$(0,M]$
total variation	$\frac{|t-1|}{2}$	0	$(0,\infty )$
Pearson’s ${\chi }^2$	${(t-1)}^2$	2	$(0,\infty )$
squared Hellinger	$2(1-\sqrt{t})$	$M^{-\frac{3}{2}}/2$	$(0,M]$
reverse relative entropy	$-\log t$	$1/M^2$	$(0,M]$
Vincze- Le Cam	$\frac{{(t-1)}^2}{t+1}$	$\frac{8}{{(M+1)}^3}$	$(0,M]$
Jensen–Shannon	$(t+1)\log \frac{2}{t+1}+t\log t$	$\frac{1}{M(M+1)}$	$(0,M]$
Neyman’s ${\chi }^2$	$\frac{1}{t}-1$	$2/M^3$	$(0,M]$
Sason’s s	$\log {(s+t)}^{{(s+t)}^2}-\log {(s+1)}^{{(s+1)}^2}$	$2\log (s+M)+3$	$[M,\infty )$, $s>e^{-3/2}$
$\alpha$-divergence	$\frac{4\left(1-t^{\frac{1+\alpha }{2}}\right)}{1-{\alpha }^2},\alpha \ne \pm 1$	$M^{\frac{\alpha -3}{2}}$	$\left\{\begin{array}{c}[M,\infty ),\alpha >3\hfill \\ (0,M],\alpha <3\hfill \end{array}\right.$

lists some $\kappa$-convex f-divergences of interest to this article.

Observe that we have taken the normalization convention on the total variation (the total variation for a signed measure $\mu$ on a space X can be defined through the Hahn-Jordan decomposition of the measure into non-negative measures ${\mu }^{+}$ and ${\mu }^{-}$ such that $\mu ={\mu }^{+}-{\mu }^{-}$, as $\parallel \mu \parallel ={\mu }^{+}(X)+{\mu }^{-}(X)$ (see [18]); in our notation, ${|\mu |}_{TV}=\parallel \mu \parallel /2$) which we denote by ${|P-Q|}_{TV}$, such that ${|P-Q|}_{TV}={\sup }_A|P(A)-Q(A)|\le 1$. In addition, note that the $\alpha$-divergence interpolates Pearson’s ${\chi }^2$-divergence when $\alpha =3$, one half Neyman’s ${\chi }^2$-divergence when $\alpha =-3$, the squared Hellinger divergence when $\alpha =0$, and has limiting cases, the relative entropy when $\alpha =1$ and the reverse relative entropy when $\alpha =-1$. If f is $\kappa$-convex on $[a,b]$, then recalling its dual divergence $f^{\ast }(x):=xf(x^{-1})$ is $\kappa a^3$-convex on $[\frac{1}{b},\frac{1}{a}]$. Recall that $f^{\ast }$ satisfies the equality $D_{f^{\ast }}(P||Q)=D_f(Q||P)$. For brevity, we use ${\chi }^2$-divergence to refer to the Pearson ${\chi }^2$-divergence, and we articulate Neyman’s ${\chi }^2$ explicitly when necessary.

The next lemma is a restatement of Jensen’s inequality.

Lemma 1.

If f is κ-convex on the range of X,

$$\mathbb{E}f(X)\ge f(\mathbb{E}(X))+\frac{\kappa }{2}\mathrm{Var}(X).$$

Proof. 

Apply Jensen’s inequality to $f(x)-\kappa x^2/2$. □

For a convex function f such that $f(1)=0$ and $c\in \mathbb{R}$, the function $\tilde{f}(t)=f(t)+c(t-1)$ remains a convex function, and what is more satisfies

$$D_f(P||Q)=D_{\tilde{f}}(P||Q)$$

since $\int c(p/q-1)qd\mu =0$.

Definition 3

(${\chi }^2$-divergence). For $f(t)={(t-1)}^2$, we write

$${\chi }^2(P||Q):=D_f(P||Q).$$

We pursue a generalization of the following bound on the total variation by the ${\chi }^2$-divergence [19,20,21].

Theorem 1

([19,20,21]). For measures P and Q,

$$\begin{array}{c}\hfill {|P-Q|}_{TV}^2\le \frac{{\chi }^2(P||Q)}{2}.\end{array}$$

(4)

We mention the work of Harremos and Vadja [20], in which it is shown, through a characterization of the extreme points of the joint range associated to a pair of f-divergences (valid in general), that the inequality characterizes the “joint range”, that is, the range of the function $(P,Q)\mapsto {(|P-Q|}_{TV},{\chi }^2(P||Q))$. We use the following lemma, which shows that every strongly convex divergence can be lower bounded, up to its convexity constant $\kappa >0$, by the ${\chi }^2$-divergence,

Lemma 2.

For a κ-convex f,

$$D_f(P||Q)\ge \frac{\kappa }{2}{\chi }^2(P||Q).$$

Proof. 

Define a $\tilde{f}(t)=f(t)-f_{+}^{\prime }(1)(t-1)$ and note that $\tilde{f}$ defines the same $\kappa$-convex divergence as f. Thus, we may assume without loss of generality that $f_{+}^{\prime }$ is uniquely zero when $t=1$. Since f is $\kappa$-convex $\varphi :t\mapsto f(t)-\kappa {(t-1)}^2/2$ is convex, and, by $f_{+}^{\prime }(1)=0$, ${\varphi }_{+}^{\prime }(1)=0$ as well. Thus, $\varphi$ takes its minimum when $t=1$ and hence $\varphi \ge 0$ so that $f(t)\ge \kappa {(t-1)}^2/2$. Computing,

$$\begin{array}{cc}\hfill D_f(P||Q)& =\int f\left(\frac{dP}{dQ}\right)dQ\hfill \\ \hfill & \ge \frac{\kappa }{2}\int {\left(\frac{dP}{dQ}-1\right)}^{2}dQ\hfill \\ \hfill & =\frac{\kappa }{2}{\chi }^2(P||Q).\hfill \end{array}$$

 □

Based on a Taylor series expansion of f about 1, Nielsen and Nock ([22], [Corollary 1]) gave the estimate

$$\begin{array}{c}\hfill D_f(P||Q)\approx \frac{f^{″}(1)}{2}{\chi }^2(P||Q)\end{array}$$

(5)

for divergences with a non-zero second derivative and P close to Q. Lemma 2 complements this estimate with a lower bound, when f is $\kappa$-concave. In particular, if $f^{″}(1)=\kappa$, it shows that the approximation in (5) is an underestimate.

Theorem 2.

For measures P and Q, and a κ convex divergence $D_f$,

$$\begin{array}{c}\hfill {|P-Q|}_{TV}^2\le \frac{D_f(P||Q)}{\kappa }.\end{array}$$

(6)

Proof. 

By Lemma 2 and then Theorem 1,

$$\begin{array}{c}\hfill \frac{D_f(P||Q)}{\kappa }\ge \frac{{\chi }^2(P||Q)}{2}\ge {|P-Q|}_{TV}.\end{array}$$

(7)

 □

The proof of Lemma 2 uses a pointwise inequality between convex functions to derive an inequality between their respective divergences. This simple technique was shown to have useful implications by Sason and Verdu in [6], where it appears as Theorem 1 and is used to give sharp comparisons in several f-divergence inequalities.

Theorem 3

(Sason–Verdu [6]). For divergences defined by g and f with $cf(t)\ge g(t)$ for all t, then

$$D_g(P||Q)\le c{D}_f(P||Q).$$

Moreover, if $f^{\prime }(1)=g^{\prime }(1)=0$, then

$$\underset{P\ne Q}{\sup }\frac{D_g(P||Q)}{D_f(P||Q)}=\underset{t\ne 1}{\sup }\frac{g(t)}{f(t)}.$$

Corollary 1.

For a smooth κ-convex divergence f, the inequality

$$D_f(P||Q)\ge \frac{\kappa }{2}{\chi }^2(P||Q)$$

(8)

is sharp multiplicatively in the sense that

$$\underset{P\ne Q}{\inf }\frac{D_f(P||Q)}{{\chi }^2(P||Q)}=\frac{\kappa }{2}.$$

(9)

if $f^{″}(1)=\kappa$.

In information geometry, a standard f-divergence is defined as an f-divergence satisfying the normalization $f(1)=f^{\prime }(1)=0,f^{″}(1)=1$ (see [23]). Thus, Corollary 1 shows that $\frac{1}{2}{\chi }^2$ provides a sharp lower bound on every standard f-divergence that is 1-convex. In particular, the lower bound in Lemma 2 complimenting the estimate (5) is shown to be sharp.

Proof. 

Without loss of generality, we assume that $f^{\prime }(1)=0$. If $f^{″}(1)=\kappa +2\epsilon$ for some $\epsilon >0$, then taking $g(t)={(t-1)}^2$ and applying Theorem 3 and Lemma 2

$$\underset{P\ne Q}{\sup }\frac{D_g(P||Q)}{D_f(P||Q)}=\underset{t\ne 1}{\sup }\frac{g(t)}{f(t)}\le \frac{2}{\kappa }.$$

(10)

Observe that, after two applications of L’Hospital,

$$\underset{\epsilon \to 0}{\lim }\frac{g(1+\epsilon )}{f(1+\epsilon )}=\underset{\epsilon \to 0}{\lim }\frac{g^{\prime }(1+\epsilon )}{f^{\prime }(1+\epsilon )}=\frac{g^{″}(1)}{f^{″}(1)}=\frac{2}{\kappa }\le \underset{t\ne 1}{\sup }\frac{g(t)}{f(t)}.$$

Thus, (9) follows. □

Proposition 2.

When $D_f$ is an f divergence such that f is κ-convex on $[a,b]$ and that $P_{\theta }$ and $Q_{\theta }$ are probability measures indexed by a set Θ such that $a\le \frac{d{P}_{\theta }}{d{Q}_{\theta }}(x)\le b$, holds for all θ and $P:={\int }_{\Theta }{P}_{\theta }d\mu (\theta )$ and $Q:={\int }_{\Theta }{Q}_{\theta }d\mu (\theta )$ for a probability measure μ on Θ, then

$$D_f(P||Q)\le {\int }_{\Theta }{D}_f(P_{\theta }||Q_{\theta })d\mu (\theta )-\frac{\kappa }{2}{\int }_{\Theta }{\int }_{\mathcal{X}}{\left(\frac{d{P}_{\theta }}{d{Q}_{\theta }}-\frac{dP}{dQ}\right)}^{2}dQd\mu ,$$

(11)

In particular, when $Q_{\theta }=Q$ for all θ

$$\begin{array}{c}{D}_f(P||Q)\hfill \\ \le {\int }_{\Theta }{D}_f(P_{\theta }||Q)d\mu (\theta )-\frac{\kappa }{2}{\int }_{\Theta }{\int }_{\mathcal{X}}{\left(\frac{d{P}_{\theta }}{dQ}-\frac{dP}{dQ}\right)}^{2}dQd\mu (\theta )\hfill \\ \le {\int }_{\Theta }{D}_f(P_{\theta }||Q)d\mu (\theta )-\kappa {\int }_{\Theta }{|P_{\theta }-P|}_{TV}^{2}d\mu (\theta )\hfill \end{array}$$

(12)

Proof. 

Let $d\theta$ denote a reference measure dominating $\mu$ so that $d\mu =\phi (\theta )d\theta$ then write ${\nu }_{\theta }=\nu (\theta ,x)=\frac{d{Q}_{\theta }}{dQ}(x)\phi (\theta )$.

$$\begin{array}{cc}\hfill D_f(P||Q)& ={\int }_{\mathcal{X}}f\left(\frac{dP}{dQ}\right)dQ\hfill \\ \hfill & ={\int }_{\mathcal{X}}f\left({\int }_{\Theta }\frac{d{P}_{\theta }}{dQ}d\mu (\theta )\right)dQ\hfill \\ \hfill & ={\int }_{\mathcal{X}}f\left({\int }_{\Theta }\frac{d{P}_{\theta }}{d{Q}_{\theta }}\nu (\theta ,x)d\theta \right)dQ\hfill \end{array}$$

(13)

By Jensen’s inequality, as in Lemma 1

$$\begin{array}{cc}\hfill f\left({\int }_{\Theta }\frac{d{P}_{\theta }}{d{Q}_{\theta }}{\nu }_{\theta }d\theta \right)\le {\int }_{\theta }f& \left(\frac{d{P}_{\theta }}{d{Q}_{\theta }}\right){\nu }_{\theta }d\theta -\frac{\kappa }{2}{\int }_{\Theta }{\left(\frac{d{P}_{\theta }}{d{Q}_{\theta }}-{\int }_{\Theta }\frac{d{P}_{\theta }}{d{Q}_{\theta }}{\nu }_{\theta }d\theta \right)}^2{\nu }_{\theta }d\theta \hfill \end{array}$$

Integrating this inequality gives

$$D_f(P||Q)\le {\int }_{\mathcal{X}}\left({\int }_{\theta }f\left(\frac{d{P}_{\theta }}{d{Q}_{\theta }}\right){\nu }_{\theta }d\theta -\frac{\kappa }{2}{\int }_{\Theta }{\left(\frac{d{P}_{\theta }}{d{Q}_{\theta }}-{\int }_{\Theta }\frac{d{P}_{\theta }}{d{Q}_{\theta }}{\nu }_{\theta }d\theta \right)}^2{\nu }_{\theta }d\theta \right)dQ$$

(14)

Note that

$${\int }_{\mathcal{X}}{\int }_{\Theta }{\left(\frac{d{P}_{\theta }}{d{Q}_{\theta }}dQ-{\int }_{\Theta }\frac{d{P}_{\theta }}{d{Q}_{{\theta }_0}}{\nu }_{{\theta }_0}d{\theta }_0\right)}^2{\nu }_{\theta }d\theta dQ={\int }_{\Theta }{\int }_{\mathcal{X}}{\left(\frac{d{P}_{\theta }}{d{Q}_{\theta }}-\frac{dP}{dQ}\right)}^{2}dQd\mu ,$$

and

$$\begin{array}{cc}\hfill {\int }_{\mathcal{X}}{\int }_{\Theta }f\left(\frac{d{P}_{\theta }}{d{Q}_{\theta }}\right)\nu (\theta ,x)d\theta dQ& ={\int }_{\Theta }{\int }_{\mathcal{X}}f\left(\frac{d{P}_{\theta }}{d{Q}_{\theta }}\right)\nu (\theta ,x)dQd\theta \hfill \\ \hfill & ={\int }_{\Theta }{\int }_{\mathcal{X}}f\left(\frac{d{P}_{\theta }}{d{Q}_{\theta }}\right)d{Q}_{\theta }d\mu (\theta )\hfill \\ \hfill & ={\int }_{\Theta }D(P_{\theta }||Q_{\theta })d\mu (\theta )\hfill \end{array}$$

(15)

Inserting these equalities into (14) gives the result.

To obtain the total variation bound, one needs only to apply Jensen’s inequality,

$$\begin{array}{cc}\hfill {\int }_{\mathcal{X}}{\left(\frac{d{P}_{\theta }}{dQ}-\frac{dP}{dQ}\right)}^{2}dQ& \ge {\left({\int }_{\mathcal{X}}\left|\frac{d{P}_{\theta }}{dQ}-\frac{dP}{dQ}\right|dQ\right)}^2\hfill \\ \hfill & =|P_{\theta }{-P|}_{TV}^2.\hfill \end{array}$$

(16)

 □

Observe that, taking $Q=P={\int }_{\Theta }{P}_{\theta }d\mu (\theta )$ in Proposition 2, one obtains a lower bound for the average f-divergence from the set of distribution to their barycenter, by the mean square total variation of the set of distributions to the barycenter,

$$\kappa {\int }_{\Theta }|P_{\theta }{-P|}_{TV}^{2}d\mu (\theta )\le {\int }_{\Theta }{D}_f(P_{\theta }||P)d\mu (\theta ).$$

(17)

An alternative proof of this can be obtained by applying $|P_{\theta }{-P|}_{TV}^2\le D_f(P_{\theta }||P)/\kappa$ from Theorem 2 pointwise.

The next result shows that, for f strongly convex, Pinsker type inequalities can never be reversed,

Proposition 3.

Given f strongly convex and $M>0$, there exists P, Q measures such that

$$D_f(P||Q)\ge {M|P-Q|}_{TV}.$$

(18)

Proof. 

By $\kappa$-convexity $\varphi (t)=f(t)-\kappa t^2/2$ is a convex function. Thus, $\varphi (t)\ge \varphi (1)+{\varphi }_{+}^{\prime }(1)(t-1)=(f_{+}^{\prime }(1)-\kappa )(t-1)$ and hence ${\lim }_{t\to \infty }\frac{f(t)}{t}\ge {\lim }_{t\to \infty }\kappa t/2+(f_{+}^{\prime }(1)-\kappa )\left(1-\frac{1}{t}\right)=\infty .$ Taking measures on the two points space $P=\{1/2,1/2\}$ and $Q=\{1/2t,1-1/2t\}$ gives $D_f(P||Q)\ge \frac{1}{2}\frac{f(t)}{t}$ which tends to infinity with $t\to \infty$, while ${|P-Q|}_{TV}\le 1$. □

In fact, building on the work of Basu-Shioya-Park [24] and Vadja [25], Sason and Verdu proved [6] that, for any f divergence, ${\sup }_{P\ne Q}\frac{D_f(P||Q)}{{|P-Q|}_{TV}}=f(0)+f^{\ast }(0)$. Thus, an f-divergence can be bounded above by a constant multiple of a the total variation, if and only if $f(0)+f^{\ast }(0)<\infty$. From this perspective, Proposition 3 is simply the obvious fact that strongly convex functions have super linear (at least quadratic) growth at infinity.

3. Skew Divergences

If we denote $Cvx(0,\infty )$ to be quotient of the cone of convex functions f on $(0,\infty )$ such that $f(1)=0$ under the equivalence relation $f_1\sim f_2$ when $f_1-f_2=c(x-1)$ for $c\in \mathbb{R}$, then the map $f\mapsto D_f$ gives a linear isomorphism between $Cvx(0,\infty )$ and the space of all f-divergences. The mapping $\mathcal{T}:Cvx(0,\infty )\to Cvx(0,\infty )$ defined by $\mathcal{T}f=f^{\ast }$, where we recall $f^{\ast }(t)=tf(t^{-1})$, gives an involution of $Cvx(0,\infty )$. Indeed, $D_{\mathcal{T}f}(P||Q)=D_f(Q||P)$, so that $D_{\mathcal{T}(\mathcal{T}(f))}(P||Q)=D_f(P||Q)$. Mathematically, skew divergences give an interpolation of this involution as

$$(P,Q)\mapsto D_f((1-t)P+tQ||(1-s)P+sQ)$$

gives $D_f(P||Q)$ by taking $s=1$ and $t=0$ or yields $D_{f^{\ast }}(P||Q)$ by taking $s=0$ and $t=1$.

Moreover, as mentioned in the Introduction, skewing imposes boundedness of the Radon–Nikodym derivative $\frac{dP}{dQ}$, which allows us to constrain the domain of f-divergences and leverage $\kappa$-convexity to obtain f-divergence inequalities in this section.

The following appears as Theorem III.1 in the preprint [26]. It states that skewing an f-divergence preserves its status as such. This guarantees that the generalized skew divergences of this section are indeed f-divergences. A proof is given in the Appendix A for the convenience of the reader.

Theorem 4

(Melbourne et al [26]). For $t,s\in [0,1]$ and a divergence $D_f$, then

$$S_f(P||Q):=D_f((1-t)P+tQ||(1-s)P+sQ)$$

(19)

is an f-divergence as well.

Definition 4.

For an f-divergence, its skew symmetrization,

$${\Delta }_f(P||Q):=\frac{1}{2}{D}_f\left(P\left|\right|\frac{P+Q}{2}\right)+\frac{1}{2}{D}_f\left(Q||\frac{P+Q}{2}\right).$$

${\Delta }_f$ is determined by the convex function

$$x\mapsto \frac{1+x}{2}\left(f\left(\frac{2x}{1+x}\right)+f\left(\frac{2}{1+x}\right)\right).$$

(20)

Observe that ${\Delta }_f(P||Q)={\Delta }_f(Q||P)$, and when $f(0)<\infty$, ${\Delta }_f(P||Q)\le {\sup }_{x\in [0,2]}f(x)<\infty$ for all $P,Q$ since $\frac{dP}{d(P+Q)/2}$, $\frac{dQ}{d(P+Q)/2}\le 2$. When $f(x)=x\log x$, the relative entropy’s skew symmetrization is the Jensen–Shannon divergence. When $f(x)={(x-1)}^2$ up to a normalization constant the ${\chi }^2$-divergence’s skew symmetrization is the Vincze–Le Cam divergence which we state below for emphasis. The work of Topsøe [11] provides more background on this divergence, where it is referred to as the triangular discrimination.

Definition 5.

When $f(t)=\frac{{(t-1)}^2}{t+1}$, denote the Vincze–Le Cam divergence by

$$\Delta (P||Q):=D_f(P||Q).$$

If one denotes the skew symmetrization of the ${\chi }^2$-divergence by ${\Delta }_{{\chi }^2}$, one can compute easily from (20) that ${\Delta }_{{\chi }^2}(P||Q)=\Delta (P||Q)/2$. We note that although skewing preserves 0-convexity, by the above example, it does not preserve $\kappa$-convexity in general. The skew symmetrization of the ${\chi }^2$-divergence a 2-convex divergence while $f(t)={(t-1)}^2/(t+1)$ corresponding to the Vincze–Le Cam divergence satisfies $f^{″}(t)=\frac{8}{{(t+1)}^3}$, which cannot be bounded away from zero on $(0,\infty )$.

Corollary 2.

For an f-divergence such that f is a κ-convex on $(0,2)$,

$${\Delta }_f(P||Q)\ge \frac{\kappa }{4}\Delta (P||Q)=\frac{\kappa }{2}{\Delta }_{{\chi }^2}(P||Q),$$

(21)

with equality when the $f(t)={(t-1)}^2$ corresponding the the ${\chi }^2$-divergence, where ${\Delta }_f$ denotes the skew symmetrized divergence associated to f and Δ is the Vincze- Le Cam divergence.

Proof. 

Applying Proposition 2

$$\begin{array}{cc}\hfill 0& =D_f\left(\frac{P+Q}{2}\left|\right|\frac{Q+P}{2}\right)\hfill \\ \hfill & \le \frac{1}{2}{D}_f\left(P\left|\right|\frac{Q+P}{2}\right)+\frac{1}{2}{D}_f\left(Q\left|\right|\frac{Q+P}{2}\right)-\frac{\kappa }{8}\int {\left(\frac{2P}{P+Q}-\frac{2Q}{P+Q}\right)}^{2}d(P+Q)/2\hfill \\ \hfill & ={\Delta }_f(P||Q)-\frac{\kappa }{4}\Delta (P||Q).\hfill \end{array}$$

 □

When $f(x)=x\log x$, we have $f^{″}(x)\ge \frac{\log e}{2}$ on $[0,2]$, which demonstrates that up to a constant $\frac{\log e}{8}$ the Jensen–Shannon divergence bounds the Vincze–Le Cam divergence (see [11] for improvement of the inequality in the case of the Jensen–Shannon divergence, called the “capacitory discrimination” in the reference, by a factor of 2).

We now investigate more general, non-symmetric skewing in what follows.

Proposition 4.

For $\alpha ,\beta \in [0,1]$, define

$$C(\alpha ):=\left\{\begin{array}{cc}1-\alpha \hfill & when\alpha \le \beta \hfill \\ \alpha \hfill & when\alpha >\beta ,\hfill \end{array}\right.$$

(22)

and

$$S_{\alpha ,\beta }(P||Q):=D((1-\alpha )P+\alpha Q||(1-\beta )P+\beta Q).$$

(23)

Then,

$$S_{\alpha ,\beta }(P||Q)\le C(\alpha )D_{\infty }{(\alpha ||\beta )|P-Q|}_{TV},$$

(24)

where $D_{\infty }(\alpha ||\beta ):=\log \left(\max \left\{\frac{\alpha }{\beta },\frac{1-\alpha }{1-\beta }\right\}\right)$ is the binary ∞-Rényi divergence [27].

We need the following lemma originally proved by Audenart in the quantum setting [28]. It is based on a differential relationship between the skew divergence [12] and the [15] (see [29,30]).

Lemma 3

(Theorem III.1 [26]). For P and Q probability measures and $t\in [0,1]$,

$$S_{0,t}(P||Q)\le {-\log t|P-Q|}_{TV}.$$

(25)

Proof of Theorem 4.

If $\alpha \le \beta$, then $D_{\infty }(\alpha ||\beta )=\log \frac{1-\alpha }{1-\beta }$ and $C(\alpha )=1-\alpha$. In addition,

$$(1-\beta )P+\beta Q=t\left((1-\alpha )P+\alpha Q\right)+(1-t)Q$$

(26)

with $t=\frac{1-\beta }{1-\alpha }$, thus

$$\begin{array}{cc}\hfill S_{\alpha ,\beta }(P||Q)& =S_{0,t}((1-\alpha )P+\alpha Q||Q)\hfill \\ \hfill & \le {(-\log t)|((1-\alpha )P+\alpha Q)-Q|}_{TV}\hfill \\ \hfill & =C(\alpha )D_{\infty }{(\alpha ||\beta )|P-Q|}_{TV},\hfill \end{array}$$

(27)

where the inequality follows from Lemma 3. Following the same argument for $\alpha >\beta$, so that $C(\alpha )=\alpha$, $D_{\infty }(\alpha ||\beta )=\log \frac{\alpha }{\beta }$, and

$$(1-\beta )P+\beta Q=t\left((1-\alpha )P+\alpha Q\right)+(1-t)P$$

(28)

for $t=\frac{\beta }{\alpha }$ completes the proof. Indeed,

$$\begin{array}{cc}\hfill S_{\alpha ,\beta }(P||Q)& =S_{0,t}((1-\alpha )P+\alpha Q||P)\hfill \\ \hfill & \le {-\log t|((1-\alpha )P+\alpha Q)-P|}_{TV}\hfill \\ \hfill & =C(\alpha )D_{\infty }{(\alpha ||\beta )|P-Q|}_{TV}.\hfill \end{array}$$

(29)

 □

We recover the classical bound [11,16] of the Jensen–Shannon divergence by the total variation.

Corollary 3.

For probability measure P and Q,

$$\mathrm{JSD}(P||Q)\le {\log 2|P-Q|}_{TV}$$

(30)

Proof. 

Since $\mathrm{JSD}(P||Q)=\frac{1}{2}{S}_{0,\frac{1}{2}}(P||Q)+\frac{1}{2}{S}_{1,\frac{1}{2}}(P||Q)$. □

Proposition 4 gives a sharpening of Lemma 1 of Nielsen [17], who proved $S_{\alpha ,\beta }(P||Q)\le D_{\infty }(\alpha ||\beta )$, and used the result to establish the boundedness of a generalization of the Jensen–Shannon Divergence.

Definition 6

(Nielsen [17]). For p and q densities with respect to a reference measure μ, $w_i>0$, such that ${\sum }_{i=1}^n{w}_i=1$ and ${\alpha }_i\in [0,1]$, define

$$J{S}^{\alpha ,w}(p:q)={\displaystyle \sum _{i=1}^n}{w}_{i}D((1-{\alpha }_i)p+{\alpha }_{i}q||(1-\overline{\alpha })p+\overline{\alpha }q)$$

(31)

where ${\sum }_{i=1}^n{w}_i{\alpha }_i=\overline{\alpha }$.

Note that, when $n=2$, ${\alpha }_1=1$, ${\alpha }_2=0$ and $w_i=\frac{1}{2}$, $J{S}^{\alpha ,w}(p:q)=\mathrm{JSD}(p||q)$, the usual Jensen–Shannon divergence. We now demonstrate that Nielsen’s generalized Jensen–Shannon Divergence can be bounded by the total variation distance just as the ordinary Jensen–Shannon Divergence.

Theorem 5.

For p and q densities with respect to a reference measure μ, $w_i>0$, such that ${\sum }_{i=1}^n{w}_i=1$ and ${\alpha }_i\in (0,1)$,

$$\log e{\mathrm{Var}}_w{(\alpha )|p-q|}_{TV}^2\le J{S}^{\alpha ,w}(p:q)\le \mathcal{A}H(w){|p-q|}_{TV}$$

(32)

where $H(w):=-{\sum }_i{w}_i\log w_i\ge 0$ and $\mathcal{A}={\max }_i|{\alpha }_i-{\overline{\alpha }}_i|$ with ${\overline{\alpha }}_i={\sum }_{j\ne i}\frac{w_j{\alpha }_j}{1-w_i}$.

Note that, since ${\overline{\alpha }}_i$ is the w average of the ${\alpha }_j$ terms with ${\alpha }_i$ removed, ${\overline{\alpha }}_i\in [0,1]$ and thus $\mathcal{A}\le 1$. We need the following Theorem from Melbourne et al. [26] for the upper bound.

Theorem 6

([26] Theorem 1.1). For $f_i$ densities with respect to a common reference measure γ and ${\lambda }_i>0$ such that ${\sum }_{i=1}^n{\lambda }_i=1$,

$$h_{\gamma }(\sum _i{\lambda }_i{f}_i)-\sum _i{\lambda }_i{h}_{\gamma }(f_i)\le \mathcal{T}H(\lambda ),$$

(33)

where $h_{\gamma }(f_i):=-\int f_i(x)\log f_i(x)d\gamma (x)$ and $\mathcal{T}={\sup }_i{|f_i-{\tilde{f}}_i|}_{TV}$ with ${\tilde{f}}_i={\sum }_{j\ne i}\frac{{\lambda }_j}{1-{\lambda }_i}{f}_j$.

Proof of Theorem 5.

We apply Theorem 6 with $f_i=(1-{\alpha }_i)p+{\alpha }_{i}q$, ${\lambda }_i=w_i$, and noticing that in general

$$h_{\gamma }(\sum _i{\lambda }_i{f}_i)-\sum _i\lambda h_{\gamma }(f_i)=\sum _i{\lambda }_{i}D(f_i||f),$$

(34)

we have

$$\begin{array}{cc}\hfill J{S}^{\alpha ,w}(p:q)& ={\displaystyle \sum _{i=1}^n}{w}_{i}D((1-{\alpha }_i)p+{\alpha }_{i}q||(1-\overline{\alpha })p+\overline{\alpha }q)\hfill \\ \hfill & \le \mathcal{T}H(w).\hfill \end{array}$$

(35)

It remains to determine $\mathcal{T}={\max }_i{|f_i-{\tilde{f}}_i|}_{TV}$,

$$\begin{array}{cc}\hfill {\tilde{f}}_i-f_i& =\frac{f-f_i}{1-{\lambda }_i}\hfill \\ \hfill & =\frac{((1-\overline{\alpha })p+\overline{\alpha }q)-((1-{\alpha }_i)p+{\alpha }_{i}q)}{1-w_i}\hfill \\ \hfill & =\frac{({\alpha }_i-\overline{\alpha })(p-q)}{1-w_i}\hfill \\ \hfill & =({\alpha }_i-{\overline{\alpha }}_i)(p-q).\hfill \end{array}$$

(36)

Thus, $\mathcal{T}={\max }_i({\alpha }_i-{\overline{\alpha }}_i){|p-q|}_{TV}=\mathcal{A}{|p-q|}_{TV},$ and the proof of the upper bound is complete.

To prove the lower bound, we apply Pinsker’s inequality, ${2\log e|P-Q|}_{TV}^2\le D(P||Q),$

$$\begin{array}{cc}\hfill J{S}^{\alpha ,w}(p:q)& ={\displaystyle \sum _{i=1}^n}{w}_{i}D((1-{\alpha }_i)p+{\alpha }_{i}q||(1-\overline{\alpha })p+\overline{\alpha }q)\hfill \\ \hfill & \ge \frac{1}{2}{\displaystyle \sum _{i=1}^n}{w}_{i}2\log e{|((1-{\alpha }_i)p+{\alpha }_{i}q)-((1-\overline{\alpha })p+\overline{\alpha }q)|}_{TV}^2\hfill \\ \hfill & =\log e{\displaystyle \sum _{i=1}^n}{w}_i{({\alpha }_i-\overline{\alpha })}^2{|p-q|}_{TV}^2\hfill \\ \hfill & =\log e{\mathrm{Var}}_w(\alpha ){|p-q|}_{TV}^2.\hfill \end{array}$$

(37)

 □

Definition 7.

Given an f-divergence, densities p and q with respect to common reference measure, $\alpha \in {[0,1]}^n$ and $w\in {(0,1)}^n$ such that ${\sum }_i{w}_i=1$ define its generalized skew divergence

$$D_f^{\alpha ,w}(p:q)={\displaystyle \sum _{i=1}^n}{w}_i{D}_f((1-{\alpha }_i)p+{\alpha }_{i}q||(1-\overline{\alpha })p+\overline{\alpha }q).$$

(38)

where $\overline{\alpha }={\sum }_i{w}_i{\alpha }_i$.

Note that, by Theorem 4, $D_f^{\alpha ,w}$ is an f-divergence. The generalized skew divergence of the relative entropy is the generalized Jensen–Shannon divergence $J{S}^{\alpha ,w}$. We denote the generalized skew divergence of the ${\chi }^2$-divergence from p to q by

$${\chi }_{\alpha ,w}^2(p:q):=\sum _i{w}_i{\chi }^2((1-{\alpha }_i)p+{\alpha }_{i}q||(1-\overline{\alpha }p+\overline{\alpha }q)$$

(39)

Note that, when $n=2$ and ${\alpha }_1=0$, ${\alpha }_2=1$ and $w_i=\frac{1}{2}$, we recover the skew symmetrized divergence in Definition 4

$$D_f^{(0,1),(1/2,1/2)}(p:q)={\Delta }_f(p||q)$$

(40)

The following theorem shows that the usual upper bound for the relative entropy by the ${\chi }^2$-divergence can be reversed up to a factor in the skewed case.

Theorem 7.

For p and q with a common dominating measure μ,

$${\chi }_{\alpha ,w}^2(p:q)\le N_{\infty }(\alpha ,w)J{S}^{\alpha ,w}(p:q).$$

Writing $N_{\infty }(\alpha ,w)={\max }_i\max \left\{\frac{1-{\alpha }_i}{1-\overline{\alpha }},\frac{{\alpha }_i}{\overline{\alpha }}\right\}$. For $\alpha \in {[0,1]}^n$ and $w\in {(0,1)}^n$ such that ${\sum }_i{w}_i=1$, we use the notation $N_{\infty }(\alpha ,w):={\max }_i{e}^{D_{\infty }({\alpha }_i||\overline{\alpha })}$ where $\overline{\alpha }{\sum }_i{w}_i{\alpha }_i$.

Proof. 

By definition,

$$J{S}^{\alpha ,w}(p:q)={\displaystyle \sum _{i=1}^n}{w}_{i}D((1-{\alpha }_i)p+{\alpha }_{i}q||(1-\overline{\alpha })p+\overline{\alpha }q).$$

Taking $P_i$ to be the measure associated to $(1-{\alpha }_i)p+{\alpha }_{i}q$ and Q given by $(1-\overline{\alpha })p+\overline{\alpha }q$, then

$$\frac{d{P}_i}{dQ}=\frac{(1-{\alpha }_i)p+{\alpha }_{i}q}{(1-\overline{\alpha })p+\overline{\alpha }q}\le \max \left\{\frac{1-{\alpha }_i}{1-\overline{\alpha }},\frac{{\alpha }_i}{\overline{\alpha }}\right\}=e^{D_{\infty }({\alpha }_i||\overline{\alpha })}\le N_{\infty }(\alpha ,w).$$

(41)

Since $f(x)=x\log x$, the convex function associated to the usual KL divergence, satisfies $f^{″}(x)=\frac{1}{x}$, f is $e^{-D_{\infty }(\alpha )}$-convex on $[0,{\sup }_{x,i}\frac{d{P}_i}{dQ}(x)]$, applying Proposition 2, we obtain

$$D\left(\sum _i{w}_i{P}_i||Q\right)\le \sum _i{w}_{i}D(P_i||Q)-\frac{{\sum }_i{w}_i{\int }_{\mathcal{X}}{\left(\frac{d{P}_i}{dQ}-\frac{dP}{dQ}\right)}^{2}dQ}{2N_{\infty }(\alpha ,w)}.$$

(42)

Since $Q={\sum }_i{w}_i{P}_i$, the left hand side of (42) is zero, while

$$\begin{array}{cc}\hfill \sum _i{w}_i{\int }_{\mathcal{X}}{\left(\frac{d{P}_i}{dQ}-\frac{dP}{dQ}\right)}^{2}dQ& =\sum _i{w}_i{\int }_{\mathcal{X}}{\left(\frac{d{P}_i}{dP}-1\right)}^{2}dP\hfill \\ \hfill & =\sum _i{w}_i{\chi }^2(P_i||P)\hfill \\ \hfill & ={\chi }_{\alpha ,w}^2(p:q).\hfill \end{array}$$

(43)

Rearranging gives,

$$\frac{{\chi }_{\alpha ,w}^2(p:q)}{2N_{\infty }(\alpha ,w)}\le J{S}^{\alpha ,w}(p:q),$$

(44)

which is our conclusion. □

4. Total Variation Bounds and Bayes Risk

In this section, we derive bounds on the Bayes risk associated to a family of probability measures with a prior distribution $\lambda$. Let us state definitions and recall basic relationships. Given probability densities ${\left\{p_i\right\}}_{i=1}^n$ on a space $\mathcal{X}$ with respect a reference measure $\mu$ and ${\lambda }_i\ge 0$ such that ${\sum }_{i=1}^n{\lambda }_i=1$, define the Bayes risk,

$$R:=R_{\lambda }(p):=1-{\int }_{\mathcal{X}}\underset{i}{\max }\left\{{\lambda }_i{p}_i(x)\right\}d\mu (x)$$

(45)

If $\ell (x,y)=1-{\delta }_x(y)$, and we define $T:=(x)\arg {\max }_i{\lambda }_i{p}_i(x)$ then observe that this definition is consistent with, the usual definition of the Bayes risk associated to the loss function ℓ. Below, we consider $\theta$ to be a random variable on $\{1,2,\dots ,n\}$ such that $\mathbb{P}(\theta =i)={\lambda }_i$, and x to be a variable with conditional distribution $\mathbb{P}(X\in A|\theta =i)={\int }_A{p}_i(x)d\mu (x)$. The following result shows that the Bayes risk gives the probability of the categorization error, under an optimal estimator.

Proposition 5.

The Bayes risk satisfies

$$R=\underset{\widehat{\theta }}{\min }\mathbb{E}\ell (\theta ,\widehat{\theta }(X))=\mathbb{E}\ell (\theta ,T(X))$$

where the minimum is defined over $\widehat{\theta }:\mathcal{X}\to \{1,2,\dots ,n\}$.

Proof. 

Observe that $R=1-{\int }_{\mathcal{X}}{\lambda }_{T(x)}{p}_{T(x)}(x)d\mu (x)=\mathbb{E}\ell (\theta ,T(X))$. Similarly,

$$\begin{array}{cc}\hfill \mathbb{E}\ell (\theta ,\widehat{\theta }(X))& =1-{\int }_{\mathcal{X}}{\lambda }_{\widehat{\theta }(x)}{p}_{\widehat{\theta }(x)}(x)d\mu (x)\hfill \\ \hfill & \ge 1-{\int }_{\mathcal{X}}{\lambda }_{T(x)}{p}_{T(x)}(x)d\mu (x)=R,\hfill \end{array}$$

which gives our conclusion. □

It is known (see, for example, [9,31]) that the Bayes risk can also be tied directly to the total variation in the following special case, whose proof we include for completeness.

Proposition 6.

When $n=2$ and ${\lambda }_1={\lambda }_2=\frac{1}{2}$, the Bayes risk associated to the densities $p_1$ and $p_2$ satisfies

$$2R=1-|p_1-p_2{|}_{TV}$$

(46)

Proof. 

Since $p_T=\frac{|p_1-p_2|+p_1+p_2}{2}$, integrating gives ${\int }_{\mathcal{X}}{p}_T(x)d\mu (x)={|p_1-p_2|}_{TV}+1$ from which the equality follows. □

Information theoretic bounds to control the Bayes and minimax risk have an extensive literature (see, for example, [9,32,33,34,35]). Fano’s inequality is the seminal result in this direction, and we direct the reader to a survey of such techniques in statistical estimation (see [36]). What follows can be understood as a sharpening of the work of Guntuboyina [9] under the assumption of a $\kappa$-convexity.

The function $T(x)=\arg {\max }_i\left\{{\lambda }_i{p}_i(x)\right\}$ induces the following convex decompositions of our densities. The density q can be realized as a convex combination of $q_1=\frac{{\lambda }_{T}q}{1-Q}$ where $Q=1-\int {\lambda }_{T}qd\mu$ and $q_2=\frac{(1-{\lambda }_T)q}{Q}$,

$$q=(1-Q)q_1+Q{q}_2.$$

If we take $p{\sum }_i{\lambda }_i{p}_i$, then p can be decomposed as ${\rho }_1=\frac{{\lambda }_T{p}_T}{1-R}$ and ${\rho }_2=\frac{p-{\lambda }_T{p}_T}{R}$ so that

$$p=(1-R){\rho }_1+R{\rho }_2.$$

Theorem 8.

When f is κ-convex, on $(a,b)$ with $a={\inf }_{i,x}\frac{p_i(x)}{q(x)}$ and $b={\sup }_{i,x}\frac{p_i(x)}{q(x)}$

$$\sum _i{\lambda }_i{D}_f(p_i||q)\ge D_f(R||Q)+\frac{\kappa W}{2}$$

where

$$W:=W({\lambda }_i,p_i,q):=\frac{{(1-R)}^2}{1-Q}{\chi }^2({\rho }_1||q_1)+\frac{R^2}{Q}{\chi }^2({\rho }_2||q_2)+W_0$$

for $W_0\ge 0$.

$W_0$ can be expressed explicitly as

$$W_0=\int (1-{\lambda }_T)Va{r}_{{\lambda }_i\ne T}\left(\frac{p_i}{q}\right)d\mu =\int \sum _{i\ne T}{\lambda }_i\frac{|p_i-{\sum }_{j\ne T}\frac{{\lambda }_j}{1-{\lambda }_T}{p}_j{|}^2}{q}d\mu ,$$

where for fixed x, we consider the variance $Va{r}_{{\lambda }_i\ne T}\left(\frac{p_i}{q}\right)$ to be the variance of a random variable taking values $p_i(x)/q(x)$ with probability ${\lambda }_i/(1-{\lambda }_{T(x)})$ for $i\ne T(x)$. Note this term is a non-zero term only when $n>2$.

Proof. 

For a fixed x, we apply Lemma 1

$$\begin{array}{cc}\hfill \sum _i{\lambda }_{i}f\left(\frac{p_i}{q}\right)& ={\lambda }_{T}f\left(\frac{p_T}{q}\right)+(1-{\lambda }_T)\sum _{i\ne T}\frac{{\lambda }_i}{1-{\lambda }_T}f\left(\frac{p_i}{q}\right)\hfill \\ \hfill & \ge {\lambda }_{T}f\left(\frac{p_T}{q}\right)+(1-{\lambda }_T)\left[f\left(\frac{p-{\lambda }_T{p}_T}{q(1-{\lambda }_T)}\right)+\frac{\kappa }{2}{\mathrm{Var}}_{{\lambda }_{i\ne T}}\left(\frac{p_i}{q}\right)\right]\hfill \end{array}$$

(47)

Integrating,

$$\begin{array}{cc}\hfill \sum _i{\lambda }_i{D}_f(p_i||q)\ge \int {\lambda }_{T}f\left(\frac{p_T}{q}\right)q& +\int (1-{\lambda }_T)f\left(\frac{-{\lambda }_T{p}_T+{\sum }_i{\lambda }_i{p}_i}{q(1-{\lambda }_T)}\right)q+\frac{\kappa }{2}{W}_0,\hfill \end{array}$$

(48)

where

$$W_0=\int \sum _{i\ne T(x)}\frac{{\lambda }_i}{1-{\lambda }_T(x)}\frac{|p_i-{\sum }_{j\ne T}\frac{{\lambda }_j}{1-{\lambda }_T}{p}_j{|}^2}{q}d\mu .$$

(49)

Applying the $\kappa$-convexity of f,

$$\begin{array}{cc}\hfill \int {\lambda }_{T}f\left(\frac{p_T}{q}\right)q& =(1-Q)\int q_{1}f\left(\frac{p_T}{q}\right)\hfill \\ \hfill & \ge (1-Q)\left(f\left(\frac{\int {\lambda }_T{p}_T}{1-Q}\right)+\frac{\kappa }{2}{\mathrm{Var}}_{q_1}\left(\frac{p_T}{q}\right)\right)\hfill \\ \hfill & =(1-Q)f((1-R)/(1-Q))+\frac{Q\kappa }{2}{W}_1,\hfill \end{array}$$

(50)

with

$$\begin{array}{cc}\hfill W_1& :={\mathrm{Var}}_{q_1}\left(\frac{p_T}{q}\right)\hfill \\ \hfill & ={\left(\frac{1-R}{1-Q}\right)}^2{\mathrm{Var}}_{q_1}\left(\frac{{\lambda }_T{p}_T}{{\lambda }_{T}q}\frac{1-Q}{1-R}\right)\hfill \\ \hfill & ={\left(\frac{1-R}{1-Q}\right)}^2{\mathrm{Var}}_{q_1}\left(\frac{{\rho }_1}{q_1}\right)\hfill \\ \hfill & ={\left(\frac{1-R}{1-Q}\right)}^2{\chi }^2({\rho }_1||q_1)\hfill \end{array}$$

(51)

Similarly,

$$\begin{array}{cc}\hfill \int (1-{\lambda }_T)f\left(\frac{p-{\lambda }_T{p}_T}{q(1-{\lambda }_T)}\right)q& =Q\int q_{2}f\left(\frac{p-{\lambda }_T{p}_T}{q(1-{\lambda }_T)}\right)\hfill \\ \hfill & \ge Qf\left(\int q_2\frac{p-{\lambda }_T{p}_T}{q(1-{\lambda }_T)}\right)+\frac{Q\kappa }{2}{W}_2\hfill \\ \hfill & =Qf\left(\frac{R}{1-Q}\right)+\frac{Q\kappa }{2}{W}_2\hfill \end{array}$$

(52)

where

$$\begin{array}{cc}\hfill W_2& :={\mathrm{Var}}_{q_2}\left(\frac{p-{\lambda }_T{p}_T}{q(1-{\lambda }_T)}\right)\hfill \\ \hfill & ={\left(\frac{R}{Q}\right)}^2{\mathrm{Var}}_{q_2}\left(\frac{p-{\lambda }_T{p}_T}{q(1-{\lambda }_T)}\frac{Q}{R}\right)\hfill \\ \hfill & ={\left(\frac{R}{Q}\right)}^2{\mathrm{Var}}_{q_2}{\left(\frac{p-{\lambda }_T{p}_T}{q(1-{\lambda }_T)}-\frac{R}{Q}\right)}^2\hfill \\ \hfill & ={\left(\frac{R}{Q}\right)}^2\int q_2{\left(\frac{{\rho }_2}{q_2}-1\right)}^2\hfill \\ \hfill & ={\left(\frac{R}{Q}\right)}^2{\chi }^2({\rho }_2||q_2)\hfill \end{array}$$

(53)

Writing $W=W_0+W_1+W_2$, we have our result. □

Corollary 4.

When ${\lambda }_i=\frac{1}{n}$, and f is κ-convex on $({\inf }_{i,x}{p}_i/q,{\sup }_{i,x}{p}_i/q)$

$$\begin{array}{cc}\hfill \frac{1}{n}\sum _i& D_f(p_i||q)\hfill \\ \hfill & \ge D_f(R||(n-1)/n)+\frac{\kappa }{2}\left(n^2{(1-R)}^2{\chi }^2({\rho }_1||q)+{\left(\frac{nR}{n-1}\right)}^2{\chi }^2({\rho }_2||q)+W_0\right)\hfill \end{array}$$

(54)

further when $n=2$,

$$\begin{array}{cc}\hfill \frac{D_f(p_1||q)+D_f(p_2||q)}{2}\ge D_f& \left(\frac{1-|p_1-p_2{|}_{TV}}{2}||\frac{1}{2}\right)\hfill \\ \hfill & +\frac{\kappa }{2}\left((1+|p_1-p_2{|}_{TV}{)}^2{\chi }^2({\rho }_1||q)+(1-|p_1-p_2{|}_{TV}{)}^2{\chi }^2({\rho }_2||q)\right).\hfill \end{array}$$

(55)

Proof. 

Note that $q_1=q_2=q$, since ${\lambda }_i=\frac{1}{n}$ implies ${\lambda }_T=\frac{1}{n}$ as well. In addition, $Q=1-\int {\lambda }_{T}qd\mu =\frac{n-1}{n}$ so that applying Theorem 8 gives

$${\displaystyle \sum _{i=1}^n}{D}_f(p_i||q)\ge n{D}_f(R||(n-1)/n)+\frac{\kappa nW({\lambda }_i,p_i,q)}{2}.$$

(56)

The term W can be simplified as well. In the notation of the proof of Theorem 8,

$$\begin{array}{cc}\hfill W_1& =n^2{(1-R)}^2{\chi }^2({\rho }_1,q)\hfill \\ \hfill W_2& ={\left(\frac{nR}{n-1}\right)}^2{\chi }^2({\rho }_2||q)\hfill \\ \hfill W_0& =\int \frac{\frac{1}{n-1}{\sum }_{i\ne T}{(p_i-\frac{1}{n-1}{\sum }_{j\ne T}{p}_j)}^2}{q}d\mu .\hfill \end{array}$$

(57)

For the special case, one needs only to recall $R=\frac{1-|p_1-p_2{|}_{TV}}{2}$ while inserting 2 for n. □

Corollary 5.

When $p_i\le q/t^{\ast }$ for $t^{\ast }>0$, and $f(x)=x\log x$

$$\sum _i{\lambda }_{i}D(p_i||q)\ge D(R||Q)+\frac{t^{\ast }W({\lambda }_i,p_i,q)}{2}$$

for $D(p_i||q)$ the relative entropy. In particular,

$$\sum _i{\lambda }_{i}D(p_i||q)\ge D(p||q)+D(R||P)+\frac{t^{\ast }W({\lambda }_i,p_i,p)}{2}$$

where $P=1-\int {\lambda }_{T}pd\mu$ for $p={\sum }_i{\lambda }_i{p}_i$ and $t^{\ast }=\min {\lambda }_i$.

Proof. 

For the relative entropy, $f(x)=x\log x$ is $\frac{1}{M}$-convex on $[0,M]$ since $f^{″}(x)=1/x$. When $p_i\le q/t^{\ast }$ holds for all i, then we can apply Theorem 8 with $M=\frac{1}{t^{\ast }}$. For the second inequality, recall the compensation identity, ${\sum }_i{\lambda }_{i}D(p_i||q)={\sum }_i{\lambda }_{i}D(p_i||p)+D(p||q)$, and apply the first inequality to ${\sum }_{i}D(p_i||p)$ for the result.  □

This gives an upper bound on the Jensen–Shannon divergence, defined as $\mathrm{JSD}(\mu ||\nu )=\frac{1}{2}D(\mu ||\mu /2+\nu /2)+\frac{1}{2}D(\nu ||\mu /2+\nu /2)$. Let us also note that through the compensation identity ${\sum }_i{\lambda }_{i}D(p_i||q)={\sum }_i{\lambda }_{i}D(p_i||p)+D(p||q)$, ${\sum }_i{\lambda }_{i}D(p_i||q)\ge {\sum }_i{\lambda }_{i}D(p_i||p)$ where $p={\sum }_i{\lambda }_i{p}_i$. In the case that ${\lambda }_i=\frac{1}{N}$

$$\begin{array}{cc}\hfill \sum _i{\lambda }_i& D(p_i||q)\hfill \\ \hfill & \ge \sum _i{\lambda }_{i}D(p_i||p)\hfill \\ \hfill & \ge Qf\left(\frac{1-R}{Q}\right)+(1-Q)f\left(\frac{R}{1-Q}\right)+\frac{t^{\ast }W}{2}\hfill \end{array}$$

(58)

Corollary 6.

For two densities $p_1$ and $p_2$, the Jensen–Shannon divergence satisfies the following,

$$\begin{array}{cc}\hfill \mathrm{JSD}(p_1||p_2)\ge D& \left(\frac{1-|p_1-p_2{|}_{TV}}{2}\left|\right|1/2\right)\hfill \\ \hfill & +\frac{1}{4}\left((1+|p_1-p_2{|}_{TV}{)}^2{\chi }^2({\rho }_1||p)+(1-|p_1-p_2{|}_{TV}{)}^2{\chi }^2({\rho }_2||p)\right)\hfill \end{array}$$

(59)

with $\rho (i)$ defined above and $p=p_1/2+p_2/2$.

Proof. 

Since $\frac{p_i}{(p_1+p_2)/2}\le 2$ and $f(x)=x\log x$ satisfies $f^{″}(x)\ge \frac{1}{2}$ on $(0,2)$. Taking $q=\frac{p_1+p_2}{2}$, in the $n=2$ example of Corollary 4 with $\kappa =\frac{1}{2}$ yields the result. □

Note that $2D((1+V)/2||1/2)=(1+V)\log (1+V)+(1-V)\log (1-V)\ge V^2\log e$, we see that a further bound,

$$\mathrm{JSD}(p_1||p_2)\ge \frac{\log e}{2}{V}^2+\frac{{(1+V)}^2{\chi }^2({\rho }_1||p)+{(1-V)}^2{\chi }^2({\rho }_2||p)}{4},$$

(60)

can be obtained for $V=|p_1-p_2{|}_{TV}$.

On Topsøe’s Sharpening of Pinsker’s Inequality

For $P_i,Q$ probability measures with densities $p_i$ and q with respect to a common reference measure, ${\sum }_{i=1}^n{t}_i=1$, with $t_i>0$, denote $P={\sum }_i{t}_i{P}_i$, with density $p={\sum }_i{t}_i{p}_i$, the compensation identity is

$${\displaystyle \sum _{i=1}^n}{t}_{i}D(P_i||Q)=D(P||Q)+{\displaystyle \sum _{i=1}^n}{t}_{i}D(P_i||P).$$

(61)

Theorem 9.

For $P_1$ and $P_2$, denote $M_k=2^{-k}{P}_1+(1-2^{-k})P_2$, and define

$$\begin{array}{c}\hfill {\mathcal{M}}_1(k)=\frac{M_k{\mathbb{1}}_{\{P_1>P_2\}}+P_2{\mathbb{1}}_{\{P_1\le P_2\}}}{M_k\{P_1>P_2\}+P_2\{P_1\le P_2\}}{\mathcal{M}}_2(k)=\frac{M_k{\mathbb{1}}_{\{P_1\le P_2\}}+P_2{\mathbb{1}}_{\{P_1>P_2\}}}{M_k\{P_1\le P_2\}+P_2\{P_1>P_2\}},\end{array}$$

then the following sharpening of Pinsker’s inequality can be derived,

$$\begin{array}{c}\hfill D(P_1||P_2)\ge (2\log e)|P_1-P_2{|}_{TV}^2+{\displaystyle \sum _{k=0}^{\infty }}{2}^k\left(\frac{{\chi }^2({\mathcal{M}}_1(k),M_{k+1})}{2}+\frac{{\chi }^2({\mathcal{M}}_2(k),M_{k+1})}{2}\right).\end{array}$$

Proof. 

When $n=2$ and $t_1=t_2=\frac{1}{2}$, if we denote $M=\frac{P_1+P_2}{2}$, then (61) reads as

$$\frac{1}{2}D(P_1||Q)+\frac{1}{2}D(P_2||Q)=D(M||Q)+\mathrm{JSD}(P_1||P_2).$$

(62)

Taking $Q=P_2$, we arrive at

$$D(P_1||P_2)=2D(M||P_2)+2\mathrm{JSD}(P_1||P_2)$$

(63)

Iterating and writing $M_k=2^{-k}{P}_1+(1-2^{-k})P_2$, we have

$$D(P_1||P_2)=2^n\left(D(M_n||P_2)+2{\displaystyle \sum _{k=0}^n}\mathrm{JSD}(M_n||P_2)\right)$$

(64)

It can be shown (see [11]) that $2^{n}D(M_n||P_2)\to 0$ with $n\to \infty$, giving the following series representation,

$$D(P_1||P_2)=2{\displaystyle \sum _{k=0}^{\infty }}{2}^k\mathrm{JSD}(M_k||P_2).$$

(65)

Note that the $\rho$-decomposition of $M_k$ is exactly ${\rho }_i={\mathcal{M}}_k(i)$, thus, by Corollary 6,

$$\begin{array}{cc}\hfill D(P_1||P_2)& =2{\displaystyle \sum _{k=0}^{\infty }}{2}^k\mathrm{JSD}(M_k||P_2)\hfill \\ \hfill & \ge {\displaystyle \sum _{k=0}^{\infty }}{2}^k\left(|M_k-P_2{|}_{TV}^2\log e+\frac{{\chi }^2({\mathcal{M}}_1(k),M_{k+1})}{2}+\frac{{\chi }^2({\mathcal{M}}_2(k),M_{k+1})}{2}\right)\hfill \\ \hfill & =(2\log e)|P_1-P_2{|}_{TV}^2+{\displaystyle \sum _{k=0}^{\infty }}{2}^k\left(\frac{{\chi }^2({\mathcal{M}}_1(k),M_{k+1})}{2}+\frac{{\chi }^2({\mathcal{M}}_2(k),M_{k+1})}{2}\right).\hfill \end{array}$$

(66)

Thus, we arrive at the desired sharpening of Pinsker’s inequality. □

Observe that the $k=0$ term in the above series is equivalent to

$$2^0\left(\frac{{\chi }^2({\mathcal{M}}_1(0),M_{0+1})}{2}+\frac{{\chi }^2({\mathcal{M}}_2(0),M_{0+1})}{2}\right)=\frac{{\chi }^2({\rho }_1,p)}{2}+\frac{{\chi }^2({\rho }_2,p)}{2},$$

(67)

where ${\rho }_i$ is the convex decomposition of $p=\frac{p_1+p_2}{2}$ in terms of $T(x)=\arg \max \{p_1(x),p_2(x)\}$.

5. Conclusions

In this article, we begin a systematic study of strongly convex divergences, and how the strength of convexity of a divergence generator f, quantified by the parameter $\kappa$, influences the behavior of the divergence $D_f$. We prove that every strongly convex divergence dominates the square of the total variation, extending the classical bound provided by the ${\chi }^2$-divergence. We also study a general notion of a skew divergence, providing new bounds, in particular for the generalized skew divergence of Nielsen. Finally, we show how $\kappa$-convexity can be leveraged to yield improvements of Bayes risk f-divergence inequalities, and as a consequence achieve a sharpening of Pinsker’s inequality.