Bounds on the Excess Minimum Risk via Generalized Information Divergence Measures

Abstract

Given finite-dimensional random vectors Y, X, and Z that form a Markov chain in that order ($Y\to X\to Z$), we derive the upper bounds on the excess minimum risk using generalized information divergence measures. Here, Y is a target vector to be estimated from an observed feature vector X or its stochastically degraded version Z. The excess minimum risk is defined as the difference between the minimum expected loss in estimating Y from X and from Z. We present a family of bounds that generalize a prior bound based on mutual information, using the Rényi and $\alpha$-Jensen–Shannon divergences, as well as Sibson’s mutual information. Our bounds are similar to recently developed bounds for the generalization error of learning algorithms. However, unlike these works, our bounds do not require the sub-Gaussian parameter to be constant, and therefore, apply to a broader class of joint distributions over Y, X, and Z. We also provide numerical examples under both constant and non-constant sub-Gaussianity assumptions, illustrating that our generalized divergence-based bounds can be tighter than the ones based on mutual information for certain regimes of the parameter $\alpha$.

1. Introduction

The excess minimum risk in statistical inference quantifies the difference between the minimum expected loss attained by estimating a (target) hidden random vector from a feature (observed) random vector and the minimum expected loss incurred by estimating the hidden vector from a stochastically degraded version of the feature vector. The aim of this work is to derive upper bounds on the excess minimum risk in terms of generalized information divergence measures such as the Rényi divergence [1], the $\alpha$-Jensen–Shannon divergence [2,3] and the Sibson mutual information [4,5].

Recently, several bounds of this nature, expressed in terms of information-theoretic measures, have appeared in the literature, including [6,7,8,9,10,11,12,13,14,15,16,17] among others. Most of these works have focused on the (expected) generalization error of learning algorithms. In [6], Xu and Raginsky established bounds on the generalization error in terms of Shannon’s mutual information between the (input) training dataset and the (output) hypothesis; these bounds are tightened in [7] by using the mutual information between individual data samples and the hypothesis. In [11], Modak et al. extend these works by obtaining upper bounds on the generalization error in terms of the Rényi divergence, employing the variational characterization of the Rényi divergence [18,19,20]. The authors also derive bounds on the probability of generalization error via Rényi’s divergence, which recover the bounds of Esposito et al. [9] (see also [8,10] for bounds expressed in terms of the f-divergence [21]). More recently, Aminian et al. [17] obtained a family of bounds on the generalization error and excess risk applicable to supervised learning settings using a so-called “auxiliary distribution method.” In particular, they derive new bounds based on the $\alpha$-Jensen–Shannon and $\alpha$-Rényi mutual information measures. Here, both measures are defined via divergences between a joint distribution and a product of its marginals: the former using the Jensen–Shannon divergence of weight $\alpha$ [3] (Equation (4.1)) (which is always finite), and the latter using the Rényi divergence of order $\alpha$. Beyond learning-theoretic settings, Rényi divergence-based measures have also been successfully applied to classification problems, including time series and pattern classification, via belief and fractal extensions of the divergence [22,23,24]. In addition to information-theoretic approaches, generalization bounds based on PAC-Bayesian theory [25,26], particularly those involving f-divergences and Rényi-type divergences, have been actively studied. Separately, generalization bounds based on the Wasserstein distance [27] have also been established as an alternative approach based on optimal transport techniques. Connections between generalization error and transportation cost inequalities were explored in [28], recovering previous mutual information-based bounds and deriving a family of new bounds. A convex analytic approach is taken in [29], where information-theoretic measures of the dependence between input and output are replaced with arbitrary strongly convex functions of the input and output joint distribution. The resulting new generalization bounds either complement prior results or improve on these. Other works concerning the analysis of generalization error include [12,30] for deep learning generative adversarial networks [31] and [16] for the Gibbs algorithm (see also the extensive lists of references therein).

In this paper, we focus on the excess minimum risk in statistical inference. Our motivation is to generalize the results of Györfi et al. [14], who derived a mutual information-based upper bound that applies to a broad class of loss functions under standard sub-Gaussianity assumptions. Related but distinct work includes [13,15], where information-theoretic bounds on excess risk are developed in a Bayesian learning framework involving training data. Lower bounds on the Bayes risk in terms of information measures were recently developed in [32]. The contributions of our paper are as follows:

• We extend the bound in [14] by introducing a family of bounds based on generalized information divergence measures, namely, the Rényi divergence, the $\alpha$-Jensen–Shannon divergence, and the Sibson mutual information, parameterized by the order $\alpha \in (0,1)$. Unlike [11] and [17], where the sub-Gaussian parameter is assumed to be constant, our setup allows this parameter to depend on the (target) random vector being estimated. This makes our bounds applicable to a broader class of joint distributions over the random vectors involved.
• For the Rényi divergence based bounds, we adopt an approach similar to that of [11], deriving upper bounds by making use of the the variational representation of the Rényi divergence.
• For the bounds involving the $\alpha$-Jensen–Shannon divergence and the Sibson mutual information, we follow the methodology of [17], employing the auxiliary distribution method together with the variational representation of the Kullback–Leibler (KL) divergence [33].
• We provide simple conditions under which the $\alpha$-Jensen–Shannon divergence bound is tighter than the other two bounds for bounded loss functions.
• We compare the bounds based on the aforementioned information divergence measures with mutual information-based bounds by providing numerical examples.

Our problem of bounding the excess minimum risk is closely related to recent work on generalization error in learning theory. In both settings, the goal is to understand how much performance is lost when a target variable is estimated from a less informative or transformed version of the input. In learning theory, this is often studied through generalization bounds, which compare the performance of a learned predictor on training and test data. As already stated, several recent works have used information-theoretic tools—such as mutual information and its generalizations—to bound the generalization error (e.g., [6,11,17]). Although these works focus on algorithm-dependent error, the structure of the bounds is similar to ours. Our bounds, instead, are on the excess minimum risk, which compares the best possible performance using full observations versus using degraded ones. Still, both approaches rely on similar tools, including variational characterizations and divergence measures. In this sense, our work takes a different but related approach by studying the basic limits of inference, rather than how well a particular algorithm performs.

This paper is organized as follows. In Section 2, we provide preliminary definitions and introduce the statistical inference problem. In Section 3, we establish a family of upper bounds on the excess minimum risk, expressed in terms of the Rényi divergence, the $\alpha$-Jensen–Shannon divergence, and the Sibson mutual information, all parameterized by the order $\alpha \in (0,1)$. We also present several numerical examples, including cases with both constant and non-constant sub-Gaussian parameters, all of which demonstrate that the proposed bounds are tighter than the mutual information bound for a range of values of $\alpha$. Additionally, Section 3 includes an analytical comparison of the proposed bounds under bounded loss functions. In Section 5, we provide concluding remarks and suggest directions for future work.

2. Preliminaries

2.1. Problem Setup

Consider a random vector $Y\in {\mathbb{R}}^p$, $p\ge 1$, that is to be estimated (predicted) from a random observation vector X taking values in ${\mathbb{R}}^q$, $q\ge 1$. Given a measurable estimator (predictor) $f:{\mathbb{R}}^q\to {\mathbb{R}}^p$ and a loss function $l:{\mathbb{R}}^p\times {\mathbb{R}}^p\to {\mathbb{R}}^{+}$, the loss (risk) realized in estimating Y by $f\left(X\right)$ is given by $l(Y,f(X\left)\right)$. The minimum expected risk in predicting Y from X is defined by

$$L_l^{*}\left(Y\right|X)=\underset{f:{\mathbb{R}}^q\to {\mathbb{R}}^p}{inf}\mathbb{E}\left[l(Y,f\left(X\right))\right]$$

(1)

where the infimum is over all measurable f.

We also consider another random observation vector Z that is a random transformation or stochastically degraded version of X obtained, for example, by observing X through a noisy channel. Here, Z takes values in ${\mathbb{R}}^r$, $r\ge 1$, and Y, X and Z form a Markov chain in this order, which we denote as $Y\to X\to Z$. We similarly define the minimum expected risk in predicting Y from Z as

$$L_l^{*}\left(Y\right|Z)=\underset{g:{\mathbb{R}}^r\to {\mathbb{R}}^p}{inf}\mathbb{E}\left[l(Y,g\left(Z\right))\right],$$

(2)

where the infimum is over all measurable predictors g. With the notation introduced above, we define the excess minimum risk as the difference $L_l^{*}\left(Y\right|Z)-L_l^{*}\left(Y\right|X)$, which is always non-negative due to the Markov chain condition $Y\to X\to Z$ (e.g., see the data processing inequality for expected risk in [13] (Lemma 1)). Our objective is to establish upper bounds to this difference using generalized information divergence measures.

In [14], the random vector Z is taken as $T\left(X\right)$, a transformation of the random vector X, where $T:{\mathbb{R}}^p\to {\mathbb{R}}^r$ is measurable. The authors derive bounds on the excess minimum risk using Shannon’s mutual information. Here, we generalize these bounds by employing a family of information-divergence measures of order $\alpha \in (0,1)$, which recover Shannon’s mutual information in the limits $\alpha \to 0$ or $\alpha \to 1$. Furthermore, we use an arbitrary random vector Z, as the degraded version of the observation X instead of $T\left(X\right)$. We also provide examples where the various generalized information divergence-based bounds perform better than the mutual information-based bounds of [14] for a certain range $\alpha$.

We next state some definitions that we will invoke in deriving our results.

2.2. Definitions

Consider two arbitrary jointly distributed random variables U and V defined on the same probability space $(\mathsf{\Omega },\mathcal{M})$ and taking values in $\mathcal{U}$ and $\mathcal{V}$, respectively. Let $P_U$ and $P_V$ be the marginal distributions of U and V, respectively, and $P_{U,V}$ be a joint distribution over $\mathcal{U}\times \mathcal{V}$. We first provide definitions for the Rényi divergence-based measures.

Definition 1 

([1,34]). The Rényi divergence of order $\alpha \in (0,\infty )$, $\alpha \ne 1$, between the two probability measures $P_U$ and $P_V$ is denoted by $D_{\alpha }(P_U\parallel P_V)$ and defined as follows. Let ν be a sigma-finite positive measure such that $P_U$ and $P_V$ are absolutely continuous with respect to ν, written as $P_U,P_V\ll \nu$, with Radon–Nikodym derivatives ${\displaystyle \frac{d{P}_U}{d\nu }}=p_U$ and ${\displaystyle \frac{d{P}_V}{d\nu }}=p_V$, respectively. Then

$$\begin{array}{cc}\hfill & D_{\alpha }(P_U\parallel P_V)=\left\{\begin{array}{cc}{\displaystyle \frac{1}{\alpha -1}}log\left[\int {\left(p_U\right)}^{\alpha }{\left(p_V\right)}^{1-\alpha }d\nu \right]\hfill & if0<\alpha <1or\alpha >1and{P}_U\ll P_V\hfill \\ +\infty \hfill & if\alpha >1and{P}_U/\ll P_V.\hfill \end{array}\right.\hfill \end{array}$$

Definition 2. 

The conditional Rényi divergence of order α between the conditional distributions $P_{V|U}$ and $Q_{V|U}$ given $P_U$ is denoted by $D_{\alpha }(P_{V|U}\parallel Q_{V|U}\left|P_U\right)$ and given by

$$D_{\alpha }(P_{V|U}\parallel Q_{V|U}\left|P_U\right)={\mathbb{E}}_{P_U}\left[D_{\alpha }(P_{V|U}\left(·\right|U)\parallel Q_{V|U}\left(·\right|U)))\right],$$

(3)

where ${\mathbb{E}}_{P_U}\left[·\right]$ denotes expectation with respect to distribution $P_U$.

Note that the above definition of conditional Rényi divergence differs from the somewhat standard one, which is given as $D_{\alpha }(P_{V|U}{P}_U\parallel Q_{V|U}{P}_U)$, e.g., see [35] (Definition 3). We adopt the above definition because it is well-tailored to our setting, which allows sub-Gaussianity parameters to be random. However, as $\alpha \to 1$, both notions of the conditional Rényi divergence recover the conditional KL divergence, which is

$$D_{\mathit{KL}}(P_{V|U}\parallel Q_{V|U}\left|P_U\right)=D_{\mathit{KL}}(P_{V|U}{P}_U\parallel Q_{V|U}{P}_U)={\mathbb{E}}_{P_U}\left[\int p_{V|U}log\left[{\displaystyle \frac{p_{V|U}}{q_{V|U}}}\right]dv\right].$$

We next provide the definitions for the $\alpha$-Jensen–Shannon divergence-based measures.

Definition 3 

([2,3]). The α-Jensen–Shannon divergence for $\alpha \in (0,1)$ between two probability measures $P_U$ and $P_V$ on a measurable space $(\mathsf{\Omega },\mathcal{M})$ is denoted by $J{S}_{\alpha }(P_U\parallel P_V)$ and given by

$$J{S}_{\alpha }(P_U\parallel P_V)=\alpha D_{\mathit{KL}}(P_U\parallel \alpha P_U+(1-\alpha )P_V)+(1-\alpha )D_{\mathit{KL}}(P_V\parallel \alpha P_U+(1-\alpha )P_V),$$

(4)

where $D_{\mathit{KL}}(·\parallel ·)$ is the KL divergence.

Definition 4. 

The conditional α-Jensen–Shannon divergence between the conditional distributions $P_{V|U}$ and $Q_{V|U}$ given $P_U$ is denoted by $J{S}_{\alpha }(P_{V|U}\parallel Q_{V|U}\left|P_U\right)$ and given by

$$J{S}_{\alpha }(P_{V|U}\parallel Q_{V|U}\left|P_U\right)={\mathbb{E}}_{P_U}\left[J{S}_{\alpha }(P_{V|U}\left(·\right|U)\parallel Q_{V|U}\left(·\right|U))\right],$$

(5)

where ${\mathbb{E}}_{P_U}\left[·\right]$ denotes expectation with respect to distribution $P_U$.

Next we define the Sibson mutual information (of order $\alpha$).

Definition 5 

([4,5]). Let $\alpha \in (0,1)\cup (1,\infty )$. The Sibson mutual information of order α between V and U is denoted by $I_{\alpha }^S(V;U)$ and given by

$$I_{\alpha }^S(V;U)=\underset{Q_U\in \mathcal{P}\left(\mathcal{U}\right)}{min}{D}_{\alpha }(P_{U,V}\parallel Q_U{P}_V),$$

(6)

where $\mathcal{P}\left(\mathcal{U}\right)$ denotes the set of probability distributions on $\mathcal{U}$.

It is known that $D_{\alpha }(P_{U,V}\parallel Q_U{P}_V)$ is convex in $Q_U$ [34], which allows for a closed-form expression for the minimizer and, consequently, for the Sibson mutual information $I_{\alpha }^S(V;U)$ [9,36]. Let $U^{*}$ denote a random variable whose distribution achieves the minimum, with the corresponding distribution $P_{U^{*}}$. Then, the Sibson mutual information of order $\alpha$ can equivalently be written as follows.

Definition 6 

([9]). Let ν be a sigma-finite positive measure such that $P_{U,V}$ and $P_U{P}_V$ are absolutely continuous with respect to $\nu \times \nu$, written as $P_{U,V},P_U{P}_V\ll \nu \times \nu$, with Radon–Nikodym derivatives(densities) ${\displaystyle \frac{d{P}_{U,V}}{d(\nu \times \nu )}}=p_{U,V}$ and ${\displaystyle \frac{d\left(P_U{P}_V\right)}{d(\nu \times \nu )}}=p_U{p}_V$, respectively. For $\alpha \in (0,1)\cup (1,\infty )$, the Sibson mutual information of order α between V and U can be written as follows:

$$I_{\alpha }^S(V;U)=D_{\alpha }(P_{U,V}\parallel P_{U^{*}}{P}_V),$$

(7)

where the distribution $P_{U^{*}}$ has density

$$p_{U^{*}}\left(u\right)={\displaystyle \frac{d{P}_{U^{*}}}{d\nu }}\left(u\right)={\displaystyle \frac{{\left({\displaystyle \int {\left(\frac{p_{U,V}(u,v)}{p_U\left(u\right)p_V\left(v\right)}\right)}^{\alpha }{p}_V\left(v\right)dv}\right)}^{\frac{1}{\alpha }}}{{\displaystyle \int {\left(\int {\left(\frac{p_{U,V}(u^{\prime },v^{\prime })}{p_U\left(u^{\prime }\right)p_V\left(v^{\prime }\right)}\right)}^{\alpha }{p}_V\left(v^{\prime }\right)d{v}^{\prime }\right)}^{\frac{1}{\alpha }}{p}_U\left(u^{\prime }\right)d{u}^{\prime }}}}{p}_U\left(u\right).$$

(8)

Remark 1. 

From Definition 6, we note that the Sibson mutual information of order α is a functional of the distributions $P_{U,V}$ and $P_{U^{*}}$. Hence, from this point onward, we denote with a slight abuse of notation the Sibson mutual information of order α between V and U by $I_{\alpha }^S(P_{U,V},P_{U^{*}})$.

We end this section with the definitions of the sub-Gaussian and conditional sub-Gaussian properties.

Definition 7. 

A real random variable U with finite expectation is said to be ${\sigma }^2$-sub-Gaussian for some ${\sigma }^2>0$ if

$$log\mathbb{E}\left[e^{\lambda (U-\mathbb{E}[U\left]\right)}\right]\le {\displaystyle \frac{{\sigma }^2{\lambda }^2}{2}}$$

(9)

for all $\lambda \in \mathbb{R}$.

Definition 8. 

A real random variable U is said to be conditionally ${\sigma }^2$-sub-Gaussian given another random variable V (i.e., under $P_{U|V}$) for some ${\sigma }^2>0$ if we have almost surely that

$$log\mathbb{E}\left[e^{\lambda (U-\mathbb{E}[U\left|V\right])}\right|V]\le {\displaystyle \frac{{\sigma }^2{\lambda }^2}{2}}$$

(10)

for all $\lambda \in \mathbb{R}$.

Throughout the paper, we omit stating explicitly that the conditional sub-Gaussian inequality holds almost surely for the sake of simplicity.

3. Bounding Excess Minimum Risk

In this section, we establish a series of bounds on the excess minimum risk based on different information-theoretic measures. Our approach combines the variational characterizations of the KL divergence [33], the Rényi divergence [18], and the Sibson mutual information [9] (Theorem 2), along with the auxiliary distribution method introduced in [17].

3.1. Rényi Divergence-Based Upper Bound

We first state the variational characterization of the Rényi divergence [18], which generalizes the Donsker–Varadhan variational formula for KL divergence [33].

Lemma 1 

([18] (Theorem 3.1)). Let P and Q be two probability measures on $(\mathsf{\Omega },\mathcal{M})$ and $\alpha \in (0,\infty )$, $\alpha \ne 1$. Let g be a measurable function such that $e^{(\alpha -1)g}\in {\mathcal{L}}^1\left(P\right)$ and $e^{\alpha g}\in {\mathcal{L}}^1\left(Q\right)$, where ${\mathcal{L}}^1\left(\mu \right)$ denotes the collection of all measurable functions with finite ${\mathcal{L}}^1$-norm. Then,

$$D_{\alpha }(P\parallel Q)\ge {\displaystyle \frac{\alpha }{\alpha -1}}log{\mathbb{E}}_P\left[e^{(\alpha -1)g\left(X\right)}\right]-log{\mathbb{E}}_Q\left[e^{\alpha g\left(X\right)}\right].$$

(11)

We next provide the following lemma, whose proof is a slight generalization of [11] (Lemma 2) and [14] (Lemma 1).

Lemma 2. 

Consider two arbitrary jointly distributed random variables U and V defined on the same probability and taking values in spaces $\mathcal{U}$ and $\mathcal{V}$, respectively. Given a measurable function $h:\mathcal{U}\times \mathcal{V}\to \mathbb{R}$, assume that $h(u,V)$ is ${\sigma }^2\left(u\right)$-sub-Gaussian under $P_V$ and $P_{V|U=u}$ for all $u\in \mathcal{U}$, where $\mathbb{E}\left[{\sigma }^2\left(U\right)\right]<\infty$. Then for $\alpha \in (0,1)$,

$$\begin{array}{cc}\hfill \left|\mathbb{E}\right[h(U,V)]-& \mathbb{E}\left[h(\overline{U},\overline{V})\right]|\le \sqrt{2\mathbb{E}\left[{\sigma }^2\left(U\right)\right]{\displaystyle \frac{D_{\alpha }(P_{V|U}\parallel P_V\left|P_U\right)}{\alpha }}},\hfill \end{array}$$

where $\overline{U}$ and $\overline{V}$ are independent copies of U and V, respectively, (i.e., $P_{\overline{U},\overline{V}}=P_U{P}_V$).

Proof. 

By the sub-Gaussian property, we have that

$$log\mathbb{E}\left[e^{(\alpha -1)\lambda h(u,V)-\mathbb{E}\left[\right(\alpha -1\left)\lambda h\right(u,V\left)\right|U=u]}|U=u\right]\le {\displaystyle \frac{{\lambda }^2{(\alpha -1)}^2{\sigma }^2\left(u\right)}{2}}$$

(12)

and

$$log\mathbb{E}\left[e^{\alpha \lambda h(u,V)-\mathbb{E}\left[\alpha \lambda h\right(u,V\left)\right]}\right]\le {\displaystyle \frac{{\lambda }^2{\alpha }^2{\sigma }^2\left(u\right)}{2}}.$$

(13)

Re-arranging the terms gives us

$$-log\mathbb{E}\left[e^{(\alpha -1)\lambda h(u,V)}|U=u\right]\ge -{\displaystyle \frac{{\lambda }^2{(\alpha -1)}^2{\sigma }^2\left(u\right)}{2}}+\mathbb{E}\left[(1-\alpha )\lambda h(u,V)\right|U=u]$$

(14)

and

$$-log\mathbb{E}\left[e^{\alpha \lambda h(u,V)}\right]\ge -{\displaystyle \frac{{\lambda }^2{\alpha }^2{\sigma }^2\left(u\right)}{2}}-\mathbb{E}\left[\alpha \lambda h(u,V)\right].$$

(15)

Note that by (12) and (13), $e^{(\alpha -1)\lambda h(u,V)}\in {\mathcal{L}}^1\left(P_{V|U=u}\right)$ and $e^{\alpha \lambda h(u,V)}\in {\mathcal{L}}^1\left(P_V\right)$. By the variational formula in (11), we have that

$$\begin{array}{c}\hfill D_{\alpha }(P_{V|U=u}\parallel P_V)\ge {\displaystyle \frac{\alpha }{\alpha -1}}log\mathbb{E}\left[e^{(\alpha -1)\lambda h(u,V)}\right|U=u]-log\mathbb{E}\left[e^{\alpha \lambda h(u,V)}\right].\end{array}$$

(16)

Substituting (14) and (15) in (16) yields

$$\begin{array}{cc}\hfill D_{\alpha }(P_{V|U=u}\parallel P_V)& \ge {\displaystyle \frac{\alpha }{1-\alpha }}\left(-{\displaystyle \frac{{\lambda }^2{(\alpha -1)}^2{\sigma }^2\left(u\right)}{2}}+\mathbb{E}\left[(1-\alpha )\lambda h(u,V)\right|U=u]\right)\hfill \\ \hfill & -{\displaystyle \frac{{\lambda }^2{\alpha }^2{\sigma }^2\left(u\right)}{2}}-\mathbb{E}\left[\alpha \lambda h(u,V)\right]\hfill \\ \hfill & =\alpha \lambda (\mathbb{E}\left[h(u,V)\right|U=u]-\mathbb{E}\left[h(u,V)\right])-{\displaystyle \frac{{\lambda }^2\alpha (1-\alpha ){\sigma }^2\left(u\right)}{2}}\hfill \\ \hfill & -{\displaystyle \frac{{\lambda }^2{\alpha }^2{\sigma }^2\left(u\right)}{2}}\hfill \\ \hfill & =\alpha \lambda (\mathbb{E}\left[h(u,V)\right|U=u]-\mathbb{E}\left[h(u,V)\right])-{\displaystyle \frac{{\lambda }^2\alpha {\sigma }^2\left(u\right)}{2}}.\hfill \end{array}$$

The left-hand side of the resulting inequality

$$\begin{array}{c}\hfill {\displaystyle \frac{{\lambda }^2\alpha {\sigma }^2\left(u\right)}{2}}-\alpha \lambda (\mathbb{E}\left[h(u,V)\right|U=u]-\mathbb{E}\left[h(u,V)\right])+D_{\alpha }(P_{V|U=u}\parallel P_V)\ge 0\end{array}$$

is a non-negative quadratic polynomial in $\lambda$. Thus, the discriminant is non-positive and we have

$$\begin{array}{cc}\hfill & {\left(\alpha (\mathbb{E}\left[h(u,V)\right|U=u]-\mathbb{E}\left[h(u,V)\right])\right)}^2\le 4\left({\displaystyle \frac{\alpha {\sigma }^2\left(u\right)}{2}}\right)D_{\alpha }(P_{V|U=u}\parallel P_V).\hfill \end{array}$$

Therefore,

$$\begin{array}{cc}\hfill & |\mathbb{E}\left[h(u,V)\right|U=u]-\mathbb{E}\left[h(u,V)\right]|\le \sqrt{{\displaystyle \frac{2{\sigma }^2\left(u\right)D_{\alpha }(P_{V|U=u}\parallel P_V)}{\alpha }}}.\hfill \end{array}$$

(17)

Since $\overline{U}$ and $\overline{V}$ are independent and $P_{\overline{V}}=P_V$, we have that

$$\mathbb{E}\left[h(u,V)\right]=\mathbb{E}\left[h(\overline{U},\overline{V})\right|\overline{U}=u].$$

Therefore, we have

$$\begin{array}{cc}\hfill |\mathbb{E}\left[h(U,V)\right]-\mathbb{E}\left[h(\overline{U},\overline{V})\right]|& =\left|\int (\mathbb{E}\left[h(U,V)\right|U=u]-\mathbb{E}\left[h(\overline{U},\overline{V})\right|\overline{U}=u])P_U\left(du\right)\right|\hfill \\ \hfill & =\left|\int (\mathbb{E}\left[h(u,V)\right|U=u]-\mathbb{E}\left[h(u,V)\right])P_U\left(du\right)\right|\hfill \\ \hfill & \le \int \left|\left(\mathbb{E}\right[h(u,V)|U=u]-\mathbb{E}\left[h\right(u,V\left)\right]\right|P_U\left(du\right)\hfill \end{array}$$

(18)

$$\begin{array}{cc}\hfill & \le \int \sqrt{{\displaystyle \frac{2{\sigma }^2\left(u\right)D_{\alpha }(P_{V|U=u}\parallel P_V)}{\alpha }}}{P}_U\left(du\right)\hfill \end{array}$$

(19)

$$\begin{array}{cc}\hfill & \le \sqrt{\int 2{\sigma }^2\left(u\right)P_U\left(du\right)}\sqrt{\int {\displaystyle \frac{D_{\alpha }(P_{V|U=u}\parallel P_V)}{\alpha }}{P}_U\left(du\right)}\hfill \end{array}$$

(20)

$$\begin{array}{cc}\hfill & =\sqrt{2\mathbb{E}\left[{\sigma }^2\left(U\right)\right]{\displaystyle \frac{D_{\alpha }(P_{V|U}\parallel P_V\left|P_U\right)}{\alpha }}},\hfill \end{array}$$

(21)

where (18) follows from Jensen’s inequality, (19) follows from (17), (20) follows from the Cauchy–Schwarz inequality and the definition of conditional Rényi divergence in (3) with $D_{\alpha }(P_{V|U}\parallel P_V\left|P_U\right)={\mathbb{E}}_U\left[D_{\alpha }(P_{V|U}\left(·\right|U)\parallel P_V)\right]$. □

Note that the Rényi divergence-based bound in Lemma 2 differs from that in [17] (Theorem 3). In our approach, we consider sub-Gaussianity under both $P_V$ and $P_{V|U=u}$ for all $u\in \mathcal{U}$, which allows for non-constant sub-Gaussian parameters. This leads to a more general bound that applies to a broader class of loss functions.

We next use Lemma 2 to derive our theorem for the Rényi divergence-based bound; its proof is an adaptation of [14] (Theorem 3).

Theorem 1. 

Let X, Y and Z be random vectors such that $Y\to X\to Z$, as described in Section 2.1. Assume that there exists an optimal estimator f of Y from X such that $l(y,f(X\left)\right)$ is conditionally ${\sigma }^2\left(y\right)$-sub-Gaussian under $P_{X|Z}$ and $P_{X|Z,Y=y}$ for all $y\in {\mathbb{R}}^p$, i.e.,

$$log\mathbb{E}\left[e^{\left(\lambda \right(l(y,f(X\left)\right))-\mathbb{E}[l(y,f(X\left)\right)\left|Z\right])}\right|Z]\le {\displaystyle \frac{{\sigma }^2\left(y\right){\lambda }^2}{2}}$$

and

$$log\mathbb{E}\left[e^{\left(\lambda \right(l(y,f(X\left)\right))-\mathbb{E}[l(y,f(X\left)\right)|Z,Y=y])}\right|Z,Y=y]\le {\displaystyle \frac{{\sigma }^2\left(y\right){\lambda }^2}{2}}$$

for all $\lambda \in \mathbb{R}$ and $y\in \mathbb{R}$, where ${\sigma }^2:\mathbb{R}\to \mathbb{R}$, satisfies $\mathbb{E}\left[{\sigma }^2\left(Y\right)\right]<\infty$. Then for $\alpha \in (0,1)$, the excess minimum risk satisfies

$$\begin{array}{cc}\hfill & L_l^{*}\left(Y\right|Z)-L_l^{*}\left(Y\right|X)\le \sqrt{{\displaystyle \frac{2\mathbb{E}\left[{\sigma }^2\left(Y\right)\right]}{\alpha }}{D}_{\alpha }(P_{X|Y,Z}\parallel P_{X|Z}\left|P_{Y,Z}\right)}.\hfill \end{array}$$

(22)

Proof of Theorem 1. 

Let $\overline{X}$, $\overline{Y}$ and $\overline{Z}$ be random variables such that $P_{\overline{Y}|\overline{Z}}=P_{Y|Z}$, $P_{\overline{X}|\overline{Z}}=P_{X|Z}$, $P_{\overline{Z}}=P_Z$ and $\overline{Y}$ and $\overline{X}$ are conditionally independent given $\overline{Z}$, i.e., $P_{\overline{Y},\overline{X},\overline{Z}}=P_{Y|Z}{P}_{X|Z}{P}_Z$.

We apply Lemma 2 by setting $U=Y$, $V=X$ and $h(u,v)=l(y,f(x\left)\right)$. Consider $\mathbb{E}\left[l\right(Y,f\left(X\right)\left)\right|Z=z\left]\right)$ and $\mathbb{E}\left[l(\overline{Y},f\left(\overline{X}\right))\right|Z=z]$ as regular expectations taken with respect to $P_{Y,X|Z=z}$ and $P_{\overline{Y},\overline{X}|Z=z}$. Since $\overline{Y}$ and $\overline{X}$ are conditionally independent given $\overline{Z}=z$ and $P_{\overline{Z}}=P_Z$, we have that

$$\begin{array}{c}|\mathbb{E}\left[l(Y,f^{\prime }\left(X\right))\right|Z=z])-\mathbb{E}\left[l(\overline{Y},f\left(\overline{X}\right))\right|Z=z]\left)\right|\hfill \\ \le \sqrt{{\displaystyle \frac{2\mathbb{E}\left[{\sigma }^2\left(Y\right)\right|Z=z]}{\alpha }}{D}_{\alpha }(P_{X|Y,Z=z}\parallel P_{X|Z=z}|P_{Y|Z=z})}.\hfill \end{array}$$

(23)

Now,

$$\begin{array}{c}\left|\mathbb{E}\left[l(Y,f(X))\right]-\mathbb{E}\left[l(\overline{Y},f(\overline{X}))\right]\right|\hfill \\ \hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em} \le \int \left|\mathbb{E}\left[l(Y,f(X))\right|Z=z])-\mathbb{E}\left[l(\overline{Y},f(\overline{X}))\right|Z=z])\right|P_Z(dz)\hfill \\ \hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em} \le \int (\sqrt{{\displaystyle \frac{2\mathbb{E}\left[{\sigma }^2(Y)\right|Z=z]}{\alpha }}}\hfill \\ \hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em} \times \sqrt{D_{\alpha }(P_{X|Y,Z=z}\parallel P_{X|Z=z}|P_{Y|Z=z})})P_Z(dz)\hfill \\ \hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em} \le \sqrt{2\int \mathbb{E}\left[{\sigma }^2(Y)\right|Z=z]P_Z(dz)}\hfill \\ \hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em} \times \sqrt{\int {\displaystyle \frac{D_{\alpha }(P_{X|Y,Z=z}\parallel P_{X|Z=z}|P_{Y|Z=z})}{\alpha }}{P}_Z(dz)}\hfill \\ \hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em} =\sqrt{{\displaystyle \frac{2\mathbb{E}\left[{\sigma }^2(Y)\right]}{\alpha }}{D}_{\alpha }(P_{X|Y,Z}\parallel P_{X|Z}|P_{Y,Z})},\hfill \end{array}$$

(24)

where the first inequality follows from Jensen’s inequality and since $P_{\overline{Z}}=P_Z$, the second inequality follows from (23), the third from the Cauchy–Schwarz inequality, and the equality follows from (3). Since $\overline{Y}$ and $\overline{X}$ are conditionally independent given $\overline{Z}$, we obtain the Markov chain $\overline{Y}\to \overline{Z}\to \overline{X}$. Then, we have

$$\begin{array}{cc}\hfill \mathbb{E}\left[l(\overline{Y},f\left(\overline{X}\right))\right])& \ge L_l^{*}\left(\overline{Y}\right|\overline{X})\hfill \\ \hfill & \ge L_l^{*}\left(\overline{Y}\right|\overline{Z})\hfill \\ \hfill & =L_l^{*}\left(Y\right|Z),\hfill \end{array}$$

(25)

where the first inequality follows since $\overline{Y}\to \overline{X}\to f\left(\overline{X}\right)$, the second inequality holds since $\overline{Y}\to \overline{Z}\to \overline{X}$ by construction, and the equality follows since $(\overline{Y},\overline{Z})$ and $(Y,Z)$ have the same distribution by construction. Since f is an optimal estimator of Y from X, we also have

$$\mathbb{E}\left[l(Y,f\left(X\right))\right])=L_l^{*}\left(Y\right|X).$$

(26)

Therefore, using (25) and (26) in (24) combined with the fact that $L_l^{*}\left(Y\right|Z)\ge L_l^{*}\left(Y\right|X)$, we arrive at the desired inequality:

$$\begin{array}{c}\hfill L_l^{*}\left(Y\right|Z)-L_l^{*}\left(Y\right|X)\le \sqrt{{\displaystyle \frac{2\mathbb{E}\left[{\sigma }^2\left(Y\right)\right]}{\alpha }}{D}_{\alpha }(P_{X|Y,Z}\parallel P_{X|Z}\left|P_{Y,Z}\right)}.\end{array}$$

□

Remark 2. 

Taking the limit as $\alpha \to 1$ of the right-hand side of (22) in Theorem 1, we have that

$$\begin{array}{cc}\hfill L_l^{*}\left(Y\right|Z)-L_l^{*}\left(Y\right|X)& \le \sqrt{2\mathbb{E}\left[{\sigma }^2\left(Y\right)\right]D_{\mathit{KL}}(P_{X|Y,Z}\parallel P_{X|Z}\left|P_{Y,Z}\right)}\hfill \\ \hfill & =\sqrt{2\mathbb{E}\left[{\sigma }^2\left(Y\right)\right](I(X;Y)-I(Z;Y))},\hfill \end{array}$$

(27)

recovering the bound in [14] (Theorem 3).

As a special case, we consider bounded loss functions, which naturally satisfy the conditional sub-Gaussian condition. The following corollary is an application of Theorem 1 under a fixed sub-Gaussian parameter. For completeness, we include the full proof.

Corollary 1. 

Suppose the loss function l is bounded, i.e., ${\parallel l\parallel }_{\infty }={sup}_{y,y^{\prime }}l(y,y^{\prime })<\infty$. Then for random vectors X, Y and Z such that $Y\to X\to Z$ as described in Section 2.1, we have the following inequality for $\alpha \in (0,1)$ on the excess minimum risk:

$$L_l^{*}\left(Y\right|Z)-L_l^{*}\left(Y\right|X)\le {\displaystyle \frac{{\parallel l\parallel }_{\infty }}{\sqrt{2}}}\sqrt{{\displaystyle \frac{D_{\alpha }(P_{X|Y,Z}\parallel P_{X|Z}\left|P_{Y,Z}\right)}{\alpha }}}.$$

(28)

Proof. 

We show that the bounded loss function l satisfies the conditional sub-Gaussian properties of Theorem 1. Since l is bounded we have that for any $f:{\mathbb{R}}^q\to {\mathbb{R}}^p$, $x\in {\mathbb{R}}^q$ and $y\in {\mathbb{R}}^p$, $l(y,f\left(x\right))\in [0,\parallel l{\parallel }_{\infty }]$. Then, by Hoeffding’s lemma [37], we can write

$$log\mathbb{E}\left[e^{\left(\lambda \right(l(y,f(X\left)\right))}\right|Z]\le \mathbb{E}\left[\lambda l(y,f\left(X\right))\right|Z])+{\displaystyle \frac{{\parallel l\parallel }_{\infty }^2{\lambda }^2}{8}}$$

and

$$\begin{array}{cc}\hfill & log\mathbb{E}\left[e^{\left(\lambda \right(l(y,f(X\left)\right))}\right|Z,Y=y]\le \mathbb{E}\left[\lambda l(y,f\left(X\right))\right|Z,Y=y])+{\displaystyle \frac{{\parallel l\parallel }_{\infty }^2{\lambda }^2}{8}}\hfill \end{array}$$

for all $\lambda \in \mathbb{R}$ and $y\in \mathbb{R}$. Rearranging the above inequalities gives us that $l(y,f(X\left)\right)$ is conditionally ${\displaystyle \frac{{\parallel l\parallel }_{\infty }^2}{4}}$-sub-Gaussian under both $P_{X|Z}$ and $P_{X|Z,Y=y}$ for all $y\in {\mathbb{R}}^p$. Then by (22), we have

$$L_l^{*}\left(Y\right|Z)-L_l^{*}\left(Y\right|X)\le {\displaystyle \frac{{\parallel l\parallel }_{\infty }}{\sqrt{2}}}\sqrt{{\displaystyle \frac{D_{\alpha }(P_{X|Y,Z}\parallel P_{X|Z}\left|P_{Y,Z}\right)}{\alpha }}}.$$

(29)

□

Remark 3. 

Taking the limit as $\alpha \to 1$ of (29) in Corollary 1 yields the mutual information-based bound:

$$\begin{array}{cc}\hfill L_l^{*}\left(Y\right|Z)-L_l^{*}\left(Y\right|X)& \le {\displaystyle \frac{{\parallel l\parallel }_{\infty }}{\sqrt{2}}}\sqrt{D_{\mathit{KL}}(P_{X|Y,Z}\parallel P_{X|Z}\left|P_{Y,Z}\right)}\hfill \\ \hfill & ={\displaystyle \frac{{\parallel l\parallel }_{\infty }}{\sqrt{2}}}\sqrt{I(X;Y)-I(Z;Y)},\hfill \end{array}$$

(30)

which recovers the bound in [14] (Corollary 1).

3.2. $\alpha$-Jensen–Shannon Divergence-Based Upper Bound

We next derive $\alpha$-Jensen–Shannon divergence-based bounds on minimum excess risk.

We consider two arbitrary jointly distributed random variables U and V defined on the same probability space and taking values in $\mathcal{U}$ and $\mathcal{V}$, respectively. Throughout this section, we work with the joint distribution $P_{U,V}$ over $\mathcal{U}\times \mathcal{V}$ and the corresponding product of marginals $P_U{P}_V$. For convenience, we also define additional distributions that will play an important role in the derivation of our bounds.

Definition 9. 

The α-convex combination of the joint distribution $P_{U,V}$ and the product of two marginals $P_U{P}_V$ is denoted by $P_{U,V}^{\left(\alpha \right)}$ and given by

$$P_{U,V}^{\left(\alpha \right)}=\alpha P_{U,V}+(1-\alpha )P_U{P}_V$$

(31)

for $\alpha \in (0,1)$.

Definition 10. 

The α-conditional convex combination of the conditional distribution $P_{V|U}$ and the marginal $P_V$ is denoted by $P_{V|U}^{\left(\alpha \right)}$ and given by

$$P_{V|U}^{\left(\alpha \right)}=\alpha P_{V|U}+(1-\alpha )P_V$$

(32)

for $\alpha \in (0,1)$.

We first provide the following lemma, whose proof, given in Appendix A, is a slight generalization of [17] (Lemma 2) and [14] (Lemma 1).

Lemma 3. 

Given a function $h:\mathcal{U}\times \mathcal{V}\to \mathbb{R}$, assume that $h(u,V)$ is ${\sigma }^2\left(u\right)$-sub-Gaussian under $P_{V|U=u}^{\left(\alpha \right)}$ for all $u\in \mathcal{U}$, where $\mathbb{E}\left[{\sigma }^2\left(U\right)\right]<\infty$. Then for $\alpha \in (0,1)$,

$$\begin{array}{cc}\hfill |{\mathbb{E}}_{P_{U,V}}\left[h(U,V)\right]-& {\mathbb{E}}_{P_U{P}_V}\left[h(U,V)\right]|\le \sqrt{2\mathbb{E}\left[{\sigma }^2\left(U\right)\right]{\displaystyle \frac{J{S}_{\alpha }(P_{U,V}\parallel P_U{P}_V)}{\alpha (1-\alpha )}}}.\hfill \end{array}$$

We next use Lemma 3 to derive our theorem for $\alpha$-Jensen–Shannon divergence-based bound. The proof of the theorem is relegated to Appendix A.

Theorem 2. 

Let X, Y and Z be random vectors such that $Y\to X\to Z$, as described in Section 2.1. Assume that there exists an optimal estimator f of Y from X such that $l(y,f(X\left)\right)$ is conditionally ${\sigma }^2\left(y\right)$-sub-Gaussian under $P_{X|Z,Y=y}^{\left(\alpha \right)}=\alpha P_{X|Z,Y=y}+(1-\alpha )P_{X|Z}$ for all $y\in {\mathbb{R}}^p$, i.e.,

$$log{\mathbb{E}}_{P_{X|Z,Y=y}^{\left(\alpha \right)}}\left[e^{\lambda \left(l(y,f\left(X\right))-{\mathbb{E}}_{P_{X|Z,Y=y}^{\left(\alpha \right)}}\left[l(y,f(X\left)\right)\right]\right)}\right]\le {\displaystyle \frac{{\sigma }^2\left(y\right){\lambda }^2}{2}}$$

for all $\lambda \in \mathbb{R}$ and $y\in \mathbb{R}$, where ${\sigma }^2:\mathbb{R}\to \mathbb{R}$ satisfies $\mathbb{E}\left[{\sigma }^2\left(Y\right)\right]<\infty$. Then for $\alpha \in (0,1)$, the excess minimum risk satisfies

$$\begin{array}{cc}\hfill & L_l^{*}\left(Y\right|Z)-L_l^{*}\left(Y\right|X)\le \sqrt{{\displaystyle \frac{2\mathbb{E}\left[{\sigma }^2\left(Y\right)\right]}{\alpha (1-\alpha )}}J{S}_{\alpha }(P_{Y,X|Z}\parallel P_{Y|Z}{P}_{X|Z}\left|P_Z\right)}.\hfill \end{array}$$

(33)

Remark 4. 

Taking the limit as $\alpha \to 0$ on the right-hand side of (33) in Theorem 2, we obtain

$$\begin{array}{cc}\hfill L_l^{*}\left(Y\right|Z)-L_l^{*}\left(Y\right|X)& \le \sqrt{2\mathbb{E}\left[{\sigma }^2\left(Y\right)\right](I(X;Y)-I(Z;Y))},\hfill \end{array}$$

recovering the bound (27) of [14] (Theorem 3). Furthermore, taking the limit as $\alpha \to 1$ of the right-hand side of (33) in Theorem 2, yields

$$\begin{array}{cc}\hfill L_l^{*}\left(Y\right|Z)-L_l^{*}\left(Y\right|X)& \le \sqrt{2\mathbb{E}\left[{\sigma }^2\left(Y\right)\right]D_{\mathit{KL}}(P_{Y|Z}{P}_{X|Z}\parallel P_{Y,X|Z}\left|P_Z\right)}\hfill \\ \hfill & =\sqrt{2\mathbb{E}\left[{\sigma }^2\left(Y\right)\right](L(X;Y)-L(Z;Y))},\hfill \end{array}$$

(34)

where $L(U;V):=D_{\mathit{KL}}(P_U{P}_V\parallel P_{U,V})$ is called the Lautum information between U and V, defined as the reverse KL divergence (i.e., the KL divergence between the product of marginals and the joint distribution) [38]. We, therefore, obtain an upper bound on the minimum excess risk in terms of the reverse KL divergence.

We close this section by specializing Theorem 2 to the case of bounded loss functions, hence obtaining a counterpart result to Corollary 1.

Corollary 2. 

Suppose the loss function l is bounded. Then for random vectors X, Y and Z such that $Y\to X\to Z$ as described in Section 2.1, we have the following inequality for $\alpha \in (0,1)$ on the excess minimum risk:

$$L_l^{*}\left(Y\right|Z)-L_l^{*}\left(Y\right|X)\le {\displaystyle \frac{{\parallel l\parallel }_{\infty }}{\sqrt{2}}}\sqrt{{\displaystyle \frac{J{S}_{\alpha }(P_{Y,X|Z}\parallel P_{Y|Z}{P}_{X|Z}\left|P_Z\right)}{\alpha (1-\alpha )}}}.$$

(35)

3.3. Sibson Mutual Information-Based Upper Bound

Here we bound the excess minimum risk based on Sibson’s mutual information. We recall from Definition 6 that U and V are jointly distributed on measurable spaces $\mathcal{U}$ and $\mathcal{V}$, with joint distribution $P_{U,V}$ and marginals $P_U$ and $P_V$, assuming that all distributions are absolutely continuous with respect to a common sigma-finite measure $\nu$, with densities $p_U$, $p_V$, and $p_{U,V}$. Let $U^{*}$ denote the random variable whose distribution $P_{U^{*}}$ attains the minimum in the definition of the Sibson mutual information $I_{\alpha }^S(P_{U,V},P_{U^{*}})$, with density $p_{U^{*}}$ as given in (8). We now define an auxiliary distribution that will be central to the derivation of the main bounds in this section.

Definition 11. 

Let $P_{\widehat{U},\widehat{V}}$ be a joint distribution on $\mathcal{U}\times \mathcal{V}$ determined by density $p_{\widehat{U},\widehat{V}}$ that is obtained by tilting (using parameter α) the densities $p_{U,V}$, $p_{U^{*}}$ and $p_V$ as follows:

$$p_{\widehat{U},\widehat{V}}(u,v)={\displaystyle \frac{{\left(p_{U,V}(u,v)\right)}^{\alpha }{\left(p_{U^{*}}\left(u\right)p_V\left(v\right)\right)}^{(1-\alpha )}}{\int \int {\left(p_{U,V}(u^{\prime },v^{\prime })\right)}^{\alpha }{\left(p_{U^{*}}\left(u^{\prime }\right)p_V\left(v^{\prime }\right)\right)}^{(1-\alpha )}d{u}^{\prime }d{v}^{\prime }}}$$

(36)

for $\alpha \in (0,1)$.

We state the following lemma based on the variational representation of the Sibson mutual information [9] (Theorem 2), which establishes a connection to the KL divergence. The proof of the lemma follows from [17] (Lemma 3) and [36] (Theorem 5.1).

Lemma 4. 

For the distributions $P_{\widehat{U},\widehat{V}}$, $P_{U,V}$ and $P_{U^{*}}{P}_V$ we have

$$\alpha D_{\mathit{KL}}(P_{\widehat{U},\widehat{V}}\parallel P_{U,V})+(1-\alpha )D_{\mathit{KL}}(P_{\widehat{U},\widehat{V}}\parallel P_{U^{*}}{P}_V)=(1-\alpha )I_{\alpha }^S(P_{U,V},P_{U^{*}}).$$

We now invoke a basic but important property of sub-Gaussian random variables that will be used later in our analysis. Specifically, the set of all sub-Gaussian random variables has a linear structure. This property is well established in the literature [39,40].

Lemma 5. 

If X is a ${\sigma }_X^2$-sub-Gaussian random variable, then for any $\alpha \in \mathbb{R}$, the random variable $\alpha X$ is $\left|\alpha \right|{\sigma }_X^2$-sub-Gaussian. If Y is a ${\sigma }_Y^2$-sub-Gaussian random variable, then the sum $X+Y$ is sub-Gaussian with parameter ${({\sigma }_X+{\sigma }_Y)}^2$.

We next provide the following lemma, whose proof is a slight generalization of [17] (Theorem 4) and [14] (Lemma 1). The proof is given in Appendix B.

Lemma 6. 

Given a function $h:\mathcal{U}\times \mathcal{V}\to \mathbb{R}$, assume that $h(u,V)$ is ${\gamma }^2\left(u\right)$-sub-Gaussian under $P_V$ for all $u\in \mathcal{U}$ and $h(U,V)$ is ${\displaystyle \frac{{\sigma }^2}{4}}$-sub-Gaussian under both $P_U{P}_V$ and $P_{U,V}$. Assume also that $log{\mathbb{E}}_{P_{U^{*}}}\left[e^{{\gamma }^2\left(U^{*}\right)}\right]<\infty$. Then for $\alpha \in (0,1)$,

$$\begin{array}{c}\hfill \left|{\mathbb{E}}_{P_{U,V}}\left[h(U,V)\right]-{\mathbb{E}}_{P_U{P}_V}\left[h(U,V)\right]\right|\le \sqrt{2((1-\alpha ){\sigma }^2+\alpha log{\mathbb{E}}_{P_{U^{*}}}\left[e^{{\gamma }^2\left(U^{*}\right)}\right]){\displaystyle \frac{I_{\alpha }^S(P_{U,V},P_{U^{*}})}{\alpha }}}.\end{array}$$

We next use Lemma 6 to derive our upper bound on the excess minimum risk in terms of the Sibson mutual information. The proof of the theorem is in Appendix B.

Theorem 3. 

Let X, Y and Z be random vectors such that $Y\to X\to Z$ form a Markov chain as described in Section 2.1. Assume that there exists an optimal estimator f of Y from X such that $l(y,f(X\left)\right)$ is conditionally ${\gamma }^2\left(y\right)$-sub-Gaussian under $P_{X|Z}$ for all $y\in {\mathbb{R}}^p$, where $log{\mathbb{E}}_{P_Z{P}_{Y^{*}|Z}}\left[e^{{\gamma }^2\left(Y^{*}\right)}\right]<\infty$, and $l(Y,f(X\left)\right)$ is conditionally ${\displaystyle \frac{{\sigma }^2}{4}}$-sub-Gaussian under both $P_{Y|Z}{P}_{X|Z}$ and $P_{Y,X|Z}$, i.e., for all $\lambda \in \mathbb{R}$

$$\begin{array}{c}\hfill log{\mathbb{E}}_{P_{X|Z}}\left[e^{(\lambda \left(l(y,f\left(X\right))\right)-{\mathbb{E}}_{P_{X|Z}}\left[l(y,f\left(X\right))\right])}\right]\le {\displaystyle \frac{{\gamma }^2\left(y\right){\lambda }^2}{2}}\end{array}$$

for all $y\in {\mathbb{R}}^p$,

$$log{\mathbb{E}}_{P_{Y|Z}{P}_{X|Z}}\left[e^{(\lambda \left(l(Y,f\left(X\right))\right)-{\mathbb{E}}_{P_{Y|Z}{P}_{X|Z}}\left[l(Y,f\left(X\right))\right])}\right]\le {\displaystyle \frac{{\sigma }^2{\lambda }^2}{8}}$$

and

$$log{\mathbb{E}}_{P_{Y,X|Z}}\left[e^{(\lambda \left(l(Y,f\left(X\right))\right)-{\mathbb{E}}_{P_{Y,X|Z}}\left[l(Y,f\left(X\right))\right])}\right]\le {\displaystyle \frac{{\sigma }^2{\lambda }^2}{8}}.$$

Then for $\alpha \in (0,1)$, the excess minimum risk satisfies

$$\begin{array}{c}\hfill L_l^{*}\left(Y\right|Z)-L_l^{*}\left(Y\right|X)\le \sqrt{{\displaystyle \frac{2((1-\alpha ){\sigma }^2+\alpha {\mathbb{E}}_{P_Z}\left[{\mathsf{\Phi }}_{Y^{*}|Z}\left({\gamma }^2\left(Y^{*}\right)\right)\right])}{\alpha }}{\mathbb{E}}_{P_Z}\left[I_{\alpha }^S(P_{Y,X|Z},P_{Y^{*}|Z})\right]},\end{array}$$

(37)

where ${\mathsf{\Phi }}_{P_U}\left(V\right)=log{\mathbb{E}}_{P_U}\left[e^V\right]$ and the distribution $P_{Y^{*}|Z}$ has density

$$\begin{array}{c}\hfill p_{Y^{*}|Z}\left(y\right|z)={\displaystyle \frac{{\left({\displaystyle \int {\left(\frac{p_{Y,X|Z}(y,x|z)}{p_{Y|Z}\left(y\right|z)p_{X|Z}\left(x\right|z)}\right)}^{\alpha }{p}_{X|Z}\left(x\right|z)dx}\right)}^{\frac{1}{\alpha }}}{{\displaystyle \int {\left(\int {\left(\frac{p_{Y,X|Z}(y^{\prime },x^{\prime }|z)}{p_{Y|Z}\left(y^{\prime }\right|z)p_{X|Z}\left(x^{\prime }\right|z)}\right)}^{\alpha }{p}_{X|Z}\left(x^{\prime }\right|z)d{x}^{\prime }\right)}^{\frac{1}{\alpha }}{p}_{Y|Z}\left(y^{\prime }\right|z)d{y}^{\prime }}}}{p}_{Y|Z}\left(y\right|z).\end{array}$$

(38)

Setting ${\gamma }^2\left(Y^{*}\right)={\sigma }^2$ and taking the limit as $\alpha \to 1$ on the right-hand side of (37) recovers the mutual information-based bound (27) of [14] (Theorem 3) in the case of a constant sub-Gaussian parameter. We conclude this section by presenting a specialization of Theorem 3 to the case of bounded loss functions.

Corollary 3. 

Suppose the loss function l is bounded. Then for random vectors X, Y and Z such that $Y\to X\to Z$ as described in Section 2.1, we have the following inequality for $\alpha \in (0,1)$ on the excess minimum risk:

$$\begin{array}{c}\hfill L_l^{*}\left(Y\right|Z)-L_l^{*}\left(Y\right|X)\le {\displaystyle \frac{{\parallel l\parallel }_{\infty }}{\sqrt{2}}}\sqrt{{\displaystyle \frac{(4-3\alpha )}{\alpha }}{\mathbb{E}}_{P_Z}\left[I_{\alpha }^S(P_{Y,X|Z},P_{Y^{*}|Z})\right]}.\end{array}$$

(39)

3.4. Comparison of Proposed Upper Bounds

In this section, we give a simple comparison of the upper bounds based on the $\alpha$-Jensen–Shannon divergence (Theorem 2) with those based on the Rényi divergence (Theorem 1) and the Sibson mutual information (Theorem 3) for bounded loss functions. Similar to [17] (Proposition 8), we provide a simple condition under which the upper bound based on the $\alpha$-Jensen–Shannon divergence is tighter than those obtained using the other two divergence measures.

Proposition 1. 

Suppose the loss function is bounded. Let X, Y, and Z be random vectors such that $Y\to X\to Z$, as described in Section 2.1. Then, for any $\alpha \in (0,1)$, the α-Jensen–Shannon divergence-based bound on the excess minimum risk,

$${\displaystyle \frac{{\parallel l\parallel }_{\infty }}{\sqrt{2}}}\sqrt{{\displaystyle \frac{J{S}_{\alpha }(P_{Y,X|Z}\parallel P_{Y|Z}{P}_{X|Z}\left|P_Z\right)}{\alpha (1-\alpha )}}},$$

is no larger than both the Rényi divergence-based bound,

$${\displaystyle \frac{{\parallel l\parallel }_{\infty }}{\sqrt{2}}}\sqrt{{\displaystyle \frac{D_{\alpha }(P_{X|Y,Z}\parallel P_{X|Z}\left|P_{Y,Z}\right)}{\alpha }}},$$

and the Sibson mutual information-based bound,

$${\displaystyle \frac{{\parallel l\parallel }_{\infty }}{\sqrt{2}}}\sqrt{{\displaystyle \frac{(4-3\alpha )}{\alpha }}{\mathbb{E}}_{P_Z}\left[I_{\alpha }^S(P_{Y,X|Z},P_{Y^{*}|Z})\right]},$$

provided that

$${\displaystyle \frac{h_b\left(\alpha \right)}{1-\alpha }}\le D_{\alpha }(P_{X|Y,Z}\parallel P_{X|Z}\mid P_{Y,Z}),\mathit{and}{\displaystyle \frac{h_b\left(\alpha \right)}{1-\alpha }}\le {\mathbb{E}}_{P_Z}\left[I_{\alpha }^S(P_{Y,X|Z},P_{Y^{*}|Z})\right],$$

(40)

where $h_b\left(\alpha \right)=-\alpha log\alpha -(1-\alpha )log(1-\alpha )$ is the binary entropy function.

Remark 5. 

The function $g\left(\alpha \right)={\displaystyle \frac{h_b\left(\alpha \right)}{1-\alpha }}$ is strictly increasing for $\alpha \in (0,1)$, ${lim}_{\alpha \to 0}g\left(\alpha \right)=0$, and ${lim}_{\alpha \to 1}g\left(\alpha \right)=\infty$. On the other hand, at least for finite alphabets, the information quantities on the right-hand sides of the inequalities in (40), in general, converge to a positive constant as $\alpha \to 0$. In this case, there always exists an ${\alpha }^{*}\in (0,1)$ such that the inequalities in (40) are satisfied for all $0<\alpha \le {\alpha }^{*}$.

Proof. 

It is known that the $\alpha$-Jensen–Shannon divergence is bounded above by the binary entropy $h_b\left(\alpha \right)$, with equality if and only if P and Q are mutually singular [3,41]. Applying this we obtain

$$\begin{array}{cc}\hfill J{S}_{\alpha }(P_{Y,X|Z}\parallel P_{Y|Z}{P}_{X|Z}\mid P_Z)& ={\mathbb{E}}_{P_Z}\left[J{S}_{\alpha }\left(P_{Y,X|Z}(·\mid Z)\parallel P_{Y|Z}(·\mid Z)P_{X|Z}(·\mid Z)\right)\right]\hfill \\ \hfill & \le {\mathbb{E}}_{P_Z}\left[h_b\left(\alpha \right)\right]\hfill \\ \hfill & =h_b\left(\alpha \right).\hfill \end{array}$$

Consequently, we obtain the bound

$$\begin{array}{c}\hfill {\displaystyle \frac{{\parallel l\parallel }_{\infty }}{\sqrt{2}}}\sqrt{{\displaystyle \frac{J{S}_{\alpha }(P_{Y,X|Z}\parallel P_{Y|Z}{P}_{X|Z}\mid P_Z)}{\alpha (1-\alpha )}}}\le {\displaystyle \frac{{\parallel l\parallel }_{\infty }}{\sqrt{2}}}\sqrt{{\displaystyle \frac{h_b\left(\alpha \right)}{\alpha (1-\alpha )}}}.\end{array}$$

Therefore, under the assumption that

$${\displaystyle \frac{h_b\left(\alpha \right)}{1-\alpha }}\le D_{\alpha }(P_{X|Y,Z}\parallel P_{X|Z}\mid P_{Y,Z})\mathrm{and}{\displaystyle \frac{h_b\left(\alpha \right)}{1-\alpha }}\le {\mathbb{E}}_{P_Z}\left[I_{\alpha }^S(P_{Y,X|Z},P_{Y^{*}|Z})\right],$$

we conclude that the $\alpha$-Jensen–Shannon bound is tighter than the other two bounds:

$$\begin{array}{cc}\hfill {\displaystyle \frac{{\parallel l\parallel }_{\infty }}{\sqrt{2}}}\sqrt{{\displaystyle \frac{J{S}_{\alpha }(P_{Y,X|Z}\parallel P_{Y|Z}{P}_{X|Z}\mid P_Z)}{\alpha (1-\alpha )}}}& \le {\displaystyle \frac{{\parallel l\parallel }_{\infty }}{\sqrt{2}}}\sqrt{{\displaystyle \frac{D_{\alpha }(P_{X|Y,Z}\parallel P_{X|Z}\mid P_{Y,Z})}{\alpha }}},\hfill \\ \hfill {\displaystyle \frac{{\parallel l\parallel }_{\infty }}{\sqrt{2}}}\sqrt{{\displaystyle \frac{J{S}_{\alpha }(P_{Y,X|Z}\parallel P_{Y|Z}{P}_{X|Z}\mid P_Z)}{\alpha (1-\alpha )}}}& \le {\displaystyle \frac{{\parallel l\parallel }_{\infty }}{\sqrt{2}}}\sqrt{{\displaystyle \frac{(4-3\alpha )}{\alpha }}{\mathbb{E}}_{P_Z}\left[I_{\alpha }^S(P_{Y,X|Z},P_{Y^{*}|Z})\right]}.\hfill \end{array}$$

□

4. Numerical Results

In this section, we present three examples where some of the proposed information divergence-based bounds outperform the mutual information-based bound. The first example considers a concatenated q-ary symmetric channel with a bounded loss function. The remaining two examples involve Gaussian additive noise channels and loss functions with non-constant sub-Gaussian parameters.

Example 1. 

We consider a concatenation of two q-ary symmetric channels, with input Y and noise variables $U_1$ and $U_2$, all taking values in $\{0,1,\dots ,q-1\}$. We assume that $Y,U_1$ and $U_2$ are independent. The input Y has distribution $p=[p_0,p_1,\dots ,p_{q-1}]$, while the noise variables $U_1$ and $U_2$ are governed by $\mathbb{P}(U_i=0)=1-{ϵ}_i$ and $\mathbb{P}(U_i=a)={ϵ}_i/(q-1)$ for all $a\in \{1,\dots ,q-1\}$ and $i=1,2$, where ${ϵ}_1,{ϵ}_2\in (0,1)$ are the crossover probabilities. The output X of the first channel is given by

$$X=(Y+U_1)modq$$

and serves as the input to the second channel. The final output Z is then given by

$$Z=(X+U_2)modq,$$

which can also be written as $Z=(Y+U_1+U_2)modq$. This construction naturally induces the Markov chain $Y\to X\to Z$.

Using a 0–1 loss function (defined as $l(y,y^{\prime })=1(y\ne y^{\prime })$, where $1\left(·\right)$ denotes the indicator function), we compute the bounds in Corollaries 1 and 2, corresponding to equations (28) and (35), respectively, as functions of $\alpha \in (0,1)$. Figure 1 compares the Rényi-based bound (28) and the αJensen–Shannon-based bound (35) with the mutual information-based bound (27). Among the two, the α-Jensen–Shannon-based bound consistently performs the best over a wide range of α values. Moreover, as q increases in the q-ary symmetric channel, both the interval of α for which the proposed bounds outperform the mutual information-based bound and the magnitude of improvement become more pronounced. For this example, we set ${ϵ}_1=0.15$ and ${ϵ}_2=0.05$. For $q=10,100,200$, we generate input distributions by sampling from a symmetric (i.e., with identical parameters) Dirichlet distribution on ${\mathbb{R}}^q$. Using a Dirichlet parameter greater than one gives balanced distributions that avoid placing too much weight on any single symbol. For $q=2,3,5$, the input distributions are explicitly specified in the figure captions.

Finally, we note that in this example, the specialized bound for bounded loss functions derived from the Sibson mutual information in Corollary 3 does not offer any improvement over the standard mutual information-based bound (27) and is, therefore, not presented. In the next two examples, we compare the αJensen–Shannon-based bound of Theorem 2 with the mutual information-based bound for loss functions with non-constant sub-Gaussian parameters.

Example 2. 

Consider a Gaussian additive noise channel with input Y and noise random variables $W_1$ and $W_2$, where $Y\sim \mathcal{N}(0,{\widehat{\sigma }}^2)$, $W_1\sim \mathcal{N}(0,{\sigma }_1^2)$, and $W_2\sim \mathcal{N}(0,{\sigma }_2^2)$. Assume that Y is independent of $(W_1,W_2)$ and $W_1$ is independent of $W_2$. Define

$$X=Y+W_1$$

and

$$Z=X+W_2=Y+W_1+W_2,$$

inducing the Markov chain $Y\to X\to Z$.

We consider the loss function $l(y,y^{\prime })=min\left\{\right|y-y^{\prime }|,|y-c\left|\right\}$ for some $c>0$. For this model, we observe that

$$\begin{array}{cc}\hfill l(y,f^{*}\left(X\right))& =min\left\{\right|y-f^{*}\left(X\right)|,|y-c\left|\right\}\le |y-c|\le \left|y\right|+\left|c\right|=\left|y\right|+c,\hfill \end{array}$$

where $f^{*}$ denotes the optimal estimator of Y from X. Thus, $l(y,f^{*}\left(X\right))$ is a non-negative random variable that is almost surely bounded by $\left|y\right|+c$. By Hoeffding’s lemma, it follows that this loss is conditionally ${\sigma }^2\left(y\right)$-sub-Gaussian under $P_{X|Z}$, $P_{X|Z,Y=y}$ and $P_{X|Z,Y=y}^{\left(\alpha \right)}$ for all $y\in {\mathbb{R}}^p$, with

$${\sigma }^2\left(y\right)={\displaystyle \frac{{\left(\right|y|+c)}^2}{4}}.$$

Furthermore, for ${\widehat{\sigma }}^2=1$ and ${\sigma }_i^2=1$, $i=1,2$, we have that

$$\begin{array}{cc}\hfill \mathbb{E}\left[{\sigma }^2\left(Y\right)\right]& =\mathbb{E}\left[{\displaystyle \frac{{\left(\right|Y|+c)}^2}{4}}\right]={\displaystyle \frac{1}{4}}\mathbb{E}\left[{\left|Y\right|}^2+c^2+2\left|Y\right|c\right]={\displaystyle \frac{1+c^2+2c\sqrt{2/\pi }}{4}}.\hfill \end{array}$$

Hence, the conditions of Theorem 2 are satisfied. Figure 2 compares the α-Jensen–Shannon-based bound in (33) with the mutual information-based bound in (27) for $c=1$. We observe that the α-Jensen–Shannon-based bound is tighter for values of α approximately in the range $(0,0.3)$.

Example 3. 

Consider a Gaussian additive noise model with input $Z\sim \mathcal{N}(0,{\widehat{\sigma }}^2)$ and two noise variables $W_1\sim \mathcal{N}(0,{\sigma }_1^2)$ and $W_2\sim \mathcal{N}(0,{\sigma }_2^2)$, all mutually independent. Let $X=Z+W_1$ and $Y=X+W_2=Z+W_1+W_2,$ inducing the Markov chain $Z\to X\to Y$, which is equivalent to the Markov chain $Y\to X\to Z$.

We again consider the loss function $l(y,y^{\prime })=min\left\{\right|y-y^{\prime }|,|y-c\left|\right\}$ for some $c>0$, and observe that $l(y,f^{*}\left(X\right))\le \left|y\right|+c$, where $f^{*}$ is the optimal estimator of Y from X. Hence, the loss is (conditionally) ${\sigma }^2\left(y\right)$-sub-Gaussian as in the previous example with ${\sigma }^2\left(y\right)={\displaystyle \frac{{\left(\right|y|+c)}^2}{4}}$. For ${\widehat{\sigma }}^2=2$, ${\sigma }_1^2=39$ and ${\sigma }_2^2=1$, the expected sub-Gaussian parameter is

$$\mathbb{E}\left[{\sigma }^2\left(Y\right)\right]={\displaystyle \frac{42+c^2+2c\sqrt{84/\pi }}{4}}.$$

Therefore, the conditions of Theorem 2 continue to hold.

In contrast to the previous example, where Y was the input and Z the degraded observation, this example reverses that direction. Figure 3 compares the α-Jensen–Shannon-based bound in (33) with the mutual information-based bound in (27) for $c=1$. We observe that the α-Jensen–Shannon-based bound is tighter for values of α approximately in the range $(0,0.7)$.

5. Conclusions

In this paper, we studied the problem of bounding the excess minimum risk in statistical inference using generalized information divergence measures. Our results extend the mutual information-based bound in [14] by developing a family of bounds parameterized by the order $\alpha \in (0,1)$, involving the Rényi divergence, the $\alpha$-Jensen–Shannon divergence, and Sibson’s mutual information. For the Rényi divergence-based bounds, we employed the variational representation of the divergence, following the approach in [11], and for the $\alpha$-Jensen–Shannon and Sibson-based bounds, we adopted the auxiliary distribution method introduced in [17].

Unlike the bounds in [11,17], which assume the sub-Gaussian parameter to be constant, our framework allows this parameter to depend on the (target) random vector, thereby making the bounds applicable to a broader class of joint distributions. We demonstrated the effectiveness of our approach through three numerical examples: one involving concatenated discrete q-ary symmetric channels, and two based on additive Gaussian noise channels. In all cases, we observed that at least one of our $\alpha$-parametric bounds is tighter than the mutual information-based bound over certain ranges of $\alpha$, with the improvements becoming more pronounced in the discrete example as the channel alphabet size q increased.

Future directions include exploring bounds under alternative f-divergence measures, developing tighter bounds for high-dimensional settings, and determining divergence rates in infinite-dimensional cases.