Lossless Transformations and Excess Risk Bounds in Statistical Inference

Abstract

We study the excess minimum risk in statistical inference, defined as the difference between the minimum expected loss when estimating a random variable from an observed feature vector and the minimum expected loss when estimating the same random variable from a transformation (statistic) of the feature vector. After characterizing lossless transformations, i.e., transformations for which the excess risk is zero for all loss functions, we construct a partitioning test statistic for the hypothesis that a given transformation is lossless, and we show that for i.i.d. data the test is strongly consistent. More generally, we develop information-theoretic upper bounds on the excess risk that uniformly hold over fairly general classes of loss functions. Based on these bounds, we introduce the notion of a $\delta$-lossless transformation and give sufficient conditions for a given transformation to be universally $\delta$-lossless. Applications to classification, nonparametric regression, portfolio strategies, information bottlenecks, and deep learning are also surveyed.

1. Introduction

We consider the standard setting of statistical inference, where Y is a real random variable, having a range $\mathcal{Y}\subset \mathbb{R}$, which is to be estimated (predicted) from a random observation (feature) vector X, taking values in ${\mathbb{R}}^d$. Given a measurable predictor $f:{\mathbb{R}}^d\to \mathcal{Y}$ and measurable loss function $\ell :\mathcal{Y}\times \mathcal{Y}\to {\mathbb{R}}_{+}$, the loss incurred is $\ell (Y,f(X\left)\right)$. The minimum expected risk in predicting Y from the random vector X is

$$L_{\ell }^{\ast }\left(Y\right|X)=\underset{f:{\mathbb{R}}^d\to \mathcal{Y}}{inf}\mathbb{E}[\ell (Y,f\left(X\right))],$$

where the infimum is over all measurable f.

Suppose that the tasks of collecting data and making the prediction are separated in time or in space. For example, the separation in time happens when the data are collected first and the statistical modeling and analysis are performed much later. Separation in space can be due, for example, to collecting data at a remote location and making predictions centrally. Such situations are modeled using a transformation $T:{\mathbb{R}}^d\to {\mathbb{R}}^{d^{\prime }}$, so that the prediction regarding Y is made from the transformed observation $T\left(X\right)$, instead of X. An important example for such a transformation is quantization, in which case $T\left(X\right)$ is a discrete random variable. Clearly, one always has $L_{\ell }^{\ast }\left(Y\right|T\left(X\right))\ge L_{\ell }^{\ast }\left(Y\right|X)$. The difference $L_{\ell }^{\ast }\left(Y\right|T\left(X\right))-L_{\ell }^{\ast }\left(Y\right|X)$ is sometimes referred to in the literature as excess risk. A part of this paper is concerned with transformations for which the excess risk is zero, no matter the underlying loss function ℓ. Such transformations are universally lossless, in the sense that they can be chosen before the cost function ℓ for the underlying problem is known. More formally, we can state the following definition.

Definition 1

(lossless transformation). For a fixed joint distribution of Y and X, a (measurable) transformation $T:{\mathbb{R}}^d\to {\mathbb{R}}^{d^{\prime }}$ is called universally lossless if for any loss function $\ell :\mathcal{Y}\times \mathcal{Y}\to {\mathbb{R}}_{+}$ we have

$$L_{\ell }^{\ast }\left(Y\right|T\left(X\right))=L_{\ell }^{\ast }\left(Y\right|X).$$

An important special transformation is feature selection. Formally, for the observation (feature) vector $X=(X^{\left(1\right)},\dots ,X^{\left(d\right)})$ and $S\subset \{1,\dots ,d\}$, consider the $\left|S\right|$-dimensional vector $X_S=(X^{\left(i\right)},i\in S)$. Typically, the dimension $\left|S\right|$ of $X_S$ is significantly smaller than d, the dimension of X. If we have

$$L_{\ell }^{\ast }\left(Y\right|X_S)=L_{\ell }^{\ast }\left(Y\right|X),$$

for all loss functions ℓ, then the feature selector $X\mapsto X_S$ is universally lossless. For fixed loss ℓ, the performance of any statistical inference method is sensitive to the dimension of the feature vector. Therefore, dimension reduction is crucial before choosing or constructing an inference method. If $X_S$ is universally lossless, then the complement feature subvector $X_{S^c}$ is irrelevant. It is an open research problem how to efficiently search a universally lossless $X_S$ with minimum size $\left|S\right|$. Since, typically, the distribution of the pair X and Y is not known and must be inferred from data, any such search algorithm needs a procedure for testing for the universal losslessness property of a feature selector.

In the first part of this paper, we give a necessary and sufficient condition for a given transformation T to be universally lossless and then construct a partitioning-based statistic for testing this condition if independent and identically distributed training data are available. With the null hypothesis being that a given transformation is universally lossless, the test is shown to be strongly consistent, in the sense that it almost surely (a.s.) makes finitely many Type I and II errors.

In many situations, requiring that a transformation T is universally lossless is too demanding. The next definition relaxes this requirement.

Definition 2

($\delta$-lossless transformation). For a fixed joint distribution of Y and X, and $\delta >0$, a transformation $T:{\mathbb{R}}^d\to {\mathbb{R}}^{d^{\prime }}$ is called universally $\delta$-lossless with respect to a class of loss functions $\mathcal{L}$, if we have

$$L_{\ell }^{\ast }\left(Y\right|T\left(X\right))-L_{\ell }^{\ast }\left(Y\right|X)\le \delta forall\ell \in \mathcal{L}.$$

In the second part of this paper, we derive bounds on the excess minimum risk $L_{\ell }^{\ast }\left(Y\right|T\left(X\right))-L_{\ell }^{\ast }\left(Y\right|X)$ in terms of the mutual information difference $I(Y;X)-I(Y;T(X\left)\right)$ under various assumptions about ℓ. With the aid of these bounds, we give information-theoretic sufficient conditions for a transformation T to be $\delta$-lossless with respect to fairly general classes of loss functions ℓ. Applications to classification, nonparametric regression, portfolio strategies, the information bottleneck method, and deep learning are also reviewed.

• Relationship with prior work

Our first result, Theorem 1, which shows that a transformation is universally lossless if and only if it is a sufficient statistic, is likely known, but we could not find it in this explicit form in the literature (a closely related result is the classical Rao–Blackwell theorem of mathematical statistics, e.g., Schervish ([1], Theorem 3.22)). Due to this result, testing with independent data whether or not a given transformation is universally lossless turns into a test for conditional independence. Our test in Theorem 2 is based on the main results in Györfi and Walk [2], but our construction is more general and we also correct an error in the proof of ([2], Theorem 1). Apart from [2], most of the results in the literature of testing for conditional independence are for real-valued random variables and/or assume certain special distribution types, typically the existence of a joint probability density function. Such assumptions exclude problems where Y is discrete and X is continuous, as is typical in classification, or problems where the observation X is concentrated on a lower dimensional subspace or manifold. In contrast, our test construction is completely distribution free and its convergence properties are also (almost) distribution free. A more detailed review of related work is given in Section 2.1.

The main result in Section 3 is Theorem 3, which bounds the excess risk in terms of the square root of the mutual information difference $I(Y;X)-I(Y;T(X\left)\right)$. There is a history of such bounds, possibly starting with Xu and Raginsky [3], where the generalization error of a learning algorithm was upper bounded using constant times the square root of the mutual information between the hypothesis and the training data (see also the references in [3,4]). This result has since been extended in various forms, mostly concentrating on providing information-theoretic bounds for the generalization capabilities of learning algorithms, instead of looking at the excess risk; see, e.g., Raginsky et al. [5], Lugosi and Neu [6], Jose and Simeone [7], and the references therein, just to mention a few of these works. The most relevant recent work relating to our bounds in Section 3 seems to be Xu and Raginsky [4], where, among other things, information-theoretic bounds were developed on the excess risk in a Bayesian learning framework; see also Hafez-Kolahi et al. [8]. The bounds in [4] are not on the excess risk $L_{\ell }^{\ast }\left(Y\right|T\left(X\right))-L_{\ell }^{\ast }\left(Y\right|X)$; they involve training data, but their forms are similar to ours. It appears that our Theorem 3 gives a bound that holds uniformly for a larger class of loss functions ℓ and joint distributions of Y and X; however, in [4], several other bounds are presented that are tighter and/or allow more general distributions, for specific fixed loss functions.

• Organization

This paper is organized as follows. In Section 2, we characterize universally lossless transformations and introduce a novel strongly consistent test for the property of universal losslessness. In Section 3, information-theoretic bounds on the excess minimum risk are developed and are used to characterize the $\delta$-losslessness property of transformations. Section 4 surveys connections with, and applications to, specific prediction problems, as well as the information bottleneck method in deep learning. The somewhat lengthy proof of the strong consistency of the test in Theorem 2 is given in Section 5. Concluding remarks are given in Section 6.

2. Testing the Universal Losslessness Property

In this section, we first give a characterization of universally lossless transformations for a given distribution of the pair $(X,Y)$. In practice, the distribution of $(X,Y)$ may not be known, but a sequence of independent and identically distributed (i.i.d.) copies of $(X,Y)$ may be available. For this case, we construct a procedure to test if a given transformation is universally lossless and prove that, under mild conditions, the test is strongly consistent.

2.1. Universally Lossless Transformations

Based on Definition 1, we introduce the null hypothesis

$$\begin{array}{c}\hfill {\mathcal{H}}_0=\{T:\mathrm{the} \mathrm{transformation}T\mathrm{is} \mathrm{universally} \mathrm{lossless}.\}\end{array}$$

(1)

A transformation (statistic) $T\left(X\right)$ is called sufficient if the random variables Y, $T\left(X\right)$, X form a Markov chain in this order, denoted by $Y\to T\left(X\right)\to X$ (see, e.g., Definition 3.8 and Theorem 3.9 in Polyanskiy and Wu [9]).

For a binary valued Y, Theorems 32.5 and 32.6 from Devroye et al. [10] imply that the statistic $T\left(X\right)$ is universally lossless if, and only if, it is sufficient. The following theorem extends this property to general Y. This result is likely known, but we could not find it in the given form.

Theorem 1.

The transformation T is universally lossless if, and only if, $Y\to T\left(X\right)\to X$ is a Markov chain.

Proof. 

Assume first that $Y\to T\left(X\right)\to X$ is a Markov chain. This is equivalent to having $\mathbb{P}(Y\in A|X,T\left(X\right))=\mathbb{P}(Y\in A\left|T\right(X\left)\right)$ almost surely (a.s.) for any measurable $A\subset \mathcal{Y}$. Then we have

$$\begin{array}{cc}\hfill L_{\ell }^{\ast }\left(Y\right|X,T\left(X\right))& =\mathbb{E}\left[\underset{y\in \mathcal{Y}}{inf}\mathbb{E}[\ell (Y,y)|X,T\left(X\right)]\right]\hfill \\ \hfill & =\mathbb{E}\left[\underset{y\in \mathcal{Y}}{inf}\mathbb{E}[\ell (Y,y)|T\left(X\right)]\right]\hfill \\ \hfill & =L_{\ell }^{\ast }\left(Y\right|T\left(X\right)).\hfill \end{array}$$

Since $L_{\ell }^{\ast }\left(Y\right|X,T\left(X\right))\le L_{\ell }^{\ast }\left(Y\right|X)\le L_{\ell }^{\ast }\left(Y\right|T\left(X\right))$ always holds, we obtain $L_{\ell }^{\ast }\left(Y\right|T\left(X\right))=L_{\ell }^{\ast }\left(Y\right|X)$ for all ℓ, so $T\left(X\right)$ is universally lossless.

Now, assume that the Markov chain condition $Y\to T\left(X\right)\to X$ does not hold. Then, there exists a measurable $A\subset \mathcal{Y}$ with $0<\mathbb{P}(Y\in A)<1$ and $B\subset {\mathbb{R}}^d$ with $\mathbb{P}(X\in B)>0$, such that

$$\mathbb{P}(Y\in A|X,T\left(X\right))\ne \mathbb{P}(Y\in A\left|T\right(X\left)\right)\mathrm{if}X\in B.$$

Let $h\left(y\right)={\mathbb{I}}_{y\in A}$, where ${\mathbb{I}}_E$ is the indicator function of event E, and define the binary valued $\widehat{Y}$ as $\widehat{Y}=h\left(Y\right)$. Then, the Markov chain condition $\widehat{Y}\to T\left(X\right)\to X$ does not hold. For this special case, Theorems 32.5 and 32.6 in [10] show that there a loss function $\widehat{\ell }:{\{0,1\}}^2\to {\mathbb{R}}_{+}$ exists, such that $L_{\widehat{\ell }}^{\ast }\left(\widehat{Y}\right|T\left(X\right))>L_{\widehat{\ell }}^{\ast }\left(\widehat{Y}\right|X)$. Finally, letting $\ell (y,y^{\prime })=\widehat{\ell }(h\left(y\right),h\left(y^{\prime }\right))$, we have

$$L_{\ell }^{\ast }\left(Y\right|T\left(X\right))=L_{\widehat{\ell }}^{\ast }\left(\widehat{Y}\right|T\left(X\right))>L_{\widehat{\ell }}^{\ast }\left(\widehat{Y}\right|X)=L_{\ell }^{\ast }\left(Y\right|X),$$

which shows that $T\left(X\right)$ is not universally lossless. □

2.2. A Strongly Consistent Test

Theorem 1 implies an equivalent form of the losslessness null hypothesis defined by (1)

$$\begin{array}{c}\hfill {\mathcal{H}}_0=\{T:Y\to T\left(X\right)\to X\mathrm{is} \mathrm{a} \mathrm{Markov} \mathrm{chain}\},\end{array}$$

(2)

or equivalently, ${\mathcal{H}}_0$ holds if and only if X and Y are conditionally independent given $T\left(X\right)$:

$${\mathcal{H}}_0:\mathbb{P}\left(X\in A,Y\in B\mid T\left(X\right)\right)=\mathbb{P}\left(X\in A\mid T\left(X\right)\right)\mathbb{P}\left(Y\in B\mid T\left(X\right)\right)\mathrm{a}.\mathrm{s}.$$

for arbitrary Borel sets $A,B$. Furthermore, we consider the general case where the alternative hypothesis ${\mathcal{H}}_1$ is the complement of ${\mathcal{H}}_0$: ${\mathcal{H}}_1={\mathcal{H}}_0^c$.

Now, assume that the joint distribution of $(X,Y,T(X\left)\right)$ is not known but instead a sample of independent and identically distributed (i.i.d.) random vectors $(X_1,Y_1,Z_1),\dots ,(X_n,Y_n,Z_n)$ having a common distribution of $(X,Y,Z)$ is given, where $Z_i=T\left(X_i\right)$ and $Z=T\left(X\right)$.The goal is to test the hypothesis ${\mathcal{H}}_0$ of conditional independence based on these data. In fact, our goal is to provide a strongly consistent test; i.e., a test that, with a probability of one, only makes finitely many Type I and II errors.

For testing conditional independence, most of the results in the literature used real valued $X,Y,Z$. Based on kernel density estimation, Cai et al. [11] introduced a test statistic and under the null hypothesis calculated its limit distribution. In Neykov et al. [12], a gap was introduced between the null and alternative hypotheses. This gap was characterized by the total variation distance, which decreased with increasing n. Under certain smoothness conditions, minimax bounds were derived. According to Shah and Peters [13], a regularity condition such as our Lipschitz condition (5) below cannot be omitted if a test for conditional independence is to be consistent. This is a consequence of their no-free lunch theorem that states that, under general conditions, if with the null hypothesis the bound on the error probability is non-asymptotic, then under the alternative hypothesis the rate of convergence of the error probability can be arbitrarily slow, which is a well-known phenomenon in nonparametric statistics. We note that these cited results, and indeed most of the results in the literature when testing for conditional independence, were for real-valued random variables and/or assumed certain special distribution types, typically the existence of a joint probability density function or that both X and Y are discrete, as in [12]. As we remarked earlier, such assumptions exclude problems where Y is discrete and X is continuous (typical in classification) or problems where the observation X is concentrated on a lower dimensional subspace or manifold. In contrast, our test construction is completely distribution-free and its convergence properties are almost distribution-free (we do assume a mild Lipschitz-type condition; see the upcoming Condition 1).

In our hypotheses testing setup, the alternative hypothesis, ${\mathcal{H}}_1$, is the complement to the null hypothesis, ${\mathcal{H}}_0$; therefore, there is no separation gap between the hypotheses. Dembo and Peres [14] and Nobel [15] characterized hypothesis pairs that admitted strongly consistent tests; i.e., tests that with a probability of one only make finitely many Type I and II errors. This property is called discernibility. As an illustration of the intricate nature of the discernibility concept, Dembo and Peres [14] demonstrated an exotic example, where the null hypothesis is that the mean of a random variable is rational, while the alternative hypothesis is that this mean minus $\sqrt{2}$ is rational. (See also Cover [16] and Kulkarni and Zeitouni [17].) The discernibility property shows up in Biau and Györfi [18] (testing homogeneity), Devroye and Lugosi [19] (classification of densities), Gretton and Györfi [20] (testing independence), Morvai and Weiss [21] and Nobel [15] (classification of stationary processes), among others.

In the remainder of this section, under mild conditions for the distribution of $(X,Y)$, we study discernibility in the context of lossless transformations for statistical inference with general risk. We will make strong use of the multivariate-partitioning-based test of Györfi and Walk [2].

Let $P_{XYZ}$ denote the joint distribution of $(X,Y,Z)$ and similarly for any marginal distribution of $(X,Y,Z)$; e.g., $P_{XZ}$ denotes the distribution of the pair $(X,Z)$. As in Györfi and Walk [2], introduce the following empirical distributions:

$$\begin{array}{cc}\hfill P_{XYZ}^n(A,B,C)& =\frac{\#\{i:(X_i,Y_i,Z_i)\in A\times B\times C,i=1,\dots ,n\}}{n},\hfill \\ \hfill P_{XZ}^n(A,C)& =\frac{\#\{i:(X_i,Z_i)\in A\times C,i=1,\dots ,n\}}{n},\hfill \\ \hfill P_{YZ}^n(B,C)& =\frac{\#\{i:(Y_i,Z_i)\in B\times C,i=1,\dots ,n\}}{n},\hfill \end{array}$$

and

$$P_Z^n\left(C\right)=\frac{\#\{i:Z_i\in C,i=1,\dots ,n\}}{n},$$

for Borel sets $A\subset {\mathbb{R}}^d$, $B\subset \mathbb{R}$, and $C\subset {\mathbb{R}}^{d^{\prime }}$.

For the sake of simplicity, assume that X, Y, and $Z=T\left(X\right)$ are bounded. Otherwise, we apply a componentwise, one-to-one scaling in the interval $[0,1]$. Obviously, the losslessness null hypothesis ${\mathcal{H}}_0$ is invariant under such a scaling. Let

$${\mathcal{P}}_n=\{A_{n,1},\dots ,A_{n,m_n}\},Q_n=\{B_{n,1},\dots ,B_{n,m_n^{\prime }}\},{\mathcal{R}}_n=\{C_{n,1},\dots ,C_{n,m_n^{\prime \prime }}\}$$

be the finite cubic partitions of the ranges X, Y, and Z, with all the cubes having common side lengths $h_n$ (thus, $h_n$ is proportional to $1/m_n^{\prime }$). As in [2], we define the test statistic

$$\begin{array}{cc}\hfill L_n& =\sum _{A\in {\mathcal{P}}_n,B\in Q_n,C\in {\mathcal{R}}_n}\left|P_{XYZ}^n(A,B,C)-\frac{P_{XZ}^n(A,C)P_{YZ}^n(B,C)}{P_Z^n\left(C\right)}\right|.\hfill \end{array}$$

(3)

Our test rejects ${\mathcal{H}}_0$ if

$$L_n\ge t_n,$$

and accepts it if $L_n<t_n$, where the threshold $t_n$ is set to

$$\begin{array}{cc}\hfill t_n& =c_1\left(\sqrt{\frac{m_n{m}_n^{\prime }{m}_n^{\prime \prime }}{n}}+\sqrt{\frac{m_n^{\prime }{m}_n^{\prime \prime }}{n}}+\sqrt{\frac{m_n{m}_n^{\prime \prime }}{n}}+\sqrt{\frac{m_n^{\prime \prime }}{n}}\right)+(logn)h_n,\hfill \end{array}$$

(4)

where the constant $c_1$ satisfies

$$c_1>\sqrt{2log2}\approx 1.177.$$

In this setup, the distribution of $(X,Y)$ is arbitrary; its components can be discrete or absolutely continuous, or a mixture of the two or even singularly continuous. It is important to note that to construct this test, there is no need to know the type of distribution.

We assume that the joint distribution of X, Y, and $Z=T\left(X\right)$ satisfies the following assumption.

Condition 1.

Let $p(·|z)$ be the density of the conditional distribution $P_{X|Z=z}=\mathbb{P}(X\in ·|Z=z)$ with respect to the distribution $P_X$ as a dominating measure and introduce the notation

$$C_n\left(z\right)=C_{n,j}ifz\in C_{n,j}.$$

Assume that for some $C^{\ast }>0$, $p\left(x\right|z)$ satisfies the condition

$$\begin{array}{cc}\hfill \int \int |p\left(x\right|z)-\frac{{\int }_{C_n\left(z\right)}p\left(x\right|z^{\prime })P_Z\left(d{z}^{\prime }\right)}{P_Z\left(C_n\left(z\right)\right)}|P_X\left(dx\right)P_Z\left(dz\right)& \le C^{\ast }{h}_n,\hfill \end{array}$$

(5)

for all n.

We note that the ordinary Lipschitz condition

$$\begin{array}{cc}\hfill \int |p\left(x\right|z)-p\left(x\right|z^{\prime })|P_X\left(dx\right)& \le \frac{C^{\ast }}{\sqrt{d^{\prime }}}\parallel z-z^{\prime }\parallel \mathrm{for}\mathrm{all}z,z^{\prime }\in {\mathbb{R}}^{d^{\prime }}\hfill \end{array}$$

(6)

implies (5). This latter condition is equivalent to

$$d_{TV}\left(P_{X|Z=z},P_{X|Z=z^{\prime }}\right)\le \frac{C^{\ast }}{2\sqrt{d^{\prime }}}\parallel z-z^{\prime }\parallel \mathrm{for}\mathrm{all}z,z^{\prime }\in {\mathbb{R}}^{d^{\prime }},$$

where $d_{TV}(P,Q)$ denotes the total variation distance between distributions P and Q. In Neykov, Balakrishnan, and Wasserman [12], condition (6) is called the Null TV Lipschitz condition.

The next theorem is an adaptation and extension of the results in Györfi and Walk [2] to this particular problem of lossless transformation. In [2], it was assumed that the sequence of partitions $\{{\mathcal{P}}_n,Q_n,{\mathcal{R}}_n\}$ is nested, while we make no such assumption. The proof, in which an error made in [2] is also corrected, is relegated to Section 5.

Theorem 2.

Suppose that X, Y, and $Z=T\left(X\right)$ are bounded and Condition 1 holds for all n. If the sequence $h_n$ satisfies

$$\underset{n\to \infty }{lim}n{h}_n^{d+1+d^{\prime }}=\infty$$

(7)

and

$$\begin{array}{c}\hfill \underset{n\to \infty }{lim}{h}_n^{d^{\prime }}logn=0,\end{array}$$

(8)

then we have the following:

(a)

Under the losslessness null hypothesis ${\mathcal{H}}_0$, we have for all $n\ge e^{C^{\ast }}$,

$$\mathbb{P}(L_n\ge t_n)\le 4e^{-(c_1^2/2-log2)m_n^{\prime \prime }},$$

(9)

and therefore, because ${\sum }_{n=1}^{\infty }\mathbb{P}(L_n\ge t_n)<\infty$ by (8) and (9), after a random sample size, the test produces no error with a probability of one.

(b)

Under the alternative hypothesis ${\mathcal{H}}_1={\mathcal{H}}_0^c$,

$$\underset{n\to \infty }{lim\; inf}{L}_n>0\mathrm{a}.s.,$$

thus, with a probability of one, after a random sample size, the test produces no error.

Remark 1.

(i)

The choice $h_n=n^{-\delta }$ with $0<\delta <1/(d+1+d^{\prime })$ satisfies both conditions (7) and (8).

(ii)

Note that using (4), $t_n$ is of order $c_1\sqrt{\frac{m_n{m}_n^{\prime }{m}_n^{\prime \prime }}{n}}+(logn)h_n$. Since we have

$$\begin{array}{c}\hfill m_n=O(1/h_n^d),m_n^{\prime }=O(1/h_n),m_n^{\prime \prime }=O(1/h_n^{d^{\prime }}),\end{array}$$

this means that $t_n$ is of order $\sqrt{1/\left(n{h}_n^{d+1+d^{\prime }}\right)}+(logn)h_n$.

An important special transformation is given by the feature selection $X_S$ defined in the Introduction. Theorem 2 demonstrates the possibility of universally lossless dimension reduction for any multivariate feature vector. Note that in the setup of feature selection, the partition ${\mathcal{P}}_n$ can be the nested version of ${\mathcal{R}}_n$ and so the calculation of the test statistic $L_n$ is easier.

3. Universally $\mathbf{\delta }$-Lossless Transformations

Here, we develop bounds on the excess minimum risk, in terms of mutual information under various assumptions about the loss function. With the aid of these bounds, we give information-theoretic sufficient conditions for a transformation T to be universally $\delta$-lossless with respect to fairly general classes of loss functions ℓ.

3.1. Preliminaries on Mutual Information

Let $P_{XY}$ denote the joint distribution of the pair $(X,Y)$ and let $P_X{P}_Y$ denote the product of the marginal distributions of X and Y, respectively. The mutual information between X and Y, denoted by $I(X;Y)$, is defined as

$$I(X;Y)=D(P_{XY}\parallel P_X{P}_Y),$$

where

$$D(P\parallel Q)=\left\{\begin{array}{cc}\int \frac{dP}{dQ}log\left(\frac{dP}{dQ}\right)dQ\hfill & \mathrm{if}P\ll Q\hfill \\ \infty \hfill & \mathrm{otherwise},\hfill \end{array}\right.$$

is the Kullback–Leibler (KL) divergence between probability distributions P and Q (here, $P\ll Q$ means that P is absolutely continuous with respect to Q with the Radon–Nikodym derivative $\frac{dP}{dQ}$). Thus, $I(X;Y)$ is always nonnegative and $I(X;Y)=0$ if and only if X and Y are independent (note that $I(X;Y)=\infty$ is possible). In this definition and throughout the paper, log denotes the natural logarithm.

For random variables U and V (both taking values in finite-dimensional Euclidean spaces), let $P_{U|V}$ denote the conditional distribution of U, given V. Furthermore, let $P_{U|V=v}$ denote the stochastic kernel (regular conditional probability) induced by $P_{U|V}$. Thus, in particular, $P_{U|V=v}\left(A\right)=\mathbb{P}(U\in A|V=v)$ for each measurable set A.

Given another random variable Z, the conditional mutual information $I(X;Y|Z)$ is defined as

$$I(X;Y|Z)=\int D(P_{XY|Z=z}\parallel P_{X|Z=z}{P}_{Y|Z=z})P_Z\left(dz\right).$$

The integral above can also be denoted by $D(P_{YX|Z}\parallel P_{Y|Z}{P}_{X|Z}|P_Z)$ and is called the conditional KL divergence. One can define

$$I(X;Y|Z=z)=D(P_{XY|Z=z}\parallel P_{X|Z=z}{P}_{Y|Z=z})$$

so that

$$I(X;Y|Z)=\int I(X;Y|Z=z)P_Z\left(dz\right).$$

(10)

From this definition it is clear that $I(X;Y|Z)=0$ if and only if X and Y are conditionally independent given Z, i.e., if and only if $Y\to Z\to X$ (or equivalently, if and only if $X\to Z\to Y$).

Another way of expressing $I(X;Y)$ is

$$I(X;Y)=\int D(P_{Y|X=x}\parallel P_Y)P_X\left(dx\right).$$

(11)

One can see that in a similar way to $I(X;Y|Z)$ can be expressed as

$$I(X;Y|Z)=\int \int D(P_{Y|X=x,Z=z}\parallel P_{Y|Z=z})P_{X|Z=z}\left(dx\right)P_Z\left(dz\right).$$

(12)

Properties of mutual information and conditional mutual information, their connections to the KL divergence, and identities involving these information measures are detailed in, e.g., Cover and Thomas ([22], Chapter 2) and Polyanskiy and Wu ([9], Chapter 3).

3.2. Mutual Information Bounds and $\delta$-Lossless Transformations

A real random variable U with finite expectation is said to be ${\sigma }^2$-sub-Gaussian for some ${\sigma }^2>0$ if

$$log\mathbb{E}\left[e^{\lambda (U-\mathbb{E}[U\left]\right)}\right]\le {\displaystyle \frac{{\sigma }^2{\lambda }^2}{2}}\mathrm{for}\mathrm{all}\lambda \in \mathbb{R}.$$

Furthermore, we say that U is conditionally ${\sigma }^2$-sub-Gaussian given another random variable V if we have a.s.

$$log\mathbb{E}\left[e^{\lambda (U-\mathbb{E}[U\left|V\right])}|V\right]\le {\displaystyle \frac{{\sigma }^2{\lambda }^2}{2}}\mathrm{for}\mathrm{all}\lambda \in \mathbb{R}.$$

(13)

The following result gives a quantitative upper bound on the excess minimum risk $L_{\ell }^{\ast }\left(Y\right|T\left(X\right))-L_{\ell }^{\ast }\left(Y\right|X)$ in terms of the mutual information difference $I(Y;X)-I(Y;T(X\left)\right)$ under certain, not too restrictive, conditions. Note that $L_{\ell }^{\ast }\left(Y\right|T\left(X\right))-L_{\ell }^{\ast }\left(Y\right|X)\ge 0$ always holds.

Given $ϵ>0$, we call an estimator $f^{\prime }:{\mathbb{R}}^d\to \mathcal{Y}$$ϵ$-optimal if $\mathbb{E}[\ell (Y,f^{\prime }\left(X\right))]<L_{\ell }^{\ast }\left(Y\right|X)+ϵ$.

Theorem 3.

Let $T:{\mathbb{R}}^d\to {\mathbb{R}}^{d^{\prime }}$ be a measurable transformation and assume that for any $ϵ>0$, there exists an ϵ-optimal estimator $f^{\prime }$ of Y from X, such that $\ell (y,f^{\prime }\left(X\right)))$ is conditionally ${\sigma }^2\left(y\right)$-sub-Gaussian given $T\left(X\right)$ for every $y\in \mathcal{Y}$, i.e.,

$$log\mathbb{E}\left[e^{\lambda \left(\ell (y,f^{\prime }\left(X\right))-\mathbb{E}[\ell (y,f^{\prime }\left(X\right))|T\left(X\right)]\right)}\mid T\left(X\right)\right]\le {\displaystyle \frac{{\sigma }^2\left(y\right){\lambda }^2}{2}}\mathit{a}.\mathit{s}$$

(14)

for all $\lambda \in \mathbb{R}$ and $y\in \mathbb{R}$, where ${\sigma }^2:\mathbb{R}\to {\mathbb{R}}_{+}$ satisfies $\mathbb{E}\left[{\sigma }^2\left(Y\right)\right]<\infty$. Then, one has

$$\begin{array}{cc}\hfill L_{\ell }^{\ast }\left(Y\right|T\left(X\right))-L_{\ell }^{\ast }\left(Y\right|X)& \le \sqrt{2\mathbb{E}\left[{\sigma }^2\left(Y\right)\right]I(Y;X|T\left(X\right))}\hfill \\ \hfill & =\sqrt{2\mathbb{E}\left[{\sigma }^2\left(Y\right)\right]\left(I(Y;X)-I(Y;T(X\left)\right)\right)}.\hfill \end{array}$$

(15)

Remark 2.

(i)

In case $I(Y;X|T\left(X\right))=\infty$, we interpret the right hand side of (15) as ∞. With this interpretation, the bound always holds.

(ii)

We show in Section 4.2 that the sub-Gaussian condition (14) holds for the regression problem with squared error $\ell (y,y^{\prime })={(y-y^{\prime })}^2$ if $Y=m\left(X\right)+N$, where N is independent noise having a zero mean and finite fourth moment $\mathbb{E}\left[N^4\right]<\infty$, and the regression function $m\left(x\right)=E\left[Y\right|X=x]$ is bounded. In particular, the bound in the theorem holds if N is normal with zero mean and m is bounded.

We note that Theorem 6 and Corollary 3 in Xu and Raginsky [4] give bounds similar to (15), in the somewhat different context of Bayesian learning. However, the conditions there exclude, e.g., regression models in the form $Y=m\left(X\right)+N$ if $\mathbb{E}\left[\right|N{|}^{\delta }]=\infty$ for some $\delta >0$.

(iii)

Although hidden in the notation, $\mathbb{E}\left[{\sigma }^2\left(Y\right)\right]$ depends on the loss function ℓ. Thus, the upper bound (15) is the product of two terms, the second of which,

$$\sqrt{I(Y;X)-I(Y;T(X\left)\right)},$$

is independent of the loss function.

(iv)

The bound in the theorem is not tight in general. In Section 4.3, an example is given in the context of portfolio selection, where the excess risk can be upper bounded by the difference $I(Y;X)-I(Y;T(X\left)\right)$.

(v)

The proof of Theorem 3 and those of its corollaries go through virtually without change if we replace $T\left(X\right)$ with any ${\mathbb{R}}^{d^{\prime }}$-valued random variable Z, such that $Y\to X\to Z$. Under the conditions of the theorem, we then have

$$\begin{array}{cc}\hfill L_{\ell }^{\ast }\left(Y\right|Z)-L_{\ell }^{\ast }\left(Y\right|X)& \le \sqrt{2\mathbb{E}\left[{\sigma }^2\left(Y\right)\right]I(Y;X|Z)}\hfill \\ \hfill & =\sqrt{2\mathbb{E}\left[{\sigma }^2\left(Y\right)\right]\left(I(Y;X)-I(Y;Z)\right)}.\hfill \end{array}$$

In fact, Theorem 3 and its corollaries hold for general random variables Y, X, and Z taking values in complete and separable metric (Polish) spaces $\mathcal{Y}$, $\mathcal{X}$, and $\mathcal{Z}$, respectively, if $Y\to X\to Z$.

The proof of Theorem 3 is based on a slight generalization of Raginsky et al. ([5], Lemma 10.2), which we state next. In the lemma, U and V are arbitrary abstract random variables defined for the same probability space and taking values in spaces $\mathcal{U}$ and $\mathcal{V}$, respectively; $\overline{U}$ and $\overline{V}$ are independent copies of U and V (so that $P_{\overline{U}\overline{V}}=P_U{P}_V$); and $h:\mathcal{U}\times \mathcal{V}\to \mathbb{R}$ is a measurable function.

Lemma 1.

Assume that $h(u,V)$ is ${\sigma }^2\left(u\right)$-sub-Gaussian for all $u\in \mathcal{U}$, where $\mathbb{E}\left[{\sigma }^2\left(U\right)\right]<\infty$. Then,

$$|\mathbb{E}\left[h(U,V)\right]-\mathbb{E}\left[h(\overline{U},\overline{V})\right]|\le \sqrt{2\mathbb{E}\left[{\sigma }^2\left(U\right)\right]I(U;V)}.$$

Proof. 

We essentially copy the proof of ([5], Lemma 10.2), where it was assumed that ${\sigma }^2\left(u\right)$ does not depend on u. With this restriction, the sub-Gaussian condition (14) in Theorem 3 would have to hold with ${\sigma }^2\left(y\right)\le {\sigma }^2$ uniformly over y. This condition would exclude regression models with independent sub-Gaussian noise and, a fortiori, models with independent noise that do not possess finite absolute moments of all orders, while our Theorem 2 can also be applied in such cases (see Section 4.2).

We make use of the Donsker–Varadhan variational representation of the relative entropy ([23] Corollary 4.15), which states that

$$D(P\parallel Q)=\underset{F}{sup}\left(\int FdP-log\int e^{F}dQ\right),$$

where the supremum is over all measurable $F:\Omega \to \mathbb{R}$, such that $\int e^{F}dQ<\infty$. Applying this with $P=P_{V|U=u}$, $Q=P_V$, and $F=\lambda h(u,V)$, we obtain

$$\begin{array}{cc}\hfill D(P_{V|U=u}\parallel P_V)& \ge \mathbb{E}\left[\lambda h(u,V)\right|U=u]-log\mathbb{E}\left[e^{\lambda h(u,V)}\right]\hfill \\ \hfill & \ge \lambda \left(\mathbb{E}\left[h\right(u,V\left)\right|U=u]-\mathbb{E}[\lambda h(u,V)]\right)-\frac{{\lambda }^2{\sigma }^2\left(u\right)}{2},\hfill \end{array}$$

(16)

where the second inequality follows from the assumption that $h(u,V)$ is ${\sigma }^2\left(u\right)$-sub-Gaussian. Maximizing the right-hand side of (16) over $\lambda \in \mathbb{R}$ gives, after rearrangement,

$$|\mathbb{E}\left[h(u,V)\right|U=u]-\mathbb{E}\left[h(u,V)\right]|\le \sqrt{2{\sigma }^2\left(u\right)D(P_{V|U=u}\parallel P_V)}.$$

(17)

Since $\overline{U}$ and $\overline{V}$ are independent, $\mathbb{E}\left[h(u,V)\right]=\mathbb{E}\left[h(\overline{U},\overline{V})\right|\overline{U}=u]$, and we obtain

$$\begin{array}{l}|\mathbb{E}\left[h(U,V)\right]-\mathbb{E}\left[h(\overline{U},\overline{V})\right]|\hfill \\ =|\int \left(\mathbb{E}\left[h(U,V)\right|U=u]-\mathbb{E}\left[h(\overline{U},\overline{V})\right|\overline{U}=u]\right)P_U\left(du\right)|\hfill \\ =|\int \left(\mathbb{E}\left[h\right(u,V\left)\right|U=u]-h(u,V)\right)P_U\left(du\right)|\hfill \\ \le \int |\mathbb{E}\left[h(u,V)\right|U=u]-h(u,V)|P_U\left(du\right)\hfill \end{array}$$

(18)

$$\le \int \sqrt{2{\sigma }^2\left(u\right)D(P_{V|U=u}\parallel P_V)}{P}_U\left(du\right)$$

(19)

$$\le \sqrt{\int 2{\sigma }^2\left(u\right)P_U\left(du\right)}\sqrt{\int D(P_{V|U=u}\parallel P_V)P_U\left(du\right)}$$

(20)

$$=\sqrt{2\mathbb{E}\left[{\sigma }^2\left(U\right)\right]I(U;V)},$$

(21)

where (18) follows from Jensen’s inequality, (19) follows from (17), in (20) we used the Cauchy–Schwarz inequality, and the last equality follows from (11). □

Proof of Theorem 3.

Let $\overline{Y}$ and $\overline{X}$ be random variables, such that $P_{\overline{Y}|T\left(\overline{X}\right)}=P_{Y\left|T\right(X)}$, $P_{\overline{X}|T\left(\overline{X}\right)}=P_{X\left|T\right(X)}$, $P_{T\left(\overline{X}\right)}=P_{T\left(X\right)}$, and $\overline{Y}$ and $\overline{X}$ are conditionally independent given $T\left(\overline{X}\right)$. Thus, the joint distribution of the triple $(\overline{Y},\overline{X},T\left(\overline{X}\right))$ is $P_{\overline{Y}\overline{X}T\left(\overline{X}\right)}=P_{Y\left|T\right(X)}{P}_{X\left|T\right(X)}{P}_{T\left(X\right)}$.

We apply Lemma 1 with $U=Y$, $V=X$, and $h(u,v)=\ell (y,f^{\prime }\left(x\right))$. Note that, using the conditions of the theorem, we can choose an $ϵ$-optimal $f^{\prime }$, such that for every y, $\ell (y,f^{\prime }\left(X\right))$ is conditionally ${\sigma }^2\left(y\right)$-sub-Gaussian given $T\left(X\right)$. Consider $\mathbb{E}[\ell (Y,f^{\prime }\left(X\right))|T\left(X\right)=z]$ and $\mathbb{E}[\ell (\overline{Y},f^{\prime }\left(\overline{X}\right)),|T\left(\overline{X}\right)=z]$ as regular (unconditional) expectations taken with respect to $P_{YX\left|T\right(X)=z}$ and $P_{\overline{Y}\overline{X}|T\left(\overline{X}\right)=z}$ respectively, and consider $I(Y;X|T\left(X\right)=z)$ as regular mutual information between random variables with the distribution $P_{YX\left|T\right(X)=z}$. Since $\overline{Y}$ and $\overline{X}$ are conditionally independent given $T\left(\overline{X}\right)=z$, Lemma 1 yields

$$\begin{array}{c}\hfill |\mathbb{E}[\ell (Y,f^{\prime }\left(X\right))|T\left(X\right)=z]-\mathbb{E}[\ell (\overline{Y},f^{\prime }\left(\overline{Y}\right))|T\left(\overline{X}\right)=z]|\\ \hfill \le \sqrt{2\mathbb{E}\left[{\sigma }^2\left(Y\right)\right|T\left(X\right)=z]I(X;Y|T\left(X\right)=z)}.\end{array}$$

Recalling that $T\left(\overline{X}\right)$ and $T\left(X\right)$ have the same distribution, and applying Jensen’s inequality and the Cauchy–Schwarz inequality as in (18) and (20), we obtain

$$\begin{array}{c}|\mathbb{E}[\ell (Y,f^{\prime }\left(X\right))]-\mathbb{E}[\ell (\overline{Y},f^{\prime }\left(\overline{X}\right))]|\\ \le \int |\mathbb{E}[\ell (Y,f^{\prime }\left(X\right))|T\left(X\right)=z]-\mathbb{E}[\ell (\overline{Y},f^{\prime }\left(\overline{X}\right))|T\left(\overline{X}\right)=z]|P_{T\left(X\right)}\left(dz\right)\\ \le \int \sqrt{2\mathbb{E}\left[{\sigma }^2\left(Y\right)\right|T\left(X\right)=z]I(Y;X|T\left(X\right)=z)}{P}_{T\left(X\right)}\left(dz\right)\\ \le \sqrt{2\int \mathbb{E}\left[{\sigma }^2\left(Y\right)\right|T\left(X\right)=z]P_{T\left(X\right)}\left(dz\right)}\sqrt{\int I(Y;X|T\left(X\right)=z)P_{T\left(X\right)}\left(dz\right)}\\ =\sqrt{2\mathbb{E}\left[{\sigma }^2\left(Y\right)\right]I(Y;X|T\left(X\right))}.\end{array}$$

(22)

On the one hand, we have

$$\mathbb{E}\left[\ell (\overline{Y};f^{\prime }\left(\overline{X}\right))\right]\ge L_{\ell }^{\ast }\left(\overline{Y}\right|\overline{X})=L_{\ell }^{\ast }\left(\overline{Y}\right|T\left(\overline{X}\right))=L_{\ell }^{\ast }\left(Y\right|T\left(X\right)),$$

(23)

where the first equality follows from Theorem 1 with the conditional independence of $\overline{Y}$ and $\overline{X}$ given $T\left(\overline{X}\right)$, and the second follows, since $(\overline{Y},T\left(\overline{X}\right))$ and $(Y,T(X\left)\right)$ have the same distribution by construction. On the other hand, $L_{\ell }^{\ast }\left(Y\right|X)\ge \mathbb{E}[\ell (Y,f^{\prime }\left(X\right))]-ϵ$. Thus, (22) and (23) imply

$$\begin{array}{cc}\hfill 0\le L_{\ell }^{\ast }\left(Y\right|T\left(X\right))-L_{\ell }^{\ast }\left(Y\right|X)& \le \mathbb{E}[\ell (\overline{Y},f^{\prime }\left(\overline{X}\right))]-\mathbb{E}[\ell (Y,f^{\prime }\left(X\right))]+ϵ\hfill \\ \hfill & \le \sqrt{2\mathbb{E}\left[{\sigma }^2\left(Y\right)\right]I(Y;X|T\left(X\right))}+ϵ,\hfill \end{array}$$

which proves the upper bound in (15), since $ϵ>0$ is arbitrary. By expanding $I(Y;X|Z)$ in two different ways using the chain rule for mutual information (e.g., Cover and Thomas ([22], Thm. 2.5.2)), and using the conditional independence of Y and $T\left(X\right)$ given X, one obtains $I(Y;X|T\left(X\right))=I(Y;X)-I(Y;T\left(X\right))$, which shows the equality in (15). □

We state two corollaries for special cases. In the first, we assume that ℓ is uniformly bounded, i.e., ${\parallel \ell \parallel }_{\infty }={sup}_{y,y^{\prime }\in \mathcal{Y}}\ell (y,y^{\prime })<\infty$. For any $c>0$, let $\mathcal{L}\left(c\right)$ denote the collection of all loss functions ℓ with ${\parallel \ell \parallel }_{\infty }\le c$. Recall the notion of a universally $\delta$-lossless transformation from Definition 2.

Corollary 1.

Suppose the loss function ℓ is bounded. Then, for any measurable $T:{\mathbb{R}}^d\to {\mathbb{R}}^{d^{\prime }}$, we have

$$\begin{array}{cc}\hfill L_{\ell }^{\ast }\left(Y\right|T\left(X\right))-L_{\ell }^{\ast }\left(Y\right|X)& \le \frac{{\parallel \ell \parallel }_{\infty }}{\sqrt{2}}\sqrt{I(Y;X)-I(Y;T(X\left)\right)}.\end{array}$$

(24)

Therefore, whenever

$$I(Y;X)-I(Y;T\left(X\right))\le \frac{2{\delta }^2}{c^2},$$

(25)

the transformation T is universally δ-lossless for the family $\mathcal{L}\left(c\right)$, i.e., $L_{\ell }^{\ast }\left(Y\right|T\left(X\right))-L_{\ell }^{\ast }\left(Y\right|X)\le \delta$ for all ℓ with ${\parallel \ell \parallel }_{\infty }\le c$.

Remark 3.

(i)

The bound of the theorem can be used to give an estimation-theoretic motivation of the information bottleneck (IB) problem; see Section 4.4.

(ii)

Let $L_{\ell }^{\ast }\left(Y\right)=L_{\ell }^{\ast }\left(Y\right|\varnothing )={inf}_{y\in \mathcal{Y}}\mathbb{E}[\ell (Y,y)]$. For bounded ℓ, the inequality

$$L_{\ell }^{\ast }\left(Y\right)-L_{\ell }^{\ast }\left(Y\right|X)\le 2\sqrt{2}{\parallel \ell \parallel }_{\infty }\sqrt{I(Y;X)}$$

was proven in Makhdoumi et al. ([24], Theorem 1) for discrete alphabets, to solve the so-called privacy funnel problem.This inequality follows from (15) by setting $Z=T\left(X\right)$ to be constant there.

(iii)

A simple self-contained proof of (24) (see below) was provided by Or Ordentlich and communicated to the second author by Shlomo Shamai [25], in response to an early version of this manuscript. The bound in (24) seems to have first appeared in published form in Hafez-Kolahi et al. ([26], Lemma 1), where the proof was attributed to Xu and Raginsky [27].

Proof of Corollary 1. 

If ℓ is uniformly bounded, then for any $f:{\mathbb{R}}^d\to \mathcal{Y}$ one has $\ell (y,f\left(x\right))\in [0,\parallel \ell {\parallel }_{\infty }]$ for all y and x. Then Hoeffding’s lemma (e.g., Boucheron et al. ([23], Lemma 2.2)) implies that for all y, $\ell (y,f(X\left)\right)$ is conditionally ${\sigma }^2$-sub-Gaussian with ${\sigma }^2=\frac{{\parallel \ell \parallel }_{\infty }^2}{4}$ given $T\left(X\right)$. Since an $ϵ$-optimal estimator $f^{\prime }$ exists for any $ϵ>0$ and $\ell (y,f^{\prime }\left(X\right))$ is conditionally ${\sigma }^2$-sub-Gaussian, given $T\left(X\right)$ using the preceding argument, (24) follows from Theorem 3. The second statement follows directly from (24) and the fact that ${\parallel \ell \parallel }_{\infty }\le c$ for all $\ell \in \mathcal{L}\left(c\right)$.

The following alternative argument by Or Ordentlich [25] is based on Pinsker’s inequality for the total variation distance in terms of the KL divergence (see, e.g., ([9], Theorem 7.9)). For bounded ℓ, this gives a direct proof of an analogue of the key inequality (22) in the proof of Theorem 3. This argument avoids Lemma 1 and the machinery introduced by the sub-Gaussian assumption.

Using the same notation as in the proof of Theorem 3 and letting $P=P_{YXZ}$ and $Q=P_{\overline{Y}\overline{X}T\left(\overline{Y}\right)}$, we have

$$\begin{array}{cc}\hfill \mathbb{E}\left[\ell (Y,f^{\prime }\left(X\right))\right]-\mathbb{E}\left[\ell (\overline{Y},f^{\prime }\left(\overline{X}\right))\right]|& =\int \ell (y,f^{\prime }\left(x\right))dP-\int \ell (y,f^{\prime }\left(x\right))dQ\hfill \\ \hfill & \le {\parallel \ell \parallel }_{\infty }{d}_{\mathrm{TV}}(P,Q)\hfill \\ \hfill & \le \frac{{\parallel \ell \parallel }_{\infty }}{\sqrt{2}}\sqrt{D(P\parallel Q)}(\mathrm{by}\mathrm{Pinsker}\prime \mathrm{s}\mathrm{inequality})\hfill \\ \hfill & =\frac{{\parallel \ell \parallel }_{\infty }}{\sqrt{2}}\sqrt{D(P_{YX\left|T\right(X)}{P}_Z\parallel P_{Y\left|T\right(X)}{P}_{X\left|T\right(X)}{P}_{T\left(X\right)})}\hfill \\ \hfill & =\frac{{\parallel \ell \parallel }_{\infty }}{\sqrt{2}}\sqrt{D(P_{YX\left|T\right(X)}\parallel P_{Y\left|T\right(X)}{P}_{X\left|T\right(X)}|P_{T\left(X\right)})}\hfill \\ \hfill & =\frac{{\parallel \ell \parallel }_{\infty }}{\sqrt{2}}\sqrt{I(X,Y|T\left(X\right))}.\hfill \end{array}$$

The rest of the proof proceeds exactly as in Theorem 3. □

In the second corollary, we do not require that ℓ be bounded but assume that an optimal estimator $f_{\ell }^{\ast }$ from X to Y exists, such that $\ell (y,f_{\ell }^{\ast }\left(X\right))$ is conditionally ${\sigma }^2\left(y\right)$-sub-Gaussian given $T\left(X\right)$, where $\mathbb{E}\left[{\sigma }^2\left(Y\right)\right]<\infty$.

Corollary 2.

Assume that an optimal estimator $f_{\ell }^{\ast }$ of Y from X exists, i.e., the measurable function $f_{\ell }^{\ast }$ satisfies $\mathbb{E}[\ell (Y,f_{\ell }^{\ast }\left(X\right))]=L_{\ell }^{\ast }\left(Y\right|X)$. Furthermore, suppose that the sub-Gaussian condition of Theorem 3 holds with $f^{\prime }=f_{\ell }^{\ast }$ (i.e., (14) holds for $f^{\prime }=f_{\ell }^{\ast })$. Then,

$$\begin{array}{cc}\hfill L_{\ell }^{\ast }\left(Y\right|T\left(X\right))-L_{\ell }^{\ast }\left(Y\right|X)& \le \sqrt{2\mathbb{E}\left[{\sigma }^2\left(Y\right)\right]\left(I(Y;X)-I(Y;T(X\left)\right)\right)}.\hfill \end{array}$$

(26)

Proof. 

The corollary immediately follows from Theorem 3, since an optimal $f_{\ell }^{\ast }$ is $ϵ$-optimal for all $ϵ>0$. □

For the next corollary, let $\widehat{\mathcal{L}}\left(c\right)$ denote the collection of all loss functions ℓ, such that

$$\ell (y,f_{\ell }^{\ast }\left(X\right))\le g_{\ell }\left(y\right)\mathrm{a}.\mathrm{s}.$$

for some function $g_{\ell }:\mathcal{Y}\to {\mathbb{R}}_{+}$ with $\mathbb{E}\left[g_{\ell }^2\left(Y\right)\right]\le c^2$.

Corollary 3.

If T is a transformation such that

$$I(Y;X)-I(Y;T\left(X\right))\le \frac{2{\delta }^2}{c^2},$$

then T is universally δ-lossless for the family $\widehat{\mathcal{L}}\left(c\right)$.

Proof. 

Since $\ell (y,f_{\ell }^{\ast }\left(X\right))$ is a.s. upper bounded by $g_{\ell }\left(y\right)$ for any $\ell \in \widehat{\mathcal{L}}\left(c\right)$, using Hoeffding’s lemma ([23], Lemma 2.2), we have that $\ell (y,f_{\ell }^{\ast }\left(X\right))$ is conditionally $\frac{g_{\ell }^2\left(y\right)}{4}$-sub-Gaussian given $T\left(X\right)$. Thus, from Corollary 2, for all $\ell \in \widehat{\mathcal{L}}\left(c\right)$, we have

$$\begin{array}{cc}\hfill L_{\ell }^{\ast }\left(Y\right|T\left(X\right))-L_{\ell }^{\ast }\left(Y\right|X)& \le \sqrt{\frac{1}{2}\mathbb{E}\left[g_{\ell }^2\left(Y\right)\right]\left(I(Y;X)-I(Y;T(X\left)\right)\right)}\hfill \\ \hfill & \le \sqrt{\frac{c^2}{2}\left(I(Y;X)-I(Y;T(X\left)\right)\right)}\hfill \\ \hfill & \le \delta \hfill \end{array}$$

if $I(Y;X)-I(Y;T\left(X\right))\le \frac{2{\delta }^2}{c^2}$. □

The next corollary generalizes and gives a much simplified proof of Faragó and Györfi [28], see also Devroye, Györfi, and Lugosi ([10], Theorem. 32.3). This result states for binary classification (Y is 0-1-valued and $\ell (y,y^{\prime })={\mathbb{I}}_{y\ne y^{\prime }}$) that if a sequence of functions $T_n:{\mathbb{R}}^d\to {\mathbb{R}}^d$ is such that $\parallel X-T_n\left(X\right)\parallel \to 0$ in probability as $n\to \infty$, then $L_{\ell }^{\ast }\left(Y\right|T_n\left(X\right))\to L_{\ell }^{\ast }\left(Y\right|X)$ as $n\to \infty$.

Corollary 4.

Assume that a sequence of transformations $T_n:{\mathbb{R}}^d\to {\mathbb{R}}^d$ is such that $T_n\left(X\right)\to X$ in distribution (i.e., $P_{T_n\left(X\right)}\to P_X$ weakly) as $n\to \infty$. Then, for any bounded loss function ℓ,

$$\underset{n\to \infty }{lim}{L}_{\ell }^{\ast }\left(Y\right|T_n\left(X\right))=L_{\ell }^{\ast }\left(Y\right|X).$$

(27)

Note that this corollary and its proof still hold without any changes if X takes values in an arbitrary complete separable metric space. For example, in the setup of function classification, X may take values in an $L_p$ function space for $1\le p<\infty$, and $T_n$ is a truncated series expansion or a quantizer. Interestingly, here the asymptotic losslessness property is guaranteed, even in the case where the sequence of transformations $T_n$ and the loss function ℓ are not matched at all.

Proof. 

If $T_n\left(X\right)\to X$ in distribution, then clearly $(Y,T_n\left(X\right))\to (Y,X)$ in distribution. Thus, the lower semicontinuity of mutual information with respect to convergence in distribution (see, e.g., Polyanskiy and Wu ([9], Equation (4.28))) implies

$$\underset{n\to \infty }{lim\; inf}I(Y;T_n\left(X\right))\ge I(Y;X).$$

Since $I(Y;T_n\left(X\right))\le I(Y;X)$ for all n, we obtain

$$\underset{n\to \infty }{lim}I(Y;T_n\left(X\right))=I(Y;X).$$

Combined with Corollary 1 (with T replaced with $T_n$), this gives

$$\begin{array}{cc}\hfill 0\le \underset{n\to \infty }{lim}{L}_{\ell }^{\ast }\left(Y\right|T_n\left(X\right))-L_{\ell }^{\ast }\left(Y\right|X)& \le \underset{n\to \infty }{lim}\frac{{\parallel \ell \parallel }_{\infty }}{\sqrt{2}}\sqrt{I(Y;X)-I(Y;T_n\left(X\right))}\hfill \\ \hfill & =0.\hfill \end{array}$$

□

4. Applications

4.1. Classification

For classification, $\mathcal{Y}$ is the finite set $\{1,\cdots ,M\}$ and the cost is the $0-1$ loss

$$\begin{array}{c}\hfill \ell (y,y^{\prime })={\mathbb{I}}_{y\ne y^{\prime }}.\end{array}$$

In this setup, the risk of estimator f is the error probability $\mathbb{P}(Y\ne f(X\left)\right)$. With the notation

$$\begin{array}{c}\hfill P_y\left(x\right)=\mathbb{P}(Y=y|X=x),\end{array}$$

the optimal estimator is the Bayes decision

$$\begin{array}{c}\hfill f^{\ast }\left(x\right)=\underset{y\in \mathcal{Y}}{\arg \max }{P}_y\left(x\right),\end{array}$$

and the minimum risk is the Bayes error probability

$$\begin{array}{c}\hfill L^{\ast }\left(X\right)=1-\mathbb{E}\left[\underset{y\in \mathcal{Y}}{max}{P}_y\left(X\right)\right].\end{array}$$

If $L^{\ast }\left(T\left(X\right)\right)=1-\mathbb{E}\left[{max}_y\mathbb{E}[P_y\left(X\right)\mid T\left(X\right)]\right]$ stands for the Bayes error probability of the transformed observation vector $T\left(X\right)$, then (24) with ${\parallel \ell \parallel }_{\infty }=1$ yields the upper bound

$$L^{\ast }\left(T\left(X\right)\right)-L^{\ast }\left(X\right)\le \frac{1}{\sqrt{2}}\sqrt{I(Y;X)-I(Y;T(X\left)\right)};$$

see also ([4], Corollary 2) for a similar bound in the context of Bayesian learning.

As a special case, the feature selector $X\mapsto X_S$ is lossless if

$$\begin{array}{c}\hfill L^{\ast }\left(X\right)=L^{\ast }\left(X_S\right).\end{array}$$

(28)

Györfi and Walk [29] studied the corresponding hypothesis testing problem. Using a k-nearest-neighbor (k-NN) estimate of the excess Bayes error probability $L^{\ast }\left(X_S\right)-L^{\ast }\left(X\right)$, they introduced a test statistic and accepted the hypothesis (28), if the test statistic is less than a threshold. Under certain mild conditions, the strong consistency of this test has been proven.

4.2. Nonparametric Regression

For the nonparametric regression problem, the cost is the squared loss

$$\begin{array}{c}\hfill \ell (y,y^{\prime })={(y-y^{\prime })}^2,y,y^{\prime }\in \mathbb{R},\end{array}$$

and the best statistical inference is the regression function

$$\begin{array}{c}\hfill m\left(X\right)=\mathbb{E}\left[Y\right|X]\end{array}$$

(here, we assume $\mathbb{E}\left[Y^2\right]<\infty$). Then, the minimum risk is the residual variance

$$\begin{array}{c}\hfill L_{\ell }^{\ast }\left(Y\right|X)=\mathbb{E}\left[{(Y-m\left(X\right))}^2\right].\end{array}$$

If $L^{\ast }\left(X\right)=L_{\ell }^{\ast }\left(Y\right|X)$ and $L^{\ast }\left(T\left(X\right)\right)=L_{\ell }^{\ast }\left(Y\right|T\left(X\right))$ denote the residual variances for the observation vectors X and $T\left(X\right)$, respectively, then

$$\begin{array}{cc}\hfill L^{\ast }\left(T\left(X\right)\right)-L^{\ast }\left(X\right)& =\mathbb{E}\left[{(Y-\mathbb{E}[m\left(X\right)\mid T\left(X\right)])}^2\right]-\mathbb{E}\left[{(Y-m\left(X\right))}^2\right]\hfill \\ \hfill & =\mathbb{E}\left[{(m\left(X\right)-\mathbb{E}[m\left(X\right)\mid T\left(X\right)])}^2\right].\hfill \end{array}$$

Note that the excess residual variance $L^{\ast }\left(T\left(X\right)\right)-L^{\ast }\left(X\right)$ does not depend on the distribution of the residual $Y-m\left(X\right)$.

Next, we show that the conditions of Corollary 2 hold with $f_{\ell }^{\ast }\left(x\right)=m\left(x\right)$ for the important case

$$Y=m\left(X\right)+N,$$

where N is a zero-mean noise variable that is independent of X and satisfies $\mathbb{E}\left[N^4\right]<\infty$, and m is bounded as $\left|m\right(x\left)\right|\le K$ for all x. For this model, we have

$$\ell (y,f^{\ast }\left(X\right))={(y-m\left(X\right))}^2\le \left(\right|y|+{\left|m\left(X\right)\right|)}^2\le {\left(\right|y|+K)}^2.$$

Thus, $\ell (y,f^{\ast }\left(X\right))$ is a nonnegative random variable a.s. bounded by ${\left(\right|y|+K)}^2$, which implies via Hoeffding’s lemma (e.g., ([23], Lemma 2.2)) that it is ${\sigma }^2\left(y\right)$-sub-Gaussian given $T\left(X\right)$ with ${\sigma }^2\left(y\right)=\frac{{\left(\right|y|+K)}^4}{4}$. We have

$$\begin{array}{cc}\hfill \mathbb{E}\left[{\sigma }^2\left(Y\right)\right]& =\frac{\mathbb{E}\left[{\left(\right|Y|+K)}^4\right]}{4}\hfill \\ \hfill & \le \frac{\mathbb{E}\left[{\left(\right|N|+|m\left(X\right)|+K)}^4\right]}{4}\hfill \\ \hfill & \le \frac{\mathbb{E}\left[{\left(\right|N|+2K)}^4\right]}{4}\hfill \\ \hfill & \le \frac{8\mathbb{E}\left[{\left|N\right|}^4+{\left(2K\right)}^4\right]}{4}\hfill \\ \hfill & \le 2\mathbb{E}\left[N^4\right]+32K^4\hfill \\ \hfill & <\infty ,\hfill \end{array}$$

thus, the conditions of Corollary 2 hold and we obtain

$$\begin{array}{cc}\hfill L_{\ell }^{\ast }\left(Y\right|T\left(X\right))-L_{\ell }^{\ast }\left(Y\right|X)& =\mathbb{E}\left[{(Y-\mathbb{E}\left[Y\right|T\left(X\right)])}^2\right]-\mathbb{E}\left[{(Y-\mathbb{E}\left[Y\right|X])}^2\right]\hfill \\ \hfill & \le \sqrt{\left(2\mathbb{E}\left[N^4\right]+32K^4\right)\left(I(Y;X)-I(Y;T(X\left)\right)\right)}.\hfill \end{array}$$

Again, the feature selection $X_S$ is called lossless, when $L^{\ast }\left(X\right)=L^{\ast }\left(X_S\right)$ holds. As a test statistic, Devroye et al. [30] introduced a 1-NN estimate of $L^{\ast }\left(X_S\right)-L^{\ast }\left(X\right)$ and proved the strong consistency of the corresponding test.

4.3. Portfolio Selection

The next example is related to the negative of the log-loss or log-utility; see Algoet and Cover [31], Barron and Cover [32], Chapters 6 and 16 in Cover and Thomas [22], Györfi et al. [33].

Consider a market consisting of $d_a$ assets. The evolution of the market in time is represented by a sequence of (random) price vectors $S_1,S_2,\dots \in {\mathbb{R}}_{+}^{d_a}$ with

$$S_n=(S_n^{\left(1\right)},\cdots ,S_n^{\left(d_a\right)}),$$

where the jth component $S_n^{\left(j\right)}$ of $S_n$ denotes the price of the jth asset in the nth trading period. Let us transform the sequence of price vectors $\left\{S_n\right\}$ into the sequence of return (relative price) vectors $\left\{R_n\right\}$, defined as

$$R_n=(R_n^{\left(1\right)},\cdots ,R_n^{\left(d_a\right)}),$$

where

$$R_n^{\left(j\right)}=\frac{S_n^{\left(j\right)}}{S_{n-1}^{\left(j\right)}}.$$

Constantly rebalanced portfolio selection is a multi-period investment strategy, where at the beginning of each trading period the investor redistributes the wealth among the assets. The investor is allowed to diversify their capital at the beginning of each trading period according to a portfolio vector $b=(b^{\left(1\right)},\cdots b^{\left(d_a\right)})$. The jth component $b^{\left(j\right)}$ of b denotes the proportion of the investor’s capital invested in asset j. Here, we assume that the portfolio vector b has nonnegative components with ${\sum }_{j=1}^{d_a}{b}^{\left(j\right)}=1$. The simplex of possible portfolio vectors is denoted by ${\Delta }_{d_a}$.

Let $S_0=1$ denote the investor’s initial capital. Then, at the beginning of the first trading period, $S_0{b}^{\left(j\right)}$ is invested into asset j, and this results in return $S_0{b}^{\left(j\right)}{R}_1^{\left(j\right)}$, and therefore at the end of the first trading period the investor’s wealth becomes

$$S_1=S_0\sum _{j=1}^{d_a}{b}^{\left(j\right)}{R}_1^{\left(j\right)}=〈b,R_1〉,$$

where $〈·,·〉$ denotes the standard inner product in ${\mathbb{R}}^{d_a}$. For the second trading period, $S_1$ is the new initial capital

$$S_2=S_1·〈b,R_2〉=〈b,R_1〉·〈b,R_2〉.$$

By induction, for the trading period n, the initial capital is $S_{n-1}$, and therefore

$$S_n=S_{n-1}〈b,R_n〉=\prod _{i=1}^n〈b,R_i〉.$$

The asymptotic average growth rate of this portfolio selection strategy is

$$\begin{array}{cc}\hfill \underset{n\to \infty }{lim}\frac{1}{n}log{S}_n& =\underset{n\to \infty }{lim}\frac{1}{n}\sum _{i=1}^{n}log〈b,R_i〉\hfill \end{array}$$

assuming a limit exists.

If the market process $\left\{R_i\right\}$ is memory-less, i.e., it is a sequence of i.i.d. random return vectors, then the strong law of large numbers implies that the best constantly rebalanced portfolio (BCRP) is the log-optimal portfolio:

$$b^{\ast }=\underset{b\in {\Delta }_{d_a}}{\arg \max }\mathbb{E}\left[log〈b,R_1〉\right],$$

while the best asymptotic average growth rate is

$$W^{\ast }=\underset{b\in {\Delta }_{d_a}}{max}\mathbb{E}\left[log〈b,R_1〉\right].$$

Barron and Cover [32] extended this setup to portfolio selection with side information. Assume that $X_1,X_2,\dots$ are ${\mathbb{R}}^d$ valued side information vectors, such that $(R_1,X_1),(R_2,X_2),\dots$ are i.i.d. and in each round n the portfolio vector may depend on $X_n$. The strong law of large numbers yields

$$\begin{array}{cc}\hfill \underset{n\to \infty }{lim}\frac{1}{n}log{S}_n& =\underset{n\to \infty }{lim}\frac{1}{n}\sum _{i=1}^{n}log〈b\left(X_i\right),R_i〉=\mathbb{E}\left[log〈b\left(X_1\right),R_1〉\right]\mathrm{a}.\mathrm{s}.\hfill \end{array}$$

Therefore, the log-optimal portfolio has the form

$$b^{\ast }\left(X_1\right)=\underset{b\in {\Delta }_{d_a}}{\arg \max }\mathbb{E}\left[log〈b,R_1〉\mid X_1\right]$$

and the best asymptotic average growth rate is

$$W^{\ast }\left(X\right)=\mathbb{E}\left[\underset{b\in {\Delta }_{d_a}}{max}\mathbb{E}\left[log〈b,R_1〉\mid X_1\right]\right].$$

Barron and Cover ([32], Thm. 2) proved that

$$W^{\ast }\left(X\right)-W^{\ast }\le I(R_1;X_1).$$

(29)

The next theorem generalizes this result by upper bounding the loss of the best asymptotic growth rate when, instead of X, only degraded side information $T\left(X\right)$ is available.

Theorem 4.

For any measurable $T:{\mathbb{R}}^d\to {\mathbb{R}}^{d^{\prime }}$,

$$W^{\ast }\left(X\right)-W^{\ast }\left(T\left(X\right)\right)\le I(R_1;X_1)-I(R_1;T\left(X_1\right))$$

assuming the terms on the right hand side are finite.

Remark 4.

(i)

As in Theorem 3, the difference $I(R_1;X_1)-I(R_1;T\left(X_1\right))$ in the upper bound is equal to $I(R_1;X_1|T\left(X_1\right))$, a quantity that is always nonnegative but may be equal to ∞. In this case, we interpret the right hand side as ∞.

(ii)

There is a correspondence between this setup of portfolio selection and the setup in previous sections. In particular, Y from the previous sections is equal to R with a range ${\mathbb{R}}_{+}^{d_a}$ and the inference is $b\left(X\right)$ taking values in ${\Delta }_{d_a}$. Then, the loss is $-log〈b\left(X\right),R〉$. If we assume that for all $j=1,\cdots d_a$,

$$\begin{array}{c}\hfill |log{R}^{\left(j\right)}|\le c_{max}\mathit{a}.\mathit{s}.,\end{array}$$

(30)

then

$$\begin{array}{c}\hfill |log〈b\left(X\right),R〉|\le c_{max}\mathit{a}.\mathit{s}.\end{array}$$

and so Corollary 1 implies

$$W^{\ast }\left(X\right)-W^{\ast }\left(T\left(X\right)\right)\le \frac{c_{max}}{\sqrt{2}}\sqrt{I(R_1;X_1)-I(R_1;T\left(X_1\right))}.$$

Note that, from the point of view of application, (30) is a mild condition. For example, for NYSE daily data $c_{max}\le 0.3$; see Györfi et al. [34].

Proof. 

Let $(R,X)$ be a generic copy of the $(R_i,X_i)$. Writing out explicitly the dependence of $W^{\ast }$ on $P_R$, we have

$$W^{\ast }\left(X\right)=\int W^{\ast }\left(P_{R|X=x}\right)P_X\left(dx\right)$$

and from (11) we have

$$I(R;X)=\int D(P_{R|X=x}\parallel P_R)P_X\left(dx\right).$$

Thus, the bound $W^{\ast }\left(X\right)-W^{\ast }\le I(R_1;X_1)$ in (29) can be written as

$$\begin{array}{cc}\hfill \int W^{\ast }\left(P_{R|X=x}\right)P_X\left(dx\right)-W^{\ast }& \le \int D(P_{R|X=x}\parallel P_R)P_X\left(dx\right).\hfill \end{array}$$

(31)

Furthermore, letting $Z=T\left(X\right)$, we have

$$\begin{array}{cc}\hfill W^{\ast }\left(X\right)-W^{\ast }\left(Z\right)& =\int W^{\ast }\left(P_{R|X=x}\right)P_X\left(dx\right)-\int W^{\ast }\left(P_{R|Z=z}\right)P_Z\left(dz\right).\hfill \end{array}$$

Since $R\to X\to Z$ is a Markov chain, $P_{R|X=x}=P_{R|X=x,Z=z}$, and we obtain

$$\begin{array}{c}\hfill W^{\ast }\left(X\right)-W^{\ast }\left(Z\right)\\ \hfill & =\int \left(\int W^{\ast }\left(P_{R|X=x,Z=z}\right)P_{X|Z=z}\left(dx\right)-W^{\ast }\left(P_{R|Z=z}\right)\right)P_Z\left(dz\right).\hfill \end{array}$$

Applying (31) with $W^{\ast }\left(P_{R|X=x}\right)$ replaced with $W^{\ast }\left(P_{R|X=x,Z=z}\right)$ and $W^{\ast }$ replaced with $W^{\ast }\left(P_{R|Z=z}\right)$ with z fixed, we can bound the expression in parentheses as

$$\begin{array}{c}\int W^{\ast }\left(P_{R|X=x,Z=z}\right)P_{X|Z=z}\left(dx\right)-W^{\ast }\left(P_{R|Z=z}\right)\\ \hfill \le \int D(P_{R|X=x,Z=z}\parallel P_{R|Z=z})P_{X|Z=z}\left(dx\right),\end{array}$$

and therefore

$$\begin{array}{c}\hfill W^{\ast }\left(X\right)-W^{\ast }\left(Z\right)\\ \hfill & \le \int \int D(P_{R|X=x,Z=z}\parallel P_{R|Z=z})P_{X|Z=z}\left(dx\right)P_Z\left(dz\right)\hfill \\ \hfill & =I(R;X|Z),\hfill \end{array}$$

(32)

where (32) follows from the alternative expression (12) of the conditional mutual information.

As in the proof Theorem 3, the conditional independence of R and $Z=T\left(X\right)$ given X implies

$$I(R;X|Z)=I(R;X)-I(R;T\left(X\right)),$$

which completes the proof. □

4.4. Information Bottleneck

Let X and Y be random variables as in Section 2. When $Y\to X\to Z$, the joint distribution $P_{YXZ}$ of the triple $(Y,X,Z)$ is determined (for fixed $P_{YX}$) by the conditional distribution (transition kernel) $P_{Z|X}$ as $P_{YXZ}=P_{YX}{P}_{Z|X}$. The information bottleneck (IB) framework can be formulated as the study of the constrained optimization problem

$$\begin{array}{cc}\mathrm{maximize}& I(Y;Z)\hfill \\ & \hfill \\ \mathrm{subject}\mathrm{to}& I(X;Z)\le \alpha \hfill \end{array}$$

(33)

for a given $\alpha >0$, where the maximization is over all transition kernels $P_{Z|X}$.

Originally proposed by Tishby et al. [35], the solution to the IB problem is a transition kernel $P_{Z|X}$, interpreted as a stochastic transformation, that “encodes” X into a “compressed” representation Z that preserves relevant information about Y through maximizing $I(Y;Z)$, while compressing X by requiring that $I(X;Z)\le \alpha$. The intuition behind this framework is that by maximizing $I(Y;Z)$, the representation Z will retain the predictive power of X with respect to Y, while the requirement $I(X;Z)\le \alpha$ makes the representation Z concise.

Note that, in case X is discrete and has finite entropy $H\left(X\right)$, setting $\alpha =H\left(X\right)$, or setting formally $\alpha =\infty$ in the general case, the constraint $I(X;Z)\le \alpha$ becomes vacuous and (assuming the alphabet of Z is sufficiently large) the resulting Z will achieve the upper bound $I(Y;Z)=I(Y;X)$, so that $I(Y;X|Z)=I(Y;X)-I(Y;Z)=0$, i.e., $Y\to Z\to X$. Thus, the solution to (33) can be considered as a stochastically relaxed version of a minimal sufficient statistic for X in predicting Y (see Goldfeld and Polyanskiy ([36], Section II.C) for more on this interpretation). Recent tutorials on the IB problem include Asoodeh and Calmon [37] and Zaidi et al. [38].

Theorem 3 and its corollaries can be used to motivate the IB principle from an estimation-theoretic viewpoint. Let

$$I\left(\alpha \right)=\underset{P_{Z|X}:I(X;Z)\le \alpha }{sup}I(Y;Z)$$

be the value function for (33) and $Z_{\alpha }$ a resulting optimal Z (assuming such a maximizer exists). From the remark after Theorem 3, we know that the bounds given in the theorem and in its corollaries remain valid if we replace $T\left(X\right)$ with a random variable Z, such that $Y\to X\to Z$. Then, for example, Corollary 1 implies that

$$L_{\ell }^{\ast }\left(Y\right|Z_{\alpha })-L_{\ell }^{\ast }\left(Y\right|X)\le c\sqrt{I(Y;X)-I\left(\alpha \right)}$$

for all ℓ such that ${\parallel \ell \parallel }_{\infty }\le \sqrt{2}c$.

Thus, the IB paradigm minimizes, under the complexity constraint $I(X;Z)$≤$\alpha$, an upper bound on the difference $L_{\ell }^{\ast }\left(Y\right|Z)-L_{\ell }^{\ast }\left(Y\right|X)$ that universally holds for all loss functions ℓ with ${\parallel \ell \parallel }_{\infty }\le \sqrt{2}c$. The resulting $Z_{\alpha }$ will then have guaranteed performance in predicting Y with respect to all sufficiently bounded loss functions. This gives a novel operational interpretation of the IB framework that seems to have been overlooked in the literature.

4.5. Deep Learning

The IB paradigm can also serve as a learning objective in deep neural networks (DNNs). Here the Lagrangian relaxation of (33) is considered. In particular, letting X denote the input and $Z^{\theta }$ the output of the last hidden layer of the DNN, where $\theta \in \Theta \subset {\mathbb{R}}^K$ is the collection of network parameters (weights), the objective is to maximize

$$I(Y;Z^{\theta })-\beta I(X;Z^{\theta })$$

(34)

over $\theta \in \Theta$ for a given $\beta >0$. The parameter $\beta$ controls the trade-off between how informative $Z^{\theta }$ is about Y, measured by $I(Y;Z^{\theta })$, and how much $Z^{\theta }$ is “compressed,” measured by $I(X;Z^{\theta })$. Clearly, larger values of $\beta$ correspond to smaller values of $I(X;Z^{\theta })$ and thus to more compression. Here, $Z^{\theta }$ is either a deterministic function of X in the form of $Z^{\theta }=T^{\theta }\left(X\right)$, where $T^{\theta }:{\mathbb{R}}^d\to {\mathbb{R}}^{d^{\prime }}$ represents the deterministic DNN, or it is produced by a stochastic kernel $P_{Z|X}^{\theta }$, parameterized by the network parameters $\theta \in \Theta$. The latter is achieved by injecting independent noise into the network’s intermediate layers.

In addition to the motivation explained in the previous section, the IB framework for DNNs can be thought as a regularization method that results in improved generalization capabilities for a network trained on data using stochastic gradient-based methods, see, e.g., Tishby and Zaslavsky [39], Shwartz-Ziv and Tishby [40], Alemi et al. [41], as well as many other references in the excellent survey article Goldfeld and Polyanskiy [36], and the special issue [42] on information bottleneck and deep learning.

As in the previous section, our Theorem 1 and corollaries can serve as a (partial) justification for setting (34) as a learning objective. Assume that after training with a given $\beta >0$, the obtained $Z^{\theta \left(\beta \right)}$ has (true) mutual information $I(Y;Z^{\theta \left(\beta \right)})$ with Y (typically, this will not be the optimal solution, since maximizing (34) is not feasible and in practice only a proxy lower bound is optimized during training, see, e.g., Alemi et al. [41]). Then, by Corollary 1 the obtained network has a guaranteed predictive performance

$$L_{\ell }^{\ast }\left(Y\right|Z^{\theta \left(\beta \right)})\le L_{\ell }^{\ast }\left(Y\right|X)+cϵ$$

for all loss functions ℓ with ${\parallel \ell \parallel }_{\infty }\le \sqrt{2}c$, where

$$ϵ=\sqrt{I(Y;X)-I(Y;Z^{\theta \left(\beta \right)})}.$$

5. Proof of Theorem 2

Proof Theorem 2.

(a)

The bounds given in the proof of Theorem 1 in [2] imply

$$\begin{array}{cc}\hfill L_n& \le J_{n,1}+J_{n,2}+J_{n,3}+J_{n,4}+J_{n,5},\hfill \end{array}$$

where

$$\begin{array}{ll}{J}_{n,1}\hfill & ={\displaystyle \sum _{A\in {\mathcal{P}}_n,B\in Q_n,C\in {\mathcal{R}}_n}}\left|P_{XYZ}^n(A,B,C)-P_{XYZ}(A,B,C)\right|,\hfill \\ J_{n,2}\hfill & ={\displaystyle \sum _{B\in Q_n,C\in {\mathcal{R}}_n}}\left|P_{YZ}(B,C)-P_{YZ}^n(B,C)\right|,\hfill \\ J_{n,3}\hfill & ={\displaystyle \sum _{A\in {\mathcal{P}}_n,C\in {\mathcal{R}}_n}}\left|P_{XZ}(A,C)-P_{XZ}^n(A,C)\right|,\hfill \\ J_{n,4}\hfill & ={\displaystyle \sum _{C\in {\mathcal{R}}_n}}\left|P_Z^n\left(C\right)-P_Z\left(C\right)\right|,\hfill \end{array}$$

and

$$\begin{array}{cc}\hfill J_{n,5}& ={\displaystyle \sum _{A\in {\mathcal{P}}_n,B\in Q_n,C\in {\mathcal{R}}_n}}\left|P_{XYZ}(A,B,C)-\frac{P_{XZ}(A,C)P_{YZ}(B,C)}{P_Z\left(C\right)}\right|.\hfill \end{array}$$

Using large deviation inequalities from Beirlant et al. [43] and in Biau and Györfi [18], Györfi and Walk [2] proved that for all ${\epsilon }_i>0$, $i=1,\cdots ,4$ and $\delta >0$,

$$\begin{array}{l}\hfill \mathbb{P}(L_n>{\epsilon }_1+{\epsilon }_2+{\epsilon }_3+{\epsilon }_4+\delta )\\ \hfill & \le \mathbb{P}(J_{n,1}>{\epsilon }_1)+\mathbb{P}(J_{n,2}>{\epsilon }_2)+\mathbb{P}(J_{n,3}>{\epsilon }_3)+\mathbb{P}(J_{n,4}>{\epsilon }_4)+{\mathbb{I}}_{J_{n,5}>\delta }\hfill \\ \hfill & \le 2^{m_n·m_n^{\prime }·m_n^{\prime \prime }}{e}^{-n{\epsilon }_1^2/2}+2^{m_n^{\prime }·m_n^{\prime \prime }}{e}^{-n{\epsilon }_2^2/2}+2^{m_n·m_n^{\prime \prime }}{e}^{-n{\epsilon }_3^2/2}+2^{m_n^{\prime \prime }}{e}^{-n{\epsilon }_4^2/2}\hfill \\ \hfill & +{\mathbb{I}}_{J_{n,5}>\delta }.\hfill \end{array}$$

(35)

We note that the bounds on the probabilities $\mathbb{P}(J_{n,i}>{\epsilon }_i)$ for $i=1,\dots ,4$ were proven in [2] without either assuming the null hypothesis ${\mathcal{H}}_0$ or using the condition that the partitions are nested. Under the null hypothesis, Györfi and Walk [2] claimed that

$$\begin{array}{c}\hfill J_{n,5}=0.\end{array}$$

As Neykov et al. [12] observed, this was incorrect. In order to resolve the gap, we show that under ${lim}_n{h}_n=0$ and condition (5) and under the null hypothesis, the last term in (35) is $o\left(1\right)$, i.e.,

$$\begin{array}{c}\hfill {\mathbb{I}}_{{\sum }_{A\in {\mathcal{P}}_n,B\in Q_n,C\in {\mathcal{R}}_n}\left|P_{XYZ}(A,B,C)-\frac{P_{XZ}(A,C)P_{YZ}(B,C)}{P_Z\left(C\right)}\right|>\delta }=0\end{array}$$

if n is large enough. The null hypothesis implies that

$$\begin{array}{l}\hfill P_{XYZ}(A,B,C)-\frac{P_{XZ}(A,C)P_{YZ}(B,C)}{P_Z\left(C\right)}\\ \hfill & =\mathbb{P}(X\in A,Y\in B,Z\in C)-\frac{P_{XZ}(A,C)\mathbb{P}(Y\in B,Z\in C)}{P_Z\left(C\right)}\hfill \\ \hfill & =\mathbb{E}\left[\mathbb{P}(X\in A,Y\in B\mid Z){\mathbb{I}}_{Z\in C}\right]-\mathbb{E}\left[\mathbb{P}(Y\in B\mid Z){\mathbb{I}}_{Z\in C}\right]\frac{P_{XZ}(A,C)}{P_Z\left(C\right)}\hfill \\ \hfill & =\mathbb{E}\left[\mathbb{P}(X\in A\mid Z)\mathbb{P}(Y\in B\mid Z){\mathbb{I}}_{Z\in C}\right]-\mathbb{E}\left[\mathbb{P}(Y\in B\mid Z){\mathbb{I}}_{Z\in C}\right]\frac{P_{XZ}(A,C)}{P_Z\left(C\right)}.\hfill \end{array}$$

Thus,

$$\begin{array}{c}\hfill {\displaystyle \sum _{A\in {\mathcal{P}}_n,B\in Q_n,C\in {\mathcal{R}}_n}}\left|P_{XYZ}(A,B,C)-\frac{P_{XZ}(A,C)P_{YZ}(B,C)}{P_Z\left(C\right)}\right|\\ \hfill & \le \sum _{A\in {\mathcal{P}}_n,B\in Q_n,C\in {\mathcal{R}}_n}\mathbb{E}\left[\mathbb{P}(Y\in B\mid Z)\left|\mathbb{P}(X\in A\mid Z){\mathbb{I}}_{Z\in C}-{\mathbb{I}}_{Z\in C}\frac{P_{XZ}(A,C)}{P_Z\left(C\right)}\right|\right]\hfill \\ \hfill & =\sum _{A\in {\mathcal{P}}_n,C\in {\mathcal{R}}_n}\mathbb{E}\left[\left|\mathbb{P}(X\in A\mid Z){\mathbb{I}}_{Z\in C}-{\mathbb{I}}_{Z\in C}\frac{P_{XZ}(A,C)}{P_Z\left(C\right)}\right|\right].\hfill \end{array}$$

Let $p(·\mid z)$ and $C_n\left(z\right)$ be as in Condition 1. Then,

$$\begin{array}{l}\hfill \sum _{A\in {\mathcal{P}}_n,C\in {\mathcal{R}}_n}\mathbb{E}\left[\left|\mathbb{P}(X\in A\mid Z){\mathbb{I}}_{Z\in C}-{\mathbb{I}}_{Z\in C}\frac{P_{XZ}(A,C)}{P_Z\left(C\right)}\right|\right]\\ \hfill & =\sum _{A\in {\mathcal{P}}_n,C\in {\mathcal{R}}_n}{\int }_C\left|{\int }_{A}p(x\mid z)P_X\left(dx\right)-\frac{{\int }_A\left[{\int }_{C}p(x\mid z^{\prime })P_Z\left(d{z}^{\prime }\right)\right]P_X\left(dx\right)}{P_Z\left(C\right)}\right|P_Z\left(dz\right)\hfill \\ \hfill & \le \int \int \left|p(x\mid z)-\frac{{\int }_{C_n\left(z\right)}p(x\mid z^{\prime })P_Z\left(d{z}^{\prime }\right)}{P_Z\left(C_n\left(z\right)\right)}\right|P_Z\left(dz\right)P_X\left(dx\right)\hfill \\ \hfill & \le C^{\ast }{h}_n,\hfill \end{array}$$

(36)

where in the last step we use condition (5). The inequalities (35) and (36) imply that

$$\begin{array}{l}\hfill \mathbb{P}\left(L_n>c_1\left(\sqrt{\frac{m_n{m}_n^{\prime }{m}_n^{\prime \prime }}{n}}+\sqrt{\frac{m_n^{\prime }{m}_n^{\prime \prime }}{n}}+\sqrt{\frac{m_n{m}_n^{\prime \prime }}{n}}+\sqrt{\frac{m_n^{\prime \prime }}{n}}\right)+(logn)h_n\right)\\ \hfill & \le 4e^{-(c_1^2/2-log2)m_n^{\prime \prime }}+{\mathbb{I}}_{C^{\ast }{h}_n>(logn)h_n}\hfill \\ \hfill & \le 4e^{-(c_1^2/2-log2)m_n^{\prime \prime }},\hfill \end{array}$$

if $n\ge e^{C^{\ast }}$. Since $m_n^{\prime \prime }$ is proportional to $1/h_n^{d^{\prime }}$, condition (8) on $h_n$ implies ${\sum }_{n=1}^{\infty }\mathbb{P}(L_n\ge t_n)<\infty$, and thus using the Borel–Cantelli lemma, after a random sample size, the test has no error with a probability of one.

(b)

This proof is a refinement of the proof of Corollary 1 in [2], in which we avoid the condition used there that the sequences of partitions ${\mathcal{P}}_n$ and $Q_n$ are nested. According to the proof of Part (a) (see the remark after (35)), we obtain that

$$\underset{n\to \infty }{lim\; inf}{L}_n\ge \underset{n\to \infty }{lim\; inf}(L_n-J_{n,5})+\underset{n\to \infty }{lim\; inf}{J}_{n,5}=\underset{n\to \infty }{lim\; inf}{J}_{n,5}\mathrm{a}.\mathrm{s}.$$

To simplify the notation, let $P_{XY|z}=P_{XY|Z=z}$, $P_{X|z}=P_{X|Z=z}$, and $P_{Y|z}=P_{Y|Z=z}$. Let $L^{\ast }$ be the expected total variation distance between $P_{XY|z}$ and $P_{X|z}{P}_{Y|z}$:

$$L^{\ast }=\int \underset{F}{sup}|P_{XY|z}\left(F\right)-P_{X|z}{P}_{Y|z}\left(F\right)|P_Z\left(dz\right),$$

where the supremum is taken over all Borel subsets F of ${\mathbb{R}}^d\times {\mathbb{R}}^{d^{\prime }}$. It suffices to prove that using the condition ${lim}_n{h}_n=0$,

$$\underset{n\to \infty }{lim\; inf}{J}_{n,5}\ge 2L^{\ast }>0.$$

One has that

$$\begin{array}{r}\hfill \sum _{A\in {\mathcal{P}}_n,B\in Q_n,C\in {\mathcal{R}}_n}\left|P_{XYZ}(A,B,C)-\frac{P_{XZ}(A,C)P_{YZ}(B,C)}{P_Z\left(C\right)}\right|\\ \hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\ge \int \sum _{A\in {\mathcal{P}}_n,B\in Q_n}\left|P_{XY|z}(A,B)-P_{X|z}\left(A\right)P_{Y|z}\left(B\right)\right|P_Z\left(dz\right)-W_n,\hfill \end{array}$$

where

$$\begin{array}{cc}\hfill W_n& \le \sum _{A\in {\mathcal{P}}_n,B\in Q_n}\int \left|\frac{{\int }_{C_n\left(z\right)}{P}_{XY|z^{\prime }}(A,B)P_Z\left(z^{\prime }\right)}{P_Z\left(C_n\left(z\right)\right)}-P_{XY|z}(A,B)\right|P_Z\left(dz\right)\hfill \end{array}$$

(37)

$$\begin{array}{cc}\hfill & +\sum _{A\in {\mathcal{P}}_n}\int \left|\frac{{\int }_{C_n\left(z\right)}{P}_{X|z^{\prime }}\left(A\right)P_Z\left(d{z}^{\prime }\right)}{P_Z\left(C_n\left(z\right)\right)}-P_{X|z}\left(A\right)\right|P_Z\left(dz\right)\hfill \end{array}$$

(38)

$$\begin{array}{cc}\hfill & +\sum _{B\in Q_n}\int \left|\frac{{\int }_{C_n\left(z\right)}{P}_{Y|z^{\prime }}\left(B\right)P_Z\left(d{z}^{\prime }\right)}{P_Z\left(C_n\left(z\right)\right)}-P_{Y|z}\left(B\right)\right|P_Z\left(dz\right)\hfill \end{array}$$

(39)

In [2], it was shown that the condition ${lim}_n{h}_n=0$ implies ${lim}_n{W}_n=0$ if the sequence of partitions ${\{{\mathcal{P}}_n,Q_n\}}_{n\ge 1}$ is nested. In order to avoid this nestedness condition, introduce the density $p(x,y|z)$ of the conditional distribution $P_{XY|z}$ with respect to the distribution $P_{XY}$ of $(X,Y)$ as a dominating measure, and similarly let $p_n(x,y|z)$ be the density of the conditional distribution ${\int }_{C_n\left(z\right)}{P}_{XY|z^{\prime }}(·,·)P_Z\left(d{z}^{\prime }\right)/P_Z\left(C_n\left(z\right)\right)$ with respect to $P_{XY}$, i.e., $p_n(x,y|z)={\int }_{C_n\left(z\right)}p(x,y|z^{\prime })P_Z\left(d{z}^{\prime }\right)/P_Z\left(C_n\left(z\right)\right)$. Then,

$$\begin{array}{l}\hfill \sum _{A\in {\mathcal{P}}_n,B\in Q_n}\left|\frac{{\int }_{C_n\left(z\right)}{P}_{XY|z^{\prime }}(A,B)P_Z\left(d{z}^{\prime }\right)}{P_Z\left(C_n\left(z\right)\right)}-P_{XY|z}(A,B)\right|\\ \hfill & \le \int \int |p_n(x,y|z)-p(x,y|z)|P_{XY}(dx,dy),\hfill \end{array}$$

and therefore the term on the right-hand side of (37) will converge to zero, as long as

$$\begin{array}{c}\hfill \int \int \int |p_n(x,y|z)-p(x,y|z)|P_{XY}(dx,dy)P_Z\left(dz\right)\to 0,\end{array}$$

which follows from ${lim}_n{h}_n=0$ using the standard technique of the bias of partitioning regression estimate for the regression function $p(·,·|z)$; see Theorem 4.2 in [44]. The terms in (38) and (39) can be dealt with analogously. Thus,

$$\begin{array}{cc}\hfill \underset{n\to \infty }{lim\; inf}{J}_{n,5}& \ge \underset{n\to \infty }{lim\; inf}\int \sum _{A\in {\mathcal{P}}_n,B\in Q_n}\left|P_{XY|z}(A,B)-P_{X|z}\left(A\right)P_{Y|z}\left(B\right)\right|P_Z\left(dz\right).\hfill \end{array}$$

For fixed z, ${lim}_n{h}_n=0$ implies

$$\begin{array}{l}\hfill {\displaystyle \underset{n\to \infty }{lim}}{\displaystyle \sum _{A\in {\mathcal{P}}_n,B\in Q_n}}\left|P_{XY|z}(A,B)-P_{X|z}\left(A\right)P_{Y|z}\left(B\right)\right|\\ \hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}=2\underset{F}{sup}|P_{XY|z}\left(F\right)-P_{X|z}{P}_{Y|z}\left(F\right)|,\hfill \end{array}$$

see Abou-Jaoude [45] and Csiszár [46]. Therefore, the dominated convergence theorem yields

$$\begin{array}{l}\hfill {\displaystyle \underset{n\to \infty }{lim}}\int {\displaystyle \sum _{A\in {\mathcal{P}}_n,B\in Q_n}}\left|P_{XY|z}(A,B)-P_{X|z}\left(A\right)P_{Y|z}\left(B\right)\right|P_Z\left(dz\right)\\ \hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}=2\int \underset{F}{sup}|P_{XY|z}\left(F\right)-P_{X|z}{P}_{Y|z}\left(F\right)|P_Z\left(dz\right)\hfill \\ \hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}=2L^{\ast }.\hfill \end{array}$$

□

Note that in the proof of Part (b) the condition (5) is not used, at all.

6. Concluding Remarks

We studied the excess minimum risk in statistical inference and under mild conditions gave a strongly consistent procedure for testing from data if a given transformation of an observed feature vector results in zero excess minimum risk for all loss functions. It is an open research problem whether a strong universal test exists, i.e., a test that is strongly consistent without any conditions on the transformation and on the underlying distribution. We also developed information-theoretic upper bounds on the excess risk that uniformly hold over fairly general classes of loss functions. The bounds were not stated in the most general form possible, in that the observed quantities were restricted to taking values in Euclidean spaces and we did not allow transformations that were random functions of the observation, both of which restrictions could be relaxed.The bounds could be sharpened, e.g., in specific cases, but in their present form are already useful. For example, they give an additional theoretical motivation for applying the information bottleneck approach in deep learning.