Information-Theoretic Generalization Bounds for Batch Reinforcement Learning

Abstract

We analyze the generalization properties of batch reinforcement learning (batch RL) with value function approximation from an information-theoretic perspective. We derive generalization bounds for batch RL using (conditional) mutual information. In addition, we demonstrate how to establish a connection between certain structural assumptions on the value function space and conditional mutual information. As a by-product, we derive a high-probability generalization bound via conditional mutual information, which was left open and may be of independent interest.

1. Introduction

Generalization is a fundamental concept in statistical machine learning. It measures how well a learning system performs on unseen data after being trained on a finite dataset. Effective generalization ensures that the learning approach captures the essential patterns in the data. Generalization in supervised learning has been studied for several decades. However, in reinforcement learning (RL), agnostic learning is generally infeasible and realizability is not a sufficient condition for efficient learning. Consequently, the study of generalization in RL poses more challenges.

In this work, we focus on batch reinforcement learning (batch RL), a branch of reinforcement learning where the agent learns a policy from a fixed dataset of previously collected experiences. This setting is favorable when online interaction is expensive, dangerous, or impractical. Batch RL, despite being a special case of supervised learning, still presents distinct challenges due to the complex temporal structures inherent in the data.

Originating from the work of [1,2], an information-theoretic framework has been developed to bound the generalization error of learning algorithms using the mutual information between the input dataset and the output hypothesis. This methodology formalizes the intuition that overfitted learning algorithms are less likely to generalize effectively. Unlike traditional approaches such as VC-dimension and Rademacher complexity, this information-theoretic framework offers the significant advantage of capturing all dependencies on the data distribution, hypothesis space, and learning algorithm. Given that reinforcement learning is a learning paradigm in which all the aforementioned aspects differ significantly from those in supervised learning, we believe this novel approach will provide us with more profound insights.

2. Preliminaries

2.1. Batch Reinforcement Learning with Function Approximation

An episodic Markov decision process (MDP) is defined by $\mathcal{M}(\mathcal{S},\mathcal{A},\mathcal{P},r,H)$. We use $\Delta (\mathcal{X})$ to denote the set of the probability distribution over the set $\mathcal{X}$. $\mathcal{M}(\mathcal{S},\mathcal{A},\mathcal{P},r,H)$ is specified by a finite state space $\mathcal{S}$, a finite action space $\mathcal{A}$, transition functions ${\mathcal{P}}_h:\mathcal{S}\times \mathcal{A}\to \Delta (\mathcal{S})$ at step $h\in \left[H\right]$, reward function $r_h:\mathcal{S}\times \mathcal{A}\to \mathbb{R}$ at step h, and H is the number of steps in each episode. We assume the reward is bounded, i.e., $r_h(s,a)\in [0,1]$ (For rewards in $[R_{min},R_{max}]$ simply rescales these bounds.), $\forall (s,a,h)$. See Figure 1 for a graphical illustration.

Let $\pi ={\{{\pi }_h:\mathcal{S}\to \Delta (\mathcal{A})\}}_{h\in \left[H\right]}$, where ${\pi }_h(·\mid s)$ is the action distribution for policy $\pi$ at state s and step h. Given a policy $\pi$, the value function $V_h^{\pi }:\mathcal{S}\to \mathbb{R}$ at step h is defined as

$$V_h^{\pi }(s):={\mathbb{E}}_{\pi }\left[\sum _{h^{\prime }=h}^H{r}_{h^{\prime }}(s_{h^{\prime }},a_{h^{\prime }})|s_h=s\right].$$

The action-value function $Q_h^{\pi }:\mathcal{S}\times \mathcal{A}\to \mathbb{R}$ at step h is defined as

$$Q_h^{\pi }(s,a):={\mathbb{E}}_{\pi }\left[\sum _{h^{\prime }=h}^H{r}_{h^{\prime }}(s_{h^{\prime }},a_{h^{\prime }})|s_h=s,a_h=a\right].$$

The Bellman operators ${\mathcal{T}}_h^{\pi }$ and ${\mathcal{T}}_h^{*}$ project functions forward by one step through the following dynamics:

$$({\mathcal{T}}_h^{\pi })(s,a)=r_h(s,a)+{\mathbb{E}}_{s^{\prime }\sim {\mathcal{P}}_h(·|s,a)}\left[{\mathbb{E}}_{a^{\prime }\sim \pi (·|s^{\prime })}\left[Q(s^{\prime },a^{\prime })\right]\right],$$

$$({\mathcal{T}}_h^{*})(s,a)=r_h(s,a)+{\mathbb{E}}_{s^{\prime }\sim {\mathcal{P}}_h(·|s,a)}\left[\underset{a^{\prime }}{max}Q(s^{\prime },a^{\prime })\right].$$

Now, we denote the dataset $Z=\left\{(s,a,r,s^{\prime },h)\right\}$, where $(s,a)\sim {\mu }_h$, $r\sim r_h(s,a)$, and $s^{\prime }\sim {\mathcal{P}}_h(·|s,a)$ for a fixed h. We also denote $\mathcal{D}={\mathcal{D}}_1\times \cdots \times {\mathcal{D}}_H$, where $(s,a,r,s^{\prime },h)\sim {\mathcal{D}}_h$. We consider batch RL with value function approximation. The learner is given a function class $\mathcal{F}={\mathcal{F}}_1\times \cdots \times {\mathcal{F}}_H$ to approximate the optimal Q-value function. Denote $f=(f_1,\cdots ,f_H)\in \mathcal{F}$. As no reward is collected in the $(H+1)\mathrm{th}$ step, we set $f_{H+1}=0$. For each $f\in \mathcal{F}$, define ${\pi }_f={\left\{{\pi }_{f_h}\right\}}_{h=1}^H$, where ${\pi }_{f_h}(a|s)=\mathbf{1}\left[a=\underset{a^{\prime }}{argmax}{f}_h(s,a^{\prime })\right]$. Next, we introduce the Bellman error and its empirical version.

Definition 1

(Bellman error). Under data distribution μ, we define the Bellman error of function $f=(f_1,\cdots ,f_H)$ as

$$\mathcal{E}(f):={\displaystyle \frac{1}{H}}\sum _{h=1}^H{\parallel f_h-{\mathcal{T}}_h^{★}{f}_{h+1}\parallel }_{{\mu }_h}^2.$$

(1)

Definition 2

(Mean squared empirical Bellman error (MSBE)). Given a dataset $Z\sim \mathcal{D}$, we define the Mean squared empirical Bellman error (MSBE) of function $f=(f_1,\cdots ,f_H)$ as

$$\begin{array}{c}\hfill L(f,Z)={\displaystyle \frac{1}{H}}\sum _{h=1}^H{\displaystyle \frac{1}{n}}\sum _{(s,a,r,s^{\prime },h)\in Z_h}{(f_h(s,a)-r-V_{f_{h+1}}(s^{\prime }))}^2\end{array}$$

where $V_{f_{h+1}}(s):={max}_{a\in \mathcal{A}}{f}_{h+1}(s,a)$.

For convenience, we denote $\ell (f_h,Z_h)={\displaystyle \frac{1}{n}}{\sum }_{(s,a,r,s^{\prime },h)\in Z_h}{(f_h(s,a)-r-V_{f_{h+1}}(s^{\prime }))}^2$.

Bellman error is used in RL as a surrogate loss function to minimize the difference between the estimated value function and the true value function under a policy. The Bellman error serves as a proxy for the optimality gap, which is the difference between the current value function and the optimal value function. Under the concentrability assumption, minimizing the Bellman error is able to reduce the optimality gap.

Lemma 1

(Bellman error to value suboptimality [3]). If there exists a constant C, such that for any policy π

$$\underset{(s,a,h)\in \mathcal{S}\times \mathcal{A}\times \left[H\right]}{sup}{\displaystyle \frac{\mathrm{d}{P}_h^{\pi }}{\mathrm{d}{\mu }_h}}(s,a)\le C$$

then for any $f\in \mathcal{F}$, we have

$$V_1^{*}(s_1)-V_1^{{\pi }_f}(s_1)\le 2H\sqrt{C·\mathcal{E}(f)}.$$

We note that $L(f,Z)$ is a biased estimate of $\mathcal{E}(f)$. A common solution is to use the double sampling method, where for each state and action in the sample, at least two next states are generated [3,4,5], and define the unbiased MSBE as:

$$L_{\mathrm{DS}}(f,\tilde{Z})={\displaystyle \frac{1}{nH}}\sum _{(s,a,r,s^{\prime },{\tilde{s}}^{\prime },h)\in \tilde{Z}}\left[{\left(f_h(s,a)-r-V_{f_{h+1}}(s^{\prime })\right)}^2-{\displaystyle \frac{1}{2}}{\left(V_{f_{h+1}}(s^{\prime })-V_{f_{h+1}}({\tilde{s}}^{\prime })\right)}^2\right].$$

Note that $L(f,Z)\in [0,4H^2]$, $L_{\mathrm{DS}}(f,\tilde{Z})\in [-2H^2,4H^2]$, and double sampling does not increase the sample size, except that it requires an additional generated ${\tilde{s}}^{\prime }\sim {\mathcal{P}}_h(·|s,a)$. Therefore, the results presented in this paper can be easily extended to the double sampling setting.

2.2. Generalization Bounds

Definition 3

(Expected generalization bounds). Given a dataset $Z\sim \mathcal{D}$ and an algorithm $\mathcal{A}$, let $L(\mathcal{A}(Z),Z)$ denote the training loss and let $L(\mathcal{A}(Z),\mathcal{D})$ denote the true loss. The expected generalization error is defined as

$$\left|{\mathbb{E}}_{Z\sim \mathcal{D}}[L(\mathcal{A}(Z),Z)-L(\mathcal{A}(Z),\mathcal{D})]\right|.$$

Definition 4

(High-probability generalization bounds). Given a dataset $Z\sim \mathcal{D}$, and an algorithm $\mathcal{A}$, let $L(\mathcal{A}(Z),Z)$ denote the training loss and let $L(\mathcal{A}(Z),\mathcal{D})$ denote the true loss. Given a failure probability δ and an error tolerance η, the high-probability generalization error is defined as

$$\mathbb{P}(|L(\mathcal{A}(Z),Z)-L(\mathcal{A}(Z),\mathcal{D})|\ge \eta )\le \delta .$$

2.3. Mutual Information

First, we define the KL-divergence of two distributions.

Definition 5

(KL-Divergence [6]). Let $\mathcal{P},\mathcal{Q}$ be two distributions over the space Ω and suppose $\mathcal{P}$ is absolutely continuous with respect to $\mathcal{Q}$. The Kullback–Leibler (KL) divergence from $\mathcal{Q}$ to $\mathcal{P}$ is

$$\mathrm{D}(\mathcal{P}\parallel \mathcal{Q})={\mathbb{E}}_{X\sim {\mathcal{P}}_X}\left[log{\displaystyle \frac{{\mathcal{P}}_X}{{\mathcal{Q}}_X}}\right],$$

where ${\mathcal{P}}_X$ and ${\mathcal{Q}}_X$ denote the probability mass/density functions of $\mathcal{P}$ and $\mathcal{Q}$ on X, respectively.

Based on KL-divergence, we can define mutual information and conditional mutual information as follows.

Definition 6

([6]). Let X, Y, and Z be arbitrary random variables, and let $D_{\mathrm{KL}}$ denote the Kullback–Leibler (KL) divergence. The mutual information between X and Y is defined as:

$$I(X;Y):=D_{\mathrm{KL}}({\mathcal{P}}_{X,Y}\parallel {\mathcal{P}}_X\otimes {\mathcal{P}}_Y).$$

The conditional mutual information is defined as:

$$I(X;Y|Z):={\mathbb{E}}_Z\left[D_{\mathrm{KL}}({\mathcal{P}}_{X,Y|Z}\parallel {\mathcal{P}}_{X|Z}\otimes {\mathcal{P}}_{Y|Z})\right].$$

Next, we introduce Rényi’s $\alpha$-Divergence, which is a generalization of KL-divergence. Rényi’s $\alpha$-Divergence has found many applications, such as hypothesis testing, differential privacy, several statistical inference, and coding problems [7,8,9,10].

Definition 7

(Rényi’s $\alpha$-Divergence [11]). Let $(\Omega ,\mathcal{F},\mathcal{P})$, $(\Omega ,\mathcal{F},\mathcal{Q})$ be two probability spaces. Let $\alpha >0$ be a positive real different from 1. Consider a measure μ, such that $\mathcal{P}\ll \mu$ and $\mathcal{Q}\ll \mu$ (such a measure always exists, e.g., $\mu =(\mathcal{P}+\mathcal{Q})/2$) and denote with $p,q$ the densities of $\mathcal{P},\mathcal{Q}$ with respect to μ. The α–divergence of $\mathcal{P}$ from $\mathcal{Q}$ is defined as follows:

$$D_{\alpha }(\mathcal{P}\parallel \mathcal{Q})={\displaystyle \frac{1}{\alpha -1}}log\int p^{\alpha }{q}^{1-\alpha }d\mu .$$

Note that the above definition is independent of the chosen measure $\mu$. With the definition of Rényi’s $\alpha$-divergence, we are ready to state the definitions of $\alpha$-mutual information and $\alpha$-conditional mutual information.

Definition 8

($\alpha$-mutual information [7]). Let $X,Y$ be two random variables jointly distributed according to ${\mathcal{P}}_{XY}$. Let ${\mathcal{Q}}_Y$ be any probability measure over $\mathcal{Y}$. For $\alpha >0$, the α-mutual information between X and Y is defined as follows:

$$I_{\alpha }(X;Y)=\underset{{\mathcal{Q}}_Y}{min}{D}_{\alpha }({\mathcal{P}}_{XY}\parallel {\mathcal{P}}_X\otimes {\mathcal{Q}}_Y).$$

Definition 9

(Conditional $\alpha$-mutual information). Let $X,Y,Z$ be three random variables jointly distributed according to ${\mathcal{P}}_{XYZ}$. Let ${\mathcal{Q}}_{Y|Z}$ be any probability measure over $\mathcal{Y}|\mathcal{Z}$. For $\alpha >0$, a conditional α-mutual information of order α between X and Y given Z is defined as follows:

$$I_{\alpha }^{Y|Z}(X;Y|Z)=\underset{{\mathcal{Q}}_{Y|Z}}{min}{D}_{\alpha }({\mathcal{P}}_{XYZ}\parallel {\mathcal{P}}_{X|Z}\otimes {\mathcal{Q}}_{Y|Z}\otimes {\mathcal{P}}_Z).$$

3. Generalization Bounds via Mutual Information

Mutual information bounds provide a direct link between the generalization error and the amount of information shared between the training data and the learned hypothesis. This offers a clear information-theoretic understanding of how overfitting can be controlled by reducing the dependency on the training data. Mutual information bounds are applicable to a wide range of learning algorithms and settings, including those with unbounded loss functions and complex hypothesis spaces. Moreover, the use of mutual information can simplify the analysis of generalization compared with traditional methods, particularly in cases where those traditional measures are difficult to compute. See Appendix A for related work.

Theorem 1

([2]). Let $\mathcal{D}$ be a distribution on Z. Let $\mathcal{A}:Z\to \mathcal{W}$ be a randomized algorithm. Let $\ell :\mathcal{W}\times Z\to \mathbb{R}$ be a loss function, which is σ-subgaussian with respect to Z. Let $L:\mathcal{W}\times Z\to \mathbb{R}$ be the empirical risk. Then

$$\left|{\mathbb{E}}_{Z\sim \mathcal{D}}[L(\mathcal{A}(Z),Z)-L(\mathcal{A}(Z),\mathcal{D})]\right|\le \sqrt{{\displaystyle \frac{2{\sigma }^2}{n}}I(\mathcal{A}(Z);Z)}.$$

The above theorem provides a bound on the expected generalization error. High-probability generalization bounds can be obtained using the $\alpha$-mutual information. Note that the $\alpha$-mutual information shares many properties with standard mutual information.

Proposition 1

([7]). For discrete random variables X and Y, the following holds:

(i) 

Data Processing Inequality: given $\alpha >0$, $I_{\alpha }(X,Z)\le min\{I_{\alpha }(X,Y),I_{\alpha }(Y,Z)\}$ if the Markov chain $X-Y-Z$ holds.

(ii) 

$I_{\alpha }(X;Y)is\mathit{\text{non-decreasing}}in\alpha .$

(iii) 

$I_{\alpha }(X,Y)\le min\{log|X|,log|Y\left|\right\}.$

(iv) 

$I_{\alpha }(X,Y)\ge 0$ with equality iff X and Y are independent.

Theorem 2

([11]). Let $\mathcal{D}$ be a distribution on Z. Let $\mathcal{A}:Z\to \mathcal{W}$ be a randomized algorithm. Let $\ell :\mathcal{W}\times Z\to \mathbb{R}$ be a loss function which is σ-subgaussian with respect to Z. Let $L:\mathcal{W}\times Z\to \mathbb{R}$ be the empirical risk. Given $\eta ,\delta \in (0,1)$ and fix $\alpha \ge 1$, if the number of samples n satisfies

$$n\ge {\displaystyle \frac{2{\sigma }^2}{{\eta }^2}}\left(I_{\alpha }(\mathcal{A}(Z),Z)+log2+{\displaystyle \frac{\alpha }{\alpha -1}}log\left({\displaystyle \frac{1}{\delta }}\right)\right).$$

then, we have

$$\mathbb{P}\left(|L(\mathcal{A}(Z),Z)-L(\mathcal{A}(Z),\mathcal{D})|\le \eta \right)\ge 1-\delta .$$

The mutual information bound can be infinite in some cases and thus be vacuous. To address this, the conditional mutual information (CMI) approach was introduced. CMI bounds normalize the information content for each data point, preventing the problem of infinite information content, particularly in continuous data distributions. This makes CMI a more robust and applicable method in scenarios where mutual information would otherwise be unbounded.

Definition 10.

Let $Z\sim {\mathcal{D}}^{2n}$ consist of $2n$ samples drawn independently from $\mathcal{D}$. Let $U\in {\{0,1\}}^n$ be uniformly random and independent from Z and the randomness of $\mathcal{A}$. Define $Z_U\in Z$, such that ${(Z_U)}_i$ is the ${(2i-U_i)}^{\mathrm{th}}$ sample in Z—that is, $Z_U$ is the subset of Z indexed by U. The conditional mutual information of $\mathcal{A}$ with respect to $\mathcal{D}$ is defined as $I(\mathcal{A}(Z_U);U|Z).$

Theorem 3

([12]). Let $\mathcal{D}$ be a distribution on Z. Let $\mathcal{A}:Z\to \mathcal{W}$ be a randomized algorithm. Let $L:\mathcal{W}\times Z\to \mathbb{R}$ be a function, such that $|L(w,z_1)-L(w,z_2)|\le \Delta (z_1,z_2)$ for all $z_1,z_2\in \mathcal{Z}$ and $w\in \mathcal{W}$ given $\Delta :Z^2\to \mathbb{R}$. Let $U\in {\{0,1\}}^n$ be uniformly random. Then

$$\left|{\mathbb{E}}_{Z\sim \mathcal{D}}[L(\mathcal{A}(Z_U),Z_U)-L(\mathcal{A}(Z_U),\mathcal{D})]\right|\le \sqrt{{\displaystyle \frac{2{\mathbb{E}}_{z_1,z_2}[\Delta {(z_1,z_2)}^2]}{n}}I(\mathcal{A}(Z_U);U|Z)}.$$

Another advantage of the CMI bounds is that they can be derived from various concepts such as VC-dimension, compression schemes, stability, and differential privacy, offering a unified framework for generalization analysis. However, because CMI is defined as an expectation, i.e., $I(X;Y|Z):={\mathbb{E}}_Z\left[D_{\mathrm{KL}}({\mathcal{P}}_{X,Y|Z}\parallel {\mathcal{P}}_{X|Z}\otimes {\mathcal{P}}_{Y|Z})\right]$, the above theorem does not provide a high-probability bound. Modifying this framework to ensure high-probability guarantees was left as future work in [12]. In the following, we use conditional $\alpha$-mutual information to address this issue.

Theorem 4.

Let $U\in {\{0,1\}}^n$ be uniformly random. Given a dataset $Z\sim {\mathcal{D}}^{2n}$ consists of $2nH$ samples. Let $\mathcal{A}:Z_U\to \mathcal{W}$ be a randomized algorithm. Let $\ell :\mathcal{W}\times Z\to \mathbb{R}$ be a loss function which is σ-subgaussian with respect to Z. Let $L:\mathcal{W}\times Z_U\to \mathbb{R}$ be the empirical risk. Given $\eta ,\delta \in (0,1)$ and fix $\alpha \ge 1$, if the number of samples n satisfies

$$n\ge {\displaystyle \frac{2{\sigma }^2}{{\eta }^2}}\left(I_{\alpha }^{\mathcal{A}(Z_U)|Z}(\mathcal{A}(Z_U);U|Z)+log2+{\displaystyle \frac{\alpha }{\alpha -1}}log\left({\displaystyle \frac{1}{\delta }}\right)\right)$$

then, we have

$$\mathbb{P}\left(|L(\mathcal{A}(Z_U),Z_U)-L(\mathcal{A}(Z_U),\mathcal{D})|\le \eta \right)\ge 1-\delta .$$

Proof. 

Let $(\mathcal{X}\times \mathcal{Y}\times \mathcal{Z},\mathcal{F},{\mathcal{P}}_{XYZ})$ be a probability space, and let $\mathcal{Q}(\mathcal{X}|\mathcal{Z})$ be the set of conditional probability measures ${\mathcal{Q}}_{X|Z}$, such that ${\mathcal{P}}_{XYZ}\ll {\mathcal{P}}_Z{Q}_{X|Z}{\mathcal{P}}_{Y|Z}$. Given $E\in \mathcal{F}$ and $z\in \mathcal{Z},x\in \mathcal{X}$, let $E_{z,x}=\{y\in \mathcal{Y}:(x,y,z)\in E\}$. We first prove that for a fixed $\alpha \ge 1$,

$$\begin{array}{c}\hfill {\mathcal{P}}_{XYZ}(E)\le {\mathbb{E}}_Z{\left[{\mathrm{ess} \sup }_{Q_{X|Z}\in \mathcal{Q}(\mathcal{X}|\mathcal{Z})}{\mathcal{P}}_{Y|Z}(E_{Z,X})\right]}^{{\displaystyle \frac{\alpha -1}{\alpha }}}exp\left({\displaystyle \frac{\alpha -1}{\alpha }}{I}_{\alpha }^{X|Z}(X;Y|Z)\right).\end{array}$$

(2)

Using the Radon–Nikodym derivative of ${\mathcal{P}}_{XYZ}$ with respect to the product measure ${\mathcal{P}}_Z{\mathcal{Q}}_{X|Z}{\mathcal{P}}_{Y|Z}$, we have

$$\begin{array}{cc}\hfill {\mathcal{P}}_{XYZ}(E)& ={\mathbb{E}}_{{\mathcal{P}}_Z{Q}_{X|Z}{\mathcal{P}}_{Y|Z}}\left[{\displaystyle \frac{d{\mathcal{P}}_{XYZ}}{d{\mathcal{P}}_Z{\mathcal{Q}}_{X|Z}{\mathcal{P}}_{Y|Z}}}{\mathcal{I}}_E\right]\hfill \end{array}$$

where ${\mathcal{I}}_E$ is the indicator function of the event E. Next, we introduce three sets of exponents ${\alpha }^{″},{\alpha }^{\prime },\alpha$, and ${\gamma }^{″},{\gamma }^{\prime },\gamma$, such that

$${\displaystyle \frac{1}{{\alpha }^{″}}}+{\displaystyle \frac{1}{{\gamma }^{″}}}={\displaystyle \frac{1}{{\alpha }^{\prime }}}+{\displaystyle \frac{1}{{\gamma }^{\prime }}}={\displaystyle \frac{1}{\alpha }}+{\displaystyle \frac{1}{\gamma }}=1.$$

By applying Hölder’s inequality three times to separate the different components of the expectation, we derive

$$\begin{array}{cc}\hfill & {\mathbb{E}}_{{\mathcal{P}}_Z{Q}_{X|Z}{\mathcal{P}}_{Y|Z}}\left[{\displaystyle \frac{d{\mathcal{P}}_{XYZ}}{d{\mathcal{P}}_Z{\mathcal{Q}}_{X|Z}{\mathcal{P}}_{Y|Z}}}{\mathcal{I}}_E\right]\hfill \\ \hfill & \le {\mathbb{E}}_{{\mathcal{P}}_Z}^{{\displaystyle \frac{1}{{\alpha }^{″}}}}\left[{\mathbb{E}}_{{\mathcal{Q}}_{X|Z}}^{{\displaystyle \frac{{\alpha }^{″}}{{\alpha }^{\prime }}}}\left[{\mathbb{E}}_{{\mathcal{Q}}_{Y|Z}}^{{\displaystyle \frac{\alpha }{{\alpha }^{\prime }}}}\left[{\left({\displaystyle \frac{d{\mathcal{P}}_{XYZ}}{d{\mathcal{P}}_Z{\mathcal{Q}}_{X|Z}{\mathcal{P}}_{Y|Z}}}\right)}^{\alpha }\right]\right]\right]{\mathbb{E}}_{{\mathcal{P}}_Z}^{{\displaystyle \frac{1}{{\gamma }^{″}}}}\left[{\mathbb{E}}_{{\mathcal{Q}}_{X|Z}}^{{\displaystyle \frac{{\gamma }^{″}}{{\gamma }^{\prime }}}}\left[{\mathbb{E}}_{{\mathcal{P}}_{Y|Z}}^{{\displaystyle \frac{{\gamma }^{\prime }}{\gamma }}}\left[{\mathcal{I}}_E^{\gamma }\right]\right]\right].\hfill \end{array}$$

By setting ${\alpha }^{″}=\alpha$ and ${\alpha }^{\prime }=1$,

$$\begin{array}{c}\hfill {\mathbb{E}}_{{\mathcal{P}}_Z}^{{\displaystyle \frac{1}{{\alpha }^{″}}}}\left[{\mathbb{E}}_{{\mathcal{Q}}_{X|Z}}^{{\displaystyle \frac{{\alpha }^{″}}{{\alpha }^{\prime }}}}\left[{\mathbb{E}}_{{\mathcal{Q}}_{Y|Z}}^{{\displaystyle \frac{\alpha }{{\alpha }^{\prime }}}}\left[{\left({\displaystyle \frac{d{\mathcal{P}}_{XYZ}}{d{\mathcal{P}}_Z{\mathcal{Q}}_{X|Z}{\mathcal{P}}_{Y|Z}}}\right)}^{\alpha }\right]\right]\right]\le exp\left({\displaystyle \frac{\alpha -1}{\alpha }}{I}_{\alpha }^{X|Z}(X;Y|Z)\right).\end{array}$$

Since ${\alpha }^{\prime }=1$ and ${\displaystyle \frac{1}{{\alpha }^{\prime }}}+{\displaystyle \frac{1}{{\gamma }^{\prime }}}=1$, we have ${\gamma }^{\prime }\to \infty$. As ${\gamma }^{\prime }\to \infty$, ${\mathbb{E}}_{{\mathcal{Q}}_{X|Z}}^{{\displaystyle \frac{{\gamma }^{″}}{{\gamma }^{\prime }}}}\left[{\mathbb{E}}_{{\mathcal{P}}_{Y|Z}}^{{\displaystyle \frac{{\gamma }^{\prime }}{\gamma }}}\left[{\mathcal{I}}_E^{\gamma }\right]\right]$ tends to the essential supremum

$$\begin{array}{c}\hfill {\mathrm{ess} \sup }_{Q_{X|Z}\in \mathcal{Q}(\mathcal{X}|\mathcal{Z})}{\mathcal{P}}_{Y|Z}(E_{Z,X}).\end{array}$$

As ${\displaystyle \frac{1}{{\gamma }^{″}}}={\displaystyle \frac{\alpha -1}{\alpha }}$, we have

$$\begin{array}{c}\hfill {\mathbb{E}}_{{\mathcal{P}}_Z}^{{\displaystyle \frac{1}{{\gamma }^{″}}}}\left[{\mathbb{E}}_{{\mathcal{Q}}_{X|Z}}^{{\displaystyle \frac{{\gamma }^{″}}{{\gamma }^{\prime }}}}\left[{\mathbb{E}}_{{\mathcal{P}}_{Y|Z}}^{{\displaystyle \frac{{\gamma }^{\prime }}{\gamma }}}\left[{\mathcal{I}}_E^{\gamma }\right]\right]\right]\le {\mathbb{E}}_Z{\left[{\mathrm{ess} \sup }_{Q_{X|Z}\in \mathcal{Q}(\mathcal{X}|\mathcal{Z})}{\mathcal{P}}_{Y|Z}(E_{Z,X})\right]}^{{\displaystyle \frac{\alpha -1}{\alpha }}}.\end{array}$$

Thus, Equation (2) holds by combining all of the inequalities.

Now, let $X=\mathcal{A}(Z_U)$ and $Y=U$. Consider the event

$$E=\left\{(X,Y,Z):\left|L(X,Z_Y)-{\mathbb{E}}_Y\left[L(X,\mathcal{D})\right]\right|\ge \eta \right\},$$

where $L(X,Z_Y)$ denotes the empirical risk defined as the average of n loss functions, and each loss function is $\sigma$-subgaussian. We can express $E_{Z,X}$, the fibers of E, with respect to Z and X, as

$$E_{Z,X}=\left\{Y:\left|L(X,Z_Y)-{\mathbb{E}}_Y\left[L(X,\mathcal{D})\right]\right|\ge \eta \right\}.$$

For any fixed Z and X, the random variable Y remains independent of Z and X under any ${\mathcal{Q}}_{X|Z}\in \mathcal{Q}(\mathcal{X}|\mathcal{Z})$. Now, using Hoeffding’s inequality, for every X and Z,

$$\begin{array}{c}\hfill {\mathcal{P}}_Y(E_{Z,X})\le 2exp\left({\displaystyle \frac{-n{\eta }^2}{2{\sigma }^2}}\right).\end{array}$$

(3)

Therefore, from Equations (2) and (3),

$$\begin{array}{cc}\hfill \mathbb{P}(E)& \le 2exp\left({\displaystyle \frac{\alpha -1}{\alpha }}·{\displaystyle \frac{-n{\eta }^2}{2{\sigma }^2}}\right)exp\left({\displaystyle \frac{\alpha -1}{\alpha }}{I}_{\alpha }^{\mathcal{A}(Z_U)|Z}(\mathcal{A}(Z_U);U|Z)\right)\hfill \\ \hfill & =2exp\left({\displaystyle \frac{\alpha -1}{\alpha }}\left(I_{\alpha }^{\mathcal{A}(Z_U)|Z}(\mathcal{A}(Z_U);U|Z)-{\displaystyle \frac{-n{\eta }^2}{2{\sigma }^2}}\right)\right).\hfill \end{array}$$

Lastly, by setting

$$n\ge {\displaystyle \frac{2{\sigma }^2}{{\eta }^2}}\left(I_{\alpha }^{\mathcal{A}(Z_U)|Z}(\mathcal{A}(Z_U);U|Z)+log2+{\displaystyle \frac{\alpha }{\alpha -1}}log\left({\displaystyle \frac{1}{\delta }}\right)\right)$$

we obtain the desired conclusion. □

4. Information-Theoretic Generalization Bounds for Batch RL

We now provide expected and high-probability generalization bounds for batch RL. The generalization bounds are derived from mutual information between the training data and the learned hypothesis. As mutual information bounds consider the data, algorithm, and hypothesis space comprehensively, they support the design of efficient learning algorithms and fine-grained theoretical analysis.

Theorem 5.

Given that dataset $Z\sim {\mathcal{D}}^n$ consists of $nH$ samples, for any batch RL algorithm $\mathcal{A}$ with output $\mathcal{A}(Z)=f=(f_1,\cdots ,f_H)\in \mathcal{F}$, the expected generalization error for the mean squared empirical Bellman error (MSBE) loss is upper bounded by

$$\begin{array}{c}\hfill \left|{\mathbb{E}}_{Z\sim \mathcal{D}}[L(\mathcal{A}(Z),Z)-L(\mathcal{A}(Z),\mathcal{D})]\right|\le \sqrt{{\displaystyle \frac{2H^2{\sum }_{h=1}^{H}I(f_h;Z_h)}{n}}}.\end{array}$$

Proof. 

We first recall the Donsker—Varadhan variational representation ([13]) of the KL-divergence between any two probability measures $\pi$ and $\rho$ on a common measurable space $(\Omega ,\mathcal{F})$

$$\begin{array}{c}\hfill D_{\mathrm{KL}}(\pi \parallel \rho )=\underset{F}{sup}\left\{{\int }_{\Omega }Fd\pi -log{\int }_{\Omega }{e}^{F}d\rho \right\}\end{array}$$

where the supremum is over all measurable functions $F:\Omega \to \mathbb{R}$, such that $e^F\in L^1(\rho )$.

Let be $Z=Z_1\cup \cdots \cup Z_H$ be a dataset where $Z_h=\left\{(s,a,r,s^{\prime },h)\right\}\sim {\mathcal{D}}_h$. Let $\mathcal{A}(Z)=f=(f_1,\cdots ,f_H)\in \mathcal{F}$ be the output of some batch RL algorithm $\mathcal{A}$. Let ${\tilde{f}}_h$ and ${\tilde{Z}}_h$ be the independent copies of $f_h$ and $Z_h$. Let

$$\begin{array}{cc}\hfill L(f,Z)& ={\displaystyle \frac{1}{H}}\sum _{h=1}^H\ell (f_h,Z_h)\hfill \\ \hfill & ={\displaystyle \frac{1}{H}}\sum _{h=1}^H{\displaystyle \frac{1}{n}}\sum _{(s,a,r,s^{\prime },h)\in Z_h}{(f_h(s,a)-r-V_{f_{h+1}}(s^{\prime }))}^2.\hfill \end{array}$$

Now, we have

$$\begin{array}{cc}\hfill I(f_h;Z_h)& =D_{\mathrm{KL}}(P_{f_h,Z_h}\parallel P_{f_h}\otimes P_{Z_h})\hfill \\ \hfill & =\underset{g}{sup}\left\{{\mathbb{E}}_{f_h,Z_h}\left[g(f_h,Z_h)\right]-log{\mathbb{E}}_{{\tilde{f}}_h,{\tilde{Z}}_h}\left[e^{g({\tilde{f}}_h,{\tilde{Z}}_h)}\right]\right\}\hfill \\ \hfill & \hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}(\text{Donsker–Varadhan} \mathrm{variational} \mathrm{representation})\hfill \\ \hfill & \ge \lambda {\mathbb{E}}_{f_h,Z_h}[\ell (f_h,Z_h)]-log{\mathbb{E}}_{{\tilde{f}}_h,{\tilde{Z}}_h}\left[e^{\lambda \ell ({\tilde{f}}_h,{\tilde{Z}}_h)}\right].\hspace{1em}\hspace{1em}\hspace{1em}(\forall \lambda \in \mathbb{R})\hfill \end{array}$$

As $\ell (f_h,Z_h)={\displaystyle \frac{1}{n}}{\sum }_{(s,a,r,s^{\prime },h)\in Z_h}{(f_h(s,a)-r-V_{f_{h+1}}(s^{\prime }))}^2$ and ${(f_h(s,a)-r-V_{f_{h+1}}(s^{\prime }))}^2\in [0,4H^2]$ for any h, it follows that

$$\begin{array}{c}\hfill log{\mathbb{E}}_{{\tilde{f}}_h,{\tilde{Z}}_h}\left[e^{\lambda (\ell ({\tilde{f}}_h,{\tilde{Z}}_h)-{\mathbb{E}}_{{\tilde{f}}_h,{\tilde{Z}}_h}[\ell ({\tilde{f}}_h,{\tilde{Z}}_h)])}\right]\le {\displaystyle \frac{2{\lambda }^2{H}^4}{n}}.\end{array}$$

Thus, we obtain

$$\begin{array}{cc}\hfill I(f_h;Z_h)& \ge \lambda \left({\mathbb{E}}_{f_h,Z_h}[\ell (f_h,Z_h)]-{\mathbb{E}}_{{\tilde{f}}_h,{\tilde{Z}}_h}[\ell ({\tilde{f}}_h,{\tilde{Z}}_h)])]\right)-{\displaystyle \frac{2{\lambda }^2{H}^4}{n}}\hfill \\ \hfill \Rightarrow {\displaystyle \frac{I(f_h;Z_h)}{\lambda }}+{\displaystyle \frac{2{\lambda }^2{H}^4}{n}}& \ge {\mathbb{E}}_{f_h,Z_h}[\ell (f_h,Z_h)]-{\mathbb{E}}_{{\tilde{f}}_h,{\tilde{Z}}_h}[\ell ({\tilde{f}}_h,{\tilde{Z}}_h)])].\hfill \end{array}$$

By optimizing the above inequality over $\lambda >0$ and $\lambda <0$, respectively, we derive

$$\begin{array}{c}\hfill -H^2\sqrt{{\displaystyle \frac{2I(f_h;Z_h)}{n}}}\le {\mathbb{E}}_{f_h,Z_h}[\ell (f_h,Z_h)]-{\mathbb{E}}_{{\tilde{f}}_h,{\tilde{Z}}_h}[\ell ({\tilde{f}}_h,{\tilde{Z}}_h)])]\le H^2\sqrt{{\displaystyle \frac{2I(f_h;Z_h)}{n}}},\end{array}$$

and thus,

$$\begin{array}{c}\hfill \left|{\mathbb{E}}_{f_h,Z_h}[\ell (f_h,Z_h)]-{\mathbb{E}}_{{\tilde{f}}_h,{\tilde{Z}}_h}[\ell ({\tilde{f}}_h,{\tilde{Z}}_h)])]\right|\le H^2\sqrt{{\displaystyle \frac{2I(f_h;Z_h)}{n}}}.\end{array}$$

(4)

Finally, we observe that

$$\begin{array}{cc}\hfill \left|{\mathbb{E}}_{Z\sim \mathcal{D}}[L(\mathcal{A}(Z),Z)-L(\mathcal{A}(Z),\mathcal{D})]\right|& =\left|{\mathbb{E}}_{Z\sim \mathcal{D}}\left[{\displaystyle \frac{1}{H}}\sum _{h=1}^H\ell (f_h,Z_h)-{\mathbb{E}}_{Z\sim \mathcal{D}}\left[{\displaystyle \frac{1}{H}}\sum _{h=1}^H\ell (f_h,Z_h)\right]\right]\right|\hfill \\ \hfill & =\left|{\displaystyle \frac{1}{H}}\sum _{h=1}^H{\mathbb{E}}_{Z_h\sim {\mathcal{D}}_h}\left[\ell (f_h,Z_h)-{\mathbb{E}}_{Z_h\sim {\mathcal{D}}_h}\left[\ell (f_h,Z_h)\right]\right]\right|\hfill \\ \hfill & ={\displaystyle \frac{1}{H}}\sum _{h=1}^H\left|{\mathbb{E}}_{f_h,Z_h}[\ell (f_h,Z_h)]-{\mathbb{E}}_{{\tilde{f}}_h,{\tilde{Z}}_h}[\ell ({\tilde{f}}_h,{\tilde{Z}}_h)])]\right|\hfill \\ \hfill & \le {\displaystyle \frac{1}{H}}\sum _{h=1}^H{H}^2\sqrt{{\displaystyle \frac{2I(f_h;Z_h)}{n}}}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}(\mathrm{By} \mathrm{Equation} (4))\hfill \\ \hfill & =\sqrt{{\displaystyle \frac{2H^2{\sum }_{h=1}^{H}I(f_h;Z_h)}{n}}}.\hfill \end{array}$$

□

The above result suggests that reducing the mutual information between the dataset $Z_h$ and the learned function $f_h$ at each step h can improve the generalization performance. Note that when the input domain is infinite, mutual information can become unbounded. To address this limitation, an approach based on conditional mutual information was introduced [12]. CMI bounds not only address the issue by normalizing the information content of each data point, but also establish connections with various other generalization concepts, as we will discuss in the next section. We now present a generalization bound using conditional mutual information.

Theorem 6.

Let $U\in {\{0,1\}}^n$ be uniformly random. Given that dataset $Z\sim {\mathcal{D}}^{2n}$ consists of $2nH$ samples, for any batch RL algorithm $\mathcal{A}$ with output $\mathcal{A}(Z_U)=f=(f_1,\cdots ,f_H)\in \mathcal{F}$, the expected generalization error for the mean squared empirical Bellman error (MSBE) loss is upper bounded by

$$\begin{array}{c}\hfill \left|{\mathbb{E}}_{Z\sim \mathcal{D}}[L(\mathcal{A}(Z_U),Z_U)-L(\mathcal{A}(Z_U),\mathcal{D})]\right|\le \sqrt{{\displaystyle \frac{2H^2{\sum }_{h=1}^{H}I(f_h;U|Z_h)}{n}}}.\end{array}$$

Proof. 

Let $U\in {\{0,1\}}^n$ be uniformly random. Let $Z=Z_1\cup \cdots \cup Z_H$ be a dataset where each $Z_h=\left\{(s,a,r,s^{\prime },h)\right\}\sim {\mathcal{D}}_h$ consists of $2n$ samples. Define $Z_U={(Z_1)}_U\cup \cdots \cup {(Z_H)}_U$. Let $\mathcal{A}(Z_U)=f=(f_1,\cdots ,f_H)\in \mathcal{F}$ be the output of some batch RL algorithm $\mathcal{A}$. Let ${\overline{f}}_h=\mathcal{A}{(Z_{\overline{U}})}_h$, ${\tilde{Z}}_h={(Z_h)}_U$ and ${\overline{Z}}_h={(Z_h)}_{\overline{U}}$. Note that $Z_h={\tilde{Z}}_h\cup {\overline{Z}}_h$. We define the disintegrated mutual information

$$I^Z(X;Y):=D_{KL}(P_{X,Y|Z}\parallel P_X{P}_Y|Z).$$

Note that $I(X;Y|Z)={\mathbb{E}}_Z\left[I^Z(X;Y)\right]$. The rest of the proof is analogous to Theorem 5. We have

$$\begin{array}{cc}\hfill I^{Z_h}(f_h;{\tilde{Z}}_h|Z_h)& =D_{\mathrm{KL}}(P_{f_h,{\tilde{Z}}_h|Z_h}\parallel P_{f_h|Z_h}\otimes P_{{\tilde{Z}}_h|Z_h})\hfill \\ \hfill & =\underset{g}{sup}\left\{{\mathbb{E}}_{f_h,{\tilde{Z}}_h|Z_h}\left[g(f_h,{\tilde{Z}}_h)\right]-log{\mathbb{E}}_{{\overline{f}}_h,{\overline{Z}}_h|Z_h}\left[e^{g({\overline{f}}_h,{\overline{Z}}_h)}\right]\right\}\hfill \\ \hfill & \hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}(\text{Donsker–Varadhan} \mathrm{variational} \mathrm{representation})\hfill \\ \hfill & \ge \lambda {\mathbb{E}}_{f_h,{\tilde{Z}}_h|Z_h}[\ell (f_h,{\tilde{Z}}_h)]-log{\mathbb{E}}_{{\overline{f}}_h,{\overline{Z}}_h|Z_h}\left[e^{\lambda \ell ({\overline{f}}_h,{\overline{Z}}_h)}\right].\hspace{1em}\hspace{1em}\hspace{1em}(\forall \lambda \in \mathbb{R})\hfill \end{array}$$

As $\ell (f_h,Z_h)={\displaystyle \frac{1}{n}}{\sum }_{(s,a,r,s^{\prime },h)\in Z_h}{(f_h(s,a)-r-V_{f_{h+1}}(s^{\prime }))}^2$ and ${(f_h(s,a)-r-V_{f_{h+1}}(s^{\prime }))}^2\in [0,4H^2]$ for any h, it follows that

$$\begin{array}{c}\hfill log{\mathbb{E}}_{{\overline{f}}_h,{\overline{Z}}_h|Z_h}\left[e^{\lambda (\ell ({\overline{f}}_h,{\overline{Z}}_h)-{\mathbb{E}}_{{\overline{f}}_h,{\overline{Z}}_h|Z_h}[\ell ({\overline{f}}_h,{\overline{Z}}_h)])}\right]\le {\displaystyle \frac{2{\lambda }^2{H}^4}{n}}.\end{array}$$

Thus, we obtain

$$\begin{array}{cc}\hfill I^{Z_h}(f_h;{\tilde{Z}}_h|Z_h)& \ge \lambda \left({\mathbb{E}}_{f_h,{\tilde{Z}}_h|Z_h}[\ell (f_h,{\tilde{Z}}_h)]-{\mathbb{E}}_{{\overline{f}}_h,{\overline{Z}}_h|Z_h}[\ell ({\overline{f}}_h,{\overline{Z}}_h)])]\right)-{\displaystyle \frac{2{\lambda }^2{H}^4}{n}}\hfill \\ \hfill \Rightarrow {\displaystyle \frac{I^{Z_h}(f_h;{\tilde{Z}}_h|Z_h)}{\lambda }}+{\displaystyle \frac{2{\lambda }^2{H}^4}{n}}& \ge {\mathbb{E}}_{f_h,{\tilde{Z}}_h|Z_h}[\ell (f_h,{\tilde{Z}}_h)]-{\mathbb{E}}_{{\overline{f}}_h,{\overline{Z}}_h|Z_h}[\ell ({\overline{f}}_h,{\overline{Z}}_h)])].\hfill \end{array}$$

By optimizing the above inequality over $\lambda >0$ and $\lambda <0$, respectively, we derive

$$\begin{array}{c}\hfill -H^2\sqrt{{\displaystyle \frac{2I^{Z_h}(f_h;{\tilde{Z}}_h|Z_h)}{n}}}\le {\mathbb{E}}_{f_h,{\tilde{Z}}_h|Z_h}[\ell (f_h,{\tilde{Z}}_h)]-{\mathbb{E}}_{{\overline{f}}_h,{\overline{Z}}_h|Z_h}[\ell ({\overline{f}}_h,{\overline{Z}}_h)])]\le H^2\sqrt{{\displaystyle \frac{2I^{Z_h}(f_h;{\tilde{Z}}_h|Z_h)}{n}}},\end{array}$$

and thus,

$$\begin{array}{c}\hfill \left|{\mathbb{E}}_{f_h,{\tilde{Z}}_h|Z_h}[\ell (f_h,{\tilde{Z}}_h)]-{\mathbb{E}}_{{\overline{f}}_h,{\overline{Z}}_h|Z_h}[\ell ({\overline{f}}_h,{\overline{Z}}_h)])]\right|\le H^2\sqrt{{\displaystyle \frac{2I^{Z_h}(f_h;{\tilde{Z}}_h|Z_h)}{n}}}.\end{array}$$

(5)

Finally, we conclude that

$$\begin{array}{cc}\hfill \left|{\mathbb{E}}_{Z\sim \mathcal{D}}[L(\mathcal{A}(Z_U),Z_U)-L(\mathcal{A}(Z_U),\mathcal{D})]\right|& =\left|{\mathbb{E}}_{Z\sim \mathcal{D}}\left[{\displaystyle \frac{1}{H}}\sum _{h=1}^H\ell (f_h,{\tilde{Z}}_h)-{\mathbb{E}}_{Z\sim \mathcal{D}}\left[{\displaystyle \frac{1}{H}}\sum _{h=1}^H\ell (f_h,{\tilde{Z}}_h)\right]\right]\right|\hfill \\ \hfill & =\left|{\displaystyle \frac{1}{H}}\sum _{h=1}^H{\mathbb{E}}_{Z_h\sim {\mathcal{D}}_h}\left[\ell (f_h,\tilde{Z})-{\mathbb{E}}_{Z_h\sim {\mathcal{D}}_h}\left[\ell (f_h,\tilde{Z})\right]\right]\right|\hfill \\ \hfill & \le {\displaystyle \frac{1}{H}}\sum _{h=1}^H\left|{\mathbb{E}}_{Z_h\sim {\mathcal{D}}_h}\left[{\mathbb{E}}_{{\overline{f}}_h,{\overline{Z}}_h|Z_h}[\ell ({\overline{f}}_h,{\overline{Z}}_h)])]-{\mathbb{E}}_{f_h,{\tilde{Z}}_h|Z_h}[\ell (f_h,{\tilde{Z}}_h)])]\right]\right|\hfill \\ \hfill & \le {\displaystyle \frac{1}{H}}\sum _{h=1}^H{H}^2{\mathbb{E}}_{Z_h\sim {\mathcal{D}}_h}\left[\sqrt{{\displaystyle \frac{2I^{Z_h}(f_h;{\tilde{Z}}_h|Z_h)}{n}}}\right]\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}(\mathrm{By} \mathrm{Equation} (5))\hfill \\ \hfill & \le {\displaystyle \frac{1}{H}}\sum _{h=1}^H{H}^2\sqrt{{\displaystyle \frac{2{\mathbb{E}}_{Z_h\sim {\mathcal{D}}_h}\left[I^{Z_h}(f_h;{\tilde{Z}}_h|Z_h)\right]}{n}}}\hfill \\ \hfill & =\sqrt{{\displaystyle \frac{2H^2{\sum }_{h=1}^{H}I(f_h;{\tilde{Z}}_h|Z_h)}{n}}}\hfill \\ \hfill & =\sqrt{{\displaystyle \frac{2H^2{\sum }_{h=1}^{H}I(f_h;U|Z_h)}{n}}}.\hfill \end{array}$$

□

Note that our setting is identical to that in [3], i.e., batch RL with value function approximation for episodic MDPs. They established a bound of the order $\tilde{O}\left(H^2\sqrt{{\displaystyle \frac{1}{n}}}+{\sum }_{h=1}^H\mathcal{R}({\mathcal{F}}_h)\right)$, where $\mathcal{R}({\mathcal{F}}_h)$ represents the Rademacher complexity of the function space ${\mathcal{F}}_h$. In contrast, our result yields an error bound of the order $O\left(H\sqrt{{\displaystyle \frac{{\sum }_{h=1}^{H}I(f_h;Z_h)}{n}}}\right)$. As demonstrated in the subsequent section, under structural assumptions like a finite pseudo-dimension or effective dimension d, this bound can be refined to $\tilde{O}\left(H^2\sqrt{{\displaystyle \frac{d}{n}}}\right)$.

Next, we proceed to derive the high-probability version of these generalization bounds using $\alpha$-mutual information.

Theorem 7.

Given a dataset $Z\sim {\mathcal{D}}^n$ consists of $nH$ samples, for any batch RL algorithm $\mathcal{A}$ with output $\mathcal{A}(Z)=f=(f_1,\cdots ,f_H)\in \mathcal{F}$, if

$$\begin{array}{c}\hfill n\ge {\displaystyle \frac{2H^4}{{ϵ}^2}}\left(I_{\alpha }(\mathcal{A}(Z);Z)+log2+{\displaystyle \frac{\alpha }{\alpha -1}}log\left({\displaystyle \frac{1}{\delta }}\right)\right),\end{array}$$

then, the generalization error for the mean squared empirical Bellman error (MSBE) loss is upper bounded by

$$\begin{array}{c}\hfill \left|L(\mathcal{A}(Z),Z)-L(\mathcal{A}(Z),\mathcal{D})\right|\le ϵ\end{array}$$

with a probability of at least $1-\delta$.

Proof. 

Let $Z=Z_1\cup \cdots \cup Z_H$ be a dataset where $Z_h=\left\{(s,a,r,s^{\prime },h)\right\}\sim {\mathcal{D}}_h$. Let $\mathcal{A}(Z)=f=(f_1,\cdots ,f_H)\in \mathcal{F}$ be the output of some batch RL algorithm $\mathcal{A}$. Let

$$\begin{array}{cc}\hfill L(f,Z)& ={\displaystyle \frac{1}{H}}\sum _{h=1}^H\ell (f_h,Z_h)\hfill \\ \hfill & ={\displaystyle \frac{1}{H}}\sum _{h=1}^H{\displaystyle \frac{1}{n}}\sum _{(s,a,r,s^{\prime },h)\in Z_h}{(f_h(s,a)-r-V_{f_{h+1}}(s^{\prime }))}^2.\hfill \end{array}$$

As $\ell (f,Z)\in [0,4H^2]$ for every f, it is $2H^2$-sub-Gaussian. By Theorem 2, we have

$$\begin{array}{c}\hfill |\ell (f_h,Z_h)-{\mathbb{E}}_{Z_h\sim {\mathcal{D}}_h}[\ell (f_h,Z_h)]|\le ϵ\end{array}$$

with probability at least $1-{\delta }^{\prime }$ for

$$\begin{array}{c}\hfill n\ge {\displaystyle \frac{8H^4}{{ϵ}^2}}\left(I_{\alpha }(f_h;Z_h)+log2+{\displaystyle \frac{\alpha }{\alpha -1}}log\left({\displaystyle \frac{1}{{\delta }^{\prime }}}\right)\right).\end{array}$$

As we have n samples at each $h\in \left[H\right]$, we require

$$\begin{array}{c}\hfill n\ge {\displaystyle \frac{8H^4}{{ϵ}^2}}\left(\underset{h}{max}{I}_{\alpha }(f_h;Z_h)+log2+{\displaystyle \frac{\alpha }{\alpha -1}}log\left({\displaystyle \frac{1}{{\delta }^{\prime }}}\right)\right).\end{array}$$

The claim is now followed by the union bound by setting ${\delta }^{\prime }=\delta /H$. □

Recall that conditional mutual information is defined as an expectation over the KL divergence. Thus, all prior works using the CMI framework have only provided bounds on the expected generalization error. We wish to establish generalization bounds with high-probability guarantees similar to Theorem 7.

Theorem 8.

Let $U\in {\{0,1\}}^n$ be uniformly random. Given that dataset $Z\sim {\mathcal{D}}^{2n}$ consists of $2nH$ samples, for any batch RL algorithm $\mathcal{A}$ with output $\mathcal{A}(Z_U)=f=(f_1,\cdots ,f_H)\in \mathcal{F}$, if

$$\begin{array}{c}\hfill n\ge {\displaystyle \frac{8H^4}{{ϵ}^2}}\left(\underset{h}{max}{I}_{\alpha }^{f_h|Z_h}(f_h;U|Z_h)+log2+{\displaystyle \frac{\alpha }{\alpha -1}}log\left({\displaystyle \frac{H}{\delta }}\right)\right).\end{array}$$

then, the generalization error for the mean squared empirical Bellman error (MSBE) loss is upper bounded by

$$\begin{array}{c}\hfill \left|L(\mathcal{A}(Z_U),Z_U)-L(\mathcal{A}(Z_U),\mathcal{D})\right|\le ϵ\end{array}$$

with probability at least $1-\delta$.

Proof. 

By substituting Theorem 2 with Theorem 4 in the proof of Theorem 7, the proof is thereby obtained. □

5. Value Functions Under Structural Assumptions

Due to the challenges stemming from large state-action spaces, long horizons, and the temporal nature of data, there is increasing interest in identifying structural assumptions for RL with value function approximation. These works include, but are not limited to, Bellman rank [14], Witness rank [15], and Eluder dimension [16]. These structural conditions aim to develop a unified theory of generalization in RL. In this section, we demonstrate that if a function class satisfies certain structural conditions reflecting a manageable complexity, the mutual information can be effectively upper bounded.

Definition 11

(Covering number). The covering number of a function class $\mathcal{F}={\mathcal{F}}_1\times \cdots \times {\mathcal{F}}_H$ under metric $\rho (f,g)={max}_h{\parallel f_h-g_h\parallel }_{\infty }$, denoted as $\mathcal{N}(\mathcal{F},ϵ)$, is the minimum integer n, such that there exists a subset ${\mathcal{F}}_{ϵ}\subseteq \mathcal{F}$ with $|{\mathcal{F}}_{ϵ}|=n$, and for any $f\in \mathcal{F}$, there exists $g\in {\mathcal{F}}_{ϵ}$, such that $\rho (x,y)\le ϵ$.

Lemma 2.

For discrete random variables $X,Y$, and Z, we have $I(X;Y|Z)\le log|X|$.

Proof. 

Denote $H(X\mid Z)$ the conditional entropy of X given Z.

$$\begin{array}{cc}\hfill I(X;Y|Z)& =H(X|Z)-H(X|Y,Z)\hfill \\ \hfill & \le H(X|Z)\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}(H(X|Y,Z)\ge 0)\hfill \\ \hfill & ={\mathbb{E}}_z\left[H(X|Z=z)\right]\hfill \\ \hfill & \le {\mathbb{E}}_z[log|X\left|\right]\hfill \\ \hfill & =log\left|X\right|.\hfill \end{array}$$

□

Theorem 9.

Suppose the function class $\mathcal{F}$ has a covering number of $\mathcal{N}(\mathcal{F},ϵ)$. Let $U\in {\{0,1\}}^n$ be uniformly random. Given that dataset Z consists of $2nH$ samples, for any batch RL algorithm $\mathcal{A}$ with output $\mathcal{A}(Z_U)=f=(f_1,\cdots ,f_H)\in \mathcal{F}$, the expected generalization error for the mean squared empirical Bellman error (MSBE) loss is upper bounded by

$$\begin{array}{c}\hfill \left|{\mathbb{E}}_{Z\sim \mathcal{D}}[L(\mathcal{A}(Z_U),Z_U)-L(\mathcal{A}(Z_U),\mathcal{D})]\right|\le \sqrt{{\displaystyle \frac{2H^{3}log(|\mathcal{N}(\mathcal{F},ϵ)|)}{n}}}+8ϵH+2{ϵ}^2.\end{array}$$

Proof. 

Let ${\tilde{Z}}_h={(Z_h)}_U$. We first define an oracle algorithm ${\mathcal{A}}^o$ capable of outputting a function ${\mathcal{A}}^o(Z_U)=f^{*}=(f_1^{*},\dots ,f_H^{*})$, such that

$$\begin{array}{c}\hfill \rho (f,f^{*})\le ϵ.\end{array}$$

Note that ${\mathcal{A}}^o$ is only used for theoretical analysis. Observe that

$$\begin{array}{cc}\hfill L(\mathcal{A}(Z_U),Z_U)& ={\displaystyle \frac{1}{H}}\sum _{h=1}^H{\displaystyle \frac{1}{n}}\sum _{(s,a,r,s^{\prime },h)\in {\tilde{Z}}_h}{(f_h(s,a)-r-V_{f_{h+1}}(s^{\prime }))}^2\hfill \\ \hfill & ={\displaystyle \frac{1}{H}}\sum _{h=1}^H{\displaystyle \frac{1}{n}}\sum _{(s,a,r,s^{\prime },h)\in {\tilde{Z}}_h}{(f_h(s,a)-f_h^{*}(s,a)+f_h^{*}(s,a)-r-V_{f_{h+1}}(s^{\prime }))}^2\hfill \\ \hfill & ={ϵ}^2+{\displaystyle \frac{1}{H}}\sum _{h=1}^H{\displaystyle \frac{1}{n}}\sum _{(s,a,r,s^{\prime },h)\in {\tilde{Z}}_h}{(f_h^{*}(s,a)-r-V_{f_{h+1}}(s^{\prime }))}^2\hfill \\ \hfill & +2ϵ{\displaystyle \frac{1}{H}}\sum _{h=1}^H{\displaystyle \frac{1}{n}}\sum _{(s,a,r,s^{\prime },h)\in {\tilde{Z}}_h}(f_h^{*}(s,a)-r-V_{f_{h+1}}(s^{\prime }))\hfill \\ \hfill & \le {ϵ}^2+L({\mathcal{A}}^o(Z_U),Z_U)+4ϵH.\hfill \end{array}$$

Thus,

$$\begin{array}{c}\hfill L(\mathcal{A}(Z_U),Z_U)-L({\mathcal{A}}^o(Z_U),Z_U)\le 4ϵH+{ϵ}^2.\end{array}$$

Bounding $|L(\mathcal{A}(Z_U),\mathcal{D})-L({\mathcal{A}}^o(Z_U),\mathcal{D})|$ is similar. Now, we have

$$\begin{array}{cc}\hfill L(\mathcal{A}(Z_U),Z_U)-L(\mathcal{A}(Z_U),\mathcal{D})& =L(\mathcal{A}(Z_U),Z_U)-L({\mathcal{A}}^o(Z_U),Z_U)+L({\mathcal{A}}^o(Z_U),Z_U)\hfill \\ \hfill & -L({\mathcal{A}}^o(Z_U),\mathcal{D})+L({\mathcal{A}}^o(Z_U),\mathcal{D})-L(\mathcal{A}(Z_U),\mathcal{D}).\hfill \end{array}$$

As $|L(\mathcal{A}(Z_U),Z_U)-L({\mathcal{A}}^o(Z_U),Z_U)|\le ϵ$ and $|L(\mathcal{A}(Z_U),\mathcal{D})-L({\mathcal{A}}^o(Z_U),\mathcal{D})|\le ϵ$, we have

$$\begin{array}{c}\hfill L(\mathcal{A}(Z_U),Z_U)-L(\mathcal{A}(Z_U),\mathcal{D})\le L({\mathcal{A}}^o(Z_U),Z_U)-L({\mathcal{A}}^o(Z_U),\mathcal{D})+8ϵH+2{ϵ}^2.\end{array}$$

By Theorem 6,

$$\begin{array}{cc}\hfill \left|{\mathbb{E}}_{Z\sim \mathcal{D}}[L({\mathcal{A}}^o(Z_U),Z_U)-L({\mathcal{A}}^o(Z_U),\mathcal{D})]\right|& \le \sqrt{{\displaystyle \frac{2H^2{\sum }_{h=1}^{H}I(f_h^{*};U|Z_h)}{n}}}\hfill \\ \hfill & \le \sqrt{{\displaystyle \frac{2H^2{\sum }_{h=1}^{H}log(|{\mathcal{F}}_{ϵ}|)}{n}}}\hspace{1em}\hspace{1em}\hspace{1em}(\mathrm{By} \mathrm{Lemma} 2)\hfill \\ \hfill & =\sqrt{{\displaystyle \frac{2H^{3}log(|{\mathcal{F}}_{ϵ}|)}{n}}}\hfill \\ \hfill & =\sqrt{{\displaystyle \frac{2H^{3}log(|\mathcal{N}(\mathcal{F},ϵ)|)}{n}}}.\hfill \end{array}$$

Therefore,

$$\begin{array}{c}\hfill \left|{\mathbb{E}}_{Z\sim \mathcal{D}}[L(\mathcal{A}(Z_U),Z_U)-L(\mathcal{A}(Z_U),\mathcal{D})]\right|\le \sqrt{{\displaystyle \frac{2H^{3}log(|\mathcal{N}(\mathcal{F},ϵ)|)}{n}}}+8ϵH+2{ϵ}^2.\end{array}$$

□

Structural assumptions on the function space typically entail a finite covering number. Next, we consider the simplest case: the pseudo-dimension. The pseudo-dimension is a complexity measure of real-valued function classes, analogous to the VC dimension used for binary classification. Although the value function space may be infinite, it remains learnable if it has a finite pseudo-dimension.

Definition 12

(VC-Dimension [17]). Given hypothesis class $\mathcal{H}\subseteq \mathcal{X}\to \{0,1\}$, its VC-dimension $\mathrm{VCdim}(\mathcal{H})$ is defined as the maximal cardinality of a set $X=\{x_1,\dots ,x_{\left|X\right|}\}\subseteq \mathcal{X}$ that satisfies $|{\mathcal{H}}_X|=2^{\left|X\right|}$ (or X is shattered by $\mathcal{H}$), where ${\mathcal{H}}_X$ is the restriction of $\mathcal{H}$ to X, namely $\{(h(x_1),\dots ,h(x_{\left|X\right|})):h\in \mathcal{H}\}$.

Definition 13

(Pseudo dimension [18]). Suppose $\mathcal{X}$ is a feature space. Given hypothesis class $\mathcal{H}\subseteq \mathcal{X}\to \mathbb{R}$, its pseudo dimension $\mathrm{Pdim}(\mathcal{H})$ is defined as $\mathrm{Pdim}(\mathcal{H})=\mathrm{VCdim}({\mathcal{H}}^{+})$, where ${\mathcal{H}}^{+}=\{(x,\xi )\mapsto 1[h(x)>\xi ]:h\in \mathcal{H}\}\subseteq \mathcal{X}\times \mathbb{R}\to \{0,1\}\}$.

Lemma 3

(Bounding covering number by pseudo dimension [19]). Given hypothesis class $\mathcal{H}\subseteq \mathcal{X}\to \mathbb{R}$ with $\mathrm{Pdim}(\mathcal{H})\le d$, we have

$$log\mathcal{N}(\mathcal{H},ϵ)\le O(dlog(1/ϵ)).$$

Corollary 1.

Suppose the function class ${\mathcal{F}}_h\subset \mathcal{F}$ has a finite pseudo dimension $\mathrm{Pdim}({\mathcal{F}}_h)=d$. For any batch RL algorithm with n training samples, the expected generalization error for the mean squared empirical Bellman error (MSBE) loss is upper bounded by $\tilde{O}(H^2\sqrt{d/n})$.

Proof. 

As $\mathrm{Pdim}({\mathcal{F}}_h)=d$ and $\mathcal{F}={\mathcal{F}}_1\times \cdots \times {\mathcal{F}}_H$, we have $log\mathcal{N}(\mathcal{F},ϵ)\le O(dHlog(1/ϵ))$. The claim follows from Theorem 9 by setting $ϵ=H\sqrt{{\displaystyle \frac{d}{n}}}$. □

A prior study on finite sample guarantees for minimizing the Bellman error, using pseudo-dimension, demonstrated a sample complexity with a dependence of $\tilde{O}(d^2)$ [5]. In contrast, our sample complexity exhibits a dependence of $\tilde{O}(d)$ on the pseudo-dimension.

Now, we introduce another complexity measure known as the effective dimension [20], which has a similar covering number to the pseudo-dimension. The effective dimension quantifies how the function class responds to data, indicating the minimum number of samples required to learn effectively.

Definition 14

($ϵ$-effective dimension of a set [20]). The ϵ-effective dimension of a set $\mathcal{X}$ is the minimum integer $d_{\mathrm{eff}}(\mathcal{X},ϵ)=n$, such that

$$\underset{x_1,\dots ,x_n\in \mathcal{X}}{sup}{\displaystyle \frac{1}{n}}logdet\left(I+{\displaystyle \frac{1}{{ϵ}^2}}\sum _{i=1}^n{x}_i{x}_i^{\top }\right)\le e^{-1}.$$

Definition 15

($ϵ$-effective dimension of a function class [20]). Given a function class $\mathcal{F}$ defined on $\mathcal{X}$, its ϵ-effective dimension $d_{\mathrm{eff}}(\mathcal{F},ϵ)=n$ is the minimum integer n, such that there exists a separable Hilbert space $\mathcal{H}$ and a mapping $\varphi :\mathcal{X}\to \mathcal{H}$, so that

• for every $f\in \mathcal{F}$, there exists ${\theta }_f\in B_{\mathcal{H}}(1)$ satisfying $f(x)={\langle {\theta }_f,\varphi (x)\rangle }_{\mathcal{H}}$ for all $x\in \mathcal{X}$,
• $d_{\mathrm{eff}}(\varphi (\mathcal{X}),ϵ)=n$, where $\varphi (\mathcal{X})=\{\varphi (x):x\in \mathcal{X}\}$.

Definition 16

(Kernel MDPs [21]). In a kernel MDP of effective dimension d, for each step $h\in \left[H\right]$, there exist feature mappings ${\varphi }_h:S\times A\to \mathcal{H}$ and ${\psi }_h:S\to \mathcal{H}$, where $\mathcal{H}$ is a separable Hilbert space, so that the transition measure can be represented as the inner product of features, i.e.,

$${\mathbb{P}}_h(s^{\prime }\mid s,a)={\langle {\varphi }_h(s,a),{\psi }_h(s^{\prime })\rangle }_{\mathcal{H}}.$$

Besides, the reward function is linear in ϕ, i.e.,

$$r_h(s,a)={\langle {\varphi }_h(s,a),{\theta }_h^r\rangle }_{\mathcal{H}}$$

for some ${\theta }_h^r\in \mathcal{H}$. Here, ϕ is known to the learner while ψ and ${\theta }_h^r$ are unknown. Moreover, a kernel MDP satisfies the following regularization conditions: for all h

• $\parallel {\theta }_h^r{\parallel }_{\mathcal{H}}\le 1$ and $\parallel {\varphi }_h{(s,a)\parallel }_{\mathcal{H}}\le 1$ for all $s,a$.
• ${∥{\sum }_{s\in S}\mathcal{V}(s){\psi }_h(s)∥}_{\mathcal{H}}\le 1$ for any function $\mathcal{V}:S\to [0,1]$.
• ${\mathit{dim}}_{\mathit{eff}}(X_h,ϵ)\le d$ for all h, where $X_h=\{{\varphi }_h(s,a):(s,a)\in S\times A\}$.

Kernel MDPs are extensions of the traditional MDPs where the transition dynamics and rewards are represented in a Reproducing Kernel Hilbert Space (RKHS). In this setup, the value functions or Q-functions are approximated using kernel methods, allowing the model to capture more complex dependencies in the data compared to linear models. To learn kernel MDPs, it is necessary to construct a function class $\mathcal{F}$.

Lemma 4

(Bounding covering number by effective dimension [21]). Let $\mathcal{M}$ be a kernel MDP of effective dimension d, then

$$log\mathcal{N}(\mathcal{F},ϵ)\le O(Hdlog(1+dH/ϵ)).$$

Corollary 2.

Suppose the function class $\mathcal{F}$ has a finite effective dimension d. For any batch RL algorithm with n training samples, the expected generalization error for the mean squared empirical Bellman error (MSBE) loss is upper bounded by $\tilde{O}(H^2\sqrt{d/n})$.

We showed that when a function class contains infinitely many elements, a finite covering number can be used to upper bound the generalization error. Just as the VC-dimension imposes a finite cardinality, various concepts in real-valued function classes, such as pseudo-dimension and effective dimension, result in a finite covering number, thereby ensuring efficient learning.

6. Discussion

In this paper, we analyzed the generalization property of batch reinforcement learning within the framework of information theory. We established generalization bounds using both conditional and unconditional mutual information. Besides, we demonstrated how to leverage the structure of the function space to guarantee generalization. Due to the merits of the information-theoretic approach, there are several appealing future research directions.

The first interesting avenue is to extend the results to the online setting. It is noteworthy that in on-policy learning, the inputs (e.g., the reward and the next state), are influenced by the output (e.g., the policy or the model), which highlights a significant disparity compared to off-policy and supervised learning. In supervised learning, a small mutual information between the input and the output indicates that the model is not overfitting. In on-policy learning, analyzing the mutual information between the input and the output can be more complicated and insightful. For example, in model-based reinforcement learning, where the model is a part of the output, a small mutual information might indicate that the learned model focuses more on the goal of maximizing the cumulative reward rather than solely capturing the transition dynamics. How to learn an effective model beyond merely fitting the transition is the central theme in decision-aware model-based reinforcement learning [22,23,24,25,26,27,28].

As in the supervised learning setting, where various algorithms such as Stochastic Gradient Descent (SGD) [29] and Stochastic Gradient Langevin Dynamics (SGLD) have been studied [30], a promising future direction is to analyze information-theoretic generalization bounds for specific reinforcement learning algorithms such as stochastic policy gradient methods.

In addition, the information-theoretic approach has the potential to unify various concepts related to generalization, such as differential privacy and stability [12,31]. It would be interesting to explore how these notions in reinforcement learning can be leveraged to guarantee generalization.

Analyzing generalization for reinforcement learning is inherently more challenging than in supervised learning [32,33,34]. Therefore, we hope that the information-theoretic approach will provide more insights into understanding the generalization of reinforcement learning.