Information-Theoretic Generalization Bounds for Meta-Learning and Applications

Abstract

Meta-learning, or “learning to learn”, refers to techniques that infer an inductive bias from data corresponding to multiple related tasks with the goal of improving the sample efficiency for new, previously unobserved, tasks. A key performance measure for meta-learning is the meta-generalization gap, that is, the difference between the average loss measured on the meta-training data and on a new, randomly selected task. This paper presents novel information-theoretic upper bounds on the meta-generalization gap. Two broad classes of meta-learning algorithms are considered that use either separate within-task training and test sets, like model agnostic meta-learning (MAML), or joint within-task training and test sets, like reptile. Extending the existing work for conventional learning, an upper bound on the meta-generalization gap is derived for the former class that depends on the mutual information (MI) between the output of the meta-learning algorithm and its input meta-training data. For the latter, the derived bound includes an additional MI between the output of the per-task learning procedure and corresponding data set to capture within-task uncertainty. Tighter bounds are then developed for the two classes via novel individual task MI (ITMI) bounds. Applications of the derived bounds are finally discussed, including a broad class of noisy iterative algorithms for meta-learning.

1. Introduction

As formalized by the “no free lunch theorem”, any effective learning procedure must be based on prior assumptions on the task of interest [1]. These include the selection of a model class and of the hyperparameters of a learning algorithm, such as weight initialization and learning rate. In conventional single-task learning, these assumptions, collectively known as inductive bias, are fixed a priori relying on domain knowledge or validation [1,2,3]. Fixing a suitable inductive bias can significantly reduce the sample complexity of the learning process, and is thus crucial to any learning procedure. The goal of meta-learning is to automatically infer the inductive bias, thereby learning to learn from past experiences via the observation of a number of related tasks, so as to speed up learning a new and unseen task [4,5,6,7,8].

In this work, we consider the meta-learning problem of inferring the hyperparameters of a learning algorithm. The learning algorithm (henceforth, called base-learning algorithm or base-learner) is defined as a stochastic mapping $P_{W|Z^m,u}$ from the input training set $Z^m=(Z_1,\dots ,Z_m)$ of m samples to a model parameter $W\in \mathcal{W}$ for a fixed hyperparameter vector u. The meta-learning algorithm (or meta-learner) infers the hyperparameter vector u, which defines the inductive bias, by observing a finite number of related tasks.

For example, consider the well-studied algorithm of biased regularization for supervised learning [9,10]. Let us denote each data point $Z=(X,Y)$ as a tuple of input features $X\in {\mathbb{R}}^d$ and label $Y\in \mathbb{R}$. The loss function $l:\mathcal{W}\times \mathcal{Z}\to \mathbb{R}$ is given as the quadratic measure $l(w,z)={(⟨w,x⟩-y)}^2$ that quantifies the loss accrued by the inferred model parameter w on a data sample z. Corresponding to each per-task data set $Z^m$, the biased regularization algorithm $P_{W|Z^m,u}$ is a Kronecker delta function centered at the minimizer of the following optimization problem

$$\begin{array}{c}\hfill \frac{1}{m}\sum _{j=1}^{m}l(w,Z_j)+\frac{\lambda }{2}\left|\right|w-{u\left|\right|}^2,\end{array}$$

(1)

which corresponds to an empirical risk minimization problem with a biased regularizer. Here, $\lambda >0$ is a regularization constant that weighs the deviation of the model parameter w from a bias vector u. The bias vector u can be then thought of as a common “mean” among related tasks. In the context of meta-learning, the objective then is to infer the bias vector u by observing data sets from a number of similar related tasks. Different meta-learning algorithms have been developed for this problem [11,12].

In the meta-learning problem under study, we follow the standard setting of Baxter [13] and assume that the learning tasks belong to a task environment, which is defined by a probability distribution $P_T$ on the space of learning tasks $\mathcal{T}$, and per-task data distributions ${\left\{P_{Z|T=\tau }\right\}}_{\tau \in \mathcal{T}}$. The data set $Z^m$ for a task $\tau$ is then generated i.i.d. according to the distribution $P_{Z|T=\tau }$. The meta-learner observes the performance of the base-learner on the meta-training data from a finite number of meta-training tasks, which are sampled independently from the task environment, and infers the hyperparameter U such that it can learn a new task, drawn from the same task environment, from fewer data samples.

The quality of the inferred hyperparameter U is measured by the meta-generalization loss, ${\mathcal{L}}_g\left(U\right)$, which is the average loss incurred on the data set $Z^m\sim P_{Z^m|T}$ of a new, previously unseen task T sampled from the task distribution $P_T$. The notation will be formally introduced in Section 2.2. While the goal of meta-learning is to infer a hyperparameter U that minimizes the meta-generalization loss ${\mathcal{L}}_g\left(U\right)$, this is not computable, since the underlying task and data distributions are unknown. Instead, the meta-learner can evaluate an empirical estimate of the loss, ${\mathcal{L}}_t\left(U\right|Z_{1:N}^m)$, using the meta-training set $Z_{1:N}^m$ of data from N tasks, which is referred to as meta-training loss. The difference between the meta-generalization loss and the meta-training loss is the meta-generalization gap,

$$\begin{array}{c}\hfill \Delta \mathcal{L}\left(U\right|Z_{1:N}^m)={\mathcal{L}}_g\left(U\right)-{\mathcal{L}}_t\left(U\right|Z_{1:N}^m),\end{array}$$

(2)

and measures how well the inferred hyperparameter U generalizes to a new, previously unseen task. In particular, if the meta-generalization gap is small, on average or with high probability, then the performance of the meta-learner on the meta-training set can be taken as a reliable estimate of the meta-generalization loss.

In this paper, we study information-theoretic upper bounds on the average meta-generalization gap ${\mathbb{E}}_{P_{Z_{1:N}^m}{P}_{U|Z_{1:N}^m}}\left[\Delta \mathcal{L}\left(U\right|Z_{1:N}^m)\right]$, where the average is with respect to the meta-training set $Z_{1:N}^m$ and the meta-learner defined by the stochastic kernel $P_{U|Z_{1:N}^m}$. Specifically, we extend the recent line of work initiated by Russo and Zhou [14], and Xu and Raginsky [15], which obtain mutual information (MI)-based bounds on the average generalization gap for conventional learning, to meta-learning. To the best of our knowledge, this is the first work that studies information-theoretic bounds for meta-learning.

The bounds on average meta-generalization gap, studied in this work, are distinct from the other well-known bounds on meta-generalization gap in literature. Broadly speaking, existing bounds on the meta-generalization gap can be grouped into two—high probability, probably approximately correct (PAC) bounds, and high probability PAC-Bayesian bounds. These upper bounds take the general form, ${\mathbb{E}}_{P_{U|Z_{1:N}^m}}\left[\Delta \left(U\right|Z_{1:N}^m)\right]\le ϵ$, that hold with probability at least $1-\delta$, for $\delta \in (0,1)$, over the meta-training set $Z_{1:N}^m$. In contrast, our work focuses on bounding ${\mathbb{E}}_{P_{Z_{1:N}^m}}{\mathbb{E}}_{P_{{U|Z}_{1:N}^m}}\left[\Delta \mathcal{L}\left(U\right|Z_{1:N}^m)\right]$ on average also over the meta-training set. Notable PAC bounds on meta-generalization gap include the bound of Baxter [13] obtained using the framework of Vapnik–Chervonenkis (VC) dimensions; and of Maurer [16], which employs the algorithmic stability [17,18] properties. In contrast, the PAC-Bayesian bounds also incorporate prior beliefs on the base-learner and the meta-learner posteriors via an auxiliary data-independent prior distribution $Q_{W|U}$ and a hyper-prior distribution $Q_U$, respectively. Most notably, PAC-Bayesian bounds include that of Pentina and Lambert [19], the tighter bound of Amit and Meir [20], and most recently, the bounds of Rothfuss et al. [21]. While the high-probability bounds are agnostic to task and data distributions, our information-theoretic bounds depend explicitly on the task and per-task data distributions, on the loss function, and on the meta-training algorithm, in accordance to prior work on information-theoretic generalization bounds.

Another general property inherited from the information-theoretic approach adopted in this paper is that the bounds on the average meta-generalization gap under study are designed to hold for arbitrary base-learners and meta-learners. As such, they generally do not result in tighter bounds as compared to non-information theoretic generalization guarantees obtained for specific meta-learning problems, such as the ridge regression problem with meta-learned bias vector mentioned above [22]. In contrast, the general purpose of the bounds in this paper is to provide insights into the number of tasks, and the number of samples per task required to ensure that the training-based metrics are a good approximation to their population counterparts.

1.1. Main Contributions

The derivation of bounds on average meta-generalization gap differs from conventional learning owing to two levels of uncertainties—environment-level uncertainty and within-task uncertainty. While within-task uncertainty results from observing a finite number m of data samples per task as in conventional learning, environment-level uncertainty results from observing a finite number N of tasks from the task-environment. The relative importance of these two forms of uncertainty depend on the use made by the meta-learner of the meta-training data. In fact, depending on how the meta-training data are used by the meta-learner, we identify two main classes of meta-training algorithms—with separate within-task training and test sets, and joint within-task training and test sets. The former class includes the state-of-the-art meta-learning algorithms, such as model agnostic meta-learning (MAML) [23], that splits the training data corresponding to each task into training and test sets, with the latter reserved for within-task validation. In contrast, the second class of algorithms, such as reptile [24], use the entire per-task data both for training and testing. Our main contributions are as follows.

• In Theorem 1, we show that, for the case with separate within-task training and test sets, the average meta-generalization gap contains only the contribution of environment-level uncertainty. This is captured by a ratio of the mutual information (MI) between the output of the meta-learner U and the meta-training set $Z_{1:N}^m$, and the number of tasks N, as

  $$\begin{array}{cc}\hfill \left|{\mathbb{E}}_{P_{Z_{1:N}^m}{P}_{U|Z_{1:N}^m}}\left[\Delta {\mathcal{L}}^{\mathrm{sep}}\left(U\right|Z_{1:N}^m)\right]\right|& \le \sqrt{\frac{2{\sigma }^2}{N}I(U;Z_{1:N}^m)},\hfill \end{array}$$

  (3)

  where ${\sigma }^2$ is the sub-Gaussianity variance factor of the meta-loss function. This is a direct parallel of the MI-based bounds for single-task learning [25].
• In Theorem 3, we then show that, for the case with joint within-task training and test sets, the bound on the average meta-generalization gap also contains a contribution due to the within-task uncertainty via the ratio of the MI between the output of the base-learner and within-task training data and the per-task data sample size m. Specifically, we have the following bound

  $$\begin{array}{cc}\hfill \left|{\mathbb{E}}_{P_{Z_{1:N}^m}{P}_{U|Z_{1:N}^m}}\left[\Delta {\mathcal{L}}^{\mathrm{joint}}\left(U\right|Z_{1:N}^m)\right]\right|& \le \sqrt{\frac{2{\sigma }^2}{N}I(U;Z_{1:N}^m)}+{\mathbb{E}}_{P_T}\left[\sqrt{\frac{2{\delta }_T^2}{m}I(W;Z^m|T=\tau )}\right],\hfill \end{array}$$

  (4)

  where ${\delta }_T^2$ is the sub-Gaussianity variance factor of the loss function $l(w,z)$ for task T.
• In Theorems 2 and 4, we extend the individual sample MI (ISMI) bound of [26] to obtain novel individual task MI (ITMI)-based bounds on the meta-generalization gap for both separate and within-task training and test sets as

  $$\begin{array}{cc}\hfill \left|{\mathbb{E}}_{P_{Z_{1:N}^m}{P}_{U|Z_{1:N}^m}}\left[\Delta {\mathcal{L}}^{\mathrm{sep}}\left(U\right|Z_{1:N}^m)\right]\right|& \le \frac{1}{N}\sum _{i=1}^N\sqrt{2{\sigma }^{2}I(U;Z_i^m)},\hfill \end{array}$$

  (5)

  and

  $$\begin{array}{cc}\hfill & \left|{\mathbb{E}}_{P_{Z_{1:N}^m}{P}_{U|Z_{1:N}^m}}\left[\Delta {\mathcal{L}}^{\mathrm{joint}}\left(U\right|Z_{1:N}^m)\right]\right|\le \frac{1}{N}\sum _{i=1}^N\sqrt{2{\sigma }^{2}I(U;Z_i^m)}+{\mathbb{E}}_{P_T}\left[\frac{1}{m}\sum _{j=1}^m\sqrt{2{\delta }_T^{2}I(W;Z_j|T=\tau )}\right].\hfill \end{array}$$

  (6)

These bounds can be seen to be tighter than the MI-based bounds in (3) and (4), respectively.

• Finally, we study the applications of the derived bounds to two meta-learning problems. The first is a parameter estimation setup that involves one-shot meta-learning and base-learning procedures, for which a closed form expression for meta-generalization gap can be derived. The second application covers a broad range of noisy iterative meta-learning algorithms and is inspired by the work of Pensia et al. [27] for conventional learning.

1.2. Related Work

For conventional learning, there exists a rich literature on diverse frameworks for deriving upper bounds on the generalization gap, i.e., on the difference between generalization and training losses. Classical bounds from statistical learning theory quantify the generalization gap in terms of measures of complexity of the model class, most notably VC dimension [28] and Radmacher complexity [29]. This approach obtains high-probability, probably approximate correct (PAC) bounds on the generalization gap with respect to the training set. An alternate line of high-probability bounding techniques relies on the notion of algorithmic stability, which measures the sensitivity of the output of a learning algorithm to the replacement of individual samples from the training data set. The pioneering work [30] has been extended to include various notions of algorithmic stability [31,32,33]. As a notable example, a distributional notion of stability in terms of differential privacy, which quantifies the sensitvity of the distribution of algorithm’s output to data set, has been studied in [34,35]. The high-probability PAC–Bayesian bounds rely on change of measure arguments and uses the Kullback–Leibler (KL) divergence between the algorithm and a data-independent prior to quantifying the algorithmic sensitivity [36,37,38].

Following the initial work of Russo and Zou [14], information-theoretic bounds on the average generalization gap for conventional learning have been widely investigated in recent years. Xu and Raginsky [25] showed that the MI between the output of the learning algorithm and its training data set yields an upper bound in expectation on the generalization gap. The bound has been shown to offer computable generalization gaurentees for noisy iterative algorithms, including stochastic gradient Langevin dynamics (SGLD) in [27]. Various refinements of the MI-based bound have since been analyzed to obtain tighter bounds. In particular, the bounds in [39] employ chaining mutual information techniques to tighten the bounds in [25], while the bound in [26] depends on the MI between the output of the algorithm and an individual data sample. The MI between the output of the algorithm and a random subset of the data set appears in the bounds introduced in [40]. The total variation information between the joint distribution of the training data and algorithmic output and the product of marginals was shown in [41] to yield a bound on the generalization gap for any bounded loss function. Subsequent works in [42,43,44] consider other information-theoretic measures, such as maximum leakage and lautum information. Most recently, a conditional mutual information (CMI)-based approach has been proposed in [45] to develop generalization bounds.

1.3. Notation

Throughout this paper, upper case letters, e.g., X, denote random variables and lower case letters, e.g., x, their realizations. We use $\mathcal{P}(·)$ to denote the set of all probability distributions on the argument set or vector space. For a discrete or continuous random variable X taking values in a set or vector space $\mathcal{X},P_X\in \mathcal{P}\left(\mathcal{X}\right)$ denotes its probability distribution, with $P_X\left(x\right)$ being the probability mass or density value at $x\in \mathcal{X}$. We denote as $P_{X^n}$ the n-fold product distribution induced by $P_X$. The conditional distribution of a random variable X given random variable Y is similarly defined as $P_{X|Y}$, with $P_{X|Y}\left(x\right|y)$ representing the probability mass or density at $X=x$ conditioned on the event $Y=y$. We use $\left|\right|·{\left|\right|}_2$ to denote the Euclidean norm of the argument vector, and $I_d$ to denote a d-dimensional identity matrix. We define the Kronecker delta $\delta (x-x_0)=1$ if $x=x_0$ and $\delta (x-x_0)=0$ otherwise.

2. Problem Definition

In this section, we define the problem of interest by introducing the key definitions of generalization gap for conventional, or single-task, learning and for meta-learning.

2.1. Generalization Gap for Single-Task Learning

Consider first the conventional problem of learning a task $\tau \in \mathcal{T}$.

As illustrated in Figure 1, each task $\tau \in \mathcal{T}$ is associated with an underlying unknown data distribution, $P_{Z|T=\tau }\in \mathcal{P}\left(\mathcal{Z}\right)$, defined in a subset or vector space. Henceforth, we use $P_{Z|\tau }$ to denote $P_{Z|T=\tau }$ for notational convenience.

Figure 1. Directed graph representing the variables involved in the definition of generalization gap (11) for single-task learning.

The training procedure, which is referred to as the base-learner, has access to a training data set $Z^m=(Z_1,Z_2,\dots ,Z_m)\sim P_{Z^m|\tau }$ of m independent and identically distributed (i.i.d.) samples drawn from distribution $P_{Z|\tau }$. The base-learner uses this data set to choose a model, or hypothesis, W from the model class $\mathcal{W}$ by using a randomized training procedure defined by a conditional distribution $P_{W|Z^m,u}$ as

$$\begin{array}{c}\hfill W\sim P_{W|Z^m,u}.\end{array}$$

(7)

The conditional distribution $P_{W|Z^m,u}$ defines a stochastic mapping from the training data set $Z^m$ to the model class $\mathcal{W}$. The training procedure (7) is parameterized by a vector $u\in \mathcal{U}$ of hyperparameters, which defines the inductive bias. As an example, the base-learner $P_{W|Z^m,u}$ may follow stochastic gradient descent (SGD) updates with hyperparameters u, including the learning rate and the initialization point.

The performance of a parameter vector $w\in \mathcal{W}$ on a data sample $z\in \mathcal{Z}$ is measured by a loss function $l:\mathcal{W}\times \mathcal{Z}\to {\mathbb{R}}_{+}$. The generalization loss for a model parameter vector $w\in \mathcal{W}$ is the average

$$\begin{array}{cc}\hfill & L_g\left(w\right|\tau )={\mathbb{E}}_{P_{Z|\tau }}\left[l(w,Z)\right],\hfill \end{array}$$

(8)

over a test example Z independently drawn from the data distribution $P_{Z|\tau }$. The subscript g is used to distinguish the generalization loss from the training loss defined below. The generalization loss cannot be computed by the learner, given that the data distribution $P_{Z|\tau }$ is unknown. Instead, the learner can evaluate the training loss on the data set $Z^m$, which is defined as the empirical average

$$\begin{array}{c}\hfill L_t\left(w\right|Z^m)=\frac{1}{m}\sum _{i=1}^{m}l(w,Z_i).\end{array}$$

(9)

The subscript t specifies that the loss is the empirical training loss.

The difference between generalization loss (8) and training loss (9) is known as generalization gap,

$$\begin{array}{c}\hfill \Delta L\left(w\right|Z^m,\tau )=L_g\left(w\right|\tau )-L_t\left(w\right|Z^m),\end{array}$$

(10)

and is a key metric that quantifies the level of uncertainty (This type of uncertainty is known as epistemic.) at the learner regarding the data distribution $P_{Z|\tau }$. The average generalization gap for the data distribution $P_{Z|\tau }$ and base-learner $P_{W|Z^m,u}$ is defined as

$$\begin{array}{c}\hfill {\mathbb{E}}_{P_{Z^m,W|\tau ,u}}\left[\Delta L\left(W\right|Z^m,\tau )\right],\end{array}$$

(11)

where the expectation is taken with respect to the joint distribution $P_{Z^m,W|\tau ,u}=P_{Z^m|\tau }{P}_{W|Z^m,u}$. A summary of the variables involved in the Definition of the generalization gap (11) can be found in Figure 1.

Intuitively, if the generalization gap is small, on average or with high probability, then the base-learner can take the performance (9) on the training set $Z^m$ as a reliable measure of the generalization loss (8) of the trained model W. Furthermore, data-dependent bounds on the generalization gap can be used as regularization terms to avoid overfitting, yielding generalized Bayesian inference problems [46,47].

2.2. Generalization Gap for Meta-Learning

As discussed, in single-task learning, the inductive bias u, defining the hyperparameters of the training procedure, must be selected a priori, i.e., without having access to task-specific data. The inductive bias determines the training data set size m needed to ensure a small generalization loss (8), since, generally speaking, richer models require more data to be trained [1]. The sample complexity can be generally reduced if one selects a suitable inductive bias based on prior information. Such prior information is typically obtained from domain knowledge on the problem under study. In contrast, meta-learning aims at automatically inferring an effective inductive bias based on data from related tasks.

To elaborate, we follow the setting of [13], in which a meta-learner observes data from a number of tasks, known as meta-training tasks, from the same task environment. A task environment is defined by a task distribution $P_T\in \mathcal{P}\left(\mathcal{T}\right)$, supported on the space $\mathcal{T}$ of tasks, and by a per-task data distribution $P_{Z|\tau }$ for each task $\tau \in \mathcal{T}$. Using the meta-training data drawn from a randomly selected subset of tasks, the meta-learner infers a hyperparameter vector $u\in \mathcal{U}$ defining the inductive bias. This is done with the goal of ensuring that, using hyperparameter u, the base-learner $P_{W|Z^m,u}$ can efficiently learn on a new task, referred to as meta-test task, drawn independently from the same task distribution $P_T$.

To elaborate, the meta-training data consist of N data sets $Z_{1:N}^m=(Z_1^m,\dots ,Z_N^m)$. Each ith data set is generated independently by first drawing a task $T_i\sim P_T$ from the task environment and then a task-specific training data set $Z_i^m\sim P_{Z^m|T_i}$. The meta-learner uses the meta-training data set $Z_{1:N}^m$ to infer a hyperparameter vector $u\in$. To this end, we consider a randomized meta-learner

$$\begin{array}{c}\hfill U\sim P_{U|Z_{1:N}^m},\end{array}$$

(12)

where $P_{U|Z_{1:N}^m}$ is a stochastic mapping from the meta-training set $Z_{1:N}^m$ to the space $\mathcal{U}$ of hyperparameters. We distinguish two different formulations of meta-learning that are often considered in the literature. In the first, the per-task data set $Z^m$ is split into training, or support, and test, or query subsets [23,48]; while, in the second, the entire data set $Z^m$ is used for both within-task training and testing [13,19,20].

2.2.1. Separate Within-Task Training and Test Sets

As seen in Figure 2, in this first approach to meta-learning, each meta-training sub data set $Z_i^m$ is split into a training set and a test set as $Z_i^m=(Z_i^{{\mathrm{m}}_{\mathrm{tr}}},Z_i^{{\mathrm{m}}_{\mathrm{te}}})$, where $Z_i^{{\mathrm{m}}_{\mathrm{tr}}}$ contains $m_{\mathrm{tr}}$ i.i.d. training examples and $Z_i^{{\mathrm{m}}_{\mathrm{te}}}$ contains $m_{\mathrm{te}}$ i.i.d. test examples, with $m=m_{\mathrm{tr}}+m_{\mathrm{te}}$. The within-task base-learner $P_{W|Z_i^{{\mathrm{m}}_{\mathrm{tr}}},u}\in \mathcal{P}\left(\mathcal{W}\right)$ maps the per-task training subset $Z_i^{{\mathrm{m}}_{\mathrm{tr}}}$ to random model parameter $W_i\sim P_{W|Z_i^{{\mathrm{m}}_{\mathrm{tr}}},u}$ for a given hyperparameter $U=u$. The test subset is used to evaluate the empirical training loss of a model w for task $T_i$ as

$$\begin{array}{c}\hfill L_t\left(w\right|Z_i^{{\mathrm{m}}_{\mathrm{te}}})=\frac{1}{m_{\mathrm{te}}}\sum _{j=1}^{m_{\mathrm{te}}}l(w,Z_{i,j}^{m_{\mathrm{te}}}),\end{array}$$

(13)

where $Z_{i,j}^{m_{\mathrm{te}}}$ denote the jth example of the test subset $Z_i^{m_{\mathrm{te}}}$. Furthermore, the overall empirical meta-training loss for a hyperparameter u is computed by summing up all meta-training tasks as

$$\begin{array}{c}\hfill {\mathcal{L}}_t^{\mathrm{sep}}\left(u\right|Z_{1:N}^m)=\frac{1}{N}\sum _{i=1}^N{L}_t^{\mathrm{sep}}\left(u\right|Z_i^m),\end{array}$$

(14)

where

$$\begin{array}{c}\hfill L_t^{\mathrm{sep}}\left(u\right|Z^m)={\mathbb{E}}_{P_{W|Z^{{\mathrm{m}}_{\mathrm{tr}}},u}}\left[L_t\left(W\right|Z^{{\mathrm{m}}_{\mathrm{te}}})\right]\end{array}$$

(15)

is the average per-task training loss over the base-learner.

Figure 2. Directed graph representing the variables involved in the definition of meta-generalization gap (18) for separate within-task training and testing sets.

We emphasize that the meta-training loss (14) can be computed by the meta-learner and used as a criterion to select the meta-learning procedure (12), since it is obtained from the meta-training data $Z_{1:N}^m$. We also note that the rationale of splitting training and test sets is that the average training loss $L_t^{\mathrm{sep}}\left(u\right|Z_i^m)$ is an unbiased estimate of the corresponding average generalization loss ${\mathbb{E}}_{P_{W|Z_i^{{\mathrm{m}}_{\mathrm{tr}}},u}}\left[L_g\left(W\right|T_i)\right]$.

The true goal of the meta-learner is to minimize the meta-generalization loss,

$$\begin{array}{c}\hfill {\mathcal{L}}_g^{\mathrm{sep}}\left(u\right)={\mathbb{E}}_{P_{T,Z^{{\mathrm{m}}_{\mathrm{tr}}}}}{\mathbb{E}}_{P_{W|Z^{{\mathrm{m}}_{\mathrm{tr}}},u}}\left[L_g\left(W\right|T)\right],\end{array}$$

(16)

where $P_{T,Z^{{\mathrm{m}}_{\mathrm{tr}}}}=P_T{P}_{Z^{{\mathrm{m}}_{\mathrm{tr}}}|T}$ and $L_g\left(W\right|T)$ are as defined in (8). Unlike the meta-training loss (14), the meta-generalization loss is evaluated on a new, meta-test task T and on the corresponding training data $Z^{{\mathrm{m}}_{\mathrm{tr}}}$. We distinguish the meta-generalization loss and meta-training loss by the subscripts g and t, respectively in (16) and (14). The difference between the meta-generalization loss (16) and the meta-training loss (14), known as the meta-generalization gap, is defined as

$$\begin{array}{c}\hfill \Delta {\mathcal{L}}^{\mathrm{sep}}\left(u\right|Z_{1:N}^m)={\mathcal{L}}_g^{\mathrm{sep}}\left(u\right)-{\mathcal{L}}_t^{\mathrm{sep}}\left(u\right|Z_{1:N}^m).\end{array}$$

(17)

The quantity of interest to us is the average meta-generalization gap, defined as

$$\begin{array}{c}\hfill {\mathbb{E}}_{P_{Z_{1:N}^m,U}}\left[\Delta {\mathcal{L}}^{\mathrm{sep}}\left(U\right|Z_{1:N}^m)\right],\end{array}$$

(18)

where the expectation is with respect to the joint distribution $P_{Z_{1:N}^m,U}=P_{Z_{1:N}^m}{P}_{U|Z_{1:N}^m}$, of the meta-training set $Z_{1:N}^m$ and of the hyperparameter U. Note that $P_{Z_{1:N}^m}$ is the marginal of the joint distribution ${\prod }_{i=1}^N{P}_{T=T_i}{P}_{Z^M|T=T_i}$.

Intuitively, if the meta-generalization gap is small, on average or with high probability, the meta learner can take the performance (14) on the meta-training data as a reliable measure of the accuracy of the inferred hyperparameter vector in terms of the meta-generalization loss (16). Furthermore, data-dependant bounds on the meta-generalization gap can be used as regularization terms to avoid meta-overfitting. Meta-overfitting occurs when the meta-trained hyperparameter yields a small meta-training loss but a large meta-test loss, due to an excessive dependence on the meta-training set [13].

2.2.2. Joint Within-Task Training and Test Sets

In the second formulation of meta-learning, as illustrated in Figure 3, the entire data set $Z_i^m$ is used for within-task training and testing. Accordingly, the meta-learner computes the meta-training loss

$$\begin{array}{c}\hfill {\mathcal{L}}_t^{\mathrm{joint}}\left(u\right|Z_{1:N}^m)=\frac{1}{N}\sum _{i=1}^N{L}_t^{\mathrm{joint}}\left(u\right|Z_i^m),\end{array}$$

(19)

where

$$\begin{array}{c}\hfill L_t^{\mathrm{joint}}\left(u\right|Z^m)={\mathbb{E}}_{P_{W|Z^m,u}}\left[L_t\left(W\right|Z^m)\right]\end{array}$$

(20)

is the average per-task training loss. Note here that in evaluating the meta-training loss in (19), the data set $Z_i^m$ is used to infer model parameters W and to evaluate the per-task training loss. The expectation in (20) is taken over the output of the base-learner W given the hyperparameter vector u. As discussed, the meta-generalization loss for hyperparameter $u\in \mathcal{U}$ is computed by randomly selecting a novel task $T\sim P_T$ as

$$\begin{array}{c}\hfill {\mathcal{L}}_g^{\mathrm{joint}}\left(u\right)={\mathbb{E}}_{P_{T,Z^m}}{\mathbb{E}}_{P_{W|Z^m,u}}\left[L_g\left(W\right|T)\right],\end{array}$$

(21)

where $P_{T,Z^m}=P_T{P}_{Z^m|T}$ and $L_g\left(W\right|T)$ is as defined in (8). In a manner similar to (17), the meta-generalization gap for a task distribution $P_T$, data distribution $P_{Z^m|T}$, meta-learning algorithm $P_{U|Z_{1:N}^m}$, and base-learner $P_{W|Z^m,U}$ is defined as

$$\begin{array}{cc}\hfill & \Delta {\mathcal{L}}^{\mathrm{joint}}\left(u\right|Z_{1:N}^m)={\mathcal{L}}_g^{\mathrm{joint}}\left(u\right)-{\mathcal{L}}_t^{\mathrm{joint}}\left(u\right|Z_{1:N}^m).\hfill \end{array}$$

(22)

Figure 3. Directed graph representing the variables involved in the definition of meta-generalization gap (22) for joint within-task training and testing sets.

The average meta-generalization gap is then given as ${\mathbb{E}}_{P_{Z_{1:N}^m,U}}\left[\Delta {\mathcal{L}}^{\mathrm{joint}}\left(U\right|Z_{1:N}^m)\right]$, where the expectation is taken over all meta-training sets and over the output of the meta-learner.

3. Information-Theoretic Generalization Bounds for Single-Task Learning

In this section, we review two information-theoretic bounds on the generalization gap (11) for conventional learning derived in [25,26]. The material covered in this section provides the necessary background for the analysis of the meta-generalization gap to be studied in the rest of the paper. Throughout this section, we fix a task $\tau \in \mathcal{T}$. Since the generalization and meta-generalization gaps measure the deviation of empirical-mean random variables representing training and meta-training losses from reference values, we will make use of tools and definitions from large-deviation theory (see, e.g., [49]). We discuss the key essential definitions below.

3.1. Preliminaries

To start, the cumulant generating function (CGF) of a random variable $X\sim P_X\in \mathcal{P}\left(\mathcal{X}\right)$ is defined as ${\Lambda }_X\left(\lambda \right)=log{\mathbb{E}}_{P_X}\left[e^{\lambda (X-{\mathbb{E}}_{P_X}\left[X\right])}\right]$. If it is well-defined, the CGF ${\Lambda }_X\left(\lambda \right)$ is convex and it satisfies the equalities ${\Lambda }_X\left(0\right)={\Lambda }_X^{\prime }\left(0\right)=0$. A random variable X with finite mean, i.e., with ${\mathbb{E}}_{P_X}\left[X\right]<\infty$, is said to ${\sigma }^2$-sub-Gaussian if its CGF is bounded as

$$\begin{array}{c}\hfill {\Lambda }_X\left(\lambda \right)\le \frac{{\lambda }^2{\sigma }^2}{2},\mathrm{for} \mathrm{all}\lambda \in \mathbb{R}.\end{array}$$

(23)

As a special case, if X is bounded in the interval $[a,b]$, i.e., if the inequality $0<a\le X\le b<\infty$ holds for some constants a and b, then X is ${(b-a)}^2/4$-sub-Gaussian.

3.2. Mutual Information (MI) Bound

We first present the mutual information (MI)-based upper bound obtained in [25]. Key to this result is the following Assumption.

Assumption 1.

The loss function $l(w,Z)$ is ${\delta }_{\tau }^2$-sub-Gaussian under $Z\sim P_{Z|\tau }$ for all model parameters $w\in \mathcal{W}$.

In particular, if the loss function is bounded, i.e., if the inequalities $-\infty <a\le l(w,z)\le b<\infty$ hold for all for $w\in \mathcal{W}$ and $z\in \mathcal{Z}$, Assumption 1 is satisfied with ${\delta }_{\tau }^2={(b-a)}^2/4$. The main result is as follows.

Lemma 1

([25]). Under Assumption 1, the following bound on the generalization gap holds for any base-learner $W\sim P_{W|Z^m,u}$

$$\begin{array}{c}\hfill |{\mathbb{E}}_{P_{Z^m,W|\tau ,u}}\left[\Delta L\left(W\right|Z^m,\tau )\right]|\le \sqrt{\frac{2{\sigma }^2}{m}I(W;Z^m)}.\end{array}$$

(24)

The proof of Lemma 1 is based on a decoupling estimate Lemma, which is reported for completeness in Lemma A1. We also note that the result in Lemma 1 can be extended to account for loss function $l(w,Z)$ with bounded CGF [14].

The bound (24) on the generalization gap is in terms of the mutual information $I(W;Z^m)$, which quantifies the overall dependence between the base-learner output W and the input training data set $Z^m$. The mutual information in (24) is hence a measure of the sensitivity of the base-learner output to the data set. Using the terminology in [25], if $I(W;Z^m)\le ϵ$, the base-learner $P_{W|Z^m,u}$ is said to be $(ϵ,P_{Z|\tau })$-MI stable, in which case the bound in (24) evaluates to $\sqrt{2{\delta }_{\tau }^2ϵ/m}$. The relationship between generalization and stability of a training algorithm is well-established [1], and the result (24) amounts to a formulation of this link in information-theoretic terms.

The traditional notion of algorithmic stability measures how much the base-learner output changes with the replacement of an individual training sample [30,50]. In the next section, we review the bound in [26] that translates this per-sample stability concept within an information-theoretic framework.

3.3. Individual Sample MI (ISMI) Bound

The MI-based bound in Lemma 1 has the disadvantage of being vacuous, i.e., $I(W;Z^m)=\infty$, for deterministic base-learning algorithms $P_{W|Z^m,u}$ defined on continuous parameter space $\mathcal{W}$. An individual sample MI (ISMI)-based bound that address this shortcoming was introduced in [26]. The ISMI bound borrows the standard algorithmic stability notion of sensitivity of the base-learner output to the replacement of any individual training sample [17,18]. Accordingly, the resulting bound is in terms of the MI between the trained parameter W and each data point $Z_i$ of the training data set $Z^m$. The bound, summarized in Lemma 2, applies under the following assumption.

Assumption 2.

The loss function $l(w,z)$ satisfies either of the following two conditions:

(a) 

Assumption 1, or

(b) 

$l(W,Z)$ is a ${\delta }_{\tau }^2$-sub-Gaussian random variable when $(W,Z)\sim P_{W|u,\tau }{P}_{Z|\tau }$, where $P_{W|u,\tau }\in \mathcal{P}\left(\mathcal{W}\right)$ is the marginal of the joint distribution $P_{W|Z^m,u}{P}_{Z^m|\tau }$.

We note that, in general, Assumption 1 does not imply Assumption 2$\left(b\right)$ (see ([40], Appendix C)), and vice versa (see [26]). There are, however, loss functions $l(w,z)$ and relevant distributions for which both the assumptions hold, including the case of loss functions $l(·,·)$ which takes values in a bounded interval $[a,b]$.

Lemma 2

([26]). Under Assumption 2, the following bound on the average generalization gap holds for any base-learner $P_{W|Z^m,u}$

$$\begin{array}{c}\hfill |{\mathbb{E}}_{P_{Z^m,W|\tau ,u}}\left[\Delta L\left(W\right|Z^m,\tau )\right]|\le \frac{1}{m}\sum _{i=1}^m\sqrt{2{\sigma }^{2}I(W;Z_i)}.\end{array}$$

(25)

For a loss function satisfying Assumption 1, the ISMI bound (25) is tighter than (24), i.e.,

$$\begin{array}{c}\hfill \frac{1}{m}\sum _{i=1}^m\sqrt{2{\delta }_{\tau }^{2}I(W;Z_i)}\le \sqrt{\frac{2{\delta }_{\tau }^2}{m}I(W;Z^m)}.\end{array}$$

(26)

The inequality in (26) follows from the chain rule of mutual information and Jensen’s inequality [26].

4. Information-Theoretic Generalization Bounds for Meta-Learning

In this section, we first derive novel MI-based bounds on the meta-generalization gap with separate within-task training and test sets, as introduced in Section 4.1, and then we consider joint within-task training and test sets, as described in Section 4.2.

4.1. Bounds on Meta-Generalization Gap with Separate Within-Task Training and Test Sets

In this section, we present two novel MI-based bounds on the meta-generalization gap (18) for the setup with separate within-task training and testing sets. The first is an MI-based bound, which is akin to Lemma 1, and the second is an individual task MI (ITMI) bound, which resembles Lemma 2 for conventional learning.

4.1.1. MI-Based Bound

In order to derive the MI-based bound, we make the following assumption on $L_t^{\mathrm{sep}}\left(u\right|Z^m)$ in (15). Throughout, we use $P_{Z^m}$ to denote the marginal of the joint distribution $P_{T,Z^m}=P_T{P}_{Z^m|T}$.

Assumption 3.

For all $u\in \mathcal{U}$, the average per-task training loss $L_t^{\mathrm{sep}}\left(u\right|Z^m)$ is ${\sigma }^2$-sub-Gaussian under $Z^m\sim P_{Z^m}$.

Distinct from the assumptions in Section 3 on loss function $l(w,z)$, we note that Assumption 3 is on the average per-task training loss $L_t^{\mathrm{sep}}\left(u\right|Z^m)$. This is because the loss function $l(w,z)$ satisfying Assumption 1 do not in general guarantee the sub-Gaussianity of $L_t^{\mathrm{sep}}\left(u\right|Z^m)$ with respect to $Z^m\sim P_{Z^m}$. However, if the loss function is bounded, Assumption 3 can be easily verified to hold, as given in the following lemma.

Lemma 3.

If the loss function $l(·,·)$ is $[a,b]-$ bounded, then $L_t^{\mathrm{sep}}(·|Z^m)$ is also $[a,b]$ bounded for all $Z^m\in {\mathcal{Z}}^m$. Consequently, $L_t^{\mathrm{sep}}\left(u\right|Z^m)$ is ${(b-a)}^2/4$-sub-Gaussian under $Z^m\sim P_{Z^m}$ for all $u\in \mathcal{U}$.

Under Assumption 3, the following theorem presents an upper bound on the meta-generalization gap (18).

Theorem 1.

Let Assumption 3 hold for the base-learner $P_{W|Z^{{\mathrm{m}}_{\mathrm{tr}}},u}$. Then, for any meta learner $P_{U|Z_{1:N}^m}$ such that the inequality $I(U;Z_{1:N}^m)<\infty$ holds, we have the following bound on the average meta-generalization gap

$$\begin{array}{cc}\hfill \left|{\mathbb{E}}_{P_{Z_{1:N}^m,U}}\left[\Delta {\mathcal{L}}^{\mathrm{sep}}\left(U\right|Z_{1:N}^m)\right]\right|& \le \sqrt{\frac{2{\sigma }^2}{N}I(U;Z_{1:N}^m)}.\hfill \end{array}$$

(27)

Proof. 

See Appendix B. □

The technical lemmas required for the proof of Theorem 1 and the theorems that follow are included in Appendix A.

In order to prove Theorem 1, one needs to overcome an additional challenge as compared to the derivation of bounds for learning reviewed in Section 3. In fact, the meta-generalization gap is caused by two distinct sources of uncertainty: $\left(a\right)$ environment-level uncertainty due to a finite number N of observed tasks, and $\left(b\right)$ within-task uncertainty resulting from the finite number m of per-task data samples. Our proof approach involves applying the single-task MI-based bound in Lemma 1 to bound the effect of both sources of uncertainties.

Towards this, we start by introducing the average training loss for the randomly selected meta-test task as

$$\begin{array}{cc}\hfill {\mathcal{L}}_{g,t}^{\mathrm{sep}}\left(u\right)& ={\mathbb{E}}_{P_{T,Z^m}}\left[L_t^{\mathrm{sep}}\left(u\right|Z^m)\right].\hfill \end{array}$$

(28)

The subscript $g,t$ denotes that the loss is generalization (g) with expectation over $P_{T,Z^m}$ at the environment level, and training (t) at the task level with $L_t^{\mathrm{sep}}\left(u\right|Z^m)$. Note that this differs from the meta-test loss ${\mathcal{L}}_g^{\mathrm{sep}}\left(u\right)$ in (16) in that the per-task loss is evaluated in (28) on the training set. With this definition, the meta-generalization gap can be decomposed as

$$\begin{array}{c}{\mathbb{E}}_{P_{Z_{1:N}^m,U}}\left[\Delta {\mathcal{L}}^{\mathrm{sep}}\left(U\right|Z_{1:N}^m)\right]\hfill \\ ={\mathbb{E}}_{P_{Z_{1:N}^m,U}}\left[({\mathcal{L}}_g^{\mathrm{sep}}\left(U\right)-{\mathcal{L}}_{g,t}^{\mathrm{sep}}\left(U\right))+({\mathcal{L}}_{g,t}^{\mathrm{sep}}\left(U\right)-{\mathcal{L}}_t^{\mathrm{sep}}\left(U\right|Z_{1:N}^m))\right].\hfill \end{array}$$

(29)

In (29), the second difference ${\mathcal{L}}_{g,t}^{\mathrm{sep}}\left(U\right)-{\mathcal{L}}_t^{\mathrm{sep}}\left(U\right|Z_{1:N}^m)$ corresponds to the environment-level uncertainty and arises from the observation of a finite number N of tasks. In fact, as N increases, the meta-training loss ${\mathcal{L}}_t^{\mathrm{sep}}\left(u\right|Z_{1:N}^m)$ almost surely tends to ${\mathcal{L}}_{g,t}^{\mathrm{sep}}\left(u\right)$ by the law of large numbers. However, the average ${\mathbb{E}}_{P_{Z_{1:N}^m,U}}\left[{\mathcal{L}}_{g,t}^{\mathrm{sep}}\left(U\right)-{\mathcal{L}}_t^{\mathrm{sep}}\left(U\right|Z_{1:N}^m)\right]$ is not equal to zero in general for finite values of N. The within-task generalization gap is instead measured by the difference ${\mathcal{L}}_g^{\mathrm{sep}}\left(u\right)-{\mathcal{L}}_{g,t}^{\mathrm{sep}}\left(u\right)$. In the setup under study with separate within-task training and test sets, this term equals zero, since, as we discussed, the average empirical loss $L_t^{\mathrm{sep}}\left(u\right|Z_i^m)$ is an unbiased estimate of the corresponding average test loss ${\mathbb{E}}_{P_{W|Z_i^{{\mathrm{m}}_{\mathrm{tr}}},u}}\left[L_g\left(W\right|T_i)\right]$ (cf. (28)). This is no longer true for joint within-task training and test sets, as we discuss in Section 4.2.

The decomposition approach adopted here follows the main steps of the bounding techniques introduced in ([16], Equation (6)). In contrast, the PAC-Bayesian bounds in [20,21] rely on a different decomposition of the meta-generalization gap. The environment and within-task generalization gaps are then separately bounded in high probability, and are combined via union bound to obtain the required PAC-Bayesian bounds.

The bound (27) relates the meta-generalization gap to the information-theoretic stability of the meta-training procedure. As first introduced here, this stability is measured by the MI $I(U;Z_{1:N}^m)$ between the hyperparameter U and the meta-training data set $Z_{1:N}^m$, in a manner similar to the MI-based bounds in Lemma 1 for conventional learning. Importantly, as we will discuss in Section 4.2, this direct parallel between learning and meta-learning no longer applies with joint within-task training and test data sets.

4.1.2. ITMI Bound

We now present the ITMI bound, which holds under the following assumption.

Assumption 4.

Either of the following assumptions on the average per-task training loss, $L_t^{\mathrm{sep}}\left(u\right|Z^m)$ holds:

(a) 

$L_t^{\mathrm{sep}}\left(u\right|Z^m)$ satisfies Assumption 3, or

(b) 

$L_t^{\mathrm{sep}}\left(U\right|Z^m)$ is ${\sigma }^2$-sub-Gaussian under $(U,Z^m)\sim P_U{P}_{Z^m}$, where $P_U$ is the marginal of the joint distribution $P_{Z_{1:N}^m,U}$ and $P_{Z^m}$ is the marginal of the joint distribution $P_{T,Z^m}$.

Assumption 4 can be seen to be implied by the sufficient conditions in Lemma 3.

Theorem 2.

Let Assumption 4 hold for the base-learner $P_{W|Z^{{\mathrm{m}}_{\mathrm{tr}}},U}$. Then, for any meta learner $P_{U|Z_{1:N}^m}$, the following bound on the meta-generalization gap (18) holds

$$\begin{array}{cc}\hfill \left|{\mathbb{E}}_{P_{Z_{1:N}^m,U}}\left[\Delta {\mathcal{L}}^{\mathrm{sep}}\left(U\right|Z_{1:N}^m)\right]\right|& \le \frac{1}{N}\sum _{i=1}^N\sqrt{2{\sigma }^{2}I(U;Z_i^m)}.\hfill \end{array}$$

(30)

where the MI $I(U;Z_i^m)$ is computed with respect to the joint distribution $P_{Z_i^m,U}$ obtained by marginalizing the probability distribution $P_{Z_{1:N}^m,U}$.

Proof. 

See Appendix B. □

As can be seen from (30), the ITMI bound on the meta-generalization gap is in terms of the MI $I(U;Z_i^m)$ between the output U of the meta learner and each per-task data set $Z_i^m$. This, in turn, quantifies the sensitivity of the meta learner output to the replacement of a single per-task data set. Moreover, under Assumption 3, the ITMI bound (30) yields a tighter bound than the MI-based bound (27). This can be seen from the following sequence of relations

$$\sqrt{\frac{1}{N}I(U;Z_{1:N}^m)}=\sqrt{\frac{1}{N}\sum _{i=1}^{N}I(U;Z_i^m|Z_{(i-1)}^m)}$$

(31a)

$$\stackrel{\left(a\right)}{\ge }\sqrt{\frac{1}{N}\sum _{i=1}^{N}I(U;Z_i^m)}$$

(31b)

$$\stackrel{\left(b\right)}{\ge }\frac{1}{N}\sum _{i=1}^N\sqrt{I(U;Z_i^m)},$$

(31c)

where $Z_{(i-1)}^m=(Z_1^m,\dots ,Z_{i-1}^m)$; $\left(a\right)$ follows, since $Z_i^m$ is independent of $Z_{(i-1)}^m$; and $\left(b\right)$ follows from Jensen’s inequality.

4.2. Bounds on Generalization Gap with Joint Within-Task Training and Test Sets

We now derive MI and ITMI-based bounds on the meta-generalization gap in (22) for the case with joint within-task training and test sets. As we will see, the key difference with respect to the case with separate within-task training and test sets is that the uncertainty due to finite number of per-task samples, measured by the second term in the decomposition (29), contributes in a non-negligible way to the meta-generalization gap. Since there is no split into separate within-task training and test sets, the average training loss with respect to the learning algorithm is given by $L_t^{\mathrm{joint}}\left(u\right|Z^m)$ in (20).

4.2.1. MI-Based Bound

In order to derive the MI-based bound, we make the following assumptions.

Assumption 5.

We consider the following assumptions.

(a) 

For each task $\tau \in \mathcal{T}$, the loss function $l(w,Z)$ satisifies Assumption 1, and

(b) 

The average per-task training loss $L_t^{\mathrm{joint}}\left(u\right|Z^m)$ in (20) is ${\sigma }^2$-sub-Gaussian for all $u\in \mathcal{U}$ when $Z^m\sim P_{Z^m}$.

An easily verifiable sufficient condition for the above assumption to hold is the boundedness of loss function $l(w,z)$, which follows in a manner similar to Lemma 3.

Theorem 3.

Let Assumption 5 hold for a base-learner $W\sim P_{W|Z^m,U}$. Then, for any meta learner $P_{U|Z_{1:N}^m}$, we have the following bound on the meta-generalization gap (22)

$$\begin{array}{cc}\hfill \left|{\mathbb{E}}_{P_{Z_{1:N}^m,U}}\left[\Delta {\mathcal{L}}^{\mathrm{joint}}\left(U\right|Z_{1:N}^m)\right]\right|& \le \sqrt{\frac{2{\sigma }^2}{N}I(U;Z_{1:N}^m)}+{\mathbb{E}}_{P_T}\left[\sqrt{\frac{2{\delta }_T^2}{m}I(W;Z^m|T=\tau )}\right].\hfill \end{array}$$

(32)

where the MI $I(W;Z^m|T=\tau )$ is evaluated with respect to the distribution $P_{Z^m,W|T=\tau }$ obtained by marginalizing the joint distribution $P_{W|Z^m,U}{P}_{Z_{1:N}^m,U}{P}_{Z^m|T=\tau }$.

Proof. 

See Appendix C. □

With joint within-task training and test sets, the bound (32) on the meta-generalization gap contains the contributions of two mutual informations. The first, $I(U;Z_{1:N}^m)$, quantifies the sensitivity of the meta learner output U to the meta-training data set $Z_{1:N}^m$. This term also appeared in the bound (27) with separate within-task training and test sets. Decomposing the meta-generalization gap in a manner analogous to (29), it corresponds to a bound on the average of the second difference. The second contribution, $I(W;Z^m|T=\tau )$, quantifies the sensitivity of the output of the base-learner $P_{W|Z^m,U}$ to the data set $Z^m$ of the meta-test task T, when the hyperparameter is randomly selected by the meta-learner $P_{U|Z_{1:N}^m}$ using the meta-training set $Z_{1:N}^m$. This second term is in line with the single-task generalization gap bounds (24), and it bounds the corresponding first difference in the decomposition (29).

We finally note that the dependence of the bound in (32) on the number of tasks N and per-task samples m is of the order $1/\sqrt{N}+1/\sqrt{m}$. Meta-generalization bounds with similar dependence have been derived in [20] using PAC-Bayesian arguments. The bounds on excess risk for representation learning also follow a similar order of dependence on N and m (c.f [51], [Thm. 2]).

4.2.2. ITMI Bound on (22)

For deriving the ITMI bound on the meta-generalization gap (22), we assume the following.

Assumption 6.

Either of the following assumptions hold:

(a) 

Assumption 5 holds, or

(b) 

For each task $\tau \in \mathcal{T}$, the loss function $l(W,Z)$ is ${\delta }_{\tau }^2$-sub-Gaussian when $(W,Z)\sim P_{W|\tau }{P}_{Z|\tau }$, where $P_{W|\tau }$ is the marginal of the joint distribution $P_{W|Z^m,U}{P}_{Z_{1:N}^m,U}{P}_{Z^m|\tau }$. The average per-task training loss $L_t^{\mathrm{joint}}\left(U\right|Z^m)$ is ${\sigma }^2$-sub-Gaussian when $(U,Z^m)\sim P_U{P}_{Z^m}$.

As in Section 4.1.2, Assumption 6 can be seen to be implied by the sufficient conditions in Lemma 3.

Theorem 4.

Under Assumption 6, for any meta learner $P_{U|Z_{1:N}^m}$, the following bound holds on the average meta-generalization gap

$$\begin{array}{cc}\hfill & \left|{\mathbb{E}}_{P_{Z_{1:N}^m,U}}\left[\Delta {\mathcal{L}}^{\mathrm{joint}}\left(U\right|Z_{1:N}^m)\right]\right|\le \frac{1}{N}\sum _{i=1}^N\sqrt{2{\sigma }^{2}I(U;Z_i^m)}+{\mathbb{E}}_{P_T}\left[\frac{1}{m}\sum _{j=1}^m\sqrt{2{\delta }_T^{2}I(W;Z_j|T=\tau )}\right],\hfill \end{array}$$

(33)

where the MI $I(U;Z_i^m)$ is evaluated with respect to $P_{Z_i^m,U}$ obtained by marginalizing $P_{Z_{1:N}^m,U}$, and the MI $I(W;Z_j|T=\tau )$ is with respect to $P_{Z_j,W|T=\tau }$ obtained by marginalizing $P_{Z^m,W|T=\tau }$.

Proof. 

See Appendix C. □

Similar to the bound in (32), the bounds on meta-generalization gap in (33) are in terms of two types of mutual information, the first describing the sensitivity of the meta-learner and the second the sensitivity of the base-learner. Specifically, the MI $I(U;Z_i^m)$ quantifies the sensitivity of the output of the meta learner to per-task data set $Z_i^m$, and the MI $I(W;Z_j|T=\tau )$ measures the sensitivity of the output of the base-learner, $P_{W|Z^m,U}$ to each data sample $Z_i$ within the training set $Z^m$ of the meta-test task T. Moreover, it can be shown, in a manner similar to (31c), that, under Assumption 5, the ITMI bound in (33) is tighter than the MI bound in (32).

4.3. Discussion on Bounds

The bounds on the average meta-generalization gap obtained in this section generalize the bounds for conventional single-task learning in Section 3. To see this, consider the task distribution $P_T=\delta (T-\tau )$ to be centered at some task $\tau \in \mathcal{T}$. Recall that in conventional learning, the hyperparameter u is fixed a priori. As such, the mutual information $I(U;Z_{1:N}^m)$ (for MI-based bounds) and $I(U;Z_i^m)$ (for ITMI-based bounds) vanishes. For the separate within-task training and test sets, this implies that the average generalization gap is zero, which follows since the per-task test loss $L_t\left(W\right|Z_i^{{\mathrm{m}}_{\mathrm{te}}})$ is an unbiased estimate of per-task generalization loss $L_g\left(W\right|T_i)$. The MI- and ITMI-based bounds for the joint within-task training and test sets then reduce to

$$\begin{array}{c}\hfill \left|{\mathbb{E}}_{P_{Z^m,W|\tau ,u}}\left[\Delta L\left(W\right|Z^m,\tau )\right]\right|\le \sqrt{\frac{2{\delta }_{\tau }^2}{m}I(W;Z^m)},\end{array}$$

(34)

and

$$\begin{array}{c}\hfill \left|{\mathbb{E}}_{P_{Z^m,W|\tau ,u}}\left[\Delta L\left(W\right|Z^m,\tau )\right]\right|\le \frac{1}{m}\sum _{j=1}^m\sqrt{2{\delta }_{\tau }^{2}I(W;Z_j)}\end{array}$$

(35)

respectively, where $I(W;Z^m)$ is evaluated with respect to the joint distribution $P_{W,Z^m|\tau ,u}$ and $I(W;Z_j)$ with respect to $P_{W,Z_j|\tau ,u}$.

The MI- and ITMI-based bounds derived in this section point that a smaller correlation between hyperparameters and meta-training set and thus small mutual information $I(U;Z_{1:N}^m)$ improves the meta-generalization gap, although this seems deleterious to performance. To clarify this contradiction, we would like to emphasize that these bounds quantify the difference between meta-generalization loss and empirical training loss, which in turn depends on the sensitivity of the meta-learner and base-learner to their input meta-training set and per-task training set, respectively. The mutual information terms in our bounds capture these sensitivities. Consequently, our bounds suggest that a meta-learner that is highly correlated to the input meta-training set (i.e., when $I(U;Z_{1:N}^m)$ is large) does not generalize well (i.e., yields large meta-generalization gap). This property aligns with a previous information-theoretic analysis for generalization in conventional learning [25].

To the best of our knowledge, the MI- and ITMI-based bounds studied here are the first bounds on the average meta-generalization gap. As discussed in the introduction, these bounds are distinct from the high-probability PAC and PAC-Bayesian bounds on the meta-generalization gap studied previously on meta-learning. Consequently, the bounds studied in this work are not directly comparable with the existing high-probability bounds.

Finally, we note that similarity between tasks is crucial to meta-learning. If the per-task data distributions $P_{Z|T=\tau }$ in the task environment are ‘closer’ to each other, a meta-learner can efficiently learn the shared characteristics of tasks, and can generalize well to new tasks from the task environment. In our setting, the statistical properties of the task environment $(P_T,{\left\{P_{Z|T=\tau }\right\}}_{\tau \in \mathcal{T}})$ dictate this similarity. Although our MI- and ITMI-based bounds do not explicitly capture this, we note that the properties of task environment are implicitly accounted for by the mutual information terms $I(U;Z_{1:N}^m)$ and $I(U;Z_i^m)$, where the meta-training data set $Z_{1:N}^m$ is generated from the task environment, and also by the sub-Gaussianity considerations in Assumptions 3–6. From preliminary studies, we believe that information-theoretic bounds that explicitly capture the impact of task similarity require a different performance metric than the average meta-generalization gap considered here, and is left to future work.

5. Applications

In this section, we consider two applications of the information-theoretic bounds proposed in Section 4.1. The first, simpler, example concerns a parameter estimation problem for which an optimized meta-learner can be obtained in closed form. In contrast, the second application covers a broad class of iterative meta-training schemes.

5.1. Parameter Estimation

To illustrate the bounds on the meta-generalization gap derived in Section 4.1, we first consider the problem of prediction for a Bernoulli process with a ‘soft’ predictor that uses only a few samples from the process, as well as meta-training data. Towards this, we consider an arbitrary discrete finite set of tasks $\mathcal{T}=\{{\tau }_1,\dots ,{\tau }_M\}$. The data distribution $P_{Z|T={\tau }_k}$ for each task ${\tau }_k\in \mathcal{T}$, $k\in \{1,\dots ,M\}$, is given as $\mathrm{Bernoulli}\left({\mu }_{{\tau }_k}\right)$ with mean parameter ${\mu }_{{\tau }_k}$. The task distribution $P_T$ is then defined over the finite set of mean parameters $\{{\mu }_{{\tau }_1},\dots ,{\mu }_{{\tau }_M}\}$. The base-learner uses training data, distributed i.i.d. from $\mathrm{Bernoulli}\left({\mu }_{{\tau }_k}\right)$ to determine the parameter $W_k$, which is used as a predictor of new observation $Z\sim \mathrm{Bernoulli}\left({\mu }_{{\tau }_k}\right)$ at test time. The loss function is defined as $l(w,z)={(w-z)}^2$, measuring the quadratic error between prediction and realized test input z. Note that the optimal (Bayes) predictor, computable in the ideal case of known distribution $P_{Z|T={\tau }_k}$, is given as $W={\mu }_{{\tau }_k}$. We now distinguish the two cases with separate and joint within-task training and test sets.

5.1.1. Separate Within-Task Training and Test Sets

The base-learner $P_{W|Z_k^{{\mathrm{m}}_{\mathrm{tr}}},u}$ for task ${\tau }_k\in \mathcal{T}$, deterministically selects the prediction

$$\begin{array}{c}\hfill W_k=\alpha D_k^{{\mathrm{m}}_{\mathrm{tr}}}+(1-\alpha )u,\end{array}$$

(36)

where $D_k^{{\mathrm{m}}_{\mathrm{tr}}}=\frac{1}{m_{\mathrm{tr}}}{\sum }_{j=1}^{m_{\mathrm{tr}}}{Z}_{k,j}^{{\mathrm{m}}_{\mathrm{tr}}}$ is an empirical average over the training set $Z_{k,j}^{{\mathrm{m}}_{\mathrm{tr}}}$, u is a hyperparameter defining a bias that can be meta-trained, and $\alpha \in [0,1)$ is a fixed scalar. Here, $Z_{k,j}^{{\mathrm{m}}_{\mathrm{tr}}}$ denote the jth data sample in the training set $Z_k^{{\mathrm{m}}_{\mathrm{tr}}}$ of task ${\tau }_k$. The bias term in (36) may help approximate the ideal Bayes predictor in the presence of limited data $Z_k^{{\mathrm{m}}_{\mathrm{tr}}}$.

The objective of the meta-learner is to infer the hyperparameter u. For a given meta-training data set $Z_{1:N}^m$, comprising of data sets from N tasks sampled according to $P_T$, the meta-learner can compute the empirical meta-training loss as

$$\begin{array}{c}\hfill {\mathcal{L}}_t^{\mathrm{sep}}\left(u\right|Z_{1:N}^m)=\frac{1}{N}\sum _{k=1}^N\frac{1}{m_{\mathrm{te}}}\sum _{j=1}^{m_{\mathrm{te}}}{(W_k-Z_{k,j}^{{\mathrm{m}}_{\mathrm{te}}})}^2,\end{array}$$

(37)

where $Z_{k,j}^{{\mathrm{m}}_{\mathrm{te}}}$ denote the jth example in the test set of $Z_k^m$, the kth sub-data set of $Z_{1:N}^m$. The meta-learner $P_{U|Z_{1:N}^m}$ then deterministically selects the minimizing hyperparameter u of the meta-training empirical loss function in (37). This optimization yields

$$\begin{array}{c}\hfill U=\frac{{(1-\alpha )}^{-1}}{N}\left(\sum _{k=1}^N{D}_k^{{\mathrm{m}}_{\mathrm{te}}}-\alpha D_k^{{\mathrm{m}}_{\mathrm{tr}}}\right),\end{array}$$

(38)

where $D_k^{{\mathrm{m}}_{\mathrm{te}}}={\sum }_{j=1}^{m_{\mathrm{te}}}{Z}_{k,j}^{{\mathrm{m}}_{\mathrm{te}}}/m_{\mathrm{te}}$. Note that $D_k^{{\mathrm{m}}_{\mathrm{te}}}$ and $D_k^{{\mathrm{m}}_{\mathrm{tr}}}$ are binomial random variables and by (38), U takes finitely many discrete values and is bounded as $-\alpha {(1-\alpha )}^{-1}\le U\le {(1-\alpha )}^{-1}$. The meta-test loss can be explicitly computed as

$$\begin{array}{c}\hfill {\mathcal{L}}_g^{\mathrm{sep}}\left(u\right)={(1-\alpha )}^2\left(u^2-2u{\mathbb{E}}_{P_T}\left[{\mu }_T\right]\right)+{\mathbb{E}}_{P_T}\left[{\alpha }^2\left({\mu }_T^2+\frac{{\mu }_T{\overline{\mu }}_T}{m_{\mathrm{tr}}}\right)+{\mu }_T-2\alpha {\mu }_T^2\right],\end{array}$$

(39)

where ${\overline{\mu }}_T=1-{\mu }_T$, and the average meta-generalization gap evaluates to

$$\begin{array}{cc}\hfill {\mathbb{E}}_{P_{Z_{1:N}^m,U}}\left[\Delta {\mathcal{P}}^{\mathrm{sep}}\left(U\right|Z_{1:N}^m)\right]& =\frac{2(1+{\alpha }^2)}{N}{\mathrm{Var}}_T+\frac{2{\mathbb{E}}_{P_T}\left[{\mu }_T{\overline{\mu }}_T\right]}{N}\left(\frac{1}{m_{\mathrm{te}}}+\frac{{\alpha }^2}{m_{\mathrm{tr}}}\right),\hfill \end{array}$$

(40)

where ${\mathrm{Var}}_T=({\mathbb{E}}_{P_T}\left[{\mu }_T^2\right]-{\left({\mathbb{E}}_{P_T}\left[{\mu }_T\right]\right)}^2)$ is the variance of ${\mu }_T$.

To compute the MI- and ITMI-based bounds on the meta-generalization gap (40), it is easy to verify that the average training loss $L_t^{\mathrm{sep}}(·|Z^m)$ is bounded, i.e., $0\le L_t^{\mathrm{sep}}(·|Z^m)\le {(1+\alpha )}^2$ for all $u\in \mathcal{U}$ and $Z^m\in {\mathcal{Z}}^m$. Thus, Assumption 3 for the MI bound and also Assumption 4 for the ITMI bound hold with ${\sigma }^2={(1+\alpha )}^4/4$. For the MI bound, we note that, since the meta-learner is deterministic, we have that $I(U;Z_{1:N}^m)=H\left(U\right)$. The ITMI bound (30) is given as

$$\begin{array}{c}\hfill |{\mathbb{E}}_{P_{Z_{1:N}^m,U}}\left[{\Delta }^{\mathrm{sep}}\left(U\right|Z_{1:N}^m)\right]|\le \frac{1}{N}\sum _{i=1}^N\sqrt{\frac{{(1+\alpha )}^4}{2}I(U;Z_i^m)}.\end{array}$$

(41)

The information-theoretic measures in (41) can be evaluated numerically as discussed in Appendix D.

For a numerical illustration, Figure 4 plots the average of the meta-generalization loss (39) and average meta-training loss (A16) along with the ITMI bound in (41) and MI bound in (27). It can be seen that the ITMI bound is tighter than MI bound and correctly predicts the decrease in the meta-generalization gap as the number N of tasks increases.

Figure 4. Comparison of the MI bound in (27) and ITMI-based bound obtained in (41) with the meta-generalization gap for meta-learning with separate within-task training and test sets. The task environment is defined by $M=12$ tasks. Other parameters are set as $\alpha =0.15$, $m_{\mathrm{tr}}=15$, $m_{\mathrm{te}}=5$.

5.1.2. Joint Within-Task Training and Testing sets

We now consider the case with joint within-task training and test sets. The base-learner $P_{W|Z_k^m,U}$ for task ${\tau }_k\in \mathcal{T}$ still uses the predictor (36), but now the empirical average over the training set is given as $D_k={\sum }_{j=1}^m{Z}_{k,j}^m/m.$ As before, the meta-learner $P_{U|Z_{1:N}^m}$ deterministically selects the minimizing hyperparameter u of the meta-training empirical loss function, $L_{Z_{1:N}^m}\left(u\right)=(1/N){\sum }_{k=1}^N(1/m){\sum }_{j=1}^m{(W_i-Z_{k,j}^m)}^2,$ yielding $U=\frac{1}{N}{\sum }_{k=1}^N{D}_k.$ As discussed in Appendix D, the meta-generalization loss for this example can also be explicitly computed and the meta-generalization gap bounds in (32) and (33) can be evaluated numerically. Figure 5 plots the average meta-generalization loss and average meta-training loss along with the MI bound in (32) and ITMI bound in (A18), as a function of per-task data samples m. The ITMI bound is seen to better reflect the decrease of the meta-training loss as a function of m.

Figure 5. Comparison of the MI- and ITMI-based bound obtained in (A18) with the meta-generalization gap for meta-learning with joint within-task training and test sets, as a function of the per-task data samples m for $N=5$ and $\alpha =0.55$. The task environment is defined by $M=9$ tasks.

5.2. Noisy Iterative Meta-Learning Algorithms

Most meta-learning algorithms are built around a nested loop structure, with the inner loop applying the base-learner on the meta-training set and the outer loop updating the hyperparameters U. In this section, we focus on a vast class of such meta-learning algorithms in which the inner loop applies training procedures dependent on the current iterate of the hyperparameter, while the outer loop updates the hyperparameter using a stochastic rule. This class includes stochastic variants of state-of-the-art algorithms such as MAML [23] and reptile [24]. We apply the derived information-theoretic bounds to study the meta-generalization performance of the mentioned class of meta-training iterative stochastic rules by focusing on the case of separate within-task training and test sets here, which is assumed e.g., by MAML. The analysis for the setup with joint within-task training and test sets can also be carried out at the cost of a more cumbersome notation.

To start, let $U^j\in {\mathbb{R}}^d$ denote the hyperparameter vector at outer iteration j, with $U^0\in {\mathbb{R}}^d$ being an arbitrary initialization. For example, in MAML, the hyperparameter U defines the initial iterate used by each base-learner in the inner loop to update the model parameter $W_{\tau }$ corresponding to task $\tau$. At each iteration $j\ge 1$, we randomly select a mini-batch of task indices $K_j\subseteq [1,\dots ,N]$ from the meta-training data $Z_{1:N}^m$, obtaining the corresponding data set $Z_{K_j}^m=(Z_{K_j}^{{\mathrm{m}}_{\mathrm{tr}}},Z_{K_j}^{{\mathrm{m}}_{\mathrm{te}}})\subseteq Z_{1:N}^m$, where $Z_{K_j}^{{\mathrm{m}}_{\mathrm{tr}}}={\left\{Z_k^{{\mathrm{m}}_{\mathrm{tr}}}\right\}}_{k\in K_j}$ and $Z_{K_j}^{{\mathrm{m}}_{\mathrm{te}}}={\left\{Z_k^{{\mathrm{m}}_{\mathrm{te}}}\right\}}_{k\in K_j}$ are the separate training and test sets for the selected tasks. For each index $k\in K_j$, in the inner loop, the base-learner selects the model parameter $W_k^j$ as a possibly stochastic function

$$\begin{array}{c}\hfill W_k^j=g(U^{j-1},Z_k^{{\mathrm{m}}_{\mathrm{tr}}}).\end{array}$$

(42)

For instance, in MAML, the function $g(U^{j-1},Z_k^{{\mathrm{m}}_{\mathrm{tr}}})\in {\mathbb{R}}^d$ in (42) represents the output of an SGD procedure that starts from initialization $U^{j-1}$ and uses the task training data $Z_k^{{\mathrm{m}}_{\mathrm{tr}}}$ to iteratively update the model parameters, producing the final iterate $W_k^j$. We denote as $W_{K_j}={\left\{W_k^j\right\}}_{k\in K_j}$ the collection of the base-learners’ outputs for all task indices $k\in K_j$ at outer iteration j.

In the outer loop, the meta-learner uses the task-specific adapted parameters $W_{K_j}$ from the inner loop and the meta-test set $Z_{K_j}^{{\mathrm{m}}_{\mathrm{te}}}$ to update the past iterate $U^{j-1}$ according to the general update rule

$$\begin{array}{c}\hfill U^j=F\left(U^{j-1}\right)+{\beta }_{j}G(U^{j-1},W_{K_j},Z_{K_j}^{{\mathrm{m}}_{\mathrm{te}}})+{\xi }_j,\end{array}$$

(43)

where $F(·)$ and $G(·,·,·)$ are arbitrary deterministic functions; ${\beta }_j$ is the step-size; and ${\xi }_j\sim \mathcal{N}(0,{\gamma }_j^2{I}_d)$ is an isotropic Gaussian noise, independently drawn for $j=1,2,\dots ,$. As an example, in MAML, the function $F(·)$ is the identity function and function $G(·,·,·)$ equals the gradient of the empirical loss $1/|K_j|{\sum }_{k\in K_j}{L}_t^{\mathrm{sep}}\left(W_k^j\right|Z_k^{{\mathrm{m}}_{\mathrm{te}}})$ in (14) with respect to $U^{j-1}$. Note, however, that MAML does not add noise, i.e., ${\gamma }_j^2=0$ for all j.

The final output of the meta-learning algorithm is then defined as an arbitrary function $U=f(U^1,\dots ,U^J),$ of all iterates. Examples of function f include the last update $f(U^1,\dots ,U^J)=f\left(U^J\right)$ and average of the updates $f(U^1,\dots ,U^J)=1/J{\sum }_{j=1}^J{U}^j$. A graphical model representation of the variables involved is shown in Figure 6.

Figure 6. A graphical model representation of the variables involved in the Definition of noisy iterative algorithms.

We now derive an upper bound on the meta-generalization gap for the general class of iterative meta-learning algorithm satisfying (42) and (43) under the following assumptions.

Assumption 7.

(1) 

For the base-learner given in (42), the average training loss $L_t^{\mathrm{sep}}\left(u\right|Z^m)$ in (15) is ${\sigma }^2$-sub-Gaussian for all $u\in$ when $Z^m\sim P_{Z^m}$;

(2) 

The meta-training data set $Z_{K_j}^m$ sampled at each iteration j is conditionally independent of the history of model-parameter vectors ${\left\{W_{K_i}\right\}}_{i=1}^{j-1}$ and hyperparameter $U^{(j-1)}=(U^1,U^2,\dots ,U^{j-1})$, i.e.,

$$\begin{array}{c}\hfill P_{Z_{K_j}^m|{\left\{Z_{K_i}^m\right\}}_{i=1}^{j-1},Z_{1:N}^m,U^{(j-1)},{\left\{W_{K_i}\right\}}_{i=1}^{j-1}}=P_{Z_{K_j}^m|{\left\{Z_{K_i}^m\right\}}_{i=1}^{j-1},Z_{1:N}^m};\end{array}$$

(44)

(3) 

The meta-parameter update function $G(·,·,·)$ is uniformly bounded, i.e., ${\left|\right|G(·,·,·)\left|\right|}_2\le L$ for some $L>0$.

Lemma 4.

Under Assumption 7, the following upper bound on the meta-generalization gap (18) holds for the class of noisy iterative meta-training algorithms (42) and (43)

$$\begin{array}{cc}\hfill {\mathbb{E}}_{P_{Z_{1:N}^m,U}}\left[{\Delta }^{\mathrm{sep}}\left(U\right|Z_{1:N}^m)\right]& \le \sqrt{\frac{2{\sigma }^2}{N}\sum _{j=1}^J\frac{d}{2}log\left(1+\frac{{\beta }_j^2{L}^2}{d{\gamma }_j^2}\right)}.\hfill \end{array}$$

(45)

Proof. 

See Appendix E. □

The bound in (45) has the same form as the generalization gap derived in [27] for conventional learning. From (45), the generalization gap can be reduced by increasing the variance ${\gamma }_j^2$ of the injected Gaussian noise. In particular, the meta-generalization gap depends on the ratios ${\beta }_j^2/{\gamma }_j^2$ between squared step size ${\beta }_j^2$ and variance ${\gamma }_j^2$. For example, SGLD sets ${\gamma }_j=\sqrt{{\beta }_j}$, and a step size ${\beta }_j$ decaying over time according to the standard Robbins–Monro conditions, in order to ensure convergence of the output samples to the generalized posterior distribution of the hyperparameters [52].

Example: To illustrate bound (45), we now consider a simple logistic regression problem that generalizes the example studied in Section 5.1. Accordingly, each data point Z corresponds to labelled data $Z=(X,Y)$, where $X\in {\{0,1\}}^d$ represents the input vector and $Y\in \{0,1\}$ represents the corresponding binary label. The data distribution $P_{Z|{\tau }_k}=P_{X|{\tau }_k}{P}_{Y|X,{\tau }_k}$ for each task ${\tau }_k\in \mathcal{T}=\{{\tau }_1,\dots ,{\tau }_M\}$ is such that $X\sim P_{X|{\tau }_k}$ is a d-dimensional Bernoulli vector obtained via d independent draws from $\mathrm{Bernoulli}\left(\nu \right)$ and Y is distributed as $Y\sim \mathrm{Bernoulli}\left(\varphi \left({\mu }_{{\tau }_k}^{T}X\right)\right)$, where $\varphi \left(a\right)=1/(1+exp(-a\left)\right)$ is the sigmoid function and ${\mu }_{{\tau }_k}\in {\mathbb{R}}^d$, with $\left|\right|{\mu }_{{\tau }_k}{\left|\right|}_2\le 1$. The task distribution $P_T$ then defines a distribution over the parameter vectors $\{{\mu }_{{\tau }_1},\dots ,{\mu }_{{\tau }_M}\}$. The base-learner uses training data generated i.i.d. from $P_{Z|{\tau }_k}$ to obtain a prediction w of the parameter vector ${\mu }_{{\tau }_k}$ for task ${\tau }_k\in \mathcal{T}$. The loss function is taken as the quadratic error $l(w,z)={(\varphi \left(w^{T}x\right)-y)}^2.$

At each iteration j, starting from initialization point $U^{j-1}$, the base-learner in (42) uses a one-step projected gradient descent algorithm on the training data set $Z_k^{{\mathrm{m}}_{\mathrm{tr}}}$ to obtain the prediction $W_k^j$ as

$$\begin{array}{c}\hfill W_k^j={\mathrm{proj}}_{\mathcal{W}}\left(U^{j-1}-\alpha {\nabla }_w{L}_t^{\mathrm{sep}}\left(w\right|Z_k^{{\mathrm{m}}_{\mathrm{tr}}}){|}_{w=U^{j-1}}\right),\end{array}$$

(46)

where $\alpha >0$ is the step-size $\mathcal{W}=\{w\in {\mathbb{R}}^d|\left|\right|w|{|}_2\le 1\}$ is the set of feasible model parameters and ${\mathrm{proj}}_{\mathcal{A}}\left(b\right)=\frac{1}{2}{min}_{a\in \mathcal{A}}\left|\right|a-{b\left|\right|}_2^2$ is the projection operator. The meta-learner (43) updates the initialization vector according to the noisy gradient descent rule

$$\begin{array}{c}\hfill U^j=U^{j-1}-{\beta }_j\left(\frac{1}{|K_j|}\sum _{k=1}^{|K_j|}{\nabla }_w{L}_t^{\mathrm{sep}}{\left(w\right|Z_k^{{\mathrm{m}}_{\mathrm{te}}})|}_{w=W_k^j}\right)+{\xi }_j,\end{array}$$

(47)

where ${\beta }_j$ is the step-size; and ${\xi }_j\sim \mathcal{N}(0,{\gamma }_j^2{I}_d)$ is isotropic Gaussian noise. This update rule corresponds to performing a first order MAML (FOMAML) [23] with the addition of noise.

For this problem, it is easy to verify that Assumption 7 is satisfied, since the loss function $l(·,·)$ is bounded in the interval $[0,1]$, whereby $L_t^{\mathrm{sep}}\left(u\right|Z^m)$ is also $[a,b]$-bounded. We also have the inequality

$$\begin{array}{c}\hfill {∥\frac{1}{|K_t|}\sum _{i=1}^{|K_t|}{\nabla }_w{L}_{Z^{{\mathrm{m}}_{\mathrm{te}}}}{\left(w\right)|}_{w=W_i^t}∥}_2\le 2\sqrt{d}{e}^{\sqrt{d}}\triangleq L.\end{array}$$

(48)

The MI bound in (45) then evaluates to

$$\begin{array}{cc}\hfill {\mathbb{E}}_{P_{Z_{1:N}^m,U}}\left[\Delta \mathcal{L}\left(U\right|Z_{1:N}^m)\right]& \le \sqrt{\frac{1}{2N}\sum _{j=1}^J\frac{d}{2}log\left(1+\frac{4{\beta }_j^2{e}^{2\sqrt{d}}}{{\gamma }_j^2}\right)}.\hfill \end{array}$$

(49)

We now evaluate the meta-training and meta-test loss, along with the bound (49) as a function of the ratio ${\gamma }_j^2/{\beta }_j^2$ in Figure 7. For the experiment, we considered a task environment of $M=20$ tasks with $\nu =0.4$, $d=3$, $N=4$ meta-training tasks with $m_{\mathrm{tr}}=10$ training data samples and $m_{\mathrm{te}}=5$ test data samples. For the inner-loop (46), we fixed step-size $\alpha ={10}^{-4}$ and for the outer-loop (47), we set $|K_t|=N$, ${\beta }_j=0.25$ and $T=200$ iterations.

Figure 7. Comparison of the meta-generalization gap with the MI-based bound in (49) as function of the ratio ${\gamma }_t^2/{\beta }_t^2$.

As suggested by Lemma 4, the meta-generalization gap decreases with addition of noise. While the MI bound (45) is generally loose, it correctly quantifies the dependence of the meta-generalization loss and the ratio ${\gamma }_j^2/{\beta }_j^2$, and it can hence serve as a useful meta-training criterion [20,48].

6. Conclusions

This work has presented novel information-theoretic upper bounds on the average generalization gap of meta-learning algorithms, thereby extending the well-studied information-theoretic approaches in conventional learning to meta-learning. The proposed bounds capture two sources of uncertainty-environment-level uncertainty and within-task uncertainty—and bound them via separate mutual information terms. Applications were also discussed, with the aim of elucidating the use of the bounds to quantify meta-overfitting and guide the choice of the meta-inductive bias, i.e., the class of inductive biases. The derived bounds are amenable to further refinements, such as those along the lines of [39,40,45]. It would also be interesting to study the meta-generalization bounds on noisy iterative meta-learning algorithms using the tighter information-theoretic bounds such as [26,40].