Unified Probabilistic Deep Continual Learning through Generative Replay and Open Set Recognition

Abstract

Modern deep neural networks are well known to be brittle in the face of unknown data instances and recognition of the latter remains a challenge. Although it is inevitable for continual-learning systems to encounter such unseen concepts, the corresponding literature appears to nonetheless focus primarily on alleviating catastrophic interference with learned representations. In this work, we introduce a probabilistic approach that connects these perspectives based on variational inference in a single deep autoencoder model. Specifically, we propose to bound the approximate posterior by fitting regions of high density on the basis of correctly classified data points. These bounds are shown to serve a dual purpose: unseen unknown out-of-distribution data can be distinguished from already trained known tasks towards robust application. Simultaneously, to retain already acquired knowledge, a generative replay process can be narrowed to strictly in-distribution samples, in order to significantly alleviate catastrophic interference.

1. Introduction

Consider an empirically optimized deep neural network for a particular task, for the sake of simplicity, say the classification of dogs and cats. Typically, such a system is trained in a closed world setting [1] according to an isolated learning paradigm [2]. That is, we assume the observable world to consist of a finite set of known instances of dogs and cats, where training and evaluation is limited to the same underlying statistical data population. The training process is treated in isolation, i.e., the model parameters are inferred from the entire existing dataset at all times. However, the real world requires dealing with sequentially arriving tasks and data originating from potentially unknown sources.

In particular, should we wish to apply and extend the system to an open world, where several other animals (and non animals) exist, there are two critical questions: (a) How can we prevent obvious mispredictions if the system encounters a new class? (b) How can we continue to incorporate this new concept into our present system without full retraining? With respect to the former question, it is well known that neural networks yield overconfident mispredictions in the face of unseen unknown concepts [3], a realization that has recently resurfaced in the context of various deep neural networks [4,5,6]. With respect to the latter question, it is similarly well known that neural networks, which are trained exclusively on newly arriving data, will overwrite their representations and thus forget encoded knowledge—a phenomenon referred to as catastrophic interference or catastrophic forgetting [7,8]. Although we have worded the above questions in a way that naturally exposes their connection: to identify what is new and think about how new concepts can be incorporated, they are largely subject to separate treatment in the respective literature. While open-set recognition [1,9,10] aims to explicitly identify novel inputs that deviate with respect to already observed instances, the existing continual learning literature predominantly concentrates its efforts on finding mechanisms to alleviate catastrophic interference (see [11] for an algorithmic survey).

In particular, the indispensable system component to distinguish seen from unseen unknown data, both as a guarantee for robust application and to avoid the requirement of explicit task labels for prediction, is generally missing from recent continual-learning works. Inspired by this gap, we set out to connect open-set recognition and continual learning. The underlying connecting element is motivated from the prior work of Bendale and Boult [12], who proposed to leverage extreme value theory (EVT) to address open-set detection in deep neural networks. The authors suggested to modify softmax prediction scores on the basis of feature space distances in blackbox discriminative models. Although this approach is promising, it alas comes with the substantial caveat that purely discriminative networks are prone to encode noise as features [13] or fall for a most simple discriminative solution that neglects meaningful features [14]. Inspired by these former insights, we set out to connect open-set recognition and continual learning, while overcoming present limitations through treatment from a generative modeling perspective.

Our specific contributions are that we propose to unify the prevention of catastrophic interference in continual learning with open-set recognition in a single model. Specifically, we extend prior EVT works [9,10,12] to a natural formulation on the basis of the aggregate posterior in variational inference with deep autoencoders [15,16]. By identifying out-of-distribution instances we can detect unseen unknown data and prevent false predictions; by explicitly generating in-distribution samples from areas of high probability density under the aggregate posterior, we can simultaneously circumvent rehearsal of ambiguous uninformative examples. This leads to robust application, while significantly reducing catastrophic interference. We empirically corroborated our approach in terms of improved out-of-distribution detection performance and simultaneously reduced the continual catastrophic interference. We further demonstrate benefits through recent deep generative modeling advances, such as autoregression [2,17,18] and introspection [19,20], validated by scaling to high-resolution color images.

1.1. Background and Related Work

1.1.1. Continual Learning

In isolated supervised learning, the core assumption is the presence of i.i.d. data at all times and training is conducted using a dataset $\mathit{D}\equiv {\left\{\left({\mathit{x}}^{\left(n\right)},y^{\left(n\right)}\right)\right\}}_{n=1}^N$, consisting of N pairs of data instances ${\mathit{x}}^{\left(n\right)}$ and their corresponding labels $y^{\left(n\right)}\in \left\{1\dots C\right\}$ for C classes. In contrast, in continual learning, data ${\mathit{D}}_t\equiv {\left\{\left({\mathit{x}}_t^{\left(n\right)},y_t^{\left(n\right)}\right)\right\}}_{n=1}^{N_t}$ with $t=1,\dots ,T$ arrives sequentially for T disjoint sets, each with number of classes $C_t$.

It is assumed that only the data of the current task is available. Without additional mechanisms, tuning on such a sequence will lead to catastrophic interference [7,8], i.e., representations of former tasks being overwritten through present optimization. A recent review of many continual-learning algorithms to prevent said interference was provided by Parisi et al. [11]. Here, we present a brief summary of the key underlying principles.

Alleviating catastrophic interference is most prominently addressed from two angles. Regularization methods, such as synaptic intelligence (SI) [21] or elastic weight consolidation (EWC) [22] explicitly constrain the weights during continual learning to avoid drifting too far away from the previous tasks’ solutions. In a related picture, learning without forgetting [23] uses knowledge distillation [24] to regularize the end-to-end functional.

Rehearsal methods on the other hand, store data subsets from distributions belonging to old tasks or generate samples in pseudo-rehearsal [25]. The central component of the latter is thus the selection of significant instances. For methods, such as incremental classifier and representation learning (iCarl) [26], it is therefore common to resort to auxiliary techniques, such as the nearest-mean classifier [27] or core sets [28]. Inspired by complementary learning systems [29], dual-model approaches sample data from a separate generative memory. In a bio-inspired incremental learning architecture (GeppNet) [30], long short-term memory [31] is used for storage, whereas generative replay [32] samples from an additional generative adversarial network (GAN) [33].

As detailed in Variational Generative Replay (VGR) [34,35], methods with a Bayesian perspective encompass a natural capability for continual learning by making use of the learned distribution. Existing works nevertheless fall into the above two categories and their combination: a prior-based approach using the former task’s approximate posterior as the new task’s prior [36] or estimating the likelihood of former data through generative replay or other forms of rehearsal [34,37]. Crucially, the success of many continual-learning techniques can be attributed primarily to the considered evaluation scenario. With the exception of VGR [34], the majority of above techniques train a separate classifier per task and thus either require the explicit storage of task labels or assume the presence of a task oracle during evaluation. This multi-head scenario prevents “cross-talk” between classifier units by not sharing them, which would otherwise rapidly decay the accuracy as newly introduced classes directly confuse existing concepts. While the latter is acceptable to limit catastrophic interference, it also signifies a major limitation in practical applications. Even though VGR [34] uses a single classifier, the researchers trained a separate generative model per task to avoid catastrophic interference in the generator.

Our approach builds upon these previous works and leverages variational inference in deep generative models. However, we propose to tie the prevention of catastrophic interference with open-set recognition through a natural mechanism based on the aggregate posterior in a single model.

1.1.2. Out-of-Distribution and Open Set Recognition

The above-mentioned literature focused their efforts predominantly on addressing catastrophic interference. Even though continual learning is the desideratum, the corresponding evaluation is thus conducted in a closed world setting, where instances that do not belong to the observed data distribution are not encountered. In reality, this is not guaranteed as users could provide arbitrary inputs or unknowingly present the system with novel inputs that deviate substantially from previously seen instances. Our models thus require the ability to identify unseen examples in the unconstrained open world and categorize them as either belonging to the already known set of classes or as presently being unknown. We provide a small overview of approaches that aim to address this question in deep neural networks. A comprehensive survey was provided by Boult et al. [1].

As the most simple approach, the aim of calibration works is to separate a known and unknown input through prediction confidence, often by fine tuning or re-training an already existing model. In out-of-distribution detector for neural networks (ODIN) [38], this is addressed through perturbations and temperature scaling, while Lee et al. [39] used a separately trained GAN to generate out-of-distribution samples from low probability densities and explicitly reduced their confidence through the inclusion of an additional loss term. Similarly, the objectosphere loss [40] defines an objective that explicitly aims to maximize entropy for upfront available unknown inputs.

As we do not have access to future data a priori, by definition, a naive conditioning or calibration on unseen unknown data is infeasible. The commonly applied thresholding is insufficient as overconfident prediction values cannot be prevented [3]. Bayesian neural network models [41] could be believed to intrinsically be able to reject statistical outliers through model uncertainty [34] and overcome this limitation of overconfident prediction values. For use with deep neural networks, it was suggested that stochastic forward passes with Monte-Carlo Dropout (MCD) [42] can provide a suitable approximation. However, the closed-world assumption in training and evaluation still persists [1]. In addition, variational approximations in deep networks [15,34,37,43] and corresponding uncertainty estimates suffer from similar overconfidence, and the distinction of unseen out-of-distribution data from already trained knowledge is known to be unsatisfactory [5,6].

A more formal approach was suggested in works based on open-set recognition [9]. The key here is to limit predictions originating from open space, that is, the area in obtained embeddings that is outside of a small radius around previously observed training examples. Without re-training, post hoc calibration or modifying loss functions, one approach to open-set recognition in deep networks is through extreme-value theory (EVT) [10,12]. Here, limiting the threat of overconfidence is based on monotonically decreasing the recognition function’s probability with respect to increasing distance of instances to the feature embedding of known training points. The Weibull distribution, as one member of the family of extreme value distributions, has been empirically demonstrated to work well in conjunction with distances in the penultimate deep network layer as the underlying feature space. On the basis of extreme values to this layer’s average activation values, the authors devised a procedure to revise the Softmax prediction values, referred to as OpenMax.

In a similar spirit, our work avoids relying on predictive values, while also moving away from empirically chosen deep neural network feature spaces. We instead propose to use EVT to bind the approximate posterior in variational inference. We thus directly operate on the underlying (lower-bound to the) data distribution and the generative factors. This additionally allows us to constrain the generative replay to distribution inliers, which further alleviates catastrophic interference.

2. Materials and Methods

2.1. Unifying Catastrophic Interference Prevention with Open Set Recognition

We first summarize the preliminaries on continual learning from a perspective of variational inference in deep generative models [15,43]. We then proceed by bridging the improved prevention of catastrophic interference in continual learning with the detection of unseen unknown data in open-set recognition.

2.1.1. Preliminaries: Learning Continually through Variational Auto-Encoding

We start with a problem scenario similar to the one introduced in “Auto-Encoding Variational Bayes” [15], i.e., we assume that there exists a data generation process responsible for the creation of the labeled data given some random latent variable $\mathit{z}$. We consider a model with a shared encoder with variational parameters $\mathit{\theta }$, decoder and linear classifier with respective parameters $\mathit{\varphi }$ and $\mathit{\xi }$. The joint probabilistic encoder learns an encoding to a latent variable $\mathit{z}$, over which a unit Gaussian prior $p\left(\mathit{z}\right)$ is placed.

Using variational inference, the encoder’s purpose is to approximate the true posterior to $p_{\mathit{\varphi }}(\mathit{x},\mathit{z})$ and $p_{\mathit{\xi }}(y,\mathit{z})$. The probabilistic decoder $p_{\mathit{\varphi }}\left(\mathit{x}\right|\mathit{z})$ and probabilistic linear classifier $p_{\mathit{\xi }}\left(y\right|\mathit{z})$ then return the conditional probability density of the input $\mathit{x}$ and target y under the respective generative model given a sample $\mathit{z}$ from the approximate posterior $q_{\mathit{\theta }}\left(\mathit{z}\right|\mathit{x})$. This yields a generative model $p(\mathit{x},\mathit{y},\mathit{z})$, for which we assume a factorization and generative process of the form $p(\mathit{x},\mathit{y},\mathit{z})=p\left(\mathit{x}\right|\mathit{z}\left)p\right(\mathit{y}\left|\mathit{z}\right)p\left(\mathit{z}\right)$. For variational inference with this model, the sum over all elements in the dataset n ∈ D in the following lower-bound is optimized:

$$p(\mathit{x},\mathit{y})\ge {\mathbb{E}}_{q_{\mathit{\theta }}\left(\mathit{z}\right|\mathit{x})}\left[log{p}_{\mathit{\varphi }}\left(\mathit{x}\right|\mathit{z})+log{p}_{\mathit{\xi }}\left(\mathit{y}\right|\mathit{z})\right]-\beta KL(q_{\mathit{\theta }}\left(\mathit{z}\right|\mathit{x})\left|\right|p\left(\mathit{z}\right)),$$

(1)

where KL denotes the Kullback-Leibler divergence. In other words, the right hand side of Equation (1) defines our loss $\mathcal{L}\left(\mathit{x},\mathit{y};\mathit{\theta },\mathit{\varphi },\mathit{\xi }\right)$. This model can be seen as employing a variant of a (semi-)supervised variational auto-encoder (VAE) [16] with a $\beta$ term [44], where, in addition to approximating the data distribution, the model learns to incorporate the class structure into the latent space. Without the blue terms, the original unsupervised VAE formulation [15] is recovered. This forms the basis for continual learning with open-set recognition as discussed in the subsequent section. An illustration of the model is shown in Figure 1.

Abstracting away from the mathematical detail and speaking informally about the intuition behind the model, we first encode a data input $\mathit{x}$ and encode it into two vectors. These vectors represent the mean and standard deviation of a Gaussian distribution. Using the reparametrization trick $\epsilon ·\sigma +\mu$, a sample from this distribution is then calculated. During training, the respective embedding, also referred to as the latent space, is encouraged to follow a unit Gaussian distribution through the minimization of the Kullback-Leibler divergence. A linear classifier that operates directly on this latent embedding to predict a class for a sample additionally ensures that the obtained distribution is clustered according to the classes.

Examples of such fits are shown in the later Figure 2. Finally, the decoder takes, as input, the latent variable and reconstructs the original data input during training. Once the model is finished training, we can also directly draw a sample from the Gaussian distribution, obtain a latent sample and generate a novel data point directly, without the need to compute the encoder first. A corresponding full and formal derivation of Equation (1), the lower-bound to the joint distribution $p(\mathit{x},\mathit{y})$ is supplied in Appendix A.1.

Without further constraints, one could continually train the above model by sequentially accumulating and optimizing Equation (1) over all currently present tasks $t=1,\dots ,T$. Being based on the accumulation of real data, this provides an upper bound to the achievable performance in continual learning. However, this form of continued training is generally infeasible if only the most recent task’s data is assumed to be available. Making use of the model’s generative nature, we can follow previous works [34,37] and estimate the likelihood of former data through generative replay:

$${\mathcal{L}}_t\left(\mathit{x},\mathit{y};\mathit{\theta },\mathit{\varphi },\mathit{\xi }\right)=\frac{1}{2}\frac{1}{N_t}\sum _{n=1}^{N_t}\mathcal{L}\left({\mathit{x}}_t^{\left(n\right)},{\mathit{y}}_t^{\left(n\right)};\mathit{\theta },\mathit{\varphi },\mathit{\xi }\right)+\frac{1}{2}\frac{1}{N_t^{\prime }}\sum _{n=1}^{N_t^{\prime }}\mathcal{L}\left({\mathit{x}}_t^{\prime \left(n\right)},{\mathit{y}}_t^{\prime \left(n\right)};\mathit{\theta },\mathit{\varphi },\mathit{\xi }\right)$$

(2)

where

$${\mathit{x}}_t^{\prime }\sim p_{\mathit{\varphi },t-1}\left(\mathit{x}\right|\mathit{z});y_t^{\prime }\sim p_{\mathit{\xi },t-1}\left(y\right|\mathit{z})\mathrm{and}\mathit{z}\sim p\left(\mathit{z}\right).$$

(3)

Here, ${\mathit{x}}_t^{\prime }$ is a sample from the generative model with its corresponding classifier label $y_t^{\prime }$. $N_t^{\prime }$ is the number of instances of all previously seen tasks. In this way, the expectation of the log-likelihood for all previously seen tasks is estimated and the dataset at any point in time ${\tilde{\mathit{D}}}_t\equiv {\left\{({\tilde{\mathit{x}}}_t^{\left(n\right)},{\tilde{y}}_t^{\left(n\right)})\right\}}_{n=1}^{{\tilde{N}}_t}=\left\{({\mathit{x}}_t\cup {\mathit{x}}_t^{\prime },y_t\cup y_t^{\prime })\right\}$ is a concatenation of past data generations and the current task’s real data.

2.1.2. Open Set Recognition and Generative Replay with Statistical Outlier Rejection

Trained naively in the above fashion, our model will unfortunately suffer from accumulated errors with each successive iteration of generative replay, similar to the current literature approaches. To avoid this, we would alternatively require the training of multiple encoders to approximate each task’s posterior individually, as in variational continual learning (VCL) [36], or train multiple generators, as in VGR [34]. We posit that the main challenge is how high-density areas under the prior $p\left(\mathit{z}\right)$ are not necessarily reflected in the structure of the aggregate posterior $q_{\mathit{\theta },t}\left(\mathit{z}\right)$ [45]. The latter refers to the practically obtained encoding [46]:

$$q_{\mathit{\theta },t}\left(\mathit{z}\right)={\mathbb{E}}_{p_{{\tilde{D}}_t}\left(\tilde{\mathit{x}}\right)}\left[q_{\mathit{\theta },t}\left(\mathit{z}\right|\tilde{\mathit{x}})\right]\approx \frac{1}{{\tilde{N}}_t}\sum _{n=1}^{{\tilde{N}}_t}{q}_{\mathit{\theta },t}\left(\mathit{z}\right|{\tilde{\mathit{x}}}^{\left(n\right)})$$

(4)

To provide intuition, we illustrate this prior-posterior discrepancy on the obtained two-dimensional latent encodings for a continually trained supervised MNIST (Modified National Institute of Standards and Technology database) [47] model in Figure 2. Here, we can make two observations: to preserve the inherent data structure, the aggregate posterior deviates from the prior. In fact, this is further amplified by the imposed necessity for linear class separation and the beta term in Equation (1); however, we note that the discrepancy is desired even in completely unsupervised scenarios [45,46].

The underlying rationale is that there needs to be a balance in the effective latent encoding overlap [48], which can best be summarized with a direct quote from the recent work of Mathieu et al. [49]: “The overlap is perhaps best understood by considering extremes: with too little the latents effectively become a lookup table; too much, and the data and latents do not convey information about each other. In either case, meaningfulness of the latent encodings is lost." (p. 4). Additional discussion on the role of beta can be found in Appendix A.2.

Thus, the generated data from low-density regions of the aggregate posterior do not generally correspond to the encountered data instances. Conversely, data instances that fall into high-density regions under the prior should not generally be considered as statistical inliers with respect to the observed data distribution; recall Figure 2. This boundary between low- and high-density regions forms the basis for a natural connection between open-set recognition and continual learning: generate from high-density regions and reject novel instances that fall into low-density regions.

Ideally, we could find a solution by replacing the prior in the KL divergence of Equation (1) with $q_{\mathit{\theta },t}\left(\mathit{z}\right)$ and, respectively, sampling $z\sim q_{\mathit{\theta },t-1}\left(\mathit{z}\right)$ in Equations (2) and (3). Even though using the aggregate posterior as a subsequent prior is the objective in multiple recent works, it can be challenging in high dimensions, lead to over-fitting or come at the expense of additional hyper-parameters [45,50,51]. To avoid finding an explicit representation for the multi-modal $q_{\mathit{\theta },t}\left(\mathit{z}\right)$, we draw inspiration from the EVT-based OpenMax approach [12] in deep neural networks. However, instead of using knowledge about extreme distances in penultimate layer activations to modify a Softmax prediction, we now propose to apply EVT on the basis of the class conditional aggregate posterior.

In this view, any sample can be regarded as statistically outlying if its distance to the classes’ latent mean is extreme with respect to what has been observed for the majority of correctly predicted data instances, i.e., the sample falls into a region of low density under the aggregate posterior and is less likely to belong to $p_{\tilde{D}}\left(\tilde{\mathit{x}}\right)$. For convenience, let us introduce the indices of all correctly classified instances at the end of task t as $m=1,\dots ,{\tilde{M}}_t$. To obtain bounds on the aggregate posterior, we first define the mean latent vector for each class for all correctly predicted seen data instances ${\overline{\mathit{z}}}_{c,t}$ and the respective set of latent distances as

$${\Delta }_{c,t}\equiv {\left\{f_d\left({\overline{\mathit{z}}}_{c,t},{\mathbb{E}}_{q_{\mathit{\theta },t}\left(\mathit{z}\right|{\tilde{\mathit{x}}}_t^{\left(m\right)})}\left[\mathit{z}\right]\right)\right\}}_{m\in {\tilde{M}}_{c,t}}\mathrm{with}{\overline{\mathit{z}}}_{c,t}=\frac{1}{|{\tilde{M}}_{c,t}|}\sum _{m\in {\tilde{M}}_{c,t}}{\mathbb{E}}_{q_{\mathit{\theta },t}\left(\mathit{z}\right|{\tilde{\mathit{x}}}_t^{\left(m\right)})}\left[\mathit{z}\right].$$

(5)

Here, $f_d$ signifies a choice of distance metric. We proceed to model this set of distances with a per class heavy-tail Weibull distribution ${\mathit{\rho }}_{c,t}=({\tau }_{c,t},{\kappa }_{c,t},{\lambda }_{c,t})$ on ${\Delta }_{c,t}$ for a given tail-size $\eta$. As these distances are based on the class conditional approximate posterior, we can thus bound the latent space regions of high density. The tightness of the bound is characterized through $\eta$, that can be seen as a prior belief with respect to the outlier quantity assumed to be inherently present in the data distribution. The choice of $f_d$ determines the nature and dimensionality of the obtained distance distribution. For our experiments, we find that the cosine distance and thus a univariate Weibull distance distribution per class seems to be sufficient. Using the cumulative distribution function of this Weibull model ${\mathit{\rho }}_t$ we can now estimate any sample’s outlier (or inlier) probability:

$${\omega }_{\mathit{\rho },t}\left(\mathit{z}\right)=\min \left(1-exp{\left(-\frac{|f_d\left({\overline{\mathit{z}}}_t,\mathit{z}\right)-{\mathit{\tau }}_t|}{{\mathit{\lambda }}_t}\right)}^{{\mathit{\kappa }}_t}\right),$$

(6)

where the minimum returns the smallest outlier probability across all classes. If this outlier probability is larger than a prior rejection probability ${\mathsf{\Omega }}_t$, the instance can be considered as unknown. Such a formulation, which we term open variational auto-encoder (OpenVAE), now provides us with the means to learn continually and identify unknown data:

• For a novel data instance, Equation (6) yields the outlier probability based on the probabilistic encoder $\mathit{z}\sim q_{\mathit{\theta },\mathit{t}}\left(\mathit{z}\right|\mathit{x})$, and a false overconfident classifier prediction can be avoided.
• To mitigate catastrophic interference, Equation (6) can be used on top of $\mathit{z}\sim p\left(\mathit{z}\right)$ to constrain the generative replay (Equation (3)) to the aggregate posterior thus avoiding the need to sample it directly.

To give an illustration of the benefits, we show the generated MNIST [47] and larger resolution flower images [52] together with their outlier percentage in Figure 3. In practical application, we discard the ambiguous examples that are due to low-density regions and thus a high outlier probability. Even though we conduct sampling with rejection, note how this is computationally efficient, as we only need to calculate the heavy probabilistic decoder for accepted statistically inlying examples, and sampling from the prior with computation of Equation (6) is almost negligible in comparison.

3. Results

Instead of presenting a single experiment for continual learning in the constant presence of outlying non-task data, we chose to empirically corroborate our proposed approach in two experimental parts. The first section is dedicated to out-of-distribution detection, where we demonstrate the advantages of EVT in our generative model formulation. We then proceed to showcase how catastrophic interference is also mitigated by confining generative replay to aggregate posterior inliers in class incremental learning.

We emphasize that whereas the sections are presented individually, our approach’s uniqueness lies in using a core underlying mechanism to unify both challenges simultaneously. The rationale behind choosing this form of presentation is to help readers better contextualize the contribution of OpenVAE with the existing literature as, to the best of our knowledge, there exists no present other work that yields adequate continual classification accuracy while being able to robustly recognize unknown data instances. As such, we will now see that existing continual-learning approaches provide no suitable mechanism to overcome the challenge of providing robust predictions when data outside the known benchmark set are included.

3.1. Open Set Recognition

We experimentally highlight OpenVAE’s ability to distinguish unknown task data from data belonging to known tasks to avoid overconfident false predictions.

Experimental Set-Up and Evaluation

In summary, our goal is two-fold. The typical goal is to train on an initial task and correctly classify the held-out or unseen test data for this task. That is, we desire a large average classification test accuracy. In addition to this, in order to ensure that this classification is robust to unknown data, we now additionally desire to have a large value for a second kind of accuracy. Our simultaneous goal is to consider all test data of already trained tasks as inlying, while successfully identifying 100% of completely unknown datasets as outliers.

For this purpose, we evaluate OpenVAE’s and other models’ capability to distinguish the in-distribution test set of a respectively trained MNIST (Modified National Institute of Standards and Technology database) [47], FashionMNIST [53], AudioMNIST [54] from the other two and several unknown datasets: Kuzushiji-MNIST (KMNIST) [55], Street-View House Numbers (SVHN) [56] and Canadian Institute for Advanced Research (CIFAR) datasets (in both versions with 10 and 100 classes) [57]. Here, the (Fourier-transformed) audio data is included to highlight the extent of the challenge, as not even a different modality is easy to detect without our proposed approach. In practice, we evaluate three criteria according to which a decision of whether a data instance is an outlier can be made:

• The classifier’s predictive entropy, as recently suggested to work surprisingly well in deep networks [58] but technically well known to be overconfident [3]. The intuition here is that the predictive entropy $-{\sum }_{y\in C}p\left(y\right|x)logp\left(y\right|x)$ considers the probability of all other classes and is at a maximum if the distribution is uniform, i.e., when the confidence in the prediction is low.
• The generative model’s obtained negative log-likelihood, to concur with previous findings [5,6] on overconfidence in generative models. On the basis of Equation (1), the intuition is that the negative log-likelihood should be much larger for unseen data.
• Our suggested OpenVAE aggregate posterior-based EVT approach, according to the outlier likelihood introduced Equation (6).

Results

Figure 4 provides a qualitative intuition behind the three criteria and respective percentage of the total dataset being considered as outlying for FashionMNIST. Consistent with Nalisnick et al. [6], we can observe that the use of reconstruction loss can sometimes distinguish between the known tasks’ test data and unknown datasets but results in failure for others. In the case of the classifier predictive entropy, depending on the exact choice of entropy threshold, generally only a partial separation can be achieved. Furthermore, both of these criteria pose the additional challenge of the results being highly dependent on the choice of the precise cut-off value. In contrast, the test data from the known tasks is regarded as inlying across a wide range of rejection priors ${\mathsf{\Omega }}_t$ for Equation (6), and the majority of other datasets is consistently regarded as outlying by our introduced OpenVAE approach.

Corresponding quantitative outlier detection accuracies are provided in

Table 1. Outlier detection values of the joint model and separate discriminative and generative models (denoted as “CNN + VAE”; discriminative convolutional neural network and variational auto-encoder), when considering 95% of the known tasks’ validation data as inlying. The percentage of detected outliers is reported based on the classifier predictive entropy, reconstruction negative log-likelihood (NLL) and our posterior-based extreme-value theory approach. Note that larger values are better, except for the test data of the trained dataset, where ideally 0% should be considered as outlying. The outlier detection values have additionally been color coded, where worse results appear in red. A deeper shading thus indicates a method’s failure to robustly recognize unknown data as such. With this color coding, we can easily see how MNIST appears to be an easy to identify dataset for all approaches; however, we notice right away that our OpenVAE is the only method (row) that does not have a single red value for any dataset combination. In fact, the lowest outlier detection accuracy of OpenVAE is a very high 94.76%.

Outlier Detection at 95% Validation Inliers (%)				MNIST	Fashion	Audio	KMNIST	CIFAR10	CIFAR100	SVHN
Trained	Model	Test Acc.	Criterion
MNIST	Dual,	99.40	Class entropy	4.160	90.43	97.53	95.29	98.54	98.63	95.51
	CNN +		Reconstruction NLL	5.522	99.98	99.97	99.98	99.99	99.96	99.98
	VAE		OpenMax	4.362	99.41	99.80	99.86	99.95	99.97	99.52
	Joint	99.53	Class entropy	3.948	95.15	98.55	95.49	99.47	99.34	97.98
	VAE		Reconstruction NLL	5.083	99.50	99.98	99.91	99.97	99.99	99.98
			OpenVAE (ours)	4.361	99.78	99.67	99.73	99.96	99.93	99.70
FashionMNIST	Dual,	90.48	Class entropy	74.71	5.461	69.65	77.85	24.91	28.76	36.64
	CNN +		Reconstruction NLL	5.535	5.340	64.10	31.33	99.50	98.41	97.24
	VAE		OpenMax	96.22	5.138	93.00	91.51	71.82	72.08	73.85
	Joint	90.92	Class Entropy	66.91	5.145	61.86	56.14	43.98	46.59	37.85
	VAE		Reconstruction NLL	0.601	5.483	63.00	28.69	99.67	98.91	98.56
			OpenVAE (ours)	96.23	5.216	94.76	96.07	96.15	95.94	96.84
AudioMNIST	Dual,	98.53	Class entropy	97.63	57.64	5.066	95.53	66.49	65.25	54.91
	CNN +		Reconstruction NLL	6.235	46.32	4.433	98.73	98.63	98.63	97.45
	VAE		OpenMax	99.82	78.74	5.038	99.47	93.44	92.76	88.73
	Joint	98.57	Class entropy	99.23	89.33	5.731	99.15	92.31	91.06	85.77
	VAE		Reconstruction NLL	0.614	38.50	3.966	36.05	98.62	98.54	96.99
			OpenVAE (ours)	99.91	99.53	5.089	99.81	100.0	99.99	99.98

. To find thresholds for the sensitive entropy and reconstruction curves, we used a $5\%$ validation split to determine the respective value at which $95\%$ of the validation data is considered as inlying before using these priors to determine outlier counts for the known tasks’ test set as well as other datasets. In an intuitive picture, we “trace” the solid green curve of Figure 4 for a validation set of the originally trained dataset, check where we intersect with the x-axis for a y-axis value of 5% and then fix the corresponding criterion’s value at this point as an outlier rejection threshold for testing. We then report the percentage of the test set being considered as an outlier, together with the percentage for various unknown datasets. In the table, we additionally extend our intuition of Figure 4 to now further investigate what would happen if we had not trained a single VAE model that learned reconstruction and classification according to Equation (1) but separate models. For this purpose, we also investigate a dual model approach, i.e., a purely discriminative deep-neural-network-based classifier and a separate unsupervised VAE (Equation (1) without blue terms).

In this way, we can showcase the advantages of a generative modeling formulation that considers the joint distribution $p(\mathit{x},\mathit{y})$ in conjunction with EVT. For instance, we can compare our values with the purely discriminative OpenMax EVT approach [59]. At the same time, this provides a justification for why the existing continual-learning approaches of the next section, especially those relying on the maintenance of multiple models, are non-ideal, as they cannot seem to adequately solve the open-set challenge.

In terms of the obtained results, with the exception of MNIST, which appears to be an easy to identify dataset for all approaches, we can make two key observations:

• Both EVT approaches generally outperform the other criteria, particularly for our suggested aggregate posterior-based OpenVAE variant, where a near perfect open-set detection can be achieved.
• Even though EVT can be applied to purely discriminative models (as in OpenMax), the generative OpenVAE model trained with variational inference consistently exhibited more accurate outlier detection. We posit that this robustness is due to OpenVAE explicitly optimizing a variational lower bound that considers the data distribution $p\left(\mathit{x}\right)$ in addition to a pure optimization of features that maximize $p\left(y\right|\mathit{x})$.

Open Set Recognition with Monte-Carlo Dropout Based Uncertainty

One might be tempted to assume that the trained weights of the individual deep neural network encoder layers are still deterministic and the failure of predictive entropy as a measure for unseen unknown data could thus primarily be attributed to uncertainty not being expressed adequately. Placing a distribution on the weights, akin to a fully Bayesian neural network, would then be expected to resolve this issue. For this purpose, we further repeat all of our experiments by treating the model weights as the random variable being marginalized through the use of Monte-Carlo Dropout (MCD) [42]. Accordingly, the models were re-trained with a Dropout probability of $0.2$ in each layer. We then conducted 50 stochastic forward passes through the entire model for prediction. The obtained open-set recognition results are reported in

Table 2. Outlier detection values of the joint model and separate discriminative and generative models (denoted as “CNN + VAE”; discriminative convolutional neural network and variational auto-encoder), when considering 95% of known tasks’ validation data is inlying. The percentage of detected outliers is reported based on classifier predictive entropy, reconstruction negative log-likelihood (NLL) and our posterior-based EVT approach. In contrast to Table 1, the results are now averaged over 50 Monte-Carlo dropout samples, with $p_{dropout}=0.2$ for each layer, per data-point, respectively, to assess the model uncertainty. Note that larger values are better, except for the test data of the trained dataset, where ideally 0% should be considered as outlying. The color coding is analogous to Table 1.

Outlier Detection at 95% Validation Inliers (%)				MNIST	Fashion	Audio	KMNIST	CIFAR10	CIFAR100	SVHN
Trained	Model	Test Acc.	Criterion
MNIST	Dual,	99.41	Class entropy	4.276	91.88	96.50	96.65	95.84	97.37	98.58
	CNN +		Reconstruction	4.829	99.99	100.0	99.90	100.0	100.0	100.0
	VAE		OpenMax	4.088	87.84	98.06	95.79	97.34	98.30	95.74
	Joint,	99.54	Class entropy	4.801	97.63	99.38	98.01	99.16	99.39	98.90
	VAE		Reconstruction	5.264	99.98	100.0	100.0	100.0	100.0	100.0
			OpenVAE (ours)	4.978	99.99	100.0	99.94	99.96	99.95	99.68
FashionMNIST	Dual,	90.58	Class entropy	75.50	5.366	70.78	74.41	49.42	49.17	38.84
	CNN +		Reconstruction NLL	55.45	5.048	59.99	99.83	99.35	99.35	99.62
	VAE		OpenMax	77.03	4.920	55.48	70.23	58.73	57.06	44.54
	Joint,	91.50	Class Entropy	85.05	4.740	67.90	78.04	63.89	66.11	59.42
	AE		Reconstruction	1.227	5.422	85.85	39.76	99.94	99.72	99.99
			OpenVAE (ours)	95.83	4.516	94.56	96.04	96.81	96.66	96.28
AudioMNIST	Dual,	98.76	Class entropy	99.97	61.26	4.996	96.77	63.78	65.76	59.38
	CNN +		Reconstruction NLL	7.334	52.37	5.100	98.19	99.97	99.90	99.96
	VAE		OpenMax	92.74	67.18	5.073	90.41	90.56	90.97	89.58
	Joint,	98.85	Class entropy	99.39	89.50	5.333	99.16	94.66	95.12	97.13
	VAE		Reconstruction NLL	15.81	53.83	4.837	41.89	99.90	99.82	99.95
			OpenVAE (ours)	99.50	99.27	5.136	99.75	99.71	99.59	99.91

.

Although MCD boosts the outlier detection accuracy, particularly for criteria, such as predictive entropy, the previous insights and drawn conclusions still hold. In summary, the joint generative model generally outperforms a purely discriminative model in terms of open-set recognition, independently of the used metric, and our proposed aggregate posterior-based EVT approach of OpenVAE yields an almost perfect separation of known and unseen unknown data. Interestingly, this was already achieved in the prior table without MCD. Resorting to the repeated model calculation of MCD thus appears to be without enough of an advantage to warrant the added computational complexity in the context of posterior-based open-set recognition, a further key advantage of OpenVAE.

3.2. Learning Classes Incrementally in Continual Learning

To showcase how our OpenVAE approach mitigates catastrophic interference in addition to successfully handling unknown data in robust prediction, we conduct an investigation of the test accuracy when learning classes incrementally.

Experimental Set-Up and Evaluation

We consider the incremental MNIST dataset (where classes arrive in groups of two) and the corresponding versions of the FashionMNIST and AudioMNIST datasets, similar to popular literature [11,21,22,32,34]. We re-emphasize that such a setting has a sole focus on mitigating catastrophic interference and does not account for the the challenges presented in the previous open-set recognition section, which we detail in the prospective discussion section. For a flexible comparison, we report our aggregate posterior-based generative replay approach in OpenVAE on both a simple multi-layer perceptron (MLP), as well as a deep convolutional neural network (CNN) based on wide residual networks (WRN). For the former, we follow previous continual-learning studies and employ a two-hidden-layer and 400-unit multi-layer perceptron [60]. For the latter, we use both encoder and decoder architectures of 14-layer wide residual networks [61,62] with a latent dimensionality of 60 [2,18]. For our statistical outlier rejection, we use a rejection prior of ${\mathsf{\Omega }}_t=0.01$ and dynamically set tail-sizes to 5% of seen examples per class.

For our own experiments, we report the mean and standard deviation of the average classification test accuracy across five experimental repetitions. If our re-implementation of related works achieved a better than original value, we report this number, otherwise the work that reported the specific best value is cited next to it. The full training details, including details on hardware and code, are supplied in Appendix A.4.

Results

In

Table 3. The accuracy ${\alpha }_T$ at the end of the last increment T = 5 for class incremental learning approaches averaged over five runs. For a fair comparison, if our re-implementation of related works achieved a better than original value, we report our number, otherwise the work that reported the specific best value is cited right next to the result. Intermediate results can be found in Appendix A.6.

	Final Accuracy ${\mathit{\alpha }}_{\mathit{T}}(\mathit{T}=5)$ [%]
Method	MNIST	FashionMNIST	AudioMNIST
MLP upper bound	98.84	87.35	96.43
WRN upper bound	99.29	89.24	97.87
EWC [22]	55.80 [63]	24.48 ± $2.86$	20.48 ± $1.73$
DGR [32]	75.47 [64]	63.21 ± $1.96$	48.42 ± $2.81$
VCL [36]	72.30 [35]	32.60 [35]	-
VGR [35]	92.22 [35]	79.10 [35]	-
Supervised VAE	60.88 ± $3.31$	62.72 ± $1.38$	69.76 ± $1.37$
OpenVAE—MLP	87.31 ± $1.22$	66.14 ± $0.50$	81.84 ± $1.44$
OpenVAE—WRN	93.24 ± $3.74$	69.88 ± $1.71$	87.72 ± $1.59$
OpenPixelVAE	96.84 ± $0.35$	80.85 ± $0.72$	90.23 ± $1.14$

, we report the final accuracy after having trained on each of the five increments. For an overall reference, we provide the achievable upper-bound continual-learning performance, i.e., accumulating all data over time and optimizing Equation (1). We can observe that our proposed OpenVAE approach provides significant improvement over generative replay with a conventional supervised VAE. In comparison with the immediately related works, our approach surpasses variational continual learning (VCL) [36], an approach that employs a full Bayesian neural network (BNN), with the additional benefit that our approach scales trivially to complex network architectures.

In contrast to variational generative replay (VGR) [34], OpenVAE initially appears to fall short. This is not surprising as VGR trains a separate GAN on each task’s aggregate posterior, an apples to oranges comparison considering that we only use a single model. Nevertheless, even in a single model, we can surpass the multi-model VGR by leveraging recent advancements in generative modeling, e.g., by making the neural architecture more complex or augmenting our decoder with autoregressive sampling [2,18] (a complementary technique to OpenVAE, often also called PixelVAE and summarized in Appendix A.3).

At the bottom of Table 3, we can see that this significantly improves upon the previously obtained accuracy. The full accuracies, along with other metrics per dataset for all intermediate steps can be found in Appendix A.6.

High-Resolution Flower Images

While the main goal of this paper is not to push the achievable boundaries of generation, we take this argument one step further and provide empirical evidence that our suggested aggregate posterior-based EVT sampling provides similar benefits when scaling to higher resolution color images. For this purpose, we consider the additional flowers dataset [52] at a resolution of $256\times 256$, investigated with five classes and increments of one class per step [65,66].

In addition to autoregressive sampling, we also include a second complementary generative modeling improvement here, called VAEs with introspection (IntroVAE) [19]. A technical description of PixelVAE and IntroVAE is detailed in Appendix A.3. For each generative modeling variant, including autoregression and introspection, we report the degradation of accuracy over time in Figure 5 and demonstrate how their respective open-set-aware version provides substantial improvements. Intuitively, this improvement is due to an increase in the visual generation quality; see the examples in the earlier Figure 3.

First, it is apparent how every OpenVAE variant improves upon its non open-set aware counterpart. We further observe that the best version, OpenIntroVAE, appears to be in the same ballpark as complex recent GAN approaches [65,66], even though they do not solve the open-set recognition challenge and conduct a simplified evaluation. The latter works use a lower resolution of $128\times 128$ (we were unable to scale to satisfying results at higher resolution) with additional distillation mechanisms, a continuously trained generator but a classifier that is trained and assessed only once at the end. We nevertheless report the respective values for intuition. We conclude that the obtained final accuracy can be competitive and is remarkably close to the achievable upper bound. A suspected initial VAEs generation quality limitation appears to be lifted with modern extensions and our proposed sampling scheme.

We also support our quantitative statements visually with a few selected generated images for the various generative variants in Figure 6. We emphasize that these examples are supposed to primarily provide visual intuition in support of the existing quantitative results, as it is difficult to draw conclusions from a perceived subjective quality from a few images alone. From a qualitative viewpoint, the OpenVAE without generative modeling extensions appears to suffer from the limitations of a traditional VAE and generates blurry images.

However, our open-set approach nevertheless provides a clearer disambiguation of classes, particularly already at the stage of task 2. The addition of introspection significantly increases the image detail, albeit still degrades considerably due to ambiguous interpolations in samples from low-density areas outside the aggregate posterior. This is again resolved by combining introspection with our proposed posterior-based EVT approach, where image quality is retained across multiple generative replay steps. From a purely visual perspective it is clear why this model outperforms the other approaches significantly in terms of quantitative accuracy values.

Interestingly, our visual inspection also hints at why the PixelVAE and its open-set variant perform much worse than perhaps initially expected. As the caveat is the same in both PixelVAE and OpenPixelVAE, we only show generated instances for the latter. From these samples, we can hypothesize why the initial performance is competitive but rapidly declines. It appears that the autoregression suffers from forgetting in terms of its long-range pixel dependency.

Whereas at the beginning, the information is locally consistent across the entire image, in each consecutive step, a further portion of subsequent pixels for old tasks is progressively replaced with uncorrelated noise. The conditioning thus appears to primarily be captured on new tasks only, resulting in interference effects. We continue this discussion alongside potential other general limitations of generative modeling variant choices in Appendix A.5.

4. Discussion

As a final piece of discussion, we would like to recall and emphasize a few important points of how our results should be interpreted and contextualized.

4.1. Presence of Unknown Data and Current Benchmarks

Perhaps most importantly, we re-iterate that OpenVAE is unique in that it provides a grounded basis to conduct continual learning in the presence of unknown data. However, as evidenced from the quantitative open-set recognition results, the inclusion of unknown data instances into continual learning would immediately result in the failure of the present continual-learning approaches at this point, simply because they lack a principled mechanism to provide robust predictions. For this reason, we show traditional incremental classification results as a proxy to assess our improved aggregate posterior-based generation quality.

Our class incremental accuracy reports in this paper should thus be interpreted with caution as they represent only a part of OpenVAE’s capability, similar to a typical ablation study. We nevertheless provided this type of comparison, in order to situate OpenVAE with respect to some existing generative continual-learning methods in terms of catastrophic forgetting, rather than presenting OpenVAE in isolation in a more realistic new setting.

4.2. State of the Art in Class Incremental Learning and Exemplar Rehearsal

Following the above subsection, we note that a fair comparison of realistic class incremental learning is further complicated due to various involved factors. In fact, multiple related works make various additional assumptions on the extra storage of explicit data subsets and the use of multiple generative models per task or even multiple classifiers. We do not make these assumptions here in favor of generality. In this spirit, we focused our evaluation on our contributions’ relevant novelty with respect to combining the detection of unknown data with the prevention of catastrophic forgetting in generative models.

The introduced OpenVAE shows that both are achievable simultaneously. At the same time, the reader familiar with the recent continual-learning literature will likely notice that some modern approaches that are attributed with state of the art in class incremental learning have not been included in our comparison. These approaches all fall into the category of exemplar rehearsal. We would like to emphasize that this is deliberate and not out of ignorance, as we see these works as purely complementary. We nevertheless wish to give deserved credit to these works and provide an outlook to one future research direction.

The primary reason for omitting a direct comparison with state of the art works in continual learning that employ exemplar rehearsal is that we believe such a comparison would be misleading. In fact, contrasting our OpenVAE against these works would imply that these methods are somehow competing. In reality, exemplar rehearsal, or the so called extraction of core sets, is an auxiliary mechanism that can be applied out-of-the-box to our experimental set-up in this work. The main premise here is that catastrophic forgetting in continual learning can be reduced by retaining an explicit subset of the original data and subsequently continuously interleaving this stored data into the training process.

Early works, such as iCarl [26] show that performance is then a function of two key aspects: the data selection technique and the memory buffer size. The former, selection of an appropriate data subset, essentially boils down to a non-continual-learning question, i.e., how to approximate the entire distribution through only a few instances. Exemplar rehearsal works thus make use of existing techniques here, such as core sets [28], herding [67], nearest mean-classifiers [27] or simply picking data samples uniformly at random [68].

The second question, on memory buffer size, has an almost trivial answer. The larger the memory buffer size, the better the performance. This is intuitive, yet also makes comparison challenging, as a memory buffer of the size of the entire dataset is analogous to what we referred to as “incremental upper bound” in our experiments. If we were to simply store the complete dataset, then catastrophic forgetting would be avoided entirely. Modern class incremental learning works make heavy use of this fact and store large portions of the original data, showing that the more data is stored, the higher the performance goes.

Primary examples include the recent works on Mnemonics Training [69], Contrastive Continual Learning (Co2L) [70] or Dark Experience Replay (DER) [71]. We do not wish to dive into a discussion here of whether or not such data storage is realistic or what size of a memory buffer should be assumed. A respective reference that questions and discusses whether storing of original data is synonymous with progress in continual learning is Greedy Sampler and Dumb Learner (GDumb) [68], where it is shown that the amount of extracted data alone amounts to a significant portion of “state-of-the-art” performance.

Primarily, we point out that the latter works all show that a larger memory buffer shows “better” class incremental learning performance, i.e., less forgetting. However, most importantly, extracting and storing parts of the original data into a separate memory buffer is an auxiliary process that is entirely complementary to our propositions of OpenVAE. As such, each of the methods referenced in this subjection is straightforward to combine with our work. Although we see such a combination as important prospective work, we leave detailed experimentation up to future investigations.

The rationale behind this choice is that inclusion of a memory buffer will inevitably additionally boost the performances of the results of Table 3, yet provide no additional insights to our main hypothesis and contribution: the proposition of OpenVAE to show that detection of unknown data for robust prediction can effectively be achieved alongside reduction of catastrophic forgetting in continual learning.

5. Conclusions

We proposed an approach to unify the prevention of catastrophic interference in continual learning with open-set recognition based on variational inference in deep generative models. As a common denominator, we introduced EVT-based bounds to the aggregate posterior. The correspondingly named OpenVAE was shown to achieve compelling results in being able to distinguish known from unknown data, while boosting the generation quality in continual learning with generative replay.

We believe that our demonstrated benefits from recent generative modeling techniques in the context of high-resolution flower images with OpenVAE provide a natural synergy to be explored in a range of future applications. We envision prospective works to employ OpenVAE as a baseline when relaxing the closed-world assumption in continual learning and allowing unknown data to appear in the investigated benchmark streams at all times in the move to a more realistic evaluation.