Learning a Video Neural Memory with Bias Mitigation for Video Class-Incremental Learning

Abstract

Video class-incremental learning retains old knowledge while learning new actions under limited replay memory. Temporal compression stores more videos but may lose temporal details and introduce discrepancies between compressed and original videos. We propose Video Neural Memory with Bias Mitigation, combining a Spatio-Temporal Neural Memorizer and Contrastive Bias Correction. The Spatio-Temporal Neural Memorizer preserves temporal and task-related information in compact neural video representations. Contrastive Bias Correction uses class-wise adaptation prompts and contrastive learning to reduce compression-induced feature bias. Experiments on three action recognition benchmarks show higher average incremental accuracy than the compared methods under the same replay-memory budgets, including a gain of 9.18% over the strongest baseline on Something-Something V2 with 18 steps.

1. Introduction

Action recognition is a fundamental task in video understanding and has been widely studied using different data modalities, including videos [1] and sensor data collected from mobile devices [2]. However, current action recognition models are required to be finetuned on continuous video task sequences in practical scenarios, leading to a sharp decrease in the performance of previous classes, a phenomenon known as the catastrophic forgetting problem [3].

Video class-incremental learning methods explore strategies to mitigate the catastrophic forgetting problem when training an action recognition model on a sequence of independent classes. Recent video class-incremental learning methods demonstrate that storing a portion of old class videos as memory is effective in overcoming the catastrophic forgetting problem. TCD [4] stores original video frames from old tasks for memory replay to mitigate catastrophic forgetting. HCE [5] proposes a correlation refinement mechanism that uses old task memory and current task data to reduce the impact of the imbalance between old and current classes. Nevertheless, a video consists of multiple frames, which requires a large memory budget. To reduce the memory cost of videos, vCLIMB [6], VCIL-CT [7], and STSA-DFS [8] downsample the original videos along the temporal dimension. SMILE [9] stores a single frame for each original video and repeats it as a video. FrameMaker [10] constructs a condensed frame for each video to further reduce the memory cost, as shown in Figure 1a. However, these methods sacrifice the temporal details during compression, resulting in temporal misalignment during video class-incremental learning. In addition, the domain bias between the condensed frames and the original videos increases as the feature space of the incremental model changes, resulting in confusion between old and current tasks in the feature space of the current task.

Figure 1. The comparison between VNMBM and FrameMaker (a) FrameMaker [10]. (b) VNMBM.

To address the aforementioned problems, we present a novel method, Video Neural Memory with Bias Mitigation (VNMBM), to maintain the temporal details of compressed videos in memory via a Spatio-Temporal Neural Memorizer (STNM) and correct domain bias through Contrastive Bias Correction (CBC), as shown in Figure 1b. Specifically, the STNM encodes spatio-temporal features by combining a spatial prototype with temporal indexes and decodes these features into compressed videos. As the original old task videos are inaccessible in the following tasks, we estimate their features via the knowledge distillation loss between the original video features and spatio-temporal features, allowing approximate original old class video features to be accessible in the following tasks. Additionally, we adopt a pixel reconstruction loss to capture the spatial details of the original videos. Therefore, the spatio-temporal details are maintained in the STNM, and the approximate original old class video features and compressed old class videos can be obtained from the old class STNM in the current task.

Moreover, we propose CBC to correct the current task model, which narrows the distance between compressed videos and original videos in the feature space of the current task to mitigate the domain bias and widens the distance between old and current task videos in the feature space of the current task to enhance the classification. Due to the lack of original old task videos in the current task, CBC first computes the approximate original old task video features via the STNM to represent the original old task video. As the original old task video features are computed in the old task feature space, we design learnable class-wise adaptation prompts to ensure the adaptation of approximate original old task video features to the current task feature space. Subsequently, CBC contrastively aligns and discriminates the video features in the current task feature space through a debiased contrastive loss. Moreover, the classification loss is designed to further correct the classifier. Therefore, domain bias between the compressed and original videos can be mitigated via CBC, improving the classification of old and current tasks.

The contributions of this paper can be summarized as follows:

• We introduce a memory-efficient STNM for video class-incremental learning, which can store compressed videos and approximate video features using a compact neural representation.
• We propose CBC to align the original and compressed video features and discriminate between the video features of old and current tasks in the feature space of the current task, thus mitigating the domain bias of the current task model.
• The proposed method outperforms existing methods under the same memory budget on three public benchmark datasets: UCF101, HMDB51 and Something-Something V2.

The remainder of this paper is organized as follows: We first review the related works in Section 2. Subsequently, we introduce the details of our proposed VNMBM in Section 3, following which we present the experiments and the analysis of the experimental results in Section 4. Finally, we provide a conclusion for our method in Section 5.

2. Related Work

This section reviews the literature related to our work from three aspects. We first introduce neural representations for videos as the basis for compact video memory representation, then review class-incremental learning methods for mitigating catastrophic forgetting, and finally discuss video class-incremental learning methods related to memory replay and video compression.

2.1. Neural Representations for Videos

The task of neural representation has gained increasing attention, aiming to train neural networks to regress the mapping between coordinate-like input and a given image or video. It has been widely applied in shape regression [11,12], image deblurring [13,14] and video compression. For video compression, some methods still rely on coordinates. MANIVR [15] proposes a phase-shift generation network to guide the frame-generation process. NIRVANA [16] divides videos into groups of frames and trains separate neural networks to perform a patch-wise prediction. Instead of taking coordinate-like inputs, NeRV [17] takes the frame index as the input to train the neural network representing videos. E-NeRV [18] improves NeRV by incorporating a disentangled spatio-temporal context. EC-NeRV [19] proposes a novel convolutional architecture that models the entropy of the weights and adopts quantization-aware training. Meanwhile, hybrid methods can represent videos via embeddings and decoders. HNeRV [20] uses content-adaptive embeddings as the input of the decoder, while DNeRV [21] proposes a collaborative content unit to merge the content embedding with the difference embedding. NeRV-Boost [22] introduces a temporal-aware affine transform module to improve representation capabilities and integrates a consistent entropy minimization technique to ensure the consistency between training and inference. Nevertheless, most existing neural video representation methods primarily focus on reconstructing the visual content of videos and are optimized mainly with reconstruction-oriented supervision. While this design is suitable for video representation and reconstruction, it does not specifically preserve the task-related feature representations required for video class-incremental learning. When neural representations are used as replay memory, the reconstructed old-task videos may therefore exhibit feature discrepancies from the original videos, introducing domain bias during incremental learning. In contrast, our STNM retains pixel-level reconstruction supervision while further introducing feature-level knowledge distillation and spatial prototypes to preserve the feature information of the original videos, thereby reducing the discrepancy between reconstructed and original video representations.

2.2. Class-Incremental Learning

Many class-incremental learning methods have been proposed to overcome the catastrophic forgetting problem. Existing methods mainly fall into three categories: regularization-based methods, memory-based methods and distillation-based methods.

• Regularization-Based Methods. These methods estimate the importance of model parameters and penalize the changes in the parameters associated with old tasks. EWC [23] applies a penalty based on the Fisher information matrix. PI [24] designs a synaptic framework to estimate importance weights. MAS [25] learns the importance of parameters in an unsupervised manner via the sensitivity of the output function. R-walk [26] improves EWC using a moving average and combines it with a modified PI, which uses the approximate Kullback–Leibler (KL) divergence to compute sensitivity. spWC [27] measures the discriminative contributions of old tasks via self-paced regularization.
• Memory-Based Methods. Memory-based methods focus on building the memory of old classes for memory replay. iCaRL [28] uses a herding strategy to construct memory. Mnemonics [29] parameterizes the memory using image-size parameters and then optimizes them and the model using bilevel optimization. MRDC [30] examines the trade-off between memory quality and quantity using JPEG [31] for data compression. CIM [32] proposes an adaptive mask generation model to construct 0-1 masks for discriminative pixels and downsample non-discriminative pixels. SRP [33] maintains feature distribution through class-feature sketching and projects examples into a lower-dimensional space via replay buffer projection.
• Distillation-Based Methods. These methods use knowledge distillation to transfer knowledge from the old model to the current model. LwF [34] first applies knowledge distillation in class-incremental learning by modifying the cross-entropy loss to align the outputs of the old model with those of the current model. Based on LwF, iCaRL [28] designs a nearest-mean-of-exemplars classifier for classification. LwM [35] utilizes an attention map for the attention distillation loss between the old model and the current model. To reduce feature and embedding drift, UCIR [36] proposes a less-forget constraint on features to preserve the previously learned knowledge. PODNet [37] further investigates feature constraints and introduces a pooled outputs distillation loss applied to both the embeddings and the outputs of intermediate layers. AFC [38] estimates the importance of each feature map for feature distillation. DKT [39] proposes a cluster-separation loss to encourage feature discrimination between old and new tasks. GR [40] proposes a distribution-aware knowledge distillation loss to alleviate the imbalanced catastrophic forgetting problem. Additionally, HFC [41] introduces a gradient-balanced relation distillation loss alongside a gradient-balanced forgetting compensation loss to tackle the heterogeneous forgetting speeds of different old classes.

2.3. Video Class-Incremental Learning

Although many class-incremental learning methods have been proposed in the image domain, video class-incremental learning methods have been rarely researched. TCD [4] uses a herding strategy to sample original videos for memory and proposes time-channel importance masks to facilitate time-channel distillation. HCE [5] proposes a hierarchical aggregation strategy and a correlation refinement mechanism for the alignment of spatio-temporal features. On the other hand, to compress the videos in memory, vCLIMB [6] downsamples the original videos in the temporal dimension and proposes a temporal consistency loss for regularization. VCIL-CT [7] employs temporal downsampling to store more exemplar videos under a fixed memory budget and restores the input sequence length through frame duplication during memory replay. STSA-DFS [8] introduces a difference-based frame selector that leverages global-local temporal differences and inter-task representation drift to adaptively preserve discriminative frames in the memory buffer. SMILE [9] stores a single frame for each video in memory and repeats a single frame to form a video for incremental training. FrameMaker [10] generates a condensed frame with an instance-specific prompt for each video to reduce memory costs. Although these compression-based methods have advanced video class-incremental learning, they do so at the cost of temporal details. In addition, a domain bias exists between the condensed frames and the original videos. In contrast to these methods, our VNMBM compactly compresses the videos and video features into neural representations and mitigates domain bias via a debiased contrastive loss.

3. Method

This section presents the proposed VNMBM. We first formulate the video class-incremental learning problem and introduce the datasets and task settings, followed by an overview of the proposed framework. We then describe the STNM for compact video memory representation and the CBC for bias correction, and finally present the overall training pipeline.

3.1. Problem Formulation

Video class-incremental learning methods aim to train an action recognition model $\mathcal{F}\left(x\right)=h\left(f\right(x\left)\right)$, where x is the input video, f is the feature extractor and h is the final classification layer. The model is trained over a sequence of tasks $\{{\mathcal{T}}^1,{\mathcal{T}}^2,\cdots ,{\mathcal{T}}^k,\cdots {\mathcal{T}}^K\}$, where ${\mathcal{T}}^k=\{(x_i^k,y_i^k),y_i^k\in Y^k\}$ contains the videos whose labels belong to the pre-defined label set $Y^k$, and $Y^a\cap Y^{b\ne a}=\varnothing$. To prevent the catastrophic forgetting problem, memory-based methods construct a memory buffer $M^k={\{(x_{i,j}^k,y_i^k),y_i^k\in Y^k\}}_{j=1}^N$ to store N representative videos for each class $y_i^k$ after the incremental training step k. During the incremental training step k, the model ${\mathcal{F}}^k$ will be trained on ${\mathcal{T}}^{\prime k}={\mathcal{T}}^k\cup M^{1:k-1}$, where $M^{1:k-1}=M^1\cup M^2\cup \cdots \cup M^{k-1}$, and evaluated on all seen tasks without task boundaries.

3.2. Datasets

We evaluated our VNMBM on three standard action recognition datasets: UCF101 [42], HMDB51 [43] and Something-Something V2 [44]. The UCF101 dataset contains 13.3K videos covering 101 classes. The HMDB51 dataset contains 6.8K videos covering 51 classes. The Something-Something V2 dataset is a large-scale motion-sensitive dataset created by crowdsourcing over 174 classes, which contains 169K videos for training and 25K videos for testing.

We follow the video class-incremental settings of TCD. For UCF101, we train the initial model on 51 classes and divide the remaining 50 classes into 5, 10, and 25 incremental tasks, corresponding to 10 × 5 steps, 5 × 10 steps, and 2 × 25 steps, respectively. For HMDB51, the model is trained on 26 classes in the initial step, and the remaining 25 classes are divided into 5 and 25 incremental tasks, corresponding to 5 × 5 steps and 1 × 25 steps, respectively. For Something-Something V2, we first train the model on 84 classes, and the remaining 90 classes are divided into 9 and 18 incremental tasks, corresponding to 10 × 9 steps and 5 × 18 steps, respectively.

3.3. Overview

In this paper, we propose Video Neural Memory with Bias Mitigation (VNMBM), a novel video class-incremental learning method comprising two core components: a Spatio-Temporal Neural Memorizer (STNM) that preserves temporal details of compressed videos in memory during memory construction, and a Contrastive Bias Correction (CBC) that mitigates domain bias after memory construction. Specifically, at  incremental training step k, we first train an action recognition model ${\mathcal{F}}^k$ following TCD [4]. Then, we adopt a herding strategy to sample N original videos from each class of the current task. Instead of storing these original videos directly in memory, we train the proposed STNM for each class of the current task, which generates the compressed videos via the corresponding spatial prototypes of the original frames and the temporal indices. During the training of STNM, the knowledge distillation loss is used to align the original current task frame features and spatio-temporal features. After training the STNMs of the current task, we store the spatial prototype, the number of frames, and the STNM for each class of the current task. If $k>1$, we retrain ${\mathcal{F}}^k$ via CBC subsequent to the memory construction process. CBC first computes the spatio-temporal features as approximate original old task video features from the old task STNMs and introduces learnable class-wise adaptation prompts to adapt the approximate original old task video features to the current task feature space. Then, CBC uses a debiased contrastive loss to align and discriminate the video features in the current task feature space, and a classification loss to further correct the classifier. Thus, VNMBM achieves compact storage of compressed videos and their approximate video features through the STNM, while effectively mitigating domain bias via the CBC.

3.4. Spatio-Temporal Neural Memorizer

Existing video class-incremental methods [4,5,6,7,8,9,10] retain frame-like representations for memory replay. In contrast to existing methods, we propose the Spatio-Temporal Neural Memorizer (STNM) to construct the neural representations of the video memory buffer for more efficient memory replay. Given the temporal index and corresponding spatial prototype of the original frame, the compressed frame and approximate frame features can be constructed using the STNM, as shown in Figure 2.

Figure 2. Illustration of STNM.

After the incremental training step k of TCD [4], we use a herding strategy [28] to select N videos from each class to construct a memory buffer $M^k$ and sample B frames with a uniform time interval for each video. For class $y_i^k\in Y^k$, the original frames of the memory buffer are defined as $M_i^k=\{x_{i,1,1}^k,\cdots ,x_{i,j,q}^k,\cdots ,x_{i,N,B}^k\}\in {\mathbb{R}}^{T\times 3\times H\times W}$, where $x_{i,j}^k$ is the j-th video in the memory buffer of class $y_i^k$, $x_{i,j,q}^k$ is the q-th frame of video $x_{i,j}^k$ and $T=N\times B$.

For frame $x_{i,j,q}^k$, we first compute the temporal index $t=B\ast (j-1)+q-1$ and normalize t between $[0,1)$. Then, we use Positional Encoding (PE) [45] to obtain temporal embedding $\Gamma \left(t\right)\in {\mathbb{R}}^{1\times (2\times l)}$:

$$\begin{array}{cc}\hfill \mathrm{PE}\left(t\right)=\Gamma \left(t\right)=& (\sin \left(b^0\pi t\right),\cos \left(b^0\pi t\right),\dots ,\sin \left(b^{l-1}\pi t\right),\hfill \\ \hfill & \cos \left(b^{l-1}\pi t\right))\hfill \end{array}$$

(1)

where b and l are the hyperparameters. After computing the temporal embedding, we construct the spatial prototype from the trained feature extractor $f^k$. For a video $x_{i,j}^k$ of class $y_i^k$, the feature extractor produces $f^k\left(x_{i,j}^k\right)\in {\mathbb{R}}^{B\times C_p}$, where B denotes the number of sampled temporal positions and $C_p$ denotes the feature dimension. Each row of $f^k\left(x_{i,j}^k\right)$ therefore represents the feature at one sampled temporal position. The spatial prototype $p_i^k\in {\mathbb{R}}^{B\times C_p}$ of class $y_i^k$ is obtained by averaging the features of all training videos in this class:

$$\begin{array}{c}\hfill p_i^k={\displaystyle \frac{1}{{\mathcal{N}}_i^k}}{\displaystyle \sum _{j=1}^{{\mathcal{N}}_i^k}}{f}^k\left(x_{i,j}^k\right),\end{array}$$

(2)

where ${\mathcal{N}}_i^k$ is the number of videos in class $y_i^k$. The spatio-temporal encoder $E_i^k$ is designed to combine temporal embedding with the spatial prototype, and the spatio-temporal feature $O_{i,j,q}^k$ is computed as follows:

$$\begin{array}{c}\hfill O_{i,j,q}^k=E_i^k(t,p_{i,q}^k)={\mathrm{FC}}_i^{k,E}(\Gamma \left(t\right))\odot p_{i,q}^k\end{array}$$

(3)

where ${\mathrm{FC}}_i^{k,E}$ is a linear layer, and $p_{i,q}^k\in {\mathbb{R}}^{1\times C_p}$ denotes the q-th row of the spatial prototype $p_i^k\in {\mathbb{R}}^{B\times C_p}$, corresponding to the q-th sampled temporal position. When $C_p>C_1$, 1D max pooling downsamples $p_{i,q}^k$ from $C_p$ to $C_1$ dimensions before its element-wise multiplication with the $C_1$-dimensional encoder output. The resulting $C_1$-dimensional spatio-temporal feature is then fed into the decoder.

Subsequently, a linear layer ${\mathrm{FC}}_i^{k,D}$ is used to compute the linear projection $o_{i,j,q}^k\in {\mathbb{R}}^{1\times (C_2\times h\times w)}$ of the spatio-temporal feature $O_{i,j,q}^k$, which is stacked with four NeRV blocks [17] as the decoder $D_i^k$ to decode the spatio-temporal feature $O_{i,j,q}^k$ into a compressed frame ${\widehat{x}}_{i,j,q}^k$. These NeRV blocks utilize $3\times 3$ convolution layers, PixelShuffle layers [46] and swish activation layers [47] for upsampling. To adjust the channel dimension to 3, we add a $1\times 1$ convolution layer at the end of the final NeRV block. Therefore, the compressed frame ${\widehat{x}}_{i,j,q}^k$ of the compressed memory buffer ${\widehat{M}}_i^k$ is constructed from the normalized temporal index t, spatial prototype $p_{i,q}^k$ and STNM $G_i^k$ of class $y_i^k$:

$$\begin{array}{c}\hfill {\widehat{x}}_{i,j,q}^k=G_i^k(t,p_{i,q}^k)=D_i^k\left(E_i^k(t,p_{i,q}^k)\right)\end{array}$$

(4)

To reduce the domain bias between the compressed frame ${\widehat{x}}_{i,j,q}^k$ and the original frame $x_{i,j,q}^k$, we first use the combination of the L1 loss and the SSIM loss [17] as the pixel reconstruction loss to supervise the STNM at the pixel-level:

$$\begin{array}{cc}\hfill {\mathcal{L}}_{\mathrm{pixel}}=& {\displaystyle \frac{1}{T}}{\displaystyle \sum _{j=1}^N}{\displaystyle \sum _{q=1}^B}[\alpha {∥x_{i,j,q}^k-{\widehat{x}}_{i,j,q}^k∥}_1+\hfill \\ \hfill & (1-\alpha )(1-\mathrm{SSIM}(x_{i,j,q}^k,{\widehat{x}}_{i,j,q}^k))]\hfill \end{array}$$

(5)

where $\alpha$ is set to 0.7, ${\parallel ·\parallel }_1$ represents the L1 norm, $\mathrm{SSIM}(·,·)$ represents the structural similarity index measure and $T=N\times B$. In addition, we use the feature extractor $f^k$ to compute the original current task frame feature and design the knowledge distillation loss ${\mathcal{L}}_{\mathrm{kd}}$ to make the expression of the original current task frame feature and spatio-temporal feature $O_{i,j,q}^k$ similar:

$${\mathcal{L}}_{\mathrm{kd}}={\displaystyle \sum _{j=1}^N}{\displaystyle \sum _{q=1}^B}{\displaystyle \sum _{c=1}^{C_p}}{I}_{q,c}^k{∥f_c^k\left(x_{i,j,q}^k\right)-O_{i,j,q,c}^k∥}_F^2.$$

(6)

where ${\parallel ·\parallel }_F$ is the Frobenius norm. When $C_p>C_1$, 1D max unpooling is applied to the $C_1$-dimensional spatio-temporal feature before computing ${\mathcal{L}}_{\mathrm{kd}}$. Using the indices obtained from prototype pooling, the unpooling produces a $C_p$-dimensional feature that matches the dimension of $f^k\left(x_{i,j,q}^k\right)$. Unpooling is used for feature matching and does not change the $C_1$-dimensional input to the decoder. When $C_p=C_1$, neither pooling nor unpooling is required. $I^k$ is obtained by normalizing the time-channel importance mask ${\widehat{I}}^k$ [4] computed from the final output of the feature extractor $f^k$. After the incremental training of task k and before memory construction, the unnormalized importance at temporal position q and channel c is estimated as

$${\widehat{I}}_{q,c}^k={\mathbb{E}}_{(x,y)\sim {\mathcal{T}}^{\prime k}}{∥{\nabla }_{f_{q,c}^k\left(x\right)}{\mathcal{L}}_{\mathrm{cls}}^k(x,y)∥}_F^2.$$

(7)

where ${\mathcal{T}}^{\prime k}={\mathcal{T}}^k\cup {\widehat{M}}^{1:k-1}$ contains the original current task videos and the compressed replay videos from previous tasks. $f_{q,c}^k\left(x\right)$ denotes the feature at the q-th sampled temporal position and the c-th channel in the final output of $f^k$ for video x. ${\mathcal{L}}_{\mathrm{cls}}^k(x,y)$ denotes the classification loss of the recognition model ${\mathcal{F}}^k$ for video x and its label y. For the first task, ${\mathcal{T}}^{\prime 1}={\mathcal{T}}^1$. The importance mask is computed from the gradients of the classification loss with respect to the recognition features and remains fixed during subsequent STNM training.

Hence, the final objective function of the STNM is given by the following equation:

$$\begin{array}{c}\hfill {\mathcal{L}}_{\mathrm{STNM}}={\mathcal{L}}_{\mathrm{pixel}}+\lambda {\mathcal{L}}_{\mathrm{kd}}\end{array}$$

(8)

where $\lambda$ is set to 0.001 for all tasks. After training the STNM in step k, we store the spatial prototype p, the number of frames T and the STNM G of each class in $Y^k$ and use them to construct ${\widehat{M}}^k$ in the following tasks.

For different memory budgets, we change the STNM architecture to obtain STNM models with different sizes. The STNM architecture is shown in Table 1. As shown in Table 1, we design NeRV blocks with different upscale factors to gradually upsample the hidden dimensions h and w from $7\times 7$ to $224\times 224$ and adjust $C_1$ and $C_2$ to obtain STNM models with different sizes. In addition, the setting of $C_1$ is related to the dimension of the original video features, as we want to store the approximate original old task video features in the STNM, and the setting of $C_2$ is decided by $C_1$ and the memory budget. We use 8-bit quantization [17] for the STNM parameters. The 8-bit quantization of the parameter tensor $\mu$ can be represented as:

$${\widehat{\mu }}_n=\min \left\{round\left({\displaystyle \frac{{\mu }_n-{\mu }_{\min }}{\mathrm{scale}}}\right),255\right\}\times \mathrm{scale}+{\mu }_{\min },\mathrm{scale}={\displaystyle \frac{{\mu }_{\max }-{\mu }_{\min }}{2^8}}.$$

(9)

where ${\mu }_n$ and ${\widehat{\mu }}_n$ denote the n-th original parameter in the parameter tensor $\mu$ and its dequantized approximation, respectively. ${\mu }_{\min }$ and ${\mu }_{\max }$ denote the minimum and maximum values of $\mu$, respectively. $round(·)$ maps a value to the nearest integer. The scale and ${\mu }_{\min }$ are stored as part of the memory budget for dequantization.

Table 1. The STNM architecture for videos with a resolution of $224\times 224$ pixels. $C_p$ denotes the spatial prototype dimension, while $C_1$ and $C_2$ are adjusted to obtain STNM models with different sizes. FC represents a linear layer with a learnable bias, and Conv represents a convolutional layer with the learnable bias. Block represents the NeRV block. Here, $\mathrm{floor}\left(x\right)$ denotes the largest integer not greater than x, and the max operation ensures a minimum of 96 output channels in Blocks 2 to 4.

Layer	Modules	Output Size ($\mathit{C}\times \mathit{H}\times \mathit{W}$)	Attributes
PE	—	$(2\times 40)\times 1\times 1$	$l=40$
Encoder.FC	—	$C_1\times 1\times 1$	—
1D max pooling	Prototype branch	$C_1\times 1\times 1$	Optional, $C_p\to C_1$
1D max unpooling	KD branch	$C_p\times 1\times 1$	Optional, $C_1\to C_p$
Decoder.FC	—	$(C_2\times 7\times 7)\times 1\times 1$	$h=7,w=7$
Resize	—	$C_2\times 7\times 7$	—
$3\times 3$ Conv	Block 1	$(C_2\times 4\times 4)\times 7\times 7$	Upscale Factor = 4
PixelShuffle	Block 1	$C_2\times 28\times 28$	—
Swish	Block 1	$C_2\times 28\times 28$	—
$3\times 3$ Conv	Block 2	$(\max (\mathrm{floor}(C_2/2),96)\times 2\times 2)\times 28\times 28$	Upscale Factor = 2
PixelShuffle	Block 2	$\max (\mathrm{floor}(C_2/2),96)\times 56\times 56$	—
Swish	Block 2	$\max (\mathrm{floor}(C_2/2),96)\times 56\times 56$	—
$3\times 3$ Conv	Block 3	$(\max (\mathrm{floor}(C_2/4),96)\times 2\times 2)\times 56\times 56$	Upscale Factor = 2
PixelShuffle	Block 3	$\max (\mathrm{floor}(C_2/4),96)\times 112\times 112$	—
Swish	Block 3	$\max (\mathrm{floor}(C_2/4),96)\times 112\times 112$	—
$3\times 3$ Conv	Block 4	$(\max (\mathrm{floor}(C_2/8),96)\times 2\times 2)\times 112\times 112$	Upscale Factor = 2
PixelShuffle	Block 4	$\max (\mathrm{floor}(C_2/8),96)\times 224\times 224$	—
Swish	Block 4	$\max (\mathrm{floor}(C_2/8),96)\times 224\times 224$	—
$1\times 1$ Conv	Block 4	$3\times 224\times 224$	—

The memory budget of each class consists of an STNM, a spatial prototype, a frame count, and the quantization parameters (scale and ${\mu }_{\min }$). All storage sizes below are expressed in bytes:

$$\mathrm{Memory}\mathrm{budget}={\mathrm{Size}}_{\mathrm{STNM}}+{\mathrm{Size}}_{\mathrm{proto}}+{\mathrm{Size}}_{\mathrm{count}}+{\mathrm{Size}}_{\mathrm{quant}}.$$

(10)

Under 8-bit quantization, each quantized STNM weight or bias is stored using one byte. Based on the architecture in Table 1, the storage required for a STNM is

$$\begin{array}{cc}\hfill {\mathrm{Size}}_{\mathrm{STNM}}=& 81C_1+49C_1{C}_2+144C_2^2+65C_2\hfill \\ \hfill & +(36C_2+4)\max \left(\mathrm{floor}\left({\displaystyle \frac{C_2}{2}}\right),96\right)\hfill \\ \hfill & +\left[36\max \left(\mathrm{floor}\left({\displaystyle \frac{C_2}{2}}\right),96\right)+4\right]\max \left(\mathrm{floor}\left({\displaystyle \frac{C_2}{4}}\right),96\right)\hfill \\ \hfill & +\left[36\max \left(\mathrm{floor}\left({\displaystyle \frac{C_2}{4}}\right),96\right)+7\right]\max \left(\mathrm{floor}\left({\displaystyle \frac{C_2}{8}}\right),96\right)+3.\hfill \end{array}$$

(11)

The spatial prototype $p\in {\mathbb{R}}^{8\times C_p}$ contains $8C_p$ elements, where $C_p$ denotes the prototype feature dimension. Since each element is stored in FP32 format, its storage requirement is

$${\mathrm{Size}}_{\mathrm{proto}}=8C_p\times 4=32C_p.$$

(12)

The frame count T is stored as a single 32-bit integer, requiring

$${\mathrm{Size}}_{\mathrm{count}}=4.$$

(13)

The quantization parameters, scale and ${\mu }_{\min }$, are stored in FP32 format, requiring four bytes per scalar. Based on the architecture in Table 1, their total storage requirement is

$$\begin{array}{cc}{\mathrm{Size}}_{\mathrm{quant}}=4[\hfill & 2C_1+130C_2+8(\max (\mathrm{floor}({\displaystyle \frac{C_2}{2}}),96)\hfill \\ \hfill & +\max (\mathrm{floor}({\displaystyle \frac{C_2}{4}}),96)+\max (\mathrm{floor}({\displaystyle \frac{C_2}{8}}),96))+20].\hfill \end{array}$$

(14)

The layer-wise storage breakdown of STNM is summarized in Table 2. The layer storage is obtained by counting the weights and biases of each learnable layer in Table 1. Under 8-bit quantization, each quantized parameter occupies one byte. The corresponding quantization storage includes the FP32 scale and ${\mu }_{\min }$ values required for dequantization. Layers without learnable parameters are excluded from the storage calculation.

Table 2. Layer-wise storage of a STNM. Layer storage includes 8-bit weights and biases, while quantization storage includes the FP32 scale and ${\mu }_{\min }$. All sizes are reported in bytes.

Layer	Modules	Layer Storage (Bytes)	Quantization Storage (Bytes)
Encoder.FC	—	$81C_1$	$8(C_1+1)$
Decoder.FC	—	$49C_2(C_1+1)$	$8(49C_2+1)$
$3\times 3$ Conv	Block 1	$144C_2^2+16C_2$	$8(16C_2+1)$
$3\times 3$ Conv	Block 2	$(36C_2+4)\max (\mathrm{floor}(C_2/2),96)$	$8[4\max (\mathrm{floor}(C_2/2),96)+1]$
$3\times 3$ Conv	Block 3	$\begin{array}{c}[36\max (\mathrm{floor}(C_2/2),96)+4]\\ \times \max (\mathrm{floor}(C_2/4),96)\end{array}$	$8[4\max (\mathrm{floor}(C_2/4),96)+1]$
$3\times 3$ Conv	Block 4	$\begin{array}{c}[36\max (\mathrm{floor}(C_2/4),96)+4]\\ \times \max (\mathrm{floor}(C_2/8),96)\end{array}$	$8[4\max (\mathrm{floor}(C_2/8),96)+1]$
$1\times 1$ Conv	Block 4	$3\max (\mathrm{floor}(C_2/8),96)+3$	32

By combining the above four components, the per-class memory budget is obtained as

$$\begin{array}{cc}\hfill \mathrm{Memory}\mathrm{budget}=& 89C_1+49C_1{C}_2+144C_2^2+585C_2\hfill \\ \hfill & +(36C_2+36)\max \left(\mathrm{floor}\left({\displaystyle \frac{C_2}{2}}\right),96\right)\hfill \\ \hfill & +\left[36\max \left(\mathrm{floor}\left({\displaystyle \frac{C_2}{2}}\right),96\right)+36\right]\max \left(\mathrm{floor}\left({\displaystyle \frac{C_2}{4}}\right),96\right)\hfill \\ \hfill & +\left[36\max \left(\mathrm{floor}\left({\displaystyle \frac{C_2}{4}}\right),96\right)+39\right]\max \left(\mathrm{floor}\left({\displaystyle \frac{C_2}{8}}\right),96\right)\hfill \\ \hfill & +32C_p+87.\hfill \end{array}$$

(15)

3.5. Contrastive Bias Correction

Although STNM mitigates domain bias between the compressed and original videos, the trained model of the current task is trained on the combination of the original current task videos and the compressed old task videos, resulting in the accumulation of domain bias. Therefore, we propose Contrastive Bias Correction (CBC) to separate the distribution of current task videos from old task videos while bringing the distribution of compressed videos and original videos closer together, as shown in Figure 3.

Figure 3. Illustration of CBC. Lock/unlock icons indicate frozen/trainable weights.

Specifically, after the step k training of the STNM, given a compressed video ${\widehat{x}}_{i,j}^k$ and a corresponding original video $x_{i,j}^k$ for the j-th video in the memory buffer of class $y_i^k$, we first compute the original current task video feature $Z_{i,j}^{oc}$ and the compressed current task video feature $Z_{i,j}^{cc}$ via the feature extractor $f^k$.

Meanwhile, we randomly select a video index $t^{\prime }\in [0,N)$ of class $y_m^{1:k-1}\in Y^{1:k-1}$, where $Y^{1:k-1}=Y^1\cup Y^2\cup \cdots \cup Y^{k-1}$. The temporal index set s of video index $t^{\prime }$ is defined as $s={\{B\ast t^{\prime }+q-1\}}_{q=1}^B$, and we compute the approximate original old task video feature $Z^{oo}$ using the following formula:

$$\begin{array}{c}\hfill Z^{oo}=E_m^{1:k-1}(s,p_m^{1:k-1})\end{array}$$

(16)

where $p_m^{1:k-1}$ is the spatial prototype of class $y_m^{1:k-1}$. Then, we feed $Z^{oo}$ into $D_m^{1:k-1}$ and obtain the compressed video ${\widehat{x}}^{\prime }$, which is used to compute the compressed old task video feature $Z^{co}$.

The approximate original old task video feature $Z^{oo}$ is close to the original old task video feature in the old task feature space. To adapt $Z^{oo}$ to the feature space of the current task k, we design a learnable class-wise adaptation prompt $P_m^{1:k-1}\in {\mathbb{R}}^{B\times C_p}$ for class $y_m^{1:k-1}$. Thus, $Z^{oo}$ can be updated as follows:

$$\begin{array}{c}\hfill Z^{oo}=Z^{oo}+P_m^{1:k-1}\end{array}$$

(17)

Domain bias between the compressed and original videos leads to confusion between old and current tasks. To reduce domain bias, the compressed and original videos should be mapped to nearby features in the feature space of the current task k, and the old and current task videos should be mapped to distant features in the feature space of the current task k. Therefore, CBC computes the distance $\mathcal{D}$ between the compressed and original videos in the feature space of the current task k as follows:

$$\begin{array}{c}\hfill \mathcal{D}={\displaystyle \sum _{i=1}^L}{\displaystyle \sum _{j=1}^N}[1-cos(Z_{i,j}^{oc},Z_{i,j}^{cc})+1-cos(Z^{co},Z^{oo})]\end{array}$$

(18)

where $cos(·,·)$ is the cosine similarity, $1-cos(·,·)$ is the cosine distance, L is the number of classes in the current task k and N is the number of compressed videos stored in the STNM.

Then, CBC computes the similarity $\mathcal{S}$ between the old and current task videos in the feature space of the current task k as follows:

$$\begin{array}{c}\hfill \mathcal{S}={\displaystyle \sum _{i=1}^L}{\displaystyle \sum _{j=1}^N}[cos(Z_{i,j}^{oc},Z^{oo})+cos(Z_{i,j}^{cc},Z^{co})]\end{array}$$

(19)

Based on the distance $\mathcal{D}$ between the compressed and original videos and the similarity $\mathcal{S}$ between the old and current task videos, CBC introduces a debiased contrastive loss ${\mathcal{L}}_{\mathrm{dc}}$ to pull the compressed and original videos together and push away old and current tasks in the feature space of the current task k, which is defined as follows:

$$\begin{array}{cc}\hfill {\mathcal{L}}_{\mathrm{dc}}=& {\displaystyle \frac{1}{4}}{\displaystyle \sum _{i=1}^L}{\displaystyle \sum _{j=1}^N}[cos(Z_{i,j}^{oc},Z^{oo})+cos(Z_{i,j}^{cc},Z^{co})+\hfill \\ \hfill & 1-cos(Z_{i,j}^{oc},Z_{i,j}^{cc})+1-cos(Z^{co},Z^{oo})]\hfill \end{array}$$

(20)

In addition, the video features $Z^{co},Z^{oo},Z_{i,j}^{oc},Z_{i,j}^{cc}$ are fed into the classification layer $h^k$, and we compute Neighborhood Component Analysis (NCA) loss [48] as classification loss ${\mathcal{L}}_{\mathrm{cls}}$ to optimize the feature extractor $f^k$ and classification layer $h^k$, which is computed from the Local Similarity Classifier (LSC) following PODNet [37].

Therefore, the final objective function of the CBC is given as follows:

$$\begin{array}{c}\hfill {\mathcal{L}}_{\mathrm{CBC}}={\mathcal{L}}_{\mathrm{cls}}+{\lambda }_1{\mathcal{L}}_{\mathrm{dc}}\end{array}$$

(21)

where ${\lambda }_1$ is set to 1 for all tasks.

3.6. Overall Training Pipeline

To provide a concise overview of the proposed VNMBM, Algorithm 1 details the complete training pipeline across three distinct stages for each incremental task. In Stage 1 (Incremental Training), our method reconstructs compressed old task video frames from the frozen STNMs using the stored spatial prototypes, the number of frames, and quantization parameters. It then trains the action recognition model using a combination of the original current task video frames and the reconstructed compressed old task video frames. During Stage 2 (Memory Construction), our method trains the new STNMs for the current classes and stores them alongside their corresponding spatial prototypes, the number of frames, and quantization parameters. Finally, in Stage 3 (Contrastive Bias Correction), our method initializes trainable class-wise adaptation prompts to retrain the action recognition model while keeping all STNMs strictly frozen, ultimately removing the prompts.

Algorithm 1 Overall Training Pipeline of VNMBM

Input: Sequential task raw data $\mathcal{T}=\{{\mathcal{T}}^1,{\mathcal{T}}^2,\dots ,{\mathcal{T}}^K\}$, where ${\mathcal{T}}^k=\{(x_i^k,y_i^k),y_i^k\in Y^k\}$ Output: The action recognition model ${\mathcal{F}}^K$   1: Initialize the action recognition model ${\mathcal{F}}^1$ for task 1   2: for task $k=1,\dots ,K$ do   3:    // Stage 1: Incremental Training, ${\mathcal{F}}^k$ is trainable, ${\left\{G_i\right\}}_{y_i^{1:k-1}\in Y^{1:k-1}}$ are frozen   4:    if $k>1$ then   5:      for each old class $y_i^{1:k-1}\in Y^{1:k-1}$ do   6:         Dequantize the STNM $G_i^{1:k-1}$ using the quantization parameters (${\mathrm{scale}}_i^{1:k-1}$ and ${\mu }_{\min ,i}^{1:k-1}$)   7:         Use the spatial prototype $p_i^{1:k-1}$, the number of frames $T_i^{1:k-1}$, and the STNM $G_i^{1:k-1}$ to reconstruct $T_i^{1:k-1}$ compressed old task video frames as ${\widehat{M}}_i^{1:k-1}$   8:         Replay compressed old task video frames ${\widehat{M}}_i^{1:k-1}$ for old class $y_i^{1:k-1}\in Y^{1:k-1}$   9:      end for 10:      Combine ${\widehat{M}}^{1:k-1}$ with original current task video frames ${\mathcal{T}}^k$ as ${\mathcal{T}}^{\prime }k={\mathcal{T}}^k\cup {\widehat{M}}^{1:k-1}$. 11:    else 12:      ${\mathcal{T}}^{\prime }k={\mathcal{T}}^k$ 13:    end if 14:    Train the action recognition model ${\mathcal{F}}^k$ based on ${\mathcal{T}}^{\prime }k$ 15:    Estimate ${\widehat{I}}^k$ from the classification gradients with respect to the recognition features of ${\mathcal{F}}^k$ on ${\mathcal{T}}^{\prime }k$ and normalize it to obtain $I^k$, without updating ${\mathcal{F}}^k$ (Equation (7)) 16:    // Stage 2: Memory Construction, ${\left\{G_i\right\}}_{y_i^k\in Y^k}$ are trainable, $f^k(·)$ is frozen, and $I^k$ is fixed 17:    for each current class $y_i^k\in Y^k$ do 18:      Initialize STNM $G_i^k$ for class $y_i^k\in Y^k$ 19:      Use a herding strategy to select N videos as $M_i^k$ and sample B frames for each video 20:      Extract spatial prototype $p_i^k$ using the feature extractor $f^k$ of ${\mathcal{F}}^k$ (Equation (2)) 21:      Use $p_i^k$, $M_i^k$, and fixed $I^k$ to train $G_i^k$ by ${\mathcal{L}}_{\mathrm{STNM}}$ (Equation (8)) 22:      Quantize the $G_i^k$ and compute the quantization parameters (${\mathrm{scale}}_i^k$ and ${\mu }_{\min ,i}^k$) 23:      Store quantized $G_i^k$, $p_i^k$, $T_i^k$, ${\mathrm{scale}}_i^k$, and ${\mu }_{\min ,i}^k$, where $T_i^k=N\times B$ is the number of frames 24:    end for 25:    // Stage 3: Contrastive Bias Correction, ${\mathcal{F}}^k$ is trainable, ${\left\{P_m\right\}}_{y_m^{1:k-1}\in Y^{1:k-1}}$ are trainable, ${\left\{G_i\right\}}_{y_i^{1:k}\in Y^{1:k}}$ are frozen 26:    if $k>1$ then 27:      Initialize the class-wise adaptation prompts ${\left\{P_m\right\}}_{y_m^{1:k-1}\in Y^{1:k-1}}$ 28:      Use ${\mathcal{T}}^k$, ${\left\{P_m\right\}}_{y_m^{1:k-1}\in Y^{1:k-1}}$, and ${\{G_i,p_i,T_i,{\mathrm{scale}}_i,{\mu }_{\min ,i}\}}_{y_i^{1:k}\in Y^{1:k}}$ to retrain the action recognition model ${\mathcal{F}}^k$ by ${\mathcal{L}}_{\mathrm{CBC}}$ (Equation (21)) 29:      Remove ${\left\{P_m\right\}}_{y_m^{1:k-1}\in Y^{1:k-1}}$ 30:    end if 31: end for 32: Return ${\mathcal{F}}^K$

4. Experiments

This section evaluates the proposed VNMBM. We first introduce the evaluation metrics and implementation details, followed by comparisons with existing methods. We then conduct ablation studies and further analyses to evaluate the contributions of the proposed components and investigate their performance under different settings.

4.1. Evaluation Metrics

Following previous video class-incremental methods [4,5,6,7,8,9,10], we apply the average incremental accuracy using CNN and NME to evaluate the performance of video class-incremental learning, which can be defined as follows:

$$\begin{array}{c}\hfill \mathrm{CNN}={\displaystyle \frac{1}{K}}{\displaystyle \sum _{k=1}^K}{a}_{\mathrm{CNN}}^k\end{array}$$

(22)

$$\begin{array}{c}\hfill \mathrm{NME}={\displaystyle \frac{1}{K}}{\displaystyle \sum _{k=1}^K}{a}_{\mathrm{NME}}^k\end{array}$$

(23)

where CNN and NME are two inference methods. CNN classifies test videos using the trained fully connected layer. For NME evaluation [28], after the completion of CBC, we use the updated feature extractor $f^k$ to compute the class-mean features from the reconstructed videos in the memory buffer. Test videos are then classified by comparing their feature embeddings with these class means. Therefore, both the class means and test video features are obtained using the recognition model after CBC when $k>1$. $a_{\mathrm{CNN}}^k$ and $a_{\mathrm{NME}}^k$ are the average accuracies using CNN and NME in all the seen categories for the incremental training step k.

Peak signal-to-noise ratio (PSNR) is adopted to evaluate the quality of compressed frames, which is the ratio of the maximum value of the pixel to the mean squared error between the original and compressed frames. The higher the PSNR, the better the quality of the compressed frames. PSNR can be denoted as follows:

$$\begin{array}{c}\hfill \mathrm{PSNR}=10·{\log }_{10}\left({\displaystyle \frac{MA{X}^2}{MSE}}\right)\end{array}$$

(24)

where $MAX$ is the maximum possible pixel value of the image, and $MSE$ is the mean squared error between the original and compressed frames.

4.2. Implementation Details

We implement our method based on TCD [4], following the training hyperparameters and data pre-processing steps of TCD. The dimensions of each frame are $3\times H\times W$, where $H=224$ and $W=224$. During the incremental training of TCD, the compressed old task videos are constructed using old task STNMs for memory replay. After the incremental training of TCD, we train the STNMs to construct the neural representations of the current task video memory buffer and use the CBC to retrain the TCD. For the STNM, b and l of PE are set to 1.25 and 40, respectively. We stack four NeRV blocks with upscale factors of 4, 2, 2, 2, and the hidden dimensions h and w are both set to 7. According to the upscale factors of the NeRV blocks, these blocks gradually upsample the hidden dimensions h and w from $7\times 7$ to $224\times 224$. We adjust the model size of the STNM based on Equation (11). The STNM configurations under different datasets and backbones are summarized in Table 3. The spatial prototype dimension $C_p$ is determined by the dimension of the backbone output features used to construct the spatial prototype, which is 512 for ResNet-34 TSM [1,49], 2048 for ResNet-50 TSM [1,49], and 1024 for CLIP ViT-L/14 [50]. The internal channel dimensions $C_1$ and $C_2$ of STNM are adjusted according to the memory budget. Specifically, $(C_1,C_2)$ are set to $(256,26)$, $(512,47)$, $(512,114)$, and $(512,304)$ for the 1.2 Mb, 2.4 Mb, 6.0 Mb, and 24.0 Mb STNMs, respectively. When $C_p>C_1$, 1D max pooling is applied to downsample the spatial prototype from $C_p$ to $C_1$ dimensions before its element-wise multiplication with the encoder output. The corresponding 1D max unpooling operation restores the spatio-temporal feature from $C_1$ to $C_p$ dimensions for feature matching. When $C_p=C_1$, these pooling and unpooling operations are not required. The detailed per-class memory budget under each experimental setting is reported in Table 4, including the quantized STNM parameters, spatial prototype, frame count, and quantization parameters. The exact total memory is calculated according to Equation (15). The quantization overhead introduced by storing the FP32 scale and ${\mu }_{\min }$ parameters is calculated as the percentage of the total per-class memory occupied by these parameters and is reported in Table 4. Mb represents Mbytes.

Table 3. STNM configurations under different datasets and backbones. Sth-Sth V2 denotes the Something-Something V2 dataset.

Dataset	Backbone	${\mathit{C}}_{\mathit{p}}$	${\mathit{C}}_1$	${\mathit{C}}_2$	Budget
UCF101	ResNet-34 TSM	512	256	26	1.2 Mb
UCF101	ResNet-34 TSM	512	512	47	2.4 Mb
UCF101	ResNet-34 TSM	512	512	114	6.0 Mb
UCF101	CLIP ViT-L/14	1024	512	114	6.0 Mb
HMDB51	ResNet-50 TSM	2048	512	114	6.0 Mb
HMDB51	CLIP ViT-L/14	1024	512	114	6.0 Mb
Sth-Sth V2	ResNet-50 TSM	2048	512	304	24.0 Mb

Table 4. Memory budget and quantization overhead of VNMBM under different experimental settings. Sth-Sth V2 denotes the Something-Something V2 dataset.

Setting	STNM	Prototype	Frame Count	Quant. Params.	Quant. Overhead	Total (Bytes)
UCF101, ResNet-34 TSM	1,200,765	16,384	4	24,864	2.00%	1,242,017 (1.2 Mb)
UCF101, ResNet-34 TSM	2,369,186	16,384	4	37,832	1.56%	2,423,406 (2.4 Mb)
UCF101, ResNet-34 TSM	5,839,317	16,384	4	72,672	1.23%	5,928,377 (6.0 Mb)
UCF101, CLIP ViT-L/14	5,839,317	32,768	4	72,672	1.22%	5,944,761 (6.0 Mb)
HMDB51, ResNet-50 TSM	5,839,317	65,536	4	72,672	1.22%	5,977,529 (6.0 Mb)
HMDB51, CLIP ViT-L/14	5,839,317	32,768	4	72,672	1.22%	5,944,761 (6.0 Mb)
Sth-Sth V2, ResNet-50 TSM	23,518,131	65,536	4	173,264	0.73%	23,756,935 (24.0 Mb)

We use the Adam optimizer to train the STNM, with a batch size of 32. The Adam optimizer uses ${\beta }_1=0.5$, ${\beta }_2=0.999$, $ϵ={10}^{-8}$, and a weight decay of 0. The initial learning rate is set to 0.004 for the 6.0 Mb, 2.4 Mb, and 24.0 Mb STNMs and 0.006 for the 1.2 Mb STNM. The STNM is trained for 300 epochs. For STNM training, we adopt a cosine annealing learning rate schedule with a warmup period of 60 epochs. We use a herding strategy [4] to select $N=80$, $N=320$, and $N=70$ videos per class for training the 6.0 Mb STNM on UCF101, the 24.0 Mb STNM on Something-Something V2, and the 6.0 Mb STNM on HMDB51, respectively. We sample $B=8$ frames with a uniform time interval for each video. After training the STNM, we use 8-bit quantization [17] for the STNM. After the training of the current task STNMs, we use CBC to train the current task TCD for 40 epochs. The batch size is set to 8 for UCF101 and 16 for HMDB51 and Something-Something V2. The SGD optimizer is deployed for CBC. The initial learning rates of the TCD and class-wise adaptation prompts are 0.0001 and 0.001, respectively. During CBC training, both learning rates are multiplied by 0.1 at epoch 20. The weight decay of CBC is set to $5\times {10}^{-4}$. For UCF101 and HMDB51, following TCD [4], we use random seeds 1000, 1993, and 2021. Since the random seed also determines the class order, these three runs correspond to three different class orders. We report the mean and standard deviation over the three runs. For Something-Something V2, following TCD [4], we use the single-seed setting with seed 1000. The complete class orders corresponding to these random seeds are provided in Appendix A Table A1, Table A2, and Table A3, respectively.

4.3. Main Results

In this section, we compare our proposed approach with the existing class-incremental learning approaches on the UCF101, HMDB51 and Something-Something V2 datasets. For the UCF101 and HMDB51 datasets, Fine-tuning, LwFMC [34], LwM [35], iCaRL [28], UCIR [36], PODNet [37], TCD [4], FrameMaker [10], and HCE [5] employ ResNet-34 TSM and ResNet-50 TSM [1,49] as their respective backbones, both initialized with ImageNet-pretrained weights [51]. In contrast, VCIL-CT [7] and STSA-DFS [8] adopt the CLIP ViT-L/14 backbone [50] for both datasets. For the TSM settings, VNMBM uses ImageNet-pretrained ResNet-34 TSM on UCF101 and ResNet-50 TSM on HMDB51. For the CLIP ViT-L/14 settings, VCIL-CT, STSA-DFS, and VNMBM use the official OpenAI CLIP ViT-L/14 pretrained weights. Following STSA-DFS [8], the feature extractor of VNMBM consists of a frozen CLIP ViT-L/14 image encoder and a trainable Spatial-Temporal Side Adaptation (STSA) module [8]. The STSA module performs temporal modeling on classification tokens and spatial modeling on patch features extracted from the CLIP image encoder, and integrates the resulting representations for action classification. Therefore, the CLIP settings of VNMBM directly use CLIP visual features without relying on TSM for feature extraction. For the Something-Something V2 dataset, UCIR, PODNet, TCD, FrameMaker and HCE use the ResNet-50 TSM as backbone, which is initialized by ImageNet pretrained weights. For a fair comparison, we evaluated our method using the same model architecture and pre-training initialization as the existing video class-incremental learning methods [4,10]. For the memory-based comparison methods, including iCaRL, UCIR, PODNet, TCD, FrameMaker, HCE, VCIL-CT, and STSA-DFS, video frames are stored as replay memory. The memory budget is set to 6.0 Mb per class for UCF101 and HMDB51 and 24.0 Mb per class for Something-Something V2. VNMBM follows the same memory budget for a fair comparison, while its replay memory is stored in the form of neural representations rather than video frames. Specifically, the memory budget of VNMBM includes the quantized STNM parameters, spatial prototypes, frame count, and quantization parameters. In Table 5, Fine-tuning, LwFMC, and LwM do not store replay exemplars, corresponding to 0 bytes of exemplar replay memory per class. For iCaRL, UCIR, PODNet, TCD, FrameMaker, HCE, VCIL-CT, and STSA-DFS, the per-class replay-memory budget corresponds to 40 RGB frames of size $3\times 224\times 224$ stored in uint8 format, requiring $3\times 224\times 224\times 40=\mathrm{6,021,120}$ bytes per class, denoted as 6.0 Mb in our experiments. FrameMaker stores 40 condensed frames within this budget. These per-class budgets apply to all reported task splits and CNN/NME columns on UCF101 and HMDB51 in Table 5.

Table 5. Comparison with existing video class-incremental learning methods on the UCF101 and HMDB51 datasets. VNMBM achieves the best performance among the compared methods under the evaluated settings. Results marked with * are taken from HCE [5], while those marked with † are taken from STSA-DFS [8]. The results of VNMBM are presented as means ± standard deviations over three class orders generated using seeds 1000, 1993, and 2021, as detailed in Appendix A.

	UCF101						HMDB51
Num. of classes	10 × 5 steps		5 × 10 steps		2 × 25 steps		5 × 5 steps		1 × 25 steps
Classifier	CNN	NME	CNN	NME	CNN	NME	CNN	NME	CNN	NME
ResNet TSM (ImageNet pretrained)
Fine-tuning *	24.97	—	13.45	—	5.78	—	16.82	—	4.83	—
LwFMC * [34]	42.14	—	25.59	—	11.68	—	26.82	—	16.49	—
LwM * [35]	43.39	—	26.07	—	12.08	—	26.97	—	16.50	—
iCaRL * [28]	—	65.34	—	64.51	—	58.73	—	40.09	—	33.77
UCIR * [36]	74.31	74.09	70.42	70.50	63.22	64.00	44.90	46.53	37.04	37.15
PODNet * [37]	73.26	74.37	71.58	73.75	70.28	71.87	44.32	48.78	38.76	46.62
TCD * [4]	74.89	77.16	73.43	75.35	72.19	74.01	45.34	50.36	40.07	46.66
FrameMaker * [10]	78.13	78.64	76.38	78.14	75.77	77.49	47.54	51.12	42.65	47.37
HCE * [5]	79.12	80.01	77.59	78.81	75.84	77.62	48.63	52.01	43.99	48.94
VNMBM (Ours)	80.79	80.39	80.36	80.17	78.60	78.69	52.88	55.18	51.15	54.19
	$\pm 0.77$	$\pm 0.53$	$\pm 1.10$	$\pm 0.89$	$\pm 1.13$	$\pm 0.69$	$\pm 0.37$	$\pm 0.85$	$\pm 1.28$	$\pm 1.33$
CLIP ViT-L/14 (OpenAI pretrained)
VCIL-CT † [7]	91.31	91.20	91.14	90.80	85.85	84.76	64.58	69.80	57.39	60.17
STSA-DFS † [8]	94.85	95.11	94.74	95.22	94.31	94.45	74.85	75.76	73.25	74.32
VNMBM (Ours)	95.85	95.91	95.55	95.51	95.11	95.01	78.19	78.78	76.73	77.02
	$\pm 0.14$	$\pm 0.19$	$\pm 0.05$	$\pm 0.11$	$\pm 0.18$	$\pm 0.20$	$\pm 0.54$	$\pm 0.59$	$\pm 0.85$	$\pm 0.77$

Table 5 reports the overall results on UCF101 and HMDB51. Under the evaluated memory budgets and backbone settings, VNMBM achieves the best performance among the compared methods across all configurations. Under the TSM setting, VNMBM surpasses HCE by up to 1.67%, 2.77%, and 2.76% in CNN accuracy on the three UCF101 settings and by up to 4.25% and 7.16% on the two HMDB51 settings, respectively. Under the CLIP ViT setting, VNMBM also outperforms the recent STSA-DFS method across all evaluated configurations. These results show that VNMBM provides favorable recognition performance under the evaluated video class-incremental learning settings and memory budgets.

Table 6 reports the overall results obtained on the large-scale motion-sensitive Something-Something V2 dataset. Our VNMBM outperforms HCE by up to 6.41% and 9.18% on the Something-Something V2 dataset with 9 steps and 18 steps, respectively. These results indicate that our method effectively maintains temporal information in the STNM while mitigating the domain bias problem caused by compression, thereby offering both high memory efficiency and superior performance.

Table 6. Comparison with existing video class-incremental learning methods on the Something-Something V2 dataset. Results marked with * are taken from HCE [5].

Num. of Classes	10 × 9 Steps		5 × 18 Steps
Classifier	CNN	NME	CNN	NME
UCIR * [36]	26.84	17.98	20.69	12.57
PODNet * [37]	34.94	27.33	26.95	17.49
TCD * [4]	35.78	28.88	29.60	21.63
FrameMaker * [10]	37.25	29.92	30.98	22.84
HCE * [5]	38.67	36.88	32.51	32.82
VNMBM (Ours)	43.76	43.29	41.69	41.28

4.4. Ablation Study

The figures in this section were prepared using Matplotlib 3.5.3 (Python 3.7.16) and Microsoft PowerPoint 2021. No additional calibration was applied to the experimental measurements.

• Effect of each component. To investigate the contributions of the individual components in STNM and CBC, we conduct ablation experiments by progressively introducing different components, and the results are summarized in Table 7. The Baseline represents TCD storing 40 frames from five videos per class under a memory budget of 6.0 Mb. The Baseline accuracies are taken from the original TCD publication [4], which does not report standard deviations. The remaining ablation configurations are evaluated using three seeds (1000, 1993, and 2021), with results reported as mean ± standard deviation.
  Table 7. Ablation study of different components of STNM and CBC on UCF101 with 10 steps. PE denotes positional encoding used for temporal encoding.

  Baseline	STNM				CBC			CNN	NME
  	${\mathcal{L}}_{\mathbf{pixel}}$	PE	${\mathcal{L}}_{\mathbf{kd}}$	Prototype	${\mathcal{L}}_{\mathbf{cls}}$	${\mathcal{L}}_{\mathbf{dc}}$	Prompt
  ✓								73.43	75.35
  ✓	✓							44.82 ± $2.21$	37.10 ± $3.47$
  ✓	✓	✓						77.22 ± $0.53$	77.30 ± $0.29$
  ✓	✓	✓	✓					78.16 ± $0.93$	78.31 ± $0.26$
  ✓	✓	✓	✓	✓				78.94 ± $0.36$	78.82 ± $0.66$
  ✓	✓	✓	✓	✓	✓			79.08 ± $0.92$	79.28 ± $0.85$
  ✓	✓	✓	✓	✓	✓	✓		79.69 ± $0.89$	79.64 ± $0.99$
  ✓	✓	✓	✓	✓	✓	✓	✓	80.36 ± $1.10$	80.17 ± $0.89$

For STNM, using only the pixel reconstruction loss ${\mathcal{L}}_{\mathrm{pixel}}$ achieves CNN and NME accuracies of 44.82% and 37.10%, respectively. ${\mathcal{L}}_{\mathrm{pixel}}$ provides pixel-level reconstruction supervision and does not specifically target perceptual or discriminative features for action recognition. After introducing positional encoding for temporal encoding, the CNN and NME accuracies increase substantially to 77.22% and 77.30%, respectively. Without temporal encoding, STNM has difficulty representing high-frequency information from the temporal input, resulting in much lower performance. Positional encoding provides a richer representation of the temporal input and improves the modeling of video content at different temporal positions. On this basis, adding the knowledge distillation loss ${\mathcal{L}}_{\mathrm{kd}}$ further improves the CNN and NME accuracies to 78.16% and 78.31%, respectively. ${\mathcal{L}}_{\mathrm{kd}}$ provides feature-level supervision to preserve information extracted from the original videos. Incorporating the spatial prototype further increases the accuracies to 78.94% and 78.82% by introducing class-related feature information, forming the complete STNM. Compared with the Baseline, the complete STNM improves the CNN and NME accuracies by 5.51% and 3.47%, respectively, showing higher recognition accuracy than frame-based replay under the same memory budget.

Based on the complete STNM, introducing ${\mathcal{L}}_{\mathrm{cls}}$ increases the CNN and NME accuracies to 79.08% and 79.28%, respectively. Further introducing ${\mathcal{L}}_{\mathrm{dc}}$ improves the accuracies to 79.69% and 79.64%. Finally, incorporating the class-wise adaptation prompt achieves 80.36% and 80.17%, respectively. These results show the progressive contributions of the components in STNM and CBC to the final recognition performance.

• Different STNM configurations. To evaluate the effect of different STNM configurations, we design the VNMBM in different sizes for each class to store 40 compressed videos per class on the UCF101 dataset with 10 steps. Then, we compute the average accuracy of each task and the mean PSNR of the compressed frames for each class to represent the per-task accuracy and per-class quality, respectively. Specifically, PSNR is calculated between each compressed frame and its corresponding original frame. The results are shown in Figure 4, which shows that the larger VNMBM can achieve better compression quality when storing the same number of videos, thereby better mitigating the catastrophic forgetting problem. Notably, we introduce the compression quality metric to explicitly explore the behavior of our neural memory paradigm. By jointly analyzing the per-task accuracy and the per-class quality in Figure 4, we observe a phenomenon of diminishing marginal returns. Specifically, the growth in per-class quality from the 2.4 Mb VNMBM to the 6.0 Mb VNMBM is higher than the growth from the 1.2 Mb VNMBM to the 2.4 Mb VNMBM. However, the corresponding increase in per-task accuracy exhibits the exact opposite trend. This phenomenon provides an additional insight: the marginal benefit of solely improving image quality diminishes for downstream classification. It reveals that under a given memory budget, relying exclusively on higher video quality is suboptimal; one must comprehensively balance both the quantity and the quality of the stored videos. This finding motivates our subsequent further exploration regarding the trade-off between video quantity and quality in the following section.
  

  Figure 4. Per-task accuracy and compressed-video quality on UCF101 with 10 steps.
• Effect of the quality of compressed videos under different memory budgets. To show the effect of the quality differences at various memory budgets, we first construct 6.0 Mb VNMBM, 2.4 Mb VNMBM and 1.2 Mb VNMBM for the 6.0 Mb, 2.4 Mb and 1.2 Mb memory budgets. Then, we store different numbers of videos in these models for various compression qualities and use the mean PSNR of the compressed frames for all the classes to measure the compression quality. The results are shown in Table 8. When the number of stored videos N is fixed, increasing the memory budget improves the reconstruction quality of the compressed videos, as indicated by the higher PSNR, and is accompanied by higher CNN and NME accuracies. For example, when $N=40$, the PSNR increases from 28.36 dB to 36.62 dB as the memory budget increases from 1.2 Mb to 6.0 Mb, while the CNN and NME accuracies increase from 74.58% and 77.35% to 79.95% and 79.91%, respectively. Specifically, increasing the memory budget from 1.2 to 2.4 Mb yields a PSNR gain of 3.09 dB, accompanied by CNN and NME accuracy gains of 3.78% and 1.90%, respectively. Increasing the budget further to 6.0 Mb yields a larger PSNR gain of 5.17 dB but smaller CNN and NME accuracy gains of 1.59% and 0.66%, respectively, indicating diminishing accuracy improvements despite greater reconstruction quality gains. When the memory budget is fixed, increasing the number of stored videos reduces the memory allocated to each video and therefore decreases the reconstruction PSNR. However, the recognition performance does not necessarily decrease accordingly. For example, under the 6.0 Mb memory budget, increasing N from 20 to 80 decreases the PSNR from 36.84 dB to 36.30 dB, while the CNN and NME accuracies increase from 76.80% and 79.34% to 80.36% and 80.17%, respectively. These results show that reconstruction quality and downstream recognition performance are not monotonically correlated under a fixed memory budget. The final recognition performance reflects a trade-off between the reconstruction quality of the compressed videos and the number of stored videos. For the 1.2 Mb memory budget, $N=40$ achieves better recognition performance than $N=20$ and $N=80$, suggesting that storing too many heavily compressed videos may reduce the benefit of increasing the number of replay videos. Therefore, we set $N=80$ for the 6.0 Mb and 2.4 Mb VNMBMs, and $N=40$ for the 1.2 Mb VNMBM. We also visualize the compressed frames under different memory budgets in Figure 5. For this visualization, all three VNMBMs encode $N=40$ videos per class. From the visualization results, it can be observed that the motion information of the compressed videos is well preserved under different memory budgets, which demonstrates that our VNMBM ensures the high quality of compressed frames.
  Table 8. Analysis of the effect of quality differences at various memory budgets on the UCF101 dataset with 10 steps. N is the number of compressed videos stored in the VNMBM. PSNR represents the mean PSNR of compressed frames for all UCF101 classes.

  Memory Per Class	1.2 Mb			2.4 Mb			6.0 Mb
  Metrics	CNN	NME	PSNR	CNN	NME	PSNR	CNN	NME	PSNR
  $N=20$	71.41	76.53	28.55	76.47	79.13	31.71	76.80	79.34	36.84
  $N=40$	74.58	77.35	28.36	78.36	79.25	31.45	79.95	79.91	36.62
  $N=80$	74.47	76.86	27.61	79.42	80.06	31.06	80.36	80.17	36.30

  

  Figure 5. Compressed frames under different memory budgets on UCF101 with 10 steps. The compressed results in panels (b–d) correspond to $N=40$ videos per class.
• Memory budget comparison. We compare VNMBM with existing approaches under different memory budgets, and the results are reported in Table 9. The HCE results are taken from [5]. Both HCE and VNMBM follow the incremental task split settings established by TCD [4]. The class orders used for VNMBM are provided in the Appendix. From these results, we observe that our method using the 2.4 Mb memory budget can achieve better performance than HCE using the 6.0 Mb memory budget. Meanwhile, the performance of our method can be further improved by using the same memory budget as existing methods. These results demonstrate the effectiveness of our method in alleviating the catastrophic forgetting problem under the limited memory budget.
  Table 9. Memory budget comparison on the UCF101 dataset with 10 steps. Existing methods store the frame-like memory, while our VNMBM stores the neural representations.

  Memory Per Class	1.2 Mb		2.4 Mb		6.0 Mb
  Classifier	CNN	NME	CNN	NME	CNN	NME
  iCaRL [28]	—	58.05	—	60.50	—	64.51
  UCIR [36]	61.92	65.52	66.43	67.58	70.42	70.50
  PODNet [37]	63.18	70.96	65.93	72.78	71.58	73.75
  TCD [4]	64.52	71.96	68.40	73.30	73.43	75.35
  FrameMaker [10]	73.64	76.98	75.19	77.43	76.38	78.14
  HCE [5]	—	—	—	—	77.59	78.81
  VNMBM (Ours)	74.58	77.35	79.42	80.06	80.36	80.17

• Effect of class-wise adaptation prompts. To evaluate the effect of class-wise adaptation prompts in CBC, we conduct experiments with different numbers of adaptation prompts. The “None” setting retains the complete STNM and uses both the classification loss ${\mathcal{L}}_{\mathrm{cls}}$ and the debiased contrastive loss ${\mathcal{L}}_{\mathrm{dc}}$ in CBC, without adaptation prompts. The results in Table 10 show that CBC with a shared adaptation prompt for all old classes outperforms CBC without the adaptation prompt, which indicates that the adaptation prompt adapts the original old task video features to the current task feature space. Moreover, CBC using class-wise adaptation prompts leads to further improvements in performance. This configuration assigns a learnable class-wise adaptation prompt for each class of old tasks and adapts the original old task video features according to the distribution of each class.
  Table 10. Ablations for different numbers of adaptation prompts in Contrastive Bias Correction on the UCF101 dataset with 10 steps.

  Num. of Adaptation Prompts	CNN	NME
  None	79.69	79.64
  One	80.12	80.02
  Class-wise	80.36	80.17

• Effect of CBC on mitigating the domain bias. The domain bias refers to the difference between compressed and original videos. To evaluate the domain bias, we calculate the per-class feature distances between compressed and original videos of the initial 51 classes at the incremental step 2 on the UCF101 dataset with 10 steps. Specifically, we feed the compressed and original videos of each class within the initial 51 classes into the feature extractor, obtaining compressed old task video features and original old task video features, respectively. For each compressed-original video pair, we then compute the Frobenius norm of the difference between their corresponding video features. The mean of the Frobenius-normalized results for each class is the per-class feature distance between compressed and original videos. A larger per-class feature distance indicates a more significant domain bias between compressed and original videos. The results are summarized in Figure 6. As shown in this figure, all classes exhibit non-zero feature distances, which proves that there is a difference between compressed and original videos, i.e., domain bias exists. Additionally, our CBC reduces the feature distances across all 51 classes, which demonstrates that CBC effectively mitigates domain bias.
  

  Figure 6. Comparison of feature distances before and after our CBC.

To further investigate cross-task confusion, we calculate the average softmax probability mass assigned to the opposite task group using compressed old-class videos and original current-class videos. Specifically, we apply softmax over all 56 seen classes, sum the probabilities assigned to the 5 current classes for each compressed old-class video or to the 51 old classes for each original current-class video, and then average over the corresponding samples. The same samples are evaluated before and after CBC. As shown in Table 11, the old-to-new probability mass decreases from 4.38% to 1.93%, corresponding to a relative reduction of 55.94%. Similarly, the new-to-old probability mass decreases from 11.43% to 4.86%, corresponding to a relative reduction of 57.48%. These results indicate reduced cross-task probability assignment after CBC, providing additional evidence beyond feature alignment.

Table 11. Cross-task probability mass before and after CBC at incremental step 2 on UCF101 with 10 steps.

Metric	Before CBC	After CBC	Relative Reduction
Old → New	4.38%	1.93%	55.94%
New → Old	11.43%	4.86%	57.48%

To further evaluate the effect of feature alignment on recognition performance, we compare the classification accuracies before and after CBC on the held-out UCF101 test set containing all 56 seen classes at incremental step 2. As shown in Table 12, the CNN and NME accuracies increase from 83.05% and 83.68% to 84.89% and 84.50%, corresponding to improvements of 1.84% and 0.82%, respectively. These results show that the reduction in feature distance after CBC is accompanied by improved classification accuracy on unseen test videos.

Table 12. Classification accuracy before and after CBC on the held-out UCF101 test set with 56 seen classes at incremental step 2.

Setting	CNN	NME
Before CBC	83.05	83.68
After CBC	84.89	84.50
Improvement	+1.84	+0.82

• Quantitative analysis of feature separability. To further evaluate the effect of CBC on class separability, we perform a quantitative analysis on the test set of the 56 seen classes at incremental step 2 on UCF101 with 10 steps. We use the feature extractors before and after CBC to extract the corresponding test features, respectively. For each stage, the mean feature of each class is independently computed as its class center. We use cosine distance as the distance metric. The average intra-class distance is calculated by first computing the average cosine distance between the samples of each class and their corresponding class center and then equally averaging the results over all 56 classes. The average inter-class distance is calculated as the average cosine distance between all pairs of different class centers. We further calculate the ratio of the inter-class distance to the intra-class distance, where a smaller intra-class distance and a larger inter-class distance indicate more compact intra-class features and better separation between different classes, respectively. As shown in Table 13, after CBC, the average intra-class distance decreases from 0.300929 to 0.298902, while the average inter-class distance increases from 0.823290 to 0.932557. The inter-to-intra distance ratio increases from 2.735829 to 3.119943. Compared with the results before CBC, the intra-class distance decreases by 0.67%, the inter-class distance increases by 13.27%, and the inter-to-intra distance ratio increases by 14.04%. These results indicate that the feature space after CBC exhibits slightly improved intra-class compactness and substantially greater inter-class separation, providing quantitative support for the effect of CBC on feature separability.
  Table 13. Quantitative comparison of feature separability before and after CBC on the 56 seen classes at incremental step 2 of UCF101 with 10 steps. ↑ indicates that higher values are better, and ↓ indicates that lower values are better.

  Setting	Intra-Class ↓	Inter-Class ↑	Inter/Intra ↑
  Before CBC	0.300929	0.823290	2.735829
  After CBC	0.298902	0.932557	3.119943

• Effect of hyperparameters $\alpha$ and ${\lambda }_1$. Following the ablation study setting of TCD, we conduct the hyperparameter sensitivity analyses on UCF101 with 10 incremental steps. We investigate the sensitivity of the mixing coefficient $\alpha$ in the pixel reconstruction loss for STNM without CBC and the weight ${\lambda }_1$ of the debiased contrastive loss in VNMBM. Specifically, $\alpha$ is varied from 0.5 to 0.9 with an interval of 0.1, while ${\lambda }_1$ is evaluated at 0.1, 0.5, 1.0, 5.0, and 10.0. We evaluate $\alpha$ and ${\lambda }_1$ sequentially rather than through a joint grid search. We first evaluate $\alpha$ using STNM without CBC, and then evaluate ${\lambda }_1$ using VNMBM with $\alpha$ fixed at 0.7. As shown in Table 14 and Table 15, $\alpha =0.7$ and ${\lambda }_1=1$ achieve the highest CNN and NME accuracies among the tested values, respectively. Therefore, we adopt these values as the default settings for all experiments. The same values are applied to HMDB51 and Something-Something V2 without additional tuning. These sensitivity analyses characterize the parameter behavior on UCF101 rather than establish the optimal values for every dataset.
  Table 14. Effect of varying $\alpha$ on the UCF101 dataset with 10 steps using STNM without CBC.

  Mixing Coefficient $\mathit{\alpha }$	CNN	NME
  0.5	78.57	77.28
  0.6	78.89	78.08
  0.7	78.94	78.82
  0.8	78.10	78.15
  0.9	77.83	77.77

  Table 15. Effect of varying ${\lambda }_1$ on the UCF101 dataset with 10 steps using VNMBM.

  Weight ${\mathit{\lambda }}_1$	CNN	NME
  0.1	78.68	79.95
  0.5	79.67	79.96
  1.0	80.36	80.17
  5.0	80.08	77.00
  10.0	80.04	75.34

• Frame reconstruction cost of STNM. To evaluate the frame reconstruction cost of our proposed STNM, we compute the FLOPs of different STNM sizes. In addition, we measure the inference speed of different STNM sizes using a single RTX 3090 GPU, which represents the number of compressed frames reconstructed by the STNM per second. The results are summarized in Table 16. These results prove that the impact of the STNM on frame reconstruction cost is acceptable.
  Table 16. Frame reconstruction costs of different STNM sizes.

  The Size of STNM	FLOPs (G)	Inference Speed (fps)
  1.2 Mb	5.30	433.83
  2.4 Mb	5.37	425.05
  6.0 Mb	5.63	408.05

• Computational efficiency analysis. We further evaluate the computational efficiency of VNMBM on UCF101 with 10 steps. The total training time is 24 h 16 min. As shown in Table 17, Stage 1 corresponds to the incremental training stage of TCD and requires 15 h 42 min. Stage 2 trains a separate STNM for each class and requires 6 h 14 min in total, while Stage 3 performs CBC and requires 2 h 20 min. These three stages account for 64.7%, 25.7%, and 9.6% of the total training time, respectively.
  Table 17. Training time breakdown of VNMBM on UCF101 with 10 steps.

  Training Stage	Time	Proportion
  Stage 1: Incremental Training	15 h 42 min	64.7%
  Stage 2: STNM Construction	6 h 14 min	25.7%
  Stage 3: CBC	2 h 20 min	9.6%
  Total	24 h 16 min	100%

As shown in Table 18, at the final incremental step, CNN and NME inference require 23.90 and 23.98 ms per video, respectively. CBC is performed only during training and does not add computation to CNN or NME inference. CNN inference directly uses the trained recognition model without STNM reconstruction. For NME evaluation, the stored compressed videos are reconstructed before constructing the class prototypes. Specifically, reconstructing the compressed videos requires 157.61 s, while constructing the class prototypes takes an additional 42.37 s, resulting in a total NME preparation time of 199.98 s. NME preparation is performed once after each incremental update, and the constructed class prototypes are reused for subsequent test videos. Therefore, the preparation cost is amortized across these videos rather than incurred per video, and it is not included in the reported per-video inference latency. In contrast, TCD directly uses stored exemplar frames to construct NME class prototypes and does not require neural reconstruction. Therefore, the 157.61 s reconstruction time represents an additional preparation cost specific to VNMBM.

Table 18. Overall inference latency of VNMBM on UCF101 with 10 steps. Inference latency is measured at the final incremental step.

Metric	Time
CNN inference	23.90 ms/video
NME inference	23.98 ms/video
STNM reconstruction	157.61 s
NME prototype construction	42.37 s
Total NME preparation	199.98 s

We further compare the training and inference costs of VNMBM with TCD, a frame-based replay method, on UCF101 with 10 steps. Training times are measured using eight NVIDIA RTX 3090 GPUs, while inference and NME preparation times are measured using a single RTX 3090 GPU. As shown in Table 19, under the same memory budget of 6.0 Mb per class, TCD stores 5 exemplar videos per class using original video frames, whereas VNMBM represents 80 videos per class using compressed neural memory. The larger replay set contributes to the longer training and NME preparation times of VNMBM. STNM construction and CBC also contribute to the training cost of VNMBM, while STNM reconstruction increases the NME preparation time. The total training times of TCD and VNMBM are 9 h 10 min and 24 h 16 min, respectively. For NME preparation, TCD directly uses stored exemplar frames to construct class prototypes, whereas VNMBM requires STNM reconstruction and prototype construction. Since both methods use the same action recognition network, their CNN and NME inference times are identical, requiring 23.90 and 23.98 ms per video, respectively.

Table 19. Comparison of training and inference costs between TCD and VNMBM on UCF101 with 10 steps. Inference and NME preparation times are measured at the final incremental step.

Method	Training Time	CNN Inference (ms/Video)	NME Inference (ms/Video)	NME Preparation (s)
TCD	9 h 10 min	23.90	23.98	2.55
VNMBM	24 h 16 min	23.90	23.98	199.98

• Visualization of feature space. In Figure 7, we use t-SNE [52] to visualize the feature space before and after our CBC at incremental step 2 on the UCF101 dataset with 10 steps. At this point, the model has completed the incremental training stage of step 2 and has seen 56 classes in total, including the 51 initial classes and the 5 classes introduced at this incremental step. The same 56 seen classes are visualized before and after CBC. The features are extracted from the test set without class balancing, and the number of samples per class follows the original test-set distribution. The t-SNE analysis is performed in Python using the scikit-learn implementation, including two output dimensions, PCA initialization, and a perplexity of 30. No additional parameter tuning is performed. The resulting two-dimensional features are visualized using Matplotlib. These visualization results demonstrate that CBC results in better clusters of classes in the feature space, and the classes are better separated after CBC, as it corrects domain bias.
  

  Figure 7. t-SNE visualization of 56 seen classes (51 initial and 5 newly introduced) before and after CBC at step 2 on UCF101 with 10 steps.
• Comparison of contrastive losses. To further analyze the debiased contrastive loss ${\mathcal{L}}_{\mathrm{dc}}$, we compare it with a standard supervised contrastive loss ${\mathcal{L}}_{\sup }$ [53] under the same experimental setting. The standard supervised contrastive loss constructs positive and negative relationships according to class labels, pulling samples from the same class closer while separating samples from different classes. Although this objective can also separate samples from old and current tasks when they belong to different classes, it is not specifically designed to model the discrepancy introduced by compressed replay. In contrast, ${\mathcal{L}}_{\mathrm{dc}}$ is designed for the domain bias introduced by compressed replay and considers both the relationships between compressed and original representations and those between old and current tasks. As shown in Table 20, using ${\mathcal{L}}_{\mathrm{cls}}$ alone achieves CNN and NME accuracies of 79.08% and 79.28%, respectively. Adding ${\mathcal{L}}_{\sup }$ results in 79.01% and 79.42%, while adding the proposed ${\mathcal{L}}_{\mathrm{dc}}$ achieves 79.69% and 79.64%. Compared with ${\mathcal{L}}_{\sup }$, ${\mathcal{L}}_{\mathrm{dc}}$ improves the CNN and NME accuracies by 0.68% and 0.22%, respectively. These results indicate that considering the compression-related and task-related feature relationships provides better recognition performance than directly applying a standard supervised contrastive objective in our setting.
  Table 20. Comparison of different loss functions on UCF101 with 10 steps.

  Loss	CNN	NME
  ${\mathcal{L}}_{\mathrm{cls}}$	79.08	79.28
  ${\mathcal{L}}_{\mathrm{cls}}+{\mathcal{L}}_{\sup }$	79.01	79.42
  ${\mathcal{L}}_{\mathrm{cls}}+{\mathcal{L}}_{\mathrm{dc}}$	79.69	79.64

• Performance on old and new tasks. At each incremental training step, VNMBM learns the current task while retaining the knowledge of previously learned tasks. Therefore, we compute the average accuracy of the old tasks and the accuracy of the current task at each incremental training step. The results on the UCF101 dataset with 10 steps are shown in Figure 8. As shown in Figure 8, VNMBM maintains favorable performance on both old and current tasks throughout the incremental learning process.
  

  Figure 8. Average accuracies of old and new tasks at each incremental step on UCF101 with 10 steps.

5. Conclusions

In this paper, we propose VNMBM to address two challenges in memory-limited video class-incremental learning: preserving temporal and task-related information during video compression and mitigating the discrepancy between compressed and original video features. A STNM compactly encodes videos and approximate video features into neural representations for memory replay. CBC uses class-wise adaptation prompts and contrastive learning to improve feature alignment and discrimination. Their combination integrates memory-efficient replay with domain-bias mitigation to alleviate catastrophic forgetting. Experiments on UCF101, HMDB51, and Something-Something V2 show higher average incremental accuracy than the compared methods under the evaluated memory budgets and task settings. Under the TSM setting, VNMBM achieves average CNN/NME accuracies of 79.92%/79.75% on UCF101, 52.02%/54.69% on HMDB51, and 42.73%/42.29% on Something-Something V2 across the evaluated task splits within each dataset. These correspond to absolute CNN/NME gains of 2.40%/0.94%, 5.71%/4.21%, and 7.14%/7.44% over HCE, respectively. These results demonstrate that VNMBM consistently improves recognition performance across all three benchmarks.

• Limitations. The current implementation uses conventional 8-bit quantization to reduce the storage cost of STNM parameters. We adopt this scheme for its simplicity and predictable storage cost, allowing us to focus on the memory representation capability of STNM and the bias correction mechanism of CBC. Both the approximate original old task video features and the reconstructed compressed old task videos used by CBC are obtained from quantized STNMs. Therefore, quantization may affect both feature preservation in STNM and the feature representations used for alignment in CBC. CBC aims to mitigate the discrepancy between these representations, although quantization-induced errors may remain. More advanced quantization and parameter compression techniques have not been explored, leaving room to improve the trade-off between memory usage and recognition performance. In addition, class-specific STNM construction and CBC introduce training overhead, while NME evaluation requires video reconstruction before class-mean computation.
• Future work. Future work will investigate more effective quantization and model compression strategies for STNM while preserving temporal and task-related information. We will also explore more efficient neural memory construction and reconstruction to reduce the additional computational cost. These directions aim to improve the balance among replay-memory usage, training cost, and recognition accuracy.