Mitigating Catastrophic Forgetting in Incremental Learning Using Hybrid Approach: Interleaving Memory Replay and Parameter Regularization for Sequential Text Classification

Abstract

Catastrophic forgetting is a major challenge for deep learning models when they are incrementally trained on a sequence of new data. Reducing this forgetting in image and video data has been the primary research focus, but less attention has been given to textual domains, where discrete token distributions and semantic shifts occur across different topics. Furthermore, standalone strategies proposed for reducing catastrophic forgetting still have room for improvement. To this end, this paper proposes a synergy of stratified memory replay with parameter regularization for a BiLSTM-based incremental learning model to mitigate catastrophic forgetting in sequential text datasets. The stratified replay mechanism replays a small buffer of historical data samples into current training phases to preserve the old data patterns, while the parameter regularization penalizes modifications to the neural network weights crucial to the past tasks. The proposed approach is evaluated in an incremental training pipeline on distinct textual datasets, including software bug reports (Task A), a news dataset (Task B), and emails (Task C). The evaluation results demonstrate that the baseline neural network experiences catastrophic forgetting as its initial dataset (Task A) accuracy drops from $87.53\%$ to $11.75\%$. The standalone experience replay approach manages to retain Task A accuracy at $80.01\%$, down from its peak of $86.56\%$, while for Task B, it achieves $93.97\%$, down from the peak of $98.37\%$. The buffer sensitivity analysis indicates the model accuracy improves with increasing replay buffer size. The parameter regularization approach effectively reduces catastrophic forgetting, but it remains less effective for disruptive text distribution sequences, resulting in noticeable forgetting on prior tasks and reduced plasticity on later tasks. Evaluations of this approach show that Task A accuracy is reduced from $87.87\%$ to $71.09\%$ after training on Task C. The proposed hybrid approach reduces forgetting and preserves Task A and Task B accuracies at $83.73\%$ and $95.04\%$, respectively. Thus, the empirical evaluations demonstrate that the proposed approach is effective and outperforms the standalone experience replay strategy and parameter regularization, limiting the forgetting on the earliest task to just $4.65\%$ compared to $6.55\%$ forgetting of the replay-based method, while allowing enough plasticity for the final task to reach $98.92\%$ accuracy.

1. Introduction

Machine learning models have achieved remarkable performance across diverse domains [1,2], demonstrating high accuracy in tasks such as classification, object detection and recognition [2,3]. Standard deep learning models are generally trained on static data, on which they have performed exceptionally well. However, when such models are incrementally trained on new data, they tend to forget previously acquired knowledge. Natural intelligence is characterized by its ability to learn new concepts continually while preserving prior knowledge. This characteristic is the motivation behind incremental learning (IL). State-of-the-art models like Large Language Models (LLMs) need to incrementally learn from a stream of new and changing data continuously [4,5]. The IL model is expected to retain the knowledge of the tasks it learned previously and utilize the accumulated knowledge in facilitating the future learning. This is a desirable attribute, referred to as Forward Knowledge Transfer [1] or transfer learning in neural networks [6,7]. This knowledge transfer property expects that after training on a new dataset related to a previous one, the performance of the model on the previous dataset should increase or at least should not decrease. However, a primary challenge to IL is catastrophic forgetting (CF) [8,9], a phenomenon in which a model tends to forget some of the past knowledge upon learning new tasks, thereby impairing its performance on previously learned tasks. Deep neural networks suffer from catastrophic forgetting, as the gradient-based weight updates can overwrite previously acquired knowledge [10,11]. Approaches have been proposed to alleviate catastrophic forgetting in different scenarios [12,13,14]. Figure 1 illustrates the contrast between natural intelligence and artificial intelligence, which is prone to catastrophic forgetting. The top of the figure shows that the natural intelligence exhibits IL, acquiring new tasks sequentially while retaining past knowledge. In contrast, the lower part of the figure shows that as artificial neural networks learn new tasks, they tend to forget previous tasks, resulting in catastrophic forgetting.

Catastrophic forgetting has its consequences. Apart from degradation of performance on old tasks, its other challenges include the computational and resource inefficiency and limited scalability and adaptability and a trade-off between stability, which is preserving past knowledge, and plasticity, which is acquiring new knowledge [15]. The transformer models like BERT are advanced models for text-based data. Even these models, alongside smaller specialized language models, are not immune to this forgetting [16,17,18,19]. To mitigate catastrophic forgetting, existing research generally proposes standalone strategies [15]. The main established strategies include regularization-based, replay-based and the architectural modification-based techniques. Regularization-based approaches such as Elastic Weight Consolidation (EWC) [10] put constraints on updates to critical parameter weights to protect prior knowledge [15]. This constraint is effective in preserving old knowledge but results in model’s reduced plasticity to learn new distinct tasks. Replay-based methods such as the experience replay (ER) [20] approach work by interleaving a memory buffer of historical datasets into training streams of new dataset [11]. This helps stabilize the parameters to reduce forgetting but a large replay buffer results in memory overhead. Architectural-based methods work by isolating the task-specific parameters or dynamically expanding neural network. This helps reduce catastrophic forgetting but leads to parameter explosion over large sets of tasks. The ER and EWC are established techniques; the contribution of this work is their synergy and empirical validation on diverse sequential textual tasks, where this combination has not been benchmarked before.

Despite many research efforts, a comprehensive and generalized solution to catastrophic forgetting remains elusive. The approaches, including memory replay methods, parameter regularization and architectural modifications, have demonstrated success in specific scenarios but may suffer from limitations of generalizability across different task sequences and network architectures [21,22]. These approaches face the trade-off between stability and plasticity. To substantially reduce forgetting of past tasks, the replay-based methods need increased memory and computational resources [23], limiting their feasibility for large datasets. Regularization techniques can be insufficient to prevent significant past-data forgetting [24]. Given such limitations of the standalone methods, this paper exploits the benefit of combining different strategies in a hybrid approach that can leverage the strengths of individual methods, which is the motivation for this research. A majority of the catastrophic forgetting mitigation approaches target image and video datasets [5,11,25,26]. The area of text-based data has its own distinct challenges such as discrete token distributions, sparse semantic shifts, and rigid syntactic structures. This area still presents research opportunities [18,27].

This work proposes a hybrid ensemble approach to mitigate catastrophic forgetting for text-based data. By integrating experience replay buffers, and regularized parameter management with the BiLSTM model, the proposed approach effectively reduces CF while also allowing enough plasticity to learn new tasks. The primary contributions of this work include the following:

• This work proposes a new hybrid approach that integrates experience replay and parameter regularization to mitigate catastrophic forgetting in incremental learning models for sequential text datasets.
• The proposed approach is evaluated across diverse textual datasets, demonstrating gains in performance compared to the established standalone techniques.
• It evaluates sensitivity analysis with varying memory buffer size allocations to examine the effects on the forgetting.

The rest of this paper is organized into the following sections. Section 2 reviews the current literature. Section 3 presents the proposed hybrid approach against catastrophic forgetting. Section 4 details the experimental setup and results. The effectiveness of the proposed approach and its limitations are discussed in Section 5. Section 6 concludes the paper and discusses future work in this direction.

2. Related Work

Traditional deep learning algorithms work well on static, identical data distributions. IL, on the other hand, has become indispensable for modern applications where data streams are inherently dynamic. This non-static data requires incremental or continual learning to accommodate the changing data patterns. But such changing data patterns have created a major problem of catastrophic forgetting, impairing the retention of previously learned knowledge [15]. Mechanistically, this catastrophic forgetting stems from the stability–plasticity dilemma caused by gradient interference. This interference happens when the gradient of a new task’s loss (${\nabla }_{\theta }{\mathcal{L}}_k$) and a previous task’s loss (${\nabla }_{\theta }{\mathcal{L}}_{k-1}$) point in different directions in parameter space; any update that reduces ${\mathcal{L}}_k$ can increase ${\mathcal{L}}_{k-1}$. Regularization methods such as EWC constrain the parameters updates along directions the Fisher Matrix identifies as important to prior tasks, while replay methods reduce interference by including gradients from ${\mathcal{L}}_{k-1}$ directly in the same update. This framing is consistent with gradient-projection approaches to IL [28,29], and motivates why combining both mechanisms is more effective than relying on either approach in isolation.

2.1. Taxonomy of Catastrophic Forgetting Mitigation Approaches

To handle the problem of the stability–plasticity dilemma, which is the trade-off between a model’s stability in preserving old knowledge and its plasticity to take in new data patterns, existing strategies can be broadly divided into three major categories: replay-based, parameter regularization-based, and architecture-based methods [15,30,31].

2.1.1. Replay-Based Techniques

Memory replay-based approaches mitigate the forgetting by re-exposing the neural network to the historical data during the incremental training of new tasks. Although joint training of shuffled complete datasets provides an upper bound for mitigating forgetting by enforcing optimization over all observed distributions simultaneously, high memory requirements or privacy-driven data retention constraints often make it restrictive. The experience replay (ER) offers a solution to re-exposing complete data during incremental training by feeding episodic memory buffers of past data. Such techniques store a compact representative subset of past exemplars, and interleave them with the current training data [32,33,34]. Concurrently calculating gradients across both new data and the previous buffered exemplars, the optimization goal is to find a joint parameter subspace satisfying multiple task boundaries [35,36]. The memory replay-based techniques introduce memory overheads that scale poorly across long task sequences. Furthermore, an over-reliance on limited exemplars can induce overfitting or a severe degradation in network plasticity [37].

Generative memory replay: Researchers have leveraged generative models and brain-inspired replay mechanisms to generate pseudo-data resembling the data distributions of previous datasets when storage constraints prevent raw exemplar retention [38,39,40]. In Deep Generative Replay [41], for example, a generator is trained alongside the main model to reconstruct past data. This reduces need for exemplar storage from previous task but introduces the challenge of training and maintaining a robust generative model.

2.1.2. Regularization-Based Approaches

Instead of re-exposing the historical data samples, the regularization-based techniques constrain the models’ optimization on new data to safeguard previous knowledge. Such methods restrict large changes to the model parameters while incrementally training on new data. Parameter regularization identifies the model weights that are most important to the prior data and penalize significant changes to such weights during incremental training of subsequent tasks, which in turn improves stability. EWC method, for instance, estimates parameter importance using diagonal elements of the Fisher Information Matrix [10], adding a quadratic penalty to the loss function. Similarly, Synaptic Intelligence (SI) tracks online parameter trajectories and accumulates importance measures throughout the learning phase [42]. The augmented loss function for EWC is formalized as

$$L\left(\theta \right)=L_{\mathrm{new}}\left(\theta \right)+\lambda \sum _i{F}_i{({\theta }_i-{\theta }_i^{\ast })}^2$$

(1)

where $L_{\mathrm{new}}\left(\theta \right)$ represents the loss on the current (new) task, $\lambda$ denotes the regularization hyperparameter, $F_i$ is computed importance score of the i-th parameter, and ${\theta }_i^{\ast }$ represents the old parameter values optimized when prior tasks training is completed.

Precisely predicting the most important parameters for previous tasks is hard, due to complex correlations between the behavior of a machine learning model and its parameters. To address this issue of parameter regularization, a more effective approach is proposed by researchers, which applies regularization in the function space of the machine learning model [43]. Functional regularization, another type of regularization, is applied at different levels of neural network representations, in which case it is also called feature distillation [44,45]. Functional regularization aims to protect against big changes to input–output mapping of a network at a set of specific inputs, called the anchor points.

2.1.3. Architecture-Based Approaches

The replay-based and regularization-based approaches learn all the incremental tasks using a single model with a shared set of parameters. This may cause inter-task interference, which results in forgetting [15]. Adding parameters that are task-specific can address this issue. Depending on whether the network architecture is fixed or not, this can be divided into two possibilities: parameter isolation and dynamic architecture. Fixed architectures leverage techniques such as iterative structural pruning, activation-based masking, or uncertainty estimation to identify and lock parameters that are vital to current task [46,47,48]. The remaining unallocated or low-importance parameters are released for future learning. Though effective, setting strict sparsity constraints can limit the model, reducing its performance on new arriving tasks. To alleviate this capacity bottleneck, dynamic architectures expand the neural network model, adding modular components or layers when current capacities are insufficient for the new data distributions [49].

2.2. Hybrid and Ensemble Approaches

Given the limitations of the individual methods against catastrophic forgetting, recent works have focused on hybrid methods that integrate multiple individual techniques to address the stability–plasticity trade-off [50]. For heterogeneous datasets, [51] employed Gradient Memory-based federated learning, combining gradient refinement with memory-based approaches to mitigate forgetting. Generative models such as GANs are being used to synthesize pseudo-data for replay. Combined with other techniques such as architectural modifications, they have demonstrated mitigating the forgetting without storing real historical data. [52] propose ZeroFlow, where they apply hybrid zero-order optimization techniques, combining first-order optimization for initial learning with zero-order methods for reducing the loss function, integrating elements of regularization and optimization within a hybrid framework. Approaches like reinforced interactive continual learning [53] leverage LLMs and reinforcement learning to learn new knowledge while handling noisy feedback and retaining prior knowledge, reducing catastrophic forgetting. The current hybrid approaches, however, suffer from high computational complexity or sensitivity to single-model parameter drift.

2.3. Incremental Learning in the Era of Large Language Models (LLMs) and Multi-Agent Systems

State-of-the-art techniques such as LLMs and multi-agent systems have transformed autonomous task execution. The continuous fine-tuning through sequential data, however, exacerbates the challenge of catastrophic forgetting [54]. Recent research on catastrophic forgetting in LLMs and multi-agent LLM systems is motivated by the fact that LLMs are updated incrementally on new data [5]. Parameter fine-tuning methods such as LoRA [55,56,57] and adapters [58] are recently being used as alternative to full-model regularization, since recomputing a full Fisher Information Matrix over billions of parameters is computationally prohibitive [34]. In complex collaborative environments, researchers have proposed the multi-agent systems to coordinate the LLMs across distinct sub-tasks. However, sequential updates to agent parameters risk degrading the capabilities and domain specific knowledge required for prior tasks. To this end, modern architectures have relied on hybrid IL paradigms, combining parameter regularization with adaptive memory replay to counter parameter degradation in LLMs [59]. Recent advances in cross-domain transfer learning and source-free domain adaptation have demonstrated necessity of aligning feature representations under shifting operational distributions without relying on continuous access to source data. Although these paradigms are effective in signal processing and industrial prognostics [60,61], adapting them to sequential text classification requires addressing discrete token distributions, vocabulary overlap drift, and semantic class imbalance over incoming tasks.

The multi-agent systems leverage fixed LLMs to mitigate weight-level catastrophic forgetting, but they face the problem of context-level memory decay and role drift over extended interactions. Ref. [62] introduced structured role-framing and shared artifact management within LLM-team workflows, preserving initial problem constraints and agent personas over multi-turn sessions. Furthermore, incorporating episodic memory architectures with cognitive task analysis enables multi-agent teams to maintain persistent spatial states during complex, dynamic coordination scenarios [63]. However, when such LLM agents accumulate thousands of evacuation trajectories over time, new spatial episodes dilute, reducing the retrieval relevance of older but highly crucial crisis patterns such as rare edge-case hazard responses. Without explicit consolidation or regularization mechanics in the memory, the system may suffer from episodic retrieval degradation. Traditional regularization methods like EWC protect crucial parameters via Fisher Information constraints, but suffer from limited plasticity when exposed to new highly disruptive text distributions [22,64]. Consequently, recent techniques deploy multi-agent coordination where memory agents apply experience replay based on parametric drift, stabilizing shared representations across shifting task distributions [59]. Multi-agent LLM systems also experience context degradation, role drift, and memory decay over long interactions, relying on memory-like mechanisms, showing necessity of the replay-based strategies [62,63]. These findings motivate the discussion in Section 5.3 and Section 6, where possible scaling of the proposed hybrid approach to adapter-based, LLM-scale IL is discussed.

3. The Proposed Hybrid Approach

The proposed approach aims to mitigate catastrophic forgetting and address the stability–plasticity trade-off in sequential text-based IL. This work proposes a BiLSTM-based hybrid IL approach that is the synergy of stratified experience replay with parameter regularization. This section details the sequential task framework, underlying neural network model, and the dual-constraint optimization strategy.

3.1. Problem Formulation and Sequential Task Setting

The proposed approach works on IL paradigm consisting of a sequence of K distinct text classification task datasets, denoted as $\mathcal{T}=\{T_1,T_2,\dots ,T_K\}$. In this study, there are three text datasets ($K=3$):

• Task A (${\mathit{T}}_{\mathbf{1}}$): Software bug report status classification (${\mathcal{D}}_1$).
• Task B (${\mathit{T}}_{\mathbf{2}}$): Topic classification from the AG News dataset (${\mathcal{D}}_2$).
• Task C (${\mathit{T}}_{\mathbf{3}}$): Spam and ham classification from the Email dataset (${\mathcal{D}}_3$).

Each task $T_k$ has its own data distribution ${\mathcal{D}}_k={\left\{(x_i^k,y_i^k)\right\}}_{i=1}^{N_k}$, where $x_i^k$ represents a text sequence and $y_i^k\in \{0,1\}$ is its corresponding binary label. These individual task datasets are trained incrementally on a BiLSTM-based model. When training on new task $T_k$, the model loses direct access to complete prior data distributions $\{{\mathcal{D}}_1,\dots ,{\mathcal{D}}_{k-1}\}$. The objective is to learn a unified set of network parameters $\theta$ that minimizes the loss or forgetting on previously trained task datasets domains:

$$\underset{\theta }{\min }{\displaystyle \sum _{k=1}^K}{\mathbb{E}}_{(x,y)\sim {\mathcal{D}}_k}\left[\mathcal{L}\left(f(x;\theta ),y\right)\right]$$

(2)

where $\mathcal{L}(·)$ denotes the binary cross-entropy loss function.

3.2. Hybrid Architecture and Vector Representation

To process sequential text datasets, a BiLSTM-based deep neural network architecture $f\left(\theta \right)$ is shared across all tasks. Raw textual inputs are passed through a tokenizer calibrated over a unified vocabulary space $\mathcal{V}$ where $\left|\mathcal{V}\right|=20,000$. The inputs are mapped to token indexes and padded or truncated to the fixed sequence length $L=150$. The neural network pipeline comprises an embedding layer mapping discrete tokens to vectors ($d_e=64$), followed by a Bidirectional Long Short-Term Memory (BiLSTM) layer (64 units per direction). The concatenated hidden states ($h\in {\mathbb{R}}^{128}$) pass through a $50\%$ dropout regularization phase and a Dense layer (32 units with ReLU activation). The last layer of the neural network is a single-node Sigmoid output layer for binary outputs.

BiLSTM-based architecture is deliberately chosen over pretrained Transformer architecture or LLMs to evaluate and address catastrophic forgetting in isolation. The proposed replay–regularization coupling approach eliminates major experimental confounds like large-scale pretraining, subword tokenization, and attention layers. This allows the reported forgetting and mitigation effects to be attributed more directly to the sequential training dynamics rather than to properties inherited from a pretrained Transformer model. The observed gains may not transfer directly to a Transformer-based or pretrained LLM architecture, and is addressed as a limitation in Section 5.3 and future work in Section 6.

The standalone approaches have their limitations. Experience replay, for instance, risks overfitting to small buffers while parameter regularization limits plasticity for future tasks apart from their memory and computation overheads. The proposed approach constructs a dual-constraint architecture running on low-level gradient optimization paths. As illustrated in Figure 2, the proposed hybrid approach bifurcates the forgetting reduction strategy into two protective mechanisms that interleave during the gradient update phase.

3.2.1. Data Constraints: Stratified Experience Replay

The upper part of Figure 2 depicts the data-space constraint. Upon completion of learning of each task $T_k$, an episodic memory buffer ${\mathcal{M}}_k\subset {\mathcal{D}}_k$ is extracted and stored. To prevent representation bias during downstream task training, a stratified sampling strategy is enforced to maintain the previous tasks’ class distributions: $|{\mathcal{M}}_k|=M$, subject to

$${\displaystyle \frac{\left|\right\{x\in {\mathcal{M}}_k:y=1\left\}\right|}{|{\mathcal{M}}_k|}}={\displaystyle \frac{\left|\right\{x\in {\mathcal{D}}_k:y=1\left\}\right|}{|{\mathcal{D}}_k|}}$$

(3)

where the buffer capacity is constrained to $M=1000$ samples per task. The left fraction of Equation (3) is the ratio of positive samples in the memory buffer while the right fraction is the ratio of positive samples in the original dataset. The equation states that the percentage of the positive class inside the small memory buffer matches the percentage of that same class in the original dataset. Stratified sampling (Equation (3)) is crucial for datasets with high class imbalance, such as Task A (Bugs, class ratio ≈ 1:7.85). Using random sampling, an episodic buffer of capacity $M=1000$ carries high variance and is vulnerable to under-representing minority class exemplars, leading to noisy gradient estimates and severe decision boundary drift during incremental training of new tasks. Stratified sampling makes sure that the empirical class distribution of the original dataset ${\mathcal{D}}_k$ is strictly preserved in memory ${\mathcal{M}}_k$. The replayed gradients with proportional class representation provide a balanced signal to stabilize the historical decision boundaries and protect prior tasks knowledge retention across sequential training tasks. During training of downstream tasks, the active batch is formed at the Interleave Junction (U), concatenating the incoming task stream with the historical buffer:

$${\mathcal{B}}_{combined}={\mathcal{D}}_k\cup \left({\displaystyle \bigcup _{j=1}^{k-1}}{\mathcal{M}}_j\right)$$

(4)

This interleaved stream is fed into the Neural Network Backbone to compute the classification loss, ${\mathcal{L}}_{base}$. Stratified sampling (Equation (3)) is important for Task A, where the underlying class ratio (≈1:7.85) makes random sampling more likely to under-represent the minority class in a buffer of only $M=1000$ samples by chance. For datasets having class distributions close to balanced, stratified and random sampling would mostly give similar performance. This suggests stratification is not a uniformly critical design choice across all tasks, but is specifically important for datasets with skewed label distributions like in the Bugs dataset.

3.2.2. Parameter Constraints: Cumulative Regularization

The lower part of Figure 2 shows the parameter regularization constraint, implemented through EWC. The proposed architecture computes a joint importance profile over all historical instances instead of calculating independent tracking constraints per task dataset, as it can lead to an over restrictive optimization. This is a deliberate design choice instead of an oversight. The model uses $\theta$ in three distinct contexts to reduce catastrophic forgetting: $\theta$ represents parameters (weights and biases) of the neural network backbone for current training task. ${\theta }_i$: The specific i-th trainable parameter in the current model. ${\theta }_{k,i}^{\ast }$ represent the historical anchor parameter representing the optimal snapshot of the i-th weight after completing Task k. $F_i$ is the Empirical Fisher Information Matrix diagonal value for parameter i, representing its importance to previous tasks.

Computing independent Fisher Matrices per task and penalizing deviation from each task’s ${\theta }_k^{\ast }$ preserves the task’s isolated importance profile, but total penalty value grows linearly with the number of tasks K, and overlapping important parameters across tasks would be penalized redundantly, progressively locking the neural network, ultimately reducing the plasticity for learning downstream tasks. The cumulative estimation used here instead blends importance signals into a single joint profile, at the cost of potentially under-protecting parameters that were highly important to only one earlier task if their signal is diluted by more recent buffer data. This refers to choosing bounded regularization cost over exact per-task accuracy, which is consistent with the result showing that the hybrid model retains strong plasticity (Task C: $98.92\%$) even after three sequential tasks.

The diagonal elements of Fisher Information Matrix F are used to calculate the importance of parameter ${\theta }_i$. For a composite of historical datasets ${\mathcal{D}}_{\mathrm{hist}}={\bigcup }_{j=1}^{k-1}{\mathcal{M}}_j$, partitioned into B batches, the parameter is formally derived by accumulating variance of first-order gradients given by

$$F_i={\displaystyle \frac{1}{B}}{\displaystyle \sum _{b=1}^B}{\left({\displaystyle \frac{\partial \mathcal{L}(f(x_b;\theta ),y_b)}{\partial {\theta }_i}}\right)}^2$$

(5)

This matrix tracks the EWC Penalty Loss (${\mathcal{L}}_{\mathrm{reg}}$) and penalizes the neural network if it attempts to alter the weights that the Fisher Matrix identifies as important to previous tasks. ${\theta }_i^{\ast }$ represents the optimal reference parameters locked at the end of training of the previous task.

The empirical Fisher in Equation (5) approximates Fisher Information Matrix using observed labels rather than marginalizing over the model’s predictive distribution which introduces bias under class imbalance. This is relevant specifically for Task A, as shown in Table 5, where the Bugs dataset has a high class imbalance (1:7.85), while News and Emails have balanced class distributions of 1:1 and 0.96:1 respectively. Consequently, gradients computed on the Bugs buffer ${\mathcal{M}}_1$ are dominated by the majority class, and the resulting Fisher estimate may under-represent parameter importance for minority class decision boundary specifically for Task A. This parameter under-representation does not meaningfully apply to News and Emails datasets due to their near-balanced class label distributions.

Mechanistic Comparison: Cumulative vs. Task-by-Task Fisher Estimation. In traditional task-by-task EWC, parameter constraints are computed sequentially after each isolated task step ($F^{\left(1\right)},F^{\left(2\right)},\dots ,F^{(k-1)}$). This results in memory overhead and overwriting critical parameters problem. The memory overhead and constraint complexity scale linearly with the number of tasks, causing over-regularization and restricting model plasticity on new data. Another drawback is because each $F^{\left(k\right)}$ is evaluated at a local task minimum, subsequent updates frequently overwrite parameters critical to earlier tasks. In contrast, our unified cumulative strategy evaluates ${\mathcal{F}}_{\mathrm{cum}}$ over the consolidated historical memory buffer ${\mathcal{M}}_{\mathrm{hist}}$. This provides three key advantages: Bounded Constraint Footprint: It maintains a single, static parameter vector regardless of sequence length, preserving model plasticity for downstream tasks. Global Importance Tracking: It evaluates parameter sensitivity across all historical task distributions simultaneously, accurately identifying parameters that are globally optimal. Optimization Stability: It consolidates conflicting regularizer penalty terms into a single, balanced restoring force during backpropagation (${\nabla }_{\theta }{\mathcal{L}}_{\mathrm{reg}}=\lambda ·{\mathcal{F}}_{\mathrm{cum}}\odot (\theta -{\theta }^{\ast })$), preventing gradient competition across task boundaries.

3.2.3. Consolidated Parameter Optimization Loop

A standard high-level neural network compiler optimizes for a single objective. To enforce the hybrid constraints, instead of standard compilation, the approach utilizes a custom automated differentiation block. As shown in the final stages of Figure 2, the optimization loop minimizes a combined objective function ${\mathcal{L}}_{\mathrm{total}}$:

$${\mathcal{L}}_{\mathrm{total}}\left(\theta \right)={\mathcal{L}}_{\mathrm{base}}({\mathcal{B}}_{\mathrm{combined}};\theta )+{\displaystyle \frac{\lambda }{2}}\sum _i{F}_i{\left({\theta }_i-{\theta }_i^{\ast }\right)}^2$$

(6)

where $\lambda =15.0$ is the value of the regularization penalty. This specific value was determined through an empirical grid search, balancing retention of historical decision boundaries against necessary gradient plasticity required to minimize ${\mathcal{L}}_{base}$ for the incoming text distributions. The gradient computes backward pass across this multi-loss scenario (${\nabla }_{\theta }{\mathcal{L}}_{\mathrm{total}}$), updating the shared parameters safely. Updated weights are then fed forward iteratively, ensuring neural network learns the distinct distributions of new tasks ${\mathcal{D}}_k$ while keeping optimization minima of previously learned task datasets.

3.2.4. Custom Gradient Dynamics via Differentiation Tape

Standard neural network optimization pipelines compute loss gradients strictly over single mini-batch distributions. To implement the hybrid optimization architecture illustrated in Figure 2, the proposed approach executes dual loss evaluation and automatic differentiation using a custom gradient tape (tf.GradientTape) context. The gradient tape is a differentiation engine that records operations performed on tensors during a forward pass to compute gradients during the backward pass. The joint loss functional ${\mathcal{L}}_{\mathrm{total}}\left(\theta \right)$ evaluated over the interleaved batch ${\mathcal{B}}_{\mathrm{combined}}={\mathcal{D}}_k\cup {\mathcal{M}}_{\mathrm{hist}}$ is formulated as

$${\mathcal{L}}_{\mathrm{total}}\left(\theta \right)={\mathcal{L}}_{\mathrm{base}}({\mathcal{B}}_{\mathrm{combined}};\theta )+{\displaystyle \frac{\lambda }{2}}{\displaystyle \sum _{i=1}^N}{F}_i{({\theta }_i-{\theta }_i^{\ast })}^2$$

(7)

During the backward pass inside the gradient tape context, parameter updates are derived via the linearity of the differentiation operator:

$${\nabla }_{\theta }{\mathcal{L}}_{\mathrm{total}}\left(\theta \right)={\nabla }_{\theta }{\mathcal{L}}_{\mathrm{base}}({\mathcal{B}}_{\mathrm{combined}};\theta )+\lambda ·F\odot (\theta -{\theta }^{\ast })$$

(8)

where ⊙ denotes the element-wise Hadamard product (element-wise product on two matrices of the same size) between the diagonal Fisher vector F and the parameter displacement vector $(\theta -{\theta }^{\ast })$. Combined gradient formulation ensures that in the case of gradient vectors being driven by incoming task samples’ (${\nabla }_{\theta }{\mathcal{L}}_{\mathrm{base}}$) conflict with historical task decision boundaries, the quadratic vector $\lambda ·F\odot ({\theta }_t-{\theta }^{\ast })$ directs the optimization trajectory back along the parameter dimensions critical to the memory of past data.

3.3. Algorithmic Implementation

The proposed approach from isolated initialization to the hybrid protective loop is depicted in Algorithm 1.

Algorithm 1 Proposed hybrid incremental learning approach

Require:  Sequential Datasets $\{{\mathcal{D}}_1,{\mathcal{D}}_2,{\mathcal{D}}_3\}$, Buffer Size $M=1000$, Penalty Strength $\lambda =15.0$   1: Initialize the shared parameter configurations $\theta$   2: Phase 1: Train $f(·;\theta )$ on ${\mathcal{D}}_1$ using standard Adam optimizer ($\eta =2\times {10}^{-3}$)   3: Extract buffer ${\mathcal{M}}_1$ from ${\mathcal{D}}_1$   4: Compute initial Fisher Map F and store the weights ${\theta }^{\ast }\leftarrow \theta$ using ${\mathcal{M}}_1$   5: for Task $k\in \{2,3\}$ do   6:    Interleave new datasets to construct ${\mathcal{D}}_{\mathrm{combined}}={\mathcal{D}}_k\cup \left({\bigcup }_{j=1}^{k-1}{\mathcal{M}}_j\right)$   7:    for each mini-batch $(X_b,y_b)\sim {\mathcal{D}}_{\mathrm{combined}}$ do   8:      Open custom tf.GradientTape() context   9:      Compute classification loss: ${\mathcal{L}}_{\mathrm{base}}=\mathrm{BCE}(f(X_b;\theta ),y_b)$ 10:      Compute quadratic penalty: ${\mathcal{L}}_{\mathrm{reg}}={\displaystyle \frac{\lambda }{2}}{\sum }_i{F}_i{({\theta }_i-{\theta }_i^{\ast })}^2$ 11:      Calculate total consolidated gradient: ${\nabla }_{\theta }{\mathcal{L}}_{total}={\nabla }_{\theta }({\mathcal{L}}_{base}+{\mathcal{L}}_{reg})$ 12:      Update parameter tracks: $\theta \leftarrow \theta -\eta ·{\nabla }_{\theta }{\mathcal{L}}_{total}$ 13:    end for 14:    Extract buffer ${\mathcal{M}}_k$ from ${\mathcal{D}}_k$ 15:    Update consolidated past validation data: ${\mathcal{D}}_{\mathrm{hist}}={\bigcup }_{j=1}^k{\mathcal{M}}_j$ 16:    Re-calculate empirical Fisher profiles F over ${\mathcal{D}}_{\mathrm{hist}}$ 17:    Update reference weight snapshots: ${\theta }^{\ast }\leftarrow \theta$ 18: end for

Algorithm 1 shows the working of the proposed hybrid approach. It takes the BiLSTM-based neural network from unconstrained IL and shifts it into a managed loop designed to preserve past-task memory.

4. Experimental Setup and Results

In this section, empirical evaluation of the proposed approach is detailed. Empirical validation on diverse sequential textual tasks is performed to benchmark the proposed approach against isolated ER and EWC techniques. The selected dataset corpora, performance metrics, experiment pipeline including preprocessing, and comparative analysis of experimental results are outlined in this section.

4.1. Dataset Configuration

To evaluate a method’s stability across varying text lengths, vocabularies, and classification complexities, multiple distinct datasets are recommended [65]. Therefore, for the proposed approach evaluation, three distinct textual datasets are utilized. Each dataset is mapped to a binary classification problem to enable sequential processing by a unified model architecture.

• Bug Reports Dataset (Task A): This is a corpus of software bug reports. This dataset contains description field where the bug is described in text form. The ’rs’ (resolution status) column in this dataset has multiple classes. The reports are categorized into bug (Class 1) if the resolution status is FIXED and non-bug (Class 0) for other status values including INVALID, WORKSFORME, DUPLICATE and WONTFIX.
• AG News Dataset (Task B): A news corpus with formal vocabulary in Description field. While the original dataset has four classes, to convert it into a binary target, it is filtered for the Business and Sports categories, mapped to Class 1 and Class 0, respectively.
• Enron Spam Dataset (Task C): This corpus of Emails has ’Message’ field with email text and ’Label’ field has two classes: legitimate emails (ham, Class 0) and unsolicited emails (spam, Class 1).

The statistical data split distributions for the three datasets for incremental training are given in

Table 1. Data distribution profiles across target task datasets.

The	Total	Data Split		Target Label Distribution
	Records	Train	Test	Class 0	Class 1
Task A (Bugs)	11,827	9461	2366	1336	10,491
Task B (News)	63,800	51,040	12,760	31,900	31,900
Task C (Emails)	33,664	26,931	6733	16,493	17,171

.

4.2. Performance Evaluation Metrics

The quantitative evaluation of the approach on BiLSTM-based IL architecture is based on three primary metrics:

• Accuracy ($\mathit{A}$): The proportion of correctly predicted observations.
• Error Rate: This is simply calculated as $1.0-A$.
• Forgetting ($\mathit{F}$): Forgetting quantifies the performance degradation for a previously learned task dataset after the model is incrementally trained on subsequent tasks. For any given Task i, forgetting metric after training on T tasks is defined as

$$F_i=\underset{t\in \{1\dots T-1\}}{\max }\left(A_{t,i}\right)-A_{T,i}$$

(9)

where $A_{t,i}$ represents the accuracy of task i at time step t.

4.3. Experimental Preprocessing Pipeline

To ensure consistency across distinct task dataset domains, all datasets pass through a unified preprocessing pipeline before training. A global vocabulary dictionary is initialized with the 20,000 most-frequent tokens across all three datasets, while leaving the unseen words to an <OOV> token. All text sequences are either zero-padded or truncated to a uniform length of 150 words to obtain fixed-shape input requirement of deep neural networks. Finally, each dataset is split as $80/20$ for training and testing subsets respectively, utilizing stratified sampling to help preserve the class distributions.

4.4. The Incremental Learning Evaluation Results

To demonstrate catastrophic forgetting and the effectiveness of the proposed mitigation approach, experiments were done and evaluated on three distinct architectural setups: Baseline Sequential, experience replay, and the proposed hybrid approach. These were evaluated over the training sequence: Phase 1 (Bugs), Phase 2 (News, with Bugs replay where applicable), and Phase 3 (Emails, with Bugs and News replay where applicable). To accelerate convergence, base learning rate of the Adam optimizer is set to $2\times {10}^{-3}$ for the initial phase and reduced to $5\times {10}^{-4}$ for subsequent replay phases.

4.4.1. The Incremental Learning Baseline

Baseline columns in

Table 2. Comparison of incremental learning approaches across sequential training phases.

Training Phase	Task	Baseline Model			Experience Replay			Param. Regularization			Proposed Hybrid
		Acc.	Err.	Forg.	Acc.	Err.	Forg.	Acc.	Err.	Forg.	Acc.	Err.	Forg.
1. Post-Bugs	Bugs (A)	87.53%	12.47%	0.00%	86.56%	13.44%	0.00%	87.87%	12.13%	0.00%	88.38%	11.62%	0.00%
	News (B)	–	–	–	–	–	–	–	–	–	–	–	–
	Emails (C)	–	–	–	–	–	–	–	–	–	–	–	–
2. Post-News	Bugs (A)	85.38%	14.62%	2.16%	85.38%	14.62%	1.18%	74.43%	25.57%	13.44%	86.22%	13.78%	2.16%
	News (B)	98.56%	1.44%	0.00%	98.37%	1.63%	0.00%	95.97%	4.03%	0.00%	98.29%	1.71%	0.00%
	Emails (C)	–	–	–	–	–	–	–	–	–	–	–	–
3. Post-Email	Bugs (A)	11.75%	88.25%	75.78%	80.01%	19.99%	6.55%	71.09%	28.91%	16.78%	83.73%	16.27%	4.65%
	News (B)	51.36%	48.64%	47.20%	93.97%	6.03%	4.40%	95.33%	4.67%	0.64%	95.04%	4.96%	3.25%
	Emails (C)	99.98%	0.01%	0.00%	99.78%	0.22%	0.00%	88.50%	11.50%	0.00%	98.92%	1.08%	0.00%

clearly demonstrates catastrophic forgetting as the model incrementally trains on downstream datasets. The model also demonstrates high plasticity by learning subsequent Task B with accuracy of $98.56\%$ and Task C with accuracy of $99.98\%$. This high accuracy shows the model has enough plasticity. But, as it over-optimizes its weights for last task (Emails) during Stage 3, the parameters crucial for previous tasks are changed. This reduces Task A’s (Bugs Dataset) accuracy to only $11.75\%$, which is a severe forgetting of $75.78\%$, visually demonstrated in Figure 3. This result proves that IL models suffer from catastrophic forgetting.

4.4.2. Mitigation Using Experience Replay

The experience replay mechanism allows a significant stabilization of the decision boundaries. By periodically reintroducing a representative subset of historical samples from previous task datasets into the current training loop, structural constraints are enforced on weight optimization. Upon completing training on Emails in Stage 3, the replay technique retains an $80.01\%$ accuracy on Task A, restricting total forgetting to only $6.55\%$. This is a massive accuracy preservation compared to the baseline IL model.

4.4.3. Forgetting Mitigation Using Parameter Regularization

To isolate the effects of the regularization component, a standalone parameter regularization (EWC-only) baseline was evaluated. As shown in

Table 3. Performance of the parameter regularization approach across sequential training phases.

Training Phase	Evaluated Task	Accuracy	Error Rate	Forgetting
1. Post-Bugs Train	Bugs (A)	87.87%	12.13%	0.00%
2. Post-News Train	Bugs (A)	74.43%	25.57%	13.44%
	News (B)	95.97%	4.03%	0.00%
3. Post-Email Train	Bugs (A)	71.09%	28.91%	16.78%
	News (B)	95.33%	4.67%	0.64%
	Emails (C)	88.50%	11.50%	0.00%

, the forgetting mitigation approach using parameter regularization successfully learns the initial tasks and retains high plasticity for Task B (95.97%). However, it struggles to maintain stability during the highly disruptive Task C (Emails) training phase. Accuracy on Task A drops to 71.09% (a 16.78% forgetting rate). Furthermore, strict weight penalties slightly inhibit the model’s plasticity for the final task, achieving only 88.50% accuracy on Task C compared to the baseline’s 99.98%. This demonstrates that regularization alone is insufficient for the sequence of varying text distributions.

4.4.4. The Proposed Hybrid Approach Performance

While the standalone memory replay reduces forgetting, the proposed hybrid approach demonstrates a better capacity to resolve the stability–plasticity dilemma. By interleaving stratified memory replay with the parameter protection, the hybrid approach maintains robust backward transfer even after the highly disruptive Emails task (Phase 3) training.

Table 2 shows that the hybrid approach successfully absorbs the final Email dataset with 98.92% accuracy while maintaining an impressive 95.04% on intermediate News task (with only 3.25% forgetting rate) and with the Task A accuracy at 83.73%. This results in highly reduced forgetting of the first-learned task (Bugs dataset) to a mere 4.65%. Figure 4 compares the performance degradation of different techniques, demonstrating that the proposed hybrid approach ensures long-term knowledge retention across distinct text distributions without compromising the plasticity to learn new tasks during incremental training.

Table 4. Component-wise ablation: Proposed hybrid approach, replay and regularization (Post-Email, Task A).

Configuration	Memory Replay	Regularization	Task A Forgetting
Baseline Sequential	✗	✗	75.78%
Regularization-only (EWC)	✗	✓	16.78%
Experience Replay-only	✓	✗	6.55%
Proposed Hybrid Approach	✓	✓	4.65%

isolates each method’s marginal contribution. Regularization alone reduces forgetting by $59.34\%$ relative to baseline, replay alone by $68.26\%$, and the hybrid approach by $71.98\%$, which translates to a further $3.72\%$ and $12.64\%$ performance gain over the standalone replay-based and regularization approaches respectively. This confirms that the two mechanisms are complementary rather than redundant as the performance of the proposed hybrid approach exceeds either individual techniques, though the marginal gain from adding regularization on top of memory replay is modest.

4.5. Buffer Size Sensitivity

The size of memory replay buffer in the replay technique affects the accuracy of the model. Results of memory buffer sensitivity analysis presented in Figure 5 and Figure 6 demonstrate that increasing the replay buffer size provides a significant boost in performance. The model based on the experience replay technique with a buffer size of 0 suffers from severe catastrophic forgetting. By the end of Phase 3, this model’s accuracy on the initial Bugs dataset plummeted to $11.75\%$, which translates to a $75.78\%$ forgetting, while the News dataset experienced a $47.20\%$ drop. Increasing its size from 500 to 1000 shows some further performance gains. However, this performance gain mostly flattens as the buffer size is increased to 2000.

Task A (Bugs) shows the highest sensitivity to buffer size among the three task datasets (forgetting drops from $75.27\%$ at $M=0$ to $12.64\%$ at $M=500$). Two measurable properties of Task A result in such high dependency on the buffer size. Firstly, the vocabulary overlap analysis (top-2000 tokens per task, using Jaccard similarity) shows Task A dataset shares significantly less vocabulary with the subsequent tasks (Bugs–News: 0.150, Bugs–Emails: 0.199) than News and Emails share with each other (0.345,

Table 5. Vocabulary overlap and class balance across task datasets (class ratios evaluated on training splits, $N_{\mathrm{train}}$).

Task Pair/Task	Vocabulary Overlap (Jaccard, Top-2000)	Class Ratio (0:1)
Bugs–News	0.150	–
Bugs–Emails	0.199	–
News–Emails	0.345	–
Task A (Bugs)	–	1069:8392 (≈1:7.85)
Task B (News)	–	25,520:25,520 (1:1)
Task C (Emails)	–	13,194:13,737 (≈0.96:1)

). The vocabulary overlap is measured at the token level and does not directly measure retained embedding-space similarity. Because later training phases reinforce very little of Bugs’ vocabulary incidentally, the model depends almost entirely on replay buffer to retain Task-A-specific representations. Secondly, Task A is also the only highly class-imbalanced dataset among the three datasets (training split ratio 1069:8392, $N_{\mathrm{train}}=9461$, approximately 1:7.85; versus a near-perfect 1:1 split for both News and Emails; Table 5); hence, a small buffer is more likely to under-represent Task A’s minority class, compounding the vocabulary effect. These two properties combine to make Task A disproportionately dependent on adequate buffer coverage, consistent with its steeper sensitivity curve in Figure 5. The empirical necessity of stratified replay is further highlighted by vocabulary overlap and class imbalance profiles summarized in Table 5. Task A shares minimal vocabulary overlap with subsequent tasks (Jaccard similarity of 0.150 with News and 0.199 with Emails), meaning later tasks provide virtually no incidental semantic reinforcement for Task A’s vocabulary. Coupled with Task A’s high class imbalance (1069:8392), random sampling would frequently fail to capture representative semantic coverage of the minority class within a small buffer. Stratified empirical replay explicitly counters this by ensuring both broad semantic coverage and structural class balance, directly explaining why Task A exhibits the highest retention sensitivity to buffer quality.

Adding an experience replay buffer reduces forgetting. Using a buffer with 500 samples restores final Bugs task accuracy to $74.09\%$ and News task accuracy to $90.47\%$. As the buffer size increases to 2000, forgetting is further minimized ($5.45\%$ and $3.86\%$ for Bugs and News, respectively), preserving the Bugs dataset accuracy at $82.42\%$. Even with a larger memory buffer size in the replay-based approach, the proposed hybrid approach outperforms it with smaller replay buffer.

Buffer size affects direct storage and compute cost.

Table 6. Total training time vs. replay buffer size (Phases 1–3 combined).

Buffer Size (M)	Total Time (s)	$\mathbf{\Delta }$ vs. Prior (s)	$\mathbf{\Delta }$ vs. Prior (%)
500	478.8	–	–
1000	529.7	+50.9	+10.6%
2000	644.5	+114.8	+21.7%

reports the total training time on our Linux machine for Phases 1–3 for each buffer size tested. Increasing the buffer from 500 to 1000 samples raised total training time by 50.9 s ($+10.6\%$), and from 1000 to 2000 samples raised it by a further 114.8 s (+21.7%), for a cumulative increase of 34.6% in going from $M=500$ to $M=2000$. This shows that doubling the buffer size doubles the number of replayed samples concatenated into training data of each downstream phase, directly increasing both memory footprint and per-epoch training time. Since Figure 5 and Figure 6 show accuracy and forgetting gains largely flattening after $M=1000-2000$ buffer, while training cost continues to grow roughly linearly over the same range, $M=1000$ represents a reasonable balance point for the task datasets studied here, capturing most of the achievable forgetting reduction without the additional $21.7\%$ training-time overhead incurred by doubling buffer to $M=2000$ exemplars from $M=1000$.

5. Discussion

The results in the previous section demonstrate that applying an isolated approach such as the replay method tends to experience more catastrophic forgetting when a model is trained incrementally, as it leans to more plasticity to learn new data. The proposed hybrid approach balances the trade-off between stability (keeping old knowledge) and plasticity (learning new data). This section concludes the hybrid model working, why it fits text data well, and its limitations.

5.1. Effectiveness of the Hybrid Model on Text Data

In learning text tasks sequentially, the vocabulary and semantics change between datasets. This leads to catastrophic forgetting as demonstrated in the baseline model that is incrementally trained from technical bug reports (Task A) to formal news (Task B) and then to informal emails (Task C). When a standard neural network incrementally trains on a new task, it changes its internal weights to fit on the new data. This process overwrites and changes the weights that are important to classify the previous tasks. The hybrid approach tackles this problem by combining two techniques:

• Feeding previous data samples with new data (memory replay): Standard sequential incremental training only feeds the model new data, allowing the network to change its weights according to the new data. By storing a small buffer of old text samples and mixing them into the new training dataset, the model is forced to learn both old and new tasks at the same time. This enables the model to reduce catastrophic forgetting of prior tasks.
• Protecting important weights (parameter regularization): While keeping old samples in a buffer reduces forgetting, a small memory buffer cannot hold everything. Over a long sequence of tasks, such a buffer becomes too small to represent the prior tasks sufficiently, which is why the purely replay-based model loses some accuracy on Task A (dropping from 85.38% to 80.01% in Table 3). The hybrid approach extends this capability by combining parameter regularization with memory replay. It calculates weights crucial to past tasks and penalizes the model if it tries to make large changes to those specific weights.

Thus, replay buffer keeps the word representations steady, regularization protects the changes in weights important to previous tasks. This combination allows the model to achieve stability (reduce forgetting) without losing plasticity to learn new tasks data distributions. The stability–plasticity trade-off can be noticed in Table 2, where the hybrid approach’s final Task C accuracy ($98.92\%$) is a little lower than both the baseline ($99.98\%$) and replay-only approach ($99.78\%$), reflecting the cost of protecting earlier tasks’ weights. This trade-off is acceptable because Task C accuracy remains above $98\%$, while performance on the initial task (Task A) is better preserved compared to the standalone ER and EWC approaches. We recommend increasing $\lambda$ or the buffer size M when backward transfer/old-task retention is the priority (for instance, in case of long-lived deployed classifiers where regressions on earlier categories are costly), and decreasing $\lambda$ or relying more on replay alone when the latest task’s performance is crucial while older tasks are less critical.

5.2. Characteristics of Incremental Learning in Text

The text-based data is more vulnerable to catastrophic forgetting, as compared to the image data. The features like edges and shapes in images are useful across many different image datasets. However, the text datasets rely on distinct vocabularies and words. The words used in software bug reports for instance are highly technical with little overlap with the formal words used in news articles or the informal text in emails. Top-vocabulary Jaccard overlap between Bugs and the other two tasks (0.150 and 0.199) is roughly half that between News and Emails (0.345), confirming that the Bugs task occupies a comparatively isolated region of the shared vocabulary space. There are two mechanistic consequences for the proposed hybrid approach. First, for the replay buffer, because task vocabularies barely overlap (technical bug-report tokens vs. informal email tokens), a buffer sampled from Task k carries little signal for stabilizing the embedding rows of tokens unique to Task $k-1$; buffer coverage must therefore scale with vocabulary divergence between tasks. Second, for Fisher-based parameter protection, importance scores concentrate disproportionately on the embedding layer, since token indices are dataset-specific identifiers rather than reusable, spatially-shared filters as in image CNNs (where a single filter for detecting edges transfers across domains). This implies that the Fisher Matrix penalty in the proposed approach primarily anchors token-embedding rows for previously seen vocabulary, while allowing the BiLSTM’s recurrent weights to adapt to new task tokens. This explains why the hybrid model retains high plasticity ($98.92\%$ on Task C) despite strong regularization.

Impact of Textual Properties on the Hybrid Approach

Text-based sequential learning has distinct challenges compared to incremental spatial domains like images. Properties such as vocabulary shift, sparse semantics, and syntactic differences directly influence both the ER- and EWC-based parameter protection mechanisms in the proposed BiLSTM architecture:

• Vocabulary Shift and Replay Buffer Dynamics: In image classification, low-level spatial features such as the edges and textures are shared across domains. Textual data however, often exhibit disjoint vocabulary distributions. As shown by the Jaccard similarity analysis (Table 5), Task A (Bugs) shares minimal token overlap with subsequent tasks ($0.150$ with News, $0.199$ with Emails). Mechanistically, when the model trains on a new task with non-overlapping vocabulary, the gradient updates for the embedding matrix $\mathbf{E}\in {\mathbb{R}}^{\left|\mathcal{V}\right|\times d_e}$ become zero for tokens unique to prior tasks. Without memory replay, such unused embedding rows remain unchanged, but the downstream recurrent (BiLSTM) and classification layers shift, rendering the old embeddings misaligned. This issue is mitigated by the stratified replay buffer as it continually injects task-specific tokens into the active training batch, forcing concurrent gradient updates across both shared and task-isolated embedding rows.
• Sparse Semantics, Syntactic Shift, and Fisher-Based Weight Protection: Syntactic variation in cases like the rigid technical bug reports vs. informal email structures, causes significant shift in the recurrent transition weights of the BiLSTM layer. In addition to that, because text tokens act as discrete sparse identifiers, the empirical Fisher Information Matrix F (Equation (5)) calculates localized, high-magnitude importance scores specifically on the specific token embedding rows associated with prior task vocabularies. The EWC quadratic penalty therefore acts as a targeted anchor, locking the specific embedding vectors of historical tokens and stabilizing key recurrent transition paths, preventing the optimizer from overwriting previously learned decision boundaries while allowing unused embedding rows and flexible recurrent capacity to adapt to new task patterns.

5.3. Limitations and Computational Trade-Offs

Although the hybrid approach reduces forgetting effectively, there are some limitations and trade-offs to consider:

• Memory Storage for Old Data: Even though text files take up much less storage space than images or videos, saving a memory buffer for large number of tasks still requires space. If a model has to learn many different task datasets, the memory needed to store these text buffers will keep growing, which might make it impractical on low-end devices.
• Privacy of Buffered Data Samples: The stratified replay buffer stores raw text exemplars, which for Task C (email dataset) may retain personally identifiable information such as names, addresses, or account details. This is a privacy concern apart from the storage cost. Named-entity redaction or anonymization can be applied to buffered samples before buffering to preserve the data privacy [66]. Another possible approach is switching to embedding-level replay, storing only fixed-length vector representation of each buffered example rather than raw text, which is harder to reconstruct. Generative replay (Section 2) can also be employed that avoids storing real samples, but at the cost of training and maintaining a separate generative model. Such privacy safeguards are not implemented in the current study, but flagged as an important direction for any deployment involving sensitive text data.
• Class Imbalance: Drawing the buffer itself from a class-balanced sampling scheme such as Class-Balancing Reservoir Sampling [67], before computing the Fisher Matrix is a possible mitigation strategy.
• Higher Computation Time: Finding out which weights are important for regularization requires extra computation and thus processing power. Because the proposed hybrid approach calculates these weight values while also processing the replay buffers, it takes more time and computing power to train per epoch than a standard baseline model.
• Need for Clear Task Transitions: The proposed approach works in an offline setting in which the model gets to know exactly when one task ends and a new one begins. In the real-world setting, data often arrives continuously without any clear labels to mark a change in the dataset. For example, a news feed may shift topics continuously rather than switching between discrete categories. This is a notable limitation in the evaluation of this work is its reliance on a task-aware, offline incremental learning, where discrete task transitions ($T_1\to T_2\to T_3$) are explicitly signaled to the neural network. This assumption allows the optimization pipeline to perform distinct operations at boundary junctions: consolidating the current dataset ${\mathcal{D}}_k$, extracting the stratified replay buffer ${\mathcal{M}}_k$, freezing reference parameter snapshot ${\theta }^{\ast }$, and computing the empirical Fisher Information Matrix F using Equation (5). Real-world textual data such as live social media feeds, news articles, and the emails rarely exhibit clear task boundaries. Such domains often experience continuous concept drift and gradual vocabulary shifts. Applying the proposed hybrid framework directly to such boundary free data streams has two main challenges:

  (i)

  Boundary Detection Constraints: Without clear boundaries, determining the optimal point to lock ${\theta }^{\ast }$ and re-estimate F becomes unclear. Calculating Fisher Matrices over mixed distributions can degrade importance estimates for key parameters.

  (ii)

  Static vs. Rolling Buffers: Stratified sampling assumes a static historical dataset ${\mathcal{D}}_k$, whereas dynamic streams require online reservoir sampling to maintain the exemplars under continuous distribution shift [67].

  • To adapt the proposed hybrid approach to task free (online) continuous learning environments, future work can integrate:

  (i)

  Unsupervised Shift Detection: Utilizing statistical drift metrics such as tracking divergence in intermediate BiLSTM latent embeddings or monitoring rolling classification entropy to autonomously trigger Fisher Matrix recomputation and buffer consolidation when a distribution shift is detected.

  (ii)

  Online Rolling Fisher Updates: Replacing discrete step-wise Fisher calculations with an Exponentially weighted Moving Average (EMA) that continuously updates parameter importance values $F_i$ over an active sliding window of text streams.

• Translating the Hybrid Approach to LLMs: While this study shows that the hybrid approach is effective on standard recurrent neural networks (BiLSTM), scaling this hybrid approach to the Transformer-based LLMs adds high computational bottlenecks due to the immense size of such networks with millions or even billions of parameters [68]. Applying diagonal Fisher Information Matrix F (Equation (5)) over billions of parameters would require substantial memory and computational resources. To make this hybrid approach computationally feasible for LLM-scale models, Adapter-Level Parameter Protection could be applied. Instead of full-parameter regularization, EWC constraints can be restricted to Parameter-Efficient Fine-Tuning (PEFT) parameters, such as Low-Rank Adaptation (LoRA) matrices or modular adapters [55,69]. By freezing the weights ${\theta }_{\mathrm{base}}$ and calculating Fisher importance only over trainable adapter subsets $\Delta {\theta }_{\mathrm{LoRA}}$, the memory requirements of F drops by several orders of magnitude [55].

6. Conclusions and Future Work

This section concludes the paper by summarizing the core research findings and highlighting the primary contributions of the proposed hybrid approach. Furthermore, it discusses current limitations and outlines potential directions for future research in incremental text classification.

6.1. Conclusions

The problem of catastrophic forgetting in deep learning models trained on sequential text datasets is investigated in this work. When a model learns new tasks incrementally, the updates to its internal parameters make the model gradually forget previously learned knowledge. This forgetting is demonstrated by using a standard baseline (BiLSTM) neural network trained sequentially on three distinct textual datasets: software bug reports (Task A), News dataset (Task B), and Emails (Task C). The baseline model experienced severe forgetting, with its accuracy on the initial bug classification task dropping from 87.53% to a mere 11.75% by the last dataset training stage. The empirical results confirm that without a mitigation strategy, the recurrent models catastrophically forget previous task distributions.

To address this challenge, a hybrid approach is proposed in this work, that synergizes stratified memory replay with parameter regularization. The replay buffer exposes the model to a small sample of historical data, keeping the shared word representations stable. The parameter regularization technique on the other hand, calculates parameters that are most critical to past tasks and penalizes the model if it makes significant changes to such parameters. Experimental results in this study suggest that increasing buffer size in memory replay-based approach improves performance of the model. Storing a reasonable number of exemplars (between 500 and 2000 samples) is sufficient to stabilize the neural network model, recovering up to $69.36\%$ of lost accuracy. The evaluation results confirm that the proposed hybrid approach resolves the stability–plasticity dilemma within textual domains:

• The proposed hybrid approach successfully learned the final Email task with high accuracy ($98.92\%$), proving it retains excellent plasticity.
• The proposed hybrid approach preserved historical knowledge with outstanding stability, maintaining an accuracy of $83.73\%$ on the earliest task (Task A) and $95.04\%$ on the intermediate task (Task B).
• Total long-term forgetting on the first task was reduced to a minor $4.65\%$, showing a massive improvement over the baseline model and outperforming standalone experience replay technique. The proposed approach with its smaller memory buffer size 1000 outperformed standalone experience replay with larger memory buffer size of 2000.

This research therefore demonstrates that combining parameter protection with stratified memory replay is an effective approach to build robust IL models for text data.

6.2. Future Work

While the proposed hybrid approach achieved convincing results, it is not without limitations. Such limitations discussed in Section 5.3 open several avenues for future work, as outlined below:

• Transition to Task-Free Online Learning: The proposed approach relies on offline task setting, where the model gets to know when a dataset changes. A future research direction is to adapt this hybrid approach to online environments where the model can autonomously detect distribution shifts in a continuous data stream.
• Addressing the Class Imbalance: Although this work reduces the impact of class imbalance by using stratified memory replay, several possibilities remain to further address class imbalance-driven bias. Replacing a buffer’s sampling with a class-balancing sampling strategy, such as Class-Balancing Reservoir Sampling [67], can ensure the buffer avoids under-representing the minority class before any Fisher Matrices are computed from it. Recent work has shown that EWC’s gradient-based importance estimates can vanish or misalign even in balanced settings once the model is confident on its training data [70]. Combining a class-balancing correction with such fixes to underlying importance estimator would help minimize bias from class imbalance versus bias from the empirical-Fisher approximation itself. Apart from that, a systematic ablation varying both buffer size M and the imbalance ratio can reveal how much of the observed noise in minority-class importance estimates is attributable to class imbalance versus the buffer’s limited capacity.
• Dynamic Replay Buffer Management: Storing raw text samples for a long sequence of tasks has the problem of adding a linear memory overhead. Future investigations can explore integrating smart sample buffer selection methods to reduce storage requirements for devices with limited memory.
• Scaling to Large Language Models (LLMs): This study validated the hybrid approach using a BiLSTM-based architecture. A next step is to empirically validate the scalability of the combined replay and regularization synergy on transformer-based language models with very large number of parameters. Another perspective is its application in LLM-based multi-agent systems. A critical direction for extending multi-agent frameworks like LLM-Teams [62] lies in mitigating long-horizon context degradation without relying on aggressive history truncation. The hybrid approach against catastrophic forgetting can act as a role-retention engine for multi-agent LLM systems, maintaining temporal coherence over long horizons without context window expansion. The Fisher Matrix computation is computationally prohibitive for models with billions of parameters, since it requires a gradient pass per parameter tracked. A more realistic path is to restrict EWC-style regularization to a small set of parameter-efficient adapters such as LoRA weights [55], computing Fisher importance only over that reduced parameter subset. Similarly, replay operating over cached hidden-state embeddings rather than raw token sequences would reduce computations, given the context-length and compute costs of replaying full historical text sequences.