Dynamic LoRA-Experts and Prototype-Ensemble Matching for Class-Incremental Learning

Featured Application

This work can be applied to intelligent vision systems that need to incrementally learn new categories while maintaining previously acquired knowledge with low storage overhead.

Abstract

Class-incremental learning (CIL) aims to continuously learn new classes without forgetting previously acquired knowledge. Recent advances in parameter-efficient fine-tuning (PEFT) based on pre-trained models (PTMs) have shown promise in this setting by integrating new tasks with minimal parameter overhead. However, these methods often suffer from knowledge degradationdue to: (1) cumulative interference caused by iterative updates, constrained gradient flows, or entangled module integration; and (2) suboptimal alignment between inference samples and specialized modules. To address these challenges, we propose Dynamic LoRA-Experts and Prototype-Ensemble Matching (DLEPEM), a novel two-stage, rehearsal-free framework. In the first stage, we allocate a task-specific LoRA-Expert for each incremental task, enabling isolated representation learning and reducing cross-task interference. In the second stage, we introduce a prototype-ensemble-matching mechanism that combines general prototypes derived from the frozen PTM with task-adaptive prototypes learned by the LoRA-Experts. This fusion facilitates both strong generalization and precise task-level discrimination. Extensive experiments on standard CIL and few-shot class-incremental learning (FSCIL) benchmarks demonstrate that DLEPEM achieves strong performance under the evaluated protocols. For instance, in CIL, it achieves 93.39% on CIFAR100 (+0.80% over EASE), 92.31% on CUB200 (+2.11% over EASE), and 91.84% on VTAB (+1.39% over EASE). In the more challenging FSCIL setting, it achieves 88.77% on CUB200, outperforming the strongest baseline by a clear margin of 5.31%. These results indicate that DLEPEM effectively mitigates catastrophic forgetting while enhancing incremental learning capability.

1. Introduction

In open-world environments, data typically arrives as a continuous stream of novel categories, a scenario formalized as class-incremental learning (CIL). Traditional machine learning models struggle under such conditions, exhibiting catastrophic forgetting [1,2], where learning from new classes disrupts existing representations and leads to severe performance degradation. Class-incremental learning seeks to navigate this challenge by balancing the acquisition of new knowledge with the preservation of prior learning, a fundamental trade-off known as the stability–plasticity dilemma [3,4].

Recent advances leverage pre-trained models (PTMs), whose robust generalization capabilities stem from large-scale supervised or self-supervised training [5], providing a compelling foundation for CIL. However, directly fine-tuning all PTM parameters across sequential tasks risks compromising this generalization and amplifying forgetting. To mitigate this, contemporary methods use parameter-efficient fine-tuning (PEFT) techniques [6], such as prompts [7,8], adapters [9,10], and LoRA [4,11], which adapt models with minimal additional parameters, thereby reducing forgetting while preserving generalization.

Despite these gains, two critical challenges remain insufficiently addressed:

1. Stability–plasticity limitations from cumulative interference:

• Shared prompt pools [7,8] are prone to overwriting earlier knowledge when exposed to shifting data distributions.
• LoRA-based strategies [4,11], though effective in constraining updates to mitigate forgetting, inadvertently restrict the plasticity needed for new task adaptation.
• Fusion-based methods [4,9] attempt to balance old and new knowledge but often degrade the fidelity of both due to forced trade-offs.

2. Inference-stage module-sample mismatches:

• Fixed PTM selection mechanisms [7,12] struggle under substantial domain shifts, resulting in suboptimal activations and degraded predictions.

These observations motivate a key question: Can we simultaneously enhance stability–plasticity dynamics and improve module-sample matches to robustly mitigate catastrophic forgetting in CIL?

To this end, we propose DLEPEM, a novel rehearsal-free framework that rethinks PEFT-based CIL by introducing two synergistic components. First, inspired by Mixture-of-Experts (MoE) architectures [13], we dynamically allocate a dedicated LoRA-Expert for each new incremental task. Unlike shared or sequentially fine-tuned modules, each LoRA-Expert exclusively encodes task-specific knowledge; only the current expert is trainable, while all prior experts are frozen. Strategically embedded within Transformer feed-forward network (FFN) and multi-head self-attention (MHA) layers, these lightweight experts preserve plasticity for new tasks while entirely isolating past parameters, thereby achieving a more favorable balance between stability and plasticity. Different from conventional token-level sparse MoE or LoRA-MoE models that learn a soft router and aggregate multiple expert outputs, DLEPEM uses MoE as a task-level organization principle: the expert bank grows with the task sequence, each old expert remains frozen, and expert activation is determined by prototype retrieval rather than differentiable token routing. Second, to mitigate inference mismatches, we introduce a prototype-ensemble strategy that jointly leverages (1) general representations from the frozen PTM backbone and (2) specialized representations from task-specific experts. By fusing these distinct feature spaces, our approach enhances sample-to-module alignment, effectively bridging generalization and specialization. Thus, DLEPEM explicitly separates within-task prediction (WTP), handled by isolated task LoRA-Experts, from module-identity inference (MII), handled by prototype-ensemble matching.

In summary, our principal contributions are threefold:

• We introduce task-level dynamic LoRA-Expert allocation into PEFT-based CIL. Unlike conventional MoE-style LoRA methods with a fixed expert pool and soft expert mixing, DLEPEM automatically adds one dedicated LoRA-Expert for each incremental task, trains only the current expert, and freezes all previous experts to reduce cross-task parameter interference.
• We enhance module-sample alignment through prototype-ensemble matching, which fuses frozen-PTM prototypes and router-based domain-specific prototypes. This design improves task-level expert retrieval when PTM-only matching is unreliable under downstream domain shift.
• Extensive experiments on six challenging CIL benchmarks validate the effectiveness of our approach, showing that DLEPEM achieves leading performance among the evaluated methods under the evaluated protocols. We further demonstrate architectural flexibility through DLEPEM-MLP, a variant that explores alternative expert integration strategies while retaining competitive results.

2. Related Work

2.1. Class-Incremental Learning

Class-incremental learning requires models to continually recognize newly introduced classes while maintaining discriminability for previously learned classes. Existing methods are commonly grouped into regularization-, rehearsal-, and architecture-based approaches [7].

Regularization-based methods [14] reduce forgetting by penalizing changes to parameters that are estimated to be important for old tasks. This strategy does not store old samples, but its effectiveness can decrease when the incremental stream contains large domain shifts or many sequential tasks [15]. Rehearsal-based methods replay raw images [15] or stored feature representations [16] to preserve old knowledge. They are often effective, but the buffer budget and possible privacy restrictions limit their applicability in rehearsal-free settings. Dynamic network methods [17] expand the model with task-specific components and freeze previous ones to reduce parameter interference, but the growing architecture may introduce non-trivial memory overhead, and some methods still rely on old data for calibration or fusion.

2.2. PEFT-Based CIL

With the strong transferability of PTMs, recent CIL methods increasingly adopt PEFT modules to adapt only a small subset of parameters while keeping most backbone weights frozen. Prompt-based methods, such as L2P [7], DualPrompt [8], S-Prompts [12], and CODA-Prompt [18], learn task-relevant prompts and retrieve or combine them during inference. These methods reduce the need for full fine-tuning, but their retrieval quality depends heavily on how well prompt keys separate tasks in the PTM feature space.

Adapter- and LoRA-based methods modify the PTM through lightweight modules. APER [10] combines PEFT-adapted and PTM features to retain generalization. LAE [9] improves compatibility across PEFT modules, but repeated feature fusion can introduce stability–plasticity trade-offs. InfLoRA [4] constrains LoRA updates through gradient-orthogonal projection to reduce interference, whereas SD-LoRA [11] decouples gradient direction and magnitude to protect early-task directions. These constraint-based strategies improve stability but may restrict plasticity for newly introduced classes. The MoE-Adapters method [19] uses an activate-freeze mechanism with a predefined expert pool, which enables inter-task collaboration but limits flexibility when the number or diversity of future tasks is unknown.

2.3. Mixture-of-Experts and Expert Retrieval

MoE architectures introduce multiple expert modules and a routing mechanism that selects or weights experts for each input [13]. In vision models, V-MoE [20] replaces part of the dense feed-forward layers in ViT with sparse expert layers, where image patches are routed to a subset of MLP experts. In multi-modal or instruction-tuning settings, MoCLE [21] integrates multiple LoRA experts to handle task diversity. These methods show that expert specialization can improve adaptation, but their routing is usually learned within a fixed or predefined expert set and often combines expert outputs through token-level or sample-level gating.

For incremental learning, the key challenge is different: the model must accommodate an open-ended task sequence while avoiding repeated updates to old task-specific parameters. Therefore, expert life cycle and inference-time expert retrieval become central design choices. A fixed expert pool may be insufficient for long or unpredictable task streams, while a learned soft gate can suffer from task-recency bias when trained only on the current task.

2.4. Our Approach

Similar to PEFT-based CIL methods [4,9,10], DLEPEM uses a PTM backbone with LoRA [22] adaptation. However, instead of repeatedly updating or blending shared PEFT modules, DLEPEM allocates one LoRA-Expert for each incremental task, trains only the current expert, and freezes all historical experts. This stage-wise expert life cycle reduces cross-task parameter interference while preserving plasticity for the new task.

DLEPEM also differs from conventional MoE-based or multi-LoRA methods in routing granularity. Existing MoE formulations usually learn an input-dependent gate to select or softly combine experts from a predefined pool. In contrast, DLEPEM performs top-1 task-level expert retrieval through an append-only prototype dictionary. Each ensemble key combines a frozen-PTM prototype with a router-domain prototype, so inference uses both general semantics and task-adaptive cues to select the appropriate expert. Thus, old-task preservation is mainly achieved by structural parameter isolation, while module-identity inference is handled by prototype-ensemble matching rather than a learned soft gate.

3. Preliminaries

Problem formulation. CIL considers a sequential stream of tasks $\mathcal{D}=\{{\mathcal{D}}_1,\dots ,{\mathcal{D}}_T\}$, where the t-th task ${\mathcal{D}}_t={\left\{({\mathbf{x}}_i,{\mathbf{y}}_i)\right\}}_{i=1}^{n_t}$ contains $n_t$ samples. Here, ${\mathbf{x}}_i\in {\mathcal{X}}_t$ denotes an input from domain ${\mathcal{X}}_t$, and ${\mathbf{y}}_i\in {\mathcal{Y}}_t$ is its corresponding label. Importantly, the label spaces are mutually exclusive across tasks, i.e., ${\mathcal{Y}}_t\cap {\mathcal{Y}}_{t^{\prime }}=\varnothing$ for $t\ne t^{\prime }$.

Following the rehearsal-free setting [7,8,18], the model only observes data from the current task during training. The training objective is to learn a model ${\mathrm{f}}_{\mathrm{\Theta }}\left(\mathbf{x}\right)={\mathbf{W}}_{cls}^{\top }\varphi \left(\mathbf{x}\right)$ that minimizes the empirical risk over the current task’s training set,

$$\mathrm{L}\left({\mathcal{D}}_t\right)={\displaystyle \frac{1}{|{\mathcal{D}}_t|}}\sum _{({\mathbf{x}}_i,{\mathbf{y}}_i)\in {\mathcal{D}}_t}\mathrm{L}\left({\mathrm{f}}_{\mathrm{\Theta }}\left({\mathbf{x}}_i\right),{\mathbf{y}}_i\right),$$

(1)

where $\varphi \left(\mathbf{x}\right)$ represents the embedded [class] token from the ViT, ${\mathbf{W}}_{cls}$ denotes the classifier weights, $|{\mathcal{D}}_t|$ is the number of examples in the current task, and $\mathrm{L}(·,·)$ represents the loss function that measures prediction error. After each task t, performance is evaluated on all classes seen so far, i.e., on the union ${\mathbf{Y}}_t={\mathcal{Y}}_1\cup \cdots \cup {\mathcal{Y}}_t$.

Mixture-of-experts. MoE architectures offer an efficient way to increase model capacity by activating only a subset of parameters per input, achieving faster training and inference compared to dense networks of equivalent scale. A typical MoE layer comprises a set of M expert networks $\mathbf{E}=\{{\mathbf{E}}_1,\dots ,{\mathbf{E}}_M\}$ and a router G that determines expert activations based on the input [23]. Each expert is commonly implemented as an FFN, while the router is parameterized by a weight matrix ${\mathbf{W}}_g$.

Formally, for an input $\mathbf{x}$, the router computes

$$\mathrm{G}\left(\mathbf{x}\right)=\mathrm{softmax}\left({\mathbf{W}}_g\mathbf{x}\right),$$

(2)

producing a soft selection over experts. The final MoE output is given by

$$\mathrm{MoE}\left(\mathbf{x}\right)={\displaystyle \sum _{i=1}^M}\mathrm{G}{\left(\mathbf{x}\right)}_i{\mathbf{E}}_i\left(\mathbf{x}\right),$$

(3)

where $\mathrm{G}{\left(\mathbf{x}\right)}_i$ represents the routing probability for expert ${\mathbf{E}}_i$. In Transformer-based architectures, MoE layers often replace standard FFN blocks to selectively route representations [24].

Low-rank adaptation. LoRA was introduced to efficiently fine-tune large pre-trained models by injecting low-rank updates into weight matrices [22]. Given a pre-trained weight matrix $\mathbf{W}\in {\mathbf{R}}^{d_{in}\times d_{out}}$, LoRA learns an additive low-rank decomposition

$$\mathbf{W}+\mathrm{\Delta }\mathbf{W}=\mathbf{W}+\mathbf{U}\mathbf{V},$$

(4)

where $\mathbf{U}\in {\mathbf{R}}^{d_{in}\times r}$, $\mathbf{V}\in {\mathbf{R}}^{r\times d_{out}}$, and the rank $r\ll min(d_{in},d_{out})$. This reduces the number of trainable parameters while maintaining expressiveness. LoRA enables cost-effective, scalable fine-tuning, making it suitable for continual learning, where efficiency and avoidance of forgetting are critical.

4. The Proposed Method

As demonstrated by HiDe-Prompt [25], CIL methods employing multi-module selection can be decomposed into two probabilistic components: module-identity inference (MII) and within-task prediction (WTP), represented by $P(\mathbf{x}\in {\mathcal{X}}_i|\mathcal{D},\mathrm{\Theta })$ and $P(\mathbf{x}\in {\mathcal{X}}_{i,j}|\mathbf{x}\in {\mathcal{X}}_i,\mathcal{D},\mathrm{\Theta })$, respectively. By Bayes’ theorem, we have

$$\begin{array}{c}\hfill P(\mathbf{x}\in {\mathcal{X}}_{i,j}|\mathcal{D},\mathrm{\Theta })=P(\mathbf{x}\in {\mathcal{X}}_{i,j}|\mathbf{x}\in {\mathcal{X}}_i,\mathcal{D},\mathrm{\Theta })P(\mathbf{x}\in {\mathcal{X}}_i|\mathcal{D},\mathrm{\Theta }).\end{array}$$

(5)

Letting $\widehat{i}$ and $\widehat{j}$ denote the ground-truth task index and class label for input $\mathbf{x}$, Equation (5) implies that improving either WTP accuracy, $P(\mathbf{x}\in {\mathcal{X}}_{\widehat{i},\widehat{j}}|\mathbf{x}\in {\mathcal{X}}_{\widehat{i}},\mathcal{D},\mathrm{\Theta })$, or MII accuracy, $P(\mathbf{x}\in {\mathcal{X}}_{\widehat{i}}|\mathcal{D},\mathrm{\Theta })$, directly enhances overall prediction performance. However, existing approaches suffer from two limitations: (1) iterative updates or the fusion of new and existing modules progressively deteriorate WTP [4,7,8,9]; and (2) MII performance, when relying solely on pre-trained features [9,12], is inherently constrained by the similarity between pre-training and downstream data distributions. To address these challenges, we propose DLEPEM, which explicitly enhances both WTP and MII via two complementary innovations:

• Dynamic LoRA-Expert

To exploit the strong generalization of the pre-trained model, we keep its weights $\mathbf{W}$ fixed throughout training. To maintain plasticity and safeguard WTP, we dynamically allocate a dedicated LoRA-Expert for each incremental task, embedded within an MoE framework. Each new LoRA-Expert is trained exclusively on its respective task while previously introduced experts remain frozen. This ensures isolated task-specific adaptation with a small number of trainable parameters, leveraging LoRA’s efficiency to effectively capture discriminative features. Figure 1a depicts expert training, while Figure 1b details the internal structure. This expert life cycle is the main difference from conventional MoE-based LoRA continual learning. DLEPEM does not repeatedly update a shared expert pool or combine all experts through a soft gate during feature computation. Instead, for task t, only ${\mathbf{E}}_t$ receives gradients, whereas $\{{\mathbf{E}}_1,\dots ,{\mathbf{E}}_{t-1}\}$ remain frozen. This design directly reduces cross-task parameter interference and protects WTP for previously learned tasks.

• Prototype-Ensemble Matching

After training each incremental task, we extract prototypes using two sources: (1) the fixed PTM for generalizable features, and (2) the router-enhanced LoRA-Expert for domain-specific nuances. Unlike prior methods that rely solely on pre-trained representations, DLEPEM combines these prototypes and associates them with their corresponding LoRA-Experts. During inference, a nearest-neighbor search in this combined prototype space determines the most appropriate expert, substantially improving MII. This design mitigates privacy concerns inherent to rehearsal-based strategies. Figure 1c illustrates our complete prototype-ensemble matching mechanism. The ensemble key for a class is constructed as the concatenation of a frozen-PTM prototype and a router-domain prototype. The former provides stable general semantics, while the latter captures downstream task-specific cues. This complements dynamic expert allocation: the task expert improves WTP once selected, and the prototype-ensemble dictionary improves MII by selecting the appropriate expert without storing raw rehearsal samples.

4.1. Dynamic LoRA-Experts

We integrate the proposed LoRA-Expert modules into the Vision Transformer (ViT) architecture [26]. In ViT, an input image is first divided into fixed-size patches, linearly projected, and augmented with positional embeddings before being processed by a Transformer encoder comprising MHA layers and multilayer perceptrons (MLPs).

To adaptively capture task-specific features in incremental learning, we dynamically introduce a LoRA-Expert at each incremental stage. This module can be inserted either as a parallel branch to the MLP (MLP-Expert, see Figure 1b) or into the attention projections of the MHA module. When LoRA is applied to ${\mathbf{W}}_q$ and ${\mathbf{W}}_v$, we refer to this variant as the QV-Expert configuration.

The modified forward computations for these components are given by

$${\mathbf{h}}^{\prime }=\mathbf{e}+\mathrm{MLP}\left(\mathbf{e}\right)+{\mathbf{E}}_t^{MLP}\left(\mathbf{e}\right),$$

(6)

$${\mathbf{h}}^{\prime }=\mathrm{Attn}\left({\mathbf{h}}_Q+{\mathbf{E}}_t^Q\left(\mathbf{e}\right),{\mathbf{h}}_K+{\mathbf{E}}_t^K\left(\mathbf{e}\right),{\mathbf{h}}_V+{\mathbf{E}}_t^V\left(\mathbf{e}\right)\right),$$

(7)

where $\mathbf{e}$ and $\mathbf{h}$ are the inputs and outputs of the original module, respectively. Here, ${\mathbf{E}}_t$ denotes the task-t LoRA-Expert module rather than a complete, independent ViT. It is a group of LoRA adapters inserted into the selected branch or projection, and each historical expert ${\mathbf{E}}_s$, $s<t$, is frozen after its task has been learned. The attention operation is defined as:

$$\mathrm{Attn}\left(\mathbf{Q},\mathbf{K},{\mathbf{V}}_{attn}\right)=\mathrm{softmax}\left({\displaystyle \frac{\mathbf{Q}{\mathbf{K}}^{\top }}{\sqrt{d}}}\right){\mathbf{V}}_{attn},$$

(8)

Here, ${\mathbf{V}}_{attn}$ denotes the attention value matrix and is distinct from the LoRA matrix ${\mathbf{V}}_t$. For N tokens, the softmax term is an attention-weight matrix $\mathbf{A}\in {\mathbf{R}}^{N\times N}$, so $\mathbf{A}{\mathbf{V}}_{attn}$ is the standard weighted sum over value vectors. The multi-head extension is omitted for clarity. Each LoRA-Expert shares the same architecture but learns distinct parameters. Under the row-vector convention, for an input $\mathbf{e}\in {\mathbf{R}}^{1\times d_{in}}$, one adapter consists of low-rank matrices ${\mathbf{U}}_t\in {\mathbf{R}}^{d_{in}\times r}$ and ${\mathbf{V}}_t\in {\mathbf{R}}^{r\times d_{out}}$, producing:

$${\mathbf{E}}_t\left(\mathbf{e}\right)=\mathbf{e}{\mathbf{U}}_t{\mathbf{V}}_t.$$

(9)

In the default ViT-B/16 QV-Expert setting, $d_{in}=d_{out}=768$ and $r=10$; for the MLP-Expert variant, $d_{in}$ and $d_{out}$ correspond to the inserted MLP branch dimensions.

In the ViT setting, we denote by $\varphi (\mathbf{x};{\mathbf{E}}_t)$ the output embeddings produced by the PTM equipped with the task-specific expert ${\mathbf{E}}_t$. For the first incremental task, we follow the standard LoRA initialization [22], setting $\mathbf{V}$ to zero and initializing $\mathbf{U}$ via Kaiming initialization [27]. For subsequent tasks, we initialize each new LoRA-Expert by copying the weights from the preceding expert, then fine-tune it while keeping all previous experts frozen. As illustrated in Figure 1a, this strategy ensures that each task-specific expert adapts independently, preserving knowledge from prior stages. This design provides a direct stability–plasticity mechanism. Let ${\theta }_t=\{{\mathbf{U}}_t,{\mathbf{V}}_t\}$ denote the LoRA parameters of the expert allocated to task t, while the PTM weights $\mathbf{W}$ remain frozen. After task s has been learned, its old expert parameters ${\theta }_s$ are not optimized when learning any later task $t>s$; hence,

$${\nabla }_{{\theta }_s}\mathcal{L}\left({\mathcal{D}}_t\right)=0,s<t.$$

(10)

Therefore, old experts remain parameter-stationary during subsequent training, which reduces forgetting caused by repeated modification of shared PEFT parameters. Meanwhile, the current expert ${\theta }_t$ is optimized with the cross-entropy objective in Equation (18) without imposing gradient-orthogonality or update-magnitude shrinking constraints, preserving plasticity for newly introduced classes. The remaining source of old-task degradation is mainly expert retrieval error, which is addressed by the prototype-ensemble-matching mechanism.

4.2. Prototype-Ensemble Matching Mechanism

When the distribution gap between the pre-trained model (PTM) and the downstream dataset is small, generalized features extracted by the frozen PTM can enable effective module-sample matching. However, since the relationship between pre-training and downstream distributions is generally unknown, it is essential to adaptively capture domain-specific features from the downstream task.

To tackle this, we introduce a dynamically updated router ${\mathbf{E}}^{router}$ that extracts domain-specific features. The router shares the same architecture as the LoRA-Experts but is continuously updated across incremental tasks.

Under typical incremental constraints, where only data from the current task is available, directly fine-tuning the router leads to overfitting, causing it to route all samples to the most recent LoRA-Expert. To mitigate this, we propose dual feature distillation mechanisms that jointly enforce stability (retaining prior knowledge) and plasticity (adapting to new tasks), regularizing router updates to ensure robust generalization.

1. Plasticity feature distillation. To encourage the router to learn category-specific features of the current task, we first compute the LoRA-Expert prototype for each class $i\in {\mathcal{Y}}_t$:

$${\mathbf{P}}_{i,t}^L={\displaystyle \frac{1}{N_{i,t}}}\sum _{({\mathbf{x}}_j,{\mathbf{y}}_j)\in {\mathcal{D}}_t}\mathbb{I}({\mathbf{y}}_j=i)\varphi ({\mathbf{x}}_j;{\mathbf{E}}_t),N_{i,t}=\sum _{({\mathbf{x}}_j,{\mathbf{y}}_j)\in {\mathcal{D}}_t}\mathbb{I}({\mathbf{y}}_j=i).$$

(11)

Here, $N_{i,t}$ is the number of current-task samples belonging to class i, and $\mathbb{I}(·)$ denotes the indicator function. We then align the router feature of each current-task sample with the LoRA-Expert prototype of its ground-truth class using a KL-divergence objective:

$${\mathcal{L}}_{PFD}={\displaystyle \frac{1}{|{\mathcal{D}}_t|}}\sum _{({\mathbf{x}}_j,{\mathbf{y}}_j)\in {\mathcal{D}}_t}\mathrm{KL}\left(\sigma ({\mathbf{P}}_{{\mathbf{y}}_j,t}^L/\tau )\parallel \sigma (\varphi ({\mathbf{x}}_j;{\mathbf{E}}_t^{router})/\tau )\right).$$

(12)

Here, $\sigma \left(\mathbf{z}\right)=\mathrm{softmax}\left(\mathbf{z}\right)$, and temperature scaling is written explicitly as $\sigma (\mathbf{z}/\tau )=\mathrm{softmax}(\mathbf{z}/\tau )$, where $\tau$ is the temperature parameter. Thus, $\sigma (·)$ in the distillation losses denotes the same softmax function as $\mathrm{softmax}(·)$ in the attention equation. These class-specific prototypes guide the router toward discriminative features of the current task, ensuring effective adaptation for new tasks.

2. Stability feature distillation. To mitigate the degradation of previously learned knowledge, we introduce a stability constraint that encourages the current router to mimic the feature distribution produced by the previous router on the current-task inputs:

$${\mathcal{L}}_{SFD}={\displaystyle \frac{1}{|{\mathcal{D}}_t|}}\sum _{({\mathbf{x}}_j,{\mathbf{y}}_j)\in {\mathcal{D}}_t}\mathrm{KL}\left(\sigma (\varphi ({\mathbf{x}}_j;{\mathbf{E}}_{t-1}^{router})/\tau )\parallel \sigma (\varphi ({\mathbf{x}}_j;{\mathbf{E}}_t^{router})/\tau )\right).$$

(13)

The overall router loss combines these objectives,

$${\mathcal{L}}_{router}=\left\{\begin{array}{cc}{\mathcal{L}}_{PFD},\hfill & t=1,\hfill \\ \alpha {\mathcal{L}}_{PFD}+(1-\alpha ){\mathcal{L}}_{SFD},\hfill & t>1,\hfill \end{array}\right.$$

(14)

where $\alpha$ weights the plasticity and stability distillation terms for the router. For the first task, ${\mathcal{L}}_{SFD}$ is omitted because no previously trained router exists. The router parameters ${\mathbf{E}}_t^{router}$ are optimized by minimizing this combined loss function. With the default $\alpha =0.04$, the router update places more weight on ${\mathcal{L}}_{SFD}$; however, this coefficient controls router regularization only and should not be interpreted as proof of an optimal global stability–plasticity balance. The main source of new-task plasticity remains the trainable current LoRA-Expert, while frozen old experts provide parameter-level stability.

At the end of each incremental stage, we compute general prototypes ${\mathbf{P}}_i^F$ using the frozen PTM $\varphi \left(\mathbf{x}\right)$ and domain-specific prototypes ${\mathbf{P}}_i^R$ using the current router ${\mathbf{E}}_t^{router}$ for each newly introduced class $i\in {\mathcal{Y}}_t$, as defined in Equation (11). The prototype dictionary is updated in an append-only manner: old entries for classes $i\in {\mathbf{Y}}_{t-1}$ are retained as historical keys and are not recomputed with later-task data. The new ensemble keys created at stage t are

$${\mathbf{K}}_t^{new}=\left\{{\mathbf{K}}_i=[{\mathbf{P}}_i^F;{\mathbf{P}}_i^R]\mid i\in {\mathcal{Y}}_t\right\}.$$

(15)

Equivalently, we define the class-i ensemble prototype vector as ${\mathbf{P}}_i=\mathrm{concat}({\mathbf{P}}_i^F,{\mathbf{P}}_i^R)=[{\mathbf{P}}_i^F;{\mathbf{P}}_i^R]$, and use it as the dictionary key ${\mathbf{K}}_i={\mathbf{P}}_i$. The semicolon in $[·;·]$ denotes vector concatenation, not matrix addition or summation. Each new key ${\mathbf{K}}_i$ is appended to Dict, together with the expert index ${\nu }_i=t$, which points to the corresponding expert ${\mathbf{E}}_t$. Thus, after task t, the dictionary covers all seen classes ${\mathbf{Y}}_t$, while only entries for ${\mathcal{Y}}_t$ are newly created. This keeps DLEPEM rehearsal-free: the persistent state consists of frozen LoRA-Experts, the current router, and compact ensemble keys, but no raw samples, old mini-batches, feature buffers, or exemplar sets.

Because the router is updated across tasks, router-domain keys from older stages may have been generated by earlier router states. DLEPEM mitigates this router-state drift in two ways. First, each key includes the frozen-PTM component ${\mathbf{P}}_i^F$, which remains comparable across stages because the PTM is fixed. Second, stability feature distillation in Equation (13) encourages ${\mathbf{E}}_t^{router}$ to preserve the behavior of ${\mathbf{E}}_{t-1}^{router}$ while adapting to the current task.

• Prototype-Ensemble Matching

Our mechanism dynamically selects the most suitable LoRA-Expert for each input by leveraging a key-value association strategy inspired by L2P [7]. Specifically, we maintain a dictionary where each seen class $i\in {\mathbf{Y}}_t$ has one ensemble key ${\mathbf{K}}_i$ and an associated expert index ${\nu }_i$:

$${\mathrm{Dict}}_t=\left\{\left({\mathbf{K}}_i,{\nu }_i\right)\mid i\in {\mathbf{Y}}_t,{\nu }_i\in \{1,\dots ,t\}\right\}.$$

(16)

Here, ${\nu }_i$ identifies the task-specific LoRA-Expert associated with class i.

During inference, given an input $\mathbf{x}$, we construct an ensemble query $\mathbf{q}\left(\mathbf{x}\right)=\mathrm{concat}({\mathbf{P}}^F,{\mathbf{P}}^R)=[{\mathbf{P}}^F;{\mathbf{P}}^R]$, where ${\mathbf{P}}^F$ is the feature from the frozen PTM and ${\mathbf{P}}^R$ is the domain-specific feature from the router. We then identify the nearest key by cosine similarity:

$$\widehat{i}=\underset{i\in {\mathbf{Y}}_t}{\mathrm{argmax}}\left(\cos (\mathbf{q}\left(\mathbf{x}\right),{\mathbf{K}}_i)\right),$$

(17)

The retrieved key ${\mathbf{K}}_{\widehat{i}}$ returns an expert identity rather than a direct class prediction. Let $\widehat{t}={\nu }_{\widehat{i}}$ denote the task index associated with ${\mathbf{K}}_{\widehat{i}}$ in ${\mathrm{Dict}}_t$; DLEPEM then selects ${\mathbf{E}}_{\widehat{t}}$ for within-task prediction. Final classification is restricted to the class set ${\mathcal{Y}}_{\widehat{t}}$ associated with the selected expert and uses the corresponding prototype weights. Thus, all stored ensemble keys participate in module-identity inference, while prototype weights from unrelated experts are not mixed during classification.

4.3. Optimization Objective and Training Procedure

DLEPEM employs a two-stage training paradigm: it first learns Dynamic LoRA-Expert modules and then optimizes the router for effective module-sample matching. Algorithm 1 provides the complete training procedure, and Algorithm 2 gives the corresponding task-agnostic inference procedure.

1. Dynamic LoRA-Expert learning: The objective function for training the LoRA-Expert is

$$\underset{{\mathbf{W}}_{cls},{\mathbf{E}}_i}{min}{\mathrm{L}}_{CE}\left({\mathbf{W}}_{cls}^{\top }\varphi \left(\mathbf{x};{\mathbf{E}}_i\right),\mathbf{y}\right),$$

(18)

where ${\mathrm{L}}_{CE}$ denotes the cross-entropy loss, and $\varphi (\mathbf{x};{\mathbf{E}}_i)$ represents the PTM equipped with the LoRA-Expert ${\mathbf{E}}_i$. During inference, DLEPEM uses LoRA-Expert-derived prototype weights for classification. Specifically, ${\mathbf{P}}^L$ denotes the prototype–weight matrix formed by the class prototypes ${\mathbf{P}}_{i,t}^L$ defined in Equation (11). Classification is performed using cosine similarity,

$$\mathrm{f}\left(\mathbf{x}\right|{\mathbf{E}}_i)={\left({\displaystyle \frac{{\mathbf{P}}^L}{\parallel {\mathbf{P}}^L{\parallel }_2}}\right)}^{\top }\left({\displaystyle \frac{\varphi (\mathbf{x};{\mathbf{E}}_i)}{\parallel \varphi (\mathbf{x};{\mathbf{E}}_i){\parallel }_2}}\right),$$

(19)

where ${\mathbf{E}}_i$ denotes the selected LoRA-Expert, and ${\mathbf{P}}^L$ contains the prototype weights associated with this expert.

Algorithm 1 Training Procedure for DLEPEM

1: Input: Pre-trained model $\varphi (·)$, incremental datasets $\{{\mathcal{D}}_1,\dots ,{\mathcal{D}}_T\}$, LoRA rank r, distillation coefficient $\alpha$.   2: Output: Trained LoRA-Experts $\{{\mathbf{E}}_1,\dots ,{\mathbf{E}}_T\}$, Router ${\mathbf{E}}_T^{router}$, Prototype Dictionary Dict.   3: Initialize: $\mathrm{Dict}\leftarrow \varnothing$, ${\mathbf{E}}_0^{router}$ with random weights.   4: for $t=1$ to T do   5:     # Stage 1: Dynamic LoRA-Expert Learning   6:     Initialize LoRA-Expert ${\mathbf{E}}_t$. If $t>1$, copy weights from ${\mathbf{E}}_{t-1}$.   7:     Train ${\mathbf{E}}_t$ and classifier ${\mathbf{W}}_{cls}$ on ${\mathcal{D}}_t$ using ${\mathcal{L}}_{CE}$ (Equation (18)).   8:     Freeze parameters of ${\mathbf{E}}_t$.   9:     # Stage 2: Router Learning and Prototype-Ensemble Building 10:     If $t>1$, freeze a copy of the previous router ${\mathbf{E}}_{t-1}^{router}$. 11:     Train router ${\mathbf{E}}_t^{router}$ on ${\mathcal{D}}_t$ using ${\mathcal{L}}_{router}$ (Equation (14)). 12:     # Append Prototype Dictionary with newly introduced classes 13:     Keep old dictionary entries for classes in ${\mathbf{Y}}_{t-1}$ unchanged. 14:     for each newly introduced class $i\in {\mathcal{Y}}_t$ do 15:         Compute general prototype ${\mathbf{P}}_i^F$ using frozen PTM $\varphi (·)$ and samples from ${\mathcal{D}}_t$. 16:         Compute domain-specific prototype ${\mathbf{P}}_i^R$ using router ${\mathbf{E}}_t^{router}$ and samples from ${\mathcal{D}}_t$. 17:         Create ensemble prototype ${\mathbf{P}}_i=[{\mathbf{P}}_i^F;{\mathbf{P}}_i^R]$ (Equation (15)). 18:         Append an entry to Dict by associating key ${\mathbf{K}}_i={\mathbf{P}}_i$ with expert index ${\nu }_i=t$ (Equation (16)). 19:     end for 20: end for 21: return $\{{\mathbf{E}}_1,\dots ,{\mathbf{E}}_T\}$, ${\mathbf{E}}_T^{router}$, Dict.

Algorithm 2 Inference Procedure for DLEPEM

1: Input: Test sample $\mathbf{x}$, frozen PTM $\varphi (·)$, frozen LoRA-Experts $\{{\mathbf{E}}_1,\dots ,{\mathbf{E}}_T\}$, current router ${\mathbf{E}}_T^{router}$, prototype dictionary Dict, and prototype weights ${\left\{{\mathbf{P}}_c^L\right\}}_{c\in {\mathbf{Y}}_T}$.   2: Output: Predicted label $\widehat{y}$.   3: Compute frozen-PTM feature ${\mathbf{P}}^F\left(\mathbf{x}\right)=\varphi \left(\mathbf{x}\right)$.   4: Compute router-domain feature ${\mathbf{P}}^R\left(\mathbf{x}\right)=\varphi (\mathbf{x};{\mathbf{E}}_T^{router})$.   5: Build the ensemble query $\mathbf{q}\left(\mathbf{x}\right)=[{\mathbf{P}}^F\left(\mathbf{x}\right),{\mathbf{P}}^R\left(\mathbf{x}\right)]$.   6: Retrieve the nearest dictionary key $\widehat{i}=arg{max}_{i\in {\mathbf{Y}}_T}\cos (\mathbf{q}\left(\mathbf{x}\right),{\mathbf{K}}_i)$.   7: Obtain the expert index $\widehat{t}={\nu }_{\widehat{i}}$ associated with ${\mathbf{K}}_{\widehat{i}}$ in Dict.   8: Extract the selected-expert feature $\mathbf{z}=\varphi (\mathbf{x};{\mathbf{E}}_{\widehat{t}})$.   9: Restrict candidate labels to ${\mathcal{Y}}_{\widehat{t}}$ and predict $\widehat{y}=arg{max}_{c\in {\mathcal{Y}}_{\widehat{t}}}\cos (\mathbf{z},{\mathbf{P}}_c^L)$. 10: return $\widehat{y}$.

2. Router learning: The router is optimized with the objective function shown in Equation (14).

As illustrated in Figure 1, we divide the class-incremental learning process into two stages: First, a new LoRA-Expert is learned for each incremental task to capture task-specific features; each expert is categorized as either an MLP-Expert or a QV-Expert depending on its insertion point. Second, we introduce a prototype-ensemble-matching mechanism that captures both general and domain-specific features, thereby improving module-sample matching. Notably, DLEPEM’s components are orthogonal to many existing approaches and can be integrated with them straightforwardly. For clarity,

Table 1. Notation used in DLEPEM.

Symbol	Definition
${\mathcal{D}}_t$, ${\mathcal{Y}}_t$	Training set and class set of task t.
${\mathbf{Y}}_t$	The set of all classes observed up to task t, i.e., ${\mathbf{Y}}_t={\mathcal{Y}}_1\cup \cdots \cup {\mathcal{Y}}_t$.
${\mathbf{E}}_t$	Task-t LoRA-Expert module, i.e., a group of low-rank LoRA adapters inserted into selected Transformer layers rather than a complete independent ViT. Historical experts are frozen after training.
${\mathbf{U}}_t$, ${\mathbf{V}}_t$, r	Low-rank matrices and rank inside each adapter. For row-vector input $\mathbf{e}\in {\mathbf{R}}^{1\times d_{in}}$, ${\mathbf{U}}_t\in {\mathbf{R}}^{d_{in}\times r}$, ${\mathbf{V}}_t\in {\mathbf{R}}^{r\times d_{out}}$, with $r=10$ by default.
${\mathbf{V}}_{attn}$	Attention value matrix in Equation (8), distinct from the LoRA matrix ${\mathbf{V}}_t$.
${\mathbf{E}}_t^{router}$	Router after learning task t, used to extract router-domain features.
$\varphi \left(\mathbf{x}\right)$, $\varphi (\mathbf{x};\mathbf{E})$	Frozen-PTM feature and the feature obtained with module $\mathbf{E}$, respectively.
${\mathbf{P}}_{i,t}^L$	LoRA-Expert prototype of class i at task t.
${\mathbf{P}}_i^F$, ${\mathbf{P}}_i^R$	Frozen-PTM prototype vector and router-domain prototype vector of class i.
${\mathbf{P}}_i$	Ensemble prototype vector of class i, defined as ${\mathbf{P}}_i=\mathrm{concat}({\mathbf{P}}_i^F,{\mathbf{P}}_i^R)=[{\mathbf{P}}_i^F;{\mathbf{P}}_i^R]$.
$[·;·]$	Vector concatenation operator; it does not denote matrix summation.
${\mathbf{K}}_i$, ${\nu }_i$	Dictionary key of class i and the associated expert index. In our implementation, ${\mathbf{K}}_i={\mathbf{P}}_i$.
$\mathbf{q}\left(\mathbf{x}\right)$	Ensemble query of test sample $\mathbf{x}$.
$\sigma (·)$, $\tau$	Softmax shorthand and temperature used in the distillation losses, where $\sigma \left(\mathbf{z}\right)=\mathrm{softmax}\left(\mathbf{z}\right)$ and $\sigma (\mathbf{z}/\tau )=\mathrm{softmax}(\mathbf{z}/\tau )$.

summarizes the main symbols used throughout the paper.

5. Experiments

5.1. Experimental Settings

Datasets: We evaluate DLEPEM in two settings: CIL and few-shot class-incremental learning (FSCIL) [28]. For CIL, we followed standard protocols [10,11] and tested on five benchmarks: VTAB [29], CIFAR100 [30], CUB200 [31], ImageNet-R [32], and OmniBenchmark [33]. VTAB comprises 50 classes, CIFAR100 contains 100 classes, CUB200 and ImageNet-R each contain 200 classes, and OmniBenchmark, the largest benchmark among them, includes 300 classes. As shown in

Table 2. Configuration of the class-incremental learning (CIL) benchmarks.

Task	CIFAR100	CUB200	ImageNet-R	OmniBenchmark	VTAB
Classes/Task	10	20	40	30	10
# of tasks	10	10	5	10	5

, we followed the common practices [4,9,11], splitting CIFAR100 into 10 tasks, CUB200 into 10 tasks, ImageNet-R into 5 tasks, OmniBenchmark into 10 tasks, and VTAB into 5 tasks. Code is available at: https://github.com/hongwei-zhao/Appl_Sci-DLEPEM-main (accessed on 13 June 2026).

For FSCIL, we adopted the settings used in prior work [34,35] on CUB200, CIFAR100, and miniImageNet [36]. As shown in

Table 3. Configuration of the few-shot class-incremental learning (FSCIL) benchmarks.

Task	CUB200	CIFAR100	miniImageNet
Base Classes	100	60	60
Incremental Tasks	10-way 5-shot	5-way 5-shot	5-way 5-shot
# of Tasks	1 + 10	1 + 8	1 + 8

, we used 100 classes in CUB200 as the base class set for the first task. The remaining 100 classes were partitioned into 10 incremental tasks, with each incremental task containing 10 new classes and the few-shot training set containing 5 examples per class (10-way 5-shot incremental task). CIFAR100 and miniImageNet were divided into 60 classes for the base task, and the remaining 40 classes were divided into eight 5-way 5-shot incremental tasks.

Comparison methods: For CIL, we compared DLEPEM against several representative and recent methods, including SimpleCIL [10], prompt-based methods (L2P [7], DualPrompt [8], CODA-Prompt [18]), LoRA-based methods (LAE [9], APER [10], InfLoRA [4], SD-LoRA [11], BiLoRA [37]), and recent PTM-based CIL methods such as EASE [38]. We also include standard full fine-tuning as a baseline, where the model is sequentially fine-tuned without any continual learning mechanism. We reproduced all baseline results in our benchmark tables under the corresponding CIL/FSCIL protocols. For fairness, all methods use the same pre-trained backbone and identical data splits; for LoRA/adapter-based methods, we set the rank to $r=10$ where applicable. Other method-specific training settings, including optimizer, training epochs, batch size, and augmentation policy, follow their original or recommended implementations.

For FSCIL, we additionally benchmarked against three recent ViT-based methods tailored for few-shot scenarios: PriViLege [34], ASP [35], and CPE-CLIP [39]. All methods leverage the same pre-trained backbone (ViT-B/16-IN21K [26]) and identical data splits to ensure fair comparisons.

Evaluation metrics: For CIL, we assessed model performance using two established metrics: $\overline{\mathcal{A}}={\displaystyle \frac{1}{T}}{\sum }_{i=1}^T{\mathcal{ACC}}_i$ and ${\mathcal{A}}_L$ [4]. Here, $\overline{\mathcal{A}}$ is the average accuracy of all T incremental stages, and ${\mathcal{A}}_L$ is the accuracy of the last incremental stage. ${\mathcal{ACC}}_i$ is defined as

$${\mathcal{ACC}}_i={\displaystyle \frac{1}{i}}{\displaystyle \sum _{j=1}^i}{a}_{i,j},$$

(20)

where $a_{i,j}$ denotes the accuracy on the j-th task after training on the i-th task. For FSCIL, ${\mathcal{A}}_{\mathrm{Base}}$ is the accuracy of the base classes in task 0. Both ${\mathcal{A}}_L$ and $\overline{\mathcal{A}}$ are defined identically to those in the standard CIL setting.

Architecture and training details: We adopted ViT-B/16-IN21K [26] pre-trained on ImageNet-21K as the backbone. Optimization was conducted using SGD with an initial learning rate of 0.02 and cosine annealing. LoRA-Experts were trained for 20 epochs and the router for 5 epochs, using a batch size of 48. We set the LoRA rank to $r=10$ and inserted LoRA-Experts into all Transformer blocks. The distillation coefficient $\alpha$ was set to 0.04. All experiments were performed on an NVIDIA A800 GPU with fixed data splits (seed 1993) and the same backbone to ensure reproducibility; results were averaged over three runs. Following SD-LoRA [11], our QV-Experts were integrated into the query and value projections of the attention module. We also evaluated a variant, DLEPEM-MLP, which introduces MLP-Experts as a parallel branch to the FFN layer.

5.2. Benchmark Comparison

Class-incremental learning: We conducted a comprehensive evaluation of DLEPEM against representative recent methods on five benchmark datasets. As shown in

Table 4. Performance comparison on CIL benchmarks using the same ViT-B/16-IN21K backbone. Results are reported as mean ± standard deviation over three runs; the best result in each column is highlighted in bold.

Method	CIFAR100 (T = 10)		CUB200 (T = 10)		ImageNet-R (T = 5)		OmniBenchmark (T = 10)		VTAB (T = 5)
	${\mathcal{A}}_{\mathit{L}}$	$\overline{\mathcal{A}}$	${\mathcal{A}}_{\mathit{L}}$	$\overline{\mathcal{A}}$	${\mathcal{A}}_{\mathit{L}}$	$\overline{\mathcal{A}}$	${\mathcal{A}}_{\mathit{L}}$	$\overline{\mathcal{A}}$	${\mathcal{A}}_{\mathit{L}}$	$\overline{\mathcal{A}}$
Full Fine-Tuning	$66.26\pm 0.18$	$76.94\pm 0.02$	$55.29\pm 0.46$	$70.30\pm 0.74$	$59.90\pm 0.05$	$72.19\pm 0.12$	$47.75\pm 0.14$	$65.86\pm 0.12$	$62.95\pm 5.94$	$80.80\pm 1.50$
SimpleCIL [10]	$81.27\pm 0.01$	$87.13\pm 0.01$	$82.28\pm 6.35$	$91.85\pm 0.00$	$65.14\pm 15.29$	$59.72\pm 0.03$	$66.98\pm 8.93$	$79.35\pm 0.01$	$80.74\pm 5.26$	$90.80\pm 0.00$
L2P [7]	$84.82\pm 0.22$	$89.78\pm 0.01$	$71.98\pm 0.07$	$81.80\pm 0.03$	$72.08\pm 0.01$	$76.76\pm 0.06$	$64.45\pm 0.07$	$74.14\pm 0.03$	$64.27\pm 0.02$	$81.84\pm 0.19$
DualPrompt [8]	$85.23\pm 0.07$	$90.32\pm 0.06$	$74.23\pm 0.10$	$84.81\pm 0.02$	$69.34\pm 0.09$	$73.58\pm 0.05$	$66.16\pm 0.06$	$74.97\pm 0.03$	$78.90\pm 0.10$	$89.82\pm 0.06$
CODA-Prompt [18]	$86.69\pm 0.00$	$91.31\pm 0.01$	$75.45\pm 0.00$	$84.65\pm 0.00$	$75.16\pm 0.08$	$80.46\pm 0.04$	$68.67\pm 0.00$	$77.79\pm 0.00$	$75.08\pm 0.00$	$87.24\pm 0.00$
APER [10]	$87.32\pm 0.01$	$92.09\pm 0.02$	$86.82\pm 0.04$	$91.84\pm 0.03$	$68.22\pm 0.07$	$75.30\pm 0.07$	$74.40\pm 0.01$	$80.62\pm 0.01$	$84.44\pm 0.01$	$86.27\pm 0.02$
LAE [9]	$85.60\pm 0.19$	$91.26\pm 0.12$	$67.91\pm 0.04$	$80.17\pm 0.04$	$71.26\pm 0.18$	$77.08\pm 0.04$	$66.14\pm 0.06$	$74.89\pm 0.03$	$68.45\pm 2.53$	$85.31\pm 0.44$
InfLoRA [4]	$86.43\pm 0.02$	$91.80\pm 0.02$	$70.07\pm 0.26$	$81.71\pm 0.17$	$77.66\pm 0.08$	$82.90\pm 0.05$	$68.38\pm 0.12$	$78.06\pm 0.03$	$74.66\pm 0.43$	$86.03\pm 0.17$
SD-LoRA [11]	$87.62\pm 0.00$	$92.10\pm 0.00$	$72.69\pm 0.00$	$83.17\pm 0.00$	$78.52\pm 0.00$	$82.74\pm 0.00$	$69.32\pm 0.00$	$77.78\pm 0.00$	$67.73\pm 0.00$	$84.44\pm 0.00$
EASE [38]	$88.13\pm 0.04$	$92.59\pm 0.05$	$84.18\pm 0.06$	$90.20\pm 0.04$	$76.95\pm 0.05$	$81.48\pm 0.06$	$67.75\pm 0.04$	$74.85\pm 0.05$	$82.34\pm 0.06$	$90.45\pm 0.04$
BiLoRA [37]	$85.30\pm 0.05$	$90.73\pm 0.04$	$73.75\pm 0.06$	$83.67\pm 0.05$	$76.33\pm 0.04$	$81.21\pm 0.07$	$68.87\pm 0.05$	$77.53\pm 0.04$	$76.76\pm 0.06$	$88.94\pm 0.05$
DLEPEM-MLP	$85.99\pm 0.34$	$92.40\pm 0.03$	$\mathbf{87.70}\pm \mathbf{0.05}$	$\mathbf{92.31}\pm \mathbf{0.29}$	$76.25\pm 0.44$	$82.37\pm 0.14$	$74.32\pm 0.00$	$81.60\pm 0.00$	$84.96\pm 0.22$	$\mathbf{91.84}\pm \mathbf{0.03}$
DLEPEM-QV	$\mathbf{88.84}\pm \mathbf{0.09}$	$\mathbf{93.39}\pm \mathbf{0.10}$	$87.56\pm 0.16$	$92.09\pm 0.08$	$\mathbf{78.77}\pm \mathbf{0.07}$	$\mathbf{83.43}\pm \mathbf{0.12}$	$\mathbf{75.53}\pm \mathbf{0.08}$	$\mathbf{82.16}\pm \mathbf{0.06}$	$\mathbf{85.18}\pm \mathbf{0.14}$	$91.11\pm 0.13$

, DLEPEM consistently delivers superior accuracy across all benchmarks. After adding the recent EASE and BiLoRA baselines, DLEPEM still achieves the best average accuracy on all five CIL benchmarks under our reproduced experimental setting. Compared with the strongest baseline in each dataset, DLEPEM improves $\overline{\mathcal{A}}$ by 0.80% on CIFAR100, 2.11% on CUB200, 0.53% on ImageNet-R, 1.54% on OmniBenchmark, and 1.39% on VTAB.

For instance, on CIFAR100, DLEPEM-QV achieves an average accuracy of 93.39%, outperforming the newly added EASE baseline by 0.80%. Following the caution of Kim and Han [40], we do not interpret high average accuracy alone as proof of an optimal stability–plasticity balance; instead, the old/new-task analysis in Section 5.7 provides behavior-level evidence of the trade-off. Figure 2 further shows that DLEPEM achieves the highest performance throughout training, underscoring its robustness.

Few-shot class-incremental learning: We further evaluated DLEPEM in the few-shot class-incremental learning setting. As shown in

Table 5. Performance comparison on FSCIL benchmarks using the same ViT-B/16-IN21K backbone. Results are reported as mean ± standard deviation over three runs; the best result in each column is highlighted in bold.

Method	CUB200 (T = 11)			CIFAR100 (T = 9)			miniImageNet (T = 9)
	${\mathcal{A}}_{\mathrm{Base}}$	${\mathcal{A}}_{\mathit{L}}$	$\overline{\mathcal{A}}$	${\mathcal{A}}_{\mathrm{Base}}$	${\mathcal{A}}_{\mathit{L}}$	$\overline{\mathcal{A}}$	${\mathcal{A}}_{\mathrm{Base}}$	${\mathcal{A}}_{\mathit{L}}$	$\overline{\mathcal{A}}$
L2P [7]	$91.50\pm 0.00$	$50.04\pm 0.00$	$66.70\pm 0.00$	$93.43\pm 0.00$	$55.75\pm 0.00$	$71.81\pm 0.00$	$96.53\pm 0.00$	$60.91\pm 0.00$	$76.28\pm 0.00$
CODA-Prompt [18]	$91.50\pm 0.00$	$53.65\pm 0.00$	$69.30\pm 0.00$	$94.05\pm 0.00$	$57.10\pm 0.00$	$73.11\pm 0.00$	$97.15\pm 0.00$	$65.55\pm 0.00$	$78.83\pm 0.00$
InfLoRA [4]	$92.45\pm 0.20$	$45.18\pm 2.50$	$66.27\pm 1.43$	$\mathbf{94.92}\pm \mathbf{0.06}$	$57.41\pm 0.31$	$74.28\pm 0.28$	$97.42\pm 0.06$	$51.52\pm 0.10$	$71.55\pm 0.03$
SD-LoRA [11]	$91.92\pm 0.00$	$56.28\pm 0.00$	$70.87\pm 0.00$	$94.60\pm 0.00$	$73.51\pm 0.00$	$78.42\pm 0.00$	$\mathbf{97.72}\pm \mathbf{0.00}$	$79.07\pm 0.00$	$84.36\pm 0.00$
CPE-CLIP [39]	$80.21\pm 0.85$	$63.32\pm 0.17$	$69.37\pm 0.37$	$88.32\pm 0.04$	$79.99\pm 0.18$	$83.38\pm 0.10$	$90.14\pm 0.05$	$81.55\pm 0.04$	$85.54\pm 0.04$
ASP [35]	$87.14\pm 0.11$	$82.86\pm 0.26$	$83.46\pm 0.22$	$91.77\pm 0.09$	$86.04\pm 0.03$	$88.54\pm 0.03$	$96.32\pm 0.15$	$93.72\pm 0.25$	$94.97\pm 0.21$
PriViLege [34]	$82.21\pm 0.35$	$75.08\pm 0.52$	$77.50\pm 0.33$	$90.88\pm 0.20$	$86.06\pm 0.32$	$88.08\pm 0.20$	$96.68\pm 0.06$	$\mathbf{94.10}\pm \mathbf{0.13}$	$\mathbf{95.27}\pm \mathbf{0.11}$
DLEPEM-MLP	$92.51\pm 0.00$	$85.37\pm 0.00$	$88.53\pm 0.00$	$94.28\pm 0.00$	$84.67\pm 0.00$	$89.00\pm 0.00$	$96.37\pm 0.00$	$89.96\pm 0.00$	$93.11\pm 0.00$
DLEPEM-QV	$\mathbf{92.68}\pm \mathbf{0.00}$	$\mathbf{86.22}\pm \mathbf{0.00}$	$\mathbf{88.77}\pm \mathbf{0.00}$	$94.00\pm 0.00$	$\mathbf{87.29}\pm \mathbf{0.00}$	$\mathbf{90.50}\pm \mathbf{0.00}$	$96.77\pm 0.00$	$93.62\pm 0.00$	$94.80\pm 0.00$

, DLEPEM consistently delivers superior accuracy and achieves leading results among the evaluated methods on multiple benchmarks. In particular, it achieves the highest last accuracy (${\mathcal{A}}_L$) and average accuracy ($\overline{\mathcal{A}}$) on CUB200 and CIFAR100. For instance, on CUB200, DLEPEM-QV achieves an average accuracy of 88.77%, outperforming the strongest baseline, ASP, by a clear margin of 5.31%. On CIFAR100, DLEPEM-QV reaches 90.50%, surpassing ASP by 1.96%. While its performance on miniImageNet is highly competitive and on par with the strongest baselines, these substantial gains on the other datasets highlight DLEPEM’s effectiveness in long-term continual learning, even under severe data sparsity. Figure 3 further illustrates that DLEPEM maintains superior performance throughout the incremental learning process, showing its robustness and adaptability in few-shot scenarios.

5.3. Ablation Study

Different components: We conducted ablation studies to assess the contribution of each component in DLEPEM (

Table 6. Ablation study on CIL and FSCIL tasks. The first two benchmarks follow CIL protocols, while the last two follow FSCIL protocols. Each metric reports MLP-Expert/QV-Expert performance.

Ablated Components	CIFAR100 (T = 10)		ImageNet-R (T = 5)		CIFAR100 (T = 9)		miniImageNet (T = 9)
	${\mathcal{A}}_{\mathit{L}}$	$\overline{\mathcal{A}}$	${\mathcal{A}}_{\mathit{L}}$	$\overline{\mathcal{A}}$	${\mathcal{A}}_{\mathit{L}}$	$\overline{\mathcal{A}}$	${\mathcal{A}}_{\mathit{L}}$	$\overline{\mathcal{A}}$
w/o Dynamic LoRA-Expert	83.13/86.72	88.85/90.82	61.17/72.37	74.31/80.14	73.03/79.73	80.39/86.59	84.26/91.11	89.90/93.31
w/o Prototype Ensemble	85.79/87.59	91.21/92.29	69.53/73.13	77.75/80.42	81.13/84.09	86.97/88.67	85.54/92.25	90.03/94.06
DLEPEM-MLP/QV	85.99/88.84	92.40/93.39	76.25/78.77	82.37/83.43	84.67/87.29	89.00/90.5	89.96/93.62	93.11/94.80

). The variant w/o Dynamic LoRA-Expert uses a single LoRA-Expert for all tasks, resulting in a significant performance drop under large domain shifts and indicating severe forgetting and task interference. In contrast, assigning a dedicated LoRA-Expert to each task better preserves the stability–plasticity trade-off across incremental steps. This result underscores the value of task-specific experts in PEFT-based continual learning. Removing the prototype-ensemble router (w/o Prototype-Ensemble) and using frozen class prototypes as keys also degrades performance, confirming the router’s critical role in effective module-sample matching. The advantage of dynamic expert allocation is most visible on ImageNet-R. Replacing the single shared expert with task-specific dynamic experts increases DLEPEM-MLP from 61.17/74.31 to 76.25/82.37 in ${\mathcal{A}}_L/\overline{\mathcal{A}}$, and increases DLEPEM-QV from 72.37/80.14 to 78.77/83.43. These gains support our claim that isolating LoRA parameters by task reduces cross-task interference, especially under domain shift.

Different routers: In addition to the prototype-ensemble strategy, we compared three module-sample-matching mechanisms: frozen-PTM prototypes ${\mathbf{P}}^F$, K-nearest-neighbor (KNN) matching, and router-domain prototypes from ${\mathbf{E}}^{router}$. For KNN, features were extracted using the frozen PTM, and performance was evaluated across k values ($k=\{1,3,5,7,9\}$), with $k=3$ yielding the best results. Both ${\mathbf{P}}^F$ and ${\mathbf{E}}^{router}$ provide single-source keys within the prototype-ensemble framework.

In

Table 7. Diagnostic accuracy of the classifier, router, and oracle expert selection on five CIL benchmarks. Each entry is reported as DLEPEM-MLP/DLEPEM-QV; the oracle expert setting uses the ground-truth task identity only for diagnostic analysis.

Metric	CIFAR100 ($\mathit{T}=10$)	CUB200 ($\mathit{T}=10$)	ImageNet-R ($\mathit{T}=5$)	OmniBenchmark ($\mathit{T}=10$)	VTAB ($\mathit{T}=5$)
CNN	$92.40/93.39$	$92.31/92.09$	$82.37/83.43$	$81.60/82.16$	$91.84/91.11$
Router	$91.88/93.68$	$93.24/93.31$	$88.43/88.87$	$84.90/85.55$	$93.45/93.56$
LoRA-Expert (Oracle)	$98.04/97.81$	$97.15/96.31$	$89.17/88.81$	$92.88/91.48$	$97.66/95.88$

, each entry is reported as DLEPEM-MLP/DLEPEM-QV. CNN reports the final average classification accuracy of each variant. Router reports the expert-selection accuracy of the learned routing module. LoRA-Expert (Oracle) reports classification accuracy when the ground-truth task/expert identity is used at test time to select the correct LoRA-Expert before normal within-expert classification.

As shown in Figure 4, the prototype ensemble consistently outperforms the baseline KNN approach. Two key insights emerge:

1. Domain-specific adaptation: Under significant domain shift (e.g., ImageNet-R $\left(T=5\right)$), the ensemble achieves larger gains, primarily due to ${\mathbf{E}}^{router}$’s ability to capture domain-specific characteristics.

2. Robust matching via integration: Combining generalized and domain-aware prototypes enables more reliable module-sample matching. Even when ${\mathbf{E}}^{router}$ underperforms ${\mathbf{P}}^F$, its ensemble compensates through mutual alignment, enhancing overall stability and accuracy. This conclusion is also consistent with the component ablation in Table 6: on ImageNet-R, replacing prototype-ensemble retrieval with frozen-prototype keys reduces ${\mathcal{A}}_L/\overline{\mathcal{A}}$ from 76.25/82.37 to 69.53/77.75 for DLEPEM-MLP and from 78.77/83.43 to 73.13/80.42 for DLEPEM-QV. The result indicates that combining PTM-general and router-domain prototypes provides more reliable MII than relying on a single feature source. Figure 4 also contains the two single-source prototype variants requested by the reviewer: the frozen-PTM prototype ${\mathbf{P}}^F$ and the router-domain prototype ${\mathbf{E}}^{router}$. Across the evaluated settings, both single-source variants, including the router-domain prototype variant, obtain lower accuracy than the prototype-ensemble strategy, which supports the complementarity of the two prototype sources. This router comparison also serves as a post-hoc compatibility check for the append-only dictionary. During evaluation, current queries are matched against stored keys accumulated from previous stages. If historical router-domain keys were incompatible with the current router state, prototype-ensemble retrieval would not consistently outperform PTM-only or KNN-style matching. The observed gains, therefore, suggest that the frozen-PTM component and stability-regularized router updates help maintain usable key-query compatibility across incremental stages.

Different pre-trained models: Beyond validating DLEPEM on ViT-B/16-IN21K, we further assessed its generalization across diverse Transformer-based architectures, including ViT-B/16-IN1K & ViT-L/16 [26], ViT-B/16-DINO [41], and ViT-B/16-SAM [42]. As a baseline, SimpleCIL [10] fine-tunes only the classifier to reflect the inherent capability of each backbone in incremental settings. We evaluated DLEPEM on the CIFAR100 and ImageNet-R benchmarks, with the results shown in Figure 5a,b. Two key observations emerge:

1.

DLEPEM yields greater improvements on larger models (e.g., ViT-L/16), highlighting its scalability with model capacity.

2.

Among similarly sized architectures (e.g., ViT-B/16 variants), DLEPEM consistently outperforms the SimpleCIL baseline, demonstrating robustness to architectural and pre-training differences.

These results show DLEPEM’s versatility and effectiveness across a wide range of Transformer-based backbones.

5.4. Analysis of Trainable Parameters and Accuracy

To further evaluate the parameter-performance trade-off, we compared the number of trainable parameters with accuracy across different methods in Figure 6. As illustrated in these plots, DLEPEM achieves a favorable balance between model size and performance, demonstrating that it uses additional parameters efficiently to improve accuracy.

We further quantified the scalability of the one-expert-per-task design. DLEPEM is rehearsal-free in the sense that it stores no raw samples, old feature batches, or exemplar buffers; nevertheless, it retains a frozen LoRA-Expert bank, the current router, and an append-only prototype dictionary. For a ViT with hidden dimension d, LoRA rank r, L Transformer blocks, and b bytes per floating-point value, the storage of one MLP-Expert, one QV-Expert, and the prototype dictionary after task t can be written as

$$\begin{array}{cc}\hfill N_{\mathrm{MLP}}& =L\left(2dr\right),\hfill \\ \hfill N_{\mathrm{QV}}& =2L\left(2dr\right),\hfill \\ \hfill M_{\mathrm{proto}}\left(t\right)& =2d|{\mathcal{Y}}_{1:t}|b.\hfill \end{array}$$

(21)

For the LoRA-Expert bank, under our default ViT-B/16-IN21K setting, $d=768$, $L=12$, $r=10$, and $b=4$ for FP32. Thus, each MLP-Expert adds 0.18 M parameters (≈0.70 MiB), and each QV-Expert adds 0.37 M parameters (≈1.41 MiB). In a 10-task setting, the frozen expert bank stores 1.84 M LoRA parameters for DLEPEM-MLP and 3.69 M for DLEPEM-QV, corresponding to approximately 2.1% and 4.3% of an 86 M-parameter ViT-B backbone, respectively. For longer task streams, the growth is linear: in a 50-task sequence, the expert bank contains 9.22 M parameters for DLEPEM-MLP and 18.43 M for DLEPEM-QV; in a 100-task sequence, it contains 18.43 M and 36.86 M parameters, respectively. The router contributes one additional same-size LoRA module and is constant with respect to the number of tasks. Therefore, the LoRA-Expert bank remains lightweight under the evaluated protocols, but it is the main component that scales with the number of tasks.

For the prototype dictionary, each ensemble prototype contains two 768-dimensional vectors: one general prototype from the frozen PTM feature space and one domain-specific prototype from the router feature space. Its memory, therefore, scales with the number of seen classes rather than directly with the number of tasks. In FP32, the dictionary requires about 0.59 MiB for 100 classes, 1.17 MiB for 200 classes, and 1.76 MiB for 300 classes. Compared with both the LoRA-Expert bank and the ViT-B backbone, this storage is small in our evaluated benchmarks, but it still grows as $O\left(\right|{\mathcal{Y}}_{1:T}\left|\right)$ with the number of seen classes.

5.5. Training and Inference Time Comparison

At inference time, task-specific LoRA-Expert selection is based on nearest-neighbor search using cosine similarity, formulated as $\widehat{i}=arg{max}_{i\in {\mathbf{Y}}_t}\cos (\mathbf{q}\left(\mathbf{x}\right),{\mathbf{K}}_i)$.

Table 8. Training and inference time comparison. Training time is the average time per epoch for each incremental task, and inference latency is measured in milliseconds per image. All methods use the same ViT-B/16-IN21K backbone for a fair comparison.

Method	CIFAR100 (T = 10)		ImageNet-R (T = 5)
	Training Time (s)	Inference Time (ms)	Training Time (s)	Inference Time (ms)
L2P [7]	102.12	3.77	50.12	3.84
DualPrompt [8]	93.21	3.44	45.16	3.58
CODA-Prompt [18]	99.42	2.99	47.53	3.08
LAE [9]	46.87	3.35	24.26	3.52
InfLoRA [4]	72.36	2.03	34.96	2.24
APER [10]	16.04	3.63	8.92	3.71
SD-LoRA [11]	79.08	1.93	32.85	2.04
DLEPEM-MLP/QV	69.46/71.37	8.39/9.26	33.3/34.15	8.47/9.08

reports training time and inference latency. The reported inference latency is measured end-to-end for one image and includes query construction with the frozen PTM and router, nearest-neighbor expert selection in the prototype dictionary, the selected LoRA-Expert forward pass, and final prototype-weight classification. DLEPEM-MLP requires $69.46$ s/epoch on CIFAR100 and $33.30$ s/epoch on ImageNet-R, which is competitive with representative PEFT-based baselines. Its inference latency is $8.39$ ms/image on CIFAR100 and $8.47$ ms/image on ImageNet-R. This latency is higher than that of SD-LoRA and InfLoRA, so the overhead should not be described as negligible. A more accurate interpretation is that DLEPEM introduces a non-negligible but still practical inference overhead in exchange for stronger expert retrieval and final accuracy.

From a deployment-memory perspective, the storage analysis above shows that DLEPEM remains lightweight relative to the ViT-B/16-IN21K backbone under the evaluated protocols. For the LoRA-Expert bank, each MLP-Expert adds about $0.70$ MiB, and each QV-Expert adds about $1.41$ MiB in FP32; in a 10-task setting, the retained expert bank accounts for about $2.1\%$ and $4.3\%$ of the 86 M-parameter backbone for DLEPEM-MLP and DLEPEM-QV, respectively. For the prototype dictionary, the storage is smaller and requires at most about $1.76$ MiB for 300 seen classes in our evaluated CIL benchmarks. Therefore, DLEPEM is suitable for GPU-based real-time or near-real-time recognition scenarios, while stricter edge deployment or much longer task streams may require additional expert compression or pruning.

5.6. Parameter Sensitivity Analysis

We investigated the sensitivity of three key hyperparameters in DLEPEM: (1) the LoRA rank r, (2) the insertion positions of LoRA-Experts in the ViT backbone, and (3) the distillation coefficient $\alpha$ for plasticity regularization.

To evaluate the effect of r and insertion positions, we conducted experiments on CUB200 ($T=10$), varying $r\in \{1,2,4,8,10,16\}$ and testing insertion ranges {0–2, 0–4, 0–8, 0–12}, where “0–2” denotes insertion into the first two Transformer layers. As shown in Figure 7a, DLEPEM maintains stable performance across various settings, demonstrating robustness to hyperparameter variations. Following prior work [4,11], we adopt $r=10$ and insert LoRA-Experts into all Transformer blocks. Similar trends are observed on other datasets.

We also analyzed the impact of the distillation coefficient $\alpha$ across datasets (Figure 7b). The two endpoints correspond to removing one distillation term: $\alpha =0$ removes plasticity feature distillation and keeps only ${\mathcal{L}}_{SFD}$, while $\alpha =1$ removes stability feature distillation and keeps only ${\mathcal{L}}_{PFD}$. The model performs consistently well for $\alpha \in (0,0.1]$, with $\alpha =0.04$ as the default.

5.7. Performance of LoRA-Based Methods Across Sequential Tasks

Kim and Han [40] show that final or average accuracy can obscure whether a CIL method is genuinely plastic or mainly stable, and they propose feature-representation diagnostics such as classifier retraining with frozen feature extractors and representation-similarity analysis. We do not reproduce that full representation-level protocol here. Instead, using the available sequential-task results, we provide a behavior-level decomposition: new-task performance is used as an empirical proxy for plasticity, and old-task performance after subsequent learning is used as an empirical proxy for stability.

To further analyze the performance characteristics of LoRA-based methods, we evaluate LAE [9], InfLoRA [4], and SD-LoRA [11] in terms of model plasticity and stability on CIFAR100 ($T=10$). As shown in Figure 8a, both InfLoRA and SD-LoRA show lower performance on new tasks because their gradient-direction constraints are designed to preserve old knowledge. While these constraints effectively mitigate catastrophic forgetting, they can compromise the model’s plasticity when learning new tasks. In contrast, DLEPEM learns task-specific LoRA modules without such restrictions, enabling stronger adaptation to new tasks while maintaining competitive performance. This comparison further distinguishes DLEPEM from constraint-based LoRA continual learning: InfLoRA and SD-LoRA protect old knowledge by restricting update directions or magnitudes, whereas DLEPEM preserves old knowledge structurally by freezing previous experts while keeping the current expert fully trainable within its low-rank subspace. Thus, DLEPEM shifts the stability–plasticity trade-off from a single shared update space to two separated factors: parameter-stationary old experts for stability and a newly trainable expert for plasticity.

LAE employs a different strategy by continuously integrating new parameters into the existing model through weighted blending. However, as demonstrated in Figure 8b, this approach leads to progressive destabilization of previously learned knowledge. DLEPEM consistently outperforms LAE on old tasks, demonstrating that our dynamic expert selection mechanism effectively preserves old knowledge without the instability issues associated with parameter blending.

6. Conclusions

We propose DLEPEM, a novel framework for CIL that combines Dynamic LoRA-Experts for task-specific adaptation and prototype-ensemble matching for improved module selection. This dual design supports a more favorable empirical stability–plasticity trade-off under the evaluated CIL protocols by combining parameter-stationary old experts for stability with a trainable expert for each new task to preserve plasticity. Extensive evaluations on six benchmarks demonstrate that DLEPEM achieves strong and competitive performance under the evaluated protocols. Although each LoRA-Expert is lightweight, the stored expert bank grows linearly with the number of tasks, and the prototype dictionary grows with the number of seen classes. Future work will explore expert compression, rank sharing, expert pruning, expert merging/distillation, different prototype fusion weights, random and expert-ID routing diagnostics, and extensions to multi-modal learning.