Dual-Modal Contrastive Learning for Continual Generalized Category Discovery

Abstract

Continual Generalized Category Discovery (C-GCD) is an emerging research direction in Open-World Learning. The model aims to incrementally discover novel classes from unlabeled data while maintaining recognition of previously learned classes, without accessing historical samples. The absence of supervision signal in incremental sessions makes catastrophic forgetting more severe than in traditional incremental learning. Existing methods primarily enhance generalization through single-modality contrastive learning, overlooking the natural advantages of textual information. Visual features capture perceptual details such as shapes and textures, while textual information helps distinguish visually similar but semantically distinct categories, offering complementary benefits. However, directly obtaining category descriptions for unlabeled data in C-GCD is challenging. To address this, we introduce a conditional prompt learning mechanism to generate pseudo-prompts as textual information for unlabeled samples. Additionally, we propose a dual-modal contrastive learning strategy to enhance vision-text alignment and exploit CLIP’s multimodal potential. Extensive experiments on four benchmark datasets demonstrate that our method achieves competitive performance. We hope this work provides new insights for future research.

1. Introduction

Traditional machine learning systems operate under the closed-set assumption, where the environment is static and the model remains fixed after deployment. However, this assumption is violated in real-world applications, where environments are inherently dynamic and uncertain. To address this limitation, Open-World Learning (OWL) [1] has emerged as a new paradigm that enables models to operate safely and evolve continuously through three integrated capabilities: rejecting unknown instances, discovering novel categories, and learning incrementally. Figure 1 illustrates the complete open-world learning pipeline. Unlike out-of-distribution (OOD) detection [2], which simply rejects unknown samples, open-world learning requires models to semantically cluster novel samples into distinct categories. This challenge is addressed by Novel/Generalized Category Discovery (N/GCD) [3,4]. And then, discovered categories are incorporated into the model through class incremental learning (CIL) [5], completing the learning cycle.

Figure 1. Overview of the open-world learning life cycle. Our work focuses on continual Generalized Category Discovery, which integrates Generalized Category Discovery and incremental learning.

This paper focuses on Continual Generalized Category Discovery (C-GCD) [6,7], which extends GCD to continual settings. The model first acquires foundational knowledge through supervised training on labeled old classes in an offline session. It then proceeds through sequential online sessions containing unlabeled data with predominantly novel classes and sparse known class samples. Crucially, all learning proceeds entirely without supervision, and data from previous sessions are inaccessible [8], simulating autonomous learning of AI agents in real-world scenarios. The objective is to discover novel classes while preserving recognition capability for all previously learned classes.

C-GCD is essentially unsupervised continual learning. Unlike supervised continual learning [9], where labeled data provides explicit guidance, C-GCD operates without supervision throughout all online sessions. This characteristic causes the model to naturally adapt its representations in the direction of gradient descent during training [10]. Consequently, when novel class samples vastly outnumber old class samples, the model tends to misclassify old samples as novel classes, resulting in task-level overfitting. Furthermore, without supervisory signals, decision boundaries become increasingly blurred as online sessions progress, as confirmed by our baseline experiments (Section 4).

Existing C-GCD methods [6,7,11,12] primarily operate within a single visual modality framework. Despite leveraging visual features extracted from self-supervised vision transformers (ViT) such as DINO [13], these methods exhibit limitations in effectively generalizing visual concepts. Recent studies [14,15] have shown that adding simple textual descriptions such as “a photo of [class]” can significantly improve classification performance. This improvement can be attributed to the fact that although certain classes exhibit visual ambiguity, their corresponding textual descriptions typically encode more discriminative features. An intuitive approach is to leverage image descriptions to provide pseudo-supervision for the corresponding images, thereby enhancing novel class discovery and mitigating catastrophic forgetting. The advantage of this approach is that the supervision information not only alleviates the previously mentioned task-level overfitting and boundary degradation, but the cross-entropy loss also achieves higher performance than non-parametric classifiers. However, a key limitation is that C-GCD cannot directly access category information from unlabeled data, which hinders existing frameworks from effectively leveraging multimodal information.

To address this limitation, we propose DMCL, a prompt-based framework that leverages Dual-Modal Contrastive Learning with multimodal self-distillation. Our framework operates in two stages. In the first stage, we introduce a prompt generation network that synthesizes textual descriptions for training images. Inspired by conditional prompt learning [15], we develop a lightweight single-layer architecture to generate image-specific tokens as initial pseudo-prompt representations, which are then embedded into CLIP’s shared feature space alongside visual features. We optimize this network through dual modal contrastive learning strategies: (1) aligning generated pseudo-prompts with ground-truth prompts via contrastive learning and mean squared error loss for labeled data, and (2) employing self-supervised cross-prediction on the labeled dataset to strengthen visual–textual correlations. In the second stage, we leverage the trained generator to produce textual descriptions for unlabeled samples in online sessions. These descriptions are fused with visual representations through an adaptive fusion strategy, and the unified features are fed into a self-distillation framework [16] to generate text-enhanced pseudo-labels. This approach improves classification performance while mitigating catastrophic forgetting in unsupervised continual learning.

The main contributions of this work are as follows:

1.

We attribute catastrophic forgetting in C-GCD to the absence of supervision. Specifically, we identify two critical issues: First, novel samples exhibit weak representational capacity, making them susceptible to semantic drift in subsequent sessions, leading to task-level overfitting. Second, semantic clusters spontaneously formed by novel samples are influenced by individual sample representations, resulting in small inter-class distances and large intra-class variance.

2.

To address insufficient supervision, we propose DMCL, which incorporates a prompt generation network to synthesize textual descriptions for unlabeled samples. By fusing textual and visual modalities, we enrich sample representations and enhance discriminability, effectively mitigating the aforementioned issues. Furthermore, we introduce dual-modal contrastive learning to strengthen vision-text alignment and a multimodal self-distillation framework to generate high-quality pseudo-labels.

3.

Extensive experiments on CIFAR-100, Tiny-ImageNet, ImageNet-100, and CUB-200 demonstrate the effectiveness of our method across both generic and fine-grained datasets. Comprehensive ablation studies and qualitative analyses confirm that our approach successfully addresses task-level overfitting and decision boundary degradation, achieving superior plasticity–stability trade-offs in continual learning scenarios.

2. Related Work

2.1. Novel Category Discovery

Novel category discovery (NCD) addresses open-world classification challenges. Early NCD methods adopted two-stage approaches derived from transfer learning principles. Deep Transfer Clustering (DTC) [17] exemplifies this paradigm by first training embeddings on labeled data, then applying them to unlabeled instances. Subsequently, contrastive learning [18,19] enabled one-stage joint training on both labeled and unlabeled data. RankStats [20] employs self-supervised learning with ranking statistics for similarity computation, while UNO [4] introduces parametric classifiers through pseudo-labeling to enable cross-entropy optimization. These advances established the foundation for NCD and GCD research.

2.2. Generalized Category Discovery

Generalized Category Discovery (GCD) extends NCD to scenarios where unlabeled data contain both known and novel classes, also termed Open-World Semi-Supervised Learning (OW-SSL) [21,22,23]. Existing GCD methods typically extract visual features using pretrained vision transformers, then fine-tune through supervised learning on labeled samples and self-supervised contrastive learning on all data. Like NCD, GCD assumes prior knowledge of novel class numbers. Classification approaches divide into two paradigms: non-parametric and parametric methods. Non-parametric Methods utilize clustering-based classifiers, typically K-means [24], for novel category discovery. These methods mitigate overfitting to known classes while providing scalability for varying class numbers. The seminal GCD work constrains labeled clusters, then applies semi-supervised K-means on unlabeled data. Subsequent works exploit semantic structures through contrastive learning and prompt mechanisms. Some approaches leverage non-parametric properties to discover classes without prior knowledge of class numbers. Parametric Methods employ learnable classification heads for predictions. SimGCD [16] demonstrates that parametric models can outperform non-parametric counterparts when pseudo-label quality is sufficient, proposing a self-distillation framework that improves pseudo-label reliability. Recent works address biases in pseudo-label generation [25] and class imbalance [26], achieving state-of-the-art performance.

2.3. Continual Category Discovery

Investigating N/GCD in continual settings represents a natural progression in open-world learning. The terms “Continual” and “Incremental” are used interchangeably, both referring to class incremental learning.

NCDwF [11] investigates NCD in continual settings where models learn from labeled data then discover novel classes from unlabeled data. FRoST introduces replay-based storage of feature prototypes, while MSc-iNCD [27] leverages pretrained self-supervised models. Grow & Merge [12] extends to continual GCD, where models access labeled data initially and sequential unlabeled data subsequently. It employs growing phases for novel class detection and merging phases for knowledge distillation. IGCD [28] studies continual GCD on iNat2021 [29], processing partially labeled images at each stage.

Recent works explore alternative strategies. CLIP-CNCD [30] leverages CLIP’s zero-shot capabilities, decoupling old knowledge preservation from novel discovery through CLIP-generated pseudo-labels with CutMix [31] augmentation. PromptCCD [6] introduces Gaussian Mixture Prompting [32] with dynamic prompt pools for adaptive selection and on-the-fly class number estimation. TSPT [33] proposes rehearsal-free Two-Step Prompt Tuning with clustering-guided consistency learning and prompt pool selection. Happy [7] addresses class imbalance and pseudo-label biases through hardness-aware prototype sampling and soft entropy regularization.

Current C-GCD research adopts different configurations for novel class emergence. This work follows [7]: labeled data at the initial stage, progressive novel class discovery from unlabeled data in subsequent stages.

3. Method

3.1. Problem Definition

Figure 2 illustrates the C-GCD task pipeline. Formally, the model operates on labeled data ${\mathcal{D}}_l^t=\left\{(x_i^t,y_i^t)\right\}$ and unlabeled data ${\mathcal{D}}_u^t=\{x_i^t,y_i^t\}$, where t denotes the session index. In our setting, labeled data is exclusively available at the initial stage (Session 0, $t=0$), also referred to as the offline session. We denote the label space of labeled data as ${\mathcal{Y}}^l$ and the complete label space including novel classes as ${\mathcal{Y}}^u$, where ${\mathcal{Y}}^l\subset {\mathcal{Y}}^u$. The model acquires foundational knowledge from ${\mathcal{D}}_l^0$ containing old classes ${\mathcal{C}}_{\mathrm{old}}^0$. Subsequently, the framework transitions into continual unsupervised discovery from Session 1 to T, where each session t presents unlabeled data ${\mathcal{D}}_u^t$ containing both previously encountered and novel categories. Throughout this process, known classes ${\mathcal{C}}_{\mathrm{known}}^t$ encompass all previously observed categories, including the old classes ${\mathcal{C}}_{\mathrm{old}}^0$ and novel classes discovered in prior sessions ${\mathcal{C}}_{\mathrm{seen}}^t$. Novel classes ${\mathcal{C}}_{\mathrm{novel}}^t$ represent categories first encountered at session t. This creates a progressive knowledge accumulation pattern where ${\mathcal{C}}_{\mathrm{seen}}^t={\mathcal{C}}_{\mathrm{novel}}^{t-1}$, meaning today’s discoveries become tomorrow’s foundation. The model’s performance is evaluated on test set ${\mathcal{D}}_{\mathrm{test}}^t\in {\mathcal{C}}_{\mathrm{known}}^t\cup {\mathcal{C}}_{\mathrm{novel}}^t$, encompassing all categories observed through session t.

Figure 2. Pipeline of Continual Generalized Category Discovery. The model first learns old classes in Session 0, then discovers novel classes while retaining known classes across sequential sessions. Novel classes discovered in session t become seen classes in session $t+1$.

3.2. Baseline Framework

A straightforward solution is to adapt off-the-shelf GCD methods to continual settings. We develop a baseline framework, termed Base-CGCD. This baseline evaluates existing approaches under continual learning and reveals inherent challenges in C-GCD. As illustrated in Figure 3, Base-CGCD adopts a self-distillation architecture that performs GCD at each session. Upon discovering novel classes, we initialize new prototype heads using K-means clustering on learned features, enabling incremental learning through iterative head expansion.

Figure 3. Overview of the DMCL framework. Top: Base-CGCD employs self-distillation for pseudo-label generation, where two augmented views are fed through a shared encoder to produce predictions and sharpened targets. Bottom-left: Prompt Generator trains a meta-module to generate pseudo-textual descriptions for unlabeled data via CLIP text encoder. And then, dual-modal contrastive learning aligns visual and textual modalities through bidirectional contrastive objectives and distribution matching loss. Bottom-right: Classifier Update shows progressive expansion from session 1 to T, where novel prototypes (red) are initialized alongside existing known prototypes (gray).

3.2.1. Representation Learning

Following [16], we employ both supervised and self-supervised learning in the ${\ell }_2$ projection space to enhance the model’s generalization capability:

$$\begin{array}{cc}\hfill & {\mathcal{L}}_{rep}^u={\displaystyle \frac{1}{\left|B\right|}}\sum _{i\in B}-\log {\displaystyle \frac{\exp (z_i^{\top }{z^{\prime }}_i/{\tau }_u)}{{\sum }_i^{i\in n}\exp (z_i^{\top }{z}_n^{\prime }/{\tau }_u)}},\hfill \end{array}$$

(1a)

$$\begin{array}{cc}\hfill & {\mathcal{L}}_{rep}^s={\displaystyle \frac{1}{|B^l|}}\sum _{i\in B^l}{\displaystyle \frac{1}{|{\mathcal{N}}_i|}}\sum _{q\in {\mathcal{N}}_i}-\log {\displaystyle \frac{\exp (z_i^{\top }{z}_q^{\prime }/{\tau }_c)}{{\sum }_i^{i\in n}\exp (z_i^{\top }{z}_n^{\prime }/{\tau }_c)}},\hfill \end{array}$$

(1b)

where the feature $z_i^t=h\left(f\left(x_i^t\right)\right)$ is ${\ell }_2$ normalized, $f(·)$ and $h(·)$ denote the backbone and the projection head, respectively, and ${\tau }_u$, ${\tau }_c$ are the temperature parameters. For clarity, the session index t is omitted in subsequent equations. In supervised contrastive learning, ${\mathcal{N}}_i$ indexes all samples in the same batch $B^l$ sharing the same label as $x_i$. The overall representation learning loss is balanced with ${\lambda }_{rep}:{\mathcal{L}}_{rep}=(1-{\lambda }_{rep}){\mathcal{L}}_{rep}^u+{\lambda }_{rep}{\mathcal{L}}_{rep}^s.$

3.2.2. Parametric Classification

We adopt the parametric classification paradigm for category discovery. Specifically, we initialize classifier weights ${\mathcal{W}}^t=\{w_1,\dots ,w_K\}$ for all classes at session t, where $K=|{\mathcal{C}}_{\mathrm{known}}^t\cup {\mathcal{C}}_{\mathrm{novel}}^t|$. During training, for each augmented view $x_i$, we compute the prediction distribution via softmax over cosine similarities between hidden feature $h_i=f\left(x_i\right)$ and prototypes ${\mathcal{W}}^t$, scaled by temperature $1/{\tau }_s$:

$$p_i^{\left(k\right)}={\displaystyle \frac{exp(z_i^{\top }{w}_k/{\tau }_s)}{{\sum }_{k^{\prime }}exp(z_i^{\top }{w}_{k^{\prime }}/{\tau }_s)}}.$$

(2)

The soft pseudo-label $q_i^{\prime }$ is generated from another augmented view $x_i^{\prime }$ using a sharper temperature ${\tau }_t<{\tau }_s$. For the offline session, we apply standard supervised cross-entropy loss:

$${\mathcal{L}}_{\mathrm{cls}}=-{\displaystyle \frac{1}{\left|B\right|}}\sum _{i\in B}{y}_{i}log{p}_i,$$

(3)

where $y_i$ denotes the one-hot label of $x_i$. For online sessions, which are entirely unsupervised, we replace ground-truth labels $y_i$ with soft pseudo-labels $q_i^{\prime }$ to construct the cross-entropy loss.

3.2.3. Incremental Update Strategy

We achieve continual discovery through iterative expansion of prototype classification heads. Specifically, as illustrated in the “Classifier Update” module of Figure 3, we perform K-means clustering on ${\mathcal{D}}_u^t$ to obtain K ${\ell }_2$-normalized cluster centroids ${\left\{c_i\right\}}_{i=1}^K$. Among these, the $K_{\mathrm{novel}}$ centroids with the lowest maximum cosine similarity to existing old class heads are selected to initialize new prototype heads:

$${\left\{w_j^{\mathrm{new}}\right\}}_{j=1}^{K_{\mathrm{novel}}}=\mathrm{top}-K_{\mathrm{novel}}\left(-\underset{i}{max}{c}_j^{\top }{w}_i^{\mathrm{old}}\right).$$

(4)

To mitigate catastrophic forgetting, we apply feature-level knowledge distillation [34] between the current model $f^t$ and the previous session model $f^{t-1}$:

$${\mathcal{L}}_{\mathrm{kd}}={\displaystyle \frac{1}{\left|B\right|}}\sum _{i\in B}\left(1-⟨f^t\left(x_i\right),f^{t-1}\left(x_i\right)⟩\right).$$

(5)

The overall optimization objectives for offline and online sessions are formulated as follows:

$$\begin{array}{cc}\hfill {\mathcal{L}}_{\mathrm{offline}}& ={\mathcal{L}}_{\mathrm{cls}}+{\mathcal{L}}_{\mathrm{rep}},\hfill \end{array}$$

(6a)

$$\begin{array}{cc}\hfill {\mathcal{L}}_{\mathrm{online}}& ={\mathcal{L}}_{\mathrm{cls}}+{\mathcal{L}}_{\mathrm{rep}}+{\mathcal{L}}_{\mathrm{kd}}.\hfill \end{array}$$

(6b)

3.2.4. Limitations of Baseline

Experiments demonstrate (Section 4) that naive adaptations fail to address the complex C-GCD problem. Although self-distillation excels in static settings, it struggles in online sessions due to insufficient supervision. Without supervisory signals, the model exhibits bias toward dominant classes with higher sample frequencies during training. Consequently, known class samples are frequently misclassified as novel classes during testing. Furthermore, unsupervised clustering yields inaccurate prototypes characterized by reduced inter-class distances and increased intra-class variance. These issues lead to severe catastrophic forgetting, with static GCD methods sometimes failing to learn effectively in incremental scenarios.

3.3. Learning Instance-Specific Prompts for C-GCD

To overcome Base-CGCD’s limitations, we leverage the inherent discriminative power of textual modality to enhance visual representations. We adopt conditional prompt learning to address the challenge of inaccessible category information in unlabeled environments, and propose dual-modal contrastive learning across textual and visual–textual modality spaces.

3.3.1. Text-Modal Contrastive Learning

We first develop a pre-processing pipeline to encode image–label pairs within each mini-batch. Category information is structured into descriptions following the template “a photo of [class name], which is a type of [coarse class name]”, then encoded into feature vectors ${\widehat{z}}_i^{\mathrm{text}}$ via CLIP’s text encoder. These ${\widehat{z}}_i^{\mathrm{text}}$ serve as ground-truth supervision for training our prompt generation network.

We introduce a meta-module $\varphi$ to map visual features to textual representations. Following meta-learning principles, $\varphi$ learns to adaptively generate prompts from visual features. Specifically, given two augmented views $x_i$ and $x_i^{\prime }$ from ${\mathcal{D}}_l^t$, we extract visual representations $z_i^{\mathrm{visual}}$ through CLIP’s frozen vision encoder, then generate $z_i^{\mathrm{text}}=\varphi \left(z_i^{\mathrm{visual}}\right)$. To align generated prompts with the ground truth, we employ textual-modal contrastive learning:

$${\mathcal{L}}_{\mathrm{con}}=-{\displaystyle \frac{1}{|B^l|}}\sum _{i\in B^l}log{\displaystyle \frac{exp(z_i^{\mathrm{text}}·{\widehat{z}}_i^{\mathrm{text}})}{{\sum }_{j=1}^{|{\mathcal{Y}}^l|}{\mathbb{1}}_{[i\ne j]}exp(z_i^{\mathrm{text}}·{\widehat{z}}_j^{\mathrm{text}})}},$$

(7)

where ${\mathcal{Y}}^l$ indicates labeled data space and mean squared error loss:

$${\mathcal{L}}_{\mathrm{mse}}={\displaystyle \frac{1}{|B^l|}}\sum _{i\in B^l}{\parallel {\widehat{z}}_i^{\mathrm{text}}-z_i^{\mathrm{text}}\parallel }^2.$$

(8)

We combine both losses as ${\mathcal{L}}_{\mathrm{dist}}={\mathcal{L}}_{\mathrm{con}}+{\mathcal{L}}_{\mathrm{mse}}$ to optimize $\varphi$ and fine-tune the last layer of the text encoder $E^{\mathrm{text}}$, ensuring $z_i^{\mathrm{text}}$ approximates the ground-truth prompts ${\widehat{z}}_i^{\mathrm{text}}$.

3.3.2. Visual–Text Cross-Modal Contrastive Learning

Subsequently, $z_i^{\mathrm{visual}}$ is processed by $\varphi$ to synthesize corresponding tokens, which are then encoded by the trainable text encoder $E^{\mathrm{text}}$ to generate textual representations $z_i^{\mathrm{text}}$. To ensure alignment between textual features and their corresponding images, we establish positive pairs by matching visual features $z_i^{\mathrm{visual}}$ from the first augmented view with pseudo-textual features $z_i^{\prime \mathrm{text}}$ from the second view, and vice versa. This bidirectional pairing strategy strengthens cross-modal alignment and enhances generalization capacity. The cross-modal contrastive losses are formulated as follows:

$$\begin{array}{cc}\hfill {\mathcal{L}}_{\mathrm{align}}^{v\to t}& =-{\displaystyle \frac{1}{\left|B\right|}}\sum _{i\in B}log{\displaystyle \frac{exp(z_i^{\mathrm{visual}}·z_i^{\prime \mathrm{text}}/{\tau }_{\alpha })}{{\sum }_{j\in B}exp(z_i^{\mathrm{visual}}·z_j^{\prime \mathrm{text}}/{\tau }_{\alpha })}},\hfill \end{array}$$

(9a)

$$\begin{array}{cc}\hfill {\mathcal{L}}_{\mathrm{align}}^{t\to v}& =-{\displaystyle \frac{1}{\left|B\right|}}\sum _{i\in B}log{\displaystyle \frac{exp(z_i^{\mathrm{text}}·z_i^{\prime \mathrm{visual}}/{\tau }_{\alpha })}{{\sum }_{j\in B}exp(z_i^{\mathrm{text}}·z_j^{\prime \mathrm{visual}}/{\tau }_{\alpha })}}.\hfill \end{array}$$

(9b)

The overall cross-modal alignment loss is ${\mathcal{L}}_{\mathrm{align}}={\mathcal{L}}_{\mathrm{align}}^{v\to t}+{\mathcal{L}}_{\mathrm{align}}^{t\to v}$. Finally, we replace the single-modality encoder in Base-CGCD with our trained multimodal encoder to extract both textual and visual features. We perform cross-attention fusion on these modalities to dynamically balance their contributions:

$$z_i^{\mathrm{enhanced}}=\mathrm{AttentionFusion}(z_i^{\mathrm{visual}},z_i^{\mathrm{text}}).$$

(10)

The enhanced features $z_i^{\mathrm{enhanced}}$ are then fed into the self-distillation framework to generate high-quality pseudo-labels for classification.

4. Experiments

4.1. Experiments Detail

Datasets

We evaluate DMCL on three generic datasets: CIFAR-100 [35], Tiny-ImageNet [36], and ImageNet-100 [37] as well as the fine-grained dataset CUB-200 [38]. Each dataset is partitioned into an offline session and multiple online sessions (3 sessions by default). In the offline session, 70% of classes serve as the initial labeled data for supervised learning. In subsequent online sessions, the remaining classes are uniformly distributed as novel classes across sessions, with each session containing unlabeled data from both newly introduced novel classes and all previously learned classes. Table 1 provides detailed dataset statistics, where “Img/Novel” and “Img/Known” denote the number of samples per novel class and known class at each session, respectively.

Table 1. Dataset configurations for Continual Category Discovery.

Dataset	Offline Session		Online Session
	Old	Img/Old	Novel	Img/Novel	Img/Known
CIFAR-100	70	400	10	400	25
ImageNet-100	70	∼1000	10	1000	60
Tiny-ImageNet	140	400	20	400	25
CUB-200	140	∼25	20	25	5

4.2. Evaluation Protocol

At each session, after training, the model is evaluated on a disjoint test set ${\mathcal{D}}_{\mathrm{test}}^t$ containing samples from both ${\mathcal{C}}_{\mathrm{novel}}^t$ and ${\mathcal{C}}_{\mathrm{known}}^t$. Following standard practice, we compute clustering accuracy (ACC) using the Hungarian algorithm [39] to find the optimal assignment between predicted cluster labels ${\widehat{y}}_i$ and ground-truth labels $y_i$:

$$\mathrm{ACC}={\displaystyle \frac{1}{M}}\sum _{i=1}^M\mathbb{1}(y_i=p\left({\widehat{y}}_i\right)),$$

(11)

where $M=|{\mathcal{D}}_{\mathrm{test}}^t|$ and p denotes the optimal permutation. We report accuracy across three subsets: “All” (all classes ${\mathcal{C}}_{\mathrm{known}}^t\cup {\mathcal{C}}_{\mathrm{novel}}^t$), “Known” (previously learned classes ${\mathcal{C}}_{\mathrm{known}}^t$), and “Novel” (newly discovered classes ${\mathcal{C}}_{\mathrm{novel}}^t$).

Implementation

In all experiments, we employ ViT-B/16 pretrained with DINO v1 [13] as the initial backbone and fine-tune only the last transformer block. Our training proceeds in two stages:

Stage 1: Prompt Generation Network Training. During the offline session (Session 0), we train the prompt generation network $\varphi$ on all labeled data ${\mathcal{D}}_l^0$ for 100 epochs. In this stage, we freeze the CLIP vision encoder and fine-tune only the last layer of the CLIP text encoder. This allows the meta-module $\varphi$ to learn effective mappings from visual features to textual representations while preserving the pretrained vision-language alignment.

Stage 2: Continual Discovery with Dual-Modal Learning. After training, we replace the DINO backbone in Base-CGCD with the trained prompt generation network $\varphi$ and CLIP encoders. At each online session (Sessions 1 to T), we train on the unlabeled data ${\mathcal{D}}_u^t$ containing both known and novel classes for 30 epochs with a batch size of 128 and a learning rate of 0.1. During this stage, we freeze the text encoder and fine-tune the vision encoder to adapt visual representations while maintaining stable textual features.

The Base-CGCD baseline follows the hyperparameter settings from [7,16]. All experiments are conducted on NVIDIA RTX A800 GPUs.

4.3. Results

4.3.1. Comparison with State-of-the-Art Methods

We compare DMCL against Base-CGCD and recent state-of-the-art C-GCD methods including GCD [3], SimGCD [16], GM [12], MetaGCD [40], PromptCCD [6], and Happy [7]. Table 2 presents comprehensive results across four datasets with three online sessions. DMCL achieves competitive or superior performance across all baselines in most scenarios.

Table 2. Results on CIFAR-100(C100), ImageNet-100(IN100), Tiny-ImageNet(TinyIN), CUB200(CUB) with 3 online sessions. Bold indicates the best performance.

Datasets	Methods	Session-1			Session-2			Session-3
		All	Known	Novel	All	Known	Novel	All	Known	Novel
C100	GCD [3]	55.68	80.12	38.57	45.16	67.62	40.87	48.96	71.71	44.98
	SimGCD [16]	48.84	74.16	31.11	35.28	61.43	30.29	36.28	64.57	31.33
	GM [12]	64.77	70.49	60.77	58.32	62.95	57.44	49.21	57.62	47.73
	MetaGCD [40]	56.20	79.59	39.83	56.63	65.81	55.05	53.65	62.76	52.05
	PromptCCD [6]	70.69	80.90	63.54	64.08	73.14	62.35	57.73	72.67	55.12
	Happy [7]	78.95	82.61	53.30	76.73	78.44	63.10	72.32	73.27	63.80
	DMCL (Ours)	83.81	87.68	65.94	78.26	79.32	77.26	67.01	66.67	66.84
IN100	GCD [3]	79.46	84.78	75.74	72.64	81.81	70.89	64.01	84.67	60.40
	SimGCD [16]	48.76	77.14	31.71	48.55	67.14	43.82	45.27	59.29	42.00
	GM [12]	75.34	76.78	74.34	63.76	73.67	62.87	64.43	74.86	62.60
	MetaGCD [40]	65.61	83.92	52.80	67.36	83.14	64.35	66.26	74.57	64.80
	PromptCCD [6]	79.56	84.24	76.29	78.58	79.71	78.36	70.33	81.33	68.40
	Happy [7]	82.43	84.52	80.10	78.04	82.61	77.24	75.20	80.21	73.24
	DMCL (Ours)	83.26	85.24	80.61	81.22	81.02	79.28	78.01	83.26	75.36
TinyIN	GCD [3]	66.40	76.06	59.64	50.28	66.38	47.21	52.32	60.19	50.94
	SimGCD [16]	46.04	65.22	32.61	34.21	56.00	30.05	32.11	53.24	28.42
	GM [12]	59.52	64.24	56.21	51.50	58.19	50.23	45.39	56.62	43.43
	MetaGCD [40]	59.41	73.90	49.27	57.21	63.71	55.96	49.17	60.76	47.14
	PromptCCD [6]	68.67	72.84	65.76	59.69	65.67	58.55	57.16	61.10	56.47
	Happy [7]	77.82	82.43	66.26	72.35	74.23	43.52	66.22	65.88	36.55
	DMCL (Ours)	75.26	78.34	73.42	74.33	76.33	58.43	68.88	71.88	62.24
CUB	GCD [3]	59.66	77.50	47.73	50.12	73.57	45.15	54.20	72.86	50.56
	SimGCD [16]	49.70	75.75	32.60	37.99	69.34	31.38	38.56	72.86	31.78
	GM [12]	44.21	65.00	30.31	29.50	65.00	21.97	42.89	65.00	38.58
	MetaGCD [40]	50.93	71.07	37.47	39.37	74.29	31.97	43.47	77.86	36.77
	PromptCCD [6]	57.08	75.00	45.11	47.38	75.00	41.52	61.89	76.43	59.05
	Happy	82.03	84.79	65.03	75.24	76.70	64.22	67.72	72.03	40.16
	DMCL (Ours)	83.42	82.18	78.82	79.01	74.62	80.22	71.22	74.64	54.12

Novel Class Discovery. DMCL demonstrates strong plasticity in discovering novel classes. On CIFAR-100, DMCL achieves 67.00% novel class accuracy at Session 3, substantially outperforming Happy at 63.80% and PromptCCD at 55.12%. On ImageNet-100, DMCL attains 74.32% at Session 3, surpassing Happy at 73.24% and significantly ahead of other methods. Notably, on CUB, DMCL achieves remarkable improvements, reaching 77.82%, 79.34%, and 48.00% novel class accuracy across three sessions, compared to Happy’s results of 65.03%, 64.22%, and 40.16%. These results demonstrate that dual-modal contrastive learning effectively enriches representations and enhances discriminability for novel category discovery.

Knowledge Retention. For previously learned classes, DMCL maintains competitive accuracy while balancing plasticity. On CIFAR-100, DMCL achieves 87.08% and 78.79% known class accuracy at Sessions 1 and 2, outperforming most baselines. On ImageNet-100, DMCL attains 86.14% and 82.72% at Sessions 1 and 2, comparable to Happy’s 84.52% and 82.61%. This validates that our approach effectively mitigates catastrophic forgetting through adaptive fusion of visual and textual modalities.

Fine-Grained Recognition. On the challenging fine-grained CUB dataset, DMCL substantially outperforms all methods across sessions. At Sessions 1 and 2, DMCL achieves overall accuracies of 82.93% and 77.98%, surpassing Happy’s 82.03% and 75.24% and significantly ahead of other methods. This highlights the benefit of incorporating textual semantics for fine-grained visual discrimination, where subtle inter-class differences require richer multimodal representations.

4.3.2. Forgetting and Discovery Analysis

To comprehensively evaluate the model’s discovery capability and resistance to catastrophic forgetting, we analyze two key metrics: the maximum forgetting rate ${\mathcal{M}}_f$ and the final discovery rate ${\mathcal{M}}_d$. The metric ${\mathcal{M}}_f$ measures the capability to maintain performance on old categories, where lower values indicate better retention. The metric ${\mathcal{M}}_d$ measures the ability to discover novel categories, where higher values indicate better discovery performance.

Both metrics are accuracy-based. The clustering accuracy on old categories ${\mathrm{ACC}}_{\mathrm{old}}^t$ and novel categories ${\mathrm{ACC}}_{\mathrm{novel}}^t$ at session t can be computed as follows:

$$\begin{array}{cc}\hfill & {\mathrm{ACC}}_{\mathrm{old}}^t={\displaystyle \frac{1}{M_{\mathrm{old}}^t}}\sum _{i=1}^{M_{\mathrm{old}}^t}\mathbb{1}(y_{i,\mathrm{old}}=p\left({\widehat{y}}_i\right)),\hfill \end{array}$$

(12a)

$$\begin{array}{cc}\hfill & {\mathrm{ACC}}_{\mathrm{novel}}^t={\displaystyle \frac{1}{M_{\mathrm{novel}}^t}}\sum _{i=1}^{M_{\mathrm{novel}}^t}\mathbb{1}(y_{i,\mathrm{novel}}=p\left({\widehat{y}}_i\right)),\hfill \end{array}$$

(12b)

where $M_{\mathrm{old}}^t$ and $M_{\mathrm{novel}}^t$ denote the number of old and novel category samples from ${\mathcal{D}}_{\mathrm{test}}^t$, and $y_{i,\mathrm{old}}$ and $y_{i,\mathrm{novel}}$ denote the ground-truth labels of old and novel samples, respectively. The forgetting and discovery metrics are then defined as ${\mathcal{M}}_f={max}_t\{{\mathrm{ACC}}_{\mathrm{old}}^0-{\mathrm{ACC}}_{\mathrm{old}}^t\}$ and ${\mathcal{M}}_d={\displaystyle \frac{1}{T}}{\sum }_{t=1}^T{\mathrm{ACC}}_{\mathrm{novel}}^t$.

Table 3 presents the forgetting and discovery analysis on CUB. Compared to the state-of-the-art method Happy, DMCL demonstrates substantial improvements on both metrics. Specifically, DMCL reduces the forgetting rate ${\mathcal{M}}_f$ from 4.18% to 3.79%, achieving the lowest forgetting among all methods. Meanwhile, DMCL significantly improves the discovery rate ${\mathcal{M}}_d$ from 53.79% to 71.05%, demonstrating enhanced plasticity for novel class learning. In terms of overall performance, DMCL attains an average accuracy of 77.90%, surpassing Happy at 77.88% and PromptCCD at 55.45%.

Table 3. Forgetting and Discovery Analysis on CUB. (↑ indicates that the higher the metric, the better, and ↓ indicates that the lower the metric, the better). Bold indicates the best performance.

Methods	CUB
	${\mathcal{M}}_{\mathit{f}}\mathbf{\downarrow }$	${\mathcal{M}}_{\mathit{d}}\mathbf{\uparrow }$	Avg(“All”) ↑
GCD [3]	9.57	56.01	54.66
GM [12]	23.24	27.63	38.87
PromptCCD [6]	6.07	74.46	55.45
Happy [7]	4.18	53.79	74.99
Ours	3.79	71.05	77.88

These results validate that dual-modal contrastive learning with adaptive fusion effectively addresses the plasticity–stability dilemma in continual category discovery. The superior performance on the fine-grained CUB dataset particularly highlights the benefit of incorporating textual semantics, where subtle inter-class differences require richer multimodal representations to maintain discriminative decision boundaries across sessions.

4.3.3. Visualization

We provide qualitative analysis to intuitively compare DMCL with the baseline framework. Figure 4 presents confusion matrices on CIFAR-100 across three online sessions. DMCL produces clearer diagonal patterns compared to Happy, indicating improved classification accuracy and reduced confusion between classes. Notably, the off-diagonal elements in DMCL are significantly sparser, demonstrating that the model effectively mitigates task-level overfitting. Specifically, DMCL exhibits substantially fewer known class samples misclassified as the novel classes introduced in the current session, validating the effectiveness of dual-modal learning in maintaining stable decision boundaries.

Figure 4. Confusion matrices on CIFAR-100. DMCL produces clearer diagonal patterns compared to Base-CGCD, indicating improved classification accuracy and reduced confusion. Moreover, DMCL effectively mitigates task-level overfitting, with significantly fewer known class samples misclassified as the 10 novel classes in the current session.

Figure 5 visualizes feature distributions using t-SNE across three sessions. Compared to Happy, DMCL achieves better class separation with reduced overlap among semantic clusters. In Session 1, DMCL shows more compact and well-separated clusters for both known and novel classes. As sessions progress, while both methods experience increased inter-class overlap due to the accumulation of classes, DMCL consistently maintains clearer cluster boundaries. This demonstrates that incorporating textual semantics through dual-modal contrastive learning enriches feature representations and enhances discriminability, thereby alleviating semantic drift across continual learning sessions.

Figure 5. t-SNE visualization of feature distributions. DMCL (bottom) achieves better class separation with reduced overlap compared to Base-CGCD (top).

4.4. Ablation Study

We conduct comprehensive ablation studies to evaluate the effectiveness of each component in DMCL. Table 4 presents results on CIFAR-100 with different configurations of dual-modal contrastive learning and fusion strategies.

Table 4. Ablation Study on DMCL Components and Multimodal Fusion Strategies. Bold indicates the best performance.

Dual-Modal CL.	Fusion				CIFAR100
	Adaptive	Average	Concate	Cross-Attention	All	Old	New
✗	✗	✗	✗	✗	45.43	72.26	36.20
✓	✓	✗	✗	✗	75.22	77.18	69.00
✓	✗	✓	✗	✗	74.89	76.24	70.23
✓	✗	✗	✓	✗	70.22	74.46	52.24
✓	✗	✗	✗	✓	76.36	77.89	70.01

The ✓ and ✗ symbols denote the presence and absence of Dual-modal Contrastive Learning, respectively.

Effectiveness of Dual-Modal Contrastive Learning. Comparing the baseline without dual-modal contrastive learning (row 1) to the full model with adaptive fusion (row 2), we observe substantial improvements: 29.79% gain in overall accuracy, 4.92% in known classes, and 32.80% in novel classes. This validates that incorporating textual semantics significantly enhances feature discriminability and mitigates catastrophic forgetting.

Fusion Strategy Analysis. We compare four fusion strategies: adaptive, average, concatenation, and cross-attention. Cross-attention fusion achieves the best overall performance at 76.36%, demonstrating the superiority of instance-specific dynamic weighting over fixed parameter approaches. Notably, cross-attention excels particularly on known classes (77.89% vs. 77.18% for adaptive), validating our theoretical analysis that sample-aware fusion better mitigates catastrophic forgetting. The mechanism allows old class samples (prone to forgetting) to emphasize stable textual anchors, while novel class samples leverage adaptive visual features for discovery.

Adaptive fusion achieves competitive performance at 75.22%. Examining the learned fusion coefficient $\gamma \approx 0.498$ reveals that visual and textual modalities contribute nearly equally to enhanced representations. This near-equal weighting provides a strong baseline but lacks the flexibility to handle samples with different plasticity–stability requirements.

Average fusion performs comparably at 74.89%, only 0.33% lower than adaptive fusion. This validates our finding that $\gamma \approx 0.5$ is optimal—when the learned parameter approaches 0.5, average fusion becomes effectively equivalent, offering a simpler alternative without learnable parameters.

Concatenation-based fusion achieves only 70.22% overall accuracy, substantially underperforming other strategies. This degradation stems from increased dimensionality and potential misalignment between visual and textual feature spaces, which complicates discriminative decision boundary learning. The particularly poor novel class performance (52.24%) suggests concatenation fails to effectively integrate complementary modality information.

Prompt Setting Analysis. Table 5 presents ablation experiments validating our design choices on CIFAR-100. The most critical component is dual-modal contrastive learning. Removing it (visual-only) causes performance to drop from 76.36% to 51.67%, a substantial 24.69% degradation. Novel class accuracy is particularly affected (27.43% drop), confirming that textual semantics are essential for distinguishing newly emerged categories.

Table 5. Ablation Study on CIFAR100 with prompt configure.

Config.	Text Type	Prompt Type	Dual-Modal CL	All	Known	Novel
Oracle	Ground-truth	Fixed	✓	82.42	84.87	76.24
Baseline	-	-	✗	45.43	72.26	36.20
w/o Cond.	Pseudo	Fixed	✓	63.78	69.45	47.32
w/o DM-CL	Pseudo	Conditional	✗	51.67	69.13	42.58
Ours	Pseudo	Conditional	✓	76.36	77.89	70.01

Bold values indicate the best performance. The ✓ and ✗ symbols denote the presence and absence of Dual-modal Contrastive Learning, respectively.

Conditional prompt generation outperforms fixed learnable prompts by 12.58% (76.36% vs. 63.78%). Instance-specific prompts capture fine-grained variations within categories, while fixed prompts apply uniformly and struggle to generalize, especially for novel classes where the gap reaches 22.69%. Our pseudo-prompt approach achieves 76.36%, reasonably close to the oracle using ground-truth text (82.42%). The 6.06% gap demonstrates that conditional generation effectively synthesizes discriminative descriptions without requiring manual annotations.

These results confirm that textual information and instance-specific generation are both essential for effective continual category discovery.

4.5. Novel Class Number Estimation

Previous experiments assume the number of novel classes is known a priori for fair comparison. However, automatic class number estimation is crucial for practical applications. Table 6 presents results under the class-agnostic setting on CIFAR-100, where we employ a hybrid estimation strategy combining Silhouette Score and Elbow Method to automatically determine novel class numbers. PromptCCD adopts a non-parametric classifier, which has inherent advantages in class-agnostic scenarios due to flexible output dimensionality. In contrast, Happy and our method employ parametric classifiers, requiring additional estimation mechanisms before training.

Table 6. Performancewithout knowing the number of novel classes. DMCL achieves competitive results in the class-agnostic setting.

Method	C100
	All	Known	Novel
GM	53.33	66.64	49.61
PromptCCD	59.12	77.62	53.70
Happy	66.80	70.56	48.84
Ours	67.20	72.00	51.26

Bold values indicate the best performance.

Despite this structural disadvantage, our method achieves the best overall accuracy at 67.20%, surpassing PromptCCD at 59.12%, Happy at 66.80%, and GM at 53.33%. For individual metrics, PromptCCD leads on both known classes at 77.62% and novel classes at 53.70%. Our method ranks second on both categories: 72.00% for known classes and 51.26% for novel classes, outperforming Happy at 70.56% and 48.84%, respectively.

These results validate that our dual-modal learning framework maintains robustness when class numbers are estimated rather than known, highlighting practical applicability in real-world continual learning scenarios.

4.6. Complexity Analysis

Table 7 presents complexity and efficiency metrics on CIFAR-100. Our model requires 92.0 M parameters, 16.87 GFLOPs per image, and 3.77 GB peak memory. Compared to visual-only baselines, the dual-modal components add approximately 6.2 M parameters (7.2% increase) and 0.67 GFLOPs (4.1% increase). This modest overhead yields substantial performance gains: 76.36% accuracy versus 65.89% for the baseline (10.47% improvement).

Table 7. Model Complexity and Efficiency Analysis on CIFAR-100.

Metric	Value
Model Complexity
Parameters	92.0 M
FLOPs per Image	16.87 G
Peak Memory Usage	3.77 GB
Inference Efficiency
Time per Sample	1.01 ms
Throughput (FPS)	986.94
Batch Time (128 samples)	253.31 ms
GOPS (Throughput)	16.65 TFLOPS
Training Cost
Time per Epoch	∼57 s
Novel Class Clustering	2–3 s

Inference efficiency remains strong at 1.01 ms per sample, achieving 986.94 FPS throughput and 16.65 TFLOPS. Training takes approximately 57 s per epoch, with novel class clustering requiring only 2–3 s per session. The pseudo-prompt generation adds negligible latency (0.05 ms per sample).

These results demonstrate a favorable trade-off: less than 5% computational overhead achieves over 10% accuracy improvement, validating our approach’s efficiency and practical applicability.

5. Discussion

5.1. Why Dual-Modal Contrastive Learning Works?

Existing works primarily rely on single visual modality for continual category discovery, overlooking the natural advantage of textual information in distinguishing novel classes under unlabeled settings. CLIP’s ability to map textual and visual features into a unified latent space provides the foundation for our dual-modal contrastive learning. Specifically, we introduce a meta-network to enable CLIP to handle unlabeled data, leveraging its inherent text-vision alignment capability to learn image-to-text mappings for generating descriptions of unknown images.

5.2. How Does Textual Features Mitigate Catastrophic Forgetting?

Textual features, in the form of pseudo-descriptions generated from corresponding images, provide semantic information that compensates for the lack of supervised signals in unsupervised continual learning. By incorporating these textual semantics, samples acquire enhanced discriminability, which alleviates semantic drift and consequently mitigates task-level overfitting.

Furthermore, as each sample obtains richer discriminative information through dual-modal representations, semantic clusters form more distinct and well-defined prototypes. During prototype initialization at each session, these high-quality prototypes derived from better-separated clusters lead to improved initial decision boundaries. This creates a virtuous cycle: better prototypes yield more accurate pseudo-labels, which in turn facilitate more reliable representation learning and prototype updates in subsequent sessions. Consequently, the model maintains stable decision boundaries for previously learned classes while adapting to novel categories, effectively balancing plasticity and stability.

5.3. How Textual Information Mitigates the Plasticity–Stability Dilemma?

Textual semantics serve dual roles: semantic anchors for stability and discriminative features for plasticity. Our adaptive fusion $\gamma \approx 0.5$ balances these uniformly, but different samples have contradictory needs. Old class samples require stability through higher text weight, while novel samples need plasticity through higher visual weight.

Cross-attention addresses this through instance-specific weighting. The bidirectional querying mechanism automatically adjusts modality emphasis: old samples attend more to stable text, novel samples to adaptive vision. This sample-aware balancing reduces forgetting while improving accuracy on both known and novel classes.

Fixed parameter fusion cannot simultaneously optimize for contradictory requirements. Cross-attention’s dynamic per-sample adjustment achieves superior plasticity–stability trade-offs, explaining its performance gains in our experiments.

6. Conclusions

This paper addresses Continual Generalized Category Discovery (C-GCD), an emerging challenge in open-world learning where models must incrementally discover novel classes from unlabeled data while preserving recognition of previously learned classes without accessing historical samples. We identify that the absence of supervisory signals in online sessions leads to two critical issues: semantic drift causing task-level overfitting, and degraded decision boundaries resulting from poor cluster quality.

To address these challenges, we propose DMCL, a dual-modal contrastive learning framework that leverages CLIP’s multimodal potential through a two-stage training paradigm. In the first stage, we introduce a prompt generation network with conditional prompt learning to synthesize textual descriptions for unlabeled samples. In the second stage, we enhance Base-CGCD with dual-modal contrastive learning and adaptive fusion, generating high-quality pseudo-labels through multimodal self-distillation. By incorporating textual semantics, our approach enriches sample representations and enhances discriminability, effectively mitigating catastrophic forgetting while improving novel class discovery.

Extensive experiments on CIFAR-100, Tiny-ImageNet, ImageNet-100, and CUB-200 demonstrate that DMCL achieves competitive or superior performance compared to state-of-the-art methods. Particularly on fine-grained datasets, our method shows substantial improvements, validating the effectiveness of multimodal learning for continual category discovery. Comprehensive ablation studies and qualitative analyses confirm that dual-modal contrastive learning successfully addresses task-level overfitting and decision boundary degradation, achieving superior plasticity–stability trade-offs.

7. Future Work

While DMCL demonstrates the effectiveness of dual-modal learning for C-GCD, several promising directions merit further investigation. First, our prompt generation currently relies on fixed templates. Future work could explore learnable prompt structures that adapt to different visual domains or leverage large language models to generate richer semantic descriptions. Second, extending our global adaptive fusion to hierarchical or attention-based mechanisms could enable fine-grained integration where modality contributions vary spatially or semantically. Third, developing robust methods for estimating novel class numbers in dual-modal settings would enhance practical applicability, potentially through multimodal clustering consistency or semantic gap analysis in the embedding space. Beyond methodological improvements, extending C-GCD to more challenging scenarios such as object detection, semantic segmentation, or domain adaptation would demonstrate broader applicability. Additionally, theoretical analysis of how textual semantics contribute to plasticity–stability trade-offs could provide principled guidance for fusion strategy design.

We hope this work provides valuable insights and inspires future research in multimodal continual learning.