HISF: Hierarchical Interactive Semantic Fusion for Multimodal Prompt Learning

Abstract

Recent vision-language pre-training models, like CLIP, have been shown to generalize well across a variety of multitask modalities. Nonetheless, their generalization for downstream tasks is limited. As a lightweight adaptation approach, prompt learning could allow task transfer by optimizing only several learnable vectors and thus is more flexible for pre-trained models. However, current methods mainly concentrate on the design of unimodal prompts and ignore effective means for multimodal semantic fusion and label alignment, which limits their representation power. To tackle these problems, this paper designs a Hierarchical Interactive Semantic Fusion (HISF) framework for multimodal prompt learning. On top of frozen CLIP backbones, HISF injects visual and textual signals simultaneously in intermediate layers of a Transformer through a cross-attention mechanism as well as fitting category embeddings. This architecture realizes the hierarchical semantic fusion at the modality level with structural consistency kept at each layer. In addition, a Label Embedding Constraint and a Semantic Alignment Loss are proposed to promote category consistency while alleviating semantic drift in training. Extensive experiments across 11 few-shot image classification benchmarks show that HISF improves the average accuracy by around 0.7% compared to state-of-the-art methods and has remarkable robustness in cross-domain transfer tasks. Ablation studies also verify the effectiveness of each proposed part and their combination: hierarchical structure, cross-modal attention, and semantic alignment collaborate to enrich representational capacity. In conclusion, the proposed HISF is a new hierarchical view for multimodal prompt learning and provides a more lightweight and generalizable paradigm for adapting vision-language pre-trained models.

1. Introduction

Massive vision-language pre-trained models like CLIP [1] have recently made great progress in multimodal understanding. Trained from hundreds of millions of image-text pairs [2], these models learn to align visual and textual information into a shared semantics space and thus provide exciting zero-shot generalization performance across many different recognition tasks. Large-scale models need to be adapted further downstream, and fine-tuning such large models is expensive and data-consuming, which implies the exploration of parameter-efficient adaptation techniques.

Swift learning has recently become a popular option for knowledge transfer while preserving small differences in the model, which allows only limited parameter updating [3]. With the addition of a few learnable prompt vectors, these methods condition the frozen backbone to execute new tasks without changing its base structure. Previous methods, such as CoOp [4] (Context Optimization) and Co-CoOp [5] (Conditional CoOp), have worked successfully by leveraging task-specific textual prompts. However, these methods are bound to single-modality optimization, and they often do not generalize well to unseen categories or cross-domain scenarios due to the insufficient multi-level multimodal interaction and semantic alignment.

To overcome these limitations, the most recent studies have extended prompt learning to a multimodal space. Representative methods MaPLe [6] and LAMM [7] have provided visual as well as text prompts, which led to better semantical correspondence between the modalities. However, these methods still suffer from shallow semantic fusion–prompts are injected at most to the input level, and their cross-modal associations are only partially captured in the Transformer framework [8,9,10]. Accordingly, the method has difficulty in modeling hierarchical semantic relations and deep (inter)action between visual and textual representations.

In this paper, we present a Hierarchical Interactive Semantic Fusion (HISF) framework for fine-grained hierarchical multimodal prompt learning. Based on a frozen CLIP backbone, HISF extends it with a dual-branch prompt structure that enables the joint optimization of visual and textual prompts across multiple Transformer layers using cross-attention to fuse category embeddings and prompt tokens. This architecture can allow for the holistic semantic representation to be effectively propagated across the network in a top-down fashion, improving both intra-class correlation and inter-class discrimination of feature maps. There is also a Label Embedding Constraint and Semantic Alignment Loss to project the label embedding space into the representations learned from the prompt while training to prevent semantic drift as well as facilitating cross-domain generalization.

The main contributions of this work are summarized as follows:

• Hierarchical Semantic Fusion Framework:

We propose a novel hierarchical prompt learning paradigm that injects multimodal prompts into multiple Transformer layers, allowing progressive semantic interaction between visual and textual modalities.

2.

Label-Guided Cross-Attention Mechanism:

A category embedding-guided cross-attention module is designed to dynamically align label semantics with multimodal prompts, achieving deep semantic binding between modalities.

3.

Semantic Alignment and Label Constraints:

We introduce a joint optimization scheme consisting of a Label Embedding Constraint and a Semantic Alignment Loss to maintain consistent semantic distribution between the label and prompt spaces.

4.

Parameter-Efficient and Robust Adaptation:

HISF requires updating only a small number of parameters while achieving superior performance on few-shot and cross-domain benchmarks, demonstrating both efficiency and generalization.

Extensive experiments on 11 benchmark datasets confirm that HISF consistently outperforms existing prompt learning methods, achieving approximately 0.7% improvement in average accuracy and stronger robustness under domain shifts. These results verify that hierarchical semantic fusion and cross-modal interaction are crucial for efficient and generalizable multimodal adaptation.

2. Related Work

2.1. Vision-Language Pre-Training

Recent progress has been made in this direction with the introduction of large-scale vision-language pre-training (VLP) [1,9,10,11,12,13], such as OSCAR [14], BLIP-2 [15], and LLaVA [16], that leads to notable improvements for multimodal grounding and transfer learning efficiency through object-semantic alignment and vision-language bootstrapping. In particular, bottom-up and top-down attention mechanisms [17] play a crucial role in visual reasoning tasks such as image captioning and VQA, providing fine-grained contextual grounding for subsequent multimodal adaptation. In early works, including CLIP [1] (Contrastive Language-Image Pre-training), ALIGN [18], and LiT [19], large-scale image-text pairs are used to jointly train visual and text encoders contrastively. The goal is to bring similar image-text pairs closer together in a shared semantic space and dissimilar pairs further apart, matching disparate modalities. These models have shown strong zero-shot generalization capacity, which can directly perform open-vocabulary recognition [1] based on the input without explicit fine-tuning.

However, those models are highly dependent on large-scale data [11] and computational capabilities that make them unsuitable in domain-specific or low-data settings. Fine-tuning these models on downstream tasks directly is inefficient and often leads to overfitting. As a result, parameter-efficient adaptation methods (in particular prompt learning) have received significant attention [20], which allow adapting to a new task by updating only a small set of additional parameters while keeping the pre-trained backbone fixed.

2.2. Prompt Learning

Prompt learning originates from natural language processing (NLP), where learnable textual tokens are inserted into the input sequence to guide the model toward specific tasks. Methods such as PET, P-Tuning, and Prefix-Tuning have proven that optimizing only a few prompt vectors can achieve performance comparable to full fine-tuning, dramatically reducing computational cost.

In the vision-language domain, this idea was first introduced to CLIP by CoOp [4] (Context Optimization), where hand-crafted textual templates were replaced by learnable contextual vectors (CoOp). As a result, this approach was able to achieve an effective enhancement of task-specific adaptation with efficiency. Nevertheless, CoOp’s learned prompts generalize poorly to new categories, as the optimization is only guided by base-class semantics. In order to address this limitation, Co-CoOp [5] (Conditional CoOp) introduced an image-conditioned prompt generator so that the learned prompts would be able to adjust on a per-sample basis. Subsequent works (e.g., PromptStyler [21], ProDA [22], and DualCoOp [23]) further explored distributional or style-aware prompt tuning. Other approaches, such as TextRefiner [24], CPT [25], and SPGFF [26], focused on fine-grained textual refinement and color-based grounding to enhance visual-text correspondence. Visual Prompting [27] further revealed that lightweight visual perturbations can serve as effective task prompts for frozen models, offering a complementary direction to textual prompt learning. More recently, CoPrompt [28] introduced a consistency-guided regularization strategy to improve generalization under few-shot conditions.

Despite these advances, most prompt learning methods focus exclusively on the textual modality. The visual encoder remains unchanged, and cross-modal semantic interactions are largely ignored. As a result, the learned prompts capture shallow contextual semantics but fail to fully exploit the complementary relationship between vision and language.

2.3. Multimodal Prompt Learning

To address the limitations of single-modality prompt learning, multimodal prompt learning (MMPL) [29] has emerged as a new paradigm that incorporates learnable prompts into both visual and textual branches. This design enables deeper semantic fusion and more balanced representation learning between modalities.

MaPLe (Multimodal Prompt Learning) [6] injected prompts into intermediate Transformer layers from both visual and textual encoders and captured them via hierarchical semantic connections. It proposed a single initialization sequence and modality-specific projections that guarantee consistency (between joint modalities) while maintaining flexibility between them. LAMM (Label Alignment for Multimodal Prompt Learning) [7] further augmented this pipeline with label embeddings and introduced a label-prompt alignment loss that explicitly aligns class semantics to the multimodal prompt. These techniques showed that alignment of semantics across modalities and class labels is a critical factor to enhance generalization in few-shot and cross-domain tasks.

Subsequent studies, such as MMRL [8], DualPrompt [30], and BiMMPL [31], have further explored multimodal fusion strategies, including bidirectional projection between visual and textual prompts. Broadly, deep multimodal fusion strategies have been categorized into encoder–decoder, attention mechanism, and graph-based methods, reflecting a shift towards more intricate interaction [32]. Other efficient fusion frameworks, such as PMF [33] and MMA [34], focus on improving cross-modal communication and balancing generalization with discriminability across Transformer layers. In addition, dynamic and efficient architectures, such as PCETL [35], demonstrate the necessity of balancing parameter efficiency and computational cost during model adaptation, highlighting the scalability issues in multimodal fusion. In addition, Multimodal Alignment Prompt (MmAP) [36] extends this idea to a multi-task setting, aligning visual and textual modalities of CLIP through shared and task-specific prompts. Furthermore, question-driven prompt generation [37] introduces a language-conditioned mechanism that utilizes large language models to enhance prompt informativeness and cross-modal alignment, offering new insights for integrating textual reasoning into visual prompt design. Building on these advances, MaPLe [6] and LAMM [7] established hierarchical multimodal prompting frameworks, which jointly align visual and textual semantics to enhance downstream adaptability.

2.4. Semantic Alignment and Label Embedding

Semantic alignment refers to the common objective across all multiple modalities learning tasks where visual and textual features correspond to matching semantics. Common prompt-instantiating methods based on contrastive objectives may generate prompts that drift semantically [33] (prompt embeddings become different from the true label semantics during training). To address this, recent methods have incorporated label embeddings as semantic anchors.

LAMM [7] introduced a label alignment mechanism to regularize the distribution of label embeddings and text prompts in one unified semantic space, leading to stabilized prompt optimization. Another branch of study proposed hierarchical semantic injection, e.g., DeepPrompt [29], where category embeddings are gradually integrated through Transformer layers. This not only helps the model capture high-level semantic abstraction but also fine-grained visual detail.

Building upon these ideas, our proposed HISF framework integrates label embeddings and cross-modal prompts through hierarchical attention layers. Similar hierarchical designs have also been discussed in recent works such as Hierarchical Prompt Tuning (HPT) [38], Learning with Enriched Inductive Biases (LwEIB) [39], and Slim Prompt-Averaged Consistency (SPAC) [40], which emphasize structured and consistent semantic learning across multiple levels of representation. Recent studies also emphasize the role of visual tokenization [41] and unsupervised prompt distillation [42] in strengthening semantic transfer, allowing models to better align class-level textual priors with image features. More recently, adaptive frameworks such as MetaPrompt [43] and ProVP [44] aimed to improve prompt robustness under domain shift by progressively refining category embeddings, providing complementary perspectives to our HISF approach.

By embedding category semantics into multiple levels of the Transformer, HISF achieves fine-grained semantic fusion and alleviates inconsistencies between visual and textual spaces.

2.5. Summary

In summary, vision-language pre-training provides a strong foundation for multimodal representation learning, while prompt learning offers a parameter-efficient path for task adaptation. Research has evolved from single-modal prompt optimization (e.g., CoOp, Co-CoOp) to multimodal prompt fusion (e.g., MaPLe, LAMM, MMRL), focusing increasingly on semantic alignment and hierarchical modeling.

Despite these advances, existing methods still face challenges in maintaining consistent cross-modal semantics throughout the Transformer architecture. Our proposed HISF framework addresses these limitations through hierarchical semantic fusion, label-guided cross-attention, and semantic alignment constraints, achieving more stable and generalized multimodal representation learning.

3. Method

In this section, we describe our proposed framework called Hierarchical Interactive Semantic Fusion (HISF) for multimodal prompt learning. HISF can improve the semantic consistency and hierarchical interaction between visual and textual modality in vision-language pre-trained models, such as CLIP. The framework is composed of three key modules including (1) Prompt Initialization to learn both visual and textual prompts, (2) semantic fusion to combine hierarchical semantics via cross-modal attention modulated between prompt tokens and category embeddings; (3) loss optimization by leveraging Label Embedding Constraint (LEC) and Semantic Alignment Loss (SAL) for semantic consistency between label space and prompt space.

The overall architecture of HISF is illustrated in Figure 1.

3.1. Overall Framework

Given an input image $I$ and its associated category label $y$, the aim of HISF is to learn a well-mapped multimodal representation so that the visual features can be well coordinated with their textual counterparts in a shared semantic space. Specifically, the visual encoder $f_v\left(·\right)$ and text encoder $f_t\left(·\right)$ of CLIP are frozen, then we introduce learnable prompts to guide the model’s adaptation. The encoded image and text features can be denoted as:

$$z_v=f_v\left(I,P_v\right), z_t=f_t\left(T,P_t\right)$$

(1)

where $P_v$ and $P_t$ denote the visual and textual prompts, respectively, and $T$ denotes the tokenized text corresponding to category $y$. The HISF framework introduces hierarchical semantic fusion across the Transformer layers of both encoders. Category embeddings are injected via cross-attention modules to enhance inter-modal communication, ensuring consistent semantic representation across hierarchical levels. The encoders progressively integrate prompt-conditioned information, yielding the final modality-specific representations $z_v$ and $z_t$, which encapsulate the fused visual and textual semantics for subsequent alignment and prediction modules.

3.2. Prompt Initialization

The Prompt Initialization module serves as the foundation for efficient adaptation. Instead of relying on hand-crafted textual templates (e.g., “a photo of a [class]”), we employ learnable context tokens initialized from a shared semantic space. These learnable tokens are divided into two branches:

Textual Prompts ($P_t$), which are appended to the text encoder’s input sequence;

Visual Prompts ($P_v$), which are inserted into the visual encoder’s token sequence.

Following MaPLe, the prompts are initialized using a shared embedding matrix $P_s\in {\mathbb{R}}^{L_p \times d}$, where $L_p$ is the number of prompt tokens and $d$ is the embedding dimension. The shared matrix ensures initial semantic coherence between modalities.

To accommodate modality-specific characteristics, we apply two linear projections:

$$P_v=W_v{P}_s, P_t=W_t{P}_s$$

(2)

where $W_v$ and $W_t$ are learnable projection matrices for the visual and textual branches. This design allows the prompts to share a common initialization while learning modality-specific representations during optimization. Through this initialization, the HISF model gains a stable starting point for cross-modal alignment and reduces training instability typically observed in multimodal prompt learning.

3.3. Hierarchical Semantic Fusion

To overcome the limitations of shallow prompting, we propose a hierarchical semantic fusion strategy that injects label semantics progressively throughout the encoder layers. The detailed operational flow of this module is illustrated in Figure 2. Unlike standard methods that only optimize prompts at the input level, our design enables deep, layer-wise interaction between the learnable prompts and the fixed category semantics.

Specifically, prompts are inserted into the visual and textual Transformers at layer $l\in \left\{L_1,L_2,L_3,\dots ,L_k\right\},$ where $k$ is the number of fusion layers. This allows progressive semantic interaction between visual tokens, textual prompts, and category embeddings.

To strengthen inter-modal communication, we introduce a cross-attention mechanism between prompts and label embedding.

For a given layer, the semantic fusion process is formulated as:

$$Q_p=LayerNorm\left(P\right)·W_q$$

(3)

$$K_l=E_i·W_k$$

(4)

$$V_l=E_i·W_v$$

(5)

where $W_q$, $W_k$, and $W_v$are learnable projection matrices for the prompt vectors and $Q_p$, $K_l$, and $V_l$ correspond to the query, key, and value matrices. In HISF, the prompt tokens serve as queries $Q_p$, while the label embeddings act as keys and values $K_l$ and $V_l$.

The updated prompt representation at layer $l$ is thus:

$$Attn=Softmax\left({\displaystyle \frac{Q_p·{K_l}^T}{\sqrt{d_K}}}\right)V_l$$

(6)

Following the cross-attention operation, we introduce a multi-layer perceptron (MLP) with a bottleneck design to perform residual semantic augmentation. This bottleneck structure functions as a non-linear projection, compressing the fused features into a lower-dimensional space before restoration. This process encourages the model to capture more compact and salient semantic representations. Meanwhile, the residual connection ensures that these augmentations refine, rather than replace the original prompt information. Consequently, this architecture improves gradient flow and stabilizes training, thereby facilitating faster convergence by efficiently integrating label-guided semantics. The calculation formula is as follows:

$$P_{fused }=P+Attn$$

(7)

$$P_{final }=P_{fused }+MLP\left(P_{fused}⨁E_i\right)$$

(8)

The prompt vector after incorporating semantic information is denoted by $P_{fused}$, where the operation symbol $⨁$ represents concatenating the blended prompt vector with label information. This is then fed into a multilayer perceptron, and the final result $P_{final }$undergoes residual connection to yield the final prompt vector.

This operation allows category semantics to directly guide prompt optimization, aligning the representation of prompts with their corresponding class meanings. By performing this operation hierarchically across Transformer layers, HISF achieves multi-level semantic fusion, where shallow layers capture local appearance information and deeper layers encode abstract semantics. To explicitly operationalize this hierarchical injection, the process proceeds layer-by-layer within the encoder. Specifically, at each fusion layer $l$, the prompt tokens are first updated via the label-guided cross-attention and refined by the bottleneck MLP as described above. These semantically enriched prompts are then concatenated with the original image (or text) tokens to form the input for the subsequent Transformer block. This interaction repeats recursively across the $k$ inserted layers. Finally, the outputs of the last Transformer layer are pooled to produce the global representations $z_v$ and $z_t$, which now encapsulate the progressively fused semantic information from both modalities.

This hierarchical fusion mechanism not only enhances intra-class feature compactness but also promotes inter-class separability, which is crucial for few-shot recognition and cross-domain generalization.

3.4. Label Embedding and Semantic Alignment

A key innovation of HISF is the introduction of label embedding (LE) and semantic alignment (SA) mechanisms, which jointly constrain the relationship between prompts and category semantics.

Let $E_y\in {\mathbb{R}}^d$ denote the label embedding corresponding to category $y$. During training, we encourage the fused prompt representation $P_{final}$ to remain semantically close to the corresponding label embedding $E_i$.

To achieve this, we define two complementary objectives:

1. Label Embedding Constraint ($L_{le}$)—encourages the prompt embeddings to align with the label embedding in semantic space:

$$L_{Le }={\displaystyle \frac{1}{N}}\sum _{i=1}^N|{|P_{final }-E_i||}_2$$

(9)

2. Semantic Alignment Loss ($L_{sa}$)—ensures consistency between the multimodal representations derived from prompts and the textual embeddings:

$$L_{sa }={\displaystyle \frac{1}{N}}\sum _{i=1}^N\left(1-cos\left(z_v,z_t\right)\right)$$

(10)

where $z_v$ and $z_t$ are the fused visual and textual features obtained after hierarchical semantic injection.

3. Class Prediction Loss ($L_{cls}$)—dominates the recognition accuracy of classification tasks, enabling the representation features obtained through prompt vector concatenation to better approximate the actual content.

$$L_{cls }={\displaystyle \frac{1}{N}}\sum _{i=1}^{N}CE\left(\tau ·sim\left(I_x,f(P_{final }+E_i)\right),y_i\right)$$

(11)

where $\tau$ is a parameter learned from CLIP, $f$ is the CLIP encoder, and $I_x$ is the presentation of image, while the cross-entropy (CE) loss is applied for the similarity score.

We use the CLIP encoder $f\left(·\right)$ and the representation of the image $I_x$, along with a parameter $\tau$ learned from CLIP. A cross-entropy (CE) loss is applied to the resulting similarity score.

The combined objective function is then expressed as:

$$L_{total }{=L}_{cls }+{\lambda }_1·L_{Le}+{\lambda }_1·L_{sa}$$

(12)

where $L_{cls}$ denotes the original CLIP contrastive loss, and ${\lambda }_1{ \mathrm{and} \lambda }_2$ are the balancing hyperparameters.

This formulation jointly optimizes the multimodal alignment and the prompt-label semantic consistency, ensuring that the learned prompts faithfully capture category-level meaning while preserving cross-modal coherence.

3.5. Complexity Analysis

A critical concern in multimodal prompting is the computational overhead. In HISF, the CLIP backbone parameters remain frozen. The additional trainable parameters are primarily introduced by the linear projection matrices ($W_q$, $W_k$, $W_v$) within the cross-attention modules across $J$ fusion layers. Since each projection matrix has a dimension of $D· D$, the total parameter complexity scales with $O\left( J·D^2\right)$, which is negligible compared to the backbone size. Regarding inference, the label embeddings $E_i$ are static so that the key and value matrices in the cross-attention modules can be pre-computed and cached. Consequently, HISF maintains an inference speed comparable to standard prompting methods while significantly enhancing representation quality.

3.6. Optimization and Training Strategy

Regarding the training configuration, it is important to clarify the status of the model parameters. To preserve the generalized knowledge acquired during pre-training, the entire CLIP backbone and the initial label embeddings derived from class names are kept frozen. The learnable parameters are strictly limited to the specific modules introduced by our HISF framework: the prompt embedding matrix, the projection matrices in the cross-attention modules, and the weights of the bottleneck MLPs. These components are jointly updated to minimize the total loss.

We apply an episodic training strategy similar to few-shot learning, where each episode includes a small portion of the base categories to form tasks. The model generalizes to new classes by transferring the hierarchical semantics to them from category embeddings. To stabilize optimization, the learning rate of prompt parameters is greater than that of the projection layers, since prompts should adapt faster. The optimizer we use is SGD, where the weight decay is only applied to projection matrices. We employ mixed-precision training for all experiments due to efficiency reasons. To summarize, the model is evaluated out-of-the-box in a few-shot and cross-domain setting without any further fine-tuning and shows high generalization capability.

3.7. Discussion

The design motivation for HISF is to model hierarchical semantic relationships in the Transformer structure explicitly. As opposed to existing approaches that treat multimodal prompts as shallow tokens, HISF integrates category-conditioned cross-attention across multiple layers of the model and therefore enables it to iteratively refine its understanding of multimodality. The hierarchical fusion structure not only strengthens the expressiveness of intra-modal features but also preserves the alignment of semantics between two modalities. Moreover, with the novel Label Embedding Constraint and Semantic Alignment Loss optimization process, we can successively regularize the learning procedure without overfitting on these tasks and achieve strong generalization to both few-shot and cross-domain settings.

In summary, HISF establishes a general and extensible framework for multimodal prompt learning, where hierarchical semantic fusion and label-guided attention serve as the core mechanisms driving efficient and interpretable model adaptation.

4. Experiments

We conduct extensive experiments to verify the effectiveness of our HISF method on 11 widely adopted image classification benchmarks. The datasets are Caltech101, ImageNet, OxfordPets, StanfordCars, Flowers102, Food101, FGVCAircraft, SUN397, UCF101, DTD, and EuroSAT. To ensure reproducibility and statistical reliability, all results reported in this paper are the average of 3 independent runs with different random seeds. All evaluations are conducted under the few-shot learning setup: we take 16 samples (called a 16-shot setting) for each class as the training set and report performance on the remaining samples. In cross-dataset transfer experiments, models learned from ImageNet are directly tested on the other datasets without any further fine-tuning to evaluate generalization stability.

4.1. Experimental Hyperparameters

We use the pre-trained ViT-B/16 model as our CLIP backbone. The dimension of the text modality embedding is 512, while the image modality embedding is 768. Prompt tokens are placed from a particular Transformer layer (the index of this layer and the number of prompt tokens are described in the source settings). All models are trained for 50 epochs by setting the batch size to 4. We use SGD implemented in PyTorch (https://pytorch.org/) (Meta AI, Menlo Park, CA, USA) as the optimizer with a learning rate of 0.0035. We perform experiments with the NVIDIA A800 GPU (NVIDIA Corporation, Santa Clara, CA, USA).

4.2. Base-to-Novel Generalization Evaluation

For base-to-novel generalization, we adopt the same data split as in MaPLe, where the dataset classes are split into base and novel classes, with 16 shots per class. The entire evaluation is performed in two steps. At the first stage, a subset of classes is chosen as (base) classes, and the model is trained on a few-shot sample belonging to those base classes. For stage two, the classes outside of state one are regarded as novel classes; we adapt on K-shot support samples from these novel classes and test on the remaining novel-class samples. We report accuracy for the base class, novel class, and their mean (H.M). The specific comparison results over 11 datasets are shown in

Table 1. Comparison of the proposed HISF method with recent state-of-the-art approaches across 11 benchmark datasets. Results for baseline methods are directly cited from their original publications to ensure fair comparison, while results of HISF are obtained from our own experiments.

Method	Average			ImageNet			Caltech101			OxfordPets
	Base	Novel	HM	Base	Novel	HM	Base	Novel	HM	Base	Novel	HM
CLIP [1]	69.34	74.22	71.70	72.43	68.14	70.22	96.84	94.00	95.40	91.17	97.26	94.12
CoOP [4]	82.69	63.22	71.66	76.47	67.88	71.92	98.00	89.81	93.73	93.67	95.29	94.47
CoOpOp [5]	80.47	71.69	75.83	75.98	70.43	73.10	97.96	93.81	95.84	95.20	97.69	96.43
ProDA [22]	81.56	72.30	76.65	75.40	70.23	72.72	98.27	93.23	95.68	95.43	97.83	96.62
KgCoOp [45]	80.73	73.60	77.00	75.83	69.96	72.78	97.72	94.39	96.03	94.65	97.76	96.18
MaPLe [6]	82.28	75.14	78.55	76.66	70.54	73.47	97.74	94.36	96.02	95.43	97.76	96.58
PromptSRC [46]	84.26	76.10	79.97	77.60	70.73	74.01	98.10	94.03	96.02	95.33	97.30	96.30
ProVP [44]	85.20	73.22	78.76	75.82	69.21	72.36	98.92	94.21	96.51	95.87	97.65	96.75
MP [43]	83.65	75.48	79.09	77.52	70.83	74.02	98.13	94.58	96.32	95.53	97.00	96.26
TCP [47]	84.13	75.36	79.51	77.27	69.87	73.38	98.23	94.67	96.42	94.67	97.20	95.92
MMA [34]	83.20	76.80	79.87	77.31	71.00	74.02	98.40	94.00	96.15	95.40	98.07	96.72
HISF	85.39	76.12	80.75	76.8	70.1	73.4	98.20	94.6	96.4	95.7	96.9	96.3
Method	StanfordCars			Flower102			Food101			FGVCAircraft
	Base	Novel	HM	Base	Novel	HM	Base	Novel	HM	Base	Novel	HM
CLIP [1]	63.37	74.89	68.65	72.08	77.80	74.83	90.10	91.22	90.66	27.19	36.29	31.09
CoOP [4]	78.12	60.40	68.13	97.60	59.67	74.06	88.33	82.26	85.19	40.44	22.30	28.75
CoOpOp [5]	70.49	73.59	72.01	94.87	71.75	81.71	90.70	91.29	90.99	33.41	23.71	27.74
ProDA [22]	74.70	71.20	72.91	97.70	68.68	80.66	90.30	88.57	89.43	36.90	34.13	35.46
KgCoOp [45]	71.76	75.04	73.36	95.00	74.73	83.65	90.50	91.70	91.09	36.21	33.55	34.83
MaPLe [6]	72.94	74.00	73.47	95.92	72.46	82.56	90.71	92.05	91.38	37.44	35.61	36.50
PromptSRC [46]	78.27	74.97	76.58	98.07	76.50	85.95	90.67	91.53	91.10	42.73	37.87	40.15
ProVP [44]	80.43	67.96	73.67	98.42	72.06	83.20	90.32	90.91	90.61	47.08	29.87	36.55
MP [43]	76.34	75.01	75.48	97.66	74.49	84.52	90.74	91.85	91.29	40.14	36.51	38.24
TCP [47]	80.80	74.13	77.32	97.73	75.57	85.23	90.57	91.37	90.97	41.97	34.43	37.83
MMA [34]	78.50	73.10	75.70	97.77	75.93	85.48	90.13	91.30	90.71	40.57	36.33	38.33
HISF	81.9	74.3	78.1	98.6	74.2	86.4	89.5	90.7	90.1	47.6	33.8	40.7
Method	SUN397			DTD			EuroSAT			UCF101
	Base	Novel	HM	Base	Novel	HM	Base	Novel	HM	Base	Novel	HM
CLIP [1]	69.36	75.35	72.23	53.24	59.90	56.37	56.48	64.05	60.03	70.53	77.50	73.85
CoOP [4]	80.60	65.89	72.51	79.44	41.18	54.24	92.19	54.74	68.69	84.69	56.05	67.46
CoOpOp [5]	79.74	76.86	78.27	77.01	56.00	64.85	87.49	60.04	71.21	82.33	73.45	77.64
ProDA [22]	78.67	76.93	77.79	80.67	56.48	66.44	83.90	66.00	73.88	85.23	71.97	78.04
KgCoOp [45]	80.29	76.53	78.36	77.55	54.99	64.35	85.64	64.34	73.48	82.89	76.67	79.65
MaPLe [6]	80.82	78.70	79.75	80.36	59.18	68.16	94.07	73.23	82.35	83.00	78.66	80.77
PromptSRC [46]	82.67	78.47	80.52	83.37	62.97	71.75	92.90	73.90	82.32	87.10	78.80	82.74
ProVP [44]	80.67	76.11	78.32	83.95	59.06	69.34	97.12	72.91	83.29	88.56	75.55	81.54
MP [43]	82.26	79.04	80.62	83.10	58.05	68.35	93.53	75.21	83.38	85.33	77.72	81.35
TCP [47]	82.63	78.20	80.35	82.77	58.07	68.25	91.63	74.73	82.32	87.13	80.77	83.83
MMA [34]	82.27	78.57	80.38	83.20	65.63	73.38	85.46	82.34	83.87	86.23	80.03	82.20
HISF	83.5	79.3	81.4	83.6	64.8	74.2	97.3	77.5	87.4	86.6	81.2	83.9

. For all baseline methods, we report the officially published results from the original papers to ensure a fair and consistent comparison. To validate the effectiveness of our proposed hierarchical fusion mechanism, we performed a paired t-test comparing HISF with its direct baseline MaPLe across the 11 datasets. The analysis yields a p-value < 0.05, indicating that our method is statistically significant and robust compared to the baseline, validating the efficacy of the proposed architectural changes.

4.3. Cross-Dataset Transfer Evaluation

For the cross-dataset transfer evaluation, we adopt these 11 datasets. We first conduct a few-shot protocol training for the models on ImageNet and then directly measure the generalization performance of the trained models across datasets on the remaining ten datasets. As shown in

Table 2. Comparison of HISF with existing methods in a cross-dataset setting. Baseline results are taken from the corresponding original papers, and HISF results are produced by our implementation.

Method	Source	Target
	ImageNet	Caltech	OxfordPets	Stanford Cars	Flowers 101	Food101	FGVC Aircraft	SUN397	DTD	EuroSAT	UCF101
CoOp [4]	71.51	93.70	89.14	64.51	68.71	85.30	18.47	68.17	41.92	46.39	66.55
CoOpOp [5]	71.02	94.43	90.14	65.32	71.88	86.06	22.94	67.36	45.73	45.37	68.21
MaPLe [6]	70.72	93.53	90.49	65.57	72.23	86.20	24.74	67.01	46.49	48.06	68.69
PromptSRC [46]	71.27	93.60	90.25	65.70	70.25	86.15	23.90	67.10	46.87	45.50	68.75
TCP [47]	71.40	93.97	91.25	64.49	71.21	86.69	23.45	67.15	44.35	51.45	68.73
MMA [34]	71.00	93.80	90.30	66.13	72.07	86.12	25.33	68.17	46.57	49.24	68.32
HISF	72.20	93.49	90.32	65.40	72.41	85.41	24.31	67.57	45.91	54.70	68.41

, HISF demonstrates varying degrees of improvement across different datasets. Notably, on datasets with significant domain shifts from the pre-training data, such as EuroSAT, HISF achieves remarkable gains compared to state-of-the-art methods like MaPLe. This validates that our hierarchical semantic fusion and label-guided alignment effectively bridge the domain gap by enforcing robust semantic consistency.

Regarding the mixed results on fine-grained datasets like Food101 and StanfordCars, where HISF performs competitively but does not show the same magnitude of improvement, we attribute this to the strong regularization effect of our design. Methods without explicit semantic constraints (e.g., CoOp) may allow the learnable prompts to drift significantly from the original text embeddings to overfit the specific visual nuances of these datasets. In contrast, HISF incorporates Label Embedding Constraints (LECs) and Semantic Alignment Loss (SAL) to strictly prevent such semantic drift. While this strict alignment might slightly limit the model’s flexibility to overfit fine-grained features in domains similar to the pre-training data, it is precisely this mechanism that guarantees the superior generalization ability observed in unseen and distinct domains (EuroSAT). Thus, the results reflect a deliberate trade-off prioritizing robust generalizability over task-specific overfitting.

4.4. Domain Generalization Evaluation

Domain generalization: We test generalized performance on ImageNet and directly train on the source domain (ImageNet), without fine-tuning the model with target data. These test sets have the same class labels as ImageNet but involve distributional shifts.

Table 3. Comparison of HISF with existing methods in the domain generalization setting. Baseline results are taken from the corresponding original papers, and HISF results are produced by our implementation.

Method	Source	Target
	ImageNet	−S	−A	−R	−V2
CLIP [1]	66.73	46.15	47.77	73.96	60.83
CoOp [4]	71.51	47.99	49.71	75.21	64.20
CoOpOp [5]	71.02	48.75	50.63	76.18	64.07
Maple [6]	70.72	49.15	50.90	76.98	64.07
PromptSRC [46]	71.27	49.55	50.90	77.80	64.35
HISF	72.20	49.51	51.20	78.21	64.43

shows the domain transfer performance of our method and other comparisons, including CLIP, CoOp, Co-CoOp, MaPLe, PromptSRC, and HISF. HISF is competitive across domain-shifted benchmarks, as reflected in Table 3 with their average numbers (e.g., on ImageNet, HISF reaches 72.20% (source)), and it improves over baselines for all the other domain shifts, like ImageNet-Sketch, ImageNet-A, ImageNet-R, and on six of the eight subsets from Imagenet-V2. This highlights the relatively higher robustness of HISF to distributional variations.

4.5. Ablation Studies

We perform a series of ablation experiments to analyze the contributions of individual design choices and loss components. The ablation studies include (1) prompt insertion depth; (2) HISF module effectiveness (Prompt Initialization PI and semantic fusion SF); and (3) loss term influences.

4.5.1. Prompt Insertion Depth

When inserting prompts into the final layers, the decline in performance is a common phenomenon observed in deep prompt learning. It is widely understood that Transformer-based models capture increasingly abstract and task-specific semantic features in deeper layers. Injecting learnable parameters at this late stage may disrupt the well-established feature space generated by the pre-trained backbone, as the added prompts have less opportunity to integrate seamlessly with the hierarchical feature extraction process. Effectively, the prompt information injected too late might conflict with the highly refined final representations, rather than guiding them, leading to suboptimal performance. The curve is given in Figure 3 (prompt depth vs. average accuracy).

4.5.2. HISF Component Effectiveness

We validate the effort of HISF components by performing module-wise ablation. PI represents the Prompt initialization, and SF is short for semantic fusion. Results of HISF on the MaPLe experimental setting and its ablated variants are shown in

Table 4. Evaluation of the impact of different HISF modules on model performance.

Method	Base	Novel	H.M
HISF (Maple setting)	82.28	75.14	78.55
HISF (w/o PI)	83.76	75.57	79.66
HISF (w/o SF)	82.45	74.96	78.71
HISF (PI + SF)	85.39	76.12	80.75

. The ablative studies confirm the significance of each module towards HISF’s performance. Taking out the Prompt Initialization module (w/o PI), there are negligible shifts in base and novel accuracies under the MaPLe setting, indicating that shared initialization has a stabilizing effect at the beginning of training. Without the semantic fusion module (w/o SF), we obtain another remarkable performance drop, verifying that the hierarchical approach plays a critical role in aligning modalities. The model achieves the best performance when both PI and SF are exploited (PI + SF) (Base 85.39, Novel 76.12, H.M 80.75), showing that prompt initialization and semantic fusion are complementary: The former establishes a consistent semantic entry point to each task, while the latter imposes deep cross-modal interplays for enhanced discrimination ability and generalizability.

4.5.3. Loss Component Influence

We test to what extent the three loss terms (label embedding loss, Semantic Alignment Loss, and classification loss) contribute to overall performance by conducting experiments to remove each part independently. The performance degrades significantly if we remove the label embedding loss, which suggests that this loss is crucial for learning discriminative class-specific representations. The degradation of cross-dataset transfer performance and domain generalization if the Semantic Alignment Loss is disabled: The model’s inference accuracy on the target datasets decreases, which shows that it can help in forming more effective semantic correspondence across different datasets for distributional change robustness. The quantitative results in

Table 5. Evaluation of the effect of different loss functions on model performance.

LCE	LS	LLP	H.M
		√	79.41
√	√	√	80.75
	√	√	80.67
√		√	80.35

collectively confirm that the three losses facilitate together to make a contribution to the base-to-novel generalization, cross-dataset transfer, and domain robustness; the combinational effect from them is crucial to obtain the performance in experiments of HISF.

4.6. Summary

The above experiments show the effectiveness of HISF under several evaluation protocols. HISF outperforms existing methods on base-to-novel generalization, cross-dataset transfer, and domain generalization. Ablation studies confirm the efficacy of timely insertion depth, Prompt Initialization, semantic fusion, and the joint loss design. These findings empirically support our claim that the hierarchical semantic injection and label-constrained alignment work together to enhance visual identification with language-based meta-training for prompt-based adaptation of vision-language pre-trained models.

5. Conclusions

In this work, we present a new multimodal prompt learning framework called HISF (Hierarchical Interleaved Semantic Fusion). In this paper, we propose a hierarchical semantic fusion (HISF) to make bidirectional information flow between visual and textual modality, thus substantially improving fine-grained semantic alignment. With hierarchical fusion and label-guided learning, richer multimodal interaction is captured, and better generalization in base as well as novel categories is obtained.

In particular, the PI module gives a uniform initialization, which stabilizes training and prevents overfitting in few-shot cases, while the SF module learns multi-level semantics to facilitate deeper interaction between modalities. Moreover, a label embedding and semantic alignment-based joint optimization approach can prevent information loss and discriminate the learned features effectively.

Extensive experiments conducted on eleven benchmark datasets validate the effectiveness of HISF compared to existing prompt learning methods. Ablation results further confirm that each proposed component contributes meaningfully to performance improvement, particularly in maintaining semantic coherence and enhancing model adaptability under limited data conditions.

Limitations: Despite these advantages, we acknowledge certain limitations. First, the explicit semantic constraints may limit performance on tasks requiring extreme fine-grained discrimination, where visual features diverge from pre-trained linguistic descriptions (e.g., Food101). Second, while parameter-efficient, the cross-attention mechanism introduces a slight increase in training latency compared to static prompting methods like CoOp. Future work will explore adaptive constraint mechanisms to dynamically balance generalization and task-specific fitting. We believe that the proposed framework offers a promising direction for developing more generalizable and interpretable multimodal prompt learning systems.