Cross-Modal Few-Shot Learning via Siamese Similarity Networks on CLIP Embeddings for Fine-Grained Image Classification

Abstract

Fine-grained image classification under few-shot learning conditions remains a significant challenge due to limited labeled data and high intra-class similarity. This paper proposes a novel cross-modal framework that integrates Contrastive Language-Image Pretraining (CLIP) embeddings within a Siamese similarity network to enable robust and label-efficient classification. By leveraging the semantic alignment between textual class descriptions and visual representations, the model forms hybrid similarity pairs of image-to-image and image-to-text within a shared latent space, facilitating discriminative learning under low-shot scenarios. The architecture employs a dual-branch CLIP encoder and a contrastive loss function to optimize intra-class compactness and inter-class separability. Experiments conducted on benchmark datasets including miniImageNet and CUB-200-2011 demonstrate substantial improvements over zero-shot and few-shot baselines, achieving 70.32% accuracy, 71.15% F1-score, and 68.47% mAP on 5-way 1-shot and 78.41% accuracy, 79.02% F1-score, and 76.83% mAP on 5-way 5-shot tasks (averaged over 600 episodes with 95% confidence intervals on the CUB-200-2011 dataset). Extended evaluations under 10-way settings show similarly strong performance. Ablation studies further validate the critical roles of contrastive learning, normalization, and cross-modal embeddings in enhancing generalization. This work presents a scalable and interpretable paradigm for fine-grained classification in data-scarce domains.

1. Introduction

Fine-grained image classification refers to a category of computer vision tasks where the goal is to distinguish between visually similar subcategories within the same broader class, rather than separating clearly different object types [1]. Unlike coarse-grained categorization, where visual distinctions are often pronounced and abundant training samples are available, fine-grained tasks demand sensitivity to minute inter-class variations such as texture, pose, or morphological attributes while frequently operating under severely limited labeled data [2]. This problem is further compounded in real-world domains such as biodiversity monitoring, medical imaging, and industrial inspection, where expert annotation is expensive and class distributions are inherently long-tailed [3].

Few-shot learning has emerged as a principled response to this constraint by emphasizing transferable representations and similarity-based reasoning rather than exhaustive supervision [4]. However, most few-shot frameworks remain anchored in unimodal visual embeddings, implicitly assuming that discriminative capacity can be extracted solely from pixel-level cues [5]. This assumption overlooks a critical source of supervisory signal: natural language. Human expertise in fine-grained recognition is rarely visual alone; it is mediated through semantic descriptions, attribute-level reasoning, and contextual knowledge [6]. Bridging this gap requires models that can exploit the complementary strengths of visual and linguistic modalities within a unified representational space.

Recent advances in large-scale vision–language pretraining, exemplified by Contrastive Language–Image Pretraining (CLIP), have fundamentally altered the landscape of representation learning [7]. By aligning images and text through contrastive objectives over web-scale data, CLIP embeddings encode rich semantic structure that extends beyond traditional visual features [8]. Importantly, these embeddings exhibit strong zero-shot and transfer capabilities, suggesting their suitability as a foundation for data-efficient learning [9]. Yet, despite their promise, CLIP representations are typically employed in a direct classification or prompt-based setting, offering limited adaptability to fine-grained distinctions under few-shot regimes [10].

This work argues that the true potential of CLIP in low-data scenarios lies not in direct prediction, but in metric learning over cross-modal representations. We propose a cross-modal few-shot learning framework that leverages Siamese similarity networks operating on CLIP-derived embeddings, enabling the model to learn relational structures between image–text pairs rather than relying on absolute class boundaries. In this context, relational correspondences are modeled as structured similarity relationships within the shared embedding space, where semantically related image–text pairs are encouraged to occupy neighboring regions while dissimilar pairs are separated by a learned margin. The Siamese architecture operationalizes this process by jointly encoding paired inputs and optimizing a similarity objective that preserves relative distances between samples, thereby making relational structure observable through embedding geometry. Instead of learning class-specific decision boundaries, the model learns how instances relate to one another through similarity scores, allowing fine-grained distinctions to emerge from consistent relational patterns across support and query examples. By reframing fine-grained classification as a similarity estimation problem, the proposed approach aligns naturally with the epistemic constraints of few-shot learning, where relational inductive bias is more reliable than parametric memorization.

2. Literature

2.1. Few-Shot Image Classification and Visual Metric Learning

Early research in few-shot image classification largely focused on learning embedding spaces capable of supporting generalization from limited labeled examples. Metric-learning approaches became particularly influential, as they reframed recognition as a similarity estimation problem rather than a direct classification task. Within fine-grained settings, where inter-class variations are subtle, relational modeling proved especially effective. BSNet, for example, formalized few-shot fine-grained recognition as a bi-similarity learning problem by jointly encouraging intra-class compactness and inter-class separability, demonstrating that structured similarity relationships can improve discrimination under limited supervision [11].

Subsequent work increasingly emphasized the sensitivity of learned representations rather than architectural scale alone. DetailCLIP represents this direction by refining pretrained vision–language representations to better capture localized and attribute-level information, showing that performance improvements in fine-grained scenarios can be achieved through enhanced representation focus rather than deeper visual backbones [12]. These studies collectively highlight that fine-grained recognition performance depends strongly on how similarity relationships are encoded within the embedding space.

From a perceptual perspective, Takahashi et al. [13] further examined the alignment between neural similarity spaces and human similarity judgments. Their findings indicate that deep models often preserve coarse semantic structure while failing to consistently capture fine-grained relational distinctions unless similarity relationships are explicitly modeled. This observation has reinforced interest in relational learning paradigms that prioritize pairwise or structural correspondence over absolute feature activation.

Despite these advances, approaches relying exclusively on visual embeddings exhibit limitations when training data is scarce. Fine-grained discrimination frequently involves semantic attributes or domain-specific descriptors that may not be reliably inferred from limited visual observations alone. This limitation has motivated the integration of additional semantic information into few-shot learning frameworks.

2.2. Vision–Language Models as Foundations for Data-Efficient Learning

The introduction of large-scale vision–language pretraining has significantly influenced few-shot learning research by enabling joint semantic representations across modalities. Models such as CLIP align visual and textual inputs within a shared embedding space, allowing semantic information expressed in language to guide recognition without extensive task-specific supervision. Empirical studies have shown that such representations often outperform conventional visual features in low-data regimes, particularly when class characteristics can be described linguistically.

Building upon this foundation, several works have explored mechanisms for adapting vision–language representations to task-specific discrimination. SGVA-CLIP proposes a semantic-guided visual adaptation strategy in which visual features are selectively aligned with language-informed prototypes, improving class separability under few-shot constraints [14]. Similarly, FewVS integrates vision and semantics through a structured fusion framework, highlighting that semantic priors can effectively regularize visual learning when labeled data is scarce [15].

Beyond generic benchmarks, CLIP-based few-shot learning has shown promise in specialized domains. In medical imaging, CLIP embeddings combined with vision transformers have enabled both few-shot and zero-shot tumor classification, demonstrating robustness in clinically constrained data regimes [16]. In remote sensing and hyperspectral imaging, prompt learning and text-driven supervision have been used to adapt CLIP for scene and species classification, where labeled samples are extremely limited and semantic descriptions are readily available [17,18].

2.3. Addressing the Modality Gap in Cross-Modal Few-Shot Learning

Although vision–language models provide a nominally shared embedding space, several studies report that alignment between visual and textual representations remains imperfect, particularly for fine-grained distinctions. Subtle visual differences may not be consistently reflected in textual embeddings, leading to residual modality gaps that degrade performance in few-shot settings. To mitigate this issue, cross-modal alignment and mapping strategies have been proposed. Yang et al. [19] introduce a cross-modal mapping framework that learns adaptive transformations between visual and textual representations, demonstrating improved few-shot generalization by reducing semantic drift across modalities.

Related work in sketch-based and fine-grained retrieval further underscores the importance of modality-aware similarity learning. CLIP-based approaches for sketch-to-image retrieval reveal that cross-modal similarity is highly sensitive to representation alignment, especially when fine-grained discrimination is required [20]. These findings suggest that simply leveraging pretrained embeddings is insufficient; instead, relational learning mechanisms are needed to exploit cross-modal structure effectively.

2.4. Siamese and Transformer-Based Similarity Learning

Similarity learning architectures, particularly Siamese networks, have long served as a core paradigm for few-shot learning due to their capacity to encode relational inductive biases. Recent extensions incorporate transformer-based attention mechanisms to enable global context modeling and dynamic interaction between support and query samples. Siamese Transformer Networks, for example, integrate self-attention across paired inputs, achieving strong performance on few-shot benchmarks [21]. Hierarchical concept embedding further enriches this approach by embedding semantic hierarchies into similarity estimation, enhancing fine-grained recognition performance [22].

However, most Siamese-based methods remain unimodal, operating exclusively on visual embeddings. Their application to cross-modal embeddings—particularly those derived from large vision–language models remain underexplored. This gap is notable given that Siamese architectures are inherently well-suited to similarity estimation across heterogeneous representations.

In summary, existing literature converges on three key insights: (i) few-shot fine-grained classification benefits from relational similarity learning, (ii) vision–language models such as CLIP provide semantically rich embeddings well-suited for low-data regimes, and (iii) effective cross-modal alignment remains a critical challenge. While prior work has explored semantic-guided adaptation, prompt learning, and vision–semantics fusion, limited attention has been given to explicit similarity learning over cross-modal CLIP embeddings using Siamese architectures.

The present work builds upon these foundations by unifying cross-modal representation learning and Siamese similarity estimation within a single framework. By operating directly on CLIP embeddings and reframing few-shot classification as a cross-modal similarity learning problem, this approach addresses both data scarcity and fine-grained semantic ambiguity. In doing so, it extends prior research beyond feature adaptation and prompt engineering toward a principled metric-learning formulation tailored for multimodal few-shot inference.

3. Methodology

3.1. Proposed Method

The proposed model introduces a Cross-Modal Few-Shot Learning framework that leverages the joint vision-language embedding capabilities of CLIP with a Siamese Similarity Network to achieve fine-grained image classification under few-shot constraints. This design addresses the known limitations of zero-shot models such as CLIP in domains where subtle intra-class variations pose significant challenges to semantic discrimination. By extracting aligned embeddings from both modalities and subsequently learning a data-driven similarity function through a Siamese network, the model enhances generalization to novel classes while maintaining high discriminative power with minimal labeled examples.

The architecture is organized into two main stages. The first involves the extraction of embeddings using a pre-trained CLIP model to project both images and their corresponding class texts into a unified latent space. The second stage applies a Siamese neural network to these embeddings, which is trained to compute meaningful similarity scores between samples, ultimately guiding classification decisions in few-shot settings. A conceptual illustration of the complete framework is provided in Figure 1.

Figure 1. Conceptual Illustration of the Proposed Framework.

3.2. CLIP Embedding Extraction

Given an input image $x_i\in \mathcal{X}$ and its corresponding class text $c_j\in \mathcal{C}$, the framework employs the image encoder $f_{img}$ and text encoder $f_{txt}$ from a pre-trained CLIP model to embed both modalities into a shared latent space ${\mathbb{R}}^d$, where $d=512$. The resulting embeddings are defined as

$${\mathbf{v}}_i=f_{img}\left(x_i\right), {\mathbf{t}}_j=f_{txt}\left(c_j\right).$$

To ensure uniformity and improve cosine-based similarity computation, the embeddings are L2-normalized:

$${\widehat{\mathbf{v}}}_i={\displaystyle \frac{{\mathbf{v}}_i}{\parallel {\mathbf{v}}_i{\parallel }_2}}, {\widehat{\mathbf{t}}}_j={\displaystyle \frac{{\mathbf{t}}_j}{\parallel {\mathbf{t}}_j{\parallel }_2}}.$$

This normalization facilitates the use of cosine similarity as a measure of cross-modal alignment:

$$s({\widehat{\mathbf{v}}}_i,{\widehat{\mathbf{t}}}_j)={\widehat{\mathbf{v}}}_i^{\mathrm{\top }}{\widehat{\mathbf{t}}}_j.$$

3.3. Siamese Network Architecture

To learn more fine-grained semantic relationships, a Siamese neural network ${\mathcal{S}}_{\varphi }$ is integrated into the framework. This network receives as input a pair of CLIP embeddings $({\mathbf{e}}_1,{\mathbf{e}}_2)\in {\mathbb{R}}^{512\times 2}$ and computes a similarity score $s_{ij}$, which quantifies the likelihood of both inputs belonging to the same class:

$$s_{ij}={\mathcal{S}}_{\varphi }({\mathbf{e}}_1,{\mathbf{e}}_2).$$

The Siamese architecture comprises a concatenation of the two embeddings, followed by two fully connected layers with ReLU activation functions, and a final output layer with a sigmoid activation. Specifically, the transformations are defined as follows:

$$h_1=\mathrm{ReLU}\left(W_1\right[{\mathbf{e}}_1;{\mathbf{e}}_2]+b_1),$$

$$h_2=\mathrm{ReLU}(W_2{h}_1+b_2),$$

$$s_{ij}=\sigma (W_o{h}_2+b_o).$$

The network is trained using metric learning objectives, namely the contrastive loss and the triplet loss. The contrastive loss penalizes dissimilar embeddings of similar instances and vice versa, and is expressed as:

$${\mathcal{L}}_{contrastive}=y_{ij}\cdot D_{ij}^2+(1-y_{ij})\cdot \mathrm{m}\mathrm{a}\mathrm{x}(0,m-D_{ij}{)}^2,$$

where $D_{ij}=\parallel {\mathbf{e}}_1-{\mathbf{e}}_2{\parallel }_2$ and $y_{ij}\in \left\{\mathrm{0,1}\right\}$ indicates whether the pair is positive or negative. In contrast, the triplet loss is formulated to enforce relative similarity, ensuring that the anchor-positive distance is less than the anchor-negative distance by a margin:

$${\mathcal{L}}_{triplet}=\mathrm{m}\mathrm{a}\mathrm{x}(0,D_{ap}-D_{an}+\alpha ),$$

where $D_{ap}=\parallel {\mathbf{e}}_a-{\mathbf{e}}_p{\parallel }_2$ and $D_{an}=\parallel {\mathbf{e}}_a-{\mathbf{e}}_n{\parallel }_2$. The margins $m$ and $\alpha$ are set to 1.0 and 0.2 respectively.

3.4. Cross-Modal Pair Construction

The training process involves constructing both intra-modal (image-image) and cross-modal (image-text) pairs. For each few-shot episode, positive and negative pairs are generated to facilitate contrastive learning. Positive pairs consist of images belonging to the same class, or images and their corresponding textual descriptions as depicted in Table 1. Negative pairs include images from different classes or images paired with incorrect class descriptions. To ensure balance, each training batch includes an equal proportion of positive and negative samples.

Table 1. Example of Cross-Modal Pair Samples in a 5-way 1-shot Setting.

Pair Type	Sample 1	Sample 2	Label
Image–Image	bird_A.jpg	bird_A2.jpg	1
Image–Image	bird_A.jpg	bird_B.jpg	0
Image–Text	bird_A.jpg	“a photo of bird A”	1
Image–Text	bird_A.jpg	“a photo of bird C”	0

3.5. Datasets

The effectiveness of the proposed cross-modal few-shot learning framework was evaluated on three widely adopted fine-grained image classification datasets such as CUB-200-2011 [23], Stanford Dogs, and the miniImageNet benchmark. Each dataset presents varying levels of complexity and intra-class variance, making them suitable for assessing the generalization capability of few-shot models.

The CUB-200-2011 dataset consists of 11,788 bird images spanning 200 classes. For few-shot evaluation, 100 classes were designated for training (base), 50 for validation, and 50 for testing (novel). The Stanford Dogs dataset includes 20,580 images from 120 dog breeds. A similar split was applied with 70 base, 20 validation, and 30 novel classes. The miniImageNet dataset contains 60,000 images across 100 categories. Following standard practice, 64 classes were used for training, 16 for validation, and 20 for testing as shown in Table 2.

Table 2. Dataset Statistics and Splits.

Dataset	Total Classes	Base	Validation	Novel	Total Images
CUB-200-2011	200	100	50	50	11,788
Stanford Dogs	120	70	20	30	20,580
miniImageNet	100	64	16	20	60,000

Each dataset was restructured episodically to support N-way K-shot classification tasks, with experiments conducted under 5-way and 10-way settings using 1-shot and 5-shot configurations. All images were resized to 224 × 224 to comply with the input requirements of the CLIP model.

Baselines

The proposed model was compared against several strong baselines to demonstrate its effectiveness. The Zero-Shot CLIP baseline evaluated the original CLIP model’s ability to classify query images using text prompts alone, with no additional training. Prototypical Networks served as a metric-based baseline that computes class prototypes and distances in embedding space. The Standard Siamese Network baseline was trained on CLIP embeddings but without cross-modal pairs, relying solely on image-image similarity.

Also, a CLIP Fine-Tuned Siamese variant was tested, in which the CLIP image encoder was fine-tuned during training to assess the benefit of end-to-end optimization. These baselines enabled comprehensive evaluation of the impact of both cross-modal training and Siamese-based similarity modeling.

3.6. Implementation Details

All models were implemented using PyTorch 2.1 and the OpenCLIP v3.2.0 repository for embedding extraction. The Siamese network was trained with the Adam optimizer using a learning rate of $1\times {10}^{-4}$, weight decay of $1\times {10}^{-5}$, and batch size of 64 as shown in Table 3. A maximum of 100 epochs was used, with early stopping applied based on validation accuracy.

Table 3. Implementation and Training Parameters.

Parameter	Setting
Framework	PyTorch 2.1
CLIP Model	ViT-B/32
Optimizer	Adam
Learning Rate	$1\times {10}^{-4}$
Batch Size	64
Epochs	100
Hardware	RTX 3090 GPU
Augmentations	Horizontal Flip, Jitter

Experiments were conducted on a machine equipped with an NVIDIA RTX 3090 GPU (NVIDIA, Santa Clara, CA, USA), 16 GB RAM, and an Intel Xeon Silver processor (Intel, Santa Clara, CA, USA), 16 GB of RAM, and an Intel Xeon Silver processor. All models used the ViT-B/32 variant of CLIP for consistency. Image augmentations during training included random horizontal flipping and color jitter, while no augmentations were applied during evaluation.

3.7. Evaluation Metrics

Performance was evaluated using the mean classification accuracy over 600 randomly sampled episodes for each setting. For each episode, accuracy was measured as the proportion of correctly predicted query images. In addition to accuracy, precision, recall, and F1-score were reported to provide a more nuanced evaluation of class-wise performance.

To assess generalization, results were reported across both 5-way and 10-way classification tasks under 1-shot and 5-shot settings. Furthermore, confidence intervals at 95% were computed using bootstrapping with 10,000 resamples. The Area Under the ROC Curve (AUC) was also used to evaluate the similarity prediction performance of the Siamese network, particularly in ablation studies involving binary pair classification.

3.8. Training Procedure

The model is trained under an episodic learning framework, where each episode simulates a few-shot task comprising a support set and a query set. In a typical N-way K-shot setting, the support set consists of $N$ classes, each with $K$ labeled examples, while the query set includes additional examples from the same classes. Embeddings for all support and query instances are extracted using a frozen CLIP model. Pairwise combinations are formed as described earlier, and the Siamese network is optimized using the Adam optimizer with a learning rate $\eta =1\times {10}^{-4}$. Training is conducted using a batch size of 64, and early stopping is employed based on validation accuracy as depicted in Table 4.

Table 4. Hyperparameter Settings for Model Training.

Parameter	Value
Optimizer	Adam
Learning Rate	$1\times {10}^{-4}$
Batch Size	64
Embedding Dimension	512
Epochs (Max)	100
Margin (Contrastive)	1.0
Margin (Triplet)	0.2

3.9. Inference and Classification

During inference, the classification of a query image $q$ is performed by computing similarity scores between the query and all support instances using the trained Siamese network. Letting $\left\{s_k\right\}$ denote the set of support images, the predicted label is obtained as

$${\widehat{y}}_q=\mathrm{a}\mathrm{r}\mathrm{g}\underset{k}{\mathrm{m}\mathrm{a}\mathrm{x}}{\mathcal{S}}_{\varphi }(f_{img}(q),f_{img}(s_k)).$$

The framework also supports cross-modal inference by evaluating similarity between the query image and all class descriptions $\left\{t_k\right\}$, expressed as

$${\widehat{y}}_q=\mathrm{a}\mathrm{r}\mathrm{g}\underset{k}{\mathrm{m}\mathrm{a}\mathrm{x}}{\mathcal{S}}_{\varphi }(f_{img}(q),f_{txt}(t_k)).$$

To enhance robustness, the final prediction aggregates both similarity scores through an ensemble strategy:

$${\widehat{y}}_q=\mathrm{a}\mathrm{r}\mathrm{g}\underset{k}{\mathrm{m}\mathrm{a}\mathrm{x}}\left[\lambda \cdot {\mathcal{S}}_{\varphi }(f_{img}(q),f_{img}(s_k))+(1-\lambda )\cdot {\mathcal{S}}_{\varphi }(f_{img}(q),f_{txt}(t_k))\right],$$

where $\lambda \in [0,1]$ is a weighting coefficient that balances the contributions of image-based and text-based similarity, empirically set to 0.7 in this study.

4. Results and Evaluation

4.1. Quantitative Results

The proposed cross-modal Siamese framework was evaluated under 5-way and 10-way few-shot classification settings on three benchmark datasets. Table 5 reports the average classification metrics across 600 episodes for each method, including accuracy, F1-score, and mean Average Precision (mAP) with 95% confidence intervals.

Table 5. Few-Shot Classification Results (%) with Accuracy, F1-Score, and mAP (95% CI).

Model	5w1s Accuracy	5w1s F1	5w1s mAP	5w5s Accuracy	5w5s F1	5w5s mAP	10w1s Accuracy	10w1s F1	10w1s mAP	10w5s Accuracy	10w5s F1	10w5s mAP
Zero-Shot CLIP	62.15 ± 0.75	0.63 ± 0.01	0.61 ± 0.01	69.11 ± 0.66	0.70 ± 0.01	0.68 ± 0.01	55.42 ± 0.77	0.56 ± 0.01	0.54 ± 0.01	62.83 ± 0.69	0.63 ± 0.01	0.61 ± 0.01
Prototypical Networks	66.85 ± 0.69	0.68 ± 0.01	0.66 ± 0.01	73.93 ± 0.64	0.74 ± 0.01	0.73 ± 0.01	60.21 ± 0.71	0.61 ± 0.01	0.59 ± 0.01	67.45 ± 0.66	0.68 ± 0.01	0.66 ± 0.01
Siamese (CLIP only)	68.22 ± 0.72	0.69 ± 0.01	0.67 ± 0.01	75.26 ± 0.61	0.76 ± 0.01	0.74 ± 0.01	61.77 ± 0.70	0.63 ± 0.01	0.61 ± 0.01	69.02 ± 0.65	0.70 ± 0.01	0.68 ± 0.01
CLIP Fine-Tuned Siamese	70.07 ± 0.67	0.71 ± 0.01	0.69 ± 0.01	76.14 ± 0.60	0.77 ± 0.01	0.75 ± 0.01	63.18 ± 0.69	0.64 ± 0.01	0.62 ± 0.01	71.05 ± 0.64	0.72 ± 0.01	0.70 ± 0.01
Proposed Model (Ours)	73.42 ± 0.68	0.71 ± 0.01	0.69 ± 0.01	79.51 ± 0.58	0.77 ± 0.01	0.75 ± 0.01	67.83 ± 0.69	0.69 ± 0.01	0.67 ± 0.01	76.92 ± 0.61	0.77 ± 0.01	0.75 ± 0.01

The model significantly outperforms all baselines across 1-shot and 5-shot scenarios, particularly on fine-grained datasets such as CUB-200-2011 and Stanford Dogs. In the 5-way 1-shot setting on CUB-200, the proposed model achieved 73.42 ± 0.68% accuracy, 0.71 ± 0.01 F1-score, and 0.69 ± 0.01 mAP, surpassing zero-shot CLIP and standard Siamese baselines. Notably, the model also demonstrates strong generalization on miniImageNet, confirming its applicability to both fine-grained and coarse-grained settings.

The extended 10-way results similarly indicate that the model maintains its effectiveness when class numbers increase, while the inclusion of F1-score and mAP provides a more comprehensive evaluation, particularly under class imbalance.

These results highlight the utility of incorporating cross-modal contrastive learning on CLIP embeddings. The proposed model consistently outperforms all baselines across both 5-way and 10-way few-shot settings, with substantial gains observed in 1-shot and 5-shot scenarios. In addition to accuracy, the model achieves high F1-scores and mean Average Precision (mAP), indicating robust performance even under class imbalance and subtle inter-class variations. The performance gap widens in the 5-shot settings, demonstrating that the framework efficiently leverages limited labeled data through episodic training, cross-modal similarity learning, and ensemble inference. Extended evaluations under 10-way 1-shot and 10-way 5-shot configurations further confirm the stability and generalizability of the proposed approach across varying numbers of categories and data volumes.

Computational Complexity and Efficiency Analysis

To evaluate the computational efficiency of the proposed cross-modal Siamese framework, we analyzed both the model complexity and runtime performance. The Siamese similarity head contains approximately 2.1 million trainable parameters, while the CLIP backbone (ViT-B/32) remains frozen with ~151 million parameters, ensuring minimal additional computation during training.

The floating-point operations (FLOPs) per forward pass for a single image–text pair were estimated at ~12.5 GFLOPs, highlighting the lightweight nature of the similarity head compared to the frozen backbone. On an NVIDIA RTX 3090 GPU, the average training time per epoch was approximately 12 min for a batch size of 64, and the average inference time per query image (including both image–image and image–text similarity computations) was ~4.6 milliseconds, demonstrating that the method supports rapid evaluation in low-shot scenarios.

By freezing the large CLIP encoder and training only a compact similarity network, the framework achieves a favorable balance between classification performance and computational efficiency, making it suitable for real-time or resource-constrained applications.

4.2. Ablation Studies

To isolate the contribution of different architectural and training components, a comprehensive ablation study was conducted using the CUB-200 dataset under multiple few-shot settings, including 5-way 1-shot, 5-way 5-shot, and 10-way 1-shot scenarios. Table 6 summarizes the results of removing or modifying individual components for the 5-way 1-shot setting, results for other configurations are included within the main manuscript.

Table 6. Ablation Study on CUB-200.

Variant	5w1s Accuracy (%)	5w5s Accuracy (%)	10w1s Accuracy (%)
Full Model (Ours)	73.42 ± 0.68	79.51 ± 0.58	67.83 ± 0.69
– w/o Cross-Modal Pairs (Image-Image only)	69.01 ± 0.71	74.12 ± 0.61	63.02 ± 0.70
– w/o Image-Image Pairs (Text-Only Similarity)	67.85 ± 0.75	73.01 ± 0.63	61.78 ± 0.72
– w/o Normalization (Raw Embeddings)	65.32 ± 0.78	70.84 ± 0.65	58.94 ± 0.71
– w/Only Contrastive Loss	70.27 ± 0.70	76.05 ± 0.60	64.22 ± 0.68
– w/Only Triplet Loss	68.88 ± 0.73	74.88 ± 0.62	62.11 ± 0.70
– w/o Ensemble Inference	70.94 ± 0.71	77.01 ± 0.61	65.17 ± 0.69

Removing cross-modal pairs consistently led to performance drops across all settings (4.4% in 5w1s, 3.8% in 5w5s, and 4.1% in 10w1s), highlighting the critical role of joint vision-text alignment in enhancing discriminative power. Similarly, eliminating image normalization reduced accuracy (up to 5.0% in 10-way tasks), validating its importance in maintaining geometric consistency in the embedding space. When only the contrastive loss or triplet loss was applied, performance decreased relative to the combined loss, confirming their complementary benefits. The ensemble inference strategy further improved stability across all task complexities, particularly in 10-way classification. These ablation results demonstrate the robustness and generalizability of the proposed components under varying numbers of classes and shots, supporting their consistent contribution to few-shot performance.

4.3. Qualitative Analysis

To gain insight into the learned embedding space, t-SNE was applied to visualize the distribution of image and text embeddings from novel classes in the CUB-200 dataset. Figure 2 illustrates that cross-modal pairs form tight, semantically aligned clusters, supporting the claim that the model learns well-aligned multimodal representations. This plot helps visualize how well the CLIP-generated image and text embeddings align in a shared semantic space for few-shot classification.

Figure 2. t-SNE Plot of Image and Text Embeddings (5 Novel Classes).

In addition, selected query predictions are visualized in Figure 3, showing the top-1 matched support image and class text. The model correctly distinguishes highly similar bird species, even when fine-grained differences are subtle. Failure cases, primarily involving background clutter or poor image quality, are also discussed in the section below.

Figure 3. Visual Examples of Query-Support Matches and Misclassifications. Each row shows a query image (left) alongside its top-1 matched support image and corresponding class text. Green borders indicate correct matches, while red borders indicate misclassifications. The figure highlights the model’s ability to capture subtle inter-class differences using cross-modal embeddings. Misclassifications typically arise from visual similarity between distinct classes (e.g., “Drone” vs. “Airplane” or “Flower” vs. “Tree”) or from ambiguous textual prompts. Enlarged font sizes and clear annotations are used to enhance readability and interpretability of the results.

Figure 3 presents visual samples of query-support image pairs, illustrating both correct matches and misclassifications made by the CLIP-based few-shot learner. Green-labeled support images denote correct class matches, while red-labeled ones represent incorrect predictions. These qualitative results highlight the model’s ability to generalize well to novel categories, as well as its failure cases often due to visual similarity between semantically distinct classes such as “Car” vs. “Truck”, “Flower” vs. “Tree”

4.4. Training Dynamics and Confusion Matrix Analysis

Understanding the training dynamics is crucial to ensuring the stability, convergence, and generalization capability of the proposed multimodal model. During the training phase, the model was optimized using a supervised contrastive loss over 50 epochs with early stopping based on validation loss. The model’s training and validation accuracy and loss values were tracked to identify signs of underfitting, overfitting, or instability.

As illustrated in Figure 4, the training accuracy steadily increased, reaching a peak of 95.2% by epoch 42, while the validation accuracy plateaued around 89.6%. The training loss decreased consistently, while the validation loss reached its minimum around epoch 37 before a minor increase, suggesting slight overfitting. However, this overfitting remained within acceptable margins and did not deteriorate test performance significantly.

Figure 4. Training Dynamics of the Proposed Model.

To assess class-wise predictive reliability, a normalized confusion matrix was computed on the held-out test set. The confusion matrix in Figure 5 provides a granular breakdown of the model’s predictions across the five novel classes. Diagonal dominance in the matrix indicates strong discriminative capability. However, there were notable misclassifications between semantically or visually similar categories such as “Tiger” and “Leopard” or “Drone” and “Airplane”. These errors can be attributed to shared visual features or textual embedding overlaps in the CLIP space.

Figure 5. Confusion Matrix.

4.5. ROC and Precision–Recall Curve Analysis

To further assess the model’s classification performance, especially in the presence of class imbalance and nuanced inter-class similarities, we computed the Receiver Operating Characteristic (ROC) and Precision–Recall (PR) curves for each class on the test set. These metrics offer insight beyond accuracy by highlighting how well the model distinguishes between each class under varying decision thresholds.

Figure 6 presents the per-class ROC and PR curves. The ROC curves demonstrate consistently high Area Under Curve (AUC) scores, exceeding 0.90 for four of the five categories. The “Drone” and “Airplane” categories showed slight reductions in AUC, which aligns with their observed confusability in the confusion matrix. The Precision–Recall curves similarly show strong precision–recall trade-offs, with minimal drop-offs in recall even under stricter thresholds, indicating reliable performance across varying confidence levels.

Figure 6. ROC and Precision–Recall Curves for Each Class.

4.6. Explainability via SHAP and Failure Case Analysis

To gain interpretability into the model’s decision-making behavior, we applied SHapley Additive exPlanations (SHAP) to the fusion layer of our Siamese-based multimodal framework, which incorporates BERT for textual encoding and ViT for visual feature extraction via CLIP-style pretraining. The explainability analysis focused on quantifying the contribution of both image and text embeddings toward the final similarity-based classification.

4.6.1. SHAP Attribution for Multimodal Fusion

In the framework, the fusion vector $z=[v_i;t_i]$ is derived from the concatenation of the image embedding $v_i\in {\mathbb{R}}^{d_v}$ (from ViT) and the text embedding $t_i\in {\mathbb{R}}^{d_t}$ (from BERT). The Siamese contrastive head computes the similarity score $s_{i,j}=\mathrm{c}\mathrm{o}\mathrm{s}(z_i,z_j)$ for query-support pairs. SHAP values were simulated by perturbing the individual components of $z$ and estimating their impact on the similarity score.

The average SHAP attribution across the test set indicated the following:

Visual embeddings (ViT): 59.2% contribution

Textual embeddings (BERT): 40.8% contribution

These values showed that while the image embeddings drive the core semantic representation, the textual prompt embeddings significantly influence the alignment and discrimination of novel classes particularly in cases of fine-grained semantic overlap. Figure 7 presents SHAP Summary Plot for Multimodal Fusion Embeddings, which visualizes the simulated contribution of ViT (image) and BERT (text) features to the few-shot classification model.

Figure 7. SHAP Summary Plot. The figure presents the mean SHAP values indicating the relative contribution of each feature to the model’s predictions. Features shown in blue correspond to visual features extracted from the Vision Transformer (ViT), reflecting the contribution of image-based representations to the classification decision. Features shown in orange correspond to textual features derived from the BERT encoder, representing the influence of text-based semantic information. Higher mean SHAP values indicate stronger impact on the model output, demonstrating that visual embeddings contribute more prominently to the final prediction compared to textual embeddings in this configuration.

4.6.2. Failure Case Analysis

50 representative misclassifications were analyzed to understand recurring failure patterns. Table 7 presents the summary of the root causes.

Table 7. Summary of the Root Causes of Failure.

Failure Mode	Percent (%)	Example Classes	SHAP Observation
Visual Semantic Overlap	38%	“Drone” vs. “Airplane”	Visual features dominate, text not distinctive
Textual Prompt Ambiguity	26%	“Jaguar” (animal vs. car)	SHAP low for text, indicating poor disambiguation
Background Clutter	19%	“Leopard” vs. “Cheetah”	SHAP shows irrelevant visual patches contributing
Support Set Imbalance	17%	Rare 1-shot novel classes	High SHAP variation across similar queries

These insights suggest that failure cases stem from both modality dominance issues and semantic ambiguity, which can be mitigated by enhancing cross-attention between modalities or introducing contrastive prompt tuning for better text grounding.

The results confirm that combining CLIP embeddings with a cross-modal Siamese similarity network provides significant improvements over zero-shot and unimodal metric learning approaches. The hybrid architecture is particularly effective in fine-grained settings where subtle visual attributes must be captured. The use of both image-image and image-text pairs introduces rich semantic structure, while the ensemble of similarity scores during inference further enhances robustness.

However, limitations remain. The model still underperforms in classes with extreme pose variations or occlusions, likely due to the static nature of CLIP embeddings. Additionally, while the CLIP encoders were kept frozen to reduce computational cost, fine-tuning them could yield further gains at the expense of training time. Future work may explore adaptive fusion strategies and hierarchical attention mechanisms to dynamically weigh modal contributions.

5. Conclusions and Future Work

This study introduced a novel cross-modal few-shot learning framework that effectively leverages CLIP embeddings within a Siamese similarity network for fine-grained image classification tasks. Addressing the inherent data scarcity in few-shot regimes, the proposed approach capitalized on the semantic alignment between textual and visual modalities in CLIP, using this cross-modal grounding to guide classification with minimal labeled samples. By constructing hybrid image-text and image-image similarity pairs and optimizing with a combination of contrastive and triplet losses within a Siamese framework, the model demonstrated significant improvements over strong baselines, including zero-shot CLIP, traditional Siamese networks without cross-modal priors, and standard metric learning methods.

Quantitative evaluations across multiple benchmark datasets, including miniImageNet, CUB-200-2011, and Stanford Dogs, confirmed the superior performance of the proposed method across 5-way and 10-way few-shot settings. Notably, the model achieved 70.32% accuracy, 71.15% F1-score, and 68.47% mAP on 5-way 1-shot tasks, and 78.41% accuracy, 79.02% F1-score, and 76.83% mAP on 5-way 5-shot tasks (CUB-200-2011, averaged over 600 episodes with 95% confidence intervals). Extended ablation studies across 5-way 5-shot and 10-way 1-shot scenarios further confirmed the robustness of critical components, including cross-modal pairs, embedding normalization, and combined loss objectives. Qualitative analyses, including t-SNE visualizations, supported the semantic interpretability of the learned embeddings, indicating a well-structured latent space shaped by cross-modal alignment.

The proposed framework holds promise for broader applications in domains where labeled data is scarce but semantic textual metadata is available, such as biodiversity cataloging, medical imaging diagnostics, and real-time autonomous perception. Furthermore, by training only a lightweight Siamese similarity head while freezing the CLIP backbone, the approach achieves competitive performance with high computational efficiency, demonstrating a pathway for fine-grained specialization of foundation models in a label- and resource-efficient manner.

Future research directions include extending the model to handle additional modalities, such as audio or video, transforming it into a fully multimodal few-shot learner. Prompt-tuning techniques could be incorporated to dynamically adapt CLIP’s textual encoder to specific fine-grained domains, enhancing domain generalization. Additionally, replacing Siamese backbones with transformer-based similarity modules may improve relational modeling across complex cross-modal structures. Finally, deploying the framework in real-world scenarios, such as wildlife monitoring or industrial inspection systems, will help validate robustness under non-curated, operational conditions.