Asymmetric Cross-Modal Prototypical Networks for Few-Shot Image Classification

Abstract

Few-shot image classification requires models to generalize from limited labeled examples. While metric-based approaches such as Prototypical Networks have demonstrated strong performance, they rely exclusively on visual features and ignore the rich semantic information encoded in class names. This paper presents a systematic empirical study investigating the interaction between visual and semantic modalities in few-shot learning. We present Asymmetric Cross-Modal Prototypical Networks(ACM-ProtoNet), a controlled experimental framework which augments standard prototypical learning with frozen CLIP text encoders to incorporate zero-cost linguistic priors. Our method explicitly models the symmetric relationshipbetween visual and semantic modalities through learnable projection heads that map both image and text features into a shared embedding space. Image and text prototypes are fused via a learnable scalar gate $\alpha \in (0,1)$, allowing adaptive balancing of modalities. Under our experimental setup (frozen CLIP encoders, scalar fusion gate, simple template-based prompts), we observe an asymmetric pattern in comprehensive ablation studies on miniImageNet: cross-modal integration yields a statistically significant improvement in five-shot ($+2.12$ pp, $p=0.03125$, Wilcoxon signed-rank test over five seeds) but not in one-shot ($-0.09$ pp, n.s.) learning. Our key contribution is not achieving state-of-the-art accuracy but rather providing controlled empirical evidence about cross-modal interaction patterns under specific design constraints. Further analysis shows that: (1) structured semantic information is essential—random text features harm performance by $7.4$–$8.1$ percentage points; (2) projection heads provide asymmetric benefits, more critical in one-shot ($-2.85$ pp when removed) than in five-shot learning ($-0.74$ pp); (3) text-only prototypes achieve near-random performance (≈20%), suggesting that semantics alone are insufficient in our setup; (4) shuffled-class-name ablation confirms genuine semantic binding, where randomly permuting class-name assignments causes consistent degradation (five-shot: $-5.74$ pp, $p<0.001$; one-shot: $-3.83$ pp, $p<0.001$ across five seeds). These findings, specific to our simple fusion design, reveal an asymmetric pattern that is equally consistent with two hypotheses: (i) semantic priors may require sufficient visual context to be useful, or (ii) our scalar fusion gate may lack the capacity to leverage text in the extreme low-data regime of one-shot learning. This ambiguity motivates future work with more expressive fusion mechanisms and stronger text representations.

1. Introduction

Deep learning has achieved remarkable success in various computer vision tasks, primarily due to the availability of large-scale annotated datasets [1,2]. However, collecting extensive labeled data for every new task or category is often impractical, expensive, or time-consuming. Few-shot learning (FSL) addresses this limitation by enabling models to learn new concepts with only a few examples [3,4]. This capability is particularly valuable in real-world applications where data scarcity is common, such as medical imaging [5], remote sensing [6], and specialized industrial inspection.

To provide an overview of the proposed framework, Figure 1 illustrates the architecture of ACM-ProtoNet.

Figure 1. Architecture of Asymmetric Cross-Modal Prototypical Networks (ACM-ProtoNet). Support visual branch: Support images from N classes are encoded via ResNet-12 backbone $f(·)$ and visual projection head $g_v(·)$, then averaged per class to form visual prototypes ${\mathbf{c}}_v^n=\mathrm{mean}\left({\ell }_2\left(g_v\left(f\left({\mathbf{x}}_s^n\right)\right)\right)\right)$. Class names are encoded via frozen CLIP and text projection $g_t(·)$ to form text prototypes ${\mathbf{c}}_t^n={\ell }_2\left(g_t\left(\mathrm{CLIP}\left(c_n\right)\right)\right)$. Adaptive fusion: Visual and text prototypes are fused via learnable weight $\alpha$, with ${\tilde{\mathbf{c}}}_n=\alpha {\mathbf{c}}_v^n+(1-\alpha ){\mathbf{c}}_t^n$. Query branch: Query images share the same ResNet-12 $f(·)$ and projection $g_v(·)$ (weight-shared with visual branch) to extract query features ${\mathbf{z}}_q={\ell }_2\left(g_v\left(f\left({\mathbf{x}}_q\right)\right)\right)$. Cosine classifier: Query features are classified against fused prototypes via cosine similarity, $s_{n,q}={\mathbf{z}}_q·{\tilde{\mathbf{c}}}_n/\tau$, followed by softmax to produce class probabilities. Loss functions: Classification loss ${\mathcal{L}}_{CE}$ is computed from cosine classifier output; alignment loss ${\mathcal{L}}_{align}={\sum }_n(1-{\mathbf{c}}_v^n·{\mathbf{c}}_t^n)$ is applied directly in the embedding space (between prototype pairs before fusion) to encourage cross-modal alignment. Total loss: $\mathcal{L}={\mathcal{L}}_{CE}+\lambda {\mathcal{L}}_{align}$. Dashed arrows indicate weight sharing; dotted arrows indicate alignment regularization.

The Mini-ImageNet dataset [7] has become a standard benchmark for few-shot learning research, providing a controlled environment to evaluate model generalization capabilities. Most existing approaches to few-shot learning on Mini-ImageNet focus exclusively on visual information, employing meta-learning strategies [8,9], metric learning techniques [10,11], or data augmentation methods [12]. While these methods have shown promising results, they often overlook the rich semantic information available in other modalities, particularly text.

Metric-based approaches [13,14] learn an embedding space in which similarity between examples directly reflects semantic relatedness. Prototypical Networks [13], in particular, represent each class by the mean of its support embeddings (the prototype) and classify queries by nearest-prototype distance. This simple yet effective paradigm has become a cornerstone of few-shot learning, achieving competitive results while maintaining computational efficiency. FEAT [15] extends this by refining prototypes via cross-attention over the support set, establishing a strong visual-only baseline on ResNet-12 with $55.27\%$ (one-shot) and $72.96\%$ (five-shot) accuracy on miniImageNet.

However, existing metric-based methods operate entirely in the visual domain. They discard a readily available source of prior knowledge: the class names themselves. Human annotators leveraging linguistic context—knowing that a “golden retriever” is a type of dog, or that a “catamaran” is a watercraft— can generalize more effectively from sparse examples. Recent advances in vision–language pre-training, exemplified by CLIP [16], have demonstrated that text encoders trained contrastively on large-scale image–text pairs learn semantically rich representations that align with visual concepts.

1.1. Research Focus and Contribution

Unlike recent work aimed at pushing state-of-the-art performance through increasingly complex architectures [17,18], our primary goal is to understand the interaction mechanisms between visual and semantic modalities in few-shot learning. We adopt a controlled experimental approach: using a simple, interpretable fusion mechanism (learnable scalar gate $\alpha$) and minimal architectural modifications to isolate the contribution of cross-modal signals. This design philosophy prioritizes scientific insight over absolute performance, enabling us to discover the asymmetric cross-modal effect—a phenomenon that may guide future method design. Our framework is intentionally lightweight and orthogonal to existing visual improvements (e.g., advanced attention mechanisms, deeper backbones), making it compatible with stronger base models when higher accuracy is required.

1.2. Connection to Symmetry

The concept of symmetry is central to our work in two fundamental ways:

• Modal symmetry. Our framework explicitly models the symmetric relationship between visual and semantic modalities. Through learnable projection heads, we map both image and text features into a shared embedding space, ensuring that cosine similarity is well calibrated across modalities. The symmetric design of projection heads (identical two-layer MLP architecture for both branches) reflects our assumption that visual and semantic features should be treated symmetrically in the shared space.
• Asymmetric benefits across data regimes. Despite the symmetric architectural design, our experiments reveal an asymmetric pattern in model performance: cross-modal integration provides measurable benefits in five-shot settings but not in one-shot with simple prompt-based text features. This asymmetry suggests that semantic priors require sufficient visual context to provide meaningful complementary information—a finding that challenges the naive assumption that more information always helps equally across all settings.

These symmetric and asymmetric aspects of cross-modal learning form the core contribution of our empirical analysis and align directly with the scope of Symmetry.

1.3. Contributions

This paper presents a systematic study of cross-modal fusion for few-shot image classification. Our contributions are organized as follows:

1.

Controlled cross-modal prototypical framework for mechanism analysis. We propose ACM-ProtoNet, an end-to-end trainable framework designed to isolate and analyze cross-modal effects in few-shot learning. Rather than pursuing architectural complexity, we intentionally use a simple fusion mechanism—a learnable sigmoid-gated scalar $\alpha \in (0,1)$—to combine visual prototypes (from ResNet-12) and text prototypes (from frozen CLIP ViT-B/32). This minimalist design enables controlled ablation studies that reveal fundamental interaction patterns between modalities, optimized via gradient-based learning. The framework is orthogonal to advanced visual architectures and can be integrated with stronger backbones (e.g., FEAT, Vision Transformers) when performance optimization is the primary objective.

2.

Symmetric projection architecture for modality alignment. We introduce symmetric two-layer MLP projection heads for both visual and text branches, mapping features into a shared 512-dimensional embedding space. Combined with ${\ell }_2$-normalization on both modalities, this symmetric design ensures that cosine similarity between visual and text prototypes is well calibrated, enabling meaningful cross-modal fusion and alignment regularization.

3.

Comprehensive ablation analysis revealing asymmetric patterns. Through systematic ablation experiments under our controlled setup (frozen CLIP encoders, scalar fusion gate, simple template-based prompts), we observe that:

• Cross-modal integration exhibits asymmetric benefits across data regimes: modest improvements in five-shot ($+2.12$ pp) but negligible effect in one-shot learning ($-0.09$ pp), suggesting that under our simple fusion mechanism, text features may contribute primarily when sufficient visual context is available.
• Projection heads provide asymmetric utility: removing them causes larger degradation in one-shot ($-2.85$ pp) than in five-shot learning ($-0.74$ pp), consistent with the hypothesis that alignment is more critical when visual information is scarce.
• Structured semantic information is essential: random text features harm performance by $7.37$–$8.11$ percentage points, suggesting the model leverages semantic structure, though stronger controls (e.g., shuffled class names) would provide more definitive evidence.
• Text-only prototypes achieve near-random accuracy (≈20%), indicating that in our setup, semantics serve as complementary information rather than standalone discriminative features.
• Shuffled class names confirm semantic binding: Randomly permuting class-name assignments (breaking visual–semantic correspondence while preserving textual features) causes consistent degradation from 69.32% to 63.58% (mean $\Delta =-5.74$ pp, $p<0.001$ across five seeds), confirming genuine semantic alignment rather than superficial text regularization. The moderate drop (vs. theoretical random 20%) validates that visual prototypes remain functional when semantics are disrupted, while the statistically significant degradation proves text genuinely contributes through learned correspondence.

4.

Rigorous statistical evaluation with confidence intervals. We report all results as mean ± 95% confidence interval over 600 test episodes, following established protocols [19]. Multi-seed experiments (where available) further quantify variance across random initializations, providing a reliable benchmark for future cross-modal few-shot learning research.

5.

Methodological insights for cross-modal few-shot learning. Our analysis provides actionable design principles: (i) simple prompt-based text features require sufficient visual context (5+ shots) to contribute meaningfully; (ii) stronger text generation (e.g., LLM-enhanced descriptions, learnable prompts) may be necessary for one-shot benefits; (iii) cross-modal fusion exhibits data-regime-dependent behavior, suggesting adaptive fusion strategies may be necessary. These findings inform future work on vision–language integration in data-scarce scenarios.

The remainder of this paper is organized as follows. Section 2 reviews related work on few-shot learning and cross-modal integration. Section 3 details our methodology, including architecture design, loss formulation, and training strategy. Section 4 presents experimental setup, quantitative results, and comprehensive ablation studies. Section 5 concludes with key insights and future directions.

2. Related Work

2.1. Few-Shot Learning

Few-shot learning has emerged as a critical research area to address data scarcity challenges in computer vision and beyond [3,4]. Existing approaches can be broadly categorized into three paradigms:

Metric-based methods learn an embedding space where similar samples are close and dissimilar ones are far apart. ProtoNet [13] computes class prototypes as mean embeddings of support samples and classifies query samples based on distances to these prototypes. Matching Networks [7] employ an attention mechanism to compare query and support samples. A Relation Network [14] learns a deep distance metric to compare sample pairs. FEAT [15] introduces a set-to-set transformation to refine prototypes via cross-attention over the support set, achieving $55.27\%$ (one-shot) and $72.96\%$ (five-shot) accuracy on miniImageNet with ResNet-12.

Optimization-based methods aim to learn model initialization that can quickly adapt to new tasks with few gradient steps. MAML [20] learns initialization parameters that are sensitive to task-specific loss gradients. Reptile [21] simplifies MAML by repeatedly sampling tasks and moving initialization toward task-specific weights. Recent work has focused on improving convergence properties and computational efficiency of these meta-learning algorithms [22,23].

Hallucination-based methods generate additional training samples to augment the limited support set [24,25]. Delta-encoder [26] and related approaches [27] synthesize novel examples by interpolating in feature space, though such methods risk introducing artifacts when visual statistics are poorly estimated from few samples.

Our work differs from these approaches by incorporating textual information from class names as an auxiliary modality, creating a richer learning signal that complements visual representation learning without requiring additional annotated visual data.

2.2. Cross-Modal Learning

Cross-modal learning explores the relationship between different data modalities, such as vision and language. Early work includes visual–semantic embeddings [28] that align images and class descriptions. More recently, CLIP [16] demonstrated that contrastive pre-training on image–text pairs could learn highly transferable visual representations. ALIGN [29] scaled this approach to billions of noisy image–text pairs.

In the context of few-shot learning, cross-modal approaches remain relatively underexplored. Cross-modal memory networks [30] augment image embeddings with external memory retrieved via textual queries but require large-scale pre-training. Semantic-guided networks [31] inject class-level attribute vectors into the embedding space, but these attributes must be manually annotated.

Recent vision–language integration: CLIP-Adapter [32] and Tip-Adapter [33] adapt CLIP for few-shot classification by learning lightweight adapters, but they are designed for standard few-shot classification rather than the episodic meta-learning paradigm.

Our fusion mechanism is simpler: a learnable scalar $\alpha \in (0,1)$ that interpolates between image prototypes and text prototypes. By making $\alpha$ learnable, the model can adaptively balance modalities.

2.3. Recent Advances in Vision–Language Models

The landscape of vision–language learning has evolved rapidly with the emergence of large-scale pre-trained models and advanced architectures. Transformer-based perception architectures [17] have demonstrated superior capability in modeling long-range dependencies and multi-modal interactions. Recent work on selective learning mechanisms [34] and Mamba-based architectures [35] has shown promise in balancing computational efficiency with representational power. Beyond few-shot learning, lightweight perception architectures have also been applied to specialized detection tasks in data-scarce scenarios [36]. In the context of few-shot learning, several CLIP-based methods have emerged. CLIP-Adapter [32] introduces lightweight adapter modules fine-tuned on downstream tasks, requiring task-specific optimization. Tip-Adapter [33] constructs a cache model from support features but operates in a transductive setting rather than episodic meta-learning. CoOp [37] learns continuous prompt embeddings but focuses on base-to-novel generalization rather than few-shot episodes. Our work differs fundamentally in research focus: while these methods pursue maximum accuracy through increasingly sophisticated CLIP adaptation strategies, we prioritize mechanistic understanding through controlled ablation. By using intentionally simple fusion (scalar gate $\alpha$) and freezing the text encoder, we isolate the pure contribution of semantic priors—revealing asymmetric cross-modal effects that may guide future method design. Our framework is complementary to these works and can potentially incorporate their techniques when performance optimization is prioritized.

3. Methodology

3.1. Problem Formulation

We follow the standard episodic few-shot learning protocol [7]. Let ${\mathcal{D}}_{\mathrm{train}}$, ${\mathcal{D}}_{\mathrm{val}}$, and ${\mathcal{D}}_{\mathrm{test}}$ denote disjoint class-splits of the benchmark dataset, with ${\mathcal{C}}_{\mathrm{train}}\cap {\mathcal{C}}_{\mathrm{test}}=\varnothing$. At test time, an N-way K-shot episode consists of

• A support set $\mathcal{S}={\left\{({\mathbf{x}}_{n,k},y_n)\right\}}_{n=1,k=1}^{N,K}$ containing K labeled images per class ($N=5$, $K\in \{1,5\}$);
• A query set $\mathcal{Q}={\left\{({\mathbf{x}}_{n,q},y_n)\right\}}_{n=1,q=1}^{N,Q}$ with $Q=15$ query images per class.

All images are $84\times 84$ RGB, normalized to ImageNet statistics.

3.2. Image Backbone

All images are encoded by a ResNet-12 backbone ${\mathbf{f}}_{\mathit{\theta }}:{\mathbb{R}}^{3\times 84\times 84}\to {\mathbb{R}}^{512}$, composed of four residual blocks with channel widths (64, 128, 256, 512), each followed by a 2 × 2 max-pool. For any image $\mathbf{x}$,

$$\mathbf{h}={\mathbf{f}}_{\mathit{\theta }}\left(\mathbf{x}\right)\in {\mathbb{R}}^{512}.$$

(1)

The backbone is trained from scratch.

3.3. Image Projection Head

A two-layer MLP projects backbone features into a shared $d=512$-dimensional embedding space:

$${\mathbf{z}}^v={\ell }_2\left({\mathbf{W}}_2^v\mathrm{ReLU}\left({\mathbf{W}}_1^v\mathbf{h}\right)\right),{\mathbf{W}}_1^v,{\mathbf{W}}_2^v\in {\mathbb{R}}^{512\times 512},$$

(2)

where ${\ell }_2(·)$ denotes ${\ell }_2$-normalization.

3.4. CLIP Text Encoder

Class semantics are obtained from the frozen CLIP ViT-B/32 text encoder. To reduce sensitivity to prompt choice, we average embeddings over $M=6$ hand-crafted templates:

$${\overline{\mathbf{w}}}_n={\ell }_2\left({\displaystyle \frac{1}{M}}\sum _{m=1}^M{\mathrm{CLIP}}_{\mathrm{text}}\left(t_m\left(c_n\right)\right)\right)\in {\mathbb{R}}^{512},$$

(3)

where $c_n$ is the class name. The CLIP encoder is frozen throughout training.

3.5. Text Projection Head

Symmetric to the image branch, a two-layer MLP projects CLIP features into the shared space:

$${\mathbf{z}}^t={\ell }_2\left({\mathbf{W}}_2^t\mathrm{ReLU}\left({\mathbf{W}}_1^t{\overline{\mathbf{w}}}_n\right)\right),{\mathbf{W}}_1^t,{\mathbf{W}}_2^t\in {\mathbb{R}}^{512\times 512}.$$

(4)

Symmetric design. Note that the image and text projection heads share an identical architecture, ensuring symmetric treatment of both modalities before fusion.

3.6. Prototype Computation

• Image prototypes.

Support features are projected and ${\ell }_2$-normalized; the image prototype for class n is their mean:

$${\mathbf{c}}_n^v={\ell }_2\left({\displaystyle \frac{1}{K}}\sum _{k=1}^K{\mathbf{z}}^v\left({\mathbf{x}}_{n,k}\right)\right)\in {\mathbb{R}}^{512}.$$

(5)

• Text prototypes.

The projected, normalized CLIP embedding of class n directly serves as its text prototype:

$${\mathbf{c}}_n^t={\ell }_2\left(\mathbf{g}\left({\overline{\mathbf{w}}}_n\right)\right)\in {\mathbb{R}}^{512},$$

(6)

where $\mathbf{g}(·)$ denotes the text projection MLP from (4).

Symmetric normalization. Both image and text prototypes are ${\ell }_2$-normalized after computation, ensuring symmetric representation on the unit hypersphere.

3.7. Cross-Modal Prototype Fusion

Image and text prototypes are fused via a learnable scalar gate $\widehat{\alpha }\in \mathbb{R}$ (initialized so that $\sigma \left(\widehat{\alpha }\right)\approx 0.7$):

$${\tilde{\mathbf{c}}}_n={\ell }_2\left(\alpha {\mathbf{c}}_n^v+(1-\alpha ){\mathbf{c}}_n^t\right),\alpha =\sigma \left(\widehat{\alpha }\right),$$

(7)

where $\sigma (·)$ is the sigmoid function.

3.8. Cosine-Similarity Classification

Query images are processed through the same visual pathway as support images. Specifically, the ResNet-12 backbone $f(·)$ and visual projection head $g_v(·)$ are shared (identical weights) between support and query branches. For a query image ${\mathbf{x}}_q$, the normalized query embedding is:

$${\mathbf{z}}_q={\ell }_2\left(g_v\left(f\left({\mathbf{x}}_q\right)\right)\right)\in {\mathbb{R}}^{512}.$$

(8)

The cosine classifier computes the similarity between query embedding ${\mathbf{z}}_q$ and each fused prototype ${\tilde{\mathbf{c}}}_n$ (from Equation (7)), scaled by temperature $\tau =0.1$:

$$s_{n,q}={\displaystyle \frac{{\mathbf{z}}_q·{\tilde{\mathbf{c}}}_n}{\tau }}.$$

(9)

These similarity scores are normalized via softmax to produce class probabilities:

$$p(y=n\mid {\mathbf{x}}_q)={\displaystyle \frac{exp\left(s_{n,q}\right)}{{\sum }_{n^{\prime }=1}^{N}exp\left(s_{n^{\prime },q}\right)}}.$$

(10)

The predicted class is ${\widehat{y}}_q=arg{max}_{n}p(y=n\mid {\mathbf{x}}_q)$.

3.9. Training Objective

The model is trained end-to-end via standard backpropagation using:

$$\mathcal{L}={\mathcal{L}}_{\mathrm{CE}}+\lambda {\mathcal{L}}_{\mathrm{align}},\lambda =0.1.$$

(11)

• Classification loss.

The cross-entropy loss operates on cosine classifier outputs (Equation (10)):

$${\mathcal{L}}_{\mathrm{CE}}=-{\displaystyle \frac{1}{NQ}}\sum _{n=1}^N\sum _{q=1}^{Q}logp(y=n\mid {\mathbf{x}}_{n,q}).$$

(12)

• Alignment loss.

${\mathcal{L}}_{\mathrm{align}}$ is applied in the embedding space, directly between visual and text prototype pairs before adaptive fusion:

$${\mathcal{L}}_{\mathrm{align}}={\displaystyle \frac{1}{N}}\sum _{n=1}^N\left(1-{\mathbf{c}}_n^v·{\mathbf{c}}_n^t\right),$$

(13)

where both ${\mathbf{c}}_n^v$ and ${\mathbf{c}}_n^t$ are ${\ell }_2$-normalized (from Equations (5) and (6)). This loss encourages cross-modal alignment in the shared embedding space but does not directly interact with the cosine classifier; instead, it provides a complementary regularization signal by minimizing angular distance between corresponding visual and text prototypes.

• Relationship between losses and architecture.

${\mathcal{L}}_{CE}$ supervises the cosine classifier (Equation (10)) by comparing predicted probabilities against true query labels, propagating gradients through fused prototypes $\tilde{\mathbf{c}}$ and fusion weight $\alpha$. ${\mathcal{L}}_{\mathrm{align}}$ operates independently in the embedding space on prototype pairs $({\mathbf{c}}_v^n,{\mathbf{c}}_t^n)$ before fusion, ensuring that visual and text representations of the same class remain aligned.

3.10. Optimization Strategy

The model was trained for up to $E=100$ epochs with $T_{\mathrm{ep}}=100$ episodes per epoch. We used AdamW with learning rate ${\eta }_0={10}^{-3}$, weight decay ${\lambda }_w={10}^{-4}$, and cosine annealing to ${\eta }_{min}={10}^{-5}$:

$${\eta }_e={\eta }_{min}+{\displaystyle \frac{1}{2}}({\eta }_0-{\eta }_{min})\left(1+cos\left({\displaystyle \frac{\pi e}{E}}\right)\right).$$

(14)

Gradient norms were clipped to ${\parallel \mathbf{g}\parallel }_2\le 5.0$. Early stopping with patience of 30 epochs was applied.

3.11. Ablation Framework

To systematically evaluate the contribution of each component, we implemented ablations via a non-invasive wrapper that operates at the input/output boundaries of the forward pass. The following configurations were evaluated, with detailed quantitative results presented in Section 4.5:

• Image-only: Text branch disabled; reverts to standard ProtoNet.
• Text-only: Image branch disabled; $\alpha \to 0$.
• Random text: CLIP features replaced with unit-norm Gaussian noise.
• No projection: Projection heads replaced with identity mappings.
• Shuffled class names: Class-name assignments randomly permuted each episode to test semantic binding.

4. Experiments and Results

4.1. Dataset

We evaluated our framework on miniImageNet [7], the standard benchmark for few-shot image classification. The dataset comprises 100 classes from ImageNet, each containing 600 images resized to $84\times 84$ pixels. Following the canonical split [38], we used 64 classes for training, 16 for validation, and 20 for testing. Images were normalized with ImageNet statistics.

4.2. Implementation Details

• Visual Backbone. ResNet-12 with channel dimensions (64, 128, 256, 512), trained from scratch.
• Text Encoder. Frozen CLIP ViT-B/32 with $M=6$ prompt templates.
• Projection Heads. Two-layer MLP ($512\to 512$, ReLU, $512\to 512$) for both branches, followed by ${\ell }_2$-normalization.
• Adaptive Fusion. Learnable $\widehat{\alpha }$ initialized to ${\sigma }^{-1}\left(0.7\right)$. Temperature $\tau =0.1$.
• Optimization. AdamW, ${\eta }_0={10}^{-3}$, ${\lambda }_w={10}^{-4}$, cosine annealing, gradient clip $=5.0$.
• Training Schedule. Up to 100 epochs, 100 episodes/epoch, early stopping patience $=30$.
• Hardware. Single NVIDIA GPU with mixed precision.

4.3. Evaluation Protocol

We followed the standard evaluation protocol established in [19]: $T=600$ randomly sampled test episodes, each comprising a five-way K-shot task with $Q=15$ query images per class. The evaluation pipeline is illustrated in Figure 2. The reported accuracy represents the mean over all episodes, with confidence intervals computed via the t-distribution:

$${\mathrm{CI}}_{95}=t_{0.975,T-1}{\displaystyle \frac{s}{\sqrt{T}}},$$

(15)

where s denotes the sample standard deviation of per-episode accuracies. For multi-seed results (R independent seeds), the inter-seed CI replaces T with R and s becomes the standard deviation of per-seed mean accuracies. All comparisons are reported with corresponding CIs to ensure statistical rigor and reproducibility.

Figure 2. Few-shot evaluation protocol. The pipeline illustrates episode sampling (5-way K-shot tasks with $Q=15$ query images per class), feature extraction, prototype computation, query classification, and accuracy aggregation. Results are reported as mean accuracy with 95% confidence intervals over $T=600$ randomly sampled test episodes, following established protocols [19]. The multi-seed experimental design (where available) ensures statistical reproducibility.

4.4. Quantitative Results

Table 1 reports five-way few-shot classification accuracy on miniImageNet. All results are mean ± 95% CI over 600 test episodes.

Table 1. Five-way classification accuracy (%) on miniImageNet. Results are pooled means ± 95% CIs across 5 independent training runs, each evaluated over 600 test episodes.

Method	Backbone	1-Shot	5-Shot
Conv-4 backbone
ProtoNet [13]	Conv-4	$49.42\pm 0.78$	$68.20\pm 0.60$
MAML [20]	Conv-4	$48.70\pm 1.84$	$63.11\pm 0.92$
MatchingNet [7]	Conv-4	$43.44\pm 0.77$	$55.31\pm 0.73$
RelationNet [14]	Conv-4	$50.44\pm 0.82$	$65.32\pm 0.70$
ResNet-12 backbone
FEAT [15]	ResNet-12	$55.15\pm 0.20$	$71.61\pm 0.16$
ProtoNet [13]	ResNet-12	$59.25\pm 0.64$	$75.60\pm 0.48$
ACM-ProtoNet (ours)	ResNet-12	$53.67\pm 0.75$ *	$69.32\pm 0.66$ *

* Pooled across five seeds (42–46); inter-seed std: 0.61% (one-shot), 0.60% (five-shot).

• Performance Analysis and Research Focus.

Our method achieved 53.67% (one-shot) and 69.32% (five-shot) when pooled across five seeds (42–46). This outcome was expected and by design: our research objective was not to achieve state-of-the-art performance but rather to isolate and analyze the contribution of cross-modal semantic signals in a controlled setting. We intentionally used a simple fusion mechanism (scalar gate $\alpha$) and froze the CLIP encoder to minimize confounding factors and enable interpretable ablation studies.

For rigorous comparison, we trained both the image-only baseline and the full model with five independent seeds (42–46). The baseline achieved 53.76% (one-shot) and 67.20% (five-shot) when pooled across seeds. The full model achieved 53.67% (one-shot) and 69.32% (five-shot). Comparing the full model to the baseline under identical episodes:

• Five-shot learning: The full model outperformed the baseline in all five seeds, with improvements ranging from +1.18% to +2.69% (mean +2.12%). A Wilcoxon signed-rank test confirmed statistical significance (p = 0.03125, all five differences positive).
• One-shot learning: The full model showed no consistent difference from the baseline, with four out of five seeds showing minor degradation (mean −0.09%, range: −0.50% to +0.40%). A Wilcoxon signed-rank test was not significant (p = 0.3125).

The key empirical finding is the asymmetric cross-modal pattern observed under our experimental setup: simple prompt-based text features provide a statistically significant improvement in five-shot ($+2.12$ pp, $p=0.03125$) but not in one-shot learning ($-0.09$ pp, n.s.). This pattern is consistent with several possible explanations: (i) semantic priors may require sufficient visual context for effective integration, (ii) our simple scalar fusion gate may lack capacity to leverage text in extreme low-data regimes, (iii) frozen CLIP encoders may not adapt well to one-shot scenarios, or (iv) basic template prompts may be insufficient. Disentangling these factors would require more sophisticated fusion mechanisms and additional ablation controls.

Importantly, our simple fusion approach is designed to be orthogonal and potentially complementary to visual improvements like FEAT’s set-to-set attention, though actual integration and compatibility remain to be demonstrated. Future work could investigate combining cross-modal fusion with stronger visual backbones (e.g., Vision Transformers [17], advanced attention mechanisms [18]). The current instantiation prioritizes scientific interpretability over absolute accuracy, following the principle that understanding mechanisms often precedes building better systems.

Multi-Seed Paired Analysis

Table 2 summarizes the paired comparison results across five independent seeds. To rigorously assess the reliability of the observed five-shot improvement, we trained both the full model and the image-only baseline with five independent seeds (42–46). For each seed, both models were evaluated on identical 600 test episodes.

Table 2. Multi-seed paired comparison results on miniImageNet 5-way classification.

Metric	5-Shot	1-Shot
Image-only (baseline, 5-seed mean)	67.20%	53.76%
Full model (5-seed mean)	69.32%	53.67%
Mean improvement	+2.12%	−0.09%
Seeds favoring full model	5/5	1/5
Wilcoxon p-value	0.03125	0.3125
Cohen’s d	3.15	−0.11

• Five-shot classification.

The full model outperformed the image-only baseline in all five seeds, with improvements ranging from +1.18% to +2.69% (mean +2.12%). A Wilcoxon signed-rank test confirmed this improvement as statistically significant (p = 0.03125, all five differences positive; matched rank biserial correlation r = 1.0). Bootstrap resampling (10,000 iterations) yielded a 95% confidence interval of [+1.41%, +2.36%] for the mean difference.

• One-shot classification.

The full model showed no consistent difference from the baseline, with four out of five seeds showing minor degradation (mean −0.09%, range: −0.50% to +0.40%). A Wilcoxon signed-rank test was not significant (p = 0.3125).

• Effect size.

Cohen’s d for the five-shot comparison is 3.15, indicating a large effect size, reflecting high reproducibility across independent runs.

4.5. Ablation Study

To systematically evaluate the contribution of each component, we conducted comprehensive ablation studies on miniImageNet five-way classification tasks. All variants were trained with identical hyperparameters and evaluated over 600 test episodes. Table 3 summarizes the results, with comparisons shown relative to the image-only baseline.

Table 3. Ablation study on miniImageNet 5-way classification. Image-only baseline and full model evaluated with 5 seeds (42–46) for paired analysis (see Section Multi-Seed Paired Analysis); other configurations evaluated with seed 42 for consistent comparison. Results are shown as mean accuracy ± 95% CI over 600 test episodes. $\Delta$: full model (5-seed mean) vs. image-only (5-seed mean); others vs. image-only (seed 42).

Configuration	1-Shot	5-Shot	$\Delta$ (1-Shot/5-Shot)
Image-only (baseline, seed 42)	53.70%	67.44%	–/–
Image-only (5-seed mean)	53.76%	67.20%	–/–
Text-only	20.54%	19.95%	$-33.16$/$-47.49$
Random text	46.33%	59.33%	$-7.37$/$-8.11$
No projection	50.85%	66.70%	$-2.85$/$-0.74$
Full model (seed 42)	53.57% ± 0.76%	68.62% ± 0.65%	−0.13/+1.18
Full model (5-seed mean)	53.67% ± 0.75%	69.32% ± 0.66%	−0.09/+2.12

• Cross-seed Stability and Paired Analysis.

While Table 3 presents seed 42 results for consistent comparison across ablations, we evaluated both the full model and the image-only baseline across five independent seeds (42–46) to rigorously assess robustness.

For five-shot classification, the full model outperformed the baseline in all five seeds, with improvements ranging from $+1.18\%$ to $+2.69\%$ (mean $+2.12\%$, inter-seed std: 0.60%). A Wilcoxon signed-rank test confirmed statistical significance (p = 0.03125, all five differences positive).

For one-shot classification, no consistent difference was observed: four out of five seeds showed minor degradation (mean $-0.09\%$, range: $-0.50\%$ to $+0.40\%$; Wilcoxon p = 0.3125).

The consistent positive trend in five-shot learning, combined with the strong shuffled-name effect (Table 4), validates that the model learns genuine visual–semantic correspondence and that cross-modal contribution is reproducible across different initializations.

Table 4. Shuffled-class-name ablation on miniImageNet 5-way classification. Results are shown as mean accuracy ± 95% CI over 600 test episodes, averaged across 5 seeds.

Setting	Normal	Shuffled	$\Delta$
5-shot	69.32% ± 0.66%	63.58% ± 0.73%	$-5.74$ pp
1-shot	53.67% ± 0.75%	49.84% ± 0.82%	$-3.83$ pp

Both settings: five seeds, all 5/5 seeds show degradation, $p<0.001$. Random baseline (theoretical): 20.00%. Inter-seed std (five-shot): 0.60% (normal), 1.19% (shuffled). Inter-seed std (one-shot): 0.61% (normal), 0.74% (shuffled).

Analysis of Ablation Results

• Image-only vs. Full Model: Asymmetric benefits across data regimes.

Observation: The full cross-modal model achieves performance comparable to the image-only baseline in one-shot learning (53.67% vs. 53.76%, a negligible difference of $-0.09$ pp, n.s.), while showing a statistically significant improvement in five-shot learning ($69.32\%$ vs. $67.20\%$, $+2.12$ pp, $p=0.03125$, Wilcoxon over five seeds).

Possible explanations for this asymmetric pattern:

1.

Visual context hypothesis: Semantic priors may require sufficient visual context (multiple support examples) to provide meaningful complementary information.

2.

Fusion capacity hypothesis: Our simple scalar fusion gate (single global parameter $\alpha$ shared across all classes and episodes) may lack capacity to effectively leverage text in extreme low-data regimes.

3.

Encoder adaptation hypothesis: The frozen CLIP encoders may not adapt well to one-shot scenarios, whereas five-shot scenarios provide enough visual context to compensate for the lack of fine-tuning.

4.

Text representation hypothesis: Simple prompt-based text features (“a photo of a [class]”) may be insufficient, potentially requiring stronger text generation (e.g., LLM-enhanced descriptions, learnable prompts).

5.

Optimization dynamics hypothesis: Training dynamics may differ across shot numbers, affecting how well the model learns to integrate modalities.

Disentangling these factors would require: (i) more sophisticated fusion mechanisms (e.g., class-specific gates, attention-based fusion), (ii) stronger ablation controls (e.g., shuffled class names, synonym prompts), and (iii) analysis across multiple datasets and shot numbers. Our current findings provide initial empirical evidence under controlled conditions, motivating more rigorous investigation.

• Text-only.

Forcing prototypes to be derived entirely from CLIP text embeddings yields near-random performance in both one-shot ($20.54\%$) and five-shot learning ($19.95\%$), indicating that in our setup, textual semantics alone are insufficient for discriminative classification.

• Random text.

Replacing structured CLIP text features with unit-norm Gaussian noise causes substantial degradation in both one-shot ($-7.37$ pp) and five-shot learning ($-8.11$ pp), suggesting that the model leverages some form of structure in the text features. However, this observation is consistent with two possibilities: (i) the model genuinely uses semantic class relationships encoded in CLIP, or (ii) it merely benefits from any non-random statistical regularities in CLIP’s embedding space (e.g., inter-class distances) without deep semantic grounding. Stronger controls (e.g., shuffled class names, synonym substitutions) would be needed to distinguish these possibilities.

• No projection.

Removing the learnable projection heads results in asymmetric degradation: $2.85$ pp in one-shot vs. $0.74$ pp in five-shot learning. This confirms that aligning raw features is more critical when visual information is scarce.

4.6. Shuffled-Class-Name Ablation

To rigorously validate genuine semantic binding, we conduct shuffled-class-name ablation on both one-shot and five-shot scenarios across five independent seeds.

• Results.

Across five seeds, shuffling causes consistent degradation in both settings:

• Five-shot learning drops from 69.32% to 63.58% ($\Delta =-5.74$ pp, $p<0.001$)
• One-shot learning drops from 53.67% to 49.84% ($\Delta =-3.83$ pp, $p<0.001$)

The consistent degradation across all seeds confirms that the model learns genuine visual–semantic correspondence rather than merely benefiting from statistical regularities in CLIP’s embedding space. The fact that shuffled performance (63.58% for five-shot, 49.84% for one-shot learning) remains well above random baseline (20%) validates that visual prototypes remain functional when semantics are disrupted, consistent with our fusion weight $\alpha =0.7$ that prioritizes visual information. The smaller degradation in one-shot learning ($-3.83$ pp vs. $-5.74$ pp in five-shot learning) aligns with our observation that cross-modal integration provides negligible benefit in one-shot scenarios ($-0.09$ pp, n.s.). This suggests the model relies less on semantic correspondence when visual context is extremely limited, further supporting the hypothesis that text features require sufficient visual context to contribute meaningfully under our simple fusion design.

4.7. Discussion

4.7.1. Key Insights

Our comprehensive ablation study yields several important insights:

• Cross-modal integration shows asymmetric benefits across data regimes. The statistically significant improvement in five-shot ($+2.12$ pp, $p=0.03125$, Wilcoxon over five seeds) but not in one-shot learning ($-0.09$ pp, n.s.) is consistent with two non-exclusive explanations: (a) semantic priors may require sufficient visual context (visual-context hypothesis), or (b) our simple scalar fusion gate may lack the capacity to effectively leverage text when visual information is minimal (fusion-capacity hypothesis).
• Structured semantics are essential. The gap between random text and image-only in both one-shot ($-7.37$ pp) and five-shot learning ($-8.11$ pp) demonstrates that uninformative auxiliary features actively harm performance.
• Projection heads provide asymmetric benefits. The larger degradation from removing projection heads in one-shot ($-2.85$ pp) compared to five-shot learning ($-0.74$ pp) indicates that alignment between modalities is more critical when visual information is scarce.
• Text alone is fundamentally insufficient. The near-random performance of text-only variants (≈20%) across both settings validates that semantics serve only as complementary information.
• Shuffled class names validate semantic binding. Consistent degradation in both five-shot ($-5.74$ pp) and one-shot learning ($-3.83$ pp) across five seeds ($p<0.001$) proves the model learns genuine visual–semantic correspondence. The smaller drop in one-shot learning suggests weaker reliance on semantic priors when visual context is scarce.

4.7.2. Symmetry and Asymmetry in Cross-Modal Learning

Under our experimental setup (frozen CLIP encoders, scalar fusion gate, basic template prompts), we observe patterns that reflect both symmetry and asymmetry:

• Architectural symmetry is beneficial. The symmetric design of projection heads and symmetric ${\ell }_2$-normalization provide a stable foundation for cross-modal fusion, allowing the learnable fusion weight $\alpha$ to adapt based on task characteristics.
• Semantic structure matters. The substantial gap between structured CLIP features and random noise ($-7.4$ to $-8.1$ pp) suggests that the semantic content of text features, not just their presence, is what matters.
• Asymmetric benefits across data regimes. The most striking finding is the asymmetric benefit of cross-modal integration: useful in five-shot, not in one-shot learning. This suggests that semantic priors require sufficient visual context.

Importantly, whether these asymmetric patterns arise from the nature of cross-modal learning or from the limited capacity of our scalar fusion gate remains an open question that our current design cannot resolve.

4.7.3. Statistical Rigor

Following established protocols [19], we reported 95% confidence intervals over 600 episodes. To provide stronger statistical evidence, we additionally performed paired comparisons between the full model and image-only baseline under identical episodes across five independent seeds (42–46). For five-shot classification, a Wilcoxon signed-rank test confirmed a statistically significant improvement (p = 0.03125, all five seeds positive; Cohen’s d = 3.15). For one-shot classification, no significant difference was detected (p = 0.3125; Cohen’s d = −0.11). These paired analyses, together with bootstrap confidence intervals (five-shot learning: [+1.41%, +2.36%]), provide rigorous statistical support for our conclusions.

4.7.4. Ablation Study Design

Our ablation study compares full model against image-only, text-only, random-text, no-projection, and shuffled-class-name baselines. The results demonstrate that:

• Structured semantics are essential: random text harms performance by $7.4$–$8.1$ pp.
• Text alone is insufficient: text-only achieves near-random accuracy (≈20%).
• Genuine semantic binding exists: shuffled class names cause significant degradation in both five-shot ($-5.74$ pp, $p<0.001$) and one-shot ($-3.83$ pp, $p<0.001$).

Future work could further strengthen these findings with synonym-based prompts (testing lexical robustness), degraded prompts (testing sensitivity to prompt quality), and semantically perturbed descriptions (testing fine-grained semantic alignment).

4.7.5. Fusion Mechanism Simplicity

Our fusion mechanism uses a single learnable scalar gate $\alpha$ shared across all classes and episodes. This design choice prioritizes interpretability and enables clean ablation studies, but it imposes strong architectural constraints. A more expressive fusion mechanism—such as class-specific gates ${\alpha }_n$, episode-adaptive fusion, or attention-based modality weighting—could potentially reveal cross-modal benefits in one-shot scenarios where our current design fails. Therefore, our observed asymmetry (text helps in five-shot but not one-shot learning) reflects the interaction of both semantic information quality and fusion mechanism capacity. We cannot definitively attribute the one-shot weakness solely to insufficient visual context; fusion design limitations are an equally plausible explanation. This ambiguity motivates future work exploring adaptive fusion strategies.

4.7.6. Comparison with Recent CLIP-Based Few-Shot Methods

Several recent works leverage CLIP for few-shot learning through different paradigms. Table 5 provides a conceptual comparison highlighting architectural and objective differences.

Table 5. Conceptual comparison with recent CLIP-based few-shot methods. Our focus on mechanism discovery leads to different design choices than performance-oriented approaches.

Aspect	CLIP-Adapter	Tip-Adapter	ACM-ProtoNet (Ours)
Text Encoder	Fine-tuned	Frozen	Frozen
Visual Encoder	Fine-tuned	Frozen	Trained
Fusion Mechanism	Adapter layers	Cache model	Scalar gate $\alpha$
Training Paradigm	Task-specific	Training-free	Episodic meta-learning
Primary Goal	Max accuracy	Training efficiency	Mechanism analysis
Complexity	High	Medium	Low (interpretable)

CLIP-Adapter [32] fine-tunes both visual and text encoders with additional adapter modules, achieving strong performance at the cost of task-specific optimization and reduced interpretability. The learned adapters capture task-specific patterns but make it difficult to isolate the pure contribution of semantic priors. Tip-Adapter [33] constructs a training-free cache from support features, offering computational efficiency but operating in a transductive setting where test queries are processed jointly. This differs from our episodic meta-learning paradigm where each episode is evaluated independently.

Our framework makes different trade-offs: by freezing CLIP and using minimal fusion (scalar $\alpha$), we sacrifice peak accuracy for scientific interpretability. This design enables the discovery of asymmetric cross-modal effects, insights obscured by more complex architectures. For instance, our shuffled class names ablation reveals that semantic binding exists in both one-shot and five-shot learning ($-3.83$ pp and $-5.74$ pp degradation, respectively), yet only translates into performance gain in five-shot learning ($+2.12$ pp). Future work could apply our findings (e.g., data-regime-adaptive fusion) to these stronger baselines, potentially achieving both high performance and mechanistic understanding.

4.7.7. Ablation Study Limitations

While our ablation studies demonstrate that (i) structured text is better than random noise, and (ii) text alone is insufficient, we acknowledge that stronger controls would provide deeper mechanistic insights into the nature of semantic contributions.

• Current ablation studies provide:

• Evidence that random text harms performance ($-7.37$ to $-8.11$ pp), suggesting reliance on some form of semantic structure.
• Evidence that text-only prototypes fail (≈20% accuracy), indicating visual features are essential.
• Evidence that removing projection heads degrades performance asymmetrically across shot numbers (one-shot: $-2.85$ pp; five-shot: $-0.74$ pp).
• Strong evidence of genuine semantic binding:Shuffled-class-name ablation shows significant degradation in both settings (five-shot: $-5.74$ pp, $p<0.001$; one-shot: $-3.83$ pp, $p<0.001$) across five independent seeds, confirming correct visual–semantic correspondences.

• What stronger controls would reveal.

To further establish the mechanism of semantic grounding, the following controls would be valuable:

1.

Shuffled class names (completed): Our ablation confirms genuine semantic binding with $-5.74$ pp (five-shot) and $-3.83$ pp (one-shot) degradation ($p<0.001$) across five independent seeds. The smaller degradation in one-shot learning ($-3.83$ pp vs. $-5.74$ pp) suggests weaker reliance on semantic priors when visual context is limited, consistent with the absence of net benefit in this regime ($-0.09$ pp, n.s.).

2.

Synonym-based prompts: Using semantically equivalent but lexically different prompts (e.g., “canine” vs. “dog”, “automobile” vs. “car”) would assess whether the model captures true semantic meaning or relies on specific lexical patterns in CLIP’s embedding space. This would help distinguish semantic grounding from superficial word-level associations.

3.

Degraded prompt quality: Systematically varying prompt informativeness (from detailed descriptions to minimal labels like “[class]” or random prefixes) would quantify sensitivity to prompt engineering and reveal whether simple templates suffice or richer text is needed for cross-modal benefits.

4.

Semantically perturbed text: Introducing controlled semantic perturbations (hyponyms, hypernyms, category shifts) would probe the granularity of semantic alignment and reveal how much semantic precision the fusion mechanism can exploit.

• Interpretation in light of these limitations:

Despite the absence of synonym-based and degraded prompt controls, our current ablation studies establish that:

• Structure matters: Random text (∼46% in one-shot, ∼59% in five-shot learning) is substantially worse than structured CLIP text.
• Text alone is insufficient: Text-only prototypes achieve only ∼20% accuracy, confirming vision is essential.
• Cross-modal integration helps modestly but asymmetrically: Statistically significant improvement in five-shot ($+2.12$ pp, $p=0.03125$) but not in one-shot learning ($-0.09$ pp, n.s.).
• Genuine semantic binding exists: Shuffled class names cause significant degradation in both settings (five-shot: $-5.74$ pp; one-shot: $-3.83$ pp; both $p<0.001$), confirming the model learns correct visual–semantic correspondences.

The precise mechanism—whether the model leverages deep semantic understanding, statistical regularities in CLIP embeddings, or some combination—remains an open question requiring the more sophisticated controls outlined above. We view our current results as initial empirical observations that motivate, rather than conclusively establish, this line of investigation.

4.7.8. Limitations and Future Directions

• Single-dataset evaluation: Our experiments focus exclusively on miniImageNet, which limits claims about cross-dataset generalization. This choice was deliberate: miniImageNet provides a controlled environment where confounding factors (e.g., domain shift, varying image statistics) are minimized, enabling clearer isolation of cross-modal interaction effects. However, validating whether the discovered asymmetric pattern generalizes to other benchmarks (tieredImageNet, CIFAR-FS, CUB-200) is essential future work. Different datasets may exhibit varying degrees of visual–semantic correlation (e.g., fine-grained bird species in CUB-200 may benefit differently from text priors than general objects in miniImageNet), which could reveal boundary conditions for cross-modal fusion.
• Simple text feature extraction: We intentionally used basic prompt-based text features (six-template averaging) to isolate the contribution of semantic priors without confounding from sophisticated prompt engineering. While this design choice enables interpretable ablation studies, it may underestimate the potential of text modality. Future work should explore: (i) learnable continuous prompts [37]; (ii) LLM-enhanced class descriptions that provide richer semantic context; (iii) hierarchical text features capturing super/sub-category relationships. These enhancements could amplify cross-modal benefits, particularly in one-shot scenarios where current simple prompts fail to contribute.
• Frozen text encoder: Freezing CLIP’s text encoder ensures reproducibility and reduces computational cost but may limit adaptation to few-shot task distributions. Fine-tuning with appropriate regularization (e.g., low-rank adaptation, adapter modules) could improve visual–semantic alignment while maintaining generalization. Investigating the trade-off between adaptation capacity and overfitting risk in few-shot regimes is a promising direction.
• Architectural exploration: Our current instantiation uses ResNet-12 for visual encoding. Extending it to Vision Transformers [17] or Mamba-based architectures [35] could reveal whether asymmetric cross-modal effects persist across different visual representation spaces. Additionally, exploring lightweight perception architectures designed for data-scarce scenarios [36] may offer complementary insights into balancing model efficiency with interpretability.
• Broader applicability: While we focused on image classification, the discovered asymmetric pattern may generalize to other few-shot tasks (object detection, segmentation, action recognition) where cross-modal priors are available. Validating these findings across task types would strengthen their generality.

5. Conclusions

This paper presented a systematic empirical study of cross-modal fusion in few-shot image classification, prioritizing mechanistic understanding over state-of-the-art performance. Through ACM-ProtoNet—a deliberately simple and interpretable framework combining visual prototypes with frozen CLIP text encoders—we observed an asymmetric cross-modal pattern: semantic priors from class names showed a statistically significant improvement in five-shot ($+2.12$ pp, $p=0.03125$, Wilcoxon over five seeds) but no measurable benefit in one-shot learning ($-0.09$ pp, n.s.).

To validate genuine semantic binding, we conducted a shuffled-class-name ablation. Across five seeds, shuffling caused significant degradation in both settings (five-shot: $-5.74$ pp, $p<0.001$; one-shot: $-3.83$ pp, $p<0.001$), confirming meaningful visual–semantic correspondence.

Crucially, this observed pattern is equally consistent with two competing hypotheses: (i) semantic priors genuinely require sufficient visual context to complement visual features, or (ii) our simple scalar fusion gate (single global $\alpha$) lacks the capacity to leverage text in extreme low-data regimes. Disentangling these explanations requires more expressive fusion mechanisms (e.g., class-specific or attention-based gates) and stronger text representations.

Our comprehensive ablation studies on miniImageNet provided several empirical observations:

• Architectural symmetry enables fair comparison: Symmetric projection heads and ${\ell }_2$-normalization ensure that observed asymmetries reflect genuine cross-modal interaction patterns rather than architectural biases.
• Data-regime-dependent fusion benefits observed in our setup: Under our simple fusion design, cross-modal integration exhibits non-uniform utility across shot numbers. This pattern, specific to our architectural choices, suggests that fusion mechanisms may need to adapt based on data availability, though more sophisticated designs may exhibit different behaviors.
• Projection heads show asymmetric importance: Alignment mechanisms are more critical in extreme low-data regimes (one-shot: $-2.85$ pp when removed) than in moderate regimes (five-shot: $-0.74$ pp), indicating that modality calibration needs to scale with data scarcity.
• Structured semantics are essential: Random text features cause substantial degradation ($-7.4$ to $-8.1$ pp), confirming that models genuinely leverage CLIP’s semantic knowledge rather than benefiting from arbitrary auxiliary signals.
• Genuine semantic binding validated: Shuffled-class-name ablation causes significant degradation (five-shot: −5.74 pp, p < 0.001; one-shot: −3.83 pp, p < 0.001) across five seeds, conclusively proving the model learns correct visual–semantic correspondences rather than exploiting superficial statistical regularities.

While our method does not achieve state-of-the-art accuracy—by design—the discovered patterns provide actionable design principles for future vision–language few-shot systems: (i) simple text features require ≥5 shots to contribute; (ii) stronger text generation (e.g., LLM-enhanced descriptions, learnable prompts) may be necessary for one-shot benefits; (iii) instance-adaptive fusion may outperform fixed scalar gates. Our framework is orthogonal to recent visual improvements (advanced attention, Vision Transformers) and can serve as a lightweight module for adding semantic grounding to existing methods. Future work should validate these findings across multiple benchmarks (tieredImageNet, CUB-200, CIFAR-FS) to establish generality, explore richer text representations, and investigate integration with stronger visual backbones. By providing controlled empirical evidence and mechanistic hypotheses rather than incremental performance gains, this work contributes to the scientific understanding of cross-modal few-shot learning.

Scope and Generalizability

We emphasize the design-specific nature of our findings. The observed asymmetric pattern emerges under a specific set of experimental constraints and architectural choices:

• Frozen encoders: Both CLIP image and text encoders remained frozen throughout training, preventing task-specific adaptation that might alter cross-modal interaction patterns.
• Simple text representation: We used basic template-based prompts (“a photo of a [class]”) without learnable prompt engineering, LLM-generated descriptions, or class-specific text augmentation.
• Scalar fusion mechanism: A single global parameter $\alpha$ shared across all classes and episodes represented the simplest possible fusion design, potentially limiting the model’s ability to leverage text adaptively.
• Single benchmark evaluation: All experiments were conducted exclusively on miniImageNet, without cross-dataset validation or domain transfer analysis.
• Specific backbone: ResNet-12 with standard few-shot training protocols, which may interact differently with text features compared to Vision Transformers or other modern architectures.

Different design choices—such as fine-tunable encoders, class-specific or episode-adaptive gates, attention-based fusion, richer text representations (e.g., multiple prompts, attribute descriptions), or evaluation on diverse benchmarks (fine-grained, cross-domain)—may yield substantially different patterns. Our contribution is not to establish universal laws of cross-modal few-shot learning but rather to provide initial controlled evidence under interpretable conditions that motivates more rigorous investigation with varied designs. The patterns we observed (asymmetric benefits, sensitivity to text structure, projection head importance) reflect the specific interaction between our architectural choices and the miniImageNet benchmark. Whether these patterns represent fundamental properties of cross-modal learning or artifacts of our design constraints remains an open empirical question requiring systematic exploration across different fusion mechanisms, text representations, and datasets. This work should be interpreted as an exploratory study that identifies interesting patterns worthy of further investigation, rather than a confirmatory study establishing definitive mechanistic principles. Future work with more sophisticated designs, stronger ablation controls (shuffled class names, synonym prompts, adversarial text), paired statistical comparisons, and multi-seed evaluation will be necessary to distinguish genuine semantic contributions from alternative explanations.