Abstract
Addressing challenges in few-shot image classification, this study introduces the Multi-Dimensional Attention and Composite Loss Former, a meta-learning model built on a Residual Network-12 backbone. The model incorporates multi-dimensional attention mechanisms and is trained with a composite loss function applied across the entire architecture. It enhances feature extraction by dynamically focusing on critical local and global information, while the composite loss optimizes classification accuracy, emphasizes hard samples, suppresses overfitting, and promotes intra-class feature compactness. Comprehensive experiments conducted on the miniImageNet and tieredImageNet datasets demonstrate that the proposed model achieves superior performance in both meta-training and meta-testing stages compared to existing benchmarks, effectively validating its robustness and generalization capabilities in few-shot learning tasks.
1. Introduction
With the advent of deep learning breakthroughs in fields such as image recognition and object detection, traditional deep neural networks have demonstrated outstanding performance on large-scale datasets. However, in many real-world applications, the scarcity of labeled samples often leads to severe overfitting, and the high cost of annotation further exacerbates this issue. Achieving efficient and accurate image classification under extremely limited sample conditions remains an open challenge. To address this problem, few-shot learning has emerged as a research focus: by leveraging prior knowledge and meta-learning strategies, models can rapidly adapt to new tasks with only a handful of examples and generalize effectively [1].
Few-shot learning faces the fundamental difficulty that models struggle to extract robust features from a very small number of examples. Pretrained models, which learn rich and generalizable representations (such as textures, edges, shapes, and semantics) from large-scale datasets, provide powerful priors that can mitigate data sparsity, accelerate convergence, and stabilize training [2]. Nonetheless, directly applying a pretrained model to few-shot tasks still encounters significant challenges. Domain shifts between the pretraining data and the target few-shot tasks often require additional fine-tuning to adapt to the new feature distribution. Moreover, the extremely limited sample size also constrains the model’s ability to capture fine-grained information, making it difficult to attain high discriminative power.
To improve the generalization of pretrained models in few-shot scenarios, researchers have integrated meta-learning frameworks with pretrained backbones. Vinyals et al. [3] proposed matching networks, and Snell et al. introduced prototypical networks [4]; both achieved initial success but did not fully exploit the global priors encoded in pretrained models. Sung et al.’s [5] relation networks captured inter-sample similarities via a learned relation metric, but their element-wise summation of feature maps in the 5-shot setting led to information loss. Finn et al.’s [6] MAML sought a good initialization such that only a few gradient steps are needed for new tasks, but it still suffered from unstable gradients and limited generalization in deeper networks [7]. Ji et al. [8] provided a theoretical analysis of ANIL (Almost No Inner Loop) convergence when only a subset of parameters is adapted, demonstrating efficiency advantages but relying on simplified model assumptions that might not hold in complex architectures. Jiang et al. [9] proposed boosting meta-training with base class information, alternating between class-level information and meta-learning loss to significantly speed convergence and improve generalization; however, its performance gains degrade when base classes are scarce or labels are noisy. Triantafillou et al. [10] constructed meta-dataset, a cross-dataset benchmark enriching evaluation scenarios but still exhibiting dataset biases and task imbalances. Wang et al. [11] introduced an inverted regularization strategy in the inner loop to enhance robustness, but the additional regularization incurred substantial computational overhead.
In summary, existing pretrained models for few-shot image classification have insufficient fine-grained feature capture and limited generalization ability; using a single cross-entropy loss fails to emphasize difficult samples or enforce intra-class compactness. This paper combines pretraining and meta-learning strategies to propose a novel meta-learning network, MACFormer, which employs a multi-dimensional attention mechanism and a composite loss function to significantly improve classification accuracy in few-shot tasks. Our main Innovation and Differentiation are as follows:
- (a)
- Uniqueness of the DualAttention Module: Unlike SE (channel-only) and CBAM (fixed kernel and MLP-based), our DualAttention module is specifically designed to address the feature distribution drift in few-shot scenarios. By replacing redundant fully connected layers with 1 × 1 convolutions, the module not only reduces parameter count by 25% (from 0.12 M to 0.09 M) but also preserves local spatial correlations during channel recalibration, which is crucial for identifying key objects from limited support samples. Furthermore, we position the module after the residual addition to refine the fused features, effectively suppressing background noise that often leads to overfitting in meta-learning. This design shifts the modeling capability from simple feature weighting to adaptive feature mapping and refinement, distinguishing it from incremental variants like CBAM. Meanwhile, the DualAttention module introduces a “spatial convolution kernel with an adjustable size k”. This design allows for a structural configuration of the receptive field based on the semantic level of the feature map: typically, a smaller kernel size (k = 3) is assigned to initial blocks for fine-grained local features, while a larger kernel (k = 7) is used in deeper blocks for global semantic information. This pre-defined structural flexibility effectively addresses the limitation of fixed kernel sizes in traditional attention modules.
- (b)
- Optimization Logic of the Composite Loss Function: Unlike the “simple superposition of multiple losses” in existing studies, the weights of the composite loss in this paper (cross-entropy loss CE, Focal Loss, Label Smoothing loss, Center Loss) (CE = 1.0, Focal Loss = 0.8, Label Smoothing = 0.2, Center Loss = 0.1) are verified through 5 groups of ablation experiments (add new Table 7 “Loss function weight combinations”, which shows that this weight combination achieves the highest accuracy (64.14%) in the miniImageNet 5-way 1-shot task). It can simultaneously address three core needs: “focus on hard samples (Focal Loss), overfitting suppression (Label Smoothing), and intra-class compactness (Center Loss)”.
- (c)
- Efficacy and Robustness Compared to Meta-Baseline: Extensive experiments on standard few-shot benchmarks demonstrate that MACFormer consistently outperforms strong baselines under both 1-shot and 5-shot settings. The proposed model shows stable performance gains on miniImageNet as well as on the more challenging tieredImageNet, which features a larger number of classes and richer semantic hierarchies. These results indicate that MACFormer not only improves classification accuracy in small-sample regimes but also exhibits robust generalization and scalability across datasets of different sizes and semantic complexities.
2. MACFormer Model
To address the insufficient extraction of local key information and the weak adaptation to difficult-to-classify samples in few-shot image classification, we propose MACFormer, a few-shot classification model that optimizes feature extraction and classification through a multi-dimensional attention mechanism and a composite loss function. The model is built on an improved ResNet-12 backbone, which introduces channel and spatial attention mechanisms during the feature extraction stage to enhance representations of both local and global features. In the classification stage, it employs a composite loss function combining cross-entropy, focal loss, label smoothing, and center loss to boost the model’s discriminative power and robustness. Figure 1 illustrates the complete network architecture and training process of the model.
Figure 1.
Meta-learning network structure flow chart.
2.1. Improved ResNet-12 Backbone
Although the traditional ResNet-12 network alleviates the vanishing-gradient problem in deep architectures, its fixed 3 × 3 convolutional receptive fields and hierarchical stacking struggle to capture fine-grained local information in extremely low-sample scenarios.
ResNet-12 contains approximately 1.2 M parameters (compared with 11.7 M for ResNet-18 and 25.6 M for ResNet-50). Because few-shot meta-learning relies on fast, iterative training over many episodic tasks per epoch, the compact footprint of ResNet-12 substantially reduces computational and memory demands. This results in shorter per-epoch runtimes, the ability to process more tasks or larger batches per update, and generally more efficient meta-training—making ResNet-12 a pragmatic choice for small-sample regimes without unduly sacrificing representational capacity. In few-shot tasks, ResNet-12 achieves the optimal cost-effectiveness—its baseline accuracy is approximately 58% in the 5-way 1-shot task on the miniImageNet dataset—which fully verifies its effectiveness Thus, there is no need to over pursue deeper network architectures.
The standard Squeeze-and-Excitation (SE) module only recalibrates features along the channel dimension, ignoring spatial variations and thus failing to balance focus between foreground and background regions [12]. The Convolutional Block Attention Module (CBAM) sequentially combines channel and spatial attention [13], but its numerous fully connected layers introduce parameter redundancy, and its fixed 7 × 7 convolution kernel limits the spatial attention range, reducing complementary advantages in few-shot settings.
To learn a more generalizable, discriminative feature extractor during meta-training and to perform efficient feature extraction on support and query samples during meta-testing, we designed the DualAttention module, thereby improving the traditional ResNet-12 backbone. The DualAttention module employs a lightweight channel-attention mechanism that uses global average pooling and global max pooling to extract channel statistics. These statistics are then processed through two convolutional layers (replacing the traditional fully connected layers), a LeakyReLU activation, and a convolutional layer with an adjustable reduction ratio to generate the channel-attention map. On the channel-weighted feature map, an adjustable spatial-attention mechanism is applied: global average pooling and global max pooling are performed along the channel dimension, and the results are concatenated. A convolutional layer with an adjustable kernel size k then reduces the channel count from two to one to produce the spatial-attention map. By selecting an appropriate k as a structural hyperparameter, the module can be optimized to focus on task-relevant spatial locations across different resolutions. The improved ResNet-12 architecture is shown in Figure 2:
Figure 2.
Improved ResNet-12 network structure diagram.
Dashed arrows in Figure 2 highlight the specific optimizations in DualAttention (convolutional layers in channel attention and tunable kernels in spatial attention) compared to CBAM. The improved ResNet-12 network is observed to enhance the model’s ability to capture key information by introducing a DualAttention module after the addition operations in each residual block. This placement is empirically found to be effective for refining fused features, which helps the model focus on discriminative regions in few-shot tasks. DualAttention has two key differences from traditional CBAM: (i) Parameter and computation optimization. CBAM’s channel attention uses two fully connected layers to achieve channel dimensionality reduction and recovery, whereas DualAttention uses two 1 × 1 convolutions with LeakyReLU non-linearity, avoiding the large number of parameters and dimensional reshuffling of fully connected layers. This results in fewer extra parameters while maintaining comparable performance. (ii) Spatial attention adapts to receptive fields: Unlike CBAM’s fixed 7 × 7 design, DualAttention treats the kernel size k as a tunable structural parameter. This allows the architecture to balance local and global information by employing a stage-aware configuration, where k is dynamically set according to the depth and semantic level of the feature maps in different residual blocks. Subsequent experiments (under identical training conditions) insert SE, CBAM, ECA, and DualAttention into residual blocks and compare accuracy; results show DualAttention outperforms CBAM in the performance trade-off. Let the input feature map be as shown in Equation (1):
where C = number of channels of the feature map (positive integer, C > 0); H = height of the feature map (positive integer, H > 0); W = width of the feature map (positive integer, W > 0). This feature map contains multi-level representations of the image and provides the raw data basis for subsequent attention computations. The channel-attention module first extracts global statistics via adaptive average pooling and adaptive max pooling as given in Equations (2) and (3):
where is the channel descriptor vector obtained by performing global average pooling on the input feature map across the spatial dimensions. Where represents the channel descriptor vector obtained by performing global max pooling on the input feature map , which effectively captures the most salient local features across spatial dimensions.
To learn the dependencies between channels efficiently, we employ a shared multi-layer perceptron (MLP) with a bottleneck structure. Here, and represent the weight matrices of the two fully connected layers, where denotes the reduction ratio. The variable corresponds to the channel dimensionality of the input feature map .
where is the channel descriptor vector obtained by performing global average pooling on the input feature map across the spatial dimensions. is the intermediate channel attention vector output by the max pooling branch. In the channel attention module, represents the reduction ratio, and denotes the ReLU activation function. The and , implemented as two 1 × 1 convolutional layers, are used to learn complex inter-channel dependencies. Dimensionality reduction via effectively decreases computational cost, while the non-linear transformation in the hidden layer extracts refined features. Simultaneously, the use of shared weights ensures that global descriptors from both average pooling and max pooling paths are mapped into a consistent feature space. Finally, the outputs from both paths and are fused via element-wise summation, and the result is normalized by a Sigmoid function to the range [0, 1], thereby generating channel attention weights that effectively recalibrate the original features.
where denotes the channel-attention map, which outputs a channel-wise recalibration weight. As defined in Equation (6), it is computed by the element-wise summation of the transformed average-pooled and max-pooled vectorized descriptors and , effectively fusing global spatial context with the most salient local activations. The Sigmoid activation function then normalizes the fused response into the range [0, 1], assigning each channel a weight that reflects its relative importance for feature enhancement.
The obtained channel-attention map is multiplied with the original feature map element-wise, where is the channel dimension determined by the backbone architecture. This yields the channel-weighted feature map as follows:
where denotes the channel-weighted feature map. As shown in Equation (7), by fusing global descriptors from both average and max pooling and using the shared MLP to extract features, we generate an importance weight for each channel, thereby effectively reweighting the original features.
To focus on spatial information, the feature map is aggregated by computing the average across all channels. This results in the spatial descriptor map , defined in Equations (8) and (9):
where denotes average pooling of along the channel dimension, producing a single-channel map that reflects the overall feature response at each spatial location, and denotes max pooling along the channel dimension, capturing the most salient feature at each spatial location. The two single-channel spatial descriptor maps are then concatenated along the channel dimension as follows:
where represents the concatenation operation in the channel dimension. Equation (11) defines the spatial-attention map . Its explicit role is to capture the spatial dependencies across the aggregated channel information, effectively highlighting where the most discriminative features are located. By applying a convolution layer to the concatenated descriptors, the model learns to enhance foreground objects and suppress irrelevant background noise.
where is the spatial attention map, and denotes a convolutional operation with a kernel size . In the final MACFormer architecture, rather than using a uniform kernel size, we adopt a stage-aware strategy: k is set to 3 in the initial residual blocks to preserve fine-grained local details, and increased to 7 in the deeper blocks to capture broader global semantic relationships. This configuration has been empirically validated to achieve superior performance. The spatial attention map and the features after channel attention processing are weighted element by element as shown in Formula (12):
where is the final feature representation after being modulated by both the channel and spatial attention modules, providing more discriminative features for the subsequent classification or other tasks. The combination of channel and spatial attention ensures that channels and spatial regions critical for classification receive higher weights, reinforcing useful features and suppressing irrelevant noise. This DualAttention design dynamically focuses on salient local and global information, significantly improving the model’s feature representation ability in few-shot scenarios.
2.2. Few-Shot Classification
The improved ResNet-12 network extracts features from meta-test samples, and then feeds the extracted features into the few-shot classification module to achieve precise feature matching and classification. After using the improved ResNet-12 network to extract feature embedding functions, the few-shot classification module constructs class prototypes based on the extracted features. It aggregates the features of samples in the support set to form representative prototype representations for each class. For a query sample, the classification module computes the cosine similarity between its feature and the prototypes of each class and performs classification according to the nearest centroid principle: the class with the highest cosine similarity is regarded as the prediction result. This design fully leverages limited sample information to construct class center representations, endowing the model with stronger adaptability and generalization capabilities in few-shot tasks.
Equation (13) defines the class prototype , which is calculated as the mean feature vector of all support samples in class . The logic behind this is based on the inductive bias of Prototypical Networks, which assumes that there exists an embedding where points cluster around a single prototype representation for each class.
where is the class-center vector for class , is the support-set samples of class , and is the extracted feature vectors. For a query sample , its cosine similarity to each class center is computed as shown in Equation (14):
where denotes cosine similarity, is the dot product, and is the vector norm. The probability distribution in Equation (15) is derived from the Metric-based Meta-learning framework. The temperature parameter (scaling factor) is introduced to scale the cosine similarities before the softmax operation. This ensures that the resulting probability distribution is sufficiently sharp, which helps with enlarging the inter-class margin and is a standard technique in few-shot learning baselines.
where is the probability of class for sample . Here, amplifies the cosine similarities for sharper distinctions. The query is assigned to the class with the highest similarity, effectively implementing a nearest-centroid classification. This design fully leverages limited samples to construct robust class centers, endowing the model with strong adaptability and generalization in few-shot tasks.
2.3. Composite Loss Function
To address common few-shot problems of class imbalance, weak inter-class separability, and insufficient focus on hard samples [14], we design a composite loss function that imposes multiple constraints during training, enhancing the model’s discrimination and generalization.
To clarify the definition of the conditional entropy used in our method, we first define the conditional entropy of the label variable given the input variable as:
where the conditional entropy measures the expected uncertainty of the label variable given the input . It is defined as the negative expectation of the log-likelihood over the data distribution. Intuitively, it quantifies how uncertain the label remains after observing the input: if fully determines , the conditional entropy is zero; otherwise, it increases as the prediction becomes more uncertain.
In Equation (17), the scalar represents the normalization factor across the total number of query samples in a single task. This ensures the loss is independent of the batch size. The indicator function depends on both the sample index and the class index , ensuring that for each sample, only the log-probability of the ground-truth class contributes to the loss.
where is the i-th input sample, is the true label of sample, and is the predicted probability.
In order to solve the problem of the traditional cross entropy loss design struggling to effectively focus on difficult-to-classify samples when facing class imbalance, focal loss is used to emphasize difficult-to-classify samples and mitigate class imbalance. The focal loss function formula is shown in Formula (18):
where is the modulation parameter. Its explicit role is to automatically down-weight the contribution of easy-to-classify samples to the total loss, thereby forcing the model to concentrate on hard samples where is low. As approaches zero for easy samples, the gradients are dominated by the more difficult, misclassified examples.
To prevent the model from becoming overconfident during training—thereby reducing its over-reliance on the training labels and improving robustness—the label-smoothing loss smooths the original one-hot labels. The adjusted label is shown in Formula (19):
where is the smoothing factor and is the total number of classes. By applying label smoothing, the adjusted labels retain the true class information while assigning a nonzero probability to other classes, thereby improving model robustness.
To further enhance feature discriminability, we introduced a center-loss term in the feature space, which encourages samples of the same class to be more tightly clustered. The center loss minimizes the Euclidean distance between each sample’s feature and its corresponding class center . This loss is formulated to reduce intra-class distances in the feature space, thereby improving feature separability. The center loss function is shown in Formula (21):
where the center loss continuously updates the distances between class centers and features , thereby enhancing the clustering of samples within the same class. The composite loss function combines the above loss terms and balances each component’s contribution to the final loss via weighting coefficients. The composite loss is computed as follows in Equation (22):
where , , and are coefficients balancing each part. By guiding the model with these multi-faceted constraints, the training process is optimized from overall accuracy, robustness, hard-sample emphasis, and feature-structure regularization simultaneously. This composite loss significantly improves MACFormer’s few-shot classification performance, enhancing attention to hard examples and intra-class consistency during training.
3. Experiments and Analysis
3.1. Dataset and Evaluation Metrics
To validate MACFormer, we conducted experiments on two widely used few-shot benchmarks derived from ILSVRC-2012: miniImageNet and tieredImageNet. miniImageNet contains 100 classes with 600 color images per class (resized to 84 × 84) and is split following common practice into 64 training, 16 validation, and 20 test classes for meta-training/meta-testing; tieredImageNet is a larger, more semantically hierarchical benchmark with 608 classes grouped into 34 high-level categories, typically split into 351 training, 97 validation, and 160 test classes with roughly 1300 images per class (84 × 84).
In our experimental setup, we evaluate the model on standard N-way K-shot tasks [12]. For each episode, N classes are randomly sampled from the test split, with K examples per class forming the support set, and additional query examples from the same classes forming the query set. We focus on the 5-way 1-shot and 5-way 5-shot settings, where 1 or 5 support samples per class are used, respectively.
We measure classification performance under few-shot conditions using average classification accuracy and the 95% confidence interval as the primary evaluation metrics [13]. By conducting comparative experiments under this protocol, we can effectively verify the actual improvements contributed by the introduced channel/spatial attention mechanisms and the composite loss function in terms of feature extraction and classification accuracy. The evaluation indicators are shown in Formulas (23) and (24):
In Equations (22) and (23), denotes the total number of randomly sampled episodes (tasks) used during the meta-testing phase. To ensure the statistical stability of the results, we define a rigorous experimental protocol for both training and evaluation. During the meta-training stage, the model is trained for 100 epochs, with each epoch consisting of 200 episodes, totaling 20,000 training tasks. Subsequently, in the meta-testing stage, we set the total number of evaluation episodes to T = 2000, which is conducted over 10 test rounds with 200 episodes per round. The final reported performance, including the mean accuracy and the 95% confidence interval, is calculated based on these 2000 meta-testing tasks, represents the classification accuracy for the i-th episode, while the term provides the 95% confidence interval, reflecting the reliability of the reported average accuracy .
3.2. Experimental Details
To ensure a fair and rigorous comparison, all baseline methods and attention variants evaluated in this study utilize the same MACFormer backbone architecture, ensuring a consistent parameter capacity across experiments. Furthermore, we maintain identical training schedules, optimization algorithms, and data augmentation strategies (including random cropping and horizontal flipping) for all comparative models. This unified experimental setup ensures that the observed performance gains are directly attributable to the proposed MACFormer components rather than variations in training protocols or model capacity.
Our experiments were implemented in Python 3.7.3 with PyTorch 1.2.0 and tensorboardX. We used four NVIDIA Tesla T4 GPUs to evaluate the 5-way 1-shot and 5-way 5-shot tasks on miniImageNet. The hyperparameter configuration of our model is summarized in Table 1:
Table 1.
Model hyperparameter configuration.
In order to further improve the feature expression ability, a multi-dimensional attention module is added after the addition operation of each residual block. This module first calculates the channel attention, extracts global statistical information through global average pooling and maximum pooling, and finally generates the channel attention weight. Then, the spatial attention is calculated, and the spatial attention map is generated by averaging and maximum pooling operations on the channel-weighted feature map in the channel dimension. Channel attention and spatial attention act on the feature map in turn, thereby optimizing the feature expression in both the channel and spatial dimensions. After the multi-dimensional attention module extracts more discriminative features, the following composite loss function is used to jointly optimize the network, where CE Loss is used to measure the difference between the category distribution predicted by the model and the true distribution, with a weight of 1.0. Focal loss reduces the weight of easy-to-classify samples and increases the attention to difficult-to-classify samples, with a weight of 0.8, α = 1, and γ = 2. Label Smoothing Loss is used to prevent model overfitting, with a smoothing factor of 0.1 and a weight of 0.2. Center Loss is used to reduce the distribution difference in samples of the same category in the feature space, with a weight of 0.1 and a learning rate of 0.5.
We apply standard data augmentations (random cropping, horizontal flipping, etc.) to improve generalization. The learning rate is initialized at 0.1 and decayed by a factor of 0.1 at the 50th and 90th epochs. We train for 100 epochs in total with a batch size of 128. During training, the feature encoder’s outputs and true labels are input into the composite-loss module for joint optimization. Through backpropagation, the entire network (feature encoder, classifier, and even the class-center vectors in center loss) is updated, yielding more discriminative feature representations.
During the meta-training phase, the model performance is regularly evaluated on the validation set to monitor the training process and guide model selection. To ensure methodological rigor and prevent any potential data leakage, the meta-test set is not accessed during training. TensorBoard is used to record key indicators such as loss values and validation accuracy. The model is saved every five epochs, and the final model configuration is selected based on the optimal performance achieved on the validation set. During the final meta-testing stage, the selected model is evaluated over 200 randomly sampled tasks to calculate the average accuracy and 95% confidence interval. This large-scale evaluation provides a statistically robust understanding of the model’s performance and ensures the stability of the reported results in few-shot learning tasks. Under the same parameter configuration and training strategy, we compare the loss and accuracy of the Meta-Baseline and MACFormer models during meta-training and few-shot meta-testing. Our model was trained and evaluated using a structured meta-learning protocol. During the meta-training stage, we trained the model for 100 epochs, where each epoch consisted of 200 episodes to ensure sufficient gradient updates. In the meta-testing stage, we followed a rigorous evaluation setting by conducting 10 rounds of testing with 200 episodes each, totaling 2000 episodes to provide a robust estimate of the model’s generalization capability. To ensure the reproducibility and transparency of our results, all experiments are conducted with a fixed random seed for both network initialization and episodic sampling. The reported mean accuracy and 95% confidence intervals are derived from a single training run followed by the meta-testing protocol of 2000 episodes, which provides a stable estimation of performance due to the large number of evaluation tasks. We observed that results remained consistent across multiple runs under this unified seed policy. Figure 3, Figure 4 and Figure 5 illustrate these comparisons for 1-shot and 5-shot tasks.
Figure 3.
Meta-training Loss and accuracy experiment comparison. (a) Meta-training Loss, (b) Meta-training classification accuracy. (on miniimagenet).
Figure 4.
Meta-test 1-shot Loss and accuracy experiment comparison. (a) Meta-test 1-shot Loss, (b) Meta-test 1-shot classification accuracy. (on miniimagenet).
Figure 5.
Meta-test 5-shot Loss and accuracy experiment comparison. (a) Meta-test 5-shot Loss, (b) Meta-test 5-shot classification accuracy. (on miniimagenet).
Figure 6, Figure 7 and Figure 8 present the 1-shot and 5-shot training accuracy as well as the corresponding training loss of the model during the meta-training and meta-testing phases on the tieredImageNet dataset, intuitively reflecting the training convergence process and performance of the model under different few-shot task settings.
Figure 6.
Meta-training Loss and accuracy experiment comparison. (a) Meta-training Loss, (b) Meta-training classification accuracy. (on tieredimagenet).
Figure 7.
Meta-test 1-shot Loss and accuracy experiment comparison. (a) Meta-test 1-shot Loss, (b) Meta-test 1-shot classification accuracy. (on tieredimagenet).
Figure 8.
Meta-test 5-shot Loss and accuracy experiment comparison. (a) Meta-test 5-shot Loss, (b) Meta-test 5-shot classification accuracy. (on tieredimagenet).
3.3. Experimental Results and Analysis
From these results, we see that MACFormer consistently achieves higher accuracy than the baseline. In particular, MACFormer significantly improves accuracy in both meta-training and meta-testing. Compared to the Meta-Baseline model, MACFormer increases the 5-way 1-shot meta-training accuracy from 58.91% to 62.68% (a gain of +3.77%) and the 5-way 5-shot meta-training accuracy from 77.76% to 79.11% (+1.35%). In the meta-testing stage, MACFormer raises 1-shot accuracy from 63.17% to 64.14% (+0.97%) and 5-shot accuracy from 79.26% to 80.69% (+1.43%). These results validate that the introduced attention mechanisms and composite loss function enhance feature expressiveness and robustness in few-shot scenarios, enabling stronger generalization under extremely limited data conditions.
To ensure comparability, the three attention mechanisms: SE, CBAM, and ECA, are all integrated with the ResNet-12 backbone after each residual block and compared with the Meta-Baseline/MACFormer under the same parameter configuration and training strategy. As can be seen in Table 2, the meta-test 1-shot/5-shot accuracies for SE-Meta-Baseline, CBAM-Meta-Baseline, and ECA-Meta-Baseline are 63.08 ± 0.33/79.36 ± 0.27; 63.24 ± 0.26/79.41 ± 0.17; and 63.16 ± 0.23/79.45 ± 0.20, respectively, showing limited overall improvement compared to the traditional Meta-Baseline (63.17 ± 0.23/79.26 ± 0.17). In contrast, the newly proposed MACFormer achieves 64.14 ± 0.28/80.69 ± 0.22 under the same experimental settings, indicating that the proposed multidimensional attention combined with composite loss can yield more significant gains in few-shot scenarios.
Table 2.
Comparison of miniImageNet experimental results on different networks.
Table 3 presents the experimental results on the tieredImageNet dataset; the MACFormer series significantly improves the performance of the corresponding baseline models. In the 1-shot task, MACFormer-Classifier (70.07 ± 0.37%) is two percentage points higher than Classifier-Baseline (68.07 ± 0.26%); MACFormer-Meta (71.52 ± 0.42%) is 2.9 percentage points higher than Meta-Baseline (68.62 ± 0.27%).
Table 3.
Comparison of tieredImageNet experimental results on different networks.
In the 5-shot task on the tieredImageNet dataset, both MACFormer-Classifier and MACFormer-Meta achieve excellent performance. MACFormer-Classifier reaches an accuracy of 84.62 ± 0.34%, which is 0.88 percentage points higher than Classifier-Baseline (83.74 ± 0.18%), while MACFormer-Meta achieves an accuracy of 84.58 ± 0.36%, a 0.84 percentage point improvement over Meta-Baseline (83.74 ± 0.18%). Consistent with the performance trend in the 1-shot task, this result further verifies that the proposed multi-dimensional attention mechanism and composite loss function can still effectively enhance feature discriminability and model generalization in few-shot scenarios with a slight increase in sample size. Moreover, the stable performance on the large-scale tieredImageNet dataset with richer semantic hierarchies highlights the robustness and scalability of MACFormer.
3.4. Ablation Study
- (a)
- To evaluate the effectiveness of each module, we perform ablation experiments by selectively removing components from MACFormer while keeping other settings fixed (miniImageNet, 5-way 1-shot/5-shot tasks, average over 200 episodes). The variants are defined as follows (each term’s weight is set to 0 when removed):
① MACFormer-FL: set the focal loss weight to 0;
② MACFormer-LS: set the label smoothing loss weight to 0;
③ MACFormer-CL: set the center loss weight to 0;
④ MACFormer-AM: remove the optimized channel + spatial attention module and only keep the original ResNet12 structure.
Table 4.
Meta-training ablation experiment results in miniImageNet.
Table 5.
Meta-testing ablation experiment results.
The ablation study evaluates the individual causal contributions of the proposed structural and optimization components. We distinguish between meta-training accuracy, which measures the model’s capacity to represent the training distribution, and meta-testing accuracy, which reflects generalization to unseen tasks.
As shown in Table 4, the results quantify the impact of each module on meta-training. Regarding the optimization strategy, disabling the focal loss drops the 1-shot/5-shot accuracy from 62.68%/79.11% to 61.72%/78.92%, confirming its role in prioritizing hard samples. Removing label smoothing causes a decline to 61.19%/78.15%, validating its necessity for preventing over-confident predictions and enhancing robustness. Excluding center loss reduces accuracy to 61.61%/78.76%, which proves its effectiveness in promoting intra-class compactness. Crucially, regarding the structural modification, removing the DualAttention module (MACFormer–AM) leads to a significant performance drop to 58.91%/77.76%. This isolation of effects demonstrates that while the composite loss refines the decision boundaries, the multi-dimensional attention module is the primary driver for capturing discriminative features in few-shot scenarios.
In meta-testing (Table 5), removing each component likewise degrades performance: e.g., without focal loss, accuracy slightly drops to 64.01%/79.55%; without label smoothing, to 63.15%/79.55%; without center loss, to 63.24%/79.48%; and without attention, to 63.17%/79.26%. These results suggest the relative contribution of each component to the overall performance. Empirical evidence indicates that the DualAttention module offers the most substantial gain, while the focal loss, center loss, and label smoothing provide complementary improvements in a descending order of impact. This hierarchy of effects is consistent with our empirical observations under the proposed framework.
Table 6 reports the ablation results on attention insertion positions conducted on the miniImageNet dataset. We evaluated 5-way 1-shot and 5-way 5-shot tasks; for each configuration we assessed performance via multiple random samplings and independent runs, reporting mean ± standard deviation. “Original ResNet-12” denotes the baseline without DualAttention; the other rows show measured performance when DualAttention is inserted before/within/after the residual block, including accuracy, standard deviation, and improvement over the baseline. This ablation examines how attention placement affects performance and provides practical guidance for module integration.
Table 6.
Ablation study on attention placement (miniImageNet).
As shown in Table 6, inserting DualAttention after the residual block yields the best performance: 64.14 ± 0.28% on 5-way 1-shot (an improvement of +0.97% over the baseline 63.17 ± 0.23%) and 80.69 ± 0.22% on 5-way 5-shot (an improvement of +1.43% over the baseline 79.26 ± 0.17%). Placement before or within the residual block also offers modest gains but to a lesser extent. This pattern supports our design rationale: features after the residual connection are more stably fused across layers, allowing attention to better filter discriminative signals and suppress early-stage noise, thus improving few-shot classification robustness and accuracy. The reported standard deviations indicate stable results. ‘—’ indicates the baseline results with no applicable improvement value.
To further explain this performance improvement, we analyzed the effect of attention placement from the perspective of the residual block’s feature fusion mechanism. Residual blocks perform feature fusion and noise suppression via shortcut connections and element-wise (residual) addition. Thus, placing the attention module after the residual block allows the attention to weight the already-fused effective features directly, avoiding spurious attention on pre-residual noisy features or on partially fused features inside the residual block. This observation aligns with the findings of CBAM [17], which report that applying attention after feature aggregation helps focus on task-relevant information. Taken together, the experimental results in Table 5 empirically validate the design choice of inserting the attention module after the residual block and provide direct experimental support for MACFormer’s architecture.
- (b)
- To verify the optimal weight ratio of the composite loss function proposed in the paper for few-shot tasks, this experiment conducts ablation studies with the miniImageNet dataset as an example, designing comparative schemes for 5-way 1-shot and 5-way 5-shot tasks. The experiment focuses on four loss terms (Cross-Entropy Loss CE, Focal Loss, Label Smoothing Loss LS, Center Loss CL), fixes the CE weight at 1.0 (the foundation for ensuring global classification accuracy), and explores the impact of different weight combinations on model performance by adjusting the weights of the other three losses (β = Focal, γ = LS, δ = CL). Finally, the rationality of the weights set in the paper (CE = 1.0, Focal = 0.8, LS = 0.2, CL = 0.1) is verified. The experimental results are shown in Table 7 below.Table 7. Loss function weight combinations (miniImageNet).
The weight design for each loss term is closely aligned with the characteristics of the miniImageNet dataset and few-shot task requirements. The effectiveness of the selected ratios is verified through a structured comparison across specific ranges: 0.4–1.2 (Focal Loss), 0.1–0.3 (Label Smoothing), and 0.05–0.15 (Center Loss). With the CE weight fixed at 1.0 as the primary benchmark, our results indicate that a focal loss weight of 0.8 achieves a robust trade-off between hard-sample mining and global stability. The Label Smoothing weight of 0.2 effectively balances overfitting suppression with label clarity, while the Center Loss weight of 0.1 provides sufficient intra-class constraint without conflicting with the classification objective. These four loss terms work synergistically, forming an empirically effective optimization configuration that yields the highest accuracy and the most stable convergence among the evaluated combinations.
3.5. Parameter Sensitivity Analysis
In our MACFormer architecture, the spatial attention module employs a stage-aware structural configuration rather than a uniform kernel size. This strategic design assigns a smaller kernel (k = 3) to the initial blocks to preserve fine-grained local details and a larger kernel (k = 7) to the deeper blocks to capture global semantic relationships. To verify the effectiveness of this multi-stage approach, we conducted a sensitivity analysis on the miniImageNet dataset. We compared our proposed configuration against models using fixed, uniform kernel sizes (k = 3, 5, 7) across all stages. The experimental results, summarized in Table 8, demonstrate that the multi-stage strategy achieves the highest accuracy, confirming its ability to better adapt to the receptive field requirements of different feature levels.
Table 8.
Performance comparison of uniform vs. multi-stage spatial kernel configurations on miniImageNet.
3.6. Efficiency and Statistical Significance Analysis
To evaluate the computational overhead introduced by the proposed Multi-dimensional Attention and Composite Loss, we conduct a comparative analysis between the vanilla ResNet-12 (Baseline) and our MACFormer. As reported in Table 9, the integration of the DualAttention module only leads to a marginal increase in computational cost. Specifically, the parameter count increases from 8.00 M to 8.04 M (+0.5%), and the FLOPs rise slightly from 2.44 G to 2.48 G. Furthermore, the inference latency only increases by 0.92 ms (from 1.14 ms to 2.06 ms). These results demonstrate that MACFormer significantly enhances feature discriminative power with negligible additional computational burden, striking an excellent balance between performance and efficiency.
Table 9.
Computational efficiency comparison.
To verify the statistical stability and significance of the performance improvements, we report the mean accuracy with 95% confidence intervals (CI). We adopted a structured experimental protocol: during the meta-training stage, the model was trained for 100 epochs, with each epoch consisting of 200 episodes, totaling 20,000 training tasks. Subsequently, in the meta-testing stage, following the standard protocol, we evaluated our model over 10 test rounds with 200 episodes per round, totaling T = 2000 episodes randomly sampled from the test set.
As summarized in Table 10, MACFormer achieves non-overlapping confidence intervals compared to the baseline models in most cases. This gap statistically confirms that the accuracy gains are not due to random fluctuations in task sampling but result from the effective feature recalibration provided by our multi-dimensional attention mechanism and the optimization of the composite loss.
Table 10.
Performance comparison on miniImageNet with 95% confidence intervals.
3.7. Comparison with Existing Attention Variants
To further clarify the architectural novelty of the proposed DualAttention module, we provide a systematic comparison with representative hybrid attention mechanisms, including BAM, GCNet, Coordinate Attention, and Triplet Attention. Table 11 summarizes the key differences in structural design, attention type, kernel strategy, parameter overhead, and insertion position.
Table 11.
Architectural comparison between DualAttention and existing attention variants.
Unlike BAM and GCNet, which rely heavily on global context modeling and introduce additional computational overhead, DualAttention adopts a lightweight dual-path design that separately enhances channel and spatial representations while maintaining low parameter complexity. Compared with Coordinate Attention and Triplet Attention—which encode cross-dimensional dependencies within individual convolutional blocks—our module is inserted after residual feature aggregation, enabling more effective multi-scale feature fusion. While, √ indicates that the corresponding module or setting is utilized, while × indicates it is not included.
Moreover, DualAttention differs from sequential attention pipelines such as CBAM by performing parallel channel–spatial enhancement, which reduces information bottlenecks and improves feature interaction efficiency. This design achieves a better trade-off between representational capacity and computational cost, making it particularly suitable for few-shot meta-learning scenarios where efficiency and generalization are critical.
Although MACFormer can be conceptually related to a Meta-Baseline framework augmented with attention modules, our approach differs in both structural integration and optimization strategy. Instead of directly inserting attention blocks into the backbone network, MACFormer introduces a dual-path feature aggregation mechanism that explicitly aligns multi-scale representations within the meta-learning pipeline. This joint design enables the coordinated optimization of representation enhancement and task-level adaptation, rather than treating attention as an independent plug-in component.
4. Conclusions
This work addresses weak local key-information extraction and limited generalization in few-shot image classification by proposing MACFormer, a ResNet-12–based meta-learning architecture. Our primary contributions are a lightweight DualAttention module inserted after residual additions that adaptively amplifies both local and global discriminative cues, and a composite loss—combining cross-entropy, Focal Loss, Label Smoothing, and Center Loss—applied across the network to jointly improve accuracy, emphasize hard samples, suppress overfitting, and tighten intra-class features. Extensive experiments on standard benchmarks verify that these design choices produce consistent, practically meaningful gains, with the attention module identified as the single largest contributor and the composite loss further stabilizing and enhancing generalization.
While these results demonstrate clear advantages, several practical and theoretical issues remain to be addressed. Efficiency improvements remain necessary for extremely resource-constrained deployments, and adaptive loss-weighting strategies merit exploration to reduce manual tuning. Future work will focus on: (1) designing even lighter attention variants and combining them with optimization/quantization techniques to lower computational and memory costs; (2) investigating adaptive, training-dynamics-driven loss-weighting methods; and (3) extending the approach to cross-domain and multimodal few-shot tasks to validate generality and engineering usability. Overall, MACFormer—through targeted attention design and multi-objective supervision—delivers reproducible, scalable performance improvements for few-shot learning and points clear directions for theoretical refinement and practical deployment.
Author Contributions
Conceptualization, Y.S. and J.L.; methodology, Y.S. and W.C.; software, Y.S.; validation, W.C. and S.L.; formal analysis, Y.S. and W.C.; investigation, Y.S. and W.C.; resources, J.L.; data curation, W.C.; writing—original draft preparation, Y.S.; writing—review and editing, W.C. and J.L.; visualization, W.C.; supervision, J.L. and S.L.; project administration, J.L.; funding acquisition, J.L. All authors have read and agreed to the published version of the manuscript.
Funding
This work was supported by the National Key Research and Development Program of China (Grant No. 2023YFC3304604).
Data Availability Statement
The data presented in this study are available in publicly accessible repositories. The miniImageNet and tieredImageNet datasets can be found at: https://www.kaggle.com/datasets/arjunashok33/miniimagenet (accessed on 14 January 2026); https://www.kaggle.com/datasets/arjun2000ashok/tieredimagenet (accessed on 14 January 2026).
Conflicts of Interest
The authors declare no conflict of interest.
Correction Statement
This article has been republished with a minor correction to the Funding statement. This change does not affect the scientific content of the article.
References
- Hospedales, T.; Antoniou, A.; Micaelli, P.; Storkey, A. Meta-learning in neural networks: A survey. IEEE Trans. Pattern Anal. Mach. Intell. (TPAMI) 2020, 44, 5149–5169. [Google Scholar] [CrossRef] [Scilit] [PubMed]
- Radford, A.; Kim, J.W.; Hallacy, C.; Ramesh, A.; Goh, G.; Agarwal, S.; Sastry, G.; Askell, A.; Mishkin, P.; Clark, J.; et al. Learning transferable visual models from natural language supervision. In Proceedings of the 38th International Conference on Machine Learning (ICML), Virtual, 18–24 July 2021; pp. 8748–8763. [Google Scholar]
- Vinyals, O.; Blundell, C.; Lillicrap, T.; Wierstra, D. Matching networks for one shot learning. In Proceedings of the Advances in Neural Information Processing Systems (NeurIPS), Barcelona, Spain, 5–10 December 2016. [Google Scholar]
- Snell, J.; Swersky, K.; Zemel, R.S. Prototypical networks for few-shot learning. In Proceedings of the Advances in Neural Information Processing Systems (NeurIPS), Long Beach, CA, USA, 4–9 December 2017. [Google Scholar]
- Sung, F.; Yang, Y.; Zhang, L.; Xiang, T.; Torr, P.H.; Hospedales, T.M. Learning to compare: Relation network for few-shot learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Salt Lake City, UT, USA, 18–23 June 2018; pp. 1199–1208. [Google Scholar]
- Finn, C.; Abbeel, P.; Levine, S. Model-agnostic meta-learning for fast adaptation of deep networks. In Proceedings of the 34th International Conference on Machine Learning (ICML), Sydney, Australia, 6–11 August 2017; pp. 1126–1135. [Google Scholar]
- Antoniou, A.; Edwards, H.; Storkey, A. How to train your MAML. In Proceedings of the International Conference on Learning Representations (ICLR), New Orleans, LA, USA, 6–9 May 2019. [Google Scholar]
- Ji, K.; Lee, J.D.; Liang, Y.; Poor, H.V. Convergence of meta-learning with task-specific adaptation over partial parameters. In Proceedings of the Advances in Neural Information Processing Systems (NeurIPS), Virtual, 6–12 December 2020. [Google Scholar]
- Jiang, W.; Liu, G.; He, D.; He, K. Boosting meta-training with base class information for few-shot learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Seattle, WA, USA, 16–22 June 2024; pp. 2341–2350. [Google Scholar]
- Triantafillou, E.; Zhu, T.; Dumoulin, V.; Lamblin, P.; Evci, U.; Xu, K.; Goroshin, R.; Gelada, C.; Swersky, K.; Manzagol, P.-A.; et al. Meta-dataset: A dataset of datasets for learning to learn from few examples. In Proceedings of the International Conference on Learning Representations (ICLR), Addis Ababa, Ethiopia, 30 April 2020. [Google Scholar]
- Wang, L.; Zhou, S.; Zhang, S.; Chu, X.; Chang, H.; Zhu, W. Improving generalization of meta-learning with inverted regularization at inner-level. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Vancouver, BC, Canada, 17–24 June 2023; pp. 16123–16132. [Google Scholar]
- Hu, J.; Shen, L.; Sun, G. Squeeze-and-excitation networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Salt Lake City, UT, USA, 18–23 June 2018; pp. 7132–7141. [Google Scholar]
- Woo, S.; Park, J.; Lee, J.Y.; Kweon, I.S. CBAM: Convolutional block attention module. In Proceedings of the European Conference on Computer Vision (ECCV), Munich, Germany, 8–14 September 2018; pp. 3–19. [Google Scholar]
- Rumelhart, D.E.; Hinton, G.E.; Williams, R.J. Learning representations by back-propagating errors. Nature 1986, 323, 533–536. [Google Scholar] [CrossRef] [Scilit]
- Chen, W.Y.; Liu, Y.C.; Kira, Z.; Wang, Y.C.F.; Huang, J.B. A closer look at few-shot classification. In Proceedings of the International Conference on Learning Representations (ICLR), New Orleans, LA, USA, 6–9 May 2019. [Google Scholar]
- Mishra, N.; Rohanian, M.; Heinrich, H.; Sugiyama, M. Meta-learning with temporal convolutions. In Proceedings of the International Conference on Learning Representations (ICLR), Vancouver, BC, Canada, 30 April–3 May 2018. [Google Scholar]
- Munkhdalai, T.; Yu, H. Learning rapid-temporal adaptations. In Proceedings of the 35th International Conference on Machine Learning (ICML), Stockholm, Sweden, 10–15 July 2018; pp. 3620–3629. [Google Scholar]
- Sun, Q.; Liu, Y.; Chua, T.S.; Schiele, B. Meta-transfer learning for few-shot learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Long Beach, CA, USA, 15–20 June 2019; pp. 403–412. [Google Scholar]
- Lee, K.; Maji, S.; Ravichandran, A.; Soatto, S. Meta-learning with differentiable convex optimization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Long Beach, CA, USA, 15–20 June 2019; pp. 10649–10657. [Google Scholar]
- Li, A.; Huang, W.; Lan, X.; Feng, J.; Li, Z.; Wang, L. Boosting few-shot learning with adaptive margin loss. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Seattle, WA, USA, 13–19 June 2020; pp. 12573–12581. [Google Scholar]
- Zhang, T.; Dai, W.; Chen, Z.; Yang, S.; Liu, F.; Zheng, H. Few-shot image classification via mutual distillation. Appl. Sci. 2023, 13, 13284. [Google Scholar] [CrossRef] [Scilit]
- Luo, X.; Wu, H.; Zhang, J.; Gao, L.; Xu, J.; Song, J. A closer look at few-shot classification again. In Proceedings of the 40th International Conference on Machine Learning (ICML), Honolulu, HI, USA, 23–29 July 2023; pp. 23103–23123. [Google Scholar]
- Chen, Y.; Liu, Z.; Xu, H.; Darrell, T.; Wang, X. Meta-baseline: Exploring simple meta-learning for few-shot learning. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), Montreal, QC, Canada, 10–17 October 2021; pp. 9062–9071. [Google Scholar]
Disclaimer/Publisher’s Note: The statements, opinions and data contained in all publications are solely those of the individual author(s) and contributor(s) and not of MDPI and/or the editor(s). MDPI and/or the editor(s) disclaim responsibility for any injury to people or property resulting from any ideas, methods, instructions or products referred to in the content. |
© 2026 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license.







