Token Injection Transformer for Enhanced Fine-Grained Recognition

Abstract

Fine-Grained Visual Classification (FGVC) involves distinguishing highly similar subordinate categories within the same basic-level class, presenting significant challenges due to subtle inter-class variations and substantial intra-class diversity. While Vision Transformer (ViT)-based approaches have demonstrated potential in this domain, they remain limited by two key issues: (1) the progressive loss of gradient-based edge and texture signals during hierarchical token aggregation and (2) insufficient extraction of discriminative fine-grained features. To overcome these limitations, we propose a Gradient-Aware Token Injection Transformer, a novel framework that explicitly incorporates gradient magnitude and orientation into token embeddings. This multi-modal feature fusion mechanism enhances the model’s capacity to preserve and leverage critical fine-grained visual cues. Extensive experiments on four standard FGVC benchmarks demonstrate the superiority of our approach, achieving 92.9% top-1 accuracy on CUB-200-2011, 90.5% on iNaturalist 2018, 93.2% on NABirds, and 95.3% on Stanford Cars, thereby validating its effectiveness and robustness.

1. Introduction

With the rapid progress of computer vision, visual classification has achieved remarkable success. Traditional methods aim to identify distinct basic-level categories such as cats, dogs, or cars, which typically exhibit substantial visual differences (Figure 1a). In contrast, fine-grained visual classification (FGVC) targets more challenging problems by distinguishing subordinate categories within the same basic-level class—for example, identifying different species of birds, breeds of dogs, or types of aircraft or flowers (see Figure 1b). This task is particularly challenging due to its low inter-class variance and high intra-class variance. Subtle differences such as feather patterns, beak shapes, or petal textures must be detected, which are often overlooked by conventional feature extraction approaches.

Recent fine-grained visual classification (FGVC) methods can be broadly categorized into part localization–based approaches and attention-based approaches. Part localization methods explicitly detect and crop discriminative regions using predefined or learned rectangular anchors [1,2]. Although effective in isolating salient parts, these approaches often suffer from imprecise localization, inclusion of irrelevant background regions, and limited modeling of inter-part relationships, especially when discriminative cues are subtle or spatially fragmented.

Attention-based approaches alleviate the need for extra annotations by directly highlighting informative regions through learned attention mechanisms. In particular, Vision Transformers (ViTs) [3] dynamically reweight patch tokens via self-attention to capture global context and part-level importance [4,5,6]. Representative works include TransFG [7], which employs counterfactual attention to suppress background clutter and refine part-aware representations; MetaFormer-style models [8,9,10], which inject auxiliary meta-information into visual tokens to enhance semantic discrimination; and dual- or multi-stream fusion methods [11,12], which integrate features across layers or streams to improve representational capacity while remaining annotation-efficient. More recently, MPSA [13] introduces part sampling attention to enhance multi-granularity representations and implicitly adapt to object boundaries.

Despite these advances, we argue that existing FGVC methods still face fundamental limitations when distinguishing visually similar categories. Specifically, they predominantly rely on high-level semantic attention or part activation, while largely overlooking the explicit modeling of low-level gradient information that encodes fine-grained edge structures and subtle texture variations. ViT partitions input images into non-overlapping patches and converts them into tokens, which are then fed into stacked self-attention layers for global aggregation. The core issue leading to feature gradient attenuation lies in two key steps:

1.

Patch Tokenization Step: Image feature gradients (capturing fine-grained spatial transitions, e.g., the edge of a car grille) are inherently defined at the pixel level. When converting pixels into fixed-size patch tokens, the local spatial gradient information is averaged within each patch, leading to an initial loss of high-frequency gradient details.

2.

Self-Attention Aggregation Step: The self-attention mechanism fuses token features based on similarity scores, prioritizing global semantic consistency over local spatial gradient preservation. For stacked attention layers, this global fusion further smooths the spatial gradient information embedded in tokens: as tokens are repeatedly aggregated across layers, the distinct gradient cues that distinguish adjacent regions (e.g., between a headlight and the car body) are progressively diluted, resulting in attenuation of image content feature gradients.

As a result, even when attention successfully localizes relevant regions, the underlying representations may lack sufficient fine-grained detail to reliably separate closely related categories.

To address these challenges, we propose the Token Injection Transformer for Enhanced Fine-Grained Representation, which explicitly models image gradient magnitude and orientation as complementary cues to visual appearance. By extracting gradient information and encoding it into gradient tokens within the same embedding space as visual tokens, the proposed framework enables direct interaction between visual semantics and gradient-based structural information. This design preserves fine-grained edge and texture cues throughout the Transformer hierarchy, thereby enhancing the discriminative capacity of the model for FGVC tasks.

The main contributions of this work are summarized as follows:

1.

We provide an in-depth analysis of how multi-layer token aggregation in Vision Transformers progressively weakens gradient-based edge and texture information, highlighting an underexplored limitation of existing attention-based FGVC models.

2.

We propose a token injection mechanism that encodes gradient magnitude and orientation into learnable tokens and integrates them with visual tokens via Transformer attention, enabling effective visual–gradient feature interaction without additional annotations.

3.

Extensive evaluation on four standard FGVC benchmarks shows that the proposed Gradient-Aware Token Injection Transformer establishes new state-of-the-art accuracies of $92.9\%$, $90.5\%$, $93.2\%$, and $95.3\%$ on CUB-200-2011, iNaturalist 2018, NABirds, and Stanford Cars, respectively, including a record-setting result on the NABirds benchmark.

2. Related Work

2.1. Fine-Grained Visual Classification

Fine-Grained Visual Classification (FGVC) aims to distinguish subordinate categories within the same basic object class (e.g., bird species or vehicle models), where inter-class differences are extremely subtle and models must focus on local discriminative features. Early approaches relied heavily on manually annotated parts or bounding boxes to guide the network’s attention to salient regions. While part-based annotation schemes provide precise localization of discriminative regions, the high annotation cost limits their scalability on large datasets.

To reduce dependence on manual annotations, subsequent studies have introduced self-supervised and weakly-supervised strategies. NTS-Net [1] employs a self-supervised mechanism for region proposal without part annotations; Bilinear CNN [14] uses outer-product operations to capture higher-order feature interactions; HBP [15] and DBTNet [2] enhance semantic representations via cross-layer bilinear structures and deep bilinear transformations, respectively. More recently, CAP [16] introduces context-aware attention pooling to improve semantic aggregation, and TransFG [7] integrates a part selection module within the Transformer framework to dynamically filter key tokens and boost classification accuracy.

In parallel, researchers have explored the incorporation of auxiliary modalities to enrich feature representations. KERL [8] fuses visual features with knowledge-graph nodes via graph neural networks; Presence-Only [9] integrates spatial-temporal priors during training; MetaFormer [10] combines textual descriptions, geolocation, and attribute information to generate more discriminative composite embeddings. Despite these advances, challenges remain in suppressing background noise and extracting the most representative fine-grained cues. To address this limitation, our work introduces Gradient Tokens, which explicitly encode high-frequency texture details and fuse them with semantic and spatial-temporal information through cross-modal attention, thereby enhancing discriminative power while preserving spatial coherence.

2.2. Transformer Architectures and Optimizations in Vision

The Transformer architecture, originally proposed by Vaswani et al. [17] for natural language processing, has become a dominant backbone due to its powerful global modeling capabilities. Dosovitskiy et al. [3] first adapted it to vision with the Vision Transformer (ViT), which partitions an image into fixed-size patches treated as tokens, achieving performance on par with or exceeding that of convolutional neural networks.

However, the quadratic complexity of global self-attention on high-resolution images has spurred numerous optimizations. ResMLP [18] replaces attention layers with pure MLP blocks to reduce computation, albeit at the expense of spatial modeling capacity; MetaFormer [19] explores pooling operations as substitutes for attention, which can lead to information loss; FourierFormer [6] seeks a balance between low- and high-frequency components but struggles to simultaneously capture both. Inspired by dual cross-attention mechanisms [5], some methods enhance interactions between global context and local high-response regions, while others [4,12] select salient tokens based on attention weights to capture subtle differences—yet often overlook explicit modeling of high-frequency textures.

Beyond attention efficiency, recent studies in related vision tasks have highlighted the importance of explicitly modeling structural cues and inductive priors to improve visual representations. For example, CausalSR [20] introduces a structural causal model with counterfactual inference to disentangle structural information from spurious correlations in super-resolution tasks, demonstrating the benefits of incorporating structured priors into deep vision models. Although targeting a different task, such findings underscore the value of explicitly encoding structural information for enhanced representation learning.

To fill this gap, we propose the Gradient Token Extraction (GTE) module within the Transformer architecture. GTE directly extracts gradient-based descriptors from feature maps and employs cross-attention with the original spatial tokens, thereby maintaining global semantic consistency while more accurately capturing fine-grained texture variations and significantly improving classification performance.

3. Methodology

The overall architecture of the proposed Gradient-Aware Token Injection Transformer is illustrated in Figure 2. Given an input image, a convolutional backbone is first employed to extract dense feature maps, which are then converted into a sequence of image tokens through overlapping patch embedding to preserve local spatial continuity. However, standard tokenization schemes mainly encode appearance statistics and tend to suppress first-order spatial variations during successive feature aggregation. To explicitly preserve fine-grained gradient cues—such as edge contours and subtle texture transitions—that are critical for fine-grained recognition, we introduce a Gradient Token Extraction (GTE) module.

The extracted gradient tokens are subsequently injected into the Transformer via a Token Fusion Unit (TFU), where they interact with image tokens and a learnable class token through multi-head self-attention. A two-stage fusion strategy is adopted to progressively refine token representations, and the class tokens from different stages are finally aggregated to produce the fine-grained classification output.

3.1. Image Tokenization

Fine-grained visual recognition requires preserving subtle local patterns while maintaining spatial coherence for subsequent global reasoning. Given an input image

$$X\in {\mathbb{R}}^{H\times W\times 3},$$

(1)

we first extract hierarchical visual features using a convolutional backbone $G(·)$:

$$Z=G\left(X\right)\in {\mathbb{R}}^{B\times D\times H_1\times W_1},$$

(2)

where B denotes the batch size, D is the channel dimension, and $H_1,W_1$ are the spatial resolutions of the backbone output. The convolutional backbone consists of a shallow convolutional stem followed by Mobile Inverted Bottleneck Convolution (MBConv) blocks equipped with squeeze-and-excitation modules, enabling efficient local feature extraction with adaptive channel-wise modulation.

This backbone is custom-designed for our framework and does not correspond to standard architectures such as EfficientNet or ConvNeXt. Its structure is fixed across all experiments and serves solely to introduce local convolutional inductive biases prior to Transformer-based global reasoning.

To transform the dense feature map Z into a sequence representation suitable for Transformer processing, we employ an overlapping patch embedding operator implemented via convolution. Using a kernel size k, stride $s<k$, and appropriate padding, the operator performs spatial downsampling and feature projection simultaneously:

$$T=P\left(Z\right)\in {\mathbb{R}}^{B\times L\times d},$$

(3)

where d denotes the embedding dimension and $L=H_t·W_t$ is the number of patch tokens arranged on a spatial grid of size $H_t\times W_t$. Compared with non-overlapping partitioning, overlapping embeddings better preserve boundary continuity and fine texture variations, which are essential for distinguishing visually similar categories.

3.2. Gradient Tokenization

Although patch embeddings encode local appearance information, they do not explicitly emphasize first-order spatial variations such as edges or contour transitions. To enhance sensitivity to such structural cues, we derive gradient-aware tokens directly from the patch embeddings.

Specifically, the patch token sequence T is reshaped back into a spatial feature map:

$$\tilde{T}=Reshape\left(T\right)\in {\mathbb{R}}^{B\times d\times H_t\times W_t},$$

(4)

which restores the two-dimensional neighborhood structure required for spatial derivative estimation.

We then compute channel-wise horizontal and vertical gradients using a single, fixed Sobel operator implemented as depth-wise convolutions. The Sobel kernels are defined as

$$K_x=\left[\begin{array}{ccc}1& 0& -1\\ 2& 0& -2\\ 1& 0& -1\end{array}\right],K_y=K_x^{\top }.$$

(5)

Applying these kernels independently to each channel yields

$$D_x=\mathcal{G}(\tilde{T};K_x),D_y=\mathcal{G}(\tilde{T};K_y),$$

(6)

where $\mathcal{G}(·)$ denotes depth-wise convolution with unit padding. The horizontal and vertical responses jointly encode gradient magnitude and directional information and are concatenated along the channel dimension:

$$G=Concat(D_x,D_y)\in {\mathbb{R}}^{B\times 2d\times H_t\times W_t}.$$

(7)

Flattening the spatial dimensions produces an initial sequence of gradient descriptors:

$$G^{♭}\in {\mathbb{R}}^{B\times L\times 2d},$$

(8)

where $L=H_t·W_t$ corresponds to the number of spatial locations and aligns with the number of visual patch tokens. To align descriptor dimensionality with the original patch embeddings, a learnable linear projection is applied:

$$T_g=G^{♭}{W}_g,W_g\in {\mathbb{R}}^{2d\times d},$$

(9)

resulting in projected gradient tokens $T_g\in {\mathbb{R}}^{B\times L\times d}$.

To reduce redundancy and suppress noisy responses while retaining spatially consistent structural patterns, we further apply a lightweight one-dimensional convolution along the token dimension:

$${\tilde{T}}_g=Conv1d\left(T_g\right)\in {\mathbb{R}}^{B\times N\times d},$$

(10)

where $N<L$ is a predefined hyperparameter that controls the number of gradient tokens retained. This design balances gradient expressiveness and computational efficiency without excessive token growth. By default, N is fixed for a given backbone configuration; specific values used in our experiments are reported in the implementation details and ablation studies.

3.3. Token Fusion Unit

Different token types capture complementary visual cues: patch tokens encode local appearance, gradient tokens emphasize structural transitions, and a learnable class token aggregates global semantic context. To jointly exploit these heterogeneous representations, we introduce a Token Fusion Unit (TFU) that enables unified modeling through self-attention.

Let $c\in {\mathbb{R}}^{1\times d}$ denote the learnable class token (broadcast across the batch), $T\in {\mathbb{R}}^{B\times L\times d}$ the patch tokens, and ${\tilde{T}}_g\in {\mathbb{R}}^{B\times N\times d}$ the refined gradient tokens. These tokens are concatenated in a global-to-local order:

$${\mathcal{S}}_0=Concat\left(c,{\tilde{T}}_g,T\right)\in {\mathbb{R}}^{B\times T_{\mathrm{tot}}\times d},$$

(11)

where $T_{\mathrm{tot}}=1+L+N$.

The fused token sequence is processed by a stack of Transformer layers with pre-normalization. For the n -th layer, the update equations are:

$${\mathcal{U}}^{\left(n\right)}={\mathcal{S}}^{\left(n\right)}+\mathcal{A}\left(LN\left({\mathcal{S}}^{\left(n\right)}\right)\right),$$

(12)

$${\mathcal{S}}^{(n+1)}={\mathcal{U}}^{\left(n\right)}+\mathcal{F}\left(LN\left({\mathcal{U}}^{\left(n\right)}\right)\right),$$

(13)

where $\mathcal{A}(·)$ denotes multi-head self-attention, $\mathcal{F}(·)$ is the position-wise feed-forward network, and $LN(·)$ represents Layer Normalization.

Within the attention module, normalized tokens are projected to queries, keys, and values:

$$(Q,K,V)=\Phi \left(LN\left(\mathcal{S}\right)\right),$$

(14)

where $Q,K,V\in {\mathbb{R}}^{B\times T_{\mathrm{tot}}\times d}$. To explicitly model interactions across different token types, a learnable bias matrix

$$R\in {\mathbb{R}}^{T_{\mathrm{tot}}\times T_{\mathrm{tot}}}$$

(15)

is incorporated to modulate the attention logits:

$$\mathcal{A}(Q,K,V)=Softmax\left({\displaystyle \frac{Q{K}^{\top }}{\sqrt{d}}}+R\right)V.$$

(16)

By jointly attending over semantic, structural, and appearance cues, the proposed Token Fusion Unit facilitates the learning of more discriminative representations for fine-grained visual classification.

4. Results

This section provides an in-depth evaluation of the proposed Gradient-Aware Token Injection Transformer across four widely adopted benchmarks for fine-grained visual classification. We begin by introducing the datasets used in our experiments, followed by a detailed description of the experimental setup. Subsequently, we conduct comparative experiments against existing state-of-the-art methods, supported by quantitative results and visual analysis. Finally, ablation studies are carried out to evaluate the effectiveness and contribution of each individual component within the proposed framework.

4.1. Datasets and Implementation Details

4.1.1. Datasets

We evaluate our method on four widely adopted fine-grained classification benchmarks (

Table 1. Dataset statistics and initial learning rates used in our experiments.

Dataset	Classes	Train	Test	LR
iNaturalist 2018 [21]	8142	437,513	24,426	5 $\times {10}^{-5}$
CUB-200-2011 [23]	200	5994	5794	5 $\times {10}^{-5}$
NABirds [22]	555	23,929	24,633	5 $\times {10}^{-5}$
Stanford Cars [24]	196	8114	8041	5 $\times {10}^{-3}$

): iNaturalist 2018 [21], NABirds [22], CUB-200-2011 [23], and Stanford Cars [24].

The iNaturalist 2018 dataset comprises large-scale, real-world RGB images spanning diverse categories such as plants, animals, and fungi, and is particularly challenging due to severe class imbalance and domain shifts. The Stanford Cars dataset focuses on fine-grained car model classification, where subtle appearance differences among hundreds of vehicle types demand strong discrimination capability. CUB-200-2011 targets bird species classification and is characterized by high inter-class similarity and large intra-class variations caused by lighting, pose, and viewpoint changes. NABirds builds upon CUB-200-2011 by incorporating a hierarchical taxonomy and introducing additional appearance variations due to seasonal changes. For all datasets, we strictly follow the officially provided training and testing splits.

All datasets consist of RGB images with varying original resolutions. Specifically, iNaturalist 2018 contains high-resolution natural images with diverse aspect ratios, typically exceeding $500\times 500$ pixels. CUB-200-2011 images are mostly centered bird photographs with resolutions around $500\times 500$ pixels. NABirds shows larger resolution variations due to varying capture conditions and camera settings. Stanford Cars images are generally high-quality RGB photographs, with resolutions ranging from approximately $300\times 300$ to above $800\times 800$ pixels. For the sake of ensuring consistency in training and evaluation, all images are resized to $384\times 384$ pixels, which helps to standardize the input data and facilitate fair comparison among different methods.

For datasets where auxiliary metadata are available (iNaturalist 2018 and CUB-200-2011), we follow prior works such as MetaFormer and incorporate the same metadata signals during training. For datasets without metadata (NABirds and Stanford Cars), all methods are evaluated under purely visual input settings. Our approach operates independently of metadata modeling (i.e., it does not rely on metadata for its core functionality) and focuses on enhancing visual representations through gradient-aware token integration.

4.1.2. Implementation Details

All experiments are conducted in a Linux environment using Python 3.8. Our implementation is based on PyTorch [25] (version 1.13.1) with torchvision version 0.14.1. We adopt the timm library (version 0.4.5) for model components and training utilities. NVIDIA Apex is used to enable mixed-precision training and performance optimization.Additional dependencies include OpenCV (opencv-python 4.5.1.48) for image processing and YACS (version 0.1.8) for configuration management.

We employ a convolutional backbone initialized with weights pretrained on iNaturalist 2021, and resize all input images to $384\times 384$ pixels. Training is conducted on two NVIDIA RTX 4090 GPUs using the AdamW optimizer [26] with a weight decay of $0.05$, and the learning rate is scheduled using cosine annealing. Standard data augmentations—including random cropping, horizontal flipping, and color jitter—are applied, together with stochastic depth regularization (maximum drop rate of $0.1$). All models are trained for a total of 300 epochs with a batch size of 16. To stabilize convergence, a warm-up phase is adopted: 20 epochs for CUB-200-2011 and 5 epochs for the remaining datasets.

4.2. Comparison with State-of-the-Art Methods

4.2.1. Results on the iNaturalist 2018 Dataset

Table 2. Comparison with state-of-the-art methods on iNaturalist 2018.

Method	Publication	Accuracy (%)
LOS [28]	ICLR 2025	70.8
LVM-LT [29]	arXiv 2025	72.7
BS-CMO [30]	CVPR 2022	74.0
PaCo [31]	ICCV 2021	75.2
ViT-L [32]	ECCV 2022	75.3
GPaCo [33]	IEEE 2023	75.4
MixMIM-B [34]	CVPR 2023	77.5
CaiT-M-36U224 [35]	ICCV 2021	78.0
MixMIM-L [34]	CVPR 2023	80.3
VL-LTR [36]	ECCV 2022	81.0
RDNet-L [37]	ECCV 2024	81.8
OMNIVORE [38]	CVPR 2022	84.1
SWAG [39]	CVPR 2022	86.0
MAE [40]	CVPR 2022	86.8
Hiera-H [41]	PMLR 2023	87.3
MetaFormer [10]	arXiv 2022	88.7
Ours	–	90.5

presents a comparison of our model with several representative methods on the iNaturalist 2018 benchmark: Geo-Aware [27], which filters predictions using a whitelist of image cues in post-processing; Presence-Only [9], which incorporates spatio-temporal priors during training; and MetaFormer [10], which fuses visual features with spatio-temporal tokens via joint attention.

Building upon these approaches, our model introduces dedicated gradient tokens that explicitly encode high-frequency image details—such as edges, fine textures, and subtle pattern variations—often underrepresented by standard attention mechanisms applied to raw pixel embeddings. The inclusion of gradient information enables the model to better distinguish between visually similar species and suppress background distractions, resulting in more precise localization of discriminative regions. These gradient tokens are effectively integrated with class and spatio-temporal tokens through a cross-modal attention mechanism, allowing the network to dynamically balance texture-based cues with semantic and contextual information.

As shown in Table 2, our model achieves a top-1 accuracy of 90.5% on the large-scale iNaturalist 2018 dataset, demonstrating that explicit modeling of gradient information can significantly enhance performance in fine-grained visual classification.

4.2.2. Results on the CUB-200-2011 Dataset

As shown in

Table 3. Comparison with state-of-the-art methods on CUB-200-2011. MetaFormer (baseline) and our method are evaluated under different pretraining datasets to isolate the effect of initialization.

Method	Publication	Accuracy (%)
PART [43]	TIP 2021	90.1
P2P-Net [44]	CVPR 2022	90.2
CAL [45]	ICCV 2021	90.6
DATL [46]	ISVC 2020	91.2
CP-CNN [47]	TIP 2022	91.4
LGTF [12]	ICCV 2023	91.5
TransIFC [48]	TMM 2023	91.0
CAMF [49]	SPL 2021	91.2
RAMS-Trans [45]	AAAI 2021	91.3
ViT-Net [50]	ICML 2022	91.6
TransFG [7]	AAAI 2022	91.7
SIM-Trans [51]	ACM 2022	91.8
IELT [42]	TMM 2023	91.8
MP-FGVR [52]	AAAI 2024	91.8
ACC-VIT [53]	AAAI 2024	91.8
DACL [5]	CVPR 2022	92.0
$A^2$QTrans [54]	TNNLS 2025	92.2
MetaFormer [10]	arXiv 2022	92.5
CGL [55]	TMM 2025	92.6
MPSA [13]	TIP 2024	92.8
MetaFormer (ImageNet-21K) [10]	arXiv 2022	91.0
MetaFormer (iNaturalist 2021) [10]	arXiv 2022	92.5
Ours (ImageNet-21K)	–	91.4
Ours (iNaturalist 2021)	–	92.9

, our model achieves a top-1 accuracy of 92.9% on CUB-200-2011, setting a new state-of-the-art performance benchmark and exceeding TransFG [7] by over 1.2%. While MetaFormer [10] leverages textual metadata to enrich representations, the Gradient-Aware Token Injection Transformer instead incorporates gradient tokens to explicitly encode fine-grained texture details. Previous methods like IELT [42] and FFVT [11] use separate attention heads to extract part features, but this can disrupt spatial coherence. To overcome this limitation, we introduce the Gradient Token Extraction (GTE) module, which restores lost texture information without sacrificing spatial structure. Embedding gradient cues in this way sharpens the model’s focus on subtle variations and underlies the improved classification performance.

4.2.3. Results on the NABirds Dataset

As shown in

Table 4. Comparison with state-of-the-art methods on NABirds.

Method	Publication	Accuracy (%)
GaRD [56]	CVPR 2021	88.0
API-Net [57]	AAAI 2020	88.1
PMG-V2 [58]	TPAMI 2022	88.4
GDSPM-Net [59]	PR 2023	89.0
LGTF [12]	ICCV 2023	90.4
TransFG [7]	AAAI 2022	90.8
IELT [42]	TMM 2023	90.8
TransIFC [48]	TMM 2023	90.9
MP-FGVR [52]	AAAI 2024	91.0
ACC-ViT [53]	PR 2024	91.4
CGL [55]	TMM 2025	91.7
MPSA [13]	TIP 2024	92.5
$A^2$QTrans [54]	TNNLS 2025	92.5
MetaFormer (ImageNet-21K) [10]	arXiv 2022	91.9
MetaFormer (iNaturalist 2021) [10]	arXiv 2022	92.8
Ours (ImageNet-21K)	–	92.3
Ours (iNaturalist 2021)	–	93.2

, our model achieves a top-1 accuracy of 93.2% on NABirds, significantly surpassing TransIFC’s 90.9% [48] and MPSA’s 92.5% [13]. The performance gain can be attributed to our method’s ability to preserve the complete token sequence, thereby enriching the feature representations. Unlike MPSA, which relies on partial region sampling [13], our model directly integrates discriminative descriptors into the feature map, allowing for more effective capture of fine-grained visual distinctions and consequently enhancing classification accuracy.

4.2.4. Results on the Stanford Cars Dataset

As shown in

Table 5. Comparison with state-of-the-art methods on Stanford Cars.

Method	Publication	Accuracy (%)
LDH-ViT [60]	PR 2025	93.1
$A^2$QTrans [54]	TNNLS 2025	93.6
DATL [46]	ISVC 2020	94.5
RsNet50-ECC [61]	TNNLS 2024	94.7
TransFG [7]	AAAI 2022	94.8
ViT-Net [50]	ICML 2022	95.0
GaRD [56]	CVPR 2021	95.1
CAMF [49]	SPL 2021	95.3
DACL [5]	AAAI 2022	95.3
API-Net [57]	AAAI 2020	95.3
GDSPM-Net [59]	PR 2023	95.3
PART [43]	TIP 2021	95.3
MetaFormer (ImageNet-21K) [10]	arXiv 2022	94.9
MetaFormer (iNaturalist 2021) [10]	arXiv 2022	95.0
Ours (ImageNet-21K)	–	95.2
Ours (iNaturalist 2021)	–	95.3

, our model achieves an accuracy of 95.3% on Stanford Cars, matching the current state-of-the-art performance. Unlike TransFG [7], which relies solely on spatial features and omits explicit gradient modeling, the Gradient-Aware Token Injection Transformer introduces dedicated gradient tokens that enhance the modeling of fine-grained details (e.g., badge textures) while preserving the overall shape information (e.g., vehicle silhouette). These gradient tokens are derived from image gradients and encode high-frequency texture cues that are often overlooked by standard attention mechanisms. By integrating these gradient cues into the spatial feature maps, the model more effectively captures subtle visual distinctions, demonstrating the potential benefit of gradient-aware modeling for fine-grained classification.

It is worth noting that our method yields tied performance with the baseline work MPSA [13]. After careful re-examination of the dataset and experimental results, we propose two potential explanations for this phenomenon:

1.

Dataset-specific feature dominance: The Stanford Cars dataset contains a large number of images with highly similar geometric structures (e.g., same car models with different colors or backgrounds). In such cases, color and texture features may play a more dominant role than fine-grained geometric details in classification tasks. Gradient tokens, which are designed to highlight geometric features, thus do not provide a significant edge over existing methods on this dataset.

2.

Limited feature discrimination: The current design of gradient tokens focuses on local geometric feature extraction but lacks a global feature aggregation mechanism. For car images with subtle differences in grilles/headlights, the local geometric cues captured by gradient tokens may not be sufficient to distinguish between similar categories, leading to saturated performance.

In future work, we plan to address this issue by exploring two directions: integrating cross-modal feature fusion to enhance feature representation and designing a dynamic weighting module to adaptively emphasize dataset-specific critical geometric regions.

4.3. Visual Analysis

4.3.1. Qualitative Attention Analysis Across Datasets

To better understand how gradient-aware token injection influences model behavior, we conduct a qualitative token-level attention analysis across all four datasets. Figure 3, Figure 4, Figure 5 and Figure 6 present representative examples from iNaturalist 2018, NABirds, CUB-200-2011, and Stanford Cars, respectively. Each figure contains three rows: the input image (top), the attention map produced by the baseline MetaFormer model without gradient token extraction (middle), and the attention map generated by our Gradient-Aware Token Injection Transformer (bottom).

From these visualizations, we observe that the baseline model frequently distributes attention over background regions or coarse object areas, reflecting its reliance on global semantic aggregation. In contrast, after injecting gradient tokens, the model exhibits more spatially concentrated attention on object contours, texture boundaries, and fine structural regions that are known to be discriminative in fine-grained recognition. Notably, this effect is most evident in regions with strong edge or texture variations, such as feather boundaries in birds and part outlines in vehicles.

These qualitative differences indicate that gradient-aware token injection does not merely reweight existing attention patterns, but instead provides explicit high-frequency structural cues that guide attention toward physically meaningful local details. Importantly, the observed attention refinement is consistent across datasets with diverse visual characteristics, suggesting that the proposed mechanism systematically mitigates the attenuation of gradient information during token aggregation. This behavior offers a qualitative explanation for the observed performance improvements, particularly in scenarios where subtle edge and texture cues are critical for distinguishing visually similar categories.

4.3.2. Gradient Token Activation Maps

To further analyze the role of the proposed Gradient Token Extraction (GTE) module, we visualize gradient token activation maps across multiple fine-grained benchmarks, including CUB-200-2011, NABirds, iNaturalist 2018, and Stanford Cars. Figure 7, Figure 8, Figure 9 and Figure 10 present representative examples from each dataset. Each visualization consists of two rows: the input image (top) and the corresponding gradient token activation map (bottom).

Across all datasets, the gradient token activations predominantly respond to high-frequency structural components, such as object contours, part boundaries, and localized texture transitions. These regions are well known to encode discriminative cues in fine-grained recognition tasks, particularly when inter-class differences are subtle. Importantly, the activation patterns remain spatially coherent under variations in pose, illumination, and background clutter, indicating that the GTE module consistently captures intrinsic structural information rather than dataset-specific appearance biases.

Unlike attention maps, which reflect the model’s allocation of importance during classification, gradient token activation maps directly reveal the type of information encoded by the injected tokens. The observed activations suggest that GTE explicitly preserves low-level gradient cues that are otherwise weakened during deep token aggregation. This complementary evidence supports the effectiveness of gradient-aware token learning as a mechanism for retaining fine-grained structural details throughout the Transformer hierarchy.

4.4. Ablation Study

4.4.1. Effectiveness of the Proposed Modules

A key component of the proposed Gradient-Aware Token Injection Transformer is the Gradient Token Extraction module (GTE), which explicitly encodes image gradient cues—such as edges and fine textures—into dedicated tokens to augment the standard visual representation. As shown in

Table 6. Ablation studies of the proposed Gradient Token Extraction (GTE) module. All results are based on the MetaFormer [10] backbone.

GTE	iNaturalist 2018	CUB-200-2011	NABirds	Stanford Cars
-	88.7	92.5	92.8	95.0
✓	90.5	92.9	93.2	95.3

, integrating the GTE module into MetaFormer [10] consistently improves performance across multiple fine-grained benchmarks. On iNaturalist 2018, the enhancement reaches nearly 2%, demonstrating the effectiveness of gradient-aware modeling in complex, real-world biodiversity scenes with significant intra-class variation. Notably, on the challenging NABirds dataset, our model achieves an accuracy of 93.2%, the highest reported to date. These results highlight that the GTE module not only provides complementary high-frequency structural cues but also generalizes well across diverse and realistic fine-grained classification scenarios.

4.4.2. Effect of Gradient Token Count on Model Performance

We study the impact of the number of gradient tokens in the GTE module. As shown in

Table 7. Effect of the number of gradient tokens on model accuracy (%) on CUB-200-2011.

Number of Gradient Tokens	0	4	8	16	20
Accuracy (%)	92.5	92.7	92.7	92.9	92.7

, performance on the CUB-200-2011 dataset peaks with 16 tokens, achieving the optimal balance between accuracy and computational efficiency. This configuration is adopted as the default. Similar trends are observed on other datasets, suggesting that 16 gradient tokens provide consistent and generalizable performance across different fine-grained classification tasks.

4.4.3. Evaluation Beyond Accuracy

To provide a more comprehensive assessment, we evaluate our model using macro and weighted F1-scores, as well as macro AUC, in addition to top-1 and top-5 accuracy. As shown in

Table 8. Performance of the model on fine-grained datasets in terms of top-1/top-5 accuracy, F1-scores, and macro AUC. All values are rounded to two decimal places.

Dataset	Acc@1 (%)	Acc@5 (%)	Macro F1	Weighted F1	Macro AUC
NABirds	93.066	99.610	0.9157	0.9290	0.9996
CUB-200-2011	92.906	98.792	0.9285	0.9281	0.9977
Stanford Cars	95.275	99.678	0.9491	0.9501	0.9987
iNaturalist2018	90.464	98.886	0.9021	0.9051	0.9993

, our model achieves consistently strong performance across all datasets, demonstrating both high accuracy and robust class-wise discrimination.

4.4.4. Confusion Matrix Analysis

To gain a deeper understanding of the model’s behavior, Figure 11 presents the normalized confusion matrices on several representative fine-grained benchmarks. Overall, most classes exhibit strong diagonal dominance, indicating consistent and reliable per-class discrimination.

Across datasets, misclassifications are sparse and mainly concentrated among visually similar classes. In particular, CUB-200-2011 and NABirds show near-perfect diagonals with only minor localized confusions, while Stanford Cars exhibits almost uniform diagonal dominance, reflecting strong discrimination for man-made object categories. In contrast, iNaturalist 2018 displays relatively more off-diagonal probability mass, which can be attributed to its long-tailed distribution and higher intra-class variance.

Overall, the confusion-matrix analysis is consistent with the quantitative results (accuracy, F1-score and macro AUC as in Table 8) and suggests that the proposed method learns robust and class-discriminative representations, with remaining errors largely driven by dataset-intrinsic challenges rather than random mispredictions.

4.5. Model Complexity Evaluation

As shown in

Table 9. Comparison of input resolution, parameters, GFLOPs, and top-1 accuracy (%) on the NABirds dataset.

Method	Input Size	Params (M)	GFLOPs	Accuracy (%)
ViT [3]	448	86.4	78.5	91.0
TransFG [7]	448	86.4	130.2	91.7
IELT [42]	448	93.5	73.2	91.8
ACC-ViT [53]	448	87.0	162.9	91.8
ViT-Net [50]	448	92.2	65.6	91.8
MPSA [13]	448	98.5	67.9	92.5
MetaFormer [10]	384	81.0	50.01	92.8
Ours	384	86.9	51.35	93.2

, the proposed method achieves an excellent trade-off between model complexity and recognition performance. Taking MetaFormer as the baseline, our model introduces only a marginal increase in parameters (86.9 M vs. 81.0 M) and FLOPs (51.35 G vs. 50.01 G), while improving accuracy from 92.8% to 93.2%. This demonstrates that the proposed design effectively enhances feature representation under a strictly controlled computational budget. All measurements are conducted on a single NVIDIA GeForce RTX 4090 with a batch size of 16, and values are averaged over multiple runs after a proper warm-up phase to ensure reliability.

Specifically, the MetaFormer baseline achieves an inference throughput of 44.16 images/s with a peak GPU memory consumption of 9170 MB. In contrast, our method yields a throughput of 39.10 images/s and a peak GPU memory usage of 11,234 MB. The slight decrease in throughput and increase in memory footprint can be directly attributed to the additional feature-enhancement blocks we introduced, which bring extra model parameters and intermediate activation computations.

Notably, this minor computational overhead is well justified by the performance gain: our method improves classification accuracy by 0.4% (from 92.8% to 93.2%) with only a marginal increase in parameters (+5.9 M, from 81.0 M to 86.9 M) and FLOPs (+1.34 G, from 50.01 G to 51.35 G). Collectively, these comprehensive results demonstrate that our proposed method maintains an excellent trade-off between recognition performance and computational cost.

5. Conclusions

We have presented a Gradient-Aware Token Injection Transformer for fine-grained visual classification. This method explicitly extracts the gradient magnitude and orientation from intermediate feature maps using a fixed Sobel operator and injects them as dedicated gradient tokens. These gradient tokens are aligned and jointly processed with visual tokens through self-attention mechanisms. By facilitating direct interaction between appearance tokens and gradient tokens, the model captures complementary structural cues, thereby enhancing its sensitivity to subtle texture-, shape-, and part-level differences.

Extensive experiments on four standard fine-grained visual classification (FGVC) benchmarks demonstrate that our approach consistently improves performance over strong baselines while only incurring a marginal increase in model complexity. This, in turn, shows that explicit gradient-token modeling serves as an effective and lightweight complement to Transformer-based recognition pipelines. The analysis of confusion matrices and the assessment of complementary metrics (accuracy, F1, macro AUC) reveal that the residual errors are structured, primarily occurring between inherently similar classes rather than randomly. This suggests that the model learns robust and class-discriminative representations.

Future work will involve replacing the fixed Sobel filtering with adaptive or learnable gradient extraction modules. We will also evaluate the token injection strategy across different backbone networks and in related structure-sensitive tasks, such as species identification, medical imaging, and remote sensing. Additionally, integrating gradient-aware tokens into multimodal or weakly supervised training regimes will be explored to further enhance the model’s robustness and interpretability. We hope that this work will inspire further exploration of explicit low-level structural cues within high-level Transformer representations for fine-grained recognition.