AMSRDet: An Adaptive Multi-Scale UAV Infrared-Visible Remote Sensing Vehicle Detection Network

Abstract

Unmanned Aerial Vehicle (UAV) platforms enable flexible and cost-effective vehicle detection for intelligent transportation systems, yet small-scale vehicles in complex aerial scenes pose substantial challenges from extreme scale variations, environmental interference, and single-sensor limitations. We present AMSRDet (Adaptive Multi-Scale Remote Sensing Detector), an adaptive multi-scale detection network fusing infrared (IR) and visible (RGB) modalities for robust UAV-based vehicle detection. Our framework comprises four novel components: (1) a MobileMamba-based dual-stream encoder extracting complementary features via Selective State-Space 2D (SS2D) blocks with linear complexity $O\left(HWC\right)$, achieving 2.1× efficiency improvement over standard Transformers; (2) a Cross-Modal Global Fusion (CMGF) module capturing global dependencies through spatial-channel attention while suppressing modality-specific noise via adaptive gating; (3) a Scale-Coordinate Attention Fusion (SCAF) module integrating multi-scale features via coordinate attention and learned scale-aware weighting, improving small object detection by 2.5 percentage points; and (4) a Separable Dynamic Decoder generating scale-adaptive predictions through content-aware dynamic convolution, reducing computational cost by 48.9% compared to standard DETR decoders. On the DroneVehicle dataset, AMSRDet achieves 45.8% mAP@0.5:0.95 (81.2% mAP@0.5) at 68.3 Frames Per Second (FPS) with 28.6 million (M) parameters and 47.2 Giga Floating Point Operations (GFLOPs), outperforming twenty state-of-the-art detectors including YOLOv12 (+0.7% mAP), DEIM (+0.8% mAP), and Mamba-YOLO (+1.5% mAP). Cross-dataset evaluation on Camera-vehicle yields 52.3% mAP without fine-tuning, demonstrating strong generalization across viewpoints and scenarios.

1. Introduction

Vehicle detection from Unmanned Aerial Vehicle (UAV) platforms has emerged as a critical technology for intelligent transportation systems, enabling applications in traffic monitoring, urban planning, and emergency response [1,2]. Compared to satellite or manned aircraft systems, UAVs provide flexible deployment, wide coverage, and real-time monitoring at lower operational costs, enabling vehicle detection across diverse urban and rural environments [3]. However, aerial vehicle detection confronts substantial challenges: vehicles span extreme scales from large trucks (hundreds of pixels) to small cars (few pixels), appear densely distributed with occlusions, exist against cluttered backgrounds, and experience varying illumination from weather and time-of-day changes [4,5].

Most vehicle detection methods employ single-modality sensors, either visible (RGB) cameras or infrared (IR) thermal imaging, each with inherent limitations [6]. Visible cameras capture rich texture and color under adequate lighting but degrade in low-light, fog, or adverse weather. Infrared sensors detect thermal radiation robustly across illumination conditions yet lack fine-grained texture and color for distinguishing similar vehicles. Multi-modal fusion addresses these complementary weaknesses: visible imagery provides spatial detail while infrared ensures robustness to lighting variations [7,8,9,10].

Multi-modal UAV vehicle detection faces four unresolved challenges that must be addressed for practical deployment. First, extreme scale variation demands effective multi-scale representation, as vehicles range from large trucks occupying hundreds of pixels to small cars spanning only a few pixels within single images. The scale difference can exceed 10× between the largest and smallest vehicles, requiring the network to maintain discriminative features across this wide range. Second, cross-modal integration must suppress modality-specific noise and handle spatial misalignment while preserving discriminative features from each sensor. Infrared and visible sensors exhibit different noise characteristics and may suffer from slight spatial misalignment due to sensor positioning, requiring robust fusion strategies. Third, UAV computational constraints require lightweight architectures balancing accuracy and inference speed for real-time operation. Onboard processing units typically have limited GPU memory and computational power compared to ground-based systems, necessitating efficient model designs. Fourth, models must generalize across altitudes, viewing angles, weather conditions, and vehicle types without extensive retraining, as UAVs operate in diverse and dynamic environments where collecting labeled data for every scenario is impractical.

Deep learning has transformed object detection, with Convolutional Neural Networks (CNNs) and Transformers achieving strong benchmark performance [7,11,12,13]. CNN-based detectors like You Only Look Once (YOLO) and RetinaNet excel at capturing local patterns through hierarchical feature extraction but struggle with long-range dependencies due to limited receptive fields [14,15]. Transformer-based detectors like DEtection TRansformer (DETR) eliminate hand-crafted components through global self-attention but suffer from quadratic spatial complexity $O\left(H^2{W}^2\right)$, limiting scalability to high-resolution aerial imagery where images can exceed 4K resolution [16,17]. State-space models (SSMs), particularly the Mamba architecture [18,19], offer an attractive alternative with linear complexity $O\left(HW\right)$ while maintaining long-range modeling capability through selective state transitions. Recent vision adaptations [20,21,22] demonstrate competitive accuracy at reduced computational cost, motivating their application to resource-constrained UAV detection scenarios.

Attention mechanisms have become fundamental for capturing long-range dependencies and cross-modal interactions in modern detectors [17,23,24]. Standard channel attention mechanisms like Squeeze-and-Excitation (SE) networks aggregate spatial information through global pooling, losing precise positional information critical for object localization [25]. Coordinate Attention (CA) [26,27] addresses this limitation by encoding spatial positions alongside channel relationships through factorized 1D horizontal and vertical attention, preserving directional positional information. For multi-scale detection, existing methods like Feature Pyramid Networks (FPNs) [28] employ fixed fusion weights that cannot adapt to varying object size distributions across different scenes. Existing attention-based fusion treats spatial locations and scales uniformly, failing to emphasize informative regions for small vehicles adaptively or adjust fusion strategies based on scene-specific scale distributions [29,30].

To address these challenges, we present AMSRDet (Adaptive Multi-Scale Remote Sensing Detector), an efficient infrared-visible fusion framework specifically designed for UAV vehicle detection. As shown in Figure 1, our approach builds upon the Real-Time DEtection TRansformer (RT-DETR) [31] architecture while introducing four key innovations tailored for multi-modal aerial detection. First, we design a MobileMamba-based dual-stream encoder that processes infrared and visible images through separate pathways, extracting hierarchical features using Selective State-Space 2D (SS2D) blocks [20,32]. The SS2D blocks achieve linear complexity $O\left(HWC\right)$ compared to standard Transformer’s quadratic complexity $O\left(H^2{W}^{2}C\right)$, enabling the efficient processing of high-resolution aerial imagery while maintaining long-range dependency modeling through selective state transitions. Second, we propose a Cross-Modal Global Fusion (CMGF) module that captures global cross-modal dependencies through a combination of spatial and channel attention mechanisms. The module employs dual-path attention to model both “where” (spatial) and “what” (channel) to attend, while an adaptive gating mechanism dynamically suppresses modality-specific noise based on local feature quality. Third, we introduce a Scale-Coordinate Attention Fusion (SCAF) module that adaptively integrates multi-scale features by combining coordinate attention with learned scale-aware weights. Unlike fixed fusion strategies, SCAF dynamically adjusts the contribution of different scales based on the global scene context, emphasizing appropriate scales for varying vehicle size distributions. Fourth, we adopt a Separable Dynamic Decoder from Mask2Former [33] that generates scale-adaptive predictions through content-aware dynamic convolution [34], replacing the computationally expensive multi-head cross-attention in standard DETR decoders while maintaining expressive power for multi-scale vehicle detection.

The main contributions of this work are summarized as follows:

• AMSRDet fuses infrared-visible modalities through state-space models and adaptive attention, achieving superior detection performance with computational efficiency for UAV vehicle detection. Compared to the baseline RT-DETR, AMSRDet improves mean Average Precision (mAP) by 1.1 percentage points while achieving 1.41× faster inference and 48.9% fewer Floating Point Operations (FLOPs).
• A MobileMamba dual-stream encoder with SS2D blocks extracts hierarchical features at linear complexity $O\left(HWC\right)$, while Cross-Modal Global Fusion captures global dependencies via spatial-channel attention with adaptive gating that suppresses modality-specific noise.
• Scale-Coordinate Attention Fusion adaptively integrates multi-scale features through coordinate attention and learned scale weights, improving small object detection by 2.5 percentage points. A Separable Dynamic Decoder generates scale-adaptive predictions via content-aware dynamic convolution.
• Experiments on DroneVehicle show that AMSRDet outperforms twenty state-of-the-art detectors (YOLOv12, DEIM, Mamba-YOLO, etc.), achieving 45.8% mAP@0.5:0.95 (81.2% mAP@0.5) at 68.3 Frames Per Second (FPS) with 28.6 M parameters and 47.2 GFLOPs. Cross-dataset evaluation yields 52.3% mAP on Camera-vehicle without fine-tuning, demonstrating 2.5–9.1 percentage points improvement over baselines.

The remainder of this paper is organized as follows. Section 2 reviews related work on UAV vehicle detection, multi-modal fusion, and state-space models. Section 3 presents the proposed AMSRDet framework including the network architecture and key components. Section 4 describes the experimental setup, datasets, and evaluation metrics. Section 5 analyzes the experimental results and ablation studies. Section 6 concludes the paper and discusses future directions.

2. Related Work

This section reviews related work on UAV vehicle detection, multi-modal fusion, state-space models, and attention mechanisms for multi-scale detection. We identify the limitations of existing approaches and position our contributions within the broader research landscape.

2.1. UAV Vehicle Detection

UAV vehicle detection has gained prominence in intelligent transportation and urban management [1,2]. Early approaches employed handcrafted features (Histogram of Oriented Gradients (HOG) [35], Scale-Invariant Feature Transform (SIFT) [36]) with traditional classifiers before deep learning enabled CNN-based detectors. Two-stage methods (Faster Region-based CNN (R-CNN) [37] and Cascade R-CNN [38]) achieve high accuracy via region proposals and multi-stage refinement at the cost of inference speed. One-stage detectors (YOLO series [11,14,39,40], RetinaNet [15]) enable real-time performance through dense prediction yet face challenges with small objects [41].

Aerial-specific adaptations address unique challenges: hybrid attention combining channel and spatial mechanisms [5], oriented bounding boxes for arbitrary vehicle angles [42], and context-aware pyramids for multi-scale detection [43]. Recent lightweight UAV detectors include PS-YOLO [44] with efficient architectures, RSW-YOLO [45] for urban vehicle detection, and AUHF-DETR [46] combining spatial attention with hybrid features for real-time performance. Transformer-based detectors offer end-to-end paradigms: DETR [13] eliminates hand-crafted components like Non-Maximum Suppression (NMS), Deformable DETR [16] accelerates convergence via deformable attention, DINO [47] applies contrastive denoising, and RT-DETR [31] achieves real-time speed through hybrid encoding with intra-scale interaction (Attention-based Intra-scale Feature Interaction, AIFI) and cross-scale fusion (Cross-scale Context Fusion Module, CCFM). These methods target single-modality visible imagery, leaving multi-modal sensor fusion underexplored.

Table 1. Comparison of representative object detection methods. “Multi-modal” indicates native support for RGB-IR fusion. Complexity refers to the dominant computational term.

Method	Type	Multi-Modal	Complexity	Real-Time	Small Object	Year
Faster R-CNN [37]	Two-stage CNN	No	$O(N·C^2)$	No	Moderate	2017
YOLOv8 [48]	One-stage CNN	No	$O\left(HWC\right)$	Yes	Moderate	2023
DETR [13]	Transformer	No	$O\left(H^2{W}^2\right)$	No	Poor	2020
RT-DETR [31]	Hybrid	No	$O\left(HWC\right)$	Yes	Good	2023
VMamba [20]	SSM	No	$O\left(HWC\right)$	Yes	Good	2024
AMSRDet (Ours)	SSM + Attention	Yes	$O\left(HWC\right)$	Yes	Excellent	2025

summarizes the comparison of representative detection methods across key dimensions including architecture type, modality support, and computational characteristics.

2.2. Multi-Modal Fusion for Object Detection

Multi-modal fusion integrates complementary sensor information for robust detection [6,49]. Early fusion concatenates modality features at the input level, sacrificing cross-modal interaction modeling. Middle fusion applies element-wise operations or attention at intermediate layers. Late fusion ensembles predictions from modality-specific detectors [50].

Deep learning enables sophisticated fusion: cross-modal attention aligning infrared-visible features [7,51], differential modules emphasizing modality-specific discriminative information [8], and graph networks modeling inter-modal relationships [9]. Recent surveys [49] comprehensively review multi-modal fusion algorithms, while efficient training strategies [52] address computational constraints through adaptive feature alignment. Defaoui et al. [10] demonstrate thermal-visible fusion for pedestrian detection, and Ma et al. [53] propose efficient multi-modal fusion transformers. However, computational expense from attention or graph operations limits real-time UAV deployment, motivating our lightweight CMGF design.

2.3. State-Space Models for Vision

State-Space Models (SSMs) offer efficient alternatives to Transformers for sequence modeling [22]. Mamba [18] achieves linear complexity via selective state-space models with input-dependent transitions. Vision Mamba [19] treats images as patch sequences for classification. VMamba [20] captures 2D dependencies through cross-scan mechanisms. U-Mamba [21] demonstrates effectiveness on medical segmentation, and Li et al. [32] extend vision Mamba for dense prediction tasks.

Object detection adaptations include LocalMamba’s local scanning preserving spatial locality [54] and PlainMamba’s simplified non-hierarchical design [55]. Mamba-YOLO [56] integrates SSMs into the YOLO framework, and MambaODet [57] targets efficient real-time detection. However, these methods lack multi-modal fusion capabilities critical for aerial vehicle detection requiring cross-modal interaction and multi-scale integration. Our MobileMamba encoder addresses this gap by designing a dual-stream architecture specifically for RGB-IR fusion.

2.4. Attention Mechanisms for Multi-Scale Detection

Attention mechanisms form core detector components [17,23]. Channel attention [25] recalibrates features via inter-channel modeling. Spatial attention [58] emphasizes informative locations through pooling and convolution. Coordinate Attention [26,27] factorizes into 1D horizontal–vertical encoding, capturing long-range dependencies with positional precision. Recent advances include bandit-based attention [59] for selective token processing in vision transformers and AttZoom [60] for adaptive feature zooming.

Multi-scale detection employs feature pyramids: FPN [28] constructs representations via top–down pathways, Path Aggregation Network (PANet) [61] adds bottom–up flow, and Bi-directional FPN (BiFPN) [12] learns weighted bi-directional fusion. Scale-wise attention [29] adaptively weights features per object size distribution, and Hao et al. [30] propose feature fusion networks for multi-modal detection. Dai et al. [62] introduce dynamic decoders for end-to-end detection. Existing methods lack explicit coordinate modeling crucial for localizing small vehicles in high-resolution aerial imagery, which our SCAF module addresses through factorized directional attention.

3. Methodology

This section presents the proposed AMSRDet framework for multi-modal UAV vehicle detection. We first describe the overall architecture and then detail each key component: the MobileMamba encoder, Cross-Modal Global Fusion module, Scale-Coordinate Attention Fusion module, Separable Dynamic Decoder, and loss function.

3.1. Overall Architecture

The overall architecture of AMSRDet is illustrated in Figure 1. Our framework is built upon the Real-Time DEtection TRansformer (RT-DETR) [31] architecture, which provides an efficient end-to-end detection framework with NMS-free post-processing and adaptable inference speed. RT-DETR employs a hybrid encoder that decouples intra-scale interaction and cross-scale fusion, along with Intersection over Union (IoU)-aware query selection for improved object query initialization.

Table 2. Comparison of RT-DETR and AMSRDet architectures. We highlight the key modifications made to adapt RT-DETR for multi-modal UAV vehicle detection.

Component	RT-DETR	AMSRDet (Ours)
Backbone	ResNet-50/101 (single-stream)	MobileMamba (dual-stream)
Input Modality	RGB only	RGB + IR
Encoder	AIFI + CCFM	AIFI + CMGF + SCAF
Cross-modal Fusion	None	CMGF module
Multi-scale Fusion	CCFM (fixed weights)	SCAF (adaptive weights)
Decoder	Standard cross-attention	Separable Dynamic Decoder
Complexity	$O(N_q·HW·C)$	$O(N_q·C·k^2)$

summarizes the key modifications from RT-DETR to AMSRDet, highlighting which components are retained, modified, or replaced.

Building on this foundation, we introduce four key adaptations specifically designed for multi-modal UAV vehicle detection: (1) a MobileMamba-based dual-stream encoder that replaces the standard CNN backbone to efficiently extract hierarchical features from infrared and visible images using selective state-space models; (2) an enhanced Efficient Transformer Encoder that incorporates cross-modal attention mechanisms to capture inter-modal dependencies beyond the original intra-scale feature interaction; (3) a multi-scale fusion module that integrates features at different scales through our proposed CMGF and SCAF modules, extending the RT-DETR cross-scale fusion capability to handle multi-modal inputs; and (4) a Separable Dynamic Decoder that improves upon the RT-DETR standard decoder by employing dynamic convolution with content-aware kernels for scale-adaptive predictions. These modifications enable AMSRDet to effectively leverage complementary information from infrared and visible modalities while maintaining the real-time performance advantages of the RT-DETR framework.

Given an input infrared image $I_{IR}\in {\mathbb{R}}^{H\times W\times 3}$ and visible image $I_{RGB}\in {\mathbb{R}}^{H\times W\times 3}$, the MobileMamba encoder extracts multi-scale features ${\left\{F_{IR}^i\right\}}_{i=3}^5$ and ${\left\{F_{RGB}^i\right\}}_{i=3}^5$ at three different scales, where $F^i\in {\mathbb{R}}^{H_i\times W_i\times C_i}$ with $H_i=H/2^i$ and $W_i=W/2^i$. Note that both RGB and IR images are represented as 3-channel tensors; the IR images are captured by thermal cameras and stored in 3-channel format for compatibility with standard image processing pipelines, though the thermal information is encoded differently from visible light RGB values. The Efficient Transformer Encoder processes these features through self-attention and cross-attention to capture intra-modal and inter-modal dependencies. The CMGF module performs global cross-modal fusion, while the SCAF module adaptively integrates multi-scale features with coordinate attention. Finally, the Separable Dynamic Decoder employs dynamic convolution to generate scale-adaptive predictions.

3.2. MobileMamba Encoder with SS2D Blocks

The MobileMamba encoder is designed to efficiently extract hierarchical features from infrared and visible images using selective state-space models. Our design is inspired by the MobileMamba architecture [20,32] but adapted for dual-stream multi-modal processing. As shown in Figure 2, each SS2D block consists of layer normalization, the SS2D module, and a Feed-Forward Network (FFN) with residual connections.

Training Strategy. The MobileMamba encoder is initialized with ImageNet-pretrained weights to leverage learned visual representations. The RGB and IR branches share the same pretrained weights initially, then are fine-tuned separately during training on the UAV dataset to adapt to modality-specific characteristics. This transfer learning strategy is essential given the relatively small size of UAV datasets compared to ImageNet.

The SS2D module processes 2D feature maps through selective state-space modeling. Given an input feature $X\in {\mathbb{R}}^{H\times W\times C}$, we first flatten it into a sequence $X_{seq}\in {\mathbb{R}}^{L\times C}$ where $L=H\times W$. The selective state-space model is defined as:

$$\begin{array}{cc}\hfill h_t& =\overline{A}{h}_{t-1}+\overline{B}{x}_t,\hfill \\ \hfill y_t& =C{h}_t,\hfill \end{array}$$

(1)

where $h_t\in {\mathbb{R}}^N$ is the hidden state, $x_t\in {\mathbb{R}}^C$ is the input at position t, $y_t\in {\mathbb{R}}^C$ is the output, and $\overline{A}\in {\mathbb{R}}^{N\times N}$, $\overline{B}\in {\mathbb{R}}^{N\times C}$, and $C\in {\mathbb{R}}^{C\times N}$ are state transition matrices. The key innovation of selective SSM is that $\overline{B}$ and C are input dependent:

$$\overline{B}=s_B\left(x_t\right),C=s_C\left(x_t\right),$$

(2)

where $s_B$ and $s_C$ are linear projections that generate input-dependent parameters. This selective mechanism enables the model to adaptively focus on relevant information while filtering out noise.

The complete SS2D block operation is formulated as

$$\begin{array}{cc}\hfill X^{\prime }& =\mathrm{LN}\left(X\right),\hfill \\ \hfill X^{″}& =\mathrm{SS}2\mathrm{D}\left(X^{\prime }\right)+X,\hfill \\ \hfill X_{out}& =\mathrm{FFN}\left(\mathrm{LN}\left(X^{″}\right)\right)+X^{″},\hfill \end{array}$$

(3)

where LN denotes layer normalization and FFN is a two-layer feed-forward network with GELU activation.

The MobileMamba encoder stacks multiple SS2D blocks at different scales to extract hierarchical features. Compared to standard Transformer encoders with self-attention complexity $O\left(L^{2}C\right)$, the SS2D module achieves linear complexity $O\left(LC\right)$, enabling the efficient processing of high-resolution aerial imagery.

3.3. Cross-Modal Global Fusion (CMGF)

The CMGF module addresses a fundamental challenge in multi-modal fusion: how to effectively integrate complementary information from infrared and visible modalities while suppressing modality-specific noise and misalignment. Traditional fusion methods either perform simple concatenation or element-wise operations, failing to capture complex cross-modal dependencies. Our CMGF module (Figure 3) employs a theoretically grounded approach that combines spatial-channel attention with cross-modal affinity learning to achieve globally coherent fusion.

Motivation and Theoretical Foundation. Multi-modal fusion in UAV imagery faces three key challenges: (1) modality-specific noise where each sensor captures artifacts unique to its imaging mechanism (e.g., thermal noise in IR, motion blur in RGB); (2) spatial misalignment due to different sensor characteristics and mounting positions; and (3) semantic gap where the same physical object exhibits different appearance patterns across modalities. To address these challenges, we design CMGF based on information theory principles. Specifically, we aim to maximize mutual information $I(F_{RGB}^G;F_{IR}^G)$ between fused features while minimizing redundancy and preserving modality-specific discriminative information.

Given infrared feature $F_{IR}^L\in {\mathbb{R}}^{H\times W\times C}$ and visible feature $F_{RGB}^L\in {\mathbb{R}}^{H\times W\times C}$ at layer L, the CMGF module first applies $1\times 1$ convolutions to project features into a shared semantic space:

$$F_m^{\prime }=\mathrm{BN}\left({\mathrm{Conv}}_{1\times 1}\left(F_m^L\right)\right)+F_m^L,m\in \{\mathrm{IR},\mathrm{RGB}\},$$

(4)

where BN denotes batch normalization, the residual connection preserves original feature information, and m indexes the modality. This projection is crucial for aligning the semantic representations of different modalities.

Dual-Path Spatial-Channel Attention. We employ a dual-path attention mechanism that simultaneously captures spatial and channel dependencies. For spatial attention, we leverage both global average pooling (GAP) and global max pooling (GMP) to extract complementary statistical information. GAP captures the average response across spatial locations, representing the overall feature distribution, while GMP identifies the most salient features, capturing discriminative peaks:

$$\begin{array}{cc}\hfill \mathrm{GAP}\left(F\right)& ={\displaystyle \frac{1}{HW}}\sum _{i=1}^H\sum _{j=1}^W{F}_{i,j},\hfill \\ \hfill \mathrm{GMP}\left(F\right)& =\underset{i,j}{max}{F}_{i,j}.\hfill \end{array}$$

(5)

The spatial attention maps are computed as

$$S{A}_m=\sigma \left({\mathrm{Conv}}_{7\times 7}\left(\mathrm{Concat}[{\mathrm{GAP}}_c\left(F_m^{\prime }\right);{\mathrm{GMP}}_c\left(F_m^{\prime }\right)]\right)\right),m\in \{\mathrm{IR},\mathrm{RGB}\},$$

(6)

where $\sigma$ denotes the sigmoid activation, ${\mathrm{GAP}}_c$ and ${\mathrm{GMP}}_c$ denote channel-wise pooling operations that reduce the channel dimension to 1, and ${\mathrm{Conv}}_{7\times 7}$ employs a large receptive field to capture spatial context. The $7\times 7$ kernel size is chosen to balance between capturing sufficient spatial context and computational efficiency.

For channel attention, we employ a shared MLP architecture with bottleneck design to model inter-channel relationships:

$$C{A}_m=\sigma (\mathrm{MLP}\left({\mathrm{GAP}}_s\left(F_m^{\prime }\right)\right)+\mathrm{MLP}\left({\mathrm{GMP}}_s\left(F_m^{\prime }\right)\right)),m\in \{\mathrm{IR},\mathrm{RGB}\},$$

(7)

where ${\mathrm{GAP}}_s$ and ${\mathrm{GMP}}_s$ denote spatial pooling operations that aggregate information across spatial dimensions, and MLP is defined as

$$\mathrm{MLP}\left(x\right)=W_2·\mathrm{ReLU}(W_1·x),$$

(8)

where $W_1\in {\mathbb{R}}^{C/r\times C}$ and $W_2\in {\mathbb{R}}^{C\times C/r}$ with reduction ratio $r=16$ to reduce parameters while maintaining representational capacity.

Cross-Modal Affinity Learning. To capture global cross-modal dependencies, we compute cross-modal affinity matrices that measure the semantic similarity between features from different modalities. We reshape features into $F^{\prime }\in {\mathbb{R}}^{\left(HW\right)\times C}$ and compute affinity through scaled dot-product attention:

$$A_{m_1\to m_2}=\mathrm{Softmax}\left({\displaystyle \frac{F_{m_1}^{\prime }{\left(F_{m_2}^{\prime }\right)}^T}{\sqrt{C}}}\right),(m_1,m_2)\in \{(\mathrm{IR},\mathrm{RGB}),(\mathrm{RGB},\mathrm{IR})\},$$

(9)

where the scaling factor $1/\sqrt{C}$ prevents the dot products from growing too large, which would cause the softmax function to have extremely small gradients. The affinity matrix $A\in {\mathbb{R}}^{\left(HW\right)\times \left(HW\right)}$ captures pairwise relationships between all spatial locations across modalities.

Cross-modal features are then computed by aggregating information based on learned affinities:

$$F_{m_1\to m_2}=\mathrm{Reshape}(A_{m_1\to m_2}·\mathrm{Flatten}\left(F_{m_2}^{\prime }\right)),(m_1,m_2)\in \{(\mathrm{IR},\mathrm{RGB}),(\mathrm{RGB},\mathrm{IR})\},$$

(10)

where Flatten reshapes features to $\left(HW\right)\times C$ and Reshape restores the spatial structure to $H\times W\times C$.

Adaptive Feature Fusion with Gating Mechanism. The final fused features incorporate spatial attention, channel attention, and cross-modal information through a gating mechanism that adaptively balances different information sources. For each modality $m\in \{\mathrm{IR},\mathrm{RGB}\}$ with its complementary modality $\overline{m}$, we compute:

$${\alpha }_m=\sigma \left({\mathrm{Conv}}_{1\times 1}\left([F_m^{\prime };F_{\overline{m}\to m}]\right)\right),$$

(11)

where $\alpha \in {\mathbb{R}}^{H\times W\times 1}$ are learned gating weights that control the contribution of cross-modal information. The final outputs are:

$$F_m^G=F_m^L+S{A}_m\odot C{A}_m\odot ({\alpha }_m\odot F_{\overline{m}\to m}+(1-{\alpha }_m)\odot F_m^{\prime }),m\in \{\mathrm{IR},\mathrm{RGB}\},$$

(12)

where ⊙ denotes element-wise multiplication, $\overline{m}$ denotes the complementary modality (IR for RGB and vice versa), and the residual connection $F^L$ ensures gradient flow and preserves low-level features.

The CMGF module effectively integrates complementary information from infrared and visible modalities through global affinity learning and adaptive gating. The dual-path attention mechanism captures both spatial and channel dependencies, while the learned gating weights dynamically suppress modality-specific noise based on local feature quality.

3.4. Scale-Coordinate Attention Fusion (SCAF)

The SCAF module addresses the challenge of detecting vehicles at vastly different scales in UAV imagery, where objects can range from large trucks occupying hundreds of pixels to small cars spanning only a few pixels. Traditional multi-scale fusion methods like FPN employ fixed fusion weights, failing to adapt to varying object size distributions across different scenes. Our SCAF module (Figure 4) introduces a principled approach that combines coordinate-aware attention with learnable scale-adaptive weighting.

Motivation and Coordinate Attention Design. Standard channel attention mechanisms like SE-Net aggregate spatial information through global pooling, which loses precise positional information. This is problematic for object detection where spatial localization is critical. Coordinate attention addresses this by decomposing 2D global pooling into two 1D feature encoding operations that preserve directional positional information. This design is particularly beneficial for vehicle detection in aerial imagery where vehicles exhibit strong directional patterns (e.g., aligned along roads).

Given multi-scale features ${\left\{F^i\right\}}_{i=3}^5$ from different stages where $F^i\in {\mathbb{R}}^{H_i\times W_i\times C_i}$, we first apply Spatial Attention Modules (SAMs) to enhance discriminative features:

$${\tilde{F}}^i=\mathrm{SAM}\left(F^i\right)=F^i\odot \sigma \left({\mathrm{Conv}}_{7\times 7}\left([\mathrm{MaxPool}\left(F^i\right);\mathrm{AvgPool}\left(F^i\right)]\right)\right),$$

(13)

where $\sigma$ is the sigmoid function and $[·;·]$ denotes channel-wise concatenation. The SAM emphasizes salient spatial regions before coordinate attention.

For coordinate attention, we perform directional pooling along horizontal and vertical axes. For a feature map ${\tilde{F}}^i\in {\mathbb{R}}^{H_i\times W_i\times C_i}$, the coordinate-aware feature encoding along direction $d\in \{h,w\}$ is

$$z_d^i=\left\{\begin{array}{cc}{\displaystyle \frac{1}{W_i}}\sum _{w=1}^{W_i}{\tilde{F}}^i(h,w,:)\in {\mathbb{R}}^{H_i\times 1\times C_i},\hfill & d=h,\hfill \\ {\displaystyle \frac{1}{H_i}}\sum _{h=1}^{H_i}{\tilde{F}}^i(h,w,:)\in {\mathbb{R}}^{1\times W_i\times C_i},\hfill & d=w,\hfill \end{array}\right.$$

(14)

where $z_h^i$ and $z_w^i$ encode aggregated features along height and width dimensions, respectively. Unlike global pooling that produces a single vector, this directional pooling preserves spatial structure along one dimension while aggregating along the other.

These directional features are concatenated and transformed through a shared transformation:

$$f^i=\delta \left(\mathrm{BN}\left({\mathrm{Conv}}_{1\times 1}\left([z_h^i;z_w^i]\right)\right)\right),$$

(15)

where $\delta$ denotes ReLU activation, BN is batch normalization, and $f^i\in {\mathbb{R}}^{(H_i+W_i)\times 1\times C_i^{\prime }}$ with $C_i^{\prime }=C_i/r$, where $r=8$ is the reduction ratio. The bottleneck design reduces computational cost while forcing the network to learn compact representations.

The intermediate feature $f^i$ is then split along the spatial dimension and processed through separate transformation branches to generate directional attention maps. For direction, $d\in \{h,w\}$:

$$f_d^i=\left\{\begin{array}{cc}{f}^i[0:H_i,:,:]\in {\mathbb{R}}^{H_i\times 1\times C_i^{\prime }},\hfill & d=h,\hfill \\ f^i[H_i:H_i+W_i,:,:]\in {\mathbb{R}}^{1\times W_i\times C_i^{\prime }},\hfill & d=w,\hfill \end{array}\right.$$

(16)

$$g_d^i=\sigma \left({\mathrm{Conv}}_{1\times 1}\left(f_d^i\right)\right),d\in \{h,w\},$$

(17)

where $g_h^i\in {\mathbb{R}}^{H_i\times 1\times C_i}$ and $g_w^i\in {\mathbb{R}}^{1\times W_i\times C_i}$ are attention weights that modulate features along height and width dimensions, respectively. The coordinate attention output is

$$F_{CA}^i={\tilde{F}}^i\odot g_h^i\odot g_w^i,$$

(18)

where ⊙ denotes element-wise multiplication with broadcasting. This factorized attention allows the network to capture long-range dependencies along both spatial dimensions with precise positional encoding.

Scale-Aware Adaptive Fusion. To fuse features from different scales, we first align them to a common spatial resolution through bilinear interpolation. Let ${\hat{F}}_{CA}^i$ denote the aligned features at resolution $H\times W$. Traditional FPN uses fixed fusion weights (typically uniform or learned but fixed after training), which cannot adapt to varying object size distributions across different scenes. We introduce a scale-aware weighting mechanism that dynamically adjusts fusion weights based on global context. For each scale $i\in \{3,4,5\}$, we extract global features and compute scale importance:

$$s^i=\mathrm{GAP}({\hat{F}}_{CA}^i)\in {\mathbb{R}}^{C_i},e^i={\mathrm{FC}}_2\left(\mathrm{ReLU}\left({\mathrm{FC}}_1\left([s^3;s^4;s^5]\right)\right)\right),$$

(19)

where ${\mathrm{FC}}_1\in {\mathbb{R}}^{d\times 3C}$ and ${\mathrm{FC}}_2\in {\mathbb{R}}^{3\times d}$ are fully connected layers with hidden dimension $d=128$. The concatenation of global features from all scales allows the network to reason about the relative importance of different scales based on the overall scene context.

The scale weights are computed through softmax normalization:

$$w^i={\displaystyle \frac{\exp \left(e^i\right)}{{\sum }_{j=3}^5\exp \left(e^j\right)}},i\in \{3,4,5\},$$

(20)

where $w^i$ represents the importance weight for scale i. The softmax ensures that weights sum to 1, providing a probabilistic interpretation of scale importance.

The final fused feature incorporates both weighted aggregation and residual connections:

$$F_{SCAF}={\mathrm{Conv}}_{3\times 3}\left(\sum _{i=3}^5{w}^i·{\hat{F}}_{CA}^i\right)+\sum _{i=3}^5{\displaystyle \frac{1}{3}}·{\hat{F}}_{CA}^i,$$

(21)

where the first term performs adaptive weighted fusion and the second term provides a uniform-weighted residual connection that ensures gradient flow and prevents the network from completely ignoring any scale.

The SCAF module enables adaptive multi-scale feature integration through coordinate attention that preserves precise positional information along horizontal and vertical directions. The scale-aware weighting mechanism dynamically adjusts fusion weights based on scene context, emphasizing appropriate scales for different vehicle size distributions.

3.5. Separable Dynamic Decoder

We adopt the Separable Dynamic Decoder architecture from Mask2Former [33], which addresses the quadratic complexity limitation of standard DETR decoders. Traditional DETR employs cross-attention between $N_q$ queries and $HW$ feature locations, resulting in $O(N_q·HW·C)$ complexity. For UAV imagery with high resolution and dense object distributions, this becomes computationally prohibitive. Following Mask2Former’s design for universal image segmentation, our decoder (Figure 5) replaces expensive multi-head cross-attention with separable dynamic convolution, achieving linear complexity while maintaining expressive power for multi-modal vehicle detection.

Motivation and Dynamic Convolution Formulation. Standard convolution uses fixed kernels that are independent of input content, limiting adaptability to varying object scales and appearances. Dynamic convolution generates input-dependent kernels, allowing the network to adapt its receptive field and feature extraction based on content. This is particularly valuable for multi-scale vehicle detection where the optimal kernel size varies with object scale.

Given aggregated features $F_{agg}\in {\mathbb{R}}^{H\times W\times C}$ from SCAF and learnable proposal kernels $K_{prop}\in {\mathbb{R}}^{N_q\times C}$ where $N_q=300$ is the number of object queries, the decoder generates initial box features through pre-attention with 2D dynamic convolution:

$$F_{box}^{\left(0\right)}=\mathrm{DyConv}2\mathrm{D}(K_{prop},F_{agg})\in {\mathbb{R}}^{N_q\times C},$$

(22)

where DyConv2D performs content-aware feature extraction. The key innovation is that convolution kernels are dynamically generated based on proposal kernels rather than being fixed parameters.

The dynamic convolution operation aggregates features from a local spatial neighborhood with content-dependent weights. For each query $q_i\in K_{prop}$, we generate $K=4$ specialized kernels:

$$\mathrm{DyConv}(q_i,F_{agg})=\sum _{k=1}^K{\pi }_k\left(q_i\right)·(W_k\ast F_{agg}),$$

(23)

where $W_k\in {\mathbb{R}}^{3\times 3\times C\times C}$ are learnable $3\times 3$ convolution kernels, ∗ denotes 2D convolution, and ${\pi }_k\left(q_i\right)$ are query-dependent attention weights computed through a lightweight attention network:

$${\pi }_k\left(q_i\right)={\displaystyle \frac{\exp ({\mathrm{FC}}_k\left(q_i\right)/\tau )}{{\sum }_{j=1}^K\exp ({\mathrm{FC}}_j\left(q_i\right)/\tau )}},$$

(24)

where ${\mathrm{FC}}_k:{\mathbb{R}}^C\to \mathbb{R}$ are linear projections and $\tau =0.1$ is a temperature parameter that controls the sharpness of the attention distribution. Lower temperature produces sharper attention, allowing the network to specialize kernels for different object scales.

Separable Dynamic Convolution Attention. The core of our decoder is the DyConvAtten module that replaces standard multi-head cross-attention with separable dynamic convolution. Standard cross-attention computes

$$\mathrm{CrossAttn}(Q,K,V)=\mathrm{Softmax}\left({\displaystyle \frac{Q{K}^T}{\sqrt{d}}}\right)V,$$

(25)

which has complexity $O(N_q·HW·d)$. Our separable dynamic convolution decomposes this into depthwise and pointwise operations. For each query $i\in [1,N_q]$,

$$\begin{array}{cc}\hfill Q_d& ={\mathrm{Linear}}_Q\left(F_{box}^{\left(n\right)}\right)\in {\mathbb{R}}^{N_q\times C},V_d={\mathrm{Linear}}_V\left(F_{agg}\right)\in {\mathbb{R}}^{H\times W\times C},\hfill \\ \hfill W_d^{\left(i\right)}& =\mathrm{reshape}\left(\mathrm{MLP}\left(Q_d^{\left(i\right)}\right)\right)\in {\mathbb{R}}^{k\times k\times 1},V_{dw}^{\left(i\right)}=W_d^{\left(i\right)}{\ast }_{dw}{V}_d[:,:,i],\hfill \end{array}$$

(26)

where ${\ast }_{dw}$ denotes depthwise convolution that operates independently on each channel, $k=3$ is the kernel size, and MLP generates dynamic kernel weights from queries. The depthwise operation has complexity $O(N_q·C·k^2)$, which is independent of spatial resolution $HW$.

The pointwise operation then aggregates across channels:

$$W_p^{\left(i\right)}=\mathrm{Softmax}\left({\mathrm{Linear}}_p\left(Q_d^{\left(i\right)}\right)\right)\in {\mathbb{R}}^C,F_{dyconv}^{\left(i\right)}=\sum _{c=1}^C{W}_p^{\left(i\right)}\left[c\right]·V_{dw}^{\left(i\right)}[:,:,c],$$

(27)

where $W_p^{\left(i\right)}$ are channel-wise attention weights. The complete DyConvAtten operation is

$$\begin{array}{cc}\hfill F_{box}^{(n+1)}& =\mathrm{LN}(\mathrm{DyConvAtten}(F_{box}^{\left(n\right)},F_{agg})+F_{box}^{\left(n\right)}),\hfill \\ \hfill F_{box}^{(n+1)}& =\mathrm{LN}(\mathrm{FFN}\left(F_{box}^{(n+1)}\right)+F_{box}^{(n+1)}),\hfill \end{array}$$

(28)

where LN denotes layer normalization, $n\in \{0,1,\dots ,N-1\}$ with $N=6$ decoder layers, and FFN is a two-layer feed-forward network:

$$\mathrm{FFN}\left(x\right)=W_2·\mathrm{GELU}(W_1·x),$$

(29)

where $W_1\in {\mathbb{R}}^{4C\times C}$ and $W_2\in {\mathbb{R}}^{C\times 4C}$ with expansion ratio 4.

Post-Attention and Prediction Heads. After N DyConvAtten blocks, we apply multi-head self-attention (MHSA) to model inter-query relationships and generate final predictions:

$$F_{post}=\mathrm{LN}(\mathrm{MHSA}\left(F_{box}^{\left(N\right)}\right)+F_{box}^{\left(N\right)}),F_{out}=\mathrm{LN}(\mathrm{FFN}\left(F_{post}\right)+F_{post}),$$

(30)

where MHSA denotes multi-head self-attention. Final predictions for bounding boxes and classes are generated through separate fully connected layers:

$$\mathrm{Boxes}={\mathrm{FC}}_{box}\left(F_{out}\right),\mathrm{Classes}={\mathrm{FC}}_{cls}\left(F_{out}\right),$$

(31)

where ${\mathrm{FC}}_{box}$ predicts box coordinates $(x,y,w,h)$ and ${\mathrm{FC}}_{cls}$ predicts class probabilities.

The Separable Dynamic Decoder provides several advantages for vehicle detection. First, dynamic convolution generates content-aware kernels that adapt to different object scales and appearances, improving detection accuracy for vehicles with large appearance variations. Second, the separable design reduces computational complexity compared to standard Transformer decoders with full cross-attention. Third, the combination of dynamic convolution attention and multi-head self-attention enables effective feature refinement and inter-query interaction.

3.6. Loss Function

UAV vehicle detection faces severe class imbalance and scale variation challenges. We design a multi-component loss function with adaptive weighting:

$${\mathcal{L}}_{total}={\lambda }_{cls}{\mathcal{L}}_{cls}+{\lambda }_{box}{\mathcal{L}}_{box}+{\lambda }_{iou}{\mathcal{L}}_{iou}+{\lambda }_{aux}{\mathcal{L}}_{aux},$$

(32)

where ${\lambda }_{cls}=2.0$, ${\lambda }_{box}=5.0$, ${\lambda }_{iou}=2.0$, and ${\lambda }_{aux}=1.0$ are balancing weights.

Focal Loss for Classification. To address extreme foreground–background imbalance, we employ focal loss [15] that down-weights well-classified examples:

$${\mathcal{L}}_{cls}=-{\displaystyle \frac{1}{N_{pos}}}\sum _{i=1}^{N_q}\sum _{c=1}^C{\alpha }_c{(1-p_{i,c})}^{\gamma }\log \left(p_{i,c}\right)·{⊮}_{y_i=c},$$

(33)

where $N_q$ is the number of queries, C is the number of classes, $N_{pos}$ is the number of positive samples, $p_{i,c}$ is the predicted probability for query i and class c, $y_i$ is the ground-truth class label, ${⊮}_{y_i=c}$ is the indicator function, ${\alpha }_c$ balances class frequencies (0.25 for vehicles, 0.75 for background), and $\gamma =2.0$ controls focusing strength. The modulating factor ${(1-p_{i,c})}^{\gamma }$ reduces loss from easy examples, focusing training on hard cases.

Smooth L1 Loss for Localization. We employ smooth L1 loss [37] combining L1 and L2 properties to prevent gradient explosion:

$${\mathcal{L}}_{box}={\displaystyle \frac{1}{N_{pos}}}\sum _{i=1}^{N_{pos}}\sum _{j\in \{x,y,w,h\}}{\mathrm{smooth}}_{L1}({\hat{b}}_i^j-b_i^j),$$

(34)

where ${\hat{b}}_i=({\hat{x}}_i,{\hat{y}}_i,{\hat{w}}_i,{\hat{h}}_i)$ and $b_i=(x_i,y_i,w_i,h_i)$ are predicted and ground-truth boxes respectively, with $(x,y)$ denoting center coordinates and $(w,h)$ denoting width and height. The smooth L1 function is defined as

$${\mathrm{smooth}}_{L1}\left(x\right)=\left\{\begin{array}{cc}0.5x^2\hfill & \mathrm{if}\left|x\right|<1\hfill \\ \left|x\right|-0.5\hfill & \mathrm{otherwise}\hfill \end{array}\right.$$

(35)

This formulation provides smooth gradients for small errors and linear behavior for large errors [37].

Complete IoU Loss. We adopt Complete IoU (CIoU) loss [63] incorporating overlap area, center distance, and aspect ratio:

$${\mathcal{L}}_{iou}={\displaystyle \frac{1}{N_{pos}}}\sum _{i=1}^{N_{pos}}\left(1-\mathrm{IoU}({\hat{b}}_i,b_i)+{\displaystyle \frac{{\rho }^2(c_{{\hat{b}}_i},c_{b_i})}{c^2}}+\alpha v\right),$$

(36)

where $\mathrm{IoU}({\hat{b}}_i,b_i)$ is the Intersection over Union between predicted and ground-truth boxes, $\rho (c_{{\hat{b}}_i},c_{b_i})$ measures the Euclidean distance between box centers $c_{{\hat{b}}_i}$ and $c_{b_i}$, c is the diagonal length of the smallest enclosing box covering both boxes, $v={\displaystyle \frac{4}{{\pi }^2}}(\arctan {\displaystyle \frac{w^{gt}}{h^{gt}}}-\arctan {\displaystyle \frac{w}{h}}{)}^2$ captures aspect ratio consistency between predicted $(w,h)$ and ground-truth $(w^{gt},h^{gt})$ dimensions, and $\alpha ={\displaystyle \frac{v}{(1-\mathrm{IoU})+v}}$ is a trade-off parameter. This formulation provides gradients for non-overlapping boxes and penalizes shape inconsistency [63].

Auxiliary Deep Supervision. Following DETR, we apply losses to all $N=6$ decoder layers for faster convergence, with Hungarian matching assigning ground-truth boxes to queries through one-to-one bipartite matching that eliminates duplicate predictions.

4. Experiments

This section describes the experimental setup including datasets, implementation details, and evaluation metrics. We then present comprehensive comparisons with state-of-the-art methods and ablation studies.

4.1. Datasets

We evaluate AMSRDet on two benchmark datasets for multi-modal vehicle detection. The selection of these datasets is motivated by their comprehensive coverage of UAV-based and ground-level multi-modal scenarios, enabling evaluation of both detection accuracy and cross-domain generalization.

DroneVehicle Dataset [41]. The DroneVehicle dataset is a large-scale benchmark for vehicle detection in UAV imagery, containing 56,878 images with 389,779 annotated vehicle instances across five categories: car, truck, bus, van, and freight car. The dataset is captured from various altitudes (5–100 m), viewing angles, and environmental conditions including different times of day, weather conditions, and urban/rural scenes. Each image is provided in both RGB and infrared modalities with a spatial resolution of $840\times 712$ pixels. The dataset is split into training (70%, 39,815 images), validation (15%, 8531 images), and test (15%, 8532 images) sets.

Table 3. Dataset statistics for DroneVehicle and Camera-vehicle benchmarks.

Dataset	Images	Instances	Categories	Resolution	Viewpoint
DroneVehicle	56,878	389,779	5	$840\times 712$	Aerial
Camera-vehicle	12,483	87,421	4	$640\times 480$	Ground-level

provides detailed statistics.

Camera-vehicle Dataset [10]. To evaluate the generalization capability of our method, we conduct cross-dataset evaluation on the Camera-vehicle dataset, which contains 12,483 RGB–infrared image pairs captured from ground-level cameras. The dataset includes diverse scenarios such as parking lots, highways, and urban streets with varying illumination conditions. Vehicle categories include car, truck, bus, and motorcycle. This dataset provides a challenging test for evaluating model robustness to domain shift from aerial to ground-level perspectives.

4.2. Implementation Details

Our model is implemented in PyTorch (v2.2.2) and trained on 4 NVIDIA RTX 4090 GPUs. The input images are resized to $640\times 640$ pixels during training and testing. We employ AdamW optimizer with initial learning rate $1\times {10}^{-4}$, weight decay $1\times {10}^{-4}$, and cosine annealing learning rate schedule. The model is trained for 300 epochs with batch size 16. Data augmentation includes random horizontal flipping, random scaling (0.5–1.5), color jittering, and mosaic augmentation. The loss weights are set to ${\lambda }_{cls}=2.0$, ${\lambda }_{box}=5.0$, and ${\lambda }_{iou}=2.0$. For the focal loss, we use $\alpha =0.25$ and $\gamma =2.0$. The number of decoder queries is set to $N_q=300$. The MobileMamba encoder uses 4 SS2D blocks at each scale with hidden dimension 256. The reduction ratio for coordinate attention is $r=16$.

4.3. Evaluation Metrics

We evaluate detection performance using the following metrics. All metrics are computed following the COCO evaluation protocol [64].

Precision (P) measures the proportion of correct positive predictions:

$$P={\displaystyle \frac{TP}{TP+FP}},$$

(37)

where $TP$ (True Positives) is the number of correctly detected objects and $FP$ (False Positives) is the number of incorrect detections.

Recall (R) measures the proportion of actual positives correctly identified:

$$R={\displaystyle \frac{TP}{TP+FN}},$$

(38)

where $FN$ (False Negatives) is the number of missed objects.

F1 Score is the harmonic mean of precision and recall:

$$F1={\displaystyle \frac{2\times P\times R}{P+R}}.$$

(39)

Mean Average Precision (mAP) is computed as the mean of Average Precision (AP) across all categories:

$$\mathrm{mAP}={\displaystyle \frac{1}{N_c}}\sum _{c=1}^{N_c}{\mathrm{AP}}_c,$$

(40)

where $N_c$ is the number of categories, and ${\mathrm{AP}}_c$ is the average precision for category c computed as the area under the precision–recall curve. We report two mAP metrics: mAP@0.5:0.95 (primary metric, averaged over IoU thresholds from 0.5 to 0.95 with step 0.05) and mAP@0.5 (IoU threshold = 0.5). The mAP@0.5:0.95 metric provides a more comprehensive evaluation of detection quality across different localization accuracies, while mAP@0.5 enables comparison with methods that only report this metric.

Frames Per Second (FPS) measures inference speed as the number of images processed per second, evaluated on a single NVIDIA RTX 4090 GPU.

Parameters (Params) counts the total number of trainable parameters in millions (M).

Floating Point Operations (FLOPs) measure computational complexity in Giga Floating Point Operations (GFLOPs) for processing a single $640\times 640$ image.

5. Results and Analysis

5.1. Comparison with State-of-the-Art Methods

We compare AMSRDet with twenty state-of-the-art object detectors spanning three categories: (1) CNN-based detectors including YOLOv5 [65], YOLOv7 [39], YOLOv8 [48], YOLOv9 [40], YOLOv10 [66], YOLOv11 [67], YOLOv12 [68], Faster R-CNN [37], RetinaNet [15], and Adaptive Training Sample Selection (ATSS) [69]; (2) Transformer-based detectors including DETR [13], Deformable DETR [16], RT-DETR [31], DINO [47], DEIM [70], and Co-DETR [71]; and (3) Mamba-based detectors including VMamba [20], Mamba-YOLO [56], and MambaODet [57].

For fair comparison, all methods are trained and evaluated under the same settings. Single-modality methods (YOLO, DETR variants, and Mamba based) are provided with early-fused RGB-IR images (6-channel concatenation) as input.

Table 4. Performance comparison on DroneVehicle dataset. Best results are in bold, second best are underlined. Methods are grouped by architecture type. All methods use RGB + IR input (6-channel early fusion for single-modality methods). Modality column indicates native fusion support: “Early” = early fusion, “Ours” = CMGF-based fusion. We report both mAP@0.5:0.95 (primary metric) and mAP@0.5 for comprehensive evaluation.

Type	Method	Modality	P (%)	R (%)	F1 (%)	mAP@0.5:0.95 (%)	mAP@0.5 (%)	FPS	Params (M)	FLOPs (G)
CNN-based	Faster R-CNN	Early	62.3	58.7	60.4	38.2	68.5	12.5	41.3	207.4
	RetinaNet	Early	64.1	59.3	61.6	39.7	70.2	18.3	36.2	145.2
	ATSS	Early	65.8	61.2	63.4	41.3	72.8	22.1	32.1	128.6
	YOLOv5	Early	68.2	63.5	65.8	42.1	74.3	58.7	46.5	109.3
	YOLOv7	Early	70.5	65.8	68.1	43.6	76.8	61.2	37.2	105.8
	YOLOv8	Early	72.1	67.3	69.6	44.2	78.2	64.5	43.6	165.3
	YOLOv9	Early	71.8	66.9	69.3	43.9	77.6	59.3	51.8	238.9
	YOLOv10	Early	72.8	68.1	70.4	44.5	78.9	67.2	29.4	98.7
	YOLOv11	Early	73.2	68.6	70.8	44.8	79.4	65.8	31.2	102.3
	YOLOv12	Early	73.6	69.2	71.3	45.1	79.8	63.4	33.8	108.5
Transformer	DETR	Early	68.5	62.3	65.3	41.8	73.6	15.6	41.5	186.4
	Deformable DETR	Early	71.3	66.2	68.7	43.2	76.2	24.3	40.1	173.2
	RT-DETR	Early	73.4	68.5	70.9	44.7	79.1	48.6	32.8	92.4
	DINO	Early	74.2	69.1	71.6	45.3	80.2	35.2	47.5	265.7
	DEIM	Early	73.8	69.3	71.5	45.0	79.6	52.7	35.6	98.3
	Co-DETR	Early	73.9	69.8	71.8	45.1	80.4	28.4	62.3	341.2
Mamba-based	VMamba	Early	71.6	67.4	69.4	43.7	77.1	52.3	44.2	156.8
	Mamba-YOLO	Early	72.4	68.2	70.2	44.3	78.4	61.5	38.7	132.4
	MambaODet	Early	72.9	68.7	70.7	44.6	78.8	58.9	41.3	145.6
Ours	AMSRDet	Ours	75.6	71.2	73.3	45.8	81.2	68.3	28.6	47.2

presents quantitative results on the DroneVehicle test set.

AMSRDet achieves the best performance across all metrics, attaining 45.8% mAP@0.5:0.95 and 81.2% mAP@0.5 with 75.6% precision and 71.2% recall. Compared to the second-best method DINO, our approach improves mAP@0.5:0.95 by 0.5 percentage points and mAP@0.5 by 1.0 percentage point while achieving 1.94× faster inference speed (68.3 vs. 35.2 FPS) and using only 60.2% parameters (28.6 M vs. 47.5 M) and 17.8% FLOPs (47.2 G vs. 265.7 G). The significant improvements in efficiency demonstrate the effectiveness of our MobileMamba encoder and Separable Dynamic Decoder design. Notably, our mAP@0.5 of 81.2% is competitive with recent state-of-the-art dual-modality methods on the DroneVehicle benchmark, while our method achieves superior computational efficiency suitable for real-time UAV deployment.

Figure 6 provides a comprehensive comparison across multiple evaluation dimensions. AMSRDet achieves balanced performance across all metrics, particularly excelling in computational efficiency (inverse parameters and FLOPs) while maintaining competitive accuracy metrics. The radar chart visualizes the superiority of our method in achieving an optimal trade-off between detection performance and computational cost.

Compared to YOLO series detectors, AMSRDet achieves 0.7–7.6 percentage points higher mAP while maintaining competitive or superior inference speed. The latest YOLOv12 [68], which introduces attention-centric design, achieves 45.1% mAP at 63.4 FPS. Our method surpasses YOLOv12 by 0.7 percentage points in mAP while achieving 7.7% faster inference (68.3 vs. 63.4 FPS) and using 15.4% fewer parameters (28.6 M vs. 33.8 M) and 56.5% fewer FLOPs (47.2 G vs. 108.5 G). Compared to YOLOv8 which is widely used in UAV applications, our method improves mAP by 1.6 percentage points (45.8% vs. 44.2%) with 5.9% faster inference and 71.4% fewer FLOPs. These results demonstrate that our multi-modal fusion strategy provides substantial accuracy gains over single-modality detectors while maintaining superior efficiency through the MobileMamba encoder.

Compared to Transformer-based detectors, AMSRDet achieves comparable or better accuracy with significantly higher inference speed and lower computational cost. Notably, compared to our baseline RT-DETR, AMSRDet improves mAP by 1.1 percentage points (45.8% vs. 44.7%) while achieving 1.41× faster inference (68.3 vs. 48.6 FPS) and using 12.8% fewer parameters (28.6 M vs. 32.8 M) and 48.9% fewer FLOPs (47.2 G vs. 92.4 G). These improvements validate that our multi-modal fusion strategy, MobileMamba encoder, and enhanced decoder design effectively extend the RT-DETR capabilities for UAV vehicle detection. The recently proposed DEIM [70], which accelerates DETR convergence through improved matching, achieves 45.0% mAP at 52.7 FPS. Our method outperforms DEIM by 0.8 percentage points in mAP with 1.30× faster inference and 52.0% fewer FLOPs (47.2 G vs. 98.3 G), demonstrating the efficiency advantages of our Separable Dynamic Decoder over standard DETR architectures. The efficiency advantage is particularly pronounced compared to Co-DETR, where our method achieves 2.41× faster inference with only 13.8% FLOPs while maintaining competitive accuracy.

Compared to Mamba-based detectors, AMSRDet demonstrates the effectiveness of our multi-modal fusion design. VMamba [20] achieves 43.7% mAP at 52.3 FPS with 44.2 M parameters. Our method surpasses VMamba by 2.1 percentage points in mAP with 1.31× faster inference and 35.3% fewer parameters. Mamba-YOLO [56], which integrates state-space models into the YOLO framework, achieves 44.3% mAP at 61.5 FPS. AMSRDet outperforms Mamba-YOLO by 1.5 percentage points in mAP with 11.1% faster inference and 26.1% fewer parameters. MambaODet [57], designed for efficient object detection, achieves 44.6% mAP at 58.9 FPS. Our method surpasses MambaODet by 1.2 percentage points in mAP with 1.16× faster inference and 30.8% fewer parameters. These comparisons validate that our dual-stream MobileMamba architecture with cross-modal fusion is more effective than single-stream Mamba-based detectors for multi-modal UAV vehicle detection, as it explicitly models cross-modal dependencies rather than simply processing concatenated multi-modal inputs.

To further analyze the benefits of multi-modal fusion, Figure 7 compares RGB-only, IR-only, and RGB+IR fusion variants under different environmental conditions and vehicle scales. The fusion approach demonstrates superior robustness across all scenarios, with particularly significant improvements in challenging conditions such as night (+24.8 percentage points over RGB-only) and low-light scenarios where single-modality sensors fail.

Figure 8 shows qualitative detection results on the DroneVehicle dataset comparing AMSRDet with representative baseline methods (YOLOv8, RT-DETR, and Mamba-YOLO). AMSRDet successfully detects vehicles of various scales, orientations, and densities under challenging conditions including occlusion, low contrast, and complex backgrounds. The multi-modal fusion enables robust detection in both day and night scenarios, where infrared modality compensates for poor visible image quality in low-light conditions. Compared to baseline methods, AMSRDet shows fewer false positives and better localization accuracy, particularly for small and occluded vehicles.

5.2. Cross-Dataset Generalization

To evaluate the generalization capability of AMSRDet, we conduct cross-dataset evaluation on the Camera-vehicle dataset without fine-tuning.

Table 5. Cross-dataset generalization results on Camera-vehicle dataset. Models are trained on DroneVehicle and tested on Camera-vehicle without fine-tuning.

Method	P (%)	R (%)	F1 (%)	mAP (%)
YOLOv5	58.3	52.1	55.0	43.2
YOLOv7	61.2	54.8	57.8	45.7
YOLOv8	63.5	57.2	60.2	47.3
RT-DETR	64.8	58.9	61.7	48.6
DINO	66.1	60.3	63.1	49.8
Co-DETR	65.7	59.7	62.6	49.2
AMSRDet (Ours)	68.4	62.8	65.5	52.3

presents quantitative results comparing our method with baseline detectors.

AMSRDet achieves 52.3% mAP on the Camera-vehicle dataset, outperforming all baseline methods by 2.5–9.1 percentage points. The strong cross-dataset performance demonstrates that our method learns robust multi-modal representations that generalize well to different viewpoints and scenarios. The CMGF module’s ability to capture global cross-modal dependencies and the SCAF module’s adaptive multi-scale fusion contribute to this generalization capability.

Figure 9 shows qualitative results on the Camera-vehicle dataset. Despite the significant domain shift from aerial to ground-level perspectives, AMSRDet maintains robust detection performance across diverse scenarios including parking lots, highways, and urban streets with varying illumination conditions.

5.3. Multi-Scale Detection Analysis

To evaluate the multi-scale detection capability of AMSRDet, we analyze performance across different vehicle scales. Figure 10 shows detection accuracy for vehicles of varying sizes, from extra small (<32 pixels) to extra large (>256 pixels).

AMSRDet achieves superior performance across all vehicle scales, with the most significant improvements on small objects (32–64 pixels) where our method achieves 42.3% AP compared to 39.8% for RT-DETR and 40.2% for YOLOv11. This validates the effectiveness of our SCAF module in adaptively integrating multi-scale features. For medium and large vehicles, AMSRDet maintains competitive or better performance, demonstrating robust scale-invariant detection capability.

5.4. Ablation Studies

To validate the effectiveness of each component in AMSRDet, we conduct comprehensive ablation studies on the DroneVehicle validation set.

Table 6. Ablation study on DroneVehicle validation set. Each row shows the contribution of individual components. RT-DETR baseline uses standard ResNet-50 backbone with single RGB modality. Bold values in mAP and FLOPs columns indicate best performance; the final row represents our complete AMSRDet model.

MobileMamba	CMGF	SCAF	Sep. Decoder	mAP (%)	FPS	FLOPs (G)
RT-DETR baseline (RGB only)
				44.7	48.6	92.4
Our modifications (RGB + IR dual-stream)
				39.2	72.5	38.4
✓				41.8	71.3	42.1
✓	✓			43.5	69.8	44.6
✓	✓	✓		44.7	68.9	46.3
✓	✓	✓	✓	45.8	68.3	47.2

presents quantitative results, and Figure 11 visualizes the progressive improvements.

The RT-DETR baseline with single RGB modality achieves 44.7% mAP at 48.6 FPS with 92.4 GFLOPs. When we replace the backbone with a dual-stream architecture for multi-modal inputs without our proposed components, the mAP drops to 39.2% due to the lack of effective cross-modal fusion mechanisms, though FPS increases to 72.5 and FLOPs decrease to 38.4 G due to the lightweight dual-stream design. Adding the MobileMamba encoder improves mAP by 2.6 percentage points to 41.8%, demonstrating the effectiveness of selective state-space modeling for efficient feature extraction from multi-modal inputs. The CMGF module further improves mAP by 1.7 percentage points to 43.5%, validating the importance of global cross-modal fusion for integrating complementary information from infrared and visible modalities. The SCAF module contributes an additional 1.2 percentage points improvement to 44.7%, showing the benefits of adaptive multi-scale feature integration with coordinate attention. Finally, the Separable Dynamic Decoder adds 1.1 percentage points to reach 45.8% mAP, confirming the advantages of dynamic convolution for scale-adaptive prediction.

The ablation study demonstrates that each component contributes positively to the final performance, and their combination achieves the best results. Compared to the RT-DETR baseline, our full model improves mAP by 1.1 percentage points while achieving 1.41× faster inference (68.3 vs. 48.6 FPS) and using 48.9% fewer FLOPs (47.2 G vs. 92.4 G), validating the effectiveness of our multi-modal fusion strategy and efficient architecture design.

5.5. Analysis of Multi-Modal Fusion

To analyze the contribution of multi-modal fusion, we compare AMSRDet with single-modality variants trained on RGB-only or IR-only data.

Table 7. Analysis of multi-modal fusion on the DroneVehicle validation set.

Modality	mAP (%)	Day mAP (%)	Night mAP (%)
RGB only	42.3	44.8	35.2
IR only	40.1	38.7	43.9
RGB + IR (Early fusion)	43.7	45.9	39.8
RGB + IR (Late fusion)	44.2	46.3	40.6
RGB + IR (AMSRDet)	45.8	47.5	44.2

presents the results.

The RGB-only model achieves 42.3% mAP with strong performance during daytime (44.8%) but significant degradation at night (35.2%). The IR-only model shows the opposite trend with 40.1% overall mAP, performing better at night (43.9%) but worse during daytime (38.7%). Simple early fusion by concatenating RGB and IR features improves overall mAP to 43.7% but still shows performance gaps between day and night scenarios. Late fusion by averaging predictions from separate RGB and IR detectors achieves 44.2% mAP with more balanced day/night performance.

AMSRDet with CMGF-based multi-modal fusion achieves 45.8% overall mAP with balanced performance across different conditions (47.5% day, 44.2% night). The 3.5 percentage points improvement over RGB-only and 5.7 percentage points over IR-only demonstrate the effectiveness of our cross-modal fusion strategy. The balanced day/night performance validates that CMGF successfully integrates complementary information from both modalities while suppressing modality-specific noise.

6. Conclusions

In this paper, we proposed AMSRDet, an adaptive multi-scale detection network for infrared-visible vehicle detection in UAV remote sensing imagery. The framework integrates four key innovations: a MobileMamba-based dual-stream encoder with SS2D blocks for efficient hierarchical feature extraction, a Cross-Modal Global Fusion module for capturing global cross-modal dependencies, a Scale-Coordinate Attention Fusion module for adaptive multi-scale feature integration, and a Separable Dynamic Decoder for scale-adaptive prediction generation. Extensive experiments on the DroneVehicle dataset demonstrate that AMSRDet achieves superior performance compared to twenty state-of-the-art detectors, attaining 45.8% mAP@0.5:0.95 (81.2% mAP@0.5) with 68.3 FPS while maintaining only 28.6 M parameters and 47.2 GFLOPs. Cross-dataset evaluation on the Camera-vehicle dataset validates strong generalization capability, achieving 52.3% mAP without fine-tuning. Ablation studies confirm the effectiveness of each component, and multi-modal fusion analysis demonstrates the advantages of our CMGF-based fusion strategy for balanced performance across diverse conditions.

Future work will focus on self-supervised pre-training strategies to leverage large-scale unlabeled UAV imagery for improving feature representations. This direction is particularly promising given the abundance of unlabeled aerial data and the cost of manual annotation. Specifically, we plan to investigate contrastive learning approaches that exploit the natural correspondence between RGB and IR modalities as a supervisory signal, potentially enabling more robust cross-modal representations without requiring additional labeled data.