Fine-Grained Multispectral Fusion for Oriented Object Detection in Remote Sensing

Highlights

What are the main findings?

• A novel visible-infrared fusion framework for oriented object detection, FGMF, is proposed to, achieving mAP₅₀ of 80.2% and 66.3% on the DroneVehicle and VEDAI datasets, respectively, with only 87.2M parameters.
• A dual-enhancement and fusion module (DEFM) is proposed for fine-grained multispectral feature calibration and fusion and an orientation aggregation module (OAM) is designed to capture directional context.

What are the implications of the main findings?

• The study provides an effective solution to the critical challenges of modality misalignment and limited orientation sensitivity, significantly advancing the robustness of object detection in complex scenarios, such as low illumination, arbitrary orientations and dense arrangement.
• The DEFM and OAM modules represent significant advancements in multispectral fusion and orientation modeling, offering transferable architectural designs that can benefit numerous vision tasks beyond remote sensing applications.

Abstract

Infrared–visible-oriented object detection aims to combine the strengths of both infrared and visible images, overcoming the limitations of a single imaging modality to achieve more robust detection with oriented bounding boxes under diverse environmental conditions. However, current methods often suffer from two issues: (1) modality misalignment caused by hardware and annotation errors, leading to inaccurate feature fusion that degrades downstream task performance; and (2) insufficient directional priors in square convolutional kernels, impeding robust object detection with diverse directions, especially in densely packed scenes. To tackle these challenges, in this paper, we propose a novel method, Fine-Grained Multispectral Fusion (FGMF), for oriented object detection in the paired aerial images. Specifically, we design a dual-enhancement and fusion module (DEFM) to obtain the calibrated and complementary features through weighted addition and subtraction-based attention mechanisms. Furthermore, we propose an orientation aggregation module (OAM) that employs large rotated strip convolutions to capture directional context and long-range dependencies. Extensive experiments on the DroneVehicle and VEDAI datasets demonstrate the effectiveness of our proposed method, yielding impressive results with accuracies of 80.2% and 66.3%, respectively. These results highlight the effectiveness of FGMF in oriented object detection within complex remote sensing scenarios.

1. Introduction

Object detection plays a crucial role in the field of remote sensing with extensive applications, such as disaster monitoring and urban planning. Due to the bird’s-eye perspective, objects in aerial images usually appear in various scales and orientations. Consequently, numerous oriented object detectors [1,2,3,4] have been proposed and achieved promising performance on challenging benchmark datasets [5,6]. However, these detectors are designed for visible images, which typically provide detailed object information, such as texture and color. These images often suffer from unfavorable conditions, such as low illumination, fog, and cloud cover, which can obscure objects and significantly degrade the detection performance. In contrast to visible imaging, infrared cameras can capture objects under adverse conditions, because of their ability to detect emitted thermal radiation. The complementary characteristics of visible (RGB) and infrared (IR) images enhance their utility, leading to their widespread application in tasks such as pedestrian detection [7,8,9] and crowd counting [10,11,12].

Although the practical significance of Infrared–Visible Oriented Object Detection (IVOOD) task is evident, one important challenge is the modality misalignment problem. Existing methods [13,14,15,16,17,18,19] generally assume perfect geometric alignment between visible–infrared image pairs and proceed with direct multimodal fusion techniques. Actually, as shown in Figure 1, the paired images are just weakly aligned due to hardware limitations such as radiation distortion and clock skew, as well as inevitable annotation inaccuracies [20]. These issues result not only in positional shifts but also in rotational discrepancies between modalities. Moreover, in extreme cases, objects may even disappear completely in one modality. These misalignments—both geometric and semantic—lead to inaccurate feature fusion, thereby degrading detection performance and highlighting the need for more robust cross-modal alignment techniques.

Another challenge in oriented object detection is limited orientation sensitivity. Objects in remote sensing images often exhibit diverse and arbitrary orientation, necessitating models that can accurately perceive and adapt to such variations. Previous approaches [1,3,4,21] often employ deformable convolutions or Rotated RoI mechanism, but often entail high computational complexity and memory consumption to capture oriented semantic information. More recently, methods [22,23,24] have utilized large square convolution kernels to obtain long-range contextual information with improved efficiency, as illustrated in Figure 2. However, these strategies fail to effectively incorporate directional priors into feature representation. Insufficient directional information impairs the modeling of target orientation and spatial relations, ultimately leading to suboptimal recognition of oriented targets.

Recently, several approaches have been introduced to tackle these challenges in RGB-IR oriented object detection tasks. For instance, TSFADet [20] predicted the deviation in position, size, and angle in the RoI head to mitigate the misalignment issue. C²Former [25] designed an inter-modality cross-attention module based on the Transformer architecture for modality calibration, but it requires substantial computational overhead. More recently, DMM [26] provided an efficient solution by employing a disparity-guided selective scan module for global context modeling without quadratic cost and a multi-scale target-aware attention module to capture features at different scales. Despite its promising performance, DMM lacks specialized components for capturing fine-grained directional information and a selective enhancement mechanism during the fusion process, limiting its overall robustness and precision.

In this paper, we propose a novel Fine-Grained Multispectral Fusion for oriented object detection method based on the one-stage detector S²ANet, called FGMF, which contains a Dual-Enhancement and Fusion Module (DEFM) and an orientation aggregation module (OAM) to cope with the above two issues. Specifically, to obtain a robust and highly discriminative fused feature representation, DEFM applies a weighted addition enhancement and a subtraction-based attention enhancement. These processes strengthen the discriminative power within each individual modality, after which the enhanced features are fed into the DMM [26] for final fusion. In addition, OAM employs several rotated strip convolutions to extract both orientation and long-range contextual information from RGB features, aiming to achieve a more robust and comprehensive understanding of the spatial and directional relationships in complex scenes. Our contributions can be summarized as follows:

• We propose a novel one-stage-based infrared–visible-oriented object detection method, named FGMF, which dynamically emphasizes object orientations and adaptively fuses complementary and similar information.
• To tackle the modality misalignment problem, we propose a Dual-Enhancement and Fusion Module with two single-modality enhancement processes to capture similar and distinct features, followed by a fusion step to achieve feature calibration.
• To handle the lack of directional priors in large square convolutional kernels, we propose an orientation aggregation module that employs a series of rotated strip convolution to encode orientation-aware features.

The remaining parts of this paper are organized as follows. Section 2 introduces related work. Section 3 elaborates on the framework of the proposed method. Section 4 details the experiments and performance evaluations that were conducted on the proposed method. Finally, Section 5 summarizes the paper.

2. Related Work

2.1. Oriented Object Detection

Oriented object detection refers to recognizing and localizing objects of interest, such as ships and vehicles, using rotated bounding boxes, which is widely used in the field of remote sensing imagery and scene text detection. Unlike horizontal detection, rotated object detection requires not only accurate localization but also precise estimation of rotation angles, which places higher demands on feature representation and spatial modeling. To achieve better rotated feature representations, a variety of methods have been developed based on horizontal detectors [27,28]. Many two-stage approaches [1,3,29] employ the rotated region of interest (RoI) extraction mechanism to align features with oriented proposals, while some one-stage methods, such as R³Det [2] and S²A-Net [4], introduce feature alignment mechanisms between rotated anchors and horizontal receptive fields to mitigate misalignment. Recently, adaptive convolution-based methods have emerged to enhance orientation sensitivity. For example, ARC [30] and GRA [31] are designed to provide the network with greater flexibility to capture the orientation information of various oriented objects by adaptively rotating the convolution kernels. Alternatively, large kernel designs such as LSKNet [23] and PKINet [24] employ large convolutional kernels to capture broader contextual information, which aids in distinguishing closely packed and rotated objects. In addition to architectural innovations, several loss functions have been proposed to improve the regression accuracy of rotated bounding boxes. CSL [32] and DCL [33] regard the angle prediction as an angular classification task to avoid the boundary discontinuity issue, while GWD [34], KLD [35], and KFIoU [36] adopt distribution-based measures that convert rotated bounding boxes into Gaussian distribution to better match the positions and orientations, which enhances rotation sensitivity and stabilize training, especially for high-precision detection tasks.

Considering the varying orientations of targets in aerial images and the limited directional sensitivity of large square convolutions, we propose an orientation aggregation module, which encodes multi-directional objects using four oriented strip convolutions to capture fine-grained orientation representations.

2.2. Infrared–Visible Image Fusion

Infrared and visible image fusion has attracted considerable attention due to its applications in tasks such as object detection [37,38,39,40] and tracking [41,42]. Fusion methods that integrate thermal signatures from infrared images and texture details from visible images can improve the perceptual robustness under challenging conditions. Based on the stage at which fusion occurs, the fusion methods can be broadly categorized into three levels: pixel-level, decision-level, and feature-level [43]. Pixel-level fusion methods directly combine raw inputs from different modalities, enabling early information interaction before feature extraction. For example, SuperYOLO [44] fuses the modalities at the pixel level, leading to a compact input representation that has a substantial reduction in computational cost. The decision-level fusion techniques, in contrast, combine the detection results from separate modality-specific branches at the final stage. Despite its ability to leverage modality-specific features for the improved performance, this strategy is often computationally expensive due to the repeated calculations across different multispectral branches. The feature-level fusion, as a balanced compromise, is commonly adopted in the field of remote sensing. Methods [20,25,40] in this field typically use multiple branches to extract modality-specific features, which are then integrated through attention mechanisms or channel concatenation. This approach could preserve structural and semantic details from the modalities, thereby improving fusion quality and further enhancing the performance of oriented object detection under variable challenging conditions.

Following the feature-level image fusion strategies, we propose a dual-enhancement and fusion module to enhance modality-specific representations and achieve more informative feature integration. The module highlights salient and complementary information from each modality through feature enhancement through additive and subtractive operations, ultimately generating consolidated and discriminative fused features.

2.3. Long-Sequence Modeling

Transformer architecture is widely used in various long-sequence modeling tasks due to its excellent global modeling capabilities [45], though it often requires high computing resources. In recent years, the Mamba architecture has emerged as an efficient alternative for long-sequence modeling. With its linear complexity, efficient state update mechanism, and streaming processing capabilities, the Mamba-based model is particularly well-suited for processing large-scale and high-resolution data. This has led to their successful application across numerous visual tasks [46,47,48,49], establishing Mamba as a powerful tool for capturing long-range dependencies. With its theoretical framework and practical applications continuously evolving, Mamba has also attracted growing interest in the domain of remote sensing vision [50,51,52]. For example, Pan-Mamba [50] introduces a channel-exchange mechanism along with cross-modal Mamba modules to achieve efficient fusion and interaction of multimodal information. RSMamba [51] proposes a dynamic multi-path activation mechanism that enhances Mamba’s capacity to model non-causal data structures, thereby improving the understanding and representation of complex semantics in remote sensing imagery. Samba [52], on the other hand, builds upon the state space model (SSM) to develop a semantic segmentation framework tailored for high-resolution remote sensing imagery, enabling efficient extraction of multi-level semantic information. ChangeMamba [53] introduces three spatio-temporal relationship modeling mechanisms, incorporated into the Mamba framework to facilitate comprehensive interaction among multi-temporal features. These efforts demonstrate the capacity of Mamba-based models to effectively tackle the distinctive challenges of geospatial vision, such as high-resolution image processing, multimodal data fusion, and complex spatial arrangements.

Inspired by Disparity-guided Multispectral Mamba [26], we propose the DEFM module, which effectively fuses visible and infrared images by leveraging the complementary and discriminative information between the two modalities, thereby improving the performance in oriented object detection.

3. Methodology

3.1. Preliminaries

State Space Models. The classical state space model (SSM) is a linear time-invariant (LTI) system that describes a system’s internal state as a set of linear equations. Specifically, given an input $x\left(t\right)\in \mathcal{R}$, it produces an output $y\left(t\right)\in {\mathcal{R}}^M$ through a hidden state $h\left(t\right)\in {\mathcal{R}}^N$. This procedure can be expressed as linear ordinary differential equations (ODEs) as follows:

$$\begin{array}{cc}\hfill \mathbf{h}\left(t\right)& =\mathbf{Ah}\left(t\right)+\mathbf{B}x\left(t\right),\hfill \end{array}$$

(1)

$$\begin{array}{cc}\hfill y\left(t\right)& =\mathbf{Ch}\left(t\right)+D\left(t\right),\hfill \end{array}$$

(2)

where $\mathbf{A}\in {\mathcal{R}}^{N\times N}$ represents the state transition matrix. $\mathbf{B}\in {\mathcal{R}}^{N\times M}$ and $\mathbf{C}\in {\mathcal{R}}^{M\times N}$ are the projection parameters. $D\in {\mathcal{R}}^1$ serves as a skip connection and is often omitted in the equation. To enhance global modeling capabilities, the input-dependent selective scanning mechanism is adopted that leverages nonlinearity and discretization by introducing a time scale parameter $\Delta$ and the zero-order hold (ZOH) [54], converting A and B into their discrete forms $\overline{\mathbf{A}}$ and $\overline{\mathbf{B}}$:

$$\begin{array}{c}\hfill \overline{\mathbf{A}}=exp(\Delta \mathbf{A}),\end{array}$$

(3)

$$\begin{array}{c}\hfill \overline{\mathbf{B}}={(\Delta \mathbf{A})}^{-1}(exp(\Delta \mathbf{A})-\mathbf{I})\Delta \mathbf{B}.\end{array}$$

(4)

After discretization, Equations (1) and (2) can also be described as follows:

$$\begin{array}{c}\hfill h_t=\overline{\mathbf{A}}{h}_{t-1}+\overline{\mathbf{B}}{x}_t,\end{array}$$

(5)

$$\begin{array}{c}\hfill y_t=\overline{\mathbf{C}}{h}_t,\end{array}$$

(6)

where $\overline{\mathbf{C}}=\mathbf{C}$. Finally, the output is obtained through a global convolution:

$$\begin{array}{c}\hfill \overline{\mathbf{K}}=\left(\mathbf{C}\overline{\mathbf{B}},\mathbf{C}\overline{\mathbf{A}}\overline{\mathbf{B}},\dots ,\mathbf{C}{\overline{\mathbf{A}}}^{\mathbf{L}-\mathbf{1}}\overline{\mathbf{B}}\right),\end{array}$$

(7)

$$\begin{array}{c}\hfill \mathbf{y}=x\overline{\mathbf{K}},\end{array}$$

(8)

where L is the length of the input sequence, and $\overline{\mathbf{K}}\in {\mathcal{R}}^L$ represents the structured convolutional kernel.

Disparity-guided Selective Scan Module. Traditional SSMs are effective for modeling long-range dependencies in 1D sequences, but they suffer from input invariant parameters, limiting their capacity for context-aware modeling. To overcome this limitation, Selective State Space Model (S6) [54], commonly known as Mamba, derives the projection parameters $\mathbf{B}$ and $\mathbf{C}$ as well as the time scale parameter $\Delta$ from the input $x_t$ to endow the model with context-aware capabilities. The input-dependent selective scanning mechanism marks a substantial improvement over the LTI systems, offering informative representation for complex sequential data. However, such 1D sequential models cannot be applied directly to visual data since it is necessary to capture 2D spatial information in images. To bridge the gap between 2D visual data and 1D sequences, VMamba [46] developed 2D Selective Scan (SS2D) mechanism that scans image patches along four distinct directions to effectively establish a global receptive field while maintaining computational efficiency.

Inspired by this paradigm, disparity-guided selective scan module (DSSM) is proposed in the field of multispectral object detection. Specifically, given the three inputs $F_{rgb},F_{ir},F_d\in {\mathcal{R}}^{N\times C\times H\times W}$, they are first flattened, followed by the concatenation of $F_d$ with $F_{rgb}$ and $F_{ir}$ to get two sequences $F_{rgb-d},F_{ir-d}\in {\mathcal{R}}^{\mathcal{N}\times \mathcal{2}\mathcal{HW}\times \mathcal{C}}$. To better capture bidirectional spatial context, reverse scans are performed to produce complementary sequences ${\tilde{F}}_{rgb-d},{\tilde{F}}_{ir-d}$. Therefore, the four sequences are processed through Equations (3)–(6), yielding four corresponding output sequences $F_{rgb-d}^{\prime },F_{ir-d}^{\prime },{\tilde{F}}_{rgb-d}^{\prime },{\tilde{F}}_{ir-d}^{\prime }\in {\mathcal{R}}^{N\times 2HW\times C}$. The output $Y_{rgb-d}$ and $Y_{ir-d}$ are obtained by combining the forward outputs with the reversed counterparts:

$$\begin{array}{c}\hfill Y_{rgb-d}=F_{rgb-d}^{\prime }+Reverse\left({\tilde{F}}_{rgb-d}^{\prime }\right),\end{array}$$

(9)

$$\begin{array}{c}\hfill Y_{ir-d}=F_{ir-d}^{\prime }+Reverse\left({\tilde{F}}_{ir-d}^{\prime }\right),\end{array}$$

(10)

where $Reverse(·)$ denotes the operation that reverses the input along the second dimension. Then, the two sequences are split along the same dimension, with the first half of modality-specific features to be maintained for the modality-specific features and the second half of differential features to be discarded. The resulting sequences $Y_{rgb-d}$ and $Y_{ir-d}$ are then reshaped to produce the final fused representations.

3.2. Overview

As shown in Figure 3, our proposed FGMF comprises four main components: a dual-stream backbone network, an orientation aggregation module (OAM), a dual-enhancement and fusion module (DEFM) and a detection head. Given a pair of infrared and visible images, we initially perform feature extraction independently through two modality-specific backbones. Subsequently, the visible features are input into an orientation aggregation module, which provides essential orientation information for objects of varying directions. Next, the two modality-specific features flow into a dual-enhancement and fusion module, which refines them with semantic interaction between infrared and visible feature maps. Finally, all bounding boxes and classification scores are obtained through the detection head. In the following, we will elaborate on the proposed DEFM and OAM in detail.

3.3. Dual-Enhancement and Fusion Module

Feature fusion plays a critical role in multimodel models. In contrast to Transformer-based approaches [20,55,56], which often suffer from quadratic computational complexity with respect to the sequence length, our proposed DEFM has two enhancement processes and a mamba-based fusion to achieve better feature representations while avoiding quadratic computational overhead. As presented in Figure 4, we firstly adopt a weighted addition enhancement approach at the first enhancement process. Select the maximum value across the channel dimension on the initial fused feature map ${\mathbf{X}}_{ini{t}_{f}use}$, and then generate the modality-specific significance weights:

$$\begin{array}{c}\hfill {\mathbf{X}}_{init\_fuse}={\mathbf{X}}_{rgb}+{\mathbf{X}}_{ir},\\ \hfill u_1={\displaystyle \frac{{\mathbf{X}}_{rgb}}{Max\left({\mathbf{X}}_{init\_fuse}\right)}},\\ \hfill u_2={\displaystyle \frac{{\mathbf{X}}_{ir}}{Max\left({\mathbf{X}}_{init\_fuse}\right)}},\end{array}$$

(11)

where $Max$ represents the operation of selecting maximum values. When $u_1\approx 1$, it means that the visible feature value at a certain position is close to the maximum fusion value, indicating that the visible mode is important at this position. $u_2\approx 1$ means that the infrared mode contributes more and has a higher weight. A new fused feature can be defined as follows:

$$\begin{array}{c}\hfill {\mathbf{W}}_{sum}=u_1·{\mathbf{X}}_{rgb}+u_2·{\mathbf{X}}_{ir}.\end{array}$$

(12)

The complementary enhancement of visible and infrared features as follows:

$$\begin{array}{cc}\hfill {\mathbf{X}}_{rgb}^1=& {\mathbf{W}}_{sum}·{\mathbf{X}}_{rgb},\hfill \\ \hfill {\mathbf{X}}_{ir}^1=& {\mathbf{W}}_{sum}·{\mathbf{X}}_{ir}.\hfill \end{array}$$

(13)

Element-wise summation fusion highlights complementary feature similarity, but it suppresses differential features between modalities. Therefore, at the second enhancement stage, the difference between the visible and infrared feature maps is computed, and then it is passed through a convolutional layer followed by a sigmoid activation function to derive the weighting factor ${\mathbf{W}}_{sig}$:

$$\begin{array}{c}\hfill {\mathbf{W}}_{sub}=\sigma \left(Conv({\mathbf{X}}_{rgb}-{\mathbf{X}}_{ir})\right),\end{array}$$

(14)

where $\sigma$ denotes the sigmoid function. This feature is used to modulate the visible and infrared features, generating the second enhanced visible ${\mathbf{X}}_{rgb}^2$ and infrared ${\mathbf{X}}_{ir}^2$ features, as well as the subtraction feature map ${\mathbf{X}}_{sub}^2$:

$$\begin{array}{c}\hfill {\mathbf{X}}_k^2={\mathbf{X}}_k+{\mathbf{W}}_{sub}·{\mathbf{X}}_k,\end{array}$$

(15)

where $k\in \{\mathrm{rgb},\mathrm{ir},\mathrm{sub}\}$ indexes the visible, infrared, and their differential features, respectively. After enhancement, the normalized visible and infrared features, denoted as ${\mathbf{X}}_{rgb}^{norm}$ and ${\mathbf{X}}_{ir}^{norm}$, along with the differentiated features ${\mathbf{X}}_{sub}^2$ are used for a high-dimensional projection to enrich modality-specific representations. Depthwise separable convolutions $DWConv(·)$ are then performed on the three features to enable effective interaction among channels:

$$\begin{array}{c}\hfill {\mathbf{X}}_k^{proj}=Project({\mathbf{X}}_{rgb}^{norm};{\mathbf{X}}_{ir}^{norm};{\mathbf{X}}_{sub}^2),\end{array}$$

(16)

$$\begin{array}{c}\hfill {\mathbf{X}}_k^{dwc}=SiLU\left(DWConv\left({\mathbf{X}}_k^{proj}\right)\right),\end{array}$$

(17)

$$\begin{array}{cc}\hfill {\mathbf{X}}^{dsmm}=& DSMM({\mathbf{X}}_{rgb}^{dwc};{\mathbf{X}}_{ir}^{dwc};{\mathbf{X}}_{sub}^{dwc}),\hfill \end{array}$$

(18)

where $LN(·)$ represents Layer Normalization, $Project(·)$ denotes a linear projection operation, and $SiLU$ means the SiLU activation function. These three features are then fed into Disparity-guided Selective Scan Module (DSSM) [26], which can obtain complementary information from another modality.

Despite its strength in modeling long-range dependencies, DSSM struggles to effectively represent correlations across channels. Therefore, a simple channel attention operation is used to capture channel-domain features, including global max pooling $GMP(·)$ and global average pooling $GAP(·)$. Specifically, for a given input feature $\mathbf{Y}$, we perform the following operations:

$$\begin{array}{cc}\hfill {\mathbf{Y}}^{avg}=& GAP\left(SiLU\right(\mathbf{Y}\left)\right),\hfill \\ \hfill {\mathbf{Y}}^{max}=& GMP\left(SiLU\right(\mathbf{Y}\left)\right),\hfill \end{array}$$

(19)

$$\begin{array}{cc}\hfill {\mathbf{Y}}^{cr}=& ReLU\left(Con{v}_{1\times 1}\left({\mathbf{Y}}^{avg}\right)\right)+ReLU(Con{v}_{1\times 1}\left({\mathbf{Y}}^{max}\right).\hfill \end{array}$$

(20)

The channel attention output is applied as follows:

$$\begin{array}{cc}\hfill {\mathbf{Y}}^{ca}=& \sigma \left({\mathbf{Y}}^{cr}\right)·\mathbf{Y}+\mathbf{Y}.\hfill \end{array}$$

(21)

3.4. Orientation Aggregation Module

The greater the use of orientation-aware information for object discrimination, the better the performance in oriented object detection tends to be. However, as shown in Figure 5, previous structures [22,23,24] that employ large convolutions for global modeling often fail to adequately capture precise object orientations. To enhance rotation and texture details for subsequent infrared and visible image fusion, as depicted in Figure 6, we propose the orientation aggregation module that models the orientation and long-range contextual relationships. Concretely, the input RGB feature is first fed into a convolution with a 3 × 3 kernel size, followed by four directional strip convolutions. A horizontal and vertical convolution capture axis-aligned patterns, while two diagonal convolutions operate on 45° and −45° rotated feature maps:

$$\begin{array}{c}\hfill \overline{\mathbf{X}}=Con{v}_{3\times 3}\left({\mathbf{X}}_{in}\right),\end{array}$$

(22)

$$\begin{array}{c}\hfill {\mathbf{X}}_{\mathrm{agg}}=\sum _{\theta \in \left\{0°,90°,45°,-45°\right\}}{\mathcal{R}}_{-\theta }\left({Conv}_{k_{\theta }}\left({\mathcal{R}}_{\theta }\left(\overline{\mathbf{X}}\right)\right)\right).\end{array}$$

(23)

Here, we adopt a strip convolution size $k_{\theta }$ of $1\times 9$ and $\mathcal{R}$ is defined as the rotated transformation of strip convolution with $\theta$ or $-\theta$. These multi-orientation features are aggregated via summation, then transformed through a pointwise convolution and GELU activation to obtain directional and long-range contextual representations:

$$\begin{array}{c}\hfill {\mathbf{X}}_{act}=GELU\left(Con{v}_{1\times 1}\left({\mathbf{X}}_{\mathrm{agg}}\right)\right).\end{array}$$

(24)

In order to focus more on the relevant spatial regions for target detection, we employ a spatial selection mechanism that utilizes the average and maximum pooling (denoted as ${\mathcal{P}}_{\mathrm{avg}}$ and ${\mathcal{P}}_{\max }$):

$$\begin{array}{c}\hfill {\mathbf{X}}_{\mathrm{avg}}={\mathcal{P}}_{\mathrm{avg}}\left({\mathbf{X}}_{act}\right),{\mathbf{X}}_{\max }={\mathcal{P}}_{\max }\left({\mathbf{X}}_{act}\right),\end{array}$$

(25)

To facilitate effective information interaction, we concatenate the spatially pooled features and employ a convolutional layer ${\mathcal{F}}^{2\to N}(·)$ to transform the aggregated features into N channels:

$$\begin{array}{c}\hfill \widehat{\mathbf{X}}={\mathcal{F}}^{2\to N}\left(Concat[{\mathbf{X}}_{\mathrm{avg}},{\mathbf{X}}_{\max }]\right).\end{array}$$

(26)

The final output of OAM is then obtained by performing an element-wise product between the input feature ${\mathbf{X}}_{in}$ and the spatial attention $\widehat{\mathbf{X}}$:

$$\begin{array}{c}\hfill {\mathbf{X}}_{out}={\mathbf{X}}_{in}·\widehat{\mathbf{X}}.\end{array}$$

(27)

3.5. Loss Function

As previously described, our proposed FGMF framework is a dual-stream feature extraction architecture. In this work, we adopt the one-stage method S2ANet [4] as the basic detector, following the same experimental configurations as current state-of-the-art (SOTA) approaches. The final loss function in FGMF consists of two components: one applied to the RGB branch and the other to the fusion branch. The overall loss function can be defined as follows:

$$\begin{array}{c}\hfill {\mathbf{L}}_{total}={\mathbf{L}}_{om{a}_{c}ls}+{\mathbf{L}}_{om{a}_{r}eg}+{\mathbf{L}}_{def{m}_{c}ls}+{\mathbf{L}}_{def{m}_{r}eg},\end{array}$$

(28)

where ${\mathbf{L}}_{om{a}_{c}ls}$ and ${\mathbf{L}}_{om{a}_{r}eg}$ denote the classification and regression loss from the RGB branch, while ${\mathbf{L}}_{def{m}_{c}ls}$ and ${\mathbf{L}}_{def{m}_{r}eg}$ represent those from the fusion branch. All the classification losses are implemented by using cross-entropy loss [57] and the regression loss employs smooth l1 loss [58] function.

4. Experiments and Analysis

4.1. Datasets

DroneVehicle [59] is the most extensively used drone-based infrared–visible dataset for arbitrary-oriented object detection, comprising 28,439 image pairs with 953,087 instances. It includes five categories: car, bus, truck, van, and freight car. The dataset covers various scenes, such as residential areas, urban roads, parking lots, and other scenarios from day to night. Officially, the dataset is split into 17,990, 1469, and 8980 image pairs for training, validation, and testing, respectively. In the experiments, the 100-pixel-wide white borders are removed from all images.

VEDAI [60] is a multispectral vehicle detection dataset, comprising 1246 pairs of infrared and visible images, each with a resolution of $512\times 512$. We select nine vehicle categories for detection: car, truck, tractor, camper, van, pick-up, boat, plane, and others. In the experiments, in line with [26], a ten-fold cross-validation protocol is employed to train and test the model.

4.2. Implementation Details

As shown in

Table 1. Server and environment parameters.

Configuration Items	Parameters
CPU	Intel(R) Xeon(R) Platinum 8336C
GPU	GTX 4090 24 G
Memory	100 G
Operating System	Ubuntu 22.04
Deep Learning Framework	MMRotate and MMDetection (Based on PyTorch 2.5.1)

, we utilize the MMRotate [61] and MMDetection [62] platforms to implement the proposed method on the server with a single GeForce RTX 4090. S²ANet is selected as the basic oriented object detector with the pretrained backbone network VMamba [46]. AdamW optimizer is adopted with the initial learning rate of 0.0001 and the weight decay of 0.05. The model was trained for 12 epochs, with the learning rate divided by 10 at epochs 8 and 11. For the DroneVehicle dataset, IR modality labels are used as ground truth, aligning with the previous studies [20,25,59]. For the VEDAI dataset, the model is trained on all nine categories, including those with fewer than 50 instances, to evaluate performance under imbalanced data conditions.

4.3. Evaluation Metrics

To better evaluate the oriented object detection performance, mean average precision (mAP) and frames per second (FPS) are selected as evaluation metrics. FPS quantifies the inference speed and mAP measures detection accuracy across different categories, which can be obtained by averaging the mean of average precision (AP) scores:

$$\begin{array}{c}\hfill mAP={\displaystyle \frac{1}{k}}\sum _{k=1}^{k}A{P}_k,\end{array}$$

(29)

where k is the number of object classes, and $A{P}_k$ represents the average precision of the $kth$ category. The area under the precision–recall (P-R) curve is defined as the AP value, providing a robust metric for assessing classifier performance.

$$\begin{array}{c}\hfill AP={\int }_0^{1}P\left(R\right)·dR,\end{array}$$

(30)

where Precision $\left(P\right)$ and Recall $\left(R\right)$ are essential metrics that measure the proportion of correctly identified samples among all the positive predicted samples and the actual positive samples:

$$\begin{array}{c}\hfill P={\displaystyle \frac{TP}{TP+FP}},\end{array}$$

(31)

$$\begin{array}{c}\hfill R={\displaystyle \frac{TP}{TP+FN}}.\end{array}$$

(32)

True positives $\left(TP\right)$, false positives $\left(FP\right)$, and false negatives $\left(FN\right)$ refer to the number of correctly identified targets, misclassified nontargets such as background or other categories, and undetected true targets, respectively. In this work, mAP with an IoU threshold of 0.5 is employed to evaluate the performance.

4.4. Ablation Studies

Effect of rotated strip convolution. As revealed in

Table 2. Ablation study on the effectiveness of rotated strip convolution with different sizes and angles in orientation aggregation module.

Exp	0°	45°	90°	−45°	mAP50
I	(3, 3)	–	–	–	78.5
II	(1, 9)	–	–	–	78.3
III	–	–	(1, 9)	–	78.4
IV	(1, 9)	–	(1, 9)	–	79.3
V	–	(1, 9)	–	(1, 9)	78.1
VI	(1, 9)	(1, 9)	(1, 9)	(1, 9)	79.6
VII	(1, 11)	(1, 11)	(1, 11)	(1, 11)	79.4

, conventional square convolution with size of $3\times 3$ achieves only 78.5% mAP₅₀, primarily due to its limited receptive field failing to capture long-range contextual dependencies and directional patterns critical for oriented object detection. Replacing the square kernel with strip convolutions at 0° (Exp II) or 90° (Exp III) alone performs comparably to the square convolution. Notably, significant gains are observed when combining complementary orientations. Employing $1\times 9$ kernels at both 0° and 90° (Exp IV) improves accuracy to 79.3%, demonstrating that orthogonal directional contexts provide mutually reinforcing information. Extending this design to a complete set of four orientations: 0°, 45°, 90°, and −45° (Exp VI) further boosts performance to 79.6% mAP₅₀, a total gain of 1.1% over the baseline. This confirms that multi-directional coverage is essential to capture the full spectrum of object rotations. However, simply increasing the kernel size is not always beneficial. Expanding from $1\times 9$ to $1\times 11$ (Exp VII) causes a slight performance drop to 79.4%, likely due to increased sensitivity to background clutter and irrelevant context as the receptive field grows beyond the scale of target structures.

Effect of each module. As depicted in

Table 3. Ablation study for each proposed module in FGMF. Enh. denotes enhancement stage.

Exp	OAM	DEFM			mAP50
		DSSM	1st Enh.	2nd Enh.
I	–	✔	–	–	79.3
II	✔	✔	–	–	79.6
III	✔	✔	✔	–	79.8
IV	✔	✔	✔	✔	80.2

, the baseline model that incorporates neither rotated features nor modality-specific enhancement achieves only 79.3%. mAP₅₀. Integrating DSSM with our proposed orientation aggregation module (OAM) yields a 0.3% improvement in mAP₅₀. The introduction of the first enhancement stage further increases mAP₅₀ by 0.2% by leveraging cross-modality complementary information, thereby enriching unimodal feature representations. Subsequent integration of the second enhancement stage explicitly amplifies discriminative differential features between visible and infrared modalities. The full configuration achieves the best performance at 80.2%, representing a total gain of 0.9% over the baseline.

4.5. Comparison with State-of-the-Art Methods

Results on DroneVehicle. To demonstrate the effectiveness of our proposed method, we provide four single-modality methods (Faster R-CNN [27], RetinaNet [28] RoI Transformer [1], and S²ANet [4] and eight multimodal object detection approaches, including CIAN [63], AR-CNN [64], UA-CMDet [59], TSFADet [20], CALNet [65], C²Former [25], E2E-MFD [66], and DMM [26] for comparisons. It can be seen from

Table 4. Comprehensive comparative experiments on the DroneVehicle dataset. The bold and underlined fonts indicate the top two performances, respectively.

Modality	Method	Venue	Year	Basic Detector	Car	Truck	Freight Car	Bus	Van	mAP50 (%)	Params (M)
RGB	Faster R-CNN [27]	NeurIPS	2015	–	79.0	49.0	37.2	77.0	37.0	55.9	41.1
	RetinaNet [28]	ICCV	2017	–	78.5	34.4	24.1	69.8	28.8	47.1	36.4
	RoI Trans [1]	CVPR	2019	–	61.6	55.1	42.3	85.5	44.8	61.6	55.1
	S2ANet [4]	TGRS	2021	–	80.0	54.2	42.2	84.9	43.8	61.0	38.6
IR	Faster R-CNN [27]	NeurIPS	2015	–	89.4	53.5	48.3	87.0	42.6	64.2	41.1
	RetinaNet [28]	ICCV	2017	–	88.8	35.4	39.5	76.5	32.1	54.5	36.4
	RoI Trans [1]	CVPR	2019	–	89.6	51.0	53.4	88.9	44.5	65.5	55.1
	S2ANet [4]	TGRS	2021	–	89.9	54.5	55.8	88.9	48.4	67.5	38.6
RGB+IR	CIAN [63]	INFORM FUSION	2019	-	89.98	62.47	60.22	88.90	49.59	70.23	-
	AR-CNN [64]	TNNLS	2021	Faster R-CNN	90.1	64.8	62.1	89.4	51.5	71.6	-
	UA-CMDet [59]	TCSVT	2022	RoI Trans	87.5	60.7	46.8	87.1	38.0	64.0	-
	TSFADet [20]	ECCV	2022	-	89.9	67.9	63.7	89.8	54.0	73.1	104.7
	CALNet [65]	ACM MM	2023	–	90.3	76.2	63.0	89.1	58.5	75.4	-
	C2Former [25]	TGRS	2024	S2ANet	90.2	68.3	64.4	89.8	58.5	74.2	100.8
	E2E-MFD [66]	NeurIPS	2024	–	90.3	79.3	64.6	89.8	63.1	77.4	-
	DMM [26]	TGRS	2025	S2ANet	90.5	77.7	73.2	90.0	65.1	79.3	88.0
	FGMF (Ours)	-	2025	S2ANet	90.5	78.5	74.7	90.3	67.1	80.2	87.2

that most multimodal methods show superior performance over single-modal detectors, confirming the critical advantage of multimodal fusion in aerial oriented object detection. Overall, the proposed FGMF achieves SOTA performance among all competitors at 80.2%, surpassing the previous approaches C²Former, E2E-MFD, and DMM by 6.0%, 2.8%, and 0.9%, respectively, which demonstrates its superiority in the IVOOD task. In the Car category, both existing approaches (CALNet, C²Former, E2E-MFD, and DMM) and our method exceed 90%. Moreover, our method is the first to surpass 90% in the Bus category, achieving 90.3%. Furthermore, significant improvements are seen in challenging cases where our proposed method attains notable oriented object detection accuracy of 74.7% on the Freight Car category and 67.1% mAP on the Van category, representing substantial gains over previous methods in these underrepresented categories. In addition, our method attains the best performance while notably requiring fewer parameters (87.2 M) than all other IVOOD approaches.

Results on VEDAI. Current multimodal fusion detection studies on the VEDAI dataset with oriented bounding box are still under limited exploration. As reported in

Table 5. Comprehensive comparative experiments on the VEDAI dataset. The bold and underlined fonts indicate the top two performances, respectively.

Modality	Method	Car	Truck	Tractor	Camping Car	Van	Pick-Up	Boat	Plane	Others	mAP50
RGB	RetinaNet [28]	48.9	16.8	15.9	21.4	5.9	37.5	4.4	21.2	14.1	20.7
	S2ANet [4]	74.5	47.3	55.6	61.7	32.5	65.1	16.7	7.1	39.8	44.5
	Faster R-CNN [27]	71.4	54.2	61.0	70.5	59.5	67.6	52.3	77.1	40.1	61.5
	RoI Trans [1]	77.3	56.1	64.7	73.6	60.2	71.5	56.7	85.7	42.8	65.4
IR	RetinaNet [28]	44.2	15.3	9.4	17.1	7.2	32.1	4.0	33.4	5.7	18.7
	S2ANet [4]	73.0	39.2	41.9	59.2	32.3	65.6	13.9	12.0	23.1	40.0
	Faster R-CNN [27]	71.6	49.1	49.2	68.1	57.0	66.5	35.6	71.6	29.5	55.4
	RoI Trans [1]	76.1	51.7	51.9	71.2	64.3	70.7	46.9	83.3	28.3	60.5
RGB+IR	C2Former + S2ANet [25]	76.7	52.0	59.8	63.2	48.0	68.7	43.3	47.0	41.9	55.6
	DMM + S2ANet [26]	77.9	59.3	68.1	70.8	57.4	75.8	61.2	77.5	43.5	65.7
	FGMF + S2ANet (Ours)	78.2	57.6	66.8	69.7	57.9	74.1	57.6	87.4	47.1	66.3

, multimodal detection methods significantly outperform unimodal approaches. Under single-modality evaluation, S²ANet [4] just achieve this on both RGB at 44.5% and IR at 40%, which highlight the challenges of modality-specific detection under poor conditions. In contrast, multimodal-based methods could achieve substantial gains under the same detector for fair comparison. Among these methods, our proposed FGMF achieves the best performance with 66.3% mAP₅₀, which is significantly higher than C²Former + S²ANet (55.6%) [25] and DMM + S²ANet (65.7%) [26]. In particular, our method achieves 87.4% mAP in the Plane category and 47.1% mAP in the Others category, exceeding the second-best method by significant margins of 9.9% and 3.6%, respectively. These results demonstrate superior capability in identifying planes and generalizing well to varying targets. Furthermore, FGMF maintains competitive accuracy across common categories such as Car, Pick-up, and Tractor. These results demonstrate that FGMF effectively leverages cross-modal information while preserving modality-specific characteristics, leading to superior detection accuracy in complex scenarios.

Visual Comparisons. For the DroneVehicle dataset, we provide some visual detection results of the compared method in Figure 7. DMM [26] exhibits significant limitations in detection accuracy, including misclassifications, missed detections, and inaccurate localization. These errors stem from its limited ability to discern subtle intermodal discrepancies and its lack of orientation modeling. In contrast, our FGMF delivers more accurate localization and identification through discriminative feature extraction, demonstrating robustness in low-visibility and densely distributed scenarios. For the VEDAI dataset, as presented in Figure 8, we visualize the results of the model in different scenarios. The objects are in complex scenarios including blur, dense arrangement, and angle change. Through these results, we find that FGMF is robust to the above challenging situations, which verifies its effectiveness.

Speed versus Accuracy. Under a batch size of 1 and an input resolution of $512\times 512$, our proposed FGMF achieves superior detection accuracy of 80.2% mAP with a competitive inference speed of 12.5 FPS, as depicted in Figure 9, demonstrating an excellent trade-off between performance and efficiency.

5. Discussion

Despite the compelling results achieved by our FGMF framework, there are still several limitations that point to valuable future research directions. First, the feature extraction pathways for the two modalities are not entirely symmetric. This architectural asymmetry can introduce a representation-level discrepancy between visible and infrared features, potentially impairing the calibration capability of the dual-enhancement and fusion module. Second, the orientation aggregation module relies on convolution kernels with a set of predefined rotation angles. Although effective in capturing dominant directional patterns, this design inherently lacks the flexibility to adapt to arbitrary orientations. As a result, the model may struggle to accurately represent objects whose orientations fall between the predefined angles, particularly in complex scenes with highly varied object layouts.

6. Conclusions

In this paper, we introduce a Fine-Grained Multispectral Fusion (FGMF) framework, specifically designed for the infrared–visible-oriented object detection task in remote sensing. Our research tackles two key problems in this domain: (1) the modality misalignment and partial or complete modality disappearance under varying environmental conditions, which presents challenges on the effective multimodal fusion process, and (2) the large orientation disparities among targets in images, which make it difficult for the models to capture rotated representations. The proposed FGMG cope with these issues through its novel design, which mainly consists of two modules: a dual-enhancement and fusion module that integrates complementary and correlated cross-modal features to address the modality misalignment issue and an orientation aggregation module that employs rotated strip convolutions to model directional context and long-range dependencies. Extensive experiments on DroneVehicle and VEDAI datasets demonstrate that FGMF delivers significant performance gains as a robust and versatile framework for infrared–visible-oriented object detection.