PMDet: Patch-Aware Enhancement and Fusion for Multispectral Object Detection

Highlights

What are the main findings?

• A patch-aware enhancement and fusion network (PMDet) is proposed for multispectralobject detection.
• A unified feature enhancement and aggregation mechanism is designed to improve cross-modal alignment and enable robust deep semantic fusion.

What are the implications of the main findings?

• The proposed method achieves consistent improvements in detection accuracy and robustness across multiple multispectral benchmarks.
• The patch-aware modeling strategy provides an effective and efficient paradigm for cross-modal feature fusion.

Abstract

Multispectral object detection addresses the limitations of single-modal approaches by fusing complementary information from visible and infrared images, thereby improving robustness in complex environments. However, the inter-modal representations are inherently misaligned due to sensing discrepancies, and the complementary cues they provide are often imbalanced, making it difficult to exploit modality-specific information effectively. Moreover, directly merging features from different modalities can introduce noise and artifacts that deteriorate the detection performance. To this end, this paper proposes a patch-aware enhancement and fusion network for multispectral object detection (PMDet). This method employs a dual-stream backbone equipped with the patch-aware Feature Enhancer (FE) module for cross-modal features alignment and enhancement. FE not only reinforces the feature representation of key regions but also helps to suppress local noise and enhance the model’s perception of fine textures and differences. Building on these enriched features, the patch-based Feature Aggregator (FA) module allows for efficient inter-modal feature interaction and semantic fusion with noise resistance. Specifically, both FE and FA modules leverage the shifted-patch design to preserve computational efficiency while enabling long-range modeling. In this regard, PMDet couples multi-scale cross-modal semantic enhancement with deep semantic fusion to form a stable and discriminative multimodal representation pipeline. Experiments on FLIR, LLVIP, and VEDAI demonstrate that the method outperforms mainstream approaches in detection accuracy and robustness, and ablation studies further verify the effectiveness of each module.

1. Introduction

Object detection, a core computer vision task for autonomous driving, security surveillance, and industrial inspection, has advanced significantly via deep learning [1,2,3,4,5,6,7,8]. Recently, diffusion-based detection frameworks have also demonstrated promising performance by modeling object localization as a generative refinement process, such as DiffusionDet [9] and MaDiNet [10]. However, visible image-based methods struggle in complex environments (e.g., low illumination, adverse weather) due to degraded texture/color cues, while infrared imagery offers stable thermal information but lacks fine-grained details. Multimodal object detection leverages diverse information sources, such as the texture and color cues from visible images and the stable thermal responses from infrared data, to overcome the limitations of single-modality perception in challenging scenarios (e.g., low illumination and adverse weather). By exploiting cross-modal alignment and complementary modeling, it significantly enhances detection robustness. Meanwhile, advances in multimodal image fusion and representation learning further promote effective information integration across heterogeneous modalities [11,12,13,14,15,16,17,18,19,20,21,22,23].

In recent years, there have been numerous studies on multispectral object detection. The rapid development of CNN [24,25] networks has significantly advanced research in multispectral object detection tasks. Early multispectral object detection methods were predominantly designed based on CNNs. Halfway Fusion systematically investigated four CNN-based fusion architectures, demonstrating that mid-level convolutional feature fusion achieves optimal detection performance by balancing visual details and semantic information [26]. In [27], an illumination-aware framework (IAF R-CNN) was proposed, which features a dynamic modality weighting based on lighting conditions and a comprehensive analysis of six fusion variants derived from Faster R-CNN. Addressing modality imbalance in pedestrian detection, MBNet was developed with a Differential Modality-Aware Fusion module and feature alignment mechanism for illumination-aware optimization [28]. The position misalignment problem was first tackled in [29] through AR-CNN, which introduced a Receptive Field Alignment module alongside RoI jittering strategy and confidence-aware fusion. For feature consistency enhancement, a cyclic fusion and refinement approach (CFR) was proposed to dynamically balance complementarity and coherence across spectral modalities [30]. Uncertainty-aware modules (UFF and UCG) were developed in [31] to predict and utilize RoI uncertainty for guiding feature distribution alignment between modalities, thereby improving feature discriminability.GAFF [32] introduces a Guided Attentive Feature Fusion method that adaptively fuses multi-spectral features through intra- and inter-modality attention modules. The approach utilizes pedestrian mask prediction to enhance features within target regions and dynamically select reliable modalities. With only three convolutional layers, GAFF significantly improves detection accuracy while minimizing computational overhead. CSAA [33] presents a novel fusion framework that combines channel switching with spatial attention mechanisms. The method dynamically selects discriminative feature channels from RGB and IR modalities while employing spatial attention to focus on salient regions. This pioneering work achieves fine-grained modality optimization through integrated channel and spatial processing. RSDet [34] proposes a coarse-to-fine fusion strategy to handle redundant features in RGB-IR detection. The framework incorporates a Redundant Spectrum Removal module for preliminary frequency-domain filtering and a Dynamic Feature Selection module for refined feature choice between modalities, enabling effective complementary fusion while eliminating irrelevant spectral components. Although the success of CNN-based methods is noted, their local receptive fields and fixed kernels hinder modeling long-range dependencies and global semantics.

The Transformer [35,36,37] can build global dependencies beyond CNN locality allowing for feature alignment and complementary modeling. For example, CrossFormer [38] introduces a novel cross-modal attention mechanism featuring Cross-scale Embedding Layer (CEL) and Long Short Distance Attention (LSDA) to maintain both local and global dependencies efficiently. CFT [39] designs a dual-stream fusion network with Transformer to effectively capture cross-modal dependencies. ICAFusion [40] proposes a dual cross-attention transformer feature fusion architecture that effectively aggregates complementary information from two modalities. However, Transformer-based architectures still face challenges in cross-modal fusion. First, the quadratic complexity of global self-attention leads to high computational and memory costs, especially for high-resolution features. Second, features from different backbone stages often exhibit cross-modal misalignment and statistical mismatch, making direct fusion prone to noise propagation and unstable semantic integration.

Swin Transformer [41] addresses the first two issues by introducing a hierarchical shifted-window mechanism: self-attention is constrained within local non-overlapping windows, yielding linear complexity, while shifted windows across layers enable long-range modeling. This provides an attractive trade-off between efficiency and global context modeling. Nonetheless, directly adopting Swin-like blocks in a dual-stream backbone is insufficient for multispectral detection. A practical cross-modal pipeline must (1) refine cross-modal consistency and selectively enhance complementary cues across multiple backbone stages, and (2) perform deep semantic fusion on the refined cross-modal features to produce a unified representation suitable for detection.

To address this issue, we propose a progressive cross-modal fusion framework that decouples patch-aware enhancement and fusion into two complementary modules:

• Feature Enhancer: Inserted at multiple stages of the dual-stream backbone and operating alongside the C3 module, the Feature Enhancer (FE) module performs patch-aware cross-modal enhancement. It maintains structural consistency between modalities while selectively reinforcing complementary cues (e.g., RGB textures and IR contours) at each scale, leading to more coherent and informative representations for subsequent processing.
• Feature Aggregator: Applied to deep feature levels (P3–P5), the Feature Aggregator (FA) module conducts patch-aware cross-modal fusion of the enhanced RGB and IR features. The module simply concatenates the dual-modal features and performs symmetric patch-level interaction and fusion, producing semantically aligned and modality-consistent representations suitable for downstream detection, without relying on any specific token interaction assumption.

An overview comparison of our method with state-of-the-art approaches is illustrated in Figure 1. Our contributions are summarized as follows:

Figure 1. The comparison of our method with some state-of-the-art methods. The x-axis represents mAP50, and the y-axis represents mAP. Both metrics are better when higher, and the size of the bubble is proportional to the mAP value. Our method achieves the best performance on both metrics.

• We identify a fundamental structural limitation in existing multispectral detection frameworks: the premature entanglement of multi-scale feature enhancement and deep semantic fusion. To overcome this issue, we propose an explicit decoupled enhancement–fusion paradigm, which hierarchically separates modality-specific enhancement from unified semantic fusion.
• Guided by this principle, we specifically design the Feature Enhancer (FE), a dedicated module for patch-aware cross-modal enhancement across backbone stages. FE is constructed to strengthen structural consistency and selectively amplify complementary modality cues before semantic aggregation.
• To complement the enhancement stage, we further design the Feature Aggregator (FA) for deep semantic fusion at high-level feature stages (P3–P5). Unlike conventional asymmetric interaction schemes, FA is deliberately built with symmetric patch-level interactions within a unified embedding space, enabling balanced and modality-consistent semantic integration.
• Extensive experiments on FLIR, LLVIP, and VEDAI demonstrate that the proposed decoupled design consistently improves localization precision and robustness, validating the effectiveness of explicitly separating enhancement from fusion.

2. Related Works

2.1. Multi-Modality Object Detection

Recent years have witnessed remarkable progress in object detection, driven by deep learning models such as the YOLO and DETR series. Nonetheless, single-modal methods are constrained by their data sources’ inherent limitations: visible images suffer performance degradation in adverse lighting, whereas infrared images lack detailed texture. To overcome these issues, multimodal object detection has emerged as a key research focus, whose central challenge is the effective fusion of complementary information from infrared and visible modalities.

This fusion process can be formally represented as shown in Equation (1):

$${\mathbf{F}}_{\mathrm{fused}}^{\left(\mathbf{i}\right)}=\mathrm{\Phi }\left({\mathbf{F}}_{\mathrm{rgb}}^{\left(\mathbf{i}\right)},{\mathbf{F}}_{\mathrm{ir}}^{\left(\mathbf{i}\right)}\right)$$

(1)

where ${\mathbf{F}}_{\mathrm{rgb}}^{\left(\mathbf{i}\right)}$ and ${\mathbf{F}}_{\mathrm{ir}}^{\left(\mathbf{i}\right)}$ denote the multi-scale features extracted from visible and infrared images at the i-th hierarchical level, respectively; ${\mathbf{F}}_{\mathrm{fused}}^{\left(\mathbf{i}\right)}$ represents the fused features at the corresponding level; $\mathrm{\Phi }(·)$ denotes the fusion function that integrates cross-modal information; and the superscript i indexes different feature scales within the hierarchical pyramid.

Early studies primarily employed Convolutional Neural Networks (CNNs), systematically investigating the impact of different fusion stages on performance. Researchers subsequently developed various fusion modules, utilizing operations such as feature addition, dot product, and channel concatenation. However, constrained by the local receptive fields inherent to CNNs, these methods often failed to capture long-range dependencies, limiting their effectiveness in complex scenarios.

The demonstrated success of Transformers in sequence modeling led to their adoption for multispectral detection. The self-attention mechanism provides a powerful tool for modeling global cross-modal context. Some methods directly employ Transformer architectures to learn complementary features, while others introduce cross-attention to build dynamic feature mappings between modalities, thereby deepening the modeling of intrinsic relationships and improving performance. Despite their superior modeling capabilities, the computational complexity and large parameter count of Transformers present deployment challenges. In response, strategies like parameter sharing have been explored to reduce costs while maintaining performance.

2.2. Swin Transformer

The Swin Transformer [41], a recently proposed hierarchical variant of Vision Transformer, exhibits significant architectural differences from traditional Transformers. In computer vision tasks, conventional Transformers typically partition input images into fixed-size patches and perform global self-attention computation after incorporating positional encodings. However, this processing approach somewhat disrupts the inherent two-dimensional spatial structure of images, leading to the loss of local correlation information and introducing high computational complexity.

In contrast, the Swin Transformer introduces a local window-based self-attention mechanism, which divides the image into non-overlapping windows and performs self-attention computation separately within each window. This design not only effectively preserves the two-dimensional structural characteristics of images but also maintains computational complexity within a reasonable range. To further achieve cross-window information interaction, the Swin Transformer proposes a shifted window partitioning strategy, which alternates between regular and shifted window configurations to establish global modeling capability without introducing additional computational overhead.

Owing to its hierarchical design and shifted window mechanism, the Swin Transformer significantly improves computational efficiency while maintaining strong representational capacity, achieving an effective balance between model performance and computational cost. It has become an important foundational architecture in visual Transformer research and has been widely applied to various vision tasks, including image classification and object detection.

3. Our Method

3.1. Architecture

The proposed multispectral object detection framework takes paired visible and infrared images as input and outputs bounding boxes with category predictions. As illustrated in Figure 2, the overall architecture adopts a dual-stream design combined with a progressive multimodal fusion mechanism. It contains three major components: (1) a dual-stream backbone equipped with the proposed Feature Enhancer inserted at multiple scales; (2) a multi-level semantic fusion stage constructed via the Feature Aggregator; and (3) a standard YOLOv5 neck and detection head. To ensure fair comparison and clean evaluation, the neck and head remain identical to the original YOLOv5 [42] design.

Figure 2. Overall architecture of the proposed multispectral object detection model. This end-to-end framework takes paired infrared and visible images as input and outputs object bounding boxes. The dual-stream backbone contains an infrared branch (structural cues) and a visible branch (texture cues). We introduce two Swin-style fusion modules: (a) architecture, the full dual-stream backbone with multi-scale feature extraction; (b) Feature Enhancer (FE) module, which achieves cross-modal feature alignment and enhancement through (d) patch-aware cross-modal feature enhancement (PFE) block and shifted-patch-aware cross-modal feature enhancement (SPFE) block; (c) Feature Aggregator (FA) module, which performs deep semantic fusion via (e) patch-aware unified fusion (PFF) block and shifted-patch-aware unified fusion (SPFF) block to integrate cross-modal information at the last three scales. Specifically, the shifted-patch strategy used in SPFE and SPFF is inherited from the Swin Transformer, facilitating efficient long-range dependency modeling.

3.1.1. Motivation

A key difficulty in multispectral object detection lies in jointly modeling the heterogeneous yet complementary information present in visible and infrared images. Visible images provide texture-rich appearance cues, whereas infrared images offer illumination-invariant structural responses; however, their distributions differ significantly across scenes and scales. Existing mid-level fusion approaches often perform fusion at only one depth or rely on simple concatenation, limiting their ability to maintain modality consistency and selectively exploit complementary cues.

To address these issues, we follow a progressive dual-stream fusion paradigm that performs multimodal enhancement across the entire backbone. The proposed Feature Enhancer is inserted at multiple hierarchical levels and facilitates cross-modal consistency preservation and complementary cue amplification. Building on these enhanced features, the Feature Aggregator further conducts unified patch-level semantic fusion at deeper stages. This coarse-to-fine design preserves local modality-specific information while enabling global contextual modeling, leading to a more coherent and discriminative multispectral representation.

3.1.2. Dual-Stream Backbone with Progressive Multimodal Enhancement

The backbone adopts a dual-stream architecture derived from YOLOv5, where visible and infrared images are processed through parallel pathways with the same hierarchical structure. At the shallow input stage, the two modalities are first separately encoded through convolution and C3 blocks. Their intermediate features are then selectively combined via concatenation operations, introducing initial cross-modal interaction.

As illustrated in Figure 2, the visible-light and infrared images are resized to the same input resolution before being fed into the backbone. The two streams adopt identical module configurations throughout the backbone, including convolutional layers and C3 blocks, so that corresponding stages maintain consistent feature dimensions. However, the parameters of the visible and infrared branches are not shared, allowing each stream to learn modality-specific representations. At each feature level, the two branches produce aligned outputs that facilitate subsequent cross-modal interaction and fusion. Specifically, the output feature sizes at P3, P4, and P5 are $80\times 80\times 256$, $40\times 40\times 512$, and $20\times 20\times 1024$, respectively.

To further refine multimodal information, the proposed Feature Enhancer modules are inserted at multiple scales throughout the backbone. Each Feature Enhancer takes both modality features as input and performs two key functions: (i) preserving cross-modal structural and contextual consistency, and (ii) selectively enhancing complementary modality-specific cues. These modules operate at the P3, P4, and P5 feature levels, ensuring that multimodal information is progressively optimized from lower to higher semantic stages.

As the network deepens, the enhanced features from both streams are passed into the Feature Aggregator, which integrates multimodal information in a unified token space. This deep fusion enables global semantic reasoning across modalities and scales, forming a comprehensive multimodal representation. The resulting fused features are subsequently forwarded to the YOLOv5 neck and detection head for final prediction.

Through this hierarchical fusion strategy—explicitly decoupling multi-scale modality enhancement (implemented by the Feature Enhancer) from deep semantic fusion (implemented by the Feature Aggregator)—the proposed architecture effectively captures shared structures, highlights complementary cues, and maintains robust feature alignment across modalities, ultimately improving multispectral object detection performance under diverse environmental conditions.

3.2. Feature Enhancer

In multispectral object detection, feature quality is fundamentally constrained by the inherent modality discrepancy between visible-light and infrared images. Due to different imaging principles and sensor characteristics, the two modalities often exhibit heterogeneous texture patterns, contrast levels, and noise distributions. These discrepancies make it difficult for a detector to maintain cross-modal consistency while effectively utilizing modality-specific complementary cues. Conventional fusion operations—such as direct concatenation or parallel independent processing—lack the ability to explicitly preserve shared structural information or selectively retrieve complementary cross-modal signals, which may result in diluted feature representations and fusion-induced artifacts.

To address these challenges, we introduce the Feature Enhancer, a lightweight yet powerful attention-based enhancement block inserted after the C3 module in the backbone. The C3 module provides strong local representation learning through pure convolution; however, its inherently local receptive field limits its ability to model long-range dependencies and cross-modal interactions. The Feature Enhancer is therefore designed to complement C3 by introducing global modeling capability and modality-aware reasoning. Together, the two modules form a hybrid design in which C3 focuses on robust local feature extraction, while the Feature Enhancer performs global enhancement and cross-modal semantic integration—leading to richer, more coherent, and more discriminative fused representations. The detailed architecture and forward workflow of this module are illustrated in Figure 2, and the corresponding implementation steps are summarized in Algorithm 1.

Algorithm 1: Feature Enhancer

A key objective of the Feature Enhancer is to simultaneously achieve cross-modal consistency preservation and selective complementary enhancement. Consistency preservation ensures that structural information shared by both modalities—such as object boundaries, shapes, and spatial layouts—remains aligned in the fused representation. Selective complementary enhancement enables the model to retrieve and reinforce modality-unique cues that are beneficial for detection, such as high-frequency textures from RGB or thermal stability from IR. By jointly enforcing these two properties, the Feature Enhancer suppresses modality-specific noise while amplifying meaningful cross-modal complementarities.

At the core of the module lies a Patch-aware Feature Enhancement (PFE) mechanism. Instead of window-based attention, PFE operates on block-level (patch-aware) units, treating each spatial block as a semantic entity and computing cross-modal attention at the patch level. This allows the network to adaptively determine how features from one modality contribute to the other, enabling flexible and effective cross-modal enhancement.

Consider the k-th Feature Enhancer in the visible-light branch. Let the visible and infrared inputs be ${\mathbf{F}}_{\mathrm{rgb}}\in {\mathbb{R}}^{H_k\times W_k\times C_k}$ and ${\mathbf{F}}_{\mathrm{ir}}\in {\mathbb{R}}^{H_k\times W_k\times C_k}$. The fused representation is obtained by channel concatenation:

$${\mathbf{F}}_{\mathrm{fused}}=\mathrm{Concat}({\mathbf{F}}_{\mathrm{rgb}},{\mathbf{F}}_{\mathrm{ir}})\in {\mathbb{R}}^{H_k\times W_k\times 2C_k}.$$

(2)

In the first enhancement stage, the LayerNorm operation and the Patch-aware cross-modal Feature Enhancement (PFE) block are sequentially applied to yield ${\mathbf{F}}_{\mathrm{out}1}$:

$${\mathbf{F}}_{\mathrm{out}1}=\mathrm{PFE}\left(\mathrm{LayerNorm}\left({\mathbf{F}}_{\mathrm{fused}}\right)\right)+{\mathbf{F}}_{\mathrm{fused}}.$$

(3)

A subsequent MLP block further refines the representation of ${\mathbf{F}}_{\mathrm{out}1}$ and obtains ${\mathbf{F}}_{\mathrm{out}2}$:

$${\mathbf{F}}_{\mathrm{out}2}=\mathrm{MLP}\left(\mathrm{LayerNorm}\left({\mathbf{F}}_{\mathrm{out}1}\right)\right)+{\mathbf{F}}_{\mathrm{out}1}.$$

(4)

To further enhance long-range dependencies and mitigate block-boundary artifacts, a Shifted Patch-aware Feature Enhancement (SPFE) operation is applied:

$${\mathbf{F}}_{\mathrm{out}3}=\mathrm{SPFE}\left(\mathrm{LayerNorm}\left({\mathbf{F}}_{\mathrm{out}2}\right)\right)+{\mathbf{F}}_{\mathrm{out}2}.$$

(5)

Finally, a second MLP produces the output:

$${\mathbf{F}}_{\mathrm{out}}=\mathrm{MLP}\left(\mathrm{LayerNorm}\left({\mathbf{F}}_{\mathrm{out}3}\right)\right)+{\mathbf{F}}_{\mathrm{out}3}.$$

(6)

The PFE operation begins with feature normalization:

$${\mathbf{F}}_{\mathrm{fused}}^{\prime }=\mathrm{LayerNorm}\left({\mathbf{F}}_{\mathrm{fused}}\right),{\mathbf{F}}_{\mathrm{fused}}^{\prime }\in {\mathbb{R}}^{H_k\times W_k\times 2C_k}.$$

(7)

It then separates the modalities:

$${\mathbf{F}}_{\mathrm{rgb}}^{\prime },{\mathbf{F}}_{\mathrm{ir}}^{\prime }=\mathrm{Split}\left({\mathbf{F}}_{\mathrm{fused}}^{\prime }\right).$$

(8)

After patch partitioning, cross-modal attention is computed as

$$\begin{array}{c}\mathbf{Q}={\mathbf{F}}_{\mathrm{rgb}}^{″}{W}^Q,\mathbf{V}={\mathbf{F}}_{\mathrm{rgb}}^{″}{W}^V,\\ \mathbf{K}={\mathbf{F}}_{\mathrm{ir}}^{″}{W}^K,\\ \mathbf{Z}=\mathrm{softmax}\left({\displaystyle \frac{\mathbf{Q}{\mathbf{K}}^{\top }}{\sqrt{D_k}}}\right)\mathbf{V}.\end{array}$$

(9)

Advantages of the Feature Enhancer:

• Cross-modal consistency preservation: Shared structural information is preserved through patch-level alignment, reducing spatial drift and ensuring coherent fused geometry.
• Selective complementary enhancement: The block adaptively extracts modality-unique cues that are beneficial for detection, reinforcing complementary signals while suppressing noisy or irrelevant components.
• Complementary to the C3 module: The C3 block captures strong local structures, whereas the Feature Enhancer introduces global context modeling and cross-modal semantic reasoning. Their synergy yields more robust and semantically consistent representations than either module alone.

Altogether, the Feature Enhancer substantially improves cross-modal representation quality by enforcing consistency, enhancing complementary cues, and compensating for the intrinsic locality of convolutional blocks. This establishes a stronger and more coherent foundation for subsequent deep fusion and object detection.

3.3. Feature Aggregator

After cross-modal feature enhancement, the next essential step is to construct a unified representation that integrates high-level semantic information from both modalities. Although attention-based cross-enhancement effectively improves each individual branch, it still treats RGB and IR features independently. This asymmetric interaction limits the ability to model balanced, modality-agnostic semantic relations. Deep fusion, in contrast, requires placing the two modalities on an equal footing, enabling the network to capture joint structures and category-level semantics within a shared feature space.

To achieve this, we propose the Feature Aggregator, a unified fusion block that replaces asymmetric attention with Patch-aware Cross-modal Feature Fusion (PFF). The Feature Aggregator first concatenates the RGB and IR representations, and then applies patch-aware self-attention to compute fusion weights jointly across both modalities. This mechanism effectively “levels the playing field,” allowing RGB and IR tokens to participate in attention computation symmetrically, which leads to more comprehensive and semantically coherent multimodal fusion. The detailed architecture and forward workflow of this module are illustrated in Figure 2, and the corresponding implementation steps are summarized in Algorithm 2.

Algorithm 2: Feature Aggregator

Overall design philosophy: Whereas the Feature Enhancer focuses on modality-specific enhancement and cross-modal consistency at the local level, the Feature Aggregator performs global, modality-agnostic fusion in deep layers. Together, the two modules form a coherent pipeline:

$$\mathrm{local}\mathrm{cross}\text{-}\mathrm{modal}\mathrm{enhancement}\to \mathrm{global}\mathrm{unified}\mathrm{fusion}.$$

This hierarchical structure provides a stable foundation for building deep cross-modal semantics.

The fusion process begins with channel concatenation:

$${\overline{\mathbf{F}}}_{\mathrm{fused}}=\mathrm{Concat}({\overline{\mathbf{F}}}_{\mathrm{rgb}},{\overline{\mathbf{F}}}_{\mathrm{ir}})\in {\mathbb{R}}^{H_m\times W_m\times 2C_m}.$$

(10)

The concatenated representation is then fed into the Feature Aggregator. As illustrated in Figure 2, the block begins with LayerNorm followed by a PFF module to obtain the first-level output:

$${\overline{\mathbf{F}}}_{\mathrm{out}1}=\mathrm{PFF}\left(\mathrm{LayerNorm}\left({\overline{\mathbf{F}}}_{\mathrm{fused}}\right)\right)+{\overline{\mathbf{F}}}_{\mathrm{fused}}.$$

(11)

The first-level output is refined using an MLP:

$${\overline{\mathbf{F}}}_{\mathrm{out}2}=\mathrm{MLP}\left(\mathrm{LayerNorm}\left({\overline{\mathbf{F}}}_{\mathrm{out}1}\right)\right)+{\overline{\mathbf{F}}}_{\mathrm{out}1}.$$

(12)

To enhance global continuity and expand the fusion receptive field, we apply Shifted PFF (SPFF):

$${\overline{\mathbf{F}}}_{\mathrm{out}3}=\mathrm{SPFF}\left(\mathrm{LayerNorm}\left({\overline{\mathbf{F}}}_{\mathrm{out}2}\right)\right)+{\overline{\mathbf{F}}}_{\mathrm{out}2}.$$

(13)

A final MLP produces the fused output:

$${\overline{\mathbf{F}}}_{\mathrm{out}}=\mathrm{MLP}\left(\mathrm{LayerNorm}\left({\overline{\mathbf{F}}}_{\mathrm{out}3}\right)\right)+{\overline{\mathbf{F}}}_{\mathrm{out}3}.$$

(14)

Patch-aware fusion mechanism: The PFF follows the self-attention principle, but attention is computed at the patch level, allowing RGB and IR tokens within each region to jointly determine fusion weights. After normalization,

$${\overline{\mathbf{F}}}_{\mathrm{fused}}^{\prime }=\mathrm{LayerNorm}\left({\overline{\mathbf{F}}}_{\mathrm{fused}}\right),$$

(15)

The queries, keys, and values are generated from the same fused representation:

$$\begin{array}{c}\mathbf{Q}={\overline{\mathbf{F}}}_{\mathrm{fused}}^{\prime }{W}^Q,\mathbf{K}={\overline{\mathbf{F}}}_{\mathrm{fused}}^{\prime }{W}^K,\mathbf{V}={\overline{\mathbf{F}}}_{\mathrm{fused}}^{\prime }{W}^V,\\ \mathbf{Z}=\mathrm{softmax}\left({\displaystyle \frac{\mathbf{Q}{\mathbf{K}}^{\top }}{\sqrt{D_k}}}\right)\mathbf{V}.\end{array}$$

(16)

Because queries, keys, and values are jointly computed from the fused tensor, RGB and IR contribute equally, enabling balanced cross-modal interaction.

Multi-scale progressive fusion: We apply the Feature Aggregator at the last three levels of the backbone (P3, P4, P5), with four blocks per level. This multi-scale design allows fusion to progress from higher resolution (capturing fine semantic details) to lower resolution (capturing global semantics), forming a coarse-to-fine multimodal representation hierarchy. The resulting fused features are:

$${\mathbf{F}}_{\mathrm{fuse}}^{\left(P3\right)}\in {\mathbb{R}}^{80\times 80\times 256},{\mathbf{F}}_{\mathrm{fuse}}^{\left(P4\right)}\in {\mathbb{R}}^{40\times 40\times 512},{\mathbf{F}}_{\mathrm{fuse}}^{\left(P5\right)}\in {\mathbb{R}}^{20\times 20\times 1024}.$$

Advantages of the Feature Aggregator.

• Unified cross-modal semantic fusion: RGB and IR features are fused symmetrically within a shared attention space, enabling modality-agnostic semantic integration.
• Patch-aware cross-modal modeling: PFF captures joint context within each patch and avoids modality biases caused by asymmetric attention schemes.
• Global semantic modeling with SPFF: Shifted patch fusion extends long-range dependencies and ensures consistent global fusion.
• Multi-scale progressive fusion: Applying the module at P3–P5 allows fusion to accumulate across resolutions, forming stronger hierarchical multimodal representations.

Overall, the Feature Aggregator serves as a powerful deep fusion module that integrates high-level semantics from both modalities, providing a unified foundation for subsequent detection heads.

In Algorithm 1 (Feature Enhancer) and Algorithm 2 (Feature Fusioner), PatchMergeBlock and ShiftPartition are adopted as basic feature manipulation operations. PatchMergeBlock is used to perform hierarchical feature downsampling by merging neighboring patches and increasing the channel dimension accordingly, enabling efficient multi-scale representation learning. This operation follows the patch merging strategy commonly adopted in Swin Transformer. ShiftPartition is designed to introduce spatial shifts before patch partitioning, allowing information exchange across neighboring regions that would otherwise be isolated. This operation is consistent with the shifted window mechanism in Swin Transformer, which enhances cross-region interaction while maintaining computational efficiency.

3.4. Neck and Head

The fused multi-scale features produced by the FA modules—${\mathbf{F}}_{\mathrm{fuse}}^{\left(P3\right)}\in {\mathbb{R}}^{80\times 80\times 256}$, ${\mathbf{F}}_{\mathrm{fuse}}^{\left(P4\right)}\in {\mathbb{R}}^{40\times 40\times 512}$, and ${\mathbf{F}}_{\mathrm{fuse}}^{\left(P5\right)}\in {\mathbb{R}}^{20\times 20\times 1024}$—are directly fed into the Neck of the detector.

We adopt the standard YOLOv5 Neck, which integrates a top-down Feature Pyramid Network (FPN) [43] and a bottom-up Path Aggregation Network (PAN) [44]. Given the fused features ${\mathbf{F}}_{\mathrm{fuse}}^{\left(l\right)}$, the FPN pathway enhances high-level semantics through top-down upsampling and lateral $1\times 1$ convolutions:

$${\mathbf{P}}_l={\varphi }_{\mathrm{td}}\left({\mathbf{F}}_{\mathrm{fuse}}^{\left(l\right)}\right),l\in \{P3,P4,P5\},$$

(17)

where ${\varphi }_{\mathrm{td}}(·)$ denotes the generic top-down fusion operation.

The PAN pathway further strengthens low-level localization through bottom-up aggregation with $3\times 3$ convolutions and downsampling:

$${\mathbf{N}}_l={\varphi }_{\mathrm{bu}}\left({\mathbf{P}}_l\right),l\in \{P3,P4,P5\},$$

(18)

where ${\varphi }_{\mathrm{bu}}(·)$ represents the bottom-up refinement process.

This results in the final enhanced multi-scale features ${\mathbf{N}}_{P3}$, ${\mathbf{N}}_{P4}$, and ${\mathbf{N}}_{P5}$.

Following YOLOv5, the detection Head applies lightweight task-specific $1\times 1$ convolutions for bounding box regression and classification:

$${\mathbf{O}}_l={\mathrm{Conv}}^{\mathrm{cls}}\left({\mathrm{Conv}}^{\mathrm{reg}}\left({\mathbf{N}}_l\right)\right),l\in \{P3,P4,P5\}.$$

(19)

The anchor-based prediction mechanism of YOLOv5 is retained, and all experiments use an input resolution of $640\times 640$ to ensure fair comparison with baseline detectors.

3.5. Loss Function

We employ the loss function formulation consistent with YOLOv5, which integrates multiple objective components as defined in Equation (20):

$${\mathcal{L}}_{\mathrm{total}}={\mathcal{L}}_{\mathrm{box}}+{\mathcal{L}}_{\mathrm{cls}}+{\mathcal{L}}_{\mathrm{obj}}$$

(20)

In this formulation, ${\mathcal{L}}_{\mathrm{box}}$ represents the bounding box regression loss, ${\mathcal{L}}_{\mathrm{cls}}$ denotes the classification loss, and ${\mathcal{L}}_{\mathrm{obj}}$ corresponds to the objectness confidence loss.

The classification loss ${\mathcal{L}}_{\mathrm{cls}}$ evaluates the accuracy of category predictions, computed using Binary Cross-Entropy (BCE) loss to measure the discrepancy between predicted class probabilities and ground-truth annotations across all candidate bounding boxes.

For bounding box regression, ${\mathcal{L}}_{\mathrm{box}}$ assesses the spatial alignment between predicted and ground-truth boxes. Following YOLOv5’s implementation, we utilize the Complete Intersection over Union (CIoU) loss as the regression objective, which provides comprehensive geometric evaluation by considering overlap area, center distance, and aspect ratio consistency.

The objectness loss ${\mathcal{L}}_{\mathrm{obj}}$ quantifies the accuracy of object presence predictions within each bounding box. Employing BCE loss, this component measures the agreement between predicted objectness scores and ground-truth labels, enabling the model to progressively refine its object localization capability.

For detailed implementation specifics of the loss functions and their respective balancing parameters, we direct readers to the original YOLOv5 documentation [42].

4. Experiments

4.1. Experimental Settings

4.1.1. Datasets

To comprehensively evaluate the effectiveness of the proposed method, this study conducts experiments on three public multispectral object detection datasets: LLVIP, FLIR (aligned version), and VEDAI. The fundamental characteristics and usage configurations of each dataset are described below.

LLVIP: LLVIP [45] is a specifically constructed dataset for visible-infrared object detection in low-light conditions, with image pairs undergoing rigorous spatial alignment during acquisition. The dataset comprises 15,488 well-registered RGB-IR image pairs, all captured under low-light conditions encompassing various nighttime and dimly lit scenarios. Following the official split, this study utilizes 12,025 image pairs for training and 3463 image pairs for testing.

FLIR: The FLIR [46] dataset represents a challenging multispectral object detection benchmark containing scenes under various illumination conditions, including daytime and nighttime. To ensure experimental fairness and reliability, this study employs an aligned version for evaluation, addressing spatial misalignment issues present in the original dataset. This aligned version contains 5142 spatially registered multispectral image pairs, with 4129 pairs used for training and 1013 pairs for testing. Annotations cover three object categories: “pedestrian”, “car”, and “bicycle”. All subsequent experiments involving the FLIR dataset are conducted based on this aligned version.

VEDAI: To further validate the generalization capability of the proposed method, evaluation is also performed on the VEDAI [47] aerial imagery dataset. Specifically designed for vehicle detection tasks, this dataset contains over 1200 aerial images with varying spatial resolutions (including 1024 × 1024 and 512 × 512), annotated with more than 3700 vehicle instances across nine distinct vehicle categories. Each image provides corresponding RGB and infrared modality data, establishing an effective benchmark for assessing multispectral object detection algorithm performance from aerial perspectives. Beyond the two ground-view datasets above, we also evaluate our method on the VEDAI dataset.

4.1.2. Evaluation Metrics

The primary evaluation metrics employed in this study are mAP50, mAP75, and mAP. The Average Precision (AP) for each category is defined as the area under the precision-recall curve, as formulated in Equation (21):

$$AP={\int }_0^{1}p\left(r\right)dr$$

(21)

where $p\left(r\right)$ represents the precision at recall level r. Precision measures the proportion of correctly detected objects among all positive predictions, calculated as shown in Equation (22):

$$\mathrm{Precision}={\displaystyle \frac{TP}{TP+FP}}$$

(22)

where $TP$ denotes True Positives (correctly detected objects) and $FP$ denotes False Positives (incorrect positive detections). Recall measures the proportion of successfully detected objects from all ground truth objects, defined in Equation (23):

$$\mathrm{Recall}={\displaystyle \frac{TP}{TP+FN}}$$

(23)

where $FN$ denotes False Negatives (undetected ground truth objects). Specifically, mAP50 represents the mean AP at IoU threshold 0.50, mAP75 denotes the mean AP at IoU threshold 0.75, and mAP indicates the average of AP values across IoU thresholds from 0.50 to 0.95 with a step size of 0.05. The final results are obtained by averaging the AP values across all object categories.

4.1.3. Implementation Details

We adopt the stochastic gradient descent (SGD) optimizer for model training, with a momentum of 0.937 and a weight decay of 0.0005. All experiments are conducted on a single NVIDIA GeForce RTX 3090 GPU(NVIDIA Corporation, Santa Clara, CA, USA) for 300 epochs with a batch size of 4. We follow the default learning rate scheduling strategy of YOLOv5, which involves two key parameters: the initial learning rate (lr0) and the final learning rate factor (lrf). A cosine learning rate decay is applied, where the learning rate gradually decreases from lr0 to lr0 × lrf. This strategy accelerates convergence in the early stages of training while maintaining stability in the later stages. To enhance the model’s generalization ability, we employ the Mosaic data augmentation method, which combines four training images into one. Regarding input resolution, the original image sizes used for the three datasets are 640 × 512 for FLIR, 1280 × 1024 for LLVIP, and 1024 × 1024 for VEDAI. During both training and testing, all images are resized to a unified input resolution of 640 × 640. The learning rate configurations for each dataset are summarized in Table 1.

Table 1. Training hyperparameters for different datasets.

Dataset	lr0	lrf	Momentum	Weight Decay
FLIR	0.10	0.02	0.937	0.0005
LLVIP	0.05	0.10	0.937	0.0005
VEDAI	0.02	0.20	0.937	0.0005

4.2. Quantitative Comparison with State-of-the-Art Methods

Comparison using the FLIR Dataset: Table 2 and Figure 3 present the comparative results between our model and several state-of-the-art methods on the FLIR dataset. Compared to single-modal (infrared or visible light) detection results, our method achieves improvements of 3.2% and 14.8% in the highest-standard mAP, demonstrating that our model effectively combines the advantages of both modalities and highlights the benefits of cross-modal fusion. Compared to CFT, which employs Transformer-based fusion, and ICAFusion, which utilizes cross-attention for fusion, our model shows improvements across all three metrics, indicating the effectiveness of our proposed fusion approach. Although RSDet achieves the highest mAP50% of 81.1%, our model outperforms all competitors in mAP75 (38.8%) and mAP50-95 (42.8%), suggesting superior localization accuracy under stricter detection thresholds.

Table 2. Quantitative comparison with state-of-the-art methods on the FLIR dataset. The best results are highlighted in bold.

Methods	Backbone	Modality	mAP50	mAP75	mAP
YOLOv5-L [42]	YOLOv5	RGB	59.8	21.6	28.0
YOLOv5-L [42]	YOLOv5	IR	74.8	35.3	39.6
YOLOv8-L [48]	YOLOv8	RGB	63.6	25.0	30.4
YOLOv8-L [48]	YOLOv8	IR	76.8	38.2	41.7
(2021) CFT [39]	YOLOv5	RGB + IR	78.7	35.5	40.2
(2023) CSAA [33]	ResNet50	RGB + IR	79.2	37.4	41.3
(2023) ICAFusion [40]	YOLOv5	RGB + IR	79.2	36.9	41.4
(2024) RSDet [34]	ResNet50	RGB + IR	81.1	-	41.4
(2024) CrossFormer [38]	YOLOv5	RGB + IR	79.3	-	42.1
Ours	Ours	RGB + IR	80.4	38.8	42.8

Figure 3. Performance comparison with state-of-the-art methods on the FLIR dataset. The bars represent mAP50-95 scores, while the red star markers indicate the corresponding mAP50 values for each method. Our method achieves the best performance with 42.8% mAP50-95 and 80.4% mAP50, demonstrating significant improvements over baseline methods, including single-modal (RGB/IR) and multi-modal fusion approaches.

Comparisons on the LLVIP Dataset: Table 3 and Figure 4 present a comparison of our model with several state-of-the-art methods on the LLVIP dataset. Compared to single-modality methods (such as Faster R-CNN and Cascade R-CNN that use only infrared images), our approach significantly improves detection accuracy by integrating information from both infrared and visible light images. When compared with other advanced multimodal detection methods, our method outperforms all of them in both metrics, achieving an excellent performance of 97.7% in mAP50 and 66.6% in the more demanding mAP. Relative to the two Transformer-based methods, CFT and ICAFusion, our approach only sees a marginal increase of 0.2% in mAP50, yet it achieves respective gains of 3.0% and 2.9% in the more stringent mAP, indicating that our method can more effectively integrate information from different modalities when dealing with complex multimodal data, thereby demonstrating higher localization accuracy and stronger robustness across various detection thresholds.

Table 3. Quantitative comparison with state-of-the-art methods on the LLVIP dataset. The best results are highlighted in bold.

Methods	Modality	Backbone	mAP50	mAP
YOLOv5-L [42]	RGB	YOLOv5	89.8	50.4
YOLOv5-L [42]	IR	YOLOv5	95.7	61.9
YOLOv8-L [48]	RGB	YOLOv8	91.9	54.0
YOLOv8-L [48]	IR	YOLOv8	95.2	62.1
(2019) Cascade R-CNN [49]	RGB	ResNet50	88.3	47.0
(2019) Cascade R-CNN [49]	IR	ResNet50	95.0	56.8
(2023) DIVFusion [19]	RGB + IR	YOLOv5	89.8	52.0
(2023) CSAA [33]	RGB + IR	ResNet50	94.3	59.2
(2024) RSDet [34]	RGB + IR	ResNet50	95.8	61.3
(2021) CFT [39]	RGB + IR	YOLOv5	97.5	63.6
(2024) ICAFusion [40]	RGB + IR	YOLOv5	97.5	63.7
(2025) FQDNet [50]	RGB + IR	YOLOv8	96.4	64.1
(2025) EI2Det [51]	RGB + IR	YOLOv8	98.0	63.9
(2025) FusionMamba [52]	RGB + IR	YOLOv8	97.0	64.3
Ours	RGB + IR	Ours	97.7	66.6

Figure 4. Performance comparison with state-of-the-art methods on the LLVIP dataset. The bars represent mAP50-95 scores, while the line with star markers indicates mAP50 scores. Our method achieves the highest performance on both metrics, demonstrating significant improvements over existing approaches.

Comparisons on the VEDAI Dataset: Table 4 presents a comparison of our method with several state-of-the-art methods on the VEDAI dataset. Specifically, compared to the single-modality baselines (YOLOv5-L), our model achieves significant improvements in the mAP metric, with gains of 13.1% and 12.3%, respectively. This fully demonstrates the effectiveness of multimodal information fusion. Compared with the other two advanced methods, our method shows clear advantages in three key metrics: an increase of 6.9% (mAP50), 9.7% (mAP75), and 5.9% (mAP) over the CFT method, and improvements of 1.2% (mAP50), 10.4% (mAP75), and 3.5% (mAP) over the more recent ICAFusion method. Particularly noteworthy is that our method achieves an excellent performance of 53.0% in the mAP75 metric, which is 9.4% higher than the second-best method, indicating that our model has stronger localization capabilities under high-precision detection requirements.

Table 4. Quantitative comparison with state-of-the-art methods on the VEDAI dataset. The best results are highlighted in bold.

Methods	Backbone	Modality	mAP50	mAP75	mAP
YOLOv5-L [42]	YOLOv5	RGB	57.3	30.6	32.4
YOLOv5-L [42]	YOLOv5	IR	58.4	32.9	33.2
(2021) CFT [39]	YOLOv5	RGB + IR	67.8	43.3	39.6
(2024) ICAFusion [40]	YOLOv5	RGB + IR	73.5	42.6	42.0
(2025) EI2Det [51]	YOLOv8	RGB + IR	75.6	43.8	44.0
Ours	Ours	RGB + IR	74.7	53.0	45.5

4.3. Qualitative Comparison with State-of-the-Art Methods

Detection Results: We randomly selected several sets of images from the FLIR, LLVIP, and VEDAI datasets and used YOLOv5L along with other multispectral object detection methods to perform detection on the images, with the detection results shown in Figure 5, Figure 6, Figure 7 and Figure 8. In Figure 5, compared to the detection results of YOLOv5L in the single-modality setting, it can be seen that our method, compared to single-modality methods, especially the detection results of YOLOv5L in RGB images, shows significant improvement, demonstrating that our multispectral object detection method can capture the complementary information of the two modalities and effectively enhance detection performance. Compared with other state-of-the-art methods, our method can provide more accurate detection results under insufficient lighting and occlusion, which is conducive to our cross-modal fusion backbone and intermediate fusion modules extracting better fusion features.

Figure 5. Qualitative comparison of our method with various benchmark methods on the FLIR dataset. The first and second rows show the RGB and infrared images annotated with GroundTruth. The third and fourth rows illustrate the detection results of the YOLOv5 method on RGB and IR images, respectively. The fifth and sixth rows represent the multispectral object detection results of the ICA and CFT methods. The last row shows the detection results of our method. Red bounding boxes indicate GroundTruth, while green boxes denote the detection results of each corresponding method. Red triangles indicate missed detections (miss), yellow triangles indicate false detections (false), and unmarked bounding boxes correspond to correct detections (true). Zoom in for more details.

Figure 6. Qualitative comparison of our method with the ICA method on the LLVIP dataset. The first row shows the RGB images annotated with GroundTruth. The second row depicts the detection results of the ICA method, while the third row presents the detection results of our method. Red bounding boxes indicate GroundTruth, and green boxes denote the detection results of each corresponding method. Red triangles indicate missed detections (miss), yellow triangles indicate false detections (false), and unmarked bounding boxes correspond to correct detections (true).

Figure 7. Qualitative comparison of our method with the CFT method on the LLVIP dataset. The first row shows the RGB images annotated with GroundTruth. The second row depicts the detection results of the CFT method, while the third row presents the detection results of our method. Red bounding boxes indicate GroundTruth, and green boxes denote the detection results of each corresponding method. Red triangles indicate missed detections (miss), and unmarked bounding boxes correspond to correct detections (true).

Figure 8. Qualitative comparison of multispectral object detection on the VEDAI dataset. The first row shows the RGB images annotated with GroundTruth. The second and third rows depict the detection results of the CFT and ICA methods, respectively, while the fourth row presents the detection results of our method. Red bounding boxes indicate GroundTruth, and green boxes denote the detection results of each corresponding method. Red triangles indicate missed detections (miss), yellow triangles indicate false detections (false), and unmarked bounding boxes correspond to correct detections (true).

Heatmap Visualization Results: To more intuitively demonstrate the performance of our model, we selected paired images from three experimental datasets to visualize heatmaps and compared our method with other fusion approaches, employing the Grad-CAM [53] method. As shown in Figure 9, Figure 10 and Figure 11 compared with other methods, our model focuses more on the target and pays less attention to parts unrelated to the target, with fewer instances of focusing on the wrong objects. Furthermore, for the targets it focuses on, our method’s attention is more concentrated on the target itself, with heatmaps that more closely fit the target, without excessive attention to the surroundings of the target.

Figure 9. Heatmap visualization of various multimodal object detection methods on the FLIR dataset. The first row shows the RGB images annotated with GroundTruth. The second and third rows depict the heatmap visualization results of the CFT and ICA methods, respectively. The last row presents our heatmap visualization results. Red bounding boxes indicate GroundTruth. The color intensity in the heatmaps reflects the response strength, where warmer colors (e.g., red/yellow) denote higher responses and cooler colors (e.g., blue) indicate lower responses.

Figure 10. Heatmap visualization of various multimodal object detection methods on the LLVIP dataset. The first row shows the RGB images annotated with GroundTruth. The second and third rows depict the heatmap visualization results of the ICA and CFT methods, respectively. The last row presents our heatmap visualization results. Red bounding boxes indicate GroundTruth. The color intensity in the heatmaps reflects the response strength, where warmer colors (e.g., green/yellow) denote higher responses and cooler colors (e.g., blue) indicate lower responses.

Figure 11. Heatmap visualization of various multimodal object detection methods on the VEDAI dataset. The first row shows the RGB images annotated with GroundTruth. The second and third rows depict the heatmap visualization results of the CFT and ICA methods, respectively. The last row presents our heatmap visualization results. Red bounding boxes indicate GroundTruth. The color intensity in the heatmaps reflects the response strength, where warmer colors (e.g., green/yellow) denote higher responses and cooler colors (e.g., blue) indicate lower responses.

Table 5 reports the inference time of different methods on the VEDAI dataset. Compared with the single-modality YOLOv5-L and several existing fusion-based approaches, our method exhibits higher inference latency. This is mainly attributed to the dual-stream architecture and the incorporation of patch-aware attention modules within both the Feature Enhancer and Feature Aggregator across multiple scales. While these components introduce additional computational overhead, they facilitate more effective cross-modal interaction and deeper semantic fusion, leading to improved detection performance as demonstrated in Table 4.

Table 5. Inference time and FPS comparison of different methods on the VEDAI dataset.

Methods	Time (ms)	FPS
YOLOv5-L [42]	8.8	113.6
(2024) ICAFusion [40]	13.9	71.9
(2021) CFT [39]	15.6	64.1
(2025) FusionMamba [52]	89.9	11.1
Ours	77.9	12.8

4.4. Ablation Study

4.4.1. Effectiveness of the Feature Enhancer and Feature Aggregator Modules

Table 6 and Table 7 present the ablation results of the Feature Enhancer and Feature Aggregator on the VEDAI and FLIR datasets, respectively. By progressively activating the two modules, we clearly observe how each component contributes to multispectral detection performance.

Table 6. Ablation study of FE and FA modules on the VEDAI dataset. The best results are highlighted in bold.

Model	mAP50	mAP75	mAP	Params
Baseline	68.8	40.5	39.8	51.7 M
Baseline + FE	66.8	46.2	42.0	145.2 M
Baseline + FE + FA	74.7	53.0	45.5	277.1 M

Table 7. Ablation study of FE and FF modules on the FLIR dataset. The best results are highlighted in bold.

Model	mAP50	mAP75	mAP	Params
Baseline	79.6	37.0	41.3	51.7 M
Baseline + FE	79.4	38.8	42.0	145.2 M
Baseline + FE + FA	80.4	38.8	42.8	277.1 M

On the VEDAI dataset, introducing only the Feature Enhancer yields improvements of 5.7% in mAP75 and 2.2% in mAP. This confirms that multi-scale patch-aware cross-modal enhancement effectively reduces feature inconsistency and strengthens modality complementarity, thereby improving localization accuracy. When the Feature Aggregator is further added, the model achieves 74.7% in mAP50, 53.0% in mAP75, and 45.5% in mAP—representing gains of 5.9%, 12.5%, and 5.7% over the baseline, respectively. These results verify the essential role of deep semantic fusion and demonstrate the effectiveness of the proposed progressive “cross-modal enhancement followed by deep fusion” paradigm.

The FLIR dataset exhibits a similar trend. With only the Feature Enhancer enabled, mAP75 and mAP increase by 1.8% and 0.7%, respectively, while mAP50 remains nearly unchanged ($-0.2\%$). When both modules are used together, the final performance reaches 80.4% in mAP50 and 42.8% in mAP, achieving improvements of 0.8% and 1.5% over the baseline. Notably, across both datasets, the Feature Enhancer alone contributes modest improvements in mAP50, while significant performance gains are consistently achieved only when the Feature Aggregator is added. This indicates a clear functional complementarity: the Feature Enhancer addresses cross-modal consistency and cue balancing, whereas the Feature Aggregator performs modality-agnostic deep semantic integration.

The different performance gains brought by the Feature Enhancer on VEDAI and FLIR can be attributed to the intrinsic characteristics of the two datasets. VEDAI consists of high-resolution aerial images with small objects and complex backgrounds, where accurate localization and fine-grained structural alignment play a critical role. In this scenario, the patch-aware multi-scale enhancement in the Feature Enhancer effectively reduces cross-modal misalignment and strengthens local structural consistency, leading to a more pronounced improvement in mAP75. In contrast, FLIR mainly contains ground-based scenes with relatively larger targets and more consistent spatial layouts, where detection performance is more dependent on high-level semantic discrimination than precise boundary alignment. As a result, the impact of low- and mid-level cross-modal enhancement alone is relatively limited. In addition, FLIR provides substantially more training samples (over 5k images) and higher instance counts per category than VEDAI, enabling the baseline model to learn more stable cross-modal representations, which further reduces the marginal gains from standalone feature enhancement.

From the perspective of parameter efficiency, the complete model contains 277.1 M parameters, substantially higher than the baseline‘s 51.7 M. This increase primarily results from the dual-stream backbone and Transformer-based modules. However, the corresponding accuracy improvements—especially on the challenging VEDAI dataset—demonstrate that the enhanced representational capability fully justifies the additional computational cost.

Overall, ablation analyses on both datasets confirm that the Feature Enhancer and Feature Aggregator improve multispectral detection from two complementary levels: cross-modal alignment at shallow-to-intermediate stages and semantic fusion at deep stages. Their synergistic operation forms a unified and effective multimodal feature processing framework.

As shown in Figure 12, brighter regions in the feature maps correspond to higher activation values, indicating areas that attract stronger model attention. Before the Feature Aggregator (FA), the two modalities exhibit different activation patterns. The infrared features show relatively fewer high-activation regions, mainly highlighting structural contours. Some target areas respond moderately due to limited texture information. In contrast, the RGB feature map contains more widespread and scattered high-activation responses, including noticeable activations in background regions such as roads and buildings, reflecting richer texture content but also increased noise. After the FA module, the fused feature map presents more concentrated and prominent activations around target regions, while background responses are suppressed. This indicates that the FA module effectively integrates complementary cues and produces more target-focused and discriminative semantic representations.

Figure 12. Visualization of module-specific feature evolution. From left to right: input RGB and infrared images, RGB and IR features before the Feature Aggregator (FA) module, and fused features after the FA module. Due to resizing to a uniform input size during training, some padding regions appear in the feature maps as areas with little or no activation. This illustrates how the FA module integrates cross-modal information and enhances semantic representations. Red bounding boxes indicate GroundTruth. The color intensity in the heatmaps reflects the response strength, where warmer colors (e.g., green/yellow) denote higher responses and cooler colors (e.g., blue) indicate lower responses.

4.4.2. Effect of the Number of Feature Aggregator Modules

To determine the optimal depth of deep semantic fusion, we conduct controlled experiments on the FLIR dataset by varying the number of Feature Aggregator modules. The quantitative results are summarized in Table 8. The relationship between module count and performance is non-monotonic.

Table 8. Ablation study results on the number of FA modules. The best results are highlighted in bold.

Number	mAP50	mAP75	mAP	Params
1	81.1	36.6	41.8	178.3 M
2	78.8	36.6	41.1	211.4 M
3	79.0	37.3	41.4	244.5 M
4	80.4	38.8	42.8	277.1 M

A single-module configuration achieves the highest mAP50 (81.1%), but performs suboptimally on stricter metrics (mAP75: 36.6%, mAP50–95: 41.8%). Increasing the number of modules to four yields more balanced improvements across all metrics (mAP50: 80.4%, mAP75: 38.8%, mAP50–95: 42.8%), though mAP50 decreases slightly by 0.7% compared to the single-module case.

Despite the linear increase in parameter count from 178.3 M to 277.1 M, the four-module configuration provides the best trade-off between accuracy and complexity. This indicates that adequate fusion depth is essential for achieving stable and complete cross-modal semantic integration. Consequently, the four-module design is adopted as the final configuration for all subsequent experiments.

4.4.3. Effect of the PFE and PFA Modules

Table 9 presents the ablation results of the proposed Patch-based Feature Enhancer (PFE) and Patch-based Feature Aggregator (PFA) on the VEDAI dataset. Removing either module leads to consistent performance degradation across all evaluation metrics, demonstrating the effectiveness of both components. Specifically, excluding PFE results in a decrease of 2.6% in mAP75 and 1.3% in mAP, while removing PFA causes a more noticeable drop of 5.6% in mAP75 and 3.2% in mAP.

Table 9. Ablation study of the proposed Patch-based Feature Enhancer (PFE) and Patch-based Feature Aggregator (PFA) on the VEDAI dataset. The best results are highlighted in bold.

Model	mAP50	mAP75	mAP
Ours	74.7	53.0	45.5
w/o PFE	74.5	50.4	44.2
w/o PFA	69.0	47.4	42.3

It is worth noting that the performance reduction is more significant under the stricter mAP75 metric than under mAP50. This phenomenon suggests that the proposed patch-based enhancement and aggregation strategies mainly contribute to improving localization accuracy and cross-modal structural alignment. In contrast, the variation in mAP50 is relatively smaller, indicating that the primary benefit of the proposed modules lies in refining bounding box precision rather than merely increasing detection recall under loose IoU thresholds.

5. Conclusions

This paper presents a hierarchical window-attention-based multispectral detection framework that systematically addresses three core challenges: cross-modal feature inconsistency, unbalanced complementary cues, and unstable deep fusion. Following the progressive design established in the abstract and introduction, the method decomposes multispectral fusion into two stages—cross-modal enhancement and deep semantic fusion—implemented through the Feature Enhancer and Feature Aggregator. The dual-stream backbone equipped with the Feature Enhancer performs multi-scale, patch-aware cross-modal enhancement, reducing structural inconsistency and selectively amplifying complementary cues. Building upon these refined representations, the Feature Aggregator conducts unified deep semantic fusion in a shared token space, producing modality-agnostic representations. Both modules adopt shifted-window local attention to ensure efficient long-range reasoning, forming a coherent hierarchical fusion pipeline. Experiments on FLIR, LLVIP, and VEDAI demonstrate that the proposed method consistently outperforms existing approaches, achieving 42.8% mAP on FLIR, 66.6% on LLVIP, and 45.5% on VEDAI. Ablation studies further verify the respective contributions of the two modules and confirm the effectiveness of the progressive “enhancement→fusion” paradigm.

While the method delivers strong performance, the dual-stream design and attention modules introduce additional computational cost. Although the proposed framework demonstrates robustness to mild spatial misalignment commonly observed in practical RGB–infrared datasets, handling severe cross-modal misalignment remains an open challenge. Future work will explore more lightweight variants and extend the proposed paradigm to broader multimodal settings such as depth or radar fusion. In addition, evaluating the framework on more challenging visible–thermal benchmarks, such as recently released tiny-object detection datasets [54], may further validate its generalization capability. Improving adaptability for real-world deployment on edge devices also represents an important direction for follow-up research.