A Multi-Stage Cross-Modal Fusion Framework for Visible–Infrared Object Detection

Abstract

In visible–infrared object detection under complex environments, cross-modal fusion often suffers from spatial misalignment, semantic inconsistency, and unstable feature responses under varying illumination conditions. To address these issues, this paper proposes a dual-branch visible–infrared object detection framework based on YOLOv11. A staged refine–interact–modulate cross-modal fusion structure (RIFN) is designed to progressively stabilize the cross-modal fusion process. Specifically, the Cross-Modal Spatial–Channel Calibration (CSK) module first suppresses cross-modal discrepancies, the bidirectional Cross-Modal Synergy (CSM) module subsequently performs complementary interaction modeling, and the Complementary Modulation (CM) module further recalibrates fused representations to improve fusion stability under complex environments. Extensive experiments conducted on the public LLVIP and M³FD datasets demonstrate the effectiveness and generalization capability of the proposed framework. Compared with DEYOLO, the proposed method improves mAP@50 by 2.4% on the M³FD dataset while reducing GFLOPs by 2.4, demonstrating superior computational efficiency. In addition, compared with the Transformer-based GM-DETR, the proposed framework improves mAP@50 by 2.1% on the LLVIP dataset and increases inference speed by 210 FPS, achieving higher detection accuracy together with better inference efficiency. Experimental results demonstrate that the proposed staged cross-modal fusion strategy can effectively enhance the robustness of visible–infrared object detection under complex illumination conditions and background interference.

1. Introduction

In object detection tasks under complex environments, relying on a single sensing modality is often insufficient to simultaneously satisfy the requirements of robustness and discriminability. Visible images exhibit inherent advantages in representing fine-grained texture details and high-level semantic information; however, their informative content degrades significantly under adverse conditions such as nighttime, rain, fog, or low-contrast scenes. In contrast, visible–infrared object detection jointly exploits visible and infrared information to achieve more robust object detection under complex illumination conditions and background interference [1]. Consequently, effectively exploiting the complementary strengths of visible and infrared modalities has become an important research direction for improving object detection performance in complex scenarios [2,3].

Early approaches predominantly adopted feature-level fusion strategies, in which features from different modalities were directly fused or concatenated within deep networks to alleviate the representational limitations of single-modality perception [4]. These methods improve visible–infrared object detection performance through feature enhancement and cascaded semantic aggregation. In low-light detection tasks, illumination compensation and contextual modeling were further introduced to mitigate feature degradation and spatial semantic information loss caused by insufficient illumination, thereby enhancing detection reliability under low-light conditions [5]. Subsequently, multiscale modeling and attention mechanisms were further introduced to strengthen complementary relationships across different feature hierarchies [6]. For example, cross-modal attention and confidence-aware mechanisms were employed to improve detection stability and discriminability [7,8].

However, as application scenarios extend toward real-world complex environments, several critical challenges associated with cross-modal fusion have become increasingly evident. On the one hand, visible and infrared data acquired in practice often suffer from spatial misalignment and weak alignment, where naive feature aggregation is prone to introducing fusion ambiguity and amplifying background noise interference [9]. On the other hand, local convolutional structures exhibit inherent limitations in modeling long-range cross-modal complementary dependencies, causing critical complementary cues to be gradually weakened during deep semantic reasoning [10]. Moreover, variations in illumination conditions may lead to unstable response distributions within fused representations, thereby further destabilizing the cross-modal fusion process under complex environments [11].

In summary, although existing methods have achieved significant progress in cross-modal object detection, maintaining robust cross-modal fusion under real-world complex environments remains a critical challenge.

In practical scenarios, fusion instability is often caused by coupled factors, including spatial–semantic discrepancies prior to interaction, insufficient complementary dependency modeling during interaction, and uneven response distributions after fusion. These issues weaken modality complementarity and further reduce the stability and detection reliability of cross-modal fusion under complex environments.

Among one-stage detection architectures, YOLOv11 provides a favorable balance between detection accuracy and computational efficiency, while its mature multi-scale architecture offers a stable backbone for staged cross-modal fusion. Considering that the proposed fusion framework already introduces additional interaction and modulation operations, adopting an excessively heavy detector architecture may further increase computational redundancy and reduce real-time inference capability. Therefore, YOLOv11 serves as a suitable baseline for constructing an efficient and lightweight visible–infrared multimodal detection framework. Nevertheless, the original YOLOv11 framework is primarily designed for single-modality perception and lacks explicit mechanisms for cross-modal alignment, complementary interaction, and fusion stability modeling.

Motivated by these observations, this study proposes a staged dual-branch visible–infrared object detection framework that progressively stabilizes the cross-modal fusion process under complex environments through discrepancy-aware refinement, complementary interaction modeling, and adaptive post-fusion modulation. The main contributions of this work are summarized as follows:

(1)

A dual-branch YOLOv11-based multimodal detection framework is proposed. The proposed framework preserves discriminative visible/infrared feature representations during feature extraction and constructs a staged refine–interact–modulate cross-modal fusion structure (RIFN), where progressive cross-modal collaborative modeling is conducted only at high-level semantic stages rather than through unconstrained direct cross-modal feature interaction, thereby reducing noise interference and cross-modal semantic inconsistency while improving fusion stability under complex environments.

(2)

A Cross-Modal Spatial–Channel Calibration (CSK) module is proposed to sequentially perform spatial discrepancy suppression and channel semantic recalibration prior to cross-modal interaction. The proposed module does not directly aggregate weakly aligned multimodal features, but instead progressively refines spatial and semantic consistency before feature interaction, thereby improving fusion stability under weakly aligned scenarios.

(3)

A Bidirectional Cross-Modal Synergy (CSM) module is designed to model long-range complementary dependencies through constrained bidirectional interaction. The proposed module does not directly perform unrestricted full-channel global interaction, but instead decomposes projected features into pass-through and interaction branches, thereby alleviating excessive modality coupling while preserving modality-specific semantic representations.

(4)

A lightweight Complementary Modulation (CM) module is further introduced to adaptively recalibrate fused single-stream representations after cross-modal interaction. The proposed module does not introduce additional cross-modal interaction, but instead suppresses illumination-driven post-fusion response imbalance through complementary spatial–channel modulation, thereby improving fusion consistency and detection robustness under complex environments.

The remainder of this paper is organized as follows. Section 2 reviews recent studies on visible–infrared object detection and cross-modal fusion methods, with particular emphasis on modality discrepancy handling and global interaction modeling. Section 3 presents the proposed dual-branch YOLOv11-based framework and details the staged refine–interact–modulate fusion strategy, including the CSK, CSM, and CM modules for discrepancy suppression, complementary dependency modeling, and adaptive recalibration, respectively. Section 4 describes the experimental settings and provides comprehensive evaluations on the LLVIP and M³FD datasets, including quantitative comparisons, qualitative analyses, and ablation studies. Finally, Section 5 concludes the paper and discusses the limitations of the proposed framework as well as potential future research directions.

2. Related Work

2.1. Cross-Modal Fusion Detection Methods for Modality Discrepancy and Misalignment

In practical application scenarios, it is often difficult to strictly register visible and infrared images, and spatial misalignment, scale inconsistency, and local distortions commonly occur between modalities. To address these challenges, existing studies have explored weak alignment modeling, geometric compensation, and fusion stability enhancement from different perspectives. Chen et al. [12] introduced cross-modal constraints at the feature level to alleviate the impact of spatial misalignment on fusion performance; however, their approach exhibits limited adaptability to local misalignment and scale variations. Lv et al. [13] proposed a dual-branch detection framework, DGE-YOLO, which enhances cross-modal feature interaction through attention-guided feature aggregation. However, the method primarily focuses on feature enhancement and lacks explicit modeling of spatial misalignment and modality discrepancy, resulting in limited fusion stability under complex or weakly aligned scenarios. Fu et al. [14] presented YOLO-Adaptor, which explicitly models geometric deviations through multimodal adapters and improves cross-modal adaptability to some extent; nevertheless, its performance gains remain limited under complex backgrounds and severe occlusion conditions.

In addition, several studies enhance feature consistency by modeling complementary information or employing iterative cross-modal interactions. Yan et al. [15] improved multispectral pedestrian detection performance by explicitly modeling cross-modal complementary information; however, their method lacks effective suppression of modality discrepancies prior to fusion. Du et al. [16] further enhanced cross-modal feature interaction by proposing a channel interaction enhancement fusion module, which strengthens the common concern information between different feature maps and improves the discriminative capability of cross-modal representations. With respect to specific application scenarios, Zhang et al. [17] and Shi et al. [18] achieved performance improvements in terms of pedestrian detection and nighttime road damage detection, respectively, by adopting object-aware and scene-driven fusion strategies; nevertheless, these methods strongly depend on task-specific priors or application scenarios, resulting in limited generalizability.

2.2. Cross-Modal Fusion Methods Based on Global Interaction Modeling

To compensate for the inherent limitations of convolutional structures in modeling long-range dependencies, recent studies have increasingly introduced global interaction mechanisms, extending cross-modal fusion toward higher-level relational modeling. Besides explicit global interaction modeling, several studies have attempted to enhance multimodal fusion stability through confidence-aware and reliability-guided fusion strategies. Zhang et al. [19] suppressed the negative influence of unreliable modalities through an evidence fusion strategy; however, the performance of this approach is highly dependent on the accuracy of confidence estimation. Li et al. [20] further constructed a confidence-aware fusion framework based on Dempster–Shafer theory to alleviate modality conflict interference, yet the method still lacks sufficient modeling capability for complex cross-modal interactions and long-range dependency representation.

Transformer-based approaches leverage self-attention and cross-modal attention mechanisms to exhibit strong global modeling capability in multispectral object detection tasks. Wang et al. [21] enhanced cross-modal global interaction through attention-based semantic fusion, thereby improving long-range dependency modeling between visible and infrared modalities. Lee et al. [22] introduced a cross-modality attention transformer to facilitate semantic interaction between visible and infrared modalities, thereby improving global contextual aggregation capability. You et al. [23] further incorporated multiscale aggregation transformers to promote complementary information exchange across different feature hierarchies. This strategy improves multiscale representation consistency. However, such methods are often associated with high computational complexity and sensitivity to feature alignment quality. Liu et al. [24] introduced an attention-enhanced feature mapping network that integrates attention-based feature fusion and graph convolutional feature mapping to alleviate both intermodal and intramodal discrepancies.

In terms of structural efficiency, the Swin transformer proposed by Liu et al. [25] reduces the computational cost of global modeling through window-based attention; however, its original design is not specifically optimized for cross-modal fusion. The hierarchical transformer introduced by Han et al. [26] enhances both local and global representation capabilities, yet additional constraints are still needed to prevent excessive cross-modal coupling during fusion.

For visible–infrared object detection tasks, Xiao et al. [27] proposed GM-DETR, which introduces the DETR framework into multispectral detection and exhibits advantages in end-to-end modeling; however, it still suffers from limitations in terms of small-object detection and training convergence speed. Within the YOLO family, Wang et al. [28] and Xie et al. [29] achieved lightweight multimodal detection by enabling feature interaction through dynamic convolution or self-attention guidance. Nevertheless, the cross-modal interaction processes in these methods lack an explicit stagewise design, making them more susceptible to noise interference in complex scenarios.

3. Methods

3.1. Overall Framework Overview

To address the insufficient suppression of modality discrepancies and inadequate cross-modal complementary modeling in traditional YOLO-based multimodal detection methods, this study constructs a dual-branch RGB–infrared object detection framework based on YOLOv11. The proposed framework adopts structurally symmetric RGB and infrared branches to separately extract texture–semantic information and structurally salient responses. Although the two branches share the same architectural design, their parameters are independently learned without weight sharing during training, thereby preserving modality-specific texture and semantic representations during feature extraction.

As illustrated in Figure 1, for the shallow feature stage, the P3 feature from the RGB branch is directly retained and fed into the neck to preserve fine-grained texture details and localization cues required for small-object detection. Cross-modal fusion is intentionally avoided at this stage because shallow visible and infrared features are more sensitive to spatial misalignment, modality discrepancy, and noise interference.

To meet the cross-modal fusion requirements at high-level semantic feature stages, this study proposes a staged refine–interact–modulate cross-modal fusion structure (RIFN). Specifically, a Cross-Modal Spatial–Channel Calibration (CSK) module is deployed at the intermediate semantic stage (P4) to alleviate spatial and semantic discrepancies between modalities prior to interaction, whereas a bidirectional cross-modal synergy (CSM) module and a complementary modulation (CM) module are further introduced at the deeper semantic stage (P5) for cross-modal complementary dependency modeling and adaptive post-fusion recalibration, respectively. In this manner, cross-modal fusion is progressively conducted only at high-level semantic stages while reducing noise interference in shallow feature representations.

After staged cross-modal fusion, the neck adopts an FPN + PAN structure to aggregate multiscale fused feature representations. Subsequently, YOLO-based decoupled detection heads are employed to predict category probabilities and bounding box coordinates at three detection scales. During training, the model is supervised using a multitask loss function consisting of binary cross-entropy (BCE) loss for classification and complete intersection over union (CIoU) loss together with distribution focal loss (DFL) for bounding box regression. During inference, non-maximum suppression (NMS) is employed to remove redundant predictions.

3.2. Refine–Interact–Modulate Neck (RIFN)

The proposed RIFN follows a staged “refine–interact–modulate” design paradigm and decouples the cross-modal fusion process into three progressively connected stages: discrepancy-aware feature refinement, complementary interaction modeling, and adaptive modulation. This structure is primarily applied to high-level semantic features, aiming to explicitly suppress cross-modal discrepancies, enable controlled and efficient cross-modal information interaction, and further enhance fusion stability through adaptive recalibration, thereby providing spatially aligned and semantically consistent fused representations for subsequent detection.

3.2.1. Cross-Modal Spatial–Channel Calibration (CSK) Module

Prior to cross-modal interactions, features from different modalities often exhibit inconsistencies in both spatial response locations and channel-wise semantic distributions, where direct interaction is prone to introducing redundant or even conflicting information.

Conventional cross-modal attention or feature alignment methods typically perform feature aggregation directly on weakly refined representations, which may further amplify spatial ambiguity and semantic inconsistency during interaction.

To address this issue, this study designs a Cross-Modal Spatial–Channel Calibration (CSK) module, which explicitly performs spatial discrepancy suppression and channel semantic recalibration in a sequential manner prior to cross-modal interaction, thereby alleviating spatial and channel alignment discrepancies among multimodal features before fusion.

The core principle of the CSK module follows a sequential processing scheme of “spatial alignment first, followed by channel semantic recalibration,” as illustrated in Figure 2. In this progressive data flow, RGB and infrared features are first processed through cross-modal spatial calibration to alleviate local spatial inconsistencies. The refined features are then forwarded to the channel recalibration stage for adaptive semantic modulation.

This ordering facilitates the suppression of cross-modal spatial misalignment noise while reducing ambiguity in the subsequent channel weight learning process, thereby enhancing fusion stability.

By decoupling spatial alignment and channel recalibration into two sequentially executed stages, CSK effectively mitigates ambiguity during attention estimation and suppresses error propagation caused by cross-modal spatial misalignment in subsequent fusion stages.

Let the same-scale features extracted from the RGB and infrared branches be denoted as $F^{rgb},F^{ir}\in {\mathbb{R}}^{C\times H\times W}$, respectively, where C, H, and W denote the channel number, feature height, and feature width. The spatial refinement stage aims to learn pixelwise cross-modal calibration weights.

Specifically, the two feature maps are first concatenated along the channel dimension to form

$$X=\left[F^{rgb};F^{ir}\right]\in {\mathbb{R}}^{2C\times H\times W}$$

(1)

where [;] denotes channel-wise concatenation.

The concatenated feature $X$ is then fed into a pointwise convolutional MLP composed of two 1 × 1 convolution layers, with ReLU and Sigmoid activations applied after the first and second layers, respectively, to generate dual spatial weight maps $S$.

No batch normalization is applied in this lightweight spatial weighting branch in order to preserve adaptive modulation responses. The output is subsequently split into two spatial weight maps, $S_1,S_2\in {\mathbb{R}}^{1\times H\times W}$.

A cross-guided strategy is then employed to perform spatial calibration:

$$X_1^{\left(s\right)}=X_1+{\alpha }_s\cdot S_2\odot X_2,X_2^{\left(s\right)}=X_2+{\alpha }_s\cdot S_1\odot X_1$$

(2)

where $\odot$ denotes elementwise multiplication, and ${\alpha }_s$ denotes a learnable scalar coefficient for spatial calibration. The parameter is implemented as a learnable scalar coefficient, initialized to 1.0 and adaptively optimized during training to control the strength of cross-modal spatial refinement. This design enables each modality to be spatially calibrated by the saliency distribution of the other modality, thereby alleviating cross-modal response peak shifts and local spatial misalignment issues.

After spatial refinement, the CSK module further performs consistency- and complementarity-aware recalibration along the channel dimension. Specifically, the spatially refined features $X_1^{\left(s\right)}$ and $X_2^{\left(s\right)}$ are concatenated to form

$$X^{\left(s\right)}=[X_1^{\left(s\right)};X_2^{\left(s\right)}]\in {\mathbb{R}}^{2C\times H\times W}$$

(3)

Subsequently, channel-wise descriptor vectors $g\in {\mathbb{R}}^{6C}$ are extracted using three statistical pooling operations, namely, average pooling, standard deviation pooling, and max pooling. These statistics characterize channel importance from complementary perspectives, including overall activation magnitude, distribution dispersion, and extreme response intensity.

The concatenated descriptor $g$ is then projected through a standard MLP composed of Linear–ReLU–Linear–Sigmoid operations to generate dual channel weight vectors $U$, which are subsequently reshaped into two sets of channel coefficients $U_1,U_2\in {\mathbb{R}}^{C\times 1\times 1}$. No batch normalization is applied in this branch to avoid suppressing inter-channel response variations during adaptive channel weighting.

Finally, cross-channel calibration is performed to obtain the refined outputs:

$$X_1^{\prime }=X_1^{\left(s\right)}+{\alpha }_c\cdot U_2\odot X_2^{\left(s\right)},X_2^{\prime }=X_2^{\left(s\right)}+{\alpha }_c\cdot U_1\odot X_1^{\left(s\right)}$$

(4)

where ${\alpha }_c$ denotes a learnable scalar coefficient for channel recalibration. Similarly, ${\alpha }_c$ is initialized to 1.0 and jointly optimized during training to regulate the contribution of cross-modal channel compensation.

Through the joint refinement of spatial and channel dimensions, the CSK module outputs a pair of aligned and redundancy-suppressed cross-modal feature representations, which are subsequently fed into the following bidirectional interaction module for further modeling.

3.2.2. Bidirectional Cross-Modal Synergy Module (CSM)

The input to the CSM consists of the two feature representations refined by the CSK module. As illustrated in Figure 3, the CSM follows an “interact–fuse–compress” processing paradigm.

Different from conventional full-channel cross-modal attention methods that directly perform global interaction over all feature dimensions, the proposed CSM first decomposes the projected features into pass-through and interaction branches through linear projection. The pass-through branch preserves stable modality-specific semantic information, whereas the interaction branch performs constrained bidirectional cross-modal attention for complementary dependency modeling.

This design enables long-range cross-modal interaction while alleviating excessive modality coupling and computational redundancy. The resulting interaction features are subsequently aggregated and projected back into fused feature representations.

Considering that convolutional features exhibit strong modeling capability within local receptive fields but are inherently limited in capturing long-range global dependencies, the two-dimensional feature maps are flattened into token sequences to facilitate attention-based modeling. However, directly performing cross-modal attention computation over full-channel tokens would not only incur substantial computational overhead but also risk degrading the inherent discriminative representations of each modality because of excessive feature coupling.

To address this issue, a projected feature decomposition strategy is introduced, in which the token representations of each modality are linearly projected and decomposed into a pass-through branch and an interaction branch:

$$\left[Y_m,Z_m\right]=Split\left(\delta \left(T_m{W}_m\right)\right),Y_m,Z_m\in {\mathbb{R}}^{N\times {\displaystyle \frac{C}{r}}}$$

(5)

$$N=H\times W$$

(6)

where N denotes the number of flattened spatial tokens. $r$ denotes the channel reduction ratio and is set to 4 in this work. $\delta (\cdot )$ represents the ReLU activation function, and $W_m$ denotes the learnable linear projection matrix for modality m. $T_m\in {\mathbb{R}}^{N\times C}$ denotes the flattened token sequence of modality m. The pass-through branch $Y_m$ directly preserves the refined features from the CSK module to maintain modality-specific semantic representations, whereas the interaction branch $Z_m$ is specifically designed for cross-modal attention-based interaction.

This “divide-and-conquer” design restricts attention computation to a subset of channels while preserving information completeness, thereby achieving an effective balance between representation capability and computational efficiency.

Within the interaction branch, the CSM adopts a bidirectional cross-attention mechanism, in which the two feature streams are mutually treated as query sources:

$$C_1=Attn\left(Z_1,Z_2\right),C_2=Attn\left(Z_2,Z_1\right)$$

(7)

$$Attn(Z_a,Z_b)=Softmax\left({\displaystyle \frac{Q_a{K_b}^T}{\sqrt{d}}}\right)V_b$$

(8)

where $Q_a$, $K_b$, and $V_b$ denote the query, key, and value projections generated from the corresponding token sequences, respectively, and d denotes the embedding dimension used for attention normalization.

This symmetric bidirectional interaction scheme enables each feature stream to actively select informative cues from the other modality for enhancement, rather than being passively aggregated.

To reduce the computational complexity in high-resolution scenarios, the CSM performs spatial reduction only on the key and value branches, which significantly decreases the attention computation cost while preserving the global modeling capability.

The interaction outputs are subsequently concatenated with the pass-through branch features along the channel dimension and projected back to the original channel size through linear mapping. With the incorporation of residual connections and normalization operations, two enhanced sequence representations are obtained for the respective modalities.

Finally, the two enhanced sequences are concatenated along the channel dimension and reshaped back into two-dimensional feature maps. The final fusion and compression are accomplished through a parallel structure consisting of a residual mapping path and a deep embedding path, yielding a unified fused feature representation.

This symmetric bidirectional interaction strategy allows each modality to actively select informative complementary cues from the other modality rather than passively accepting the fused results. As a result, more stable and complementary cross-modal representations can be obtained for subsequent fusion.

3.2.3. Complementary Modulation Module (CM)

Although the fused features obtained after CSK and CSM exhibit improved cross-modal semantic consistency, pre-interaction refinement and complementary interaction modeling mainly focus on discrepancy suppression and dependency aggregation before or during cross-modal interaction. However, even after deep multimodal integration, the resulting representations may still exhibit uneven spatial–channel response distributions under complex illumination conditions, which cannot be fully alleviated by pre-interaction calibration alone. Such illumination-driven imbalance is particularly evident in complex nighttime or low-contrast scenarios, where dominant modality responses may suppress complementary cues and consequently affect feature stability.

To address this issue, a complementary modulation (CM) module is introduced along the fusion pathway. As shown in Figure 4, the fused feature representations are first decomposed into primary and auxiliary branches for complementary modulation. The auxiliary branch generates spatial guidance information to modulate the primary branch features, whereas the primary branch simultaneously produces channel attention information for reverse recalibration of the auxiliary branch. The modulated features are subsequently aligned and integrated to generate the final recalibrated representations.

Notably, although the CM module is incorporated within the overall RIFN framework, it differs from CSK and CSM in that it does not introduce additional cross-modal interactions. Instead, its core function is to recalibrate the already fused single-stream representations, thereby correcting residual spatial–channel response imbalance introduced during the fusion process.

The core objective of the CM module is to explore the complementary relationships among different channel subsets within a single fused feature stream through a primary–auxiliary branch design and a spatial–channel complementary modulation mechanism. Given an input fused feature map $X\in {\mathbb{R}}^{C\times H\times W}$, the feature is first asymmetrically split along the channel dimension into a primary branch and an auxiliary branch:

$$X=\left[X_m, X_s\right], X_m\in {\mathbb{R}}^{\left(C-C_s\right)\times H\times W}, X_s\in {\mathbb{R}}^{C_s\times H\times W}$$

(9)

$$C_s=max(\lfloor C/r\rfloor ,1)$$

(10)

where $C_s$ denotes the auxiliary branch channel number, ⌊·⌋ represents the floor operation, and $r$ denotes the channel compression ratio and is set to 4 in this work.

The primary branch $X_m$ is responsible for the main feature representation and adopts a cascaded structure consisting of two $3\times 3$ convolutional layers followed by a $1\times 1$ convolutional layer, which expands the receptive field while achieving channel alignment.

In contrast, the auxiliary branch $X_s$ retains only a small subset of channels and employs a single $1\times 1$ convolution for linear projection, serving as a modulation guidance signal.

In the spatial modulation stage, the auxiliary branch feature $X_s$ is used to generate a spatial attention map $A_s$, which is subsequently applied to the primary branch feature $X_m$:

$$M_s=A_s\odot X_m$$

(11)

where $A_s$ denotes the spatial attention map, $X_m$ represents the primary branch feature, $M_s$ denotes the spatially modulated feature, and $\odot$ represents element-wise multiplication.

In the channel modulation stage, the primary branch feature $X_m$ is taken as input to model channel importance $A_c$ through depth-wise separable convolution and global average pooling. The resulting channel attention is then reversely applied to the auxiliary branch feature:

$$M_c=A_c\odot X_s$$

(12)

where $A_c$ denotes the channel attention map, ${\mathrm{X}}_s$ represents the auxiliary branch feature, and $M_c$ denotes the channel-modulated feature.

Before fusion, the auxiliary branch feature is projected to the same channel dimension as the primary branch through channel alignment. Then, the spatially modulated primary branch features and the channel-modulated auxiliary branch features are fused through element-wise integration to produce the output of the CM module:

$$Y=M_s+M_c$$

(13)

Unlike feature reconstruction structures, the CM module is designed as a lightweight adaptive modulation unit that focuses on refining the fused representations generated by the preceding CSK and CSM stages. Therefore, an additional residual shortcut from the input feature to the output is intentionally omitted to avoid reintroducing uncalibrated responses that may weaken the modulation effect and reduce the suppression capability for response imbalance under complex environments. Meanwhile, the primary branch itself preserves the dominant semantic representation, thereby providing stable feature propagation during the complementary modulation process.

Overall, the CM module achieves fine-grained recalibration of fused features without introducing additional cross-modal interactions. This design effectively enhances discriminability across both spatial and channel dimensions, thereby providing more stable fused representations for subsequent detection tasks.

4. Experiments and Results

In this section, we describe the experimental setup, including implementation details, evaluation metrics, and datasets. Comparative experiments and transfer evaluations are subsequently conducted against state-of-the-art object detection models to demonstrate the superiority and generalizability of the proposed method through both qualitative and quantitative analyses. Finally, ablation studies are performed to analyze the contributions of key components within the model and to validate their impact on fusion performance.

4.1. Experimental Configuration

All the experiments were conducted under a Linux operating system environment and accelerated using an NVIDIA RTX 4090D GPU with 24 GB of memory (NVIDIA Corporation, Santa Clara, CA, USA). The CPU operates at a base frequency of 2.6 GHz with 18 physical cores, and the system is equipped with 90 GB of RAM. The experiments were implemented using Python 3.10 and PyTorch 2.1, and the deep learning environment was built upon CUDA 12.8 to support efficient parallel computation during both training and inference.

With respect to training settings, the model was trained with a batch size of 16 and an input image size of 640 × 640, with data caching enabled (cache = True) and the number of parallel data loading workers set to 8. Automatic mixed precision (AMP) training was adopted to improve training efficiency. The optimizer employed the default automatic optimization strategy provided by the YOLOv11 framework, with an initial learning rate of 0.01, a momentum coefficient of 0.937, and a weight decay factor of 0.0005. A warm-up strategy was applied during the first 3 epochs. During data preprocessing and augmentation, visible and infrared image pairs were synchronously processed to preserve cross-modal spatial correspondence. Geometric augmentations, including resizing, flipping, and Mosaic operations, were jointly applied to both modalities using shared transformation parameters. In addition, identical normalization preprocessing was adopted for both modalities following the default YOLOv11 data pipeline, without introducing modality-specific normalization strategies.

To accommodate dataset-specific characteristics, different numbers of training epochs were adopted: 150 epochs for the M³FD dataset and 100 epochs for the LLVIP dataset. This difference was determined according to the convergence behavior observed during training. Specifically, the M³FD dataset contains more complex scene variations and object categories, requiring more training epochs to achieve stable convergence, whereas the LLVIP dataset converged earlier. Training convergence was verified based on the stabilization trend of validation performance during training.

4.2. Evaluation Protocol and Metrics

In this study, the precision (P), recall (R), mean average precision (mAP), and number of parameters (Params) are adopted as evaluation metrics to assess model performance. The mAP metric represents the mean of the average precision over all the object categories. The params metric denotes the total number of model parameters, which quantifies the computational and memory resources required by the model. The specific formulations of these metrics are defined as follows:

$$Precision={\displaystyle \frac{TP}{TP+FP}}$$

(14)

$$Recall={\displaystyle \frac{TP}{TP+FN}}$$

(15)

$$mAP={\displaystyle \frac{1}{N}}{\sum }_{i=1}^N{\left(P\right(R\left)dR\right)}_i$$

(16)

where TP, FP, and FN denote true positives, false positives, and false negatives, respectively, and N represents the number of object categories. During evaluation, the confidence threshold followed the default YOLOv11 setting of 0.25, while the IoU threshold for non-maximum suppression (NMS) was set to 0.7. In addition, GFLOPs denotes the number of floating-point operations required for a single forward inference pass and was measured under an input resolution of 640 × 640. FPS was measured on a single RTX 4090D GPU with batch size 1 and an input resolution of 640 × 640, including both forward inference and post-processing stages.

4.3. Dataset Description

Visible–infrared object detection integrates the complementary information provided by visible texture features and infrared thermal responses to achieve robust object localization and category recognition under complex environments. In such scenarios, single-modality perception often suffers from illumination degradation, background interference, partial object occlusion, and unstable feature responses. Visible images can provide rich texture and semantic details, whereas infrared images are able to maintain stable thermal saliency under low-illumination conditions. The combination of the two modalities therefore enables more reliable multimodal perception in complex environments.

LLVIP is a visible–infrared paired dataset designed for low-light vision tasks. The dataset contains 16,836 strictly aligned visible–infrared image pairs collected from 26 different low-light surveillance scenes, where all image pairs are synchronized in both time and space. The annotations mainly focus on pedestrian targets under low-illumination conditions. The visible and infrared images have resolutions of 1920 × 1080 and 1280 × 720, respectively. Due to its strictly aligned image pairs and pedestrian-centric annotations under extremely low-light environments, LLVIP is widely used for visible–infrared pedestrian detection research.

M³FD is a multi-scenario visible–infrared object detection dataset containing six object categories, including People, Car, Bus, Motorcycle, Lamp, and Truck. The dataset contains 4200 well-aligned visible–infrared image pairs collected under diverse environments, illumination conditions, seasons, and weather scenarios. The visible and infrared images have resolutions of 1024 × 768 and 640 × 512, respectively. Compared with LLVIP, M³FD covers more diverse traffic and urban scenes with greater object-category complexity. Meanwhile, due to viewpoint differences and complex real-world environments, the dataset still exhibits certain cross-modal spatial inconsistencies, which introduces additional challenges for robust visible–infrared feature fusion and localization. Therefore, M³FD is particularly suitable for evaluating the robustness and generalization capability of multimodal detection methods under complex real-world scenarios.

In this work, both datasets were divided into training, validation, and testing subsets with a ratio of 7:2:1.

4.4. Comparison of Results

To comprehensively evaluate the effectiveness of the proposed model while ensuring fair and relevant comparisons, the experiments on the LLVIP dataset include not only the single-modality baseline YOLOv11n but also dual-branch multimodal detection models, namely, YOLOv12-D and YOLOv13-D. These two models are internally implemented multimodal variants based on the original YOLOv12 and YOLOv13 architectures, respectively. Specifically, two structurally symmetric branches were employed to independently extract visible and infrared features, while same-scale high-level semantic features were fused within the neck through feature concatenation and convolutional aggregation. The original neck topology, detection heads, and training strategy were retained to ensure fair comparisons. This design enables the evaluation to focus on performance differences among similar multimodal frameworks rather than a direct comparison between single-modality and multimodal approaches.

In addition, a representative multimodal fusion method, DEYOLO, and the Transformer-based multispectral detector GM-DETR is incorporated to construct a hierarchical evaluation paradigm consisting of “single-modality baseline–YOLO-based multimodal frameworks–global interaction-based multimodal detectors.” For fair evaluation, all methods were tested under the same dataset split, input resolution, and evaluation protocol. Considering the architectural differences between DETR-based and YOLO-based detectors, the training strategy and optimization settings of GM-DETR followed its official implementation to ensure stable convergence.

Qualitative Analysis: Figure 5 presents a qualitative comparison of detection results on the LLVIP dataset, which primarily consists of pedestrian scenes captured under nighttime and low-light conditions, thereby imposing stringent requirements on the robustness of object detection models. As observed from the visualization results, the single-modality method YOLOv11n tends to suffer from missed detections, low confidence scores, and incomplete bounding boxes under low-illumination conditions. Although YOLOv12-D and YOLOv13-D introduce dual-branch multimodal architectures that improve detection performance to some extent, they still exhibit unstable responses and localization deviations in complex backgrounds and strong interference scenarios.

DEYOLO improves pedestrian visibility in low-light environments to a certain extent by leveraging the infrared modality; however, its detection results still exhibit noticeable fluctuations across different scenarios. GM-DETR further improves global contextual modeling and achieves relatively stable responses under low-light conditions. In contrast, the proposed method maintains more stable detection performance under varying low-light conditions. By further performing internal recalibration on the fused features, the model effectively suppresses background interference and highlights critical pedestrian regions, thereby achieving higher and more consistent detection confidence in complex nighttime scenes.

Quantitative Analysis:

Table 1. Comparison results on the LLVIP dataset.

Model	Precision (%)	Recall (%)	mAP50 (%)	mAP50–95 (%)	Params (M)	GFLOPs	FPS
yolov11n	88.2	79.4	86.7	48.2	2.58	6.3	526
yolov12-D	88.8	79.2	86.3	46.1	6.5	15.8	301
yolov13-D	85.7	75.3	83.0	43.3	6.44	16.1	297
DEYOLO	89.9	81.6	90.1	52.5	6.0	16.6	352
GM-DETR	91.6	84.2	91.8	54.1	11.8	28.7	168
Ours	94.0	88.5	93.9 ± 0.18	56.2 ± 0.14	6.91	14.2	378

presents the quantitative comparison results on the LLVIP dataset. The proposed method was repeatedly evaluated under three different random seeds, achieving 93.9 ± 0.18% mAP@50 and 56.2 ± 0.14% mAP@50–95, demonstrating stable visible–infrared detection performance under low-light environments. Compared with the single-modality baseline YOLOv11n, the mean Precision and Recall are improved by 5.8% and 9.1%, respectively, demonstrating the effectiveness of introducing multimodal complementary information for pedestrian detection under low-light conditions. Although the proposed method introduces higher parameter complexity and computational cost than YOLOv11n, it achieves significant improvements in Precision, Recall, mAP@50, and mAP@50–95, indicating that it can effectively improve multimodal feature discrimination under low-light environments.

The proposed method further outperforms the dual-branch multimodal baselines YOLOv12-D and YOLOv13-D in terms of Precision, Recall, mAP@50, and mAP@50–95, indicating that the performance improvement is not solely derived from multimodal inputs, but also benefits from the proposed staged cross-modal fusion strategy. Furthermore, compared with DEYOLO and GM-DETR, the proposed method not only achieves higher detection accuracy, but also further reduces GFLOPs to 14.2 and increases FPS to 378, while maintaining lower computational overhead and higher inference efficiency. These results demonstrate that the proposed framework achieves a favorable balance between detection accuracy and runtime efficiency, while maintaining robust pedestrian detection capability under complex low-light environments.

4.5. Cross-Scenario Evaluation on the M³FD Dataset

To further validate the generalizability and cross-scenario adaptability of the proposed method, cross-dataset evaluations were conducted on the M³FD dataset. Similar to the primary experiments, the single-modality baseline YOLOv11n and the dual-branch multimodal models YOLOv12-D and YOLOv13-D are adopted as comparison methods, together with the representative multimodal detection approaches DEYOLO and GM-DETR, thereby constructing a multilevel evaluation framework consistent with the main experiments. By assessing model performance under different data distributions and complex environmental conditions, the stability and effectiveness of the proposed staged cross-modal fusion strategy are systematically analyzed, further demonstrating the generalizability and practical application potential of the proposed model. All models were independently trained and evaluated on each dataset, and no cross-dataset fine-tuning or transfer learning strategy was adopted.

Qualitative Analysis: Figure 6 presents the detection results of different methods on the M³FD dataset under complex traffic scenarios, including tunnel entrances, backlighting conditions, and densely populated multi-object scenes. YOLOv11n tends to suffer from blurred object boundaries and missed detections under uneven illumination and complex backgrounds. Although YOLOv12-D and YOLOv13-D adopt dual-branch multimodal architectures that alleviate information loss caused by low-light conditions to some extent, they still exhibit unstable responses and localization deviations in scenarios involving spatial misalignment and scale variation. DEYOLO further improves multimodal fusion performance; however, it may still produce dispersed responses and category confusion in complex traffic scenes. GM-DETR enhances global contextual modeling and achieves relatively stable responses under complex environments; however, localization deviations can still be observed in densely distributed and small-target traffic scenes.

In contrast, the proposed method yields more concentrated and stable target responses across various challenging scenarios. Benefiting from the staged “refinement–interaction–modulation” fusion strategy, the proposed method effectively suppresses cross-modal interference and preserves clear target boundaries under complex illumination and background conditions. These results further demonstrate the stable cross-scenario detection capability of the proposed method.

Quantitative Analysis:

Table 2. Experimental results on the M³FD dataset.

Model	Precision (%)	Recall (%)	mAP50 (%)	mAP50–95 (%)	Params (M)	GFLOPs	FPS
yolov11n	79.8	60.5	69.2	43.6	2.58	6.3	370
yolov12-D	78.0	60.5	67.4	43.2	6.5	15.8	218
yolov13-D	77.6	60.3	68.3	43.5	6.44	16.1	203
DEYOLO	78.2	69.2	73.4	40.9	6.0	16.6	254
GM-DETR	82.4	67.5	74.3	47.8	11.8	28.7	124
Ours	85.0	68.3	75.8 ± 0.24	50.3 ± 0.19	6.91	14.2	286

presents the overall quantitative comparison results of different methods on the M³FD dataset. The proposed method was likewise repeatedly evaluated under three different random seeds, achieving 75.8 ± 0.24% mAP@50 and 50.3 ± 0.19% mAP@50–95, which further demonstrates the stability and reproducibility of the proposed framework under complex traffic scenarios. Compared with the single-modality baseline YOLOv11n, the mean mAP@50 and mAP@50–95 are improved by 6.6% and 6.7%, respectively, demonstrating the effectiveness of introducing multimodal complementary information under complex traffic scenarios. Furthermore, the proposed method also outperforms the dual-branch multimodal baselines YOLOv12-D and YOLOv13-D, further improving the mean mAP@50–95 metric by 7.1% and 6.8%, respectively, indicating that the proposed staged cross-modal fusion strategy further contributes to more robust localization performance beyond the benefit of multimodal inputs alone.

Compared with GM-DETR and DEYOLO, the proposed method maintains lower GFLOPs and higher FPS, demonstrating that the proposed staged fusion strategy achieves a favorable trade-off between detection accuracy and runtime efficiency under complex traffic scenarios. Although it does not achieve the highest Recall and introduces slightly higher parameter complexity than DEYOLO, it still achieves the best overall mAP performance, demonstrating a more favorable balance among detection accuracy, robustness, and computational efficiency under complex traffic scenarios. Furthermore, the performance improvement achieved under the stricter mAP@50–95 metric further demonstrates the effectiveness of the proposed method in maintaining stable visible–infrared detection performance under complex environments.

Since the LLVIP dataset mainly focuses on single-category pedestrian detection, class-wise evaluation is only conducted on the multi-category M³FD dataset. The detailed class-wise comparison results are presented in

Table 3. Experimental results of per-class accuracy on the M³FD dataset.

Class	yolov11n	yolov12-D	yolov13-D	DEYOLO	GM-DETR	Ours
People	61.3	59.8	59.6	77.3	76.8	75.6
Car	85.7	85.1	84.9	86.8	87.2	87.9
Bus	79.3	77.4	78.4	81.5	82.0	82.8
Motorcycle	66.5	64.3	63.1	63.6	67.8	70.0
Lamp	52.6	49.0	49.7	55.2	60.5	64.2
Truck	70.2	69.0	73.8	76.2	71.5	74.7
all	69.2	67.4	68.3	73.4	74.3	75.8

. The proposed method achieves the best overall mAP performance across all categories and obtains the highest accuracy on Car, Bus, Motorcycle, and Lamp. Among these categories, the improvement for Lamp is particularly notable, where the proposed method significantly outperforms all comparison methods. This improvement may be attributed to the fact that lamp targets are usually small, illumination-sensitive, and easily affected by complex background interference. Benefiting from the staged cross-modal fusion strategy, the proposed method can better preserve discriminative visible–infrared complementary cues while suppressing unstable background responses.

The Motorcycle category also shows clear improvements over all comparison methods, indicating that the proposed method is effective for small-scale and low-contrast targets with complex structural characteristics. In contrast, the improvement for Truck is relatively limited. Although the proposed method still outperforms YOLOv11n, YOLOv12-D, YOLOv13-D, and GM-DETR on this category, its performance remains slightly lower than that of DEYOLO. This may be because truck targets usually occupy larger spatial regions and exhibit relatively stable structural representations, reducing the relative benefit brought by fine-grained cross-modal interaction and recalibration. Similarly, the performance on the People category is slightly lower than that of DEYOLO and GM-DETR, which may be related to dense pedestrian distributions, partial occlusion, and cross-modal spatial inconsistencies in complex traffic scenes. Nevertheless, the proposed method still achieves the best overall category-level performance, demonstrating stronger robustness and more balanced generalization capability across different object categories.

4.6. Ablation Study

To verify the effectiveness of each key component and their synergistic contributions to cross-modal object detection, ablation experiments are conducted on both the LLVIP and M³FD datasets for the Cross-Modal Spatial–Channel Calibration (CSK) module, bidirectional cross-modal synergy (CSM) module, and complementary modulation module (CM). The performance is evaluated using mAP@50 and mAP@50–95 as the primary metrics.

Table 4. Ablation results on the M³FD dataset.

Configuration	M3FD mAP50 (%)	M3FD mAP50–95 (%)	LLVIP mAP50 (%)	LLVIP mAP50–95 (%)
Baseline	69.2	43.6	86.7	48.2
Baseline + CSK	73.1	48.1	90.6	52.1
Baseline + CSM	72.3	46.5	89.7	51.0
Baseline + CM	71.2	46.1	88.9	50.3
Baseline + CSK + CSM	74.0	49.2	92.2	54.5
Baseline + CSK + CM	74.7	49.3	92.4	54.8
Baseline + CSM + CM	75.0	49.4	92.8	55.0
Full model	75.8	50.3	93.9	56.2

reports the ablation results under different module combinations, where the baseline corresponds to the YOLOv11n model without additional cross-modal enhancement modules. Introducing CSK alone increases mAP@50 from 69.2% to 73.1% and mAP@50–95 from 43.6% to 48.1% on the M³FD dataset, while improving mAP@50 from 86.7% to 90.6% and mAP@50–95 from 48.2% to 52.1% on LLVIP. These results indicate that reducing spatial misalignment and channel-wise inconsistency before cross-modal interaction is beneficial for improving feature representation quality across different scenarios.

In comparison, incorporating CSM or CM individually also improves detection performance on both datasets, although the gains are relatively smaller than those achieved by CSK. When CSK and CSM are jointly enabled, mAP@50–95 further increases to 49.2% on M³FD and 54.5% on LLVIP, suggesting that complementary interaction modeling becomes more effective after preliminary cross-modal calibration.

Combining CSK with CM also leads to further performance improvement, achieving 49.3% mAP@50–95 on M³FD and 54.8% on LLVIP. In addition, the CSM + CM configuration achieves competitive performance, reaching 49.4% mAP@50–95 on M³FD and 55.0% on LLVIP, which indicates that interaction modeling and adaptive modulation can still provide effective feature enhancement without explicit calibration. Nevertheless, the full model achieves the best overall performance on both datasets, with mAP@50–95 reaching 50.3% on M³FD and 56.2% on LLVIP, demonstrating that the three modules provide complementary benefits within the staged refine–interact–modulate framework.

Figure 7 presents the visualization analysis results under different ablation configurations. As shown in Figure 7b, the baseline model exhibits relatively scattered activation distributions and significant background interference, where non-target regions such as buildings, poles, and road textures produce strong responses, indicating that the model still suffers from insufficient cross-modal feature discrimination capability. Meanwhile, within the red-boxed vehicle regions, the target activations remain relatively dispersed, and several small or distant objects fail to form stable attention regions.

After introducing the CSK module in Figure 7c, the activation regions become more concentrated around the vehicle targets, while several background responses are suppressed. In particular, the activation distributions around the central vehicle and distant small targets become more focused on the target bodies, demonstrating that CSK effectively improves cross-modal spatial consistency and alleviates feature misalignment between modalities.

With the further introduction of the CSM module, as illustrated in Figure 7d, the model exhibits stronger target-aware capability and more continuous contextual responses across multiple object regions. Compared with the CSK-only configuration, the activation distributions among the red-boxed regions show stronger semantic correlation, especially through the formation of more continuous response patterns between adjacent vehicle targets, indicating that CSM contributes to enhanced long-range cross-modal dependency modeling and complementary semantic interaction.

Finally, as shown in Figure 7e, the full model further reorganizes the activation distributions, resulting in smoother and more stable feature responses that are more concentrated around target regions. Compared with the previous configurations, the vehicle regions within the red boxes exhibit more continuous and stable target responses, while redundant activations over background structures are noticeably reduced. This demonstrates that the CM module can effectively recalibrate fused feature responses dynamically, thereby improving feature stability and discriminative capability under complex nighttime environments. Overall, the visualization results further validate the effectiveness of the proposed progressive “refinement–interaction–modulation” fusion strategy.

5. Conclusions

To address the issues of spatial misalignment, semantic inconsistency, and illumination-driven response imbalance in visible–infrared object detection under complex environments, this paper constructs a dual-branch multimodal detection framework based on YOLOv11 and proposes a staged cross-modal fusion strategy, systematically optimizing cross-modal collaborative modeling from both architectural design and fusion mechanism perspectives.

Experimental analyses across different datasets demonstrate that stage-wise decomposition in the cross-modal fusion process plays a crucial role in improving detection performance. Specifically, performing discrepancy-aware refinement prior to cross-modal interaction helps alleviate feature conflicts between modalities. Building upon this, further recalibration of the fused features can reduce uneven spatial–channel response distribution, thereby improving fusion stability and detection reliability under complex environments. In addition, the experimental results indicate that the proposed method maintains stable performance across different data distributions and scene conditions, demonstrating strong generalization capability.

Despite the effectiveness of the proposed method, several challenges still remain. Although the proposed framework demonstrates strong performance on the LLVIP and M³FD datasets, further validation under more diverse real-world scenarios involving severe weather conditions, large-scale viewpoint variations, and stronger cross-modal misalignment is still necessary. In addition, the proposed staged fusion strategy introduces additional architectural complexity due to the multi-stage cross-modal interaction design, and its performance may still be affected when one modality suffers from severe quality degradation or unstable sensing conditions. Although the proposed framework achieves a favorable balance between detection accuracy and computational efficiency, further optimization of the dual-branch interaction structure may still benefit lightweight deployment on resource-constrained devices.

In future work, we will further explore more fine-grained geometric alignment strategies, lightweight global interaction mechanisms, and model compression techniques to improve robustness, efficiency, and practical deployment capability under more challenging real-world environments.