YOLO-CMFM: A Visible-SAR Multimodal Object Detection Method Based on Edge-Guided and Gated Cross-Attention Fusion

Highlights

What are the main findings?

• Proposes YOLO-CMFM for multimodal remote sensing object detection.
• Designs LEGA, BCA, and CGG for edge-guided and gated cross-attention fusion.

What are the implications of the main finding?

• YOLO-CMFM sets a new SOTA with robust, transferable multimodal detection.
• Robust performance across diverse remote sensing imaging conditions.

Abstract

To address the challenges of cross-modal feature misalignment and ineffective information fusion caused by the inherent differences in imaging mechanisms, noise statistics, and semantic representations between visible and synthetic aperture radar (SAR) imagery, this paper proposes a multimodal remote sensing object detection method, namely YOLO-CMFM. Built upon the Ultralytics YOLOv11 framework, the proposed approach designs a Cross-Modal Fusion Module (CMFM) that systematically enhances detection accuracy and robustness from the perspectives of modality alignment, feature interaction, and adaptive fusion. Specifically, (1) a Learnable Edge-Guided Attention (LEGA) module is constructed, which leverages a learnable Gaussian saliency prior to achieve edge-oriented cross-modal alignment, effectively mitigating edge-structure mismatches across modalities; (2) a Bidirectional Cross-Attention (BCA) module is developed to enable deep semantic interaction and global contextual aggregation; (3) a Context-Guided Gating (CGG) module is designed to dynamically generate complementary weights based on multimodal source features and global contextual information, thereby achieving adaptive fusion across modalities. Extensive experiments conducted on the OGSOD 1.0 dataset demonstrate that the proposed YOLO-CMFM achieves an $mAP@50$ of 96.2% and an $mAP@50$:95 of 75.1%. While maintaining competitive performance comparable to mainstream approaches at lower IoU thresholds, the proposed method significantly outperforms existing counterparts at high IoU thresholds, highlighting its superior capability in precise object localization. Also, the experimental results on the OSPRC dataset demonstrate that the proposed method can consistently achieve stable gains under different kinds of imaging conditions, including diverse SAR polarizations, spatial resolutions, and cloud occlusion conditions. Moreover, the CMFM can be flexibly integrated into different detection frameworks, which further validates its strong generalization and transferability in multimodal remote sensing object detection tasks.

1. Introduction

With the rapid advancement of multi-source remote sensing platforms such as satellites and unmanned aerial vehicles (UAVs), deep learning-based remote sensing object detection has become a key research area in intelligent remote sensing interpretation. It has been extensively applied in fields including maritime surveillance, urban planning, and disaster early-warning systems [1,2]. The traditional visible remote sensing imagery provides abundant spatial and texture details, enabling clear visual representation of the geometric and spectral characteristics of ground objects. However, visible imaging is strongly constrained by external illumination conditions. Under low-light or cloud-covered environments, target features are vulnerable to interference, resulting in decreased detection accuracy and unstable detection performance [3].

In contrast, the synthetic aperture radar (SAR), which operates via active microwave imaging, is capable of penetrating clouds and fog, thus providing stable observation under all-weather, day-and-night conditions. These characteristics make SAR particularly advantageous in complex environments. However, owing to the coherent nature of SAR imaging, it is inherently affected by speckle noise, limited texture representation, and less intuitive semantic information, all of which hinder accurate visual interpretation [4]. Current remote sensing object detection methods predominantly rely on single-modality data, limiting their ability to exploit the complementary relationships among heterogeneous modalities. As a result, these methods struggle to sufficiently characterize target features and generalize well across diverse imaging conditions [5]. Therefore, a critical challenge in multimodal remote sensing object detection is how to effectively fuse the complementary information between visible and SAR modalities to enhance detection accuracy and robustness.

With the rapid advancement of deep learning, multimodal remote sensing object detection has gained increasing attention, giving rise to a variety of fusion strategies. Fundamentally, multimodal fusion is a multi-source data integration process that combines information from different sensors, spectral bands, or imaging mechanisms (such as visible, infrared, SAR, hyperspectral, and LiDAR) to achieve complementary feature representation and robust ground-object recognition [6]. From the perspective of fusion hierarchy, existing methods can generally be classified into data-level fusion, feature-level fusion, and decision-level fusion [7]. Data-level fusion integrates multiple source images at the pixel or band level during the input stage through techniques such as channel stacking, weighted averaging, or principal component transformation. Although this approach maximizes the use of raw information, it is highly sensitive to spatial misalignment and noise interference, which may introduce artifacts or modality inconsistency. Feature-level fusion jointly models multimodal features within the intermediate layers of deep neural networks. By facilitating cross-modal feature interaction, it strengthens semantic representation and enhances complementary information. This has become the mainstream strategy in the current research field of multimodal remote sensing object detection. Decision-level fusion combines the independent outputs of different modalities at the final prediction stage using strategies such as voting, confidence weighting, or multi-branch ensemble learning. While this approach offers flexibility, information exchange occurs only after decision-making, limiting its ability to exploit fine-grained complementary features across modalities. The schematic comparison of these three fusion paradigms is presented in Figure 1.

Figure 1. Illustration of multi-modal fusion strategies.

Existing research on multimodal remote sensing object detection primarily centers on feature-level fusion, where deep neural architectures are employed to enable high-level semantic collaboration across modalities. In recent years, feature-level fusion has advanced substantially, giving rise to various improvement strategies. Early methods typically adopted feature concatenation or weighted fusion, directly stacking or linearly combining features from different modalities. Although such approaches are conceptually simple and can improve detection performance to some extent, the absence of modality alignment and deep semantic interaction mechanisms often amplifies redundant information and propagates noise across modalities. For example, Sharma et al. [8] applied multimodal fusion through feature concatenation and cross-product operations, yet their method still exhibited limited performance in complex scenarios.

To mitigate the degradation of low-quality modalities, several studies have incorporated super-resolution reconstruction into detection frameworks. For example, Zhang et al. [9] introduced auxiliary super-resolution learning to enhance the detailed representation of small objects, which improved detection accuracy but incurred additional training overhead and created a strong dependence on reconstruction quality, ultimately resulting in unstable performance in visible-SAR multimodal scenarios.

With the advent of Transformer architectures [10] and cross-modal attention mechanisms, researchers began to pay attention to global dependency modeling across modalities. Fang et al. [11] were among the first to leverage the long-range modeling capability of Transformers to capture cross-modal contextual associations, thereby enhancing the comprehensiveness of multimodal feature interaction. Shen et al. [12] further improved discriminative representation through query-guided cross-attention. However, the absence of adaptive modulation still resulted in fixed inter-modal weighting, limiting the flexibility of fusion.

Building upon these foundations, several studies have emphasized structural optimization and semantic enhancement at the feature-fusion level. Guo et al. [13] incorporated a cosine-similarity-guided feature decomposition and fusion module into YOLOv8, achieving lightweight cross-modal alignment through common-specific feature decomposition. Fei et al. [14] proposed an attention-guided differential fusion mechanism based on YOLOv5 to strengthen complementary feature modeling and salient-region guidance. Wang et al. [15] designed a dual-branch asymmetric attention backbone and feature fusion pyramid, enabling semantic-detail synergy from backbone to fusion layers.

It is noteworthy that most existing multimodal detection research has focused on visible-infrared settings, primarily because these modalities share relatively similar imaging mechanisms, which facilitates feature alignment. Moreover, several large-scale multimodal datasets are publicly available, such as M3FD [16], VEDAI [17], and DroneVehicle [18]. In contrast, visible-SAR multimodal detection faces considerably greater challenges. On one hand, visible and SAR modalities exhibit substantial differences in imaging mechanisms, statistical distributions, and semantic representations, where visible imagery emphasizes spectral and texture information, whereas SAR primarily captures geometric structure. These disparities lead to distinct feature-space distributions, making direct alignment and fusion difficult. On the other hand, high-quality and well-registered visible-SAR datasets are difficult to obtain, and publicly available resources remain limited, thereby constraining large-scale empirical research.

Therefore, achieving efficient and adaptive visible-SAR multimodal feature fusion that fully leverages cross-modal complementarity and structural consistency remains a key challenge in multimodal remote sensing object detection. In summary, current approaches still exhibit three major limitations when applied to visible-SAR multimodal detection tasks as follows: (1) The lack of explicit edge and structural modeling weakens spatial detail representation during fusion and degrades boundary discrimination of detected objects. (2) Cross-modal attention mechanisms suffer from insufficient semantic alignment, which limits their ability to distinguish subtle target features. (3) Static fusion strategies without dynamic regulation cannot adjust modality contributions according to scene variations and target characteristics, thereby reducing fusion robustness.

To address the aforementioned challenges, this paper proposes a visible-SAR multimodal remote sensing object detection framework based on the edge-guided and gated cross-attention method, namely YOLO-CMFM. The proposed YOLO-CMFM fully utilizes the complementary characteristics of visible and SAR modalities to enable efficient, adaptive, and robust cross-modal fusion. The main contributions of this study are summarized below.

• A visible-SAR multimodal object detection method, namely YOLO-CMFM, is proposed. This method, built upon the Ultralytics YOLOv11 framework, incorporates a novel Cross-Modal Fusion Module (CMFM), which establishes a progressive fusion mechanism characterized by edge guidance, contextual interaction and gated regulation. While preserving the integrity of feature representation, the proposed method enables hierarchical semantic collaboration and dynamic information balancing from shallow to deep layers, thereby enhancing the effectiveness and stability of multimodal feature fusion. The proposed CMFM consists of the following three key components, i.e., the Learnable Edge-Guided Attention (LEGA) module, the Bidirectional Cross-modal Attention (BCA) module, and the Context-Guided Gate (CGG) module:
  • The LEGA module integrated a learnable Gaussian saliency prior to performing joint channel-spatial modulation and edge-constrained modeling of multimodal features, which can effectively suppress SAR speckle noise and mitigate cross-modal edge-structure misalignment.
  • The BCA module utilizes a bidirectional cross-attention mechanism to achieve deep semantic interaction and global contextual aggregation across modalities, and meanwhile to reduce computational complexity through channel reduction and reconstruction.
  • The CGG module incorporates multimodal source features and global contextual information to dynamically generate complementary gating weights across modalities, which can effectively suppress redundant information and promote adaptive feature fusion.
• Extensive experiments were conducted on the OGSOD 1.0 and OSPRC datasets, which cover diverse spatial resolutions, SAR polarization modes, and cloud occlusion conditions. The results demonstrate that compared with classical single-modal and multimodal detection models, the proposed YOLO-CMFM achieves superior overall performance, attaining a competitive $mAP@50$ of 96.2% and a substantially improved $mAP@50$:95 of 75.1%.
  Particularly, the model can gain more notable increases at higher IoU thresholds, showing its strengths in precise object localization. Moreover, YOLO-CMFM can still maintain stable performance under diverse imaging conditions, and the proposed CMFM can also be seamlessly integrated into various detection frameworks, which further verifies its robustness, generality and transferability in multimodal object detection.

2. Materials and Methods

2.1. Overall Framework

The overall architecture of the proposed YOLO-CMFM is presented in Figure 2. The model is built upon the single-stage object detection framework of Ultralytics YOLOv11 [19], selected for its lightweight and high-efficiency end-to-end design, which provides strong multi-scale feature extraction capability and has been extensively validated across diverse detection tasks [20,21,22].

Figure 2. Overall architecture of the proposed YOLO-CMFM method. It employs a dual-branch backbone to independently extract multi-scale features from visible and SAR images. A Cross-Modal Fusion Module is integrated into the P3, P4, and P5 layers to enable adaptive semantic interaction and cross-modal feature fusion. The fused representations are subsequently aggregated and upsampled in the detection head to produce multi-scale predictions for small, medium, and large objects.

The proposed YOLO-CMFM fully leverages the complementary properties of visible and SAR modalities. Through cross-modal attention and gated interaction mechanisms, it achieves deep semantic collaboration and adaptive feature fusion across layers, thereby enhancing detection accuracy and robustness under complex imaging conditions.

The overall architecture consists of three major components: (1) a dual-branch feature extraction network (Backbone), (2) a Cross-Modal Fusion Module (CMFM), and (3) a multi-scale feature aggregation and decoupled detection head (Head).

The overall workflow is as follows: First, the dual-branch backbone independently extracts multi-scale features from the visible and SAR images. Then, the CMFM is embedded into the aligned feature layers (P3, P4, P5) to enable deep semantic interaction and adaptive cross-modal fusion. Finally, the fused features are aggregated and upsampled along the detection path to integrate multi-scale semantic information and are subsequently passed to the decoupled detection head for object classification and bounding-box regression.

To comply with the predefined input specifications of YOLOv11, the original images are uniformly resized to $640\times 640$ pixels using aspect-ratio-preserving bilinear interpolation, thereby ensuring sufficient extraction of multi-scale features. Let the input images be the visible-light modality $I_{\mathrm{vis}}$ and the SAR modality $I_{\mathrm{sar}}$, with their corresponding multi-scale feature representations denoted as $\left\{F_{\mathrm{vis}}^i\right\}$ and $\left\{F_{\mathrm{sar}}^i\right\}$, where $i\in \{3,4,5\}$ indicates different feature levels. The functions of each network component are described as follows:

(1)

Backbone

To preserve the independence and complementarity of feature representations between the visible and SAR modalities, a parameter-unshared dual-branch feature extraction structure is employed. Each branch is composed of convolutional blocks and C3K2 modules, generating multi-scale features through progressive downsampling. The visible branch focuses on capturing rich texture and spectral details, whereas the SAR branch emphasizes geometric structure and edge modeling, forming complementary semantic-level representations. This establishes a solid foundation for subsequent cross-modal fusion.

(2)

CMFM

At each aligned feature level, a CMFM is embedded to facilitate deep semantic interaction and adaptive fusion between the two modalities. The CMFM consists of three submodules, which together constitute a progressive fusion strategy characterized by sequential edge guidance, context interaction, and gated regulation. These submodules operate in a mutually complementary manner rather than functioning as a simple cascaded pipeline, which substantially enhances the sufficiency and adaptivity of cross-modal feature exchange and semantic integration.

(3)

Head

The fused multi-scale features are aggregated and upsampled to further integrate semantic and spatial information before being fed into the decoupled detection head of YOLOv11. The detection head comprises separate classification and regression branches, which operate at small, medium, and large scales to perform object category recognition and bounding-box regression, respectively.

2.2. CMFM

The structure of the proposed CMFM is presented in Figure 3. This module preserves the integrity of feature representation while achieving structural alignment and adaptive fusion between modalities. Given a pair of same-scale features $(F_{\mathrm{vis}},F_{\mathrm{sar}})\in {\mathbb{R}}^{B\times C\times H\times W}$, the fused output feature is denoted as $F_{\mathrm{fuse}}\in {\mathbb{R}}^{B\times C\times H\times W}$, where B, C, H, and W denote the batch size, channel number, height, and width of the feature maps, respectively. The CMFM adheres to a design philosophy of edge guidance, context interaction, and gated regulation, and is composed of three submodules.

Figure 3. Detailed structure of the proposed CMFM. It incorporates three coordinated submodules. (1) The LEGA module (green) achieves edge-oriented cross-modal alignment with Gaussian-guided saliency prior and CBAM attention. (2) The BCA module (red) performs deep semantic interaction through bidirectional cross-modal attention and global contextual aggregation, and integrates sequence downsampling, which reduces the standard $\mathcal{O}\left(N^2\right)$ attention complexity to $\mathcal{O}({\displaystyle \frac{N^2}{R}})$. (3) The CCG module (blue) adaptively balances visible and SAR contributions through complementary gating.

2.2.1. LEGA

In visible-SAR scenarios, the accuracy of target recognition and the continuity of object boundaries are highly dependent on the quality of edge structures. However, speckle noise in SAR images often obscures salient edge information, thereby weakening boundary distinctiveness [23]. To alleviate the structural discrepancies and noise-induced degradation between visible and SAR modalities, the LEGA module is designed to enhance edge features, as illustrated in the green section of Figure 3.

The core idea is to enhance edge-region responses through a learnable Gaussian saliency prior, which adaptively strengthens local edge structures while preserving background smoothness. The Gaussian prior provides a differentiable saliency-guided mechanism that aligns with the natural spatial diffusion characteristics of edges [24,25]. In addition, the LEGA module maintains a stable residual flow, effectively suppressing gradient oscillations during the early training stage. The computational process is organized into the following two components.

(1)

Structural Saliency Mask Generation

Let the input feature be denoted as $F_{\mathrm{in}}\in {\mathbb{R}}^{B\times C\times H\times W}$. The LEGA module generates a refined saliency mask M through a composite Gaussian-guided subnetwork $G_{\theta }$, which operates in three stages.

First, a fixed two-dimensional Gaussian kernel $K\in {\mathbb{R}}^{k\times k}$ is constructed to serve as a consistent geometric prior. Its weights are initialized according to the standard Gaussian distribution:

$$G(x,y)={\displaystyle \frac{1}{2\pi {\sigma }^2}}exp\left(-{\displaystyle \frac{x^2+y^2}{2{\sigma }^2}}\right)$$

(1)

where $(x,y)$ denotes spatial coordinates and $\sigma$ controls the smoothing scale. In this study, we set $k=5$ and $\sigma =1.0$, and the kernel is normalized such that $\sum K=1$. Crucially, this layer is set as non-trainable to introduce a deterministic structural bias.

Second, the input feature $F_{\mathrm{in}}$ is processed by a depthwise convolutional layer utilizing the kernel K. This ensures that spatial filtering is applied independently to each channel, effectively suppressing speckle noise while preserving key semantic information. The response is stabilized via Batch Normalization and a ReLU activation function to yield the preliminary saliency feature $F_{gaussian}$,

$$F_{gaussian}=\mathrm{ReLU}\left(\mathrm{BN}(F_{\mathrm{in}}\ast K)\right)$$

(2)

where ∗ denotes the depthwise convolution operation.

Third, to correct the boundary coarseness inherent in Gaussian smoothing, $F_{gaussian}$ is fused with the original context $F_{\mathrm{in}}$ and fed into a learnable refinement block ${\mathcal{F}}_{\mathrm{refine}}$. This block employs a bottleneck structure (cascaded $1\times 1$, $3\times 3$, and $1\times 1$ convolutions) to align the Gaussian response with precise object contours, producing the final mask M,

$$M=G_{\theta }\left(F_{\mathrm{in}}\right)={\mathcal{F}}_{\mathrm{refine}}(F_{\mathrm{in}}+F_{gaussian})$$

(3)

(2)

Edge-Guided Modulation and Adaptive Enhancement

Subsequently, the feature M is modulated through an affine transformation to obtain the edge-enhanced feature $F^{\prime }$,

$$F^{\prime }=\alpha (F_{\mathrm{in}}\odot M)+\beta F_{\mathrm{in}}$$

(4)

where ⊙ represents element-wise multiplication, and $\alpha$, $\beta$ are learnable scalar coefficients that control the balance between the enhanced and residual components. Subsequently, a convolution-normalization-activation operation is applied to extract locally enhanced structural information,

$$F^{″}=\mathrm{ReLU}\left(\mathrm{Conv}\left(F^{\prime }\right)\right)$$

(5)

Given that different channels contribute unequally to semantic representation, a CBAM (Convolutional Block Attention Module) [26] is introduced to generate a joint channel-spatial attention map, thereby enhancing feature discriminability and selectivity,

$$A=\sigma \left(\mathrm{CBAM}\left(F^{″}\right)\right)$$

(6)

where $\sigma$ denotes the sigmoid function. Finally, the enhanced feature A is fused back into the main branch through a controlled residual connection, producing the output feature $F_{\mathrm{out}}$,

$$F_{\mathrm{out}}=F_{\mathrm{in}}+\gamma (A\odot F^{″})$$

(7)

where $\gamma$ is a learnable scaling factor, initialized to zero and gradually increased during training to enable smooth and stable transition from identity mapping to edge-guided modulation.

2.2.2. BCA

To achieve deep semantic interaction and adaptive feature alignment between the visible and SAR modalities, this study designs the BCA module. As illustrated in the red section of Figure 3, the module consists of three main components.

(1)

Channel Decoupling and Feature-wise Linear Modulation (FiLM) [27]

Given the input features of the visible and SAR modalities, $F_{\mathrm{vis}},F_{\mathrm{sar}}\in {\mathbb{R}}^{B\times C\times H\times W}$, channel-wise separation is first performed to obtain local semantic features and cross-modal query features,

$$[y_i,z_i]=\mathrm{ReLU}\left(\mathrm{Linear}\left(F_i\right)\right)i\in \{\mathrm{vis},\mathrm{sar}\}$$

(8)

where $y_i\in {\mathbb{R}}^{B\times C_r\times H\times W}$, $C_r=\frac{C}{2}$ represents the local semantic feature that preserves modality-specific semantic information for subsequent fusion; $z_i\in {\mathbb{R}}^{B\times C_r\times H\times W}$ represents the cross-modal query feature for interaction modeling.

In the reduced feature space, edge priors $F_{\mathrm{lega}}^i$ generated by the LEGA module are employed to modulate query features through a FiLM transformation,

$$z_i^{\prime }=z_i\odot (1+{\lambda }_i{F}_{\mathrm{lega}}^i)i\in \{\mathrm{vis},\mathrm{sar}\}$$

(9)

where ${\lambda }_i$ is a learnable modulation coefficient that controls the modulation strength. This design leverages edge-enhanced saliency information to guide cross-modal matching, ensuring that the fusion process focuses more on structurally salient regions.

(2)

Bidirectional Cross-Modal Attention

To enable global semantic interaction between visible and SAR modalities, a bidirectional cross-attention structure is constructed based on the classical self-attention mechanism [10]. The computation is defined as:

$$\mathrm{Attention}(Q,K,V)=\mathrm{Softmax}\left({\displaystyle \frac{Q{K}^T}{\sqrt{d_{\mathrm{head}}}}}\right)V$$

(10)

where $Q,K,V\in {\mathbb{R}}^{B\times C_r\times N}$ denote the query, key, and value vectors, respectively, all generated from the input sequence $z_i^{\prime }$ through linear projection. $N=H\times W$ represents the sequence length, and $d_{\mathrm{head}}$ denotes the feature dimensionality of each attention head.

The computational complexity of standard attention is $\mathcal{O}\left(N^2\right)$, which increases quadratically with the sequence length N. However, for remote sensing images with large spatial resolutions, such complexity becomes prohibitively expensive. To address this issue, inspired by the sequence downsampling strategy proposed by Zhao et al. [28], this study performs sparsified modeling along the sequence-length dimension to obtain reduced query, key, and value representations, which enables efficient modeling of global dependencies while achieving substantial feature compression:

$$\begin{array}{cc}\hfill Q_i& =\mathrm{Linear}\left(z_i^{\prime }\right)\hfill \\ \hfill [K_i,V_i]& =\mathrm{Linear}\left(\mathrm{Conv}\left(z_i^{\prime }\right)\right)\hfill \end{array}$$

(11)

where $Q_i\in {\mathbb{R}}^{B\times C_r\times N}$, and $K_i,V_i\in {\mathbb{R}}^{B\times C_r\times {\displaystyle \frac{N}{R}}}$, with R denoting the downsampling factor. In this study, we adopt the efficient design from [28] by setting $R=4$. This configuration effectively reduces the computational complexity from $\mathcal{O}\left(N^2\right)$ to $\mathcal{O}({\displaystyle \frac{N^2}{R}})$.

Next, a bidirectional cross-modal interaction mechanism is constructed. The visible modality first serves as the query $Q_{\mathrm{vis}}^i$, while the SAR modality provides the key $K_{\mathrm{sar}}^i$ and value $V_{\mathrm{sar}}^i$ for one round of cross-attention. The process is then reversed, with the SAR modality querying the visible modality. To further enhance the interaction capability between modalities and better capture complex cross-modal patterns, a multi-head attention mechanism is introduced. Specifically, for each attention head $(1\le i\le \mathrm{heads})$, an independent attention map is computed, and the outputs from all heads are subsequently aggregated. The computation is defined as

$$\left\{\begin{array}{c}\hfill z_{\mathrm{vis}}^i=\mathrm{Softmax}\left({\displaystyle \frac{Q_{\mathrm{vis}}^i{\left(K_{\mathrm{sar}}^i\right)}^T}{\sqrt{d_{\mathrm{head}}}}}\right)V_{\mathrm{sar}}^i\\ \hfill z_{\mathrm{sar}}^i=\mathrm{Softmax}\left({\displaystyle \frac{Q_{\mathrm{sar}}^i{\left(K_{\mathrm{vis}}^i\right)}^T}{\sqrt{d_{\mathrm{head}}}}}\right)V_{\mathrm{vis}}^i\end{array}\right.i=1,2,\dots ,\mathrm{heads}.$$

(12)

$$\left\{\begin{array}{cc}\hfill z_{\mathrm{vis}}^{″}& =\mathrm{Concat}\left(z_{\mathrm{vis}}^i\right)\hfill \\ \hfill z_{\mathrm{sar}}^{″}& =\mathrm{Concat}\left(z_{\mathrm{sar}}^i\right)\hfill \end{array}\right.i=1,2,\dots ,\mathrm{heads}.$$

(13)

The attended features $z_{\mathrm{vis}}^{″},z_{\mathrm{sar}}^{″}$ are then concatenated with their respective local semantic features and linearly projected back to the original channel dimension to obtain the cross representations $F_{\mathrm{vis}}^{\mathrm{cross}},F_{\mathrm{sar}}^{\mathrm{cross}}$,

$$\left\{\begin{array}{c}\hfill F_{\mathrm{vis}}^{\mathrm{cross}}=\mathrm{Linear}\left(\mathrm{Concat}(z_{\mathrm{vis}}^{″},y_i)\right)+F_{\mathrm{vis}}\\ \hfill F_{\mathrm{sar}}^{\mathrm{cross}}=\mathrm{Linear}\left(\mathrm{Concat}(z_{\mathrm{sar}}^{″},y_i)\right)+F_{\mathrm{sar}}\end{array}\right.$$

(14)

This bidirectional interaction structure preserves modality symmetry while ensuring comprehensive global contextual alignment, significantly improving scene consistency and boundary discriminability of the target features.

(3)

Global Context Construction

To further integrate the global semantic relationships between visible and SAR modalities, a global contextual construction module is introduced to aggregate high-level semantic dependencies. The cross features from both branches are concatenated along the channel dimension and globally pooled using GAP (Global Average Pooling) to obtain a unified semantic descriptor. A lightweight MLP is then applied to generate a compact cross-modal global context embedding, where the MLP is implemented with two convolutional layers.

$$\mathrm{C}=\mathrm{MLP}\left(\mathrm{GAP}\left([F_{\mathrm{vis}}^{\mathrm{cross}},F_{\mathrm{sar}}^{\mathrm{cross}}]\right)\right)$$

(15)

where $C\in {\mathbb{R}}^{B\times C_{\mathrm{ctx}}\times 1\times 1}$ denotes the fused global contextual vector, and $C_{ctx}$ is the dimension of the global semantic embedding. It serves as a global semantic prior for the CGG module, guiding adaptive modality weighting and enforcing semantic consistency during cross-modal fusion.

2.2.3. CGG

During the fusion stage, the contribution of different modality features to target recognition is highly dynamic. To address this issue, the CGG module is proposed to achieve adaptively adjusted modality contributions through joint spatial-channel gating. Inspired by dynamic selection mechanisms [29] and gated fusion strategies [30], this design enables context-conditioned adaptive gating, selectively enhancing complementary and discriminative modality features while effectively suppressing redundant or conflicting information. As illustrated in the blue section of Figure 3, the CGG module takes as input the dual-branch outputs of the visible and SAR modalities together with the global context generated by the BCA module, which can be divided into two parts.

(1)

Context-Conditioned Complementary Gating

Let $F_{vis}$ and $F_{sar}$ denote the visible and SAR feature, respectively, and C represent the global contextual embedding obtained from the previous stage. The contextual vector C is first broadcast across the spatial dimensions to generate an expanded context tensor $C^{\prime }\in {\mathbb{R}}^{B\times C_{ctx}\times H\times W}$. $F_{vis}$, $F_{sar}$, and $C^{\prime }$ are then concatenated along the channel dimension to form the joint representation $Z\in {\mathbb{R}}^{B\times (2C+C_{ctx})\times H\times W}$,

$$Z=[F_{\mathrm{vis}},F_{\mathrm{sar}},C^{\prime }]$$

(16)

A pointwise convolution followed by a Sigmoid activation is applied to generate a pixel-wise gating map $g\in {[0,1]}^{B\times C\times H\times W}$,

$$g=\sigma \left(\mathrm{Conv}\right(Z\left)\right)$$

(17)

where $\sigma$ denotes the sigmoid function. This operation adaptively assigns gating weights across both spatial and channel dimensions, enabling refined feature reweighting between modalities. To prevent instability caused by simultaneous amplifying or suppressing both modalities within the same spatial region, a complementary gating strategy is adopted,

$$\left\{\begin{array}{cc}\hfill G_{\mathrm{vis}}& =g\hfill \\ \hfill G_{\mathrm{sar}}& =1-g\hfill \end{array}\right.$$

(18)

This complementary mechanism, similar to soft attention [29], ensures that at each spatial location only one modality predominates, effectively reducing redundancy and conflict between modalities. The gated features ${\widehat{F}}_{\mathrm{vis}},{\widehat{F}}_{\mathrm{sar}}$ are computed as

$$\left\{\begin{array}{cc}\hfill {\widehat{F}}_{\mathrm{vis}}& =G_{\mathrm{vis}}\odot F_{\mathrm{vis}}\hfill \\ \hfill {\widehat{F}}_{\mathrm{sar}}& =G_{\mathrm{sar}}\odot F_{\mathrm{sar}}\hfill \end{array}\right.$$

(19)

where ⊙ denotes element-wise multiplication.

(2)

Residual Scaling and Lightweight Refinement

To alleviate feature distribution mismatch and stabilize training, learnable residual scaling factors ${\delta }_1,{\delta }_2$ (initialized to 0.5) are introduced after gating fusion, forming a residual smoothing path that preserves gradient flow.

$$F_g={\widehat{F}}_{vis}+{\widehat{F}}_{sar}+{\delta }_1{F}_{vis}+{\delta }_2{F}_{sar}$$

(20)

When $G\approx 0.5$ and ${\delta }_1={\delta }_2=0.5$, the operation reduces to additive fusion, $F_g\approx F_{vis}+F_{sar}$, ensuring stable initialization and allowing the model to progressively learn modality-specific weighting differences during training.

To further align modality distributions and enhance feature discriminability, a lightweight refinement head module is applied using convolution and nonlinear activation for channel-domain normalization and mapping.

$$F_{\mathrm{fuse}}=\mathrm{ReLU}\left(\mathrm{Conv}\left(F_{\mathrm{g}}\right)\right)$$

(21)

This operation compresses redundant channels and strengthens semantic responses, producing the final fused feature map $F_{\mathrm{fuse}}$, which is subsequently passed to the detection head for multi-scale aggregation, classification, and bounding-box regression.

Overall, the CGG module performs pixel-wise adaptive modality selection in the spatial domain and weighted residual refinement in the channel domain. It effectively suppresses redundancy and modality bias while enhancing fusion stability and robustness, thereby significantly improving the consistency and complementarity of cross-modal feature integration.

2.3. Loss Function

To enable end-to-end optimization of multimodal feature fusion, the proposed CMFM is seamlessly integrated into the Ultralytics YOLOv11 architecture while preserving the original detection head design and loss functions, thereby facilitating joint training from multimodal feature extraction to final detection prediction. The detection head contains two independent branches for object classification and bounding-box regression, and the total loss is formulated as

$${\mathcal{L}}_{\mathrm{total}}={\lambda }_1{\mathcal{L}}_F+{\lambda }_2{\mathcal{L}}_{\mathrm{CIoU}}+{\lambda }_3{\mathcal{L}}_{\mathrm{DFL}}$$

(22)

where ${\mathcal{L}}_F$ denotes the Focal Loss for addressing class imbalance; ${\mathcal{L}}_{\mathrm{CIoU}}$ represents the Complete IoU Loss for bounding-box regression; ${\mathcal{L}}_{\mathrm{DFL}}$ is the Distribution Focal Loss for refine boundary localization. The weighting coefficients ${\lambda }_1$, ${\lambda }_2$, ${\lambda }_3$ are set to 7.5, 0.5, and 1.5, respectively, consistent with the original YOLOv11 configuration.

3. Results

3.1. Datasets

To comprehensively evaluate the detection performance and generalization ability of the proposed YOLO-CMFM, two representative visible-SAR multimodal remote sensing object detection datasets are selected, i.e., OGSOD 1.0 and OSPRC.

(1)

OGSOD 1.0 Dataset

The OGSOD 1.0 dataset [31] is a newly released large-scale benchmark for visible-SAR multimodal remote sensing object detection. It contains 14,665 pairs of training images and 3666 pairs of testing images, with more than 48,000 annotated instances in total. Each sample consists of a co-registered pair of visible and SAR images. The annotated objects cover three representative categories: Bridge, Harbor, and Oil Tank.

The SAR images are collected from the GF-3 satellite (3 m resolution), while the visible images are sourced from Google Earth (10 m resolution). To ensure spatial consistency, the images from both modalities are mapped to the same spatial resolution using ENVI software (version 6.0) and precisely aligned at the pixel level, and the co-registered image pairs are cropped into slices with dimensions of $256\times 256$ pixels. This dataset provides high-quality co-registered imagery for investigating cross-modal object detection and has become an important benchmark in the visible-SAR domain. Figure 4a presents typical examples of visible-SAR image pairs in the OGSOD 1.0 dataset, and Figure 4b summarizes the statistical characteristics of target categories and scale distributions.

Figure 4. (a) Visualization of visible and SAR image pairs in OGSOD 1.0, where bridges, harbors, and oil tanks are marked with green, blue, and red boxes, respectively. (b) Statistical characteristics of the OGSOD 1.0 dataset, where the bar chart depicts the number of annotated instances across three scales (small, medium, large), and the donut chart summarizes the overall category proportions, revealing that the dataset is predominantly composed of small- and medium-sized targets.

(2)

OSPRC Dataset

The OSPRC dataset [31] is a smaller-scale paired visible-SAR dataset designed for systematic evaluation of the effects of SAR polarization modes (VV/VH), visible resolutions (10/20 m), and cloud occlusion conditions on detection performance. It consists of five subsets, each containing 3300 paired images. All images are spatially aligned using ENVI software, and the co-registered image pairs are cropped into $256\times 256$ pixel patches. The training and testing sets are split with an 8:2 ratio. The annotated object categories are consistent with those of OGSOD, namely Bridge, Harbor, and Oil Tank, enabling a robust assessment of model generalization under cross-scene conditions. Representative examples under different imaging scenarios from OSPRC are shown in Figure 5.

Figure 5. Representative examples in the OSPRC dataset. Each row illustrates a different remote-sensing imaging condition, including visible images at 10 m resolution, visible images at 20 m resolution, visible images affected by cloud occlusion, SAR imagery acquired in VH polarization, and SAR imagery acquired in VV polarization.

In summary, the OGSOD 1.0 and OSPRC datasets jointly cover a diverse range of imaging modes and environmental conditions, providing a solid data foundation for evaluating the stability, adaptability, and generalization capability of the proposed YOLO-CMFM across various visible-SAR imaging scenarios.

3.2. Experimental Settings

All experiments in this study were conducted on a workstation equipped with an NVIDIA GeForce RTX 4090 GPU (24 GB) and an Intel Xeon Gold 6530 CPU. The software environment consists of Python 3.9.18, CUDA 11.8, and PyTorch 2.1. Unless otherwise specified, the input image size was set to 640 × 640, with visible images using three channels and SAR images using a single channel. The batch size was fixed at 16, and all models were trained for 400 epochs in an end-to-end manner. Stochastic Gradient Descent (SGD) was adopted as the optimizer, with an initial learning rate of $1\times {10}^{-2}$, a momentum of 0.937, and a weight decay of $5\times {10}^{-4}$.

Data augmentation strategies followed those in the default Ultralytics YOLOv11 implementation, including Mosaic and Mixup. To ensure fair comparisons, random seeds were fixed across all experiments, and identical hyperparameter configurations were applied during both training and inference.

3.3. Evaluation Metrics

To quantitatively evaluate the detection performance of different methods, this study adopts mean Average Precision ($mAP$) as the primary evaluation metric and conducts comprehensive evaluations under varying Intersection over Union (IoU) thresholds.

A predicted bounding box is regarded as a True Positive (TP) when its IoU with the ground truth (GT) box exceeds a given threshold t; otherwise, it is classified as a False Positive (FP). Any ground-truth object that is not detected is counted as a False Negative (FN). The IoU is defined as follows:

$$IoU(B_p,B_g)={\displaystyle \frac{|B_p\cap B_g|}{|B_p\cup B_g|}}$$

(23)

where $B_p$ and $B_g$ denote the predicted and ground truth bounding boxes, respectively. Precision and Recall at threshold t are computed as

$$\left\{\begin{array}{c}\hfill \mathrm{Precision}\left(t\right)={\displaystyle \frac{TP}{TP+FP}}\\ \hfill \mathrm{Recall}\left(t\right)={\displaystyle \frac{TP}{TP+FN}}\end{array}\right.$$

(24)

For each object class i, the Average Precision $A{P}_i$ is defined as the integral of the Precision-Recall Curve,

$$\begin{array}{cc}\hfill A{P}_i\left(t\right)& ={\int }_0^1{P}_i\left(r\right)dr\hfill \\ \hfill & ={\int }_0^1{P}_{\mathrm{recision}}\left(\mathrm{Recall}\right)d\mathrm{Recall}\hfill \end{array}$$

(25)

where $P_i\left(r\right)$ represents the precision as a function of recall for class i. The $mAP$ is then obtained by averaging over all object classes:

$$mAP\left(t\right)={\displaystyle \frac{1}{n}}\sum _{i=1}^{n}A{P}_i\left(t\right)$$

(26)

This study employs the following two widely used evaluation metrics:

• $mAP@50$: The $mAP$ computed at an IoU threshold of 0.5, reflecting the model’s detection capability under standard overlap conditions.
• $mAP@50$:95: The $mAP$ evaluated across IoU thresholds from 0.5 to 0.95 (step size 0.05) offers a comprehensive and rigorous measure of localization accuracy and model robustness. The mathematical formulation is as follows:

$$mAP@50:95={\displaystyle \frac{1}{n}}\sum _{i=1}^n{\int }_0^1{P}_i\left(r\right)dr$$

(27)

Compared with $mAP@50$, the $mAP@50$:95 metric is more sensitive to localization errors and thus serves as a more rigorous measure for evaluating multimodal object detection performance.

3.4. Experimental Results and Analysis

To systematically validate the effectiveness and generalization capability of the proposed YOLO-CMFM, a comprehensive set of experiments was conducted from four perspectives: (1) performance comparison with mainstream detection methods; (2) robustness verification under different imaging conditions; (3) evaluation of performance gains across different detection frameworks; (4) ablation study of individual submodules.

All experiments were performed on the OGSOD 1.0 and OSPRC multimodal datasets. Model evaluation was performed using the $mAP@50$ and $mAP@50$:95 metrics to comprehensively assess detection accuracy and robustness. In addition, heatmap visualizations were performed to further interpret the model’s feature learning behavior.

3.4.1. Performance Comparison with Mainstream Detection Methods

To evaluate the detection performance of the proposed YOLO-CMFM, extensive experiments were conducted on the OGSOD 1.0 dataset and compared with several mainstream detection approaches. The comparison includes three categories: (1) Classical single-modal models: Faster R-CNN [32], RT-DETR-L [33], RT-DETR-ResNet50 [33], YOLOv11n [19], YOLOv11s [19], and YOLOv11m [19]; (2) Representative multimodal detection methods: TarDAL [16], TINet [34], SuperYOLO [9], ICAFusion [12], and CFT [11]; (3) Simple fusion baselines built upon the YOLOv11s: Add (feature addition) and Multi (element-wise multiplication). The experimental results are summarized in Table 1.

Table 1. Performance comparison of different detection methods on the OGSOD-1.0 dataset.

Method	Modality	${\mathit{AP}}_{50}\uparrow$			$\mathit{mAP}@50\uparrow$	$\mathit{mAP}@50$:$95\uparrow$	Params (M) ↓	GFLOPs ↓	FPS ↑
		Bridge	Harbor	Oil Tank
Single Model Detectors
Faster R-CNN [32]	visible	63.2	78.5	5.2	48.9	23.9	82.4	137.7	17.7
Faster R-CNN [32]	SAR	29.0	55.9	0.9	28.6	11.9	82.4	137.7	15.7
RT-DETR-L [33]	visible	91.8	94.0	83.4	89.6	52.7	32.0	103.4	21.1
RT-DETR-L [33]	SAR	90.1	95.5	83.2	89.6	51.7	32.0	103.4	21.4
RT-DETR-ResNet50 [33]	visible	90.6	92.4	88.5	90.5	49.1	41.9	125.6	20.0
RT-DETR-ResNet50 [33]	SAR	89.6	92.9	86.5	89.7	51.1	41.9	125.6	20.7
YOLOv11n [19]	visible	96.0	99.5	87.5	94.3	62.6	2.6	6.4	181.7
YOLOv11n [19]	SAR	88.9	99.4	68.1	85.5	55.0	2.6	6.4	192.7
YOLOv11s [19]	visible	97.7	99.5	89.7	95.6	67.9	9.4	21.3	167.3
YOLOv11s [19]	SAR	95.7	99.5	82.2	92.5	63.3	9.4	21.3	166.0
YOLOv11m [19]	visible	98.1	99.5	90.5	96.1	72.1	20.0	67.7	134.5
YOLOv11m [19]	SAR	97.3	99.5	89.9	95.6	68.7	20.0	67.7	134.3
Multimodal Detectors
TarDAL [16]	visible-SAR	96.7	95.1	73.6	94.3	–	–	–	–
TINet [34]	visible-SAR	92.1	90.5	70.9	88.3	–	–	–	–
SuperYOLO [9]	visible-SAR	94.9	96.9	90.4	94.7	53.8	4.8	56.2	140.8
ICAFusion [12]	visible-SAR	96.5	99.5	88.0	94.8	64.1	20.2	29.8	96.0
CFT [11]	visible-SAR	97.9	99.5	90.7	95.7	62.9	44.8	35.8	55.3
Multimodal Detectors (Ours)
YOLOv11s-Add (Baseline)	visible-SAR	98.0	99.3	89.6	95.7	69.5	13.2	31.9	143.5
YOLOv11s-Multi (Baseline)	visible-SAR	97.9	99.5	88.7	95.4	68.6	13.2	31.9	145.0
YOLO-CMFM (n)	visible-SAR	97.4	99.5	87.7	94.9	66.0	6.3	18.5	75.3
YOLO-CMFM (s)	visible-SAR	98.1	99.5	90.2	96.0	70.8	23.2	64.2	60.5
YOLO-CMFM (m)	visible-SAR	98.2	99.5	90.9	96.2	75.1	48.4	212.3	58.6

Rows with a gray background indicate different categories of detection methods. The best and second best results are labeled by red and blue respectively. ↑ (resp. ↓) indicates that higher (resp. lower) values are better. All results were reproduced by our implementation except those of TarDAL and TINet, which are directly cited from [35].

The results show that among single-modal detection models, the SAR modality consistently yields lower $mAP$ scores than the visible modality. This performance disparity indicates that the intrinsic speckle noise and sparse texture characteristics of SAR imagery hinder effective feature representation, thereby limiting detection accuracy. The overall performance of multimodal detection methods is mostly superior to that of classical single-modal models, underscoring the effectiveness and necessity of cross-modal fusion in visible-SAR remote sensing scenarios. Although the simple fusion baselines provide moderate improvements over single-modal models, their lack of alignment and feature-selection mechanisms constrains their fusion effectiveness and prevents them from fully exploiting inter-modal complementary information.

In contrast, the proposed YOLO-CMFM achieves superior performance, attaining an $mAP@50$ of 96.2% and an $mAP@50$:95 of 75.1%, the highest among all evaluated methods. A detailed analysis of the results further highlights the accuracy and efficiency of our method. Specifically, the n, s, and m versions of YOLO-CMFM consistently exhibit improved accuracy compared to their corresponding single-modal baselines, validating the robustness of the fusion module across different model scales. In terms of parameter efficiency, YOLO-CMFM (s) maintains a parameter count comparable to ICAFusion but improves $mAP@50$:95 by 6.7%, while YOLO-CMFM (m) achieves a substantial 12.2% increase in $mAP@50$:95 compared to CFT under similar model complexity. Furthermore, regarding the trade-off between accuracy and efficiency, although the introduction of the CMFM incurs additional parameters and computational overhead, it yields significant gains in localization accuracy. Notably, YOLO-CMFM (m) achieves the best results across all accuracy metrics while maintaining an inference speed of 58.6 FPS. This speed is well above the standard real-time threshold of 30 FPS [36], demonstrating that our method successfully strikes an optimal balance between high-precision detection and real-time inference capability.

This substantial performance improvement is attributed to the edge-guided feature enhancement and context-aware gating mechanisms incorporated into YOLO-CMFM. The former reinforces geometric consistency between visible and SAR features by leveraging edge structures, while the latter adaptively modulates cross-modal feature weights according to semantic context, effectively reducing feature redundancy induced by modality-distribution discrepancies. Together, these mechanisms enable more efficient cross-modal feature interaction and support highly adaptive fusion.

To further validate the effectiveness of YOLO-CMFM, Figure 6 presents a comparison of the Precision-Recall (PR) curves of YOLO-CMFM (s) and different methods under IoU thresholds of 0.5, 0.75, and 0.90. It is evident that under the relatively lenient IoU threshold of 0.5, all methods achieve high precision and recall. However, as the threshold becomes more stringent (IoU = 0.75 and 0.90), the PR curves of YOLO-CMFM remain noticeably smoother, exhibiting a much slower decline in performance than the other methods. Across the full recall spectrum, YOLO-CMFM consistently maintains superior precision and recall, clearly outperforming all competing models. These results suggest that YOLO-CMFM provides stronger feature correspondence and false-positive suppression under high IoU thresholds, underscoring its distinct advantages in precise object localization.

Figure 6. PR curves of different object detectors on the OGSOD-1.0 dataset under IoU thresholds of 0.5, 0.75, and 0.90. (a) PR curves of classical single-modal detectors. (b) PR curves of representative multimodal detectors. (c) PR curves of simple fusion baselines. Across all settings, the proposed YOLO-CMFM demonstrates consistently higher precision and recall, particularly under stricter IoU thresholds (0.75 and 0.90), highlighting its capability in precise object localization.

3.4.2. Robustness Verification Under Different Imaging Conditions

To further assess the stability and generalization capability of the proposed YOLO-CMFM under varying imaging conditions, experiments were conducted on the OSPRC dataset. In these experiments, YOLOv11s served as the baseline detector for both visible and SAR single-modal inputs, while the proposed YOLO-CMFM was employed for the visible-SAR multimodal configuration. The experimental results for all settings are summarized in Table 2.

Table 2. Robustness evaluation of the proposed YOLO-CMFM under different imaging conditions on the OSPRC dataset.

Dataset	Modality	${\mathit{AP}}_{50}\uparrow$			$\mathit{mAP}@50\uparrow$	$\mathit{mAP}@50$:$95\uparrow$
		Bridge	Harbor	Oil Tank
OSPRC-10-VH	visible	88.5	99.4	58.4	82.1	50.2
	SAR	82.3	99.3	43.7	75.1	45.9
	visible-SAR	91.7	99.3	60.3	83.8 (↑1.7)	53.2 (↑3.0)
OSPRC-10-VV	visible	88.5	99.4	58.1	81.9	50.1
	SAR	84.3	99.4	46.2	76.6	47.4
	visible-SAR	91.1	99.4	57.8	82.7 (↑0.8)	53.0 (↑2.9)
OSPRC-20-VH	visible	86.9	99.4	53.9	79.9	47.8
	SAR	82.3	99.3	43.7	75.1	45.9
	visible-SAR	91.3	99.2	55.3	82.0 (↑2.1)	51.0 (↑3.2)
OSPRC-20-VV	visible	86.9	98.9	53.9	79.9	47.8
	SAR	84.3	99.4	46.2	76.6	47.4
	visible-SAR	91.7	99.4	56.4	82.5 (↑2.6)	51.3 (↑3.5)
OSPRC-Cloud-VH	visible	48.3	66.8	40.0	51.7	23.0
	SAR	82.3	99.3	43.7	75.1	45.9
	visible-SAR	88.9	99.4	53.0	81.4 (↑6.3)	50.4 (↑4.5)
OSPRC-Cloud-VV	visible	48.3	66.8	40.0	51.7	23.0
	SAR	84.3	99.4	46.2	76.6	47.4
	visible-SAR	91.1	99.4	58.6	82.9 (↑6.3)	52.0 (↑4.6)

↑ indicates that higher values are better. (↑xx) indicates the performance gain of YOLO-CMFM over the best single-modality model, where xx denotes the absolute improvement.

From the overall results, YOLO-CMFM delivers substantial performance gains across all imaging conditions, demonstrating strong adaptability and generalization capability when handling multi-source and heterogeneous data. Specifically, YOLO-CMFM improves $mAP@50$ by approximately 0.8–6.3% and $mAP@50$:95 by 2.9–4.6% over the single-modal models, indicating its effectiveness in mitigating feature degradation caused by variations in imaging resolution and polarization.

It is particularly noteworthy that in cloud-occlusion scenarios, the visible single-modal model exhibits a marked performance degradation due to severe interference from cloud layers. Its $mAP@50$ drops to 51.7%, and $mAP@50$:95 decreases to only 23.0%, indicating substantial deterioration of visible-image features under cloudy conditions. In contrast, YOLO-CMFM leverages the structural compensation provided by SAR imagery to effectively recover geometric and textural cues, yielding improvements of 6.3 and 4.6 percentage points in $mAP@50$ and $mAP@50$:95, respectively. These findings demonstrate that YOLO-CMFM can adaptively adjust modality weight distributions based on scene characteristics, dynamically enhancing the contribution of SAR features when the visible modality becomes degraded, thereby achieving robust cross-modal complementarity and maintaining stable detection performance.

3.4.3. Performance Gain Evaluation Across Different Detection Frameworks

To evaluate the generality and transferability of the proposed CMFM, it was integrated into several representative Ultralytics YOLO detection frameworks, including YOLOv5s [37], YOLOv6s [38], YOLOv8s [39], YOLOv9s [40], YOLOv10s [41], YOLOv11s [19], and YOLOv12s [42]. Experiments were conducted on the OGSOD 1.0 dataset. For the single-modal setting, visible and SAR inputs were processed using the original YOLO models, whereas the visible-SAR multimodal setting employed the CMFM-enhanced versions. The detection performance of all models under both single-modal and multimodal configurations is summarized in Table 3.

Table 3. Performance comparison of CMFM integration across different YOLO frameworks on the OGSOD-1.0 dataset.

Method	Modality	${\mathit{AP}}_{50}\uparrow$			$\mathit{mAP}@50\uparrow$	$\mathit{mAP}@50$:$95\uparrow$
		Bridge	Harbor	Oil Tank
YOLOv5s	visible	96.7	99.5	89.7	95.3	64.9
YOLOv5s	SAR	93.6	99.5	79.8	91.0	60.1
YOLOv5s-CMFM	visible-SAR	97.9 (↑1.2)	99.5	90.6 (↑0.9)	96.0 (↑0.7)	69.7 (↑4.8)
YOLOv6s	visible	94.7	99.3	88.0	94.0	61.6
YOLOv6s	SAR	89.9	99.3	72.1	87.1	55.7
YOLOv6s-CMFM	visible-SAR	95.6 (↑0.9)	99.3	88.3 (↑0.3)	94.4 (↑0.4)	63.2 (↑1.6)
YOLOv8s	visible	97.2	99.5	89.6	95.4	66.8
YOLOv8s	SAR	94.9	99.5	83.5	92.6	62.5
YOLOv8s-CMFM	visible-SAR	98.2 (↑1.0)	99.5	89.7 (↑0.1)	95.8 (↑0.4)	71.7 (↑4.9)
YOLOv9s	visible	97.3	99.5	88.6	95.2	67.2
YOLOv9s	SAR	93.5	99.5	80.5	91.2	61.1
YOLOv9s-CMFM	visible-SAR	98.0 (↑0.7)	99.5	89.6 (↑1.0)	95.7 (↑0.5)	69.8 (↑2.6)
YOLOv10s	visible	96.5	99.5	85.2	93.7	67.2
YOLOv10s	SAR	94.3	99.4	80.0	91.2	63.5
YOLOv10s-CMFM	visible-SAR	97.9 (↑1.4)	99.5	89.3 (↑4.1)	95.6 (↑1.9)	71.8 (↑4.6)
YOLOv11s	visible	97.7	99.5	89.7	95.6	67.9
YOLOv11s	SAR	95.7	99.5	82.2	92.5	63.3
YOLOv11s-CMFM	visible-SAR	98.1 (↑0.4)	99.5	90.2 (↑0.5)	96.0 (↑0.4)	70.8 (↑2.9)
YOLOv12s	visible	97.5	99.5	90.2	95.7	67.1
YOLOv12s	SAR	95.9	99.5	82.1	92.5	64.3
YOLOv12s-CMFM	visible-SAR	98.2 (↑0.7)	99.5	90.8 (↑0.6)	96.2 (↑0.5)	71.7 (↑4.6)

↑ indicates that higher values are better. (↑xx) indicates the performance gain of the visible-SAR model over the best single-modality model, where xx denotes the absolute improvement.

From the overall results, the CMFM consistently delivers stable and notable performance gains across different YOLO frameworks. The improvements in $mAP@50$ range from 0.2% to 1.8%, whereas the gains in the more challenging $mAP@50$:95 metric reach 1.6–4.9%, demonstrating the strong architectural compatibility and cross-model transferability of CMFM. Among all models, YOLOv12s-CMFM achieves the highest $mAP@50$ performance of 96.2%. YOLOv10s-CMFM attains the best $mAP@50$:95 score of 71.8%, highlighting its superior high-precision detection capability.

From a modality perspective, the visible single-modal models generally outperform their SAR counterparts, primarily because the OGSOD 1.0 dataset contains numerous scenes rich in textures and spectral details, where visible imagery naturally provides stronger feature representation. When integrated with CMFM, the multimodal models facilitate deep cross-modal feature interaction and adaptive channel-spatial fusion, effectively enhancing complementary information and suppressing redundancy. Across different object categories, Harbor detection is nearly saturated, Bridge shows consistent improvements of 0.4–1.2%, and Oil Tank exhibits the most significant gains (0.1–4.1%), demonstrating the clear advantage of CMFM for small- and medium-sized target detection.

To further validate the enhanced feature representation capability provided by the CMFM, Grad-CAM-based visualizations [43] were generated to examine the attention distributions of different models. Representative examples are shown in Figure 7. The resulting heatmaps highlight the regions that the models attend to during inference, providing an intuitive assessment of their target-focus capability and background suppression performance [44].

Figure 7. Grad-CAM visualizations of attention responses for different models, where red regions denote strong activation responses and blue regions correspond to weak responses. The visible-only models tend to exhibit diffused attention in texture-rich backgrounds, while SAR-only models produce edge-enhanced but noisy and spatially inconsistent activations. In contrast, the CMFM-enhanced multimodal models generate more concentrated and accurately aligned high-response regions around true targets, with significantly suppressed activation in irrelevant background areas. The colored bounding boxes represent the detection results of the models, with green, red, and blue boxes corresponding to the categories bridge, harbor, and oil tank, respectively.

The visible single-modal model can capture the primary region of the target. However, its attention frequently diffuses into texture-rich background regions, and noticeable attention drifts occur in certain areas. The SAR single-modal model exhibits strong responses along structural edges, but the presence of random noise and speckle results in blurred salient regions and poor boundary consistency. In contrast, the CMFM-enhanced multimodal models demonstrate markedly improved target localization. The activation responses are more concentrated, the high-response areas align closely with the actual target regions, and irrelevant background activation is substantially reduced.

Overall, the CMFM strengthens multimodal feature representation by enabling discriminative feature enhancement and semantically adaptive aggregation. These advancements substantially improve the model’s capability to identify key objects and achieve more precise spatial focus. Moreover, the consistent performance gains observed across various YOLO frameworks underscore the CMFM’s transferability and practical potential as a versatile cross-modal fusion component.

3.4.4. Ablation Study of Individual Submodules

To evaluate the independent contributions and synergistic effects of each submodule within the overall fusion mechanism, comprehensive ablation experiments were conducted on the OGSOD1.0 dataset. Using YOLOv11s-Add as the baseline model, LEGA, BCA, and CGG were incrementally introduced. The results are summarized in Table 4.

Table 4. Ablation study of the submodules on the OGSOD-1.0 dataset.

Index	LEGA	BCA	CGG	$\mathit{mAP}@50$ ↑	$\mathit{mAP}@50$:95 ↑
1				95.7	69.5
2		✓		95.7	69.8
3			✓	95.9	69.9
4	✓	✓		95.9	70.6
5	✓		✓	95.8	70.8
6	✓	✓	✓	96.0 (↑0.3)	70.8 (↑1.3)

✓ indicates that the corresponding submodule is enabled in the method. ↑ indicates that higher values are better. (↑xx) indicates the performance gain of the method over the baseline, where xx denotes the absolute improvement.

Overall, the major performance improvement is concentrated in the stricter $mAP@50$:95 metric, with a gain of 1.3%, whereas the improvement in $mAP@50$ is relatively small at only 0.3%. This suggests that the baseline model is already close to saturation under the more lenient IoU threshold, while the proposed fusion mechanism primarily enhances precise object localization, thereby yielding more substantial gains under stricter matching conditions.

Regarding the individual contributions of each module, incorporating the BCA module improves $mAP@50$:95 by approximately 0.3 percentage points, indicating that the bidirectional cross-modal interaction effectively facilitates complementary fusion between visible and SAR features at the global semantic level. Introducing the gating mechanism alone also yields a slight improvement, but when combined with contextual information, the gain expands to 1.3 percentage points. This demonstrates that global contextual cues play a critical role in modality-weight allocation, enabling the model to adaptively adjust the contributions of visible and SAR modalities based on scene semantics, thereby effectively suppressing redundant features and mitigating interference.

When the LEGA and BCA modules are jointly applied through the FiLM mechanism, the $mAP@50$:95 improves by approximately 1.1 percentage points. This result confirms the effectiveness of edge-saliency guidance for cross-modal alignment, enabling the model to more perceive structural boundaries and salient target regions, thereby enhancing geometric consistency and discriminative feature accuracy.

When all three modules operate together, the model achieves its best overall performance, with improvements of 0.3 and 1.3 percentage points in $mAP@50$ and $mAP@50$:95, respectively, compared with the baseline. The progressive performance gains observed with the addition of each module indicate that they collectively form a complementary hierarchy across different semantic levels. Collectively, these components constitute a progressive fusion architecture that substantially enhances detection accuracy and robustness, demonstrating that the design of the proposed YOLO-CMFM provides structural interpretability.

4. Discussion

The proposed YOLO-CMFM delivers consistent and significant performance improvements across diverse imaging conditions and various detection frameworks, underscoring its effectiveness in deep cross-modal semantic interaction and robust feature modeling.

Notably, in visually degraded scenarios such as cloud occlusion, where visible single-modal performance deteriorates sharply, YOLO-CMFM maintains high detection accuracy by leveraging SAR information, highlighting its ability to adaptively regulate modality contributions according to scene characteristics.

Moreover, the proposed YOLO-CMFM achieves superior performance under high-IoU threshold conditions compared with other detection models, demonstrating its significant advantages in precise object localization, and false-positive suppression. Grad-CAM visualizations reveal that YOLO-CMFM concentrates its attention more accurately on meaningful structural regions while effectively suppressing background clutter and SAR speckle noise, providing strong evidence for the validity of the proposed fusion mechanism.

From a mechanistic perspective, the performance gains of YOLO-CMFM arise from the synergistic interaction of its three core components. Specifically, the following apply:

• The LEGA module leverages a learnable edge-saliency prior to achieve edge-oriented cross-modal structural alignment, effectively mitigating SAR noise interference and visible boundary inconsistencies;
• The BCA module establishes deep interactions and global contextual associations within the joint semantic space of the two modalities, thereby enhancing cross-modal semantic consistency and complementarity;
• The CGG module employs global contextual cues to dynamically weight modality contributions, enabling the model to adaptively balance visible texture details and SAR structural information under varying scene conditions.

Together, these modules unify cross-modal structural alignment, deep semantic fusion, and adaptive regulation without modifying the original detection head or training paradigm, thereby providing an interpretable and transferable fusion framework for visible-SAR multimodal object detection.

Despite these advantages, we acknowledge the trade-off between accuracy and model complexity. As analyzed in Section 3.4, the proposed YOLO-CMFM increases the parameter count and computational overhead compared to the single-modal baseline. Although a larger single-modal model (e.g., YOLOv11m) may achieve comparable performance in favorable visible conditions due to increased parameters, it lacks the inherent robustness required for complex multimodal scenarios (e.g., night-time or cloud-occluded scenes). Therefore, the proposed YOLO-CMFM successfully strikes a balance, providing significant gains in localization accuracy and environmental adaptability while maintaining real-time inference capability.

In future work, further studies will be conducted to validate the effectiveness of YOLO-CMFM across different modality combinations (e.g., infrared, LiDAR, and hyperspectral data) and to investigate its scalability with an increased number of input modalities, assessing its generalization capability in more complex multimodal remote sensing scenarios. Furthermore, addressing the parameter overhead discussed above, we aim to explore lightweight design strategies to further reduce the computational cost of multimodal detectors, enabling a stricter comparison with baselines at the same parameter scale.

5. Conclusions

Due to the substantial differences in imaging mechanisms, noise statistics, and semantic representations between visible and SAR imagery, cross-modal feature alignment becomes challenging and information fusion is often suboptimal. To address these challenges, this paper proposes a multimodal remote sensing object detection method, namely YOLO-CMFM.

Built upon the Ultralytics YOLOv11 framework, the proposed YOLO-CMFM constructs a novel CMFM to accomplish multimodal feature fusion, which comprises an LEGA module, a BCA module, and a CGG module. It adopts a progressive fusion strategy of edge perception, contextual interaction and gated modulation, which greatly helps to enhance the detection performance with respect to structural consistency, semantic complementarity and fusion adaptivity.

Experimental results on multimodal remote sensing datasets, including OGSOD 1.0 and OSPRC, demonstrate that while the proposed YOLO-CMFM maintains competitive detection accuracy comparable to mainstream methods at lower IoU thresholds, it yields substantial performance gains at higher IoU thresholds, highlighting its superior capability in precise object localization. Moreover, despite the additional complexity introduced by the fusion module, the proposed method strikes a balance between high-precision detection and real-time inference capability.

Furthermore, the proposed CMFM can be flexibly embedded into various object detection frameworks, and YOLO-CMFM consistently delivers stable performance gains across varying resolutions, polarization modes, and cloud-occlusion conditions, which also validates its strong transferability, robustness and environmental adaptability.