PCFD-Net: A Parallel Collaborative Fusion-Detection Network for SAR and Optical Imagery

Highlights

What are the main findings?

• A new parallel collaborative training paradigm, PCFD-Net, is proposed to bridge the long-standing separation between SAR–optical image fusion and object detection in remote sensing.
• Through the joint use of CSMDE, dual discriminators, and multi-scale feature injection, PCFD-Net enables effective bidirectional interaction between fusion and detection, and consistently outperforms state-of-the-art methods on both fusion quality and detection accuracy.

What are the implications of the main findings?

• The results show that fusion and detection should be optimized as mutually constrained tasks rather than as isolated or weakly cascaded modules, providing a practical framework for multimodal remote sensing perception.
• The proposed bidirectional collaborative strategy offers a transferable design principle for other low-level and high-level remote sensing tasks, such as change detection, oriented object detection, and multimodal scene understanding.

Abstract

Synthetic aperture radar (SAR)–optical image fusion and object detection are two closely related tasks in remote sensing. Fusion can provide richer texture and structural cues for downstream detection, while detection can, in turn, provide object-level location and semantic information to improve fusion. However, effectively integrating these two tasks within a unified training framework remains challenging. Their optimization objectives are inherently different: fusion emphasizes cross-modal information preservation and structural fidelity, whereas detection focuses more on discriminative target representation. As a result, direct joint training often leads to mutual interference rather than mutual reinforcement. In addition, most existing joint frameworks remain serial or unidirectional, limiting effective bidirectional knowledge transfer between fusion and detection. To address these issues, we propose PCFD-Net (Parallel Collaborative Fusion-Detection Network), which consists of a fusion branch, a detection branch, and a bidirectional interaction branch, and unifies fused image generation and oriented object detection within a single training framework through explicit bidirectional interaction. The fusion branch employs dual ResNet-50 encoders, a multi-scale attention fusion module, and a progressive decoder, while the detection branch is built on YOLOv8. The key component of the proposed framework is the bidirectional interaction branch. On the one hand, the multi-scale fused features generated by the fusion branch are injected into the detection backbone to enhance the exploitation of cross-modal intermediate representations. On the other hand, we develop CSMDE (Category Semantic–Modality Disentangled Embedding), which disentangles category-discriminative and modality-preference semantics to map detector category outputs into instance-level semantic embeddings. These embeddings, together with object locations, are further fed into a dual-discriminator mechanism to reversely constrain the fusion branch, thereby strengthening SAR-discriminative target preservation and optical background structure consistency. Experiments on the M4-SAR and OGSOD1.0 datasets demonstrate that PCFD-Net consistently outperforms representative fusion and detection methods, achieving superior fusion quality and stronger downstream detection performance.

1. Introduction

Synthetic aperture radar (SAR) and optical imagery are two representative modalities for Earth observation [1,2,3]. Optical images provide rich textures, edges, and appearance cues, but are sensitive to illumination, cloud and haze occlusion, and weather variations [4]. By contrast, SAR imagery supports all-weather, all-day sensing and captures stable scattering responses under adverse conditions, yet suffers from speckle noise, less natural textures, and weaker background readability [5]. Their complementary properties make SAR–optical fusion a key problem in multimodal remote sensing perception [6,7,8]. For downstream object detection, transforming this complementarity into representations more favorable for localization and recognition is crucial for improving robustness and accuracy in complex scenes [9]. Consequently, SAR–optical image fusion and fusion-oriented detection have become important topics in intelligent remote sensing interpretation [10,11].

Early studies typically treated image fusion and object detection as separate tasks. With the development of deep learning, CNN-based fusion methods have substantially improved cross-modal image fusion through deep feature extraction and end-to-end reconstruction [12,13], while modern detectors such as YOLO [14], Faster R-CNN [15], and DETR [16] have significantly advanced object perception in remote sensing imagery. However, these two research lines remain largely disconnected: fusion methods are mainly optimized for visual quality, structural preservation, or information enrichment, whereas detection methods usually operate directly on fused or single-modality images. As a result, fusion is not explicitly guided by detection-oriented target representation, and detection cannot in turn regulate how target and background regions are formed during fusion, leading to only limited gains in both visual fidelity and detection performance.

Recent studies have begun to explore joint fusion-detection optimization [17,18,19], which can be broadly divided into two categories. The first combines independently optimized models, as shown in Figure 1a, by training fusion and detection separately and then feeding the fused output into the detector, or by freezing a pretrained fusion model before training the detector. The second adopts cascaded joint optimization [20], as shown in Figure 1b, where one task is used to constrain or assist the other. Although these strategies acknowledge the dependence between fusion and detection, they still face two major limitations. First, the objectives of the two tasks are not fully aligned: fusion favors cross-modal completeness and structural stability, whereas detection requires discriminative target representations, so simple cascading often causes mutual interference during training. Second, existing frameworks are mostly unidirectional, with either fusion serving detection or detection providing auxiliary supervision for fusion, and therefore lack explicit bidirectional interaction at intermediate layers.

Figure 1. Three paradigms for jointly performing image fusion and object detection (OD). (a) Separate optimization: the fusion network $G$ and the detection network $\psi$ are trained independently with fusion loss $L_f$ and detection loss $L_d$, respectively. The SAR and optical images, $I_s$ and $I_o$, are first fused into $I_f$, which is then fed to $\psi$ for detection. (b) Cascaded optimization: one task is used to constrain the other, e.g., the detection network $\psi$ guides the optimization of the fusion network $G$ through $L_f$ and $L_d$, or vice versa. (c) Parallel collaborative optimization: the fusion network $G$ and the detection network $\psi$ are jointly optimized through explicit bidirectional interaction, where multi-scale fused features $H_f$ and detector-derived object locations $S$ are used for cross-branch gradient propagation.

To address these issues, we draw inspiration from the perspective of a continuous two-player cooperative game [21] and treat the fusion and detection branches as two participants that interact continuously toward a shared objective, as shown in Figure 1c. Based on this view, we propose a parallel collaborative fusion-detection network for SAR and optical imagery. The fusion branch is formulated as an adversarial fusion network with dual ResNet-50 encoders, a multi-scale attention fusion module, and a progressive decoder, while a target discriminator and a background discriminator respectively constrain SAR-discriminative target features and optical structure fidelity. The detection branch adopts a standard YOLOv8 architecture in the fused-image domain. More importantly, the two branches are linked through explicit bidirectional interaction: multi-scale fused features are injected into the detection backbone, while detector-predicted target regions and category semantics are fed to the target discriminator to reversely guide fusion through semantic conditioning. During training, both branches are jointly optimized through differentiable interactions; during inference, the discriminators are removed, while the target-enhancement and structure-preserving capability learned during training remains internalized in the retained fusion generator and detection branch.

The main contributions of this work are summarized as follows:

• We propose PCFD-Net, a parallel collaborative fusion-detection network for SAR and optical imagery, which unifies fusion and detection through explicit bidirectional interaction.
• We develop a Category Semantic–Modality Disentangled Embedding (CSMDE) module that maps detection semantics into continuous instance-level embeddings while disentangling category-discriminative and modality-preference information.
• We design a dual-discriminator mechanism for semantic-conditioned target discrimination and background structure discrimination, enabling region-decoupled fusion optimization.
• Extensive experiments demonstrate that the proposed framework consistently improves both fused image quality and downstream detection performance over existing methods.

2. Related Work

2.1. Deep Learning-Based SAR–Optical Image Fusion

The objective of SAR–optical image fusion is to preserve SAR target scattering cues together with optical textures and scene structures in a unified representation. Early methods mainly relied on multi-scale transforms, sparse representation, and optimization models with hand-crafted fusion rules [22], but their representation capacity is limited in the presence of SAR speckle noise, structural mismatch, and target shape variation [23].

Recent studies have shifted to deep learning-based fusion [24,25]. Encoder decoder architectures, especially U-Net-style frameworks, are widely used to extract multi-scale features and reconstruct fused images [26]. Sun et al. [27] improved a symmetric U-Net with non-uniform convolution kernels to enhance target-aware representation, while Gazzea et al. [28] employed an enhanced U-Net for SAR–optical fusion in high-resolution surface structure estimation. To better handle the modality gap, dual-branch and heterogeneous architectures have also been introduced. Luo et al. [29] improved texture and structure preservation through a dual-branch decoder and feature interaction mechanism, and Gu et al. [30] modeled optical and SAR features using Transformer and CNN branches, respectively. In addition, attention and context modeling have been widely adopted to improve adaptive fusion. Li et al. [31], Ma et al. [32], and Guo et al. [33] enhanced cross-modal interaction through collaborative attention, structural attention, and multi-scale context modeling, respectively.

Although these methods have substantially improved image quality and information preservation, most of them remain oriented toward full-image reconstruction, visual fidelity, or structural enhancement, with limited consideration of consistency with downstream object detection.

2.2. Deep Learning-Based SAR–Optical Object Detection Methods

Joint SAR–optical object detection aims to exploit modality complementarity in target saliency, texture detail, and scene structure to improve robustness and accuracy in complex environments. Compared with single-modality detection, such methods can alleviate the sensitivity of optical imagery to illumination and weather conditions [34] while benefiting from the stable scattering responses of SAR imagery.

Existing methods can be broadly categorized into image-level, feature-level, and decision-level fusion detection. Image-level methods first generate a fused image and then perform detection on the fused result. Representative examples include the total variation-constrained fusion-detection method of Ye et al. [35] and MSFE-Net of Guo et al. [36]. Feature-level methods perform joint modeling through intermediate interaction between SAR and optical features within the detector. Wang et al. [37] introduced attention-based cross-modal fusion in the backbone, while He et al. [38] improved feature consistency and complementarity through cross-modal alignment and contrastive constraints. Decision-level methods fuse independent modality-specific detections through post-processing. Ahmed et al. [39] combined parallel SAR and optical detectors with improved non-maximum suppression, and Zhang et al. [40] fused multimodal detections through neighborhood matching and confidence calibration.

Despite their effectiveness, these methods still have clear limitations. Image-level methods depend strongly on front-end fusion quality, feature-level methods usually confine collaboration within the detector, and decision-level methods lack deep interaction during representation learning. More importantly, most existing methods model fusion and detection as a unidirectional process, with limited exploration of how high-level detection semantics can reversely guide fusion.

2.3. Joint Frameworks for Multiple Vision Tasks

Joint learning for multiple vision tasks has become an active topic in computer vision and intelligent remote sensing interpretation, motivated by the shared structural and semantic information across tasks. Existing frameworks have generally evolved from stepwise optimization to cascaded optimization, and more recently to parallel collaborative and unified optimization.

Early stepwise methods first trained a fusion network and then used its outputs to train a detector or segmenter. While simple, such strategies do not effectively inject downstream semantic demands into the fusion process. Cascaded frameworks subsequently introduced high-level task losses into fusion training or used fusion as a front-end enhancement module. Representative examples include TarDAL [20], which jointly models fusion and detection through bilevel optimization and target-aware adversarial learning, and MetaFusion [41], which bridges low-level and high-level representations through meta-feature embedding. Although these methods strengthen task coupling, their interaction is still largely confined to loss-level supervision, and multi-stage optimization may reduce efficiency and stability.

More recent frameworks emphasize parallel collaboration and unified optimization. CRTFS [42] jointly optimizes fusion and salient object detection through cross-task feature interaction, while UniFusOD [43] further introduces region attention and gradient coordination to alleviate optimization conflicts between fusion and detection. These studies move closer to true collaborative learning. However, for SAR–optical imagery, existing methods still rarely integrate region-level structural constraints, category-semantic guidance, and the unified modeling of training-stage collaboration with inference-stage lightweight deployment. This gap motivates the framework proposed in this work.

3. Materials and Methods

This section describes PCFD-Net from the task-level optimization principle to the module-level realization. The methodological contribution lies in coordinating SAR–optical image fusion and oriented object detection through explicit cross-branch interaction, rather than in introducing a new encoder or detector backbone. We first formulate the parallel collaborative objective, then present the fusion and detection branches, and finally describe the semantic-conditioned dual-discriminator supervision and stage-wise training strategy.

3.1. Parallel Collaborative Optimization

The central idea of PCFD-Net is to couple SAR–optical image fusion and oriented object detection through two complementary interaction paths, rather than optimizing the tasks independently or arranging them in a purely sequential cascade. Fusion emphasizes complementary information preservation and structural fidelity, whereas detection requires discriminative object representations. PCFD-Net therefore adopts a parallel collaborative architecture with stage-wise initialization and alternating joint optimization. Figure 2 illustrates the principle of collaborative optimization for the dual tasks of fusion and detection. Figure 3 illustrates the overall architecture of the PCFD-Net network.

Figure 2. Collaborative optimization of fusion and detection dual tasks.

Figure 3. Network structure of PCFD-Net.

In the fusion-to-detection path, the fused image and intermediate multi-scale representations refine the detector at three prediction scales. In the detection-to-fusion path, detector-derived geometry and category semantics construct target-level and background-level constraints for fusion. The interaction is activated explicitly during collaborative fine-tuning; the preceding stages initialize the task-specific branches.

Let the input SAR and optical images be denoted by $I_s$ and $I_o$, respectively, and let ${\mathrm{I}}_{\mathrm{f}}$ denote the fused image. The fusion and detection branches are represented by $G(·;w_f)$ and $\psi (·;w_d)$, where $w_f$ and $w_d$ are their learnable parameters. Their task-specific objectives are retained as follows:

$$\underset{w_f}{\min } L_f(G(I_s,I_o,S;w_f)),$$

(1)

$$\underset{w_d}{\min } L_d(\psi (I_f,H_f;w_d),\gamma ),$$

(2)

where $\gamma$ denotes the oriented object annotations and $H_f$ is the set of multi-scale features generated by the fusion branch. Here, $G$ denotes the complete fusion-generation process optimized under object-level semantic feedback and background-level adversarial regularization, rather than only the pixel-domain forward generator. Accordingly, $S$ is a loss-level semantic-structural condition instead of an image tensor concatenated with $I_s$ and $I_o$. It is defined as:

$$S={\left\{(b_i,q_i,e_i)\right\}}_{i=1}^{N_t},{ q}_i=Softmax(a_i),$$

(3)

where $N_t$ is the number of selected target instances, $b_i$ is the oriented box geometry, $a_i$ and $q_i$ are the category logits and probability distribution, and $e_i$ is the CSMDE condition. During Stage I, the geometry and category information are supplied by ground-truth annotations. During Stage III, they are obtained from top-ranked pre-NMS predictions.

The pre-NMS top-k operation determines discrete routing indices and is therefore not claimed to be differentiable. Once an instance is selected, however, its continuous box tensor and fused-image values remain connected to the rotated grid sampler. Thus, target-level gradients propagate through bilinear sampling to the fused representation and weakly to continuous localization parameters, while the discrete selection identity receives no gradient. The category distribution is stop-gradient stabilized with respect to the detector classification head but remains an optimization target for CSMDE. Rotated NMS is used only during inference.

Let $w_e$, $w_t$, and $w_b$ denote the parameters of CSMDE, the target discriminator, and the background discriminator. During the main-network step, the two discriminators are frozen and the fusion, detection, and embedding parameters are updated by:

$$w_j^{(k + 1)}=w_j^k - {\eta }_j{\nabla }_{w_j}{\mathcal{L}}_{main},j\in \{f,d,e\},$$

(4)

During the discriminator step, generated samples and semantic conditions are detached from the main computational graph, and the discriminator parameters are updated independently:

$$\begin{array}{c}{w}_t^{(k + 1)}=w_t^k-{\eta }_t{\nabla }_{w_t}{\mathcal{L}}_{D_t},\\ w_b^{(k + 1)}=w_b^k-{\eta }_b{\nabla }_{w_b}{\mathcal{L}}_{D_b}.\end{array}$$

(5)

The term parallel collaborative optimization therefore refers to structural parallelism and a coupled objective solved by alternating parameter updates. It does not imply simultaneous minimization of opposing adversarial objectives, a fully differentiable post-processing chain, or guaranteed convergence to a global optimum.

3.2. Fusion Branch

The fusion branch generates an image that retains informative SAR scattering responses and optical spatial structure while exposing intermediate fused representations to the detection branch. It consists of two modality-specific encoders, Multi-Scale Attention Fusion (MAF) blocks, and a progressive decoder.

3.2.1. Modality-Specific Encoder

Because SAR and optical images exhibit different intensity statistics and texture characteristics, they are processed by two ResNet-50 encoders without weight sharing. A three-layer Conv-BN-SiLU stabilization stem precedes the SAR encoder and uses a residual shortcut to suppress local speckle fluctuations without discarding salient scattering responses. The encoders produce four feature levels at spatial scales $\mathrm{H}/4$, $\mathrm{H}/8$, $\mathrm{H}/16$, and $\mathrm{H}/32$, denoted by $\{F_s^{(l)}{\}}_{l=1}^4$ and $\{F_o^{(l)}{\}}_{l=1}^4$. Standard residual operations are not expanded because the innovation lies in their subsequent cross-modal and cross-task use.

3.2.2. Multi-Scale Attention Fusion Block

At each level, the paired features are first aligned by $1\times 1$ projections and transformed into a joint representation $J^{(l)}$. Channel attention identifies globally informative responses, while spatial attention locates regions in which the modalities contribute differently. Their product produces the SAR preference map, and the optical preference is defined as its complement:

$$M_s^{(l)}=\sigma \left(A_c(J^{(l)})\right)\odot \sigma \left(A_p(J^{(l)})\right), M_o^{(l)}=1-M_s^{(l)},$$

(6)

$$A_c=MLP\left(GAP\left(·\right)\right)+MLP(GMP(·)),$$

(7)

$$A_p={Conv}_{7\times 7}([{Mean}_c(·)\hspace{0.17em}\Vert \hspace{0.17em}{Max}_c(·)]),$$

(8)

where $A_c$ and $A_p$ denote channel and spatial-attention transformations. Complementary weighting avoids redundant independent gates and makes the regional modality trade-off explicit. A lightweight residual path compensates for local information attenuated by the weighting operation, yielding:

$$F_f^{(l)}=M_s^{(l)}\odot {\tilde{F}}_s^{\left(l\right)} +{ M}_o^{\left(l\right)}\odot {\tilde{F}}_o^{\left(l\right)} +{ R}^{(l)}\left([{\tilde{F}}_s^{(l)}\parallel {\tilde{F}}_o^{(l)}]\right),$$

(9)

where ${\tilde{F}}_s^{(l)}$ and ${\tilde{F}}_o^{(l)}$ are aligned modality features, $R^{(l)}$ is a lightweight residual refinement block, and $\odot$ denotes element-wise multiplication. Applying MAF at all four levels produces the fused hierarchy $\{F_f^{(l)}{\}}_{l=1}^4$.

3.2.3. Progressive Decoder

The fused features differ substantially in both semantic level and spatial resolution. Deeper features contain stronger target semantics and broader context, whereas shallower features preserve richer structural details. To combine them effectively, we adopt a top-down progressive decoder.

Starting from the deepest fused feature, the decoder gradually upsamples the current decoding feature and concatenates it with the fused feature at the corresponding higher resolution. Each merged feature is then refined by a lightweight decoding block. This progressive design propagates high-level semantics to finer resolutions while restoring local structures through same-scale fused features. After the final decoding stage, two additional upsampling and convolution operations are applied to recover the original resolution and generate the final fused image.

Although the direct output of the fusion branch is the fused image $I_f$, the branch is not merely a reconstruction module. The intermediate fused features generated by the MAF blocks encode complementary SAR–optical information and provide structurally informative representations for detection. Therefore, to match the multi-scale requirements of the detection branch, we select $\{F_f^2,F_f^3,F_f^4\}$ as the cross-branch interaction features, as detailed in Section 3.3.1.

3.3. Detection Branch and Semantic Conditioning

The detection branch uses YOLOv8-OBB to predict object categories and oriented bounding boxes from $I_f$. Its standard backbone, neck, and prediction head are retained; the methodological changes are residual multi-scale feature injection and the conversion of detector outputs into semantic conditions for fusion.

3.3.1. Residual Multi-Scale Feature Injection

Let $P_k$ denote the native detector feature at level $k\in \{3,4,5\}$. The corresponding fusion feature is bilinearly resized, aligned by a $1\times 1$ projection ${\varphi }_k$, and added residually:

$${\tilde{P}}_k=P_{k }+ {\varphi }_k\left(Resize(F_f^{(k-1)},P_k)\right),k\in \{3,4,5\}.$$

(10)

Consequently, $F_f^{(2)}$, $F_f^{(3)}$, and $F_f^{(4)}$ refine P3, P4, and P5 without replacing the detector hierarchy. For candidate $\mathrm{i}$, the detector predicts $b_i=(x_i,y_i,w_i,h_i,{\theta }_i)$ and class logits $a_i\in {\mathbb{R}}^K$, from which $q_i=\mathrm{S}\mathrm{o}\mathrm{f}\mathrm{t}\mathrm{m}\mathrm{a}\mathrm{x}(a_i)$ is obtained. No separate objectness variable is introduced.

3.3.2. Category Semantic–Modality Disentangled Embedding

A conventional category embedding mainly encodes class identity, but does not indicate how SAR and optical modalities contribute to a specific target region. This limitation is non-negligible in SAR–optical fusion-detection, because targets from the same category may require different modality preferences under different local imaging conditions. For example, a bridge region may rely more on SAR scattering continuity when optical textures are weak, whereas a harbor region may benefit more from optical structural details when SAR responses are noisy. Therefore, the semantic condition used to guide fusion should encode not only category-discriminative information, but also region-dependent modality preference.

The proposed Category Semantic–Modality Disentangled Embedding (CSMDE) addresses this issue by combining multi-temperature category representations with a lightweight modality-preference statistic supplied by MAF. It should be emphasized that temperature smoothing itself does not directly distinguish SAR and optical features. Instead, the detector category distribution provides a category-consistent semantic basis, while the regional modality statistic from MAF introduces local SAR–optical contribution information. The two temperature branches therefore reorganize the same category evidence at different sharpness levels, and the modality-preference branch is further conditioned by the regional modality preference. Figure 4 shows the Structure of CSMDE.

Figure 4. Structure of CSMDE. Detector probabilities are reshaped at two temperatures and mapped to independent prototype spaces. A regional modality statistic from MAF conditions the modality-preference branch; decorrelation and distribution reconstruction constrain the two subspaces, and their coupled representation provides the target-discriminator condition.

The detector distribution is reshaped at two temperatures:

$$q_i^{sem}=Softmax\left({\displaystyle \frac{log(q_i + ϵ)}{{\tau }_s}}\right),{ q}_i^{mod}=Softmax\left({\displaystyle \frac{log(q_i + ϵ)}{{\tau }_m}}\right),$$

(11)

where $ϵ$ ensures numerical stability. The lower-temperature branch sharpens category discrimination, whereas the smoother branch preserves uncertainty useful for modality-preference modeling. Let $E_{sem}\in {\mathbb{R}}^{K\times d_s}$ and $E_{mod}\in {\mathbb{R}}^{K\times d_m}$ be independent learnable prototype matrices. A regional SAR-preference statistic $m_i$ is obtained by averaging the MAF weight map over $b_i$, and the prototype representations are:

$$m_i=ROIAvg(M_s,b_i),{ h}_i^{sem}=q_i^{sem}{E}_{sem}, { h}_i^{mod}=q_i^{mod}{E}_{mod},$$

(12)

The two branches are projected into a common space and coupled element-wise:

$$u_i=W_s{h}_i^{sem}, { v}_i=W_m[h_i^{mod}\parallel m_i], e_i=\delta (u_i\odot v_i).$$

(13)

The element-wise coupling allows the category-discriminative semantic branch to provide semantic identity constraints, while the modality-preference branch adjusts the condition according to the local modality preference. The resulting embedding is used as the semantic condition of the target discriminator.

The motivation for disentanglement is that category identity and modality preference have different invariance properties. Category semantics should remain relatively stable for targets of the same class, whereas modality preference may vary with local SAR scattering strength, optical texture quality, background clutter, and registration conditions. If these two factors are mixed into a single embedding, the discriminator may receive ambiguous semantic conditions, making it more difficult to consistently constrain target preservation during fusion. Therefore, CSMDE encourages the two subspaces to be complementary rather than redundant.

Here, disentangled denotes lightweight subspace decorrelation rather than statistical independence. The normalized orthogonality penalty is:

$${\mathcal{L}}_{dec}={\displaystyle \frac{1}{N_t}}{\displaystyle \sum _{i=1}^{N_t}}{\left({\displaystyle \frac{u_i^{\mathsf{T}}{v}_i}{‖u_i{\Vert }_2‖v_i{\Vert }_2+ϵ}}\right)}^2.$$

(14)

Two lightweight reconstruction heads map the subspace representations back to their temperature-smoothed distributions:

$${\tilde{q}}_i^{sem}=Softmax(R_s(u_i)), {\tilde{q}}_i^{mod}=Softmax(R_m(v_i)),$$

(15)

Here, $R_s: {\mathbb{R}}^d\to {\mathbb{R}}^K$ and $R_m: {\mathbb{R}}^d\to {\mathbb{R}}^K$ denote two independently parameterized lightweight reconstruction heads for the category-discriminative semantic and modality-preference subspaces, respectively. Each head uses a fully connected projection to map its input representation to $K$-dimensional class logits, which are subsequently normalized by Softmax. The two heads do not share parameters, allowing each subspace to preserve its branch-specific information during distribution reconstruction.

$${\mathcal{L}}_{dist}={\displaystyle \frac{1}{N_t}}{\sum }_{i=1}^{N_t}[D_{KL}(\mathrm{s}\mathrm{g}(q_i)\parallel {\tilde{q}}_i^{sem})+D_{KL}(\mathrm{s}\mathrm{g}(q_i)\parallel {\tilde{q}}_i^{mod})]$$

(16)

where $\mathrm{s}\mathrm{g}(·)$ denotes stop-gradient. Thus, the reconstruction objective updates CSMDE without perturbing the detector classification head. Ground-truth-aligned categories stabilize the prototypes during Stage I, whereas predicted distributions drive CSMDE during collaborative fine-tuning.

It should be noted that CSMDE is used during training to generate semantic conditions for the target discriminator. Once the discriminators are removed at inference, CSMDE is no longer required in the deployment graph.

3.4. Dual-Discriminator Adversarial Supervision

A single global discriminator would impose one distributional preference on regions with different purposes. Object regions should retain category-consistent SAR responses, whereas non-object regions should preserve optical edges and structural continuity. PCFD-Net therefore uses a semantic-conditioned target discriminator and a background structure discriminator.

3.4.1. Semantic-Conditioned Target Discriminator

Stage I uses ground-truth OBBs to define target regions. Stage III uses top-ranked pre-NMS predictions, avoiding rotated NMS in the training path. A differentiable rotated sampler implemented with an affine grid and bilinear grid sampling extracts paired $32\times 32$ patches from $I_s$ and $I_f$; out-of-bounds samples are zero padded. The selected continuous box coordinates are not detached, whereas the discrete top-k identity is treated as a fixed routing decision.

The target discriminator is a lightweight projection-conditioned PatchGAN. Let $z_i=H_t(r_i)$ be the local patch representation. Its score is:

$$D_t(z_i,e_i)=u_t(z_i) + ⟨v_t(z_i),e_i⟩,$$

(17)

where the first term evaluates SAR-like target realism and the projection term measures consistency with $e_i$. SAR target patches are treated as real and fused target patches as generated. The discriminator and generator-side hinge losses are:

$$\begin{array}{c}{\mathcal{L}}_{D_t}={\mathbb{E}}_i\left[max(0,1-D_t(z_{s,i},e_i)) + max(0,1 + D_t(z_{f,i},e_i))\right],\\ {\mathcal{L}}_{G_t}=-{\mathbb{E}}_i[D_t(z_{f,i},e_i)].\end{array}$$

(18)

The losses are averaged over valid target ROIs. If no valid target is available, the target-discriminator update is skipped and the corresponding loss is set to zero.

3.4.2. Background Structure Discriminator

The background discriminator operates in the Sobel-gradient domain. Let $M_{obj}$ be the union of ground-truth masks in Stage I or predicted masks in Stage III. A slight dilation excludes uncertain target boundaries from background supervision. The optical and fused background representations are:

$$\begin{array}{c}{B}_o=(1-{\tilde{M}}_{obj})\odot ‖{\nabla }_{Sobel}{I}_o{\Vert }_2,\\ B_f=(1-{\tilde{M}}_{obj})\odot ‖{\nabla }_{Sobel}{I}_f{\Vert }_2.\end{array}$$

(19)

where ${\tilde{M}}_{obj}$ is the dilated object mask. The optical background is treated as real and the fused background as generated. Their hinge losses are:

$$\begin{array}{c}{\mathcal{L}}_{D_b}=\mathbb{E}\left[max(0,1-D_b(B_o)) + max(0,1 + D_b(B_f))\right],\\ {\mathcal{L}}_{G_b}=-\mathbb{E}[D_b(B_f)].\end{array}$$

(20)

The two discriminators therefore impose region-specific constraints: $D_t$ promotes semantically consistent SAR characteristics within targets, while $D_b$ constrains the fused background toward the optical structural distribution. Both discriminators are used only during training.

3.5. Loss Functions and Stage-Wise Training

The main objective combines image reconstruction, oriented detection, semantic embedding, and generator-side adversarial terms. To avoid expanding standard operations, only the complete objectives required to reproduce the collaborative optimization are given below.

The fusion reconstruction loss combines structural similarity and gradient preservation:

$${\mathcal{L}}_{fus}=[1-\mathrm{S}\mathrm{S}\mathrm{I}\mathrm{M}(I_f,I_s)]+[1-\mathrm{S}\mathrm{S}\mathrm{I}\mathrm{M}(I_f,I_o)]+{\Vert |\nabla I_f|-\max (|\nabla I_s|,|\nabla I_o|)\Vert }_1$$

(21)

The detection branch follows the YOLOv8-OBB objective without an independent objectness or additional angle term:

$${\mathcal{L}}_{det}={\lambda }_{cls}{\mathcal{L}}_{cls}+{\lambda }_{obb}{\mathcal{L}}_{obb}+{\lambda }_{dfl}{\mathcal{L}}_{dfl},$$

(22)

Here, ${\mathcal{L}}_{cls}$ is the binary cross-entropy classification loss between the predicted class logits and their assigned category targets. ${\mathcal{L}}_{obb}$ is the probabilistic-IoU-based oriented bounding-box regression loss, which jointly constrains the center position, spatial extent, and orientation of each decoded rotated box. ${\mathcal{L}}_{dfl}$ denotes the distribution focal loss, which models the box-regression offsets as discrete probability distributions to improve localization precision. The coefficients ${\lambda }_{cls}$, ${\lambda }_{obb}$, and ${\lambda }_{dfl}$ balance the contributions of the three terms. The rotation angle is optimized through the decoded box geometry in ${\mathcal{L}}_{obb}$, rather than through an additional independent angle loss.

The CSMDE objective combines distribution reconstruction and subspace decorrelation:

$${\mathcal{L}}_{sem}={\mathcal{L}}_{dist}+{\lambda }_{dec}{\mathcal{L}}_{dec}$$

(23)

The main-network objective is:

$${\mathcal{L}}_{main}={\lambda }_{fus}{\mathcal{L}}_{fus}+{\lambda }_{det}{\mathcal{L}}_{det}+{\lambda }_T{\mathcal{L}}_{G_t}+{\lambda }_B{\mathcal{L}}_{G_b}+{\lambda }_{sem}{\mathcal{L}}_{sem} .$$

(24)

The adopted balancing coefficients are ${\lambda }_{fus}=1.0$, ${\lambda }_{det}=1.0$, ${\lambda }_T=0.5$, ${\lambda }_B=0.2$, and ${\lambda }_{sem}=0.1$. Target losses are normalized by the number of valid ROIs, background losses by the number of background patches, and fusion losses by the number of images in the batch. Loss warm-up and gradient clipping are used to stabilize the joint stage without introducing dynamic task weighting.

Training is conducted over 250 epochs using a three-stage optimization schedule, with the stage boundaries located between epochs 50/51 and 130/131. In Stage I (epochs 1–50), the fusion generator, CSMDE, and the target and background discriminators are optimized using ground-truth target geometry and category labels. The active objectives include the structural and gradient reconstruction terms, the two generator-side adversarial terms, the two discriminator objectives, and the CSMDE semantic objective. As shown in Figure 5a–c and e, the fusion reconstruction and semantic losses decrease progressively, while the generator-side adversarial and discriminator losses enter bounded fluctuating regimes after their initial reductions. Meanwhile, the increasing SCD and VIF values in Figure 5g indicate that the fusion branch progressively improves its information-preservation capability. These observations show that Stage I establishes stable fusion-side and semantic priors without evident numerical instability.

Figure 5. Training dynamics of the three-stage optimization strategy over 250 epochs. Stage I (epochs 1–50) optimizes the fusion branch, CSMDE, and both discriminators; Stage II (epochs 51–130) trains only the detector with the fusion-related modules frozen; Stage III (epochs 131–250) jointly optimizes the full framework. (a) ${\mathcal{L}}_{fus}$ (b) ${\mathcal{L}}_{G_t}$ and ${\mathcal{L}}_{G_b}$. (c) ${\mathcal{L}}_{D_t}$ and ${\mathcal{L}}_{D_b}$. (d) ${\mathcal{L}}_{det}$. (e) ${\mathcal{L}}_{dist}$, ${\mathcal{L}}_{dec}$, and ${\mathcal{L}}_{sem}$. (f) validation $A{P}_{75}$ and $mA{P}_{50:95}$. (g) Validation SCD and VIF. Vertical dashed lines denote stage boundaries. Blank intervals indicate inactive objectives due to module freezing. The short transient at the beginning of Stage III is caused by network unfreezing and the transition to prediction-driven collaborative optimization.

In Stage II (epochs 51–130), the fusion branch, CSMDE, and both discriminators are frozen, and the fusion branch is maintained in evaluation mode. Only the detector is optimized, while residual feature injection remains active using fixed fusion features. Accordingly, the fusion, adversarial, discriminator, and semantic objectives are inactive and are not plotted during this stage. As shown in Figure 5d,f, the detection loss decreases steadily, accompanied by continuous improvements in ${\mathrm{A}\mathrm{P}}_{75}$ and ${mAP}_{50:95}$. This behavior indicates that the detector adapts to the same forward feature-interaction path subsequently used during collaborative optimization. In addition, the constant SCD and VIF values throughout Stage II in Figure 5g are consistent with the unchanged fused outputs produced by the frozen fusion branch.

In Stage III (epochs 131–250), the fusion branch, detection branch, and CSMDE are unfrozen and jointly optimized using the complete objective in Equation (24). For each mini-batch, the target and background discriminators are first updated once using detached generated samples. A fresh forward pass is then performed with both discriminators frozen, after which the fusion generator, detector, and CSMDE are updated once. At the beginning of this stage, the target regions and semantic conditions are switched from ground-truth annotations to selected pre-NMS predictions. This change in supervision, together with network unfreezing and the activation of cross-branch gradients, produces a short adaptation transient, manifested as temporary increases in several losses and slight decreases in the validation metrics near epoch 131. The active losses subsequently decrease or remain within bounded regimes, while both detection and fusion metrics recover and continue to improve. These training dynamics provide evidence that the proposed stage-wise schedule is implemented consistently and supports numerically stable joint adaptation. They should not, however, be interpreted as a standalone theoretical proof of convergence or of the superiority of the three-stage strategy over alternative training schedules. Throughout Stage III, gradients remain defined with respect to the selected continuous prediction tensors and bilinearly sampled image values before inference-time post-processing.

During inference, the two discriminators are removed. The fusion branch, residual feature-injection path, YOLOv8-OBB detector, and lightweight CSMDE mapping are retained in the joint mode, producing the fused image and oriented detection results. Rotated NMS is applied only to the final detection predictions.

4. Experimental Results

4.1. Datasets and Experimental Settings

To support training and evaluation, we reconstruct a SAR–optical multimodal fusion-detection benchmark from M4-SAR [9] and OGSOD1.0 [44]. The official M4-SAR dataset provides six categories, namely Bridge, Harbor, Oil Tank, Playground, Airport, and Wind Turbine, whereas OGSOD1.0 provides Bridge, Harbor, and Storage Tank/Oil Tank in oriented bounding box format. In this work, the final category taxonomy is defined as Bridge, Harbor, Oil Tank, Playground, Airport, and Ship. The Wind Turbine category is excluded because, after registration-quality-based screening, only a limited number of valid image pairs and target instances remain, which is insufficient for stable category-wise evaluation. The Ship category is introduced through supplementary annotation on well-aligned SAR–optical pairs. The final reconstructed benchmark contains 51,393 aligned image pairs and 183,986 annotated instances.

During benchmark reconstruction, image pairs were first subjected to automatic pair screening based on local registration quality verification. Specifically, we employed the HSC-Prior [45] model to assess cross-modal local alignment and remove pairs with unreliable geometric correspondence. Pairs with severe cross-modal misalignment, annotation inconsistency, or strong SAR contamination were discarded, and the remaining samples were further checked manually when needed. After screening, all retained image pairs were resized to 256 × 256 by isotropic scaling with zero padding, and all labels were unified into the YOLO-OBB four-point format. For image pairs with reliable registration, supplementary ship annotations were generated by a candidate-detection-plus-manual-verification pipeline. After reconstruction, the dataset was split into training, validation, and test sets at a ratio of 7:1:2 at the scene level, so that patches from the same scene or geographic region did not appear in different subsets. This setting reduces evaluation bias caused by scene overlap. The source-wise and category-wise statistics of the reconstructed benchmark are summarized in Table 1.

Table 1. Composition of SAR–optical fusion-detection benchmark.

Source-Wise Image-Pair Statistics	Count	Ratio (%)	Final Category Distribution	Bri.	Har.	Ply.	Apr.	Shp.	Oil.
M4-SAR	36,842	71.69	Instances	62,418	18,764	27,985	5143	28,420	41,256
OGSOD1.0	14,551	28.31	Ratio (%)	33.93	10.20	15.21	2.80	15.45	22.42
Total image pairs	51,393	100.00	Total instances	183,986

All experiments are implemented in Python 3.10 and conducted on two NVIDIA RTX 4090 GPUs. For the fusion branch, the dual encoders are initialized with ImageNet-pretrained ResNet-50 weights, while the remaining modules are randomly initialized. The fusion branch is optimized using AdamW, with ${\beta }_1=0.9$, ${\beta }_2=0.999$, and weight decay $1\times 10^{-4}$. The detection branch follows the official YOLOv8-OBB implementation and is optimized using SGD with momentum 0.937 and weight decay $5\times 10^{-4}$.

Before training, the input data are normalized. SAR images are first log-compressed and then normalized to $[0,1]$, while optical images are linearly normalized. Data augmentation includes random horizontal flipping, random vertical flipping, random scale perturbation, and photometric jittering for optical images only. All geometric augmentations are synchronously applied to the SAR image, optical image, and their annotations. For the CSMDE module, four key hyperparameters are involved: the temperature parameters ${\tau }_s$ and ${\tau }_m$, and the prototype dimensions $d_s$ and $d_m$. Based on the ablation study, we set ${\tau }_s=0.5$, ${\tau }_m=1.5$, and $d_s=d_m=64$. A detailed analysis is provided in Section 4.4.4.

To stabilize optimization, we adopt a stage-wise training strategy. In the first stage, the fusion generator, CSMDE, and both discriminators are trained; ground-truth OBBs define target ROIs, and ground-truth category labels stabilize the semantic prototypes. The initial learning rate is set to $1\times {10}^{-4}$, and the model is trained for 50 epochs. In the second stage, the fusion branch, CSMDE, and both discriminators are frozen, the fusion branch is placed in evaluation mode, and only the detector is trained for 80 epochs with an initial learning rate of $1\times {10}^{-2}$; residual feature injection remains active with fixed fusion features. In the final stage, the fusion branch, detection branch, and CSMDE are jointly optimized for 120 epochs. During each joint-training iteration, the target and background discriminators are first updated once using detached generated samples; a fresh forward pass is then performed, the discriminators are frozen, and the main network is updated once. The main network and the two discriminators use independent optimizers. The total batch size is 16, and a cosine annealing schedule is used throughout training. During inference, both discriminators are removed, while the fusion branch, residual feature-injection path, YOLOv8-OBB detector, and lightweight CSMDE mapping are retained to output the fused image and oriented detection results.

4.2. Fusion Performance Analysis

To evaluate the effectiveness of the fusion branch, we compare the proposed method with six representative fusion methods, including TarDAL [20], EMMA [46], MetaFusion [41], CoCoNet [47], SeAFusion [48], and Diff-IF [49]. These methods cover different technical paradigms, including attention-enhanced fusion, unified representation learning, context modeling, task-driven fusion, diffusion-based generation, and collaborative fusion-detection methods.

For methods originally developed for infrared–visible fusion, such as TarDAL, SeAFusion, and MetaFusion, we used SAR-adapted implementations by preserving their core architectures and optimization strategies while replacing only infrared-specific priors or unavailable supervision signals with the closest task-compatible alternatives. In particular, TarDAL uses OBB-rasterized target masks during training instead of infrared saliency supervision, SeAFusion is adapted from segmentation-guided fusion to detection-guided fusion using the available OBB annotations, and MetaFusion retains its original YOLOv5s-based meta-feature embedding mechanism while being retrained on SAR–optical pairs. EMMA, CoCoNet, and Diff-IF, which are general multimodal fusion frameworks, are also retrained on the same reconstructed SAR–optical benchmark. In this way, all compared methods are evaluated under the same preprocessing pipeline, scene-level data split, and testing protocol.

Fusion performance is evaluated qualitatively and quantitatively. For quantitative comparison, we use a set of complementary metrics, including AG, SF, SD, SCD, PSNR, EI, VIF, EN, MI, Qabf, and SSIM, to assess sharpness, contrast, information preservation, visual fidelity, and structural consistency.

Figure 6 shows representative fusion results on M4-SAR and OGSOD1.0. We focus on two typical scenarios from M4-SAR, namely the harbor–ship scene and the urban bridge scene, while OGSOD1.0 exhibits similar trends. In the harbor scene, many comparison methods introduce strong SAR responses indiscriminately to preserve target saliency, which damages shoreline textures and blurs the structural relationship between docked ships and the surrounding background. By contrast, our method preserves ship and harbor scattering cues while suppressing the intrusion of onshore strong scattering into the optical background, resulting in clearer target contours and more complete harbor structures. In the urban bridge scene, comparison methods either overemphasize SAR responses and produce high-frequency interference around the bridge, or overpreserve optical appearance and weaken bridge saliency. Our method more stably maintains bridge continuity and local prominence while preserving the surrounding texture hierarchy and spatial continuity.

Figure 6. Visual comparison of fusion results. The first three rows are from the M4-SAR dataset, whereas the last three rows are from the OGSOD1.0 dataset; the third and sixth rows show images with simulated cloud and fog. The target position is within the red box.

We further evaluate robustness under cloud contamination by introducing simulated cloud interference into a harbor scene in M4-SAR and a dense-ship scene in OGSOD1.0. When the optical modality is degraded, most comparison methods propagate cloud-induced artifacts to the fused image, causing blurred background regions, local brightness drift, and poor separation between targets and contaminated areas. In contrast, our method better suppresses cloud-induced structural corruption while maintaining clear responses for ships, harbor boundaries, and other salient regions.

These qualitative results reflect the effect of the proposed collaborative training strategy. By explicitly interacting with the detection branch during training, the fusion branch learns representations that are more favorable for target perception. At the same time, the dual discriminators impose differentiated constraints on target and background regions, while category semantics further guide target-oriented fusion. Their joint effect yields more stable target enhancement and background preservation in complex and degraded scenes.

Table 2 and Table 3 report the quantitative results on M4-SAR and OGSOD1.0. Overall, the proposed method achieves the best or near-best performance on most metrics, with particularly consistent advantages on AG, SF, SD, EI, VIF, MI, and Qabf, indicating strong performance in image sharpness, contrast, visual fidelity, and local structural fusion. Although our method is not always the best on SCD, PSNR, or SSIM, this is expected because the proposed framework does not optimize for global difference preservation or pixel-level fidelity alone. Instead, it emphasizes region-sensitive fusion representations that better support downstream detection. In this sense, higher values on sharpness-related metrics are meaningful only if background structure and texture consistency are preserved rather than being sacrificed for indiscriminate high-frequency enhancement.

Table 2. Comparison of fusion evaluation metrics of M4-SAR for different SOTA fusion methods.

Method	AG	SF	SD	SCD	PSNR	EI	VIF	EN	MI	Qabf	SSIM
TarDAL	18.74	40.92	52.86	1.38	14.21	55.34	0.36	7.51	1.63	0.46	0.87
EMMA	19.86	42.48	54.73	1.44	14.68	57.89	0.38	7.58	1.69	0.49	0.89
MetaFusion	21.05	46.31	57.84	1.47	14.11	59.75	0.40	7.63	1.78	0.54	0.90
CoCoNet	20.11	44.76	59.02	1.56	13.76	58.43	0.39	7.79	1.74	0.50	0.88
SeAFusion	20.48	43.95	55.61	1.49	15.07	60.82	0.42	7.60	1.82	0.56	0.91
Diff-IF	23.62	56.84	60.97	1.43	14.02	62.31	0.47	7.55	2.04	0.60	0.95
Ours	24.71	58.07	63.84	1.51	14.63	63.95	0.53	7.70	2.18	0.65	0.94

Table 3. Comparison of fusion evaluation metrics of OGSOD1.0 for different SOTA fusion methods.

Method	AG	SF	SD	SCD	PSNR	EI	VIF	EN	MI	Qabf	SSIM
TarDAL	17.92	38.76	50.43	1.34	13.87	53.16	0.34	7.42	1.58	0.44	0.84
EMMA	18.65	40.15	52.11	1.41	14.24	55.03	0.36	7.49	1.64	0.47	0.86
MetaFusion	19.84	43.82	55.73	1.45	13.79	57.92	0.38	7.54	1.73	0.52	0.85
CoCoNet	19.12	42.37	57.46	1.53	13.44	56.71	0.37	7.71	1.69	0.48	0.86
SeAFusion	19.36	41.88	53.84	1.47	14.56	58.26	0.41	7.51	1.77	0.55	0.90
Diff-IF	22.08	54.36	59.68	1.40	13.68	60.14	0.46	7.46	1.98	0.59	0.91
Ours	22.94	55.71	61.92	1.49	14.08	61.87	0.50	7.63	2.11	0.63	0.92

Beyond visual quality, a fused image should also provide a more effective input representation for detection. We therefore further evaluate the detection adaptability of fused images on M4-SAR. Specifically, we train and test the same YOLOv8 detector using four types of inputs: (1) optical images, (2) SAR images, (3) fused images generated by the six comparison methods, and (4) fused images generated by our method.

Figure 7 and Table 4 show the corresponding detection results. In general, detectors trained on optical images outperform those trained on SAR images alone, and most fusion methods further improve detection performance. However, the fused images generated by TarDAL and MetaFusion perform worse than pure optical images. This is mainly because most test samples from the M4-SAR dataset possess favorable optical image quality, and the proportion of samples suffering severe optical degradation is limited. Moreover, these two collaborative fusion-detection approaches are designed for optical-infrared imagery; their strategy of extracting infrared salient regions to boost target saliency cannot be directly applied to SAR images. Under such conditions, fusion methods that indiscriminately introduce SAR speckle, over-bright responses, or irrelevant textures may corrupt the originally clean optical structure and reduce the utility of the fused image for detection. By contrast, the fused images generated by our method achieve the best detection performance, with more evident gains in categories such as Ship and Harbor, indicating that the proposed fusion branch organizes complementary SAR–optical information into a representation more suitable for downstream detection.

Figure 7. Visual comparison of detection results on fused images.

Table 4. Comparison of detection evaluation metrics for different SOTA fusion methods.

Method	AP50							AP75	mAP
	Bri.	Har.	Ply.	Apr.	Shp.	Oil.	ALL
Optical	70.2	84.1	80.3	86.5	68.4	83.4	78.8	64.1	55.6
SAR	63.0	78.5	73.4	81.2	60.9	76.8	72.3	57.9	49.8
TarDAL	68.5	82.0	78.4	84.3	65.7	81.1	76.7	61.4	53.2
EMMA	70.7	84.4	80.6	86.7	69.2	83.8	79.2	64.6	55.9
MetaFusion	69.4	82.7	79.1	85.1	66.8	82.0	77.5	62.0	53.9
CoCoNet	72.3	85.0	81.1	87.0	69.8	84.4	79.9	65.0	56.4
SeAFusion	71.8	85.6	81.5	87.4	70.5	84.9	80.3	65.6	57.0
Diff-IF	73.6	85.4	81.3	87.2	70.1	84.6	80.4	65.3	56.7
Ours	77.0	86.8	82.7	88.5	72.6	86.1	82.3	67.2	58.6

4.3. Detection Performance Analysis

To evaluate the effectiveness of the detection branch within the proposed parallel collaborative framework, we compare our method with several representative state-of-the-art cross-modal detection methods, including COMO [50], TarDAL [20], MHFNet [51], CFT [52], MetaFusion [41], CLANet [53], CSSA [54], CMADet [55], ICAFusion [56], MMIDet [57], and E2E-OSDet [9]. Our detector is built on the fusion generator, multi-scale feature injection, and a YOLOv8-based OBB detection network. Following standard practice, we report AP₅₀, AP₇₅, and mAP, together with category-wise detection results.

Table 5 summarizes the quantitative comparison. The proposed method achieves the best overall performance, with particularly clear gains on AP₇₅ and mAP, indicating more accurate localization and stronger overall detection capability. At the category level, the improvements are especially pronounced for Bridge and Ship, which is consistent with the fact that these categories are more sensitive to structural continuity and SAR-discriminative scattering responses. This suggests that the advantage of the proposed method is not limited to object presence recognition, but also extends to more precise localization and more robust category discrimination.

Table 5. Comparison of detection evaluation metrics for different SOTA detection methods.

Method	AP50							AP75	mAP
	Bri.	Har.	Ply.	Apr.	Shp.	Oil.	ALL
COMO	72.3	90.8	88.7	90.5	73.6	89.6	84.3	66.3	57.9
TarDAL	71.1	89.2	84.3	89.5	71.9	88.3	82.4	65.7	57.2
MHFNet	73.7	91.4	89.9	90.4	74.5	90.1	85.0	65.6	58.0
CFT	75.8	92.5	91.6	90.3	76.7	91.0	86.3	68.9	59.9
MetaFusion	73.0	90.1	86.4	89.9	75.5	89.0	84.0	67.6	58.3
CLANet	74.8	92.2	91.3	91.6	76.3	90.8	86.2	68.5	59.6
CSSA	73.3	91.7	88.9	91.6	75.4	89.8	85.1	66.4	58.0
CMADet	70.9	90.7	86.4	91.7	73.9	88.2	83.6	63.5	55.7
ICAFusion	74.7	91.9	91.0	91.8	76.9	90.6	86.2	67.3	58.8
MMIDet	74.9	92.6	91.7	91.4	77.2	91.2	86.5	68.6	59.8
E2E-OSDet	77.7	90.7	91.8	92.1	78.8	91.5	87.1	70.3	61.4
Ours	80.1	92.8	92.6	93.0	81.2	92.4	88.7	72.1	63.3

The performance gain does not arise from a single component, but from the overall effect of collaborative training. Multi-scale fusion feature injection improves the detector’s ability to exploit cross-modal intermediate representations, the dual discriminators guide the generator toward region representations more compatible with detection, and the Category Semantic–Modality Disentangled Embedding further enhances category consistency in target regions. Together, these components enable the detection branch to better integrate complementary SAR–optical cues.

We further provide Grad-CAM [58] visualizations to illustrate the attention behavior of different detection settings. As shown in Figure 8, detectors using optical-only or SAR-only inputs often produce scattered high-response regions and are more easily affected by background textures or local interference. When the fused images generated by our method are used as input to YOLOv8, the attention responses become noticeably more concentrated on the target regions, indicating that the fused images provide more detection-friendly representations. More importantly, the complete dual-modal detection branch of the proposed method produces the most compact and target-aligned responses under the same fused-image input, with high-activation regions better matching object contours and key discriminative areas while suppressing background interference.

Figure 8. Grad-CAM heatmaps of the proposed PCFD-Net. (a) Ground truth (GT), (b) Optical image, (c) SAR image, (d) Detection results of YOLOv8 on fused images generated by PCFD-Net, (e) Detection outputs of PCFD-Net.

4.4. Ablation Study

Our ablation study is designed to answer three questions:

(1)

Whether the core components of the parallel collaborative framework are all necessary;

(2)

Whether the distribution consistency constraint in CSMDE is essential for stabilizing cross-task semantic representation;

(3)

Whether the temperature parameters and prototype dimensions in CSMDE are reasonably configured.

Accordingly, we conduct three groups of ablations: network structure ablation, distribution consistency loss ablation, and CSMDE parameter ablation.

4.4.1. Network Structure Ablation

To evaluate the individual and joint contributions of the main components, we construct four progressively enhanced variants: Variant-A, which retains only the fusion generator; Variant-B, which further introduces the target and background discriminators; Variant-C, which additionally includes the detection branch and cross-branch feature interaction but without CSMDE; and Variant-D, which corresponds to the full model. Experiments are conducted on the M4-SAR dataset.

Table 6 reports the results. From Variant-A to Variant-B, the fusion metrics improve most directly, and both SCD and VIF increase noticeably. The detection mAP also rises, indicating that the dual discriminators first improve the functional decoupling of target and background regions. In particular, the target discriminator encourages the preservation of SAR-discriminative target cues, while the background discriminator suppresses irrelevant interference in non-target regions, thereby improving the detection utility of the fused image.

Table 6. Comparison of quantitative metrics for network architecture ablation study.

Variant	Gen.	Dual Discri.	Det. Branch	CSMDE	SCD	VIF	MI	SSIM	mAP
A	✓	✗	✗	✗	1.26	0.37	1.70	0.86	54.8
B	✓	✓	✗	✗	1.37	0.41	1.84	0.89	57.9
C	✓	✓	✓	✗	1.50	0.52	2.03	0.92	62.4
D	✓	✓	✓	✓	1.51	0.53	2.18	0.94	63.3

From Variant-B to Variant-C, the gain in detection mAP is more pronounced than that in fusion metrics, suggesting that the main benefit at this stage comes from more effective cross-branch interaction rather than further improvement of the fused image alone. Specifically, by injecting multi-scale fused features into the detection branch, the detector can directly exploit multimodal intermediate representations instead of relying solely on the final fused image. At the same time, the detection loss can reversely shape the fused features through the interaction path, making the fusion branch more detection-oriented.

From Variant-C to Variant-D, introducing CSMDE further improves both fusion and detection performance. Although the gain is smaller than in the previous two stages, it is more targeted, as CSMDE strengthens category consistency in target-region representation. Without CSMDE, the target discriminator mainly enforces generic SAR saliency. With CSMDE, the target region is further guided toward category-consistent SAR-discriminative expression, resulting in additional gains for both tasks.

Overall, these structural ablations show that the benefits of the proposed method arise from the full collaborative framework rather than from any single module. The dual discriminators mainly improve region-level functional decoupling, multi-scale feature interaction strengthens the detector’s use of multimodal intermediate representations, and CSMDE further refines category-consistent target modeling. Their contributions are progressive and complementary.

4.4.2. Ablation on the Dual-Discriminator Mechanism

To further validate the proposed dual-discriminator design, we conduct a controlled ablation study in which all variants share the same fusion generator, YOLOv8-OBB detection branch, multi-scale feature injection, reconstruction losses, and training schedule, while only the discriminator configuration is changed. The detailed settings are summarized in Table 7. In addition, DD-6 is introduced as a shuffled-semantic control to verify whether the gain of semantic conditioning comes from correct instance-level semantics rather than from extra capacity.

Table 7. Structural composition of ablation variants.

Variant	Target Discriminator	Semantic Condition	Background Discriminator
DD-0	✗	✗	✗
DD-1	Unconditional	✗	✗
DD-2	CSMDE-conditioned	CSMDE	✗
DD-3	✗	✗	✓
DD-4	Unconditional	✗	✓
DD-5	CSMDE-conditioned	CSMDE	✓
DD-6	CSMDE-conditioned	Shuffled CSMDE	✓

The results in Table 8 show that the two discriminators play complementary roles. The target discriminator improves target-region discriminability and detection-oriented fusion, as indicated by the gains from DD-0 to DD-1 and from DD-4 to DD-5. The background discriminator mainly suppresses non-target interference and improves background fidelity, as reflected by the reduction in FP/image and the increase in VIF from DD-0 to DD-3 and from DD-1 to DD-4. Moreover, the CSMDE-conditioned target discriminator consistently outperforms the unconditional version, as shown by the improvements from DD-1 to DD-2 and from DD-4 to DD-5. The performance drop observed in DD-6 further confirms that the gain depends on correct semantic conditioning rather than on parameter count or auxiliary inputs.

Table 8. Ablation study of the dual-discriminator mechanism.

Variant	SCD	VIF	Qabf	AP75	mAP	FP/Image
DD-0	1.44	0.47	0.56	69.4	60.5	1.82
DD-1	1.47	0.49	0.59	70.3	61.4	1.58
DD-2	1.49	0.50	0.62	71.0	62.1	1.51
DD-3	1.46	0.51	0.58	69.9	61.0	1.34
DD-4	1.50	0.52	0.63	71.3	62.4	1.18
DD-5	1.51	0.53	0.65	72.1	63.3	1.05
DD-6	1.50	0.51	0.62	71.1	62.3	1.19

The gradient-domain maps in Figure 9 are generated by visualizing the response patterns of the fused images under different discriminator settings; they reflect how each configuration reshapes the fused representation in terms of target emphasis and background suppression. As shown in the figure, DD-0 exhibits more dispersed responses, DD-1/DD-2 increase target focus, DD-3 suppresses background clutter, and DD-5 achieves the most balanced response pattern. The shuffled-semantic control DD-6 shows weaker compactness than DD-5, consistent with the quantitative degradation. Overall, the visualization confirms that the target discriminator, background discriminator, and semantic conditioning contribute in complementary ways.

Figure 9. Gradient-domain visualization of the dual-discriminator ablation study.

4.4.3. Ablation on the Distribution Consistency Loss

The distribution consistency loss ${\mathcal{L}}_{\mathrm{d}\mathrm{i}\mathrm{s}\mathrm{t}}$ is introduced to ensure that the disentangled embeddings in CSMDE remain aligned with the basic semantic structure of the original category distribution. To verify its effectiveness, we remove this loss from the full model and compare the resulting fusion and detection performance. In addition, we report the average KL divergences of the category-discriminative semantic branch and the modality-preference branch, denoted by $D_{KL}^{sem}$ and $D_{KL}^{mod}$, respectively. These values are computed from the probability distributions of test instances in the corresponding branches and are used to measure the regularity of the semantic spaces. We further visualize the instance-level semantic embeddings on the test set using t-SNE. Figure 10 shows the visualization results.

Figure 10. t-SNE visualization of CSMDE semantic embeddings: (a,b) category-discriminative semantic branch; (c,d) modality-preference branch.

Table 9 shows that removing ${\mathcal{L}}_{\mathrm{d}\mathrm{i}\mathrm{s}\mathrm{t}}$ degrades both fusion and detection performance. At the same time, the KL divergence of both branches increases, indicating that the semantic spaces become less regular. This trend is also evident in the t-SNE visualization: without ${\mathcal{L}}_{\mathrm{d}\mathrm{i}\mathrm{s}\mathrm{t}}$, different categories show more overlap and larger intra-class dispersion in both the category-discriminative semantic space and the modality-preference space; after introducing ${\mathcal{L}}_{\mathrm{d}\mathrm{i}\mathrm{s}\mathrm{t}}$, the clusters become more compact and their boundaries more separable, especially for intermediate categories that are more easily affected by surrounding categories.

Table 9. Comparison of quantitative metrics for ${\mathcal{L}}_{\mathrm{d}\mathrm{i}\mathrm{s}\mathrm{t}}$ ablation study.

Setting	SCD	VIF	SSIM	mAP	${\mathit{D}}_{\mathit{K}\mathit{L}}^{\mathit{s}\mathit{e}\mathit{m}}$	${\mathit{D}}_{\mathit{K}\mathit{L}}^{\mathit{m}\mathit{o}\mathit{d}}$
w/o ${\mathcal{L}}_{dist}$	1.47	0.51	0.93	62.4	0.173	0.241
w ${\mathcal{L}}_{dist}$	1.51	0.53	0.94	63.3	0.118	0.164

Notably, $D_{KL}^{mod}$ remains consistently larger than $D_{KL}^{sem}$, which is consistent with the design of CSMDE. The category-discriminative semantic branch mainly captures shared category-discriminative information and is therefore easier to regularize into a compact distribution. By contrast, the modality-preference branch must describe the cross-modal discrepancy between SAR and optical imagery, making its distribution intrinsically more complex and harder to align. After introducing ${\mathcal{L}}_{dist}$, both branches become better regularized, while the modality-preference branch still preserves greater distributional flexibility. This indicates that ${\mathcal{L}}_{dist}$ does not simply compress the feature space, but instead reduces unnecessary drift while preserving meaningful modality variation.

These results show that ${\mathcal{L}}_{dist}$ improves not only the numerical performance, but also the stability and interpretability of the semantic embeddings. By regularizing the two semantic branches, it enhances the reliability of the instance-level semantic condition used by the target discriminator, which in turn improves both fusion quality and detection accuracy.

4.4.4. CSMDE Ablation and Hyperparameter Analysis

To further examine the role of the proposed CSMDE, we analyze both its structural contribution and its hyperparameter behavior in this subsection. The objective is to clarify whether the performance gain arises from a genuinely improved semantic-conditioned mechanism rather than from a simple increase in model complexity, and to explain the rationale of the adopted dual-temperature configuration and prototype dimensions.

We first compare the full CSMDE with two simpler semantic-conditioned baselines, namely a plain learnable category embedding and a single-temperature variant. Here, the learnable embedding baseline directly assigns one trainable embedding vector to each category, while the single-temperature variant retains the same semantic-conditioned branch but replaces the dual-temperature formulation with a shared temperature. All three settings share the same backbone, detector, dual-discriminator setting, training schedule, and data split, so that the influence of semantic-conditioned design can be isolated. Figure 11 shows the ROI-level similarity between local features and category-semantic prototypes. The plain learnable embedding produces relatively diffuse responses and stronger leakage to surrounding structural clutter. The single-temperature variant improves spatial concentration to some extent, but the activation remains relatively broad in complex scenes. By contrast, the full CSMDE yields more target-consistent responses while still preserving weak background residuals, which is more consistent with realistic semantic prototype behavior in SAR–optical fusion scenarios.

Figure 11. ROI-level similarity between local features and category-semantic prototypes under different semantic-conditioned designs. From left to right: contextual ROI patches, Learnable Embedding, Single-Temperature, and Full CSMDE.

The effect of CSMDE can also be observed in the embedding space. Figure 12 presents the pairwise cosine-similarity matrices of ROI semantic embeddings learned by the three variants. The matrices are arranged by category, so that brighter diagonal blocks indicate higher intra-class compactness and darker off-diagonal regions indicate lower inter-class confusion. Compared with the learnable embedding baseline, the single-temperature setting already yields a more organized embedding structure. The full CSMDE further strengthens this tendency, producing the clearest diagonal blocks and the weakest off-diagonal interference. This result indicates that the proposed design improves not only the final task performance, but also the category consistency and separability of the learned ROI-level semantic representations.

Figure 12. Pairwise cosine-similarity matrices of ROI semantic embeddings learned by different semantic-conditioned variants. Brighter diagonal blocks indicate higher intra-class compactness, while darker off-diagonal regions indicate lower inter-class confusion.

To assess whether the dual-temperature design introduces additional overfitting risk, we further compare the training and validation performance of the three semantic-conditioned variants. The representative train–validation gap statistics are reported in Table 10. Although the full CSMDE slightly improves training accuracy, its validation gain is more pronounced, and the train–validation gap does not increase. Instead, the gap becomes slightly smaller than that of the simpler baselines. This trend suggests that the richer semantic-conditioned mechanism does not simply fit the training set more aggressively, but improves the generalization of category-consistent target modeling. In other words, the added complexity of CSMDE does not manifest as evident overfitting in our setting.

Table 10. Train–validation gap comparison of different semantic-conditioned variants.

Method	Train mAP50	Val mAP50	Gap
Learnable Embedding	83.8	78.9	4.9
Single-Temperature	84.1	79.4	4.7
Full CSMDE	84.6	80.3	4.3

The temperature parameters are then analyzed through the original grid-search experiments reported in Figure 13. This figure serves as the parameter-selection basis of the proposed dual-temperature design. In our formulation, ${\tau }_s$ controls the sharpness of the semantic distribution, while ${\tau }_m$ controls the smoothness of the modality-preference distribution. The results in Figure 13 show that these two temperatures play complementary roles rather than duplicating the same operation. When ${\tau }_s$ is too large, the semantic distribution becomes overly smooth and the dominant category information is weakened. When it is too small, the distribution becomes excessively sharp and more sensitive to local noise or imperfect early-stage predictions. Similarly, an excessively small ${\tau }_m$ makes the modality-preference branch overly confident, whereas an excessively large ${\tau }_m$ weakens useful discriminative cues. The grid-search results consistently indicate that $({\tau }_s,{\tau }_m)=(0.5,1.5)$ provides the best trade-off and is therefore adopted in the final model.

Figure 13. Ablation of temperature parameters in CSMDE. (a) mAP under different $({\tau }_s,{\tau }_m)$ settings. (b) SCD under different $({\tau }_s,{\tau }_m)$ settings.

In addition to the temperature study, we further analyze the prototype dimensions of the category-discriminative semantic and modality-preference branches. The corresponding quantitative comparison is reported in Table 11 and Figure 14, where $d_s=d_m=64$ denote the dimensions of the category-discriminative semantic prototype space and the modality-preference prototype space, respectively. The prototype dimension controls the representation capacity of the embedding branch: if the dimension is too small, the semantic space may be insufficient to preserve meaningful class structure, whereas an excessively large dimension may introduce redundancy and weaken generalization. The results in Table 11 and Figure 14 show that moderate prototype dimensions achieve the best balance between representational sufficiency and compactness. This observation is consistent with the role of CSMDE as a structured semantic-conditioned module: the gain comes from better semantic organization rather than from blindly increasing feature dimensionality.

Table 11. Comparison of quantitative metrics for prototype dimension ablation study.

(${\mathit{d}}_{\mathit{s}}\mathbf{,}{\mathit{d}}_{\mathit{m}}$)	SCD	mAP	Params (M)	FLOPs (G)
(32, 32)	1.47	62.5	0.38	0.82
(64, 32)	1.49	62.9	0.42	0.87
(64, 64)	1.51	63.3	0.47	0.93
(128, 64)	1.50	63.1	0.56	1.02

Figure 14. Prototype dimension ablation.

4.5. Computational Efficiency Analysis

To evaluate the practical cost of PCFD-Net, we compare the parameters, FLOPs, and inference time of representative fusion and fusion-detection methods. During inference, the target and background discriminators are removed; only the fusion branch, residual feature-injection path, detection branch, and lightweight CSMDE mapping are retained.

As shown in Table 12, PCFD-Net Fusion has more parameters than lightweight fusion networks because it uses two modality-specific ResNet-50 encoders to separately model SAR scattering responses and optical texture structures. However, its FLOPs remain moderate. The fusion branch requires 18.51 G FLOPs, which is close to TarDAL and much lower than MetaFusion, CoCoNet, and Diff-IF. Its inference time is 9.5 ms, indicating that the proposed fusion branch increases representation capacity without introducing prohibitive computational cost.

Table 12. Forward-inference complexity comparison of representative fusion methods.

Method	Params (M)	FLOPs (G)	Time (ms)
TarDAL	0.29	19.24	2.8
MMIF-EMMA	1.52	8.94	6.5
MetaFusion	2.25	148.71	8.3
CoCoNet	9.13	1846.98	180
SeAFusion	0.17	11.12	2.3
Diff-IF	23.73	696.60	784
PCFD-Net Fusion	55.05	18.51	9.5

Table 13 reports the end-to-end inference cost. PCFD-Net is not the lightest model, mainly due to the dual-encoder fusion branch and the unified fusion-detection pipeline. Nevertheless, its total FLOPs are 21.6 G and the inference time is 17.5 ms per image, which remains practical for near-real-time remote sensing interpretation. Unlike methods that only perform fusion or detection, PCFD-Net simultaneously outputs the fused image and oriented detection results, while enabling bidirectional task interaction during training. Therefore, the additional cost is exchanged for stronger target preservation, better cross-modal representation, and improved detection performance. Overall, PCFD-Net provides a reasonable accuracy-efficiency trade-off rather than pursuing the minimum computational complexity.

Table 13. Forward-inference complexity comparison of detection and end-to-end fusion-detection methods.

Method	Params (M)	FLOPs (G)	Time (ms)
COMO	22.6	6.78	12.5
TarDAL	10.30	23.73	6.9
MHFNet	16.1	7.27	7.6
CFT	53.8	7.95	12.4
MetaFusion	12.25	151.46	12.0
CLANet	48.2	13.35	9.5
CSSA	13.5	5.47	5.3
CMADet	41.5	7.90	13.9
ICAFusion	29.0	6.68	8.2
MMIDet	53.8	7.95	12.7
E2E-OSDet	24.7	10.18	7.5
PCFD-Net	68.7	21.6	17.5

5. Discussion

By integrating a fusion branch, a YOLOv8-based detection branch, CSMDE, and dual discriminators within a parallel collaborative framework, PCFD-Net achieves improved performance in both SAR–optical image fusion and downstream object detection. The results indicate that explicit bidirectional interaction is effective for exploiting cross-modal complementarity and producing fusion representations that are more favorable for detection.

Nevertheless, the proposed framework still has several limitations. First, the parallel collaborative training strategy increases computational cost and may introduce optimization instability, especially during the joint training stage. Second, its performance depends on the registration quality of SAR–optical image pairs, and severe cross-modal misalignment may weaken both feature interaction and region-level supervision. In practical remote sensing applications, scene sizes are often much larger than the 256 × 256 image pairs used for training. In such cases, the framework can be deployed in a sliding-window or overlapping tiling manner, followed by result merging with rotated non-maximum suppression. Since both discriminators are removed during inference, the deployed model consists only of the fusion generator and the detection branch, making patch-wise deployment feasible for large-scale scenes.

The current benchmark also involves targets of different scales and scene types, including small ships, oil tanks, and narrow bridges in cluttered backgrounds. The results suggest that the proposed framework is not limited to only large objects, but its effectiveness may still vary with target scale, background complexity, and modality alignment. In particular, sea-surface ship detection benefits more directly from SAR saliency and background suppression, whereas bridge and airport detection relies more heavily on optical structural continuity. More challenging scenarios, such as severe misalignment, extremely dense clutter, or broader multimodal tasks, may require stronger alignment strategies, adaptive tiling, or more robust collaborative optimization mechanisms. These directions will be explored in future work.

6. Conclusions

This paper addresses the long-standing separation and weak collaboration between fusion and detection in SAR–optical multimodal perception. We propose PCFD-Net, a parallel collaborative optimization framework that unifies cross-modal image fusion and object detection through explicit bidirectional interaction between the fusion and detection branches. By incorporating the Category Semantic–Modality Disentangled Embedding (CSMDE) module and the dual-discriminator design, the proposed method preserves stronger SAR-discriminative cues in target regions while maintaining optical structural fidelity in background regions. Extensive experiments demonstrate that the proposed framework consistently improves both fusion quality and downstream detection performance, confirming the practical value of parallel collaborative optimization for SAR–optical multimodal perception.

More broadly, this work suggests that treating low-level fusion generation and high-level target detection as peer tasks within a unified collaborative framework is an effective way to improve multimodal remote sensing perception. Future work will extend this framework to more challenging scenarios, such as change detection, oriented object detection, and multimodal semantic segmentation, while further exploring lighter and more robust collaboration mechanisms for efficient deployment in practical remote sensing applications.