Local Information-Driven Hierarchical Fusion of SAR and Visible Images via Refined Modal Salient Features

Abstract

Compared to other multi-source image fusion tasks, visible and SAR image fusion faces a lack of training data in deep learning-based methods. Introducing structural priors to design fusion networks is a viable solution. We incorporated the feature hierarchy concept from computer vision, dividing deep features into low-, mid-, and high-level tiers. Based on the complementary modal characteristics of SAR and visible, we designed a fusion architecture that fully analyze and utilize the difference of hierarchical features. Specifically, our framework has two stages. In the cross-modal enhancement stage, a CycleGAN generator-based method for cross-modal interaction and input data enhancement is employed to generate pseudo-modal images. In the fusion stage, we have three innovations: (1) We designed feature extraction branches and fusion strategies differently for each level based on the features of different levels and the complementary modal features of SAR and visible to fully utilize cross-modal complementary features. (2) We proposed the Layered Strictly Nested Framework (LSNF), which emphasizes hierarchical differences and uses hierarchical characteristics, to reduce feature redundancy. (3) Based on visual saliency theory, we proposed a Gradient-weighted Pixel Loss (GWPL), which dynamically assigns higher weights to regions with significant gradient magnitudes, emphasizing high-frequency detail preservation during fusion. Experiments on the YYX-OPT-SAR and WHU-OPT-SAR datasets show that our method outperforms 11 state-of-the-art methods. Ablation studies confirm each component’s contribution. This framework effectively meets remote sensing applications’ high-precision image fusion needs.

1. Introduction

Due to the inherent theoretical and technical limitations of hardware devices, images captured by a single sensor or under a single shooting condition often fail to effectively and comprehensively represent the imaging scene [1]. Image fusion naturally emerges as a solution by integrating meaningful information from different source images to produce a single image that contains richer information, thereby enhancing its utility for subsequent applications [2]. In the realm of remote sensing, image fusion tasks encompass a variety of modalities, each with unique characteristics and applications. Among these, the fusion of visible and synthetic aperture radar (SAR) images is of significant importance and has garnered considerable attention in recent years.

SAR actively emits pulsed electromagnetic waves and receives reflected signals to acquire surface information about the targets [3]. Unlike visible imaging technologies, SAR can penetrate clouds, rain, snow, and fog, providing all-weather and all-day imaging capabilities that are not affected by weather and lighting conditions. Although SAR is rich in spatial information, it lacks the spectral information that visible imagery provides, which is critical for numerous remote sensing applications [4]. Furthermore, SAR images often suffer from poor interpretability and are affected by speckle noise [5]. Visible images depend on sunlight reflected from Earth’s objects. Although two differently structured objects may appear identical in visible imagery because of their similar spectral responses, these differences can be distinguished in SAR imagery. Therefore, SAR and visible image fusion is advantageous for generating images with enriched spatial and spectral information. The analysis of fused images enhances our understanding and interpretation of the imaged area, serving a wide range of applications [4].

Early image fusion methods relied on mathematical transformations for manual analysis and fusion rule design in spatial or transform domains, also known as traditional fusion methods. These include multi-scale transform-based [6], sparse representation-based [7], subspace-based [8], saliency-based [9], and total variation-based methods [10]. However, these methods have limitations. They often enforce uniform transformations across different source images, neglecting inherent differences and leading to poorly expressive features. Additionally, traditional fusion strategies are too coarse for deep feature fusion, limiting performance. The rapid development of deep learning has shown superiority in various fields. Its powerful feature learning and generalization capabilities make it suitable for remote sensing image fusion tasks [11]. Common methods include convolutional neural networks (CNNs) [12], auto-encoder (AE) networks [13], generative adversarial networks (GANs) [14], and transformer-based approaches [15]. These techniques are crucial for autonomous feature learning and the efficient processing of large-scale remote sensing data.

The optimal fusion of complementary features constitutes a fundamental paradigm in image fusion research, where systematic investigation of feature decomposition methodologies has profoundly advanced theoretical understanding and algorithmic development in this domain. The seminal work by Vese and Osher [16] pioneered the application of finite difference techniques for the decomposition of images into cartoon and texture components, establishing the foundation for exploiting modality-specific characteristics. In the context of visible and SAR image fusion, Ye et al. [17] developed the VSFF framework to extract complementary features from optical and SAR modalities. However, conventional approaches rely on hand-crafted transformation heuristics, consequently limiting their adaptability to complex inter-modal discrepancies. In deep learning, Ye et al. [18] proposed SOSTF by incorporating structural–textural decomposition with customized loss functions. Its unsupervised learning approach, while advantageous in avoiding manual labeling, still relies on predefined structural and textural components, which hinders its ability to fully capture the complex interactions between SAR and visible images. Studies in other multi-modal fusion fields [19,20,21] have also delved into the hierarchical decomposition of complementary features. However, limitations still exist. On the one hand, there is a lack of a detailed refinement of the decomposed features at each hierarchical level and differentiated processing. In addition, complex decomposition methods and the insufficient use of hierarchical characteristics continue to present challenges, such as a loss of detail and the blurring of edges. It should be noted that the development of deep learning methodologies specifically for SAR-visible fusion remains relatively underexplored. To our knowledge, beyond the aforementioned works [17,18], Kong et al. [22] developed a semi-supervised approach that employs dense UGAN with the Gram–Schmidt transformation, which nevertheless requires annotated data. A major bottleneck that hinders progress in this field is the paucity of SAR-visible paired datasets, where both the scarcity and suboptimal quality of existing public repositories present significant obstacles. Although contemporary fusion algorithms focus predominantly on model architecture design, the quality and quantity of raw input images constitute critical determinants that significantly influence the effectiveness of deep learning methodologies [23]. Addressing these data-related challenges is imperative for advancing SAR-visible image fusion.

To address these technical challenges and compensate for the lack of detailed analysis and full utilization of hierarchical feature differences in existing studies, this paper proposes a method for the local information-driven hierarchical fusion of SAR and visible images via refined modal salient features. The methodology comprises two orthogonal processing phases: cross-modal enhancement and modal fusion. In the cross-modal enhancement phase, to mitigate the effects of data quality limitations on fusion network training, we employ CycleGAN generators [24] to produce pseudo-modal images. This approach establishes cross-modal correlations through CycleGAN’s cycle-consistency constraints, effectively enabling cross-modal interaction via bidirectional generation. Subsequently, the generated pseudo-modal images are concatenated with the source images along the dimension o of the channel to achieve cross-modal enhancement. In the modal fusion phase, to fully exploit and utilize complementary modality information, we classify features into low-level, middle-level, and high-level groups [25] according to their degree of abstraction, based on computer vision principles [26]. In particular, the texture features of visible images and the structural contour features of SAR images are defined as low-level and middle-level features, respectively. In contrast to other fusion networks [20,21] based on complementary feature decomposition, our methodology explicitly distinguishes characteristic properties across different feature hierarchies and employs a targeted topological design to construct differentiated processing branches for feature extraction and fusion, along with a strict hierarchical reconstruction mechanism. Specifically, feature extraction addresses SAR’s high dynamic range characteristics. A modified ResNet-50 architecture serves as the backbone network with hierarchically partitioned subnetworks optimized for distinct feature levels. For feature fusion, we implement saliency-aware selective fusion for low-/middle-level features while maintaining maximum integrity preservation of high-level semantic-carrying features. For image reconstruction, we proposed a Layered Strictly Nested Framework (LSNF) that systematically preserves and propagates refined hierarchical depth features by establishing strict intra-layer and inter-layer nested connections. To address detail degradation and edge blurring in the fusion process, we devise a Gradient-Weighted Pixel Loss (GWPL) that incorporates human visual system sensitivity priors. Unlike conventional pixel-wise losses, such as the mean squared error (MSE), which treat all regions equally, the GWPL prioritizes high-frequency components during the fusion process, which aligns with the cognitive principle that humans perceive scenes through structural contrasts. In summary, the main contributions of this paper are as follows.

• To mitigate the effects of data quality limitations on the training of fusion networks, we propose a two-stage hierarchical fusion paradigm for SAR-visible images. The cross-modal enhancement stage employs CycleGAN generators to synthesize pseudo-modal images, effectively facilitating cross-modal interaction between SAR-visible images and enhancing cross-modal consistency. In the fusion stage, departing from conventional decomposition-based methods, we introduce computer vision-guided hierarchical decomposition to explicitly refine modality salient features and implement strict layer-wise processing of complementary characteristics.
• To fully exploit and utilize complementary modality information while jointly considering the characteristic properties of hierarchical features and the high dynamic range specific to SAR images, we construct a topology where differentiated feature extraction and fusion branches target distinct hierarchical features and devise an LSNF. This hierarchical differentiation mechanism facilitates improved learning of complementary cross-modal features between visible and SAR modalities.
• To address detail degradation and edge blurring in the fusion process, we formulate a GWPL that guides pixel-level optimization through feature saliency spatial variations, thereby bridging global structural constraints with local detail preservation and enhancing detail representation in fused results.

Experimental results on two datasets of different scales demonstrate that our method achieves superior image fusion performance compared to existing deep learning-based and traditional methods. This is evident in both human visual perception and objective metrics, with particularly outstanding performance in metrics related to image detail quality.

2. Materials and Methods

In this section, we detail the framework and workflow of the method. Our basic idea for fusion is to first convert the visible images from the RGB space to the YCbCr space, extract the Y channel images, and perform grayscale fusion with the SAR images, then combine it with the Cb and Cr channel images to reconstruct the fused color images. For simplicity, the visible images discussed below refer to visible grayscale images. The proposed framework (Figure 1) employs a generative adversarial network with dual discriminators. The generator’s workflow comprises four sequential processing stages: First, source SAR and visible images are processed by CycleGAN generators (Figure 2) to produce pseudo-modality images, thereby establishing bidirectional interactions between the SAR and visible domains. These generated images are then channel-concatenated with the original inputs for cross-modal enhancement. Subsequently, as illustrated in Figure 3, the enhanced representations undergo multi-layer deep feature extraction. The extracted features then progress through hierarchical fusion operations. Finally, multi-level fused features are reconstructed into the output image via an LSNF.

2.1. CycleGAN Generators for Cross-Modal Enhancement

To address the issue of data support required for training networks, we input the real SAR images $I_{SAR\_Real}$ and the real visible images $I_{Vis\_Real}$ into the two generators, $G_{S\to V}$ and $G_{V\to S}$, of the CycleGAN network as shown in Figure 2:

$$I_{SAR\_Fake}=G_{V\to S}\left(I_{Vis\_Real}\right)$$

(1)

$$I_{Vis\_Fake}=G_{S\to V}\left(I_{SAR\_Real}\right)$$

(2)

The pseudo-SAR images $I_{SAR\_Fake}$ and the pseudo-visible images $I_{Vis\_Fake}$ are utilized to enhance the quality of the data by supplementing additional information with the real images. These pseudo-images are generated to provide complementary features that may not be present in the real images, thereby enriching the dataset. Ultimately, they serve as input to the multi-layer deep feature extraction module of the fusion network. The integration of real and pseudo-images is performed through a convolutional operation, which is defined as follows:

$$I_{SAR}=\mathrm{Conv}\left(\mathrm{Concat}(I_{SAR\_Real},I_{SAR\_Fake})\right)$$

(3)

$$I_{Vis}=\mathrm{Conv}\left(\mathrm{Concat}(I_{Vis\_Real},I_{Vis\_Fake})\right)$$

(4)

Here, Conv represents a convolutional layer that processes the concatenated real and pseudo-images, extracting meaningful features for subsequent analysis. This approach ensures that the fusion network receives a comprehensive set of features, which improves its ability to perform accurate and robust image fusion.

2.2. Multi-Layer Deep Feature Extraction

Given the lack of research on the fusion of different depth characteristics at the bottom, middle, and high levels in existing methods, this module considers using the Resnet-50 [27] architecture for its excellent performance in obtaining depth features. The same modified Resnet-50 architecture is used as the backbone, with two branches to separately extract multi-layer depth features of SAR and visible images. Since SAR images typically have a higher dynamic range [28], instead of normalizing to 0 to 1, we normalize the pixels to −1 to 1. This better captures the negative features in the input data. We modify the activation function from Relu [29] to Leaky_Relu [30] in the original Resnet-50 architecture, and the activation functions used in the modules are also designed or modified to Leaky_Relu, as shown in Figure 3a. First, the input SAR and visible images are used to extract multi-depth features through the backbone network:

$$Out{1}_S,Out{2}_S,Out{3}_S,Out{4}_S,Out{5}_S=F_S\left(I_{SAR}\right)$$

(5)

$$Out{1}_V,Out{2}_V,Out{3}_V,Out{4}_V,Out{5}_V=F_V\left(I_{VIS}\right)$$

(6)

In this study, the modules $F_S\left(\right)$ and $F_V\left(\right)$ are designated as feature extraction components for the SAR and visible branches, respectively. $Out1$, $Out2$, $Out3$, $Out4$, and $Out5$ denote depth features extracted from the backbone, ranging from low to high levels. The subscript indicates whether the output is from the SAR or the visible branch. Specifically, the distinction between low-, middle-, and high-level features is not clearly defined and is usually based on their abstraction level. As shown in Figure 4, we perform a visual analysis of the outputs $Out1$, $Out2$, $Out3$, $Out4$, and $Out5$, categorizing them into distinct feature levels based on complexity and abstraction. $Out1$ signifies low-level features, capturing fundamental patterns and edges crucial for initial image comprehension, with a dimensionality of 32. $Out2$ and $Out3$ represent middle-level features, which extend to low-level features by capturing more intricate patterns and textures. Their dimensionalities, 128 and 256, respectively, offer a richer depiction of image content, facilitating enhanced differentiation of structures. $Out4$ and $Out5$ are identified as high-level features, encapsulating the most abstract and complex representations, capturing semantic information and intricate details. With dimensionalities of 512 and 1024, they demonstrate the ability to represent detailed and nuanced aspects of the images. This hierarchical division from low- to high-level features ensures a comprehensive analysis that effectively captures both fundamental and complex information in the feature extraction process.

Subsequently, acknowledging that low-level features in images of the same scene across different modalities are generally more consistent and stable [31], we integrated the DenseASPP module [32] to enhance these low-level features. This module is specifically designed to densify feature maps, thereby capturing richer and more intricate details within the images. This enhancement not only improves the granularity of the feature representation but also significantly bolsters the robustness and accuracy of the fusion results by providing a more stable foundation for subsequent processing stages.

In the final stage, the densification of low-level features using the DenseASPP module, along with the seamless transmission of middle- and high-level features, culminates in the output of the multi-depth feature extraction module. The equations below describe the process:

$$Lo{w}_S=\mathrm{DenseASPP}\left(Out{1}_S\right)$$

(7)

$$Lo{w}_V=\mathrm{DenseASPP}\left(Out{1}_V\right)$$

(8)

For both SAR and visible modalities, the outputs Out2 and Out3 capture perceptible structural details with progressively enriched feature combinations through channel-wise expansion, forming complex pattern representations categorized as mid-level features (designated $Middle2$ and $Middle1$, respectively). In addition, the outputs Out4 and Out5 encode abstract global contextual information beyond direct perceptual details, constituting high-level features (identified as $High2$ and $High1$).

In this context, $Lo{w}_S$, $Middle{1}_S$, $Middle{2}_S$, $High{1}_S$, and $High{2}_S$ denote the feature hierarchy for the SAR image. Similarly, $Lo{w}_V$, $Middle{1}_V$, $Middle{2}_V$, $High{1}_V$, and $High{2}_V$ represent the corresponding features of the visible image. This structured approach ensures that all feature levels are used effectively, maintains the integrity and richness of the data throughout the extraction process, and ultimately contributes to more precise and reliable image fusion.

2.3. Deep Feature Hierarchical Fusion

As depicted in Figure 3b, we integrated the lightweight attention module CBAM [33] into the fusion component to effectively merge low- and middle-level features. This module enhances feature representation by selectively highlighting advantageous modal features, thereby improving the overall quality of the fusion. For the integration of high-level features, an adder mechanism is employed to ensure the retention of all feature information, which is crucial for guiding the reconstruction of the fused image while preserving semantic integrity. To illustrate the fusion process, we present the merging of $Lo{w}_S$ and $Lo{w}_V$ as a representative example, demonstrating the methodology applied to the fusion of features at the low level and the middle level.

The channel attention $M_C\left(F\right)$ and spatial attention $M_S\left(F\right)$ mechanisms are defined as follows:

$$M_C\left(F\right)=\sigma (\mathrm{MLP}\left(\mathrm{AvgPool}\left(F\right)\right)+\mathrm{MLP}\left(\mathrm{MaxPool}\left(F\right)\right))$$

(9)

$$M_S\left(F\right)=\sigma \left(f^{7\times 7}\left([\mathrm{AvgPool}\left(F\right);\mathrm{MaxPool}\left(F\right)]\right)\right)$$

(10)

The fusion of low-level features is achieved through a two-step process. Initially, the low-level features of the SAR and visible modalities, denoted as $Lo{w}_S$ and $Lo{w}_V$, are concatenated and processed through a convolutional layer. This operation is defined as

$$Lo{w}_x=\mathrm{Conv}\left(\mathrm{Concat}(Lo{w}_S,Lo{w}_V)\right)$$

(11)

Here, Conv represents a convolutional layer that extracts integrated features from the concatenated input, enhancing the representation by capturing interactions between the modalities. Subsequently, the fused features are enhanced through spatial and channel attention mechanisms, respectively, denoted as $M_S$ and $M_C$. This enhancement is expressed as

$$Lo{w}_F=M_S(M_C\left(Lo{w}_x\right)\times Lo{w}_x)\times (M_C\left(Lo{w}_x\right)\times Lo{w}_x)$$

(12)

This approach ensures that the essential characteristics of each modality are preserved and effectively combined, resulting in robust and comprehensive feature fusion. Using attention mechanisms, the fusion process selectively emphasizes important features, thereby enhancing the quality and robustness of the final output.

2.4. Layered Strictly Nested Framework (LSNF)

Once the deeply fused features have been obtained at various levels, the focus shifts to how to effectively apply these features in a layered manner. Drawing inspiration from previous works [34,35], in which nested connection-based decoders were found to be effective in preserving feature information, we further advanced this concept. We proposed the Layered Strictly Nested Framework (LSNF) (illustrated in part (c) of Figure 3). This framework purposefully nests features within the same level in the topological structure according to their hierarchical differences, instead of nesting all features without discrimination. The design of the LSNF better highlights hierarchical differences and leverages hierarchical characteristics to reduce feature redundancy, and this framework significantly reduces the demand for computational resources. Specifically, we denote k as the characteristic level, where $k\in$ {1—high-level, 2—middle-level, and 3—low-level}. We use $m_k\in \{1,2,3,\dots ,M_k\}$ to represent channel levels from high to low within the kth feature level. In addition, we use $n\in \{1,2,3\dots ,N_k\}$ to distinguish the outputs $x^{(k,m_k,n)}$ of different convolutional groups. Here, $M_k$ is determined by the network design, and $N_k$ follows the conditional relationship (13), where the dependency between $N_k$ and $M_k$ explicitly reflects the hierarchical nest strategy of the characteristics.

$$N_k=\left\{\begin{array}{cc}{M}_k\hfill & \mathrm{if}k=1\hfill \\ M_k+1\hfill & \mathrm{if}k=2,3\hfill \end{array}\right.$$

(13)

The hierarchical feature nesting mechanism is illustrated in Figure 5. The outputs of the convolutional groups are defined with respect to (14) and (15):

$$\left\{\begin{array}{c}{x}^{(1,m_1,n)}={\mathrm{High}}_F^{m_1}(n=1)\hfill \\ x^{(1,m_1,n)}=Conv\left[{\left.x^{(1,m_1,i)}\right|}_{i=1}^{n-1},\mathrm{up}\left(x^{(1,m_1-1,n-1)}\right)\right](m_1\ge n>1)\hfill \end{array}\right.$$

(14)

$$\left\{\begin{array}{c}{x}^{(k,m_k,n)}=\mathrm{Middle}/{\mathrm{Low}}_F^{m_k}(n=1)\hfill \\ x^{(k,1,2)}=Conv\left[x^{k,1,1},\mathrm{up}\left(x^{(k-1,M_{k-1},N_{k-1})}\right)\right](n=2,m_k=1)\hfill \\ x^{(k,m_k,n)}=Conv\left[{\left.x^{(k,m_k,i)}\right|}_{i=1}^{n-1},\mathrm{up}\left(x^{(k,m_k-1,n-1)}\right)\right](m_k\ge n\ge 2)\hfill \end{array}\right.$$

(15)

Here, ${\mathrm{High}}_F^{m_k}$ and $\mathrm{Middle}/{\mathrm{Low}}_F^{m_k}$ represent the fused features at level $m_k$, derived from the feature extraction and fusion branches. Conv indicates the processing performed by the convolutional group, while $\mathrm{up}$ denotes the upsampling procedure. In our approach, the parameters are set as follows: $M_1=2$, $M_2=2$, and $M_3=1$.

2.5. Discriminator Structure

As shown in Figure 6, the two discriminators share the same architecture. Each consists of seven convolutional layers followed by a linear layer. Without batch normalization, the first convolutional module comprises a convolutional layer and a Leaky_ReLU activation function. The next six modules have a consistent structure: each includes a convolutional layer, a batch normalization layer, and a Leaky_ReLU activation function. All convolutional layers use a kernel size of $3\times 3$ with a stride of 2, facilitating a rapid reduction in the width and height of the feature maps. Batch normalization is implemented using BN. The final linear layer transforms the flattened feature map into an output that quantifies the relative distance between the generated image and the real image. To optimize parameter efficiency, the weights of the second through seventh convolutional layers are shared.

2.6. Loss Function

Here, we focus on the loss function settings during the fusion phase. In the fusion phase, we constructed a generative adversarial network (GAN), which naturally divides the loss into generator loss and discriminator loss.

2.6.1. Loss Function of the Generator

The generator loss function, denoted as $Los{s}_{\mathrm{G}}$, is a composite of two primary components: content loss, $Los{s}_{\mathrm{Content}}$, and adversarial loss, $Los{s}_{\mathrm{GAN}}$. Content loss ensures that the generated images retain the essential features and structure of the input images, thereby preserving semantic integrity during the generation process. Adversarial loss, on the other hand, drives the generator to produce outputs that are indistinguishable from real images, enhancing the realism of the generated content. The formulation of the generator’s loss function is given by

$$Los{s}_{\mathrm{G}}={\lambda }_{1}Los{s}_{\mathrm{Content}}+{\lambda }_{2}Los{s}_{\mathrm{GAN}}$$

(16)

where ${\lambda }_1$ and ${\lambda }_2$ are weighting factors that balance the contributions of the content and adversarial losses, respectively. By appropriately tuning these parameters, the generator can be optimized to achieve a desirable trade-off between content fidelity and visual realism.

2.6.2. Content Loss

In this section, we propose an innovative Gradient-Weighted Pixel Loss (GWPL) to ensure that the generated image retains as much detailed information from both the SAR image and the visible image as possible while preserving the overall structure and pixel integrity within a local scope.

First, for the input image I, the Sobel operator [36] is employed to compute its gradient:

$$\left\{\begin{array}{c}{G}_x\left(I\right)=I\otimes K_x\hfill \\ G_y\left(I\right)=I\otimes K_y\hfill \end{array}\right.$$

(17)

where $K_x$ and $K_y$ are the horizontal and vertical kernels of the Sobel operator, and ⊗ denotes the convolution operation. The gradient magnitude is then calculated as

$$G\left(I\right)=|G_x\left(I\right)|+|G_y\left(I\right)|$$

(18)

For the real SAR image $I_{SAR}$, the real visible image $I_{Vis}$, and the fused image $I_F$, we compute their gradient magnitude matrices $G\left(SAR\right)$, $G\left(Vis\right)$, and $G\left(F\right)$ as (17) and (18). The joint gradient matrix is defined as

$$G_{joint}=max(G\left(Vis\right),G\left(SAR\right))$$

(19)

The gradient loss is defined as the L1 loss [37] between the fused image gradient matrix and the joint gradient matrix:

$$Los{s}_{\mathrm{grad}}={\parallel G\left(F\right)-G_{joint}\parallel }_1$$

(20)

Specifically, we define the weight matrices for SAR and visible images as follows:

$$\left\{\begin{array}{c}{W}_S(i,j)=\left\{\begin{array}{cc}1,\hfill & \mathrm{if}G{\left(SAR\right)}_{(i,j)}=G_{joint(i,j)}\hfill \\ 0,\hfill & \mathrm{otherwise}\hfill \end{array}\right.\hfill \\ W_V(i,j)=\left\{\begin{array}{cc}1,\hfill & \mathrm{if}G{\left(Vis\right)}_{(i,j)}=G_{joint(i,j)}\hfill \\ 0,\hfill & \mathrm{otherwise}\hfill \end{array}\right.\hfill \end{array}\right.$$

(21)

The GWPL is then formulated as

$$\begin{array}{c}\hfill Los{s}_{\mathrm{GWPL}}={\displaystyle \frac{1}{N}}\left(\parallel W_S(i,j)(I_F-I_{SAR}){\parallel }_2^2+{\parallel W_V(i,j)(I_F-I_{Vis})\parallel }_2^2\right)\end{array}$$

(22)

Here, N represents the number of pixels in the input image. The content loss, derived by combining the Structural Similarity Index (SSIM) [38], is formulated as follows:

$$Los{s}_{\mathrm{SSIM}}={\mathrm{w}}_1(1-SSI{M}_{F,\mathrm{SAR}})+{\mathrm{w}}_2(1-SSI{M}_{F,\mathrm{Vis}})$$

(23)

$$Los{s}_{\mathrm{Content}}=\alpha Los{s}_{\mathrm{grad}}+\beta Los{s}_{\mathrm{GWPL}}+\gamma Los{s}_{\mathrm{SSIM}}$$

(24)

In these equations, ${\mathrm{w}}_1$ and ${\mathrm{w}}_2$ are weighting factors that determine the relative importance of the SSIM loss components for the SAR and visible images, respectively. These weights allow for the adjustment of the balance between preserving the structural similarity of the fused image with respect to each modality. By tuning ${\mathrm{w}}_1$ and ${\mathrm{w}}_2$, one can emphasize the preservation of details from SAR or visible images, depending on the specific requirements of the application. The parameters $\alpha$, $\beta$, and $\gamma$ are additional weighting factors that balance the contributions of the gradient loss ($Los{s}_{\mathrm{grad}}$) [39], $Los{s}_{\mathrm{GWPL}}$, and the SSIM loss ($Los{s}_{\mathrm{SSIM}}$) to the overall content loss. This formulation ensures a comprehensive approach to maintaining both the structural integrity and detailed information in the fused image.

2.6.3. Adversarial Loss

Adversarial loss is a key component in facilitating the interactive update of parameters between the generator and the discriminator. The goal is for the discriminator’s evaluation of the generated image to approach a target value of 0.9, rather than 1. This adjustment helps mitigate potential issues such as gradient vanishing, which can occur when the discriminator’s output is too close to the extremes of 0 or 1. In this framework, two discriminators are employed to evaluate the generated images, resulting in two components of adversarial loss for the generator, and they are defined as follows:

$$Los{s}_{GA{N}_{Vis}}={\displaystyle \frac{1}{N}}{\displaystyle \sum _{n=1}^N}{(D_V\left(I_F^{\left(n\right)}\right)-0.9)}^2$$

(25)

$$Los{s}_{GA{N}_{SAR}}={\displaystyle \frac{1}{N}}{\displaystyle \sum _{n=1}^N}{(D_S\left(I_F^{\left(n\right)}\right)-0.9)}^2$$

(26)

In these equations, $n\in N$ denotes the index of the fused image, $I_F^{\left(n\right)}$ represents the n-th fused image, and $D_V\left(I_F^{\left(n\right)}\right)$ and $D_S\left(I_F^{\left(n\right)}\right)$ are the outputs of the visible and SAR discriminators, respectively, when evaluating the generated image.

The overall adversarial loss, which combines the contributions from both discriminators, is expressed as

$$Los{s}_{\mathrm{GAN}}=Los{s}_{GA{N}_{Vis}}+Los{s}_{GA{N}_{SAR}}$$

(27)

This formulation ensures that the generator is effectively trained to produce images that are indistinguishable from real images, thereby enhancing the realism and quality of the generated outputs.

2.6.4. Loss Function of the Discriminator

The loss function $Los{s}_{\mathrm{D}}$ of the discriminator is designed to evaluate the input images, with the objective of classifying the real samples toward 0.9 and the generated samples toward 0.1. This approach still helps maintain a stable training process by avoiding extreme values, which can lead to issues such as gradient vanishing. The loss functions for the two discriminators are defined as follows:

$$\begin{array}{cc}\hfill Los{s}_{D_{Vis}}=& {\displaystyle \frac{1}{N}}{\displaystyle \sum _{n=1}^N}{(D_V\left(I_{Vis}^{\left(n\right)}\right)-0.9)}^2+{\displaystyle \frac{1}{N}}{\displaystyle \sum _{n=1}^N}{(D_V\left(I_F^{\left(n\right)}\right)-0.1)}^2\hfill \end{array}$$

(28)

$$\begin{array}{cc}\hfill Los{s}_{D_{SAR}}=& {\displaystyle \frac{1}{N}}{\displaystyle \sum _{n=1}^N}{(D_S\left(I_{SAR}^{\left(n\right)}\right)-0.9)}^2+{\displaystyle \frac{1}{N}}{\displaystyle \sum _{n=1}^N}{(D_S\left(I_F^{\left(n\right)}\right)-0.1)}^2\hfill \end{array}$$

(29)

In these equations, $n\in N$ represents the index of the image, $I_{Vis}^{\left(n\right)}$ and $I_{SAR}^{\left(n\right)}$ denote the n-th real visible and SAR images, respectively, while $I_F^{\left(n\right)}$ is the n-th generated (fused) image. The terms $D_V\left(I_{Vis}^{\left(n\right)}\right)$ and $D_S\left(I_{SAR}^{\left(n\right)}\right)$ are the outputs of the visible and SAR discriminators when evaluating real images, and $D_V\left(I_F^{\left(n\right)}\right)$ and $D_S\left(I_F^{\left(n\right)}\right)$ are their outputs for the generated images.

This loss ensures that discriminators are effectively trained to distinguish between real and generated images, thus enhancing the overall robustness and accuracy of the adversarial network.

3. Experimental Results

Here we cover the details of implementing and configuring our SAR and visible image fusion network. The experiment is performed to demonstrate the rationality of our model and the structure of the network.

3.1. Experimental Setup

3.1.1. Datasets

The primary dataset used in our experiments consists of 90 pairs of SAR and visible images, each with a sub-meter resolution of $512\times 512$, sourced from the YYX-OPT-SAR dataset [40]. These high-resolution images provide detailed information about the shape, structure, and texture of the objects, making them ideal for evaluating the performance of fusion algorithms in terms of structural and textural integration. The fusion results derived from these images offer a compelling assessment of the effectiveness of the fusion method. Furthermore, to assess the performance of the fusion method in different datasets, we used medium-resolution SAR and visible image pairs from the WHU-OPT-SAR dataset [41]. The original images in this dataset have a resolution of $5556\times 3704$, with the optical images containing four channels. To conform to the requirement for three channels of visible and to better illustrate the fusion effect, we excluded the fourth $\alpha$ channel and cropped the images to select 2566 pairs of $512\times 512$ SAR and visible image pairs. Examples of these images are presented in Figure 7.

3.1.2. Metrics

We used seven metrics to quantitatively evaluate the fusion results, which fully cover all four aspects of the evaluation system [42,43]:

• Information-Based:

  -

  Entropy (EN) quantifies the amount of information contained in an image, with higher values indicating richer information diversity.

  -

  The standard deviation (SD) reflects the contrast and intensity distribution of the fused image, where a higher SD suggests a better utilization of the dynamic range.

  -

  The edge-based metric (Qabf) evaluates the edge information transferred from source images to the fused result, with higher values implying superior edge preservation.

• Image-Feature-Based:

  -

  The average gradient (AG) measures the sharpness and texture details of an image; higher AG values correspond to richer gradient information.

  -

  The spatial frequency (SF) characterizes the overall activity level of the image details, where an elevated SF indicates enhanced edge and texture representation.

• Structural-Similarity-Based:

  -

  The Structural Similarity Index (SSIM) assesses the structural consistency between the fused and source images, with values closer to one denoting minimal structural distortion.

• Human Perception-Inspired:

  -

  Visual information fidelity (VIF) quantifies perceptual similarity through natural scene statistics and human visual system modeling, where higher scores align better with human visual expectations.

3.1.3. Implementation Details

The computational experiments were executed on a cloud-based infrastructure utilizing NVIDIA L20 GPUs with CUDA 11.3 acceleration, managed through a Miniconda3 environment (Python 3.8, Ubuntu 20.04). Our training pipeline comprised two sequential phases: domain adaptation via CycleGAN pretraining for 2400 iterations to generate photorealistic pseudo-images, followed by fusion network optimization with dataset-specific configurations. We maintained a uniform batch size of two across all experiments while adapting training durations to dataset characteristics—55 epochs for the YYX dataset versus 25 epochs for the WHU dataset. The optimization strategy employed Adam with differential learning rates: $1.0\times {10}^{-2}$ for generator networks versus $1.0\times {10}^{-5}$ for discriminators, implementing a scheduled decay policy that reduced learning rates by 25% every 10 epochs. Loss function hyperparameters were carefully calibrated through empirical validation: Equation (16) maintained equilibrium through identical weighting coefficients (${\lambda }_1={\lambda }_2=1$), while structural preservation in Equation (23) utilized symmetric weights ($w_1=w_2=0.5$). Feature-space constraints in Equation (24) were governed by the coefficient triad $\alpha =8$, $\beta =15$, and $\gamma =8$, which were selected to balance gradient magnitudes across different network components. This parametric configuration demonstrated stable convergence behavior while preserving multi-modal feature representations throughout the optimization trajectory.

3.2. Comparison with State-of-the-Art Methods

Our experimental evaluation establishes a rigorous comparison framework using two benchmark datasets, incorporating both cutting-edge deep learning techniques and classical fusion methodologies. The evaluation encompasses eight representative deep learning approaches: FusionGAN [14], ResNetFusion [44], and DenseFuse [45] as foundational architectures, followed by more recent developments including TarDal [46], ReCoNet [47], SDNet [48], CDDFuse [49], and MACTFusion [50]. For traditional baselines, we implement three well-established methods: weighted least-squares (WLS) optimization [51], perceptual quality-oriented Hybrid-MSD [52], and the recent variational framework VSFF [17]. To ensure methodological fairness, the supervised models (TarDal and ReCoNet) employ their official pre-trained implementations without architectural modifications, while other deep learning competitors undergo full training convergence on our experimental datasets. All conventional methods strictly adhere to their original parameter configurations as specified in respective publications. This comprehensive evaluation protocol facilitates a direct performance comparison across different methodological paradigms while maintaining implementation consistency.

3.2.1. Qualitative Comparison

Figure 8 and Figure 9 provide a qualitative comparison of the fusion results. In Figure 8, our proposed method effectively fuses SAR and visible images with sub-meter resolution, retaining both significant structural and spectral information conducive to interpretation. The red rectangle highlights areas where our method excels in filling in the “bad areas” in the source SAR images with visible images, ensuring seamless integration with adjacent regions. This results in a more coherent and visually appealing image, enhancing interpretability.

In Figure 9, our method demonstrates its ability to fuse medium-resolution SAR and visible images effectively. It preserves crucial target contours and maintains good contrast in building information within the red rectangle. The method successfully balances detail retention and noise suppression, providing a clear and detailed representation of the scene. This balance is crucial for applications requiring precise image analysis and interpretation.

3.2.2. Quantitative Comparison

We conducted a comprehensive quantitative comparison between our proposed method and state-of-the-art techniques on two datasets: the compact YYX dataset and the extensive WHU dataset.

Table 1. Quantitative results of the YYX datasets.

Method	EN	Qabf	SD	AG	SF	SSIM	VIF
WLS [51]	6.992	0.539	35.601	12.086	32.839	0.967	0.499
Hybrid-MSD [52]	7.018	0.565	38.034	12.867	34.572	0.978	0.543
VSFF [17]	7.333	0.44	49.279	13.126	33.263	0.919	0.49
FusionGAN [14]	6.855	0.314	36.152	7.188	17.926	0.325	0.477
DenseFuse [45]	6.853	0.408	30.961	8.498	22.092	0.914	0.442
ResNetFusion [44]	6.537	0.57	32.044	11.384	32.698	0.872	0.7
SDNet [48]	6.434	0.445	27.009	9.94	27.274	0.903	0.418
TarDal [46]	6.261	0.178	24.679	5.186	13.143	0.656	0.294
RecoNet [47]	6.461	0.338	40.681	8.044	17.008	0.781	0.42
CDDFuse [49]	7.197	0.618	41.234	12.546	32.799	0.929	0.651
MACTFusion [50]	7.271	0.561	41.735	12.2	32.58	0.914	0.484
Ours	7.118	0.62	40.277	13.244	34.402	0.966	0.638

Note: The dataset contains 90 image pairs (81 for training and 9 for testing). Color coding: red = best, green = second-best, and blue = third-best.

and

Table 2. Quantitative results of the WHU datasets.

Method	EN	Qabf	SD	AG	SF	SSIM	VIF
WLS [51]	6.775	0.662	39.67	15.69	42.722	0.733	0.545
Hybrid-MSD [52]	6.687	0.688	48.527	16.854	46.773	0.947	0.655
VSFF [17]	6.241	0.202	24.767	7.881	23.634	0.784	0.465
FusionGAN [14]	7.223	0.569	47.873	17.242	44.19	0.471	0.434
DenseFuse [45]	6.582	0.562	34.942	11.539	31.36	0.807	0.518
ResNetFusion [44]	6.21	0.74	48.291	16.812	47.194	0.954	0.834
SDNet [48]	6.457	0.655	36.128	14.322	38.646	0.682	0.512
TarDal [46]	6.386	0.272	36.766	8.185	21.446	0.574	0.319
RecoNet [47]	6.398	0.204	25.028	7.902	16.805	0.649	0.309
CDDFuse [49]	6.849	0.749	51.654	18.777	51.274	0.783	0.837
MACTFusion [50]	7.027	0.62	41.183	17.893	54.201	0.651	0.441
Ours	7.036	0.73	52.443	19.541	52.833	0.845	0.744

Note: The dataset contains 2566 image pairs (2552 for training and 14 for testing). Color coding: red = best, green = second-best, and blue = third-best.

summarize the evaluation results using seven established metrics, EN, Qabf, SD, AG, SF, SSIM, and VIF, with the top three performances marked in red, green, and blue, respectively. In the YYX dataset (Table 1), traditional methods, including WLS, Hybrid-MSD, and VSFF, demonstrate competitive performance, achieving notable scores in multiple metrics. However, our method continues to maintain relatively good performance. The results of the WHU dataset (Table 2) reveal distinct advantages of deep learning approaches, particularly our proposed method which achieves top-three performance across all metrics. In particular, our method excels at preserving important details and textures, as evidenced by its high scores in AG (19.541) and SF (52.833). These results align with our original intention of using detailed information to guide the training process. Furthermore, consistent high performance across different datasets and various metrics highlights the robustness and adaptability of our approach. The impressive results on the larger WHU dataset provide strong evidence that our approach can effectively leverage large amounts of data for more comprehensive training, leading to better performance. In particular, the high scores in AG and SF indicate that our method is capable of preserving fine details and textures, which are crucial for accurate image interpretation. This is consistent with our goal of enhancing the quality of image fusion by leveraging detailed information. In summary, the quantitative results validate the effectiveness of our proposed method in achieving high-quality image fusion, particularly on larger datasets. We have reason to believe that as dataset sizes continue to increase, our method will demonstrate even better performance due to more thorough training and generalization capabilities. Our approach is thus well-suited for SAR and visible image fusion tasks at different resolutions, and its consistent performance across datasets highlights its robustness and adaptability.

3.3. Ablation Studies

In this section, we validate the effectiveness of different modules and design choices through ablation experiments conducted on the YYX datasets. As shown in

Table 3. Ablation experiment results in the YYX datasets.

Exp	Configuration	EN	AG	SF	SD	VIF	Parameter Quantity
I	Leaky_Relu with Normalized to [−1, 1] → Relu with Normalized to [0, 1]	7.111	12.497	32.011	38.71	0.561	27,939,735
II	3 CBAM with 2 Addition → 5 CBAM	7.046	12.481	32.99	38.266	0.621	30,726,257
III	3 CBAM with 2 Addition → 5 Addition	7.057	12.568	32.819	38.299	0.595	27,762,769
IV	LSNF → Nest Connection	7.068	12.536	32.637	38.83	0.61	35,372,257
V	with DenseASPP → out DenseASPP	7.064	12.762	33.643	39.009	0.621	27,737,841
VI	with CycleGAN generators → out CycleGAN generators	7.099	12.969	33.709	39.671	0.635	27,938,865
VII	$Los{s}_{GWPL}$→$Los{s}_{MSE}$	7.276	11.724	29.807	42.982	0.578	27,939,441
Ours		7.118	13.244	34.402	40.277	0.638	27,939,441

Note: Boldface denotes results obtained by our method; underline indicates results that outperform ours.

, we use metrics such as EN, AG, SF, SD, VIF, and the number of model parameters to quantitatively assess fusion performance. The model parameters directly impact the model’s performance and complexity. They are computed on the basis of the specific architecture and configuration of the model. Generally, a higher number of parameters may lead to better performance due to the model’s increased capacity to learn complex patterns. However, it also results in greater computational complexity and resource demands. Thus, finding a balance between quantity, performance, and complexity is vital for optimal model design. The results of our method are highlighted in bold in the table, while results from other ablation configurations that outperform our method are underlined. These experiments demonstrate the contribution of each component to the overall performance, confirming the robustness and efficiency of our design. The analysis provides insights into how each module enhances the fusion process, ensuring optimal balance between detail preservation and computational efficiency.

3.3.1. Improved Resnet-50 and [−1, 1] Normalization

We verify that the validity of the normalized and modified Leaky_Relu activation function used [−1, 1]. In Exp. I, we revert to the [0, 1] normalization and the original Relu activation function, and the experimental results show that the modified Leaky_ReLU activation function combined with the [−1, 1] normalization is able to retain more details when dealing with negative values. This normalization extends the dynamic range of the data, allowing the network to better capture subtle changes in features during training. Leaky_ReLU avoids ReLU’s ’death’ problem by allowing negative values to pass, thus retaining more useful information during the feature extraction phase. In addition, a slight reduction in the number of parameters also helps to improve the computational efficiency and training speed of the model. Although this approach may have some impact on structural consistency, it can provide higher image quality and better visual effects in tasks that require rich detail.

3.3.2. Fusion Method of Joint Attention Mechanism CBAM and Addition

We verify the necessity of the proposed fusion method of the joint attention mechanism CBAM and addition. In Exp. II, we used a complete fusion method using the CBAM. Similarly, in Exp. III, we used a full fusion method using an adder. The experimental results show that although simple adder fusion can completely retain the feature information and slightly reduce the number of parameters, it lacks the ability to distinguish the importance of features, which is not good for the effective fusion of deep features. However, the fusion method using the CBAM can selectively highlight features, but it will lead to a significant increase in computational complexity and the number of model parameters, especially when dealing with high-dimensional features, and may introduce unnecessary computational overhead. In addition, attention mechanisms can overemphasize certain features, causing other important features to be ignored, thus losing information. We use the CBAM to select the features at the low and middle levels, and we used addition to retain the high-level semantic features completely so as to obtain the fusion image with rich details and complete semantic features in the reconstruction of the image.

3.3.3. Layered Strictly Nested Framework (LSNF)

We verify the validity of the proposed LSNF. In Exp. IV, we revert to the dense nested connection architecture [35], and the experimental results show that our LSNF firstly has a significant reduction in the number of model parameters, and secondly, such refined feature connections make the features interact more fully during the transfer process. This architecture not only improves the transmission efficiency of features but also enhances the diversity and expression ability of the features. This approach improves the visual quality and detail of the image while keeping the model lightweight.

3.3.4. Densely Connected Atrous Spatial Pyramid Pooling (DenseASPP) Module

We validated the necessity of using the DenseASPP module. In Exp. V, we removed this module, and the experimental results show that the DenseASPP module plays an important role in the low-level feature branches of the feature extraction phase, although using this module results in a small increase in the number of model parameters. It can capture multi-scale context information through intensive hollow convolution operations and enhance the feature expression ability. The module helps improve feature resolution and detail retention, especially when dealing with complex scenes, enabling better extraction and fusion of detail information. In addition, the use of the DenseASPP module allows the model to process diverse input data more efficiently while maintaining high accuracy, improving overall performance and robustness.

3.3.5. Modal Cross-Enhancement with CycleGAN Generators

We verify the necessity of using the bidirectional generation capability of the CycleGAN generator network to generate pseudo-modal images in the cross-modal enhancement stage. In Exp. VI, we directly input the source image into the fusion network, and the experimental results show that pre-mode cross-enhancement by generating pseudo-modal images can help overcome the differences between different modes. This approach provides a more consistent representation of features before fusion, allowing the fusion network to integrate information from different modes more efficiently, thereby preserving more detail and texture. This is especially important for tasks that require high precision and rich detail. However, in some applications, if the requirements for details are not high, direct fusion can achieve better results, simplifying the process and reducing the computational overhead. Therefore, whether or not to use modal cross-enhancement can be balanced according to specific task requirements.

3.3.6. ${\mathrm{Loss}}_{\mathrm{GMPL}}$

Finally, in Exp. VII, we modify the definition of Equation (22) by using a simple MSE loss [53] instead of a GMPL and show that while there is a small EN score advantage in the overall error, this approach ignores the weight allocation of image details. GMPL enables a sharper focus on edges and detailed areas within images. It enhances the retention of details by assigning greater weights to these critical regions. This approach aligns well with the human visual perception system, which prioritizes high-gradient areas. In contrast, simple MSE loss treats every pixel uniformly, often leading to blurred details and information loss. Thus, GMPL is particularly beneficial for tasks demanding high detail retention, as it delivers more precise image reconstruction that is more attuned to human visual perception.

4. Conclusions

This paper presented a novel local information-driven hierarchical fusion method for SAR and visible images. This method operates in two phases: In the cross-modal enhancement phase, CycleGAN generator-driven pseudo-modal image generation is employed to strengthen cross-modal consistency. In the fusion phase, the deep features are hierarchically decomposed. Guided by modality prior knowledge and the distinctive characteristics of hierarchical features, differentiated feature extraction and fusion branches are designed. The Layered Strictly Nested Framework (LSNF) establishes strict intra-layer and inter-layer nested connections to reduce computation and feature redundancy. The Gradient-Weighted Pixel Loss (GWPL) constraint increases the expression of high-frequency information, emphasizing edge details while maintaining global structural fidelity. Experiments on two datasets show that this method achieves state-of-the-art performance, highlighting its robustness, efficiency, and potential for practical applications. This approach not only provides a new solution for modality-adaptive remote sensing image fusion but also offers a novel perspective on feature interpretation. However, the model has limitations, such as performance degradation in small-scale datasets and room for improvement in visual perceptual quality. Future research will focus on lightweight variants for real-time deployment and extending the method to other modalities.