Multi-Scale Frequency-Aware Representation Learning for Infrared and Visible Image Fusion

Highlights

What are the main findings?

• A multi-scale frequency-aware fusion framework (MSF-Net) is proposed, which achieves a superior balance between thermal-target saliency and structural-detail preservation by integrating hybrid spatial-frequency encoding and structure-guided refinement.
• Extensive experiments on MSRS, M³FD, and TNO datasets demonstrate that MSF-Net outperforms nine SOTA across six core metrics, showing strong robustness in complex scenarios (e.g., smoke, low light, and remote-sensing urban scenes).

What are the implications of the main findings?

• The hybrid spatial-frequency representation learning paradigm provides an efficient alternative to self-attention for global context modeling, offering a new direction for low-level vision tasks like cross-modal image fusion.
• The proposed method’s strong generalization and degradation resistance enable reliable applications in remote sensing, autonomous driving, and military surveillance, supporting downstream tasks such as target detection and scene understanding.

Abstract

Infrared and visible image fusion aims to integrate complementary information from heterogeneous sensors for remote sensing and Earth-observation applications. To achieve a better balance between global contextual modeling and local structural preservation, we propose MSF-Net, a multi-scale frequency-aware fusion network with a hierarchical design. The proposed framework consists of two main stages: multi-scale feature extraction with frequency-domain interaction and hierarchical cross-modal fusion. Specifically, a hybrid spatial-frequency encoding block (HSFEB) is designed as the basic building unit, which combines a spatial-frequency interaction module (SFIM) for global context aggregation in the frequency domain and a structure-guided feature refinement module (SGFRM) for preserving local structural details. In addition, a hierarchical feature fusion module (HFFM) is introduced to progressively integrate cross-modal and cross-scale features in a coarse-to-fine manner. A joint loss function, composed of intensity and structural constraints, is adopted to supervise the fusion process. Extensive experiments on three public benchmarks, MSRS, M³FD, and TNO, demonstrate that MSF-Net achieves superior performance over nine SOTA methods in both qualitative and quantitative evaluations. The results show that the proposed method effectively enhances thermal targets, preserves structural details, and maintains good visual naturalness under diverse remote-sensing scenarios.

1. Introduction

Infrared and visible image fusion aims to integrate complementary information from heterogeneous sensors into a single representation with improved perceptual quality [1]. Visible images usually provide rich texture details and fine spatial structures under favorable illumination conditions, whereas infrared images are insensitive to illumination variations and can effectively highlight salient thermal targets in low-light environments [2,3]. By combining the complementary advantages of these two modalities, infrared and visible image fusion has been widely applied in tasks such as visual perception enhancement, scene understanding, and nighttime observation, and it also shows potential in remote sensing and related Earth-observation scenarios [4,5,6].

Despite extensive research efforts, infrared and visible image fusion remains a challenging problem due to the inherent modality discrepancy between the two imaging sources [7]. Infrared and visible images differ significantly in imaging mechanisms, intensity distributions, and semantic characteristics, making direct feature aggregation suboptimal [8]. Early fusion approaches mainly relied on handcrafted rules and multi-scale transforms, such as pyramid-based and wavelet-based methods [9,10], to combine source images. Although these methods are computationally efficient and interpretable, their limited representation capacity often leads to insufficient preservation of texture details or degraded thermal saliency in complex scenes. More recently, deep learning-based methods have achieved notable improvements by learning fusion strategies directly from data. However, effectively modeling both global contextual information and local structural details across heterogeneous modalities remains an open challenge [11,12,13].

Multi-scale representation learning has been demonstrated to be an effective strategy for addressing the complexity of image fusion tasks [14,15,16]. Features extracted at different spatial scales capture complementary information, ranging from global scene structures to fine-grained local textures. This property is particularly important for infrared and visible image fusion, where thermal targets and visual details often dominate at different resolutions [17,18,19]. In parallel, frequency-domain modeling has recently attracted increasing attention in low-level vision tasks due to its ability to efficiently capture global contextual information. By transforming spatial features into the frequency domain, long-range dependencies can be modeled without introducing excessive computational overhead. These observations motivate the exploration of multi-scale frequency-aware representations for infrared and visible image fusion [20,21,22].

In this paper, we propose MSF-Net, a multi-scale frequency-aware network for infrared and visible image fusion. The proposed framework adopts a hierarchical design and extracts feature representations at four spatial scales using a multi-scale frequency-aware encoder. Within the encoder, frequency-domain interactions are explicitly introduced to aggregate global context, while structure-guided feature refinement is employed to preserve local structural fidelity [23]. Furthermore, a hierarchical feature fusion module is designed to progressively integrate multi-scale features from infrared and visible modalities. Extensive experiments on public infrared and visible image fusion benchmarks demonstrate that the proposed method achieves competitive performance compared with SOTA approaches. Moreover, the proposed framework is general and can be readily extended to infrared and visible image fusion tasks in remote-sensing scenarios.

The main contributions of this paper are summarized as follows:

We propose a multi-scale frequency-aware representation learning framework for infrared and visible image fusion.

We introduce a hybrid spatial-frequency feature extraction strategy to explicitly incorporate frequency-domain modeling into multi-scale encoding.

We design a structure-guided feature refinement mechanism to enhance local structural consistency after global modeling.

We develop a hierarchical feature fusion module to progressively integrate cross-modal and cross-scale information.

Extensive experimental results demonstrate the effectiveness and generalization ability of the proposed framework.

2. Related Work

2.1. Traditional Image Fusion Methods

Traditional image fusion methods mainly rely on handcrafted rules and classical signal processing techniques [24,25]. Early approaches typically adopt multi-scale decomposition strategies to separate source images into different components and then fuse them using heuristic criteria [26]. Representative methods include pyramid-based fusion approaches such as the Laplacian pyramid, as well as transform-based techniques based on wavelet, contourlet, and nonsubsampled contourlet transforms [27,28,29,30]. These methods combine coefficients at different scales using activity-level measures, local energy, or saliency-driven rules. Sparse representation has also been widely explored for image fusion. In these methods, source images are projected onto over-complete dictionaries, and fusion is performed by selecting or combining sparse coefficients according to specific criteria. Low-rank representation-based fusion methods further exploit global image structures by decomposing source images into low-rank and sparse components. Although traditional fusion methods are computationally efficient and interpretable, their performance heavily depends on manually designed fusion rules and prior assumptions, which limit their robustness in complex scenes.

2.2. AE-Based Fusion Methods

With the advent of deep learning, autoencoder-based fusion methods have emerged as early learning-based approaches for image fusion [31]. These methods typically employ encoder-decoder architectures to learn latent representations of source images, where fusion is conducted in the feature space, and the fused image is reconstructed by a decoder. One of the earliest works applied convolutional autoencoders to infrared and visible image fusion, demonstrating the effectiveness of data-driven feature learning. Subsequent AE-based methods introduced deeper convolutional autoencoders, dense connections, and improved reconstruction losses to enhance detail preservation and fusion quality [32]. Some studies have further integrated attention mechanisms into AE frameworks to emphasize salient features from each modality [33]. Despite their advantages, most AE-based fusion methods primarily rely on spatial-domain convolutions, which may limit their ability to capture long-range dependencies and global contextual information.

2.3. CNN-Based Fusion Methods

Beyond autoencoder-based frameworks, a large body of work directly employs convolutional neural networks for image fusion [34,35]. CNN-based fusion methods typically extract features from source images using convolutional layers and perform fusion through feature concatenation, weighted summation, or element-wise operations, followed by convolutional mixing. Representative methods include DeepFuse, which demonstrates that CNNs can learn effective fusion rules without handcrafted priors [36]. Subsequent CNN-based fusion methods further improve fusion performance by incorporating multi-scale feature extraction, residual learning, and attention mechanisms. For example, residual fusion networks leverage skip connections to stabilize training and preserve structural information, while attention-based CNNs adaptively weight modality-specific features to highlight salient regions [37,38,39]. Although CNN-based methods achieve strong performance and efficiency, their reliance on local convolutional operations makes it challenging to model global contextual relationships, particularly for high-resolution images.

2.4. Transformer-Based Fusion Methods

Transformer-based models have recently been introduced into image fusion to address the limited receptive field of convolutional architectures [40]. By leveraging self-attention mechanisms, transformer-based fusion methods can model long-range dependencies and global interactions across the entire image. Several studies have explored pure transformer architectures for fusion, while others have adopted hybrid CNN-transformer designs to balance global modeling and local detail preservation [41,42]. Representative transformer-based fusion methods apply self-attention to model cross-modal interactions between infrared and visible features, enabling adaptive feature weighting across spatial locations. Some approaches further incorporate hierarchical transformers or window-based attention to improve efficiency [43]. However, transformer-based fusion methods often suffer from high computational and memory costs, especially when applied to high-resolution images, motivating the search for more efficient global modeling strategies.

2.5. Frequency-Domain Modeling in Image Representation Learning

Frequency-domain analysis has long been a fundamental tool in signal processing and image analysis. Classical techniques such as the Fourier transform and the discrete cosine transform decompose images into spectral components that encode global structural information. Unlike spatial-domain representations, frequency-domain components inherently capture information aggregated from the entire image, providing a global perspective on image content [44,45]. In recent years, frequency-domain modeling has been revisited in deep learning-based image representation learning. Several studies have integrated Fourier transforms into neural network architectures to enhance global context modeling. Examples include Fourier-based convolutions and hybrid spatial-frequency architectures that combine spatial-domain feature extraction with frequency-domain interactions [46]. These designs enable efficient global feature mixing without introducing the high computational cost associated with self-attention mechanisms. Frequency-domain modeling has demonstrated effectiveness in various low-level vision tasks, suggesting its potential as a general representation learning paradigm [47]. Recent studies have also explored frequency-domain and transformer-based interaction mechanisms in related image reconstruction tasks. For example, frequency-domain cross-interaction strategies have shown effectiveness in enhancing global information exchange in hyperspectral image super-resolution [48]. However, unlike these methods, our work focuses on infrared-visible image fusion, where the key challenge lies in jointly preserving thermal-target saliency and visible structural details under heterogeneous modality discrepancies.

3. Methods

3.1. Architecture of MSF-Net

Given an infrared image $I^{ir}\in {ℝ}^{H\times W}$ and a visible image $I^{vis}\in {ℝ}^{H\times W}$, the objective of infrared and visible image fusion is to generate a fused image $I^{fus}\in {ℝ}^{H\times W}$ that simultaneously preserves the salient thermal targets from the infrared modality and the rich structural details from the visible modality. To achieve this goal, we propose a multi-scale frequency-aware fusion framework termed MSF-Net, whose overall architecture is illustrated in Figure 1. MSF-Net adopts a two-stage framework that balances global contextual modeling and local structural preservation in an efficient manner. The first stage focuses on multi-scale feature extraction with explicit frequency-domain modeling, while the second stage performs hierarchical cross-modal feature fusion to progressively integrate infrared and visible information across different spatial resolutions.

Figure 1. Overall architecture of the proposed MSF-Net. (a) Hybrid spatial-frequency encoding block. (b) Hierarchical feature fusion module.

In the first stage, the infrared and visible images are processed by two weight-sharing encoders to extract modality-specific feature representations. Each encoder is constructed by stacking multiple hybrid spatial-frequency encoding blocks (HSFEBs) arranged in a hierarchical manner, producing feature maps at four spatial scales. Instead of relying on self-attention mechanisms for global modeling, each HSFEB explicitly introduces frequency-domain interactions to capture long-range dependencies, while complementary spatial-domain operations are retained to preserve local structures. This design enables efficient global-local representation learning without incurring excessive computational overhead. Specifically, at each scale, the input features are successively refined through a series of HSFEBs. Within each block, frequency-domain feature interaction is first performed to aggregate global contextual information, followed by structure-guided refinement to enhance local details and suppress potential artifacts introduced by global mixing. Through progressive downsampling, the encoder captures complementary information ranging from fine-grained textures at shallow levels to high-level structural representations at deeper levels, forming a multi-scale feature pyramid for both modalities.

In the second stage, the extracted multi-scale infrared and visible features are fed into a hierarchical feature fusion module (HFFM). The HFFM performs cross-modal fusion at each scale and further propagates high-level semantic information to lower-resolution features via a top-down aggregation pathway. By jointly modeling cross-modal complementarity and cross-scale dependencies, the fusion module produces a unified representation that integrates thermal saliency and visual detail in a coherent manner. Finally, the fused features are projected back to the image space to generate the output fused image.

3.2. Hybrid Spatial-Frequency Encoding Block

To effectively balance global contextual modeling and local structural preservation in infrared and visible image fusion, we design a hybrid spatial-frequency encoding block (HSFEB) as the fundamental building unit of the proposed MSF-Net. Each HSFEB integrates frequency-domain interaction and structure-guided refinement in a unified residual framework, enabling efficient global-local representation learning at multiple spatial scales.

Let $H^{t-1}\in {ℝ}^{H\times W\times C}$ denotes the input feature map of the $t\text{-}\mathrm{th}$ HSFEB. Inspired by frequency-aware hierarchical architectures, the block adopts a two-step residual update strategy, where global context aggregation is first introduced through frequency-domain modeling, followed by structure-aware feature refinement. The overall formulation of the HSFEB can be expressed as

$$H^{t-\frac{1}{2}}=H^{t-1}+\mathrm{SFIM}(\mathrm{BN}(H^{t-1})),$$

(1)

$$H^t=H^{t-\frac{1}{2}}+\mathrm{SGFRM}(\mathrm{BN}(H^{t-\frac{1}{2}}),G^{t-1}),$$

(2)

where $\mathrm{B}\mathrm{N}(\cdot )$ denotes batch normalization, $\mathrm{S}\mathrm{F}\mathrm{I}\mathrm{M}(\cdot )$ represents the spatial-frequency interaction module, $\mathrm{S}\mathrm{G}\mathrm{F}\mathrm{R}\mathrm{M}(\cdot )$ denotes the structure-guided feature refinement module, and $G^{t-1}$ is the structure guidance feature propagated from the previous block.

The first update stage introduces explicit frequency-domain interaction to the input features. By transforming spatial representations into the frequency domain and performing learnable spectral modulation, the SFIM efficiently aggregates global contextual information across the entire spatial extent. Compared with self-attention-based global modeling, this frequency-aware design provides a computationally efficient alternative while retaining a global receptive field. The residual connection in Equation (1) ensures stable optimization and preserves the original spatial information.

While frequency-domain interactions enhance global consistency, they may also weaken local structural details due to global feature mixing. To compensate for this effect, the second update stage employs a structure-guided refinement mechanism. The SGFRM adaptively modulates the intermediate features using learned structural cues, selectively enhancing informative local structures and suppressing redundant responses. The structure guidance feature $G^t$ is progressively updated across encoding blocks, allowing the refinement process to evolve in accordance with hierarchical feature abstraction.

By cascading multiple HSFEBs at each scale, the encoder is able to capture complementary information ranging from fine-grained textures at shallow layers to high-level structural representations at deeper layers. This hybrid spatial-frequency design enables MSF-Net to effectively integrate global context and local details, which is particularly important for infrared and visible image fusion, where thermal saliency and visual structure must be simultaneously preserved.

3.2.1. Spatial-Frequency Interaction Module

The spatial-frequency interaction module (SFIM) is designed to efficiently capture global contextual information by explicitly modeling feature interactions in the frequency domain. Unlike spatial-domain convolutions that operate on local neighborhoods, frequency-domain representations inherently encode global image characteristics, making them well-suited for long-range dependency modeling in infrared and visible image fusion. The specific structure of SFIM is shown in Figure 2.

Figure 2. Overview of the spatial-frequency interaction module. FFT and IFFT denote the fast Fourier transform and inverse fast Fourier transform, respectively. PConv denotes pixel-wise convolution.

Given the normalized input feature map $X=\mathrm{BN}(H^{t-1})\in {ℝ}^{H\times W\times C}$, SFIM first performs spatial-domain feature embedding to extract local patterns and enrich channel-wise representations. Specifically, a pixel-wise convolution is applied to expand the feature channels, followed by channel splitting:

$$U={\mathrm{Conv}}_{1\times 1}(X)\in {ℝ}^{H\times W\times \gamma C},$$

(3)

$$[U_1,U_2]=\mathrm{Split}(U),$$

(4)

where $\gamma$ denotes the channel expansion ratio. To capture multi-scale local information, depth-wise convolutions with different kernel sizes are applied to the split features:

$$S=\varphi (\mathrm{Concat}({\mathrm{DWConv}}_{3\times 3}(U_1),{\mathrm{DWConv}}_{5\times 5}(U_2))),$$

(5)

where $\varphi (\cdot )$ denotes a nonlinear activation function. The resulting feature $S$ serves as a spatial-domain representation that preserves local structural details. To introduce global feature interaction, the spatial features $S$ are transformed into the frequency domain using a two-dimensional Fourier transform:

$$[A,B]=\mathcal{F}(S),$$

(6)

where A and B denote the real and imaginary components of the frequency spectrum, respectively. The spectral components are then concatenated and modulated via a learnable spectral transformation:

$$M=\varphi (\mathrm{BN}({\mathrm{Conv}}_{1\times 1}(\mathrm{Concat}(A,B)))),$$

(7)

$$[{\displaystyle \tilde{\mathbf{A}},\tilde{\mathbf{B}}}]=\mathrm{Split}(M),$$

(8)

where the learnable transformation enables adaptive reweighting of frequency components, facilitating global context aggregation across the entire spatial extent. After spectral modulation, the frequency-domain features are projected back to the spatial domain via the inverse Fourier transform:

$$F={\mathcal{F}}^{-1}({\displaystyle \tilde{\mathbf{A}},\tilde{\mathbf{B}}})\in {ℝ}^{H\times W\times \gamma C},$$

(9)

where F encodes globally aggregated contextual information. Finally, the spatial-domain feature S and the frequency-enhanced feature F are fused and compressed to produce the SFIM output:

$$\mathrm{SFIM}(X)={\mathrm{Conv}}_{1\times 1}(S+F).$$

(10)

Through this spatial-frequency interaction process, SFIM effectively combines local structural representations with global contextual cues. Compared with attention-based global modeling, the frequency-domain interaction provides a computationally efficient alternative while retaining a global receptive field. The output of SFIM is subsequently refined by structure-guided feature modulation, which is introduced in the following subsection.

3.2.2. Structure-Guided Feature Refinement Module

While the frequency-domain interaction in SFIM effectively aggregates global contextual information, the global mixing of features may weaken local structural details that are critical for perceptual quality in infrared and visible image fusion. To address this issue, we introduce a structure-guided feature refinement module (SGFRM), which adaptively enhances local structural consistency by leveraging learned structural cues. Let $Y=H^{t-\frac{1}{2}}\in {ℝ}^{H\times W\times C}$ denote the intermediate feature after frequency-domain interaction. The SGFRM consists of two cooperative branches: a main refinement branch responsible for local feature enhancement and a structure guidance branch that generates structure-sensitive modulation signals. The specific structure of SGFRM is shown in Figure 3.

Figure 3. Overview of the structure-guided feature refinement module.

Instead of relying on task-specific priors, the structure guidance feature is self-generated from intermediate representations. Specifically, the guidance feature $G^{t-1}$ propagated from the previous block is first updated using a lightweight transformation:

$$G^t=\mathrm{GeLU}({\mathrm{Conv}}_{3\times 3}(G^{t-1})),$$

(11)

the guidance feature $G^{t-1}$ propagated from the previous block is updated through a 3 × 3 convolution followed by a GeLU activation. Through progressive updates across encoding blocks, $G^t$ gradually evolves to align with hierarchical feature abstraction, encoding structure-sensitive cues such as edges and intensity transitions. To refine the intermediate features, the updated structure guidance feature $G^t$ is employed to generate an adaptive modulation map:

$${\mathbf{M}}^t=\sigma ({\mathrm{Conv}}_{1\times 1}(G^t)),$$

(12)

where $\sigma (\cdot )$ denotes the sigmoid function. The modulation map ${\mathbf{M}}^t$ acts as a soft structural calibration signal, selectively emphasizing informative local regions while suppressing redundant responses. The refined feature representation is then obtained via residual modulation:

$${\mathbf{Y}}^{\prime }=Y+Y\odot M^t,$$

(13)

where $\odot$ denotes element-wise multiplication. Compared with hard masking strategies, this residual modulation preserves the original feature distribution while adaptively enhancing structure-consistent responses. Finally, a pixel-wise convolution is applied to integrate the refined features and adjust channel representations:

$$\mathrm{SGFRM}(Y,G^{t-1})={\mathrm{Conv}}_{1\times 1}({\mathbf{Y}}^{\prime }).$$

(14)

By explicitly introducing structure-guided modulation after frequency-domain interaction, the SGFRM compensates for potential structural degradation caused by global context aggregation. This design enables the network to jointly benefit from frequency-aware global modeling and structure-preserving local refinement, which is essential for achieving high-quality infrared and visible image fusion.

3.3. Hierarchical Feature Fusion Module

After multi-scale feature extraction using stacked HSFEBs, the infrared and visible branches produce hierarchical feature representations at four spatial scales. To effectively integrate cross-modal complementarity and cross-scale dependencies, we design a hierarchical feature fusion module (HFFM) that progressively fuses infrared and visible features in a coarse-to-fine manner. Let ${\{F_i^{ir},F_i^{vis}\}}_{i=1}^4$ denote the infrared and visible feature maps extracted at the $i$-th scale, where $i=1$ corresponds to the highest spatial resolution and $i=4$ corresponds to the lowest. The HFFM consists of three key stages: same-scale cross-modal fusion, top-down cross-scale aggregation, and multi-scale alignment with final fusion.

At each scale, infrared and visible features are first fused to form a modality-complementary representation. Specifically, features from the two modalities are concatenated along the channel dimension and compressed via a convolutional transformation:

$$C_i=\mathrm{GeLU}({\mathrm{Conv}}_{3\times 3}(\mathrm{Concat}(F_i^{ir},F_i^{vis}))),i\in \{1,2,3,4\},$$

(15)

the concatenated cross-modal features are further transformed by a 3 × 3 convolution and a GELU activation to obtain the fused representation at each scale. This operation explicitly models cross-modal interactions at each scale while reducing feature redundancy. To further enhance the global context at the coarsest level, an optional context expansion module can be applied to $C_4$, such as pyramid pooling or atrous spatial pyramid pooling:

$${C^{\prime }}_4=\epsilon (C_4),$$

(16)

where $\mathcal{E}(\cdot )$ denotes the selected context expansion operator.

To propagate high-level semantic information to finer scales, HFFM adopts a top-down aggregation strategy similar to feature pyramid networks. Starting from the coarsest scale, higher-level features are successively upsampled and fused with features at the adjacent finer scale. For the transition from scale $i+1$ to scale $i$, the aggregation process is formulated as

$$U_i={\mathrm{Conv}}_{3\times 3}(\uparrow ({C^{\prime }}_{i+1})),$$

(17)

$${C^{\prime }}_i=\varphi ({\mathrm{Conv}}_{3\times 3}(\mathrm{Concat}(C_i,U_i))),i=3,2,1,$$

(18)

where $\uparrow (\cdot )$ denotes spatial upsampling by a factor of 2. Through this progressive aggregation, semantic information captured at deeper layers is effectively injected into shallower features, enhancing global consistency while preserving local details. After top-down aggregation, the refined features $C_i^{\prime }$ at different scales encode complementary information. To further integrate multi-scale cues, features from all scales are spatially aligned to the highest resolution and jointly fused. Specifically, the aggregated features from scales $i=2,3,4$ are resized to match the spatial resolution of $C_1^{\prime }$ and transformed via convolutional layers:

$$D_i={\mathrm{Conv}}_{3\times 3}(\mathrm{Resize}({C^{\prime }}_i,H_1,W_1)),i\in \{2,3,4\},$$

(19)

where $(H_1,W_1)$ denotes the spatial size of the finest scale. The final fused representation is then obtained by concatenating and mixing all aligned features:

$$F^{fus}=\varphi ({\mathrm{Conv}}_{3\times 3}(\mathrm{Concat}({C^{\prime }}_1,D_2,D_3,D_4))).$$

(20)

Through hierarchical cross-modal fusion and top-down multi-scale aggregation, the proposed HFFM effectively integrates thermal saliency from infrared images and structural details from visible images across different spatial resolutions. This design complements the frequency-aware feature extraction in the encoder and provides a robust fused representation for subsequent image reconstruction.

3.4. Loss Function

The objective of infrared and visible image fusion is to generate a fused image that preserves both salient intensity information and fine structural details from the source images. To this end, we design a fusion loss function that jointly enforces intensity consistency and structural preservation.

To ensure that the fused image retains the most informative intensity responses from both modalities, we adopt a max-intensity strategy. Specifically, a reference intensity image is constructed by taking the element-wise maximum of the infrared and visible images. The intensity loss is defined as

$${\mathcal{L}}_{\mathrm{int}}=\left|\right|\max (I^{vis},I^{ir})-I^{fus}|{|}_1.$$

(21)

This loss encourages the fused image to preserve salient thermal responses from the infrared image while maintaining strong intensity contrast from the visible image. Although intensity consistency is important, it is insufficient for maintaining fine structural details such as edges and textures. Therefore, we further introduce a gradient-based loss to explicitly constrain structural fidelity. A Sobel operator is employed to extract gradient representations along horizontal and vertical directions:

$$\nabla I=|I\ast K_x|+|I\ast K_y|,$$

(22)

where $\ast$ denotes convolution, and $K_x$, $K_y$ are the Sobel kernels. To preserve the most salient structures from both modalities, a joint gradient reference is constructed using the element-wise maximum of the infrared and visible gradients. The gradient loss is formulated as

$${\mathcal{L}}_{\mathrm{grad}}=\left|\right|\max (\nabla I^{vis},\nabla I^{ir},)-\nabla I^{fus}|{|}_1.$$

(23)

This term effectively promotes edge sharpness and texture clarity in the fused image. The final fusion loss is defined as a weighted sum of the intensity and gradient losses:

$${\mathcal{L}}_{\mathrm{total}}={\mathcal{L}}_{\mathrm{int}}+\lambda {\mathcal{L}}_{\mathrm{grad}},$$

(24)

where $\lambda$ is a weighting parameter that balances intensity preservation and structural consistency. In our implementation, $\lambda$ is empirically set to 10 for all experiments.

This combined loss function provides effective supervision for infrared and visible image fusion by simultaneously enforcing global intensity consistency and local structural preservation. The implementation of the proposed loss function follows the formulation described above.

4. Experiments and Results

4.1. Datasets and Training Details

To rigorously evaluate the effectiveness and generalization of the proposed MSF-Net for infrared and visible image fusion, extensive experiments are conducted on three widely adopted public benchmarks that cover diverse real-world scenes, illumination conditions and complex degradations. All image pairs are pre-aligned by the original dataset providers, and we perform pair-consistent preprocessing and data augmentations to preserve spatial correspondence between IR and VIS modalities throughout the experiment. The selected datasets are MSRS (2017) [49], M³FD (2022) [50], and TNO (2017) [51], whose core characteristics and evaluation values for fusion tasks are elaborated as follows.

MSRS is dedicated to remote sensing-related road and urban scenes, including buildings, sidewalks, lane markings and moving pedestrians and vehicles, captured under both daytime and nighttime conditions. Frequent high-frequency structural details (facades, window frames, and road signs) coexist with dynamic thermal targets and varying illumination, which makes this dataset particularly informative for assessing semantic-level and multi-scale fusion quality and verifying whether thermal saliency and fine structural cues can be fused consistently across different spatial scales. M³FD contains IR-VIS pairs with diverse degradations, including rainy and smoky weather, defocus blur, over- or underexposure, and illumination extremes, with scenes often involving target occlusions and depth-of-field changes that disrupt naive fusion rules. This dataset stresses the generalization of fusion methods under complex real-world degradations, requiring robust fusion performance to maintain edge integrity and global brightness balance while suppressing fusion artifacts and ghosting. The TNO Image Fusion collection centers on nighttime and low-light outdoor scenarios, including urban streets, rural roads and military camps. IR images capture salient thermal targets (pedestrians, vehicles, and low-light buildings) under poor illumination, while VIS images provide complementary structural context (edges, geometric contours, and surface textures). Typical challenges in this dataset include severe illumination imbalance, inherent sensor thermal noise, glare around light sources, and sparse high-contrast structures, making it well-suited for evaluating the cross-modality complementarity and contrast preservation capability of fusion methods.

For unified training and testing, all IR-VIS image pairs across the three datasets are resized or cropped to 256 × 256, which is a standard resolution for IR-VIS fusion tasks. The proposed MSF-Net is implemented in the PyTorch framework and trained from scratch on the combined training split of the three datasets using an NVIDIA RTX A6000 GPU for accelerated training and inference. The core hyperparameters of MSF-Net are determined by considering the trade-off among representation capacity, computational cost, and fusion stability. Specifically, the number of scales is set to four, since this configuration provides sufficient hierarchical context for infrared-visible fusion while avoiding excessive redundancy and optimization difficulty at deeper levels. In the spatial-frequency interaction module, the channel expansion ratio γ is introduced to enrich channel-wise representations before spectral interaction and is set to 4 to balance feature diversity and efficiency. Multi-branch depth-wise convolutions with kernel sizes 3 and 5 are adopted to capture local structures under different receptive fields, enabling the encoder to preserve both fine textures and medium-range spatial patterns. For optimization, we adopt the Adam optimizer with an initial learning rate of $1\times {10}^{-4}$. In the loss function, the weighting parameter λ is set to 10 to balance intensity preservation and structural consistency, where a too small value tends to weaken structural enhancement, while an excessively large value may overemphasize gradient information and affect overall intensity balance. During training, pair-consistent data augmentations, including random horizontal or vertical flipping and random rotation, are applied to the training set to expand sample diversity, while no augmentation is used on the test set to ensure the authenticity of evaluation results.

4.2. Comparison Methods and Evaluation Metrics

To fully verify the superiority of the proposed MSF-Net in infrared and visible image fusion, we compare it with nine state-of-the-art fusion methods covering GAN-based, CNN-based, semantic-driven, and frequency-domain frameworks. All comparison methods are implemented with their official open-source codes, retrained and evaluated under the same experimental setup (datasets, resolution, and hardware) as MSF-Net to ensure a fair comparison; hyperparameters of each method are set strictly according to the original papers to guarantee optimal performance. The comparison methods are listed as follows:

FusionGAN (2019) [52]: A pioneering GAN-based fusion method that adopts an adversarial game between generator and discriminator. As an end-to-end framework, it avoids manual fusion rules, retains infrared thermal radiation and visible texture details, and generalizes to cross-resolution fusion without upsampling noise.

DenseFuse (2018) [53]: A classic CNN-based fusion method integrated with dense blocks, enabling full reuse of multi-layer features from encoders. It adopts addition and L1-norm fusion strategies to integrate salient features and reconstructs fused images via a decoder, achieving effective fusion for both grayscale and RGB images.

NestFuse (2020) [54]: A fusion method based on nest connection and spatial/channel attention mechanisms. It extracts multi-scale deep features via an encoder, fuses them using dual-attention models, and reconstructs fused images through a nested connection-based decoder, preserving rich salient features.

U2Fusion (2020) [55]: A unified unsupervised end-to-end fusion network that handles multi-modal, multi-exposure, and multi-focus tasks. It uses adaptive information preservation degrees to guide similarity learning, adopts elastic weight consolidation to avoid catastrophic forgetting, and leverages DenseNet for feature fusion and image reconstruction.

RFN-Nest (2021) [56]: An end-to-end residual fusion network based on nest connection. It adopts a two-stage training strategy, uses residual fusion networks for learnable feature fusion, and leverages detail-preserving and feature-enhancing loss functions to balance visible details and infrared salient features.

LRRNet (2023) [57]: A lightweight fusion network guided by low-rank representation learning. It uses LLRR blocks to decompose source images into low-rank and sparse parts and adopts a detail-to-semantic multi-level loss function to balance visible details and infrared salient features.

FSFusion (2024) [20]: A full-scale hierarchical encoder-decoder fusion method with cascading edge prior. It adopts a triple fusion mechanism (edge, shallow, and full-scale semantic fusion) and a dual-attention strategy and uses a loss function with edge constraints to enhance target contrast and detail preservation.

SPDFusion (2024) [58]: A semantic prior knowledge-driven fusion method based on GAN. It leverages pre-trained semantic segmentation to extract multi-scale semantic information, integrates it via semantic feature perception and embedding modules, and uses regional semantic adversarial loss to enhance fusion quality for visual perception and advanced tasks.

SFINet (2025) [59]: A semantic feature interactive learning network for full-time fusion. It integrates multi-scale feature extraction, dual-attention fusion, and semantic interaction modules and is guided by a segmentation network to balance visual quality and semantic information for both day and night scenes.

To quantitatively evaluate the fusion performance from multiple perspectives (contrast, detail richness, information retention, and visual consistency), we adopt six widely used objective metrics. For all metrics, higher values indicate better fusion performance. The definitions and formulas are given below:

• Standard Deviation (SD)

SD reflects the gray-level contrast and dynamic range of the fused image. It is defined as

$$SD=\sqrt{{\displaystyle \frac{1}{H\times W}}{{\displaystyle {\sum }_{i=1}^H{\displaystyle {\sum }_{j=1}^W(I(i,j)-\mu )}}}^2},$$

(25)

where $I(i,j)$ is the pixel value at position $(i,j)$ of the fused image, H and W are the height and width of the image, and μ is the mean value of the fused image.

2.

Average Gradient (AG)

AG measures the local gray-level variation rate, reflecting the clarity and edge sharpness of the fused image:

$$AG={\displaystyle \frac{1}{(H-1)(W-1)}}{\displaystyle {\sum }_{i=1}^{H-1}{\displaystyle {\sum }_{j=1}^{W-1}\sqrt{\frac{{\Delta }_{x}I{(i,j)}^2+{\Delta }_{y}I{(i,j)}^2}{2}},}}$$

(26)

where ${\Delta }_{x}I(i,j)=I(i+1,j)-I(i,j)$ and ${\Delta }_{y}I(i,j)=I(i,j+1)-I(i,j)$ are the horizontal and vertical gradients, respectively.

3.

Spatial Correlation Coefficient (SCD)

SCD measures the overall spatial correlation between the fused image and the two source images, reflecting the information retention ability:

$$SCD={\displaystyle \frac{1}{2}}\left({\displaystyle \frac{\mathrm{cov}(F,I_R)}{{\sigma }_F{\sigma }_{I_R}}}+{\displaystyle \frac{\mathrm{cov}(F,I_V)}{{\sigma }_F{\sigma }_{I_V}}}\right),$$

(27)

where $F$ is the fused image, $I_R$ and $I_V$ are the IR and VIS source images, cov(·) denotes the covariance, and σ denotes the standard deviation.

4.

Entropy (EN)

EN quantifies the information richness of the fused image based on information theory:

$$EN=-{\displaystyle {\sum }_{l=0}^{L-1}p(l){\log }_{2}p(l),}$$

(28)

where $L$ is the number of gray levels (typically 256), and $p(l)$ is the normalized histogram probability of the gray level $l$.

5.

Feature-Based Fusion Quality Index (Q_abf)

Q_abf evaluates the visual quality by measuring the similarity of edge features between the fused image and the source images, combining spatial-frequency and phase consistency:

$$Q_{\mathrm{abf}}={\displaystyle \frac{1}{H\times W}}{\displaystyle {\sum }_{i=1}^H{\displaystyle {\sum }_{j=1}^W\left(\lambda (i,j)Q_{\mathrm{abf}}^R(i,j)+(1-\lambda (i,j))Q_{\mathrm{abf}}^V(i,j)\right)},}$$

(29)

where $Q_{\mathrm{abf}}^R$ and $Q_{\mathrm{abf}}^V$ are the edge similarity between the fused image and the IR/VIS source images, respectively, and $\lambda (i,j)$ is the weight balancing the contribution of the two modalities.

6.

Spatial-Frequency (SF)

SF reflects the overall activity and detail richness of the fused image by integrating row and column frequency components:

$$SF=\sqrt{R{F}^2+C{F}^2},$$

(30)

where the row frequency $RF$ and column frequency $CF$ are defined as

$$RF=\sqrt{{\displaystyle \frac{1}{H(W-1)}}{\displaystyle {\sum }_{i=1}^H{\displaystyle {\sum }_{j=1}^{W-1}{(I(i,j+1)-I(i,j))}^2}}},$$

(31)

$$CF=\sqrt{{\displaystyle \frac{1}{(H-1)W}}{\displaystyle {\sum }_{i=1}^{H-1}{\displaystyle {\sum }_{j=1}^W{(I(i+1,j)-I(i,j))}^2}}}.$$

(32)

4.3. Experiments on MSRS Dataset

4.3.1. Visual Quality Comparison

The MSRS dataset mainly contains nighttime urban scenes, where the fusion task requires a good balance between thermal-target enhancement and building-texture preservation. As shown in Figure 4, the compared methods show clear differences across various scenes, including dark roads, pedestrian crossings, plazas, and building entrances. Some methods highlight pedestrians but weaken surrounding structures, resulting in blurred road boundaries and softened building details. Others preserve more visible appearance information yet fail to sufficiently emphasize pedestrians in low-illumination regions. By contrast, the proposed MSF-Net produces more balanced fused results, in which pedestrians remain clearly visible while road surfaces, facades, and surrounding scene structures are better preserved.

Figure 4. Qualitative comparison of our method with 9 SOTA fusion methods on the MSRS dataset.

Figure 5 further enlarges representative local regions, where the red-box regions focus on pedestrian saliency and contour clarity, and the yellow-box regions focus on building and window-detail preservation. FusionGAN produces dark results with artifacts, weakening pedestrian contrast and blurring window boundaries. DenseFuse retains part of the scene content, but the pedestrian-background separation is still weak, and the window structures remain unclear. NestFuse and LRRNet over-smooth local details, causing vague pedestrian contours and softened façade edges. U2Fusion and RFN-Nest preserve some visible information, but the pedestrian target is still insufficiently emphasized in the dark background. FSFusion and SPDFusion improve local quality to some extent yet still show slightly unnatural brightness or texture transitions. SFINet achieves a relatively balanced result, but its pedestrian contour clarity and window-edge sharpness are still inferior to those of MSF-Net. In contrast, MSF-Net preserves clearer pedestrian silhouettes, sharper window-frame structures, and a more natural local appearance, demonstrating a better balance between thermal-target enhancement and structural-detail preservation in nighttime urban scenes.

Figure 5. Enlarged local comparisons on the MSRS dataset.

4.3.2. Quantitative Analysis

Quantitative results on the MSRS dataset are summarized in Table 1, and Figure 6 further shows the distribution of each metric across the test set. The white dot in each violin plot denotes the mean value.

Table 1. Average quantitative results in the MSRS dataset.

Methods	SD	AG	SCD	EN	Qabf	SF
FusionGAN2019	6.45131	2.78923	1.00880	5.72162	0.22524	0.03431
DenseFuse2018	7.86112	4.03377	1.50720	6.25689	0.54049	0.05051
NestFuse2020	8.23331	4.95555	1.66925	6.50137	0.66172	0.06484
U2Fusion2020	6.74172	4.35839	1.28649	5.61742	0.45469	0.05740
RFN-Nest2021	7.99375	3.47389	1.52765	6.30544	0.45619	0.03952
LRRNet2023	7.89974	4.38349	0.80705	6.27378	0.49791	0.05700
FSFusion2024	8.05644	4.84654	1.68577	6.43238	0.63169	0.06398
SPDFusion2024	8.46868	5.82902	1.65419	6.68656	0.68454	0.07480
SFINet2025	8.45115	4.90788	1.27321	6.61244	0.61367	0.06046
Ours	8.47947	6.09322	1.69260	6.73947	0.71066	0.07815

The red font indicates the best performance, and the blue font indicates the second-best performance in each metric.

Figure 6. Quantitative comparison of our method with 9 SOTA fusion methods on the MSRS dataset.

Overall, the proposed MSF-Net achieves the best performance across all six evaluation metrics on the MSRS dataset. On the average gradient (AG) metric, MSF-Net significantly outperforms all comparison methods, indicating stronger preservation of edge and texture details in the fused images. For the feature-based fusion quality index (Q_abf), MSF-Net also leads, indicating that its fused results exhibit better visual quality and cross-modal feature consistency. From the violin plot distribution, MSF-Net’s plots for multiple metrics are concentrated in the high-value region with no obvious low-score fluctuations, while some comparison methods show relatively dispersed distributions. This directly reflects that MSF-Net maintains stable, high performance across various scenes in MSRS, rather than relying on a few favorable samples. On the spatial correlation coefficient (SCD) metric, MSF-Net’s distribution is not only concentrated but also at the top of the value range, confirming its balanced ability to preserve information from both infrared and visible sources. In summary, the quantitative results on the MSRS dataset demonstrate that MSF-Net is well-suited for remote-sensing urban scenarios, showing outstanding advantages in detail preservation, information balance, and visual quality.

4.4. Experiments on M³FD Dataset

4.4.1. Visual Quality Comparison

The M³FD dataset contains more challenging multispectral scenes under degraded conditions, such as smoke, rain, haze, and low-visibility environments, where the fusion task requires not only clear thermal-target enhancement but also robust preservation of scene details under interference. Therefore, the qualitative analysis on this dataset focuses on the balance between degradation robustness, smoke or rain penetration, and detail preservation. As shown in Figure 7, the compared methods exhibit clear performance differences across various degraded scenes, including rainy roads, smoke-covered open areas, mountainous backgrounds, and low-visibility nighttime roads. Some methods can preserve thermal targets to some extent but suffer from blurred background structures and weakened scene interpretability under smoke or rain interference. Other methods retain partially visible content yet fail to sufficiently suppress degradation effects, causing reduced target clarity or over-smoothed structural details. In contrast, the proposed MSF-Net produces more balanced fused results across these challenging scenarios, where thermal targets remain more distinguishable while road textures, vegetation regions, mountainous contours, and surrounding scene structures are better preserved.

Figure 7. Qualitative comparison of our method with 9 SOTA fusion methods on the M³FD dataset.

Figure 8 further enlarges representative local regions from a typical smoke-degraded M³FD scene, where the red-box regions focus on pedestrian saliency and contour clarity under smoke interference, and the yellow-box regions focus on structural-detail preservation in degraded areas. Specifically, FusionGAN almost fails in this scene, producing severe information loss in both target and background regions. DenseFuse preserves some target response, but both the pedestrian contours and the background structures are heavily blurred. NestFuse and LRRNet show obvious smoothing effects, leading to weakened pedestrian visibility and indistinct building edges behind the smoke. U2Fusion and RFN-Nest retain part of the thermal targets, but their smoke penetration ability is still limited, and the structural details in the yellow-box regions remain unclear. FSFusion, SPDFusion, and SFINet achieve relatively improved results yet still suffer from reduced edge sharpness or insufficient scene clarity in the degraded regions. In contrast, MSF-Net preserves clearer pedestrian silhouettes in the red-box regions and recovers more recognizable building structures and edge details in the yellow-box regions, demonstrating stronger robustness in smoke-degraded environments and a better balance between target enhancement and detail preservation.

Figure 8. Enlarged local comparisons on the M³FD dataset.

4.4.2. Quantitative Analysis

Table 2 reports the average quantitative results on the M³FD dataset, and Figure 9 shows the metric distributions across all test samples. The white dot denotes the mean value.

Table 2. Average quantitative results in the M³FD dataset.

Methods	SD	AG	SCD	EN	Qabf	SF
FusionGAN2019	8.66041	3.77069	1.24463	6.72627	0.42718	0.05554
DenseFuse2018	8.59568	4.09303	1.72214	6.81115	0.54357	0.05778
NestFuse2020	9.46070	4.70140	1.65628	6.81537	0.59671	0.06950
U2Fusion2020	9.18792	5.53532	1.71948	6.93779	0.53271	0.07134
RFN-Nest2021	8.76891	3.40521	1.81287	7.06300	0.41878	0.03969
LRRNet2023	9.66308	4.65058	1.34892	6.48011	0.51766	0.06491
FSFusion2024	9.60246	4.74238	1.64982	6.82149	0.59614	0.07019
SPDFusion2024	10.05661	6.59157	1.60571	6.91599	0.57186	0.08300
SFINet2025	8.86562	5.87690	1.45366	6.74685	0.53613	0.07033
Ours	9.95718	6.62262	1.75559	7.10943	0.62603	0.08498

The red font indicates the best performance, and the blue font indicates the second-best performance in each metric.

Figure 9. Quantitative comparison of our method with 9 SOTA fusion methods on the M³FD dataset.

In terms of metric performance, MSF-Net maintains leading results on core metrics of the M³FD dataset. On the entropy (EN) metric, it ranks first among all methods, indicating that even in scenes with strong background interference, MSF-Net’s fused results retain richer valid information. On the average gradient (AG) metric, MSF-Net also performs prominently, indicating that the fused images retain clearer edge structures and better local detail under degraded conditions. Analysis of the violin plot distribution reveals that MSF-Net’s plots for multiple metrics are characterized by high-value concentration without obvious low-score outliers, while some comparison methods show relatively discrete distributions. This difference indicates that MSF-Net has stronger stability in degraded scenes and avoids performance degradation caused by environmental interference. On the spatial correlation coefficient (SCD) metric, MSF-Net’s distribution is among the top, proving that it can still balance the preservation of infrared thermal targets and visible structural information even when the quality of source images is compromised. Overall, the M³FD results suggest that MSF-Net generalizes well to degraded scenes and shows potential for practical remote-sensing applications.

4.5. Experiments on TNO Dataset

4.5.1. Visual Quality Comparison

The TNO dataset mainly contains nighttime and low-illumination surveillance scenes, where the fusion task requires both thermal-target saliency and contour integrity for reliable scene interpretation. As shown in Figure 10, the compared methods exhibit clear differences across various scenarios, including roads, urban intersections, building entrances, vegetated areas, and pedestrian walkways. Some methods enhance thermal targets but weaken surrounding scene structures, resulting in blurred roads, softened background contours, or reduced environmental interpretability. Other methods preserve part of the scene’s appearance yet fail to sufficiently highlight pedestrians or small targets in dark regions. In contrast, the proposed MSF-Net produces more balanced fused results across different scenes, where pedestrians remain more distinguishable while road boundaries, building outlines, and surrounding structures are better preserved. These characteristics are important for applications such as military surveillance and security monitoring, where both target visibility and scene awareness are critical.

Figure 10. Qualitative comparison of our method with 9 SOTA fusion methods on the TNO dataset.

Figure 11 further enlarges representative local regions from a typical TNO scene, where the red-box regions focus on the contour integrity of the helicopter tail structure, and the yellow-box regions emphasize the preservation of fine structural details such as the support frame and landing-skid components. FusionGAN produces dark fused results with blurred target boundaries, making the tail structure difficult to distinguish clearly. DenseFuse preserves part of the target intensity, but the local contours remain indistinct, and the fine support details are weak. NestFuse and LRRNet show obvious smoothing effects, leading to vague tail edges and loss of small structural components. U2Fusion and RFN-Nest retain the overall target shape but still fail to preserve sufficient sharpness in the highlighted structural regions. FSFusion, SPDFusion, and SFINet achieve relatively improved local clarity, yet some fine structural details are still softened or partially weakened. In contrast, MSF-Net preserves clearer tail boundaries and more recognizable fine support structures in both enlarged regions, demonstrating better contour integrity and structural-detail preservation for surveillance-oriented scenarios.

Figure 11. Enlarged local comparisons on the TNO dataset.

4.5.2. Quantitative Analysis

Table 3 summarizes the quantitative results on the TNO dataset, and Figure 12 shows the metric distributions across all test samples. The white dot denotes the mean value.

Table 3. Average quantitative results in the TNO dataset.

Methods	SD	AG	SCD	EN	Qabf	SF
FusionGAN2019	8.31567	3.20321	1.45580	6.50420	0.25839	0.03139
DenseFuse2018	8.83504	4.72455	1.86524	6.74024	0.46562	0.04747
NestFuse2020	8.96820	4.99184	1.78725	6.95941	0.51336	0.05309
U2Fusion2020	8.98294	6.27574	1.83306	6.90394	0.45177	0.05853
RFN-Nest2021	9.00065	3.70674	1.84668	6.93005	0.38640	0.03252
LRRNet2023	8.97169	4.92748	1.62642	6.91205	0.39545	0.04946
FSFusion2024	8.98243	4.91514	1.78509	6.94582	0.50899	0.05243
SPDFusion2024	9.14409	6.76896	1.75757	7.06529	0.51876	0.06752
SFINet2025	8.76164	5.94353	1.65723	6.91066	0.50743	0.05831
Ours	9.23977	6.08548	1.88272	7.08096	0.52206	0.06495

The red font indicates the best performance, and the blue font indicates the second-best performance in each metric.

Figure 12. Quantitative comparison of our method with 9 SOTA fusion methods on the TNO dataset.

In terms of metric performance, MSF-Net achieves leading results on key metrics of the TNO dataset. On the standard deviation (SD) metric, it outperforms all comparison methods, indicating that its fused images have a richer dynamic range and can clearly distinguish thermal targets from backgrounds in low-light environments. On the entropy (EN) metric, MSF-Net also ranks first, reflecting that its fused results retain sufficient information even in scenes with limited visual cues. Observation of the violin plot distribution shows that MSF-Net’s plots for average gradient (AG) and Q_abf are concentrated in the high-value region without obvious low-score trailing, while some comparison methods show relatively dispersed distributions. This feature indicates that MSF-Net can stably maintain detail preservation and visual consistency in low-light scenes. On the spatial correlation coefficient (SCD) metric, MSF-Net’s distribution is among the top, proving that it can effectively balance the preservation of infrared thermal information and visible structural context, meeting the core fusion requirements of the TNO dataset. The experimental results on the TNO dataset demonstrate that MSF-Net is well-adapted to low-light remote-sensing scenarios, achieving an efficient balance between thermal-target saliency and detail preservation.

4.6. Parameter Sensitivity Analysis

To further verify the rationality of the loss-weight setting, we conducted a parameter sensitivity analysis on the weighting factor λ in the proposed fusion loss using the MSRS dataset. Specifically, λ was set to [1, 5, 10, 15, 20], while all the other settings were kept unchanged. The corresponding results are shown in Figure 13, where four representative metrics, namely AG, Q_abf, SCD, and EN, are reported for evaluation.

Figure 13. Parameter sensitivity analysis of the loss weight λ on the MSRS dataset.

When λ is set to a relatively small value, the structural constraint in the loss function is comparatively weak, which limits the preservation of salient edges and fine details in the fused images. As λ increases, the fusion performance improves steadily. Among all the tested settings, λ = 10 achieves the best results on all four metrics. This indicates that the proposed loss function reaches the most favorable balance between intensity preservation and structural consistency at this setting. When λ is further increased to 15 or 20, the performance shows a slight decline. This suggests that over-emphasizing the structural term may weaken the overall balance of the fused results, although the method still remains relatively stable within a reasonable parameter range. Therefore, λ = 10 is selected as the default setting in all experiments.

4.7. Ablation Studies

To verify the effectiveness of the core designs of MSF-Net, including multi-scale frequency-aware encoding, structure-guided refinement, and hierarchical fusion, a systematic ablation study is conducted on the MSRS dataset. The complete MSF-Net serves as the baseline model, and three variant models are constructed by removing key modules one by one to isolate and analyze the contribution of each component from both quantitative and qualitative perspectives. All ablation experiments maintain consistent training parameters (learning rate, batch size, number of iterations, loss function weight) with the baseline to ensure fairness. Three ablation variants are constructed for comparison. V1 removes the frequency interaction component, V2 removes the structure-guided refinement component, and V3 simplifies the hierarchical fusion strategy by discarding cross-scale aggregation. All variants are trained under the same settings as the baseline, and six objective metrics are used for evaluation.

Table 4 presents the quantitative test results of the baseline model and each ablation variant (bold values indicate the best performance):

Table 4. Results of the ablation experiments.

Model	SD	AG	SCD	EN	Qabf	SF
w/o SFIM	8.01259	4.98768	1.58926	6.52184	0.63248	0.06541
w/o SGFRM	8.13579	5.20138	1.70151	6.59346	0.64872	0.06893
Simplified HFFM	8.20191	5.43272	1.63188	6.62051	0.66535	0.07015
Baseline	8.47947	6.09322	1.69260	6.73947	0.71066	0.07815

Boldface is used to indicate the best result for each metric.

Quantitative results indicate that V1 (w/o SFIM) exhibits the most significant declines in average gradient and feature-based fusion quality index, verifying the core role of SFIM in modeling global long-range dependencies through the Fourier transform, which compensates for the local receptive field limitation of pure spatial convolution. V2 (w/o SGFRM) has lower average gradient and spatial-frequency metrics than the baseline, demonstrating that this module effectively compensates for the weakening of local details caused by frequency-domain fusion and enhances edge and texture information through structure-guided residual modulation. V3 (Simplified HFFM) shows obvious decreases in spatial correlation coefficient and entropy after canceling cross-scale aggregation, proving that its design of “cross-modal fusion at the same scale + top-down semantic transmission” can efficiently integrate complementary information of coarse-scale thermal targets and fine-scale textures.

In summary, the ablation results confirm that each component contributes positively to the final performance, and the complete MSF-Net achieves the best overall balance among detail preservation, information integration, and visual consistency.

4.8. Computational Complexity Analysis

To further evaluate the efficiency of the proposed MSF-Net, we compare its computational complexity with several representative methods in terms of parameter count, FLOPs, and inference time, as shown in Table 5. U2Fusion has the smallest model size and FLOPs, whereas RFN-Nest exhibits the highest computational cost and the slowest inference speed. SFINet achieves the fastest inference speed, but its parameter count is the largest among the compared methods. By contrast, MSF-Net uses 11.871 M parameters and 44.329 G FLOPs, which are lower than those of SFINet and RFN-Nest, while its inference time is 0.104 s/image, faster than U2Fusion, RFN-Nest, and SPDFusion, and only slightly slower than SFINet. These results show that MSF-Net achieves a favorable trade-off between computational efficiency and model complexity.

Table 5. Computational complexity comparison of different fusion methods in terms of parameter count, FLOPs, and inference time.

Method	Parameters (m)	FLOPs (G)	Inference Time (s)
U2Fusion2020	0.783	23.876	0.134
RFN-Nest2021	7.932	98.125	1.655
SPDFusion2024	2.483	48.235	0.183
SFINet2025	16.894	66.893	0.085
Ours	11.871	44.329	0.104

5. Discussion

Although the proposed MSF-Net achieves strong performance on the MSRS, M³FD, and TNO datasets, several limitations remain. As shown in Figure 14, the current model may still encounter difficulties in some challenging lighting conditions. In particular, under rainy scenes with wet-road reflections or strong-light interference, the fused results may exhibit locally saturated high-brightness regions, weakened structural boundaries, or reduced local contrast. In such cases, the overall scene information is still preserved, but the discrimination of target contours and fine details around reflective or over-exposed regions can become less reliable. These failure cases suggest that the current feature interaction and refinement strategy still has room for improvement when handling complex illumination patterns, especially those caused by specular reflection, glare, and strong local light sources. In addition, for small or weak targets embedded in cluttered backgrounds, the enhancement effect may still be limited in some cases, indicating that more target-aware feature modulation could be beneficial.

Figure 14. Representative failure cases under rain-induced reflection and strong-light interference.

Moreover, the proposed multi-scale frequency-aware framework introduces additional computational cost compared with simpler convolution-based fusion models. Although the complexity analysis shows that MSF-Net achieves a favorable trade-off between model complexity and runtime efficiency, its scalability on native high-resolution remote-sensing imagery still requires further investigation. These observations suggest several directions for future work, including more robust illumination-aware feature interaction, stronger target-aware enhancement strategies, and lightweight optimization for high-resolution remote-sensing applications.

6. Conclusions

This paper proposes MSF-Net, a multi-scale frequency-aware representation learning framework for infrared and visible image fusion in remote-sensing scenarios. To address the difficulty of simultaneously preserving infrared thermal saliency and visible structural details, the proposed method introduces a hybrid design that combines spatial-domain local representation learning with frequency-domain global interaction modeling. Specifically, the proposed framework integrates the spatial-frequency interaction module (SFIM) to model global complementary information, the structure-guided feature refinement module (SGFRM) to enhance local structural fidelity, and the hierarchical feature fusion module (HFFM) to progressively aggregate cross-modal and cross-scale features. Through this design, MSF-Net is able to achieve a better balance between target enhancement, detail preservation, and visual consistency.

Extensive experiments on the MSRS, M³FD, and TNO datasets demonstrate the effectiveness of the proposed method in different infrared-visible fusion scenarios. On the MSRS dataset, MSF-Net shows a strong capability in balancing building-texture preservation and thermal-target enhancement in nighttime urban scenes. On the M³FD dataset, the proposed method exhibits better robustness under smoke, rain, haze, and other degraded conditions, with improved target visibility and scene-detail preservation. On the TNO dataset, MSF-Net produces fused results with clearer target saliency and contour integrity, which is beneficial for low-illumination surveillance scenarios. In addition to the qualitative improvements, the quantitative results also verify that MSF-Net achieves competitive and often superior performance compared with representative fusion methods.

Furthermore, the ablation studies confirm the effectiveness of the main components of the proposed framework, and the computational complexity analysis shows that MSF-Net achieves a favorable trade-off between fusion performance and computational efficiency. Overall, these results indicate that the proposed method is a promising solution for infrared and visible image fusion in remote-sensing applications.