Significance-Preserving Progressive Network for Infrared and Visible Image Fusion

Highlights

What are the main findings?

• A significance-preserving progressive network (SiPFusion) is proposed, featuring a progressive extraction process that integrates hierarchical multi-scale local features and Transformer-based global dependencies.
• A novel significance-preserving fusion module with a spatial selection mechanism and a complementary significance similarity loss function were uniquely designed.

What are the implications of the main finding?

• The progressive feature extraction and spatial selection mechanism effectively coordinate heterogeneous global-local features, ensuring the adaptive integration of target saliency and texture details.
• The designed modules improve image fidelity and structural consistency in complex scenes, enabling SiPFusion to achieve strong overall performance across three benchmarks, with leading results on most metrics while maintaining competitive artifact suppression performance.

Abstract

Fusing infrared and visible images can effectively compensate for the inherent limitations of each modality in different scenes, resulting in fused images that contain richer information. However, existing methods often struggle to balance global dependency modeling with local detail preservation and to effectively coordinate heterogeneous local and global features during fusion. To address these issues, this paper proposes a Significance-Preserving Progressive Fusion Network (SiPFusion). First, a progressive feature extraction framework was designed, which hierarchically extracts multi-scale local features using CNNs and then models long-range dependencies across scales via a Transformer-based global module. To adaptively integrate local-global complementary features, a significance-preserving fusion module was designed to obtain significance attention maps with a spatial selection mechanism, enabling dynamic fusion of multi-source features. Furthermore, we propose a significance similarity loss function that leverages intermediate feature guidance to enhance structural consistency and preserve salient-region information in the fused image. Extensive experiments on the MSRS, RoadScene, and TNO datasets demonstrate that SiPFusion achieves competitive visual quality and strong overall quantitative performance against 15 state-of-the-art fusion methods, obtaining leading results on most evaluated metrics.

1. Introduction

Infrared and visible image fusion is an important branch of image fusion and a research hotspot worldwide, playing a crucial role in both civilian and military applications [1,2,3]. Visible images are generated based on the reflection of light, with a wavelength range of approximately 380 nm to 780 nm aligning with human visual perception [4]. They provide detailed resolution and abundant texture details. However, they are highly vulnerable to challenging weather conditions, including poor visibility, nighttime [5], and fog. In contrast, infrared images, generated based on the radiation emitted by objects with a wavelength range of about 8 µm to 14 µm [6], can capture thermal targets in all weather conditions, due to their independence from external illumination sources. However, infrared images suffer from low resolution, high noise, poor contrast, and blurry edges, making it difficult to reveal texture details and providing poor visual quality [7,8]. By fusing these two types of images, the expressive power of the scene is greatly enhanced. The fused image not only shows infrared targets clearly but also provides richer texture information, significantly improving the overall visual quality. Therefore, infrared and visible image fusion technology is widely applied in many fields, like tracking [9], surveillance [10,11,12], and target detection [13,14].

Existing multi-source image fusion methods can be broadly categorized into traditional image fusion methods and deep learning-based image fusion methods. Traditional fusion methods primarily include multi-scale transform-based fusion methods [15,16], sparse representation-based fusion methods [17,18], subspace-based fusion frameworks [19,20], and saliency-based fusion methods [21,22]. These methods perform well in specific scenarios but often require adjustments to fusion strategies in response to environmental factors such as illumination and background variations. Consequently, traditional image fusion methods exhibit limited generalizability and require considerable time for the manual design of appropriate fusion workflows [23].

In recent years, deep learning-based methods for infrared and visible image fusion have attracted extensive attention within the image fusion community. Compared with traditional image fusion methods, deep learning-based approaches offer greater adaptability and enable end-to-end optimization, thereby reducing their reliance on manually designed fusion rules [3]. By designing effective network architectures and loss functions, these methods guide models to generate fused images enriched with more comprehensive visual information. Moreover, the robustness of these algorithms can be further enhanced by expanding the quantity and diversity of training datasets. Current deep learning-based image fusion algorithms primarily focus on addressing three critical challenges in the fusion process: feature extraction, feature fusion, and image reconstruction. Based on the adopted network architectures, deep learning-based infrared and visible image fusion methods can be categorized into three main categories: convolutional neural network (CNN)-based fusion methods [24,25,26,27], generative adversarial network (GAN)-based fusion methods [28,29,30,31,32] and autoencoder (AE)-based fusion methods [33,34,35,36,37,38,39,40,41,42,43,44,45].

CNN-based image fusion methods leverage convolutional kernels for effective local and multi-level feature extraction. In [46], an unsupervised end-to-end network was proposed to estimate the importance of source images, enabling adaptive information retention. Tang et al. [27] introduced SeAFusion, a semantic-aware real-time fusion network that connected image fusion with high-level vision tasks. However, CNN-based methods struggle with the lack of ground truth, as their performance heavily depends on loss functions and network design. To address this, GAN-based methods, such as [29], used adversarial generation to reduce the impact of loss functions and network architecture. Ma et al. [28] introduced GANMcC with multi-class constraints, which improved contrast and texture detail in fused images.

Despite providing a viable solution for image fusion, GAN-based methods face challenges such as high training complexity, instability during training and resource inefficiency. In contrast, AE-based image fusion methods circumvent these issues and offer advantages such as flexible fusion strategies, strong adaptability and robust performance, making them a more practical choice in certain applications. AE-based methods typically consist of an encoder and a decoder. Initially, a pre-trained encoder is employed to extract features from multi-source images. These extracted deep features are then fused using manually designed fusion strategies. Finally, a decoder is utilized to reconstruct the image from the fused features.

Currently, most AE methods primarily use CNNs for feature extraction, which inherently limits the network’s ability to capture global features. As a result, some researchers have introduced Transformers into the design of feature extraction networks. The Transformer, being an effective network module for extracting long-distance dependencies, performs well in capturing global information in images [47]. However, since the Transformer was originally designed for natural language processing, this isolated usage increases global information but can lead to the loss of local information [48]. Consequently, the integration of CNNs and Transformers has emerged as a promising research direction, aiming to combine the advantages of both local and global feature representations. However, how to effectively coordinate the interaction between these heterogeneous features remains a critical challenge. Some studies, such as CDDFuse [41,49] employ a dual-branch architecture combining a CNN and Transformer to capture both local and global features independently. Other approaches apply a Transformer as a fusion module to establish interactions between different modalities or feature sources [50,51]. In addition, some recent studies [52,53] further exploit multi-scale receptive field features by feeding them into different Transformer modules, aiming to enrich the representation diversity. Despite these advances, most existing methods treat features of different spatial scales or receptive fields independently, without explicitly modeling the global structural relationships across feature scales and hierarchies. This may result in ineffective fusion of texture and structure information, compromising the visual quality and structural consistency of the fused image.

To address these challenges, this study proposes a progressive network architecture that first employs CNNs to extract hierarchical and scale-diverse local features and subsequently leverages a Transformer to model global dependencies across the resulting representations. This gradual integration allows the Transformer module to further process the extracted features, capturing both fine-grained details and long-range contextual relationships. Specifically, the module performs spatial-domain and scale-aware transformations to integrate complementary information while preserving spatial structural cues. By exploiting the self-attention mechanism, it facilitates the modeling of structural dependencies among important regions at different scales, thereby enhancing global contextual representation. Not only can it extract global information from features with different receptive fields, but it also lays the foundation for the alignment of multi-scale features in the subsequent fusion process. Additionally, a significance-preserving fusion module is introduced. This module dynamically adjusts the fusion weights at the feature level using a spatial selection mechanism to generate significance attention maps, which preserve the effective information in the salient regions of the source images. Furthermore, we propose a significance similarity metric. It adaptively computes the global structural correlation of images based on the extracted global features and calculates the similarity loss of salient regions using the attention maps from the fusion module. This approach ensures the retention of effective information in important regions, thereby meeting the requirements of multimodal image fusion tasks in complex scenes.

The central distinction of SiPFusion lies in the progressive coupling of these stages. The network first uses convolution-based multi-scale local feature extraction to organize fine textures, object boundaries, and intermediate-scale structures and then applies Transformer-based global feature extraction to model long-range contextual dependencies on the enriched local representations. This progressive local-to-global pathway enables the global branch to benefit from reliable local details rather than operating on isolated or single-scale features. The significance-preserving fusion module further generates spatial selection masks for the visible/infrared local and global feature branches, allowing adaptive feature selection across both modalities and representation levels. Moreover, the local selection masks and global features are reused in the significance similarity loss so that the feature-level fusion decisions are explicitly linked with output-level salient-region preservation and global structural consistency.

The main contributions of this paper can be concluded as follows:

• We propose a fusion framework based on progressive feature extraction and significance preserving for the task of visible and infrared image fusion. Unlike existing fusion strategies, this framework explicitly considers the adaptive integration of multi-scale local information with the globally structured representations of multi-scale local features, rather than merely reconstructing fused data based on the global correlations of local features. This strategy enables comprehensive cross-scale feature interaction and improves global contextual representation, facilitating the retention of effective information in important regions.
• A significance-preserving fusion module is employed to adaptively fuse the extracted features. In this module, a spatial selection mechanism is used to perform adaptive weighting between features from different receptive fields of the multimodal inputs, improving the network’s capability to merge the most critical spatial context areas efficiently.
• We designed a significance similarity metric in the loss function, which guides the global and salient region similarity of the fused image based on intermediate features. This improves structural consistency and enhances the perceptual quality of the fused image.

The rest of this article is organized as follows. In Section 2, we will review related works, including AE-based image fusion methods and Transformer-based image fusion methods. In Section 3, we will present our proposed method in detail. Section 4 will present experimental results, including experimental settings and both qualitative and quantitative comparisons, along with ablation studies. Section 5 will provide a conclusion and summary of the paper.

2. Related Works

2.1. AE-Based Image Fusion Methods

Autoencoder (AE)-based fusion methods have attracted considerable attention due to their powerful feature representation capabilities and end-to-end training flexibility. In the image fusion process, the autoencoder extracts and compresses the features of both the infrared and visible images, capturing their key characteristics. These features are then fused to generate a combined feature representation. Specifically, infrared images usually provide thermal radiation information in low-light or nighttime environments, while visible images contain rich visual details under normal lighting conditions.

In the encoding networks of these methods, various architectural enhancements have been proposed to improve feature extraction. For instance, DenseFuse [33], NestFuse [34], and RFN-Nest [35] integrated dense connections, nested skip paths, and residual dense blocks, respectively, to enhance feature reuse and gradient flow. More recent designs, such as Res2Fusion [36] and MTDFusion [37], employed Res2Net modules and triple-dense structures to further strengthen multi-scale feature representation.

To better capture image information for varying frequency and modality distributions, decomposition-based strategies were introduced [54]. DIDFuse [38] separated source images into high-frequency and low-frequency components, while DRF [39] decomposed inputs into scene-relevant and modality-specific components. MACCNet [40] extended this idea with a dual-branch encoder combining multi-scale attention and cross-convolution to extract complementary features from infrared and visible modalities.

In parallel, recent works have explored two-stage training strategies to improve fusion performance by decoupling different learning objectives. For instance, FTSFN [55] employs a two-stage fusion strategy in which modality-specific features are first transferred and then enhanced through a supplementary module. This separation allows the network to better preserve complementary details from both infrared and visible images. Similarly, DANet [56] adopts a two-stage training paradigm that integrates a diffusion model into an autoencoder framework. Although these methods demonstrate impressive results, the staged training schemes often increase the training complexity and may suffer from suboptimal global-local interaction due to their sequential optimization process. In contrast, our method adopts a progressive network that unifies local and global feature modeling, enabling more coordinated feature modeling across modalities and scales.

2.2. Transformer-Based Image Fusion Methods

A Transformer is a deep learning architecture based on the self-attention mechanism that completely discards recurrent and convolutional structures. By adopting self-attention, the Transformer enables parallel computation, significantly improving training efficiency. Due to their highly parallelizable computation, strong ability to model long-range dependencies, and flexible input–output structure, Transformer-based image fusion methods have attracted increasing attention.

To overcome the limited receptive field of convolutional networks, several studies have incorporated Transformers into image fusion frameworks. THFuse [57] integrates Transformer blocks into a CNN-based pipeline to enhance the modeling of global and local structures. STFNet [42] replaces handcrafted fusion strategies with a Transformer-based feature fusion module that explicitly models contextual relationships. DATFuse [58] employs a dual-attention Transformer to preserve long-range complementary information across modalities, whereas YDTR [59] adopts a Y-shaped dual-branch architecture with a dynamic Transformer to extract thermal and texture information. Trans2Fuse [60] further combines self-supervised learning with a Transformer-based fusion network to improve feature representation.

Recent studies have explored more specialized Transformer-based fusion designs. ATFusion [61] employs alternating cross-attention to capture inter-modal discrepancies and complementary information. The one-stream one-stage pure Transformer framework [62] integrates feature extraction and fusion into a unified Transformer pipeline. D2Fusion [63] explicitly decomposes and fuses high- and low-frequency components in both the spatial and frequency domains. In addition, recent Mamba-based approaches [64] have been investigated as alternative mechanisms for modeling long-range dependencies.

Existing hybrid fusion frameworks can be further distinguished by how they organize local and global representations and establish cross-modal interactions. CDDFuse [41] employs parallel Lite-Transformer and invertible CNN branches to represent low-frequency shared information and high-frequency modality-specific information with a correlation-driven objective. SwinFusion [65] models cross-domain interactions using Swin Transformer blocks. CrossFuse [66] introduces cross-attention between deep modality features extracted by an autoencoder, whereas PPIFuse [67] combines modality-specific physical priors with multi-scale cross-attention. TransFusion [68] integrates cross-modal channel perception and deep-feature-guided spatial attention with a feature mutual-information objective. From the perspective of degradation-robust representation learning, URFusion [69] separates intrinsic content modeling from appearance representation assignment.

Taken together, these methods advance infrared and visible image fusion through representation decomposition, cross-modal interaction, physical-prior modeling, and degradation-robust learning. Building on these advances, SiPFusion approaches fusion from the perspective of cross-stage coordination. Its progressive local-to-global feature modeling provides representations for significance-aware feature selection, while the resulting intermediate masks and global features are further employed by S-SIM to guide output-level information preservation. This mechanism establishes a direct connection among feature extraction, feature fusion, and learning supervision.

3. The Proposed Method

In this part, the proposed significance-preserving progressive network named SiPFusion will be introduced in detail, including the architecture of the network, the encoder, the significance-preserving fusion module, the decoder and the loss function.

3.1. Network Framework

The overall architecture of the proposed SiPFusion network is shown in Figure 1.

Figure 1. The overall framework of the proposed SiPFusion. The model contains a multi-scale local feature extraction (MLFE) module and a Transformer-based global feature extraction (TGFE) module. The multi-scale local feature extraction module utilizes MP blocks to extract local features from the source images. The Transformer-based global feature extraction module consists of several Transformer layers, which capture global dependencies based on the local features. The features extracted through the progressive structure are then fused using the fusion module, and finally, the fused image is reconstructed through the decoder.

The network consists of three main components: the encoder, the fusion module, and the decoder. Given an RGB visible image $I_{vi}^{RGB}\in {\mathbb{R}}^{H\times W\times 3}$, we first convert it to the YCbCr color space and use only its luminance channel as the visible input to the fusion network. For simplicity, this single-channel luminance component is denoted as $I_{vi}\in {\mathbb{R}}^{H\times W\times 1}$, while the infrared input is denoted as $I_{ir}\in {\mathbb{R}}^{H\times W\times 1}$. Both $I_{ir}$ and $I_{vi}$ are passed through the multi-scale local feature extraction module to obtain feature ${\mathsf{\Phi }}_I^L,{\mathsf{\Phi }}_V^L$, which can be expressed as

$${\mathsf{\Phi }}_I^L=\mathrm{MLFE}\left(I_{ir}\right),$$

(1)

$${\mathsf{\Phi }}_V^L=\mathrm{MLFE}\left(I_{vi}\right),$$

(2)

where ${\mathsf{\Phi }}_I^L$ and ${\mathsf{\Phi }}_V^L$ represent the final local feature representations of the infrared and visible branches used for subsequent feature fusion. During the MLFE process, the outputs of the B multi-scale perception (MP) blocks are concatenated to form modality-specific intermediate multi-scale feature maps, denoted as ${\mathbf{F}}_{ir}$ and ${\mathbf{F}}_{vi}$ for the infrared and visible branches, respectively. These concatenated feature maps are then fed into the Transformer-based global feature extraction module for further global dependency modeling, which can be expressed as

$${\mathsf{\Phi }}_I^G=\mathrm{TGFE}\left({\mathbf{F}}_{ir}\right),$$

(3)

$${\mathsf{\Phi }}_V^G=\mathrm{TGFE}\left({\mathbf{F}}_{vi}\right),$$

(4)

where TGFE indicates the global feature extraction operation of this module, which consists of L Transformer layers. This operation ultimately produces the feature tensor that contains both local and global information. Then, the four feature maps generated by the two modules are concatenated, which can be denoted as

$$\mathsf{\Phi }=\mathrm{Concat}\left({\mathsf{\Phi }}_I^L,{\mathsf{\Phi }}_V^L,{\mathsf{\Phi }}_I^G,{\mathsf{\Phi }}_V^G\right),$$

(5)

where Concat refers to the concatenation operation. Afterward, the concatenated feature map $\mathsf{\Phi }$ is processed through the significance-preserving fusion (SPF) module to obtain the final fused feature $\widehat{\mathbf{F}}$. This procedure can be formulated as

$$\widehat{\mathbf{F}}=\mathrm{SPF}\left(\mathsf{\Phi }\right)$$

(6)

Finally, the fused feature is reconstructed to the original size using a decoder, resulting in the final fused image $I_f$:

$$I_f={\mathsf{\psi }}_{\mathrm{D}}\left(\widehat{\mathbf{F}}\right),$$

(7)

where ${\mathsf{\psi }}_{\mathrm{D}}$ represents the decoder block.

3.2. Progressive Structure-Based Encoder Network

Considering that both local salient target information and the global correlations in infrared and visible images play a crucial role in understanding the image scene and maintaining global structural consistency, we designed a network architecture in the encoder that combines a multi-scale local feature extraction module with a Transformer-based global feature extraction module. This approach further enhances the network’s ability to simultaneously integrate both the detailed features and overall characteristics of the source images.

3.2.1. Multi-Scale Local Feature Extraction Module

As the local feature extractor of SiPFusion, the MLFE module aims to capture local contextual information at multiple receptive-field scales, with particular emphasis on salient targets and fine texture details. Given a source image $I\in \{I_{ir},I_{vi}\}$, a shallow $3\times 3$ convolution first projects the input into a feature representation with a base channel width of $C=64$. The resulting feature is then progressively processed by B sequential MP blocks. Instead of using only the output of the last block, MLFE retains the output of each MP block and concatenates them along the channel dimension to aggregate features extracted at different depths. This process is formulated as

$$\begin{array}{c}{\mathbf{F}}^{\left(0\right)}={\mathcal{C}}_{3\times 3}\left(I\right),\hfill \\ {\mathbf{F}}^{\left(i\right)}=\mathrm{MP}\left({\mathbf{F}}^{(i-1)}\right),i=1,2,\dots ,B,\hfill \\ \mathbf{F}=\mathrm{Concat}\left({\mathbf{F}}^{\left(1\right)},{\mathbf{F}}^{\left(2\right)},\dots ,{\mathbf{F}}^{\left(B\right)}\right),\hfill \end{array}$$

(8)

where ${\mathcal{C}}_{3\times 3}(·)$ denotes the shallow feature projection, ${\mathbf{F}}^{\left(i\right)}$ represents the output of the i-th MP block, and $\mathbf{F}$ denotes the concatenated multi-scale local feature for a single modality branch. When the input image is $I_{ir}$ or $I_{vi}$, this concatenated feature is denoted as ${\mathbf{F}}_{ir}$ or ${\mathbf{F}}_{vi}$, respectively, and is used as the input to TGFE.

As illustrated in Figure 2, each MP block contains three parallel convolutional branches with kernel sizes of $1\times 1$, $3\times 3$, and $5\times 5$, respectively. Their output channels are set to C, $C/2$, and $C/4$. Since $C=64$ in our experiments, the three branches produce 64, 32, and 16 channels, respectively. All branches use a stride of 1 with appropriate padding to preserve the spatial resolution of the feature maps. The outputs of the three branches are concatenated along the channel dimension, producing a 112-channel intermediate feature. A subsequent $1\times 1$ convolution projects this feature back to 64 channels, after which a residual connection is applied between the projected feature and the block input. Therefore, each MP block preserves the spatial and channel dimensions of its input and produces ${\mathbf{F}}^{\left(i\right)}\in {\mathbb{R}}^{H\times W\times 64}$. In our implementation, B is set to 5. Concatenating the outputs of the five MP blocks therefore produces $\mathbf{F}\in {\mathbb{R}}^{H\times W\times (B·C)}={\mathbb{R}}^{H\times W\times (5·64)}={\mathbb{R}}^{H\times W\times 320}$. The resulting $H\times W\times 320$ feature tensor is subsequently fed into the global feature extraction stage.

Figure 2. The architecture of MP block.

3.2.2. Transformer-Based Global Feature Extraction Module

The MLFE module is a CNN-based local extractor and thus has inherent locality, which limits its ability to model global dependencies across the entire image, especially for distant targets. To enhance global dependency learning, we further introduce a TGFE module.

In our implementation, the concatenated MLFE features ${\mathbf{F}}_{ir}$ and ${\mathbf{F}}_{vi}$ are fed into TGFE to extract global representations for the infrared and visible branches, respectively. TGFE starts with a downsampling pyramid that progressively reduces the spatial resolution while expanding the receptive field, producing three feature maps at resolutions $H/2$, $H/4$, and $H/16$ (with channel sizes 640, 1024, and 1024, respectively). The $H/8$ level is intentionally omitted to limit further growth in the number of spatial tokens used for global self-attention. During decoding, the $H/16$ feature is directly upsampled to $H/4$ and fused with the corresponding skip feature, while intermediate-scale spatial information is retained through the $H/4$ and $H/2$ skip connections. The coarsest feature map is then projected by patch embedding to a hidden dimension D (set to 128 in our configuration) and reshaped into a token sequence of length $N_{tok}=(H/16)·(W/16)$. The TGFE module contains L Transformer layers (LayerNorm, multi-head attention, and MLP) to capture long-range dependencies for global contextual modeling.

After Transformer encoding, the token sequence is reshaped back to a 2D feature map of size $(H/16,W/16,D)$ and decoded by progressive upsampling with three-stage skip fusion. Specifically, at each stage, the upsampled feature map is concatenated with the corresponding downsampling-stage feature map at the same spatial resolution and then projected by convolution (denoted as “C: Concat & Conv” in Figure 1). The decoding path restores the resolution in the order $(H/16,W/16)\to (H/4,W/4)\to (H/2,W/2)\to (H,W)$ and finally outputs a full-resolution global feature map ${\mathsf{\Phi }}^G\in {\mathbb{R}}^{H\times W\times 64}$ for each modality (${\mathsf{\Phi }}_I^G$ and ${\mathsf{\Phi }}_V^G$).

3.3. Significance-Preserving Fusion Module

In terms of fusion strategy, we adopt a significance-preserving mechanism to maintain the salient regions in the feature maps of the multi-source images, thereby improving the network’s capability to concentrate on the most informative spatial context regions. The structure of SPF module is illustrated in Figure 3.

Figure 3. The architecture of SPF Block.

Initially, as shown in Equation (5), we concatenate the four feature maps of the infrared and visible images obtained from the progressive structure. Then, operations are performed on $\mathsf{\Phi }$ through average pooling and max pooling at the feature channel level:

$$\begin{array}{cc}\hfill {\mathbf{PM}}_{avg}& ={\mathcal{P}}_{avg}\left(\mathsf{\Phi }\right),\hfill \\ \hfill {\mathbf{PM}}_{max}& ={\mathcal{P}}_{max}\left(\mathsf{\Phi }\right),\hfill \end{array}$$

(9)

where ${\mathcal{P}}_{avg}(·)$ and ${\mathcal{P}}_{max}(·)$ represent the average pooling operation and the max pooling operation, respectively, while ${\mathbf{PM}}_{avg}$ and ${\mathbf{PM}}_{max}$ are their corresponding saliency feature descriptors. To facilitate the interaction of salient information between different source features, the saliency feature descriptors ${\mathbf{PM}}_{avg}$ and ${\mathbf{PM}}_{max}$ are aggregated into a single significance feature descriptor. This aggregated descriptor is then transformed into the corresponding $N_{map}$ significance attention maps through a convolution operation ${\mathcal{C}}^{2\to N_{map}}(·)$, which can be described as

$$\widetilde{\mathbf{PM}}={\mathcal{C}}^{2\to N_{map}}\left(\mathrm{Concat}\left({\mathbf{PM}}_{avg},{\mathbf{PM}}_{max}\right)\right),$$

(10)

where $N_{map}=4$, and $\widetilde{\mathbf{PM}}$ represents the four significance attention maps generated for the local and global features of the two modalities. Specifically, the four attention maps are denoted as ${\widetilde{\mathbf{PM}}}_V^L$, ${\widetilde{\mathbf{PM}}}_V^G$, ${\widetilde{\mathbf{PM}}}_I^L$, and ${\widetilde{\mathbf{PM}}}_I^G$, which correspond one-to-one to ${\mathsf{\Phi }}_V^L$, ${\mathsf{\Phi }}_V^G$, ${\mathsf{\Phi }}_I^L$, and ${\mathsf{\Phi }}_I^G$, respectively. The Sigmoid activation function is then applied to each attention map to obtain its corresponding significance selection mask:

$${\widehat{\mathbf{PM}}}_m^s=\rho \left({\widetilde{\mathbf{PM}}}_m^s\right),m\in \{V,I\},s\in \{L,G\},$$

(11)

where m denotes the visible or infrared modality, s denotes the local or global feature level, and $\rho (·)$ denotes the Sigmoid activation function.

The four extracted features are weighted by their corresponding selection masks and then summed to obtain the fused feature tensor:

$$\begin{array}{c}\hfill \widehat{\mathbf{F}}={\widehat{\mathbf{PM}}}_V^L\odot {\mathsf{\Phi }}_V^L+{\widehat{\mathbf{PM}}}_V^G\odot {\mathsf{\Phi }}_V^G+{\widehat{\mathbf{PM}}}_I^L\odot {\mathsf{\Phi }}_I^L+{\widehat{\mathbf{PM}}}_I^G\odot {\mathsf{\Phi }}_I^G,\end{array}$$

(12)

where ⊙ denotes element-wise multiplication. Each selection mask weights only its corresponding feature map, and all four masks participate in the adaptive feature fusion performed by SPF.

Although SPF adopts an attention-based weighting form, its purpose is not generic feature refinement but adaptive multi-branch feature fusion. Specifically, SPF receives the visible local feature, visible global feature, infrared local feature, and infrared global feature and generates branch-specific spatial selection maps to adaptively select complementary information across both modality and representation level. Moreover, the local selection maps corresponding to the visible and infrared local features are reused by the SRS term in S-SIM, thereby linking feature-level fusion with output-level salient-region preservation. This design makes SPF a fusion-oriented selection module rather than a standalone attention-refinement block.

3.4. Decoder

The decoder performs a decoding operation on the features fused by the SPF module. The decoder consists of $1\times 1$ and $3\times 3$ convolution layers, a Leaky ReLU activation function, followed by another $3\times 3$ convolution layer with a Sigmoid activation function. This process ultimately restores the features to their original spatial dimensions and outputs a single-channel fused luminance image $I_f$. During inference, $I_f$ is combined with the $Cb$ and $Cr$ channels retained from the original visible image. The resulting YCbCr image is then converted back to the RGB color space to obtain the final color fused image.

3.5. Loss Function

The loss function plays a guiding role during the training phase of the network. In this paper, we propose a hybrid loss function, which includes gradient loss, mean squared error (MSE) loss and a significance similarity (S-SIM) loss that we propose. The formula for the total loss function is as follows:

$$L={\lambda }_1{L}_{grad}+{\lambda }_2{L}_{MSE}+{\lambda }_3{L}_{S\text{-}SIM},$$

(13)

where ${\lambda }_1$,${\lambda }_2$, and ${\lambda }_3$ are weighting coefficients used to balance the numerical contributions of the gradient, MSE, and S-SIM losses, respectively.

Gradient loss is calculated by computing the maximum gradient between the fused image and the two source images. The gradient values at corresponding locations are subtracted, and the squared differences are averaged across all pixels. Gradient loss helps the fused image retain the detail and texture information from the source images. The formula can be expressed as

$$L_{grad}={\displaystyle \frac{1}{HW}}{\displaystyle \sum _{i=1}^{HW}}{\left(▽I_f\left(i\right)-\max \left(▽I_{ir}\left(i\right),▽I_{vi}\left(i\right)\right)\right)}^2,$$

(14)

where $▽I\left(i\right)$ represents the gradient of the image at pixel i, and $HW$ denotes the total number of pixels in the image.

MSE loss can evaluate the fusion performance by calculating the squared differences between the fusion image and the source images at each pixel position, which has certain advantages for network training. It can be expressed as

$$\begin{array}{cc}\hfill L_{MSE}=& {\alpha }_1\left({\displaystyle \frac{1}{HW}}{\displaystyle \sum _{i=1}^{HW}}{\left(I_f\left(i\right)-I_{ir}\left(i\right)\right)}^2\right)\hfill \\ \hfill & +{\alpha }_2\left({\displaystyle \frac{1}{HW}}{\displaystyle \sum _{i=1}^{HW}}{\left(I_f\left(i\right)-I_{vi}\left(i\right)\right)}^2\right),\hfill \end{array}$$

(15)

where ${\alpha }_1,{\alpha }_2$ represent the balancing parameters for the MSE of the infrared and visible images. In order to ensure that the fused image retains as much information as possible from both source images, we assume that the contributions of the two source images are equal. Therefore, both ${\alpha }_1$ and ${\alpha }_2$ are set to 1.

In order to better preserve the global structural similarity and the effective information of the salient regions in the image, we designed the S-SIM loss. The S-SIM loss function primarily consists of two components: the salient region similarity (SRS) loss and the global structural similarity (GSS) loss. Its formulation is as follows:

$$\begin{array}{cc}\hfill L_{S\text{-}SIM}=& {\beta }_1{L}_{SRS}+{\beta }_2{L}_{GSS},\hfill \end{array}$$

(16)

where ${\beta }_1$ and ${\beta }_2$ are weighting coefficients used to balance the numerical contributions of the SRS and GSS terms, respectively.

The goal of the SRS loss is to preserve informative intensity content in salient local regions. Among the four selection masks generated by SPF, ${\widehat{\mathbf{PM}}}_V^L$ and ${\widehat{\mathbf{PM}}}_I^L$, which correspond to the visible and infrared local features ${\mathsf{\Phi }}_V^L$ and ${\mathsf{\Phi }}_I^L$, respectively, are directly reused to construct the spatial source-selection weights ${\mathbf{W}}_1$ and ${\mathbf{W}}_2$. The weights are defined as

$$\left\{\begin{array}{c}{\mathbf{W}}_1=\delta \left({\widehat{\mathbf{PM}}}_V^L-{\widehat{\mathbf{PM}}}_I^L\right),\hfill \\ {\mathbf{W}}_2=1-{\mathbf{W}}_1,\hfill \end{array}\right.$$

(17)

where $\delta (·)$ denotes the step function. During training, the step function is implemented using the differentiable approximation $\delta \left(x\right)\approx \sigma \left((x-\mathrm{thr})/\tau \right)$, where $\sigma \left(z\right)=1/(1+\exp (-z\left)\right)$ is the standard sigmoid function, $\mathrm{thr}$ is the decision threshold, and $\tau$ controls the sharpness of the approximation. We set $\mathrm{thr}=0$ and $\tau =0.1$ in all experiments.

Using these spatial weights, the SRS loss is calculated as

$$L_{SRS}={\displaystyle \frac{1}{HW}}{\displaystyle \sum _{i=1}^{HW}}\left[{\mathbf{W}}_1\left|I_f\left(i\right)-I_{vi}\left(i\right)\right|+{\mathbf{W}}_2\left|I_f\left(i\right)-I_{ir}\left(i\right)\right|\right],$$

(18)

where $HW$ denotes the total number of pixels. Thus, the fused image is adaptively constrained toward the source modality assigned a larger local selection weight at each spatial location.

To better preserve effective scene-related information such as global positions, we further design the GSS loss as a global scale similarity metric loss function. Based on the ${\mathsf{\Phi }}_V^G$ and ${\mathsf{\Phi }}_I^G$ obtained from the Transformer-based feature extraction module, we calculate their means ${\mu }_{vi},{\mu }_{ir}$ to obtain the weights ${\mathbf{W}}_{\mathbf{3}}$ and ${\mathbf{W}}_{\mathbf{4}}$ for the global structural similarity. The specific calculation formula is as follows:

$$\left\{\begin{array}{c}{\mu }_{vi}=\mathrm{MEAN}\left({\mathsf{\Phi }}_V^G\right)\hfill \\ {\mu }_{ir}=\mathrm{MEAN}\left({\mathsf{\Phi }}_I^G\right)\hfill \end{array}\right.,$$

(19)

$$\left\{\begin{array}{c}{\mathbf{W}}_{\mathbf{3}}={\displaystyle \frac{{\mu }_{vi}}{{\mu }_{vi}+{\mu }_{ir}}}\hfill \\ {\mathbf{W}}_{\mathbf{4}}={\displaystyle \frac{{\mu }_{ir}}{{\mu }_{vi}+{\mu }_{ir}}}\hfill \end{array}\right.,$$

(20)

where $\mathrm{MEAN}(·)$ denotes the global averaging operation. The normalized weights ${\mathbf{W}}_3$ and ${\mathbf{W}}_4$ are used to adaptively balance the visible- and infrared-specific structural similarity terms according to the global responses of ${\mathsf{\Phi }}_V^G$ and ${\mathsf{\Phi }}_I^G$. The structural similarity is calculated as follows:

$$\begin{array}{c}\hfill \varpi (X,F)={\displaystyle \frac{\left(2{\mu }_X{\mu }_F+C_1\right)\left(2{\sigma }_{XF}+C_2\right)}{\left({\mu }_X^2+{\mu }_F^2+C_1\right)\left({\sigma }_X^2+{\sigma }_F^2+C_2\right)}},\end{array}$$

(21)

where $\varpi (X,F)$ represents the calculation of the structural similarity between source image X and fused image F, ${\mu }_X$ refers to the mean of the image X, and ${\sigma }_X$ and ${\sigma }_{XF}$ represent the variance and covariance of the images, respectively.

Since a higher SSIM value indicates better consistency between the fused image and the source image, the loss is computed as $1-SSIM$ in the loss function. Finally, the GSS loss is calculated by weighting the loss based on the global features obtained from the weight map.

$$\begin{array}{cc}\hfill L_{GSS}=& {\mathbf{W}}_3\left[1-\varpi \left(I_{vi},I_f\right)\right]+{\mathbf{W}}_4\left[1-\varpi \left(I_{ir},I_f\right)\right].\hfill \end{array}$$

(22)

4. Experiments

In this section, we first introduce the experimental settings of this paper, including the datasets, evaluation metrics, and details of the experimental process. Next, we present the comparison algorithms and the results of both qualitative and quantitative analyses. Finally, we elaborate on the ablation experiments.

4.1. Experimental Settings

4.1.1. Dataset Description

The infrared and visible image fusion experiments conducted in this paper used three publicly available datasets: MSRS [26], RoadScene [24], and TNO [70]. The MSRS and RoadScene datasets contain pairs of infrared and visible images from various daytime and nighttime road scenes, while the TNO dataset includes pairs from a broader range of scenes, such as military environments.

Specifically, 1294 image pairs from the MSRS dataset were used for training, while the remaining 150 pairs were retained for evaluation. The trained model was further evaluated on 25 image pairs from RoadScene and 25 image pairs from TNO. No fine-tuning was performed during testing.

4.1.2. Evaluation Metrics

To evaluate the fusion results from complementary perspectives, we employed five metrics [71]: mean squared error ($MSE$), peak signal-to-noise ratio ($PSNR$), fusion artifacts ($N^{AB/F}$), correlation coefficient ($CC$), and multi-scale structural similarity ($MS\text{-}SSIM$). In Section 4.1.2, $MSE$ refers to the evaluation metric rather than the training loss term $L_{MSE}$ defined in Section 3.5. The evaluation $MSE$ was computed as a dual-reference measure between the fused image and the two source images. All metrics were computed on single-channel images. The symbols $I_{vi}$, $I_{ir}$, and $I_f$ denote the visible luminance image, infrared image, and fused luminance image, respectively. Each metric was first calculated for every test image pair and then averaged over the corresponding test set.

For the metrics requiring normalized intensities, the images were mapped from $[0,255]$ to $[0,1]$ as

$${\overline{I}}_x={\displaystyle \frac{I_x}{255}},x\in \{vi,ir,f\}.$$

(23)

The $MSE$ measures the average pixel-wise discrepancy between the fused image and the two source images. Since infrared-visible image fusion does not provide a ground-truth fused image, the pairwise errors between the fused image and each source image were first calculated as

$$\begin{array}{cc}\hfill MS{E}_{x,f}& ={\displaystyle \frac{1}{HW}}{\displaystyle \sum _{i=1}^{HW}}{\left({\overline{I}}_f\left(i\right)-{\overline{I}}_x\left(i\right)\right)}^2,x\in \{vi,ir\},\hfill \end{array}$$

(24)

$$\begin{array}{cc}\hfill MSE& ={\displaystyle \frac{MS{E}_{vi,f}+MS{E}_{ir,f}}{2}},\hfill \end{array}$$

(25)

where H and W denote the image height and width, respectively. A lower $MSE$ indicates a smaller average pixel-wise discrepancy between the fused image and the two source images.

The $PSNR$ was calculated from the dual-reference $MSE$ defined above:

$$PSNR=10{\log }_{10}\left({\displaystyle \frac{I_{\max }^2}{MSE}}\right),$$

(26)

where $I_{\max }=1$ denotes the maximum possible intensity value after normalization to $[0,1]$. A higher $PSNR$ indicates a smaller average pixel-level distortion between the fused image and the two source images.

The artifact measure $N^{AB/F}$ evaluates the amount of artificial edge information introduced into the fused image. It is defined as

$$N^{AB/F}={\displaystyle \frac{{\displaystyle {\displaystyle \sum _{i=1}^H}{\displaystyle \sum _{j=1}^W}AM(i,j)\left[\left(1-Q^{vi,f}(i,j)\right){\omega }^{vi}(i,j)+\left(1-Q^{ir,f}(i,j)\right){\omega }^{ir}(i,j)\right]}}{{\displaystyle {\displaystyle \sum _{i=1}^H}{\displaystyle \sum _{j=1}^W}\left[{\omega }^{vi}(i,j)+{\omega }^{ir}(i,j)\right]}}},$$

(27)

where $Q^{vi,f}$ and $Q^{ir,f}$ denote the edge-information preservation measures between the fused image and the visible and infrared source images, respectively. The artifact indicator is defined as

$$AM(i,j)=\left\{\begin{array}{cc}1,\hfill & g_f(i,j)>g_{vi}(i,j)\mathrm{and}{g}_f(i,j)>g_{ir}(i,j),\hfill \\ 0,\hfill & \mathrm{otherwise},\hfill \end{array}\right.$$

(28)

where $g_{vi}$, $g_{ir}$, and $g_f$ denote the edge strengths of the visible, infrared, and fused images, respectively, while ${\omega }^{vi}$ and ${\omega }^{ir}$ denote the corresponding edge-importance weights. A lower $N^{AB/F}$ indicates fewer artifacts introduced during fusion.

The $CC$ evaluates the linear correlation between the fused image and the two source images:

$$\begin{array}{cc}\hfill C{C}_{x,f}& ={\displaystyle \frac{{\displaystyle {\displaystyle \sum _{i=1}^{HW}}\left(I_x\left(i\right)-{\mu }_x\right)\left(I_f\left(i\right)-{\mu }_f\right)}}{{\displaystyle \sqrt{\sum _{i=1}^{HW}{\left(I_x\left(i\right)-{\mu }_x\right)}^2\sum _{i=1}^{HW}{\left(I_f\left(i\right)-{\mu }_f\right)}^2}}}},x\in \{vi,ir\},\hfill \end{array}$$

(29)

$$\begin{array}{cc}\hfill CC& ={\displaystyle \frac{C{C}_{vi,f}+C{C}_{ir,f}}{2}},\hfill \end{array}$$

(30)

where ${\mu }_x$ and ${\mu }_f$ denote the mean intensities of the source and fused images, respectively. A higher $CC$ indicates a stronger average linear correlation between the fused image and the two source images.

The standard pairwise $MS\text{-}SSIM$ measures structural similarity between two images across multiple spatial scales. For a source image $I_x$ and the fused image $I_f$, it is calculated as

$$\mathrm{MSSSIM}\left({\overline{I}}_x,{\overline{I}}_f\right)={\left[l_J\left({\overline{I}}_x,{\overline{I}}_f\right)\right]}^{w_J}{\displaystyle \prod _{j=1}^J}{\left[c_j\left({\overline{I}}_x,{\overline{I}}_f\right)s_j\left({\overline{I}}_x,{\overline{I}}_f\right)\right]}^{w_j},$$

(31)

where $l_J(·)$ denotes the luminance comparison at the coarsest scale, while $c_j(·)$ and $s_j(·)$ denote the contrast and structural comparisons at scale j, respectively. The symbol $w_j$ denotes the standard scale weight [72].

Because infrared-visible image fusion involves two source images, the $MS\text{-}SSIM$ reported in this paper is defined as the sum of the two pairwise scores:

$$MS\text{-}SSIM=\mathrm{MSSSIM}\left({\overline{I}}_{vi},{\overline{I}}_f\right)+\mathrm{MSSSIM}\left({\overline{I}}_{ir},{\overline{I}}_f\right).$$

(32)

Each pairwise MSSSIM value lies within $[0,1]$; therefore, the summed dual-reference $MS\text{-}SSIM$ reported in this paper lies within $[0,2]$. A higher value indicates stronger overall structural similarity between the fused image and the two source images.

4.1.3. Implementation Details

Our experiments were conducted using the PyTorch 2.4.0 framework on an NVIDIA A800 GPU with 80 GB of memory. In the preprocessing stage, the training samples were randomly divided into $128\times 128$ patches. The training process was run for a total of 60 epochs. The batch size was set to 128. The optimizer used was Adam with the initial learning rate set to ${10}^{-4}$. For the network hyperparameters, the number of MP blocks B was set to 5. The TGFE module consisted of $L=8$ Transformer layers.

For the overall loss function in Equation (13), the weights were empirically selected through preliminary calibration experiments based on the observed numerical scales of the unweighted loss terms. Since the unweighted gradient loss was considerably smaller than the MSE and S-SIM losses, ${\lambda }_1$ was set to 50 to maintain an effective gradient-preservation constraint. The MSE and S-SIM terms were assigned weights of ${\lambda }_2=5$ and ${\lambda }_3=2$, respectively. For the S-SIM loss in Equation (16), the internal weights were set to ${\beta }_1=1$ and ${\beta }_2=5$ following the same scale-balancing principle. The resulting coefficient configuration was fixed for all reported experiments.

4.1.4. Comparison with State-of-the-Arts

We evaluated the proposed method on three datasets and compared it with 15 state-of-the-art methods, namely, one traditional method (MDLatLRR [16]), four CNN-based methods (FusionDN [24], ReCoNet [73], SeAFusion [27], PIAFusion [26]), one GAN-based method (TarDAL [31]), four AE-based methods (NestFuse [34], DIDFuse [38], DeFusion [74], CrossFuse [66]), two Transformer-based methods (SwinFusion [65], CDDFuse [41]), and three recent methods published in 2025 (PPIFuse [67], TransFusion [68], and URFusion [69]).

4.2. Qualitative Analysis

Figure 4, Figure 5 and Figure 6 present qualitative comparisons between SiPFusion and 15 representative fusion methods on the MSRS, RoadScene, and TNO datasets, respectively. The highlighted regions were used to examine the preservation of scene intensity, salient targets, and local structural details.

Figure 4. Qualitative comparison of SiPFusion with 15 state-of-the-art methods on MSRS dataset (01035N).

Figure 5. Qualitative comparison of SiPFusion with 15 state-of-the-art methods on RoadScene dataset (FLIR_09336).

Figure 6. Qualitative comparison of SiPFusion with 15 state-of-the-art methods on TNO dataset (002).

As shown in Figure 4, several comparison methods exhibit uneven intensity distributions or local texture distortions in the green-boxed road region. CrossFuse presents noticeable brightness discontinuities, whereas the results produced by ReCoNet, DIDFuse, and TransFusion appear relatively dark, making some low-light structures less distinguishable. In comparison, SiPFusion maintains a coherent road-region representation with smoother intensity transitions and fewer visually apparent high-frequency fluctuations. In the red-boxed low-light region, the tree branches remain recognizable, while the overall scene intensity is preserved consistently.

In Figure 5, the green-boxed region contains a salient vehicle affected by surrounding headlight illumination. SiPFusion preserves the principal vehicle contours and local intensity information without introducing visually dominant halo-like artifacts. In the red-boxed region, which contains densely distributed tree branches, SiPFusion maintains distinguishable branch structures without excessive sharpening. These observations indicate a balanced preservation of salient intensity information and local structural details.

In Figure 6, the green-boxed region highlights the railing structure in front of the store entrance. SiPFusion preserves the principal contours and individual segments of the railing, allowing these structures to remain distinguishable in the fused image. In the red-boxed region, recognizable patterns on the store sign are retained without visually conspicuous artifacts. The resulting image therefore maintains a balanced integration of visible texture and infrared intensity information.

To complement the visual observations, we evaluated the six highlighted regions in Figure 4, Figure 5 and Figure 6. All calculations were performed on the luminance channel using identical region coordinates for all methods. Let ${\mathcal{R}}_k$ denote the k-th highlighted region, where $K=6$. The local dual-source correlation coefficient and local dual-source structural similarity are defined as

$$\begin{array}{cc}\hfill \mathrm{LCC}& ={\displaystyle \frac{1}{K}}{\displaystyle \sum _{k=1}^K}{\displaystyle \frac{CC(I_{vi}^{{\mathcal{R}}_k},I_f^{{\mathcal{R}}_k})+CC(I_{ir}^{{\mathcal{R}}_k},I_f^{{\mathcal{R}}_k})}{2}},\hfill \end{array}$$

(33)

$$\begin{array}{cc}\hfill \mathrm{LSSIM}& ={\displaystyle \frac{1}{K}}{\displaystyle \sum _{k=1}^K}{\displaystyle \frac{SSIM(I_{vi}^{{\mathcal{R}}_k},I_f^{{\mathcal{R}}_k})+SSIM(I_{ir}^{{\mathcal{R}}_k},I_f^{{\mathcal{R}}_k})}{2}}.\hfill \end{array}$$

(34)

As shown in Table 1, SiPFusion achieves the highest average LCC and competitive LSSIM performance. These results demonstrate that SiPFusion maintains strong aggregate consistency with the visible and infrared source images while competitively preserving local structural information.

Table 1. Localized quantitative comparison over the six highlighted regions in Figure 4, Figure 5 and Figure 6. The best and second-best results are highlighted in bold and underlined, respectively. ↑ indicates that higher values are better.

Method	LCC ↑	LSSIM ↑
MDLatLRR	0.4921	0.4102
FusionDN	0.5996	0.5014
ReCoNet	0.5537	0.4054
SeAFusion	0.5595	0.6351
PIAFusion	0.5589	0.6406
TarDAL	0.5574	0.5515
SwinFusion	0.5560	0.5509
CDDFuse	0.5357	0.6240
NestFuse	0.5900	0.5308
DIDFuse	0.5854	0.4467
DeFusion	0.5563	0.6658
CrossFuse	0.5604	0.5825
PPIFuse	0.4859	0.5793
TransFusion	0.5669	0.5019
URFusion	0.5489	0.4896
SiPFusion	0.6249	0.6350

4.3. Quantitative Analysis

From Table 2, Table 3 and Table 4, SiPFusion achieves consistently strong quantitative performance against 15 competitive baselines, including recently published methods.

Table 2. Quantitative comparisons on the MSRS dataset. In the table, bold is best, and underlined is second-best. ↑ and ↓ indicate that higher and lower values are better, respectively.

	$\mathit{MSE}$ ↓	$\mathit{PSNR}$ ↑	${\mathit{N}}^{\mathit{AB}/\mathit{F}}$ ↓	$\mathit{CC}$ ↑	$\mathit{MS}\text{-}\mathit{SSIM}$ ↑
MDLatLRR	0.04808	13.4962	0.1999	0.5782	1.0753
FusionDN	0.07565	11.2669	0.0726	0.6201	1.2853
ReCoNet	0.04242	14.0533	0.0315	0.5956	1.2580
SeAFusion	0.05386	13.2431	0.0427	0.6113	1.3421
PIAFusion	0.05871	12.9121	0.0800	0.6025	1.3292
TarDAL	0.05065	13.0972	0.0294	0.6414	1.3599
SwinFusion	0.05714	13.0542	0.0841	0.6017	1.3107
CDDFuse	0.05204	13.3920	0.0444	0.5973	1.3252
NestFuse	0.04865	13.8166	0.0505	0.6035	1.3291
DIDFuse	0.04618	13.6180	0.0284	0.6349	1.2646
DeFusion	0.03679	14.9569	0.0123	0.5836	1.3325
CrossFuse	0.04631	14.0026	0.0173	0.5494	1.3150
PPIFuse	0.05654	13.0708	0.0819	0.6016	1.3062
TransFusion	0.03681	14.6409	0.0073	0.6388	1.3402
URFusion	0.04952	13.1567	0.0681	0.5817	1.2904
Ours	0.02925	15.8090	0.0100	0.6877	1.4203

Table 3. Quantitative comparisons on the RoadScene dataset. In the table, bold is best, and underlined is second-best. ↑ and ↓ indicate that higher and lower values are better, respectively.

	$\mathit{MSE}$ ↓	$\mathit{PSNR}$ ↑	${\mathit{N}}^{\mathit{AB}/\mathit{F}}$ ↓	$\mathit{CC}$ ↑	$\mathit{MS}\text{-}\mathit{SSIM}$ ↑
MDLatLRR	0.05415	12.9204	0.1091	0.5106	1.0436
FusionDN	0.05530	13.0634	0.0256	0.5642	1.2831
ReCoNet	0.04870	13.6551	0.0292	0.5431	1.2973
SeAFusion	0.06635	12.1833	0.1086	0.5287	1.2555
PIAFusion	0.06719	12.2132	0.4545	0.5041	1.2788
TarDAL	0.04734	13.6449	0.0303	0.5064	1.2310
SwinFusion	0.08715	10.9973	0.1517	0.5312	1.2573
CDDFuse	0.05904	12.7755	0.0656	0.5424	1.2598
NestFuse	0.06694	12.0919	0.0264	0.5279	1.2455
DIDFuse	0.05021	13.2285	0.0468	0.5584	1.2597
DeFusion	0.04131	14.4211	0.0033	0.5258	1.2523
CrossFuse	0.08318	11.2965	0.0162	0.4628	1.1641
PPIFuse	0.07665	11.7237	0.0926	0.4921	1.1999
TransFusion	0.04176	14.1606	0.0079	0.5470	1.2637
URFusion	0.09629	10.3485	0.0175	0.5420	1.2572
Ours	0.03700	14.6731	0.0104	0.5826	1.2952

Table 4. Quantitative comparisons on the TNO dataset. In the table, bold is best, and underlined is second-best. ↑ and ↓ indicate that higher and lower values are better, respectively.

	$\mathit{MSE}$ ↓	$\mathit{PSNR}$ ↑	${\mathit{N}}^{\mathit{AB}/\mathit{F}}$ ↓	$\mathit{CC}$ ↑	$\mathit{MS}\text{-}\mathit{SSIM}$ ↑
MDLatLRR	0.05226	13.1926	0.0887	0.4657	1.1515
FusionDN	0.05807	12.6321	0.0363	0.4834	1.2758
ReCoNet	0.05409	13.0883	0.0153	0.4780	1.3002
SeAFusion	0.07894	11.7012	0.0879	0.4695	1.3017
PIAFusion	0.07493	11.6930	0.0556	0.4531	1.2952
TarDAL	0.05820	12.8749	0.0354	0.4578	1.2759
SwinFusion	0.08005	11.4756	0.0528	0.4599	1.2940
CDDFuse	0.06888	12.1624	0.0432	0.4727	1.3108
NestFuse	0.06749	12.4418	0.0189	0.4674	1.3262
DIDFuse	0.05687	12.7122	0.0319	0.4963	1.2606
DeFusion	0.05569	13.3174	0.0061	0.4542	1.2943
CrossFuse	0.06520	12.4143	0.0180	0.4054	1.2082
PPIFuse	0.08873	11.1007	0.0434	0.4235	1.2758
TransFusion	0.05432	13.0251	0.0074	0.4759	1.3239
URFusion	0.06892	11.8386	0.0214	0.4839	1.3290
Ours	0.04388	13.9964	0.0078	0.5226	1.3711

On MSRS, SiPFusion attains the best $MSE$, $PSNR$, $CC$, and $MS\text{-}SSIM$ while remaining competitive on $N^{AB/F}$ compared with the lowest-artifact results.

On RoadScene, our method achieves the best $MSE$, $PSNR$, and $CC$ and the second-best $MS\text{-}SSIM$, indicating improved fidelity and cross-modal consistency under challenging illumination and headlight glare conditions.

On TNO, SiPFusion again obtains the best $MSE$, $PSNR$, $CC$, and $MS\text{-}SSIM$, demonstrating robust generalization to diverse scenes. These improvements can be attributed to the complementary contributions of the proposed components.

First, the progressive local-to-global feature extraction (MLFE → TGFE) enhances cross-scale alignment and global dependency modeling, which reduces reconstruction error (lower $MSE$/higher $PSNR$) and benefits structure-related similarity (higher $MS\text{-}SSIM$).

Second, the SPF module performs spatial selection to emphasize informative regions and suppress irrelevant responses, leading to higher correlation with the sources (higher $CC$) and more stable structural preservation.

Third, the proposed S-SIM loss explicitly constrains salient-region fidelity and global structural consistency during training, further improving perceptual and structural quality.

While SiPFusion does not obtain the lowest $N^{AB/F}$ values on the three datasets, it achieves strong fidelity-, correlation-, and structure-related performance while maintaining competitive artifact suppression, resulting in a favorable overall balance across the evaluated metrics.

4.4. Computational Efficiency

To provide a more comprehensive evaluation of computational efficiency, we report the average runtime, parameter count, and FLOPs of SiPFusion and the comparison methods. The input resolution was set to $640\times 480$ for the computational evaluation.

As shown in Table 5, SiPFusion exhibits a moderate computational cost in terms of runtime, parameter count, and FLOPs. Although it is not among the most computationally efficient methods, this cost is associated with the progressive multi-scale feature extraction and Transformer-based global dependency modeling used to enhance local-detail preservation and global-context representation. We therefore acknowledge computational efficiency as a limitation of the current implementation. Future work will investigate lightweight attention mechanisms, efficient Transformer variants, token reduction, and model compression techniques to reduce inference latency while maintaining competitive fusion performance.

Table 5. Computational-efficiency comparison using an input resolution of $640\times 480$. Runtime is measured over 100 images. The best and second-best results are highlighted in bold and underlined, respectively. ↓ indicate that lower values are better.

Method	Runtime (s/100 Images) ↓	Params (M) ↓	FLOPs (G) ↓
MDLatLRR	7.5729	–	–
FusionDN	42.2988	1.1629	1031.4253
ReCoNet	8.4293	0.0036	6.4365
SeAFusion	69.4869	0.1669	101.9904
PIAFusion	10.4043	1.1761	721.9839
TarDAL	44.5793	0.2966	181.7248
SwinFusion	66.5644	0.9737	700.1487
CDDFuse	37.3328	1.1883	1104.9173
NestFuse	18.5203	2.7328	714.2179
DIDFuse	4.3965	0.2609	227.9080
DeFusion	176.1607	7.8745	141.1020
CrossFuse	88.1156	23.3102	271.9851
PPIFuse	20.3642	4.2485	155.5785
TransFusion	32.4340	53.1364	1822.6623
URFusion	19.8884	0.2433	126.7053
Ours	47.9313	18.0205	816.3377

4.5. Ablation Study

To investigate the contribution of each component in SiPFusion, we conducted an ablation study with eight settings. The ablation design (progressive strategy, feature extraction branches, fusion strategy, and loss terms) and the corresponding quantitative results are summarized in Table 6. In addition, Figure 7 provides qualitative comparisons (with local zoom-in regions) for all ablation settings to visually validate the reported effects. Specifically, the green boxes focus on the intensity/visibility of pedestrians (salient thermal targets), while the red boxes highlight thin-structure textures (e.g., tree-branch details), which together offer an intuitive validation of the quantitative trends.

Table 6. Ablation study of SiPFusion.The circled numbers denote different experimental settings. The checkmark and multiplication sign indicate that the corresponding component is included and excluded, respectively. “©” indicates that the SPF module is replaced with simple concatenation. The best and second-best results are highlighted in bold and underlined, respectively. ↑ and ↓ indicate that higher and lower values are better, respectively.

Setting	Ablation Design						Metrics
	Progressive	MLFE	TGFE	SPF	${\mathit{L}}_{\mathit{SRS}}$	${\mathit{L}}_{\mathit{GSS}}$	$\mathit{MSE}$ ↓	$\mathit{PSNR}$ ↑	${\mathit{N}}^{\mathit{AB}/\mathit{F}}$↓	$\mathit{CC}$ ↑	$\mathit{MS}\text{-}\mathit{SSIM}$ ↑
①	×	✓	✓	✓	✓	✓	0.11701	9.3823	0.0623	0.2262	0.8118
②	×	✓	×	✓	✓	×	0.06270	12.1141	0.0120	0.0458	1.2864
③	×	×	✓	✓	×	✓	0.10580	9.8144	0.0027	0.3163	0.9824
④	✓	✓	✓	©	×	✓	0.03823	14.4220	0.0793	0.1536	0.6932
⑤	✓	✓	✓	✓	×	✓	0.03088	15.5145	0.0125	0.6734	1.4135
⑥	✓	✓	✓	✓	✓	×	0.03200	15.4315	0.0103	0.6743	1.4061
⑦	✓	✓	✓	✓	×	×	0.03465	14.9193	0.0169	0.6703	1.3474
⑧	✓	✓	✓	✓	✓	✓	0.02925	15.8090	0.0100	0.6877	1.4203

Figure 7. Qualitative comparisons of the ablation settings in Table 6. (a) Infrared image, (b) visible image, (c–i) correspond to Settings 1–7, and (j) is the full model (Setting 8). The local magnified regions are shown for clearer inspection of pedestrian intensity/visibility (green boxes) and fine texture details such as tree branches (red boxes).

In Table 6, “Progressive” refers to the organizational relationship between MLFE and TGFE. With Progressive enabled, the multi-scale local features produced by MLFE are used as the input to TGFE, forming the proposed serial MLFE–TGFE pathway. In Setting 1, Progressive is disabled while both MLFE and TGFE remain enabled. MLFE and TGFE therefore operate in parallel, with MLFE extracting local features and TGFE extracting global features independently before their outputs are provided to the subsequent fusion process. For Settings 2 and 3, only one feature-extraction module is retained, and the Progressive connection is therefore not applied.

The availability of the S-SIM terms depends on their corresponding intermediate features. $L_{GSS}$ is calculated using the global features ${\mathsf{\Phi }}_V^G$ and ${\mathsf{\Phi }}_I^G$ produced by TGFE and is therefore unavailable when TGFE is removed in Setting 2. Similarly, $L_{SRS}$ is calculated using the local selection masks ${\widehat{\mathbf{PM}}}_V^L$ and ${\widehat{\mathbf{PM}}}_I^L$ corresponding to the local features produced by MLFE and is therefore unavailable when MLFE is removed in Setting 3.

Effect of the progressive strategy. Comparing Setting 1 and Setting 8 (with all other components unchanged) shows that enabling the progressive local-to-global pipeline consistently improves fidelity- and structure-related metrics (lower $MSE$ and higher $PSNR$/$MS\text{-}SSIM$) as well as source consistency (higher $CC$). As shown in Figure 7, the progressive pipeline yields more reliable pedestrian intensity and target visibility in the green boxes while better preserving thin-structure textures (e.g., branch continuity and sharpness) in the red boxes, indicating that progressive cross-scale alignment and global dependency modeling are crucial for high-quality fusion.

Effect of the feature extraction branches (MLFE and TGFE). We evaluated MLFE by comparing Settings 3 and 5 and TGFE by comparing Settings 2 and 6. It should be noted that these matched comparisons are not strictly single-variable ablations because removing either MLFE or TGFE also disables the progressive connection between the two modules. Therefore, the results reflect the contribution of each feature-extraction module within the proposed progressive local-to-global pathway. In both matched comparisons, introducing the corresponding feature-extraction module produces clear improvements in $MSE$, $PSNR$, and $MS\text{-}SSIM$. Qualitatively, as shown in Figure 7, removing TGFE tends to cause less stable intensity organization for salient pedestrians in the green-boxed regions, leading to darker or lower-contrast targets and weakened source consistency. Removing MLFE often results in over-smoothing and loss of fine textures, which is particularly evident in the thin branches within the red-boxed regions. These observations suggest that MLFE and TGFE are complementary: MLFE preserves fine-grained local details, whereas TGFE contributes to global structural consistency.

Effect of the significance-preserving fusion module. To evaluate the fusion strategy, we compared the simple concatenation baseline (Setting 4) with the SPF-based fusion (Setting 7) with the same progressive encoder and without SRS/GSS. SPF yields consistent gains across all metrics, including better reconstruction fidelity ($MSE$/$PSNR$), fewer artifacts ($N^{AB/F}$), higher correlation ($CC$), and stronger structural similarity ($MS\text{-}SSIM$). The visual comparison in Figure 7 also shows that simple concatenation is more prone to suppressing salient-target intensity/visibility (green boxes) and attenuating thin-structure textures (red boxes), whereas SPF better preserves pedestrian intensity while retaining sharper branch details, validating the effectiveness of spatial selection.

Effect of SRS and GSS in S-SIM. We further disentangle the two terms in S-SIM by evaluating GSS-only (Setting 5), SRS-only (Setting 6), the model without S-SIM (Setting 7), and full S-SIM (Setting 8). Specifically, the contribution of GSS is evaluated by comparing Settings 6 and 8, which differ only in whether GSS is enabled. The contribution of SRS is evaluated by comparing Settings 5 and 8, which differ only in whether SRS is enabled. Comparing Settings 7 and 8 further evaluates the combined contribution of the two terms. Both single-term variants outperform the setting without S-SIM, while combining them achieves the best overall performance across fidelity ($MSE$/$PSNR$), artifact suppression ($N^{AB/F}$), source consistency ($CC$), and structural similarity ($MS\text{-}SSIM$). In Figure 7, SRS-only tends to better preserve the intensity and visibility of salient pedestrians in the green-boxed regions, whereas GSS-only more effectively stabilizes structural consistency and reduces texture breakage on thin branches in the red boxes. Combining both terms simultaneously maintains salient-target intensity and fine texture fidelity, confirming the complementary roles of SRS and GSS in fusion.

To further examine the weighting strategy in GSS, we compared the proposed adaptive GSS weights with a fixed-weight variant. In the fixed-weight variant, the adaptive coefficients ${\mathbf{W}}_3$ and ${\mathbf{W}}_4$ were replaced with ${\mathbf{W}}_3={\mathbf{W}}_4=0.5$, while the remaining network architecture, loss function, training data, and optimization settings were kept unchanged. We also included the variant without GSS for reference. This comparison evaluated whether the global-feature-based adaptive weighting provides advantages over uniform weighting for the two source-specific structural similarity terms.

As shown in Table 7, the fixed-weight GSS variant improves $MSE$ and $PSNR$ compared with removing GSS, but it does not consistently improve $N^{AB/F}$, $CC$, or $MS\text{-}SSIM$. In contrast, the proposed adaptive GSS weighting strategy achieves the best performance across all five metrics. These results indicate that using global feature responses to adjust the relative contributions of the visible- and infrared-specific structural constraints is more effective than assigning equal weights to the two terms.

Table 7. Ablation comparison of the GSS weighting strategy. The best and second-best results are highlighted in bold and underlined, respectively. ↑ and ↓ indicate that higher and lower values are better, respectively.

GSS Weighting Strategy	$\mathit{MSE} \downarrow$	$\mathit{PSNR} \uparrow$	${\mathit{N}}^{\mathit{AB}/\mathit{F}} \downarrow$	$\mathit{CC} \uparrow$	$\mathit{MS}\text{-}\mathit{SSIM} \uparrow$
Without GSS	0.03200	15.4315	0.0103	0.6743	1.4061
Fixed weights (${\mathbf{W}}_3={\mathbf{W}}_4=0.5$)	0.03128	15.5396	0.0108	0.6696	1.4033
Adaptive weights (Ours)	0.02925	15.8090	0.0100	0.6877	1.4203

In addition to the component ablation settings above, we further compared SPF with representative attention-style fusion modules. Specifically, we replaced SPF with CBAM-style [75] and NestFuse-style [34] fusion modules while keeping the remaining network architecture, loss function, training data, and optimization settings unchanged. The CBAM-style variant applies channel attention and spatial attention independently to each of the four feature branches and then sums the attended features. The NestFuse-style variant computes spatial and channel activity weights from the four feature branches and performs weighted fusion. These variants preserve the same input/output interface as SPF and only change the fusion rule. When the original S-SIM loss is used, the SRS masks are taken from the attention or weighting maps corresponding to the visible local and infrared local branches, while GSS is computed from the same TGFE global representations ${\mathsf{\Phi }}_V^G$ and ${\mathsf{\Phi }}_I^G$.

As shown in Table 8, SPF achieves the best $MSE$, $PSNR$, and $MS\text{-}SSIM$ among the compared fusion modules, indicating stronger pixel-level fidelity and multi-scale structural preservation. Although the CBAM-style variant obtains a lower $N^{AB/F}$ and a higher $CC$, SPF maintains competitive artifact suppression and source correlation while improving fidelity and structural similarity. Compared with the NestFuse-style variant, SPF performs better across all evaluated metrics. These results indicate that the proposed branch-specific spatial selection strategy is more suitable for the four-branch local-global feature representation in SiPFusion than directly applying conventional attention-style feature refinement or activity-weighted fusion.

Table 8. Ablation comparison of different fusion modules. All variants use the same backbone, loss function, training data, and optimization settings. The best and second-best results are highlighted in bold and underlined, respectively. ↑ and ↓ indicate that higher and lower values are better, respectively.

Fusion Module	$\mathit{MSE} \downarrow$	$\mathit{PSNR} \uparrow$	${\mathit{N}}^{\mathit{AB}/\mathit{F}} \downarrow$	$\mathit{CC} \uparrow$	$\mathit{MS}\text{-}\mathit{SSIM} \uparrow$
CBAM-style	0.03140	15.5635	0.0085	0.7080	1.4177
NestFuse-style	0.03491	15.0180	0.0209	0.6348	1.3670
SPF (Ours)	0.02925	15.8090	0.0100	0.6877	1.4203

5. Conclusions

This paper proposes a significance-preserving progressive fusion network for infrared and visible images. The algorithm first uses multi-scale blocks to extract multi-scale information from the source images and then inputs these multi-scale features into a Transformer-based feature extraction module to capture global dependency information. Subsequently, the SPF module adaptively maximizes the retention of useful information from the source images. Finally, the designed S-SIM function further strengthens global structural consistency while focusing on retaining effective information in salient regions. The proposed method is compared with 15 advanced algorithms on three datasets—MSRS, RoadScene, and TNO—and evaluated using five metrics: $MSE$, $PSNR$, $N^{AB/F}$, $CC$, and $MS\text{-}SSIM$. The proposed method achieves leading performance on most evaluated metrics and competitive overall results across the three datasets, although it does not consistently obtain the lowest $N^{AB/F}$ values. Overall, SiPFusion effectively preserves spatial structures and texture details while limiting visually apparent noise and artifacts, demonstrating its potential for infrared and visible image fusion. Despite the promising performance, our current implementation has a limitation in inference efficiency due to the progressive multi-scale encoder and the Transformer-based global modeling. In future work, we will investigate more efficient architectures to reduce latency, including lightweight attention and token reduction strategies, efficient Transformer variants, and model compression techniques such as pruning, quantization, and knowledge distillation, while maintaining high fusion quality. Future work will also investigate task-oriented evaluations, such as object detection and semantic segmentation, to further assess the practical impact of the fused images on downstream perception tasks.

6. Patents

The work reported in this manuscript has resulted in the following patent: Zhejiang University. A Method and System for Infrared and Visible Image Fusion Based on Global Integration of Multi-scale Information and Salient Region Preservation (in Chinese). Chinese Patent Application No. 202510249011.1, authorized on 27 October 2025.