SDFNet: Semantic Driven Frequency-Domain Infrared and Visible Image Fusion Network

Highlights

What are the main findings?

• The proposed SDFNet effectively incorporates semantic priors into the fusion process and performs frequency-domain cross-modal interaction, enabling the fused images to preserve complementary visual information and useful semantic cues.
• Experiments on the MSRS and TNO datasets demonstrate competitive fusion performance across multiple quantitative metrics, while SDFNet achieves the best downstream semantic segmentation performance among the compared methods.

What are the implications of the main findings?

• These findings suggest that incorporating high-level semantic priors into image fusion can help bridge low-level multimodal fusion and downstream perception tasks, providing fused representations that better preserve task-relevant semantic information.
• The frequency-domain interaction strategy further demonstrates the potential of global cross-modal modeling to achieve effective feature interaction while maintaining a competitive computational profile, providing a practical basis for subsequent high-level vision tasks.

Abstract

Infrared and visible image fusion aims to integrate complementary thermal target information and visible structural details for multimodal perception and remote sensing applications. However, existing deep learning methods often underutilize semantic information during fusion and mainly perform cross-modal interaction in the spatial domain, limiting the exploitation of complementary representations. To address these issues, we propose SDFNet, a semantic driven frequency-domain infrared and visible image fusion network based on an encoder–decoder architecture. Guided by semantic priors extracted from a pre-trained segmentation model, SDFNet introduces a frequency-domain feature aggregation module to enhance cross-modal feature interaction and semantic information preservation. Extensive experiments on the MSRS dataset demonstrate the effectiveness of the proposed method. On the 361-pair MSRS test set, SDFNet ranks among the top two on five of the six evaluated fusion metrics, achieving EN, AG, SF, SCD, and Qabf scores of 6.73, 3.78, 11.59, 1.74, and 0.69, respectively, while obtaining a competitive VIF score of 1.02. In the downstream semantic utility evaluation, SDFNet achieves the highest mIoU of 78.76 among the compared methods, further demonstrating its ability to preserve task-relevant semantic information.

1. Introduction

Multimodal image fusion (MIF) serves as a fundamental paradigm for overcoming the inherent physical constraints of individual sensors [1]. As an important branch, infrared and visible image fusion (IVIF) integrates heterogeneous data sources to leverage their complementary physical characteristics [2]. Specifically, while visible sensors capture reflective spectra (400–750 nm) to provide high-resolution textural details and intricate spatial structures [3], their efficacy degrades sharply under adverse conditions such as low light, heavy smoke, or inclement weather [2]. Conversely, long-wave infrared (LWIR, 8–14 μm) sensors detect thermal radiation and can highlight thermally salient targets, such as pedestrians and vehicles, under low-illumination or nighttime conditions [4]. Nevertheless, infrared images generally contain fewer fine textures and high-frequency spatial details than visible images. This limitation arises not only from the longer operating wavelength and optical or detector-resolution constraints, but also from thermal-physics-driven texture loss: thermal radiation distributions may exhibit relatively smooth spatial variations and thus contain less recoverable fine-scale texture [5]. Synergistically fusing these modalities can compensate for their individual limitations and provide more informative representations for downstream perception tasks [6].

Historically, IVIF methodologies are categorized into traditional frameworks and deep learning-based paradigms [2]. Traditional image fusion methods mainly involve multi-scale transforms [7,8], sparse representation [9], subspace-based methods [10], and other handcrafted fusion strategies [11]. Although foundational, these handcrafted pipelines often struggle to model complex nonlinear cross-modal relationships. With the development of deep learning, CNN-based approaches have been employed for hierarchical feature extraction [12,13], followed by Transformer-based architectures that capture global context and long-range spatial dependencies [14,15]. Concurrently, Generative Adversarial Networks (GANs) have also been explored to optimize the fusion process through adversarial learning [16].

Despite these advances, many existing fusion methods are primarily optimized using pixel-level or feature-level reconstruction constraints and lack explicit high-level semantic guidance for different semantic regions. Consequently, semantically important targets or structures may not receive adequate attention during fusion, even when the resulting images achieve competitive low-level objective metrics. Recent semantic-guided fusion methods have demonstrated the potential of incorporating high-level semantic knowledge into the fusion process [17]. Beyond semantic guidance, existing methods also encounter limitations in the representation of complementary information in the frequency domain. From a signal-processing perspective, frequency-domain transforms decompose image signals into magnitude and phase components: the magnitude spectrum characterizes the strength distribution of different spatial-frequency components, whereas phase information plays an important role in preserving spatial structures and positional relationships [18]. Recent studies have begun to combine frequency-domain representations with Transformer mechanisms to model spectral information together with global dependencies [19,20]. Nevertheless, the joint exploitation of high-level semantic priors, spatial-domain local saliency, and cross-modal frequency-domain global interactions remain insufficiently explored. Therefore, effectively coordinating semantic guidance with complementary information from the spatial and frequency domains remains an important challenge for infrared and visible image fusion.

To address the above issues, we propose SDFNet, a semantic driven frequency-domain fusion network. In our method, we aim to generate semantically rich fused images guided by the semantic knowledge inherent in the provided visible and infrared images. A pre-trained semantic segmentation model is utilized to extract semantic prior information from both modalities, which is then employed to guide feature extraction, attention allocation, and loss function design in the fusion network. In particular, we incorporate a frequency domain fusion transformer to enhance the robustness of the multimodal coupled features for both the fusion process and downstream high-level vision tasks. Specifically, we integrate the frequency-domain attention interaction module (FDAI) into the frequency-domain feature aggregation module (FFAM) to model cross-modal global relationships through a multi-view self-attention mechanism. The combined action of these mechanisms compels the fusion network to precisely capture semantic information at the feature level.

The main contributions of this paper can be summarized as follows:

(1)

We propose a novel semantic driven frequency-domain infrared and visible image fusion network, termed SDFNet, which explicitly incorporates semantic priors extracted from a pre-trained semantic segmentation network into the fusion process to enhance cross-modal feature representation and semantic information preservation.

(2)

We design a Frequency-Domain Feature Aggregation Module (FFAM) that integrates semantic-guided local saliency enhancement, frequency-domain cross-modal interaction, and semantic feature refinement, enabling effective aggregation of local structural information and long-range cross-modal dependencies.

(3)

We introduce a Semantic Driven Region-Adaptive Loss that utilizes semantic class probabilities to construct soft semantic weighting maps for intensity supervision, adaptively regulating the preservation of thermally salient information and complementary visible information during training.

Extensive qualitative and quantitative experiments, together with ablation and downstream semantic segmentation evaluations, validate the effectiveness of the proposed method.

The rest of the paper is structured as follows: Section 2 reviews recent works related to the proposed method. Section 3 describes the proposed SDFNet in detail. Section 4 presents comprehensive experimental evaluations, including comparative experiments, downstream semantic utility evaluation, ablation studies, and complexity analysis. Section 5 concludes this work.

2. Related Work

Infrared and visible image fusion (IVIF) aims to integrate salient thermal information from infrared images with rich structural and textural information from visible images, thereby generating a more informative representation of the observed scene. This section reviews representative learning-based IVIF methods from four perspectives: CNN-based fusion, transformer-based fusion, frequency-domain and spatial–frequency fusion, and semantic driven fusion.

2.1. CNN-Based Fusion Methods

Convolutional neural networks have laid an important foundation for learning-based image fusion. As an early study applying CNNs to image fusion, Liu et al. [21] employed a convolutional network to learn the mapping from source image patches to fusion decision maps for multi-focus image fusion, demonstrating the feasibility of replacing manually designed activity measures and fusion rules with learned representations. For infrared and visible image fusion, Li et al. [22] proposed DenseFuse, which employs a densely connected encoder to extract deep features and predefined fusion strategies to integrate multimodal information. Xu et al. [23] subsequently proposed U2Fusion, a unified unsupervised framework that estimates adaptive information-preservation degrees and accommodates multiple image fusion tasks within a single model.

CNN-based methods provide effective mechanisms for extracting local texture and structural information. Nevertheless, the inherent locality of convolutional operators can limit the explicit modeling of long-range spatial correlations and global cross-modal dependencies, motivating subsequent research on architectures with stronger global interaction capabilities.

2.2. Transformer-Based Fusion Methods

Transformer architectures have been increasingly introduced into image fusion to model long-range dependencies and cross-modal relationships. Ma et al. [24] proposed SwinFusion, which employs intra-domain self-attention and inter-domain cross-attention within a shifted-window architecture to model long-range dependencies within and across modalities. Tang et al. [25] combined dual-attention-based feature extraction with Transformer modeling to capture global complementary information. Zhao et al. [26] proposed CDDFuse, which combines Lite Transformer blocks for low-frequency global representation with invertible neural network blocks for high-frequency local representation, together with a correlation-driven constraint for decomposing modality-shared and modality-specific information. Li et al. [27] further introduced a hybrid CNN–Transformer framework with cross-attention to explicitly model complementary relationships between infrared and visible features.

These methods demonstrate the effectiveness of attention-based global modeling for cross-modal feature interaction. Nevertheless, their interaction mechanisms are primarily constructed in learned feature spaces, while the explicit exploitation of modality-dependent frequency characteristics is relatively limited. This has motivated increasing attention to frequency-domain and spatial–frequency collaborative modeling.

2.3. Frequency-Domain and Spatial–Frequency Fusion Methods

Frequency-domain representations provide a complementary perspective for characterizing global structures, local details, and modality-dependent image information. Zheng et al. [28] combined frequency integration with spatial compensation by exploiting the Fourier characteristics of infrared and visible images. Hu et al. [29] developed SFDFusion, where a dual-modality refinement module extracts complementary spatial features and an FFT-based frequency-domain fusion module integrates spectral information. Shi et al. [19] proposed Frefusion, which performs feature extraction and Transformer-based multimodal interaction in the frequency domain. Recent studies have further explored coordinated spatial–frequency representation learning. Wang et al. [30] combined Transformer-based global modeling with spatial–frequency detail enhancement in a dual-branch framework. Chen et al. [31] further proposed STSFusion, which integrates spatial feature modeling with adaptive frequency-domain guidance and semantic segmentation supervision to jointly preserve structural details and task-relevant information.

These studies demonstrate a progression from explicit spectral processing toward coordinated spatial–frequency representation and task-aware fusion. Nevertheless, frequency-domain interaction and high-level semantic guidance are still often realized through dedicated frequency modules and separate semantic supervision or reconstruction pathways. The direct use of hierarchical semantic priors to regulate cross-modal interaction within frequency-aware feature representations remains comparatively less explored.

2.4. Semantic Driven Fusion Methods

Conventional fusion objectives mainly emphasize pixel intensity, gradients, structural similarity, and other low-level visual properties, whereas high-level semantic information can provide additional guidance for identifying task-relevant regions and regulating modality-specific information preservation. Tang et al. [32] connected infrared–visible image fusion with semantic segmentation and propagated semantic supervision back to the fusion module during training, thereby introducing high-level task requirements into fusion optimization. Tang et al. [33] subsequently proposed PSFusion, in which hierarchical semantic features are progressively injected into the fusion pathway together with scene-fidelity constraints.

More recent studies have investigated semantic guidance beyond conventional segmentation supervision. Wu et al. [34] proposed SAGE, which extracts high-level semantic priors from the Segment Anything Model and transfers the semantic knowledge to a compact fusion network through distillation, avoiding dependence on SAM during inference. Tang et al. [35] further introduced degradation and semantic dual priors to jointly guide restoration and complementary information aggregation. Xing et al. [36] introduced a semantic-guided mixture-of-experts framework, where a segmentation head guides the dynamic selection of complementary multimodal features for task-aware fusion.

Overall, recent IVIF research has evolved from local feature aggregation toward global cross-modal modeling, spatial-frequency collaborative representation, and semantic driven fusion [37]. Although existing studies have explored frequency representation and semantic guidance from different perspectives, further opportunities remain for jointly modeling global cross-modal dependencies, complementary frequency-domain information, and hierarchical semantic cues. This observation provides an important motivation for the design of our method.

3. Materials and Methods

In this section, we first give an overview of the overall process of SDFNet. Next, we discuss the core modules of SDFNet and present their architectures. Finally, we introduce the loss functions used in our method.

3.1. Framework Overview

The detailed architecture of SDFNet is shown in Figure 1. The framework consists of three parts, i.e., a semantic segmentation network, a shared encoder and a decoder. To extract high-level semantic priors from the input visible (I_vi) and infrared (I_ir) images, we employ a publicly released BiSeNet pretrained on the CamVid dataset as a frozen semantic extractor. Its parameters remain fixed throughout SDFNet training and are used only to generate semantic features and probability maps. BiSeNet adopts a bilateral architecture to capture decouples spatial details and categorical context via a Spatial Path (SP), a Context Path (CP), and a Feature Fusion Module (FFM). Given that the SP effectively preserves high-resolution spatial information and provides a substantial channel capacity, we specifically utilize it to generate the multi-scale semantic feature maps $S_{\left\{\mathit{ir}\right||\mathit{vi}{\}}^n},n=1,2,3$. This extraction process is formulated as:

$$S_{\left\{\mathit{ir}\right||\mathit{vi}{\}}^n}={\mathit{Net}}_{\mathit{seg}}\left(I;{\theta }_s\right),I\in \left\{I_{\mathit{vi}},I_{\mathit{ir}}\right\}$$

(1)

where θ_s represents the frozen parameters of the segmentation network, which remain non-trainable during the fusion process.

Figure 1. Network architecture of the proposed SDFNet.

During the feature encoding stage, I_vi and I_ir are propagated through two symmetric, independent branches to extract multi-scale deep representations. To balance the preservation of fine-grained textures and the abstraction of high-level semantics, each path consists of four convolutional layers with specific receptive field designs. Specifically, the initial layer employs a 7 × 7 kernel with a stride of 1, aimed at capturing extensive local textures while maintaining spatial resolution. The subsequent three layers utilize 3 × 3 kernels with a stride of 2 to achieve hierarchical downsampling and abstract multi-scale feature maps, denoted as $F_{\left\{\mathit{ir}\right||\mathit{vi}{\}}^n}$ (n = 1, 2, 3).

To bridge the cross-modal gap and facilitate effective information convergence, the multi-scale encoded features $F_{\left\{\mathit{ir}\right||\mathit{vi}{\}}^n}$ along with their corresponding structural semantic priors $S_{\left\{\mathit{ir}\right||\mathit{vi}{\}}^n}$ are fed into the frequency-domain feature aggregation module (FFAM). The FFAM is dedicated to enhancing unimodal local feature expressions while promoting non-local cross-modality feature interactions within the frequency domain. By incorporating $S_{\mathit{fuse}}^n$ as an additional input, which is formed by concatenating the semantic feature maps $S_{\mathit{vi}}^n$ and $S_{\mathit{ir}}^n$, the module aggregates the dual-stream inputs and generates the cross-modal coupled features, denoted as $F_{\mathit{fuse}}^n$.

The decoder incorporates a progressive semantic injection module (PSIM) [33] to reconstruct the final fused image from the coupled representations. Rather than employing primitive feature concatenation, the decoder consists of multiple cascaded Semantic Injection Modules (SIM) that top-down modulate features between adjacent scales. Concretely, given two adjacent feature layers denoted as $F_{\mathit{fuse}}^{n+1}$ and $F_{\mathit{fuse}}^n$, high-level semantic context from the deeper layer is progressively injected into the shallower layer. The feature flow strictly executes as follows: the rich semantic information from the third layer $F_{\mathit{fuse}}^3$ is first injected into the second layer $F_{\mathit{fuse}}^2$ and the updated second-layer signal is subsequently injected into the first layer $F_{\mathit{fuse}}^1$, the process can be described as:

$$\gamma =\mathrm{Sigmoid}\left({\mathit{Conv}}_{3\times 3}^{\uparrow }\left(F_{\mathit{fuse}}^{n+1}\right)\right)$$

(2)

$$\beta ={\mathit{Conv}}_{3\times 3}^{\uparrow }\left(F_{\mathit{fuse}}^{n+1}\right)$$

(3)

$$F_{\mathit{fuse}}^{n\prime }=\left(\mathrm{IN}\left(\mathrm{ReLU}\left({\mathit{Conv}}_{3\times 3}\left(F_{\mathit{fuse}}^n\right)\right)\right)\odot \gamma \right)+\beta$$

(4)

where the superscript arrow ↑ denotes an upsampling operation, IN (.) denotes the non-parametric instance normalization operation and the operator ⊙ represents the element-wise multiplication.

Finally, having absorbed comprehensive semantic context and spatial details from all preceding layers, the enriched first-layer feature $F_{\mathit{fuse}}^{1\prime }$ undergoes a decoding operation. In this operation, the features are first processed by a dense connection module consisting of three convolutional blocks to enhance fine-grained details. Finally, the refined features are projected back into the image space via a 3 × 3 convolutional layer followed by a Tanh activation function, yielding the high-fidelity fused image I_fuse. Therefore, the complete forward process of the proposed framework can be summarized as:

I_fuse = Net_SDF(I_vi,I_ir;W_f,θ_s)

(5)

where Net_SDF represents the fusion network, W_f represents the learnable parameters, and θ_s represents the frozen parameters.

3.2. Frequency-Domain Feature Aggregation Module

To effectively integrate multi-scale cross-modal features with semantic information, we propose a frequency-domain feature aggregation module (FFAM). As illustrated in Figure 2, the module comprises three core components: dual semantic-guided coordinate attention module (SGCA), a frequency-domain attention interaction module (FDAI), and a semantic feature enhanced module (SFE). Specifically, the SGCA leverages a semantic-aware mechanism to explicitly enhance the representations of local salient features in monomodal data, while the FDAI maps the features to the frequency domain, employing a multi-view self-attention mechanism to model global cross-modal dependencies. Finally, the SFE utilizes semantic priors to refine the aggregated cross-modal representations. This collective interaction empowers the FFAM to generate highly robust representations, thereby simultaneously satisfying the structural demands of image fusion and enhancing performance on downstream perception tasks.

Figure 2. The structure of FFAM.

3.2.1. Semantic-Guided Coordinate Attention Module

The inherent physical disparities between infrared and visible sensors induce significant cross-modal asymmetry in feature representations, which is mainly reflected in the complementary characteristics of thermal saliency and geometric structure information. To reinforce these modality-specific salient features, we propose a semantic-guided coordinate attention module (SGCA). As illustrated in the upper-right part of Figure 2, by leveraging contextual semantic priors, the SGCA utilizes an attention-guided weighted coordinate pooling strategy to accurately localize salient targets while suppressing background clutter.

Given the backbone visual feature map $F_{\left\{\mathit{ir}\right||\mathit{vi}{\}}^n}\in R^{C\times H\times W}$ and the auxiliary semantic feature map $S_{\left\{\mathit{ir}\right||\mathit{vi}{\}}^n}$, the network first computes a spatial semantic attention map M_s ∈ R^C×H×W, which explicitly highlights the spatial distribution of semantic targets. Utilizing M_s as a modulating weight, a normalized aggregation is performed on $F_{\left\{\mathit{ir}\right||\mathit{vi}{\}}^n}$ along the vertical (Y-axis) and horizontal (X-axis) dimensions. Specifically, the semantic-weighted coordinate feature responses $z_c^h\left(h\right)$ at height h and $z_c^w\left(w\right)$ at width w for the c-th channel and are formalized as:

$$M_s=\mathrm{Sigmoid}\left({\mathit{Conv}}_{1\times 1}\left(S_{\left\{\mathit{ir}\right||\mathit{vi}{\}}^n}\right)\right)$$

(6)

$$z_c^h\left(h\right)={\displaystyle \frac{{\sum }_{w=1}^W{F}_c\left(h,w\right)·M_{s,c}\left(h,w\right)}{{\sum }_{w=1}^W{M}_{s,c}\left(h,w\right)+ϵ}}$$

(7)

$$z_c^w\left(w\right)={\displaystyle \frac{{\sum }_{h=1}^H{F}_c\left(h,w\right)·M_{s,c}\left(h,w\right)}{{\sum }_{h=1}^H{M}_{s,c}\left(h,w\right)+ϵ}}$$

(8)

where ϵ is a minimal constant (set to 10⁻⁵ in this work) initialized to ensure numerical stability. Dividing the aggregated features by the sum of effective attention weights guarantees that the extracted 1D directional feature vectors z^h and z^w maintain a highly robust signal-to-noise ratio, regardless of the target scale.

After acquiring the directional feature vectors, we concatenate them spatially and apply a dimension reduction convolution to generate the intermediate feature f, where the channel dimension is compressed by a ratio r. Subsequently, f is split along the spatial dimension back into f^h and f^w. We then apply independent 1 × 1 convolutions and Sigmoid activations to generate the final directional attention weights g^h and g^w:

$$f=\mathrm{SiLU}\left({\mathit{Conv}}_{1\times 1}^{\downarrow }\left(\left[z^h,z^w\right]\right)\right)$$

(9)

$$g^h=\mathrm{Sigmoid}\left({\mathit{Conv}}_{1\times 1}^{\uparrow }\left(f^h\right)\right)$$

(10)

$$g^w=\mathrm{Sigmoid}\left({\mathit{Conv}}_{1\times 1}^{\uparrow }\left(f^w\right)\right)$$

(11)

Finally, the output feature $F_{\left\{\mathit{ir}\right||\mathit{vi}{\}}^n}^{\mathit{out}}$ is obtained by applying element-wise multiplication between the original feature $F_{\left\{\mathit{ir}\right||\mathit{vi}{\}}^n}$ and the generated directional attention weights:

$$F_{\left\{\mathit{ir}\right||\mathit{vi}{\}}^n}^{\mathit{out}}=F_{\left\{\mathit{ir}\right||\mathit{vi}{\}}^n}\times g^h\times g^w$$

(12)

This gating operation enhances spatially relevant responses while suppressing less informative background features. The directional attention weights along the horizontal and vertical dimensions help preserve positional dependencies and emphasize structurally informative regions, thereby producing coordinate-aware representations for subsequent frequency-domain cross-attention.

3.2.2. Frequency-Domain Attention Interaction Module

Spatial-domain attention mechanisms are commonly employed to model long-range dependencies in image representations. However, performing feature interaction exclusively in the spatial domain may not fully exploit the global representation characteristics provided by frequency-domain transformations. Motivated by this consideration, we propose a frequency-domain attention interaction module (FDAI), as illustrated in Figure 3. FDAI transforms cross-modal features into the frequency domain and performs attention-based interaction to capture long-range dependencies between infrared and visible representations. Through this design, complementary cross-modal information can be jointly modeled from a frequency-domain perspective, providing an alternative interaction mechanism to conventional spatial-domain attention. The effectiveness of this design is further examined through a controlled comparison with its spatial-domain counterpart in Section 4.4.3.

Figure 3. The structure of FDAI.

Initially, a layer normalization (LN) is applied to the input infrared and visible features, followed by a standard 1 × 1 convolution and a 3 × 3 depth-wise convolution to generate the corresponding query (Q_ir, Q_vi,), key (K_ir, K_Vi), and value (V_ir, V_Vi). In this module, the value features encapsulate critical spatial location information. Relying on the strict pixel-level alignment of the multimodal source images, we aggregate the infrared and visible value features to synthesize a cross-modal spatial representation V_cross endowed with rich local details:

V_cross = V_ir + V_vi

(13)

Subsequently, the Fast Fourier Transform (FFT) is employed to transform the Query and Key features from the spatial domain to the frequency domain. According to the convolution theorem, spatial global convolution corresponds to standard element-wise multiplication in the frequency domain. Therefore, instead of relying on resource-intensive spatial matrix multiplications, we perform distinct frequency-domain element-wise multiplications to synthesize global spatial filters. This unified filtering mechanism is designed to globally modulate the semantic feature maps from different modalities. Specifically, we design a dual-branch global filtering paradigm: the mono-modal global filtering branch modulates the cross-modal value representations, whereas the cross-modal global filtering branch modulates the mono-modal value representations. This strategy efficiently exploits complementary semantic structures and mitigates modality discrepancies without aggressive spatial downsampling. The calculation of the global modulated feature maps can be formulated as:

$$\left\{\begin{array}{c}{F}_{\mathit{mono}}^{\mathit{vi}}={\mathit{Conv}}_{1\times 1}\left(\mathrm{LN}\left({\mathcal{F}}^{-1}\left(\mathcal{F}\left(Q_{\mathit{vi}}\right)\odot \mathcal{F}\left(K_{\mathit{vi}}\right)\right)\right)\odot V_{\mathit{cross}}\right)\\ F_{\mathit{mono}}^{\mathit{ir}}={\mathit{Conv}}_{1\times 1}\left(\mathrm{LN}\left({\mathcal{F}}^{-1}\left(\mathcal{F}\left(Q_{\mathit{ir}}\right)\odot \mathcal{F}\left(K_{\mathit{ir}}\right)\right)\right)\odot V_{\mathit{cross}}\right)\end{array}\right.$$

(14)

$$\left\{\begin{array}{c}{F}_{\mathit{cro}}^{\mathit{vi}}={\mathit{Conv}}_{1\times 1}\left(\mathrm{LN}\left({\mathcal{F}}^{-1}\left(\mathcal{F}\left(Q_{\mathit{ir}}\right)\odot \mathcal{F}\left(K_{\mathit{vi}}\right)\right)\right)\odot V_{\mathit{vi}}\right)\\ F_{\mathit{cro}}^{\mathit{ir}}={\mathit{Conv}}_{1\times 1}\left(\mathrm{LN}\left({\mathcal{F}}^{-1}\left(\mathcal{F}\left(Q_{\mathit{vi}}\right)\odot \mathcal{F}\left(K_{\mathit{ir}}\right)\right)\right)\odot V_{\mathit{ir}}\right)\end{array}\right.$$

(15)

where $ℱ$(⋅) and $ℱ$⁻¹(⋅) denote the FFT and IFFT, respectively. Finally, all the attention feature maps are convolved by a channel shrinking convolution layer with kernel size of 1 × 1 to obtain the cross-modality semantic feature maps F_out.

This unified spatial-frequency dual-domain paradigm integrates local spatial saliency modeling with global frequency-domain interaction, enabling complementary cross-modal information to be captured from multiple perspectives while alleviating potential conflicts between heterogeneous modality features. Consequently, it provides an effective mechanism for integrating local structural details and global cross-modal dependencies in multimodal image fusion.

3.2.3. Semantic Feature Enhanced Module

As illustrated in the lower branch of Figure 2, the semantic feature enhanced module (SFE) is engineered to integrate high-frequency semantic boundary priors into the fused feature representations. Initially, a spatial-domain channel alignment operation is executed to establish dimensional consistency across heterogeneous feature spaces. Formally, the input semantic feature map S_fuse is projected via a 1 × 1 convolutional layer to strictly match the channel dimension of the backbone feature map, yielding the aligned representation S′_fuse. Subsequently, a 3 × 3 depth-wise convolutional layer is applied to $S_{\mathit{fuse}}^{\prime }$ to extract the preliminary semantic edge map S_edge:

$$S_{\mathit{fuse}}^{\prime }=\mathrm{SiLU}\left(\mathrm{BN}\left({\mathit{Conv}}_{1\times 1}\left(S_{\mathit{fuse}}\right)\right)\right)$$

(16)

$$S_{\mathit{edge}}={\mathit{DwConv}}_{3\times 3}\left(S_{\mathit{fuse}}^{\prime }\right)$$

(17)

To provide a deterministic structural prior for capturing high-frequency boundaries, the weights of this depth-wise convolutional layer are initialized using a classic Laplacian kernel K_Lap, defined as:

$$K_{\mathit{Lap}}=\left[\begin{array}{ccc}-1& -1& -1\\ -1& 8& -1\\ -1& -1& -1\end{array}\right]$$

(18)

Following edge extraction, both the fused backbone feature map F_out and the semantic edge map S_edge are mapped into the frequency domain via FFT. Within the spectral domain, these representations are decoupled into amplitude spectra $\mathrm{(𝒜}$), which capture energy and luminance distributions, and phase spectra ($𝒫$), which preserve structural layouts and geometric contours, according to:

$$\mathcal{F}\left(F_{\mathit{out}}\right)={\mathcal{A}}_{\mathcal{m}\mathcal{a}\mathcal{i}\mathcal{n}}\odot e^{j{\mathcal{P}}_{\mathcal{main}}}$$

(19)

$$\mathcal{F}\left(S_{\mathit{edge}}\right)={\mathcal{A}}_{\mathit{edge}}\odot e^{j{\mathcal{P}}_{\mathit{edge}}}$$

(20)

where j is the imaginary unit.

To enrich fine-grained textures, the semantic edge amplitude spectrum 𝒜_edge is injected into the backbone amplitude spectrum 𝒜_{𝓂𝒶𝒾𝓃} via a learnable scaling factor α ∈ ℝ^C. Finally, the enhanced amplitude spectrum is coupled with the original backbone phase spectrum $𝒫$_{𝓂𝒶𝒾𝓃}, and the IFFT is applied to project the synthesized spectral representation back into the spatial domain, generating the final enhanced feature map F_fuse:

$$F_{\mathit{fuse}}={\mathcal{F}}^{-1}\left(\right({\mathcal{A}}_{\mathcal{main}}+\mathsf{\alpha }\odot {\mathcal{A}}_{\mathcal{edge}})\odot e^{j{\mathcal{P}}_{\mathcal{main}}})$$

(21)

3.3. Semantic Driven Region-Adaptive Loss

In this work, the loss function of SDFNet consists of three parts, which are intensity loss L_int, gradient loss L_grad, and structural loss L_ssim. In summary, the total loss is defined as:

L_total = L_int + λ₁L_grad + λ₂L_ssim

(22)

where λ₁, λ₂ are hyperparameters for balancing the weights of various loss functions.

To incorporate semantic information into the reconstruction objective, the frozen semantic segmentation network takes I_ir as input and produces a semantic probability distribution P ∈ ℝ^C×H×W. Within its output label space, cars, pedestrians, and cyclists are defined as the infrared-preferred semantic set C_ir. This selection represents a manually specified, task-oriented class prior for road-scene fusion, since infrared observations can provide useful target cues for these dynamic objects, particularly under insufficient visible illumination or contrast.

Although the class preference is predefined, the spatial weighting is adaptively determined from the predicted semantic probabilities for each input. Specifically, the infrared-preferred weight is defined as:

$$W_{\mathit{ir}}\left(x,y\right)=\sum _{c\in C_{\mathit{ir}}}{P}_c\left(x,y\right)$$

(23)

and its complementary semantic weight is

W_comp(x,y) = 1 − W_ir(x,y)

(24)

Here, W_comp(x,y) represents the probability mass of all semantic categories outside C_ir, rather than a strict background mask. Accordingly, the intensity loss is formulated as:

$$L_{\mathit{int}}={\displaystyle \frac{1}{\mathit{HW}}}||W_{\mathit{ir}}\odot \left(I_{\mathit{fuse}}-I_{\mathit{ir}}\right){\left|\right|}_1+{\displaystyle \frac{1}{\mathit{HW}}}||W_{\mathit{comp}}\odot \left(I_{\mathit{fuse}}-I_{\mathit{vi}}\right){\left|\right|}_1$$

(25)

where || ⋅ ||₁ denotes the L₁ norm, and I_fuse, I_ir, and I_vi denote the fused, infrared, and visible images, respectively. This formulation encourages infrared intensity preservation in the predefined infrared-preferred semantic regions while retaining visible intensity information in the complementary regions.

It should be noted that C_ir is a task-specific semantic prior rather than a universally optimal class partition. When the semantic network or its output taxonomy changes, the preferred semantic categories may need to be redefined accordingly. Therefore, the current formulation is primarily designed for road-scene fusion, while reducing the dependence on manually specified class preferences remains a direction for future work.

To optimize the preservation of visible-light textures and useful infrared information in complex scenarios such as strong lighting and smoke obscuration, we introduce a gradient loss term L_grad to maintain gradient consistency between the fused image and the source images, which is calculated as shown below:

$$L_{\mathit{grad}}={\displaystyle \frac{1}{\mathit{HW}}}{‖\left|\nabla I_f\right|-\max \left(\left|\nabla I_{\mathit{ir}}\right|,\left|\nabla I_{\mathit{vi}}\right|\right)‖}_1$$

(26)

where ∇ represents the gradient operator, and |⋅| denotes the absolute operation.

To further maintain useful characteristics from the infrared and visible images, we introduce SSIM to construct structural loss for ensuring that the fusion result reveals structurally similar to the source images, which is formulated as:

$$L_{\mathit{ssim}}=1-{\displaystyle \frac{1}{2}}\left(\mathrm{SSIM}\left(I_f,I_{\mathit{ir}}\right)+\mathrm{SSIM}\left(I_f,I_{\mathit{vi}}\right)\right)$$

(27)

4. Experiments and Discussions

In this section, we first introduce the experimental configurations and implementation details in Section 4.1. In Section 4.2, the proposed method is compared with nine recent and representative image fusion methods through qualitative and quantitative evaluations. Section 4.3 further evaluates the downstream semantic utility of the fused images through semantic segmentation experiments, examining their ability to preserve task-relevant semantic information. In Section 4.4, a series of ablation experiments are conducted to investigate the contributions of the proposed modules and design choices. Finally, Section 4.5 analyzes the computational complexity and inference efficiency of SDFNet.

4.1. Experimental Configurations and Implementation Details

We conduct training on MSRS training set (1083 pairs) and subsequently assess its performance on MSRS test set (361 pairs) and TNO (25 pairs). MSRS dataset primarily depicts complex road scenes during both daytime and nighttime, with a spatial resolution of 480 × 640. It includes nine object classes: background, pedestrians, cars, parked vehicles, bicycles, guardrails, speed bumps, curves, and traffic cones.

To comprehensively verify the effectiveness of the proposed method, we conduct validation experiments at both the pixel and semantic levels. The pixel-level evaluation comprises qualitative and quantitative assessments. Specifically, the qualitative analysis involves the subjective appraisal of fused images based on human visual perception, focusing on clarity and contrast. In contrast, the quantitative analysis includes six metrics, i.e., entropy (EN), average gradient (AG), spatial frequency (SF), the sum of the correlations of differences (SCD), gradient-based fusion performance (Qabf), and visual information fidelity (VIF). EN indicates the information richness of the fusion image. AG reflects the richness of the image texture information. SF measures the overall level of activity in the image. SCD represents the correlation between the fused image and the source image. Qabf evaluates the image quality by calculating the blurriness and noise level of the image. VIF measures the information fidelity from the perspective of human visual perception. For all six metrics, higher values indicate superior fusion quality. Furthermore, at the semantic level, we employ the mean Intersection over Union (mIoU) to accurately evaluate the segmentation performance across the 9 distinct categories.

To provide a more up-to-date and technically relevant evaluation, SDFNet is compared with nine recent and representative infrared-visible image fusion methods, including U2Fusion [23] and TarDAL [38] as conventional and task-driven baselines, DATFuse [25] and CDDFuse [26] as Transformer-based and feature-decomposition methods, SeAFusion [32] and STSFusion [31] as semantic-guided approaches, SFDFusion [29] and TSFI-Fusion [30] as spatial-frequency and frequency-domain methods, and PPIFuse [4] as a physical-prior-guided method.

The proposed framework is implemented in PyTorch 2.3 with CUDA 11.8 and trained on a single NVIDIA GeForce RTX 4090 GPU. The fusion network is optimized using Adam with a batch size of 32 for 200 epochs. A warm-up cosine annealing learning-rate schedule is adopted. Specifically, during the first 5 epochs, the learning rate is linearly increased from 10⁻⁴ to 10⁻³. Subsequently, cosine annealing is applied over the remaining 195 epochs, gradually reducing the learning rate to 10⁻⁶. The loss weighting factors λ₁ and λ₂ are set to 100 and 10, respectively. No additional data augmentation is applied during training, and the random seed is fixed to 3407 for reproducibility. For downstream semantic evaluation, BANet is optimized using SGD with a momentum of 0.9, a weight decay of 5 × 10⁻⁴, and a base learning rate of 0.01. The learning rate is warmed up for the first 1000 iterations and subsequently decayed using a Poly schedule with a power of 0.9. The batch size is set to 16, and the network is trained for 80,000 iterations. The final checkpoints obtained after 200 epochs and 80,000 iterations are used for fusion and downstream semantic evaluation.

For semantic prior extraction, each infrared image is read as an 8-bit single-channel grayscale image and replicated into three identical channels to satisfy the input requirement of the pretrained BiSeNet. To assess whether the extracted semantic information is informative for infrared inputs, representative daytime and nighttime examples are presented in Figure 4. The predicted semantic responses are primarily concentrated in regions corresponding to cars, pedestrians, and cyclists, although some deviations remain around small objects and object boundaries. These results indicate that the frozen semantic network can provide useful target related spatial cues to guide the subsequent fusion process. In the color-coded segmentation maps, pedestrians are shown in dark olive, cars in dark purple, and cyclists in blue.

Figure 4. Qualitative validation of semantic information extracted from MSRS infrared images under daytime and nighttime conditions. From left to right: infrared input, ground truth, semantic prediction, and target probability map used in the semantic driven region-adaptive loss.

4.2. Comparison with Recent and Representative Methods

To comprehensively evaluate the performance of SDFNet, fusion experiments are conducted on two widely used benchmark datasets, MSRS and TNO. The proposed method is compared with several recent and representative image fusion methods, and its performance is systematically evaluated through both qualitative visual comparisons and quantitative metric-based analyses.

(1)

Fusion Results on the MSRS Dataset

The MSRS dataset encompasses diverse urban traffic environments under both daytime and nighttime conditions. To evaluate fusion performance under different illumination conditions, representative image pairs are selected for qualitative comparison. As shown in Figure 5, salient targets and background regions are highlighted by boxes. In the daytime scene, U2Fusion and TarDAL exhibit relatively weak contrast and limited detail preservation, while DATFuse and SFDFusion retain the overall scene structure but lose some fine textures. SeAFusion preserves relatively clear structural information, although some thermal targets are insufficiently emphasized and local textures are weakened. CDDFuse achieves relatively good overall contrast and edge preservation, whereas certain background details appear slightly blurred and the distinction between salient targets and their surroundings remains limited. PPIFuse, TSFI-Fusion, and STSFusion further improve scene clarity and local structural preservation, but varying degrees of imbalance between thermal-target saliency and visible-texture retention can still be observed in some regions. In the nighttime scene, the reduced amount of visible-light information makes it more challenging to simultaneously enhance salient targets and preserve background details. U2Fusion and TarDAL show insufficient representation of structures in dark regions, whereas DATFuse, SFDFusion, and TSFI-Fusion improve overall scene visibility but still exhibit varying degrees of texture degradation. CDDFuse, PPIFuse, and STSFusion preserve more structural and edge information, although some weak background details or thermal targets remain insufficiently emphasized. In comparison, SDFNet provides a more balanced integration of infrared thermal saliency and visible structural information, resulting in clear target–background distinction while maintaining relatively rich scene details under both daytime and nighttime conditions.

Figure 5. Qualitative comparison of the proposed SDFNet on two scenes (i.e., daytime and nighttime) of the MSRS dataset.

Table 1 reports the quantitative results of the 10 methods on the MSRS test set comprising 361 image pairs. SDFNet achieves the highest EN and SCD values of 6.73 and 1.74, respectively. For EN, its advantage over the second-ranked TSFI-Fusion is relatively small, with scores of 6.73 and 6.71, indicating comparable performance in terms of information richness. A more evident difference is observed for SCD, where SDFNet achieves 1.74, compared with 1.69 for SeAFusion and 1.62 for both CDDFuse and TSFI-Fusion. This result indicates that SDFNet performs particularly well in preserving complementary information from the source modalities. SDFNet also achieves the second-best AG and SF values of 3.78 and 11.59, respectively, whereas STSFusion obtains the highest scores of 4.09 and 12.56. This suggests that SDFNet maintains competitive gradient and spatial-detail information, although STSFusion performs better on these two detail-oriented metrics. For Qabf, SDFNet and CDDFuse both achieve 0.69, while PPIFuse obtains the highest value of 0.72. In terms of VIF, SDFNet achieves 1.02, compared with 1.07 for PPIFuse and 1.05 for CDDFuse, while still outperforming most of the remaining methods. Overall, the results show that SDFNet does not exhibit a uniform advantage across all evaluation metrics but achieves competitive and well-balanced performance.

Table 1. Quantitative performance comparison of different image fusion methods on the MSRS dataset. For each metric, the best and second-best results are highlighted in bold and underlined, respectively.

Methods	EN	AG	SF	SCD	Qabf	VIF
U2Fusion [23]	5.37	1.88	9.07	1.24	0.42	0.54
TarDAL [38]	5.28	2.13	5.98	0.71	0.18	0.42
SeAFusion [32]	6.65	3.68	11.11	1.69	0.67	0.97
DATFuse [25]	6.48	3.60	10.93	1.41	0.64	0.91
CDDFuse [26]	6.70	3.74	11.56	1.62	0.69	1.05
SFDFusion [29]	6.67	3.55	11.07	1.61	0.66	1.01
PPIFuse [4]	6.67	3.65	11.32	1.63	0.72	1.07
TSFI-Fusion [30]	6.71	3.67	11.15	1.62	0.67	1.01
STSFusion [31]	6.61	4.09	12.56	1.63	0.59	0.95
Ours	6.73	3.78	11.59	1.74	0.69	1.02

(2)

Fusion Results on the TNO Dataset

To evaluate the cross-dataset generalization ability, SDFNet and all competing methods are trained exclusively on the MSRS dataset and directly evaluated on TNO without additional training or fine-tuning. This unified cross-dataset protocol ensures a consistent evaluation setting for all compared methods.

Figure 6 presents the qualitative comparison results, where two representative regions of interest (ROIs) are enlarged for clearer visualization. U2Fusion, TarDAL, and DATFuse exhibit relatively weak target saliency or degraded background details in some regions. SeAFusion and CDDFuse preserve comparatively rich structural and thermal information, although local contrast and target boundaries remain less distinct in certain areas. TSFI-Fusion provides good complementary-information preservation, but some fine textures and edges are weakened in complex regions. SFDFusion and PPIFuse retain relatively clear background structures, whereas several salient thermal targets are less prominent. STSFusion preserves sharp spatial details, although the balance between target saliency and background information can still be improved. In comparison, SDFNet maintains clear target boundaries while preserving relatively rich background structures, resulting in a favorable balance between thermal-target saliency and scene-detail preservation.

Figure 6. Qualitative comparison of the proposed SDFNet on the TNO dataset.

Table 2 presents the quantitative comparison of the 10 methods on 25 TNO image pairs. SDFNet achieves the best performance in EN and SF and the second-best results in AG and Qabf, while maintaining competitive performance in SCD and VIF. Overall, the results demonstrate that SDFNet provides a well-balanced fusion performance across multiple evaluation criteria rather than favoring a single metric. Moreover, since the model is directly transferred from MSRS to TNO without fine-tuning, these results also indicate its favorable cross-dataset generalization capability.

Table 2. Quantitative performance comparison of different image fusion methods on the TNO dataset. For each metric, the best and second-best results are highlighted in bold and underlined, respectively.

Methods	EN	AG	SF	SCD	Qabf	VIF
U2Fusion [23]	6.40	4.03	9.88	1.56	0.43	0.54
TarDAL [38]	6.84	3.91	8.68	1.52	0.32	0.53
SeAFusion [32]	7.10	4.94	12.41	1.72	0.51	0.69
DATFuse [25]	6.58	3.74	10.09	1.45	0.51	0.70
CDDFuse [26]	7.12	4.90	13.15	1.76	0.54	0.77
SFDFusion [29]	7.08	4.50	11.39	1.51	0.53	0.85
PPIFuse [4]	7.03	4.57	12.11	1.56	0.58	0.92
TSFI-Fusion [30]	7.12	4.94	13.18	1.63	0.52	0.81
STSFusion [31]	7.18	5.36	13.18	1.59	0.53	0.74
Ours	7.27	5.21	13.27	1.71	0.55	0.79

4.3. Downstream Semantic Utility Evaluation

To evaluate the downstream semantic utility of the fused images, we further conduct comparative experiments on a downstream semantic segmentation task. BANet [39] is adopted as an independent downstream evaluation network. For each fusion method, the corresponding fused training and test images are first generated using the same MSRS data split. A separate BANet with the same architecture is then independently trained on the fused training images produced by each method and evaluated on its corresponding fused test images. All BANet models are trained using identical settings. Class-wise Intersection over Union (IoU) and mean Intersection over Union (mIoU) are used as the quantitative metrics. Figure 7 presents the qualitative segmentation results on the MSRS dataset under representative daytime and nighttime conditions. For clearer visualization, one representative region of interest (ROI) is highlighted by a red box and enlarged in each scene. In the color-coded segmentation maps, persons are shown in dark olive and cars in dark purple.

Figure 7. Qualitative semantic segmentation results of compared methods and SDFNet on the MSRS dataset.

In the daytime example, differences among the fusion methods are mainly reflected in the completeness of small-target segmentation and the delineation of object boundaries. Some competing methods produce incomplete or locally degraded predictions in the highlighted region, whereas recent semantic-aware and spatial-frequency-based approaches generally provide more stable segmentation results. In the nighttime example, the reduced visible-light information further increases the difficulty of preserving weak targets and fine structural cues, resulting in varying degrees of omission or boundary degradation for several methods. In comparison, SDFNet produces segmentation maps that are more consistent with the ground truth under both illumination conditions, with relatively complete target regions and clear semantic boundaries. These qualitative results indicate that the proposed fusion strategy effectively preserves information beneficial to downstream semantic understanding.

Table 3 reports the class-wise IoU and overall mIoU results on the MSRS dataset, with all values expressed as percentages. The differences among semantic categories are related to both their visual characteristics and the information-preservation mechanisms of the fusion methods. For Background, all methods achieve relatively high IoU values because this category occupies a large proportion of the scene and generally forms spatially continuous regions, while SDFNet achieves the highest value of 98.67%. For Car, SDFNet also obtains the best IoU of 91.05%. Vehicles generally exhibit salient thermal responses together with distinctive visible structural information, and the semantic-guided cross-modal interaction in SDFNet facilitates the joint preservation of these complementary cues.

Table 3. Quantitative semantic segmentation results of different fusion methods on the MSRS dataset. The best and second-best results are highlighted in bold and underlined, respectively.

Methods	IoU									mIoU
	Background	Car	Person	Bike	Curve	Car Stop	Guardrail	Color Cone	Bump
U2Fusion [23]	98.49	89.78	72.93	70.99	62.84	72.13	79.25	63.59	77.12	76.35
TarDAL [38]	98.50	90.04	73.17	71.13	61.92	72.94	73.08	64.77	75.67	75.69
SeAFusion [32]	98.61	90.38	74.56	72.09	65.26	74.38	84.34	66.21	78.35	78.24
DATFuse [25]	98.58	90.01	74.33	71.90	64.48	73.11	83.28	65.17	80.78	77.96
CDDFuse [26]	98.57	90.13	74.28	72.01	64.51	74.01	84.11	66.13	80.26	78.22
SFDFusion [29]	98.59	90.01	74.12	71.92	64.66	74.45	84.75	66.52	80.34	78.37
PPIFuse [4]	98.60	90.14	74.33	72.02	64.87	74.49	84.43	66.07	80.14	78.34
TSFI-Fusion [30]	98.61	90.27	74.59	72.14	65.47	75.02	84.45	65.75	80.41	78.52
STSFusion [31]	98.62	90.42	74.48	72.10	65.05	74.35	84.55	66.25	80.50	78.48
Ours	98.67	91.05	74.52	72.12	65.18	74.12	84.96	67.09	81.16	78.76

For relatively small foreground objects, the differences among the leading methods become much smaller. TSFI-Fusion achieves the highest Person IoU of 74.59%, while SDFNet obtains a comparable value of 74.52%. For Bike, TSFI-Fusion and SDFNet achieve 72.14% and 72.12%, respectively. Since pedestrians and bicycles usually occupy relatively small image regions and contain fine object boundaries, their segmentation is more sensitive to the preservation of local spatial and high-frequency information. The competitive performance of TSFI-Fusion on these categories is consistent with its spatial-frequency detail modeling, while the close results of SDFNet indicate that semantic guidance combined with frequency-domain interaction can also effectively preserve small-target information.

For Curve and Car Stop, TSFI-Fusion achieves the highest IoUs of 65.47% and 75.02%, whereas SDFNet obtains 65.18% and 74.12%, respectively. These categories are characterized by elongated structures, relatively weak local contrast, or fine spatial patterns and therefore rely strongly on local structural information. In contrast, SDFNet achieves the highest IoUs for Guardrail, Color cone, and Bump, reaching 84.96%, 67.09%, and 81.16%, respectively. These categories require simultaneous preservation of object boundaries, local structures, and sufficient target–background discrimination. The results suggest that the combination of semantic guidance and frequency-domain interaction in SDFNet provides a favorable balance between structural-detail preservation and semantic discriminability.

Overall, SDFNet achieves the highest mIoU of 78.76%, followed by TSFI-Fusion at 78.52% and STSFusion at 78.48%. The differences among these top-performing methods are modest. Therefore, rather than showing a uniform advantage across all semantic categories, SDFNet maintains competitive performance across different object types and achieves the best overall segmentation performance in terms of mIoU.

4.4. Ablation Experiments

4.4.1. Component-Wise Ablation

To systematically evaluate the individual and complementary contributions of the proposed components, we conduct controlled ablation experiments on SGCA, FDAI, and SFE. When SGCA or SFE is disabled, it is replaced by an identity mapping, while FDAI is replaced by element-wise addition of the aligned infrared and visible features to avoid additional structural or dimensional changes. The baseline disables all three modules. To further assess the capability of each component independently, three single-component variants containing only SGCA, FDAI, or SFE are additionally evaluated. All other network components and training settings are kept unchanged.

Figure 8 presents representative qualitative fusion and downstream segmentation results under daytime and nighttime scenes. The single-component variants already recover different aspects of the fused information compared with the baseline, indicating that each module provides an independent contribution. Removing SGCA weakens the emphasis on semantically salient regions, while removing FDAI leads to more evident degradation of background textures and low-contrast structures. Without SFE, local structural details and object boundaries are less well preserved. These differences are also reflected in the downstream segmentation results, where the ablated variants exhibit varying degrees of semantic omission. Overall, the qualitative results indicate complementary roles of SGCA, FDAI, and SFE in semantic-aware feature enhancement, cross-modal interaction, and local structural preservation.

Figure 8. The fusion and segmentation results of ablation experiments on MSRS dataset.

The quantitative results in Table 4 further confirm the effectiveness of the proposed components. Introducing SGCA, FDAI, or SFE individually improves all six metrics over the baseline, demonstrating that each module provides an independent contribution to fusion performance. Among the three single-component variants, FDAI achieves the largest overall improvement, highlighting its important role in cross-modal feature interaction. SGCA also yields consistent gains across all metrics, while SFE consistently improves the baseline and contributes to spatial-detail preservation. Pairwise combinations further improve the overall performance, demonstrating the complementary effects among the three components. The complete SDFNet achieves the best results across all six metrics, further validating the effectiveness of their joint integration.

Table 4. Ablation study of SGCA, FDAI, and SFE on the MSRS dataset. The best results are highlighted in bold.

Components			MSRS Evaluation Metrics
SGCA	FDAI	SFE	EN	AG	SF	SCD	Qabf	VIF
×	×	×	5.23	1.85	4.86	0.86	0.35	0.51
√	×	×	5.60	2.34	7.21	1.17	0.44	0.67
×	√	×	6.13	3.14	8.72	1.29	0.53	0.79
×	×	√	5.34	2.05	7.24	1.09	0.41	0.58
×	√	√	6.55	3.41	11.32	1.54	0.61	0.89
√	×	√	5.91	2.56	9.68	1.42	0.51	0.76
√	√	×	6.61	3.64	11.21	1.61	0.63	0.97
√	√	√	6.73	3.78	11.59	1.74	0.69	1.02

4.4.2. Ablation on Semantic Prior and Semantic Driven Supervision

To further investigate the role of semantic guidance in SDFNet, we conduct a 2 × 2 controlled ablation study on two components: the semantic prior P incorporated into the fusion network and the semantic driven region-adaptive supervision L. Four configurations are evaluated: G₀₀ (P = 0, L = 0), G₁₀ (P = 1, L = 0), G₀₁ (P = 0, L = 1), and G₁₁ (P = 1, L = 1), where G₁₁ denotes the full SDFNet. When L = 0, the semantic driven region-adaptive loss is replaced by a non-semantic content loss that assigns equal weights to the infrared and visible modalities:

$$L_{\text{non-semantic}}={\displaystyle \frac{1}{2\mathit{HW}}}\left(||I_f-I_{\mathit{ir}}{||}_1+||I_f-I_{\mathit{vi}}{||}_1\right)$$

(28)

While the remaining loss terms and training settings are kept unchanged. When P = 0, semantic information is removed from the fusion network without altering the remaining network structure or feature dimensions. Specifically, the semantic modulation map in SGCA is replaced by an all-ones map (M_s = 1), while the semantic input to SFE is replaced by a zero-valued tensor with the same shape as the original semantic feature. No additional source-image features are introduced to compensate for the removed semantic information. Notably, G₀₁ uses semantic information only during training and requires no semantic prior during inference. This controlled design enables the effects of the semantic prior and semantic driven supervision to be evaluated separately.

As shown in Table 5, both forms of semantic guidance contribute to improved fusion performance. Compared with the non-semantic baseline G₀₀, both G₁₀, which incorporates only the semantic prior, and G₀₁, which uses only the semantic driven region-adaptive loss, achieve improvements across all six fusion metrics. This indicates that each component contributes independently to the fusion process. When the two components are jointly employed, G₁₁ achieves the best overall fusion performance. A consistent trend is also observed in the downstream segmentation evaluation, where both G₁₀ and G₀₁ achieve higher mIoU than G₀₀, while G₁₁ achieves the highest mIoU of 78.76%. These results support the complementary contributions of the semantic prior and the semantic driven region-adaptive loss to fusion quality and downstream semantic preservation.

Table 5. Controlled ablation study of the semantic prior and semantic driven region-adaptive supervision in SDFNet. The best result for each metric is shown in bold.

Configuration	P	L	EN	AG	SF	SCD	Qabf	VIF	mIoU
G00	×	×	6.38	3.27	10.63	1.48	0.58	0.87	72.55
G10	√	×	6.60	3.61	11.23	1.65	0.66	0.96	76.17
G01	×	√	6.49	3.45	10.96	1.59	0.62	0.92	74.08
G11(full)	√	√	6.73	3.78	11.59	1.74	0.69	1.02	78.76

4.4.3. Frequency-Domain Versus Spatial-Domain Interaction

To directly evaluate the contribution of frequency-domain interaction, we construct a spatial-domain counterpart, termed SDAI, using the same projection and fusion framework as FDAI. SDAI retains the same input/output dimensions, Q/K/V projections, four-branch organization, residual connections, branch-wise output projections, and final fusion projection. The surrounding semantic-guidance modules, loss functions, and training settings are also kept unchanged. Thus, the principal difference between the two variants lies in whether feature interaction is performed in the spatial or frequency domain.

SDAI performs attention directly at the native spatial resolution of each feature level. After flattening the spatial dimensions, the projected features are represented as Q,K,V ∈ ℝ^C×HW, and each branch computes A = Softmax(Q^⊤K) followed by value aggregation using VA^⊤. The two intra-modal branches use the summed value representation (V_vi + V_ir), whereas the two cross-modal branches use the corresponding single-modality value features. The aggregated outputs are then reshaped to their original spatial dimensions and processed using the same projection, normalization, residual, and fusion operations as the corresponding FDAI branches.

As shown in Table 6, FDAI achieves higher EN, AG, SF, SCD, and Qabf than SDAI, whereas SDAI obtains a slightly higher VIF. In particular, the improvements in AG and SF indicate better preservation of gradient and fine detail information, while the higher SCD reflects stronger preservation of complementary information from the source modalities. Although the advantage is not uniform across all metrics, FDAI achieves better results on five of the six fusion metrics compared with its structurally comparable spatial-domain counterpart. These results provide direct empirical support for the effectiveness of frequency-domain interaction within the proposed framework.

Table 6. Controlled comparison between spatial-domain and frequency-domain interaction in SDFNet. The best result for each metric is shown in bold.

Interaction	EN	AG	SF	SCD	Qabf	VIF
SDAI	6.68	3.66	11.32	1.69	0.67	1.04
FDAI	6.73	3.78	11.59	1.74	0.69	1.02

4.5. Complexity Analysis

To quantitatively characterize the computational cost of SDFNet, we compare its model complexity and inference efficiency with several representative infrared-visible image fusion methods, including SeAFusion, CDDFuse, SFDFusion, PPIFuse, TSFI-Fusion, and STSFusion. The selected baselines include including recent Transformer-based methods and approaches involving spatial-domain or frequency-domain feature interaction related to the design of SDFNet. For consistency with the inference-time evaluation on the MSRS dataset, all computational complexity measurements are conducted using infrared-visible image pairs with a resolution of 640 × 480. We report the number of model parameters, FLOPs, and inference time per image pair. All inference-time measurements are performed under the same hardware environment, batch size, and numerical precision.

For SDFNet, the reported computational statistics correspond to the complete inference pipeline used in our experiments. Although the semantic feature extractor is kept frozen during fusion-network training, its computational cost is included in the evaluation because semantic prior extraction is required during inference. This setting provides a more transparent estimate of the practical inference cost of the proposed framework.

As shown in Table 7, SDFNet exhibits a competitive computational profile among the compared methods. At an input resolution of 640 × 480, SDFNet contains 1.38 M parameters and requires 79.22 G FLOPs. Its FLOPs are substantially lower than those of CDDFuse, TSFI-Fusion, and STSFusion and remain close to PPIFuse, although they are higher than those of SeAFusion and SFDFusion. In terms of inference efficiency, SDFNet requires 0.312 s per infrared-visible image pair, which is faster than CDDFuse, PPIFuse, TSFI-Fusion, and STSFusion, while SeAFusion and SFDFusion achieve shorter inference times. These results show that SDFNet does not achieve the lowest computational cost among all compared methods; instead, it maintains a reasonable balance between fusion performance and computational cost.

Table 7. Computational complexity and inference efficiency comparison on 640 × 480 infrared-visible image pairs from the MSRS dataset. “↓” indicates that a lower value is better.

Method	Params (M) ↓	FLOPs (G) ↓	Inference Time(s) ↓
SeAFusion [32]	0.17	50.99	0.172
CDDFuse [26]	1.19	547.73	0.412
SFDFusion [29]	0.14	42.80	0.073
PPIFuse [4]	4.24	77.25	0.375
TSFI-Fusion [30]	1.08	198.80	0.513
STSFusion [31]	1.42	530.48	0.537
SDFNet	1.38	79.22	0.312

5. Conclusions

In this paper, we propose SDFNet, a semantic driven frequency-domain framework for infrared and visible image fusion. By exploiting multi-scale semantic priors extracted from a frozen pre-trained segmentation network, SDFNet incorporates high-level semantic guidance into cross-modal feature aggregation and fusion optimization. Specifically, the proposed frequency-domain feature aggregation module integrates semantic-guided feature enhancement, frequency-domain cross-modal interaction, and semantic boundary refinement to effectively exploit the complementary information of infrared and visible modalities, while the semantic driven region-adaptive loss adaptively regulates the preservation of thermal saliency and visible information during training. Extensive qualitative and quantitative experiments demonstrate that SDFNet achieves strong overall fusion performance compared with recent representative methods. Comprehensive ablation studies further verify the effectiveness of the proposed modules, semantic guidance, and frequency-domain interaction. Complexity analysis also shows that SDFNet maintains relatively low parameter and computational complexity compared with several representative fusion methods. In addition, downstream semantic evaluation confirms its capability to preserve task-relevant semantic information, indicating the potential of SDFNet for multi-sensor intelligent perception applications.