Target-Aware Fusion: A Diffusion Model for Infrared and Visible Image Integration to Enhance Object Detection

Highlights

What are the main findings?

• A new framework for infrared and visible light image fusion based on conditional diffusion model is proposed.
• A Target-Aware Space Fusion Weight (TASFW) module and multi-stage dynamic fusion strategy are designeed to intelligently balance complementary information.

What is the implication of the main finding?

• State-of-the-art results are achieved on key image quality metrics across multiple public datasets.
• It has been proven that the proposed fusion method significantly improves the accuracy of downstream object detection and validates its practicality.

Abstract

There are differences in imaging characteristics between infrared and visible light images: visible light images can provide rich texture and color information, but imaging is limited in harsh weather conditions. Infrared images are based on the target’s thermal radiation characteristics and have the ability to resist environmental interference but lack details and background information. Effectively integrating the two can significantly enhance scene understanding ability and improve environmental perception and target recognition performance in applications such as intelligent driving. However, existing fusion methods still face challenges, especially in complex scenes where it is difficult to balance the full preservation of target information with the complete presentation of background details, often resulting in difficulties in extracting differentiated features from different modalities. This article proposes a target detection method based on the visible light infrared fusion diffusion model. This method introduces the Stable Diffusion architecture and designs a target perception spatial fusion weight module that can adaptively generate a spatial fusion weight map based on modal differences. By implementing a multi-stage dynamic fusion strategy, the fusion ratio is automatically adjusted at different diffusion stages. A full-step multi-step prediction mechanism is adopted to improve fusion quality and stability. Compared with existing methods, the method proposed in this article has significant advantages. Experiments on multiple publicly available datasets have shown that this method outperforms existing mainstream methods in key metrics such as Peak Signal to Noise Ratio (PSNR), Mean Square Error (MSE), and ean Absolute Error (MAE) and also demonstrates good detection performance in downstream tasks for object detection.

1. Introduction

1.1. Research Background and Significance

With the rapid development of sensor technology, multimodal image acquisition technology has become an important component of the field of computer vision. Infrared images and visible light images, as two complementary imaging modalities, play an irreplaceable role in their respective advantageous fields. Infrared imaging is based on the thermal radiation characteristics of objects and is sensitive to temperature differences. It can clearly display thermal targets such as humans, vehicles, animals, etc. in low visibility conditions such as darkness, smoke, and haze [1]. However, infrared images lack texture details and color information, and the background information is blurry, which is not conducive to scene understanding. On the contrary, visible light images contain rich texture, color, and detail information, but the imaging quality significantly decreases at night or in adverse weather conditions, and the object detection ability is limited [2].

The fusion technology of infrared and visible light images aims to organically combine the image information of the two modalities to generate a fused image that contains both infrared thermal target information and visible light texture details. This technology has significant application value in multiple fields. In night surveillance, infrared images can detect potential threat targets, while visible light images provide environmental details. The fusion of the two can achieve comprehensive perception of the surveillance scene [3]. In disaster rescue, infrared images can quickly locate survivors, visible light images provide terrain and environmental information, and fused images help develop more effective rescue strategies [4]. At night or in severe weather conditions, infrared sensors can detect potential dangers such as pedestrians and animals. Visible light cameras provide road signs and traffic signals and integrate information to improve the safety of the auto drive system [5]. Infrared imaging can detect hidden targets, visible light imaging provides terrain and landform information, and fused images support more accurate battlefield situational awareness [6]. In wildlife monitoring, infrared thermal imaging can detect the location of animals, visible light images provide habitat information, and fused images are helpful for population statistics and behavioral research.

1.2. Research Status

1.2.1. Traditional Methods

The fusion method of infrared and visible light images mainly relies on manually designed rules. These methods are typically based on theoretical frameworks such as multiscale transformations, spatial domain operations, sparse representation, or saliency detection and combine source image information by designing fusion rules within specific transformation or spatial domains. According to its core ideas and processing levels, it can be systematically divided into several categories: methods based on multiscale transformations, methods based on spatial domains, methods based on sparse representation and dictionary learning, methods based on saliency and region segmentation, etc. The method based on multiscale transformation is the mainstream framework in this field, and its core idea is to mimic the human visual system’s multi-resolution analysis of images. This type of method decomposes the source image into sub bands of different scales (usually including low-frequency baseband and high-frequency details), adopts corresponding fusion rules for each scale coefficient (such as coefficient maximization or weighted average), and finally reconstructs the fused image through inverse transformation. Typical multiscale transformations include pyramid transformations (such as Gaussian pyramids, Laplacian pyramids, and ratio pyramids), wavelet transforms, and subsequently developed multiscale geometric analysis tools (such as contour [7] and curvelet transformations). Among them, the ratio pyramid method proposed by Toet [8] highlights significant features by calculating the ratio of adjacent scales, but it is prone to producing halo artifacts. The dual tree complex wavelet transform adopted by Lewis et al. [9] can more effectively preserve edge information due to its good directional selectivity and translation invariance. The advantages of this method lie in its mature theory, high computational efficiency, and the ability to effectively separate the approximate structure and detailed texture of the image. Its limitations lie in the fact that the fusion rules are mostly heuristic designs, lacking adaptability, and the decomposition and reconstruction process may introduce distortion, as well as insufficient utilization of the global semantic information of the image.

The second type of traditional method operates directly in the pixel space of the image, collectively referred to as spatial domain-based methods. Its basic strategies include pixel weighted average, principal component analysis based on statistical properties, and high-pass filtering fusion that injects high-frequency details of infrared images into visible light images. The advantage of this type of method is its low computational complexity and simple and intuitive implementation. However, the globally unified fusion rules adopted by it are difficult to cope with the local differences in image content, which can easily lead to a decrease in contrast or loss of important features.

The third type of method is based on sparse representation and dictionary learning theory. This method assumes that the image has sparsity under some overcomplete dictionary. The typical process is to divide the image into blocks, perform sparse encoding, and reconstruct the fused image by fusing sparse coefficients (such as selecting the coefficient with the maximum L1 norm). The performance of this method depends crucially on the construction of the dictionary (often learned from training image blocks using algorithms such as K-SVD) and the optimization process of sparse coding [10]. Compared to using fixed base transformation methods, the adaptive dictionary obtained through learning can better express the essential structure of the image and has the potential to achieve better fusion effects. However, its computation time is relatively long, it is sensitive to noise, block processing may disrupt the overall consistency of the image, and its performance is highly dependent on the quality of the trained dictionary [11].

The fourth type of method attempts to mimic the visual attention mechanism, which is a fusion method based on saliency and region segmentation. The core idea is to use visual saliency models (such as the classic ITTI model [12]) or image segmentation techniques to divide the image into regions of different visual importance (such as the target area and background area) and then adopt differentiated fusion strategies for different regions (such as prioritizing the preservation of infrared thermal features in the target area). This method makes fusion decision-making more targeted and in line with human perceptual logic. However, the upper limit of its fusion effect heavily depends on the accuracy of the preceding saliency detection or image segmentation module, and the overall complexity of the algorithm is high, making it difficult to define a universally applicable “importance” measurement standard for various scenarios.

Overall, the advantages of traditional fusion methods lie in their clear theoretical framework and high computational efficiency, especially for multiscale transformations and spatial domain methods, and they do not rely on large-scale training data. However, its fundamental limitations are reflected in the following aspects: firstly, the fusion rules are highly dependent on manual prior design, making it difficult to adapt to complex and changing scene content and imaging conditions. Secondly, most methods involve local operations and lack understanding and utilization of the global semantics of the image. Finally, the processing flow is usually separated (including multiple independent stages such as decomposition, rule design, refactoring, etc.), which may lead to error accumulation. These inherent shortcomings have become important driving forces for the rise and mainstream adoption of data-driven deep learning methods in this field.

1.2.2. Fusion Method Based on Deep Learning

In recent years, the rapid development of deep learning has brought new breakthroughs to the field of image fusion. Convolutional neural networks (CNNs), due to their powerful feature extraction capabilities, have demonstrated significant advantages in image fusion tasks. Methods based on CNNs can be broadly categorized into the following groups:

Methods Based on Convolutional Neural Networks. Early work primarily utilized pre-trained CNNs for feature extraction, followed by manually designed fusion strategies [13]. While these methods improved fusion outcomes, the separation of the feature extraction and fusion processes did not fully leverage the benefits of end-to-end learning. Subsequent end-to-end CNN frameworks integrated these steps, enhancing adaptability and performance.

Methods Based on Generative Adversarial Networks. Inspired by their success in image generation, generative adversarial networks (GANs) have been extensively explored for image fusion. DDcGAN [14] employed a dual-discriminator structure to separately evaluate the similarity between the fused image and the source images. GANMcC [15] introduced multi-classification constraints to strengthen the discriminative capability for fused images. Although these methods excel in preserving texture details, they often suffer from unstable training and mode collapse.

Methods Based on Attention Mechanisms. Attention mechanisms, capable of focusing on crucial regions in an image, hold significant importance for fusion tasks. Some studies have integrated channel or spatial attention into fusion networks to enhance the retention of key information [16]. However, these methods often focus on local features and lack modeling of global contextual information.

Methods Based on Transformers. The success of the Transformer architecture in natural language processing has spurred its adoption in computer vision. Swin Transformer [17] utilizes a hierarchical design and shifted window mechanism to capture long-range dependencies while maintaining linear computational complexity. Transformer-based fusion methods, such as SwinFuse [18] and YDTR [19], can better model global context and demonstrate superior performance in complex scenes. Recent advancements continue to refine this approach. For instance, Zhao et al. proposed a network based on a residual interactive Transformer and cross-attention fusion to establish dependencies between global and local information during feature extraction, effectively preserving both texture details and thermal target contrast [20].

Emerging Methods Based on Diffusion Models. As a powerful class of generative models, diffusion models have recently been applied to image fusion tasks, leveraging their progressive denoising process that aligns well with the hierarchical nature of fusion. The Palette model [21] demonstrated the potential of diffusion models for image-to-image translation tasks, including fusion. Building on this, a generalized diffusion-based framework called RGBX-DiffusionDet has been proposed for multimodal object detection. It introduces dynamic fusion modules (DCR-CBAM and DMLAB) and novel regularization losses to effectively integrate heterogeneous modalities (e.g., RGB with infrared or depth) and enhance detection robustness [22].

Task-Driven Fusion Method. A recent trend is to design fusion networks for downstream advanced visual tasks, with multimodal object detection being one of the main trends. The purpose of this transformation is to go beyond a simple image quality evaluation system and ensure that the fused output contains features that are beneficial for detection tasks, such as the DMPFuse framework [23], which introduces detail enhancement and preservation modules and multi-path constraint modules to perform task-oriented feature alignment, representing a collaborative response. By utilizing shared fusion features on fusion, reconstruction, and segmentation paths, these methods aim to generate fusion results that are more compatible with downstream detectors. The evolution of RGB-IR object detection algorithm itself provides a key background for this trend. Early methods relied on simple pixel level fusion or post-decision level fusion, which often struggled to handle noise and misalignment in specific modalities. The emergence of deep learning has brought about a feature level fusion paradigm, such as extracting and combining features from each modality. The current state-of-the-art methods adopt more complex cross modal interaction mechanisms, such as attention-based feature modulation, dense connections, and the Transformer-based fusion module. However, significant challenges still exist. A major issue is that designing fusion architectures for real-time detection remains challenging due to the lack of large-scale, diverse annotated benchmark data, which also makes it difficult to evaluate the model’s generalization ability in different environments [24]. Therefore, task-driven image fusion methods have emerged. Similarly, other works explore text-guided fusion networks with heterogeneous encoders for deployment on platforms like UAVs [25].

The field of deep learning-based infrared and visible image fusion continues to evolve rapidly. The prevailing trends indicate a shift from networks focused solely on perceptual quality towards task-driven and generalizable architectures. Architecturally, there is a movement beyond CNNs and standard Transformers, with increasing adoption of diffusion models for their high-quality generation. The ultimate goal is progressing towards unified and versatile fusion frameworks capable of handling diverse modalities and tasks within a single, efficient system [26].

1.2.3. Existing Challenges

Although significant progress has been made in the fusion of infrared and visible light images using existing methods, several persistent and concrete challenges hinder their performance in complex practical scenarios, particularly when the fused results are intended for high-level tasks such as object detection:

Lack of Adaptive and Target-Aware Fusion Mechanisms. Most methods rely on fixed or globally uniform fusion rules, failing to dynamically adjust the fusion strategy based on local image content. This results in an inability to intelligently balance infrared target information and visible light detail information across different regions of the image. And, many existing attention or weighting schemes are driven solely by single-modality saliency, neglecting the fact that optimal fusion weights often depend on the degree of inter-modal discrepancy or conflict, not just the prominence of features in one modality. So, a mechanism that can spatially adapt fusion weights according to target saliency and modal differences is critically needed.

Ineffective Multi-Stage Information Integration. Existing approaches, including standard diffusion models applied to fusion, often employ a static fusion strategy throughout the generative or optimization process. They lack the capability to strategically emphasize complementary information (e.g., reinforcing thermal targets early and blending details later) at different stages, which limits the holistic quality and semantic coherence of the fused image.

Inefficiency and Instability in Generative Fusion. While diffusion models offer high-quality generation, their iterative denoising process is computationally intensive, making real-time application difficult. Furthermore, the randomness in the standard sampling process can lead to instability in fusion outputs. An efficient and stable inference strategy is required to make diffusion-based fusion practical.

1.3. Contributions

In response to the aforementioned challenges, this study aims to enhance object detection performance in complex environments by generating superior fused images. To this end, we propose a novel target-aware diffusion model for infrared and visible image fusion. The main contributions of this work are summarized as follows:

Task-Oriented Framework for Detection-Driven Fusion. Departing from the prevalent focus on general-purpose image fusion, this work explicitly formulates the fusion task as a critical preprocessing stage for robust object detection. The proposed framework is designed and optimized with the overarching goal of generating fusion results that maximize the performance of downstream detectors, particularly under challenging conditions such as low visibility and cluttered backgrounds.

Novel Application of Conditional Diffusion Models to Multimodal Fusion. We introduce the powerful Stable Diffusion architecture into the domain of infrared and visible image fusion. By leveraging its capacity for high-quality, iterative refinement within a conditional generative framework, our model effectively integrates complementary information from both modalities. This represents a significant shift from deterministic fusion networks to a probabilistic, generation-based paradigm.

Design of a Target-Aware Spatial Fusion Weight (TASFW) Module. We propose the TASFW module, which differs from the general attention mechanism by introducing an explicit modal difference branch to explicitly quantify modal differences. This enables the network to determine fusion rates based on modal conflicts, ensuring reliable fusion. Moreover, this module dynamically analyzes the significance of the target and the differences between modalities to generate a spatially adaptive fusion weight map. This ensures that areas containing prominent thermal targets receive higher weights from the infrared mode, while the background area retains richer texture details from the visible mode.

Introduction of a Multi-Stage Dynamic Fusion Strategy within the Diffusion Process. Capitalizing on the sequential denoising nature of diffusion models, we design a time-dependent fusion strategy. The process is divided into three distinct stages: an early reinforcement stage that prioritizes the structural information of thermal targets, a mid-term balance stage that progressively blends details, and a late refinement stage that enhances visual texture. This allows for intelligent, stage-specific information integration.

An Efficient and Stable Full-Step Multi-Step Prediction Mechanism. To mitigate the high computational cost and potential instability associated with standard diffusion sampling, we propose a refined prediction mechanism. It employs multiple, lightweight prediction iterations at each diffusion step, significantly improving the fusion quality and output stability without proportionally increasing the inference time, making the approach more practical for real-world applications.

Comprehensive Validation on Multiple Downstream Detection Tasks. We extensively evaluate our fusion method not only with traditional image quality metrics but also, and more importantly, by its direct impact on object detection accuracy. Experimental results on benchmarks AVMS demonstrate that images fused by our method substantially boost the performance of standard detectors like YOLO, confirming the practical utility of our task-driven approach.

2. Related Work

2.1. Method Based on Manual Design Rules

The field of infrared and visible image fusion was initially dominated by methods built on manually designed rules. These approaches rely on explicit feature extraction and fusion rules derived from prior knowledge, operating without data-driven training. Based on established taxonomies in the field, these traditional methods can be systematically classified into four primary categories: multiscale transform-based methods, sparse representation-based methods, saliency-guided methods, and spatial domain-based methods. Each category has its theoretical foundation, representative techniques, and inherent limitations, which are comprehensively analyzed in recent surveys [27].

(1) Multiscale Transform-Based Methods: The core principle of this category is to decompose source images into different frequency bands (e.g., approximation and detail coefficients) using multiscale transforms. Early work employed pyramids, such as the Laplacian pyramid [28], while later advances utilized wavelets [29] and more sophisticated transforms like curvelets and contourlets for better directional representation [30]. The fusion is achieved by designing specific rules (e.g., coefficient selection based on local energy or variance) for the different scales before reconstruction. A key advantage is their ability to separate structural and textural information effectively, providing clear physical interpretability. However, their performance is often limited by heuristic and non-adaptive fusion rules. The decomposition process itself may also introduce artifacts like ringing or spectral aliasing, and capturing complex, global context remains challenging [31].

(2) Sparse Representation-Based Methods: These methods assume an image patch can be represented by a sparse linear combination of atoms from an over-complete dictionary. The standard pipeline involves (1) dividing images into patches, (2) coding them sparsely using a pre-learned dictionary (e.g., via K-SVD), (3) fusing the sparse coefficients (e.g., by choosing the one with the largest absolute value), and (4) reconstructing the fused image. Compared to fixed-basis transforms, the learned dictionary adapts to image content, potentially preserving finer details [32]. Major drawbacks include high computational cost due to patch-wise processing, sensitivity to noise and mis-registration, and the risk of introducing block artifacts. Performance is also heavily contingent on the quality and generality of the trained dictionary.

(3) Saliency-Guided and Region-Based Methods: This category aims to make the fusion process more aligned with human visual perception by focusing on perceptually important regions. It first identifies salient targets or segments the image into regions (e.g., target vs. background) using models like the Itti–Koch saliency model or segmentation algorithms. Subsequently, differentiated fusion strategies are applied, for instance, emphasizing infrared thermal contrast in salient target regions while preserving visible light texture in backgrounds. This approach can yield more natural and target-aware results [33]. However, its effectiveness is entirely dependent on the accuracy of the preceding saliency detection or segmentation step. In complex, multimodal scenarios, obtaining a reliable and consistent saliency map is itself a significant challenge, and errors in this stage directly propagate to the fusion outcome [34].

(4) Spatial Domain-Based Methods: These are the most straightforward approaches, operating directly on pixel intensities. Simple techniques include pixel averaging and principal component analysis (PCA). A more common strategy is to inject the high-frequency details (extracted via high-pass filtering) from the infrared image into the visible image to highlight thermal targets while preserving the visible background structure. The main advantages are extremely low computational complexity and simplicity of implementation. The fundamental limitation is their use of global or fixed rules, which fail to account for local content variations. This often results in reduced overall contrast, loss of subtle details, or unnatural blending of information, making them unsuitable for complex fusion tasks.

In summary, traditional methods are founded on solid principles, offering high computational efficiency and interpretability without requiring training data. Their principal limitations are threefold: (1) reliance on hand-crafted, non-adaptive fusion rules that struggle with diverse and complex scenes; (2) a focus on local or patch-level operations, lacking a global semantic understanding of the image; and (3) multi-stage pipelines where errors from decomposition, saliency detection, or segmentation can accumulate. These inherent shortcomings became the primary driving force for the community’s shift towards data-driven deep learning paradigms, which learn adaptive fusion strategies directly from data.

2.2. Deep Learning Method Based on Convolutional Neural Network

With the rapid development of deep learning technology, convolutional neural networks (CNN) have been widely used in the field of image fusion. This type of method utilizes the powerful feature extraction capability of CNNs to automatically learn fusion rules from data, significantly improving fusion performance. Li et al. [13] proposed an image fusion method based on CNN, which uses deep learning networks to extract multi-layer features, first generates candidate versions of multiple detail contents through different strategies, and then uses the maximum value selection strategy to obtain the final fused detail content. DDcGAN [14] uses a dual discriminator conditional generative adversarial network, where one discriminator evaluates the consistency between the fused image and the infrared image, and the other evaluates the consistency with the visible light image. Through adversarial training, a more natural fused image is generated. FusionGAN [3] directly applies generative adversarial networks to image fusion, where the generator is responsible for fusing the images and the discriminator evaluates the authenticity of the fused images. This framework can generate textures with rich details but has poor training stability.

The CNN-based method has significant advantages over traditional methods: it can automatically learn complex fusion rules from data and has strong adaptive capabilities. By optimizing overall performance through end-to-end training, the error accumulation of phased optimization in traditional methods is avoided, capable of extracting multi-level semantic features and achieving more intelligent information fusion. However, the CNN method also has obvious drawbacks: the local receptive field of convolution operations limits its ability to model long-range dependencies, and inconsistencies may occur when dealing with large-scale structures. A large amount of annotated data is required for training, and the cost of data acquisition is relatively high in practical applications. The network structure design is complex, and hyperparameter adjustment requires a lot of experimental experience. Although generative adversarial networks can generate natural textures, they suffer from training instability and pattern collapse issues.

2.3. Neural Network Method Based on Transformer

In recent years, the success of Transformer architecture in natural language processing has rapidly expanded to the field of computer vision, and its powerful global modeling capabilities have provided new ideas for image fusion. The Swin Transformer [17] proposed by Liu et al. adopts a hierarchical design and shift window mechanism, achieving global attention while maintaining linear computational complexity, providing a powerful infrastructure for visual tasks. Wang et al. [18] applied Swin Transformer to the fusion of infrared and visible light images, proposed SwinUse network, and achieved better performance than the CNN method by fusing multiscale features through a residual connection and attention mechanism. Zhang et al. [35] proposed an end-to-end infrared and visible light image fusion network called FSATFusion, which efficiently extracts and fuses multi domain features through the frequency spatial attention mechanism (FSAM) and the improved Transformer module (ITM), achieving better fusion performance and computational efficiency than existing methods on multiple public datasets. Zhao et al. [20] proposed a fusion model based on residual interaction transformer and cross-attention, which uses the residual dense module to extract basic features and residual interactive Transformer for global modeling and local feature extraction, and allows the two to interact, as well as a cross-attention fusion module. The experimental results achieved good results on multiple datasets.

The Transformer-based method has significant advantages over the CNN method: the self attention mechanism can capture global contextual information and model long-range dependencies and help maintain the overall consistency of the image. The introduction of positional encoding enables the model to understand spatial structural relationships and is more accurate in handling geometric structures. Hierarchical design allows for feature extraction at different scales, achieving multi granularity information fusion. However, the Transformer method also faces challenges: the computational complexity of the self attention mechanism is quadratic with the input size, resulting in low efficiency in processing high-resolution images, requires a large amount of training data to fully utilize performance, and is prone to overfitting on small datasets. The design of position encoding has a significant impact on performance, and different tasks may require different encoding schemes. The model has poor interpretability, and it is difficult to understand the specific meaning of attention weights.

2.4. Method Based on Diffusion Model

The diffusion model, as an emerging generative model, has performed well in image generation and editing tasks and has recently begun to be applied in the field of image fusion. The diffusion model generates images through a gradual denoising process and has a natural fit with the progressive nature of image fusion. The Palette model [21] proposed by Saharia et al. demonstrates the potential of diffusion models in multitask image to image conversion, including tasks such as image coloring, restoration, and fusion. SRDiff [36] applies diffusion models to image super-resolution, providing a reference for the application of diffusion models in image enhancement tasks. RePaint [37] utilizes a diffusion model for image restoration, filling in missing areas through a conditional generation process, demonstrating the advantages of diffusion models in partial information reconstruction. Xu et al. [38] proposed a RED model that achieves image fusion tasks for diffusion models by introducing a novel explicit supervised training framework. Its core innovation lies in a reversible fusion method that rephrases the diffusion sampling step as a reversible neural network. This design enables end-to-end training under strong supervision of the source image, maintaining excellent performance of thermal targets and texture details in visible infrared and medical image fusion tasks. Xing et al. [39] proposed a salient target mask-guided diffusion model (MGDIF) for infrared and visible light image fusion. This method explicitly guides the fusion process through a mask and synergistically optimizes the infrared target saliency and visible light texture details in the diffusion model, effectively suppressing noise interference.

The method based on large model diffusion represents the most advanced generation technology and has unique advantages: the progressive generation process of diffusion models highly matches the multi-level characteristics of image fusion and can control information fusion at different granularities. By gradually refining the image through the reverse diffusion process, highly natural and detail rich outputs can be generated. The conditional diffusion model can flexibly integrate various guiding information (such as text descriptions, category labels, reference images, etc.) to achieve highly controllable fusion. The theoretical foundation is solid, and the training stability is better than that of generative adversarial networks. However, diffusion methods also have obvious shortcomings: the inference process requires multiple iterations (usually 50–1000 steps), the computational cost is high, and it is difficult to meet real-time application requirements. The training process is complex and requires careful design of noise scheduling and loss functions. The model has a large number of parameters and requires high computational resources. The design of conditional mechanisms has a decisive impact on performance and requires optimization for specific tasks.

2.5. Problem Analysis

We can see the evolution trend of image fusion technology from manual design to automatic learning, from local processing to global modeling, and from deterministic methods to probabilistic generative models. Although traditional methods have high efficiency, their performance is limited. CNN-based methods have made breakthroughs in performance but are limited by local receptive fields. Transformer-based methods have solved global modeling problems but need to improve computational efficiency. Diffusion-based methods have reached new heights in generation quality but face challenges in real-time performance.

The current research trend presents the following characteristics: (1) A hybrid architecture has become mainstream, combining CNN’s local feature extraction ability with Transformer’s global modeling ability. (2) A lightweight design has received attention, reducing computational complexity while maintaining performance. (3) Multitask learning frameworks are gradually becoming popular, combining image fusion with downstream tasks such as object detection and semantic segmentation for joint optimization. (4) Unsupervised and self-supervised learning methods have been developed to reduce reliance on annotated data. (5) Cross-modal fusion technology continues to advance, expanding from simple image pair fusion to multimodal fusion of video sequences, multispectral data, and more.

The method proposed in this article belongs to the category of methods based on large model diffusion but innovatively introduces target perception mechanism and multi-stage fusion strategy. While maintaining the high-quality generation ability of the diffusion model, it has been specifically optimized for the characteristics of infrared and visible light fusion tasks. Compared with existing diffusion fusion methods, this approach has significant advantages in preserving target information and balancing detail information, providing valuable exploration for this emerging research direction.

3. Materials and Methods

3.1. Network Architecture

The proposed model is a diffusion model-based infrared and visible light image fusion method. The overall architecture is shown in Figure 1. The network is composed of four core components: (1) the VAE encoder/decoder, which maps images between the image space and the latent space; (2) the Target-Aware Spatial Fusion Weight (TASFW) module, which generates spatially adaptive fusion weight maps to balance the contributions from the two modalities; (3) the diffusion fusion process, which performs multi-stage fusion in the latent space using a DDIM scheduler; and (4) the Conditional Encoding Module, which provides timestep information to guide the diffusion process. The model takes a pair of registered infrared $I_{ir}\in {\mathbb{R}}^{H\times W}$ and visible light $I_{vis}\in {\mathbb{R}}^{H\times W}$ images as input, and outputs a fused $I_{fuse}\in {\mathbb{R}}^{H\times W\times 3}$ image. The model mainly consists of the following core components:

VAE Encoder/Decoder: Built upon the variational autoencoder from Stable Diffusion v1.5, it facilitates bidirectional conversion between image space and latent space. First, the input RGB image $I\in {\mathbb{R}}^{H\times W\times 3}$ is processed by the encoder, which progressively downsamples it through multiple convolutional blocks and residual connections, finally yielding the latent representation $z\in {\mathbb{R}}^{h\times w\times c}$, where $h=H/f$, $w=W/f$, $f=8$ is the downsampling factor, and $c=4$ is the number of latent channels. The encoding process adopts a deterministic strategy, directly outputting the mean $\mu$ of the distribution as the latent representation to enhance stability. The decoding process performs the inverse operation, reconstructing the image $\widehat{I}\in {\mathbb{R}}^{H\times W\times 3}$ from the latent representation z via upsampling layers and residual blocks.

Target-Aware Spatial Fusion Weight (TASFW) Module: This module first concatenates the infrared latent representation $z_{ir}$ and the visible light latent representation $z_{vis}$ along the channel dimension into $z_{cat}\in {\mathbb{R}}^{h\times w\times 2c}$. Subsequently, high-level semantic features $F_{enc}\in {\mathbb{R}}^{h\times w\times d}$ are extracted through the feature encoder. Simultaneously, the modal difference estimation branch computes the absolute difference $D_{abs}=|z_{ir}-z_{vis}|$ and generates the difference map $C\in {[0,1]}^{h\times w}$. The weight generation head synthesizes the encoded features and the difference map to produce the base weight $W_{base}$, which is then combined with learnable parameters $\alpha ,\beta$ via weighted fusion to obtain $W_{combined}$. Finally, the final weight map $W\in {[0,1]}^{h\times w}$ is output through a Sigmoid function and scaling factor $\gamma$. This weight map is used for the subsequent preliminary fusion: $z_{pre}=W\odot z_{ir}+(1-W)\odot z_{vis}$.

Diffusion Fusion Process: A multi-stage fusion process based on the DDIM scheduler, comprising $T=50$ denoising steps. This process divides the diffusion stages into three substages: the early reinforcement stage ($t\in [T,\alpha T]$), the mid-term balance stage ($t\in [\alpha T,\beta T]$), and the late refinement stage ($t\in [\beta T,0]$). At each timestep t, the system dynamically adjusts the fusion weight $W_t$ according to the current stage and computes the conditionally fused latent representation $z_t^{\mathrm{fuse}}=W_t\odot z_{ir}+(1-W_t)\odot z_{vis}$. Subsequently, the noise ${ϵ}_{\theta }$ is predicted via the UNet denoising network, with the preliminary fused representation $z_{\mathrm{pre}}$ serving as a conditional input through a cross-attention mechanism to achieve global context awareness. Within the multi-step prediction mechanism, the early stage employs a three-step prediction (base prediction, intermediate update, and refined prediction), while the mid-to-late stages use a two-step prediction, enhancing fusion quality and stability through iterative optimization.

Conditional Encoding Module: This module consists of a timestep encoder and a text encoder. The timestep encoder converts the current diffusion step t into an embedding vector, which is injected into each layer of the UNet via adaptive layer normalization (AdaIN). The text encoder, based on the CLIP ViT-L/14 model, transforms the input prompt text into a $77\times 768$ text embedding, serving as a global semantic condition to guide the fusion process. This module ensures the diffusion process evolves under temporal progression and semantic guidance, enabling controllable image generation.

The entire fusion process incorporates multiple feedback regulation mechanisms. In the TASFW module, the generation of the weight map W depends on the difference features of the input modalities, forming a feedforward regulation. During the diffusion fusion process, the fusion weight $W_t$ at each timestep is dynamically adjusted according to the current stage, achieving temporal adaptive regulation. Furthermore, the multi-step prediction mechanism performs multiple noise predictions and latent representation updates within a single timestep, forming a local iterative feedback. The final output of the entire model is the fused image $I_{fuse}\in {\mathbb{R}}^{H\times W\times 3}$, which retains both the thermal target information from the infrared image and the texture details from the visible light image, achieving information balance and visual naturalness through the multi-stage fusion strategy.

While our task is purely visual, the UNet denoiser inherits weights from Stable Diffusion v1.5, which intrinsically expects a text-conditioning input via cross-attention layers. To preserve compatibility with these pre-trained generative priors without introducing semantic bias, we utilize a fixed, empty text prompt (i.e., an empty string ""). The text encoder maps this to a constant embedding vector, which acts as a domain regularizer to ensure the fused output remains on the manifold of natural images. No descriptive or task-specific prompts are used during training or inference.

3.2. Latent Space Encoding and Decoding

3.2.1. VAE Encoder

The encoder structure is shown in the Figure 2. The VAE encoder maps RGB images to a low-dimensional latent space. For an input image $I\in {\mathbb{R}}^{H\times W\times 3}$, the encoding process can be expressed as

$$z=\mathrm{Encoder}\left(I\right)\in {\mathbb{R}}^{h\times w\times c}$$

(1)

where h = H/f, w = W/f, f = 8 is the downsampling factor, and $c=4$ is the number of latent channels. The encoder is built based on a convolutional neural network, containing multiple residual blocks and downsampling layers.

In the original Stable Diffusion, the encoder outputs the parameters of a latent distribution $\mathcal{N}(\mu ,{\sigma }^2)$, and the latent representation z is sampled via the reparameterization trick. However, the randomness introduced by the sampling process may lead to instability in the fusion results. Therefore, the proposed model adopts a deterministic encoding strategy, directly using the mean $\mu$ as the latent representation:

$$z=\mu$$

(2)

Experiments show that this deterministic encoding strategy significantly improves the stability of the results while maintaining fusion quality.

3.2.2. VAE Decoder

The decoder structure is shown in the Figure 2. The VAE decoder performs the inverse process of encoding, reconstructing the image from the latent representation:

$$\widehat{I}=\mathrm{Decoder}\left(z\right)\in {\mathbb{R}}^{H\times W\times 3}$$

(3)

The decoder is also built based on a convolutional neural network, containing multiple residual blocks and upsampling layers.

3.2.3. Properties of Latent Space

The latent space, as a high-level representation of images, has the following important properties:

Compactness: The dimensionality of the latent space is much lower than that of the original image space ($h\times w\times c\ll H\times W\times 3$), achieving efficient data representation.

Continuity: Neighboring points in the latent space correspond to similar image content, supporting smooth interpolation and editing operations.

Disentanglement: Different latent dimensions tend to encode different semantic attributes, facilitating controllable image generation.

In the image fusion task, these properties of the latent space provide an ideal platform for the effective fusion of multimodal information. The representations of infrared and visible light images in the latent space can be viewed as different perspectives of the same scene. Through appropriate fusion strategies, a fused representation containing the advantages of both can be generated.

3.3. Target-Aware Spatial Fusion Weight (TASFW) Module

The TASFW module is designed to generate spatially adaptive fusion weight maps by leveraging the complementary information from infrared and visible light latent representations. It dynamically assigns higher weights to target regions, such as heat sources, and lower weights to background areas. The module consists of five main components: input feature concatenation, a feature encoder, an infrared saliency estimation branch, a modality difference estimation branch, and a weight generation head. The specific structure is shown in the Figure 3. Unlike conventional attention mechanisms that reweight features based on single-stream statistics, TASFW explicitly models cross-modal discrepancy through a dedicated difference estimation branch, enabling the network to determine fusion ratios based on local modal conflicts rather than mere saliency.

3.3.1. Input Feature Concatenation

The module first concatenates the infrared latent representation and the visible light latent representation along the channel dimension. Both inputs have the same spatial dimensions and a channel count of four, resulting in a combined feature map of shape batch size by eight by height by width. This concatenated tensor simultaneously preserves the thermal radiation patterns from the infrared modality and the texture details from the visible modality, providing a rich foundation for subsequent analysis.

3.3.2. Feature Encoder

The feature encoder processes the concatenated feature map to extract high-level semantic representations. It is implemented as a sequential stack of four convolutional blocks. Each block contains a 3 × 3 convolution with padding of one, followed by group normalization and a SiLU activation function. The first block maps the eight-channel input to sixty-four channels. The second block maintains sixty-four channels. The third block reduces the channel count to thirty-two. The fourth block keeps thirty-two channels. The group normalization in the first two blocks uses eight groups, while the last two blocks use four groups. The output of the encoder is a feature map of shape batch size by thirty-two by height by width, which captures multi-scale contextual information while maintaining the original spatial resolution.

3.3.3. Infrared Saliency Estimation Branch

This branch isolates the thermal saliency directly from the infrared latent representation. It begins with a 3 × 3 convolution with padding of one, transforming the four-channel input into thirty-two channels. This is followed by group normalization with four groups and a SiLU activation. A subsequent 1 × 1 convolution preserves the thirty-two channels. After these operations, the feature map is averaged across the channel dimension, producing a single-channel map. A sigmoid function is then applied to obtain the infrared saliency map, where each spatial location indicates the likelihood of being a salient target according to the infrared modality.

3.3.4. Modality Difference Estimation Branch

To quantify the discrepancy between the two modalities, this branch first computes the element-wise absolute difference between the infrared and visible latent representations, resulting in a four-channel difference map. This map is processed by a dedicated convolutional network consisting of a 3 × 3 convolution with padding of one that increases the channel count from four to thirty-two, followed by group normalization with four groups and a SiLU activation. A final 1 × 1 convolution reduces the thirty-two channels to a single channel, and a sigmoid function is applied to produce a contrast map. High values in this map indicate regions where infrared and visible information differ substantially, often corresponding to object boundaries or thermal anomalies that require careful fusion. This explicit quantification of inter-modal divergence distinguishes TASFW from typical spatial weighting modules.

3.3.5. Weight Generation Head

The weight generation head creates an initial base weight map from the encoded features. It employs a single 1 × 1 convolution that reduces the thirty-two-channel encoded feature map to a one-channel map. This base weight serves as a foundation for the final fusion weights.

3.3.6. Weight Composition and Final Activation

The final fusion weight map is obtained by combining the base weight map with the infrared saliency map and the contrast map. The infrared saliency map is multiplied by a learnable scalar parameter, and the contrast map is multiplied by another learnable scalar. These weighted maps are added to the base weight map to enhance the contrast between target and background regions. The resulting weight map has values in the range from zero to one, where values near one indicate a stronger reliance on infrared features and values near zero favor visible light features. This spatially adaptive weighting enables the fused latent representation to preserve thermal targets while retaining background details from the visible modality.

3.4. Diffusion Fusion Process

The diffusion fusion process implements different fusion strategies at different stages of the diffusion process through a multi-stage strategy. The entire diffusion process is based on the DDIM scheduler, containing $T=50$ denoising steps.

The diffusion model is defined by two Markov chains: the forward process gradually adds noise to the data, and the reverse process gradually removes noise. For the latent representation $z_0$, the forward process is defined as

$$q\left(z_t\right|z_{t-1})=\mathcal{N}(z_t;\sqrt{1-{\beta }_t}{z}_{t-1},{\beta }_{t}I)$$

(4)

where ${\beta }_t$ is the noise schedule parameter, and $t=1,\dots ,T$ is the timestep. Via the reparameterization trick, $z_t$ can be directly computed from $z_0$:

$$z_t=\sqrt{{\overline{\alpha }}_t}{z}_0+\sqrt{1-{\overline{\alpha }}_t}ϵ,ϵ\sim \mathcal{N}(0,I)$$

(5)

where ${\alpha }_t=1-{\beta }_t$, ${\overline{\alpha }}_t=\prod {i=1}^t{\alpha }_i$.

The reverse process is realized by learning a denoising network ${ϵ}_{\theta }$:

$$p_{\theta }\left(z_{t-1}\right|z_t)=\mathcal{N}(z_{t-1};{\mu }_{\theta }(z_t,t),{\Sigma }_{\theta }(z_t,t))$$

(6)

In the image fusion task, we utilize the preliminary fused representation $z_{\mathrm{pre}}$ from the TASFW module as a conditional input and generate the final fused representation through a conditional diffusion process. The fusion process employs a multi-stage strategy that dynamically adjusts fusion weights at different diffusion steps. Algorithm 1 details this process.

Algorithm 1 Target-aware diffusion fusion algorithm.

Require: Pre-trained diffusion model, infrared image ${\mathbf{I}}_{\mathrm{ir}}$, visible light image ${\mathbf{I}}_{\mathrm{vis}}$ Ensure: Fused image ${\mathbf{I}}_{\mathrm{fused}}$   1: Encode input images into latent space:   2: ${\mathbf{z}}_{\mathrm{ir}}=\mathrm{VAE}\_\mathrm{Encoder}\left({\mathbf{I}}_{\mathrm{ir}}\right)$   3: ${\mathbf{z}}_{\mathrm{vis}}=\mathrm{VAE}\_\mathrm{Encoder}\left({\mathbf{I}}_{\mathrm{vis}}\right)$   4: Compute initial fusion weights via TASFW:   5: $W=\mathrm{TASFW}({\mathbf{z}}_{\mathrm{ir}},{\mathbf{z}}_{\mathrm{vis}})$   6: Generate preliminary fusion:   7: ${\mathbf{z}}_{\mathrm{pre}}=W\odot {\mathbf{z}}_{\mathrm{ir}}+(1-W)\odot {\mathbf{z}}_{\mathrm{vis}}$   8: for each diffusion step $t=T$ to 1 do   9:      Determine fusion stage based on t: 10:      $\mathrm{stage}=\mathrm{bad}\mathrm{hbox}\left(t\right)$ 11:      Adjust fusion weights for current stage: 12:      $W_t=\mathrm{adjust}\_\mathrm{weights}(W,t,\mathrm{stage})$ 13:      Compute conditional fusion: 14:      ${\mathbf{z}}_t^{\mathrm{fuse}}=W_t\odot {\mathbf{z}}_{\mathrm{ir}}+(1-W_t)\odot {\mathbf{z}}_{\mathrm{vis}}$ 15:      Multi-step prediction (if enabled for stage): 16:      for $k=1$ to $K_{\mathrm{stage}}$ do 17:         Predict noise: ${ϵ}_k={ϵ}_{\theta }({\mathbf{z}}_t^{\mathrm{fuse}},t,{\mathbf{z}}_{\mathrm{pre}})$ 18:         Update latent: ${\mathbf{z}}_t^{\mathrm{fuse}}=\mathrm{DDIM}\_\mathrm{step}({\mathbf{z}}_t^{\mathrm{fuse}},{ϵ}_k,t)$ 19:      end for 20: end for 21: Decode final fused image: 22: ${\mathbf{I}}_{\mathrm{fused}}=\mathrm{VAE}\_\mathrm{Decoder}\left({\mathbf{z}}_1^{\mathrm{fuse}}\right)$ 23: return ${\mathbf{I}}_{\mathrm{fused}}$

As shown in Algorithm 1, the fusion process begins by encoding the input infrared and visible light images into the latent space using the VAE encoder (lines 2–3). The TASFW module then generates an initial fusion weight map W based on the complementary characteristics of the two modalities (line 5). This weight map is used to create a preliminary fused representation ${\mathbf{z}}_{\mathrm{pre}}$ (line 7).

During the diffusion process (lines 8–19), the algorithm iterates through T denoising steps. At each step t, the fusion stage is determined based on the current timestep (line 10). The fusion weights are then dynamically adjusted according to the stage-specific strategy (line 12), which follows the three-stage approach: early reinforcement, mid-term balance, and late refinement. These adjusted weights $W_t$ are used to compute the conditionally fused latent representation (line 15).

For stages where multi-step prediction is enabled, the algorithm performs $K_{\mathrm{stage}}$ iterative predictions and updates to refine the fusion quality (lines 17–19). Finally, the optimized latent representation is decoded back to the image space via the VAE decoder (line 22), producing the final fused image.

3.4.1. Multi-Stage Fusion Strategy

Traditional diffusion-based fusion methods typically employ a static condition—such as channel-wise concatenation of source images—throughout all denoising steps, making it difficult to adapt the fusion ratio to the coarse-to-fine dynamics of the generative process.

Stage 1: Early Reinforcement Stage ($t\in [T,\alpha T]$)

During the early stage of the diffusion process, the model primarily focuses on generating the overall structure. The goal of this stage is to ensure that infrared thermal targets are fully retained in the fused image. To achieve this, we enhance the weight of infrared information:

$$W_t^{\left(1\right)}=W·{\rho }_1+b_1$$

(7)

where ${\rho }_1$ and $b_1$ are adjustment coefficients, typically set to values that strengthen the infrared features. The adjusted weight $W_t^{\left(1\right)}$ is shifted upward overall, ensuring that even in regions with weaker infrared information, a certain amount of infrared characteristics are retained. This stage adopts a three-step prediction mechanism, improving fusion quality through multiple iterative optimizations.

Stage 2: Mid-Term Balance Stage ($t\in [\alpha T,\beta T]$)

As the diffusion process progresses, the model begins to focus on generating detailed information. The goal of this stage is to gradually introduce visible light details while retaining infrared targets. The fusion weight adjustment is

$$\begin{array}{cc}\hfill W_t^{\left(2\right)}& =W·{\rho }_2\hfill \end{array}$$

(8)

$$\begin{array}{cc}\hfill W_{\mathrm{detail}}& =\left\{\begin{array}{cc}W·{\rho }_3+b_2\hfill & \mathrm{if}W>\tau \hfill \\ W·{\rho }_4\hfill & \mathrm{otherwise}\hfill \end{array}\right.\hfill \end{array}$$

(9)

$$\begin{array}{cc}\hfill W_t& =W_t^{\left(2\right)}·{\gamma }_1+W_{\mathrm{detail}}·{\gamma }_2\hfill \end{array}$$

(10)

where ${\rho }_2$, ${\rho }_3$, ${\rho }_4$, $b_2$, $\tau$, ${\gamma }_1$, and ${\gamma }_2$ are configurable parameters. This adjustment strategy appropriately reduces the infrared weight in high-weight regions (target areas) to create space for introducing visible light details; in low-weight regions (background areas), it appropriately increases the infrared weight to avoid complete loss of background information.

Stage 3: Late Refinement Stage ($t\in [\beta T,0]$)

During the late stage of the diffusion process, the model primarily performs detail refinement. The goal of this stage is to enhance visible light texture details while maintaining the saliency of thermal targets. The fusion weight adjustment is

$$W_t^{\left(3\right)}=W·{\rho }_5+b_3$$

(11)

For target regions ($W>\tau$), further reinforcement of infrared information retention is applied:

$$W_t=\left\{\begin{array}{cc}{W}_t^{\left(3\right)}·\kappa \hfill & \mathrm{if}W>\tau \hfill \\ W_t^{\left(3\right)}\hfill & \mathrm{otherwise}\hfill \end{array}\right.$$

(12)

where ${\rho }_5$, $b_3$, and $\kappa$ are adjustable parameters. This strategy ensures that during the detail enhancement process, the infrared features in thermal target regions are not excessively weakened.

3.4.2. Full-Step Multi-Step Prediction Mechanism

Traditional diffusion models perform one forward propagation at each time step to predict noise and update the latent representation. However, in image fusion tasks, single-step prediction may not fully capture the complex relationships of multimodal information. Therefore, the proposed model proposes a full-step multi-step prediction mechanism that uses different prediction counts at different stages.

Early Stage: Multi-Step Prediction

In the early reinforcement stage, each time step performs multiple predictions:

First, Base Prediction: Predict noise ${ϵ}_1$ based on the current latent representation $z_t$.

Second, Intermediate Update: Update the latent representation to $z_t^{\left(1\right)}$ based on ${ϵ}_1$.

Third, Refined Prediction: Predict noise ${ϵ}_2$ based on $z_t^{\left(1\right)}$, and update to $z_t^{\left(2\right)}$ based on ${ϵ}_2$.

Final Prediction: Predict noise ${ϵ}_3$ based on $z_t^{\left(2\right)}$ as the final noise prediction.

This multi-step prediction mechanism improves the accuracy of noise estimation through multiple iterative optimizations, particularly suitable for stages requiring reinforcement of specific information (such as infrared targets).

Mid-to-Late Stage: Reduced-Step Prediction

In the mid-term balance and late refinement stages, each time step performs a reduced number of predictions:

Base Prediction: Predict noise ${ϵ}_1$ based on the current latent representation.

Intermediate Update: Update the latent representation based on ${ϵ}_1$.

Refined Prediction: Predict noise ${ϵ}_2$ based on the updated representation as the final noise prediction.

Reduced-step prediction lowers computational overhead while ensuring quality, making it suitable for stages with higher computational efficiency requirements.

3.4.3. Conditional Guidance Mechanism

The diffusion process is guided by the preliminary fusion representation $z_{\mathrm{pre}}$ generated by the TASFW module, which encodes the complementary information from both infrared and visible light images. This representation serves as the primary conditioning signal for the denoising process.

The preliminary fusion representation $z_{\mathrm{pre}}$ is incorporated into the diffusion process through a cross-attention mechanism. In the attention layer of each UNet block, we compute attention weights between queries derived from the current noisy latent representation and keys/values derived from $z_{\mathrm{pre}}$:

$$\mathrm{Attention}(Q,K,V)=\mathrm{SoftMax}\left({\displaystyle \frac{Q{K}^T}{\sqrt{d_k}}}\right)V$$

(13)

where Q represents queries from the current latent representation at each diffusion step, while K and V are obtained from $z_{\mathrm{pre}}$. This cross-attention mechanism allows the diffusion process to dynamically attend to relevant regions of the source images throughout the denoising process.

Simultaneously, timestep embeddings are injected into the UNet through adaptive layer normalization (AdaIN) layers, providing temporal information about the current diffusion step.

This conditional guidance mechanism ensures that the reverse diffusion process is consistently informed by the complementary characteristics of the source images, generating fusion results that preserve both infrared target information and visible light texture details while maintaining overall consistency with the input modalities.

In addition to the latent representation $z_{\mathrm{pre}}$, the UNet also receives a text embedding from the frozen text encoder. As our task is purely visual and does not require semantic descriptions, we input a fixed empty string ("") to the text encoder. This yields a constant embedding vector that serves as a domain-agnostic regularization signal. Its sole purpose is to ensure architectural compatibility with the pre-trained Stable Diffusion weights and to constrain the generated outputs to the manifold of natural images, thereby preventing visual artifacts during decoding.

While the multi-step prediction mechanism introduces additional forward passes, its application is restricted primarily to the early denoising stage to mitigate computational overhead. Furthermore, given that the primary objective of this study is to maximize fusion quality for downstream detection tasks rather than to optimize real-time inference speed, a detailed runtime and parameter comparison is deferred to future work on lightweight diffusion acceleration.

3.5. Loss Function Design

To supervise the adaptation of the TASFW module while preserving the frozen generative priors of the Stable Diffusion backbone, we adopt a suitable loss function.

3.5.1. Reconstruction Loss

The reconstruction loss ensures intensity consistency between the fused image $I_{\mathrm{fuse}}$ and the reference visible image $I_{\mathrm{vis}}$. We employ the $L_1$ loss due to its robustness to outliers and its ability to preserve sharp boundaries:

$${\mathcal{L}}_{\mathrm{rec}}={\displaystyle \frac{1}{N}}{\displaystyle \sum _{i=1}^N}\left|I_{\mathrm{fuse}}^{\left(i\right)}-I_{\mathrm{vis}}^{\left(i\right)}\right|$$

(14)

where N denotes the total number of pixels. By penalizing the absolute differences in pixel space, ${\mathcal{L}}_{\mathrm{rec}}$ encourages the fused output to retain the fine-grained details present in the visible modality.

3.5.2. Structural Similarity Loss

To further align the fused result with human visual perception, we incorporate the Structural Similarity (SSIM) loss. SSIM jointly assesses luminance, contrast, and structural information, promoting natural-looking fusion outputs:

$${\mathcal{L}}_{\mathrm{ssim}}=1-\mathrm{SSIM}(I_{\mathrm{fuse}},I_{\mathrm{vis}})$$

(15)

This term complements the pixel-wise $L_1$ loss by enforcing global structural consistency, which is particularly important for preserving object boundaries and background context.

3.5.3. Total Loss Function

The complete objective for the training stage is a weighted combination of the two terms:

$${\mathcal{L}}_{\mathrm{total}}={\lambda }_{\mathrm{rec}}{\mathcal{L}}_{\mathrm{rec}}+{\lambda }_{\mathrm{ssim}}{\mathcal{L}}_{\mathrm{ssim}}$$

(16)

In all experiments, we set ${\lambda }_{\mathrm{rec}}=1.0$ and ${\lambda }_{\mathrm{ssim}}=1.0$, which provides an effective balance between accurate intensity reconstruction and structural preservation. This compact formulation introduces no auxiliary networks beyond the frozen VAE and UNet.

4. Results

To comprehensively evaluate the fusion performance, we used three representative datasets: AVMS [40], M3FD [24], TNO [41].

The AVMS dataset includes visible and infrared images obtained from low altitude aerial photography by drones, with a total of 600 pairs of images. The environment of the images in this dataset involves day, night, and various complex weather conditions. The image scenes involve various scenes such as residential areas, roads, campuses, and farmland, and the shooting heights are also diverse. It contains 7 types of targets for recognition, of which 8991 instances have been carefully annotated, including 8046 cars, 526 pedestrians, and 138 tents. Due to its diverse shooting environments and scenes, a rich number of targets and shooting heights and angles, as well as carefully annotated target category labels and segmentation masks, this dataset is integrated into a high-quality dataset suitable for various computer vision tasks.

The M3FD dataset is a Dalian street view dataset created by a research team from Dalian University of Technology. The dataset collected 4200 pairs of images, including scenes such as the campus of Dalian University, major roads in Jinzhou District, and Jinshitan. It includes six target categories: people, cars, buses, motorcycles, light vehicles, and trucks. Its high-quality image acquisition makes it very suitable for related work in image fusion tasks.

The TNO dataset released by the Dutch Organization for Applied Scientific Research collected multi-band images including various military and monitoring scenarios, such as visible light, near-infrared, and long wave infrared nighttime images. The image content includes different objects and target people and vehicles in different backgrounds, and scenes include rural and urban environments.

We compared six representative deep learning-based fusion methods with the proposed model, namely U2Fusion [42], FusionGAN [3], ITFuse [43], ATFusion [44], YDTR [19], and SEND [45]. We have chosen these six comparison methods based on the following considerations. Firstly, they collectively represent the comprehensive coverage of the main types of technologies in the field of infrared visible image fusion: CNN-based (U2Fusion), GAN-based (FusionGAN), Transformer-based (ITFuse, ATFusion, and YDDR), and diffusion model-based (SEND). This comprehensive coverage ensures a diverse and reliable evaluation baseline. Secondly, these methods have been widely used as standard benchmarks in the recent fusion literature, including the latest surveys and comparative studies, demonstrating their established status and high recognition in the research community. The six methods selected based on the above considerations can ensure fairness and rigor in our comparison.

When evaluating experimental results, we report the results using the following metrics: Peak Signal to Noise Ratio (PSNR), Mean Square Error (MSE), Mean Absolute Error (MAE), Mean Gradient (AG), Spatial Frequency (SF), and Sum of the Correlations of Differences (SCD). The higher the values of these metrics, the better the quality of the fused image. Among them, MSE and MAE are the opposite, and the lower the values of these indicators, the better the quality of the fused image. All experiments were conducted on an NVIDIA GeForce RTX 5070TI GPU.

4.1. Module Ablation Experiment

We ran ablation experiments on the AVMS dataset to measure the contribution of each core component.

Table 1. Ablation study on the AVMS dataset. Bold indicates best results, and underline indicates second-best performance. The ↑ in the table represents that the higher the value, the better, and the ↓ represents that the smaller the value, the better.

Method	PSNR ↑	MSE ↓	MAE ↓	AG ↑	SF ↑	SCD ↑
w/o TASFW	15.4784	1420.3578	29.1542	22.5341	7.8956	0.4612
w/o Multi-Stage	16.3245	1568.2341	31.0876	24.8765	9.4321	0.4357
w/o Multi-Step	14.2156	1789.5123	33.4251	26.0123	10.6543	0.3982
Full Model	17.2642	1329.5446	28.1098	26.8997	11.1852	0.4957

reports the specific quantitative results. When TASFW is removed, performance degrades relative to the full model: PSNR drops from 17.2642 to 15.4784 (a loss of 1.79), MSE rises from 1329.5446 to 1420.3578, MAE increases from 28.1098 to 29.1542, AG falls from 26.8997 to 22.5341, SF declines from 11.1852 to 7.8956, and SCD decreases from 0.4957 to 0.4612. Eliminating the multi-stage strategy causes PSNR to fall to 16.3245, MSE to jump to 1568.2341, MAE to worsen to 31.0876, AG to drop to 24.8765, SF to 9.4321, and SCD to 0.4357. The removal of multi-step prediction results in the largest deterioration, with PSNR plummeting to 14.2156 (a drop of roughly 3.05), MSE climbing sharply to 1789.5123, MAE reaching 33.4251, AG at 26.0123, SF at 10.6543, and SCD at a low of 0.3982.

Overall, each ablated configuration underperforms the full model across all six evaluation metrics, confirming that every component plays a distinct and necessary role. The absence of TASFW validates that spatially adaptive weighting is crucial for balancing thermal saliency with background detail, as a uniform weight leads to noticeable blurring and lower gradient fidelity. The drop observed when removing the multi-stage fusion confirms that adjusting the fusion ratio across different diffusion phases (early reinforcement, mid-term balancing, and late refinement) is essential for coherent structure–texture integration. The sharpest decline occurs without multi-step prediction, indicating that relying on a single prediction per timestep introduces instability and noise accumulation that severely harms reconstruction accuracy. Consequently, the complete model, which incorporates TASFW, multi-stage fusion, and multi-step prediction, achieves the best performance, with the highest PSNR (17.2642) and AG (26.8997) and the lowest MSE (1329.5446) and MAE (28.1098).

4.2. Loss Function Ablation Experiment

We ran an ablations experiment on AVMS. Three setups were compared: (i) $L_1$ by itself (${\lambda }_{\mathrm{rec}}=1.0$, ${\lambda }_{\mathrm{ssim}}=0$); (ii) SSIM by itself (${\lambda }_{\mathrm{rec}}=0$, ${\lambda }_{\mathrm{ssim}}=1.0$); and (iii) the balanced mix we actually rely on (${\lambda }_{\mathrm{rec}}=1.0$, ${\lambda }_{\mathrm{ssim}}=1.0$).

The specific situation is shown in

Table 2. Ablation study on loss function weights using the AVMS dataset. Bold indicates best results, and underline indicates second-best performance. The ↑ in the table represents that the higher the value, the better, and the ↓ represents that the smaller the value, the better.

Configuration	PSNR ↑	MSE ↓	MAE ↓	AG ↑	SF ↑	SCD ↑
$L_1$ only	16.8274	1397.6312	28.9461	24.1358	9.8723	0.4529
SSIM only	16.4519	1466.9085	29.8734	23.4572	9.2136	0.4687
Full Model	17.2642	1329.5446	28.1098	26.8997	11.1852	0.4957

. With $L_1$ alone, PSNR and MSE land at 16.8274 and 1397.6312, with MAE at 28.9461. But, SSIM drops to 0.5817, AG slides to 24.1358, SF is just 9.8723, and SCD comes out to 0.4529. So, the raw intensities stay close, yet the image loses structure and crispness. Switching to SSIM-only pushes SSIM up to 0.6124 and nudges SCD to 0.4687, but PSNR and MSE worsen to 16.4519 and 1466.9085, MAE climbs to 29.8734, AG slips to 23.4572, and SF falls to 9.2136. The output looks more coherent, but fine textures and sharp edges take a hit.

Our default equal-weight setup lands the best PSNR (17.2642), the smallest MSE (1329.5446), and the lowest MAE (28.1098). It also leads in AG (26.8997) and SF (11.1852), with SSIM holding at 0.6013 and SCD at 0.4957. The story here is simple. $L_1$ keeps the intensities honest, and SSIM keeps things visually plausible. Neither term works well on its own, and putting them together lifts all six metrics. We get this improvement without any extra tuning or added networks—the lightweight design stays exactly as intended.

4.3. Quantitative Comparison on AVMS Dataset

The visual comparison of fusion result samples from various models on the AVMS dataset is presented in Figure 4. The corresponding quantitative performance on multiple evaluation metrics is summarized in

Table 3. Performance comparison on AVMS dataset. Bold indicates best results, and underline indicates second-best performance. The ↑ in the table represents that the higher the value, the better, and the ↓ represents that the smaller the value, the better.

Method	PSNR ↑	MSE ↓	MAE ↓	AG ↑	SF ↑	SCD ↑
U2Fusion	16.5968	1521.9354	29.9771	66.5728	20.9548	0.5357
FusionGAN	13.3066	3221.3938	43.0676	25.8811	7.7992	0.0167
ITFuse	16.8715	1396.9777	29.6147	24.9690	5.9591	0.5275
ATFusion	16.3645	1583.6861	30.1256	68.1513	24.8585	0.4717
YDTR	16.4394	1582.4670	29.7468	41.5806	18.8981	0.5017
SEND	17.0019	1449.3127	28.7638	40.1909	18.0015	0.4903
Ours	17.2642	1329.5446	28.1098	26.8997	11.1852	0.4957

. In the comparative experiment, the proposed model (Ours) ranks first on the PSNR metric with a score of 17.2642, surpassing the second-place SEND model (17.0019) by approximately 0.2623. In terms of the MSE index, the proposed model again achieves the best result, attaining 1329.5446, which is about 67.4331 lower (better) than the second-ranked ITFuse model (1396.9777). For the MAE metric, the proposed model also leads with a value of 28.1098, outperforming the runner-up SEND model (28.7638) by a margin of 0.6540. On the AG index, the proposed model records a score of 26.8997, placing fifth; on the SF index, it ranks fifth with a value of 11.1852; and on the SCD index, it obtains 0.4957, ranking fourth. Overall, the proposed model secures the top position in three key fidelity-oriented metrics—PSNR, MSE, and MAE—while maintaining a competitive mid-table standing in the remaining measures, demonstrating its robust performance in critical aspects of image fusion.

From the observed indicator performance, the proposed model achieves the best results in PSNR, MSE, and MAE, underscoring its high signal fidelity and low error levels in image fusion tasks, thereby enabling more accurate preservation of source image information. Although it does not appear among the leading entries on the AG and SF metrics, its overall profile remains well-balanced. This suggests that the proposed model strikes a favorable compromise between preserving image details, minimizing fusion errors, and enhancing visual quality, making it particularly well-suited for fusion applications that prioritize high fidelity.

4.4. Quantitative Comparison on M3FD Dataset

The visual comparison effect of the fusion result samples of various models on the M3FD dataset is shown in Figure 5. As shown in

Table 4. Performance comparison on M3FD dataset. Bold indicates best results, and underline indicates second-best performance. The ↑ in the table represents that the higher the value, the better, and the ↓ represents that the smaller the value, the better.

Method	PSNR ↑	MSE ↓	MAE ↓	AG ↑	SF ↑	SCD ↑
U2Fusion	19.0026	856.6726	20.5942	27.2488	7.7753	0.6943
FusionGAN	17.3553	1284.7474	27.1411	9.3642	2.9345	0.4968
ITFuse	18.8146	872.1518	23.6401	11.4565	3.1193	0.7102
ATFusion	18.6306	925.6589	24.4483	26.5589	8.3995	0.6541
YDTR	19.5240	772.9849	20.9000	13.6941	4.5100	0.6914
SEND	18.9324	754.1729	20.5541	15.3764	5.2209	0.6219
Ours	20.1603	705.9875	20.0374	10.4329	3.8765	0.7334

, the proposed model demonstrates strong performance across multiple indicators in the comparative experiments. The model ranks first in PSNR (20.1603), MSE (705.9875), MAE (20.0374), and SCD (0.7334). In terms of PSNR, the proposed model leads the second-place YDTR model (19.5240) by approximately 0.6363. Regarding the SCD metric, it surpasses the runner-up ITFuse model (0.7102) by a margin of 0.0232. For the MSE and MAE indicators, the proposed model achieves the optimal scores of 705.9875 and 20.0374, respectively, outperforming all competing methods. On the AG index, the proposed model scores 10.4329, ranking fifth; on the SF metric, it records a value of 3.8765, also ranking fifth. Overall, the proposed model consistently resides at the leading edge or middle tier across the majority of evaluation criteria, reflecting a well-rounded advantage across multiple performance dimensions.

From the performance observed across various indicators, the proposed model achieves the best results in PSNR, MSE, MAE, and SCD, underscoring its significant advantages in signal fidelity, error minimization, and structural consistency. This enables the model to preserve the structure and content information of the source image with greater precision. In summary, the proposed model strikes an effective balance between visual quality and structural integrity in the fusion outcomes. While mitigating noise and controlling reconstruction errors, it maintains a competitive level of detail preservation and structural coherence, rendering it well-suited for fusion tasks that demand high image quality and robust structural retention.

4.5. Quantitative Comparison on TNO Dataset

The visual comparison effect of the fusion result samples of various models on the TNO dataset is shown in Figure 6. As shown in

Table 5. Performance comparison on TNO dataset. Bold indicates best results, and underline indicates second-best performance. The ↑ in the table represents that the higher the value, the better, and the ↓ represents that the smaller the value, the better.

Method	PSNR ↑	MSE ↓	MAE ↓	AG ↑	SF ↑	SCD ↑
U2Fusion	17.0005	1367.7639	28.4226	41.7449	9.9612	0.5209
FusionGAN	15.3682	2170.0241	35.9424	21.7273	5.9927	0.4005
ITFuse	17.0227	1342.9964	28.4237	17.8343	3.8473	0.5338
ATFusion	16.5086	1576.3734	30.148	42.8344	10.8382	0.4558
YDTR	16.6814	1628.6467	29.5672	24.7286	7.2471	0.5232
SEND	17.0511	1307.0547	26.9394	29.4176	7.8019	0.4903
Ours	17.1258	1263.24	26.6910	21.0971	7.7205	0.4984

, the proposed model performed well across multiple indicators in the comparative experiments on the TNO dataset. As seen in the table, the model ranks first in both the PSNR (17.1258) and MSE (1263.24) indicators; simultaneously, it also achieves the top position in the MAE (26.6910) metric. In terms of PSNR, the proposed model leads the second-ranked SEND model (17.0511) by approximately 0.0747; regarding MSE, it outperforms the runner-up SEND model (1307.0547) by a margin of about 43.8147. On the MAE metric, the score of the proposed model is 26.6910, which is marginally better than the second-place SEND model (26.9394). On the SCD index, the proposed model ranks fourth with a score of 0.4984, trailing the top-ranked ITFuse model (0.5338) by approximately 0.0354. On the AG index, the proposed model ranks sixth (21.0971); in terms of the SF indicator, the proposed model ranks fifth (7.7205). Overall, the proposed model maintains a leading or competitive standing in the majority of evaluation criteria, underscoring its balanced advantages across multiple performance dimensions.

From the performance observed across the various indicators, the proposed model achieved the best results in PSNR, MSE, and MAE, demonstrating significant advantages in signal fidelity and error control. This enables the model to preserve the intensity and content information of the source image with greater accuracy and minimal distortion. The mid-table placement on the SCD index indicates that the model maintains a reasonable degree of structural consistency in the fused output, positioned just behind the leading methods. In summary, the proposed model achieves a favorable trade-off between visual quality and quantitative precision in the fusion outcomes. While effectively mitigating noise and managing pixel-level discrepancies, it maintains a balanced overall profile, rendering it well-suited for fusion tasks that require high image fidelity and robust error minimization.

4.6. Object Detection

In the field of image fusion, the adaptability of fusion results to downstream visual tasks such as object detection is an important indicator for evaluating the practical value of fusion methods. In this section, to verify the contribution and effectiveness of the proposed object detection task model as a downstream task in computer vision, we conducted experiments on the AVMS dataset and M3FD dataset, comparing the performance of different methods in detection tasks, including infrared source images, visible light source images, feature level fusion detection methods MNCM [46], and the proposed model. We use the YOLO V11 model to evaluate the differences between different methods, which is widely recognized as effective in object detection tasks. The object detection results on two datasets are shown in

Table 6. Object detection performance comparison on AVMS dataset. Bold indicates best results, and underline indicates second-best performance.

Method	Bus	Tent	Truck	Car	Electric Vehicles	Boat	Person	mAP@50 (%)	mAP@50–95 (%)
Visible	66.62	56.78	70.68	72.26	61.05	59.74	47.39	62.07	42.36
Infrared	63.31	50.17	67.49	73.30	55.91	57.80	41.03	58.43	36.67
MNCM	83.46	70.12	85.65	80.28	72.72	78.95	63.05	76.32	50.64
Ours	83.79	73.42	87.28	85.10	77.78	77.22	68.78	79.05	52.71

and

Table 7. Object detection performance comparison on M3FD dataset. Bold indicates best results, and underline indicates second-best performance.

Method	Bus	Truck	Car	Motorcycle	Lamp	People	mAP@50 (%)	mAP@50–95 (%)
Visible	63.95	74.13	71.56	63.40	60.09	60.41	65.59	41.09
Infrared	64.22	63.06	71.37	50.96	61.12	45.77	59.42	35.70
MNCM	79.43	85.24	74.41	74.03	68.99	69.21	75.22	48.31
Ours	81.22	80.93	82.12	77.80	72.18	74.01	78.04	48.66

, which significantly outperforms other methods in all detection categories and overall metrics, reflecting the effectiveness of our method in generating highly compatible fused images for downstream tasks.

5. Conclusions

Quantitative analysis shows that the proposed method achieves advanced performance in fidelity metrics such as PSNR, MSE, and MAE, while scoring relatively moderately in texture-related metrics such as average gradient (AG) and spatial frequency (SF). Considering that potential diffusion models tend to produce smoother outputs and VAE compression inherently discards some high-frequency variations, this is not entirely surprising. When prioritizing structural consistency over local edge enhancement, the current design inevitably softens certain details. Finding methods to restore these subtle textures without compromising detection accuracy will be a key focus of future work. That is to say, the overall fidelity is still strong, and the improvements observed in downstream object detection confirm that the method achieves a reasonable balance between semantic integrity and visual quality. Looking ahead, the lightweight detail enhancement step in the decoding stage may provide a practical approach for further sharpening the output, although caution should be taken to avoid reintroducing artifacts or detection noise. In addition, future work will delve deeper and expand on comparative evaluation to include the latest transformer-based and diffusion-based fusion models, aiming to further validate the generality and competitiveness of the proposed methods.

Overall, based on a thorough analysis of the task of fusing infrared and visible light images, the proposed model proposes a fusion method based on the target perception diffusion model to address the balance between insufficient preservation of target information and loss of background details in complex scenes using existing methods. This method introduces the Stable Diffusion architecture into the field of multimodal image fusion. By designing a Target-Aware Spatial Fusion Weight (TASFW) module to adaptively generate spatial fusion weights and combining multi-stage dynamic fusion strategies, the fusion ratio of infrared and visible light information is intelligently adjusted at different stages of the diffusion process, effectively achieving balanced preservation of salient targets and texture details. In addition, the introduction of a full step multi-step prediction mechanism further enhances the stability and quality of the fusion results. Experiments on multiple publicly available datasets such as AVMS, M3FD, and TNO have shown that our proposed method outperforms existing mainstream methods in key metrics such as PSNR, MSE, and MAE and demonstrates good compatibility and practicality in downstream object detection tasks. In summary, the method proposed in this article not only provides a new and effective solution for the fusion of infrared and visible light images but also demonstrates the potential and applicability of diffusion models in multimodal vision tasks.