DCAFuse: A Differential Cross-Attention Transformer Network for Infrared and Visible Image Fusion in UAV-Based Wilderness Search and Rescue

Highlights

What are the main findings?

• An aligned infrared-visible multimodal dataset, UAV-MSR, is constructed for UAV-based wilderness search and rescue. It contains 3889 image pairs captured across diverse scenarios and weather conditions.
• A novel DCAFuse network is proposed for infrared and visible image fusion, integrating the differential cross-attention mechanism to address complementary information loss and target brightness degradation.

What are the implications of the main finding?

• The proposed DCAFuse network effectively preserves fine-grained complementary information and enhances target representation, generating high-quality fused images with superior visual performance.
• The proposed dataset and network provide data and technical support for search and rescue missions, enhancing the capability of UAVs to perceive targets and the environment.

Abstract

Infrared and visible image fusion is critical for unmanned aerial vehicle (UAV) wilderness search and rescue. By integrating thermal radiation of the targets and texture details of the scenario, it enables accurate search for the wounded and comprehensive perception of disaster areas, thereby significantly improving emergency rescue efficiency. To alleviate data scarcity, we construct UAV-MSR, an infrared-visible dataset for casualty search, comprising 3889 paired images captured under diverse weather, illumination, and scenarios. Existing Transformer-based fusion methods mainly focus on high-intensity pixels while inadequately modeling low-intensity complementary features, resulting in blurred details and degraded target contrast in fused images. To this end, we propose a novel differential cross-attention Transformer network to address the issue of complementary information loss. Specifically, the encoder integrates convolution operations for local detail extraction and self-attention mechanisms for global context modeling. Then, we design a differential cross-attention guided feature fusion module to enhance the representation and preservation of detailed complementary features. Furthermore, a pixel loss function with a segmentation strategy is employed to improve the saliency of the target, enabling the fused image to facilitate subsequent target detection tasks. Experimental results and ablation studies demonstrate that the proposed method achieves notable performance and generalization ability. In summary, this work delivers a multimodal dataset and an efficient infrared-visible image fusion network to enable comprehensive perception for UAVs in wilderness search and rescue scenarios.

1. Introduction

With the advantages of high maneuverability, wide coverage, and no risk of casualties, unmanned aerial vehicle (UAV) optical systems have become a crucial technological equipment in emergency rescue [1,2,3], replacing manual entry into disaster areas to search for missing persons. However, in wilderness search and rescue (SAR) scenarios, human targets are often occluded by vegetation. Under such conditions, the visible imaging system suffers from insufficient illumination and occlusion [4], resulting in a significant decline in target perception capability. To overcome the limitations of single-modal imaging, researchers have turned to multimodal fusion technology [5]. By integrating complementary information from different modalities, it effectively captures more complete and comprehensive scene information.

The multimodal fusion problem of infrared and visible image fusion (IVIF) holds a unique significance in the above SAR scenarios [6], where their complementary characteristics are illustrated in Figure 1. Specifically, visible images capture high-resolution scene color and texture details consistent with human visual perception, providing intuitive and reliable visual references for on-site SAR path planning. However, under complex conditions such as low light, smoke, or target occlusion, their ability to perceive human targets is significantly limited. Thermal imaging systems can overcome the constraints of occlusion and illumination to stably acquire human thermal radiation [7], and achieve robust target localization. Nevertheless, they fail to sense the color and texture information of scenarios. To address these limitations, IVIF tasks aim to explore the synergistic correlations among different modalities and integrate their complementary information. This process generates fused images with richer information, stronger scene representation, and improved target perception, thereby benefiting high-level semantic tasks such as segmentation, object detection, and scene understanding [8].

Figure 1. Samples of the UAV-MSR dataset. Top: visible images. Bottom: infrared images. Close-ups are shown in red boxes.

To alleviate the scarcity of multimodal data in wilderness SAR scenarios, we construct UAV-MSR, a dedicated UAV infrared-visible dataset for casualty search, consisting of 3889 image pairs in total. The dataset was collected at different flight altitudes and under various weather conditions, encompassing human targets occluded by vegetation in a variety of complex environments. The image pairs in the dataset are spatially and temporally synchronized. This dataset provides a substantial data foundation for SAR operations and drives further advancements in image fusion algorithms.

Deep learning has seen extensive applications in IVIF tasks, attributed to its powerful nonlinear fitting capability [9]. The deep learning-based image fusion methods can be divided into three categories: convolutional neural network (CNN)-based, generative model-based, and Transformer-based methods [8]; related works are described in Section 2. Generative model-based methods learn the pixel distribution of source images to generate fused images without label supervision, but suffer from training instability. CNN-based methods typically use convolutional operations for feature extraction and reconstruction, with fusion rules such as L1-norm or weighted summation designed to achieve feature fusion [10]. These fusion rules lack in-depth exploration of cross-modal feature heterogeneity, making it challenging to model the complex complementary relationships between multi-source images [11]. Furthermore, constrained by limited receptive fields, convolutional operations excel at capturing local details but struggle to extract global features [12,13].

Transformer has delivered leading performance in image fusion tasks, owing to its ability to model long-range dependencies and capture global contextual information [14]. Typically, these methods adopt a parallel dual-branch Transformer architecture, which performs cross-domain interaction by treating the two modalities as mutual queries [15,16,17]. However, such dual-branch architectures inevitably introduce high computational burdens due to stacked modules. In addition, the cross-attention mechanism tends to assign greater weights to high-intensity pixels, while its modeling capacity for low-intensity pixels remains limited, causing relatively weak complementary information to be easily overwhelmed or lost during fusion. Therefore, designing a cross-modal interaction paradigm with a simple architecture and the ability to deeply explore complementary information remains a critical challenge.

To address the aforementioned issues, this paper proposes DCAFuse, a novel IVIF network based on the differential cross-attention mechanism to sufficiently preserve complementary information from the source images. DCAFuse adopts a collaborative encoder architecture that integrates a convolution block for extracting local details and a self-attention block for extracting global contextual features. To achieve comprehensive cross-modal feature interaction, we design a feature module consisting of a spatial attention module (SAM) and a channel attention module (CAM). The SAM employs differential cross-attention to guide the network to focus more on relatively weak complementary features, then utilizing standard cross-attention to fuse the salient features. The CAM models channel dependencies and adaptively recalibrate channel responses to enhance feature representation. Furthermore, a segmented pixel loss function is employed to balance the preservation of salient targets and background details, enhance the contrast of the target, and enable the network to adapt to diverse illumination conditions. Overall, the key contributions of this work are outlined as follows:

1.

We construct a multimodal UAV-MSR dataset comprising 3889 aligned infrared-visible image pairs with 7573 annotated objects, captured across diverse scenarios and weather conditions. This dataset provides a data foundation for SAR missions and algorithm development in complex environments.

2.

We propose a novel IVIF network based on the differential cross-attention mechanism, which deeply explores the interdependencies between different modalities to achieve adequate fusion of complementary information and effectively enhance target contrast.

3.

Quantitative and qualitative experiments on four datasets demonstrate the superior fusion performance and the promising generalization capability of the proposed method compared to state-of-the-art algorithms.

The remainder of this paper is organized as follows. The related works are summarized in Section 2. The UAV-MSR dataset is introduced in Section 3. The proposed DCAFuse framework and loss function are thoroughly depicted in Section 4. The experimental results and discussion are presented in Section 5. Finally, the conclusion is given in Section 6.

2. Related Work

2.1. CNN-Based Fusion Methods

The architecture of CNN-based fusion methods typically comprises three stages: feature extraction, fusion, and reconstruction [9]. Early works employed backbone networks such as DenseNet, ResNet, and U-Net for feature extraction, followed by customized fusion strategies to integrate the extracted features. Zhang et al. proposed element-wise max, min, and mean rules to fuse features extracted by convolutional networks [18]. Li et al. incorporated a dense block into the encoding network to construct DenseFuse and designed additive and L1-norm fusion strategies [10]. Subsequently, Li et al. developed RFN-Nest, using a residual architecture to reduce the limitations of manually designed fusion rules [11]. IGNet integrated GNN and CNN by representing infrared and visible features as graph structures for cross-modal interaction [19]. Over the past few years, a growing number of networks have adopted hybrid CNN–Transformer architectures to enhance cross-modal feature fusion and interaction [20].

2.2. Generative Model-Based Fusion Methods

Generative model-based fusion methods learn the data distribution of source images to generate fused images. Ma et al. first proposed FusionGAN, pioneering the application of generative adversarial networks (GANs) to image fusion [21]. The same team later developed DDcGAN, a dual-discriminator network that preserves information from both modalities [22]. Based on DDcGAN, Song et al. added a third discriminator to preserve the difference information between infrared and visible images [23]. In addition, Yin et al. introduced pyramid attention into the GAN framework [24], proposing a residual attention weight fusion rule for multi-scale feature fusion.

In recent years, diffusion models have excelled in image generation, driven by their stable training and intrinsic interpretability. Yue et al. modeled the distributions of multi-channel input data and fused diffusion features to directly generate the three-channel fused image [25]. Yi et al. proposed Diff-IF, a conditional diffusion model with dynamically tailored fusion knowledge priors, enabling high-quality fused images that adhere to the fusion knowledge distribution [26]. Cao et al. proposed DDIF, a supervised diffusion model that incorporates coarse-grained style modulation and fine-grained wavelet modulation [27], effectively decoupling entangled representations to better suit the fusion task. Diffusion models have shown significant promise in image fusion, but further exploration is needed in inference efficiency and task-specific adaptation.

2.3. Transformer-Based Fusion Methods

Transformer has made significant progress in computer vision tasks, owing to the global context modeling capability of the attention mechanism [9]. Early works introduced self-attention (SA) into fusion networks for deep feature extraction. Wang et al. proposed SwinFuse, a Transformer backbone with SA for long-range dependency modeling [28], demonstrating stronger feature representation than CNN. Fu et al. proposed a patch pyramid Transformer for multi-scale feature extraction [29]. Subsequently, Transformer has been increasingly applied in cross-modal feature fusion. Ma et al. constructed an attention-guided feature fusion module in SwinFusion for feature interaction and fusion via iterative dual-branch SA and cross-attention (CA) [15]. Tang et al. developed DATFuse by integrating a dual-attention residual module with a Transformer module for precise fusion [30].

Recently, increasing research has focused on fine-grained interaction modeling using Transformer architectures. Li et al. presented a two-stage autoencoder-based fusion framework that embeds a reversed softmax function into CA to preserve irrelevant information [16]. Zhao et al. used lite Transformer and invertible neural networks to disentangle features into high and low frequencies, promoting fusion by enhancing low-frequency correlations and suppressing high-frequency ones [31]. Park et al. developed spatial and channel Transformers to suppress redundant features of source images [32]. Zhou et al. proposed a high-order interaction paradigm to model spatial fine-grained and global statistical cross-modal synergies [33]. Wu et al. presented a fully connected Transformer architecture with three parallel branches for feature interaction [34].

The CA has been widely used in feature fusion, but it inadequately models relatively weak complementary information. Furthermore, the trade-off between efficiency and performance in fusion architectures warrants further investigation.

3. Data Acquisition and Preprocessing

We constructed UAV-MSR, an aligned infrared-visible dataset for SAR operations targeting missing persons in wilderness environments, with a focus on occluded individuals. This section introduces the acquisition, registration, and annotation of the dataset.

3.1. Data Acquisition

The DJI Matrice 30T UAV, equipped with visible and infrared cameras, was employed for data collection [7], as shown in Figure 2a, and the key parameters are shown in Table 1. The infrared camera has a response band of 8–14 μm. The UAV flew at approximately 1 m/s, with a gimbal pitch angle of $-90°$. The flight path was pre-programmed via ground station software to ensure consistent coverage across different flights. The longitudinal and lateral overlap ratios were set to 80% and 70%, respectively, with images captured at intervals of 3 s. Human participants were positioned near vegetation occluders in three postures: lying supine, lying on the side, and curled up. The images were acquired in natural daylight (6:00 AM to 5:00 PM). The maximum experimental ambient temperature was 27 °C, and the minimum was −3 °C. A total of 3889 image pairs were collected from 12 scenarios, 3 altitudes (as shown in Figure 3), and various weather conditions (as shown in Figure 4). The distribution of images in the UAV-MSR dataset across different altitudes, weather conditions, and target occlusion conditions is presented in Figure 5.

Figure 2. Dataacquisition and registration. (a) DJI Matrice 30T system and source visible and infrared images. (b) Image pairs after registration.

Table 1. Key parameters of the infrared and visible cameras mounted on the M30T.

Camera	Focal Length (mm)	Pixel Pitch (μm)	Image Size (Pixels)
Infrared	9.1	12	Normal: 640 × 512
Visible	4.5	1.6	4000 × 3000

Figure 3. Sample images at different altitudes. The pixel sizes of the human target are as follows: approximately 25 × 49 pixels at 30 m, 18 × 32 pixels at 50 m, and 12 × 21 pixels at 70 m. Arrows indicate the target persons.

Figure 4. Sample images under different weather conditions. Arrows indicate the target persons.

Figure 5. Image distribution of the UAV-MSR dataset. (a) Number of images at different altitudes. (b) Number of images under different weather conditions. (c) Number of bounding boxes under different occlusion conditions.

3.2. Image Registration

The source images are misaligned due to differences in imaging systems and field sizes. Image registration establishes pixel-level correspondence between the two modalities. First, radial distortion of infrared images is corrected by Zhang’s checkerboard calibration algorithm [35]. Next, visible images are cropped and resized to match the field of view and size of the infrared images. The image pairs are then aligned via the pretrained MINIMA matching model [36]. Figure 2b shows the aligned images at a resolution of 640 × 512 pixels. We select 20 samples each for unoccluded targets and targets occluded by vegetation to analyze the registration errors via root mean square error (RMSE). The results in Table 2 verify the stability of registration performance.

Table 2. Statistical analysis of image registration error. The reprojection error threshold is set to 3 pixels.

Condition	Metric	Mean	Std	Min	Max
Unoccluded	RMSE	1.0046	0.1209	0.763	1.230
Occluded	RMSE	1.0218	0.1170	0.856	1.318

3.3. Annotations

Human targets are difficult to distinguish from the background in visible images. Therefore, we annotate them on infrared images [37] where they are more prominent. Objects at the image edges or invisible to the human eye are excluded from annotation. In total, 7573 objects were annotated.

3.4. Limitations

The dataset was collected only at altitudes of 30, 50, and 70 m, without samples collected at lower or higher heights. This limits the comprehensive assessment of the model’s adaptability to varying target scales.

In this experiment, the obstruction was only qualitatively described as vegetation, and its density was not quantitatively characterized. Subsequent research will introduce multi gradient vegetation density experiments to further enhance the generality of the dataset.

The dataset was collected by UAV. Constrained by the data acquisition equipment and experimental conditions, the diversity of the dataset is limited, and some samples exhibit background repetition. We plan to introduce more terrain and scenarios in future work to enrich the UAV-MSR dataset.

4. Proposed Method

In this section, the architectural schematic of the proposed DCAFuse is first introduced, and then the design of the feature extraction module, feature fusion module and loss function are detailed.

4.1. Overview Framework

As illustrated in Figure 6a, the proposed framework consists of three core components: feature extraction, feature fusion, and feature reconstruction. The inputs are a visible image $I_v\in {\mathbf{R}}^{\mathrm{H}\times \mathrm{W}\times 3}$ and an infrared image $I_r\in {\mathbf{R}}^{\mathrm{H}\times \mathrm{W}\times 1}$, where $\mathrm{H}$ and $\mathrm{W}$ denote the height and width, respectively. We extract local features using convolution layers and global features using self-attention mechanism from $I_v$ and $I_r$, obtaining features $F_v$ and $F_r\in {\mathbf{R}}^{\mathrm{H}\times \mathrm{W}\times \mathrm{C}}$, which can be expressed as:

$$\left\{F_v,F_r\right\}=\left\{{\mathrm{H}}_{\mathrm{FE}}\left(I_v\right),{\mathrm{H}}_{\mathrm{FE}}\left(I_r\right)\right\}$$

(1)

where ${\mathrm{H}}_{\mathrm{FE}}$ denotes feature extraction and $\mathrm{C}$ is the number of output channel, set to 16. Then, these features are fed into the fusion module, which incorporates both spatial and channel dimensions to generate the fused feature $F_f\in {\mathbf{R}}^{\mathrm{H}\times \mathrm{W}\times \mathrm{C}}$. The process can be formulated as:

$$F_f=H_{\mathrm{CAM}}\left(H_{\mathrm{SAM}}\left(F_v,F_r\right)\right)$$

(2)

where ${\mathrm{H}}_{\mathrm{SAM}}$ denotes the spatial attention module and ${\mathrm{H}}_{\mathrm{CAM}}$ denotes the channel attention module. Finally, the convolutional layers and ReLU activation layers are adopted to reconstruct the fused features and obtain the final fused image:

$$I_f={\mathrm{H}}_{\mathrm{FR}}\left(F_f\right)$$

(3)

where ${\mathrm{H}}_{\mathrm{FR}}$ denotes feature reconstruction.

Figure 6. Architectural schematic of the proposed DCAFuse. (a) Workflow of the DCAFuse. (b) Spatial attention module. (c) Channel attention module.

4.2. Feature Extraction

The feature extraction module employs a cascaded CNN–Transformer hybrid architecture, as illustrated in Figure 7. The CNN branch precisely extracts local textural and structural features, which integrates a convolutional block with a cross central difference convolution (C-CDC) block [38]. The C-CDC block further constrains the receptive field to horizontal/vertical or diagonal sub-neighborhoods to capture fine-grained directional details, thereby enhancing the representation of local details.

Figure 7. Feature extraction module.

The Transformer branch is designed to capture intra-domain global contextual features and model long-range dependencies. The self-attention (SA) mechanism performs global modeling via the interaction among query $\mathbf{Q}$, key $\mathbf{K}$, and value $\mathbf{V}$ [14]. For a feature matrix X, the three components are derived by:

$$\mathbf{Q}=X{\mathbf{W}}^Q,\mathbf{K}=X{\mathbf{W}}^K,\mathbf{V}=X{\mathbf{W}}^V$$

(4)

where ${\mathbf{W}}^Q$, ${\mathbf{W}}^K$, and ${\mathbf{W}}^V$ indicate linear transformations. The attention is computed by the dot-product operation as follows:

$$Attention\left(\mathbf{Q},\mathbf{K},\mathbf{V}\right)=Softmax\left({\displaystyle \frac{\mathbf{Q}{\mathbf{K}}^T}{\sqrt{d_k}}}\right)\mathbf{V}$$

(5)

where $d_k$ is the dimension of $\mathbf{K}$.

The outputs of the two branches are fused via an addition operation:

$$\begin{array}{cc}\hfill F_v& ={\mathrm{H}}_{\mathrm{CNN}}\left(\mathrm{Conv}\left(I_v\right)\right)+{\mathrm{H}}_{\mathrm{MSA}}\left(\mathrm{Conv}\left(I_v\right)\right)\hfill \\ \hfill F_r& ={\mathrm{H}}_{\mathrm{CNN}}\left(\mathrm{Conv}\left(I_r\right)\right)+{\mathrm{H}}_{\mathrm{MSA}}\left(\mathrm{Conv}\left(I_r\right)\right)\hfill \end{array}$$

(6)

where $\mathrm{Conv}$ denotes 1 × 1 convolution, ${\mathrm{H}}_{\mathrm{CNN}}$ denotes CNN branch, and ${\mathrm{H}}_{\mathrm{MSA}}$ denotes multi-head SA branch. This CNN-SA hybrid architecture integrates local detail extraction with global dependency modeling to produce more comprehensive feature representations.

4.3. Cross-Domain Feature Fusion

To efficiently fuse the extracted features, we construct a composite attention fusion module as the core component of the proposed network, which consists of a spatial attention module and a channel attention module, as illustrated in Figure 6b,c.

4.3.1. Spatial Attention Module

Complementary information: As shown in Figure 8, the green regions represent highly consistent information shared between the two modalities [31]. The blue regions represent the unique and discriminative features, which are complementary information [16]. The core of image fusion is to fully exploit complementary information for a more informative fused result. Cross-attention (CA) effectively captures salient complementary information, such as human targets in infrared images. For relatively weak complementary information (e.g., buildings and trees in infrared images), CA struggles to fully perceive these subtle features, potentially causing blurring artifacts in the fused image. This phenomenon is validated in the ablation experiments. To this end, we design a differential cross-attention guided fusion architecture to comprehensively explore and fuse complementary information.

Figure 8. Complementary information in the image fusion task. Green region: relevant information. Blue region: complementary information.

Differential cross-attention guided fusion: After deep feature extraction, feature fusion is performed via the proposed differential cross-attention (DCA) guided fusion module, which consists of two Transformer blocks, as shown in Figure 6b.

The features $F_v$ and $F_r$ are first fed into the DCA block to enhance the weak complementary features. The design of DCA is depicted in Figure 9. In the context of infrared and visible image fusion, $\mathbf{Q}$ is derived from the visible image, while $\mathbf{K}$ and $\mathbf{V}$ are derived from the infrared image [39], which are computed as follows:

$$\left\{\left.{\mathbf{Q}}_{\mathbf{1}},{\mathbf{K}}_{\mathbf{1}},{\mathbf{V}}_{\mathbf{1}}\right\}\right.=\left.\left\{F_v{\mathbf{W}}^Q\right.,F_r{\mathbf{W}}^K,F_r{\mathbf{W}}^V\right\}$$

(7)

Figure 9. Differential cross-attention mechanism.

The cross-attention mechanism calculates attention weights by constructing a similarity matrix between query and key features. The multi-head cross-attention (MCA) is employed to establish cross-modal interaction [15], which is denoted as:

$$F_{CA}=\mathrm{MCA}\left({\mathbf{Q}}_1,{\mathbf{K}}_1,{\mathbf{V}}_1\right)$$

(8)

Affected by dot-product similarity and softmax normalization, cross-attention tends to assign higher weights to salient features, resulting in representation degradation in low-intensity features [40]. To address this, we introduce a differential approach to guide the model toward focusing more on these relatively weak complementary features, expressed as:

$$F_{DCA}={\mathrm{LN}({\mathbf{V}}_{\mathbf{1}}-F}_{CA})+{\mathbf{Q}}_1$$

(9)

where $\mathrm{LN}(·)$ represents layer normalization. The output of the DCA block is as follows:

$$F_{DCA}^{out}=\mathrm{MLP}\left(\mathrm{LN}\left(F_{DCA}\right)\right)+F_{DCA}$$

(10)

where $\mathrm{MLP}(·)$ consists of two fully connected layers with a GeLU activation function in between.

Following the DCA block, a CA block is adopted to further refine and integrate the global salient features across modalities. The overall process can be represented as:

$$\begin{array}{c}\left\{{\mathbf{Q}}_2,{\mathbf{K}}_2,{\mathbf{V}}_2\right\}=\left\{F_{DCA}^{out}{\mathbf{W}}^Q,F_r{\mathbf{W}}^K,F_r{\mathbf{W}}^V\right\}\\ F_{CA}=\mathrm{MCA}\left({\mathbf{Q}}_2,{\mathbf{K}}_2,{\mathbf{V}}_2\right)+{\mathbf{Q}}_2\\ F_{SAM}^{out}=\mathrm{MLP}\left(\mathrm{LN}\left(F_{CA}\right)\right)+F_{CA}\end{array}$$

(11)

As presented in Equation (11), ${\mathbf{Q}}_2$ derived from $F_{DCA}^{out}$ aggregates cross-domain information via attention-weighted with ${\mathbf{K}}_2$ and ${\mathbf{V}}_2$ from $F_r$. The cascaded cross-domain fusion structure, incorporating DCA and CA, fully enhances and integrates the complementary information between different modalities.

4.3.2. Channel Attention Module

After the spatial attention module, a channel attention module is deployed to model the interdependencies among channel dimensions and adaptively recalibrate channel-wise feature responses, as illustrated in Figure 6c. The channel attention module integrates average pooling and max pooling operations [41], which is formulated as:

$$\begin{array}{c}{\mathbf{M}}_c\left(F\right)=\mathrm{AvgPool}\left(F\right)+\mathrm{MaxPool}\left(F\right)\\ F_f=\sigma \left({\mathbf{W}}_i\left({\mathbf{M}}_c\left(F\right)\right)\right)\end{array}$$

(12)

where $F=\mathrm{concat}(F_{SAM}^{out},F_r)$, $\sigma$ denotes the sigmoid function, and ${\mathbf{W}}_i$ includes two linear transformations and a ReLU function.

Following the complete fusion of cross-domain complementary information, the decoder reconstructs the fused image by projecting the fused features into the image domain, consisting of several convolutional layers and ReLU layers.

4.4. Loss Function

The model is trained in an end-to-end pipeline. A dedicated loss function is developed to adequately capture the inherent characteristics of source images, ensuring the enhancement of salient information while retaining texture details. The loss function is formulated as a weighted aggregation of structural similarity (SSIM), gradient, and pixel loss:

$$L_{total}={\lambda }_1{L}_{ssim}+{\lambda }_2{L}_{grad}+{\lambda }_3{L}_{pixel}$$

(13)

where ${\lambda }_1,{\lambda }_2,{\lambda }_3$ are trade-off hyperparameters. The SSIM loss assesses the degree of similarity within the image pair through a joint evaluation of their brightness, contrast, and structural information [32], which is formulated as:

$$L_{ssim}=\left(1-\mathrm{SSIM}\left(I_f,I_v\right)\right)+\left(1-\mathrm{SSIM}\left(I_f,I_r\right)\right)$$

(14)

where $\mathrm{SSIM}(·)$ is the structural similarity function.

The gradient information usually characterizes the texture details and structure of an image [13]. Therefore, we employ gradient loss to guide the fused image in preserving complex texture information, which is formulated as:

$$L_{grad}={\displaystyle \frac{1}{HW}}{∥\nabla I_f-\mathrm{Max}(\nabla I_v,\nabla I_r)∥}_1$$

(15)

where ∇ is the gradient operation, $\mathrm{Max}(·)$ is element-wise maximum selection, and ${\parallel ·\parallel }_1$ represents the matrix L1-norm.

We adopt a segmented pixel loss to balance between salient feature preservation and detail information retention during network training, defined as:

$$L_{pixel}={\displaystyle \frac{1}{HW}}{∥I_f-M(I_v,I_r)∥}_1$$

(16)

where $\mathrm{M}(·)$ represents element-wise aggregation operation, which is a segmented mapping function that assigns weights to pixels based on their intensity levels. For the top $\alpha \%$ of pixels with the highest intensity values, maximum selection is applied. For the remaining pixels, mean aggregation is adopted. The pixel intensity value is calculated as: ${\widehat{p}}_{i,j}=\nabla I_{i,j}·I_{i,j}$. In the experiments, the threshold $\alpha$ is set to 20.

5. Results

5.1. Experimental Settings

5.1.1. Implementation Details

The network is implemented in PyTorch 2.0.0 (CUDA 11.8) and trained on a single NVIDIA GeForce GTX 4060 GPU with a batch size of 4, a patch size of 128, and 60,000 iterations. The Adam optimizer is employed with an initial learning rate of $1\times {10}^{-4}$, which is reduced by a factor of 0.75 at 15,000 and 25,000 iterations.

5.1.2. Datasets

To validate the efficacy of the proposed DCAFuse, comprehensive experiments are conducted on our constructed dataset and three public datasets. Our UAV-MSR dataset comprises 2779 training pairs, 510 validation pairs, and 549 test pairs. To evaluate the generalization ability of the proposed method, we also tested on three public datasets: RoadScene (221 pairs) [42], MSRS (361 pairs) [43], and M3FD (300 pairs) [44].

5.1.3. Compared Methods

To evaluate the performance of our method, we compare it with nine state-of-the-art (SOTA) image fusion approaches on the above four datasets, including SwinFusion [15], Diff-IF [26], CrossFuse [16], DFVO [45], AMLCA [46], ATFuse [47], SHIP [33], FDFuse [48], and LUT-Fuse [49].

5.1.4. Evaluation Metrics

Six objective metrics are employed to evaluate the quality of fused images, including entropy (EN), Sum of Correlations of Differences (SCD), spatial frequency (SF), visual information fidelity (VIF), mean squared error (MSE), and structural similarity index measure (SSIM). Additionally, the average rank (Avg.rank) is calculated based on these metrics to provide an overall performance assessment.

5.2. Fusion Results on UAV-MSR Dataset

5.2.1. QualitativeComparison

Figure 10 presents the qualitative comparison results of four typical samples from the UAV-MSR dataset. Compared with other fusion networks, the proposed DCAFuse possesses two prominent strengths. First, it is more adept at preserving the delicate and even weak radiation information in infrared images. As indicated by the red-highlighted regions of Figure 10a,b, SwinFusion, Diff-IF, CrossFuse, AMLCA, and SHIP fail to retain the integrity of tree trunk details in infrared images. DFVO and LUT-Fuse suffer from brightness degradation in human target. Although ATFuse and FDFuse produce relatively satisfactory results, the proposed method achieves the best visual effects, simultaneously preserving complete infrared information and enhancing prominent targets.

Figure 10. Qualitative results of different fusion methods on UAV-MSR dataset. (a) 931.jpg. (b) 787.jpg. (c) 1776.jpg. (d) 77.jpg. Close-ups are shown in red and green boxes. The same applies to the following figures.

Second, our method can favorably balance target saliency and detail preservation. As illustrated in Figure 10c,d, AMLCA introduces considerable artificial noise, while CrossFuse, DFVO and LUT-Fuse exhibit insufficient infrared target saliency and blurred contours. In the green boxes of Figure 10c,d, our fusion result preserves fine leaf textures, whereas ATFuse and SHIP produce unnatural shadows that are absent in the input images. Overall, our method generates high-quality fusion images consistent with human visual perception.

5.2.2. QuantitativeComparison

Table 3 shows the quantitative assessment results with six metrics on the UAV-MSR dataset. The proposed method achieves the best average ranking, demonstrating superior overall performance across all metrics. In particular, DCAFuse obtains the optimal values on EN, SCD, VIF, MSE, and SSIM, indicating that our fused images are more informative, contain clearer details, and introduce fewer noises. In addition, our method ranks second in SF, showing its effective capability in preserving spatial frequency and structural information. Consistent with the qualitative observations, these quantitative results further validate the superiority of our model in integrating complementary information.

Table 3. Quantitative results on UAV-MSR dataset. ↑: higher better. ↓: lower better. Bold: optimal performance. Underlined: second-best performance. The same applies to the following tables.

Method	EN ↑	SCD ↑	SF ↑	VIF ↑	MSE ↓	SSIM ↑	Avg.Rank
SwinFusion	6.603	1.245	15.558	0.791	0.023	1.057	6.33
Diff-IF	6.634	1.495	16.198	0.658	0.024	0.812	6.33
CrossFuse	6.608	1.423	14.056	0.570	0.018	0.960	7.50
DFVO	6.682	1.562	17.052	0.582	0.017	0.941	4.50
AMLCA	6.652	1.293	14.511	0.557	0.021	1.033	7.17
ATFuse	6.718	1.283	15.428	0.776	0.015	1.040	4.00
SHIP	6.605	1.269	16.134	0.769	0.022	1.014	6.67
FDFuse	6.672	1.484	15.006	0.589	0.021	0.967	6.00
LUT-Fuse	6.677	1.395	14.835	0.592	0.016	1.046	5.00
DCAFuse	6.800	1.604	16.216	0.799	0.015	1.091	1.17

5.3. Generalization Analysis on Public Dataset

5.3.1. Qualitative Comparison

The fusion results on the RoadScene dataset are shown in Figure 11. When handling images with localized exposure area, the fused images of SwinFusion, Diff-IF, CrossFuse, AMLCA, ATFuse, and LUT-Fuse exhibit significant deficiencies, including infrared detail loss and severe blurring, such as the tree trunk in the red box in Figure 11a and the wires in the enlarged area in Figure 11b. For the human target in the red box of Figure 11a, SHIP and FDFuse struggle to highlight the valuable information from infrared images, while DFVO reduces the brightness of radiation information. Our method more effectively preserves and enhances complementary information from the source images and produces more naturally looking fused images, benefiting from the selective retention of background infrared pixels.

Figure 11. Qualitative results on RoadScene dataset. (a) FLIR_04598.jpg. (b) FLIR_00311.jpg.

For the MSRS dataset, the fusion results are presented in Figure 12. CrossFuse and DFVO weakened the brightness of the human target, as seen in the green box of Figure 12a. In addition, the results generated by other algorithms are blurry with weak infrared details, as illustrated by the distant tree trunks and buildings within the two red boxes of Figure 12. By contrast, the proposed method exhibits a stable and reliable capability to capture comprehensive information, even weak thermal radiation, while preserving more details.

Figure 12. Qualitative results on MSRS dataset. (a) 01020N.png. (b) 01152N.png.

For the M3FD dataset, the fusion results are presented in Figure 13. Due to smoke interference, Diff-if, CrossFuse, AMLCA, SHIP, and LUT-Fuse result in blurred target contours, as shown in the two enlarged boxes of Figure 13a. However, other methods struggle to balance infrared radiation information with visible light texture information during the fusion process. As illustrated in Figure 10b, DCAFuse achieves a more favorable trade-off between salient targets and fine details, resulting in optimal visual fusion performance among all compared methods.

Figure 13. Qualitative results on M3FD dataset. (a) 00787.png. (b) 00910.png.

5.3.2. Quantitative Comparison

Table 4, Table 5 and Table 6 show the quantitative evaluation results on the three public datasets. DCAFuse achieves the best performance on EN, SCD, and SSIM, demonstrating that our method produces fused images with richer texture details, higher sharpness, and stronger correlation with the source images. Furthermore, it maintains favorable performance on SF and MSE metrics, suggesting that our model performs stably and reliably under challenging conditions, including low illumination, thick smoke, and exposure scenarios.

Table 4. Quantitative results on RoadScene dataset.

Method	EN ↑	SCD ↑	SF ↑	VIF ↑	MSE ↓	SSIM ↑	Avg.Rank
SwinFusion	7.069	1.443	14.004	0.669	0.054	1.002	4.83
Diff-IF	7.159	1.234	14.955	0.632	0.051	0.822	6.67
CrossFuse	7.178	1.269	12.092	0.577	0.048	0.744	7.50
DFVO	7.062	1.418	14.861	0.510	0.049	0.933	6.33
AMLCA	7.380	1.317	13.928	0.627	0.046	0.888	5.33
ATFuse	7.222	1.335	14.882	0.709	0.045	0.946	3.17
SHIP	7.202	1.299	16.082	0.659	0.051	0.901	4.83
FDFuse	6.918	1.301	15.942	0.571	0.063	0.880	7.33
LUT-Fuse	6.740	1.495	14.863	0.531	0.068	0.924	7.00
DCAFuse	7.258	1.501	16.638	0.639	0.036	0.991	1.83

Table 5. Quantitative results on MSRS dataset.

Method	EN ↑	SCD ↑	SF ↑	VIF ↑	MSE ↓	SSIM ↑	Avg.Rank
SwinFusion	6.525	1.569	10.892	0.931	0.033	0.955	5.00
Diff-IF	6.669	1.624	11.459	0.842	0.039	0.961	4.67
CrossFuse	6.498	1.355	9.616	0.878	0.034	0.872	7.00
DFVO	6.244	1.289	11.426	0.684	0.035	0.872	7.67
AMLCA	6.599	1.631	10.930	0.924	0.037	0.963	4.50
ATFuse	6.388	1.584	11.093	0.931	0.031	0.812	5.50
SHIP	6.579	1.554	11.638	0.982	0.039	0.920	4.83
FDFuse	6.159	1.321	10.907	0.713	0.035	0.840	8.33
LUT-Fuse	6.527	1.638	12.502	0.835	0.036	0.927	4.67
DCAFuse	6.529	1.648	11.738	0.938	0.033	0.972	2.00

Table 6. Quantitative results on M3FD dataset.

Method	EN ↑	SCD ↑	SF ↑	VIF ↑	MSE ↓	SSIM ↑	Avg.Rank
SwinFusion	6.799	1.396	14.144	0.837	0.058	0.936	4.50
Diff-IF	6.790	1.322	14.080	0.782	0.059	0.932	6.50
CrossFuse	6.667	1.484	11.059	0.606	0.046	0.869	7.67
DFVO	6.652	1.583	16.712	0.709	0.038	0.884	4.67
AMLCA	6.746	1.575	11.608	0.699	0.045	0.929	5.83
ATFuse	6.916	1.356	14.176	0.822	0.046	0.908	4.67
SHIP	6.837	1.305	15.141	0.812	0.058	0.826	5.83
FDFuse	6.605	1.232	14.649	0.795	0.076	0.935	7.00
LUT-Fuse	6.809	1.576	15.838	0.610	0.066	0.926	5.50
DCAFuse	6.929	1.639	15.551	0.765	0.045	0.938	2.33

Overall, quantitative and qualitative results demonstrate that our method delivers higher-quality results for IVIF tasks compared with SOTA methods. Experiments on public datasets confirm the robust generalization of the proposed method, verifying its effectiveness across diverse illumination conditions and scenarios.

5.4. Performance Analysis on Detection Tasks

To further evaluate the quality of fused images produced by different algorithms in subsequent vision tasks, we employ the pretrained YOLOv11 model as the baseline detector for human target detection on the UAV-MSR dataset. As shown in the detection results in Figure 14, Diff-IF and DFVO miss some thermal targets, which compromises the overall detection performance. As shown in Table 7, our fused images achieve the highest mAP_0.5 of 0.7637 and the second-best mAP_0.5:0.95 of 0.4962, facilitating more accurate detection of human targets. In contrast, the fused images generated by our method can visually enhance target saliency and facilitate subsequent detection algorithms to achieve more accurate results. In addition, we further compare the detection performance of visible (VIS), infrared (IR) and fused images. As shown in Table 8, the fused images have the best performance on both metrics, which fully demonstrates the effectiveness of multimodal fusion.

Figure 14. Detection performance of the fused images on UAV-MSR dataset.

Table 7. Detection performance comparison on UAV-MSR dataset.

Metrics	SwinFusion	Diff-IF	CrossFuse	DFVO	AMLCA	ATFuse	SHIP	FDFuse	LUT-Fuse	DCAFuse
mAP0.5	0.7153	0.7250	0.6407	0.6269	0.7209	0.7396	0.7242	0.7104	0.7438	0.7637
mAP0.5:0.95	0.5029	0.4779	0.4626	0.4231	0.4758	0.4930	0.4752	0.4809	0.4925	0.4964

Table 8. Detection performance of different image modalities on the UAV-MSR dataset.

Metrics	VIS	IR	Fused
mAP0.5	0.4861	0.7150	0.7637
mAP0.5:0.95	0.1682	0.4265	0.4964

5.5. Ablation Studies

In this section, five ablation experiments on the UAV-MSR dataset analyze the effects of key components, including the SA block, DCA block, spatial and channel attention module, the number of the feature fusion model, and the loss function.

5.5.1. Effect of SA on Feature Extraction

To evaluate the impact of the SA block on feature extraction, two architectures (CNN and CNN-SA) are utilized to extract features. Figure 15 visualizes the extracted features from both architectures. The CNN-SA branch produces sharper thermal targets and richer background textures, demonstrating its advantage in global and local feature extraction. From the results in Table 9, it is evident that with the introduction of the SA block, the model’s performance has been considerably improved.

Figure 15. Feature visualization comparison of CNN and CNN-SA branches.

Table 9. Ablation study results on UAV-MSR dataset. Bold shows the optimal performance.

Method	EN	SCD	SF	VIF	MSE	SSIM
Proposed	6.800	1.604	16.216	0.799	0.015	1.091
w/o SA	6.677	1.395	14.835	0.592	0.016	1.046
w/o DCA	6.672	1.484	15.006	0.589	0.021	0.967
w/o Spatial	6.789	1.591	15.494	0.771	0.015	1.076
w/o Channel	6.775	1.587	15.261	0.736	0.015	1.089

5.5.2. Effect of DCA Block

To investigate the effectiveness of the DCA block on feature fusion, we removed it and replaced it with the standard CA mechanism (denoted as w/o DCA). Figure 16 presents the feature visualization comparison of the first five channels between the DCA and w/o DCA approaches, with the source images shown in Figure 15. For salient complementary features, such as human targets, DCA and w/o DCA perform similarly and effectively capture this prominent information. However, for weak complementary features, as illustrated in the red boxes of Figure 16 (showing the texture of tree trunks), w/o DCA struggles to fully capture these subtle details. In contrast, DCA overcomes this limitation and successfully preserves fine-grained complementary features. The quantitative results in Table 9 further validate the effectiveness of DCA.

Figure 16. Feature visualization comparison of DCA and w/o DAC. Close-ups are shown in red boxes.

5.5.3. Effect of Spatial and Channel Attention

To analyze the necessity of SAM and CAM, we conducted ablation experiments by removing each module individually (denoted as w/o Spatial and w/o Channel). Results in Table 9 demonstrate a significant performance degradation when either module is removed. Specifically, removing SAM impairs the retention of structural details, while excluding CAM weakens the preservation of meaningful multi-modal features. This ablation study verifies that both attention modules play an indispensable role in our fusion framework, and their collaborative learning effectively improves the quality of the fused image.

5.5.4. Effect of Iterations Number N

To determine the optimal setting of the iteration number N of the feature fusion module, we evaluate the model performance for N = 1, 2, 3, and 4. The quantitative comparison in Table 10 shows that as N increases, both the information content and the overall visual perception of the fused image are improved. However, when N is increased from 3 to 4, the performance improvement becomes very limited compared to the increase in computational cost. Thus, with both performance and efficiency taken into account, N = 3 is selected.

Table 10. Ablation study results of iteration number N on UAV-MSR dataset.

Method	EN	SCD	SF	VIF	MSE	SSIM
N = 1	6.796	1.579	16.161	0.759	0.016	1.065
N = 2	6.777	1.587	16.183	0.769	0.015	1.075
N = 3	6.800	1.604	16.216	0.799	0.015	1.091
N = 4	6.818	1.602	16.160	0.811	0.015	1.093

5.5.5. Effect of Pixel Loss Function

To determine the impact of the segmented operation within the pixel loss function, we conducted six experiments with different segmentation thresholds: $\alpha$ = 100, 80, 60, 40, 20, and 0, where $\alpha =100$ and $\alpha =0$ denote the maximum and average pixel loss, respectively. As illustrated in Figure 17, the metrics increase monotonically as $\alpha$ decreases. This is because the averaging operation gradually becomes dominant, enabling more effective preservation of background details. The ablation study results demonstrate that the model achieves optimal performance when $\alpha$ = 20. Specifically, the maximum operation is applied to the top 20% of pixels by intensity, while the average operation is applied to the remaining pixels. We further compare the fusion results of the segmented pixel loss ($\alpha =20$) with widely used maximum pixel loss. As shown in Figure 18, the segmented pixel loss function largely preserves the texture details of visible images while prominently highlighting infrared thermal targets, leading to improved visual quality.

Figure 17. Ablation studies of the segmented pixel loss with different values of $\alpha$.

Figure 18. Qualitative comparison of fusion results with maximum and segmented pixel loss.

5.6. Computational Efficiency

The computational efficiency comparison with SOTA fusion methods is presented in Table 11, including three aspects: parameter count (size), floating point operations (FLOPs), and average inference time. As can be observed from the results, LUT-Fuse achieves the highest computational efficiency among all compared methods. In addition, AMLCA and SHIP achieve the second-best results in FLOPs and average inference time respectively. Notably, our proposed DCAFuse has a relatively small parameter size and balances lightweight design and practical performance well. Nevertheless, there is still room for further optimization in its inference speed.

Table 11. Computational efficiency evaluation of different fusion methods on 20 image pairs from the UAV-MSR dataset.

Metrics	SwinFusion	Diff-IF	CrossFuse	DFVO	AMLCA	ATFuse	SHIP	FDFuse	LUT-Fuse	DCAFuse
Size (M)	0.97	23.74	23.31	9.72	14.36	1.05	56.78	0.52	0.07	0.38
FLOPs (G)	318.65	637.65	140.68	781.31	8.06	31.05	167.30	241.60	2.58	96.96
Time (ms)	1179.62	1730.05	853.16	236.99	446.52	61.16	58.19	289.58	4.65	268.91

6. Conclusions

In this paper, we construct an aligned infrared-visible dataset UAV-MSR for casualty search and rescue in jungle scenarios. We then propose a novel fusion network for IVIF tasks, named DCAFuse. The proposed network introduces a differential cross-attention mechanism to guide the model in deeply exploiting cross-modal complementary information while effectively enhancing target saliency. Experimental evaluations on four datasets confirm that our DCAFuse yields superior fusion performance under various illumination conditions. In summary, this study constructs a dedicated SAR dataset and proposes an effective feature fusion network, providing technical support for UAV-based situational awareness and victim search in complex wilderness environments.

Although the cross-attention mechanism is efficient in image fusion, it exhibits certain limitations with respect to computational complexity and operational efficiency, making it difficult to meet the requirements of real-time processing. Future work will concentrate on reducing computational overhead by exploring lightweight attention designs. In addition, the optimal threshold for segmented pixel loss may vary across datasets depending on factors such as thermal contrast and background complexity. In future work, we will explore adaptive methods to determine this threshold based on the characteristics of the input data.