FIRGAN: High-Fidelity Infrared–Visible Image Fusion for UAV Detection Tasks

Abstract

Infrared–visible image fusion aims to generate informative representations for both visual perception and downstream vision tasks. However, existing methods often prioritize visual quality while overlooking task adaptability, optimization efficiency, and effective spatial-frequency representation learning. To address these issues, we propose FIRGAN, a high-fidelity multi-modal fusion framework for downstream detection applications that jointly optimizes fusion quality and detection-relevant feature representation. Specifically, an SFFM is introduced to exploit MSCN-derived statistical priors for adaptive spectrum enhancement. A Dual-branch Fusion Network is designed to collaboratively model local structural details and global semantic dependencies through complementary CNN and Transformer branches. In addition, an IQA-guided optimization strategy performs quality-aware patch selection to improve training efficiency and robustness. Extensive experiments on public infrared–visible fusion datasets demonstrate that the proposed method achieves competitive visual quality and quantitative performance while providing more discriminative representations for downstream UAV detection tasks. These results verify the effectiveness of the proposed framework for multi-modal fusion for downstream detection applications.

1. Introduction

Infrared–visible image fusion aims to integrate complementary information from infrared and visible modalities into a unified representation. Infrared images provide robust thermal responses and salient target information under adverse illumination conditions, whereas visible images contain abundant texture and structural details. By combining the advantages of both modalities, fused images can significantly improve scene perception and support downstream vision tasks such as object detection, tracking, and intelligent surveillance [1,2].

Existing fusion methods primarily focus on improving visual quality and image reconstruction performance. Traditional approaches based on multi-scale transforms and sparse representation preserve modality-specific information through handcrafted feature extraction [1,3]. More recently, deep learning-based methods have achieved remarkable progress by learning hierarchical feature representations directly from data. Representative convolutional neural network (CNN)-based methods, such as DenseFuse, FusionGAN, IFCNN, and RFN-Nest [4,5,6,7], have substantially improved fusion quality and structural preservation. Transformer-based fusion methods have recently attracted increasing attention [8,9,10]. Benefiting from self-attention mechanisms, Transformers can effectively model global interactions and improve semantic consistency. However, high visual quality does not necessarily guarantee superior performance in downstream detection tasks. As a result, fusion for downstream detection applications has emerged as an important research direction, aiming to jointly optimize fusion quality and task-specific feature representation.

Despite recent advances, existing fusion methods still face several challenges. First, many approaches focus primarily on image reconstruction while paying limited attention to the discriminative features required by downstream detection models. Consequently, the generated fused images may exhibit satisfactory visual quality but insufficient task adaptability. Second, most fusion frameworks treat all training samples equally during optimization. Since different image regions contribute unequally to feature learning, redundant or low-information patches may dominate the training process, leading to reduced optimization efficiency and limited representation capability. Third, although frequency-domain information contains rich structural and texture cues, existing methods often rely on fixed spectral operations or directly manipulate frequency coefficients without considering local image characteristics [11,12,13]. Therefore, informative frequency components cannot be effectively exploited for adaptive multi-modal representation learning.

To address these issues, we propose a multi-modal fusion framework for downstream detection applications termed FIRGAN. Specifically, a Statistical Frequency Feature Module (SFFM) is introduced to establish a bridge between local statistical priors and frequency-domain representation learning. By leveraging mean-subtracted contrast normalization (MSCN)-derived statistical information to generate adaptive frequency weights, the proposed module selectively enhances informative spectral components and improves spatial-frequency feature representation. Moreover, a Dual-branch Fusion Network is designed to collaboratively model local structural details and global semantic dependencies through complementary CNN and Transformer branches. To further improve training efficiency, an image quality assessment (IQA)-guided optimization strategy is developed to perform quality-aware patch selection and ranking, enabling the network to focus on informative regions during optimization.

Extensive experiments on public infrared–visible fusion datasets demonstrate that the proposed method achieves superior visual quality and competitive quantitative performance. In addition, the generated fused images provide more discriminative representations for downstream unmanned aerial vehicle (UAV) detection tasks, validating the effectiveness of the proposed framework. Figure 1 provides a representative visual comparison on a UAV traffic scene. Several competing methods exhibit either weakened thermal responses, local color bias, or blurred vehicle and lane boundaries. In contrast, FIRGAN preserves salient vehicle responses from the infrared image while retaining clearer lane markings and background structures from the visible image. The enlarged regions further show a more balanced treatment of local contrast and texture, motivating the joint use of statistical frequency enhancement and local–global feature fusion.

Figure 1. Visual comparison of different image fusion algorithms.

The main contributions of this work are summarized as follows:

• A multi-modal fusion framework for downstream detection applications is proposed to jointly preserve infrared target saliency and visible structural details, generating fused images suitable for downstream UAV vision tasks.
• An SFFM is introduced to leverage MSCN-derived statistical priors for adaptive frequency weighting, enabling informative spectrum enhancement and robust spatial-frequency representation learning.
• A Dual-branch Fusion Network is designed to collaboratively capture local structural details and global semantic dependencies through complementary CNN and Transformer branches.
• An IQA-guided optimization strategy is developed to perform quality-aware patch selection, improving training efficiency and fusion robustness.

2. Related Work

2.1. Infrared–Visible Fusion

Infrared–visible image fusion aims to integrate thermal saliency from infrared images and rich structural details from visible images into a unified representation. Early fusion methods mainly relied on multi-scale transforms, sparse representation, and optimization-based models, such as wavelet transform, pyramid decomposition, and sparse coding techniques [1]. Although these approaches preserve modality-specific information, their handcrafted feature extraction limits representation capability in complex scenes.

With the success of deep learning, CNN-based fusion methods have achieved significant improvements by learning hierarchical feature representations directly from data. Representative methods, including DenseFuse [4], FusionGAN [1,5], IFCNN [6] and RFN-Nest [7], demonstrate superior feature extraction and image reconstruction capabilities compared with traditional approaches. However, convolutional operations primarily focus on local neighborhoods and often struggle to capture long-range dependencies.

2.2. Transformer-Based Fusion

Transformer-based image fusion methods have attracted considerable attention due to their ability to model long-range dependencies and global contextual information through self-attention mechanisms. SwinFusion [8] employs a hierarchical Swin Transformer to capture multi-scale contextual representations while maintaining computational efficiency using window-based self-attention. DATFuse [14] introduces a dual-attention Transformer architecture that enhances cross-modal interaction by jointly exploiting spatial and channel attention, thereby improving complementary feature aggregation between infrared and visible images. TGFuse [13] further integrates CNNs and Transformers into a hybrid architecture, enabling complementary learning of local structural details and global semantic dependencies.

Although these methods have demonstrated impressive performance, their improvements are mainly achieved through architectural optimization and spatial feature interaction, while explicit frequency-domain representation learning and training optimization remain insufficiently explored. In contrast, the proposed FIRGAN incorporates a Statistical Frequency Feature Module (SFFM) to explicitly enhance informative frequency components using local statistical priors. Furthermore, a CNN–Transformer Dual-branch Fusion Network, together with an IQA-guided optimization strategy, jointly improves local–global feature interaction, frequency representation capability, and optimization efficiency, leading to more faithful infrared–visible image fusion and robustness for downstream visual perception.

2.3. Frequency-Domain Representation Learning

Frequency-domain representation learning has attracted increasing attention because frequency components provide complementary information to spatial features, including textures, edges, and structural patterns. Recent studies, such as FEDFormer [12], FFTFormer [11], and FreqFusion [15], incorporate Fourier-based representations to enhance feature interactions and improve reconstruction quality. These methods demonstrate the effectiveness of frequency-domain information in capturing global structures and fine details. However, most existing approaches rely on fixed spectral operations or directly manipulate frequency coefficients without considering local image characteristics. Consequently, they cannot adaptively identify informative frequency components under different image contents. To address this issue, we propose SFFM, which leverages MSCN-derived statistical priors to generate adaptive frequency weights for content-aware spectrum enhancement and robust spatial-frequency representation learning.

2.4. Task-Driven Fusion and Quality-Aware Optimization

Recent infrared–visible image fusion research has increasingly shifted from visual quality enhancement to improving the applicability of fused images for downstream detection. Representative detection-driven methods, including DetFusion [16], TDFuse [17], and TarDAL [18], jointly optimize the fusion network and object detector by introducing detection supervision into the training process. DetFusion improves target discriminability through detection-aware learning, TDFuse combines detail and semantic feature injection to balance visual fidelity and detection performance, and TarDAL directly formulates image fusion as a task-driven learning problem. Although these methods improve detection accuracy, their optimization objectives are tightly coupled with specific detectors and task-specific losses, limiting their generalization across different downstream applications. In contrast, FIR-GAN follows an independently trained image-fusion paradigm. Its optimization does not involve a detector or detection-related loss, and object detection is used only as an external downstream evaluation. In addition, existing quality-aware methods generally employ IQA as an auxiliary supervision signal while treating all training samples equally [19,20]. AQFS-Net [21], which introduces adaptive quality-aware fusion and foreground-aware saliency guidance into a visible–infrared object detection network. AQFS-Net performs feature-level multi-modal fusion within a detector and is directly optimized using detection-related supervision. Different from these approaches, FIRGAN introduces an IQA-guided optimization strategy that performs offline quality-aware patch ranking and sample selection, together with the proposed SFFM for adaptive frequency-domain enhancement. Moreover, downstream object detection is used only as an independent evaluation protocol under identical detector settings, providing an objective assessment of the practical effectiveness of the generated fusion images while preserving the generality of the proposed framework.

3. Method

3.1. Overall Framework

As shown in Figure 2, FIRGAN is a infrared–visible image fusion framework for downstream detection applications that integrates an IQA-guided optimization strategy, an SFFM, and a Dual-branch Fusion Network. First, the IQA-guided indexing module constructs quality-aware training samples to improve optimization efficiency. The selected infrared and visible image pairs are then processed by SFFM, which jointly models spatial statistics and frequency-domain information to enhance discriminative feature representation. Subsequently, a Dual-branch Fusion Network consisting of a CNN Fuse Net and a Transformer Fuse Net captures complementary local textures and global semantic dependencies for effective cross-modal fusion. Finally, adversarial learning is employed to improve visual realism and structural consistency, generating high-fidelity fused images for downstream object detection tasks.

Figure 2. Overall architecture of FIRGAN. The proposed framework consists of an IQA-guided optimization strategy, a Dual-Branch Generator Network, and a discriminator. The IQA-guided module constructs quality-aware indices for efficient training, while the generator integrates statistical frequency representation and CNN–Transformer fusion to produce high-fidelity fused images. Adversarial learning further enhances visual realism and structural consistency.

3.2. IQA-Guided Optimization Strategy

As shown in Figure 3, the proposed IQA-guided optimization strategy is designed to reduce redundant quality evaluation during training while improving the informativeness of training samples. Unlike conventional IQA-assisted fusion methods that repeatedly invoke quality assessment networks in each iteration, the proposed strategy decouples quality estimation from network optimization. It first evaluates local patch quality, then constructs reusable quality-aware indices, and finally uses these indices to guide generator training. The pseudocode is given in Algorithm A1 in Appendix A.

Figure 3. IQA-guided optimization strategy. The proposed method first evaluates patch-level image quality to construct a quality-aware index table. The generated indices are subsequently used for patch ranking, sample reorganization, and index-guided training, significantly reducing redundant IQA computation while improving optimization efficiency and fusion quality.

Quality-aware Patch Evaluation Given an infrared image $I_{ir}$ and a visible image $I_{vis}$, the paired inputs are first divided into N corresponding patch pairs:

$$P={(P_{ir}^i,P_{vis}^i)}_{i=1}^N$$

(1)

Each patch pair represents a local multi-modal region with specific structural, textural, and semantic characteristics. Since different regions contribute unequally to fusion learning, treating all patches with the same importance may lead to inefficient optimization. For example, target regions, strong edges, and texture-rich areas usually provide more useful information than smooth background regions.

To measure the perceptual and structural importance of each patch, an IQA model is used to compute a quality score:

$$Q_i=\mathrm{IQA}(P_{ir}^i,P_{vis}^i),$$

(2)

where $Q_i$ denotes the quality response of the i-th patch pair. The score reflects local information richness, structural clarity, and perceptual reliability. In practice, quality-related indicators such as BRISQUE [19], SSIM [22], entropy [23], or their weighted combination can be adopted to characterize patch quality. Thus, the final quality score is denoted as $S_i$, and the calculation procedure is shown in Algorithm A1 of Appendix A.

The quality-aware evaluation has two purposes. First, it identifies informative patches that contain salient targets, clear boundaries, or rich textures. Second, it suppresses the influence of low-quality patches, such as blurred, weak-texture, over-exposed, or redundant background regions. Therefore, the fusion model can focus more on regions that are important for both visual reconstruction and downstream detection.

Index Construction

After obtaining the quality scores of all patch pairs, the patches are ranked according to their quality values:

$$R=\mathrm{Sort}(Q_1,Q_2,\dots ,Q_N)$$

(3)

where R is the set of $r_i$ for $i=1,2,\dots ,N$. For an aligned infrared–visible image pair of size $(H\times W)$, the number of paired patches is determined:

$$N=(\lfloor {\displaystyle \frac{H-p}{s}}\rfloor +1)(\lfloor {\displaystyle \frac{W-p}{s}}\rfloor +1)$$

(4)

where $\left(p\right)$ and $\left(s\right)$ denote the patch size and stride, respectively.

Based on the ranking results, an index table is constructed:

$$I={(i,Q_i,r_i)}_{i=1}^N$$

(5)

where i is the original patch position, $Q_i$ is the quality score, and $r_i$ denotes the ranking order. This index table records the relationship between spatial position, perceptual quality, and training priority.

The index construction process converts dynamic IQA evaluation into a static lookup mechanism. Instead of recalculating quality scores in every training iteration, the model directly retrieves patch pairs according to the precomputed index table. This greatly reduces computational cost and avoids instability caused by repeatedly invoking IQA networks during adversarial training.

Furthermore, the index table can support two sampling modes. In deterministic mode, high-quality patches are preferentially selected according to their ranking order. This helps the network learn discriminative structures more effectively. The resulting quality-aware index sequence is precomputed offline and reused throughout training. In randomized mode, patches are shuffled under quality constraints, which introduces data diversity and reduces the risk of overfitting to fixed patch sequences. The randomized mode is retained as an optional implementation and was not used to produce the reported results. The sampling rate is set to $100\%$, namely $\rho =1.0$ and $M=N$. All N patch indices are sampled without replacement using a fixed random seed of 42. Consequently, every candidate patch is used exactly once in each reordered sequence, and the randomized mode changes only the quality-aware presentation order rather than discarding patches or reducing the number of training samples. The R-index table supports two modes, as elaborated in Algorithm A1 in Appendix A. Thus, the index construction not only improves efficiency but also serves as a quality-aware data reorganization strategy.

In deterministic mode, all candidate patches are stably sorted in descending order according to their final quality scores. When two patches have the same score, their original indices are used as the secondary sorting criterion. The resulting quality-aware index sequence is precomputed offline and reused throughout training. The randomized mode is retained as an optional implementation and was not used to produce the reported results. When this mode is enabled, the sampling probability is defined as set out in Appendix A.

Index-guided Training

During training, the generator does not receive all patches uniformly. Instead, training samples are selected or reordered based on the constructed quality-aware index table. Given the index table I, the input patch sequence is reorganized as

$$\tilde{P}=\mathrm{Reorder}(P,I)$$

(6)

The reordered patch sequence is then fed into the generator for feature extraction and fusion. Since high-quality and information-rich patches appear with higher priority or higher probability, the generator receives stronger supervision from structurally meaningful regions. This improves the learning of target saliency, edge preservation, and texture reconstruction.

The proposed index-guided training also benefits adversarial optimization. In conventional generative adversarial network (GAN)-based fusion, the generator and discriminator may be affected by unstable sample distributions, especially when low-quality or uninformative patches dominate the mini-batch. By introducing quality-aware indexing, the training distribution becomes more informative and balanced, which helps stabilize generator–discriminator optimization.

Overall, the proposed IQA-guided optimization strategy provides a bridge between perceptual quality assessment and efficient fusion learning. It reduces repeated IQA computation, prioritizes informative regions, and improves training stability. As a result, the framework achieves faster convergence and better fusion quality, as verified by the complexity analysis where the training time is reduced from 125 h to 3.85 h while multiple image quality metrics are simultaneously improved.

3.3. Generator Network

The generator network aims to produce a high-fidelity fused image by jointly exploiting modality-specific frequency characteristics and complementary spatial representations. As illustrated in Figure 4, the proposed generator consists of two major components: an SFFM and a Dual-branch Fusion Network.

Figure 4. Architecture of the proposed generator network. Infrared and visible features are first extracted by the SFFM. The resulting representations are subsequently fused through a Dual-Branch Network consisting of a CNN Fuse Net and a Transformer Fuse Net, followed by a fusion decoder to generate the final fused image.

Given infrared and visible images $I_{ir}$ and $I_{vi}$, modality-specific features are first extracted by the backbone network:

$$F_{ir}=E_{ir}\left(I_{ir}\right),F_{vi}=E_{vi}\left(I_{vi}\right),$$

(7)

where $E_{ir}$ and $E_{vi}$ denote the infrared and visible feature extractors, respectively.

Subsequently, the extracted features are enhanced by SFFM to exploit informative frequency components and suppress redundant responses. The enhanced features are then forwarded to the Dual-branch Fusion Network, where local texture information and global semantic dependencies are collaboratively modeled through CNN and Transformer branches. Finally, the fused representation is reconstructed to generate the output image.

3.3.1. Statistical Frequency Feature Module—SFFM

Although convolutional features contain abundant spatial information, they often lack explicit modeling of frequency-domain characteristics. To address this limitation, we introduce the SFFM, which utilizes local statistical priors to generate adaptive frequency weights for spectrum enhancement in Figure 5.

Figure 5. Architecture of the proposed SFFM. The module utilizes MSCN-derived statistical priors to generate adaptive frequency weights for frequency spectrum enhancement, followed by inverse frequency reconstruction and residual feature refinement.

Statistical Prior Extraction

Given an input feature map, $X\in {\mathbb{R}}^{H\times W\times C}$ local statistical information is first extracted using mean subtracted contrast normalization (MSCN):

$${\widehat{X}}_i={\displaystyle \frac{X_i-{\mu }_i}{{\sigma }_i+C}}$$

(8)

where ${\mu }_i$ and ${\sigma }_i$ denote the local mean and standard deviation obtained by Gaussian filtering, and C is a stabilization constant, where X is the set of $X_i$ for $i=1,2,\dots ,N$.

Unlike conventional IQA applications that directly use MSCN coefficients, we employ the local statistical responses as structural priors for frequency modeling. These statistics provide valuable cues regarding local contrast variation, texture activity, and structural saliency.

Adaptive Frequency Weight Generation

To identify statistically informative spatial responses while maintaining dimensional compatibility with the subsequent frequency-domain representation, a Top-K operation is applied to the MSCN-derived statistical feature map. Given an input feature map $X\in {\mathbb{R}}^{H\times W\times C}$, the rFFT2 operation produces a one-sided frequency representation $F\in {\mathbb{C}}^{H\times W_{\mathrm{rFFT}}\times C}$, and K is dynamically defined as $K=W_{\mathrm{rFFT}}=⌊{\displaystyle \frac{W}{2}}⌋+1$.

The Top-K operation is performed along the width dimension ($\dim =2$) of the MSCN-derived statistical map:

$$I_K={\mathrm{TopKIndex}}_{\dim =2}\left(\widehat{X},K\right),$$

(9)

where $I_K\in {\mathbb{R}}^{H\times K\times C}$ contains the position indices associated with the K largest statistical responses.

The selected responses are then processed by a lightweight weight generator consisting of stacked convolution, group normalization, and activation layers:

$$W_f=\mathsf{\Phi }\left(I_K\right),$$

(10)

where $W_f$ denotes the generated frequency weight.

Compared with fixed frequency filtering strategies, the proposed mechanism dynamically adapts frequency emphasis according to local image characteristics.

Frequency-domain Enhancement

Meanwhile, the original feature map is transformed into the frequency domain using a two-dimensional Fourier transform:

$$F=\mathcal{F}\left(LN\right(X\left)\right).$$

(11)

where $\mathcal{F}(·)$ denotes the fast Fourier transform (FFT) function.

The generated frequency weight is used to recalibrate the spectrum:

$$F^{\prime }=W_f\odot F,$$

(12)

where ⊙ denotes element-wise multiplication.

Through this operation, informative frequency components are selectively amplified while redundant responses are suppressed.

The enhanced spectrum is subsequently transformed back into the spatial domain:

$$X_f={\mathcal{F}}^{-1}\left(F^{\prime }\right),$$

(13)

where ${\mathcal{F}}^{-1}(·)$ denotes the inverse fast Fourier transform (IFFT) function.

A lightweight refinement layer consisting of LayerNorm and MLP is further adopted to stabilize feature distribution and improve representation quality:

$$X_r=\mathrm{MLP}\left(\mathrm{LN}\left(X_f\right)\right).$$

(14)

Adaptive spectral weighting may change the scale and variance of reconstructed responses after IFFT. LayerNorm therefore normalizes the refined spatial feature before projection, reducing channel-wise distribution shifts and isolated response peaks. The MLP applies nonlinear channel mixing to re-encode useful frequency-enhanced patterns and suppress unstable responses. The residual path adds the original feature X, preserving low-frequency structure and providing a direct gradient route. Consequently, the refinement layer stabilizes optimization without discarding the input information required for faithful reconstruction. Finally, a residual connection is introduced to preserve the original feature information:

$$F_{out}=X_r+X,$$

(15)

where X represents the input feature and $F_{out}$ denotes the output of the SFFM. This design enables the module to effectively integrate frequency-enhanced representations with the original spatial features, thereby improving both structural fidelity and feature discriminability.

Consequently, SFFM establishes an effective bridge between spatial statistical priors and frequency-domain representation learning, enabling adaptive spectrum enhancement without introducing significant computational overhead. The proposed SFFM leverages MSCN-derived statistical priors to generate adaptive frequency weights, enabling informative spectrum enhancement and robust spatial-frequency representation learning.

3.3.2. Dual-Branch Fusion Structure

Given the enhanced infrared and visible features ($F_{ir}$) and ($F_{vi}$), a feature aggregation operation is first performed to construct a unified multi-modal representation. Specifically, the two modality-specific features are concatenated along the channel dimension and projected into a common feature space:

$$F_0={\varphi }_p\left(\mathrm{Concat}(F_{ir},F_{vi})\right),$$

(16)

where $\mathrm{Concat}(·)$ denotes channel-wise concatenation and ${\varphi }_p(·)$ represents a linear projection layer used to reduce feature redundancy and align modality distributions.

The resulting feature $F_0$ serves as the shared input of the Dual-branch Fusion Network and is simultaneously forwarded to two parallel branches. The CNN Fuse Branch focuses on local texture preservation and fine-grained structural modeling, whereas the Transformer Fuse Branch captures long-range dependencies and global semantic relationships. By jointly exploiting local and global representations, the proposed architecture achieves more effective multi-modal feature fusion.

CNN Fuse Net

The CNN Fuse Net is designed to model fine-grained local structures and preserve modality-specific spatial details. Given the aggregated feature $F_0$, a series of convolutional blocks are employed to extract local representations:

$$F_{cnn}={\varphi }_{cnn}\left(F_0\right),$$

(17)

where ${\varphi }_{cnn}(·)$ denotes the CNN-based fusion operator.

Benefiting from localized receptive fields and hierarchical feature aggregation, the CNN branch effectively captures texture patterns, edge information, local contrast variations, and object boundaries. Moreover, convolutional operations naturally preserve spatial continuity and structural consistency, making this branch particularly suitable for maintaining visible texture details and infrared target contours. The resulting feature $F_{cnn}$ provides a high-fidelity local representation that complements the global semantic features learned by the Transformer branch.

Transformer Fuse Net

The Transformer Fuse Net is introduced to capture long-range dependencies and global contextual relationships that are difficult to model using convolutional operations alone. Given the aggregated feature $F_0$, a sequence of Transformer blocks is employed to learn global feature representations:

$$F_{trans}={\varphi }_{trans}\left(F_0\right),$$

(18)

where ${\varphi }_{trans}(·)$ denotes the Transformer-based fusion operator.

Benefiting from the self-attention mechanism, the Transformer branch establishes interactions among spatially distant regions and enables more effective cross-modal information aggregation. This global modeling capability improves semantic consistency and enhances the representation of salient targets and contextual structures. Consequently, the Transformer Fuse Net provides complementary global features that effectively supplement the local representations extracted by the CNN branch.

Fuse Block

The Fuse Block is designed to integrate the complementary representations generated by the CNN and Transformer branches. Given the local feature $F_{cnn}$ and global feature $F_{trans}$, a cross-modal interaction operation is first performed to facilitate information exchange between local structural details and global contextual semantics:

$$F_f={\varphi }_{fuse}(F_{cnn},F_{trans}),$$

(19)

where ${\varphi }_{fuse}(·)$ denotes the feature fusion operator.

Through adaptive feature aggregation, the Fuse Block effectively combines texture details, edge information, and semantic context into a unified representation. The resulting feature preserves both local fidelity and global consistency, providing a robust foundation for the subsequent fusion decoder.

3.4. Training Procedure

The overall training procedure of FIRGAN is summarized in Algorithm A2. First, an IQA-guided indexing strategy is employed to evaluate local patch quality and construct an offline index set. The generated index is used to prioritize informative training samples during optimization. For each selected infrared–visible image pair, modality-specific features are extracted independently and enhanced by the proposed SFFM. The enhanced features are then concatenated and fed into the CNN–Transformer Dual-branch Fusion Network to jointly capture local structural details and global contextual dependencies. The fused representation is subsequently decoded to generate the final fusion image. During training, the discriminator [24] is optimized to distinguish fused images from real samples, while the generator is updated using content-preservation, structural-consistency, and adversarial losses. Through alternating optimization, FIRGAN learns to generate visually informative and structurally consistent fusion results.

4. Results

This section presents the datasets and experimental configurations, followed by a comparative evaluation of the proposed FIRGAN against six representative deep learning-based image fusion approaches, including the recently published SPDFusion method. Furthermore, the effectiveness of FIRGAN is quantitatively assessed by investigating its influence on downstream object detection performance.

4.1. Datasets and Implementation Details

4.1.1. Dataset

In this study, multiple objective evaluation metrics were adopted to comprehensively assess fusion performance, including DeltaE [25] for color difference, AG [26] for average gradient, EN [23] for information entropy, QAB/F [27] for edge information preservation, VIF [28] for visual information fidelity, SD [29] for standard deviation, and SF [26] for spatial frequency.

The DroneVehicle dataset offers unique advantages in terms of scene diversity, multi-modal complexity, and fine-grained annotations, enabling a thorough and challenging evaluation of multi-modal image fusion methods from the perspectives of practical relevance, methodological robustness, and evaluation completeness. Accordingly, experiments were conducted on both the widely used TNO dataset and the multi-modal detection dataset DroneVehicle, accompanied by extensive quantitative and qualitative analyses.

TNO (https://figshare.com/articles/TNO_Image_Fusion_Dataset/1008029 (accessed on 10 September 2026)). The TNO dataset is a widely adopted public benchmark for infrared and visible image fusion, comprising multispectral nighttime images collected from a variety of military-related scenes. Following common practice in deep learning-based image fusion, we selected 42 image pairs for evaluation.

DroneVehicle (https://github.com/VisDrone/DroneVehicle (accessed on 10 September 2026)). Proposed by Ding et al., the DroneVehicle dataset consists of well-aligned infrared and visible image pairs for vehicle detection. It contains 28,439 RGB–infrared image pairs captured across diverse environments, including urban roads, residential areas, parking lots, and other scenarios under both daytime and nighttime conditions. As such, it serves as a valuable benchmark for comparing the fusion performance of different methods. For the DroneVehicle dataset, 30 image pairs are used to evaluation the proposed fusion network, while 5000 image pairs are reserved for downstream detection evaluation.

4.1.2. Implementation Details

All experiments were carried out on a Windows 10 platform. The software environment included PyTorch version 1.11.0, CUDA 11.3.1, and Python version 3.9.16. The experiments were executed on a workstation equipped with an Intel(R) Core(TM) i9-10920X CPU and an NVIDIA GeForce RTX 3090 GPU with 24 GB of memory. Both model training and testing were conducted on the same hardware platform. The training process was performed for a total of 100 epochs, using an initial learning rate of 0.0001. An Adam optimizer with weight decay was adopted to minimize the average logistic regression loss, with the weight decay factor set to 0.0001. All experimental results reported in the manuscript were obtained using the deterministic mode. For the IQA-guided optimization strategy, the BRISQUE, SSIM, and entropy scores are normalized to $[0, 1]$ within each candidate patch set. Since a lower raw BRISQUE score indicates better perceptual quality, it is inversely normalized to the same range. For this, $\alpha =\beta =0.5$, $a=b=0.5$.

4.2. Main Results

To demonstrate the usefulness of our work, we used seven representative deep learning-based methods on the TNO dataset and including DenseFuse [4], FusionGAN [5], PIAFusion [30], SeAFusion [31], ATGAN [24], Tardal [18], SPDFusion [32] and Ours (the proposed method), and compared the experimental results of FusionGAN, Diffusion [33], Tarsal, ATGAN, SPDFusion and Ours (the proposed method) on Drone Vehicle.

4.2.1. Results on TNO

Quantitative Analysis

High-quality image fusion aims to preserve detailed texture, structural information, and luminance characteristics from the source images. These properties are quantitatively assessed using metrics such as visual information fidelity (VIF) and standard deviation (SD), as summarized in Table 1.

Table 1. Comparison of performance metrics for different methods on the TNO dataset for infrared–visible image fusion tasks.

	SD	MI	VIF	AG	EN	${\mathit{Q}}_{\mathit{f}}^{\mathit{ab}}$	SF
Densefuse	34.825	2.301	0.658	3.56	6.819	0.446	8.985
FusionGAN	30.663	2.335	0.422	2.421	6.558	0.234	6.275
PIAFusion	37.141	3.357	0.74	3.828	6.814	0.529	9.62
SeAFusion	44.243	2.838	0.704	4.98	7.133	0.487	12.253
ATGAN	43.822	1.819	0.743	4.891	6.357	0.285	12.302
Tardal	45.506	4.269	0.818	4.657	6.658	0.405	31.836
SPDFusion	43.065	2.688	0.701	5.23	7.11	0.502	13.009
Ours	45.175	2.132	1.098	5.873	7.167	0.307	13.677

Bold values indicate the best result, and italic values represent the second-best result.

The VIF metric evaluates the extent to which visual information from the source images is retained in the fused image. A higher VIF value indicates better perceptual fidelity. As shown in Table 1, the proposed method achieves a VIF of 1.098, which is substantially higher than those obtained by existing methods, including Tardal (0.818) and PIAFusion (0.740). This result highlights the ability of the proposed approach to preserve perceptually important information and accurately convey fine details from multiple modalities.

The SD metric measures the dispersion of pixel intensities, with larger values corresponding to higher contrast and richer texture representation. Tardal (45.506), Ours (45.175), and SeAFusion (44.243) achieve relatively high SD values, indicating that these methods are effective in maintaining contrast and dynamic range while preserving abundant structural details.

Effective infrared–visible image fusion further requires robust feature alignment that balances cross-modal semantic consistency and feature distribution coordination. This property can be evaluated using edge-related metrics such as average gradient (AG) and spatial frequency (SF), as reported in Table 1.

AG reflects edge sharpness, whereas SF characterizes the activity level of spatial details. Both metrics are closely related to edge preservation and spatial feature alignment. The proposed method achieves the highest AG value (5.873) among all compared methods, demonstrating its strong capability in preserving sharp edges and fine details. Regarding SF, TarDAL achieves the highest value of 31.836, while the proposed method obtains the second-highest value of 13.677, slightly exceeding SPDFusion (13.009) and the other compared methods. These results suggest that the proposed method enhances texture details while effectively avoiding artifacts or noise amplification caused by excessive enhancement.

The entropy (EN) metric evaluates the information content and overall fidelity of the fused image. As shown in Table 1, the proposed method achieves the highest EN value (7.167), surpassing PIAFusion (6.814) and ATGAN (6.357). This indicates that the fused images generated by the proposed approach exhibit more natural brightness and contrast, as well as improved global information preservation and scene reconstruction quality.

Nevertheless, FIR-GAN does not attain the leading MI or $Q_{abf}$ values. This result reflects the fact that different fusion metrics emphasize different properties and that their corresponding performances cannot be improved simultaneously. In particular, the proposed quality-aware and frequency-adaptive optimization favors perceptual fidelity, informative structural details, and controlled frequency enhancement rather than maximizing intensity dispersion or edge-energy transfer alone. Therefore, the results represent a favorable but non-uniform trade-off among perceptual fidelity, information content, edge preservation, and spatial-frequency characteristics, rather than universal superiority under every metric.

In summary, by combining statistically driven frequency-domain attention with Transformer-based spatial-domain fusion, the proposed method effectively bridges the gap between frequency and spatial feature interactions. This dual-domain collaborative framework provides a novel paradigm for multi-modal image fusion and achieves competitive performance across multiple quantitative metrics confirms its advantages in detail preservation and visual consistency, offering valuable guidance for the design of future fusion architectures.

Figure 6 presents qualitative comparison results on the TNO dataset for infrared–visible image fusion, including the infrared and visible source images, as well as fusion results produced by DenseFuse, FusionGAN, PIAFusion, SeAFusion, ATGAN, Tardal, SPDFusion, and the proposed method.

Figure 6. Quantitative Quantitative metric trends of different image fusion algorithms.

Qualitative Analysis

As illustrated in Figure 7, existing comparison methods are able to fuse information from infrared and visible modalities to a certain degree; however, they often exhibit limitations such as low contrast, blurred details, or prominent noise artifacts.

Figure 7. Visual fused images generated by different image fusion algorithms.

In the smoke scene (second row of Figure 7), the proposed method effectively highlights the thermal flame targets from the infrared modality while simultaneously preserving fine-grained tree textures from the visible modality. Compared with ATGAN, which suffers from noticeable texture blurring, Ours produces sharper and more complete details. In the outdoor scene shown in the eighth row, Ours achieves competitive performance compared with Tardal in retaining building structures and vegetation textures, as Tardal presents local texture distortions, demonstrating the multi-scene detail preservation capability of the proposed approach.

Infrared–visible image fusion further requires coherent feature alignment, encompassing both semantic consistency and spatial coordination across modalities. In the human target scene (third row), Ours establishes a more natural semantic relationship between infrared human silhouettes and the visible background, whereas ATGAN shows a clear separation between target and background features. This indicates stronger feature alignment at the semantic “target–scene” level. Similarly, in the indoor human scene (sixth row), Ours achieves more harmonious cross-modal fusion of human motion and building structures, highlighting its advantage in maintaining semantic consistency. In the vegetation and target scene (fourth row), Ours provides more precise spatial alignment and edge consistency between infrared small targets and visible vegetation. In contrast, Tardal exhibits evident spatial distortions, including misalignment between vegetation textures and target contours, underscoring the geometric alignment capability of the proposed method.

In summary, through the targeted design of the statistically driven frequency-domain attention module (SFFM) and the Transformer Fuse Net, the proposed method achieves notable improvements in both visual fidelity and feature alignment. From a visual fidelity perspective, Ours preserves fine details and luminance information across diverse scenes. From a feature alignment perspective, it demonstrates stronger cross-modal consistency at both semantic and spatial levels. This effectively addresses the trade-off between high-fidelity reconstruction and precise feature alignment in infrared–visible image fusion. By producing clearer, more accurate, and visually natural fused images, the proposed method enhances target recognition accuracy and efficiency, and provides high-quality data for subsequent image analysis and vision-based applications.

4.2.2. Results on DroneVehicle

Quantitative Analysis

Table 2 summarizes the quantitative performance of FusionGAN, Diff (Diffusion), Tardal, SPDFusion, and Ours (the proposed method) on the infrared–visible image fusion task, evaluated using seven commonly adopted metrics, namely standard deviation (SD), DeltaE, visual information fidelity (VIF), average gradient (AG), entropy (EN), $Q_f^{ab}$, and spatial frequency (SF). The experimental results are analyzed in detail below. Visual Fidelity Evaluation: High-quality fusion requires the fused image to preserve color consistency, fine textures, and luminance structures from the source images. These characteristics can be effectively assessed using DeltaE, VIF, and SD, as reported in Table 2. DeltaE measures the color difference between the fused image and the source images, with lower values indicating better color fidelity. The proposed method achieves the lowest DeltaE value of 7.405, substantially outperforming Tardal (8.536) and FusionGAN (11.145). This result indicates that Ours maintains superior color consistency and effectively preserves color information across modalities. To further demonstrate this advantage, visual comparisons of DeltaE maps on the DroneVehicle dataset are provided in Figure 8.

Table 2. Comparison of performance metrics for different methods on the Drone Vehicle dataset in the infrared–visible image fusion task.

	SD	DeltaE	VIF	AG	EN	${\mathit{Q}}_{\mathit{f}}^{\mathit{ab}}$	SF
FusionGAN	72.057	11.145	0.557	2.463	4.775	0.421	12.921
Diffusion	61.133	17.951	0.636	3.584	5.029	0.525	14.797
ATGAN	76.721	13.654	0.783	4.879	5.186	0.541	18.057
Tardal	90.274	8.536	0.496	2.992	4.749	0.402	15.075
SPDFusion	66.180	13.477	0.556	4.661	5.291	0.533	18.588
Ours	74.859	7.405	0.880	5.377	5.526	0.465	19.578

Bold values indicate the best result, and italic values represent the second-best result.

Figure 8. DeltaE chromatic aberration image comparison in the Drone Vehicle dataset.

As observed from the DeltaE comparisons, Ours produces the most natural color gradations and achieves accurate target–background color separation, with smooth transitions that are highly consistent with the source images. This highlights the effectiveness of the proposed method in multi-modal color feature extraction and alignment. In contrast, FusionGAN exhibits disordered color distributions, Diffusion and ATGAN tend to generate over-saturated colors, Tardal suffers from noticeable color bias, and SPDFusion introduces local color artifacts. These limitations, arising from color fusion distortion or cross-modal misalignment, lead to degraded DeltaE performance and further validate the effectiveness of the proposed color-domain design.

It can also be observed from Table 2 that the DeltaE values of different methods vary significantly across data samples, indicating that color fidelity remains sensitive to scene content. Nevertheless, the proposed method demonstrates overall stability and achieves competitive performance.

In terms of visual information fidelity, Ours achieves the highest VIF score of 0.880, markedly higher than Diff (0.636) and SPDFusion (0.556), indicating its strong capability to preserve visual details on the DroneVehicle dataset. SD reflects the dispersion of gray-level values, where higher values correspond to richer detail representation. However, its SD value of 74.859 is lower than those of TarDAL and ATGAN, while its $Q_{abf}$ value of 0.465 is also not the highest among the compared methods. These differences reveal an inherent trade-off among fusion-quality criteria. SD primarily characterizes global intensity dispersion, whereas $Q_{abf}$ emphasizes the transfer of edge information from the source images.

Feature Alignment Evaluation: Infrared–visible image fusion further requires effective feature alignment to ensure semantic consistency and spatial coherence across modalities. This can be evaluated using AG and SF.

The proposed method achieves the highest AG (5.377) and SF (19.578) among all compared methods, indicating superior edge preservation and spatial detail representation. These results suggest that Ours produces more coherent spatial feature distributions and achieves stronger cross-modal semantic alignment in UAV–vehicle scenes, such as improved correspondence between vehicle contours and surrounding textures.

In summary, the proposed method demonstrates outstanding performance across multiple key metrics, particularly in visual fidelity, image sharpness, information richness, and spatial detail representation, with especially strong performance in color fidelity. Optimizing perceptual and structural consistency may therefore not simultaneously maximize every contrast- or edge-oriented metric. Overall, FIR-GAN achieves competitive multi-metric performance, with particular strengths in perceptual fidelity, color consistency, information preservation, and spatial-detail representation.

Figure 9 further illustrates the performance trends of FusionGAN, Tardal, Diff, and Ours across the seven evaluation metrics. Overall, Ours obtains the best or second-best results on key indicators such as DeltaE, VIF, AG, EN, and SF. Although performance fluctuations are observed across different samples, the proposed method maintains stability and robustness, further validating its effectiveness in infrared–visible image fusion tasks.

Figure 9. Quantitative metric trends of different algorithms on the DroneVehicle dataset.

In summary, our method demonstrates significant advantages in key metrics such as DeltaE, VIF, AG, EN, and SF, generating fused images with good visual information preservation, high clarity, rich information content, and excellent spatial detail. Although there are some fluctuations in different metrics and data points, the overall stability and competitive performance of our method prove its effectiveness in infrared–visible image fusion tasks.

Qualitative Analysis

Figure 10 presents qualitative comparisons of infrared–visible image fusion results on the DroneVehicle dataset, including the Ir Source, Vis Source, and fusion results produced by FusionGAN, Diffusion, Tardal, and Ours (the proposed method).

Figure 10. Comparison of results of different methods on the Drone Vehicle dataset for infrared–visible image fusion task.

From the visual results, FusionGAN is able to merge information from both modalities to a limited extent; however, the overall contrast remains relatively low. The visible-light textures are insufficiently preserved, leading to blurred structural details. In several scenes, such as parking lots and road intersections, the contours of vehicles are not clearly distinguishable from the background, which weakens target saliency. Diffusion-based fusion exhibits moderate performance in terms of information integration, but noticeable over-smoothing effects can be observed. Fine-grained details, including road markings, vehicle edges, and scene textures, are partially suppressed. As a result, the fused images appear visually smooth but lack sharpness, reducing the clarity of both targets and background structures. Tardal can enhance infrared targets in some scenarios; however, it suffers from evident color distortion and local texture degradation. In nighttime scenes, illumination appears unnatural, and inconsistencies between visible textures and infrared targets are observed. Moreover, spatial misalignment artifacts occasionally occur, resulting in blurred edges and reduced geometric consistency.

In contrast, the proposed method demonstrates superior visual performance across diverse scenes. It effectively preserves the detailed textures and color information from the visible modality while simultaneously enhancing salient infrared targets. Vehicle contours, lane markings, and environmental structures remain clear and well-defined, and the target–background contrast is significantly improved. Furthermore, the fused images produced by the proposed method exhibit consistent color representation and sharp edges, indicating effective cross-modal feature alignment and robust detail preservation. These qualitative results confirm that the proposed method achieves a better balance between visual fidelity and feature alignment, producing fusion outputs that are more natural, informative, and suitable for downstream perception tasks.

To comprehensively evaluate the robustness and generalization capability of the proposed fusion framework, qualitative experiments are conducted under diverse real-world scenarios with varying illumination conditions, target densities, and background complexities. As shown in Figure 11, the selected examples include extremely low-illumination environments, urban traffic scenes, dense parking areas, strong nighttime illumination interference, and weak-texture nighttime conditions. These challenging scenarios are specifically chosen to verify whether the proposed method can simultaneously preserve thermal target saliency, structural details, and perceptual consistency under heterogeneous multi-modal inputs. Furthermore, the visualization analysis aims to demonstrate the effectiveness of the proposed framework in maintaining discriminative representations for downstream detection tasks, particularly in complex environments where conventional fusion methods often suffer from structural degradation, contrast attenuation, or detail loss.

Figure 11. Comparison of results of different methods on the Drone Vehicle dataset for infrared–visible image fusion task. (a) The extremely low-illumination scenario, (b) the urban traffic scene, (c) the dense parking scenario, (d) the highly crowded target scene, (e) the nighttime scene with strong illumination interference, (f) the weak-texture nighttime scenario.

In the extremely low-illumination scenario shown in Figure 11a, the visible image suffers from severe information degradation, while the infrared modality only preserves coarse thermal responses. In contrast, the proposed method effectively restores structural contours of buildings and parking areas while retaining thermal target saliency, resulting in significantly improved scene interpretability. For urban traffic scenes in Figure 11b, the proposed method successfully integrates infrared target prominence with visible-texture details, producing clearer vehicle boundaries and road structures without introducing noticeable contrast attenuation. This indicates stronger cross-modal feature complementarity. As illustrated in Figure 11c,d, under dense parking scenarios with highly crowded targets, the proposed method preserves inter-object boundaries and spatial distribution patterns more effectively. Compared with conventional fusion strategies, the fused results exhibit enhanced local contrast and more discriminative structural representations, which are beneficial for downstream object detection and localization. In the nighttime scene with strong illumination interference shown in Figure 11e, the proposed method suppresses over-exposure diffusion caused by bright light sources while maintaining surrounding target details. This demonstrates its capability in dynamic range coordination and illumination-aware fusion. Finally, in the weak-texture nighttime scenario shown in Figure 11f, where the visible modality provides extremely limited information, the proposed method still preserves meaningful target distributions and scene structures by leveraging complementary infrared responses. The fused image exhibits improved target visibility and spatial perception, validating the robustness and generalization ability of the proposed framework under challenging nighttime environments.

4.3. Fusion Target Detection

To demonstrate the adaptability of the FIRGAN fusion model to downstream tasks, this paper performs fine-grained detection on 5000 pairs of infrared and visible-light image pairs from the DroneVehicle dataset. The fine-grained detection experiment used the yolov5 [34] detection model in conjunction with the OBB module [35] to perform rotationally oriented fine-grained detection. For detection evaluation, the YOLOv5-OBB detector is trained from scratch using only the visible images of the DroneVehicle training set. To ensure a fair comparison, all fusion methods employ the same detector, identical preprocessing pipeline, and the same input image resolution. The experimental results are shown in Figure 12. To comprehensively assess the effectiveness of the proposed fusion strategy in infrared–visible object detection, extensive quantitative evaluations were conducted. The comparative results presented in Table 3 demonstrate that the proposed method consistently outperforms all baseline approaches by a substantial margin. Specifically, the proposed model achieves an mAP@0.5 of 76.1% and an mAP@0.5:0.95 of 57.1%, surpassing the best-performing competing method by 10.3% and 9.7%, respectively.

Figure 12. Detection radar performance of different fusion algorithms on the DroneVehicle dataset.

Table 3. Comparison of dectection performance metrics for different methods on the Drone Vehicle dataset in the infrared–visible image fusion task.

	Model	mAP0.5	mAP0.5:0.95	Recall	Car	Truck	Bus	Feright Car
Full	UA-CMDet	64.1	-	-	87.51	60.7	87.08	46.8
Dataset	Yolov5-obb	71	45.1	65.9	89	67.3	91.1	50.1
	ATGAN	50.5	30.9	64	81.1	32.5	64.9	23.7
	FusionGAN	65.8	47.4	71.6	89.5	50.0	79.7	43.8
process	Diffusion	50.5	23.7	60.3	78.1	25.6	68.5	29.7
5000 pairs	Tardal	57.4	37.7	66.1	83.8	40.5	73	32.5
	SPDFusion	63.7	43.2	71.9	84.5	46.4	76.6	47.2
	Ours	76.1	57.1	74.2	92.9	69.8	82.3	59.5

Bold values indicate the best result, and italic values represent the second-best result.

Notably, the proposed approach exhibits pronounced performance gains for object categories that are traditionally challenging to detect due to low contrast or limited texture information, such as trucks and vans, with improvements of 23.4% and 12.3%, respectively. In addition, a recall rate of 74.2% is achieved, indicating that the fused representations produced by the proposed framework facilitate more comprehensive and reliable object discovery.

The observed performance improvements can be primarily attributed to two complementary aspects of the proposed fusion framework. First, from the perspective of visual fidelity, the method effectively retains detailed visible-light textures while simultaneously emphasizing salient thermal cues, thereby providing richer and more discriminative appearance and structural information for the detection network. This capability is particularly advantageous for improving the detectability of medium- and large-scale objects. Second, from the perspective of feature alignment, the integration of a frequency-domain enhanced attention mechanism enables more coherent fusion of heterogeneous modality features. By aligning low-frequency structural information and selectively enhancing high-frequency details, the proposed framework reduces modality-induced discrepancies and preserves both geometric and semantic consistency in the fused representations. Together, these design choices result in more stable feature distributions for downstream detection, leading to consistent improvements across multiple evaluation metrics. These findings collectively confirm the effectiveness and robustness of the proposed method in multi-modal UAV object detection tasks.

4.4. Ablation Study

The Original model trained without IQA-guided optimization requires approximately 125 h to complete the full training process. After introducing the IQA-guided patch-generation and sample-organization strategy, the training time is reduced to approximately 3.85 h. Therefore, Org+IQA is adopted as the controlled baseline for the subsequent architectural ablations. Specifically, (FFM, SFFM) and (SFFM, FuseNet) are separately incorporated into Org+IQA while keeping the patch strategy, training schedule, optimizer, and other implementation settings unchanged. This protocol enables the effects of SFFM and FuseNet to be evaluated under consistent and computationally feasible training conditions.

Table 4 presents the ablation results of different frequency-domain design choices. After introducing the FFM module, the AG value decreases from 4.92 to 4.20, indicating a reduction in local gradient information. Nevertheless, FFM substantially improves color fidelity, as evidenced by the decrease in DeltaE from 18.59 to 12.13. Therefore, its contribution is primarily reflected in reducing cross-modal color discrepancy rather than enhancing image sharpness. Meanwhile, the SF metric is reduced from 17.72 to 15.52, suggesting that the introduced frequency-aware modeling suppresses redundant high-frequency noise and alleviates excessive spatial fluctuation. However, the decreases in VIF and EN imply that purely frequency-domain enhancement may weaken part of the global information representation and image entropy due to over-concentration on local frequency responses.

Table 4. Ablation study of frequency-domain designs under the IQA-guided optimization strategy.

Model	VIF	AG	EN	CC	DeltaE	SF
Org+IQA	0.78	4.92	5.23	0.88	18.59	17.72
Org+IQA+FFM	0.72	4.20	5.12	0.87	12.13	15.52
Org+IQA+SFFM	0.70	4.62	5.19	0.89	13.84	17.19

Table 4 indicates that the SFFM module introduces a trade-off among different fusion-quality criteria rather than uniformly improving all metrics. Compared with the baseline, SFFM reduces DeltaE from 18.59 to 13.84 and slightly increases CC from 0.88 to 0.89, while VIF, AG, EN, and SF decrease. Nevertheless, compared with the conventional FFM variant, SFFM recovers AG from 4.20 to 4.62, EN from 5.12 to 5.19, CC from 0.87 to 0.89, and SF from 15.52 to 17.19. These results indicate that statistical frequency modeling partially alleviates the loss of structural and texture information caused by conventional frequency-domain enhancement while maintaining a lower color discrepancy than the baseline. Therefore, the primary role of SFFM is not to independently improve every image-quality metric, but to provide complementary frequency-domain representations for the complete fusion framework, whose overall effectiveness depends on its collaboration with the other components.

A detailed interpretation of the ablation results presented in Table 5 is provided below. The newly introduced Org+IQA configuration establishes a direct correspondence with the IQA-guided baseline used in Table 4. Compared with Org, introducing the IQA-guided strategy increases AG from 4.72 to 4.92 and EN from 5.07 to 5.23 while maintaining the same VIF value of 0.78. However, DeltaE increases from 14.51 to 18.59 and SF decreases slightly from 18.05 to 17.72, indicating that quality-aware sample organization alone does not uniformly improve all fusion-quality metrics. Therefore, Org+IQA is adopted as the controlled baseline for evaluating the subsequent architectural components. Based on this configuration, incorporating SFFM reduces DeltaE from 18.59 to 13.84, demonstrating improved color fidelity, although VIF, AG, EN, and SF decrease to varying degrees. Incorporating FuseNet reduces DeltaE further to 8.20 and increases SF from 17.72 to 19.06, but it also introduces trade-offs in VIF, AG, and EN. These observations indicate that SFFM and FuseNet emphasize different aspects of multi-modal representation and do not independently improve every evaluation criterion.

Table 5. Ablation results of the IQA-guided optimization strategy and main network components.

Metric	Org	Org+IQA	Org+IQA+SFFM	Org+IQA+FuseNet	Ours
FPS	403.23	223.4	202.43	156.25	90.99
VIF	0.78	0.78	0.70	0.62	0.88
DeltaE	14.51	18.59	13.84	8.2	7.41
AG	4.72	4.92	4.62	4.82	5.38
EN	5.07	5.23	5.19	5.09	5.53
SF	18.05	17.72	17.19	19.06	19.58

Bold values indicate the best result, and italic values represent the second-best result.

When the IQA-guided optimization strategy, SFFM, and FuseNet are jointly integrated, the complete model achieves the best performance across all five fusion-quality metrics. Specifically, Ours attains a VIF value of 0.88, exceeding Org and Org+IQA (both 0.78), Org+IQA+SFFM (0.70), and Org+IQA+FuseNet (0.62). It also obtains the lowest DeltaE value of 7.41, compared with 14.51 for Org, 18.59 for Org+IQA, 13.84 for Org+IQA+SFFM, and 8.20 for Org+IQA+FuseNet, demonstrating improved color fidelity and reduced cross-modal chromatic discrepancy. Regarding structural-detail and information preservation, Ours achieves the highest AG, SF, and EN values of 5.38, 19.58, and 5.53, respectively. These results demonstrate that the complete framework more effectively preserves visual information, spatial edges, and fine-scale structural details. Overall, the improvements achieved by the complete model confirm that IQA-guided optimization, SFFM, and FuseNet provide complementary contributions, enabling their joint integration to compensate for the limitations observed when each component is introduced independently.

To further evaluate the effectiveness of different components in the proposed framework, an ablation study is conducted under a challenging nighttime traffic scenario, as illustrated in Figure 13.

Figure 13. An ablation study of different components under a challenging nighttime traffic scenario. (a) presents the baseline result, (b) introduces the statistical frequency-aware attention module (SFFM), (c) further incorporates the TransFuse branch for cross-modal feature interaction, and (d) corresponds to the complete proposed framework.

As shown in Figure 13a, the baseline method suffers from noticeable detail degradation and insufficient structural contrast under low-illumination conditions. Although the main road structure can be roughly preserved, vehicle boundaries and lane markings remain blurry, and local regions exhibit obvious texture inconsistency and illumination imbalance. This indicates that the baseline lacks sufficient capability for high-frequency detail preservation and cross-modal complementary representation. After introducing the SFFM module in Figure 13b, the fused result exhibits significantly enhanced local contrast and sharper structural details. Benefiting from the statistical frequency-domain attention mechanism, high-frequency components such as lane markings, vehicle contours, and roadside textures are more effectively preserved. However, excessive enhancement of local frequency responses also introduces slight over-smoothing and brightness diffusion in certain regions, resulting in limited global structural consistency and insufficient semantic coordination between modalities. Figure 13c further incorporates the TransFuse branch to strengthen long-range dependency modeling and cross-modal semantic interaction. Compared with Figure 13b, the global structural coherence and spatial continuity are noticeably improved, particularly in the road regions and object distributions. Nevertheless, although the TransFuse branch enhances semantic consistency, certain local texture regions still exhibit relatively weak edge sharpness and incomplete suppression of illumination interference, indicating that global interaction alone is insufficient for simultaneously maintaining fine-grained detail fidelity and stable frequency representation.

Finally, the complete framework shown in Figure 13d achieves the most balanced fusion performance by jointly integrating frequency-aware representation learning and cross-modal semantic interaction. The proposed method not only preserves clearer vehicle boundaries, lane structures, and roadside textures, but also effectively suppresses noise amplification and illumination diffusion. Compared with the previous variants, the fused result exhibits superior structural consistency, local detail fidelity, and perceptual naturalness. These observations demonstrate that the proposed framework benefits substantially from the complementary collaboration between statistical frequency modeling and Transformer-based cross-modal interaction, leading to more robust and discriminative fusion representations for downstream detection tasks.

In summary, the ablation study clearly shows that the proposed method achieves competitive compared with the original model and baseline variants across both major dimensions—high-fidelity reconstruction and feature alignment—thus demonstrating the capability of the framework to jointly optimize these two fundamental objectives in multi-modal image fusion.

4.5. Complexity Analysis

To verify the index training optimization strategy proposed in this paper, we conducted a complexity analysis on 30 pairs of publicly available infrared and visible-light fused images in the DroneVehicle dataset based on the ATGAN fusion algorithm framework. The computational efficiency of the proposed IQA-guided optimization strategy is summarized in Table 6. Unlike the conventional training pipeline, the proposed strategy improves efficiency from two aspects.

Table 6. Complexity analysis of index training optimization strategy.

Method	Patch Strategy	IQA Index	Time/Epoch (s)	Training FPS
Baseline	80 × 80/14	×	3522.68	10.82
Baseline + IQA-guided	100 × 100/80	✓	5.64	223.4
FIRGAN (without IQA-guided)	80 × 80/14	×	3644.95	10.46
FIRGAN (with IQA-guided)	100 × 100/80	✓	6.17	204.21

First, local image-quality scores are computed only once during the offline index construction stage and are directly reused during subsequent training, thereby avoiding repeated IQA evaluation. Second, a new patch generation strategy is adopted by increasing the patch size from 80 × 80 to 100 × 100 pixels and the stride from 14 to 80 pixels, which substantially reduces the number of overlapping training patches. Consequently, the computational overhead associated with sample generation and iterative optimization is significantly reduced. Although FIRGAN introduces additional computation due to the SFFM and CNN–Transformer dual-branch architecture, the proposed IQA-guided strategy effectively compensates for this overhead, resulting in a favorable trade-off between training efficiency and fusion performance.

5. Discussion

The proposed FIRGAN demonstrates the effectiveness of jointly optimizing fusion quality and downstream detection performance through quality-aware optimization, adaptive frequency-domain enhancement, and local–global collaborative feature learning. Theoretically, it provides a new perspective for integrating frequency-domain representation learning with multi-modal fusion for downstream detection applications. In practice, the efficient training strategies and frameworks for downstream detection improve the feasibility of deployment in scenarios such as intelligent surveillance and UAV inspection. Nevertheless, the proposed method still faces limitations in adaptive parameter tuning, patch partition dependence, and cross-scenario generalization. In future work, we will explore adaptive optimization mechanisms and broader multi-modal applications, including RGB–thermal, multispectral, and SAR–visible fusion, to further improve robustness, scalability, and task adaptability.

6. Conclusions

Driven by the demand for collaborative optimization between multi-modal image fusion and downstream visual detection tasks, this paper investigated the key challenges of high-fidelity feature representation, efficient model optimization, and fusion for downstream detection applications learning. To address these issues, a multi-modal fusion framework for downstream detection applications, termed FIRGAN, was proposed. The framework integrates an SFFM for adaptive spectrum enhancement, a Dual-branch Fusion Network for complementary local–global feature modeling, and an IQA-guided optimization strategy for quality-aware training. Through the collaborative effects of frequency-domain representation learning, dual-branch feature interaction, and quality-guided optimization, the proposed method achieves improved visual fidelity, structural preservation, and fusion robustness. Extensive experiments demonstrate that FIRGAN not only produces high-quality fused images but also provides more discriminative representations for downstream detection tasks. Future work will focus on task-driven fusion optimization and lightweight multi-modal perception frameworks for real-world intelligent vision systems.