Visual Attention Fusion Network (VAFNet): Bridging Bottom-Up and Top-Down Features in Infrared and Visible Image Fusion

Abstract

Infrared and visible image fusion aims to integrate useful information from the source image to obtain a fused image that not only has excellent visual perception but also promotes the performance of the subsequent object detection task. However, due to the asymmetry between image fusion and object detection tasks, obtaining superior visual effects while facilitating object detection tasks remains challenging in real-world applications. Addressing this issue, we propose a novel visual attention fusion network for infrared and visible image fusion (VAFNet), which can bridge bottom-up and top-down features to achieve high-quality visual perception while improving the performance of object detection tasks. The core idea is that bottom-up visual attention is utilized to extract multi-layer bottom-up features for ensuring superior visual perception, while top-down visual attention determines object attention signals related to object detection tasks. Then, a bidirectional attention integration mechanism is designed to naturally integrate two forms of attention into the fused image. Experiments on public and collection datasets demonstrate that VAFNet not only outperforms seven state-of-the-art (SOTA) fusion methods in qualitative and quantitative evaluation but also has advantages in facilitating object detection tasks.

1. Introduction

Due to theoretical and technical limitations, single modality images are unable to effectively and comprehensively describe imaging scenes [1,2]. Therefore, multi-modal image fusion technology has gradually become a research hotspot in the field of information fusion [3]. Among them, infrared images can highlight useful information by capturing thermal radiation. On the contrary, visible images can clearly outline the useful feature by capturing the light reflected from the surface of objects. The infrared and visible image fusion technology can utilize the advantages of both to extract symmetrical useful information for obtaining a fused image that can stably describe the imaging scene, which has become a widely used pre-processing module for high-level visual tasks such as object detection [4,5,6,7,8].

In recent years, scholars have proposed many infrared and visible image fusion methods, including traditional image fusion methods and deep learning-based image fusion methods. Traditional image fusion methods use hand-crafted features to obtain the fused image, such as multi-scale transformations, sparse representation, subspace and hybrid models. Deep learning-based image fusion methods, in contrast, can automatically extract useful features through network frameworks [9,10]. However, in order to achieve superior visual effects, these fusion methods mainly focus on the pixel relationships between image pairs, but object features are ignored. Unfortunately, for object detection tasks, the key to accurately locating interesting regions lies in obtaining rich object features. Therefore, due to the semantic asymmetry between image fusion tasks and object detection tasks, both traditional and deep learning image fusion methods are unable to improve the performance of subsequent object detection tasks. Some image fusion methods use object detection networks as constraints to train the image fusion network, forcing the fused image to contain object features [11,12]. Unfortunately, the effectiveness of cascaded optimization is limited [6]. Not only that, in order to achieve excellent visual perception, the computational complexity of existing fusion methods is increasing, which is not conducive to deploying pre-processing modules for object detection tasks because they cannot generate fusion images in real time.

The human visual system has powerful feature extraction capabilities with extremely high computing efficiency [13,14,15,16]. We therefore resort to our visual system for inspiration. Numerous neuroscience studies have proven that the human visual system relies on both bottom-up and top-down visual attention. In bottom-up process, the novel or salient visual stimuli can be automatically associated [17]. In contrast, attention can be focused on task-relevant stimuli by a top-down process [18]. Obviously, bottom-up visual attention can ensure that the fused image preserves rich visual features and top-down visual attention can be used to extract object features that are useful for the object detection task. Therefore, this paper studies a joint bottom-up and top-down visual attention fusion network to obtain the fused image with excellent visual perception and rich object features.

However, some works relied solely on either top-down or bottom-up visual attention to obtain fusion results [19,20,21], which neglects the cooperation between bottom-up and top-down visual attention, resulting in a restricted effect. In order to integrate top-down and bottom-up visual attention, Fang et al. [22] used dense connections to combine top-down signals with the top layer of the bottom-up signals. Lin et al. [23] directly added bottom-up and top-down attention signals to segment objects. However, although top-down signals preserve object features, the saliency is reduced due to the increase in network depth and narrow receptive fields. Therefore, directly combining bottom-up and top-down signals will heavily rely on the learning ability of the fusion network, which is typically proportional to the size of the training dataset. Unfortunately, obtaining a large amount of training data is very difficult, especially in the field of infrared and visible image fusion. In addition, these methods cannot learn the relevant relationships between top-down and bottom-up signals, which may limit their natural combination due to semantic differences.

To overcome the above problem, we propose a novel visual attention fusion network for infrared and visible image fusion, which can achieve excellent visual perception while improving the performance of object detection tasks. Firstly, since bottom-up visual attention is automatically triggered by external novel or salient stimuli, we design a visual attention extraction module (VAEM) to obtain the multi-layer bottom-up feature of infrared and visible images in the bottom-up process. The multi-layer bottom-up feature of this module integrates both intensity and orientation features, effectively enhancing the visual effect of fused images. Secondly, in the top-down process, a cross-modal attention module (CMAM) is developed to extract significant features from multi-layer bottom-up features, then the multi-layer significant features are merged to obtain object attention signals through a cross-scale attention module (CSAM). Finally, in order to fully leverage the advantages of bottom-up and top-down visual attention, a bidirectional attention integration mechanism is proposed to naturally integrate multi-layer bottom-up feature and object attention signals, which includes an object attention enhancement module (OAEM) and attention guidance module (AGM) to overcome the loss of object feature saliency and semantic differences between bottom-up and top-down signals, respectively. In particular, the top-down process utilizes multi-layer bottom-up features obtained from the bottom-up process to improve object feature extraction, while the bottom-up process further integrates the object attention signal obtained from the top-down process to obtain fine-grained visual features. Thus, in VAFNet, top-down and bottom-up visual attention work in a collaborative manner. The specific contributions in this study are as follows:

• We propose a real-time visual attention image fusion network, which can achieve superior performance in both excellent visual perception and improve object detection tasks.
• We design a bidirectional attention integration mechanism to integrate multi-layer bottom-up features and object attention signals, which can effectively enhance object attention signals and extract relevant features of two forms of attention.
• A series of experiments on public and collected datasets show that the proposed method not only has better qualitative and quantitative evaluation results than seven SOTA methods but also can serve as a pre-processing module for the object detection task.

The remainder of this paper is organized as follows. In Section 2, we introduce the related works of existing image fusion methods and visual attention mechanisms. Section 3 describes our VAFNet in detail. Section 4 analyzes the experimental results of the proposed method, followed by Section 5, which presents the conclusions of this work.

2. Related Work

In this section, the existing fusion methods and visual attention mechanism are briefly described.

2.1. Image Fusion Methods

2.1.1. Traditional Image Fusion Methods

In recent years, due to strong interpretability and mature theoretical foundations, many traditional infrared and visible image fusion methods have been proposed. According to the adopted theory, these fusion methods can be roughly classified into four categories, including multi-scale transformation, sparse representation, subspace and hybrid models. The multi-scale transformation utilizes decomposition tools to extract high-frequency and low-frequency information from the source images and then reconstructs the fused image by inverse transformation. Although this method can preserve rich texture in the fused image, it is difficult to extract prominent objects [24,25,26]. The sparse representation can use sparse bases from an overcomplete dictionary to extract prominent objects of the source image, but its ability to retain details is insufficient [27]. The subspace method can map the source image to a low-dimensional space to obtain key information. Therefore, this method has good computational efficiency, but it is prone to losing background information [28]. The above theories have their own advantages, so the hybrid model aims to improve the quality of the fused images by combining their advantages [29]. However, complex hybrid models may not be able to meet the requirements of real-time image fusion. In addition, traditional image fusion methods focus on improving the visual effect of fused images and fail to integrate object semantic information, which limits the contribution of fusion results to object detection tasks.

2.1.2. Deep Learning-Based Image Fusion Methods

In the past few years, since deep learning has a powerful feature extraction ability and stability, it has been successfully applied in the community of image fusion. Yang et al. [30] proposed a texture conditional generative adversarial network to keep the gradient changes of the visible image in the fusion results. Rao et al. [31] combined a convolutional neural network and transformer to extract local and global information to improve the resolution of fused images. Su et al. [32] designed an auto-encoder to obtain fused images with better visual effects. However, it has been proven that solely considering the visual effects of fused images is difficult to improve the performance of the object detection task [33,34]. In order to preserve the object features in the fused image, Liu et al. [12] constructed a cascaded training framework for object detection and image fusion networks. Similarly, Tang et al. [11] used a high-level task to constrain image fusion networks for obtaining the fused image with a salient object. Unfortunately, using the fusion output directly as a link between the image fusion module and the object detection network fails to effectively guide the fusion process, as it overlooks the pixel-level semantic cues provided by the detection network.

2.2. Visual Attention for Image Fusion

The human visual system exhibits remarkable capability in extracting critical features from complex scenes with high computational efficiency, which offers benefits in many visual tasks, such as visual question answering and image classification [35,36,37]. It has been proven that the visual attention can be divided into bottom-up and top-down processes [38]. On the one hand, bottom-up visual attention is driven by external visual stimuli, such as direction and intensity. Wang et al. [39] utilized bottom-up visual attention to obtain visual information during feature extraction. Liu et al. [20] proposed an improved ITTI visual attention model to extract intensity and directional features of source images. These methods demonstrate that bottom-up attention can obtain fused images with excellent visual perception. On the other hand, top-down visual attention can modulate the visual system’s responsiveness to different stimuli based on prior knowledge and experience of perceptual objects. This mechanism enables selective enhancement of object-relevant information while suppressing task-irrelevant distractors in complex scenes [40]. Fang et al. [41] utilized top-down attention to detect salient objects. Wang et al. [42] enhance the object semantics by top-down attention. These works demonstrate that top-down visual attention can effectively capture an object’s semantic information.

Therefore, by integrating bottom-up and top-down visual attention, a fused image with excellent visual quality and rich object semantics can be obtained. However, early attempts at integrating bottom-up and top-down visual attention merely concatenated features along the channel dimension or performed direct element-wise addition [22,23], which placed excessive reliance on the fusion network’s learning capacity. Crucially, due to the inherent asymmetry between bottom-up and top-down features, it is essential to explicitly model their cross-modal correlations to achieve plausible integration.

3. Method

3.1. Overview

Figure 1 shows the network architecture of VAFNet, which has three main components: (a) bottom-up process, (b) top-down process and (c) bidirectional attention integration mechanism. Firstly, in the bottom-up process, VAEM is utilized to extract the multi-layer bottom-up features of infrared and visible images, respectively (see Section 3.2). Secondly, in the top-down process, the multi-layer bottom-up features of the infrared and visible image are fed into CMAM to obtain multi-scale significant features, and then CSAM is utilized to integrate the multi-scale significant features for obtaining object attention signals (see Section 3.3). Finally, in order to reconstruct the fused image, we use a bidirectional attention integration mechanism to combine multi-scale attention maps and object attention signals(see Section 3.4).

3.2. Bottom-Up Process

The bottom-up process work is shown in Figure 1a. Specifically, bottom-up attention is driven by external stimuli, which exhibit varying saliency levels across different spatial locations. Therefore, the VAEM is employed to extract multi-layer bottom-up features, ensuring that salient external stimuli are accurately preserved in the fused image. Moreover, since infrared and visible images possess distinct characteristics, the process of extracting multi-layer bottom-up features can be expressed as,

$$\left\{{\varphi }_i^{vis},{\varphi }_i^{ir}\right\}=\left\{\begin{array}{cc}VAEM\left(I_{vis},I_{ir}\right)& ifi=1\\ VAEM\left({\varphi }_{i-1}^{vis},{\varphi }_{i-1}^{ir}\right)& ifi=2,3,4\end{array}\right.$$

(1)

where ${\varphi }_i^{vis}\in R^{H\times W\times C}$ and ${\varphi }_i^{ir}\in R^{H\times W\times C}$ represent the i-th layer bottom-up features of the visible $I_{vis}\in R^{H\times W\times 3}$ and infrared images $I_{ir}\in R^{H\times W\times 3}$, respectively. $VAEM\left(·\right)$ represents the bottom-up feature extraction operation, and its framework is illustrated in Figure 2. Firstly, we employ a convolutional layer with kernels of $3\times 3$ and a stride of 1 to extract shallow features ${\gamma }_i\in R^{H\times W\times C}$ from the input. After the convolutional layer, Sigmoid is used as the activation function. Then, as higher intensity information is more likely to attract attention, we utilize the Gaussian kernel function to extract intensity features, which can be expressed as,

$${\mathsf{\Gamma }}_i=Gaussian\left(w,\sigma \right)\ast {\gamma }_i$$

(2)

where $Gaussian\left(·\right)$ represents the Gaussian kernel, w is the Gaussian filtering window, and $\sigma$ is the Gaussian kernel parameters. ${\mathsf{\Gamma }}_i\in R^{H\times W\times C}$ is the intensity features; ∗ illustrates the convolution operation.

Moreover, regions with richer orientations in the source image tend to attract more attention, so the Gabor filter [43] is utilized to extract orientation features, which can be represented as,

$$O_i=\sum _{\theta \in \left\{0^{\circ },{45}^{\circ },{90}^{\circ },{135}^{\circ }\right\}}Gabor\left(\theta \right)\ast {\gamma }_i$$

(3)

where $Gabor\left(·\right)$ is Gabor filter, $\theta$ is the filtering direction, $\theta \in \left\{0^{\circ },{45}^{\circ },{90}^{\circ },{135}^{\circ }\right\}$. $O_i\in R^{H\times W\times C}$ is the orientation features.

Furthermore, based on the extracted intensity and texture features, we propose a linear normalization operator for visual salient feature extraction. Specifically, this operator first sums the intensity features and textural features, then applies normalization to generate a visual saliency map. In this saliency map, higher values indicate regions with greater attentional priority. Subsequently, the saliency map is multiplied by ${\gamma }_i$ to obtain the visual salient feature. This process can be described as,

$${\psi }_i=Nor\left({\mathsf{\Gamma }}_i\oplus O_i\right)\otimes {\gamma }_i$$

(4)

where ${\psi }_i\in R^{H\times W\times C}$ is the visual salient feature, $Nor\left(·\right)$ is the normalization operation to the range of $0-1$, ⊕ represents element-wise addition, ⊗ is the Hadamard product operation, which represents an element-wise multiplication operation between two features of identical dimensions [44].

Finally, to prevent the degradation of visual features as network depth increases, residual connections are employed to integrate features ${\psi }_i$ and ${\gamma }_i$, which can be expressed as,

$${\varphi }_i={\psi }_i\oplus {\gamma }_i$$

(5)

where ${\varphi }_i$ are the i-th layer bottom-up features.

3.3. Top-Down Process

The top-down process includes two components: CMAM and CSAM, as shown in Figure 3a and b, respectively. It takes the multi-layer bottom-up features of the infrared and visible images as input and computes significant features by CMAM. Then, CSAM is used to merge the ith significant features for obtaining object attention signals.

3.3.1. Cross Modal Attention Module

Since the multi-layer bottom-up features of infrared and visible imagesb can prove important clues for focusing on object-relevant stimuli, the ${\varphi }_i^{ir}$ and ${\varphi }_i^{vis}$ are connected to form cross modal features ${\kappa }_i\in R^{H\times W\times 2C}$. Next, we calculate the maximum values $p_i^{max}\in R^{1\times 1\times 2C}$ and average values $p_i^{avg}\in R^{1\times 1\times 2C}$ of ${\kappa }_i\in R^{H\times W\times 2C}$ in channel [45], then the difference between $p_i^{max}$ and $p_i^{avg}$ is calculated to obtain channel attention features, which can be expressed as,

$${\mathrm{Z}}_i={\langle {\displaystyle \frac{1}{1+e^{\left(p_i^{max}-p_i^{avg}\right)}}},{\kappa }_i\rangle }_C$$

(6)

where ${\mathrm{Z}}_i\in R^{H\times W\times 2C}$ is channel attention features, and ${\langle ·,·\rangle }_C$ is channel element-wise multiplication.

In addition, we obtain the spatial maximum value $q_i^{max}\in R^{H\times W\times 1}$ and spatial average value $q_i^{avg}\in R^{H\times W\times 1}$ of ${\mathrm{Z}}_i$ through max pooling and average pooling [46], and then the significant features in each layer are obtained by Equation (7),

$${\nu }_i={\langle {\displaystyle \frac{1}{1+e^{\left(q_i^{max}-q_i^{avg}\right)}}},{\mathrm{Z}}_i\rangle }_{H\times W}$$

(7)

where ${\nu }_i\in R^{H\times W\times 2C}$ is the significant features, ${\langle ·,·\rangle }_{H\times W}$ is the spatial element-wise multiplication.

3.3.2. Cross-Scale Attention Module

In order to efficiently combine significant features, the ${\nu }_i\in R^{H\times W\times 2C}$ is input into a convolutional layer with kernels of $1\times 1$ and stride of 1 to adjust the number of channels, which can be expressed as,

$${\zeta }_i=con{v}_{1\times 1}\left({\nu }_i\right)$$

(8)

where ${\zeta }_i\in R^{H\times W\times C}$ is the output layers of the convolutional layer, and $con{v}_{n\times n}\left(·\right)$ is the standard convolution operation with the kernel set to $n\times n$ and stride set to 1. After the convolutional layer, the leaky rectified linear unit (LReLU) is used as the activation function.

After resizing all ith significant features, Hadamard product operations are applied to obtain object attention signals, which can be expressed as,

$$\stackrel{\sim }{\zeta }={\otimes }_{i=1}^4\left({\zeta }_i\right)$$

(9)

where $\stackrel{\sim }{\zeta }\in R^{H\times W\times C}$ is object attention signals. ${\otimes }_{i=1}^4\left(·\right)$ denotes the Hadamard product of multi-scale attention feature.

3.4. Bidirectional Attention Integration Mechanism

The whole structure of the bidirectional attention integration mechanism is shown in Figure 1c. Specifically, the saliency of object features in the fused image is crucial for improving the performance of object detection tasks. However, due to the gradual deepening of the network and narrow receptive fields, it is extremely easy to reduce the saliency of object attention signals. Therefore, OAEM is considered to enhance the $\stackrel{\sim }{\zeta }\in R^{H\times W\times C}$. In addition, the proposed AGM can be utilized to guide the multi-layer bottom-up features through enhanced object attention signals for obtaining global attention signals. Last, the global attention signals are used to reconstruct the fused image, which can be expressed as,

$$F_i=\left\{\begin{array}{cc}con{v}_{3\times 3}\left({\phi }_i\right)& ifi=4\\ con{v}_{3\times 3}\left(F_{i+1}\copyright {\phi }_i\right)& ifi=3,2,1\end{array}\right.$$

(10)

where © is the channel connection operation. After the $F_4,F_3,F_2$ convolutional layer, LReLU is used as the activation function. ${\phi }_i\in R^{H\times W\times 2C}$ is the global attention signals. The $F_1$ is followed by the Tanh activation function. In the end, we get $F_1\in R^{H\times W\times 3}$ as the fused image.

3.4.1. Object Attention Enhancement Module

OAEM considers enhancing the saliency of object features from two aspects. On the one hand, we adopt a multi-branch structure to increase the richness of object features, which can reduce the saliency attenuation caused by network depth. On the other hand, in order to expand the receptive field, an atrous convolution is applied to obtain richer object features. The structure of OAEM is shown in Figure 3c. OAEM includes two branches with atrous convolution and one residual branch, where two branches perform $1\times 3$ and $3\times 1$ convolution to adjust the number of channels and then use atrous convolutions to extract object features of different shapes, which can be expressed as,

$$\stackrel{\wedge }{\mathrm{{\rm Y}}}=f_{3\times 3}\left(Con{v}_{1\times 3}\left(Con{v}_{3\times 1}\left(\stackrel{\sim }{\zeta }\right)\right)\right)$$

(11)

$$\stackrel{\vee }{\mathrm{{\rm Y}}}=f_{3\times 3}\left(Con{v}_{3\times 1}\left(Con{v}_{1\times 3}\left(\stackrel{\sim }{\zeta }\right)\right)\right)$$

(12)

where $f_{3\times 3}\left(·\right)$ represents atrous convolution with a dilation rate of 5 and the kernel set to $3\times 3$. $\stackrel{\wedge }{\mathrm{{\rm Y}}}\in R^{H\times W\times {\displaystyle \frac{C}{2}}}$, and $\stackrel{\vee }{\mathrm{{\rm Y}}}\in R^{H\times W\times {\displaystyle \frac{C}{2}}}$ are the output data of two branches after atrous convolution.

Then, the residual branch connects two branches with an atrous convolution to enhance object features, which can be expressed as,

$$\mathrm{{\rm Y}}=\stackrel{\wedge }{\mathrm{{\rm Y}}}\copyright \stackrel{\vee }{\mathrm{{\rm Y}}}\copyright \stackrel{\sim }{\zeta }$$

(13)

where $\mathrm{{\rm Y}}\in R^{H\times W\times 2C}$ is the enhanced object attention signal.

3.4.2. Attention Guidance Module

In order to efficiently obtain global attention signals, the key issue is obtaining the correlation of multi-layer bottom-up features and object attention signals to guide their natural combination. Inspired by the impressive performance of transformer networks in capturing contextual relationships [47], we design AGM to obtain global attention signals, as shown in Figure 3d. Firstly, the multi-layer bottom-up features of infrared and visible are connected to obtain cross-modal features ${\kappa }_i$. In addition, we take ${\kappa }_i$ as query(Q) and $\mathrm{{\rm Y}}$ as key(K) to extract the correlation weight, which is expressed as,

$$At{t}_i=Softmax\left({\displaystyle \frac{Q·K_i}{\sqrt{d_k}}}\right)$$

(14)

where $At{t}_i\in R^{H\times W\times 2C}$ is the correlation weight.

Then, in order to reduce the semantic difference between object attention signals and multi-layer bottom-up features, we take the enhanced object attention signal as value (V) to multiply $At{t}_i$ to obtain the modulated object attention signals. Finally, the global attention signals can be extracted by adding the modulated object attention signals and multi-layer bottom-up features. This process can be expressed as,

$${\phi }_i=At{t}_i·V\oplus {\kappa }_i$$

(15)

where ${\phi }_i\in R^{H\times W\times 2C}$ is the global attention signals.

3.5. Loss Function

In order to guide the bottom-up process in VAFNet to extract strength features, we present a strength loss $l_{str}$ to measure the pixel-level differences between the fused image and the source image, which can be expressed as,

$$l_{str}={\displaystyle \frac{1}{H\times W\times C}}\left({∥F-I^{vis}∥}_2+{∥F-I^{ir}∥}_2\right)$$

(16)

where ${∥·∥}_2$ is L2 norm, $F\in R^{H\times W\times 3}$ is the fused image. $I^{vis}\in R^{H\times W\times 3}$ and $I^{ir}\in R^{H\times W\times 3}$ are the visible and infrared image, respectively. H, W and C are the height, width and channel number.

Then, we apply a structure loss $l_{ssim}$ to force the bottom-up process for obtaining orientation features, which can be expressed as,

$$l_{ssim}=SSIM\left(F,I^{vis}\right)+SSIM\left(F,I^{ir}\right)$$

(17)

where $SSIM\left(·\right)$ is a structural similarity measurement.

In addition, in order to encourage a top-down process to extract object features, the proposed object loss can be defined as,

$$l_{object}={\displaystyle \frac{1}{H\times W\times C}}\left({∥F,{\mathrm{\Psi }}^{ir}∥}_1+{∥F,{\mathrm{\Psi }}^{vis}∥}_1\right)$$

(18)

where ${∥·∥}_1$ is L1 norm, $l_{object}$ is the object loss, ${\mathrm{\Psi }}^{ir}$ and ${\mathrm{\Psi }}^{vis}$ are object regions extracted from infrared and visible images using object saliency detection algorithms [48].

Finally, we combine the strength, structure, and object loss to train VAFNet, which can be expressed as,

$$l_{total}=l_{str}+{\omega }_1{l}_{ssim}+{\omega }_2{l}_{object}$$

(19)

where $l_{total}$ is the final loss, and ${\omega }_1$ and ${\omega }_2$ are the balance coefficient.

4. Experiment

In this section, we perform a series of experiments on the public and collected datasets to comprehensively validate the performance of our fusion method. Firstly, we describe the experimental settings and implementation details. Secondly, the comparative, generalization and surveillance application experiments are discussed and analyzed. Thirdly, we also complete the evaluation of infrared-visible object detection. Finally, the computational efficiency of the proposed method is analyzed.

4.1. Experimental Settings

To demonstrate the advantages of our VAFNet, the comparative and generalization experiments were conducted on the MSRS [49] and TNO public datasets, respectively. Moreover, we also collected 23 pairs of aligned infrared and visible images during daytime and nighttime for surveillance application experiments. The imaging equipment and acquisition site are shown in Figure 4. We compare VAFNet with seven SOTA infrared and visible image fusion methods, including MDA [50], SFINet [51], SpTFuse [52], U2Fusion [53], LRRNet [54], RES2Fusion [55] and SeAFusion [11]. The implementation code of the seven SOTA methods is publicly available, and the parameter selection is based on default values.

In addition, we selected four metrics for quantitative evaluation, including entropy (EN) [56], spatial frequency (SF) [57], standard deviation (SD) [58] and visual information fidelity (VIF) [59]. EN can measure the information richness of the fused image, with a larger EN indicating a richer amount of information. SF describes the clarity of fused images, with larger SF representing clearer fused images. SD indicates the degree of pixel distribution in fused images, with a larger SD indicating higher contrast. VIF can evaluate the visual fidelity of the fusion results; a higher VIF value means the fused image is closer to human visual perception.

4.2. Implementation Details

We utilized 1106 pairs of registered infrared and visible images from the MSRS and collection dataset to train VAFNet, which includes 562 pairs of daytime scene images and 544 pairs of nighttime scene images. The parameters of VAFNet were updated by the Adam optimizer and the learning rate was set to $1\times {10}^{-4}$. For the hyperparameters of Equation (19), we set ${\omega }_1=5$ and ${\omega }_2=1$. Our VAFNet was implemented on the Microsoft Windows 10 platform, NVIDIA GeForce RTX 4070 super graphics Processing Unit (GPU), 32 GB Random Access Memory and Intel i5-13400F central processing unit (CPU).

4.3. Comparative Experiment

In order to illustrate the advantages of the proposed fusion method, we compared the VAFNet with LRRNet, MDA, RES2Fusion, SeAFusion, SFINet, SpTFuse and U2Fusion on the MSRS dataset.

4.3.1. Qualitative Evaluation

Since image fusion methods need to be applied around the clock, we chose daytime and nighttime scenes for qualitative evaluation. The qualitative evaluation results of VAFNet and seven SOTA fusion methods are shown in Figure 5. In daytime scenes, infrared images have prominent objects, while visible images have rich details. Therefore, the fused image with excellent visual effects should have prominent infrared objects and rich textures. As shown in the first and second rows of Figure 5, due to insufficient feature extraction ability, LRRNet and MDA cannot preserve the rich details of visible images. In contrast, SpTFuse and U2Fusion cannot effectively preserve infrared objects. Although SFINet, RES2Fusion and SeAFusion can extract the infrared objects and visible texture to a certain extent, they cannot effectively overcome the interference of uneven lighting. To clearly demonstrate the ability to preserve textures, we enlarged a small texture region with a red box in the bottom right corner. In addition, a prominent object is displayed within the yellow box to illustrate the effectiveness of extracting interesting targets. We can see that due to the ability to accurately extract visual saliency features from infrared and visible images, only our method has significant objects and rich texture information.

In nighttime scenes, although infrared images are not easily affected, visible images always contain many distorted regions due to insufficient lighting. Therefore, excellent fusion methods not only preserve infrared objects but also need to extract features from distorted regions to comprehensively express the imaging scene. As shown in the third and fourth rows of Figure 5, each fusion method can combine complementary information of infrared and visible images to some extent. However, from the enlarged texture region (as shown in the red box) and target region (as shown in the yellow box), it can be seen that apart from the fusion results of SeAFuison and VAFNet, there is some distortion in the fused images of the other six fusion methods. Compared to SeAFusion, VAFNet has richer texture details thanks to the stability of the visual attention mechanism. In summary, the proposed fusion method has better qualitative evaluation performance than the other seven fusion methods.

4.3.2. Quantitative Evaluation

Table 1. Mean and standard of quantitative evaluation results on 56 image pairs from the MSRS dataset. Blue and green represent the best and runner-up values, respectively. ↑ describes that the larger the value, the better the performance.

Method	EN↑	SF↑	SD↑	VIF↑
LRRNET	5.399 ± 0.663	4.674 ± 0.961	22.626 ± 8.047	0.513 ± 0.120
MDA	5.329 ± 0.471	4.523 ± 0.735	17.831 ± 4.531	0.614 ± 0.061
RES2Fusion	4.896 ± 1.056	5.028 ± 0.972	32.774 ± 9.056	0.827 ± 0.097
SeAFusion	5.965 ± 0.544	5.637 ± 0.860	33.169 ± 8.225	1.001 ± 0.077
SFINet	6.074 ± 0.513	5.272 ± 0.824	28.012 ± 7.268	0.785 ± 0.111
SpTFuse	5.981 ± 0.547	5.256 ± 0.806	32.126 ± 8.299	0.988 ± 0.073
U2Fusion	4.244 ± 0.706	4.096 ± 0.919	18.176 ± 5.180	0.477 ± 0.063
VAFNet	6.452 ± 0.359	6.851 ± 0.874	36.451 ± 4.421	1.060 ± 0.210

and Figure 6 show the quantitative evaluation results of EN, SF, SD and VIF metrics for 56 pairs of images from the MSRS dataset. For MI, VAFNet achieved the maximum value, indicating that our proposed method can generate fused images with rich complementary information. For SF and SD, our method has great advantages, which indicates that VAFNet has abundant textures and significant objects. Finally, the evaluation results of VIF indicate that the fused images generated by VAFNet conform to human visual perception. In summary, compared with the other seven fusion methods, the proposed fusion method has better quantitative evaluation results.

4.4. Generalization Experiment

To evaluate the reliability of our fusion method, we perform generalization experiments on the TNO dataset. Our VAFNet was trained on the MSRS and collection dataset, and tested on the TNO dataset.

4.4.1. Qualitative Evaluation

Figure 7 shows the qualitative evaluation results of VAFNet and other seven comparative methods. It can be seen that the fusion results of MDA, SFINet, SpTFuse, U2Fusion, LRRNet and Res2Fusion not only weaken the contrast of infrared objects but also have blurry textures. To visually demonstrate the fusion performance of the proposed method, we enlarged a small texture region in the red box and displayed an object region in the yellow box. We can observe that only the fusion results of SeAFusion and VAFNet retain the significance of infrared targets. However, compared to SeAFusion, the proposed method has clearer textures.

4.4.2. Quantitative Evaluation

We also selected nine pairs of infrared and visible images from the TNO dataset for qualitative evaluation, as shown in

Table 2. Mean and standard of quantitative evaluation results on 9 image pairs from the TNO dataset. Blue and green represent the best and runner-up values, respectively. ↑ describes that the larger the value, the better the performance.

Method	EN↑	SF↑	SD↑	VIF↑
LRRNet	7.177 ± 0.399	8.095 ± 1.487	44.302 ± 12.874	0.970 ± 0.208
MDA	6.589 ± 0.306	6.320 ± 0.846	28.665 ± 7.255	0.812 ± 0.153
RES2Fusion	6.188 ± 0.537	5.200 ± 1.461	23.206 ± 9.078	0.741 ± 0.108
SeAFusion	7.067 ± 0.234	8.946 ± 1.642	40.161 ± 10.384	1.061 ± 0.303
SFINet	6.940 ± 0.325	8.148 ± 1.220	38.649 ± 9.845	0.857 ± 0.182
SpTFuse	6.810 ± 0.388	5.756 ± 1.409	32.796 ± 11.088	0.915 ± 0.101
U2Fusion	6.465 ± 0.550	7.798 ± 1.666	25.813 ± 9.795	0.765 ± 0.147
VAFNet	7.416 ± 0.185	9.124 ± 1.850	48.841 ± 3.766	1.042 ± 0.302

and Figure 8. The results demonstrate that VAFNet outperforms other methods in EN, SF, and SD, indicating its superior capabilities in preserving information richness, maintaining optimal clarity, and retaining object details. However, due to the considerable difficulty in collecting large-scale training datasets, the proposed method achieves a secondary ranking on the VIF metric.

4.5. Evaluation on Surveillance Application

To evaluate the performance of the proposed method in practical surveillance tasks, we collected 23 pairs of infrared and visible images for application evaluation.

4.5.1. Qualitative Evaluation

Figure 9 shows the quantitative evaluation results of our VAFNet and seven SOTA fusion methods. It can be seen that VAFNet not only has better visual effects but also has prominent objects (as shown in the yellow box) in daytime scenes. Moreover, for nighttime scenes, the proposed method has a lighting enhancement effect because VAFNet can accurately extract complementary information from dark regions. In summary, the application experiment shows that VAFNet can provide comprehensive imaging scene information for surveillance tasks.

4.5.2. Quantitative Evaluation

Table 3. Mean and standard of quantitative evaluation results on 23 image pairs form the collection dataset. Blue and green represent the best and runner-up values, respectively. ↑ describes that the larger value, the better performance.

Method	EN↑	SF↑	SD↑	VIF↑
LRRNet	7.069 ± 0.542	9.076 ± 1.533	45.272 ± 9.645	0.817 ± 0.131
MDA	6.791 ± 0.498	7.901 ± 1.315	33.212 ± 8.180	0.727 ± 0.105
RES2Fusion	7.113 ± 0.507	8.835 ± 1.738	44.663 ± 11.189	0.116 ± 0.192
SeAFusion	7.294 ± 0.405	9.520 ± 1.450	48.948 ± 9.929	0.950 ± 0.129
SFINet	6.955 ± 0.453	8.121 ± 1.584	39.058 ± 9.006	0.362 ± 0.387
SpTFuse	7.335 ± 0.473	8.451 ± 1.355	49.471 ± 11.994	0.876 ± 0.099
U2Fusion	6.676 ± 0.664	9.200 ± 1.854	33.300 ± 9.809	0.092 ± 0.112
VAFNet	7.337 ± 0.275	9.232 ± 1.235	49.773 ± 5.285	0.799 ± 0.195

and Figure 10 show the quantitative evaluation results of our VAFNet and seven SOTA fusion methods from the collection dataset. It can be seen that the proposed method ranks first in EN and SD. The best EN demonstrates that VAFNet can effectively integrate complementary information. The best SD represents that our fusion method can effectively extract salient objects. In addition, SF has the second-best evaluation result, indicating that the fusion result of VAFNet has good clarity. Finally, although our VIF value is not optimal, it also indicates that the fusion results of the proposed method are consistent with human visual perception.

4.6. Evaluation on Infrared-Visible Object Detection

The infrared and visible fusion technology has been widely used as a preprocessing module for object detection tasks. Therefore, we use the popular YOLOV5 to evaluate the performance of each fusion method in object detection tasks. Specifically, we selected 80 pairs of infrared and visible images from the MSRS dataset as the test set and labeled them with two key categories: people and cars. Figure 11 provides some visual examples to illustrate the advantages of our VAFNet in improving the performance of object detection tasks. In the first row of Figure 11, YOLOV5 is unable to detect a car in the infrared image due to the lack of texture. On the contrary, due to lighting factors, YOLOV5 cannot detect pedestrians in the visible image. However, since VAFNet and other seven methods can combine complementary information of infrared and visible images to a certain extent, the fusion results of various fusion methods can enable YOLOV5 to detect pedestrians and cars simultaneously. However, due to the proposed method, maintaining the best object intensity and clearest texture information, VAFNet has the optimal confidence level. It is worth noting that there are similar results in the second row of Figure 11, which proves that VAFNet can effectively improve the performance of object detection.

In addition, precision (P), recall (R), and the mean average Precision (mAP) are used to measure detection performance. The AP@50, AP@70 and AP@90 are the AP value at Intersection over Union (IoU) 0.5, 0.7 and 0.9. mAP@[0.5:0.95] represents the average precision within the range of IoU from 0.5 to 0.95. The detection performance of YOLOV5 is shown in

Table 4. Object detection performance (mAP) of visible, infrared and fused images on the MSRS dataset. Blue represents the best result, and green represents the second best result.

Method	P	R	AP@50			AP@70			AP@90			mAP@[0.5:0.95]
			Person	Car	All	Person	Car	All	Person	Car	All	Person	Car	All
Infrared	0.9203	0.7337	0.9493	0.6835	0.8164	0.9061	0.6294	0.7677	0.2375	0.1938	0.2157	0.7208	0.4959	0.6084
Visible	0.7806	0.6606	0.4375	0.9255	0.6815	0.3197	0.9199	0.6198	0.0032	0.3993	0.2012	0.2412	0.7514	0.4963
MDA	0.9485	0.7856	0.8165	0.9557	0.8861	0.7640	0.9538	0.8589	0.0999	0.4473	0.2736	0.5714	0.7938	0.6826
SFINet	0.9488	0.8756	0.9366	0.9830	0.9598	0.8908	0.9830	0.9369	0.1397	0.5667	0.3532	0.6771	0.8278	0.7524
SpTFuse	0.9156	0.8873	0.9281	0.9752	0.9516	0.8891	0.9665	0.9278	0.1757	0.4843	0.3300	0.6868	0.8174	0.7521
U2Fusion	0.9467	0.8324	0.8699	0.9604	0.9152	0.8499	0.9513	0.9006	0.1160	0.6232	0.3696	0.6348	0.8206	0.7277
LRRNet	0.9612	0.7414	0.7350	0.9440	0.8395	0.6598	0.9440	0.8019	0.0266	0.6059	0.3162	0.4748	0.7944	0.6346
RES2Fusion	0.9235	0.7973	0.9039	0.8864	0.8951	0.8269	0.8719	0.8494	0.0663	0.3922	0.2292	0.6170	0.7210	0.6690
SeAFusion	0.9108	0.9038	0.9372	0.9742	0.9557	0.8707	0.9604	0.9156	0.1684	0.5440	0.3562	0.6726	0.8171	0.7448
VAFNet	0.9639	0.9476	0.9912	0.9911	0.9911	0.9896	0.9912	0.9904	0.3041	0.7751	0.5396	0.8014	0.8838	0.8426

. We can see that VAFNet exhibits high accuracy and recall, and overall has better accuracy under different IoUs. In summary, the proposed method can effectively improve the performance of object detection tasks.

4.7. Efficiency Comparison

In practical applications, the image fusion methods must have excellent computational efficiency to achieve real-time processing.

Table 5. Mean and standard of running time on the MSRS, TNO and collection dataset (unit: second). Blue represents the best values, and green represents the runner-up values.

Method	MSRS	TNO	Collection
MDA	0.2527 ± 0.0065	0.9739 ± 0.0168	1.1855 ± 0.0113
SFINet	1.6688 ± 0.3591	1.3579 ± 0.4125	8.5686 ± 1.7780
SpTFuse	0.6259 ± 0.0137	1.9306 ± 0.0390	2.4026 ± 0.0434
U2Fusion	0.6514 ± 0.0193	2.0656 ± 0.0349	2.4314 ± 0.0444
LRRNet	0.0967 ± 0.3240	0.4529 ± 0.7805	0.3330 ± 0.5065
RES2Fusion	1.0121 ± 0.0154	8.4371 ± 0.0797	13.2681 ± 0.1103
SeAFusion	0.0488 ± 0.3246	0.2966 ± 0.7789	0.1344 ± 0.5090
VAFNet	0.0054 ± 0.0007	0.0050 ± 0.0003	0.0053 ± 0.0005

shows the average running time of VAFNet and the other seven fusion methods in MSRS, TNO, and the collection dataset. All fusion methods run on a GPU. It can be seen that VAFNet not only has the best running efficiency, but also meets the real-time requirement of processing 30 frames per second (FPS) for object detection tasks [60]. This is because visual attention technology has the ability to process large amounts of information with extremely low energy consumption. In conclusion, VAFNet can serve as a reliable preprocessing module for object detection tasks.

5. Conclusions

To fill the gap between image fusion and object detection tasks, we proposed a novel visual attention fusion network for infrared and visible image fusion. Inspired by the human visual system, our network can achieve bottom-up and top-down inference to extract visual and object features, respectively. Most importantly, bottom-up and top-down processes work in a collaborative manner. In addition, we also present a bidirectional attention integration mechanism to naturally combine two forms of attention. Comparative and generalization experiments show that the proposed fusion method outperforms seven SOTA methods in both qualitative and quantitative evaluations. In addition, the evaluation results applied to surveillance and object detection tasks reveal that our fusion network cannot only comprehensively express imaging scenes but also effectively improve the performance of object detection tasks. In terms of computational efficiency, the average running time of VAFNet also meets real-time processing requirements. In the future, we will expand the training dataset collection to enhance VAFNet’s robustness in practical engineering applications.