An Improved Infrared and Visible Image Fusion Using an Adaptive Contrast Enhancement Method and Deep Learning Network with Transfer Learning

Abstract

Deep learning (DL) has achieved significant attention in the field of infrared (IR) and visible (VI) image fusion, and several attempts have been made to enhance the quality of the final fused image. It produces better results than conventional methods; however, the captured image cannot acquire useful information due to environments with poor lighting, fog, dense smoke, haze, and the noise generated by sensors. This paper proposes an adaptive fuzzy-based preprocessing enhancement method that automatically enhances the contrast of images with adaptive parameter calculation. The enhanced images are then decomposed into base and detail layers by anisotropic diffusion-based edge-preserving filters that remove noise while smoothing the edges. The detailed parts are fed into four convolutional layers of the VGG-19 network through transfer learning to extract features maps. These features maps are fused by multiple fusion strategies to obtain the final fused detailed layer. The base parts are fused by the PCA method to preserve the energy information. Experimental results reveal that our proposed method achieves state-of-the-art performance compared with existing fusion methods in a subjective evaluation through the visual experience of experts and statistical tests. Moreover, the objective assessment parameters are conducted by various parameters ($FMI$, $SSI{M}_a$, $API$, $\mathrm{EN}$, ${\mathrm{Q}}_{\mathrm{F}}^{\mathrm{AB}}$, and $N_F^{AB}$) which were used in the comparison method. The proposed method achieves 0.2651 to 0.3951, 0.5827 to 0.8469, 56.3710 to 71.9081, 4.0117 to 7.9907, and 0.6538 to 0.8727 gain for $FMI$, $SSI{M}_a$, $API$, $\mathrm{EN}$, and ${\mathrm{Q}}_{\mathrm{F}}^{\mathrm{AB}}$, respectively. At the same time, the proposed method has more noise reduction (0.3049 to 0.0021) that further justifies the efficacy of the proposed method than conventional methods.

1. Introduction

The individual image taken by IR or VI cannot extract all features due to the limitation of sensor systems. In particular, the IR images can extract thermal information about the foreground, such as hidden targets, camouflage identification, movement of objects, and so on [1]. Nonetheless, it cannot extract background information such as vegetation and soil due to different spectrums, resulting in the loss of spatial features [2]. In contrast, the VI images extract the background information, such as details of objects and edges, but they cannot provide information about thermal targets. The primary objective of image fusion is to acquire desired characteristics so that composite images can simultaneously produce background information and thermal targets perceptually. The IR and VI image fusion is a process of amalgamating both images to provide one final fused image that extracts all substantial features from both images without adding artifacts or noise. Recently, the fusion of IR and VI images has garnered a lot of attention as it has played a vital role in understanding image recognition and classification. It has opened a new path for many real-world applications, including object detection such as surveillance, remote sensing [3], vehicular ad hoc network (VANET), navigation, and so on [4,5,6]. Based on different applications, the fusion of images can be performed using pixel-level-, feature-level-, or decision-level-based fusion schemes [7]. In a pixel-based scheme, the pixels of both images are merged directly to obtain a final fused image, whereas a feature-based scheme first extracts the features from given images and then combines the extracted features [8]. On the contrary, the decision-based scheme extracts all the features from given images and makes decisions based on specific criteria to fuse the desired features [9].

For decades, numerous signal processing algorithms have been contemplated in image fusion to extract features such as multi-scale transform-based methods [10,11], sparse-representation-based [12,13], edge-preserving filtering [14,15], neural network [16], subspace-based [17] and hybrid-based methods [3]. During the last five years, deep learning (DL)-based [18] methods have received painstaking attention in image fusion due to their outstanding feature extraction from source images.

The PCA-based fusion method is employed in [17], which preserves energy features in the fused image. This scheme calculates the eigenvalues and eigenvectors for source images that use the highest eigenvector. Then, principal components are obtained from eigenvectors corresponding to the highest eigenvalues. This method helps to eliminate the redundant information with a robust operation. However, this method fails to extract detailed features such as edges, boundaries, and textures. The discrete cosine transform (DCT)-based image fusion algorithm is inferred by [18]. This method employs a correlation coefficient to calculate the change in the resultant image after passing from a low-pass filter. This method is unable to capture smooth contours and textures from source images. The improved discrete wavelet transform (DWT)-based fusion scheme is exploited in [19], which preserves high-frequency features with fast computation. This method decomposes the source image into one low-frequency sub-band and three high-frequency sub-bands (low-high sub-band, high-low sub-band, and high-high sub-band). Nevertheless, this method introduces artifacts and noise because it uses down sampling that causes shift variance, which degrades fused image quality. The author in [20] used the hybrid method using DWT and PCA, wherein DWT decomposes the source images and then decomposed images are fused by the PCA method. Although this method can preserve energy features and high-frequency features, it fails to capture sharp edges and boundaries due to limited directional information. Authors in [21] have used discrete stationary wavelet transform (DSWT) that decomposes the source images into one low-frequency sub-band and three high-frequency sub-bands (low-high sub-band, high-low sub-band, and high-high sub-band) without using down sampling. The resultant image produces better results for spectral information but fails to capture spatial features.

The contourlet transform-based method is applied in [22], which overcomes the issue of directional information. The edges and textures are captured precisely, but Gibb’s effect is introduced due to the property of shift variance. The non-sub-sampled contourlet transform (NSCT) is pinpointed in [11] that uses non-subsampled pyramid filter banks (NSPFB) and non-subsampled directional filter banks (NSDFB). It has a shift-invariant property and flexible frequency selectivity with fast computation. This method addresses the aforementioned issues, but the fused image suffers from uneven illumination, and the noise is generated due to the environment. The median average filter-based fusion scheme using hybrid DSWT and PCA is highlighted by Tian et al. in [23]. The combined filter is used to eliminate the noise, and then DSWT is applied that decomposes the images into sub-frequency bands. The sub-frequency bands are then fed to PCA for fusion. Nevertheless, this method has limited directional information, and it cannot preserve sharp edges and boundaries. The author in [24] has used a morphology-hat transform (MT) using a hybrid contourlet and PCA algorithm. The morphological processing top–bottom operation removes noise, while the energy features are preserved by principal component analysis (PCA), and edges are precisely obtained by contourlet transform. However, the fused image suffers due to the shift variance property that introduces Gibb’s effect, and hence the final image quality is degraded. The spatial frequency (SF)-based non-subsampled shearlet transform (NSST) along with pulse code neural network (PCNN) is designed by authors in [16]. The images are decomposed by NSST and these decomposed images are fed into a spatial frequency pulse code neural network (SF-PCNN) for fusion. This method produces better results due to shift-invariant and flexible frequency selectivity, but its image quality is affected due to uneven illumination. Hence, the fused image loses some energy information and edges are not captured smoothly.

The sparse representation (SR) has been employed in many applications in the last decade. This method learns an over-complete dictionary (OCD) from many high-quality images, and then the learned dictionary sparsely represents input images. The sparse-based schemes use the sliding window method to divide input images into various overlapping patches that help to reduce artifacts. The author in [12] constructed a fixed over complete discrete cosine transform dictionary to fuse IR and VI images. Kim et al. in [25] designed a scheme for learning a dictionary based on image patch clustering and the PCA. This scheme reduces the dimensions due to the robust operation of the PCA without affecting the fusion performance. The author in [26] applied an SR method and a histogram of gradient (HOG). The features obtained from the HOG are taken for classifying the image patches to learn various sub-dictionaries. The author in [27] applied a convolutional sparse representation (CSR) scheme. It decomposes the input images into base and detail layers. The detailed parts are fused by sparse coefficients obtained by utilizing the CSR model, and base parts are fused using the choose-max selection fusion rule. The final fused image is reconstructed by combining the base and detailed parts. This method has more ability to preserve detailed features than simple SR-based methods; however, this method uses a window-based averaging strategy that leads to the loss of essential features due to the value of the window size. The latent low-rank representation (LatLRR)-based fusion method is used in [28]. The source images are firstly decomposed into low rank and saliency parts by LatLRR. After that, a weighted averaging scheme fuses low-rank parts, while the sum strategy fuses the saliency parts. Finally, the fused image is acquired by merging the low-rank and saliency parts. This method captures smooth edges but fails to capture more energy information because it uses a weighted-averaging strategy that leads to losing some energy information.

Edge-preserving-based filtering methods have become an active research tool for many image fusion applications in the last two decades. The given images are decomposed by filtering into one base layer and more detailed layers. The base layer is acquired by an edge-preserving filter that accurately gets foremost changes in intensity. In contrast, the detail layers consist of a series of different images that acquire coarse resolution at fine scales. These filtering-based approaches retain spatial consistency and reduce the halo effects around textures and edges. The cross bilateral filtering (CBF) method is applied in [15]. This scheme calculates the weights by measuring the strength of the image by subtracting the CBF from the source images. After that, these weights are directly multiplied with source images using the normalization of weights. This direct multiplication leads to introducing artifacts and noise. The author in [29] designed an image decomposition filtering-based scheme based on L1 fidelity with L0 gradient. This method retains the features of the base layer, and it obtains good edges, textures, and boundaries around the objects. Zhou et al. in [30] used a new scheme using Gaussian and bilateral filters capable of decomposing the image multi-scale edges and retaining fine texture details.

The role of deep learning (DL) has received considerable attention in the last five years, and it has been employed in the field of image fusion [31,32,33]. The convolutional neural network (CNN)-based image fusion scheme is proposed in [34]. This method employs image patches consisting of various blur copies of the original image for training their network to acquire a decision map. The fused image is obtained using that decision map and the input images. A new fully CNN-based IR and VI fusion-based method is proposed in [35] that uses local non-subsampled shearlet transform (LNSST) to decompose the given images into low-frequency and high-frequency sub-bands. The high-frequency coefficients are fed into CNN to obtain the weight map. After that, the average gradient fusion strategy is used for these coefficients, while the local energy fusion scheme is used for low-frequency coefficients.

Table 1. Overall summarization of all methods in related work.

Methods	Transform	Fusion Strategy	Drawback
Multi-scale decomposition (MSD)	Discrete wavelet transform (DWT) [19], discrete stationary wavelet transform (DSWT) [21], discrete cosine transform (DCT) [18], contourlet transform [22], non-subsampled contourlet transform (NSCT) [11], and non-subsampled shearlet transform (NSST) [36]	Averaging, window [19], choose-max, [21] weighted averaging, activity-level measurement, region based, guided filtering based [37]	Fails to provide sufficient spatial information
Edge preserving-based filtering methods	Cross bilateral filtering (CBF) [15], guided filtering, mean-median filtering [23], Gaussian and bilateral filtering [30], isotropic diffusion, and anisotropic diffusion [14]	Choose-max [14], average gradient, local energy, and window based [34]	Introduces reversal artifacts and halo effects
Sparse representation (SR) methods	Orthogonal matching pursuit (OMD) [25], joint sparsity model, over-complete dictionary (OCD) [25], and simultaneous OMD [27]	Window based [19], averaging, choose-max, and the substitution of sparse coefficients [26]	Limited ability in detail preservation, dimension disaster, and high sensitivity to miss-registration
Deep learning-based methods	Convolutional neural network (CNN) [38,39], VGG-19, residual network (ResNet), full convolutional networks (FCN) [35], and Siamese network	Averaging, activity-level measurement [24], normalization method, average gradient, and local similarity strategy [3]	Limited ability to restore spatial information
Hybrid methods	Contourlet transform with principal component analysis (PCA) [24], mean-median-based DSWT with PCA [23], morphological method with contourlet transform, NSCT with pulse code neural network (PCNN) [16], and sparse-based convolutional neural network (CSR) [27]	Window, averaging, choose-max [38], normalization scheme, activity level measurement, and average gradient [40]	Miss-alignment, noise is generated due to environment or fusion, and the complexity of architecture

presents the overall summary of related work.

In a nutshell, the main motive of any fusion method is to extract all useful features from given images to produce a composite image that comprises all salient features from both individual images without introducing artifacts or noise. Though various fusion algorithms fulfill the vital motive of image fusion by producing one final fused image with the aid of combining the features from individual images, the quality of the final fused image is still not up to the mark. As a result, the quality of the final fused image is still degraded due to contrast variation in the background, uneven illumination, environments with fog, dust, haze, and dense smoke, the presence of noise, and improper fusion strategy. In such a context, this work bridges the technological gap of recent works by proposing an improved image fusion scheme, thereby overcoming the aforementioned shortcomings. To the best of our knowledge, this is the first time that an image fusion scheme using an adaptive fuzzy-based enhancement method and a deep convolutional network model through the guidance of transfer learning with multiple fusion strategies is employed. The proposed hybrid method outperforms the traditional and recent fusion methods in subjective evaluation and objective quality assessments.

The main contributions of this paper are outlined as:

• The robust and adaptive fuzzy set-based image enhancement method is applied as a preprocessing task that automatically enhances the contrast of IR and VI images for better visualization with adaptive parameter calculation. The enhanced images are then decomposed into base and detail layers using an anisotropic diffusion-based method.
• The resultant detailed parts are fed to four convolutional layers of the VGG-19 network through transfer learning to extract the multi-layer features. Afterward, $L_{1-norm}$ and averaging-based fusion strategies along with the max selection rule are applied to obtain the fused detail contents.
• The base parts are fused by the principal component analysis (PCA) method to preserve the local energy from base parts. At last, the final fused image is obtained by a linear combination of base and detailed parts.

The outline of this paper is as follows. Section 2 presents the materials and methods. The results are summarized in Section 3. The discussion is elaborated in Section 4. Finally, Section 5 draws the conclusion and future work.

2. Materials and Methods

The main motive of any image fusion method is to acquire a final fused image that extracts all the useful features from individual source images by eliminating the redundant information without introducing artifacts and noise. Many recent image fusion methods have been proposed so far, and they achieve satisfactory results compared with traditional fusion schemes. Nevertheless, the fused image suffers from contrast variation, uneven illumination, blurring effects, noise, and improper fusion strategies. Moreover, the input images taken in environments with dense fog, haze, dust, cloud, and improper illumination affect the quality of the final fused image. We propose an effective image fusion method that circumvents the mentioned shortcomings.

The proposed method first uses a preprocessing contrast enhancement technique named adaptive fuzzy set-based enhancement (FSE). This method automatically enhances the contrast of IR and VI images for better visualization with adaptive parameter calculation. These enhanced images are decomposed into base and detail parts by the anisotropic diffusion method using the partial differential equation. This method prevents diffusion across the edges; it removes the noise and preserves the edges. The PCA method is employed for a fusion of base parts. It can preserve the local energy information while eliminating the redundant data with its robust operation. The four convolution layers of the VGG-19 network through the guidance of transfer learning are applied to detailed parts. This model extracts all the useful salient features with better visual perception, while it saves time and resources during the training without requiring a lot of data. Furthermore, the $L_{1-norm}$ averaging fusion strategy and choose-max selection rule removes the noise in smooth regions while restoring the edges precisely.

The proposed method consists of a series of sequential steps presented in Figure 1. Each step in the proposed method performs a specific task to improve the fused image quality. We will elaborate on each step in detail in sub-sections.

2.1. Fuzzy Set-Based Enhancement (FSE) Method

Capturing the IR and VI image is quite a simple task; however, it is challenging for the human visual system (HVS) to obtain all information due to environments with dense fog, haze, cloud, poor illumination, and noise. In addition, IR and the visible images taken at night vision have low contrast, and they have more ambiguity and uncertainty in the pixel values that arise during the sampling and quantization process. The pixel intensity evaluation needs to be carried out if the image has low contrast. The fuzzy domain is the best approach to alleviate the above-mentioned shortcomings. We have used the FSE method that improves the visualization of IR and VI automatically. This method is adaptive and robust for the parameter calculation of image enhancement.

The fuzzy set for an image I of $\mathrm{a}\times \mathrm{b}$ size with L gray levels each having membership value is given as:

$$\mathrm{I}={\displaystyle \underset{\mathrm{i}=1}{\overset{\mathrm{a}}{\cup }}{\displaystyle \underset{\mathrm{j}=1}{\overset{\mathrm{b}}{\cup }}\frac{{\mathsf{\mu }}_{\mathrm{i}\mathrm{j}}}{{\mathrm{x}}_{\mathrm{i}\mathrm{j}}}}}$$

(1)

where, ${\mathsf{\mu }}_{\mathrm{i}\mathrm{j}}$ represents the membership function for each value of intensity at $\mathrm{i},\mathrm{j}$, and ${\mathrm{x}}_{\mathrm{i}\mathrm{j}}$ is the intensity value of each pixel. The membership function typifies the appropriate property of images such as edges, brightness, darkness, and textural that can be described globally for the whole image. The block diagram for FSE is given in Figure 2.

We will discuss each step that is involved in FSE in detail as follows:

2.1.1. Fuzzification

The IR or VI images are fed as input I. Then, the image is transformed from spatial domain to fuzzy domain by Equation (2).

$${\mathsf{\mu }}_{\mathrm{i}\mathrm{j}}={[1+\frac{{\mathrm{x}}_{\max }-{\mathrm{x}}_{\mathrm{i}\mathrm{j}}}{{\mathrm{F}}_{\mathrm{d}}}]}^{-{\mathrm{F}}_{\mathrm{e}}}$$

(2)

where, ${\mathrm{x}}_{\max }$ indicates the maximum intensity level in an image I, whereas ${\mathrm{F}}_{\mathrm{d}}$ and ${\mathrm{F}}_{\mathrm{e}}$ represent the fuzzifier dominator and exponential.

The importance of fuzzifiers given in Equation (2) is to reduce the ambiguity and uncertainty in a given image. Many of the researchers have chosen the value of exponential fuzzifier as 2. Instead of selecting a static value for ${\mathrm{F}}_{\mathrm{e}}$, we have determined the adaptive calculation. As we used IR and the VI images (VI taken at night) that have a poor resolution, the general trials using the static value of ${\mathrm{F}}_{\mathrm{e}}$ cannot acquire good results. Therefore, keeping the above considerations, we compute the value of ${\mathrm{F}}_{\mathrm{e}}$ using the sigmoid [41] function that is calculated by Equation (3):

$${\mathrm{F}}_{\mathrm{e}}=\frac{1}{(1+{\exp }^{(-\mathrm{m})})}$$

(3)

where, m indicates the mean value of an image.

The dominator fuzzifier ${\mathrm{F}}_{\mathrm{d}}$ is obtained as:

$${\mathrm{F}}_{\mathrm{d}}=\frac{{\mathrm{x}}_{\max }-{\mathrm{x}}_{\min }}{{0.5}^{\frac{-1}{{\mathrm{F}}_{\mathrm{e}}}}-1}$$

(4)

2.1.2. Modification of Membership Function

The contrast intensification operator (CIO) is applied to ${\mathsf{\mu }}_{\mathrm{i}\mathrm{j}}$ to modify the membership function value that is computed by Equation (5):

$$\begin{array}{l}{\mathsf{\mu }}_{\mathrm{ij}}^{\prime }=\left\{\begin{array}{ll}2{[{\mathsf{\mu }}_{\mathrm{i}\mathrm{j}}]}^2& 0\le {\mathsf{\mu }}_{\mathrm{i}\mathrm{j}}\le {\mathrm{x}}_{\mathrm{c}}\\ 1-2{[1-{\mathsf{\mu }}_{\mathrm{i}\mathrm{j}}]}^2& {\mathrm{x}}_{\mathrm{c}}\le {\mathsf{\mu }}_{\mathrm{i}\mathrm{j}}\le 1\end{array}\right\}\end{array}$$

(5)

The ${\mathrm{x}}_{\mathrm{c}}$ is the cross-over point that ascertains the value of ${\mathsf{\mu }}_{\mathrm{ij}}^{\prime }\ne {\mathrm{x}}_{\mathrm{c}}$. There is a monotonical increment in the ${\mathsf{\mu }}_{\mathrm{i}\mathrm{j}}$ value because of the incremental ${\mathrm{x}}_{\mathrm{c}}$ value from 0 to 1 due to CIO, and we have set 0.5 as the value for ${\mathrm{x}}_{\mathrm{c}}$.

2.1.3. Defuzzification

It is a phenomenon of transforming modified fuzzification data from fuzzy domain to spatial domain using Equation (6):

$${\mathrm{D}}_{\mathrm{i}\mathrm{j}}={\mathrm{x}}_{\max }-{\mathrm{F}}_{\mathrm{d}}\times ({({\mathsf{\mu }}_{\mathrm{ij}}^{\prime })}^{\frac{-1}{{\mathrm{F}}_{\mathrm{e}}}})+{\mathrm{F}}_{\mathrm{d}}$$

(6)

where ${\mathsf{\mu }}_{\mathrm{ij}}^{\prime }$ indicates the modified member function.

2.2. Anisotropic Diffusion

The anisotropic diffusion (AD) is a way to decompose the source image into a base layer and detailed layer used for image fusion in this paper. This method employs the intra region smoothing procedure to obtain coarse resolution images; hence, the edges are sharper at each coarse resolution. This scheme will acquire the sharp edges while smoothing the homogenous areas more precisely by employing partial differential equations (PDE). An AD employs flux function to ascend the image diffusion that can be computed by Equation (7):

$${\mathrm{I}}_{\mathrm{t}}=\mathrm{R}(\mathrm{x},\mathrm{y},\mathrm{t})\mathrm{\Delta }\mathrm{I}+\nabla \mathrm{R}.\nabla \mathrm{I}$$

(7)

where ${\mathrm{I}}_{\mathrm{t}}$ represents the image diffusion, $\mathrm{R}(\mathrm{x},\mathrm{y},\mathrm{t})$ = rate of diffusion, $\nabla$ = gradient operator, $\mathrm{\Delta }$ = Laplacian operator, and $\mathrm{t}$ is iteration.

We will use forward time central space (FTCS) to solve the PDE. The reason to use FTCS is to maintain diffusion stability, which can be assured by the maximum principle [42]. The equation to resolve the PDE can be computed by Equation (8) as:

$${\mathrm{I}}_{\mathrm{i},\mathrm{j}}^{\mathrm{t}+1}={\mathrm{I}}_{\mathrm{i},\mathrm{j}}^{\mathrm{t}}+\mathrm{\lambda }[{\mathrm{R}}_{\mathrm{N}}.{\overline{\nabla }}_{\mathrm{N}}{\mathrm{I}}_{\mathrm{i},\mathrm{j}}^{\mathrm{t}}+{\mathrm{R}}_{\mathrm{S}}.{\overline{\nabla }}_{\mathrm{S}}{\mathrm{I}}_{\mathrm{i},\mathrm{j}}^{\mathrm{t}}+{\mathrm{R}}_{\mathrm{W}}.{\overline{\nabla }}_{\mathrm{W}}{\mathrm{I}}_{\mathrm{i},\mathrm{j}}^{\mathrm{t}}+{\mathrm{R}}_{\mathrm{E}}.{\overline{\nabla }}_{\mathrm{E}}{\mathrm{I}}_{\mathrm{i},\mathrm{j}}^{\mathrm{t}}]$$

(8)

where $\mathrm{\lambda }$ represents the stability constant in the range $0\le \mathrm{\lambda }\le 0.25$, ${\mathrm{D}}_{\mathrm{i},\mathrm{j}}^{\mathrm{t}+1}$ indicates the image coarse resolution in forward time at $\mathrm{t}+1$ iteration that relies on former ${\mathrm{I}}_{\mathrm{i},\mathrm{j}}^{\mathrm{t}}$. The ${\overline{\nabla }}_{\mathrm{N}},{\overline{\nabla }}_{\mathrm{S}},{\overline{\nabla }}_{\mathrm{E}}$, and ${\overline{\nabla }}_{\mathrm{W}}$ show the neighboring differences in the north, south, east, and west directions, and it can be computed by Equation (9):

$$\left\{\begin{array}{c}{\overline{\nabla }}_{\mathrm{N}}{\mathrm{I}}_{\mathrm{i},\mathrm{j}}={\mathrm{I}}_{\mathrm{i}-1,\mathrm{j}}-{\mathrm{I}}_{\mathrm{i},\mathrm{j}},\\ {\overline{\nabla }}_{\mathrm{S}}{\mathrm{I}}_{\mathrm{i},\mathrm{j}}={\mathrm{I}}_{\mathrm{i}+1,\mathrm{j}}-{\mathrm{I}}_{\mathrm{i},\mathrm{j}},\\ {\overline{\nabla }}_{\mathrm{E}}{\mathrm{I}}_{\mathrm{i},\mathrm{j}}={\mathrm{I}}_{\mathrm{i},\mathrm{j}+1}-{\mathrm{I}}_{\mathrm{i},\mathrm{j}},\\ {\overline{\nabla }}_{\mathrm{W}}{\mathrm{I}}_{\mathrm{i},\mathrm{j}}={\mathrm{I}}_{\mathrm{i},\mathrm{j}-1}-{\mathrm{I}}_{\mathrm{i},\mathrm{j}}\end{array}\right\}$$

(9)

Similarly, the flux functions ${\mathrm{R}}_{\mathrm{N}},{\mathrm{R}}_{\mathrm{S}},{\mathrm{R}}_{\mathrm{E}}$, and ${\mathrm{R}}_{\mathrm{W}}$ will be updated at each iteration in all four directions as:

$$\left\{\begin{array}{c}{\mathrm{R}}_{{\mathrm{N}}_{\mathrm{i},\mathrm{j}}}^{\mathrm{t}}=\mathrm{h}(\Vert (\nabla \mathrm{I}{)}_{\mathrm{i}+1/2,\mathrm{j}}^{\mathrm{t}}\Vert )=\mathrm{h}(|{\overline{\nabla }}_{\mathrm{N}}{\mathrm{I}}_{\mathrm{i},\mathrm{j}}^{\mathrm{t}}|),\\ {\mathrm{R}}_{{\mathrm{S}}_{\mathrm{i},\mathrm{j}}}^{\mathrm{t}}=\mathrm{h}(\Vert (\nabla \mathrm{I}{)}_{\mathrm{i}-1/2,\mathrm{j}}^{\mathrm{t}}\Vert )=\mathrm{h}(|{\overline{\nabla }}_{\mathrm{S}}{\mathrm{I}}_{\mathrm{i},\mathrm{j}}^{\mathrm{t}}|),\\ {\mathrm{R}}_{{\mathrm{E}}_{\mathrm{i},\mathrm{j}}}^{\mathrm{t}}=\mathrm{h}(\Vert (\nabla \mathrm{I}{)}_{\mathrm{i},\mathrm{j}+1/2}^{\mathrm{t}}\Vert )=\mathrm{h}(|{\overline{\nabla }}_{\mathrm{E}}{\mathrm{I}}_{\mathrm{i},\mathrm{j}}^{\mathrm{t}}|),\\ {\mathrm{R}}_{{\mathrm{W}}_{\mathrm{i},\mathrm{j}}}^{\mathrm{t}}=\mathrm{h}(\Vert (\nabla \mathrm{I}{)}_{\mathrm{i},\mathrm{j}-1/2}^{\mathrm{t}}\Vert )=\mathrm{h}(|{\overline{\nabla }}_{\mathrm{W}}{\mathrm{I}}_{\mathrm{i},\mathrm{j}}^{\mathrm{t}}|),\end{array}\right\}$$

(10)

The $\mathrm{h}(.)$ in Equation (10) is the monotonical decrement function with $\mathrm{h}(0)=1$, and can be computed as:

$$\mathrm{h}(\nabla \mathrm{I})=\frac{1}{1+{(\frac{\Vert \nabla \mathrm{I}\Vert }{\mathrm{p}})}^2}$$

(11)

$$\mathrm{h}(\nabla \mathrm{I})={\mathrm{e}}^{-{(\frac{\Vert \nabla \mathrm{I}\Vert }{\mathrm{p}})}^2}$$

(12)

where p represents a free parameter used to determine the validity of the region boundary based on its edge intensity level.

Equation (11) is more useful if a given image contains wide regions instead of narrow regions. Wherein, Equation (12) is more flexible if an image has high-edge resolution than poor-edge resolution; however, the parameter p is fixed.

Let the input images ${\mathrm{I}}_{\mathrm{n}}{(\mathrm{x},\mathrm{y})}_{\mathrm{n}=1}$ having $\mathrm{p}\times \mathrm{q}$ size be fed into AD that are well-aligned and co-registered. These images are infiltrated into the edge-preserving smoothing AD process for acquiring the base layer, and it is computed as Equation (13):

$${\mathrm{B}}_{\mathrm{n}}(\mathrm{x},\mathrm{y})={\mathrm{A}}_{\mathrm{d}\mathrm{f}}({\mathrm{I}}_{\mathrm{n}}(\mathrm{x},\mathrm{y}))$$

(13)

where ${\mathrm{A}}_{\mathrm{d}\mathrm{f}}({\mathrm{I}}_{\mathrm{n}}(\mathrm{x},\mathrm{y}))$ indicates the AD process for the n-th base layer, while ${\mathrm{B}}_{\mathrm{n}}(\mathrm{x},\mathrm{y})$ is the acquired n-th base layer. After acquiring the base layer, the detail layer is computed by subtracting the base layer from the source image as given in Equation (14):

$${\mathrm{D}}_{\mathrm{n}}(\mathrm{x},\mathrm{y})={\mathrm{I}}_{\mathrm{n}}(\mathrm{x},\mathrm{y})-{\mathrm{B}}_{\mathrm{n}}(\mathrm{x},\mathrm{y})$$

(14)

It can be seen in Figure 3 that AD is employed in an enhanced image obtained from the FSE scheme. The AD decomposes the image into the base layer and detail layer, and their base and detail images are shown in Figure 3. We have applied AD to only one enhanced image of FSE, and we will apply the AD to another image in the same pattern to obtain a base and detail layer for another image.

2.3. Fusion Strategy of Base Layers

The base layers contain the energy information, and it is vital to preserve the energy information from both base layers to extract more features. Though the average fusion scheme can be applied, it cannot preserve the complete information. The PCA is utilized for the fusion of low-frequency images that is a dimension reduction technique that minimizes the complexity of a model and avoids overfitting. This method compresses a dataset into a lower-dimensional feature subspace while preserving relevant information at a high level of detail. It takes a subset of the original features and uses that knowledge to create a new feature subspace. It locates the most elevated variance directions in high-dimensional data and projects them onto a new subspace with the same or fewer dimensions as the original. The PCA transforms correlated variables into uncorrelated forms that choose features with significant values of eigenvalues $\mathrm{\lambda }$ and eigenvectors $\mathrm{\xi }$. After that, the principal components ${\mathrm{U}}_{\mathrm{C}1}$ and ${\mathrm{U}}_{\mathrm{C}2}$ are computed from these highest eigenvalues $\mathrm{\lambda }$ and eigenvectors $\mathrm{\xi }$. All the principal components are orthogonal to each other, which helps to eliminate redundant information and makes fast computation. The PCA method used for the fusion of base layers is discussed below.

1. The proposed method is applicable for a fusion of two, or more than two, base layers; however, we consider only two base layers for simplicity. Let ${\mathrm{B}}_1(\mathrm{x},\mathrm{y})$ and ${\mathrm{B}}_2(\mathrm{x},\mathrm{y})$ be the two base layers obtained after decomposition from the AD process. The first step is to acquire a column matrix, Z, from these two base layers.

2. Obtain the variance V and covariance ${\mathrm{C}}_{\mathrm{v}}$ matrix ${\mathrm{C}}_{\mathrm{Z}}$ from column matrix Z for both base layers ${\mathrm{B}}_1(\mathrm{x},\mathrm{y})$ and ${\mathrm{B}}_2(\mathrm{x},\mathrm{y})$. The V and ${\mathrm{C}}_{\mathrm{v}}$ is calculated by Equation (15):

$$\mathrm{Z}=\left[\begin{array}{cc}{\mathrm{V}}_1& {\mathrm{C}}_{\mathrm{v}}(1,2)\\ {\mathrm{C}}_{\mathrm{v}}(1,2)& {\mathrm{V}}_2\end{array}\right]$$

(15)

$${\mathrm{C}}_{\mathrm{v}}=\frac{({\displaystyle \sum ({\mathrm{x}}_1-{\overline{\mathrm{x}}}_1)\times ({\mathrm{x}}_2-{\overline{\mathrm{x}}}_2)})}{\mathrm{T}-1}$$

(16)

In Equations (15) and (16), ${\mathrm{V}}_1,{\mathrm{V}}_2,{\mathrm{C}}_{\mathrm{V}}(1,2)$ indicates the variance and covariance, whereas T represents the number of iterations.

3. After obtaining the variance and co-variance, the next step is to obtain the eigen values $\mathrm{\lambda }$ and eigen vectors $\mathrm{\xi }$. The $\mathrm{\lambda }$ is computed by Equation (17):

$$\mathrm{D}\mathrm{e}\mathrm{t}\left|\mathrm{Z}-\mathrm{\lambda }\mathrm{I}\right|=0$$

(17)

$$(({\mathrm{V}}_1-\mathrm{\lambda })\times ({\mathrm{V}}_2-\mathrm{\lambda }))\times {\mathrm{C}}_{\mathrm{v}}{(1,2)}^2=0$$

(18)

Now we compute the $\mathrm{\xi }$ by Equation (19):

$${\mathrm{\xi }}_1=\left[\begin{array}{c}{\mathrm{\xi }}_1(1)\\ {\mathrm{\xi }}_2(2)\end{array}\right] \mathrm{and} {\mathrm{\xi }}_2=\left[\begin{array}{c}{\mathrm{\xi }}_2(1)\\ {\mathrm{\xi }}_2(2)\end{array}\right]\mathrm{for} {\mathrm{C}}_{\mathrm{Z}}$$

(19)

4. The next step is to determine if the uncorrelated components ${\mathrm{U}}_{\mathrm{c}1}$ and ${\mathrm{U}}_{\mathrm{c}2}$ match the highest eigenvalues $({\mathrm{\lambda }}_{\max }=\max ({\mathrm{\lambda }}_1,{\mathrm{\lambda }}_2))$. Let ${\mathrm{\xi }}_{\max }$ represent the eigenvector corresponding to ${\mathrm{\lambda }}_{\max }$, then ${\mathrm{U}}_{\mathrm{c}1}$ and ${\mathrm{U}}_{\mathrm{c}2}$ are computed as:

$${\mathrm{U}}_{\mathrm{c}1}=\frac{{\mathrm{\xi }}_{\max }(1)}{{\displaystyle \sum _{\mathrm{n}}{\mathrm{\xi }}_{\max }(\mathrm{n})}},{\mathrm{U}}_{\mathrm{c}2}=\frac{{\mathrm{\xi }}_{\max }(2)}{{\displaystyle \sum _{\mathrm{n}}{\mathrm{\xi }}_{\max }(\mathrm{n})}}$$

(20)

5. The final fused base layer can be calculated by Equation (21) as:

$$\overline{\mathrm{B}}(\mathrm{x},\mathrm{y})={\mathrm{U}}_{\mathrm{c}1}{\mathrm{B}}_1(\mathrm{x},\mathrm{y})+{\mathrm{U}}_{\mathrm{c}2}{\mathrm{B}}_2(\mathrm{x},\mathrm{y})$$

(21)

The block diagram for a fusion of base layers by PCA method is shown in Figure 4. The two base layers that are decomposed by AD are fed to PCA and the final fused base layer is obtained after the fusion process of PCA. The symbols ${\mathrm{U}}_{\mathrm{c}1}$ and ${\mathrm{U}}_{\mathrm{c}2}$ in Figure 4 represent the uncorrelated components for both base layers. These uncorrelated components are obtained from the highest eigenvalues and eigenvectors that are orthogonal to each other, and its computation would be fast due to the elimination of redundant data.

2.4. Fusion Strategy for Detail Layers

Most traditional methods use the choose-maximum fusion rule to extract the deep features. Nonetheless, this fusion scheme significantly loses detailed information that affects the overall fused image quality. In this paper, we extract the deep features by four convolutional layers of VGG-19 through the guidance of transfer learning. The activity level maps are acquired from desired extracted deep features by $L_{1-norm}$ and the averaging method. Then, soft-max and up-sampling are employed to these activity level maps to generate the weight maps. After that, these weight maps are convolved with IR and VI images to obtain the four candidates. Finally, the maximum strategy is applied to these four candidates to obtain the final fused detailed layer. The whole phenomena for a fusion of detailed layers are shown in Figure 5.

We will discuss each step more precisely in sub-sections.

2.4.1. Feature Extraction by VGG-19 through Transfer Learning

We have used a VGG-19 network that consists of convolution layers, ReLU (rectified linear unit) layers, pooling layers, and fully connected layers [43]. The main purpose for using the VGG-19 network is its feasibility which has a $3\times 3$ receptive field for each convolution layer with a fixed stride of 1 pixel and spatial padding of 1 pixel. We took only four convolution layers from VGG-19 through the guidance of transfer learning. Transfer learning helps to reduce the training effort. We took four convolution layers (conv1-1, conv2-1, conv3-1, and conv4-1) and chose the first convolution layer from each convolution group because the first convolution layer contains complete significant information.

Let $\mathsf{\beta }$ represent the number of detailed images, and we took two detailed images that are ${\mathrm{X}}_1^{\mathrm{d}}$ and ${\mathrm{X}}_2^{\mathrm{d}}$. Assume that ${\mathsf{\phi }}_{\mathsf{\beta }}^{\mathrm{i},\mathrm{m}}$ represents the feature map of the $\mathsf{\beta }-\mathrm{t}\mathrm{h}$ detailed layer at $\mathrm{i}-\mathrm{t}\mathrm{h}$ convolution layer, and m shows the number of channels of the $\mathrm{i}-\mathrm{t}\mathrm{h}$ layer, so $\mathrm{m}\in \{1,2,.....,\mathrm{M}\}$. The M is the maximum number of channels, and it can be calculated as $64\times 2^{\mathrm{i}-1}$ by Equation (22).

$${\mathsf{\phi }}_{\mathsf{\beta }}^{\mathrm{i},\mathrm{m}}={\mathrm{\Phi }}_{\mathrm{i}}({\mathrm{X}}_{\mathsf{\beta }}^{\mathrm{d}})$$

(22)

Each ${\mathrm{\Phi }}_{\mathrm{i}}(.)$ represents a layer in VGG-19, and we chose four layers, i.e., $\mathrm{i}\in \{1,2,3,4\}$ conv1-1, conv2-1, conv3-1, and conv4-1.

Suppose ${\mathsf{\phi }}_{\mathsf{\beta }}^{\mathrm{i},1:\mathrm{M}}(\mathrm{x},\mathrm{y})$ indicate the contents of ${\mathsf{\phi }}_{\mathsf{\beta }}^{\mathrm{i},\mathrm{m}}$ at $(\mathrm{x},\mathrm{y})$ position, and ${\mathsf{\phi }}_{\mathsf{\beta }}^{\mathrm{i},1:\mathrm{M}}(\mathrm{x},\mathrm{y})$ is an M-dimensional vector. Consequently, the phenomena of our fusion strategy is elaborated in the next section, which is also shown in Figure 5.

2.4.2. Feature Processing by Multiple Fusion Strategies

Inspired by [44], we employ the $L_{1-norm}$ method to quantify the amount of information in the detailed images, which is called the activity level measurement of detailed images. After acquiring the feature maps ${\mathsf{\phi }}_{\mathsf{\beta }}^{\mathrm{i},\mathrm{m}}$, the next step is to find the initial and final activity level maps. The initial activity level map ${\mathsf{\delta }}_{\mathsf{\beta }}^{\mathrm{i}}$ is obtained by using the $L_{1-norm}$ scheme that is computed as given in Equation (23) and depicted in Figure 5:

$${\mathsf{\delta }}_{\mathsf{\beta }}^{\mathrm{i}}={\Vert {\mathsf{\phi }}_{\mathsf{\beta }}^{\mathrm{i},1:\mathrm{M}}(\mathrm{x},\mathrm{y})\Vert }_1$$

(23)

The next step is to find the final activity level map ${\widehat{\mathsf{\delta }}}_{\mathsf{\beta }}^{\mathrm{i}}$. The ${\widehat{\mathsf{\delta }}}_{\mathsf{\beta }}^{\mathrm{i}}$ is computed by Equation (24):

$${\widehat{\mathsf{\delta }}}_{\mathsf{\beta }}^{\mathrm{i}}=\frac{{\displaystyle \sum _{\mathrm{u}=-\mathrm{r}}^{\mathrm{r}}{\displaystyle \sum _{\mathrm{v}=-\mathrm{r}}^{\mathrm{r}}{\mathsf{\delta }}_{\mathsf{\beta }}^{\mathrm{i}}(\mathrm{x}+\mathrm{u},\mathrm{y}+\mathrm{v})}}}{{(2\mathrm{r}+1)}^2}$$

(24)

where r represents the size of the block, which helps our fusion method avoid misregistration. The larger value of r makes fusion more robust to prevent misregistration. Though some detail could be lost by choosing a higher value of r, we need to determine the value of r carefully. In our case, we have selected $\mathrm{r}=1$ because it preserves all information at this value while it also avoids misregistration.

It can be seen in Figure 5 that we have applied a soft-max that is used for computing the weight map ${\mathrm{W}}_{\mathsf{\beta }}^{\mathrm{i}}$ from ${\widehat{\mathsf{\delta }}}_{\mathsf{\beta }}^{\mathrm{i}}$. The ${\mathrm{W}}_{\mathsf{\beta }}^{\mathrm{i}}$ is calculated by Equation (25):

$${\mathrm{W}}_{\mathsf{\beta }}^{\mathrm{i}}(\mathrm{x},\mathrm{y})=\frac{{\widehat{\mathsf{\delta }}}_{\mathsf{\beta }}^{\mathrm{i}}(\mathrm{x},\mathrm{y})}{{\displaystyle \sum _{\mathrm{n}=1}^{\mathrm{\gamma }}{\widehat{\mathsf{\delta }}}_{\mathsf{\beta }}^{\mathrm{i}}(\mathrm{x},\mathrm{y})}}$$

(25)

where $\mathrm{\gamma }$ indicates the number of activity level maps. As we have taken two source images, therefore, $\mathrm{\gamma }=2$ in our case, we will obtain four pairs of initial weight maps to distinguish scales because we have chosen four convolutional layers through transfer learning.

The pooling layer is used in the VGG-19 network that acts as a sub-sampling; hence, it reduces the size of feature maps. Therefore, after every pooling layer, the feature maps are resized to $\raisebox{1ex}{$1$}\!\left/ \!\raisebox{-1ex}{$\mathrm{s}$}\right.$ times where s denotes the pooling stride, and the stride of the pooling operator is 2. Thus, the size of the feature maps would be $\frac{1}{2^{\mathrm{i}-1}}$ for each different layer of the convolution group, so $\mathrm{i}\in \{1,2,3,4\}$ for the first, second, third, and fourth convolution layer in each convolution group. Therefore, after obtaining the initial weight maps, we use the up-sampling operator to resize the weight map size that equals the size of detail content. The final weight maps using the up-sampling operator can be computed by Equation (26):

$${\widehat{\mathrm{W}}}_{\mathsf{\beta }}^{\mathrm{i}}(\mathrm{x}+\mathrm{p},\mathrm{y}+\mathrm{q})={\mathrm{W}}_{\mathsf{\beta }}^{\mathrm{i}}(\mathrm{x},\mathrm{y}),\mathrm{p},\mathrm{q}\in \{0,1,\dots (2^{\mathrm{i}-1}-1)\}$$

(26)

Now, the four weight maps that are acquired by the up-sampling operator are convolved with the detail layer of IR and VI images to obtain the candidates, and it is computed by Equation (27):

$${\mathrm{D}}^{\mathrm{i}}(\mathrm{x},\mathrm{y})={\displaystyle \sum _{\mathrm{n}=1}^{\mathrm{\gamma }}{\widehat{\mathrm{W}}}_{\mathrm{n}}^{\mathrm{i}}(\mathrm{x},\mathrm{y})\times }{\mathrm{X}}_{\mathrm{n}}^{\mathrm{d}}(\mathrm{x},\mathrm{y}),\mathrm{\gamma }=2$$

(27)

The fused detail content $\overline{\mathrm{D}}$ is finally computed by employing a choose-max value from four candidate fused images, as shown in Figure 5 and Equation (28).

$$\overline{\mathrm{D}}(\mathrm{x},\mathrm{y})=\max \left|{\mathrm{D}}^{\mathrm{i}}(\mathrm{x},\mathrm{y})\right|,\mathrm{i}\in \{1,2,3,4\}$$

(28)

2.5. Reconstruction

The reconstruction of the final fused image is obtained by simple linear superposition of the fused base image and the fused detailed image as given in Equation (29).

$$\mathrm{F}(\mathrm{x},\mathrm{y})=\overline{\mathrm{B}}(\mathrm{x},\mathrm{y})+\overline{\mathrm{D}}(\mathrm{x},\mathrm{y})$$

(29)

3. Results

The experiments are conducted on IR and VI datasets that prove the efficacy of the proposed method over conventional and recent image fusion methods. Though there is no universal evaluation standard for comparing the image fusion quality, subjective and objective evaluation is the best approach to compare the results of image fusion methods accurately. We took all the IR and VI from the TNO dataset [45], and all the images are preregistered that can be directly used for fusion. We compared our proposed method with the cross bilateral fusion (CBF) [15] scheme, discrete cosine transform (DCT)-based [18] method, improved DWT [19] fusion method, convolution sparse representation (CSR) [27], and latent low-rank representation (LatLRR)-based [28] methods. The simulation results are implemented using MATLAB 2016b on a core i5 3.2 GHz processor with 8 GB RAM.

3.1. Subjective Evaluation

The subjective evaluation is a good approach to compare the difference among different image fusion methods by the human visual system (HVS). The quality of images can be judged based on image brightness, contrast, and other features that can reflect the visual effect of the image.

It can be seen in Figure 6c that the fused image acquired by CBF has more artifacts and the image details are not clear. This is because the CBF method obtains weight by directly measuring the strength of horizontal and vertical details, leading to artifacts and noise. The fused image obtained by the DCT method in Figure 6d is also affected by gradient reversal artifacts, and it contains halo effects. Since the DCT method uses cross-correlation using a low filter that can capture the energy information from source images, the detailed information cannot be preserved, such as edges and contours that affect the overall quality of a fused image. The fused image produced by improved DWT in Figure 6e has better visual effects than CBF and DCT, but the image is blurred, and the details are unclear. It is well known that the DWT method has limited ability to preserve detailed information because it has limited directional information. Moreover, it cannot preserve more spatial (energy information) details because this method fails to capture energy information; due to that, the image is blurred. The fused result obtained by the CSR-based method in Figure 6f produces good results, and it is less noisy, but the overall information in the image is still blurred, and characters on billboards are not vivid, as can be seen in the red box. Since the CSR method used an averaging strategy to avoid insensitivity to misregistration, it uses a window with size r. The window size r results in losing some spatial details because the energy and edges of objects have different sharpness. The LatLRR-based method in Figure 6g achieves good results, and the letters on the billboard are also more vivid than existing methods, but the contrast of an image is still not satisfactory. It is due to using the weighted averaging strategy for a fusion of low-rank, and it is well known that the weighted-averaging approach is limited to preserve more spatial information. The fused image acquired by the proposed method has high quality in terms of detail extraction such as edges and contours due to the novel fusion strategy of $L_{1-norm}$ and the averaging strategy along with the choose-max selection fusion rule in the VGG-19 network. Besides, the proposed method has very negligible noise due to the unique characteristics of the fuzzy set approach. It uses the exponential fuzzifier that utilizes adaptive calculation to remove the noise and adjust contrast.

The CFB method in Figure 7c contains more artificial noise points, and the significant features are not obvious. Its visual effect is poor compared with other fusion methods. The fused image obtained by the DCT method has the presence of gradient reversal artifacts. In addition, the details are fuzzy, and there are numerous artificial edges and contours in the whole image. The fused images obtained by the CSR and LatLRR methods almost look similar. They have fewer artifacts and noise, resulting in a more informative image. They restore the thermal objects with more details, but they lose some spatial details such as small windows, and other details are not very clear, as can be seen in the red boxes. The fused image obtained by the DWT method captures more salient features from both source images. On the other side, there is a large amount of additional structure around the prominent targets, which is not present in the source images but is introduced by noise, resulting in a distorted image. It can be seen in Figure 7h that the proposed method preserves whole energy information due to the unique characteristics of PCA because it transforms the correlated variable into an uncorrelated form that chooses those features which have significant values of eigenvalues $\mathrm{\lambda }$ and eigenvectors $\mathrm{\xi }$. In addition, the fused image restores more contrast, structural information, smooth patterns, and contours.

It can be depicted in Figure 8 that the CBF method has obvious artifacts and salient features are unclear. The fused image obtained by DCT also has noise and gradient reversal artifacts. Hence, the targets are ambiguous to detect from fused images, thus resulting in a worse visual effect. The fused image acquired by the CSR method has a better visual effect than CBF and DCT. Contrarily, the details are still fuzzy in the fused image, and the contrast is low. Moreover, it can be seen in the red box in Figure 8f that the bench is blurred. The CSR method uses the choose-max strategy for the fusion of low frequency, and this fusion rule is not effective in capturing energy information. The DWT and LatLRR methods have more details, and the bench, which is highlighted by a red box is vivid. The contrast of DWT is more than the latent-rank method, but the DWT image has more artificial noise, especially surrounding the bench, while the latLRR method has less noise than DWT and the other three methods. The proposed method achieves a better result than the aforementioned methods. The red highlighted bench is clear, and the water transparency in the lake is also more vivid because the proposed method uses AD that employs the intra region smoothing procedure to obtain coarse resolution images. We examined various values for free parameters $\mathrm{\lambda },\mathrm{t}$ and k, and we have chosen $\mathrm{\lambda }=0.15,\mathrm{t}=10$ and k = 30 for smoothing because it gives good results at these values. This makes the edges sharper at each coarse resolution. Furthermore, the contrast of the image is bright with very little noise, which shows that the image can preserve all salient features from both source images.

Figure 9c–h shows the fusion results of other methods and proposed methods. The CBF method contains substantial gradient reversal artifacts that result in distortion in the image. The DCT method contains most of the information from the IR image while it loses information from the VI image. Hence, the fused image results in the loss of detailed information, and the overall quality of an image is worse. The DWT and LatLRR methods almost resemble the fused image, thereby obtaining better results than CBF and DCT methods. Nevertheless, the fused image is fuzzy and blurred as can be observed in the red boxes, and it has less contrast that fails to capture all salient features from the source images. The fused image obtained by the CSR method has better fusion results than the other four methods and has better contrast. On the contrary, the IR component of the CSR method has domination, and due to that, it cannot preserve more detailed features from the VI image. The fused image produced by the proposed method has very little noise. Besides, due to using a novel $L_{1-norm}$ and averaging strategy along with the choose-max selection fusion rule in the VGG-19 network, the fused image preserves all salient features such as edges and textures from both source images. Moreover, it can capture all patterns with better contrast, as can be seen in the red boxes. Hence, the proposed methods show their efficacy compared with all other methods.

It can be seen in Figure 10c,d that the fusion quality of CBF and DCT is significantly worse due to the misalignment of the source images. There are gradient reversal artifacts and noise in the CFB image due to high sensitivity because the CBF method obtains weight by measuring the strength of horizontal and vertical details directly that introduces noise and artifacts. In contrast, the DCT image quality is affected as it uses cross-correlation with a low pass filter, so it discards useful energy information that produces blurred images. The DWT, CSR, and LatLRR methods have almost similar fused quality visually, but the DWT has more noise because DWT uses down sampling, due to which there is a shift variance that introduces artifacts. Besides, the textures are not smooth due to the loss of detailed information during down sampling phenomena that cause Gibb’s effect. The proposed method again shows dominance compared with other fusion methods. It can be observed in the red boxes that the proposed method has smooth patterns, textures, and edges due to the unique fusion rule of $L_{1-norm}$ and averaging strategy along with the choose-max selection fusion rule in the VGG-19 network. Furthermore, the left–right bottom red boxes show that the proposed method restores more salient features with very smooth background because of choosing the specific values for free parameters ($\mathrm{\lambda },\mathrm{t}$ and k) in the anisotropic diffusion process. In addition, the contrast of a fused image is more vivid due to the fuzzy set enhancement method that uses an adaptive calculation for the exponential fuzzifier that proves the efficacy of the proposed method compared with other methods.

Due to space limitations, we cannot discuss the results of each fused image. We have only presented the final fused image of the proposed method and five other methods in Figure 11. It can be observed that the proposed method achieves better results than other methods.

3.1.1. Statistical Evaluation

We use two statistical tests, such as the Wilcoxon signed-rank test [46] and the Friedman test [47], to illustrate the superiority of the proposed method over other state-of-the-art (SOTA) methods. The Wilcoxon signed-rank test observes the significant difference between the input images and the generated images. In contrast, the Friedman test computes a significant variation in image quality between different approaches. For the former, we created two kinds of image pairs for a test, the IR image vs. the fused image, and the VI image vs. the fused image, to see the effects of different methods on image fusion. The lower the p-value, the more likely it is to reject the null hypothesis, implying that the output images differ from the input images and carry more information about the IR/VI images. It can be seen in

Table 2. The results of the Wilcoxon signed-rank and Friedman tests for different methods.

Methods	Wilcoxon Signed-Rank Test		Friedman Test
	IR-Fused	VI-Fused	PSNR
CBF	3.80 × 10−17	5.68 × 10−18	2.63 × 10−16
CSR	5.67 × 10−18	3.87 × 10−18
DCT	3.89 × 10−18	3.88 × 10−18
DWT	3.88 × 10−18	5.62 × 10−18
LatLRR	3.86 × 10−18	5.68 × 10−18
Proposed	3.52 × 10−18	3.58 × 10−18

that the p-value of the IR-fused images produced by the proposed method is the lowest, indicating that our approach can effectively fuse the input images from different sources (IR and VI). For the latter, we employ peak-signal-to-noise-ratio (PSNR) as the ranking metric to test the significant difference among the other methods. As shown in Table 2, the result is small enough to exhibit the superiority of the proposed method over SOTA methods.

3.1.2. User Case

We performed a user case in which we invited ten field experts to provide a ranking for each method on five pairs of images. The detailed discussion is described below:

We conducted a user study on five datasets to demonstrate that our method outperforms other methods. We invited ten experts to rank the results of all approaches according to their choice. Each expert was asked to rate six results (the proposed approach and five other methods) using a scale of one point (least preferred) to six points (most favorite) for each pair of source images. Experts conducted the subjective analysis on the following four points: (1) precise details, (2) no loss of information, (3) contrast of an image, and (4) noise in the fused image. The experts were assigned anonymous results in random order to avoid subjective bias.

Furthermore, the user study was performed in the same environment (room, light, and monitor). We calculated the average points obtained by each approach on five pairs of source images after the experts finished grading all of the simulation results.

Table 3. User case (average points acquired by each method on five pairs of source images).

Dataset-1	CBF	DCT	DWT	CSR	LatLRR	Proposed
1	1.5	2	2.5	4	4.5	5.5
2	2	3	4	5	5	5.5
3	2.5	1.5	4	4.5	4	5
4	1.5	1.5	2.5	3.5	3.5	4.5
5	2.5	2	3.5	3.5	4.5	4.5

presents the average points obtained by each method by experts on five pairs of source images.

It can be observed from Figure 12 that experts prefer the average results produced by the proposed method.

3.2. Objective Evaluation

The objective evaluation is used to analyze the difference based on mathematical calculation quantitatively. We compared our method with conventional fusion methods on six objective evaluation assessment parameters.

3.2.1. Feature Mutual Information (FMI)

The FMI [48] is used to measure the mutual information of the image features such as edges, contrast, details, and so on. It can be computed by Equation (30):

$$FMI=M{I}_{A^{\prime },F^{\prime }}+M{I}_{B^{\prime },F^{\prime }}$$

(30)

$$M{I}_{X^{\prime }{F}^{\prime }}={\displaystyle \sum _{x,f}{H}_{X^{\prime },F^{\prime }}(x^{\prime },f^{\prime })}\log \frac{H_{X^{\prime },F^{\prime }(x^{\prime },f^{\prime })}}{H_{X^{\prime }}(x^{\prime })H_{F^{\prime }}(f^{\prime })},X^{\prime }\in \{A^{\prime },B^{\prime }\}$$

(31)

where, $H_{X^{\prime }}(x^{\prime })$ and $H_{F^{\prime }}(f^{\prime })$ represent the marginal histogram of an input image $X$ and the fused image $F$. The $H_{X^{\prime },F^{\prime }}(x^{\prime },f^{\prime })$ denotes the joint histogram of $X$ and $F$. $X^{\prime }$ indicates the feature map of input image $A$ and $B$, whereas, $A^{\prime },B^{\prime },F^{\prime }$ are the feature maps of IR, VI, and the fused image, respectively.

3.2.2. Modified Structural Similarity Index ($SSI{M}_a$)

The $SSI{M}_a$ [49] is used to compute the distortion and loss in the image structure. The quality of the fused image would be better if the value of $SSI{M}_a$ is larger because larger values correspond to a fused image with more structure. The fused image will have more distortion and loss if the value of $SSI{M}_a$ is smaller. The higher value of $SSI{M}_a$ is written in bold letters in Table 4. It can be calculated as Equation (32):

$$SSI{M}_a(F)=(SSI{M}_{A,F}+SSI{M}_{B,F})\times 0.5$$

(32)

where $SSIM(.)$ denotes the structural similarity, $F$ represents the fused image, and $A,B$ denotes the IR and VI image.

3.2.3. Average Pixel Intensity (API)

The $API$ [3] is used to measure the contrast of the fused image. The quality of the fused image would be better if the value of API is larger. It is obtained by Equation (33):

$$API=\frac{{\displaystyle \sum _{i=1}^m{\displaystyle \sum _{j=1}^{n}f(i,j)}}}{m\times n}$$

(33)

where $f(i,j)$ is the intensity of the pixel at $(i,j)$, and $m\times n$ represents the image size.

3.2.4. Entropy (EN)

The entropy [18] determines the amount of information available in the image. It is computed by Equation (34):

$$\mathrm{E}\mathrm{N}=-{\displaystyle \sum _{\mathrm{l}=0}^{\mathrm{L}-1}{\mathrm{p}}_{\mathrm{l}}{\log }_2{\mathrm{p}}_{\mathrm{l}}}$$

(34)

where, L represents the number of gray levels and ${\mathrm{p}}_{\mathrm{l}}$ indicates the normalized histogram of the corresponding gray level in the fused image. The larger value of EN refers to a fused image with more information.

3.2.5. Quality Ratio of Edges from Sources Images to Fused Image (${\mathrm{Q}}_{\mathrm{F}}^{\mathrm{A}\mathrm{B}}$)

The ${\mathrm{Q}}_{\mathrm{F}}^{\mathrm{A}\mathrm{B}}$ [37] is used to compute the amount of edges information transferred from input images to the final fused image. It is calculated by Equation (35):

$${\mathrm{Q}}_{\mathrm{F}}^{\mathrm{A}\mathrm{B}}=\frac{{\displaystyle \sum _{\mathrm{i}=1}^{\mathrm{N}}{\displaystyle \sum _{\mathrm{j}=1}^{\mathrm{M}}{\mathrm{Q}}^{\mathrm{A}\mathrm{F}}(\mathrm{i},\mathrm{j}){\mathrm{w}}^{\mathrm{A}}(\mathrm{i},\mathrm{j})+{\mathrm{Q}}^{\mathrm{B}\mathrm{F}}(\mathrm{i},\mathrm{j}){\mathrm{w}}^{\mathrm{B}}(\mathrm{i},\mathrm{j})}}}{{\displaystyle \sum _{\mathrm{i}=1}^{\mathrm{N}}{\displaystyle \sum _{\mathrm{j}=1}^{\mathrm{M}}({\mathrm{w}}^{\mathrm{A}}(\mathrm{i},\mathrm{j})+{\mathrm{w}}^{\mathrm{B}}(\mathrm{i},\mathrm{j}))}}}$$

(35)

where ${\mathrm{Q}}^{\mathrm{A}\mathrm{F}},{\mathrm{Q}}^{\mathrm{B}\mathrm{F}}$ represents the amount of edge information transferred from image A to F, and B to F at location $(\mathrm{i},\mathrm{j})$, respectively. The A, B, and F are input into image 1, image 2, and the fused image, respectively. The ${\mathrm{w}}^{\mathrm{A}}$ and ${\mathrm{w}}^{\mathrm{B}}$ indicate the weights in image A and image B. The higher the value of ${\mathrm{Q}}_{\mathrm{F}}^{\mathrm{A}\mathrm{B}}$, the more edges information will be transferred from the input images to the fused image.

3.2.6. Noise ratio from the source image to the fused image $N^{\raisebox{1ex}{$AB$}\!\left/ \!\raisebox{-1ex}{$F$}\right.}$

The $N^{\raisebox{1ex}{$AB$}\!\left/ \!\raisebox{-1ex}{$F$}\right.}$ [18] is the index of fusion on the basis of artifacts and noise. The larger value corresponds to more artifacts and noise in the fused image, while the image will have less artifacts and noise if the value is smaller. The smaller value is written by bold letters in Table 4 for ease of reading. It is computed by Equation (36):

$$N^{\raisebox{1ex}{$AB$}\!\left/ \!\raisebox{-1ex}{$F$}\right.}=\frac{{\displaystyle \sum {\forall }_{n,m}{N}_{n,m}(w_{n,m}^A+w_{n,m}^B)}}{{\displaystyle \sum {\forall }_{n,m}(w_{n,m}^A+w_{n,m}^B)}}$$

(36)

where $N_{n,m}$ indicates the fusion loss and $w_{n,m}^A$ and $w_{n,m}^B$ represent the weight maps of the IR and VI images.

Table 4. The average values of FMI, $SSI{M}_a$, API, $N_F^{AB}$, EN, and ${\mathrm{Q}}_{\mathrm{F}}^{\mathrm{A}\mathrm{B}}$.

Methods	CBF [15]	DCT [18]	Improved DWT [19]	CSR [27]	LatLRR [28]	Proposed
FMI	0.2651	0.2746	0.3041	0.3439	0.3561	0.3951
$SSI{M}_a$	0.5827	0.5971	0.6349	0.7412	0.7798	0.8469
API	56.3710	56.4928	58.9314	63.0178	66.7153	71.9081
$N_F^{AB}$	0.3049	0.3015	0.2916	0.0197	0.0038	0.0021
EN	4.0117	4.1537	4.8471	6.1517	6.8924	7.9907
${\mathrm{Q}}_{\mathrm{F}}^{\mathrm{A}\mathrm{B}}$	0.6538	0.7142	0.7867	0.8191	0.8306	0.8727

Table 4. The average values of FMI, $SSI{M}_a$, API, $N_F^{AB}$, EN, and ${\mathrm{Q}}_{\mathrm{F}}^{\mathrm{A}\mathrm{B}}$.

Methods	CBF [15]	DCT [18]	Improved DWT [19]	CSR [27]	LatLRR [28]	Proposed
FMI	0.2651	0.2746	0.3041	0.3439	0.3561	0.3951
$SSI{M}_a$	0.5827	0.5971	0.6349	0.7412	0.7798	0.8469
API	56.3710	56.4928	58.9314	63.0178	66.7153	71.9081
$N_F^{AB}$	0.3049	0.3015	0.2916	0.0197	0.0038	0.0021
EN	4.0117	4.1537	4.8471	6.1517	6.8924	7.9907
${\mathrm{Q}}_{\mathrm{F}}^{\mathrm{A}\mathrm{B}}$	0.6538	0.7142	0.7867	0.8191	0.8306	0.8727

We performed fusion on 18 test images, and their average values of FMI, $SSI{M}_a$, API, EN, ${\mathrm{Q}}_{\mathrm{F}}^{\mathrm{A}\mathrm{B}}$, and $N_F^{AB}$ are presented in

Table 5. The evaluation criteria for objective assessment parameters.

Quality/Parameter	FMI	$\mathit{S}\mathit{S}\mathit{I}{\mathit{M}}_{\mathit{a}}$	API	EN	${{Q}}_{{F}}^{{A}{B}}$	${\mathit{N}}_{\mathit{F}}^{\mathit{A}\mathit{B}}$
Poor	<0.28	<0.6	<55	<4	<0.7	>0.3
Fair	0.28–0.32	0.60–0.70	55–60	4–5	0.7–0.75	0.02–0.3
Good	0.33–0.37	0.71–0.81	61–65	5–6	0.75–0.8	0.003–0.02
Excellent	>0.37	>0.81	>65	>6	>0.8	<0.003

. Due to space limitations, we provided the average values for all mentioned parameters. In addition, we also provided a line graph and scattering plot in Figure 13, Figure 14, Figure 15, Figure 16, Figure 17 and Figure 18, respectively. The primary purpose of using the two different graphs is for ease of comprehension.

The larger values of $FMI$, $SSI{M}_a$ $API$, EN, and ${\mathrm{Q}}_{\mathrm{F}}^{\mathrm{A}\mathrm{B}}$ result in a good quality fused image with more salient features from both source images, while the smaller values correspond to a fused image with less similar structure. On the contrary, the fused image will have less artifacts and noise if the value of $N_F^{AB}$ is smaller, while the higher value results in a fused image with more artifacts and noise. The larger values for $FMI$, $SSI{M}_a$, $API$, EN, ${\mathrm{Q}}_{\mathrm{F}}^{\mathrm{A}\mathrm{B}}$, and the smaller values of $N_F^{AB}$ are written in bold letters in Table 4 for differentiation purposes. It can be observed in Table 4 and Figure 13 that CBF and DCT have fewer values for $FMI$, which means the fused image restores less salient features from source images. The improved DWT produces better results than CBF and DCT; however, the CSR and latent-based methods achieve better results than DWT. The proposed method outperforms all other methods that acquire a better quality fused image with more salient features. Similarly, it can be observed in Table 4, Figure 14 and Figure 15, that the proposed method achieves significant values for $SSI{M}_a$ and $API$ which shows the optimal fusion performance compared with other methods by retaining more structure and contrast from the source images. Compared with other methods, the latent-based method’s performance is better than the other three methods, whereas the CSR values are closer to the latent-based method.

Similarly, it can be seen in Table 4, Figure 17 and Figure 18, that the proposed method has higher values for EN and ${\mathrm{Q}}_{\mathrm{F}}^{\mathrm{A}\mathrm{B}}$ than the other methods. The yellow triangles indicate the proposed method in scattering plotting in Figure 17 and Figure 18. It can be seen that the proposed method achieves higher values for EN ${\mathrm{Q}}_{\mathrm{F}}^{\mathrm{A}\mathrm{B}}$ except for a few images. Moreover, the CBF (square boxes) and DCT methods (circle boxes) in Figure 17 and Figure 18 produce poorer results than all other methods. Furthermore, it can be observed from Table 4 and Figure 16 that the proposed method has significantly fewer values of $N_F^{AB}$, which proves that the fused images have fewer artifacts and noise than other methods. In this regard, it can be concluded that the proposed method produces better visual results and obtains better values for objective evaluation analysis, demonstrating the supremacy of the proposed method.

We also present the evaluation criteria for FMI, $SSI{M}_a$, API, EN, ${\mathrm{Q}}_{\mathrm{F}}^{\mathrm{A}\mathrm{B}}$, and $N_F^{AB}$ for objective quality parameters to estimate the results. The evaluation criteria for quality parameters in presented in Table 5.

It can be observed from evaluation criteria that the proposed method obtained the best performance for all evaluation parameters while the performance of CBF and DCT methods exhibited poor performance.

3.3. Computation Time Comparison

The average computation time for the proposed method and the other compared fusion methods is presented in

Table 6. The average computation time for proposed and other fusion methods (unit: seconds).

Method	CBF	DCT	Improved DWT	CSR	LatLRR	Proposed
Time	8.6	11.3	10.7	259.6	313.4	72.3

. The experiments are conducted for computation time (t) in seconds on MATLAB 2016b with corei5 3.20 GHz processor and 8 GB RAM. It can be depicted in Table 6 that the computation time for LatLRR is longer than other methods, which shows it has the poorest timeliness. The CBF method is the fastest, which shows it has the best timeliness; however, its performance is the worst on subjective and objective parameters. The computation time of the proposed method is longer than CBF, DCT, and improved DWT, but the better visual effects can compensate for the reduced timeliness. The computation time for the proposed method is shorter than CSR and LatLRR, while it also produces better visual and objective performance. Therefore, the computation time for the proposed method is still acceptable.

4. Discussion

Implementing the image fusion method that extracts all salient features from individual source images into one composite fused image is the hot research topic of future development in IR and VI applications. This proposed work is one of the attempts in the IR and IV image fusion research area. Our method not only emphasizes the crucial objects by eliminating the noise but also preserves the essential and rich detailed information that is helpful for many applications such as target detection, recognition, tracking, and other computer vision applications.

Various hybrid fusion methods amalgamate the advantages of different schemes to enhance the image fusion quality. In our hybrid method, each stage improves the image quality, which can be observed in Figure 2, Figure 3, Figure 4 and Figure 5, respectively. Our hybrid method extensively uses the fuzzy set-based enhancement, anisotropic diffusion, and VGG-19 network through transfer learning with multiple fusion strategies and PCA. The FSE method automatically enhances the contrast of IR and IV with its adaptive parameter calculation. In addition, this method is robust in environments with fog, dust, haze, rain, and cloud. The AD method using PDE is applied that decomposes the enhanced images of FSE into the base and detailed images. It has the attribute of removing noise from smooth regions while retaining sharp edges. The PCA method is employed in the fusion of the base parts that restrain the distinction between the background and targets since it can restore the energy information. Besides, it eliminates redundant information with its robust operation. The VGG-19 network using transfer learning is adapted as it has a significant ability to extract specific salient features. In addition, we have used four convolution layers of VGG-19 through transfer learning that efficiently saves time and resources during the training without requiring a lot of data. At last, the $L_{1-norm}$ and averaging fusion strategy along with the choose-max selection fusion rule is applied, which efficiently removes the noise and retains the sharp edges.

All in all, from a visual perspective, it can be observed from Figure 6, Figure 7, Figure 8, Figure 9, Figure 10 and Figure 11, and experts ranking in Table 2 and Figure 12, that the proposed method produced a better quality fused image than the other fusion methods, demonstrating the supremacy of our method. Similarly, the proposed method has better values in the indexes of $FMI$, $SSI{M}_a$, $API$, EN, ${\mathrm{Q}}_{\mathrm{F}}^{\mathrm{A}\mathrm{B}}$, and $N_F^{AB}$ objective evaluation assessment that further justify the dominancy of the proposed work.

Although the proposed work shows supremacy over recent fusion methods, this work still has limitations. The proposed work is implemented for the general IR and VI image fusion framework. However, the IR and VI image fusion for a specific application needs particular details based on that application. Therefore, we will extend this work for specific applications such as night vision and surveillance. The computation time of the proposed method is not very favorable. In future research, we will emphasize reducing the consumption time while sustaining the improved quality of a fused image.

5. Conclusions

A novel IR and VI image fusion method is proposed in this work. This work contains several stages of novel techniques. We use fuzzy set-based contrast enhancement to automatically adjust the contrast of images with its adaptive calculation and an anisotropic diffusion to remove noise from smooth regions while capturing sharp edges. Moreover, a deep learning network based on four convolutional layers named VGG-19 through transfer learning is exploited to detail images to restore substantial features. Concurrently, it saves time and resources during training and restores spectral information. The extracted detailed features are fed into $L_{1-norm}$ and the averaging fusion scheme with a max selection rule that helps to remove the noise in smooth regions, thereby resulting in sharp edges. The PCA method is applied to base parts to preserve the essential energy information by eliminating the redundant information with its robust operation. Finally, a fused image is obtained by the superposition of base and detail parts containing substantial rich and detailed features with negligible noise. The proposed method is compared with other methods on subjective and objective assessments. The subjective analysis is performed by statistical tests (Wilcoxon signed-rank and Friedman) and experts’ visual experience. In contrast, the objective evaluation is conducted by computing the quality parameters (FMI, $SSI{M}_a$, API, EN, ${\mathrm{Q}}_{\mathrm{F}}^{\mathrm{A}\mathrm{B}}$, and $N_F^{AB}$). The proposed method exhibits superiority by obtaining a smaller value for statistical tests than other methods from a subjective perspective. Besides, experts also prefer the average results produced by the proposed method. In addition, the proposed method further justifies its efficacy for objective parameters by achieving 0.2651 to 0.3951, 0.5827 to 0.8469, 56.3710 to 71.9081, 4.0117 to 7.9907, 0.6538 to 0.8727 gain for $FMI$, $SSI{M}_a$, $API$, $\mathrm{E}\mathrm{N}$, and ${\mathrm{Q}}_{\mathrm{F}}^{\mathrm{A}\mathrm{B}}$ respectively

Intuitively, it can be concluded that the proposed method achieves optimal fusion performance. Our next research direction is to implement a framework for other image fusion tasks such as multimodal remote sensing, medical imaging, and multi-focus. In addition, we can apply transfer learning to other deep convolutional networks in image fusion to improve the fusion quality.