Multi-Sensor Image Fusion Using Optimized Support Vector Machine and Multiscale Weighted Principal Component Analysis

Abstract

Multi-sensor image fusion is used to combine the complementary information of source images from the multiple sensors. Recently, conventional image fusion schemes based on signal processing techniques have been studied extensively, and machine learning-based techniques have been introduced into image fusion because of the prominent advantages. In this work, a new multi-sensor image fusion method based on the support vector machine and principal component analysis is proposed. First, the key features of the source images are extracted by combining the sliding window technique and five effective evaluation indicators. Second, a trained support vector machine model is used to extract the focus region and the non-focus region of the source images according to the extracted image features, the fusion decision is therefore obtained for each source image. Then, the consistency verification operation is used to absorb a single singular point in the decisions of the trained classifier. Finally, a novel method based on principal component analysis and the multi-scale sliding window is proposed to handle the disputed areas in the fusion decision pair. Experiments are performed to verify the performance of the new combined method.

1. Introduction

Multi-sensor image fusion is a synthesis technique that can fuse source images from multiple sensors into a high-quality image with comprehensive information [1,2,3]. The technique is widely used in visual sensor networks, such as military defense, security monitoring, and image inpainting. In digital photography, it is difficult for the single-lens reflex camera to take an image that can present all objects into focus [4,5]. To obtain all-in-focus images, multisource images from the same scene with different focuses are fused into one signal image, which is named the multi-focus image fusion [6]. Most of the existing multi-focus image fusion methods can be classified into two strategies: signal processing-based fusion methods (such as transform domain methods, spatial domain methods, the hybrid methods), and machine learning-based fusion methods (such as artificial neural network, fuzzy system, and support vector machine).

Generally, the transform domain-based fusion methods include three stages: first, the source images are transformed to obtain the decomposed sub-band coefficients of each image; then, a certain fusion rule is performed to integrate the corresponding sub-band coefficients to obtain the fused coefficients; at last, the fused coefficients are used to obtain the fused image by inverse transformation [7,8,9]. The classical signal processing-based fusion methods include principal component analysis (PCA) [10], discrete wavelet transform (DWT) [11], nonsubsampled operation-based transform (such as nonsubsampled shearlet transform, non-subsampled contourlet transform, stationary wavelet transform) [12], multi-resolution singular value decomposition (MSVD) [13], discrete cosine harmonic wavelet transform (DCHWT) [14], and so on. However, the conventional image fusion methods may produce unpredictable errors between the transform and inverse transform, and these errors may produce the problem of image distortion and artifacts.

With the development of neural networks, researchers are devoted to introducing deep learning into image fusion, especially the field of multi-focus image fusion, which can model as a pixel classification task [15,16,17,18,19]. In recent years, image fusion methods based on deep learning models have emerged and shown great development potential in some situations [20,21]. Liu et al. [15], in 2017, applied a deep convolutional neural network (DCNN) to multi-focus image fusion. This method regarded image fusion as a binary classification problem, but it was still a fusion method based on the spatial domain method that may have the block effect. To solve this problem, Mustafa et al. [22] proposed a multi-focus image fusion method, which combined the feature extraction, fusion and reconstruction task together as a complete unsupervised end-to-end model. With the development of generative adversarial networks (GANs), it has shown great capacity in the field of image fusion. Guo et al. [23] proposed a multi-focus image fusion method based on conditional generative adversarial network (cGANs), which achieved good image fusion performance. However, the image fusion methods based on deep learning also have some limitations, for example, a mass of samples and computational resources are needed for training a good model with plenty of time; moreover, many hyper-parameters are adjusted manually [24]. Considering the tradeoff of calculated quantity and fusion performance, shallow machine learning methods also have some superiorities in image fusion because these methods require limited computing resources and fewer training samples. The support vector machine (SVM), which can be regarded as a classical shallow learning model with a hidden layer, is normally trained by using some extracted features to distinguish the focused and unfocused regains that are employed for generating fusion decisions [18,19]. Because of the lack of feature extraction capability for the shallow machine learning model, it is necessary to employ a given feature extraction method to present the image features (such as texture, structure, and edge), which has great significance on the improvement of image fusion performance.

In this work, a novel multi-focus image fusion method based on SVM, multiscale PCA, and the feature extraction method is introduced. The method first uses the sliding window technique to extract the detailed features of different source images. Then, the focused and unfocused areas of source images are extracted by a pre-trained SVM. In the fusion stage, the fusion decisions of different source images are combined with a set of logic operations, and then CV is carried out to optimize the decisions. At last, a new pixel-weighted image fusion scheme is designed based on multi-scale PCA to process the disputed decisions at the same positions of different source images. The contributions of this work are summarized as follows.

• This work designs a regional feature extraction method based on five image fusion evaluation metrics and the extracted regional features are then employed as the input of an SVM model to produce pixel fusion decisions. This design can avoid inputting the complete image into SVM.
• An SVM-based spatial image focus detection method is introduced to distinguish the focused and unfocused regions for integrating different source images, and the new method requires a few training samples to identify the focused and unfocused areas.
• A multi-scale weighted image fusion method based on PCA is proposed to handle the disputed regions that come from the same position of the decision masks of different source images. The proposed multi-scale image fusion method based on PCA has better performance compared to the conventional PCA methods.

The remaining sections of the paper are presented as follows. In Section 2, the basic theories of the SVM and PCA-based image fusion method are briefly reviewed. In Section 3, the proposed image fusion method is reported. The experimental results and analysis are described in Section 4. Section 5 concludes this work.

2. Related Work

The related work and basic theories of the multi-focus image fusion method based on SVM and PCA are briefly reviewed in this sub-section.

2.1. Multi-Focus Image Fusion

The multi-focus image fusion method can fuse the multiple images with different focuses to obtain a fully focused image. In 2016, a multi-focus image fusion method based on SVM and hybrid wavelet was proposed by Yu et al. [19]. In this method, multi-focus image fusion was regarded as a binary classification problem: focus and non-focus. However, this method introduced some noise when obtaining the fused image. In 2018, Siddique et al. [25] proposed an image fusion method based on color-principal component analysis (C-PCA), which was divided into three stages: first, color PCA and enhanced color properties were used to generate the intermediate images; second, the salient features of an image were extracted by Laplacian of Gaussian; third, the spatial frequency was used as the focus measurement to obtain the final fused image. In 2020, Tyagi et al. [26] proposed a hybrid and parallel processing fusion technique for multi-focus images based on stationary wavelet transform (SWT) and principal component analysis (PCA). Recently, more and more researchers have carried out research on multi-focus image fusion methods based on deep learning. In 2018, Tang et al. [20] proposed a pixel-wise CNN (p-CNN) that can recognize the focused and defocused pixels in source images from its neighborhood information for multi-focus image fusion. More recently, the end-to-end modeling of multi-focus image fusion based on U-shape networks was proposed by Li et al. [27] However, multi-focus image fusion based on deep learning usually consumes a lot of computing resources and time, which was the limitation of this method. To solve this problem, a shallow machine learning approach is applied to the proposed method.

2.2. SVM Model and Its Application in Image Fusion

SVM is a generalized linear classifier with a supervised learning style, and its decision boundary is obtained by the maximum-margin hyperplane learned according to the samples [28]. In this work, the multi-focus image fusion problem is handled as a classification task, thus SVM can be employed for the pixel-level image fusion task. The theory of SVM can be defined by:

$$\begin{array}{l}\underset{\omega ,b}{\min }\frac{{\Vert \omega \Vert }^2}{2}\\ s.t.y_i({\omega }^T{x}_i+b)\ge 1,i=1,2,\dots m\end{array}$$

(1)

where $\omega =({\omega }_1;{\omega }_2;\dots ;{\omega }_d)$, $\omega$ is the normal vector that determines the hyperplane direction; b represents the displacement term and determines the distance of the hyperplane and origin.

For the nature of linear indivisibility of samples, a kernel function can be employed to map the features of samples from low-dimensional space into high-dimensional space, thus the samples are separable in high-dimensional space. Therefore, radial basis function (RBF) kernel function is employed to address this problem, which is defined as:

$$\kappa ({\chi }_i,{\chi }_j)=\exp (-\frac{{\Vert {\chi }_i-{\chi }_j\Vert }^2}{2{\sigma }^2})$$

(2)

In a practical problem, it is very difficult to find a proper kernel function to make samples completely separable in the feature spaces, and it is also difficult to determine whether the samples that are completely separable, which is caused by an overfitting problem. Thus, the soft margin is introduced as sacrificing some samples that must be properly divided for the maximum classification interval. The basic SVM model with a soft margin is defined by:

$$\begin{array}{l}\underset{\omega ,b,{\xi }_i}{\min }\frac{{\Vert \omega \Vert }^2}{2}+C{\displaystyle \sum _{i=1}^m{\xi }_i}\\ s.t.y_i({\omega }^T{x}_i+b)\ge 1-{\xi }_i,i=1,2,\dots ,m\end{array}$$

(3)

where ${\xi }_i$ is the slack variable, which is utilized to record the wrongly classified samples; C ≥ 0, and the constant C is called a penalty parameter, which controls the tolerance for error samples.

In this work, particle swarm optimization (PSO) is employed to obtain the optimized settings of SVM automatically, and the parameters are penalty parameter C and RBF kernel parameter g [29]. PSO is widely used for parameter optimization problems. In the solution space, each particle of PSO describes a solution for a given problem. Moreover, the best solution of all particles in each iteration is called the locally optimal solution. The best solution of the swarm is called the global optimal solution. The particle iteratively adjusts its trajectory to find local and global optimal solutions. As a result, it can find a set of optimized parameters for SVM instead of repeated trials manually.

2.3. PCA-Based Image Fusion

PCA is a popular descending dimension method that can maintain the key features of the input variable, such as the image. In PCA-based image fusion methods, the principal components of two different source images are employed to obtain the global fusion weight [10]. However, the global fusion weight calculated by classical PCA-based image fusion cannot effectively present the detailed features of the source image. In [30], the authors described a hierarchical PCA image fusion method that can take into consideration the window-based image information to obtain regional weights; however, they only consider a single-scale image feature, which is not enough for obtaining a good fusion performance. The processes of conventional PCA-based image fusion method are described in Algorithm 1.

Algorithm 1 PCA-based image fusion

0: Input: two source images: ${{\displaystyle im}}_1$, ${{\displaystyle im}}_2$.

1: Decentralize input image pixels.

2: Convert each image into column vectors and constitute a new matrix $M$.

3: Calculate the covariance matrix $COV$ of the matrix $M$.

4: Produce a diagonal $D$ of eigenvalues and a full matrix $V$ whose columns are the correspond eigenvectors.

5: Obtain the fused weight, and the calculation process is defined as follows:

6: $\mathit{i}\mathit{f}(D(1,1)>D(2,2))$

7:   $a=V(:,1)./sum(V(:,1))$

8: $\mathit{e}\mathit{l}\mathit{s}\mathit{e}$

9:    $a=V(:,2)./sum(V(:,2))$

10: Fuse two source images, and the calculate process define as follows:

11:   $F=a(1)\times i{m}_1+a(2)\times i{m}_2$

12: Output: fused image $F$.

3. The Proposed Image Fusion Method

The scheme of our proposed image fusion algorithm is shown in Figure 1. According to the proposed scheme, the processes of the proposed image fusion method can be divided into three steps: (1) the detailed features of the focused and unfocused regions in the source images are extracted using a given sliding window, which is marked as the red box; (2) an SVM is trained by the extracted features and labels, and then two decision masks are produced by the pre-trained SVM model, which is marked as the blue box; (3) the undisputed decisions of the given source image pair are first extracted, and then the pixels that are corresponding to the undisputed decisions are fused to obtain F1, which is marked as the yellow box; (4) the disputed decisions of a given source image pair are extracted, and then the pixels in the disputed decisions are fused with the proposed multiscale weighted PCA (MWPCA) to obtain F2, which is marked as the green box roughly. Finally, the fused image is obtained by logic operation with F1 and F2.

3.1. Our Proposed Image Fusion Method Based on Pixel Classification

This sub-section introduces our proposed multi-focus image fusion method based on pixel classification, which includes image feature extraction, RBF-SVM training, and parameter settings.

3.1.1. Feature Extraction

According to Figure 1, two source images are traversed by a sliding window which can extract five features. These features, which represent the regional degree of focus around the pixel, constitute the feature vectors that are input into the SVM model. Given a pixel $im(i,j)$, this method employs a $n\times n$ window is used to calculate the regional features of its surrounding pixels. Moreover, the perimeter boundary of the source image is useful to represent the regional features of the boundary pixels, thus a mirroring method is used to expand the boundary area according to the defined window. The size of the expanded area is:

$$s=(n-1)/2$$

(4)

where s represents the size of expanding area, n is the sliding window size; and the step size is set as 1 to traverse all pixels of source images.

To present the regional features of the source image and achieve the goal of the descending dimension, five important image fusion metrics are selected based on our repeated trials. The used five metrics are employed to present the detailed features of a given image in the sliding window. When the window slides to a pixel position, five metrics are calculated to form a feature vector. The input features of SVM are formed when the sliding window traverses all pixels of source images. These metrics are standard deviation (STD), spatial frequency (SF), average gradient (AG), energy of image gradient (EIG), and sum-modified Laplacian (SML) [19,31]. In this subsection, $im(i,j)$ represents the value of pixel $(i,j)$, and $M$ and $N$ are source image sizes.

STD can be employed to analyze the statistical distribution and contrast information of a given image, which is presented as follows:

$$STD=\sqrt{\frac{1}{M\times N}{\displaystyle \sum _{i=1}^M{\displaystyle \sum _{j=1}^N{(im(i,j)-\mu )}^2}}}$$

(5)

where $\mu$ is the mean value (MV) and defined by:

$$\mu =\frac{1}{M\times N}{\displaystyle \sum _{i=1}^M{\displaystyle \sum _{j=1}^{N}im(i,j)}}$$

(6)

SF presents the spatial activity of a given image, which is described as follows:

$$SF=\sqrt{R{F}^2+C{F}^2}$$

(7)

$$RF=\sqrt{\frac{1}{M\times N}{\displaystyle \sum _{i=1}^{M-1}{\displaystyle \sum _{j=1}^{N-1}{[im(i,j)-im(i,j-1)]}^2}}}$$

(8)

$$CF=\sqrt{\frac{1}{M\times N}{\displaystyle \sum _{j=1}^{N-1}{\displaystyle \sum _{i=1}^{M-1}{[im(i,j)-im(i-1,j)]}^2}}}$$

(9)

where $RF$ presents the row frequency of a given image, $CF$ presents the column frequency.

AG evaluates the sharpness of a given image by different directions to show the details and texture information of the image, which is shown as follows:

$$AG=\frac{1}{M\times N}{\displaystyle \sum _{i=1}^M{\displaystyle \sum _{j=1}^N\sqrt{\frac{1}{2}{(im(i,j)-im(i+1,j))}^2+{(im(i,j)-im(i,j+1))}^2}}}$$

(10)

EIG can present the gradient information of an image by considering the features between the adjacent pixels, which is shown as follows:

$$D(im)={\displaystyle {\sum }_j{\displaystyle {\sum }_i({\left|im(i+1,j)-im(i,j)\right|}^2+{\left|im(i,j+1)-im(i,j)\right|}^2)}}$$

(11)

SML is an improved version of the basic definition of energy of Laplacian to present the gradient information of an image, which is shown as follows:

$${{\displaystyle {{\displaystyle \nabla }}^2}}_{ML}im(i,j)=|2im(i,j)-im(i-\beta ,j)-im(i+\beta ,j)|+2im(i,j)-im(i,j-\beta )-im(i,j+\beta )|$$

(12)

where $\beta$ is set as 1 to adjust the variation of features in a given image, and SML is shown as follows:

$$SML={\displaystyle \sum _{x=i-N}^{x=i+N}{\displaystyle \sum _{y=i-N}^{y=i+N}{\nabla }^2{}_{ML}im(x,y)for{\displaystyle \sum _{y=i-N}^{y=i+N}{\nabla }^2{}_{ML}im(x,y)\ge T}}}$$

(13)

where $T$ presents the discrimination threshold value, $N$ is the window size of SML.

3.1.2. SVM Model Training and Fusion Decision Mask

For SVM training, we first cut the focused and unfocused areas from the multi-focus image into blocks. The feature extraction methods are used to build a training dataset. Here, 0 represents unfocused, 1 represents focused. The PSO method is utilized to find the optimized parameters of SVM. The C and g of SVM with the best accuracy will be selected as the optimized parameters of SVM. Thus, a classification model is trained successfully.

The trained SVM model is utilized to judge the focused regions and the unfocused regions in the given source image that should be disposed as the training set. A pair of given source images (${{\displaystyle im}}_1$,${{\displaystyle im}}_2$) is disposed in the following steps:

• Traverse a given source image using sliding windows to get a set of pixel vectors.
• Calculate five indicators in each sliding window to obtain the regional feature of the central pixel in the given source image.
• The trained SVM model is used to mask each pixel as “1” or “0”, which means each pixel in the given source image is determined whether it belongs to the focus or the non-focus area.
• The decision results are reconstituted into the image fusion masks.

The focused and unfocused regions of a source image pair must be complementary, which means the fusion decisions of the corresponding pixels of different source images are complementary. However, the fusion decisions obtained by the SVM model may not be perfectly complementary because the decisions are not quite correct. Figure 2 present two groups of fusion decisions which are obtained by the source image pairs “head” and “wine bottle”. We can find that some fusion decisions from different source images are disputed. Therefore, we cannot decide which pixel should be fused into the final image. For example, some disputed fusion decisions are marked by the red arrows in Figure 2.

3.2. Our Proposed Multiscale Image Fusion Method Based on PCA

In this work, multiscale weighted PCA (MWPCA) is proposed to handle the fusion masks generated by the SVM model. The local features of the source images are regarded as a key factor in multi-focus image fusion. Thus, a novel image fusion method based on PCA joint sliding window is employed to fuse the source images, in which the fusion weight of each pixel in the dispute area is calculated [32]. Since each size of the sliding window only reflects the regional features in a single-scale, the windows with different sizes are simultaneously combined with PCA to get the corresponding fusion weights. Thus, the regional features of the source images can be represented in multi-scales. To enhance the fusion results, MWPCA is used to handle the disputed area by considering the regional feature of the source images. MWPCA is also an integrated fusion method, and the scheme of the MWPCA is shown in Figure 3.

The fused images of our proposed MWPCA are obviously better than those of conventional PCA-based methods, and the experiments are shown in Section 4. The processes of MWPCA are shown in Algorithm 2.

Algorithm 2 MWPCA

0: Input: source images: ${{\displaystyle im}}_1$, ${{\displaystyle im}}_2$

1: Define a group of sliding windows with different sizes $D_n=\left\{im{w}_{n,1},im{w}_{n,2}\right\}$. The ${{\displaystyle imw}}_{n,1}$, ${{\displaystyle imw}}_{n,1}$  are sizes of the sliding window n.

2: Get a defined sliding window.

3:   Traverse two source images with the defined sliding window.

4:      Put the outputs of the window pairs ${{\displaystyle D}}_n$ into PCA to calculate eigenvector ${{\displaystyle V}}_n$.

5:      Select the eigenvector ${{\displaystyle V}}_n$ which is corresponding to the largest eigenvalue.

6:      Generate weighted vector ${{\displaystyle \alpha }}_n$ which is calculated as follows:

7:      $\{\begin{array}{l}{{\displaystyle \alpha }}_n^1=V_{n_1}/V_n\\ {{\displaystyle \alpha }}_n^2=V_{n_2}/V_n\end{array}$       where ${{\displaystyle \alpha }}_n^1$ and $V_{n_1}$ is the first value in vector ${{\displaystyle \alpha }}_n$ and ${{\displaystyle V}}_n$; ${{\displaystyle \alpha }}_n^2$ and $V_{n_2}$ is the       second value in vector ${{\displaystyle \alpha }}_n$ and ${{\displaystyle V}}_n$.

8:      Output weighted vector ${{\displaystyle \alpha }}_n$; and record ${{\displaystyle \alpha }}_n^1$ and ${{\displaystyle \alpha }}_n^2$.

9:      The weighted values are calculated with the corresponding pixel value in images ${{\displaystyle im}}_1$ and ${{\displaystyle im}}_2$, as follows:

10:      $Y_n={{\displaystyle \alpha }}_n^1\times i{m}_1(i,j)+{{\displaystyle \alpha }}_n^2\times i{m}_2(i,j)$

11:    Repeat the steps above 3–9 until the sliding windows pairs traverse all the pixels in the source images ${{\displaystyle im}}_1$ and ${{\displaystyle im}}_2$, and finally get a weighted image ${}_{{{\displaystyle Y}}_n}$

12: Repeat the steps above 2–8 for different sizes of sliding windows to get n weight images and calculate the second fused image, as:

13: $F_2=\frac{1}{n}\times Y_1+\frac{1}{n}\times Y_2+\dots +\frac{1}{n}\times Y_n$

14: Output: fused image $F$.

3.3. Our Proposed Multi-Focus Image Fusion Strategy

According to the proposed scheme shown in Figure 1, this sub-section introduces the proposed multi-focus image fusion strategy that consists of three steps. First, the undisputed fusion decisions are directly integrated through the results obtained by SVM. Second, the disputed decisions of a given source image pair are extracted, and then the pixels that correspond to the disputed decisions are fused with the proposed MWPCA. Finally, the fused results obtained from the above two stages are synthesized by a logic operation. Figure 1 shows the image fusion strategy. In the first stage, consistency verification (CV) [33] is employed to remove the single singular decisions to correct the misclassifications of the trained SVM, thus an optimized mask is produced. $M_1$ and $M_2$ represent a pair of optimized image fusion decisions, and the integrated results of $M_1$ and $M_2$ are denoted as $M_3$ and $M_4$ respectively. The size of the decision mask is represented as $(x,y)$. The process is shown in Algorithm 3.

Algorithm 3 Fusion Strategy

0: Input: $M_1$ and $M_2$.

1: Output: $M_3$ and $M_4$.

2: function JUDGEMENT ($M_1$,$M_2$.)

3:    for $i=1$→x do

4:    for $j=1$→y do

5:     if ${{\displaystyle M}}_1(i,j)\&\&(1-{{\displaystyle M}}_2(i,j))=1$

6:      ${{\displaystyle M}}_3(i,j)=1$

7:     else

8:     ${{\displaystyle M}}_3(i,j)=0$

9:    if ${{\displaystyle M}}_1(i,j)\&\&(1-{{\displaystyle M}}_2(i,j))=0$

10:     ${{\displaystyle M}}_4(i,j)=1$

11:   else

12:     ${{\displaystyle M}}_4(i,j)=0$

13:    end for

14:   end for

15:   return $M_3$,$M_4$

16: end function

Then, the given multi-focus image pair is fused by the corresponding masks as follows:

$$F_1(i,j)=M_3(i,j)\times i{m}_1(i,j)+M_4(i,j)\times i{m}_2(i,j)$$

(14)

where ${{\displaystyle F}}_1$ is the preliminary fused image.

The disputed fusion decisions are integrated by logic “XOR” operation in the second stage, which is introduced as follows:

$$M_5(i,j)=M_3(i,j)\oplus M_4(i,j)$$

(15)

where $\oplus$ presents the logical “XOR” operation, and $M_5$ is the fusion decisions of disputed areas.

To dispose disputed area $M_5$, two source images are inputted into MWPCA to obtain the secondary fused image ${{\displaystyle F}}_2$. The fusion decision $M_5$ and the fused image ${{\displaystyle F}}_2$ are used to produce the tertiary fused image ${{\displaystyle F}}_3$ that is the complementary set of ${{\displaystyle F}}_1$.

$$F_3(i,j)=(1-M_5(i,j))\times F_2(i,j)$$

(16)

Finally, ${{\displaystyle F}}_1$ and ${{\displaystyle F}}_3$ are integrated to get the fused image $F$:

$$F=F_1+F_3$$

(17)

4. Experimental Results and Analysis

This section first shows two experiments to verify the validity of the proposed MWPCA. Conventional PCA [10,34] and single-scale PCA-based weight (SWPCA) are used to compare with our proposed MWPCA. To further verify the effectiveness of the proposed image fusion method, some popular image fusion algorithms are also employed to compare with our proposed model by six widely-used image metrics. In the feature extraction stage, the sliding window size is set as 9 × 9. In the SVM model training, the libsvm package provided by Professor Lin Zhiren from Taiwan university is used to train and test the performance of the model. The parameters of SVM are optimized by PSO, as g = 400 and c = 0.005. After our repeated experiments, MWPCA with four-scales is suitable for the proposed method. The experimental images are six pairs of popular multi-focus images, which are shown in Figure 4. The evaluation metrics are: edge-based on a similarity measure (Q^AB/F), mutual information (MI), STD, SF, feature mutual information (FMI), and AG.

The comparison methods are: DWT [35], gradient pyramid (GP) [36], MSVD [11], convolutional sparse representation (CSR) [37], fsd pyramid (FSD) [34], discrete cosine harmonic wavelet transform (DCHWT) [14], multi-scale guided image and video fusion (MGFF) [38], multi-exposure and multi-focus image fusion in gradient domain (MMGD) [39], stationary wavelet transform (SWT) [40], image fusion method with Laplacian pyramid transform and pulse coupled neural networks (LPPCNN) [15], image fusion method with fourth order partial differential equations (FPED) [17], image fusion method with boosted random walks-based algorithm (BRWIF) [16]. The proposed image fusion method is denoted by SVM-MWPCA.

Figure 5, Figure 6, Figure 7, Figure 8, Figure 9 and Figure 10 display the source images and the fused images of different image fusion methods. The experiments show that some previous methods cannot fuse the source images effectively. In Figure 5, DWT, GRP, and MSVD cannot fuse the detailed features of the source images, thus the fused images are distorted to some extent. We can clearly see that a good fusion image is not obtained by the FPED method, especially at the junction of focus and multi-focus images. In Figure 6, the fused images of DWT, MSVD, FSD, DCHWT, MMGD, and FPED have obvious distortion. In particular, the fusion image obtained by the FPED method has a serious loss of details. The images fused by our proposed fusion method are superior to those of other methods in terms of edges, details, and textures, and our fused images are most similar to the source images. The enlarged images confirm the above situations. In Figure 7, the fused images of GRP and MSVD have obvious distortion, and the results are worse than other methods. In Figure 8, we found that apart from the FPED method, it is difficult to judge the difference of the fused images of different methods by human eyes. In Figure 9, the fused images obtained by GRP, MSVD, CSR, FSD, DCHWT, and MMGD cannot effectively represent the details of the source images, especially the clear and fuzzy edges. In Figure 10, the difference among the fused images cannot be recognized very well by human eyes, thus some evaluation metrics are employed to verify the performance of different methods. In general, our proposed image fusion method generally produces better visual effect when compared with these of other comparison methods.

By employing the experimental data in

Table 1. Evaluation indexes of the fused images in source image pair “head”.

	QAB/F	MI	STD	SF	FMI	AG
PCA	0.4706	4.8861	56.6482	17.6389	0.7880	9.2269
SWPCA	0.7556	5.3798	60.0055	27.8297	0.7912	14.3047
MWPCA	0.7681	5.4800	60.2857	28.7283	0.7910	14.7699
DWT	0.7037	5.5991	58.9731	34.3882	0.7847	17.9566
GP	0.7240	4.7167	57.1455	32.0316	0.7820	16.7318
MSVD	0.5839	4.6095	58.1864	31.5158	0.7759	16.4240
CSR	0.7763	5.0208	59.3009	31.4408	0.7867	16.0126
FSD	0.7230	4.7195	57.1382	32.0335	0.7812	16.7339
DCHWT	0.7413	5.3390	61.9662	33.3022	0.7863	17.3192
MGFF	0.7697	4.7094	67.4080	32.2792	0.7857	16.5804
MMGD	0.7707	3.5486	58.3318	42.6904	0.7830	23.0448
SWT	0.7716	4.9537	59.8963	34.3623	0.7870	17.9258
LPPCNN	0.7945	6.1172	60.3058	34.4343	0.7885	17.9337
BRWIF	0.7985	7.4392	62.9315	33.5246	0.7894	17.2717
FPED	0.4409	4.5751	57.2435	23.4089	0.7419	11.7576
SVM-MWPCA	0.8039	8.0117	63.2394	34.2334	0.7871	17.7114

,

Table 2. Evaluation indexes of the fused images in source image pair “office”.

	QAB/F	MI	STD	SF	FMI	AG
PCA	0.5238	5.2868	59.3659	14.2466	0.8891	5.6931
SWPCA	0.6406	5.0307	62.1808	18.2687	0.8892	6.9143
MWPCA	0.6485	5.0476	62.5232	18.7837	0.8896	7.0665
DWT	0.6142	4.9734	64.0152	24.7496	0.8857	9.3293
GP	0.6261	4.7114	58.2125	20.0419	0.8839	7.5131
MSVD	0.4490	4.7764	59.6761	19.0134	0.8768	6.9902
CSR	0.6688	5.0989	63.3177	22.6544	0.8896	8.0580
FSD	0.6247	4.6958	58.1493	20.0650	0.8841	7.5303
DCHWT	0.6446	5.0586	66.0015	23.9874	0.8873	8.9441
MGFF	0.6415	4.7869	71.1440	22.9285	0.8860	8.2877
MMGD	0.6519	3.8418	58.3996	18.4592	0.8800	7.9927
SWT	0.6685	5.0893	65.7365	25.1233	0.8897	9.2708
LPPCNN	0.6993	6.1530	66.3773	25.3597	0.8907	9.3527
BRWIF	0.7030	6.8735	67.8244	24.9213	0.8898	9.0464
FPED	0.2969	3.6333	61.9545	29.8877	0.7831	14.6584
SVM-MWPCA	0.7123	7.3404	68.3761	25.5976	0.8899	9.3341

,

Table 3. Evaluation indexes of the fused images in source image pair “boat”.

	QAB/F	MI	STD	SF	FMI	AG
PCA	0.5431	5.1153	46.6333	12.5158	0.8620	5.7731
SWPCA	0.6742	5.2069	47.5695	15.2652	0.8662	7.0489
MWPCA	0.6825	5.2338	47.6727	15.5633	0.8666	7.1840
DWT	0.6845	5.0368	50.1138	20.7918	0.8639	9.7672
GP	0.6830	4.7941	45.1592	16.7996	0.8640	7.8353
MSVD	0.5900	4.8608	47.3814	17.2383	0.8566	7.9434
CSR	0.7227	5.3809	48.8383	18.7100	0.8655	8.4201
FSD	0.6792	4.7847	45.1333	16.8615	0.8635	7.8745
DCHWT	0.6994	5.3117	49.1159	19.1425	0.8638	8.8718
MGFF	0.6656	5.2320	50.1745	17.4931	0.8602	7.8530
MMGD	0.6999	3.1949	53.4840	24.3752	0.8585	12.0170
SWT	0.7229	5.6105	50.3197	20.8033	0.8665	9.6473
LPPCNN	0.7481	6.6954	50.2184	20.8252	0.8682	9.6278
BRWIF	0.7484	7.4085	50.3348	20.6533	0.8678	9.4998
FPED	0.5229	4.6411	47.1768	15.7169	0.8354	7.6312
SVM-MWPCA	0.7419	7.0596	50.2332	20.3844	0.8678	9.3424

,

Table 4. Evaluation indexes of the fused images in source image pair “flora”.

	QAB/F	MI	STD	SF	FMI	AG
PCA	0.5819	6.0339	53.5386	10.5907	0.9281	3.3497
SWPCA	0.7612	6.2699	54.8591	17.4445	0.9313	4.7632
MWPCA	0.7653	6.2761	54.8703	17.7690	0.9311	4.8457
DWT	0.7345	6.7702	56.9598	20.1160	0.9267	5.8923
GP	0.7231	5.8700	55.2217	17.3743	0.9261	4.9874
MSVD	0.4997	5.5913	53.7290	13.8294	0.9138	3.9497
CSR	0.7348	5.6427	55.1162	18.6646	0.9292	5.0100
FSD	0.7218	5.8701	55.2136	20.3445	0.9238	6.6982
DCHWT	0.7621	6.4891	55.1291	20.0433	0.9299	5.6974
MGFF	0.6954	5.1374	59.4223	16.2773	0.9219	5.2609
MMGD	0.7753	6.4414	57.2308	20.1240	0.9237	6.6162
SWT	0.5705	8.1594	55.3159	20.2009	0.9296	5.8092
LPPCNN	0.7957	7.2606	66.2374	26.3672	0.8952	10.8548
BRWIF	0.7563	9.5549	55.0321	20.5896	0.9261	5.5549
FPED	0.5738	5.7000	53.5262	10.9073	0.9274	3.4974
SVM-MWPCA	0.8078	8.5955	55.2287	20.8230	0.9294	5.8873

,

Table 5. Evaluation indexes of the fused images in source image pair “wine bottle”.

	QAB/F	MI	STD	SF	FMI	AG
PCA	0.4771	5.3686	61.3207	14.2079	0.8909	5.8415
SWPCA	0.7393	5.7836	63.6277	20.9928	0.8937	8.5928
MWPCA	0.7445	5.8380	63.7936	21.4030	0.8936	8.7532
DWT	0.7410	5.5586	66.2019	26.2938	0.8938	10.8986
GP	0.7139	5.2029	61.2615	20.6140	0.8884	8.5945
MSVD	0.3893	4.9802	61.4478	19.6539	0.8781	8.2799
CSR	0.7587	5.6724	64.4768	21.0188	0.9062	7.5273
FSD	0.7125	5.1914	61.2383	20.6312	0.8884	8.6113
DCHWT	0.7444	5.7979	65.4574	25.0666	0.8938	10.2544
MGFF	0.6941	4.9002	68.3972	23.0008	0.8828	9.2334
MMGD	0.7421	3.9215	54.7576	22.4013	0.8757	10.0876
SWT	0.7698	5.8674	66.2809	26.2314	0.8936	10.9360
LPPCNN	0.7957	7.2606	66.2374	26.3672	0.8950	10.8548
BRWIF	0.7974	8.1721	66.5152	26.3142	0.8950	10.7810
FPED	0.4536	5.2547	61.3751	14.8585	0.8840	6.8271
SVM-MWPCA	0.7990	8.2273	66.5312	26.3726	0.8951	10.8262

and

Table 6. Evaluation indexes of the fused images in source image pair “bread”.

	QAB/F	MI	STD	SF	FMI	AG
PCA	0.5523	6.8742	71.0656	12.5951	0.9027	4.8738
SWPCA	0.6783	6.9589	71.6105	14.9799	0.9054	5.7183
MWPCA	0.6867	6.9877	71.6875	15.3606	0.9055	5.8529
DWT	0.6968	6.8028	72.6304	22.8224	0.9034	8.7466
GP	0.6914	6.3610	70.1465	20.1881	0.9026	7.7378
MSVD	0.4527	6.4728	71.1557	15.9845	0.8867	6.2920
CSR	0.7249	7.0324	72.3162	21.0188	0.9062	7.5273
FSD	0.6869	6.3925	70.0854	18.7237	0.9037	7.0463
DCHWT	0.7210	7.3458	72.5087	20.4694	0.9102	7.5167
MGFF	0.6876	6.6399	73.5398	18.1895	0.9050	6.5866
MMGD	0.7094	4.1615	66.1882	20.2428	0.8941	8.4643
SWT	0.7424	7.2949	72.9396	21.4359	0.9083	8.0236
LPPCNN	0.7544	7.9824	73.0340	21.7214	0.9084	8.1477
BRWIF	0.7627	8.9794	73.6095	23.0642	0.9071	8.7368
FPED	0.5816	6.8390	71.1643	14.3646	0.8889	5.7833
SVM-MWPCA	0.7634	8.9921	73.5932	23.0513	0.9070	8.7515

, we can find that the proposed MWPCA has the largest values of Q^AB/F and MI in source images “head”, “office”, “boat”, “wine bottle”, and “bread” when compared with the conventional PCA and SWPCA methods. For the source image “flora”, the MWPCA method has the best values of Q^AB/F. Q^AB/F and MI are the two most crucial evaluation metrics in image fusion. MWPCA has the largest values in almost all of the rest evaluated metrics. The fused images obtained by MWPCA have much better clarity than those of conventional PCA methods. Thus, the fusion image obtained by the proposed MWPCA has better visual effects and more superior objective indicators.

The comparison of the evaluation indexes of different image fusion methods is provided in Table 1, Table 2, Table 3, Table 4, Table 5 and Table 6. Generally, four digits are used in the field of image fusion because some indicators are approximate. Among the above evaluation metrics, Q^AB/F and MI are the most important parameters to evaluate the fused image quality. The Q^AB/F metrics indicate how much edge information from the source image is retained. The MI metrics indicate how much source image information is transferred to a fused image. Other indicators include metrics as auxiliary indicators. The higher the evaluation metrics value, the higher the fused image quality. Table 1 shows that the Q^AB/F and MI values of the proposed method are the largest in “head”. Table 2 shows that the Q^AB/F and MI values of the proposed method are the largest in “office”. Table 3 displays the values of Q^AB/F are the second largest in “boat”, which is only 0.0062 below the maximum. Table 4 shows our proposed image fusion method can obtain the best values for the source image pair “flora” in Q^AB/F indexes. Table 5 shows that the proposed image fusion method can obtain the best values for the source image pair “wine bottle” in Q^AB/F, MI, and SF. Table 6 shows that the proposed method can obtain the best values for the source image pair “bread” in Q^AB/F, MI, STD, and AG. According to these experiments, we can find that our proposed image fusion method always has the best values of Q^AB/F and MI. Among other indicators, metric values fluctuate due to the calculation method. The STD, SF, and AG are independent of the source images and only depend on the fused images. Therefore, STD, SF, and AG are not always effective to analyze the fused images. However, the values of STD, SF, and AG are better than most of the other methods. To sum up, our proposed image fusion method has better performance compared with those of other comparison methods.

5. Conclusions

This work proposes a novel multi-focus image fusion method based on SVM and an improved multi-scale PCA-based pixel weighted method. Moreover, the logic operations are also employed to optimize the fusion decisions. The experimental results reveal that the fused images obtained by our proposed method are superior to those of other comparison fusion methods. The used regional feature extraction method can present the important information of the focused and unfocused regions in source images, and the proposed image fusion method can cover the shortage of the misclassification of SVM. Moreover, our new proposed multiscale PCA-based image fusion is used to handle the disputed regions to overcome the weakness of conventional PCA methods, and the experiments confirmed the performance of the new PCA-based method. Our future research will be aimed at exploring some new local feature extraction methods. Moreover, the advanced machine learning methods are also expected to be applied to image fusion.