A Novel Geometric Dictionary Construction Approach for Sparse Representation Based Image Fusion

Abstract

Sparse-representation based approaches have been integrated into image fusion methods in the past few years and show great performance in image fusion. Training an informative and compact dictionary is a key step for a sparsity-based image fusion method. However, it is difficult to balance “informative” and “compact”. In order to obtain sufficient information for sparse representation in dictionary construction, this paper classifies image patches from source images into different groups based on morphological similarities. Stochastic coordinate coding (SCC) is used to extract corresponding image-patch information for dictionary construction. According to the constructed dictionary, image patches of source images are converted to sparse coefficients by the simultaneous orthogonal matching pursuit (SOMP) algorithm. At last, the sparse coefficients are fused by the Max-L1 fusion rule and inverted to a fused image. The comparison experimentations are simulated to evaluate the fused image in image features, information, structure similarity, and visual perception. The results confirm the feasibility and effectiveness of the proposed image fusion solution.

1. Introduction

High-quality images can help increase the accuracy and efficiency of image processing and related analysis. However, a single sensor cannot capture sufficient information in one scenario. For obtaining more information, image fusion techniques are used to combine multiple images from the same scenario. Now image fusion techniques are widely used in different areas, such as computer vision, medical diagnosis and treatment, and remote sensing. Various image fusion algorithms are proposed and used in diverse applications.

According to the spatial and transformative features of fusion domain, these methods could be categorized into two main categories: spatial-domain-based methods and transform-domain-based methods [1,2]. Spatial-domain-based methods directly choose clear pixels, blocks, or regions from source images to compose a fused image [3,4,5]. Some simple methods, such as averaging or max pixel schemes, are performed on single pixel to generate fused image. However, these methods may reduce the contrast and edge intensity of the fused result. In order to improve the quality of fused image, some advanced algorithms, such as block-based and region-based algorithms, were developed. Li et al. proposed a scheme by dividing images into blocks and chose the focused one by comparing spatial frequencies (SF) first; then, the fused results are produced by consistency verification [6,7]. Although block-based methods improve the contrast and sharpness of integrated image, they may cause a block effect in the integrated image [8,9].

Different from spatial-domain fusion methods, transform-domain methods transform source images into a few corresponding coefficients and transform bases first [10,11]. Then, the coefficients are fused and inverted to an integrated image. Multi-scale transform (MST) and wavelet based algorithms are conventional transform approaches applied to transform-domain-based image fusion [12,13,14], such as wavelet transform [15,16], shearlet [17,18], curvelet [19], dual tree complex wavelet transform [20,21], and nonsubsampled contourlet transform (NSCT) [22]. MST decomposition methods have attracted great attention in the image processing field, and are widely used in image fusion fields. However, MST-based methods need priori knowledge to select an optimal transform basis [23]. As each MST method has its own limitations, one MST method is difficult to fit all kinds of images [12].

Recently, sparse-representation based methods show great performance in image de-noising [24], image de-blurring [25], image target tracking [26,27], and image super-resolution [28,29]. Sparse-representation based methods decompose an image patch using a few number of bases or atoms of a fixed or trained dictionary.

In the image fusion field, a sparse-representation based method was first proposed by Yang and Li [30]. They applied the Discrete Cosine Transform(DCT) dictionary and orthogonal matching pursuit (OMP) method to sparse-representation based multi-focus image fusion. Liu et al. [31] presented a sparse-representation based method using an NSCT filter for image decomposition and the DCT dictionary for sparse coding of image patches. Yin et al. [32] used a dual-tree complex shearlet transform dictionary for image fusion, which enhanced the contrast of image details.

Previously mentioned methods used a fixed dictionary for spare representation. However, a fixed dictionary cannot adaptively change according to input images. As the dictionary is one of the most crucial parts in sparse representation, a trained dictionary according to input images has better performance in describing source images. Selecting a good over-complete dictionary is the main issue of a sparse representation-based image reconstruction and fusion techniques. Mostly, the DCT basis or wavelet basis are often used for an over-complete dictionary. Since such dictionaries formed with transform bases do not rely on input image data, they are fixed regardless of type of sensors, context of images, or applications. While a fixed dictionary can easily be implemented, their performance is somewhat limited depending on the type of data and application. To make a dictionary adaptive to input image data, a dictionary learning method has been developed by Aharon et al. [33]. Yin et al. [34] developed an image fusion method based on K-means generalized singular value decomposition (K-SVD) [33], which also explored the sparse parameter in image fusion. Wang et al. [35] proposed an approximate K-SVD-based sparse representation method for image fusion and exposure fusion to reduce computation costs in dictionary learning. To make the trained dictionary more informative, Kim et al. [36] proposed a compact dictionary learning method called joint clustering patches dictionary learning (JCPD). JCPD used image pixel clustering and principal component analysis (PCA) bases to train sub-dictionaries in dictionary construction. Fusion results showed that the detailed information from source images was perfectly reserved. Zhu et al. [9,37,38] presented an image patch clustering method and applied it to corresponding sub-dictionary training process. Their method improved the detailed features in medical image fusion. All of these sparse-representation based image fusion methods only used one dictionary for sparse coding of all image patches, which may cause redundancy in the constructed dictionary.

Geometric information, as one type of the most important image information, including edges, contours, and textures of image, can remarkably influence the quality of image perception [39,40]. This information can be used in patch classification as a supervised dictionary prior to improving the performance of the trained dictionary [41,42]. In this paper, a geometric classification based dictionary learning method is proposed for sparse-representation based image fusion. Instead of grouping the pixels of images, the proposed geometric classification method groups image blocks directly by the geometric similarity of each image block. Since a sparse-representation based fusion method uses image blocks for sparse coding and coefficient fusion, extracting underlying geometric information from image-block groups is an efficient way to construct a dictionary. Moreover, the geometric classification can group image blocks based on edge and sharp line information for dictionary learning, which can improve the accuracy of sparse representation. This paper has two main contributions.

• A geometric-information based classification method is proposed and applied to a sub-dictionary learning of image patches. The proposed classification method can accurately split source image patches into different groups for sub-dictionary learning based on the corresponding geometry features. Sub-dictionary bases extracted from each image-patch group contain the key geometry features of source images. These extracted sub-dictionary bases are trained to form informative and compact sub-dictionaries for image fusion.
• A dictionary combination method is developed to construct an informative and compact sub-dictionary. Each image patch of a fused image is composed of corresponding source image patches using a constructed-sub-dictionary (CSD). According to the classification of geometry features, each source image patch is trained and categorized into a group of sub-dictionaries. Corresponding image patches, that appear at the same place of the two source images, at most have two groups of sub-dictionary. Redundant geometric information of source image patches is eliminated.

The remaining sections of this paper are structured as follows: Section 2 proposes the geometric sub-dictionary learning method and integrated image fusion framework; Section 3 compares and analyzes experimentation results; and Section 4 concludes this paper.

2. Geometry-Based Image Fusion Framework

This section presents the proposed image fusion method. The proposed method consists of geometric similarity based sub-dictionary learning and sparse representation based image fusion processes. In the sub-dictionary learning step, images are split into image patches first. The image patches are clustered into a few groups based on the geometric similarity. The K-SVD method is used in sub-dictionary training. In the image fusion step, image patches of source images are sparse coded using an assembled dictionary. The assembled dictionary consists of sub-dictionaries, which are corresponding to the groups of input image patches. When image patches are sparse coded, the coded coefficients are fused by using the Max-L1 fusion rule [30].

2.1. Dictionary Learning

The proposed dictionary learning method is shown in Figure 1, in which source images are split into 8 × 8 image patches. In the proposed method, source images are split into 8 × 8 image patches by sliding windows. These image patches are transformed to vectors of 1 × 64 in a linewise direction and normalized between 0 and 1. Then, these image patches can be clustered into a few groups for sub-dictionary learning. These sub-dictionaries can preserve key information of each image patch group. There are six specific groups of sub-dictionaries in this paper.

The geometric image patches clustering method can classify all of the image patches into three main groups, such as smooth, dominant orientation, and stochastic patch group. In the proposed method, image patches are classified into smooth and non-smooth first. A variance based method is proposed for grouping smooth and non-smooth patches. For image patches $(p_1,p_2,...,p_n)$, the corresponding variances are $(v_1,v_2,...,v_n)$. If the variance $c_i$ of ith image patch meets $c_i<\delta$, the ith image patch $p_i$ is considered as a smooth image patch.

In this way, image patches can be classified into smooth and non-smooth patches. Non-smooth patches can be further clustered into dominant orientation and stochastic patch group, by calculating the dominant orientations of patches. The dominant orientation estimation method is based on the singular value decomposition (SVD). The gradient of an image pixel $g_i$ can be calculated by Equation (1).

$$\begin{array}{c}\hfill \begin{array}{c}\hfill g_i=\left[\left(\partial i\right(x,y)/\partial x),\left(\partial i\right(x,y)/\partial y)\right]\end{array},\end{array}$$

(1)

where $g_i$ is the gradient map.

The gradient map of an image patch $p_i=[i_1,i_2,...,i_n]$, which consists of n pixels, is shown in Equation (2):

$$\begin{array}{c}\hfill \begin{array}{c}\hfill g_{p_i}=\left[g_1,g_2,...,g_n\right]\end{array},\end{array}$$

(2)

where $g_{p_i}$ is the gradient map of $p_i$.

Performing an SVD on $g_{p_i}$ can obtain $g=SV{D}^T$. Extracting the first column of V can obtain the dominant orientation of the gradient field $v_1$. The second column $v_2$ in V is the subdominant orientation of the gradient field. If the corresponding singular values of $v_1$ and $v_2$ are remarkably different, the dominant measure [43] R can be calculated using Equation (3):

$$\begin{array}{c}\hfill \begin{array}{c}\hfill R=\frac{S_{1,1}-S_{2,2}}{S_{1,1}+S_{2,2}}\end{array}.\end{array}$$

(3)

After SVD decomposition, $S_{1,1}$ and $S_{2,2}$ are row 1-column 1 and row 2-column 2 values of singular value matrix V, respectively. If R is smaller than a significance level threshold $R^{*}$, the image patch is considered as a stochastic pattern.

In order to differentiate the geometric information of dominant orientation patches, the dominant orientation image patches can be further classified to different groups according to the directions of image patches. The direction d of dominant orientation image patches can be estimated using gradient field $v_1$, which is shown in Equation (4):

$$\begin{array}{c}\hfill \begin{array}{c}\hfill d=arctan\left(\frac{v_1\left(2\right)}{v_1\left(1\right)}\right)\end{array}.\end{array}$$

(4)

In the proposed dictionary learning framework, dominant orientation image patches are classified into four groups, such as horizontal, right-direction, vertical, and left-direction patch group that correspond to 0, 45, 90, and 135 degree group, respectively. In Equation (4), when d is close to the horizontal, right-direction, vertical, or left-direction patch group, d is clustered into the corresponding group. For each group, a sub-dictionary can be trained by the stochastic coordinate coding (SCC) algorithm shown in Algorithm 1, which is extremely fast. H represents a Hessian matrix of the objective function. To obtain the learning rate, SCC uses the Hessian matrix of objective function. According to the second order stochastic gradient descent, it should inverse the Hessian matrix as the learning rate [44]. z is obtained by using the simultaneous orthogonal matching pursuit (SOMP) algorithm shown in Algorithm 2 to sparse code image patches based on dictionary D. The trained sub-dictionaries for different geometric groups are shown at the bottom of Figure 1.

In the SOMP algorithm, K is the sum of image patch x and sparse coefficient z, $x_k$ is k-th image patch, and $z_k$ is k-th sparse coefficient. In this paper, it assumes that the source images are all noise free. Thus, a small global error is set, i.e., $\epsilon =0.01$.

Algorithm 1 SCC Algorithm.

Input:  Image patches of Wth cluster $P^w=(p_1^w,p_2^w,...,p_n^w)\in {\mathbb{R}}^{64\times n}$ Output:  Sub-dictionary $S=S_n^{\partial }\in {\mathbb{S}}^{64\times n}$, and $z_i=z_i^{\partial }$ for $i=1,2,...,n$  Initialize $S_1^1$, $H=0$, and $z_i^0=0$ for $i=1,2,...,n$  for $k=1$ to ∂ do   for $i=1$ to n do    Get image patch $p_i^n$    Update $z_i^k$ via one or a few steps of coordinate descent: $z_i^k\leftarrow CD(D_i^k,z_i^{k-1},x_i)$    Update the Hessian matrix and the learning rate: $H\leftarrow H+z_i^k{\left(z_i^k\right)}^T,{\eta }_{i,j}^k=1/h_{jj}$    Update the support of the dictionary via SGD (Stochastic Gradient Descent): $d_{i+1}^k\leftarrow d_{i,j}^k-{\eta }_{i,j}^k{z}_{i,j}(S_i^k{z}_i^k-x_i)$    if $i=n$ then     Set $S_1^{k+1}=S_{n+1}^k$    end if   end for  end for

Algorithm 2 SOMP Algorithm.

Input:  Dictionary D, image patches ${\left\{x_k\right\}}_{k=1}^K$, $x_k\in R^n$, threshold $\epsilon$, an empty matrix $\Phi$ Output:  Sparse coefficients ${\left\{z_k\right\}}_{k=1}^K,z_j\in R^{\mathrm{w}}$  Initialize the residuals $r_k^{\left(0\right)}=x_k$, for $k=1,2,...,K$, set iteration counter $l=1$.  Select the index ${\widehat{t}}_l$ which indicates the next best coefficient atom to simultaneously provide good reconstruction for all signals by solving: ${\widehat{t}}_l=arg\underset{t=1,2,...,T}{max}{\displaystyle \sum _{k=1}^K}\left|〈r_k^{l-1},d_t〉\right|$.  Update sets $\Phi =\left[{\Phi }_{l-1},d_{\widehat{t_l}}\right]$.  Compute new coefficients (sparse representations), approximations, and residuals as:  ${\alpha }_k^{\left(l\right)}=\underset{\alpha }{argmin}{∥x_k-{\Phi }_l\alpha ∥}_2={\left({{\Phi }_l}^T{\Phi }_l\right)}^{-1}{{\Phi }_l}^T{x}_k$, for $k=1,2,...,K$,  ${\widehat{x}}_k^{\left(l\right)}={\Phi }_t{\alpha }_k^{\left(l\right)}$, for $k=1,2,...,K$,  $r_k^{\left(l\right)}=x_k-{{\widehat{x}}_k}^{\left(l\right)}$, for $k=1,2,...,K$.  Increase the iteration counter $l=l+1$, if ${\displaystyle \sum _{k=1}^K}{∥r_k^{\left(l\right)}∥}_2^2>{\epsilon }^2$, go back to step 2.

2.2. Image Sparse Coding and Fusion

When all image-patch groups are trained, source images can be fused by using the trained sub-dictionaries. The proposed image sparse-representation and fusion method are shown in Figure 2. In the proposed solution, the fused image patch can be obtained by corresponding sub-dictionaries. It is an efficient way to decrease the size of learned dictionaries. All image patches are clustered into six groups. Any two aligned image patches of source images at most belong to two groups. Even if two aligned image patches belong to the same group, two corresponding sub-dictionaries are different. Since one group only has a sub-dictionary, it only needs two sub-dictionaries to represent all information of two image blocks.

The proposed fusion method uses sliding windows and sets overlap as six, so the splitting overlap of source images has six image patches. Suppose two source images for fusion have already been split into image patches. According to the classification method mentioned in the previous section, all of these image patches are clustered into a few groups. The pair of image patches from the same location of source images are sparse coded by using a CSD. In accordance with the classified image patch groups, one CSD of two corresponding source image patches at most consists of two sub-dictionaries. The CSD construction algorithm shown in Algorithm 3 combines the corresponding sub-dictionaries of two source image patches to obtain dictionary D.

When all of the CSDs are constructed, any pair of source image patches can be sparse coded by using the corresponding CSD and SOMP algorithm. Assume that there are K registered source images, $I_1,...,I_j$ with size of $M\times N$. The Max-L1 fusion rule takes the following steps.

• Step 1: Use the sliding window technique to divide each source image $I_j$, from left-top to right-bottom, into patches of size $8\times 8$, i.e., the size of the atom in the dictionary. These image patches are vectorized to image pixel vectors in the linewise direction. The obtained image pixel vectors only have one dimension.
• Step 2: For the ith image patch $x_{ji}$ of one source image $I_j$, it can be sparse coded using the trained dictionary D.
• Step 3: When all of the image patches are sparse coded, the corresponding image patches of each image use Equation (5) to do fusion:

  $$\begin{array}{c}\hfill \begin{array}{c}\hfill {{\mathbf{z}}^f}_i=\sum _{j=1}^k{\mathbf{z}}_{ji}\times O_{ji},where\left\{\begin{array}{c}{O}_{ji}=1,ifmax({∥{\mathbf{z}}_{j1}∥}_1,{∥{\mathbf{z}}_{j2}∥}_1...,{∥{\mathbf{z}}_{jk}∥}_1)={∥{\mathbf{z}}_{ji}∥}_1\\ O_{ji}=0,otherwise\end{array}\right.\end{array},\end{array}$$

  (5)

  where $z_{ji}$ is a sparse coefficient corresponding to the i-th image patch in j-th image $p_{ji}$.
• Step 4: Fused coefficients are inversely transformed to fused image pixel vectors, using Equation (6), and transform these vectors back to the fused image patches. Then, it reconstructs the fused image by using fused image patches. The dictionary D in Equation (6) is the same as dictionary D in Algorithm 3:

  $$\begin{array}{c}\hfill \begin{array}{c}\hfill I^f=D{z}^f\end{array}.\end{array}$$

  (6)

Algorithm 3 CSD Construction Algorithm.

Input:  Sub-dictionaries $S=(S_1,S_2...S_n)$, image patches for fusion $i_1$ and $i_2$ Output:  CSD D  Figure out sub-dictionaries $S_j\in S$, $S_k\in S$, that correspond to image patch groups of $i_1$ and $i_2$.  if$j=k$then   Set $D=S_j$  else   Set $D=[S_j,S_k]$  end if

3. Experiments and Analyses

To test the efficiency, the proposed image fusion approach is applied to multi-focus, medical, and visible-infrared images, respectively.

• Twenty pairs of $520\times 520$ size multi-focus images are obtained from the Lytro-multi-focus data-set http://mansournejati.ece.iut.ac.ir.
• Thirty pairs of medical images are from www.med.harvard.edu/aanlib/home.html. All of them are $256\times 256$ size.
• Ten pairs of visible-infrared images are obtained from from www.quxiaobo.org consisting of four $320\times 240$ and six $256\times 256$ image pairs.

Figure 3a–f show the selected sample pairs of multi-focus, medical, and visible-infrared images, respectively.

This paper assumes that the input image pairs are precisely co-aligned. All image pairs are from the standard library. They have the same size. The proposed solution can also be applied to multiple images. In this section, one experiment of each image type is chosen and presented respectively in the following sections. To show the efficiency of the proposed method, the state-of-the-art dictionary learning based sparse-representation fusion approaches K-SVD and JCPD , which were proposed by Li et al. in 2012 [45] and Kim et al. in 2016 [36], respectively, are used for comparison. The experiments are evaluated by both subjective and objective assessments. Five popular image fusion quality metrics are used in this paper for the quantitative evaluation. The larger the metric value is, the better the performance is. The patch size of all sparse-representation-based methods including the proposed method are set to $8\times 8$. To avoid blocking artifacts, all experiments use the sliding window scheme [36,45,46]. The overlapping region of the sliding window is set to six pixels in each vertical and horizontal direction of all experiments. All experiments are performed using a 2.60GHz single processor of an Intel(R) Core(TM) i7-4720HQ CPU Laptop with 12.00GB RAM. To compare fusion results fairly, all experiments in this paper are programmed by Matlab code in a Matlab 2014a environment.

3.1. Objective Evaluation Methods

Five mainstream objective evaluation metrics are implemented for the quantitative evaluation. These metrics include edge retention ($Q^{AB/F}$) [47], mutual information (MI) [48], visual information fidelity (VIF) [49],the Yang proposed fusion metric ($Q_Y$) [50,51], and the Chen–Blum metric ($Q_{CB}$) [51,52]. The above five solutions are classical approaches used in multi-focus, multi-modality medical, and infrared-visible image fusion. $Q^{AB/F}$ is the image feature-based metric. MI is the information theory-based metric. $Q_Y$ is the image structural similarity-based metric. $Q_{CB}$ and VIF are human perception inspired fusion metrics. According to objective assessment [51,53,54], these metrics can objectively evaluate the fused image in image feature, information, structure similarity, and visual perception. Thus, our paper chooses these metrics. For the fused image, the sizes of $Q^{AB/F}$, MI, VIF, $Q_Y$, and $Q_{CB}$ become bigger, and the corresponding fusion results are better.

3.1.1. Mutual Information

MI for images can be formalized as Equation (7):

$$\begin{array}{c}\hfill \begin{array}{c}\hfill MI=\sum _{i=1}^L \sum _{j=1}^L{h}_{A,F}(i,j)lo{g}_2\frac{h_{A,F}(i,j)}{h_A\left(i\right)h_{F\left(j\right)}}\end{array},\end{array}$$

(7)

where L is the number of gray-level, $h_{A,F}(i,j)$ is the gray histogram of image A and F. $h_A\left(i\right)$ and $h_F\left(j\right)$ are edge histogram of image A and F. Edge histogram is used to present the edge information of image [48]. MI of the fused image can be calculated by Equation (8):

$$\begin{array}{c}\hfill \begin{array}{c}\hfill MI(A,B,F)=MI(A,F)+MI(B,F)\end{array},\end{array}$$

(8)

where $MI(A,F)$ represents the $MI$ value of input image A and fused image F; $MI(B,F)$ represents the $MI$ value of input image B and fused image F.

3.1.2. $Q^{AB/F}$

$Q^{AB/F}$ metric is a gradient-based quality index to measure how well the edge information of source images is conducted to the fused image. It is calculated by:

$$\begin{array}{c}\hfill \begin{array}{c}\hfill Q^{AB/F}=\frac{{\sum }_{i,j}(Q^{AF}(i,j)w^A(i,j)+Q^{BF}(i,j)w^B(i,j))}{{\sum }_{i,j}(w^A(i,j)+w^B(i,j))}\end{array},\end{array}$$

(9)

where $Q^{AF}=Q_g^{AF}{Q}_0^{AF}$, $Q_g^{AF}$ and $Q_0^{AF}$ are the edge strength and orientation preservation values at location (i,j). $Q^{BF}$ can be computed similarly to $Q^{AF}$ . $w_A(i,j)$ and $w_B(i,j)$ are the weights of $Q^{AF}$ and $Q^{BF}$, respectively.

3.1.3. Visual Information Fidelity

$VIF$ is the novel full reference image quality metric. $VIF$ quantifies the mutual information between the reference and test images based on the Natural Scene Statistics (NSS) theory and the Human Visual System (HVS) model. It can be expressed as the ratio between the distorted test image information and the reference image information, and the calculation equation of $VIF$ is shown in Equation (10):

$$\begin{array}{c}\hfill \begin{array}{c}\hfill VIF=\frac{{\sum }_{i\in subbands}I(\overrightarrow{C^{N,i}};\overrightarrow{F^{N,i}})}{{\sum }_{i\in subbands}I(\overrightarrow{C^{N,i}};\overrightarrow{E^{N,i}})}\end{array},\end{array}$$

(10)

where $I(\overrightarrow{C^{N,i}};\overrightarrow{F^{N,i}})$ and $I(\overrightarrow{C^{N,i}};\overrightarrow{E^{N,i}})$ represent the mutual information, which are extracted from a particular subband in the reference and the test images, respectively. Here, subband means the frequency of human eye sensory. Thus, this subband is used to evaluate the visual performance objectively [49,55]. $\overrightarrow{C^N}$ denotes N elements from a random field, and $\overrightarrow{E^N}$ and $\overrightarrow{F^N}$ are visual signals at the output of HVS model from the reference and the test images, respectively.

To evaluate the VIF of fused image, an average of VIF values of each input image and the integrated image is proposed [49]. The evaluation function of VIF for image fusion is shown in Equation (11):

$$\begin{array}{c}\hfill \begin{array}{c}\hfill VIF(A,B,F)=\frac{VIF(A,F)+VIF(B,F)}{2}\end{array},\end{array}$$

(11)

where $VIF(A,F)$ is the $VIF$ value between input image A and fused image F; $VIF(B,F)$ is the $VIF$ value between input image B and fused image F.

3.1.4. $Q_Y$

Yang et al. proposed a structural similarity-based way for fusion assessment [50]. Yang’s method is shown in Equation (12):

$$\begin{array}{c}\hfill \begin{array}{c}\hfill Q_Y=\left\{\begin{array}{c}\begin{array}{c}\lambda \left(\omega \right)SSIM(A,F\left|\omega \right.)+(1-\lambda \left(\omega \right))SSIM(B,F\left|\omega \right.),\hfill \\ SSIM(A,B\left|\omega \right.)\ge 0.75,\hfill \end{array}\\ \begin{array}{c}max\left\{SSIM(A,F\left|\omega \right.),SSIM(A,B\left|\omega \right.)\right\},\hfill \\ SSIM(A,B\left|\omega \right.)<0.75,\hfill \end{array}\end{array}\right.\end{array}\end{array}$$

(12)

where $\lambda \left(\omega \right)$ is the local weight, and $SSIM(A,B)$ is a structural similarity index measure for images A and B. The detail of $\lambda \left(\omega \right)$ and $SSIM(A,B)$ can be found in [50,51].

3.1.5. $Q_{CB}$

The Chen–Blum metric is a human perception inspired fusion metric. The Chen–Blum metric consists of five steps:

The first step is filtering image $I(i,j)$ in the frequency domain. $I(i,j)$ is transformed to the frequency domain and gets $I(m,n)$. Filtering $I(m,n)$ by the contrast sensitive function (CSV) [52,56] filter $S\left(r\right)$, where $r=\sqrt{m^2+n^2}$. In this image fusion metric, $S\left(r\right)$ is in polar form. $\tilde{I}(m,n)$ can be obtained by $\tilde{I}(m,n)=I(m,n)\times S\left(r\right)$.

In the second step, local contrast is computed. Considering the band-pass filters of a pyramid transform, which can be obtained as the difference of two neighboring low-pass filters. For the $Q_{CB}$ metric, Peli’s contrast C is used in this paper, and it can be defined as:

$$\begin{array}{c}\hfill \begin{array}{c}\hfill C(i,j)=\frac{{\varphi }_k(i,j)\times I(i,j)}{{\varphi }_{k+1}(i,j)\times I(i,j)}-1\end{array}.\end{array}$$

(13)

A common choice for ${\varphi }_k(i,j)$ would be a Gaussian kernel that is shown as follows:

$$\begin{array}{c}\hfill \begin{array}{c}\hfill G_k(i,j)=\frac{1}{\sqrt{2\pi }{\sigma }_k}{e}^{\frac{i^2+j^2}{2{\sigma }_k^2}}\end{array},\end{array}$$

(14)

where k and k + 1 stand for two neighboring low-pass filters ${\sigma }_k=2^k$.

In the third step, the masked contrast map for input image $I_A(i,j)$ is calculated as:

$$\begin{array}{c}\hfill \begin{array}{c}\hfill C_A^{{}^{\prime }}=\frac{t{\left(C_A\right)}^p}{h{\left(C_A\right)}^p+Z}\end{array}.\end{array}$$

(15)

Here, $t,h,p,q$ and Z are real scalar parameters that determine the shape of the nonlinearity of the masking function [52].

In the fourth step, the saliency map of $I_A(i,j)$ can be calculated by Equation (16):

$$\begin{array}{c}\hfill \begin{array}{c}\hfill {\lambda }_A(i,j)=\frac{C{{}_A^{{}^{\prime }}}^2(i,j)}{C{{}_A^{{}^{\prime }}}^2(i,j)+C{{}_B^{{}^{\prime }}}^2(i,j)}\end{array}.\end{array}$$

(16)

The information preservation value is computed as Equation (17):

$$\begin{array}{c}\hfill \begin{array}{c}\hfill Q_{AF}(i,j)=\left\{\begin{array}{c}\frac{C_A^{{}^{\prime }}(i,j)}{C_F^{{}^{\prime }}(i,j)},if{C}_A^{{}^{\prime }}(i,j)<C_F^{{}^{\prime }}(i,j),\\ \frac{C_F^{{}^{\prime }}(i,j)}{C_A^{{}^{\prime }}(i,j)},otherwise.\end{array}\right.\end{array}\end{array}$$

(17)

In the fifth step, the Global quality map can be calculated:

$$\begin{array}{c}\hfill \begin{array}{c}\hfill Q_{GQM}(i,j)={\lambda }_A(i,j)Q_{AF}(i,j)+{\lambda }_B(i,j)Q_{BF}(i,j)\end{array}.\end{array}$$

(18)

Then, the value of $Q_{CB}$ can be obtained by averaging the global quality map:

$$\begin{array}{c}\hfill \begin{array}{c}\hfill Q_{CB}=\overline{\sum _{i,j}{Q}_{GQM}(i,j)}\end{array}.\end{array}$$

(19)

3.2. Image Quality

To show the efficiency of proposed method, the comparison of fused images is provided. It compares the quality of the fused image based on visual effects, the accuracy of focused region detection, and the objective evaluations.

3.2.1. Multi-Focus Comparison

Figure 4a,b are the source multi-focus images. To show the details of the fused image, two image blocks are highlighted and magnified, which are marked by red and blue frames, respectively. The image block in the red frame is out of focus in Figure 4a, and the image block in the blue frame is out of focus in Figure 4b. The corresponding image blocks in blue and red frames are totally focused in Figure 4a,b, respectively. Figure 4c–e show the fused images of K-SVD, JCPD, and the proposed method, respectively.

The difference and performance of the algorithms to the fused images by three different methods are difficult to figure out visually. In order to evaluate of fusion performances objectively, $Q^{AB/F}$, MI, VIF, $Q_Y$, and $Q_{CB}$ are also used as image fusion quality measures. The fusion results of multi-focus images using three different methods are shown in

Table 1. Fusion performance comparison of multi-focus image pairs.

	${\mathit{Q}}^{\mathit{AB}/\mathit{F}}$	$\mathit{MI}$	$\mathit{VIF}$	${\mathit{Q}}_{\mathit{Y}}$	${\mathit{Q}}_{\mathit{CB}}$
K-SVD	0.4753	4.5992	0.7705	0.6897	0.6408
JCPD	0.5331	4.5586	0.7571	0.7403	0.6317
Proposed	0.5374	4.9561	0.7778	0.7420	0.6613

.

The best results of each evaluation metric are highlighted by bold-face in Table 1. According to Table 1, the proposed method has the best performance in all five types of evaluation metrics. Particularly, for the objective evaluation metric $Q^{AB/F}$, the proposed method obtains higher results than other two comparison image fusion methods. Since $Q^{AB/F}$ is a gradient-based quality metric to measure how well the edge information of source images is conducted to the fused image, it means that the proposed method can get a better fused image with edge information.

3.2.2. Medical Comparison

The “brain” images are a pair of PET (Positron Emission Tomography) and MRI (Magnetic Resonance Image) images shown in Figure 5a,b, respectively. PET images show the image of brain slices that produces a 3D image of functional processes in the human body. MRI images also show the image of brain slices that contain clear information of soft tissues. K-SVD, JCPD and the proposed method are employed to merge PET and MRI images into a clear image with soft tissues and functional processes information. The corresponding fusion results are shown in Figure 5c–e, respectively. Figure 5f–k show the enlarged details in red and green frames of the fused images in Figure 5c–e, respectively. Three fused images of different approaches have high quality in details, contrast, sharpness, and brightness.

Table 2. Fusion performance comparison of medical image pairs.

	${\mathit{Q}}^{\mathit{AB}/\mathit{F}}$	$\mathit{MI}$	$\mathit{VIF}$	${\mathit{Q}}_{\mathit{Y}}$	${\mathit{Q}}_{\mathit{CB}}$
K-SVD	0.2886	1.8554	0.2831	0.3294	0.6700
JCPD	0.2880	1.8575	0.2829	0.3290	0.6680
Proposed	0.3088	1.8869	0.2878	0.3591	0.6854

shows the objective evaluations of fusion results. Compared with K-SVD and JCPD, the proposed method gets the largest values in all five objective evaluations.

3.2.3. Visible-Infrared Comparison

The proposed solution is used to fuse two sample images from the same scenario of the downtown street scene. One is a visible image and the other one is an infrared image shown in Figure 6a,b, respectively. In Figure 6a,b, the walking person is marked in the red frame and the letters in the shade marked in the blue frame are dark, respectively. The fused images of K-SVD, JCPD and proposed method shown in Figure 6c–e are compared. The enlarged details in the red and blue frames of the fused images in Figure 6c–e are shown in Figure 6f–k, respectively. The walking person and the letters in the shade are clear in all three fused images. The objective evaluations of each visible-infrared image fusion solution are demonstrated in

Table 3. Fusion performance comparison of visible-infrared image pairs.

	${\mathit{Q}}^{\mathit{AB}/\mathit{F}}$	$\mathit{MI}$	$\mathit{VIF}$	${\mathit{Q}}_{\mathit{Y}}$	${\mathit{Q}}_{\mathit{CB}}$
K-SVD	0.4784	1.7713	0.3585	0.5670	0.5370
JCPD	0.5648	1.4563	0.3173	0.6562	0.4653
Proposed	0.6449	1.8398	0.3597	0.7647	0.5437

. Similarly, the proposed solution has the best performances in all five objective evaluations.

4. Conclusions

This paper proposes a novel sparse-representation based image fusion framework, which integrates geometric dictionary construction. A geometric image patch classification approach is presented to cluster image patches from different source images based on the similarity of image geometric structure. A few compact and informative sub-dictionaries are extracted from each image patch cluster by SCC. The extracted sub-dictionaries are combined into a dictionary for sparse representation. Then, image patches are sparsely coded into coefficients by the trained dictionary. To obtain better edge and corner details of fusion results, the proposed solution also chooses image block size adaptively and selects optimal coefficients during the image fusion process. The sparsely coded coefficients are fused by the Max-L1 rule and inverted to the fused image. The proposed method is compared with existing mainstream sparse-representation based methods in three aspects, including multi-focus, medical, and visible-infrared comparison. The experimentation results prove that the proposed method has the best performance in all three image scenarios. It means that geometric information of the source image can not only reduce the size of the learned dictionary efficiently and effectively, but also obtain a high-quality fused image. In the future, it will explore more details in geometric information to enhance fusion performance. Denoising, inpainting, and other image processing techniques will be integrated into the current solution.