A Geometric Dictionary Learning Based Approach for Fluorescence Spectroscopy Image Fusion

Abstract

In recent years, sparse representation approaches have been integrated into multi-focus image fusion methods. The fused images of sparse-representation-based image fusion methods show great performance. Constructing an informative dictionary is a key step for sparsity-based image fusion method. In order to ensure sufficient number of useful bases for sparse representation in the process of informative dictionary construction, image patches from all source images are classified into different groups based on geometric similarities. The key information of each image-patch group is extracted by principle component analysis (PCA) to build dictionary. According to the constructed dictionary, image patches are converted to sparse coefficients by simultaneous orthogonal matching pursuit (SOMP) algorithm for representing the source multi-focus images. At last the sparse coefficients are fused by Max-L1 fusion rule and inverted to fused image. Due to the limitation of microscope, the fluorescence image cannot be fully focused. The proposed multi-focus image fusion solution is applied to fluorescence imaging area for generating all-in-focus images. The comparison experimentation results confirm the feasibility and effectiveness of the proposed multi-focus image fusion solution.

1. Introduction

Following the development of cloud computing, cloud environment provides more and more strong computation capacity to process various images [1,2,3]. Due to the limited depth-of-focus of optical lenses in imaging sensors, one-time focus cannot guarantee to obtain all focused image in the whole scene [4]. In another word, it is difficult to obtain an image that contains all relevant objects in focus. In microscope, only the objects at a certain distance away from the lens can be captured in focus and sharply, and other objects are out of focus and blurred. A fuzzy multi-sensor data fusion Kalman filter model was proposed by Rodger to reduce failure risk in an integrated vehicle health maintenance system (IVHMS) [5]. In fluorescence spectroscopy, one scene at least contains several objects. For ensuring the accuracy and efficiency of fluorescence image processing, it usually captures multiple images of the same scene to guarantee all objects are in focus. However, it costs too much on viewing and analyzing a series of images separately. Multi-focus image fusion is an effective technique to resolve this problem by combining complementary information from multiple images of the same scene into a sharp and accurate one [6,7]. The integrated image can precisely describe the corresponding scene, which is beneficial for further analysis and understanding.

As one of the most widely recognized image fusion technologies, a large number of multi-focus image fusion methods have been proposed. According to fusion domain, these methods could be categorized into two main categories: spatial-domain-based methods and transform-domain-based methods [8]. Spatial-domain-based methods directly choose clear pixels, blocks, or regions from source images to compose a fused image [9,10]. Some simple methods, such as averaging or max pixel schemes, are performed on single pixel to generate a fused image. However, these methods may reduce the contrast and edge intensity of fused result. In order to improve the performance of fused image, some advanced schemes, such as block-based and region-based algorithms, were proposed. Li et al. proposed a scheme by dividing images into blocks and chose the focused one by comparing spatial frequencies (SF) first; then the fused results were produced by consistency verification [11]. Although block-based methods improve the contrast and sharpness of integrated image, they may cause block effect in integrated image [12].

Different from spatial-domain fusion methods, transform-domain methods transform source images into a few corresponding coefficients and transform bases first [13]. Then the coefficients are fused and inverted to an integrated image. Multi-scale transform (MST) and wavelet based algorithms are the most conventional transform approaches applied to transform-domain-based image fusion [14,15]. Wavelet transform [16,17], shearlet [18,19], curvelet [20], dual tree complex wavelet transform [21,22], nonsubsampled controulet transform (NSCT) [23] are usually used in MST-based methods. MST decomposition methods have attracted great attention in image processing field and achieved great performance in image fusion fields. However, MST-based methods are difficult to select an optimal transform basis without priori knowledge [24,25]. As each MST method has its own limitations, one MST method is difficult to fit all kinds of images [14].

In recent years, sparse-representation-based methods, as the subsets of MST fusion methods, are applied to image fusion. Other than MST methods, sparse-representation-based methods usually use trained bases, which can adaptively change according to input images without priori knowledge [26,27]. Due to adaptive learning feature, sparse representation is widely applied to image de-noising [28,29], image deblurring [30,31], image inpainting [32], image fusion [33], and super-resolution [34].

Yang and Li [35] took the first step for applying sparse representation theory to image fusion field. They proposed a multi-focus image fusion method with a fixed-dictionary. Li and Zhang [36] applied morphologically filtering sparse feature to matrix decomposition method for improving the accuracy of sparse representation in image fusion. Wang and Liu [37] proposed an approximate K-SVD based sparse representation method for image fusion and exposure fusion to reduce computation costs. Kim and Han [38] proposed a joint clustering dictionary learning method for image fusion. They used the steering kernel regression to strength the local geometric features of source images first. Then K-means clustering method was applied to image pixels clustering based on the local image features. The proposed method by Kim and Han can significantly group the pixels of input source images into a few classes. The principle components of local features are extracted from each group to construct a dictionary, which can make a compact and informative dictionary. This method reduces the sparse coding time by using the constructed compact dictionary. However, the performance of Kim and Han’s method depends on the presetting cluster number, that is difficult to confirm before clustering. The number of local geometric-feature types is usually used as the presetting cluster number, but the actual clustering results do not exactly follow the geometric features every time. So the clustering results cannot reach the expectations all the time. To optimize the weakness of existing clustering methods, a geometric information based image block classification method is proposed in this paper.

Geometric information, as one type of the most important image information, including edges, contours, and textures of image, and so on, can remarkably influence the quality of image perception [39]. The information can be used in patch classification as a supervised dictionary prior to improve the performance of trained dictionary [40,41]. In this paper, a geometric classification based dictionary learning method is proposed for sparse-representation based image fusion. Instead of grouping the pixels of images, the proposed geometric classification method groups image blocks directly by the geometric similarity of each image block. Since sparse-representation based fusion method uses image blocks for sparse coding and coefficients fusion, extracting underlying geometric information from image-block groups is an efficient way to construct a dictionary. Moreover, the geometric classification can group image blocks based on edge, sharp line information for dictionary learning, which can improve the accuracy of sparse representation. The proposed dictionary learning approach consists of three steps:

• In the first step, input source images are split into small blocks. According to the similarity of geometric patterns, these blocks are classified into smooth patches, stochastic patches, and dominant orientation patches.
• In the second step, it does principal component analysis (PCA) on each type of patches to extract crucial bases. The extracted PCA bases are used to construct the sub-dictionary of each image patch group.
• In the last step, the obtained sub-dictionaries are merged into a complete dictionary for sparse-representation-based fusion approach and the integrated sparse coefficients are inverted to fused image.

This paper has two main contributions.

• A geometric image patches classification method is proposed for dictionary learning. The proposed geometry classification method can accurately split source image patches into different image patch groups for dictionary learning. Dictionary bases extracted from each image patch group have good performance, when they are used to describe the geometric features of source images.
• A PCA-based geometry dictionary construction method is proposed. The trained dictionary with PCA bases is informative and compact for sparse representation. The informative feature of trained dictionary ensures that different geometric features of source images can be accurately described. The compact feature of trained dictionary can speed up the sparse coding process.

The rest sections of this paper are structured as follows: Section 2 proposes the geometric dictionary learning method and multi-focus image fusion framework; Section 3 compares and analyzes experimentation results; and Section 4 concludes this paper.

2. Geometry Dictionary Construction and Fusion Scheme

2.1. Dictionary Learning Analysis

During the sparse representation of the dictionary construction process, the key step is to construct an over-complete dictionary that covers the important information of input image. Since geometry information of image plays an important role in describing an image, an informative dictionary should take the geometric information into consideration. Smooth and non-smooth information as two important types of geometry information can be used to classify source images.

A multiple geometry information classification approach was proposed for single image super-resolution reconstruction (SISR) [39]. According to the geometric features, a large number of high-resolution (HR) image patches were randomly extracted and clustered to generate corresponding geometric dictionaries for sparse representations of local patches in low-resolution (LR) images. The geometric features were classified into three types, smooth patch, dominant orientation patch, and stochastic patch. According to the estimated dominant orientation, the dominant orientation patches could be divided into different directional patches. Rather than adaptively selecting one dictionary, the recovered patches were weighted in the learned geometric dictionaries to characterize the local image patterns, by a subsequent patch aggregation to estimate the whole image. To reduce the redundancy in image recovery, the self-similarity constraints were added on HR image patch aggregation. Both LR and HR residual images were estimated from the recovered image and compensated for the subtle details of reconstructed image.

In SISR, the source images are classified as smooth, dominant orientation, and stochastic patches. Dominant orientation and stochastic patches are non-smooth patches. Since the detection of orientation in finite dimensional Euclidean spaces corresponded to fitting an axis or a plane by Fourier transformation of an n-dimensional structure, Bigün verified dominant orientation and stochastic patches can be separated by orientation estimation in the spatial domain based on the error of the fit [42].

In dictionary learning, the redundancy of trained dictionary is usually not considered. A redundant dictionary not only has high computational complexity, but cannot also obtain the best effect in image representation. Different methods are proposed to reduce the redundancy in learning process to construct the compact dictionary. Elad [43] estimated the maximum a posteriori probability (MAP) by a compact dictionary. An effective image interpolation method by nonlocal autoregressive modeling (NARM) was developed and embedded in SRM by Dong [44] to enhance the effectiveness of SRM in image interpolation by reducing the coherence between the sampling matrix and the sparse representation dictionary and minimizing the nonlocal redundancy. To improve the performance of sparse representation-based image restoration, a non-locally centralized sparse representation (NCSR) model proposed by Dong [45] suppressed the sparse coding noise in image restoration.

The sparse representation-based approach linearly combines a few atoms extracted from an over-complete dictionary as an image patch. The promising results have been shown in various image restoration applications [38,45]. Based on the classification of image patches, this paper proposed sparse representation-based approach that uses PCA algorithm to construct more informative and compact dictionary [38]. The proposed solution cannot be extended to other sparse-representation applications. Most sparse representation methods are based on a large number of sample images. The corresponding learned dictionaries are expected to be applied to different scenic source images. The proposed solution uses sparse feature to do image sparse representation and fusion. Therefore, the learned dictionary of proposed solution does not need to be extended to different source images, and only specializes for experimented source images. It uses PCA classification to reduce the redundancy in dictionary and enhance the performance.

There are many popular dictionary learning methods like KSVD. However, this paper cannot use other dictionary learning methods to substitute PCA to do dictionary learning. It compares PCA with KSVD to explain the benefits of using PCA in dictionary learning. For sparse representation-based image reconstruction and fusion techniques, it is difficult to select a good over-complete dictionary. To obtain an adaptive dictionary, Aharon [46] developed KSVD that learned an over-complete dictionary by using a set of training images or updates a dictionary adaptively by applying SVD operations. The image patches used by KSVD in dictionary learning were extracted either from globally trained dictionary (natural images) or adaptively trained dictionary (input images). Although KSVD shows better performance in dictionary learning than other methods, it takes high costs to do dictionary learning with a large number of training images iteratively. The high computational complexity constrains the learned dictionary size of KSVD in practical usage. Kim [38] firstly applied clustering-based dictionary learning solution to image fusion. It clustered patches from different source images together based on local structural similarities. Only a few principal components of each cluster were analyzed and used to construct corresponding sub-dictionary. All learned sub-dictionaries were combined to form a compact and informative dictionary that can describe local structures of all source images effectively. The redundancies of dictionary (i.e., reducing the size of dictionary) were eliminated to reduce the computation loads of the proposed PCA-based dictionary learning solution. In various comparison experimentations, the proposed PCA-based dictionary learning solution (JPDL) had better performance than other traditional multi-scale transform-based and sparse representation-based methods, including KSVD. Comparing with KSVD, JPDL as a PCA-based method not only obtained a more compact dictionary, but also took less computational costs in dictionary learning process.

2.2. Geometry Dictionary Construction

According to geometric patterns used in SISR [39], this paper classifies the image into three patch types: smooth patches, stochastic patches, and dominant orientation patches. The smooth patches, stochastic patches and dominant orientation patches mainly describe the structure information, texture information, and edge information respectively. Based on a priori knowledge, image patches can be classified into three groups for sub-dictionaries learning. Additionally, to obtain more compact sub-dictionary for each image patch group, this paper uses PCA method to extract the important information from each group. PCA bases of each group are used as the bases of corresponding sub-dictionary. The obtained dictionary can be more compact and informative by combining the PCA-based sub-dictionaries [38].

This paper learns three different sub-dictionaries rather other one dictionary. Three different sub-dictionaries contain more detailed information and structure information, and less redundant information. It means that although the obtained sub-dictionaries are compact, they can still contain more useful information. Three types of geometric patches extract different image information from source images respectively.

• Smooth patches describe the structure information of source images, such as the background.
• Dominant orientation patches describe the edge information to provide the direction information in source images.
• Stochastic patches show the texture information to make up the missing detailed information that is not represented in dominant orientation patches.

Three learned dictionaries specialize in smooth, dominant orientation, and stochastic patches respectively. The structure information, edge information, and texture information can be accurately and completely shown in corresponding sub-dictionary. Comparing with one learned dictionary, three learned sub-dictionaries can not only supply different information of source images respectively, but also make up the deficiencies mutually to enhance the quality of merged dictionary. So each sub-dictionary is important to compose a compact and informative dictionary in the proposed approach.

The proposed geometric approach is shown in Figure 1 that has two main steps. In the first step, the input source images $I_i$ to $I_k$ are split to several small image blocks $p_{in},i\in (1,2...k),n\in (1,2...w)$, where i is the source image number, and n is the patch number. The total block number of each input image is w. Then according to the similarity of geometric patterns, the obtained image blocks are classified into three groups, smooth patch group, stochastic patch group, and dominant orientation patch group. In the second step, it does PCA analysis on each sub-class for extracting corresponding PCA bases as sub-dictionary. Then these obtained sub-dictionaries are composed to construct a complete dictionary for image sparse representation.

2.3. Geometric-Structure-Based Patches Classification

Image blocks of a source image can be classified by various image features in describing underlying relationships. According to geometry descriptions, the multi-focus source images can be divided to smooth, stochastic, and dominant orientation patches. The edges of focused area are usually sharp that contain dominant orientation patches. The out-of-focused areas are smooth that contain a large number of smooth blocks. Besides that, there are also lots of stochastic blocks in source images. Grouping image blocks into different classes for dictionary learning is an efficient way to improve the accuracy of image description.

To obtain different geometry sub-classes, this paper uses a geometric-structure-based approach to partition image into several sub-classes first. Then based on the classified image blocks, it gets the corresponding sub-dictionaries.

Before doing geometry analysis, it needs to divide the input image into $\sqrt{w}\times \sqrt{w}$ small image blocks $P_I=(p_1,p_2,...,p_n)$ first. Each image patch $p_i,i\in (1,2,...n)$ is converted into $1\times w$ image vectors $v_i,i\in (1,2,...n)$. After obtaining vectors , the variance $C_i$ of pixels in each image vector can be calculated. After obtaining variance, it needs to choose the threshold δ to evaluate whether image block is smooth. If $C_i<\delta$, image block $p_i$ is smooth, otherwise image block $p_i$ is not smooth [39]. Based on the threshold δ, the classified smooth and non-smooth patches are shown in Figure 2a,b respectively.

According to the classified smooth and non-smooth patches shown in Figure 2a,b, it is clear to find that the smooth patches have similar structure information of input images. Oppositely, non-smooth patches are different and cannot be directly classified into one class.

Non-smooth patches may also be classified. According to geometric patterns, non-smooth patches could be divided into stochastic and dominant orientation patches. The separation method of stochastic and dominant orientation image patches consists of two steps. In the first step, the gradient of each pixel is calculated. In every image vector $v_i,i\in (1,2,...n)$, the gradient of each pixel $k_{ij}$, $j\in (1,2,...,w),i\in (1,2,...n)$ is composed by its x and y coordinate gradient $g_{ij}\left(x\right)$ and $g_{ij}\left(y\right)$. The gradient value of each pixel $k_{ij}$ in image patch $v_i$ is $g_{ij}=(g_{ij}\left(x\right),g_{ij}\left(y\right))$. The $(g_{ij}\left(x\right)$, $g_{ij}\left(y\right))$ can be calculated by $g_{ij}\left(x\right)=\partial k_{ij}(x,y)/\partial x$, $g_{ij}\left(y\right)=\partial k_{ij}(x,y)/\partial y$. For each image vector $v_i$, the gradient $G_i$ is $G_i={(g_{i1},g_{i2},...,g_{iw})}^T$, where $G_i\in {\mathbb{R}}^{w\times 2}$. In the second step, the gradient value of each image patch is decomposed by Equation (1):

$$\begin{array}{c}\hfill \begin{array}{c}\hfill G_i=\left[\begin{array}{c}{G}_{i1}\\ G_{i2}\\ ...\\ G_{iw}\end{array}\right]=U_i{S}_i{V}_i^T\end{array},\end{array}$$

(1)

where $U_i{S}_i{V}_i^T$ is the singular value decomposition of $G_i$. $S_i$ is a diagonal $2\times 2$ matrix for representing energy in dominant directions [47]. When $S_i$ is obtained, the dominant measure R can be calculated. The calculation method of R is shown in Equation (2):

$$\begin{array}{c}\hfill \begin{array}{c}\hfill R=\frac{S_{1,1}-S_{2,2}}{S_{1,1}+S_{2,2}}\end{array},\end{array}$$

(2)

The smaller the R is, the more stochastic the corresponding image patch is [48]. In this case, a threshold $R^{*}$ should be calculated for distinguishing stochastic and dominant orientation patches. To find the threshold $R^{*}$, a probability density function (PDF) of R is calculated. According to [42], the PDF of R can be calculated by Equation (3).

$$\begin{array}{c}\hfill \begin{array}{c}\hfill P\left(R\right)=4(w-1)R\frac{{(1-R^2)}^{w-2}}{{(1+R^2)}^w}\end{array},\end{array}$$

(3)

The PDF of dominant measure R of patches with different sizes is shown in Figure 3.

A PDF significance test is implemented to distinguish stochastic and dominant orientation patch by the threshold $R^{*}$ [42]. If R is less than $R^{*}$, the image patch can be considered as stochastic patch. The stochastic and dominant orientation patches separated by the proposed method are shown in Figure 4. The Figure 4a shows the stochastic image patches, which contain some texture and detailed information. Dominant orientation image patches are shown in Figure 4b, which mainly contain the edge information.

As shown in Figure 3, when R increases, $P\left(R\right)$ converges to zero. It chooses the value of R as the threshold $R^{*}$, when $P\left(R\right)$ reaches zero for the first time.

2.4. PCA-Based Dictionary Construction

When the geometric classification finished, image patches with similar geometric structure are classified into a few groups. In this work, the compact and informative dictionary is trained by combining the principal components of each geometric group. Since patches in the same geometric group can be well-approximated by a small number of PCA bases, top m most informative principal components are chosen to form a sub-dictionary [49] as Equation (4).

$$\begin{array}{c}\hfill \begin{array}{c}\hfill B_x=[b_1,b_2...,b_j],s.t.p=\underset{p}{\arg \max }\left\{\sum _{j=p+1}^q{L}_j>\delta \right\}\end{array},\end{array}$$

(4)

where $B_x$ denotes the sub-dictionary of the xth cluster, and q is the total number of atoms in each cluster. Each $B_x$ consists of p eigenvector atoms. $L_j$ is the eigenvalue of the corresponding jth eigenvector $d_j$. The eigenvalues are sorted in descending order (i.e., $L_1>L_2>...>L_n>0$). δ is a parameter to control the amount of approximation with rank p. Usually, δ is set to a proportional to the dimension of input image to avoid over-fitting [49]. After the sub-dictionaries are obtained, a dictionary D is constructed by combining sub-dictionaries together as Equation (5).

$$\begin{array}{c}\hfill \begin{array}{c}\hfill D=[B_1,B_2...,B_n]\end{array},\end{array}$$

(5)

where n is the total number of clusters.

2.5. Fusion Scheme

The fusion scheme of proposed method is shown in Figure 5, which consists of two steps.

In the first step, each input image $I_i$ is split into n image patches with the size of $\sqrt{w}\times \sqrt{w}$. These image patches are resized to $w*1$ vectors $p_1^i,p_2^i,...,p_n^i$. The resized vectors are sparse coded with trained dictionary to sparse coefficients $z_1^i,z_2^i,...,z_n^i$. In the second step, the sparse coefficients are fused by ’Max-L1’ fusion rule [50,51,52]. Then the fused coefficients are inverted to fused image by the trained dictionary.

3. Experiments and Analyses

In comparison experiments, three pairs of color fluorescence images are used to test the proposed multi-focused image fusion approach. All three multi-focus image pairs are modified to the size of $256\times 256$ for comparison purpose. To show the efficiency of proposed method, the state of art dictionary learning based sparse-representation fusion schemes KSVD [50] and JPDL [38], which were proposed by Li in 2012 and Kim in 2016 respectively, are used in comparison experiments. The comparison experiments are evaluated by both subjective and objective assessments. Four popular image fusion quality metrics are used in this paper for the quantitative evaluation. The patch size of all sparse-representation-based methods including the proposed method are set to $8\times 8$. To avoid blocking artifacts, all experiments use sliding window scheme [38,50]. The overlapped region of sliding window scheme is set to 4-pixel in each vertical and horizontal direction of all experiments. All experiments were implemented using Matlab, version 2014a; MathWorks: Natick, MA, 2014, and Visual Studio, version Community 2013; Microsoft: Redmond, WA, 2013, mixed programming on an Intel(R) Core(TM), version i7-4720HQ; Intel: Santa Clara, CA, 2015, CPU @ 2.60 GHz Laptop with 12.00 GB RAM.

3.1. Objective Evaluation Methods

Five mainstream objective evaluation metrics are implemented for the quantitative evaluation. These metrics include edge retention ($Q^{AB/F}$) [53,54], mutual information (MI) [55], visual information fidelity (VIF) [56], Yang proposed fusion metric ($Q_Y$) [57,58], and Chen-Blum metric ($Q_{CB}$) [58,59]. For the fused image, the sizes of $Q^{AB/F}$, MI, VIF, $Q_Y$, and $Q_{CB}$ become bigger, the corresponding fusion results are better.

3.1.1. Mutual Information

MI for images can be formalized as Equation (6).

$$\begin{array}{c}\hfill \begin{array}{c}\hfill MI=\sum _{i=1}^L\sum _{j=1}^L{h}_{A,F}(i,j)lo{g}_2\frac{h_{A,F}(i,j)}{h_A\left(i\right)h_{F\left(j\right)}}\end{array},\end{array}$$

(6)

where L is the number of gray-level, $h_{R,F}(i,j)$ is the gray histogram of image A and F. $h_A\left(i\right)$ and $h_F\left(j\right)$ are edge histogram of image A and F. MI of fused image can be calculated by Equation (7).

$$\begin{array}{c}\hfill \begin{array}{c}\hfill MI(A,B,F)=MI(A,F)+MI(B,F)\end{array},\end{array}$$

(7)

where $MI(A,F)$ represents the MI value of input image A and fused image F; $MI(B,F)$ represents the MI value of input image B and fused image F.

3.1.2. $Q^{AB/F}$

$Q^{AB/F}$ metric is a gradient-based quality index to measure how well the edge information of source images conducted to the fused image. It is calculated by:

$$\begin{array}{c}\hfill \begin{array}{c}\hfill Q^{AB/F}=\frac{{\sum }_{i,j}(Q^{AF}(i,j)w^A(i,j)+Q^{BF}(i,j)w^B(i,j))}{{\sum }_{i,j}(w^A(i,j)+w^B(i,j))}\end{array},\end{array}$$

(8)

where $Q^{AF}=Q_g^{AF}{Q}_0^{AF}$, $Q_g^{AF}$ and $Q_0^{AF}$ are the edge strength and orientation preservation values at location (i, j). $Q^{BF}$ can be computed similarly to $Q^{AF}$ . $w_A(i,j)$ and $w_B(i,j)$ are the weights of $Q^{AF}$ and $Q^{BF}$ respectively.

3.1.3. Visual Information Fidelity

VIF is the novel full reference image quality metric. VIF quantifies the mutual information between the reference and test images based on Natural Scene Statistics (NSS) theory and Human Visual System (HVS) model. It can be expressed as the ratio between the distorted test image information and the reference image information, the calculation equation of VIF is shown in Equation (9).

$$\begin{array}{c}\hfill \begin{array}{c}\hfill VIF=\frac{{\sum }_{i\in subbands}I(\overrightarrow{C^{N,i}};\overrightarrow{F^{N,i}})}{{\sum }_{i\in subbands}I(\overrightarrow{C^{N,i}};\overrightarrow{E^{N,i}})}\end{array},\end{array}$$

(9)

where $I(\overrightarrow{C^{N,i}};\overrightarrow{F^{N,i}})$ and $I(\overrightarrow{C^{N,i}};\overrightarrow{E^{N,i}})$ represent the mutual information, which are extracted from a particular subband in the reference and the test images respectively. $\overrightarrow{C^N}$ denotes N elements from a random field, $\overrightarrow{E^N}$ and $\overrightarrow{F^N}$ are visual signals at the output of HVS model from the reference and the test images respectively.

To evaluate the VIF of fused image, an average of VIF values of each input image and the integrated image is proposed [56]. The evaluation function of VIF for image fusion is shown in Equation (10).

$$\begin{array}{c}\hfill \begin{array}{c}\hfill VIF(A,B,F)=\frac{VIF(A,F)+VIF(B,F)}{2}\end{array},\end{array}$$

(10)

where $VIF(A,F)$ is the VIF value between input image A and fused image F; $VIF(B,F)$ is the VIF value between input image B and fused image F.

3.1.4. $Q_Y$

Yang et al. proposed structural similarity-Based way for fusion assessment [57]. The Yang’s method is shown in Equation (11).

$$\begin{array}{c}\hfill \begin{array}{c}\hfill Q_Y=\left\{\begin{array}{c}\begin{array}{c}\lambda \left(\omega \right)SSIM(A,F\left|\omega \right.)+(1-\lambda \left(\omega \right)(SSIM(B,F\left|\omega \right.),\hfill \\ SSIM(A,B\left|\omega \right.)\ge 0.75,\hfill \end{array}\\ \begin{array}{c}max\left\{SSIM(A,F\left|\omega \right.),SSIM(A,B\left|\omega \right.)\right\},\hfill \\ SSIM(A,B\left|\omega \right.)<0.75\hfill \end{array}\end{array}\right.\end{array},\end{array}$$

(11)

where $\lambda \left(\omega \right)$ is the local weight, $SSIM(A,B)$ is a structural similarity index measure for images A and B. The detail of $\lambda \left(\omega \right)$ and $SSIM(A,B)$ can be found in [57,58].

3.1.5. $Q_{CB}$

Chen-Blum metric is human perception inspired fusion metrics. Chen-Blum metric consists of 5 steps:

In the first step, filtering image $I(i,j)$ in frequency domain. $I(i,j)$ is transformed to frequency domain and get $I(m,n)$. Filtering $I(m,n)$ by contrast sensitive function(CSV) [59] filter $S\left(r\right)$, where $r=\sqrt{m^2+n^2}$. In this image fusion metric $S\left(r\right)$ is in polar form. $\tilde{I}(m,n)$ can be got by $\tilde{I}(m,n)=I(m,n)*S\left(r\right)$.

In the second step, local contrast is computed. For $Q_{CB}$ metric, Peli’s contrast is used and it can be defined as:

$$\begin{array}{c}\hfill \begin{array}{c}\hfill C(i,j)=\frac{{\varphi }_k(i,j)*I(i,j)}{{\varphi }_{k+1}(i,j)*I(i,j)}-1\end{array}.\end{array}$$

(12)

A common choice for ${\varphi }_k(i,j)$ would be

$$\begin{array}{c}\hfill \begin{array}{c}\hfill {\varphi }_k(i,j)=\frac{1}{\sqrt{2\pi }{\sigma }_k}{e}^{\frac{i^2+j^2}{2{\sigma }_k^2}}\end{array}.\end{array}$$

(13)

with a standard deviation ${\sigma }_k=2$.

In the third step, The masked contrast map for input image $I_A(i,j)$ is calculated as:

$$\begin{array}{c}\hfill \begin{array}{c}\hfill C_A^{\prime }=\frac{t{\left(C_A\right)}^p}{h{\left(C_A\right)}^p+Z}\end{array}.\end{array}$$

(14)

Here, $t,h,p,q$ and Z are real scalar parameters that determine the shape of the nonlinearity of the masking function [59,60].

In the fourth step, the saliency map of $I_A(i,j)$ can be calculated by Equation (15),

$$\begin{array}{c}\hfill \begin{array}{c}\hfill {\lambda }_A(i,j)=\frac{{C_A^{\prime }}^2(i,j)}{{C_A^{\prime }}^2(i,j)+{C_A^{\prime }}^2(i,j)}\end{array}.\end{array}$$

(15)

The information preservation value is computed as Equation (16),

$$\begin{array}{c}\hfill \begin{array}{c}\hfill Q_{AF}(i,j)=\left\{\begin{array}{c}\frac{C_A^{\prime }(i,j)}{C_F^{\prime }(i,j)},if{C}_A^{\prime }(i,j)<C_F^{\prime }(i,j),\\ \frac{C_F^{\prime }(i,j)}{C_A^{\prime }(i,j)},otherwise.\end{array}\right.\end{array}\end{array}$$

(16)

In the fifth step, the Global quality map can be calculated:

$$\begin{array}{c}\hfill \begin{array}{c}\hfill Q_{GQM}(i,j)={\lambda }_A(i,j)Q_{AF}(i,j)+{\lambda }_B(i,j)Q_{BF}(i,j)\end{array}.\end{array}$$

(17)

Then the value of $Q_{CB}$ can be got by average the global quality map:

$$\begin{array}{c}\hfill \begin{array}{c}\hfill Q_{CB}=\overline{\sum _{i,j}{Q}_{GQM}(i,j)}\end{array}.\end{array}$$

(18)

3.2. Image Quality Comparison

To show the efficiency of proposed method, the quality comparison of fused images is demonstrated. It compares the quality of fused image based on visual effect, the accuracy of focused region detection, and the objective evaluations.

3.2.1. Comparison Experiment 1

The source fluorescence image is obtained from public website [61]. Figure 6a,b are the source multi-focus fluorescence images. To show the details of fused image, two image blocks are highlighted and magnified, which are squared by red and blue square respectively. The image block in red square is out of focus in Figure 6a, and the image block in blue square, is out of focus in Figure 6b. The corresponding image block in blue and red square are totally focused in Figure 6a,b respectively. Figure 6c–e show the fused images of KSVD, JPDL, and proposed method, respectively.

The difference and performance of fused images by three different methods are difficult to figure out by eyes. In order to evaluate of fusion performances objectively, $Q^{AB/F}$, MI, VIF, $Q_Y$, and $Q_{CB}$ are also used as image fusion quality measures. The fusion results of multi-focus fluorescence images using three different methods are shown in

Table 1. Fusion Performance Comparison - 1 of Multi-focus Fluorescence Image Pairs.

	${\mathit{Q}}^{\mathit{AB}/\mathit{F}}$	MI	VIF	${\mathit{Q}}_{\mathit{Y}}$	${\mathit{Q}}_{\mathit{CB}}$
KSVD	0.4966	2.4247	0.5778	0.5960	0.5287
JPDL	0.5815	2.9258	0.6972	0.6944	0.7243
Proposed Solution	0.6226${}^{*}$	3.4773	0.7428	0.7386	0.7974

* The highest result in each column is marked in bold-face.

.

The best results of each evaluation metric are highlighted by bold-faces in Table 1. According to Table 1, it can figure out that the proposed method has the best performance in all five types of evaluation metrics. Particularly, for the objective evaluation metric $Q^{AB/F}$, the proposed method obtains higher result than other two comparison image fusion methods. Since $Q^{AB/F}$ is a gradient-based quality metric to measure how well the edge information of source images is conducted to the fused image, it means the proposed method can get better fused image with edge information.

3.2.2. Comparison Experiment 2 and 3

Similarly, the source fluorescence images shown in Figure 7 and Figure 8a,b are obtained from public websites [62,63] respectively. In a set of source images, two images (a) and (b) focus on different items. The source images are fused by KSVD, JPDL, and proposed method to get a totally focused image, and the corresponding fusion results are shown in Figure 7 and Figure 8c–e respectively.

Objective metrics of multi-focus comparison experiment 2 and 3 are shown in

Table 2. Fusion Performance Comparison - 2 of Multi-focus Fluorescence Image Pairs.

	${\mathit{Q}}^{\mathit{AB}/\mathit{F}}$	MI	VIF	${\mathit{Q}}_{\mathit{Y}}$	${\mathit{Q}}_{\mathit{CB}}$
KSVD	0.5600	2.4747	0.6099	0.6161	0.5477
JPDL	0.6952	3.1357	0.7013	0.7355	0.7423
Proposed Solution	0.7692${}^{*}$	3.8982	0.7488	0.8058	0.8206

* The highest result in each column is marked in bold-face.

and

Table 3. Fusion Performance Comparison - 3 of Multi-focus Fluorescence Image Pairs.

	${\mathit{Q}}^{\mathit{AB}/\mathit{F}}$	MI	VIF	${\mathit{Q}}_{\mathit{Y}}$	${\mathit{Q}}_{\mathit{CB}}$
KSVD	0.6837	2.4624	0.8171	0.7855	0.6454
JPDL	0.8557	2.9358	0.8943	0.9325	0.7665
Proposed Solution	0.8979${}^{*}$	3.9154	0.9129	0.9790	0.8237

* The highest result in each column is marked in bold-face.

respectively to evaluate the quality of fused images. According to the metric results, the proposed method has the best performance in all five objective evaluations in comparison experiment 2 and 3. So the proposed method has the best overall performance among all comparison methods.

3.3. Processing Time Comparison

Table 4. Processing Time Comparison.

	Experiment - 1	Experiment - 2	Experiment - 3
KSVD	164.02 s	215.09 s	108.65 s
JPDL	123.68 s	171.93 s	73.58 s
Proposed Solution	76.23 s${}^{*}$	103.96 s	25.47 s

* The highest result in each column is marked in bold-face.

compares the processing times of three comparison experimentations. The proposed solution has lower computation costs than KSVD and JPDL in image fusion process. Compared with KSVD, the dictionary construction method of proposed solution does not use any iterative way to extract the underlying information of images, which is not efficient in dictionary construction. Although JPDL and the proposed method both cluster image pixels or patches based on geometric similarity, the proposed method does not use the iterative method of Steering Kernel Regression (SKR), which is time consuming.

4. Conclusions

This paper proposes a novel sparse representation-based image fusion framework, which integrates geometric dictionary construction. A geometric image patch classification approach is presented to group image patches from different source images based on the similarity of image geometric structure. The proposed method extracts a few compact and informative sub-dictionaries from each image patch cluster by PCA and these sub-dictionaries are combined into a dictionary for sparse representation. Then image patches are sparsely coded into coefficients by the trained dictionary. For obtaining better edge and corner details of fusion results, the proposed solution also chooses image block size adaptively and selects optimal coefficients during the image process. The sparsely coded coefficients are fused by Max-L1 rule and inverted to the fused image. The proposed method is compared with existing mainstream sparse representation-based methods in various experiments. The experimentation results proves that the proposed method has good fusion performance in different image scenarios.