A Novel Image Fusion Method Via Ratio Weighted and Condition Weighted Based on Shearlet Transform ^†

Abstract

In this paper, a new image fusion method is proposed based on the shearlet transform. The new image is reconstructed by image fusion, which provides richer visual information than the original images. The image could be decomposed by shearlet transform in any scale and any direction, and the detail information can be caught easily. The multi-scale and multiple directions decomposition coefficients are obtained through shearlet transformation. The fusion results are analyzed and compared with the measurement of human visual system and objective evaluation. The experiment results show the advantages of the new fusion method compare to other classical fusion algorithm. The simulation results of the multimodal images are adopted to demonstrate that the algorithm based on shearlet transform is able to obtain fused images of higher clarity and complementary information compared with any other methods.

1. Introduction

Image fusion is a process of combining two or more source images to form a new one with more sharpness and reliability. Through utilizing a certain algorithms, image fusion produces a result image with the best aspects of the source images. The fused image reflects the multifaceted information of the source images and is more suitable for human visual perception or computer processing [1]. Image fusion will become a power subject that integrates technologies such as multi-sensors, signal processing, and image processing. It has been widely used in target object tracking, edge detection, identification and other agriculture areas. As the same, it can be widely used in medical imaging and diagnostics, security and surveillance, intelligent transportation and other civilian areas. In the actual clinical application, a single mode of image cannot provide enough information to doctors in need, often need different modes of image fusion together, to get more abundant information for better understanding the comprehensive information of diseased tissue.

At present the research of image fusion technology can be roughly divided into two categories: spatial domain fusion algorithm and multi-resolution frequency domain fusion algorithm [2]. Many scholars abroad for multimodal image fusion technology has also made a related study, such as image fusion methods based on curvelet transform [3,4,5], based on vision [6,7], based on discrete wavelet transform [8,9,10]. The associated system forms an affine system, the transform can be regarded as matrix coefficients of a unitary representation of a special group, there is an MRA-structure associated with the systems. The shearlets satisfy all these properties in addition to showing optimal behavior with respect to the detection of directional information. Shearlet transform [11,12] is a image representation of multi-resolution, local area, and multi direction. It inherited the anisotropy of shearlet transform scales relationship, in a sense, it can be thought as the digital realization way of curvelet transform approximation. Shearlet transform is a real 2D image sparse representation, good directivity and anisotropic characteristics, to better the image of the edge profile information capture the different scale and the direction of the subband in [13], is effectively used in the field of image fusion. In this paper, based on the research of the image fusion technology and shearlet transform, this paper proposes a new image fusion method based on shearlet transform ratio weight and condition weight fusion algorithm.

2. Basic Shearlet Theory [14]

The basic idea for the definition of continuous shearlets is the usage of a 2-parameter dilation group, which consists of products of parabolic scaling matrices and shear matrices. Hence the con-tinuous shearlets depend on three parameters, the scaling parameter a > 0, the shear parameter s ∈ R and the translation parameter t ∈ R², and they are defined by Eq. (1):

$${\psi }_{a,s,t}(x)=a^{-3/4}\psi (D_{a,s^{-1}(x-t)}),\mathit{where}{D}_{a,s}=[a,-a^{1/2}\mathit{s;}0,a^{1/2}].$$

(1)

The mother shearlet function ψ is defined almost like a tensor product by Eq. (2):

$${\psi }_{{\xi }_1,{\xi }_2}={\psi }_1({\xi }_1){\psi }_2({\xi }_2/{\xi }_1),$$

(2)

where ψ₁ is a wavelet and ψ₂ is a bump function. The figure on the right hand side illustrates the behavior of the continuous shearlets in frequency domain assuming that ψ₁ and ψ₂ are chosen to be compactly supported in frequency domain.

The associated continuous shearlet transform again depends on the scaling parameter a, the shear parameter s and the translation parameter t, and is defined by Eq. (3):

SHf(a,s,t)=<f,ψ_a,s,t>

(3)

This transform can also be regarded as matrix coefficients of the unitary representation Eq. (4):

$$(\sigma (a,s,t)\psi )(x)={\psi }_{a,s,t}(x)=a^{-3/4}\psi (D_{a,s^{-1}(x-t)})$$

(4)

of the shearlet group S = R⁺ × R × R² with multiplication given by Eq. (5):

$$(a,s,t)·(a^{\prime },s^{\prime },t^{\prime })=(a{a}^{\prime },s+s^{\prime }{a}^{1/2},t+D_{a,s{t}^{\prime }})$$

(5)

The continuous shearlet transform precisely resolves the wavefront set, since the location param-eter t does detect the location of singularities, whereas the shear parameter shows the direction perpendicular to the direction of the singularity.

The Discrete Theory;

By sampling the continuous shearlet transform on an appropriate discrete set of the scaling, shear, and translation parameters, it is possible to obtain a frame or even a Parseval frame for L2(R). We will briefly discuss two possibilities to discretize the continuous shearlet transform.

To obtain the discrete shearlets, we sample the three parameters as a_j= 2^j(j ∈ Z), s_j, k = ka_j^1/2 = k2^j/2(k ∈ Z), and t_j, k, m = D_aj, s_j, k(m ∈ Z²). We choose the mother shearlet function ψ in a similar fashion as in the continuous case, i.e., we now choose ψ to be a discrete wavelet and ψ₂ to be bump function with certain weak additional properties. The tiling of the frequency plane is illustrated in the figure on the left hand side.

3. Fusion Method

3.1. Image Fusion Algorithm Based on Shearlet Transform

As the introduction above of the basic theories, we propose a novel method that is based on the combination ratio weighted and condition weighted and shearlet transform through gradi-ent combined with mutual information similarity criterion for image registration. The premise of image fusion is that the two images have been carried out for better image registration. In this paper before the image fusion, only when the two image pixels of the same spatial location are corresponding to the same anatomical structure, image fusion has application value. Shear-let transform implementation structure by Laplacian Pyramid (LP) and Direction Filter Banks (DFB), the transformation is the multi-scale analysis and direction analysis [15], respectively. With similar to the profile section of the base structure of the approximation image, using the base structure is changing with scale bar structure of aspect ratio, directional and anisotropy, the image of the line and can have a more sparse representation. We take the fusion process of two multi-focus images for instance and the two image pixels of the same spatial location are corresponding to the same anatomical structure. The proposed method is composed four main phases detailed as follows:

Step 1 The two source images are decomposed at different scales and directions by shearlet transform. Firstly, the singular points in the two-dimensional image are captured by using multi-scale decomposition in shearlet transform. Then the high frequency signal is distributed by using direction filter banks, and the singular points of the same direction are composted the one coefficient.

Step 2 The fusion rules are analyzed. The low frequency sub-band and high frequency sub-band are optimized by choosing the appropriate fusion rules. The approximation and most of the energy of the original image information are retained in the low-frequency sub-band. The general picture of the whole characteristic is as much as possible to keep by dealing with the low frequency part.

Step 3 Produce the result fused image on shearlet inverse transform.

Step 4 The result of evaluation can be analysed. That is to carry on objective quantitative evaluation on the basis of subjective qualitative visual evaluation.

3.2. Fusion Rules for Low Frequency Subband

The choice of fusion rules will affect the final effect of image fusion. After contourlet transform, the low frequency sub-band mainly reflect the detail of the image. For low-frequency sub-band images mainly adopt the fusion rules of a pixel value of extreme value method, the pixel values average method, etc. The pixel values of extremum method is that two fusing images are merged based on the minimum or maximum principle of image gray values to get the final image. The method proved feasible and it is easy to implement. But it cannot be integrated the original medical image information, makes the distortion of the fusion results because of the extreme value. Pixel values average method includes the direct average method and weighted average method. Average operation is to smooth the image, so it can make the fusion image on the edge and contour part becomes relatively vague. And the overall image contrast will be affected. At the same time, the characteristics of the pixel in the surrounding area cannot be reflected. In order to retain more image information effectively, ratio analysis fusion rule is adopted for low frequency sub-band in this article. Low frequency sub-band coefficients fusion rules are as follows: C_a(m, n) and C_b(m, n) are the low frequency coefficients of the two original images. C_f(m, n) is the low frequency coefficients of the final fusion image. And a is the ratio weighted coefficient in low frequency subband coefficients. Low frequency fusion rules are as follows in Eq. (6):

$$A=\{\begin{array}{l}{\displaystyle \frac{C_a(m,n)\times a_1}{C_b(m,n)\times a_1}}\\ C_a(m,n)>C_b(m,n)\\ {\displaystyle \frac{C_a(m,n)\times a_2}{C_b(m,n)\times a_2}}\\ C_a(m,n)<C_b(m,n)\end{array}$$

(6)

The proposed ratio weighted fusion method can integrate the information and retain the fea-tures of source images more effectively compared with the traditional algorithm. The sensitive information can be extracted effectively by weighted coefficient. Using this algorithm to image fusion will obtain better fusion effect than the average algorithm.

3.3. Fusion Rules for High Frequency Subband

In the fusion scheme, the high frequency subband coefficients are fused by the main image con-dition weighted analysis method. a and b are the source images. a_L(L) = (L = 2, …, Len) is defined as the i level pyramid decomposition of the image a. And b_L(L) = (L = 2, …, Len) is defined as the level i pyramid decomposition of the image b. They store the high frequency subband coefficients by decomposition of directional filter bank. Suppose that C_a(m, n), C_b(m, n) and C_f(m, n) are the high frequency components of a, b and the final fusion image, respectively. m denotes the level number of multi-scale decompositions, and mth level. The absolute values of different scales high frequency subband coefficients are calculated. The larger absolute value is selected as the weighted values. Then the fusion rules by region energy method are adopted in smaller absolute value high frequency subband coefficients. The calculation formula of the absolute value weighted coefficient with the main image representation is as follows:

Step 1: The coefficients of C_a(m, n) and C_b(m, n) are both taken to be the feeding input to stimulate the corresponding high frequency subband coefficients.

Step 2: Initialize U_ij[0] = T_ij[0] = Y_ij[0] = 0, E_ij[0] = max(|C_a(m, n)|, |C_b(m, n)|).

4. The Result and Analysis of the Experiment

In the image fusion experiments, and their performances are compared and discussed. Compared different fusion results based on shearlet transform of original (shown in Figure 1 (a) and (h), respectively) images are given in the first Figure 2 below. And compared different fusion results based on shearlet transform and different methods of original images are given in the second Figure 3 below. The result fusion images based on shearlet transform are always shown in last column. Further, the superiority of the proposed original image fusion method is assessed by comparing its results with the following existing fusion methods: Method 1: pixels minimum method. Method 2: pixels maximum method. Method 3: average and absolute maximum fusion rules. Method 4: the different fusion method such as the statistical fusion method of Principal Component Analysis (PCA) method [17]. Method 5: multi-resolution decomposition of the Laplacian pyramid transform method. Method 6: wavelet decomposition method [16].

Figure 1. The original image and position scrambling images: (a)-(h).

Figure 2. Fusion results of different methods for original images: (a) Method 1, (b) Method 2, (c) Method 3, (d) Method 4.

Figure 3. Fusion results of different methods for original image pair: (a) Method 1, (b) Method 2, (c) Method 3, (d) Method 4.

The fusion image based on shearlet transform ratio weighting and conditions weighted is shown in Figure 2. The fusion image based on minimum method in the low frequency subband and high frequency subband is shown in Figure 3. The fusion image based on maximum method in the low frequency subband and high frequency subband is shown in Figure 4. The fusion image based on average method in the low frequency subband and maximum method [10] in the high frequency subband is shown in Figure 5. For the experiment analysis of the fusion results produced by the various methods, it is observed that some conclusion can be recorded: maximum fusion rules can make the brightness of the fused images compared to the minimum method fusion rules. Further, these subjective evaluations of the proposed and traditional methods are also supported by the quantitative results measures as mentioned above as shown in Table 1. And the dis-tortion of the fusion results will be producing in this method. Pixel greatly method, average and the fusion effect of absolute value maximum fusion rules are more effective than the pixel minimum method fusion rules. This paper presents the fusion rule to make the fusion images contain the information of original images as much as possible in the information entropy and mutual information index. These objective outcomes can be shown in Table 1 and Table 2 from different datasets using different fusion methods. Table 2 also shows the fusion results of different fusion methods with seven objective indicators. The different fusion method such as the statistical fusion method of Principal Component Analysis (PCA) method [17], multi-resolution decompo-sition of the Laplacian pyramid transform method and wavelet decomposition method [16] are compared with the proposed algorithm in this paper. Image fusion result based on Laplacian pyramid transform is shown in Figure 4. Image fusion result based on principal component anal-ysis is shown in Figure 5. Image fusion result based on wavelet decomposition method [15,16] is shown in Figure 5. Then used different fusion rules in different subbands, where the big ab-solute value rule was adopted in highpass subbands and average rule was adopted in lowpass subband.

Figure 4. Fusion results of different methods for original image pair: (a) Method 3, (b) Method 4, (c) Method 5, (d) Method 6.

Figure 5. Fusion results of different methods for original image pair: (a) Method 3, (b) Method 4, (c) Method 5.

Table 1. Evaluation of image fusion results based on different fusion rules.

	Information Entropy	Mutual Information	Cross Entropy	Correlation Coefficient
Method1	5.2014	9.3514	1.6214	0.4011
Method2	5.4418	9.9847	0.9542	0.9773
Method3	5.1946	9.2457	0.8023	0.6244
Proposed method	5.8106	11.1410	0.8142	1.0010

Table 2. Evaluation of image fusion results based on different fusion rules.

	Information Entropy	Root Mean Square Error	Peak Signal to Noise Ratio	Average Gradient
Method4	4.9639	2.8412	9.2140	0.0514
Method5	5.0356	1.9502	10.0214	0.0312
Method6	4.9801	2.3142	10.0424	0.0512
Proposed method	6.0102	1.7412	11.0014	0.0812

5. Conclusion

Shearlet transform is a kind transformation of multi-resolution decomposition and multi-direction decomposition. The support interval of shearlet base has a fixed structure which can be changed according to the changed ratio of the scales 2D image geometric structure can be obtained based on the shearlet transform. And its application in medicine is also explored, which further sup-ports the methods advanced in the paper. This paper proposes a novel images fusion method based on shearlet transform ratio weight and condition weight fusion algorithm. The experimen-tal results show that, the proposed fusion method can integrate the information and retain the features of source images more effectively compared with the traditional algorithm in the subjec-tive and objective evaluation index. The experimental results show that new algorithm improve the entropy of the image, protect the characters of the image and look well with the evaluation of the information entropy, mutual information and average gradient. But this does not mean this technology has improved. There are still many theoretical and technical issues to be resolved in the medical image fusion. The proposed algorithm is applied to the other mode of medical image fusion is the research emphasis in the future.