Next Article in Journal
Multiple Observations for Secret-Key Binding with SRAM PUFs
Next Article in Special Issue
A Foreground-Aware Framework for Local Face Attribute Transfer
Previous Article in Journal
On Explaining Quantum Correlations: Causal vs. Non-Causal
Previous Article in Special Issue
Unsupervised Exemplar-Domain Aware Image-to-Image Translation
 
 
Font Type:
Arial Georgia Verdana
Font Size:
Aa Aa Aa
Line Spacing:
Column Width:
Background:
Article

Pulse Coupled Neural Network-Based Multimodal Medical Image Fusion via Guided Filtering and WSEML in NSCT Domain

Department of Electronic Engineering, Tsinghua University, Beijing 100084, China
*
Author to whom correspondence should be addressed.
Entropy 2021, 23(5), 591; https://doi.org/10.3390/e23050591
Submission received: 22 March 2021 / Revised: 26 April 2021 / Accepted: 30 April 2021 / Published: 11 May 2021
(This article belongs to the Special Issue Advances in Image Fusion)

Abstract

:
Multimodal medical image fusion aims to fuse images with complementary multisource information. In this paper, we propose a novel multimodal medical image fusion method using pulse coupled neural network (PCNN) and a weighted sum of eight-neighborhood-based modified Laplacian (WSEML) integrating guided image filtering (GIF) in non-subsampled contourlet transform (NSCT) domain. Firstly, the source images are decomposed by NSCT, several low- and high-frequency sub-bands are generated. Secondly, the PCNN-based fusion rule is used to process the low-frequency components, and the GIF-WSEML fusion model is used to process the high-frequency components. Finally, the fused image is obtained by integrating the fused low- and high-frequency sub-bands. The experimental results demonstrate that the proposed method can achieve better performance in terms of multimodal medical image fusion. The proposed algorithm also has obvious advantages in objective evaluation indexes VIFF, QW, API, SD, EN and time consumption.

1. Introduction

In recent years, numerous medical image processing algorithms are being extensively used for visualizing complementary information. Medical image fusion is a very effective technique in combining the important information obtained from the multimodal images into one single composite image and enhance the diagnostic accuracy [1,2]. Medical images can be divided into the following categories: Computed tomography (CT), magnetic resonance imaging (MRI), positron emission tomography (PET), single-photon emission CT (SPECT) etc. Usually, there is no single imaging method that can reflect the complete tissue information; medical image fusion technology can retain the diagnostic information of input image to the maximum extent [3,4]. Figure 1 shows the example of image fusion, it involves not only medicine, but also a multifocus image and remote sensing image. In this paper, we mainly discuss the application of multimodal medical image fusion.
At present, a lot of image fusion techniques have been proposed by the researchers, and these image fusion methods are broadly categorized as spatial domain and transform domain [5,6]. The spatial domain-based image fusion methods have high computational efficiency, but these methods suffer from poor contrast and spatial localization [7,8]. In terms of technical development, many multiscale transform decomposition methods have been introduced to design an effective platform that provide better localization of an image contour and texture details [9]. These transforms include the discrete wavelet transform (DWT) [10], stationary wavelet transform (SWT) [11], dual-tree complex wavelet transform (DTCWT) [12], curvelet transform (CVT) [13], contourlet transform (CNT) [14], surfacelet transform [15], non-subsampled contourlet transform (NSCT) [16], shearlet transform (ST) [17], non-subsampled shearlet transform (NSST) [18], adjustable non-subsampled shearlet transform (ANSST) [19] etc. Iqbal et al. [20] proposed a novel multifocus image fusion scheme utilizing discrete wavelet transform and guided image filtering, which can provide outperformance fusion results both on qualitative and quantitative comparisons. Wang et al. [21] introduced a technique for multifocus image fusion based on discrete wavelet transform and convolutional neural network (CNN), leading to better fusion results than traditional DWT-based fusion algorithm. DTCWT is an extension of DWT and has translation invariance. Aishwarya et al. [22] proposed an image fusion method utilizing DTCWT and adaptive combined clustered dictionary, leading to good performance than the conventional multiscale transform-based algorithms and the state-of-the-art sparse representation-based algorithms.
Due to the limited ability in capturing directional information in two-dimensional space about the wavelets based methods, most wavelet transforms cannot generate an optimal representation for images. In order to address the aforementioned problem, a series of multi-scale geometric analysis (MGA) theory including curvelet, contourlet and shearlet have been introduced by the scientist, these methods accelerate the development of image fusion technology. Mao et al. [23] proposed an image fusion technique based on curvelet transform and sparse representation. Chen et al. [24] introduced an approach for multi-source optical remote sensing image fusion based on principal component analysis and curvelet transform. Li et al. [25] introduced the non-subsampled contourlet transform into the medical image fusion based on fuzzy entropy and regional energy. Wu et al. [26] conducted another NSCT-based work using pulse coupled neural network (PCNN) for infrared and visible image fusion. Li et al. [27] proposed an image fusion scheme based on parameter-adaptive pulse coupled neural network (PAPCNN) and improved sum-modified-laplacian (ISML) in non-subsampled shearlet transform (NSST) domain, leading to a good fusion performance.
In recent years, the sparse representation-based, convolutional neural network-based, edge-preserving filter-based techniques also achieve successfully in the field of image fusion. Xing et al. [28] proposed an image fusion method based on Taylor expansion theory and convolutional sparse representation with gradient penalties scheme. Liu et al. [29] introduced an adaptive sparse representation (ASR) for multimodal image fusion and denoising. Liu et al. [30] proposed an image fusion technique using deep convolutional neural network (DCNN), leading to state-of-the-art image fusion performance in terms of visual quality and objective assessment. Li et al. [31] introduced the guided image filtering for image fusion (GFF), and the calculation efficiency is relatively high. The main image fusion models mentioned above can be summarized as shown in Table 1.
The image fusion methods, based on transform domain, mainly use different energy functions to construct the weight of the source image for image fusion. Although the details of each source image can be well-preserved, the space continuity of the high- and low-frequency coefficients in the transform domain is often not considered, the fused image will introduce artificial texture, which will affect the image fusion effect. In this paper, a novel fusion model with pulse coupled neural network (PCNN) and weighted sum of eight-neighborhood-based modified Laplacian (WSEML) in NSCT domain is proposed for multimodal medical image fusion. The guided filtering is introduced to enhance the spatial continuity of the image, and then the corresponding artificial texture is suppressed and the gray level of the fused image is enhanced. The contributions of the proposed framework can be summarized as follows: (1) The multiscale NSCT decomposition is used to decompose the input source images into low- and high-frequency components; (2) the PCNN is adopted to fuse the low-frequency components, and the WSEML integrating guided image filtering is utilized to fuse the high-frequency components. The guided image filtering is a good edge-preserving filter, the proposed model can efficiently capture the spatial information and suppress noise; (3) the effectiveness of the proposed work is authenticated utilizing the extensive experimental fusion results and comparisons with the state-of-the-art image fusion algorithms.
The rest of this work is organized as follows. Preliminaries is briefly reviewed in Section 2. The proposed fusion algorithm is illustrated in Section 3. The experimental results and discussions are shown in Section 4. The conclusions are presented in Section 5.

2. Preliminaries

2.1. Non-Subsampled Contourlet Transform

The non-subsampled contourlet transform (NSCT) is an improved model of contourlet, NSCT adopts the multiscale, multidirectional analysis and shift-invariance. It is applied for image decomposition into one low-frequency and several high-frequency sub-bands. The decomposition model utilizes non-subsampled pyramid (NSP) to generate low-frequency and high-frequency components and then the non-subsampled directional filter bank (NSDFB) is adopted to generate several sub-image components [32]. The overview of NSCT is depicted in Figure 2. NSCT is recognized as an effective method for image fusion [25,26], and it is selected as the multiscale transform for proposed fusion algorithm in this paper.

2.2. Pulse Coupled Neural Network

Pulse coupled neural network (PCNN) is a feedback network, and it is widely used in the field of image fusion. In particular, it is reasonable to apply the PCNN model to merge the low-frequency components generated by the NSCT. The PCNN model is described as follows [16]:
F i j n   = S i j
L i j n   = e α L L i j n 1   +   V L p q W i j , p q Y i j , p q n 1
U i j n   = F i j n     1 + β L i j n
θ i j n   = e α θ θ i j n 1   +   V θ Y i j n 1
Y i j n   =   1 if   U i j n   > θ i j n 0 else
T i , j = T i , j n 1   +   Y i , j n
where F i j shows the feeding input and Sij denotes the external input stimulus signal, the linking input L i j depicts the sum of neurons firing times in linking range, W i j , p q represents the synaptic gain strength, α L denotes the decay constants, V L and V θ present the amplitude gain, β shows the linking strength, U i j is the total internal activity, θ i j represents the threshold, n is the iteration times, Yij is the pulse output of PCNN, T i j denotes the firing times. Figure 3 shows the architecture of the PCNN model.

2.3. Guided Image Filter

Guided image filter is a linear filtering, we suppose that the filtering output image q is the linear transform of the guidance image I in a window ω k centered at the pixel k [33]:
q i = a k I i + b k , i ω k
where ω k presents the square window of size 2 r + 1   ×   2 r + 1 . The linear coefficients a k , b k are constant in the ω k , and they could be estimated by minimizing the cost function in the window ω k :
E a k , b k   = i ω k a k I i + b k p i 2 + ε a k 2
where ε represents the regularization parameter penalizing large a k . The linear coefficients a k , b k can be calculated by the following:
a k = 1 ω i ω k I i p i μ k p ¯ k σ k 2 + ε
b k = p ¯ k a k μ k
where μ k and σ k 2 denote the mean and variance of I in ω k , ω shows the number of pixels in ω k , and p ¯ k represents the mean of p in ω k , it can be calculated by the following:
p ¯ k = 1 ω i ω k p i
In order to keep the q i value unchanged in different windows, all the possible data of a k , b k are first averaged, the filtering output can be computed by
q i = 1 ω k | i ω k a k I i + b k   = a ¯ i I i + b ¯ i
where a ¯ i and b ¯ i present the mean of a k and b k , respectively; they can be computed by
a ¯ i = 1 ω k ω i a k
b ¯ i = 1 ω k ω i b k
In this work, the G r , ε p , I is utilized to denote the guided filtering operation, r and ε denote the parameters which control the size of filter kernel and blur extent, respectively. p refers to the input image, and I represents the guidance image. The guided image filter is used to process the high-frequency components generated by NSCT.

3. Proposed Fusion Method

3.1. Overview

The proposed multimodal medical image fusion algorithm in this work is shown in Figure 4. The input source images are assumed to be well registered with the size 256 × 256, the detailed image fusion approach consists of four parts, namely NSCT decomposition, low-frequency sub-bands fusion, high-frequency sub-bands fusion, and NSCT reconstruction.

3.2. Detailed Fusion Algorithm

Step 1: NSCT decomposition
Suppose the registered input source images A and B are decomposed by NSCT transform with L-level, and generate the corresponding decomposition low- and high-frequency sub-bands L A , L B and H A l , k , H B l , k , respectively.
Step 2: Low-frequency sub-band fusion
The low-frequency sub-band contains the approximate information of the source images, in this section, the PCNN based fusion rule is applied to keep more useful information. According to the PCNN model described from Equations (1)–(6), the fusion rule is depicted in the following:
D F i , j   =   1 i f   T A , i j N   T B , i j N 0 e l s e
L F i , j   =   L A i , j i f     D i j N = 1 L B i , j e l s e
where T A , i j N and T B , i j N are the PCNN firing times, N presents the total number of iterations; D i j represents the decision map, L F i , j denotes the fused low-frequency sub-band.
Step 3: High-frequency sub-bands fusion
The high-frequency sub-bands contain the plentiful edge and texture detail information of the input images, in order to extract the details information, the weighted sum of eight-neighborhood-based modified Laplacian (WSEML) is adopted, and it is defined as follows [34]:
WSEML S i , j   = m = r r   n = r r W m + r + 1 , n + r + 1   ×   EML S i + m , j + n
EML S i , j   = | 2 S i , j S i 1 , j S i + 1 , j | + 2 S i , j S i , j 1 S i , j + 1 + 1 2 2 S i , j S i 1 , j 1 S i + 1 , j + 1 + 1 2 2 S i , j S i 1 , j + 1 S i + 1 , j 1
where S A , B ; W denotes the weighting matrix, and it can be calculated by the following:
W = 1 16 1 2 1 2 4 2 1 2 1
For the high-frequency coefficients, the fusion rule based on WSEML is adopted, and then the two zero-value matrixes mapA and mapB are initialized, and the matrixes are computed by the following:
m a p A i , j   =   1 i f   W S E M L H A l , k i , j   W S E M L H B l , k i , j 0 e l s e
m a p B ( i , j ) = 1 m a p A i , j
In order to enhance the spatial continuity of the high-frequency coefficients, the guided filter on mapA and mapB is adopted, and the corresponding coefficients H A l , k and H B l , k are utilized as the guided images:
m a p A = G r , ε m a p A , H A l , k
m a p B = G r , ε m a p B , H B l , k
where mapA and mapB should be normalized, the fused high-frequency coefficients H F l , k i , j can be generated by the following Equation:
H F l , k i , j   = m a p A × H A l , k i , j   +   m a p B × H B l , k i , j
Step 4: NSCT reconstruction
The final fused image is generated by performing inverse NSCT transform over the merged fusion sub-bands L F , H F l , k .

3.3. Extension to Color Image Fusion

The proposed medical image fusion algorithm is extended to fuse the anatomical and functional image in this section. The anatomical image contains the CT and MRI, and the functional image usually denotes the PET and SPECT. When solving the gray image and color image fusion, the image color space conversion is adopted, in this paper, the RGB to YUV color space is used to compute the anatomical and functional image fusion work [34]. The framework of the anatomical and functional image fusion is shown in Figure 5.

4. Experimental Results and Discussions

4.1. Experimental Setup

In this section, to explore the effectiveness of the proposed multimodal medical image fusion algorithm, we evaluate the method on the two public datasets http://www.imagefusion.org and http://www.med.harvard.edu/AANLIB/home.html (accessed on 10 February 2021). Figure 6 shows the selected public gray source image fusion pairs, all the CT and MRI source images have the same size with 256 × 256. Figure 7 denotes the selected anatomical and functional (MRI-PET/SPECT) images with the size 256 × 256, and all the source images are pre-registered. In addition, eight state-of-the-art fusion approaches are used to compare with the proposed scheme, namely image fusion based on non-subsampled contourlet transform (NSCT) [16], image fusion using dual-tree complex wavelet transform (DTCWT) [12], guided image filtering for image fusion (GFF) [31], image fusion utilizing ratio of low-pass pyramid (RP) [13], image fusion via adaptive sparse representation (ASR) [29], deep convolutional neural network based image fusion (DCNN) [30], image fusion using convolutional sparsity based morphological component analysis (CSMCA) [35], single-scale structural image decomposition (SSID) [36]. In this paper, the pyramid filter and directional filter with the parameters “9–7” and “pkva"; the NSCT decomposition level is 4, and the corresponding directions are 4, 4, 4, 4; the parameters of the PCNN is set as p × q , α L = 0.06931 , α θ = 0.2 , β = 3 , V L = 1.0 , V θ = 20 , W = 0.707 1 0.707 1 0 1 0.707 1 0.707 , and the iterative number is N = 200 ; the parameters r and ε of guided image filer are set as 3 and 1, respectively. For the parameters in the comparison algorithms, corresponding parameter values are as described in the original papers proposed by the scholars. Table 2 summarizes the tested algorithms and the parameter setup. All of the experiments run in win7, MATLAB R2018b software. The hardware is Intel(R) Core(TM) i5-2520M CUP (2.50 GHz) and 12-GB memory.
The proposed medical image fusion technique is evaluated and compared with other classical fusion algorithms by qualitative and quantitative analyses. In terms of qualitative analysis, it is based on human visual system such as image details, image contrast and image brightness etc. As for quantitative analysis, multiple evaluation metrics are selected to assess the proposed fusion algorithm and the comparison fusion algorithms, which include the visual information fidelity (VIFF) [37,38,39,40,41], weighted fusion quality index (QW) [42,43], average pixel intensity (API) [44], standard deviation (SD) [44], entropy (EN) [44,45,46,47,48] and time (seconds). VIFF measures the visual information fidelity of the fused image by computing the distortion of the images, a larger VIFF means the fused image has higher visual information fidelity; the QW addresses the distortions of coefficient correlation, illumination and contrast between the source images and fused image, a larger QW means less distortion of image quality; API measures an index of contrast, a larger API reflects the fused image has higher contrast; SD measures the amount of information contained in the fused image from the perspective of statistics and reflects the overall contrast, a larger SD reflects the fused image contains more information and higher contrast; the computation of EN value is based on information theory, and it measures the amount of information in the fusion image, a larger EN means the fused image contains more information. The low computation time shows that the algorithm is efficient. Among the examined quantitative metrics, VIFF and QW are reference-based metrics, while API, SD and EN are no-reference evaluation metrics. The fusion method takes the anatomical or functional image as the reference, it is easy to introduce the interference information from the source images into the fusion image. In order to comprehensively evaluate the fusion performance from different perspectives, this study uses reference-based and no-reference-based indicators. The corresponding fusion results and metrics data as shown in Figure 8, Figure 9, Figure 10, Figure 11 and Figure 12 and Table 3, Table 4, Table 5, Table 6 and Table 7.

4.2. Comparison of Gray Image Fusion

Figure 8, Figure 9 and Figure 10 represent the gray medical image fusion results generated by different image fusion approaches. Figure 8 depicts the fused results of the methods on the first group gray medical images. Figure 9 presents the fusion results of the algorithms on the second group gray medical images. Figure 10 shows the fused results of the methods on other gray medical images.
With regard to the visual performance, the edge information in Subfigure (a) of Figure 8 and Figure 9 denotes that the fused images of NSCT have lost some details of MRI images, and the results have some noise, which affects the doctor’s observation. From the Subfigure (b) of Figure 8 and Figure 9 generated by the DTCWT method have the low contrast and brightness. We can denote the blocking artifacts are generated by GFF algorithm as shown in Subfigure (c) of Figure 8 and Figure 9, due to the guided image filtering is a non-linear filter, it needs the same or better guidance image to implement the smoothing process. The fused images calculated by RP and DCNN schemes as shown in Subfigures (d) and (f) of Figure 8 and Figure 9, respectively, and the results produce certain kinds of distortions, especially the Figure 8f obtained by DCNN, almost all the information of MRI image is lost in the fusion image. ASR algorithm can generate the block effect and the gradient contrast is poor, which could be denoted from Subfigure (e) of Figure 8 and Figure 9. It can be seen from Subfigure (g) of Figure 8 and Figure 9 that the fused results computed by CSMCA approach lead to information loss. The fusion results calculated by the SSID and proposed techniques are relatively high-quality, and they are depicted in Subfigures (h) and (i) of Figure 8 and Figure 9, the results of the proposed method retain more image information and the brightness is higher.
In order to reduce the influence of individual subjective judgment on image fusion quality as far as possible, the objective evaluation indicators are introduced, and the corresponding index values are shown in Table 3, Table 4 and Table 5. From the Table 3, in terms of QW, API, SD and EN, the proposed approach generates superb performance, although the best data for VIFF and Time are generated by GFF and SSID, with 0.4863 and 0.1608, respectively. From the Table 4, we can see that the values of VIFF, QW, API and SD obtained by the proposed fusion scheme are the best, while the best data for EN and Time are generated by GFF and SSID, with 5.3836 and 0.0721, respectively. In order to analyze the universality of the fusion algorithms more objectively, we take the average values of the index data obtained from nine groups of gray medical images computed by the nine fusion methods, as shown in Table 5, in addition to the EN and Time values, the other four metric values obtained by the proposed algorithm are the best.

4.3. Comparison of Anatomical and Functional Image Fusion

In this section, nine groups of color medical images (MRI-PET/SPECT) are used to assess the fused results of the proposed fusion technique, and the corresponding comparative analysis is given. The typical MRI-PET fusion results of the techniques are given in Figure 11. From the Figure 11, we can denote that the fused images such as Figure 11a–c,f generated by the NSCT, DTCWT, GFF, and DCNN algorithms, respectively, suffer from color distortion. Figure 11d–e are the fusion results computed by the RP and ASR methods, respectively; it can be clearly denoted that the fused results still exists the color distortion, but the image contrast and brightness have improved. The fused image computed by the CSMCA is shown in Figure 11g, and the artificial textures are appeared, the fusion effects are undesirable. The fused images calculated by SSID and the proposed methods are depicted in Figure 11h,i, respectively, the two fused images are similar, but the proposed method has a better fusion performance and higher brightness. Figure 12 shows the fused results of different algorithms on the other eight groups of anatomical and functional images.
The quantitative assessments on the fused images of Figure 11 corresponding to the first group anatomical and functional images are tabulated in Table 6. We can see that the metrics data of API, SD and EN computed by the proposed algorithm are the best compared with other state-of-the-art fusion strategies, while the best data for VIFF, QW and Time are computed by RP and SSID, with 0.8443, 0.8471and 0.0806, respectively.
Here, the average of the six metrics calculated by the various fusion approaches on the selected nine groups anatomical and functional images in Figure 7 are recorded, as shown in Table 7. In contrast to the other fusion techniques, there is a remarkable enhancement on the metrics API, SD and EN. The overall comparative analysis shows that the proposed scheme works better in terms of anatomical and functional images fusion, demonstrating its effectiveness.
From the anatomical-anatomical image fusion results and anatomical-functional image fusion results aforementioned, the proposed algorithm has obvious advantages in subjective and objective evaluations compared with other state-of-the-art fusion algorithms. The PCNN fusion rule and GIF-WSEML fusion rule are used in the NSCT domain, the combination of the two fusion models denotes better preservation of spatial and spectral features. The fusion images can provide accurate location of defected tissues, and provide meaningful quantitative explanation for clinical diagnosis. Given that there are many parameters in this algorithm, it needs continuous manual debugging to select the appropriate values of parameters to achieve the optimal fusion effect.

5. Conclusions

In this paper, a practical multimodal medical image fusion algorithm based on PCNN and GIF-WSEML in non-subsampled contourlet transform domain is introduced. For sub-bands fusion, two different rules are adopted, the low-frequency sub-bands are fused by PCNN model, and the weighted sum of eight-neighborhood-based modified Laplacian integrating guided image filtering (GIF-WSEML) is used to merge the high-frequency sub-bands. The nine groups of anatomical-anatomical images and nine groups of anatomical-functional images are used to simulate by the proposed framework and other conventional fusion approaches. The comparative experimental fusion results conducted on both gray and color medical image datasets demonstrate that the proposed fusion algorithm has a better performance with improved brightness and contrast of multimodal medical images, and the objective metrics such as VIFF, QW, API, SD and EN computed by the proposed method also have obvious advantages. Compared to DTCWT, GFF, RP and SSID, the time consuming of the proposed method is high, so reducing the operation time and improving the real-time performance of the algorithm are the problems we need to solve in the future.

Author Contributions

The experimental measurements and data collection were carried out by L.L. and H.M. The manuscript was written by L.L. with the assistance of H.M. All authors have read and agreed to the published version of the manuscript.

Funding

This work was supported by the Shanghai Aerospace Science and Technology Innovation Fund under Grant No. SAST2019-048.

Institutional Review Board Statement

Not applicable.

Informed Consent Statement

Not applicable.

Data Availability Statement

Not applicable.

Conflicts of Interest

The authors declare no conflict of interest.

Abbreviations

PCNNpulse coupled neural network
WSEMLweighted sum of eight-neighborhood-based modified Laplacian
GIFguided image filtering
NSCTnonsubsampled contourlet transform
CTcomputed tomography
MRImagnetic resonance imaging
PETpositron emission tomography
SPECTsingle-photon emission CT
DWTdiscrete wavelet transform
SWTstationary wavelet transform
DTCWTdual-tree complex wavelet transform
CVTcurvelet transform
CNTcontourlet transform
STshearlet transform
NSSTnonsubsampled shearlet transform
ANSSTadjustable nonsubsampled shearlet transform
CNNconvolutional neural network
MGAmulti-scale geometric analysis
PAPCNNparameter-adaptive pulse coupled neural network
ISMLimproved sum-modified-laplacian
DCNNdeep convolutional neural network
GFFguided image filtering for image fusion
NSPnonsubsampled pyramid
NSDFBnonsubsampled directional filter bank
RPratio of low-pass pyramid
ASRadaptive sparse representation
CSMCAconvolutional sparsity based morphological component analysis
SSIDsingle-scale structural image decomposition
VIFFvisual information fidelity
QWweighted fusion quality index
APIaverage pixel intensity
SDstandard deviation
ENentropy

References

  1. Singh, S.; Anand, R.S. Multimodal medical image sensor fusion model using sparse K-SVD dictionary learning in nonsubsampled shearlet domain. IEEE Trans. Instrum. Meas. 2020, 69, 593–607. [Google Scholar] [CrossRef]
  2. Kong, W.; Miao, Q.; Lei, Y. Multimodal sensor medical image fusion based on local difference in non-subsampled domain. IEEE Trans. Instrum. Meas. 2019, 68, 938–951. [Google Scholar] [CrossRef]
  3. Wang, Z.; Cui, Z.; Zhu, Y. Multi-modal medical image fusion by Laplacian pyramid and adaptive sparse representation. Comput. Biol. Med. 2020, 123, 103823. [Google Scholar] [CrossRef] [PubMed]
  4. Zhang, L.; Zeng, G.; Wei, J. Multi-modality image fusion in adaptive-parameters SPCNN based on inherent characteristics of image. IEEE Sens. J. 2020, 20, 11820–11827. [Google Scholar] [CrossRef]
  5. Liu, Y.; Zhou, D.; Nie, R. Robust spiking cortical model and total-variational decomposition for multimodal medical image fusion. Biomed. Signal Process. Control 2020, 61, 101996. [Google Scholar] [CrossRef]
  6. Zhang, Y.; Liu, Y.; Sun, P. IFCNN: A general image fusion framework based on convolutional neural network. Inf. Fusion 2020, 54, 99–118. [Google Scholar] [CrossRef]
  7. Ma, J.; Liang, P.; Yu, W. Infrared and visible image fusion via detail preserving adversarial learning. Inf. Fusion 2020, 54, 2020. [Google Scholar] [CrossRef]
  8. Liu, Y.; Wang, L.; Cheng, J. Multi-focus image fusion: A survey of the state of the art. Inf. Fusion 2020, 64, 71–91. [Google Scholar] [CrossRef]
  9. Liu, Y.; Chen, X.; Wang, Z. Deep learning for pixel-level image fusion: Recent advances and future prospects. Inf. Fusion 2018, 42, 158–173. [Google Scholar] [CrossRef]
  10. Talal, T.; Attiya, G. Satellite image fusion based on modified central force optimization. Multimed. Tools Appl. 2020, 79, 21129–21154. [Google Scholar] [CrossRef]
  11. Liu, S.; Chen, J.; Rahardja, S. A new multi-focus image fusion algorithm and its efficient implementation. IEEE Trans. Circuits Syst. Video Technol. 2020, 30, 1374–1384. [Google Scholar] [CrossRef]
  12. Singh, R.; Srivastava, R. Multimodal medical image fusion in dual tree complex wavelet transform domain using maximum and average fusion rules. J. Med. Imaging Health Inform. 2012, 2, 168–173. [Google Scholar] [CrossRef]
  13. Liu, Y.; Liu, S.; Wang, Z. A general framework for image fusion based on multi-scale transform and sparse representation. Inf. Fusion 2015, 24, 147–164. [Google Scholar] [CrossRef]
  14. Do, M.N.; Vetterli, M. The contourlet transform: An efficient directional multiresolution image representation. IEEE Trans. Image Process. 2005, 14, 2091–2106. [Google Scholar] [CrossRef] [Green Version]
  15. Li, B.; Peng, H. Multi-focus image fusion based on dynamic threshold neural P systems and surfacelet transform. Knowl. Based Syst. 2020, 196, 105794. [Google Scholar] [CrossRef]
  16. Qu, X.; Yan, J.; Xiao, H. Image fusion algorithm based on spatial frequency-motivated pulse coupled neural networks in nonsubsampled contourlet transform domain. Acta Autom. Sin. 2008, 34, 1508–1514. [Google Scholar] [CrossRef]
  17. Guo, K.; Labate, D. Optimally sparse multidimensional representation using shearlets. Siam J. Math. Anal. 2007, 39, 298–318. [Google Scholar] [CrossRef] [Green Version]
  18. Li, L.; Wang, L.; Wang, Z. A novel medical image fusion approach based on nonsubsampled shearlet transform. J. Med. Imaging Health Inform. 2019, 9, 1815–1826. [Google Scholar] [CrossRef]
  19. Vishwakarma, A.; Bhuyan, M.K. Image fusion using adjustable non-subsampled shearlet transform. IEEE Trans. Instrum. Meas. 2019, 68, 3367–3378. [Google Scholar] [CrossRef]
  20. Iqbal, M.; Riaz, M. A multifocus image fusion using highlevel DWT components and guided filter. Multimed. Tools Appl. 2020, 79, 12817–12828. [Google Scholar]
  21. Wang, Z.; Li, X.; Duan, H. Multifocus image fusion using convolutional neural networks in the discrete wavelet transform domain. Multimed. Tools Appl. 2019, 78, 34483–34512. [Google Scholar] [CrossRef]
  22. Aishwarya, N.; Bennila, T.C. Visible and Infrared image fusion using DTCWT and adaptive combined clustered dictionary. Infrared Phys. Technol. 2018, 93, 300–309. [Google Scholar] [CrossRef]
  23. Mao, Q.; Zhu, Y.; Lv, C. Image fusion based on multiscale transform and sparse representation to enhance terahertz images. Opt. Express 2020, 28, 25293–25307. [Google Scholar] [CrossRef] [PubMed]
  24. Chen, C.; He, X.; Guo, B. A pixel-level fusion method for multi-source optical remote sensing image combining the principal component analysis and curvelet transform. Earth Sci. Inform. 2020, 13, 1005–1013. [Google Scholar] [CrossRef]
  25. Li, W.; Lin, Q.; Wang, K. Improving medical image fusion method using fuzzy entropy and nonsubsampling contourlet transform. Int. J. Imaging Syst. Technol. 2020, 30, 204–214. [Google Scholar] [CrossRef]
  26. Wu, C.; Chen, L. Infrared and visible image fusion method of dual NSCT and PCNN. PLoS ONE 2020, 15, e0239535. [Google Scholar]
  27. Li, L.; Si, Y.; Wang, L. A novel approach for multi-focus image fusion based on SF-PAPCNN and ISML in NSST domain. Multimed. Tools Appl. 2020, 79, 24303–24328. [Google Scholar] [CrossRef]
  28. Xing, C.; Wang, M.; Dong, C. Using taylor expansion and convolutional sparse representation for image fusion. Neurocomputing 2020, 402, 437–455. [Google Scholar] [CrossRef]
  29. Liu, Y.; Wang, Z. Simultaneous image fusion and denoising with adaptive sparse representation. IET Image Process. 2015, 9, 347–357. [Google Scholar] [CrossRef] [Green Version]
  30. Liu, Y.; Chen, X.; Peng, H. Multi-focus image fusion with a deep convolutional neural network. Inf. Fusion 2017, 36, 191–207. [Google Scholar] [CrossRef]
  31. Li, S.; Kang, X.; Hu, J. Image fusion with guided filtering. IEEE Trans. Image Process. 2013, 22, 2864–2875. [Google Scholar]
  32. Da, A.; Zhou, J.; Do, M. The nonsubsampled contourlet transform: Theory, design, and applications. IEEE Trans. Image Process. 2006, 15, 3089–3101. [Google Scholar]
  33. He, K.; Sun, J.; Tang, X. Guided image filtering. IEEE Trans. Pattern Anal. Mach. Intell. 2013, 35, 1397–1409. [Google Scholar] [CrossRef]
  34. Yin, M.; Liu, X.; Liu, Y. Medical image fusion with parameter-adaptive pulse coupled neural network in nonsubsampled shearlet transform domain. IEEE Trans. Instrum. Meas. 2019, 68, 49–64. [Google Scholar] [CrossRef]
  35. Liu, Y.; Chen, X.; Ward, R. Medical image fusion via convolutional sparsity based morphological component analysis. IEEE Signal Process. Lett. 2019, 26, 485–489. [Google Scholar] [CrossRef]
  36. Li, H.; Qi, X.; Xie, W. Fast infrared and visible image fusion with structural decomposition. Knowl. Based Syst. 2020, 204, 106182. [Google Scholar] [CrossRef]
  37. Han, Y.; Cai, Y.; Cao, Y. A new image fusion performance metric based on visual information fidelity. Inf. Fusion 2013, 14, 127–135. [Google Scholar] [CrossRef]
  38. Li, L.; Ma, H.; Jia, Z. A novel multiscale transform decomposition based multi-focus image fusion framework. Multimed. Tools Appl. 2021, 80, 12389–12409. [Google Scholar] [CrossRef]
  39. Li, L.; Si, Y. Enhancement of hyperspectral remote sensing images based on improved fuzzy contrast in nonsubsampled shearlet transform domain. Multimed. Tools Appl. 2019, 78, 18077–18094. [Google Scholar] [CrossRef]
  40. Zhang, H.; Le, Z.; Shao, Z.; Xu, H.; Ma, J. MFF-GAN: An unsupervised generative adversarial network with adaptive and gradient joint constraints for multi-focus image fusion. Inf. Fusion 2021, 66, 40–53. [Google Scholar] [CrossRef]
  41. Li, L.; Si, Y. Brain image enhancement approach based on singular value decomposition in nonsubsampled shearlet transform domain. J. Med. Imaging Health Inform. 2020, 10, 1785–1794. [Google Scholar] [CrossRef]
  42. Liu, Z.; Blasch, E. Objective assessment of multiresolution image fusion algorithms for context enhancement in night vision: A comparative study. IEEE Trans. Pattern Anal. Mach. Intell. 2012, 34, 94–109. [Google Scholar] [CrossRef] [PubMed]
  43. Wang, L.; Li, B.; Tian, L. EGGDD: An explicit dependency model for multi-modal medical image fusion in shift-invariant shearlet transform domain. Inf. Fusion 2014, 19, 29–37. [Google Scholar] [CrossRef]
  44. Kumar, B.K.S. Image fusion based on pixel significance using cross bilateral filter. Signal Image Video Process. 2015, 9, 1193–1204. [Google Scholar] [CrossRef]
  45. Du, J.; Li, W. Two-scale image decomposition based image fusion using structure tensor. Int. J. Imaging Syst. Technol. 2020, 30, 271–284. [Google Scholar] [CrossRef]
  46. Ma, J.; Zhou, Y. Infrared and visible image fusion via gradientlet filter. Comput. Vis. Image Underst. 2020, 197, 103016. [Google Scholar] [CrossRef]
  47. Xu, H.; Ma, J.; Zhang, X. MEF-GAN: Multi-exposure image fusion via generative adversarial networks. IEEE Trans. Image Process. 2020, 29, 7203–7216. [Google Scholar] [CrossRef]
  48. Chen, J.; Li, X.; Luo, L. Infrared and visible image fusion based on target-enhanced multiscale transform decomposition. Inf. Sci. 2020, 508, 64–78. [Google Scholar] [CrossRef]
Figure 1. The example of image fusion.
Figure 1. The example of image fusion.
Entropy 23 00591 g001
Figure 2. The overview of NSCT [32]. (a) Non-subsampled filter bank structure; (b) Idealized frequency partitioning.
Figure 2. The overview of NSCT [32]. (a) Non-subsampled filter bank structure; (b) Idealized frequency partitioning.
Entropy 23 00591 g002
Figure 3. Architecture of the PCNN model.
Figure 3. Architecture of the PCNN model.
Entropy 23 00591 g003
Figure 4. The schematic diagram of the proposed fusion method.
Figure 4. The schematic diagram of the proposed fusion method.
Entropy 23 00591 g004
Figure 5. Process flow for the proposed algorithm for anatomical and functional image in YUV color space.
Figure 5. Process flow for the proposed algorithm for anatomical and functional image in YUV color space.
Entropy 23 00591 g005
Figure 6. Test gray medical images.
Figure 6. Test gray medical images.
Entropy 23 00591 g006
Figure 7. Test anatomical and functional images.
Figure 7. Test anatomical and functional images.
Entropy 23 00591 g007
Figure 8. Fusion results of the first group gray medical images. (a) NSCT; (b) DTCWT; (c) GFF; (d) RP; (e) ASR; (f) DCNN; (g) CSMCA; (h) SSID; (i) Proposed method.
Figure 8. Fusion results of the first group gray medical images. (a) NSCT; (b) DTCWT; (c) GFF; (d) RP; (e) ASR; (f) DCNN; (g) CSMCA; (h) SSID; (i) Proposed method.
Entropy 23 00591 g008
Figure 9. Fusion results of the second group gray medical images. (a) NSCT; (b) DTCWT; (c) GFF; (d) RP; (e) ASR; (f) DCNN; (g) CSMCA; (h) SSID; (i) Proposed method.
Figure 9. Fusion results of the second group gray medical images. (a) NSCT; (b) DTCWT; (c) GFF; (d) RP; (e) ASR; (f) DCNN; (g) CSMCA; (h) SSID; (i) Proposed method.
Entropy 23 00591 g009aEntropy 23 00591 g009b
Figure 10. Simulation results of other seven groups of gray medical images in Figure 6. From top to bottom, the fusion results of NSCT, DTCWT, GFF, RP, ASR, DCNN, CSMCA, SSID and proposed method are in turn.
Figure 10. Simulation results of other seven groups of gray medical images in Figure 6. From top to bottom, the fusion results of NSCT, DTCWT, GFF, RP, ASR, DCNN, CSMCA, SSID and proposed method are in turn.
Entropy 23 00591 g010
Figure 11. Fusion results of the first group anatomical and functional images. (a) NSCT; (b) DTCWT; (c) GFF; (d) RP; (e) ASR; (f) DCNN; (g) CSMCA; (h) SSID; (i) Proposed method.
Figure 11. Fusion results of the first group anatomical and functional images. (a) NSCT; (b) DTCWT; (c) GFF; (d) RP; (e) ASR; (f) DCNN; (g) CSMCA; (h) SSID; (i) Proposed method.
Entropy 23 00591 g011aEntropy 23 00591 g011b
Figure 12. Simulation results of other eight groups of anatomical and functional images in Figure 7. From top to bottom, the fusion results of NSCT, DTCWT, GFF, RP, ASR, DCNN, CSMCA, SSID and proposed method are in turn.
Figure 12. Simulation results of other eight groups of anatomical and functional images in Figure 7. From top to bottom, the fusion results of NSCT, DTCWT, GFF, RP, ASR, DCNN, CSMCA, SSID and proposed method are in turn.
Entropy 23 00591 g012
Table 1. The classifications and methods of main image fusion models.
Table 1. The classifications and methods of main image fusion models.
CategoriesMethods
Multiscale transform decompositiondiscrete wavelet transform (DWT) [10], stationary wavelet transform (SWT) [11], dual-tree complex wavelet transform (DTCWT) [12], curvelet transform (CVT) [13], contourlet transform (CNT) [14],
surfacelet transform [15], non-subsampled contourlet transform (NSCT) [16], shearlet transform (ST) [17], nonsubsampled shearlet transform (NSST) [18], adjustable non-subsampled shearlet transform (ANSST) [19]
Sparse representationconvolutional sparse representation [28],
adaptive sparse representation (ASR) [29]
Deep learningdeep convolutional neural network (DCNN) [30]
Edge-preserving filterguided image filtering [31]
Table 2. All tested algorithms and the parameter settings.
Table 2. All tested algorithms and the parameter settings.
MethodsParameter Setting
NSCT [16]PCNN is set as p × q , α L = 0.06931 , α θ = 0.2 , β = 0.2 , V L = 1.0 , V θ = 20 , W = 0.707 1 0.707 1 0 1 0.707 1 0.707 , and N = 200 ; the NSCT decomposition direction numbers are [4, 4, 4, 4].
DTCWT [12]L = 4
GFF [31] r 1 = 45 ,   ε 1 = 0.3 ,   r 2 = 7 ,   ε 2 = 10 6
RP [13]L = 4
ASR [29]dictionary size: 256, ε = 0.1 ,   C = 1.15 ,   σ = 0 , the number of sub-dictionaries: 7
DCNN [30]patch size = 16 × 16, convolutional layer: kernel size = 3 × 3, stride = 1, max-pooling layer: kernel size = 2 × 2, stride = 2
CSMCA [35] L = 6 ,   λ c = λ t = max 0.6 0.1 × i ,   0.005 ,   i [ 1 ,   L ]
SSID [36]r = 15
ProposedPCNN is set as p × q , α L = 0.06931 , α θ = 0.2 , β = 3 , V L = 1.0 , V θ = 20 , W = 0.707 1 0.707 1 0 1 0.707 1 0.707 , and N = 200 ; the NSCT decomposition direction numbers are [4, 4, 4, 4], r = 3 ,   ε = 1
Notes: NSCT (non-subsampled contourlet transform), DTCWT (dual-tree complex wavelet transform), GFF (guided image filtering for image fusion), RP (ratio of low-pass pyramid), ASR (adaptive sparse representation), DCNN (deep convolutional neural network), CSMCA (convolutional sparsity based morphological component analysis), SSID (single-scale structural image decomposition).
Table 3. Objective assessment of different fusion methods on the first group gray images.
Table 3. Objective assessment of different fusion methods on the first group gray images.
VIFFQWAPISDENTime/s
NSCT0.34400.783340.371949.92116.628423.3362
DTCWT0.37470.748132.511342.95036.22580.2269
GFF0.48630.833750.193053.71136.79200.2579
RP0.22560.528936.466951.58196.05000.2034
ASR0.37440.752631.515040.04836.177891.1108
DCNN0.23980.694922.383452.24473.473775.3303
CSMCA0.47520.803037.262050.74386.3268200.6023
SSID0.44230.798851.289752.42706.65800.1608
Proposed0.45940.843853.290555.15116.800017.9221
Table 4. Objective assessment of different fusion methods on the second group gray images.
Table 4. Objective assessment of different fusion methods on the second group gray images.
VIFFQWAPISDENTime/s
NSCT0.47280.832456.261969.61785.229122.4744
DTCWT0.48300.832652.186265.55214.93100.1799
GFF0.48500.844854.531165.90815.38360.2404
RP0.35820.546455.545670.04424.57440.1278
ASR0.46800.816451.534663.93704.156087.0228
DCNN0.46380.827960.447674.83794.525078.8741
CSMCA0.49400.844453.232267.48994.3896205.1055
SSID0.51220.842655.888870.37514.57380.0721
Proposed0.51510.849260.644375.12315.052418.5094
Table 5. Average objective assessment of different fusion methods on the nine groups gray medical images in Figure 6.
Table 5. Average objective assessment of different fusion methods on the nine groups gray medical images in Figure 6.
VIFFQWAPISDENTime/s
NSCT0.52100.776159.899665.10866.121823.7192
DTCWT0.51810.771354.418259.91315.78970.1778
GFF0.50950.781360.066662.80366.06360.2568
RP0.37010.575858.804664.23015.64150.1428
ASR0.48240.758453.692957.29585.3715106.4758
DCNN0.54390.767465.352873.72305.139080.0550
CSMCA0.54730.782256.859963.20755.4745199.1734
SSID0.59700.793466.251770.05405.65400.0848
Proposed0.61210.807270.536374.29155.968519.0577
Table 6. Objective assessment of different fusion methods on the first group anatomical and functional images.
Table 6. Objective assessment of different fusion methods on the first group anatomical and functional images.
VIFFQWAPISDENTime/s
NSCT0.26510.798643.136464.89964.764828.2017
DTCWT0.59010.825043.453362.99234.64930.1937
GFF0.18990.807533.874664.03594.40730.2377
RP0.84430.847145.867468.70584.72890.1570
ASR0.31500.760242.949661.12354.199785.6910
DCNN0.20160.804936.441263.07644.545180.3691
CSMCA0.30880.792644.441963.94664.5383193.1375
SSID0.36750.683753.545174.46864.67020.0806
Proposed0.39050.773757.731080.62454.916920.5294
Table 7. Average objective assessment of different fusion methods on the nine groups anatomical and functional images in Figure 7.
Table 7. Average objective assessment of different fusion methods on the nine groups anatomical and functional images in Figure 7.
VIFFQWAPISDENTime/s
NSCT0.50160.894639.888356.04954.710126.5087
DTCWT0.73960.903435.757350.12174.74620.2026
GFF0.49470.899538.814155.13864.65840.2475
RP0.62230.787838.440053.63704.65220.1562
ASR0.46880.834235.242148.28894.373692.3286
DCNN0.49520.893639.650756.79824.664179.5171
CSMCA0.38010.679829.890942.20794.1895186.4474
SSID0.54250.869041.108556.06594.66060.0828
Proposed0.54840.896843.711359.62734.884719.3064
Publisher’s Note: MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Share and Cite

MDPI and ACS Style

Li, L.; Ma, H. Pulse Coupled Neural Network-Based Multimodal Medical Image Fusion via Guided Filtering and WSEML in NSCT Domain. Entropy 2021, 23, 591. https://doi.org/10.3390/e23050591

AMA Style

Li L, Ma H. Pulse Coupled Neural Network-Based Multimodal Medical Image Fusion via Guided Filtering and WSEML in NSCT Domain. Entropy. 2021; 23(5):591. https://doi.org/10.3390/e23050591

Chicago/Turabian Style

Li, Liangliang, and Hongbing Ma. 2021. "Pulse Coupled Neural Network-Based Multimodal Medical Image Fusion via Guided Filtering and WSEML in NSCT Domain" Entropy 23, no. 5: 591. https://doi.org/10.3390/e23050591

Note that from the first issue of 2016, this journal uses article numbers instead of page numbers. See further details here.

Article Metrics

Back to TopTop