Combining Regional Energy and Intuitionistic Fuzzy Sets for Infrared and Visible Image Fusion
Abstract
:1. Introduction
2. Related Works
2.1. Basic Principle of NSST
2.2. Fuzzy Set Theory
3. Proposed Method
3.1. NSST Decomposition
3.2. The Rule for LowFrequency Components
 (1)
 The prefusion based on RE
 (2)
 The final fusion based on IFS
3.3. The Rule for HighFrequency Components
3.4. NSST Reconstruction
Algorithm 1. The proposed REIFSNSST fusion algorithm. 
Input: Infrared image (IR), Visible image (VIS) 
Out: Fused image (F). 

4. Experimental Results
4.1. Datasets
4.2. Experimental Setting
 (1)
 The computer is configured as 2.6 Hz Intel Core CPU and 4GB memory, and all experimental codes run on the Matlab2017 platform.
 (2)
 In the proposed method, the ‘maxflat’ is chosen as the pyramid filter. The numbers of decomposition level and directions are 3 and {16,16,16}, respectively.
 (3)
 In the RENSST and IFSNSST methods, the parameters of NSST are the same as that of the proposed method. The calculation of RE and IFS are the same as that of the proposed method.
 (4)
 In Bala and Gauss methods, the ‘97′ and ‘pkva’ are chosen as the pyramid filter and the directional filter respectively, and the decomposition scale is 3.
 (5)
 In the MDLatLRR method, the decomposition level selection 2.
 (6)
 The parameters of the other 9 methods are set following the best parameter setting reported in the corresponding papers.
4.3. Quantitative Evaluation
 (1)
 Entropy (E) [43]
 (2)
 Average Gradient (AG) [43]
 (3)
 Mutual Information (MI) [44]
 (4)
 Cross Entropy (CE) [44]
 (5)
 Spectral Distortion (SPD) [45]
 (6)
 Peak signal to noise ratio (PSNR) [44]
4.4. Fusion Results on the TNO Dataset
4.4.1. Comparison with RENSST and IFSNSST Methods
4.4.2. Comparison with the StateoftheArt Methods
4.4.3. Analysis
 (1)
 The proposed method can transfer more detailed textures features of shrub and tree to the resulting image.
 (2)
 The proposed method can preserve obvious infrared targets information in the resulting image.
 (3)
 The proposed method can improve the image contrast and brightness.
4.5. Fusion Results on the RoadScene Dataset
4.6. The Computational Complexity Analysis
5. Conclusions
Author Contributions
Funding
Institutional Review Board Statement
Informed Consent Statement
Data Availability Statement
Conflicts of Interest
References
 Hu, W.; Hu, H. Discriminant Deep Feature Learning based on joint supervision Loss and Multilayer Feature Fusion for heterogeneous face recognition. Comput. Vis. Image Underst. 2019, 184, 9–21. [Google Scholar] [CrossRef]
 Sun, H.; Liu, Q.; Wang, J.; Ren, J.; Wu, Y.; Zhao, H.; Li, H. Fusion of Infrared and Visible Images for Remote Detection of LowAltitude SlowSpeed Small Targets. IEEE J. Sel. Top. Appl. Earth Obs. Remote Sens. 2021, 14, 2971–2983. [Google Scholar] [CrossRef]
 Huang, H.; Dong, L.; Xue, Z.; Liu, X.; Hua, C. Fusion algorithm of visible and infrared image based on anisotropic diffusion and image enhancement. PLoS ONE 2021, 16, e0245563. [Google Scholar]
 Jose, J.; Gautam, N.; Tiwari, M.; Tiwari, T.; Suresh, A.; Sundararaj, V.; Mr, R. An image quality enhancement scheme employing adolescent identity search algorithm in the NSST domain for multimodal medical image fusion. Biomed. Signal Proces. Control 2021, 66, 102480. [Google Scholar] [CrossRef]
 Jin, X.; Huang, S.; Jiang, Q.; Lee, S.J.; Wu, L.; Yao, S. Semisupervised Remote Sensing Image Fusion Using Multiscale Conditional Generative Adversarial Network with Siamese Structure. IEEE J. Sel. Top. Appl. Earth Obs. Remote Sens. 2021, 14, 7066–7084. [Google Scholar] [CrossRef]
 Qi, G.; Chang, L.; Luo, Y.; Chen, Y.; Zhu, Z.; Wang, S. A Precise MultiExposure Image Fusion Method Based on Lowlevel Features. Sensors 2020, 20, 1597. [Google Scholar] [CrossRef] [PubMed] [Green Version]
 Liu, Y.; Dong, L.; Ji, Y.; Xu, W. Infrared and Visible Image Fusion through Details Preservation. Sensors 2019, 19, 4556. [Google Scholar] [CrossRef] [PubMed] [Green Version]
 Zhang, S.; Liu, F. Infrared and visible image fusion based on nonsubsampled shearlet transform, regional energy, and cooccurrence filtering. Electron. Lett. 2020, 56, 761–764. [Google Scholar] [CrossRef]
 Srivastava, R.; Prakash, O.; Khare, A. Local energybased multimodal medical image fusion in curvelet domain. IET Comput. Vis. 2016, 10, 513–527. [Google Scholar] [CrossRef]
 Liu, X.; Zhou, Y.; Wang, J. Image fusion based on shearlet transform and regional features. AEUInt. J. Electron. Commun. 2014, 68, 471–477. [Google Scholar] [CrossRef]
 Versaci, M.; Calcagno, S.; Morabito, F.C. Fuzzy Geometrical Approach Based on Unit HyperCubes for Image Contrast Enhancement. In Proceedings of the 2015 IEEE International Conference on Signal and Image Processing Applications (ICSIPA), Kuala Lumpur, Malaysia, 19–21 October 2015; pp. 488–493. [Google Scholar]
 Zhang, X.F.; Yan, H.; He, H. Multifocus image fusion based on fractionalorder derivative and intuitionistic fuzzy sets. Front. Inf. Technol. Electron. Eng. 2020, 21, 834–843. [Google Scholar] [CrossRef]
 Tirupal, T.; Mohan, B.C.; Kumar, S.S. Multimodal medical image fusion based on Sugeno’s intuitionistic fuzzy sets. ETRI J. 2017, 39, 173–180. [Google Scholar] [CrossRef]
 Seng, C.H.; Bouzerdoum, A.; Amin, M.G.; Phung, S.L. Probabilistic Fuzzy Image Fusion Approach for Radar Through Wall Sensing. IEEE Trans. Image Process. 2013, 22, 4938–4951. [Google Scholar] [CrossRef] [PubMed]
 Ma, J.; Ma, Y.; Li, C. Infrared and visible image fusion methods and applications: A survey. Inform. Fusion 2019, 45, 153–178. [Google Scholar] [CrossRef]
 Zhou, Z.H.; Tan, M. Infrared Image and Visible Image Fusion Based on Wavelet Transform. Adv. Mater. Res. 2013, 756–759, 2850–2856. [Google Scholar] [CrossRef]
 Kou, L.; Zhang, L.; Zhang, K.; Sun, J.; Han, Q.; Jin, Z. A multifocus image fusion method via region mosaicking on Laplacian pyramids. PLoS ONE 2018, 13, e0191085. [Google Scholar] [CrossRef] [PubMed]
 Guo, L.; Dai, M.; Zhu, M. Multifocus color image fusion based on quaternion curvelet transform. Opt. Express 2012, 20, 18846–18860. [Google Scholar] [CrossRef] [PubMed]
 Mosavi, M.R.; Bisjerdi, M.H.; RezaiRad, G. Optimal TargetOriented Fusion of Passive Millimeter Wave Images with Visible Images Based on Contourlet Transform. Wireless Pers. Commun. 2017, 95, 4643–4666. [Google Scholar] [CrossRef]
 Adu, J.; Gan, J.; Wang, Y.; Huang, J. Image fusion based on nonsubsampled contourlet transform for infrared and visible light image. Infrared Phys. Technol. 2013, 61, 94–100. [Google Scholar] [CrossRef]
 Fan, Z.; Bi, D.; Gao, S.; He, L.; Ding, W. Adaptive enhancement for infrared image using shearlet frame. J. Opt. 2016, 18, 085706. [Google Scholar] [CrossRef]
 Guorong, G.; Luping, X.; Dongzhu, F. Multifocus image fusion based on nonsubsampled shearlet transform. IET Image Process. 2013, 7, 633–639. [Google Scholar] [CrossRef]
 ElHoseny, H.M.; ElRahman, W.A.; ElShafai, W.; ElBanby, G.M.; ElRabaie, E.S.M.; Abd ElSamie, F.E.; Faragallah, O.S.; Mahmoud, K.R. Efficient multiscale nonsubsampled shearlet fusion system based on modified central force optimization and contrast enhancement. Infrared Phys. Technol. 2019, 102, 102975. [Google Scholar] [CrossRef]
 Kong, W.; Wang, B.; Lei, Y. Technique for infrared and visible image fusion based on nonsubsampled shearlet transform and spiking cortical model. Infrared Phys. Technol. 2015, 71, 87–98. [Google Scholar] [CrossRef]
 Guo, K.; Labate, D. Optimally Sparse Multidimensional Representation Using Shearlets. SIAM J. Math. Anal. 2007, 39, 298–318. [Google Scholar] [CrossRef] [Green Version]
 Easley, G.; Labate, D.; Lim, W.Q. Sparse directional image representations using the discrete shearlet transform. Appl. Comput. Harmon. Anal. 2008, 25, 25–46. [Google Scholar] [CrossRef] [Green Version]
 Deschrijver, G.; Kerre, E.E. On the relationship between some extensions of fuzzy set theory. Fuzzy Sets Syst. 2003, 133, 227–235. [Google Scholar] [CrossRef]
 Bai, X. Infrared and Visual Image Fusion through Fuzzy Measure and Alternating Operators. Sensors 2015, 15, 17149–17167. [Google Scholar] [CrossRef] [PubMed] [Green Version]
 Saeedi, J.; Faez, K. Infrared and visible image fusion using fuzzy logic and populationbased optimization. Appl. Soft Comput. 2012, 12, 1041–1054. [Google Scholar] [CrossRef]
 Pal, S.K.; King, R. Image enhancement using smoothing with fuzzy sets. IEEE Trans. Syst. Man Cybern. 1981, 11, 494–500. [Google Scholar]
 Selvaraj, A.; Ganesan, P. Infrared and visible image fusion using multiscale NSCT and rollingguidance filter. IET Image Process. 2020, 14, 4210–4219. [Google Scholar] [CrossRef]
 Yu, S.; Chen, X. Infrared and Visible Image Fusion Based on a Latent LowRank Representation Nested with Multiscale Geometric Transform. IEEE Access 2020, 8, 110214–110226. [Google Scholar] [CrossRef]
 Cai, H.; Zhuo, L.; Zhu, P.; Huang, Z.; Wu, X. Fusion of infrared and visible images based on nonsubsampled contourlet transform and intuitionistic fuzzy set. Acta Photon. Sin. 2018, 47, 610002. [Google Scholar] [CrossRef]
 Dai, Z.; Wang, Q. Research on fusion method of visible and infrared image based on PCNN and IFS. J. Optoelectron. Laser 2020, 31, 738–744. [Google Scholar]
 Bavirisetti, D.P.; Xiao, G.; Liu, G. In MultiSensor Image Fusion Based on Fourth Order Partial Differential Equations. In Proceedings of the 2017 20th International Conference on Information Fusion (Fusion), Xi’an, China, 10–13 July 2017; pp. 1–9. [Google Scholar]
 Ma, J.; Zhou, Z.; Wang, B.; Zong, H. Infrared and visible image fusion based on visual saliency map and weighted least square optimization. Infrared Phys. Technol. 2017, 82, 8–17. [Google Scholar] [CrossRef]
 Jingchao, Z.; Suzhen, L.; Dawei, L.; Lifang, W.; Xiaoli, Y. Comparative Study of Intuitionistic Fuzzy Sets in Multiband Image Fusion. Infrared Technol. 2018, 40, 881. [Google Scholar]
 Du, Q.; Xu, H.; Ma, Y.; Huang, J.; Fan, F. Fusing infrared and visible images of different resolutions via total variation model. Sensors 2018, 18, 3827. [Google Scholar] [CrossRef] [PubMed] [Green Version]
 Li, H.; Wu, X.J. Infrared and visible image fusion using latent lowrank representation. arXiv 2018, arXiv:1804.08992. [Google Scholar]
 Anantrasirichai, N.; Zheng, R.; Selesnick, I.; Achim, A. Image fusion via sparse regularization with nonconvex penalties. Pattern Recogn. Lett. 2020, 131, 355–360. [Google Scholar] [CrossRef] [Green Version]
 Li, H.; Wu, X.J.; Kittler, J. MDLatLRR: A novel decomposition method for infrared and visible image fusion. IEEE Trans. Image Process. 2020, 29, 4733–4746. [Google Scholar] [CrossRef] [PubMed] [Green Version]
 Li, H.; Wu, X.J.; Kittler, J. RFNNest: An endtoend residual fusion network for infrared and visible images. Inf. Fusion 2021, 73, 72–86. [Google Scholar] [CrossRef]
 Xing, X.X.; Liu, C.; Luo, C.; Xu, T. Infrared and visible image fusion based on nonlinear enhancement and NSST decomposition. EURASIP J. Wirel. Commun. Netw. 2020, 2020, 162. [Google Scholar] [CrossRef]
 Yang, Y.C.; Li, J.; Wang, Y.P. Review of image fusion quality evaluation methods. J. Front. Comput. Sci. Technol. 2018, 12, 1021–1035. [Google Scholar]
 Guo, Q.; Liu, S. Performance analysis of multispectral and panchromatic image fusion techniques based on two wavelet discrete approaches. Optik 2011, 122, 811–819. [Google Scholar] [CrossRef]
Pictures  Algorithm  E  AG  MI  CE  SPD  PSNR 

set2  RENSST  7.0231  7.8391  1.5491  0.4003  20.2137  18.6581 
IFSNSST  7.4985  8.2866  1.6744  0.4143  17.9667  17.4212  
Proposed  7.5003  8.1158  2.1463  0.3881  9.1239  21.3318  
set4  RENSST  6.6502  6.6127  1.3885  1.1583  27.0561  16.3921 
IFSNSST  7.1960  6.9817  1.5673  0.4401  23.0697  16.2754  
Proposed  7.2065  6.8724  1.7395  0.2678  16.6751  17.9727  
set5  RENSST  7.1367  5.0369  2.1408  0.8499  30.3678  14.9696 
IFSNSST  7.4193  5.2169  2.1355  0.7068  22.5062  15.6012  
Proposed  7.5317  5.2223  2.2287  0.4438  17.1070  16.7450  
set6  RENSST  6.7152  5.1251  2.3817  2.5125  30.7200  17.3706 
IFSNSST  7.1657  5.6524  2.7181  1.4545  23.0795  14.9469  
Proposed  7.1764  5.2818  3.0924  1.2185  10.6998  22.1216 
Algorithm  E  AG  MI  CE  SPD  PSNR 

FPDE  6.6385  5.1961  1.5334  1.1573  25.8568  17.8237 
VSM  6.5374  5.0431  1.1345  1.5274  30.3846  15.9714 
Bala  6.7515  2.5025  1.3897  0.6556  28.1342  16.8945 
Gauss  6.7573  3.3446  1.4076  1.5874  27.1725  17.3515 
DRTV  7.0767  4.9648  1.9273  0.8521  60.3667  10.0039 
LATLRR  6.6468  3.3042  1.1525  1.2658  31.8783  15.9835 
SR  6.6610  3.4351  1.7760  1.6768  27.7083  17.1956 
MDLatLRR  6.6913  3.9260  1.8946  0.4422  134.4224  5.3021 
RFN  6.6424  2.7265  1.1956  1.4660  30.4971  15.5273 
Proposed  7.1322  5.6427  2.0628  0.2455  14.5461  19.2055 
Images  FPDE  VSM  Bala  Gauss  DRTV  LATLRR  SR  MDLatLRR  RFN  Proposed 

set1  11.0281  2.1048  32.5587  33.2193  0.8448  105.9846  6.0897  150.6458  10.6317  3.1674 
set2  18.6852  3.6156  49.1194  49.3599  1.3805  111.3046  10.266  186.3398  11.4100  4.8280 
set3  10.2954  2.2599  31.4941  31.0160  0.8172  99.6353  5.9340  180.6707  11.9672  3.1971 
set4  1.7641  0.8080  22.3608  19.6283  0.2517  33.6849  1.7168  80.0596  12.7350  1.3497 
set5  10.8291  2.1046  32.4518  32.0341  0.8023  106.767  6.3531  161.4593  13.6248  3.1962 
Images  FPDE  VSM  Bala  Gauss  DRTV  LATLRR  SR  MDLatLRR  RFN  Proposed 

set1  5.3412  2.0977  36.0400  37.9651  0.4194  104.1814  3.1564  153.4153  9.7678  2.3402 
set2  2.6183  3.0216  17.0855  19.6853  0.2962  75.0036  1.7318  95.97211  10.8458  1.5268 
set3  7.6224  4.6006  24.3446  25.7651  0.5192  112.2796  3.1698  192.0942  11.7744  1.9909 
set4  2.7065  5.3109  14.0975  15.7018  0.2460  55.0571  1.5370  76.53124  12.1473  1.2707 
set5  6.1202  6.8466  23.5806  25.0931  0.4660  103.0710  3.0122  162.6771  13.0335  1.9509 
Publisher’s Note: MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations. 
© 2021 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https://creativecommons.org/licenses/by/4.0/).
Share and Cite
Xing, X.; Luo, C.; Zhou, J.; Yan, M.; Liu, C.; Xu, T. Combining Regional Energy and Intuitionistic Fuzzy Sets for Infrared and Visible Image Fusion. Sensors 2021, 21, 7813. https://doi.org/10.3390/s21237813
Xing X, Luo C, Zhou J, Yan M, Liu C, Xu T. Combining Regional Energy and Intuitionistic Fuzzy Sets for Infrared and Visible Image Fusion. Sensors. 2021; 21(23):7813. https://doi.org/10.3390/s21237813
Chicago/Turabian StyleXing, Xiaoxue, Cong Luo, Jian Zhou, Minghan Yan, Cheng Liu, and Tingfa Xu. 2021. "Combining Regional Energy and Intuitionistic Fuzzy Sets for Infrared and Visible Image Fusion" Sensors 21, no. 23: 7813. https://doi.org/10.3390/s21237813