RI-MFM: A Novel Infrared and Visible Image Registration with Rotation Invariance and Multilevel Feature Matching
Abstract
:1. Introduction
- We propose to generate feature descriptors using a concentric circle-based feature-description algorithm. When the feature descriptor is generated, the description of the main direction of the feature point is enhanced by introducing the centroid, and the rotation invariance is guaranteed by using the concentric circles. This produces more robust feature descriptors and reduces dimensionality, speeding up computation.
- We propose a multi-level feature-matching algorithm with improved offset consistency to match feature points. We use the length and angle of the connection between the correct matching points to be basically the same, and redesign the matching algorithm to achieve a better matching effect. Finally, the matching points with higher positioning accuracy can be obtained by iterative screening using the RANSAC algorithm.
- Experimental results of our proposed infrared and visible image registration method on public and homemade datasets show that it achieves higher localization accuracy and correct matching rate than existing state-of-the-art image registration methods.
2. Related Work
2.1. Region-Based Methods
2.2. Feature-Based Methods
3. Proposed Method
- Image pre-processing: For better robustness and generalization of the proposed registration algorithm, the input infrared and visible images should be pre-processed. First of all, if the device acquires color images, the color images need to be converted to grayscale images. Then, the infrared image is noise-reduced and dynamic range adjusted using an image-enhancement algorithm, which aims to enhance the edge and texture detail information of the image. Finally, the infrared and visible image resolutions are scaled to the same size.
- Feature extraction: At first, the Canny edge-detection algorithm is used to obtain the edge contour maps of infrared and visible images. Additionally, then, the SIFT algorithm is used to detect and localize the edge contour map. Finally, two sets of feature-point sets of infrared and visible images are obtained.
- Feature description based on concentric circles: After obtaining two sets of feature points of infrared and visible images, it is necessary to construct corresponding feature descriptors for each feature point. First, the centroid is calculated using the area-integration method. Then, based on the coordinates of the centroid and the feature point, the principal direction is calculated. Finally, the concentric-ring area is constructed to generate the feature-point descriptors.
- Multi-level feature matching based on improved offset consistency: The most important feature of feature matching is to search for the most correct matching-point pair in the two images according to the obtained feature descriptor and determine whether it is a feature point in the same scene. Firstly, coarse matching of feature points is performed using Euclidean distance. After that, the feature points are finely matched using an improved offset consistency judgment criterion to further eliminate the incorrect matches. Finally, in order to eliminate the incorrect matching points with insignificant differences among the candidate matching points, the RANSAC algorithm is used for iterative screening to obtain matching points with higher accuracy.
3.1. Image Preprocessing
3.2. Feature Detection
3.3. Feature Description
3.4. Feature Matching
4. Experimental Analysis
4.1. Dataset
4.2. Evaluation Metrics
- RMSEThe RMSE is to detect the positioning accuracy of feature points, and RMSE is defined as:To distinguish the location of each pair of matching points in the alignment results, is defined as the correct matching point. The rest are false match points.
- CMRTo quantitatively evaluate the matching accuracy of the matching method, the CMR evaluation index is introduced. CMR is defined as follows:
- ARTThe efficiency of each method is judged by counting the average running time of different infrared and visible image-registration methods on the same experimental platform.
4.3. Experiments on the CVC Dataset
4.3.1. Qualitative Evaluation
4.3.2. Quantitative Evaluation
4.4. Experiments on Homemade Datasets
4.4.1. Qualitative Evaluation
4.4.2. Quantitative Evaluation
5. Conclusions
Author Contributions
Funding
Institutional Review Board Statement
Informed Consent Statement
Data Availability Statement
Conflicts of Interest
References
- Li, H.; Ding, W.; Cao, X.; Liu, C. Image registration and fusion of visible and infrared integrated camera for medium-altitude unmanned aerial vehicle remote sensing. Remote Sens. 2017, 9, 441. [Google Scholar] [CrossRef]
- Dwith Chenna, Y.N.; Ghassemi, P.; Pfefer, T.J.; Casamento, J.; Wang, Q. Free-form deformation approach for registration of visible and infrared facial images in fever screening. Sensors 2018, 18, 125. [Google Scholar] [CrossRef] [PubMed]
- Dou, J.; Qin, Q.; Tu, Z. Multi-Modal Image Registration Based on Local Self-Similarity and Bidirectional Matching. Pattern Recognit. Image Anal. 2021, 31, 7–17. [Google Scholar] [CrossRef]
- Zhao, X.; Zhang, J.; Yang, C.; Song, H.; Shi, Y.; Zhou, X.; Zhang, D.; Zhang, G. Registration for optical multimodal remote sensing images based on FAST detection, window selection, and histogram specification. Remote Sens. 2018, 10, 663. [Google Scholar] [CrossRef]
- Zhu, D.; Zhan, W.; Jiang, Y.; Xu, X.; Guo, R. MIFFuse: A Multi-Level Feature Fusion Network for Infrared and Visible Images. IEEE Access 2021, 9, 130778–130792. [Google Scholar] [CrossRef]
- Xiao, R.; Cheng, F.; Shi, J.; Wang, Y.; Li, C. An Infrared and Visible Fusion Framework Based on a Novel Decomposition Method. Symmetry 2022, 14, 786. [Google Scholar] [CrossRef]
- Alonso, M.T.; López-Martínez, C.; Mallorquí, J.J.; Salembier, P. Edge enhancement algorithm based on the wavelet transform for automatic edge detection in SAR images. IEEE Trans. Geosci. Remote Sens. 2010, 49, 222–235. [Google Scholar] [CrossRef]
- Jiang, Y.; Liu, Y.; Zhan, W.; Zhu, D. Lightweight Dual-Stream Residual Network for Single Image Super-Resolution. IEEE Access 2021, 9, 129890–129901. [Google Scholar] [CrossRef]
- Chen, Y.; Zhang, X.; Zhang, Y.; Maybank, S.J.; Fu, Z. Visible and infrared image registration based on region features and edginess. Mach. Vis. Appl. 2018, 29, 113–123. [Google Scholar] [CrossRef]
- Ma, Y.; Wang, Y.; Mei, X.; Liu, C.; Dai, X.; Fan, F.; Huang, J. Visible/infrared combined 3D reconstruction scheme based on nonrigid registration of multi-modality images with mixed features. IEEE Access 2019, 7, 19199–19211. [Google Scholar] [CrossRef]
- Jhan, J.P.; Rau, J.Y. A generalized tool for accurate and efficient image registration of UAV multi-lens multispectral cameras by N-SURF matching. IEEE J. Sel. Top. Appl. Earth Obs. Remote Sens. 2021, 14, 6353–6362. [Google Scholar] [CrossRef]
- Inglada, J.; Giros, A. On the possibility of automatic multisensor image registration. IEEE Trans. Geosci. Remote Sens. 2004, 42, 2104–2120. [Google Scholar] [CrossRef]
- Woo, J.; Stone, M.; Prince, J.L. Multimodal registration via mutual information incorporating geometric and spatial context. IEEE Trans. Image Process. 2014, 24, 757–769. [Google Scholar] [CrossRef]
- Yang, T.; Tang, Q.; Li, L.; Song, J.; Zhu, C.; Tang, L. Nonrigid registration of medical image based on adaptive local structure tensor and normalized mutual information. J. Appl. Clin. Med. Phys. 2019, 20, 99–110. [Google Scholar] [CrossRef]
- Zhang, J.; Wang, Y.; Dai, J.; Cavichini, M.; Bartsch, D.U.G.; Freeman, W.R.; Nguyen, T.Q.; An, C. Two-Step Registration on Multi-Modal Retinal Images via Deep Neural Networks. IEEE Trans. Image Process. 2021, 31, 823–838. [Google Scholar] [CrossRef]
- Bay, H.; Tuytelaars, T.; Gool, L.V. SURF: Speeded up robust features. In Proceedings of the European Conference on Computer Vision, Graz, Austria, 7–13 May 2006; pp. 404–417. [Google Scholar]
- Lowe, D.G. Object recognition from local scale-invariant features. Proc. Seventh IEEE Int. Conf. Comput. Vis. 1999, 2, 1150–1157. [Google Scholar]
- Zeng, Q.; Adu, J.; Liu, J.; Yang, J.; Xu, Y.; Gong, M. Real-time adaptive visible and infrared image registration based on morphological gradient and C_SIFT. J. Real-Time Image Process. 2020, 17, 1103–1115. [Google Scholar] [CrossRef]
- Xiong, X.; Xu, Q.; Jin, G.; Zhang, H.; Gao, X. Rank-based local self-similarity descriptor for optical-to-SAR image matching. IEEE Geosci. Remote Sens. Lett. 2019, 17, 1742–1746. [Google Scholar] [CrossRef]
- Fischler, M.A.; Bolles, R.C. Random sample consensus: A paradigm for model fitting with applications to image analysis and automated cartography. Commun. ACM 1981, 24, 381–395. [Google Scholar] [CrossRef]
- Canny, J. A computational approach to edge detection. In Proceedings of the IEEE Transactions on Pattern Analysis and Machine Intelligence, Kerkyra, Greece, 20–27 September 1986; pp. 679–698. [Google Scholar]
- Chen, J.; Cheng, B.; Zhang, X.; Long, T.; Chen, B.; Wang, G.; Zhang, D. A TIR-Visible Automatic Registration and Geometric Correction Method for SDGSAT-1 Thermal Infrared Image Based on Modified RIFT. Remote Sens. 2022, 14, 1393. [Google Scholar] [CrossRef]
- Koz, A.; Efe, U. Geometric-and Optimization-Based Registration Methods for Long-Wave Infrared Hyperspectral Images. Remote Sens. 2021, 13, 2465. [Google Scholar] [CrossRef]
- Zhu, D.; Zhan, W.; Jiang, Y.; Xu, X.; Guo, R. IPLF: A Novel Image Pair Learning Fusion Network for Infrared and Visible Image. IEEE Sens. J. 2022, 22, 8808–8817. [Google Scholar] [CrossRef]
- Son, D.M.; Kwon, H.J.; Lee, S.H. Visible and Near Infrared Image Fusion Using Base Tone Compression and Detail Transform Fusion. Chemosensors 2022, 10, 124. [Google Scholar] [CrossRef]
- Xiao, W.; Zhang, Y.; Wang, H.; Li, F.; Jin, H. Heterogeneous Knowledge Distillation for Simultaneous Infrared-Visible Image Fusion and Super-Resolution. IEEE Trans. Instrum. Meas. 2022, 71, 1–15. [Google Scholar] [CrossRef]
- Li, Q.; Lu, L.; Li, Z.; Wu, W.; Liu, Z.; Jeon, G.; Yang, X. Coupled GAN with relativistic discriminators for infrared and visible images fusion. IEEE Sens. J. 2019, 21, 7458–7467. [Google Scholar] [CrossRef]
- Xie, X.; Zhang, Y.; Ling, X.; Wang, X. A new registration algorithm for multimodal remote sensing images. In Proceedings of the IGARSS 2018-2018 IEEE International Geoscience and Remote Sensing Symposium, Valencia, Spain, 22–27 July 2018; pp. 7011–7014. [Google Scholar]
- Zabalza, J.; Ren, J.; Zheng, J.; Zhao, H.; Qing, C.; Yang, Z.; Du, P.; Marshall, S. Novel segmented stacked autoencoder for effective dimensionality reduction and feature extraction in hyperspectral imaging. Neurocomputing 2016, 185, 1–10. [Google Scholar] [CrossRef]
- Loncomilla, P.; Ruiz-del Solar, J.; Martínez, L. Object recognition using local invariant features for robotic applications: A survey. Pattern Recognit. 2016, 60, 499–514. [Google Scholar] [CrossRef]
- Fan, J.; Wu, Y.; Li, M.; Liang, W.; Cao, Y. SAR and optical image registration using nonlinear diffusion and phase congruency structural descriptor. IEEE Trans. Geosci. Remote Sens. 2018, 56, 5368–5379. [Google Scholar] [CrossRef]
- Sotiras, A.; Davatzikos, C.; Paragios, N. Deformable medical image registration: A survey. IEEE Trans. Med. Imaging 2013, 32, 1153–1190. [Google Scholar] [CrossRef]
- Barnea, D.I.; Silverman, H.F. A class of algorithms for fast digital image registration. IEEE Trans. Comput. 1972, 100, 179–186. [Google Scholar] [CrossRef]
- Chen, S.; Li, X.; Zhao, L.; Yang, H. Medium-low resolution multisource remote sensing image registration based on SIFT and robust regional mutual information. Int. J. Remote Sens. 2018, 39, 3215–3242. [Google Scholar] [CrossRef]
- Wu, Y.; Ma, W.; Su, Q.; Liu, S.; Ge, Y. Remote sensing image registration based on local structural information and global constraint. J. Appl. Remote Sens. 2019, 13, 016518. [Google Scholar] [CrossRef]
- Wolberg, G.; Zokai, S. Image registration for perspective deformation recovery. Autom. Target Recognit. X 2000, 4050, 259–270. [Google Scholar]
- Cao, S.Y.; Shen, H.L.; Chen, S.J.; Li, C. Boosting structure consistency for multispectral and multimodal image registration. IEEE Trans. Image Process. 2020, 29, 5147–5162. [Google Scholar] [CrossRef]
- Cole-Rhodes, A.A.; Johnson, K.L.; LeMoigne, J.; Zavorin, I. Multiresolution registration of remote sensing imagery by optimization of mutual information using a stochastic gradient. IEEE Trans. Image Process. 2003, 12, 1495–1511. [Google Scholar] [CrossRef]
- Xu, X.; Li, X.; Liu, X.; Shen, H.; Shi, Q. Multimodal registration of remotely sensed images based on Jeffrey’s divergence. ISPRS J. Photogramm. Remote Sens. 2016, 122, 97–115. [Google Scholar] [CrossRef]
- Glocker, B.; Komodakis, N.; Paragios, N.; Navab, N. Approximated curvature penalty in non-rigid registration using pairwise mrfs. In Proceedings of the International Symposium on Visual Computing, Las Vegas, NV, USA, 30 November–2 December 2009; pp. 1101–1109. [Google Scholar]
- Hasan, M.; Pickering, M.R.; Jia, X. Robust automatic registration of multimodal satellite images using CCRE with partial volume interpolation. IEEE Trans. Geosci. Remote Sens. 2012, 50, 4050–4061. [Google Scholar] [CrossRef]
- Yan, X.; Zhang, Y.; Zhang, D.; Hou, N.; Zhang, B. Registration of multimodal remote sensing images using transfer optimization. IEEE Geosci. Remote Sens. Lett. 2020, 17, 2060–2064. [Google Scholar] [CrossRef]
- Kori, A.; Krishnamurthi, G. Zero shot learning for multi-modal real time image registration. arXiv 2019, arXiv:1908.06213. [Google Scholar]
- Moravec, H.P. Obstacle Avoidance and Navigation in the Real World by a Seeing Robot Rover. Ph.D. Thesis, Stanford University, Stanford, CA, USA, 1980. [Google Scholar]
- Yang, W.; Wang, X.; Moran, B.; Wheaton, A.; Cooley, N. Efficient registration of optical and infrared images via modified Sobel edging for plant canopy temperature estimation. Comput. Electr. Eng. 2012, 38, 1213–1221. [Google Scholar] [CrossRef]
- Chen, X.; Liu, L.; Zhang, J.; Shao, W. Registration of multimodal images with edge features and scale invariant PIIFD. Infrared Phys. Technol. 2020, 111, 103549. [Google Scholar] [CrossRef]
- Barath, D.; Noskova, J.; Ivashechkin, M.; Matas, J. MAGSAC++, a fast, reliable and accurate robust estimator. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Seattle, WA, USA, 13–19 June 2020; pp. 1304–1312. [Google Scholar]
- Liu, X.; Liu, Y.; Zhaorong, L. Application of Maximum a Posteriori Algorithm in Remote Sensing Image Reconstruction. Acta Opt. Sin. 2013, 33, 206–212. [Google Scholar]
- Kallel, F.; Hamida, A.B. A new adaptive gamma correction based algorithm using DWT-SVD for non-contrast CT image enhancement. IEEE Trans. Nanobiosci. 2017, 16, 666–675. [Google Scholar] [CrossRef] [PubMed]
- Sun, Z.; Han, B.; Li, J.; Zhang, J.; Gao, X. Weighted guided image filtering with steering kernel. IEEE Trans. Image Process. 2019, 29, 500–508. [Google Scholar] [CrossRef] [PubMed]
- Tang, C.; Tian, G.Y.; Chen, X.; Wu, J.; Li, K.; Meng, H. Infrared and visible images registration with adaptable local-global feature integration for rail inspection. Infrared Phys. Technol. 2017, 87, 31–39. [Google Scholar] [CrossRef]
- Ma, J.; Zhao, J.; Jiang, J.; Zhou, H.; Guo, X. Locality preserving matching. Int. J. Comput. Vis. 2019, 127, 512–531. [Google Scholar] [CrossRef]
- Du, Q.; Fan, A.; Ma, Y.; Fan, F.; Huang, J.; Mei, X. Infrared and visible image registration based on scale-invariant PIIFD feature and locality preserving matching. IEEE Access 2018, 6, 64107–64121. [Google Scholar] [CrossRef]
- Aguilera, C.; Barrera, F.; Lumbreras, F.; Sappa, A.D.; Toledo, R. Multispectral image feature points. Sensors 2012, 12, 12661–12672. [Google Scholar] [CrossRef]
- Zhang, J.; Ma, W.; Wu, Y.; Jiao, L. Multimodal remote sensing image registration based on image transfer and local features. IEEE Geosci. Remote Sens. Lett. 2019, 16, 1210–1214. [Google Scholar] [CrossRef]
- Yu, K.; Zheng, X.; Fang, B.; An, P.; Huang, X.; Luo, W.; Ding, J.; Wang, Z.; Ma, J. Multimodal Urban Remote Sensing Image Registration Via Roadcross Triangular Feature. IEEE J. Sel. Top. Appl. Earth Obs. Remote Sens. 2021, 14, 4441–4451. [Google Scholar] [CrossRef]
- Li, Y.; Stevenson, R.L. Multimodal image registration with line segments by selective search. IEEE Trans. Cybern. 2016, 47, 1285–1298. [Google Scholar] [CrossRef]
Methods | EG-SURF | LPM | SI-PIIFD | Ours |
---|---|---|---|---|
ART(s) | 5.5982 | 16.2643 | 15.2845 | 18.5826 |
Methods | EG-SURF | LPM | SI-PIIFD | Ours |
---|---|---|---|---|
ART(s) | 3.6854 | 14.2961 | 12.5884 | 13.2691 |
Publisher’s Note: MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations. |
© 2022 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https://creativecommons.org/licenses/by/4.0/).
Share and Cite
Zhu, D.; Zhan, W.; Fu, J.; Jiang, Y.; Xu, X.; Guo, R.; Chen, Y. RI-MFM: A Novel Infrared and Visible Image Registration with Rotation Invariance and Multilevel Feature Matching. Electronics 2022, 11, 2866. https://doi.org/10.3390/electronics11182866
Zhu D, Zhan W, Fu J, Jiang Y, Xu X, Guo R, Chen Y. RI-MFM: A Novel Infrared and Visible Image Registration with Rotation Invariance and Multilevel Feature Matching. Electronics. 2022; 11(18):2866. https://doi.org/10.3390/electronics11182866
Chicago/Turabian StyleZhu, Depeng, Weida Zhan, Jingqi Fu, Yichun Jiang, Xiaoyu Xu, Renzhong Guo, and Yu Chen. 2022. "RI-MFM: A Novel Infrared and Visible Image Registration with Rotation Invariance and Multilevel Feature Matching" Electronics 11, no. 18: 2866. https://doi.org/10.3390/electronics11182866