Real-Time Video Stitching for Mine Surveillance Using a Hybrid Image Registration Method
Abstract
:1. Introduction
2. Materials and Methods
2.1. Image Preprocessing for Defogging
2.2. Hybrid Detection Method Based on Moravec Optimization
2.3. Feature Point Matching Using KD Tree
2.4. Image Registration
3. Experiment
3.1. Dataset
3.2. Evaluation
3.3. Results
3.4. Discussion
4. Conclusions
Author Contributions
Funding
Conflicts of Interest
References
- IEA. Coal Information 2019; IEA: Paris, France, 2019; Available online: https://www.iea.org/reports/coal-information-2019 (accessed on 6 August 2020).
- Wang, X.; Meng, F. Statistical analysis of large accidents in China’s coal mines in 2016. Nat. Hazards 2018, 92, 311–325. [Google Scholar] [CrossRef]
- Jo, B.W.; Khan, R.M.A. An Event Reporting and Early-Warning Safety System Based on the Internet of Things for Underground Coal Mines: A Case Study. Appl. Sci. 2017, 7, 925. [Google Scholar]
- Zhang, F.; Xu, Z.; Chen, W.; Zhang, Z.; Zhong, H.; Luan, J.; Li, C. An Image Compression Method for Video Surveillance System in Underground Mines Based on Residual Networks and Discrete Wavelet Transform. Electronics 2019, 8, 1559. [Google Scholar] [CrossRef] [Green Version]
- Wei, W.; Xia, X.; Wozniak, M.; Fan, X.; Damaševičius, R.; Li, Y. Multi-sink distributed power control algorithm for cyber-physical-systems in coal mine tunnels. Comput. Netw. 2019, 161, 210–219. [Google Scholar] [CrossRef]
- Singh, A.; Kumar, D.; Hötzel, J. IoT based information and communication system for enhancing underground mines safety and productivity: Genesis, taxonomy and open issues. Ad Hoc Netw. 2018, 78, 115–129. [Google Scholar] [CrossRef]
- Kumar, D. Application of modern tools and techniques for mine safety disaster management. J. Inst. Eng. Ser. D 2016, 97, 77–85. [Google Scholar] [CrossRef]
- Dong, G.; Wei, W.; Xia, X.; Woźniak, M.; Damaševičius, R. Safety risk assessment of a pb-zn mine based on fuzzy-grey correlation analysis. Electronics 2020, 9, 130. [Google Scholar] [CrossRef] [Green Version]
- Zhou, B.; Duan, X.; Ye, D.; Wei, W.; Woźniak, M.; Damaševičius, R. Heterogeneous image matching via a novel feature describing model. Appl. Sci. 2019, 9, 4792. [Google Scholar] [CrossRef] [Green Version]
- Zhou, B.; Duan, X.; Wei, W.; Ye, D.; Wozniak, M.; Damasevicius, R. An adaptive local descriptor embedding Zernike moments for image matching. IEEE Access 2019, 7, 183971–183984. [Google Scholar] [CrossRef]
- Wei, L.; Zhong, Z.; Lang, C.; Yi, Z. A survey on image and video stitching. Virtual Real. Intell. Hardw. 2019, 1, 55–83. [Google Scholar] [CrossRef]
- He, B.; Yu, S. Parallax-Robust Surveillance Video Stitching. Sensors 2016, 16, 7. [Google Scholar] [CrossRef] [PubMed]
- Zhu, M.; Wang, W.; Liu, B.; Huang, J. Efficient Video Panoramic Image Stitching Based on an Improved Selection of Harris Corners and a Multiple-Constraint Corner Matching. PLoS ONE 2013, 8, e81182. [Google Scholar] [CrossRef] [PubMed]
- Zhou, B.; Duan, X.; Ye, D.; Wei, W.; Woźniak, M.; Połap, D.; Damaševičius, R. Multi-level features extraction for discontinuous target tracking in remote sensing image monitoring. Sensors 2019, 19, 4855. [Google Scholar] [CrossRef] [PubMed] [Green Version]
- Chen, X.; Ma, H.; Wan, J.; Li, B.; Xia, T. Multi-view 3D Object Detection Network for Autonomous Driving. In Proceedings of the 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Honolulu, HI, USA, 21–26 July 2017. [Google Scholar] [CrossRef] [Green Version]
- Bonny, M.Z.; Uddin, M.S. Feature-based image stitching algorithms. In Proceedings of the 2016 International Workshop on Computational Intelligence (IWCI), Dhaka, Bangladesh, 12–13 December 2016. [Google Scholar] [CrossRef]
- Adel, E.; Elmogy, M.; Elbakry, H. Image Stitching based on Feature Extraction Techniques: A Survey. Int. J. Comput. Appl. 2014, 99, 1–8. [Google Scholar] [CrossRef]
- Zhu, H.; Zou, K.; Li, Y.; Cen, M.; Mihaylova, L. Robust Non-Rigid Feature Matching for Image Registration Using Geometry Preserving. Sensors 2019, 19, 2729. [Google Scholar] [CrossRef] [Green Version]
- Alomran, M.; Chai, D. Feature-based panoramic image stitching. In Proceedings of the 14th International Conference on Control, Automation, Robotics and Vision (ICARCV), Phuket, Thailand, 13–15 November 2016. [Google Scholar] [CrossRef]
- Ho, T.; Schizas, I.D.; Rao, K.R.; Budagavi, M. 360-degree video stitching for dual-fisheye lens cameras based on rigid moving least squares. In Proceedings of the 2017 IEEE International Conference on Image Processing (ICIP), Beijing, China, 17–20 September 2017. [Google Scholar] [CrossRef] [Green Version]
- Yeh, S.-H.; Lai, S.-H. Real-time video stitching. In Proceedings of the 2017 IEEE International Conference on Image Processing (ICIP), Honolulu, HI, USA, 21–26 July 2017. [Google Scholar] [CrossRef] [Green Version]
- Babu, V.M.M.; Santha, T. Efficient brightness adaptive deep-sea image stitching using biorthogonal multi-wavelet transform and harris algorithm. In Proceedings of the 2017 International Conference on Intelligent Computing and Control (I2C2), Coimbatore, India, 23–24 June 2017. [Google Scholar] [CrossRef]
- Ruan, J.; Xie, L.; Ruan, Y.; Liu, L.; Chen, Q.; Zhang, Q. Image Stitching Algorithm Based on SURF and Wavelet Transform. In Proceedings of the 7th International Conference on Digital Home (ICDH), Guilin, China, 30 November–1 December 2018. [Google Scholar] [CrossRef]
- Aung, N.L.; Victor, D.K.; Ye, K.Z.; Htet, Z.W. The study of the process of stitching video images in real time. In Proceedings of the IEEE Conference of Russian Young Researchers in Electrical and Electronic Engineering (EIConRus), Moscow, St. Petersburg, Russia, 29 January–1 February 2018. [Google Scholar] [CrossRef]
- Lu, Y.; Gao, K.; Zhang, T.; Xu, T. A novel image registration approach via combining local features and geometric invariants. PLoS ONE 2018, 13, e0190383. [Google Scholar] [CrossRef] [Green Version]
- Nie, Y.; Su, T.; Zhang, Z.; Sun, H.; Li, G. Dynamic Video Stitching via Shakiness Removing. IEEE Trans. Image Process. 2018, 27, 164–178. [Google Scholar] [CrossRef]
- Fang, F.; Wang, T.; Fang, Y.; Zhang, G. Fast Color Blending for Seamless Image Stitching. IEEE Geosci. Remote Sens. Lett. 2019, 16, 1115–1119. [Google Scholar] [CrossRef]
- Lee, K.; Sim, J. Stitching for multi-view videos with large parallax based on adaptive pixel warping. IEEE Access 2018, 6, 26904–26917. [Google Scholar] [CrossRef]
- Yang, X.; Liu, Z.; Qiao, H.; Su, J.; Ji, D.; Zang, A.; Huang, H. Graph-based registration and blending for undersea image stitching. Robotica 2020, 38, 396–409. [Google Scholar] [CrossRef]
- Park, K.; Shim, Y.; Lee, M.; Ahn, H. Multi-frame based homography estimation for video stitching in static camera environments. Sensors 2020, 20, 92. [Google Scholar] [CrossRef] [PubMed] [Green Version]
- Zheng, J.; Wang, D.; Geng, Z. Coal Mine Video Data Detail Enhancement Algorithm Based on L0 Norm and Low Rank Analysis. Eur. J. Electr. Eng. 2019, 21, 55–60. [Google Scholar] [CrossRef] [Green Version]
- Li, D.; Qian, J.; Liu, Z.; Yang, P. Stitching technology of coal mine video with complex environment. J. China Coal Soc. 2011, 36, 878–884. [Google Scholar]
- Xu, M. Comparison and research of fisheye image correction algorithms in coal mine survey. Iop Conf. Ser. Earth Environ. Sci. 2019, 300. [Google Scholar] [CrossRef]
- Kim, B.; Choi, K.; Park, W.; Kim, S.; Ko, S. Content-preserving video stitching method for multi-camera systems. IEEE Trans. Consum. Electron. 2017, 63, 109–116. [Google Scholar] [CrossRef]
- Liu, Q.; Su, X.; Zhang, L.; Huang, H. Panoramic video stitching of dual cameras based on spatio-temporal seam optimization. Multimed. Tools Appl. 2018. [Google Scholar] [CrossRef]
- Li, J.; Wang, Z.; Lai, S.; Zhai, Y.; Zhang, M. Parallax-tolerant image stitching based on robust elastic warping. IEEE Trans. Multimed. 2018, 20, 1672–1687. [Google Scholar] [CrossRef]
- Chen, J.; Wan, Q.; Luo, L.; Wang, Y.; Luo, D. Drone image stitching based on compactly supported radial basis function. IEEE J. Sel. Top. Appl. Earth Obs. Remote Sens. 2019, 12, 4634–4643. [Google Scholar] [CrossRef]
- Kang, L.; Wei, Y.; Jiang, J.; Xie, Y. Robust cylindrical panorama stitching for low-texture scenes based on image alignment using deep learning and iterative optimization. Sensors 2019, 19, 5310. [Google Scholar] [CrossRef] [Green Version]
- Kang, J.; Kim, J.; Lee, I.; Kim, K. Minimum Error Seam-Based Efficient Panorama Video Stitching Method Robust to Parallax. IEEE Access 2019, 7, 167127–167140. [Google Scholar] [CrossRef]
- Krishnakumar, K.; Gandhi, S.I. Video stitching using interacting multiple model based feature tracking. Multimed. Tools Appl. 2019, 78, 1375–1397. [Google Scholar] [CrossRef]
- Krishnakumar, K.; Indira Gandhi, S. Video stitching based on multi-view spatiotemporal feature points and grid-based matching. Visual Comput. 2019. [Google Scholar] [CrossRef]
- Lin, L.; Ding, Y.; Wang, L.; Zhang, M.; Li, D. Line-preserving video stitching for asymmetric cameras. Multimed. Tools Appl. 2019, 78, 14591–14611. [Google Scholar] [CrossRef]
- Du, C.; Yuan, J.; Dong, J.; Li, L.; Chen, M.; Li, T. GPU based parallel optimization for real time panoramic video stitching. Pattern Recognit. Lett. 2020, 133, 62–69. [Google Scholar] [CrossRef] [Green Version]
- Kakli, M.U.; Cho, Y.; Seo, J. Parallax-tolerant video stitching with moving foregrounds. In Asian Conference on Pattern Recognition, ACPR 2019; Lecture Notes in Computer Science; Springer: Cham, Switzerland, 2020; Volume 12047, pp. 625–639. [Google Scholar] [CrossRef]
- Nazaré, T.S.; da Costa, G.B.P.; Contato, W.A.; Ponti, M. Deep Convolutional Neural Networks and Noisy Images. In Progress in Pattern Recognition, Image Analysis, Computer Vision, and Applications; Springer: Cham, Switzerland, 2018; pp. 416–424. [Google Scholar] [CrossRef]
- Algan, G.; Ulusoy, I. Image Classification with Deep Learning in the Presence of Noisy Labels: A Survey. arXiv 2019, arXiv:1912.05170. [Google Scholar]
- Moravec, H.P. Towards Automatic Visual Obstacle Avoidance. In Proceedings of the 5th International Joint Conference on Artificial Intelligence, Cambridge, MA, USA, 22 August 1977; p. 584. [Google Scholar]
- Lowe, D.G. Object recognition from local scale-invariant features. In Proceedings of the International Conference on Computer Vision, 2, Kerkyra, Corfu, Greece, 20–25 September 1999; pp. 1150–1157. [Google Scholar] [CrossRef]
- Dongmei, W.; Siqi, Z. Research on image enhancement algorithm of coal mine dust. In Proceedings of the 2018 International Conference on Sensor Networks and Signal Processing, SNSP 2018, Xi’an, China, 28–31 October 2018; pp. 261–265. [Google Scholar]
- He, K.; Sun, J.; Tang, X. Single Image Haze Removal Using Dark Channel Prior. IEEE Trans. Pattern Anal. Mach. Intell. 2011, 33, 2341–2353. [Google Scholar] [CrossRef]
- Liu, S.; Rahman, M.A.; Wong, C.Y.; Lin, S.C.F.; Jiang, G.; Kwok, N. Dark channel prior based image de-hazing: A review. In Proceedings of the 5th International Conference on Information Science and Technology (ICIST), Istanbul, Turkey, 21–23 March 2015. [Google Scholar] [CrossRef]
- Lee, S.; Yun, S.; Nam, J.-H.; Won, C.S.; Jung, S.-W. A review on dark channel prior based image dehazing algorithms. J. Image Video Proc. 2016, 4. [Google Scholar] [CrossRef] [Green Version]
- Li, Y.; You, S.; Brown, M.S.; Tan, R.T. Haze visibility enhancement: A Survey and quantitative benchmarking. Comput. Vis. Image Underst. 2017, 165, 1–16. [Google Scholar] [CrossRef] [Green Version]
- Fischler, M.A.; Bolles, R.C. Random Sample Consensus: A Paradigm for Model Fitting with Applications to Image Analysis and Automated Cartography. Commun. ACM 1981, 24, 381–395. [Google Scholar] [CrossRef]
- Chaiyasarn, K.; Kim, T.-K.; Viola, F.; Cipolla, R.; Soga, K. Distortion-Free Image Mosaicing for Tunnel Inspection Based on Robust Cylindrical Surface Estimation through Structure from Motion. J. Comput. Civ. Eng. 2016, 30, 04015045. [Google Scholar] [CrossRef]
- Deng, F.; Yang, J. Panoramic Image Generation Using Centerline- Constrained Mesh Parameterization for Arbitrarily Shaped Tunnel Lining. IEEE Access 2020, 8, 7969–7980. [Google Scholar] [CrossRef]
- Guo, H.; Liu, S.; He, T.; Zhu, S.; Zeng, B.; Gabbouj, M. Joint video stitching and stabilization from moving cameras. IEEE Trans. Image Process 2016, 25, 5491–5503. [Google Scholar] [CrossRef] [PubMed]
- Nabil, S.; Balzarini, R.; Devernay, F.; Crowley, J. Designing Objective Quality Metrics for Panoramic Videos based on Human Perception. In Proceedings of the Irish Machine Vision and Image Processing Conference, IMVIP 2018, Ulster, UK, 29–31 August 2018; pp. 189–192. [Google Scholar]
- Yoon, J.; Lee, D. Real-Time Video Stitching Using Camera Path Estimation and Homography Refinement. Symmetry 2018, 10, 4. [Google Scholar] [CrossRef] [Green Version]
- Wang, Z.; Bovik, A.C.; Sheikh, H.R.; Simoncelli, E.P. Image quality assessment: From error visibility to structural similarity. IEEE Trans. Image Process. 2004, 13, 600–612. [Google Scholar] [CrossRef] [PubMed] [Green Version]
- Zhao, C.; Zhang, H.; Chen, J.; Fu, W. Region-based parallax-tolerant image stitching. In Proceedings of the Tenth International Conference on Graphics and Image Processing (ICGIP 2018), Chengdu, China, 12–14 December 2018. [Google Scholar] [CrossRef]
- Zhu, Z.-H.; Fu, J.-Y.; Yang, J.-S.; Zhang, X.-M. Panoramic Image Stitching for Arbitrarily Shaped Tunnel Lining Inspection. Comput. Aided Civ. Infrastruct. Eng. 2016, 31, 936–953. [Google Scholar] [CrossRef]
- Huang, H.; Li, Q.; Zhang, D. Deep learning based image recognition for crack and leakage defects of metro shield tunnel. Tunn. Undergr. Space Technol. 2018, 77, 166–176. [Google Scholar] [CrossRef]
- Kim, C.N.; Kawamura, K.; Shiozaki, M.; Tarighat, A. An image-matching method based on the curvature of cost curve for producing tunnel lining panorama. J. JSCE 2018, 6, 78–90. [Google Scholar] [CrossRef]
- Konishi, S.; Imaizumi, N.; Enokidani, Y.; Nagaya, J.; Machijima, Y.; Akutagawa, S.; Murakoshi, K. Effective water leakage detection by using an innovative optic fiber sensing for aged concrete lining of urban metro lines in Tokyo. In Proceedings of the Tunnels and Underground Cities: Engineering and Innovation meet Archaeology, Architecture and Art, Naples, Italy, 3–9 May 2019; CRC Press: Boca Raton, FL, USA, 2019; pp. 2383–2392. [Google Scholar] [CrossRef]
- Zhao, S.; Zhang, D.M.; Huang, H.W. Deep learning–based image instance segmentation for moisture marks of shield tunnel lining. Tunn. Undergr. Space Technol. 2020, 95, 103156. [Google Scholar] [CrossRef]
- Liu, Y.; Song, J. Using the internet of things technology constructing digital mine. In Procedia Environmental Sciences, 10(PART B); Elsevier BV: Amsterdam, The Netherlands, 2011; pp. 1104–1108. [Google Scholar] [CrossRef] [Green Version]
- Wang, J.; Wang, Y.; Fu, J. Crucial technology research and demonstration of digital mines. J. China Coal Soc. 2016, 41, 1323–1331. [Google Scholar] [CrossRef]
Standard Deviation | Gradient Mean | Information Entropy | |
---|---|---|---|
Original images | 32.1948 | 1.7294 | 6.7359 |
Defogged images | 32.8739 | 2.4861 | 7.2056 |
Video Dataset Number | #1 Tytyri | #2 Sonderhausen | #3 Târgu Ocna | #4 Xiaoyadaokou |
---|---|---|---|---|
Resolution | 540p | 720p | 540p | 540p |
Number of frames | 280 | 450 | 300 | 120 |
Processing time (s) | 13.03 | 21.02 | 14.26 | 5.58 |
Stability score | 0.638 | 0.529 | 0.482 | 0.572 |
Stitching score | 1.02 | 1.11 | 0.89 | 0.94 |
Trajectory stability score | 1.25 | 1.13 | 1.32 | 1.18 |
MVSQA | 0.24 | 0.29 | 0.31 | 0.28 |
© 2020 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (http://creativecommons.org/licenses/by/4.0/).
Share and Cite
Bai, Z.; Li, Y.; Chen, X.; Yi, T.; Wei, W.; Wozniak, M.; Damasevicius, R. Real-Time Video Stitching for Mine Surveillance Using a Hybrid Image Registration Method. Electronics 2020, 9, 1336. https://doi.org/10.3390/electronics9091336
Bai Z, Li Y, Chen X, Yi T, Wei W, Wozniak M, Damasevicius R. Real-Time Video Stitching for Mine Surveillance Using a Hybrid Image Registration Method. Electronics. 2020; 9(9):1336. https://doi.org/10.3390/electronics9091336
Chicago/Turabian StyleBai, Zongwen, Ying Li, Xiaohuan Chen, Tingting Yi, Wei Wei, Marcin Wozniak, and Robertas Damasevicius. 2020. "Real-Time Video Stitching for Mine Surveillance Using a Hybrid Image Registration Method" Electronics 9, no. 9: 1336. https://doi.org/10.3390/electronics9091336
APA StyleBai, Z., Li, Y., Chen, X., Yi, T., Wei, W., Wozniak, M., & Damasevicius, R. (2020). Real-Time Video Stitching for Mine Surveillance Using a Hybrid Image Registration Method. Electronics, 9(9), 1336. https://doi.org/10.3390/electronics9091336