MCBM-SLAM: An Improved Mask-Region-Convolutional Neural Network-Based Simultaneous Localization and Mapping System for Dynamic Environments
Abstract
:1. Introduction
- (1)
- This paper adds an attention mechanism module to the existing neural network to solve the problem of incomplete dynamic object segmentation;
- (2)
- An image mismatch rejection algorithm incorporating grid motion statistics with adaptive margins is proposed;
- (3)
- Re-addition of static feature points on potential dynamic features using chi-square test and motion constraints.
2. Related Work
3. System Framework
3.1. SLAM System Framework
3.2. Mask R-CNN Network Based on Improved Attention Mechanism
3.3. Mask R-CNN Network Based on Attention Mechanism
3.4. Cardinality Experiment and Motion Consistency Detection
4. Experimental Analysis
4.1. Experiments on SLAM Algorithm in Dynamic Environment
4.1.1. Experimental Analysis on TUM RGBD Dataset
4.1.2. Experimental Analysis on KITTI Dataset
5. Conclusions
Author Contributions
Funding
Data Availability Statement
Conflicts of Interest
Abbreviations
Abbreviations in the Abstract | Full Name |
SLAM | Simultaneous locali localization and mapping |
Mask R-CNN | Mask Region-CNN |
KITTI | Karlsruhe Institute of Technologyand Toyota Technological Institute |
TUM-RGBD | The RGB-D dataset proposed by the tum Computer Vision Group |
ORB-SLAM2 | Simultaneous localization and mapping algorithm based on ORB features |
Dyna-SLAM | Simultaneous localization and mapping algorithm in a dynamic environment |
References
- Mur-Artal, R.; Tardós, J.D. Orb-Slam2: An Open-Source Slam System for Monocular, Stereo, and Rgb-d Cameras. IEEE Trans. Robot. 2017, 33, 1255–1262. [Google Scholar] [CrossRef]
- Campos, C.; Elvira, R.; Rodríguez, J.J.G.; Montiel, J.M.; Tardós, J.D. Orb-Slam3: An Accurate Open-Source Library for Visual, Visual–Inertial, and Multimap Slam. IEEE Trans. Robot. 2021, 37, 1874–1890. [Google Scholar] [CrossRef]
- Schenk, F.; Fraundorfer, F. RESLAM: A Real-Time Robust Edge-Based SLAM System. In Proceedings of the 2019 International Conference on Robotics and Automation (ICRA), Montreal, QC, Canada, 20–24 May 2019; pp. 154–160. [Google Scholar]
- Su, P.; Luo, S.; Huang, X. Real-Time Dynamic SLAM Algorithm Based on Deep Learning. IEEE Access 2022, 10, 87754–87766. [Google Scholar] [CrossRef]
- Bescos, B.; Fácil, J.M.; Civera, J.; Neira, J. DynaSLAM: Tracking, Mapping, and Inpainting in Dynamic Scenes. IEEE Robot. Autom. Lett. 2018, 3, 4076–4083. [Google Scholar] [CrossRef]
- Yu, C.; Liu, Z.; Liu, X.-J.; Xie, F.; Yang, Y.; Wei, Q.; Fei, Q. DS-SLAM: A Semantic Visual SLAM towards Dynamic Environments. In Proceedings of the 2018 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), Madrid, Spain, 1–5 October 2018; pp. 1168–1174. [Google Scholar]
- Xing, Z.; Zhu, X.; Dong, D. DE-SLAM: SLAM for Highly Dynamic Environment. J. Field Robot. 2022, 39, 528–542. [Google Scholar] [CrossRef]
- Zhong, Y.; Hu, S.; Huang, G.; Bai, L.; Li, Q. WF-SLAM: A Robust VSLAM for Dynamic Scenarios via Weighted Features. IEEE Sens. J. 2022, 22, 10818–10827. [Google Scholar] [CrossRef]
- Lyu, L.; Ding, Y.; Yuan, Y.; Zhang, Y.; Liu, J.; Li, J. Doc-Slam: Robust Stereo Slam with Dynamic Object Culling. In Proceedings of the 2021 7th International Conference on Automation, Robotics and Applications (ICARA), Prague, Czech Republic, 4–6 February 2021; pp. 258–262. [Google Scholar]
- Yin, H.; Li, S.; Tao, Y.; Guo, J.; Huang, B. Dynam-SLAM: An Accurate, Robust Stereo Visual-Inertial SLAM Method in Dynamic Environments. IEEE Trans. Robot. 2022, 39, 289–308. [Google Scholar] [CrossRef]
- Li, A.; Wang, J.; Xu, M.; Chen, Z. DP-SLAM: A Visual SLAM with Moving Probability towards Dynamic Environments. Inf. Sci. 2021, 556, 128–142. [Google Scholar] [CrossRef]
- Cheng, S.; Sun, C.; Zhang, S.; Zhang, D. SG-SLAM: A Real-Time RGB-D Visual SLAM towards Dynamic Scenes with Semantic and Geometric Information. IEEE Trans. Instrum. Meas. 2022, 72, 7501012. [Google Scholar] [CrossRef]
- Ni, J.; Wang, X.; Gong, T.; Xie, Y. An Improved Adaptive ORB-SLAM Method for Monocular Vision Robot under Dynamic Environments. Int. J. Mach. Learn. Cybern. 2022, 13, 3821–3836. [Google Scholar] [CrossRef]
- Wang, C.; Luo, B.; Zhang, Y.; Zhao, Q.; Yin, L.; Wang, W.; Su, X.; Wang, Y.; Li, C. DymSLAM: 4D Dynamic Scene Reconstruction Based on Geometrical Motion Segmentation. IEEE Robot. Autom. Lett. 2020, 6, 550–557. [Google Scholar] [CrossRef]
- Chen, J.; Xie, F.; Huang, L.; Yang, J.; Liu, X.; Shi, J. A Robot Pose Estimation Optimized Visual SLAM Algorithm Based on CO-HDC Instance Segmentation Network for Dynamic Scenes. Remote Sens. 2022, 14, 2114. [Google Scholar] [CrossRef]
- Wu, W.; Guo, L.; Gao, H.; You, Z.; Liu, Y.; Chen, Z. YOLO-SLAM: A Semantic SLAM System towards Dynamic Environment with Geometric Constraint. Neural Comput. Appl. 2022, 34, 6011–6026. [Google Scholar] [CrossRef]
- Zhang, R.; Zhang, X. Geometric Constraint-Based and Improved YOLOv5 Semantic SLAM for Dynamic Scenes. ISPRS Int. J. Geo-Inf. 2023, 12, 211. [Google Scholar] [CrossRef]
- Gou, R.; Chen, G.; Yan, C.; Pu, X.; Wu, Y.; Tang, Y. Three-Dimensional Dynamic Uncertainty Semantic SLAM Method for a Production Workshop. Eng. Appl. Artif. Intell. 2022, 116, 105325. [Google Scholar] [CrossRef]
- Liu, Y.; Wu, Y.; Pan, W. Dynamic RGB-D SLAM Based on Static Probability and Observation Number. IEEE Trans. Instrum. Meas. 2021, 70, 8503411. [Google Scholar] [CrossRef]
- Zhang, Y.; Liu, R.; Yu, H.; Zhou, B.; Qian, K. Visual Loop Closure Detection with Instance Segmentation and Image Inpainting in Dynamic Scenes Using Wearable Camera. IEEE Sens. J. 2022, 22, 16628–16637. [Google Scholar] [CrossRef]
- Dai, W.; Zhang, Y.; Li, P.; Fang, Z.; Scherer, S. Rgb-d Slam in Dynamic Environments Using Point Correlations. IEEE Trans. Pattern Anal. Mach. Intell. 2020, 44, 373–389. [Google Scholar] [CrossRef]
- Rosinol, A.; Violette, A.; Abate, M.; Hughes, N.; Chang, Y.; Shi, J.; Gupta, A.; Carlone, L. Kimera: From SLAM to Spatial Perception with 3D Dynamic Scene Graphs. Int. J. Robot. Res. 2021, 40, 1510–1546. [Google Scholar] [CrossRef]
- Cheng, J.; Zhang, H.; Meng, M.Q.-H. Improving Visual Localization Accuracy in Dynamic Environments Based on Dynamic Region Removal. IEEE Trans. Autom. Sci. Eng. 2020, 17, 1585–1596. [Google Scholar] [CrossRef]
- Han, S.; Xi, Z. Dynamic Scene Semantics SLAM Based on Semantic Segmentation. IEEE Access 2020, 8, 43563–43570. [Google Scholar] [CrossRef]
- Zang, Q.; Zhang, K.; Wang, L.; Wu, L. An Adaptive ORB-SLAM3 System for Outdoor Dynamic Environments. Sensors 2023, 23, 1359. [Google Scholar] [CrossRef]
- Yuan, C.; Xu, Y.; Zhou, Q. PLDS-SLAM: Point and Line Features SLAM in Dynamic Environment. Remote Sens. 2023, 15, 1893. [Google Scholar] [CrossRef]
- Zhang, X.; Wang, X.; Zhang, R. Dynamic Semantics SLAM Based on Improved Mask R-CNN. IEEE Access 2022, 10, 126525–126535. [Google Scholar] [CrossRef]
- Gong, H.; Gong, L.; Ma, T.; Sun, Z.; Li, L. AHY-SLAM: Toward Faster and More Accurate Visual SLAM in Dynamic Scenes Using Homogenized Feature Extraction and Object Detection Method. Sensors 2023, 23, 4241. [Google Scholar] [CrossRef]
- He, K.; Gkioxari, G.; Dollár, P.; Girshick, R. Mask R-Cnn. In Proceedings of the IEEE International Conference on Computer Vision, Venice, Italy, 22–29 October 2017; pp. 2961–2969. [Google Scholar]
- Girshick, R. Fast R-Cnn. In Proceedings of the IEEE International Conference on Computer Vision, Santiago, Chile, 7–13 December 2015; pp. 1440–1448. [Google Scholar]
Sequence | SD(m) | ||
---|---|---|---|
ORB-SLAM2 | Dyna-SLAM | MCBM-SLAM | |
s_static | 0.0042 | 0.0039 | 0.0040 |
w_halfsphere | 0.3085 | 0.0192 | 0.0187 |
w_static | 0.1925 | 0.0048 | 0.0042 |
w_xyz | 0.3267 | 0.0085 | 0.0065 |
Sequence | ORB-SLAM2 | Dyna-SLAM | MCBM-SLAM | |||||||||
---|---|---|---|---|---|---|---|---|---|---|---|---|
Median | Mean | Min | Max | Median | Mean | Min | Max | Median | Mean | Min | Max | |
s_static | 0.012 | 0.011 | 0.010 | 0.012 | 0.011 | 0.010 | 0.009 | 0.012 | 0.009 | 0.010 | 0.007 | 0.012 |
w_halfsphere | 0.916 | 0.976 | 0.828 | 1.210 | 0.058 | 0.115 | 0.041 | 0.299 | 0.038 | 0.036 | 0.027 | 0.040 |
w_static | 0.437 | 0.429 | 0.394 | 0.445 | 0.015 | 0.015 | 0.014 | 0.016 | 0.008 | 0.007 | 0.006 | 0.009 |
w_xyz | 0.771 | 0.726 | 0.590 | 0.800 | 0.044 | 0.094 | 0.020 | 0.215 | 0.022 | 0.023 | 0.017 | 0.025 |
Sequence | Dyna-SLAM | MCBM-SLAM | ||||||
---|---|---|---|---|---|---|---|---|
Median | Mean | Min | Max | Median | Mean | Min | Max | |
s_static | 8.33% | 9.09% | 10% | - | 25.00% | 9.09%% | 30% | - |
w_halfsphere | 93.67% | 79.65% | 95.05% | 75.29% | 95.85% | 96.31% | 96.74% | 96.69% |
w_static | 96.57% | 96.50% | 96.45% | 96.40% | 98.17% | 98.37% | 98.48% | 97.98% |
w_xyz | 94.29% | 87.05% | 96.61% | 73.13% | 96.15% | 96.83% | 97.12% | 96.88% |
Serial Number | RMSE (m) | |
---|---|---|
ORB-SLAM2 | MCBM-SLAM | |
00 | 0.9466 | 1.0745 |
01 | 3.4375 | 3.3125 |
02 | 6.0386 | 5.7456 |
03 | 0.3011 | 0.2745 |
04 | 0.1849 | 0.1645 |
05 | 0.6173 | 0.6352 |
06 | 1.3647 | 1.3746 |
07 | 0.4165 | 0.3625 |
08 | 6.6482 | 6.6901 |
09 | 2.6057 | 2.6845 |
10 | 2.050 | 1.9150 |
Disclaimer/Publisher’s Note: The statements, opinions and data contained in all publications are solely those of the individual author(s) and contributor(s) and not of MDPI and/or the editor(s). MDPI and/or the editor(s) disclaim responsibility for any injury to people or property resulting from any ideas, methods, instructions or products referred to in the content. |
© 2023 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https://creativecommons.org/licenses/by/4.0/).
Share and Cite
Wang, X.; Zhang, X. MCBM-SLAM: An Improved Mask-Region-Convolutional Neural Network-Based Simultaneous Localization and Mapping System for Dynamic Environments. Electronics 2023, 12, 3596. https://doi.org/10.3390/electronics12173596
Wang X, Zhang X. MCBM-SLAM: An Improved Mask-Region-Convolutional Neural Network-Based Simultaneous Localization and Mapping System for Dynamic Environments. Electronics. 2023; 12(17):3596. https://doi.org/10.3390/electronics12173596
Chicago/Turabian StyleWang, Xiankun, and Xinguang Zhang. 2023. "MCBM-SLAM: An Improved Mask-Region-Convolutional Neural Network-Based Simultaneous Localization and Mapping System for Dynamic Environments" Electronics 12, no. 17: 3596. https://doi.org/10.3390/electronics12173596
APA StyleWang, X., & Zhang, X. (2023). MCBM-SLAM: An Improved Mask-Region-Convolutional Neural Network-Based Simultaneous Localization and Mapping System for Dynamic Environments. Electronics, 12(17), 3596. https://doi.org/10.3390/electronics12173596