A Generic Image Processing Pipeline for Enhancing Accuracy and Robustness of Visual Odometry
Abstract
:1. Introduction
2. Related Works
2.1. Image Filtration
2.2. Non-Maximal Suppression
2.3. Outliers Rejection
3. Proposed Pipeline
3.1. Image Pre-Processing
3.2. Features Detection and Matching
3.2.1. Suppression via Square Covering
3.2.2. Angle-Based Outliers Rejection (AOR) for Feature Matching
Algorithm 1: AOR Algorithm |
Set R and fordo
end Calculate as in Equation (7). fordo
end where is the set of matched features inliers. |
4. Experimental Work
4.1. Motion Estimation
4.1.1. Stereo/RGB-D Visual Odometry
4.1.2. Monocular Visual Odometry
4.2. Datasets
4.2.1. KITTI Vision Benchmark Dataset
4.2.2. TUM RGB-D Dataset
4.2.3. Images from Omnidirectional Robot
5. Results and Discussion
5.1. KITTI Vision Benchmark Dataset/Stereo VO
5.1.1. Pose Accuracy Comparison
5.1.2. Effect of AOR
5.1.3. Computational Cost
5.2. TUM RGB-D Dataset/RGB-D VO
5.2.1. Pose Accuracy Comparison
5.2.2. Computational Cost
5.3. Summit XL Steel Sequences/Monocular VO
5.3.1. Pose Accuracy Comparison
5.3.2. Computational Cost
5.4. Discussion
6. Conclusions and Future Works
Author Contributions
Funding
Data Availability Statement
Acknowledgments
Conflicts of Interest
Abbreviations
MDPI | Multidisciplinary Digital Publishing Institute |
DOAJ | Directory of Open Access Journals |
TLA | Three Letter Acronym |
LD | Linear Dichroism |
References
- Zhang, J.; Singh, S. LOAM: Lidar Odometry and Mapping in Real-time. In Proceedings of the Robotics: Science and Systems Conference, Rome, Italy, 13–15 July 2014; Volume 2. [Google Scholar]
- Quist, E.B.; Niedfeldt, P.C.; Beard, R.W. Radar odometry with recursive-RANSAC. IEEE Trans. Aerosp. Electron. Syst. 2016, 52, 1618–1630. [Google Scholar] [CrossRef]
- Drawil, N.M.; Amar, H.M.; Basir, O.A. GPS localization accuracy classification: A context-based approach. IEEE Trans. Intell. Transp. Syst. 2012, 14, 262–273. [Google Scholar] [CrossRef]
- Yi, J.; Zhang, J.; Song, D.; Jayasuriya, S. IMU-based localization and slip estimation for skid-steered mobile robots. In Proceedings of the IEEE/RSJ International Conference on Intelligent Robots and Systems, San Diego, CA, USA, 29 October–2 November 2007; pp. 2845–2850. [Google Scholar]
- Lee, S.; Song, J.B. Robust mobile robot localization using optical flow sensors and encoders. In Proceedings of the IEEE International Conference on Robotics and Automation (ICRA), New Orleans, LA, USA, 26 April–1 May 2004; Volume 1, pp. 1039–1044. [Google Scholar]
- Shim, J.H.; Cho, Y.I. A mobile robot localization via indoor fixed remote surveillance cameras. Sensors 2016, 16, 195. [Google Scholar] [CrossRef] [Green Version]
- Scaramuzza, D.; Fraundorfer, F. Visual odometry [tutorial]. IEEE Robot. Autom. Mag. 2011, 18, 80–92. [Google Scholar] [CrossRef]
- Fraundorfer, F.; Scaramuzza, D. Visual odometry: Part ii: Matching, robustness, optimization, and applications. IEEE Robot. Autom. Mag. 2012, 19, 78–90. [Google Scholar] [CrossRef] [Green Version]
- Strasdat, H.; Montiel, J.; Davison, A.J. Scale drift-aware large scale monocular SLAM. Robot. Sci. Syst. VI 2010, 2, 7. [Google Scholar]
- Engel, J.; Schops, T.; Cremers, D. LSD-SLAM Large-scale direct monocular SLAM. In Proceedings of the European Conference on Computer Vision; Springer: Berlin/Heidelberg, Germany, 2014; pp. 834–849. [Google Scholar]
- Mur-Artal, R.; Montiel, J.M.M.; Tardos, J.D. ORB-SLAM a versatile and accurate monocular SLAM system. IEEE Trans. Robotics 2015, 31, 1147–1163. [Google Scholar] [CrossRef] [Green Version]
- Qu, X.; Soheilian, B.; Paparoditis, N. Landmark based localization in urban environment. ISPRS J. Photogramm. Remote. Sens. 2018, 140, 90–103. [Google Scholar] [CrossRef]
- Yang, N.; Wang, R.; Cremers, D. Feature-based or direct An evaluation of monocular visual odometry. arXiv 2017, arXiv:1705.04300. [Google Scholar]
- Tomono, M. 3-D localization and mapping using a single camera based on structure-from-motion with automatic baseline selection. In Proceedings of the IEEE International Conference on Robotics and Automation, Barcelona, Spain, 18–22 April 2005; pp. 3342–3347. [Google Scholar]
- Furukawa, Y.; Hernández, C. Multi-view stereo: A tutorial. Found. Trends Comput. Graph. Vis. 2015, 9, 1–148. [Google Scholar] [CrossRef] [Green Version]
- Hartley, R.; Zisserman, A. Multiple View Geometry in Computer Vision; Cambridge University Press: Cambridge, UK, 2003. [Google Scholar]
- Sabry, M.; Al-Kaff, A.; Hussein, A.; Abdennadher, S. Ground Vehicle Monocular Visual Odometry. In Proceedings of the IEEE Intelligent Transportation Systems Conf. (ITSC), Auckland, NZ, USA, 27–30 October 2019; pp. 3587–3592. [Google Scholar]
- Zhou, D.; Dai, Y.; Li, H. Ground-Plane-Based Absolute Scale Estimation for Monocular Visual Odometry. IEEE Trans. Intell. Transp. Syst. 2019, 21, 791–802. [Google Scholar] [CrossRef] [Green Version]
- Zhou, D.; Dai, Y.; Li, H. Reliable scale estimation and correction for monocular visual odometry. In Proceedings of the IEEE Intelligent Vehicles Symposium (IV), Gotenburg, Sweden, 19–22 June 2016; IEEE: Piscataway, NJ, USA, 2016; pp. 490–495. [Google Scholar]
- Li, R.; Wang, S.; Long, Z.; Gu, D. Undeepvo Monocular visual odometry through unsupervised deep learning. In Proceedings of the IEEE International Conference on Robotics and Automation (ICRA), Brisbane, Australia, 21–25 May 2018; pp. 7286–7291. [Google Scholar]
- Zhan, H.; Garg, R.; Saroj Weerasekera, C.; Li, K.; Agarwal, H.; Reid, I. Unsupervised learning of monocular depth estimation and visual odometry with deep feature reconstruction. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA, 18–22 June 2018; pp. 340–349. [Google Scholar]
- Steinbrücker, F.; Sturm, J.; Cremers, D. Real-time visual odometry from dense RGB-D images. In Proceedings of the IEEE International Conference on Computer Vision Workshops (ICCV Workshops), Barcelona, Spain, 7 November 2011; pp. 719–722. [Google Scholar]
- Henry, P.; Krainin, M.; Herbst, E.; Ren, X.; Fox, D. RGB-D mapping: Using Kinect-style depth cameras for dense 3D modeling of indoor environments. Int. J. Robot. Res. 2012, 31, 647–663. [Google Scholar] [CrossRef]
- Li, X.; Zhang, C. Robust RGB-D Visual Odometry Based on the Line Intersection Structure Feature in Low-Textured Scenes. In Proceedings of the 5th IEEE International Conference on Cloud Computing and Intelligence Systems (CCIS), Nanjing, China, 23–25 November 2018; pp. 390–394. [Google Scholar]
- Lowe, D.G. Distinctive image features from scale-invariant keypoints. Int. J. Comput. Vis. 2004, 60, 91–110. [Google Scholar] [CrossRef]
- Bay, H.; Tuytelaars, T.; Van Gool, L. Surf Speeded up robust features. In Proceedings of the European Conference on Computer Vision, Graz, Austria, 7–13 May 2006; Springer: Berlin/Heidelberg, Germany, 2006; pp. 404–417. [Google Scholar]
- Rublee, E.; Rabaud, V.; Konolige, K.; Bradski, G. ORB: An efficient alternative to SIFT or SURF. In Proceedings of the IEEE International Conference on Computer Vision, Washington, DC, USA, 6–13 November 2011; pp. 2564–2571. [Google Scholar]
- BAYRAKTAR, E.; Boyraz, P. Analysis of feature detector and descriptor combinations with a localization experiment for various performance metrics. Turk. J. Electr. Eng. Comput. Sci. 2017, 25, 2444–2454. [Google Scholar] [CrossRef]
- Dong, X.; Dong, X.; Dong, J.; Zhou, H. Monocular visual-IMU odometry: A comparative evaluation of detector-descriptor-based methods. IEEE Trans. Intell. Transp. Syst. 2019, 21, 2471–2484. [Google Scholar] [CrossRef] [Green Version]
- Alismail, H.; Kaess, M.; Browning, B.; Lucey, S. Direct visual odometry in low light using binary descriptors. IEEE Robot. Autom. Lett. 2016, 2, 444–451. [Google Scholar] [CrossRef]
- Engel, J.; Sturm, J.; Cremers, D. Semi-dense visual odometry for a monocular camera. In Proceedings of the IEEE International Conference on Computer Vision, Sydney, Australia, 1–8 December 2013; pp. 1449–1456. [Google Scholar]
- Yang, N.; Wang, R.; Gao, X.; Cremers, D. Challenges in monocular visual odometry Photometric calibration, motion bias, and rolling shutter effect. IEEE Robot. Autom. Lett. 2018, 3, 2878–2885. [Google Scholar] [CrossRef] [Green Version]
- Bailo, O.; Rameau, F.; Joo, K.; Park, J.; Bogdan, O.; Kweon, I.S. Efficient adaptive non-maximal suppression algorithms for homogeneous spatial keypoint distribution. Pattern Recognit. Lett. 2018, 106, 53–60. [Google Scholar] [CrossRef]
- Geiger, A.; Lenz, P.; Urtasun, R. Are we ready for autonomous driving? The kitti vision benchmark suite. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Providence, RI, USA, 16–21 June 2012; pp. 3354–3361. [Google Scholar]
- Sturm, J.; Engelhard, N.; Endres, F.; Burgard, W.; Cremers, D. A benchmark for the evaluation of RGB-D SLAM systems. In Proceedings of the IEEE/RSJ International Conference on Intelligent Robots and Systems, Algarve, Portugal, 7–12 October 2012; pp. 573–580. [Google Scholar]
- Ramanath, R.; Snyder, W.E.; Yoo, Y.; Drew, M.S. Color image processing pipeline. IEEE Signal Process. Mag. 2005, 22, 34–43. [Google Scholar] [CrossRef]
- Campbell, J.; Sukthankar, R.; Nourbakhsh, I.; Pahwa, A. A robust visual odometry and precipice detection system using consumer-grade monocular vision. In Proceedings of the IEEE International Conference on Robotics and Automation, Barcelona, Spain, 18–22 April 2005; pp. 3421–3427. [Google Scholar]
- Desai, A.; Lee, D.J. Visual odometry drift reduction using SYBA descriptor and feature transformation. IEEE Trans. Intell. Transp. Syst. 2016, 17, 1839–1851. [Google Scholar] [CrossRef]
- Zhao, X.; Min, H.; Xu, Z.; Wu, X.; Li, X.; Sun, P. Image antiblurring and statistic filter of feature space displacement: Application to visual odometry for outdoor ground vehicle. J. Sens. 2018, 2018, 2987819. [Google Scholar] [CrossRef] [Green Version]
- Yang, W.; Zhai, X. Contrast Limited Adaptive Histogram Equalization for an Advanced Stereo Visual SLAM System. In Proceedings of the IEEE International Conference on Cyber-Enabled Distributed Computing and Knowledge Discovery (CyberC), Guilin, China, 17–19 October 2019; pp. 131–134. [Google Scholar]
- Zhang, J.; Ila, V.; Kneip, L. Robust visual odometry in underwater environment. In Proceedings of the IEEE OCEANS-MTS/IEEE Kobe Techno-Oceans (OTO), Kobe, Japan, 28–31 May 2018; pp. 1–9. [Google Scholar]
- Wu, M.; Lam, S.K.; Srikanthan, T. A framework for fast and robust visual odometry. IEEE Trans. Intell. Transp. Syst. 2017, 18, 3433–3448. [Google Scholar] [CrossRef]
- Jiang, Y.; Xu, Y.; Liu, Y. Performance evaluation of feature detection and matching in stereo visual odometry. Neurocomputing 2013, 120, 380–390. [Google Scholar] [CrossRef]
- Mou, W.; Wang, H.; Seet, G. Efficient visual odometry estimation using stereo camera. In Proceedings of the 11th IEEE International Conference on Control & Automation (ICCA), Sapporo, Japan, 6–9 July 2014; pp. 1399–1403. [Google Scholar]
- Rosten, E.; Drummond, T. Machine learning for high-speed corner detection. In Proceedings of the European Conference on Computer Vision, Tel Aviv, Israel, 23–27 October 2006; Springer: Berlin/Heidelberg, Germany, 2006; pp. 430–443. [Google Scholar]
- Aladem, M.; Rawashdeh, S.A. Lightweight visual odometry for autonomous mobile robots. Sensors 2018, 18, 2837. [Google Scholar] [CrossRef] [Green Version]
- Brown, M.; Szeliski, R.; Winder, S. Multi-image matching using multi-scale oriented patches. In Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR’05), San Diego, CA, USA, 20–25 June 2005; Volume 1, pp. 510–517. [Google Scholar]
- Calonder, M.; Lepetit, V.; Ozuysal, M.; Trzcinski, T.; Strecha, C.; Fua, P. BRIEF: Computing a local binary descriptor very fast. IEEE Trans. Pattern Anal. Mach. Intell. 2011, 34, 1281–1298. [Google Scholar] [CrossRef] [PubMed] [Green Version]
- Buczko, M.; Willert, V. How to distinguish inliers from outliers in visual odometry for high-speed automotive applications. In Proceedings of the IEEE Intelligent Vehicles Symposium (IV), Gotenburg, Sweden, 19–22 June 2016; pp. 478–483. [Google Scholar]
- Fan, H.; Zhang, S. Stereo Odometry Based on Careful Frame Selection. In Proceedings of the 10th IEEE International Symposium on Computational Intelligence and Design (ISCID), Hangzhou, China, 9–10 December 2017; Volume 2, pp. 177–180. [Google Scholar]
- Kitt, B.; Geiger, A.; Lategahn, H. Visual odometry based on stereo image sequences with ransac-based outlier rejection scheme. In Proceedings of the IEEE Intelligent Vehicles Symposium (IV), La Jolla, CA, USA, 21–24 June 2010; pp. 486–492. [Google Scholar]
- Zuiderveld, K. Contrast limited adaptive histogram equalization. In Proceedings of the Graphics Gems IV; Academic Press Professional, Inc.: Cambridge, MA, USA, 1994; pp. 474–485. [Google Scholar]
- Mount, D.M.; Netanyahu, N.S.; Le Moigne, J. Efficient algorithms for robust feature matching. Pattern Recognit. 1999, 32, 17–38. [Google Scholar] [CrossRef] [Green Version]
- Weisstein, E.W. Law of Cosines from MathWorld—A Wolfram Web Resource. Available online: https://mathworld.wolfram.com/LawofCosines.html (accessed on 10 June 2020).
- Weisstein, E.W. Angular Distance from MathWorld—A Wolfram Web Resource. Available online: https://mathworld.wolfram.com/AngularDistance.html (accessed on 10 June 2020).
- Lu, X.; Yaoy, J.; Li, H.; Liu, Y.; Zhang, X. 2-line exhaustive searching for real-time vanishing point estimation in Manhattan world. In Proceedings of the Winter Conference on Applications of Computer Vision (WACV), Santa Rosa, CA, USA, 24–31 March 2017; IEEE: Piscataway, NJ, USA, 2017; pp. 345–353. [Google Scholar]
- Quigley, M.; Conley, K.; Gerkey, B.; Faust, J.; Foote, T.; Leibs, J.; Wheeler, R.; Ng, A.Y. ROS an open-source Robot Operating System. ICRA Workshop Open Source Softw. 2009, 3, 1–5. [Google Scholar]
- Robotnik. SUMMIT- XL STEEL Datasheet. Available online: www.robotnik.eu (accessed on 10 June 2020).
- Nister, D. An efficient solution to the five-point relative pose problem. IEEE Trans. Pattern Anal. Mach. Intell. 2004, 26, 756–770. [Google Scholar] [CrossRef]
- Rousseeuw, P.J. Least median of squares regression. J. Am. Stat. Assoc. 1984, 79, 871–880. [Google Scholar] [CrossRef]
- Krejci, T. kitti2bag. Available online: https://github.com/tomas789/kitti2bag/ (accessed on 1 February 2020).
- TUM. Submission form for automatic evaluation of RGB-D SLAM results. Available online: https://vision.in.tum.de/data/datasets/rgbd-dataset/online_evaluation (accessed on 10 June 2020).
- Achtelik, M.; vicon bridge. GitHub Repository. Available online: https://github.com/ethz-asl/vicon_bridge (accessed on 1 October 2020).
Sequence Number | VO without Pipeline | VO with AOR | VO with Pipeline | |||
---|---|---|---|---|---|---|
Tr (%) | Rot (\deg/m) | Tr (%) | Rot (deg/m) | Tr (%) | Rot (deg/m) | |
0 | 1.85 | 0.011 | 1.80 | 0.011 | 1.68 | 0.011 |
1 | 5.14 | 0.013 | 5.33 | 0.013 | 4.72 | 0.012 |
2 | 6.99 | 0.036 | 1.82 | 0.016 | 1.89 | 0.017 |
4 | 3.91 | 0.007 | 2.75 | 0.006 | 0.33 | 0.007 |
5 | 2.38 | 0.011 | 2.75 | 0.012 | 2.43 | 0.009 |
6 | 2.62 | 0.010 | 2.74 | 0.010 | 2.88 | 0.010 |
7 | 2.11 | 0.016 | 1.95 | 0.014 | 2.07 | 0.015 |
8 | 2.57 | 0.012 | 2.65 | 0.011 | 1.92 | 0.012 |
9 | 2.70 | 0.019 | 2.86 | 0.019 | 1.78 | 0.019 |
10 | 2.65 | 0.020 | 2.82 | 0.02 | 2.02 | 0.020 |
Overall | 3.29 | 0.015 | 2.71 | 0.013 | 2.18 | 0.013 |
Seq. No. | VO without AOR | VO with AOR | ||
---|---|---|---|---|
Tr (%) | Rot (deg/m) | Tr (%) | Rot (deg/m) | |
0 | 1.85 | 0.011 | 1.80 | 0.011 |
1 | 5.14 | 0.013 | 5.33 | 0.013 |
2 | 6.99 | 0.036 | 1.82 | 0.0158 |
4 | 3.91 | 0.007 | 2.75 | 0.0056 |
5 | 2.38 | 0.011 | 2.75 | 0.012 |
6 | 2.62 | 0.010 | 2.74 | 0.010 |
7 | 2.11 | 0.016 | 1.95 | 0.014 |
8 | 2.57 | 0.012 | 2.65 | 0.011 |
9 | 2.70 | 0.019 | 2.86 | 0.019 |
10 | 2.65 | 0.020 | 2.82 | 0.020 |
Overall | 3.29 | 0.015 | 2.71 | 0.013 |
VO Type | Comp. Time (mean ± std [ms]) | Tr RMSE (%) | Rot RMSE (deg/m) |
---|---|---|---|
Vanilla | 116 ± 31 | 3.29 | 0.0154 |
CLAHE | 176 ± 36 | 2.88 | 0.0136 |
CLAHE and SSC | 181 ± 34 | 2.86 | 0.0136 |
AOR Only | 106 ± 24 | 2.71 | 0.0134 |
Full Pipeline | 160 ± 35 | 2.18 | 0.0133 |
Seq. Name | VO without Pipeline | VO with Pipeline | ||
---|---|---|---|---|
Tr (m) | Rot (deg) | Tr (m) | Rot (deg) | |
fr1/xyz | 0.24 | 8.80 | 0.04 | 2.08 |
fr1/desk | 0.43 | 19.5 | 0.07 | 3.55 |
fr1/desk2 | 0.46 | 23.8 | 0.09 | 6.74 |
fr1/room | 0.32 | 23.5 | 0.12 | 5.73 |
fr2/pioneer_360 | 0.18 | 6.50 | 0.10 | 3.58 |
fr2/pioneer_slam | 0.10 | 3.60 | 0.09 | 2.38 |
fr2/pioneer_slam2 | 0.07 | 2.82 | 0.07 | 2.00 |
fr2/pioneer_slam3 | 0.07 | 2.03 | 0.05 | 1.44 |
fr2/desk | 0.19 | 5.20 | 0.02 | 0.64 |
Overall | 0.23 | 10.6 | 0.07 | 3.12 |
VO Type | Comp. Time (mean ± std (ms)) | Tr RMSE (m) | Rot RMSE (deg) |
---|---|---|---|
Vanilla | 118 ± 26 | 0.229 | 10.6 |
CLAHE | 179 ± 48 | 0.213 | 10.7 |
CLAHE and SSC | 185 ± 53 | 0.210 | 9.98 |
AOR Only | 169 ± 36 | 0.135 | 10.4 |
Full Pipeline | 176 ± 40 | 0.073 | 3.13 |
Seq. Name | VO without Pipeline | VO with Pipeline | ||
---|---|---|---|---|
Tr (m) | Rot (deg) | Tr (m) | Rot (deg) | |
Rectangle 1 | 0.49 | 1.67 | 0.34 | 1.51 |
Rectangle 2 | 0.25 | 1.69 | 0.25 | 1.52 |
Circle | 0.50 | 2.09 | 0.48 | 2.07 |
Overall | 0.31 | 1.75 | 0.27 | 1.67 |
VO Type | Comp. Time (mean ± std (ms)) | Tr RMSE (m) | Rot RMSE (deg) |
---|---|---|---|
Vanilla | 148 ± 28 | 0.315 | 1.68 |
CLAHE | 165 ± 47 | 0.350 | 1.68 |
CLAHE and SSC | 168 ± 42 | 0.302 | 1.74 |
AOR Only | 132 ± 19 | 0.287 | 1.68 |
Full Pipeline | 158 ± 34 | 0.275 | 1.67 |
Dataset | Tr (%) | Rot (%) | Comp. Time (ms) |
---|---|---|---|
KITTI | −33% | −13% | +44 |
TUM | −68% | −70% | +58 |
Summit | −12% | −0.5% | +10 |
Publisher’s Note: MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations. |
© 2022 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https://creativecommons.org/licenses/by/4.0/).
Share and Cite
Sabry, M.; Osman, M.; Hussein, A.; Mehrez, M.W.; Jeon, S.; Melek, W. A Generic Image Processing Pipeline for Enhancing Accuracy and Robustness of Visual Odometry. Sensors 2022, 22, 8967. https://doi.org/10.3390/s22228967
Sabry M, Osman M, Hussein A, Mehrez MW, Jeon S, Melek W. A Generic Image Processing Pipeline for Enhancing Accuracy and Robustness of Visual Odometry. Sensors. 2022; 22(22):8967. https://doi.org/10.3390/s22228967
Chicago/Turabian StyleSabry, Mohamed, Mostafa Osman, Ahmed Hussein, Mohamed W. Mehrez, Soo Jeon, and William Melek. 2022. "A Generic Image Processing Pipeline for Enhancing Accuracy and Robustness of Visual Odometry" Sensors 22, no. 22: 8967. https://doi.org/10.3390/s22228967