A Multi-Sensory Guidance System for the Visually Impaired Using YOLO and ORB-SLAM †
Abstract
:1. Introduction
- We evaluate the potential of ORB-SLAM and YOLO as assistive methods for the blind guidance system. A multi-sensory blind guidance system is proposed and implemented to combine both the tactile and auditory sensations;
- We propose improving the conventional SLAM accuracy by generating a dense navigation map. Our proposed system utilizes the position information from the ORB-SLAM that can be fused using the Bresenham algorithm [20]. A dense navigation map is developed by transforming coordinates to feature points. A local obstacle avoidance algorithm is developed to identify short-range obstacles through point cloud filtering and angle sampling;
- We implemented the overall system as a prototype for our proposed equipment. Experiments are performed in four different environments. Results demonstrate that our proposed system can accurately reconstruct the map of the surrounding environment in various scenarios. Our system is lighted and weighted and does not require an external power supply. Trial experiments show that it can be useful auxiliary equipment for the community of vision impairment and enables the visually impaired person to move about safely.
2. Related Work
3. System Principles
3.1. SLAM Mapping
- For each point in the map, we calculate its average distance d to the nearest K points. This calculation is iterated for each point in the input point cloud. Therefore, an array containing the d values of each point can be obtained, denoted as distance;
- For all the points in the input point cloud, it is assumed that the elements in the distance array follow a Gaussian distribution, the array is a sample, and the sample capacity is the number of points in the point cloud. The shape of the Gaussian distribution curve is determined by the mean and standard deviation of the sample, and the corresponding points whose d values are outside the standard range are considered outliers and will be removed.
3.2. Navigation and Path Planning
Algorithm 1: The proposed PPA Algorithm. |
3.3. YOLO-Based Object Detector
3.4. System Composition
4. Experimental Results
4.1. Experimental Setups
4.2. Map Building Experiments
4.2.1. Experimental Scene Selection
4.2.2. Field Simulation
4.2.3. Point Cloud Map Establishing Experiments
4.2.4. Dense Point Cloud Map and Occupied Grid Map Reconstructing Experiments
4.2.5. Experimental Data Analysis
4.3. Navigation
4.4. Object Detection
5. Work Limitations
6. Conclusions
Author Contributions
Funding
Institutional Review Board Statement
Informed Consent Statement
Data Availability Statement
Conflicts of Interest
References
- Rui, C.; Liu, Y.; Shen, J.; Li, Z.; Xie, Z. A Multi-Sensory Blind Guidance System Based on YOLO and ORB-SLAM. In Proceedings of the 2021 IEEE International Conference on Progress in Informatics and Computing (PIC), Shanghai, China, 17–19 December 2021; pp. 409–414. [Google Scholar] [CrossRef]
- Khan, S.; Nazir, S.; Khan, H.U. Analysis of navigation assistants for blind and visually impaired people: A systematic review. IEEE Access 2021, 9, 26712–26734. [Google Scholar] [CrossRef]
- Plikynas, D.; Žvironas, A.; Budrionis, A.; Gudauskis, M. Indoor navigation systems for visually impaired persons: Mapping the features of existing technologies to user needs. Sensors 2020, 20, 636. [Google Scholar] [CrossRef] [PubMed] [Green Version]
- Tapu, R.; Mocanu, B.; Zaharia, T. Wearable assistive devices for visually impaired: A state of the art survey. Pattern Recognit. Lett. 2020, 137, 37–52. [Google Scholar] [CrossRef]
- Islam, M.M.; Sadi, M.S.; Zamli, K.Z.; Ahmed, M.M. Developing walking assistants for visually impaired people: A review. IEEE Sensors J. 2019, 19, 2814–2828. [Google Scholar] [CrossRef]
- Haymes, S.A.; Johnston, A.W.; Heyes, A.D. Relationship between vision impairment and ability to perform activities of daily living. Ophthalmic Physiol. Opt. 2002, 22, 79–91. [Google Scholar] [CrossRef]
- Krueger, F.; Mitchell, K.C.; Deshpande, G.; Katz, J.S. Human–dog relationships as a working framework for exploring human–robot attachment: A multidisciplinary review. Anim. Cogn. 2021, 24, 371–385. [Google Scholar] [CrossRef]
- Jian-Yu, E.; Mihailovic, A.; Garzon, C.; Schrack, J.A.; Li, T.; West, S.K.; Friedman, D.S.; Gitlin, L.N.; Ramulu, P.Y. Importance and severity dependence of physical activity by GPS-tracked location in glaucoma patients. Am. J. Ophthalmol. 2021, 230, 276–284. [Google Scholar]
- Saud, S.N.; Raya, L.; Abdullah, M.I.; Isa, M.Z.A. Smart navigation aids for blind and vision impairment people. In Proceedings of the International Conference on Computational Intelligence in Information System; Springer: Berlin/Heidelberg, Germany, 2021; pp. 54–62. [Google Scholar]
- Sen, A.; Sen, K.; Das, J. Ultrasonic blind stick for completely blind people to avoid any kind of obstacles. In Proceedings of the 2018 IEEE Sensors, New Delhi, India, 28–31 October 2018; pp. 1–4. [Google Scholar]
- Gigl, T.; Janssen, G.J.; Dizdarevic, V.; Witrisal, K.; Irahhauten, Z. Analysis of a UWB indoor positioning system based on received signal strength. In Proceedings of the 4th IEEE Workshop on Positioning, Navigation and Communication, Hannover, Germany, 22–22 March 2007; pp. 97–101. [Google Scholar]
- Roy, P.; Chowdhury, C. A survey of machine learning techniques for indoor localization and navigation systems. J. Intell. Robot. Syst. 2021, 101, 63. [Google Scholar] [CrossRef]
- Taheri, H.; Xia, Z.C. SLAM; definition and evolution. Eng. Appl. Artif. Intell. 2021, 97, 104032. [Google Scholar] [CrossRef]
- Kanwal, N.; Bostanci, E.; Currie, K.; Clark, A.F. A navigation system for the visually impaired: A fusion of vision and depth sensor. Appl. Bionics Biomech. 2015, 2015, 479857. [Google Scholar] [CrossRef] [Green Version]
- Bai, J.; Lian, S.; Liu, Z.; Wang, K.; Liu, D. Virtual-blind-road following-based wearable navigation device for blind people. IEEE Trans. Consum. Electron. 2018, 64, 136–143. [Google Scholar] [CrossRef] [Green Version]
- Mur-Artal, R.; Montiel, J.M.M.; Tardos, J.D. ORB-SLAM: A versatile and accurate monocular SLAM system. IEEE Trans. Robot. 2015, 31, 1147–1163. [Google Scholar] [CrossRef] [Green Version]
- Chen, Z.; Liu, X.; Kojima, M.; Huang, Q.; Arai, T. A wearable navigation device for visually impaired people based on the real-time semantic visual SLAM system. Sensors 2021, 21, 1536. [Google Scholar] [CrossRef] [PubMed]
- Redmon, J.; Farhadi, A. YOLOv3: An incremental improvement. arXiv 2018, arXiv:1804.02767. [Google Scholar]
- Xu, Q.; Lin, R.; Yue, H.; Huang, H.; Yang, Y.; Yao, Z. Research on small target detection in driving scenarios based on improved YOLO network. IEEE Access 2020, 8, 27574–27583. [Google Scholar] [CrossRef]
- Rashid, A.T.; Ali, A.A.; Frasca, M.; Fortuna, L. Path planning with obstacle avoidance based on visibility binary tree algorithm. Robot. Auton. Syst. 2013, 61, 1440–1449. [Google Scholar] [CrossRef]
- Real, S.; Araujo, A. Navigation systems for the blind and visually impaired: Past work, challenges, and open problems. Sensors 2019, 19, 3404. [Google Scholar] [CrossRef] [Green Version]
- Simoes, W.C.; Machado, G.S.; Sales, A.M.; de Lucena, M.M.; Jazdi, N.; de Lucena, V.F., Jr. A review of technologies and techniques for indoor navigation systems for the visually impaired. Sensors 2020, 20, 3935. [Google Scholar]
- Budrionis, A.; Plikynas, D.; Daniusis, P.; Indrulionis, A. Smartphone-based computer vision travelling aids for blind and visually impaired individuals: A systematic review. Assist. Technol. 2020, 34, 178–194. [Google Scholar] [CrossRef]
- Peng, H.; Song, G.; You, J.; Zhang, Y.; Lian, J. An indoor navigation service robot system based on vibration tactile feedback. Int. J. Soc. Robot. 2017, 9, 331–341. [Google Scholar] [CrossRef]
- Zhao, Y.; Kupferstein, E.; Rojnirun, H.; Findlater, L.; Azenkot, S. The effectiveness of visual and audio wayfinding guidance on smartglasses for people with low vision. In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems; Association for Computing Machinery: New York, NY, USA, 2020; pp. 1–14. [Google Scholar]
- Romeo, K.; Pissaloux, E.; Gay, S.L.; Truong, N.T.; Djoussouf, L. The MAPS: Toward a novel mobility assistance system for visually impaired people. Sensors 2022, 22, 3316. [Google Scholar] [CrossRef] [PubMed]
- Metsiritrakul, K.; Suchato, A.; Punyabukkana, P. Obstacle avoidance feedback system for the blind using stereo sound. In Proceedings of the 11th International Convention on Rehabilitation Engineering and Assistive Technology, Kobe, Japan, 25–28 July 2017. [Google Scholar]
- Lee, C.W.; Chondro, P.; Ruan, S.J.; Christen, O.; Naroska, E. Improving mobility for the visually impaired: A wearable indoor positioning system based on visual markers. IEEE Consum. Electron. Mag. 2018, 7, 12–20. [Google Scholar] [CrossRef]
- Presti, G.; Ahmetovic, D.; Ducci, M.; Bernareggi, C.; Ludovico, L.; Barate, A.; Avanzini, F.; Mascetti, S. WatchOut: Obstacle sonification for people with visual impairment or blindness. In Proceedings of the 21st International ACM SIGACCESS Conference on Computers and Accessibility (ASSETS), Pittsburgh, PA, USA, 28–30 October 2019; pp. 402–413. [Google Scholar]
- Chang, W.J.; Chen, L.B.; Chen, M.C.; Su, J.P.; Sie, C.Y.; Yang, C.H. Design and implementation of an intelligent assistive system for visually impaired people for aerial obstacle avoidance and fall detection. IEEE Sens. J. 2020, 20, 10199–10210. [Google Scholar] [CrossRef]
- Masud, U.; Saeed, T.; Malaikah, H.M.; Islam, F.U.; Abbas, G. Smart assistive system for visually impaired people obstruction avoidance through object detection and classification. IEEE Access 2022, 10, 13428–13441. [Google Scholar] [CrossRef]
- Qiu, H.; Wang, C.; Wang, J.; Wang, N.; Zeng, W. Cross view fusion for 3D human pose estimation. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), Seoul, Korea, 27 October–2 November 2019; pp. 4342–4351. [Google Scholar]
- Ashish, D.; Munjal, S.; Mani, M.; Srivastava, S. Path finding algorithms. In Emerging Technologies in Data Mining and Information Security; Springer: Berlin/Heidelberg, Germany, 2021; pp. 331–338. [Google Scholar]
- Ahn, M.S.; Chae, H.; Noh, D.; Nam, H.; Hong, D. Analysis and noise modeling of the intel realsense D435 for mobile robots. In Proceedings of the 16th IEEE International Conference on Ubiquitous Robots (UR), Jeju, Korea, 24–27 June 2019; pp. 707–711. [Google Scholar]
- Macenski, S.; Foote, T.; Gerkey, B.; Lalancette, C.; Woodall, W. Robot Operating System 2: Design, architecture, and uses in the wild. Sci. Robot. 2022, 7, eabm6074. [Google Scholar] [CrossRef]
- Redmon, J.; Divvala, S.; Girshick, R.; Farhadi, A. You only look once: Unified, real-time object detection. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Las Vegas, NV, USA, 27–30 June 2016; pp. 779–788. [Google Scholar]
Type | Parameter |
---|---|
Camera | Intel RealSense D435 [34] |
Hardware Platform | Raspberry Pi 4B |
Operating System | Ubuntu 20.04 LTS |
Software Platform | ROS2 Galactic Geochelone [35] |
Name | Speed Specification | |
---|---|---|
Keyframes Number | Matches | |
A | 117 | 195 |
B | 85 | 408 |
C | 46 | 251 |
D | 100 | 176 |
Method | mAP | FPS |
---|---|---|
Fast R-CNN | 70.0 | 0.5 |
SSD321 | 45.4 | 16 |
YOLOv3-320 | 51.5 | 45 |
Our work | 55.3 | 35 |
Publisher’s Note: MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations. |
© 2022 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https://creativecommons.org/licenses/by/4.0/).
Share and Cite
Xie, Z.; Li, Z.; Zhang, Y.; Zhang, J.; Liu, F.; Chen, W. A Multi-Sensory Guidance System for the Visually Impaired Using YOLO and ORB-SLAM. Information 2022, 13, 343. https://doi.org/10.3390/info13070343
Xie Z, Li Z, Zhang Y, Zhang J, Liu F, Chen W. A Multi-Sensory Guidance System for the Visually Impaired Using YOLO and ORB-SLAM. Information. 2022; 13(7):343. https://doi.org/10.3390/info13070343
Chicago/Turabian StyleXie, Zaipeng, Zhaobin Li, Yida Zhang, Jianan Zhang, Fangming Liu, and Wei Chen. 2022. "A Multi-Sensory Guidance System for the Visually Impaired Using YOLO and ORB-SLAM" Information 13, no. 7: 343. https://doi.org/10.3390/info13070343