SOMN_IA: Portable and Universal Device for Real-Time Detection of Driver’s Drowsiness and Distraction Levels
Abstract
:1. Introduction
2. Related Works
3. Proposed System
3.1. Frame Separation (Stage-1)
3.2. Analysis of Consecutive Results (Stage 2)
3.2.1. Analysis of Consecutive Results (First Part)
3.2.2. Analysis of Consecutive Results (Second Part)
3.2.3. Drowsiness Detection Using S-CNN’s Prediction Results
3.3. Implementation of a S-CNN (Stage 3)
3.3.1. Database Construction
3.3.2. Architecture of the Proposed S-CNN
3.4. Isolated Prediction Error Correction (Stage 4)
3.5. Activation of Relevant Alarms
3.5.1. Alarms for Driver’s Distraction Level
3.5.2. Alarms for Driver’s Drowsiness Level
3.5.3. Information Transmitted by TS Alarm
4. Hardware Implementation
4.1. Components That Make Up the Front Part of the SOMN_IA
4.2. Components That Make Up the Back Part of the SOMN_IA
4.3. Internal Composition of SOMN_IA
4.4. Connection Diagram in the Vehicle
5. Experimental Results
5.1. Software Performance of the SOMN_IA
5.2. Global Performance of the SOMN_IA
5.3. Performance Comparison
5.4. Evaluation of Proposed System in Real-World Conditions
6. Conclusions
Supplementary Materials
Author Contributions
Funding
Informed Consent Statement
Data Availability Statement
Acknowledgments
Conflicts of Interest
References
- Mexico Ranked Seventh Worst Place in the Traffic Accident in the World (Spanish), Instituto Nacional de Salud Pública. Available online: https://www.insp.mx/avisos/4761-seguridad-vial-accidentes-transito.html (accessed on 22 May 2022).
- Causes of Traffic Accidents in Mexico (Spanish), by the National Institute of Statistics, Geographics and Informatics in Mexico (INEGI). Available online: https://www.inegi.org.mx/app/tabulados/interactivos/?px=ATUS_2&bd=ATUS&idrt=168&opc=t (accessed on 22 May 2022).
- Accidents and Their Causes (Spanish), Mexican National Committee for Security (CNS). Available online: http://www.cns.gob.mx/portalWebApp/appmanager/portal/desk?_nfpb=true&_pageLabel=portals_portal_page_m2p1p2&content_id=830068&folderNode=830052&folderNode1=810277 (accessed on 22 May 2022).
- 2019 Traffic Safety Culture Index, Foundation for Traffic Safety. 2020. Available online: https://aaafoundation.org/2019-traffic-safety-culture-index/ (accessed on 10 April 2022).
- Chacon-Murguia, M.; Prieto-Resendiz, C. Detecting driver drowsiness: A survey of system designs and technology. IEEE Consum. Electron. Mag. 2015, 4, 107–119. [Google Scholar] [CrossRef]
- Wang, J.; Zhu, S.; Gong, Y. Driving safety monitoring using semisupervised learning on time series data. IEEE Trans. Intell. Transp. Syst. 2010, 11, 728–737. [Google Scholar] [CrossRef]
- Wu, B.-F.; Chen, Y.-H.; Yeh, C.-H.; Li, Y.-F. Reasoning-based framework for driving safety monitoring using driving event recognition. IEEE Trans. Intell. Transp. Syst. 2013, 14, 1231–1241. [Google Scholar] [CrossRef]
- Kokonozi, A.K.; Michail, E.M.; Chouvarda, I.C.; Maglaveras, N.M. A study of heart rate and brain system complexity and their interaction in sleep-deprived subjects. In Proceedings of the Computing in Cardiology, Bolonya, Italia, 14 September 2008. [Google Scholar]
- Vidente, V.; Laguna, P.; Bartra, A.; Bailón, R. Drowsiness detection using heart rate variability. Med. Biol. Eng. Comput. 2016, 54, 827–937. [Google Scholar]
- Zhang, C.; Wang, H.; Fu, R. Automated detection of driver fatigue based on entropy and complexity measures. IEEE Trans. Intell. Transp. Syst. 2014, 15, 168–177. [Google Scholar] [CrossRef]
- Flores-Monroy, J.; Nakano-Miyatake, M.; Perez-Meana, H.; Sanchez-Perez, G. Visual-based real time driver drowsiness detection system using CNN. In Proceedings of the International Conference on Electrical Engineering, Computing Science and Automatic Control, IEEE, Mexico City, Mexico, 10 November 2021. [Google Scholar]
- Flores-Monroy, J.; Nakano-Miyatake, M.; Perez-Meana, H.; Escamilla-Hernandez, E.; Sanchez-Perez, G. A CNN-based driver’s drowsiness and distraction detection system. In Proceedings of the 14th Mexican Conference on Pattern Recognition, Chihuahua, Mexico, 22–25 June 2022. [Google Scholar]
- Phan, A.-C.; Nguyen, N.-H.-Q.; Trieu, T.-N.; Phan, T.-C. An Efficient Approach for Detecting Driver Drowsiness Based on Deep Learning. Appl. Sci. 2021, 11, 8441. [Google Scholar] [CrossRef]
- Tamanani, R.; Muresan, R.; Al-Dweik, A. Estimation of driver vigilance status using real-time facial expression and deep learning. IEEE Sens. Lett. 2021, 5, 6000904. [Google Scholar] [CrossRef]
- Viola, P.; Jones, M. Robust real-time face detection. Int. J. Comput. Vis. 2004, 57, 137–154. [Google Scholar] [CrossRef]
- Bazarevsky, V.; Kartynnik, Y.; Vakunov, A.; Raveendran, K.; Grundmann, M. BlazeFace: Sub-millisecond Neural Face Detection on Mobile GPUs. In Proceedings of the Computer Vision & Pattern Recognition, Long Beach, CA, USA, 15–20 June 2019. [Google Scholar]
- Anber, S.; Alsaggaf, W.; Shalash, W. A Hybrid Driver Fatigue and Distraction Detection Model Using AlexNet Based on Facial Features. Electronics 2022, 11, 285. [Google Scholar] [CrossRef]
- Magán, E.; Paz-Sesmero, M.; Alonso-Weber, J.M.; Sanchis, A. Driver Drowsiness Detection by Applying Deep Learning Techniques to Sequences of Images. Appl. Sci. 2022, 12, 1145. [Google Scholar] [CrossRef]
- Jabbar, R.; Al-Khalifa, K.; Kharbeche, M.; Alhajyaseen, W.; Jafari, M.; Jiang, S. Real-time Driver Drowsiness Detection for Android Application Using Deep Neural Networks Techniques. Procedia Comput. Sci. 2018, 130, 400–407. [Google Scholar] [CrossRef]
- Uma, S.; Eswari, R. Accident prevention and safety assistance using IOT and machine learning. J. Reliab. Intell. Environ. 2021, 8, 79–103. [Google Scholar] [CrossRef]
- Fernández-Villán, A.; Usamentiaga-Fernández, R.; Casado-Tejedor, R. Sistema Automático Para la Detección de Distracción y Somnolencia en Conductores por Medio de Características Visuales Robustas. Rev. Iberoam. Autom. Inform. Ind. 2017, 14, 307–328. [Google Scholar] [CrossRef]
- Pattarapongsin, P.; Neupane, B.; Vorawan, J.; Sutthikulsombat, H.; Horanont, T. Real-time drowsiness and distraction detection using computer vision and deep learning. In Proceedings of the ACM International Conference Proceeding Series, 1, Toronto, ON, Canada, 22–24 July 2020. [Google Scholar]
- Hashemi, M.; Mirrashid, A.; Shirazi, A.B. Driver safety development: Real-time driver drowsiness detection system based on convolutional neural network. SN Comput. Sci. 2020, 8, 1–10. [Google Scholar] [CrossRef]
- Pan, G.; Sun, L.; Wu, Z.; Lao, S. Eyeblink-based anti-spoofing in face recognition from a generic webcamera. In Proceedings of the IEEE International Conference on Computer Vision, Rio de Janeiro, Brazil, 14–21 October 2007; pp. 1–8. [Google Scholar]
- Kwon, K.A.; Shipley, R.; Edirisinghe, M.; Ezra, D.G.; Rose, G.; Best, S.M.; Cameron, R.E. High-speed camera characterization of voluntary eye blinking kinematics. J. R. Soc. Interface 2013, 10, 1–6. [Google Scholar] [CrossRef] [PubMed]
- Weng, C.H.; Lai, Y.H.; Lai, S.H. Driver drowsiness detection via a hierarchical temporal deep belief network. In Proceedings of the Asian Conference on Computer Vision, Taipei, Taiwan, 20–24 November 2016; pp. 117–133. [Google Scholar]
- Sandler, M.; Howard, A.; Zhu, M.; Zhmoginov, A.; Chen, L. MobileNetV2: Inverted residu-als and linear bottlenecks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA, 18–23 June 2018; pp. 4510–4520. [Google Scholar]
- Simonyan, K.; Zisserman, A. Very deep convolutional networks for large-scale image recognition. In Proceedings of the International Conference on Learning Representations (ICLR), San Diego, CA, USA, 7–9 May 2015. [Google Scholar]
- He, K.; Zhang, X.; Ren, S.; Sun, J. Deep residual learning for image recognition. In Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, Las Vegas, NV, USA, 27–29 June 2016; pp. 770–778. [Google Scholar]
- Szegedy, C.; Liu, W.; Jia, Y.; Sermanet, P.; Reed, S.; Anguelov, D.; Erhan, D.; Vanhouckel, V.; Rabinovich, A. Going Deeper with Convolutions. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Boston, MA, USA, 7–12 June 2015; pp. 1–9. [Google Scholar]
- Chollet, F. Xception: Deep Learning with Depthwise Separable Convolutions. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Honolulu, HI, USA, 21–26 July 2017; pp. 1800–1807. [Google Scholar]
- Ghoddoosian, R.; Galib, M.; Athitsos, V. A Realistic Dataset and Baseline Temporal Model for Early Drowsiness Detection. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Long Beach, CA, USA, 15–20 June 2019; pp. 178–187. [Google Scholar]
Variable | 2016 | 2017 | 2018 | 2019 | 2020 |
---|---|---|---|---|---|
Full events (absolute number) | 360,051 | 367,789 | 365,281 | 362,729 | 301,678 |
Driver | 91% | 91% | 92% | 92% | 95% |
Pedestrian or passenger | 1% | 1% | 1% | 1% | 1% |
Vehicle failure | 1% | 1% | 1% | 1% | 1% |
Bad road condition | 3% | 3% | 3% | 3% | 2% |
Other causes | 4% | 4% | 3% | 3% | 1% |
Face Without Drowsy Features | Face with Drowsy Features | Face with Distraction Features |
---|---|---|
Models | Architecture | Additional Settings | Model Size | Trainable Params | Accuracy (%) | |
---|---|---|---|---|---|---|
Selected Model | Input Image (64,64,3) 3 × (2D Conv, BN, 2 × 2 MaxPooling) Flatten, 2 × FC (128), Dp (0.2), FC (3) | 2D Conv | 32 | 3.98 MB | 340,835 | 95.77 |
32 | ||||||
64 | ||||||
Max Pooling | 6 × 6 × 64 | |||||
31 × 31 × 32 | ||||||
14 × 14 × 32 | ||||||
Model 2 | Image input (64,64,3) 3 × (2D Conv, BN, 2 × 2 MaxPooling) Flatten, 2 × FC (128), FC (3) | 2D Conv | 32 | 5.42 MB | 700,547 | 95.77 |
32 | ||||||
128 | ||||||
Max Pooling | 31 × 31 × 32 | |||||
14 × 14 × 64 | ||||||
6 × 6 × 128 | ||||||
Model 3 | Image input (64,64,1) 3 × (2D Conv, BN, 2 × 2 MaxPooling) Flatten, 2 × FC (128), Dp (0.2), FC (3) | 2D Conv | 32 | 3.98 MB | 340,259 | 95.5 |
32 | ||||||
64 | ||||||
Max Pooling | 31 × 31 × 32 | |||||
14 × 14 × 32 | ||||||
6 × 6 × 64 |
Model | Accuracy | Input Image Size | Model Size (MB) | Total Params | Operation Speed (fps) | |
---|---|---|---|---|---|---|
Mediapipe | Haar | |||||
Proposed | 0.9577 | 64 × 64 | 3.98 | 341,091 | 21 | 21 |
MobileNetV2 [27] | 0.9675 | 64 × 64 | 10.30 | 2,590,083 | 17 | 16 |
VGG16 [28] | 0.9777 | 64 × 64 | 56.80 | 14,850,179 | 15 | 16 |
ResNet50 V2 [29] | 0.9760 | 64 × 64 | 92.50 | 24,116,419 | 14 | 13 |
InceptionV3 [30] | 0.9600 | 75 × 75 | 84.50 | 21,938,275 | 13 | 12 |
Xception [31] | 0.9713 | 64 × 64 | 81.90 | 21,390,187 | 16 | 12 |
Condition | Detection Accuracy (%) |
---|---|
Without glasses under normal illumination | 96.67 |
With glasses under normal illumination | 95.83 |
Without glasses under IR illumination | 95.91 |
With glasses under IR illumination | 94.67 |
Global performance | 95.77 |
System/Research | Accuracy | Monitoring in Any Lighting Condition | |||||
---|---|---|---|---|---|---|---|
Drowsiness | Distraction | FPS | Hardware Implementation | Database | Real Time | Video | |
Proposed | 0.9506 | 0.9638 | 21 | SOMN_IA (Portable specific device) | NTHU-DDD | ✓ | ✓ |
[13] | 0.97 | × | - | No | Self-built | × | ✓ |
[14] | 0.88 | × | 8.4 | No | UTA-RLDD + Self-built | × | ✓ |
[17] | 0.9965 | 15 | No | NTHU-DDD | × | ✓ | |
[18] | 0.93 | × | 10 | No | UTA-RLDD | × | × |
[19] | 0.81 | × | 7 | Mobile device (Android Smartphone) | NTHU-DDD | × | ✓ |
[20] | 0.965 | × | 15 | Raspberry Pi + cloud computing | Self-built | ✓ | ✓ |
[21] | 0.9311 | × | 10 | Mobile device (Smartphone) | CEW | × | ✓ |
[22] | 0.95 | × | - | JetsonNano + GPU server | FDDB WFLW | × | ✓ |
[23] | 0.9815 | × | - | No | ZJU Eyeblink dataset | × | ✓ |
Publisher’s Note: MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations. |
© 2022 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https://creativecommons.org/licenses/by/4.0/).
Share and Cite
Flores-Monroy, J.; Nakano-Miyatake, M.; Escamilla-Hernandez, E.; Sanchez-Perez, G.; Perez-Meana, H. SOMN_IA: Portable and Universal Device for Real-Time Detection of Driver’s Drowsiness and Distraction Levels. Electronics 2022, 11, 2558. https://doi.org/10.3390/electronics11162558
Flores-Monroy J, Nakano-Miyatake M, Escamilla-Hernandez E, Sanchez-Perez G, Perez-Meana H. SOMN_IA: Portable and Universal Device for Real-Time Detection of Driver’s Drowsiness and Distraction Levels. Electronics. 2022; 11(16):2558. https://doi.org/10.3390/electronics11162558
Chicago/Turabian StyleFlores-Monroy, Jonathan, Mariko Nakano-Miyatake, Enrique Escamilla-Hernandez, Gabriel Sanchez-Perez, and Hector Perez-Meana. 2022. "SOMN_IA: Portable and Universal Device for Real-Time Detection of Driver’s Drowsiness and Distraction Levels" Electronics 11, no. 16: 2558. https://doi.org/10.3390/electronics11162558
APA StyleFlores-Monroy, J., Nakano-Miyatake, M., Escamilla-Hernandez, E., Sanchez-Perez, G., & Perez-Meana, H. (2022). SOMN_IA: Portable and Universal Device for Real-Time Detection of Driver’s Drowsiness and Distraction Levels. Electronics, 11(16), 2558. https://doi.org/10.3390/electronics11162558