Drowsiness Detection System Based on PERCLOS and Facial Physiological Signal †
Abstract
:1. Introduction
2. Technical Background
2.1. Physiological Signal of PPGI
2.2. Heart Rate Variability Analysis
2.3. PRVA
2.4. Autonomic Nervous System
2.5. LF/HF Ratio Analysis
2.6. Eyes Detection
2.6.1. Open and Closed Eyes Detection Method
2.6.2. Percentage of Eyelid Closure over the Pupil over Time (PERCLOS)
2.7. Brain Wave Analysis
3. Proposed System Architecture
3.1. System Architecture
3.2. Red/Green Channel of PPGI
3.3. Determination of the ROI
3.4. PRVA
3.4.1. PPI Detection
3.4.2. Power Density Spectrum
3.5. Open and Closed Eyes Detection
3.5.1. Enter Sleep State
3.5.2. Judgment Conditions
- (1)
- Condition 1: LF/HF ratio(Now)
- (2)
- Condition 2: LF/HF ratio(Now)—LF/HF ratio(Avg)
- (3)
- Condition 3: PERCLOS
3.5.3. Drowsiness Judgment Analysis
4. Experimental Evaluation
4.1. Experimental Setting
4.2. Webcam Specifications and Controls
4.3. Experimental Flow
4.4. Experimental Results
5. Conclusions
Author Contributions
Funding
Institutional Review Board Statement
Informed Consent Statement
Data Availability Statement
Acknowledgments
Conflicts of Interest
References
- The National Highway Traffic Safety Administration(NHTSA). Available online: http://www.nhtsa.gov/ (accessed on 1 July 2017).
- Nordbakke, S.; Sagberg, F. Sleepy at the wheel: Knowledge, symptoms and 51 behaviour among car drivers. Transp. Res. Part F 2007, 10, 1–10. [Google Scholar] [CrossRef]
- Cho, D.; Lee, B. Non-contact robust heeart rate estimation using HSV color model and matrix based IIR filter in the face video imaging. In Proceedings of the 38th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC), Orlando, FL, USA, 16–20 August 2016; pp. 3847–3850. [Google Scholar]
- Qian, D.; Wang, B.; Qing, X.; Zhang, T.; Zhang, Y.; Wang, X. Bayesian nonnegative CP decomposition-based feature extraction algorithm for drowsiness detection. IEEE Trans. Neural Syst. Rehabil. Eng. 2017, 25, 1297–1308. [Google Scholar] [CrossRef] [PubMed]
- Fujiwara, K.; Abe, E.; Kamata, K.; Nakayama, C.; Suzuki, Y.; Yamakawa, T.; Hiraoka, T.; Kano, M.; Sumi, Y.; Masuda, F.; et al. Heart rate vartiability-based driver drowsiness detection and its validation with EEG. IEEE Trans. Biomed. Eng. 2019, 66, 1769–1778. [Google Scholar] [CrossRef] [PubMed]
- Guo, B.N.; Lin, C.Y. A study to vital signs measured using a webcam. JITA 2012, 6, 112–118. [Google Scholar]
- Chang, R.C.-H.; Wang, C.-Y.; Kao, Y.-Y. Implementation of a novel intelligent drowsiness detection warning system. In Proceedings of the 2021 IEEE International Conference on Consumer Electronics, Penghu, Taiwan, 15–17 September 2021. [Google Scholar]
- Sunagawa, M.; Shikii, S.I.; Nakai, W.; Mochizuki, M.; Kusukame, K.; Kitajima, H. Comprehensive drowsiness level detection model combining multimodal information. IEEE Sens. J. 2020, 20, 3709–3717. [Google Scholar] [CrossRef]
- Chang, C.W. Hough Transform Based Drowsy Driver Detection System and Embedded System Implementation. Master’s Thesis, Nat. Chung Hsing University, Taichung, Taiwan, 2014. [Google Scholar]
- Chen, C.H. The Study of the Variation Trend for Diastolic Pressure of the Surgical Patients utilizing Non-Invasive Plethysmography Signal. Master’s Thesis, Sun Yat-sen Univiversity, Kaogsiung, Taiwan, 2004. [Google Scholar]
- Lin, C.H. Contact-Free Measurement of Cardiac Pulse Using IR Video Imaging. Master’s thesis, Nat. Chung Hsing University, Taichung, Taiwan, 2015. [Google Scholar]
- Tarassenko, L.; Villarroel, M.; Guazzi, A.; Jorge, J.; Clifton, D.; Pugh, C. Non-contact video-based vital sign monitoring using ambient light and auto-regressive models. Physiol. Meas. 2014, 35, 807. [Google Scholar] [CrossRef]
- Poh, M.-Z.; McDuff, D.J.; Picard, R.W. Non-contact, automated cardiac pulse measurements using video imaging and blind source separation. Opt. Express 2010, 18, 10762–10774. [Google Scholar] [CrossRef] [PubMed]
- Reed, M.J.; Robertson, C.; Addison, P. Heart rate variability measurements and the prediction of ventricular arrhythmias. Qjm 2005, 98, 87–95. [Google Scholar] [CrossRef] [PubMed] [Green Version]
- Akselrod, S.; Gordon, D.; Ubel, F.A.; Shannon, D.C.; Barger, A.C.; Cohen, R.J. Power spectrum analysis of heart rate fluctuation: A quantitative probe of beat-to-beat cardiovascular control. Science 1981, 213, 220–222. [Google Scholar] [CrossRef]
- Bigger, J.T.; Fleiss, J.L.; Steinman, R.C.; Rolnitzky, L.M.; Kleiger, R.E.; Rottman., J.N. Correlations among time and frequency domain measures of heart period variability two weeks after acute myocardial infarction. Am. J. Cardiol. 1992, 69, 891–898. [Google Scholar] [CrossRef]
- Task Force of the European Society of Cardiology the North American Society of Pacing Electrophysiology-Heart Rate Variability. Available online: https://circ.ahajournals.org/content/93/5/1043 (accessed on 1 July 2017).
- Wang, K.P.; Ho, T.Y.; Ou, S.F.; Lin, C.C.; Hsieh, K.S. Analysis of Heart Rhythm Variability. Taiwan Med. J. 2009, 52, 290–293. [Google Scholar]
- Pinheiro, N.; Couceiro, R.; Henriques, J.; Muehlsteff, J.; Quintal, I.; Gonçalves, L.; Carvalho, P. Can PPG be used for HRV analysis? In Proceedings of the 2016 IEEE 38th Annual International Conference of the Engineering in Medicine and Biology Society (EMBC), Orlando, FL, USA, 6–20 August 2016; pp. 2945–2949. [Google Scholar]
- Sengthipphany, T.; Tretriluxana, S.; Chitsakul, K. Comparison of Heart Rate statistical parameters from Photoplethysmographic signal in resting and exercise conditions. In Proceedings of the Electrical Engineering/Electronics, Computer, Telecommunications and Information Technology (ECTI-CON), 2015 12th International Conference, Hua Hin, Thailand, 24–27 June 2015; pp. 1–5. [Google Scholar]
- KingNet National Internet Hospital “Sympathetic Nerve” and “Parasympathetic Nerve”. Available online: http://hospital.kingnet.com.tw/essay/essay.html?pid=18479 (accessed on 1 May 2022).
- Autonomic Nervous Medicine Association Health Education Manual for HRV Detection and Treatment of Autonomic Nervous Disorders. Available online: http://hrvtw.blogspot.tw/2010/09/hrvq.html (accessed on 1 May 2022).
- Heseltine, T.; Pears, N.; Austin, J. Evaluation of image pre-processing techniques for eigenface based face recognition. In Proceedings of the Second International Conference on Image and Graphics (SPIE), Heifei, China, 16–18 August 2002; pp. 677–685. [Google Scholar]
- Park, I.; Ahn, J.-H.; Byun, H. Efficient measurement of eye blinking under various illumination conditions for drowsiness detection systems. In Proceedings of the ICPR 2006—18th International Conference on Pattern Recognition, HongKong, 20–24 August 2006; pp. 383–386. [Google Scholar]
- Sivanath, S.K.; Muralikrishnan, S.A.; Thothadri, P.; Raja, V. Eyeball and blink controlled firing system for military tank using LabVIEW. In Proceedings of the 2012 4th International Conference on Intelligent Human Computer Interaction (IHCI), Kharagpur, India, 27–29 December 2012; pp. 1–4. [Google Scholar]
- International Organization of Societies for Electrophysiological Technology. Available online: https://www.oset.org/Guidelines.html (accessed on 1 May 2022).
- Nayak, C.S.; Anilkumar, A.C. EEG Normal Waveforms. [Updated 8 May 2022]. In StatPearls [Internet]; StatPearls Publishing: Treasure Island, FL, USA, 2022. [Google Scholar]
- Britton, J.W.; Frey, L.C.; Hopp, J.L. Electroencephalography (EEG): An Introductory Text and Atlas of Normal and Abnormal Findings in Adults, Children, and Infants [Internet]; St. Louis, E.K., Frey, L.C., Eds.; American Epilepsy Society: Chicago, IL, USA, 2016. [Google Scholar]
- Knyazev, G.G.; Slobodskoj-Plusnin, J.Y.; Bocharov, A.V. Event-related delta and theta synchronization during explicit and implicit emotion processing. Neuroscience 2009, 164, 1588–1600. [Google Scholar] [CrossRef] [PubMed]
- Putman, P.; van Peer, J.; Maimari, I.; van der Werff, S. EEG theta/beta ratio in relation to fear-modulated response-inhibition, attentional control, and affective traits. Biol. Psychol. 2010, 83, 73–78. [Google Scholar] [CrossRef] [PubMed]
- Del Giudice, R.; Blume, C.; Wislowska, M.; Wielek, T.; Heib, D.P.; Schabus, M. The Voice of Anger: Oscillatory EEG Responses to Emotional Prosody. PLoS ONE 2016, 11, e0159429. [Google Scholar] [CrossRef] [PubMed]
- Keller, A.S.; Payne, L.; Sekuler, R. Characterizing the roles of alpha and theta oscillations in multisensory attention. Neuropsychologia 2017, 99, 48–63. [Google Scholar] [CrossRef] [PubMed] [Green Version]
- Wascher, E.; Rasch, B.; Sänger, J.; Hoffmann, S.; Schneider, D.; Rinkenauer, G.; Heuer, H.; Gutberlet, I. Frontal theta activity reflects distinct aspects of mental fatigue. Biol. Psychol. 2013, 96, 57–65. [Google Scholar] [CrossRef] [PubMed]
- Klimesch, W. EEG alpha and theta oscillations reflect cognitive and memory performance: A review and analysis. Brain Res. Rev. 1999, 29, 169–195. [Google Scholar] [CrossRef]
- Van Lier, H.; Drinkenburg, W.H.; Van Eeten, Y.J.; Coenen, A.M. Effects of diazepam and zolpidem on EEG. beta frequencies are behavior-specific in rats. Neuropharmacology 2004, 47, 163–174. [Google Scholar] [CrossRef] [PubMed]
- NeuroSky, Inc. Brain Wave Signal (EEG); NeuroSky, Inc.: San Jose, CA, USA, 2009. [Google Scholar]
- Shih, M.H. Design and Embedded System Implementation of Driver Eye Detection Algorithm with Facial Features. Master’s Thesis, Nat. Chung Hsing University, Taichung, Taiwan, 2014. [Google Scholar]
- Islam, M.S. Local gradient pattern-A novel feature representation for facial expression recognition. JAIDM 2014, 2, 33–38. [Google Scholar]
- Szypulska, M.; Piotrowski, Z. Prediction of fatigue and sleep onset using HRV analysis. In Proceedings of the 19th International Conference Mixed Design of Integrated Circuits and Systems-MIXDES 2012, Warsaw, Poland, 24–26 May 2012; pp. 543–546. [Google Scholar]
- Huikuri, H.V.; Pikkuja, S.M.; Airaksinen, K.J.; Ika, M.J.; Rantala, A.O.; Kauma, H.; Kesa Niemi, Y.A. Sex-related differences in autonomic modulation of heart rate in middle-aged subjects. Circulation 1996, 94, 122–125. [Google Scholar] [CrossRef] [PubMed]
Index | Definition | Clinical Significance |
---|---|---|
Total Power, TP (ms2) | Set the sampling frequency to ≤0.4 Hz, and sum the variation frequency bands of all normal heartbeat intervals | Global Heart Rate Variability Assessment |
Very Low Frequency Power, VLFP (ms2) | Set the sampling frequency to ≤0.04 Hz, and the variation in the normal heartbeat interval in the very low frequency range | Physiological significance is unknown |
Low Frequency Power, LFP (ms2) | Set the sampling frequency to 0.04~0.15 Hz, and the variation of the normal heartbeat interval in the low frequency range | Sympathetic nerve activity |
High Frequency Power, HFP (ms2) | Set the sampling frequency to 0.15~0.4 Hz, and the variation of the normal heartbeat interval in the high frequency range | Parasympathetic nerve activity |
LF/HF | The ratio of high and low frequency power | Balance of autonomic nervous activity |
Classification | PERCLOS Value |
---|---|
Awake | PERCLOS < 0.075 |
Questionable | 0.075 < PERCLOS < 0.15 |
Drowsy | PERCLOS > 0.15 |
PERCLOS Index | Definition |
---|---|
P70 | Pupil coverage over 70% |
P80 | Pupil coverage over 80% |
EM | Pupil coverage over 50% |
Mode | Range |
---|---|
1 | X5 < Ratio |
2 | X4 < Ratio < X5 |
3 | X3 < Ratio < X4 |
4 | X2 < Ratio < X3 |
5 | X1 < Ratio < X2 |
6 | X0 < Ratio < X1 |
Mode | Range |
---|---|
1 | β1 × SD < Diff |
2 | β2 × SD < Diff ≤ β1 × SD |
3 | β3 × SD < Diff ≤ β2 × SD |
4 | β4 × SD < Diff ≤ β3 × SD |
5 | Diff ≤ β4 × SD |
Mode | Range |
---|---|
0 | PERCLOS < 180 |
1 | 180 < PERCLOS < 270 |
2 | 270 < PERCLOS |
Condition 2 | 1 | 2 | 3 | 4 | 5 | |
---|---|---|---|---|---|---|
Condition 1 | ||||||
1 | 1 | 1 | 1 | 1 | 1 | |
2 | 1 | 1 | 1 | 2 | 2 | |
3 | 1 | 2 | 2 | 2 | 2 | |
4 | 2 | 2 | 2 | 3 | 3 | |
5 | 2 | 3 | 3 | 3 | 3 | |
6 | 3 | 3 | 3 | 3 | 3 |
Drowsiness Level | Condition |
---|---|
0 | 1. When the previous level 2 is level 0, the judgment is changed to level 0.2. When the proportion of indicator 1 appears ≥66%. |
1 | 1. When index1 is not found.2. When the proportion of index 3 is ≥33% and the total proportion of index 2 is ≥50%. |
2 | 1. When the proportion of index 3 is ≥66% and the total proportion of index 2 is ≥80%.2. Change the judgment to level 2 when three levels of level 1 are observed in a row. |
Condition 3 | 0 | 1 | 2 | |
---|---|---|---|---|
Level | ||||
0 | 0 | 0 | 1 | |
1 | 0 | 1 | 2 | |
2 | 1 | 2 | 2 |
Chipset | SMI |
---|---|
Resolution | 640 × 480 |
Output format | YCbCr422 |
Infrared wavelength | 850 ± 50 nm |
Output interface | USB 2.0 |
Sample rates (kHz) | 30 |
Predicted | Awake Judgment | Doze Judgment | |
---|---|---|---|
Actual | |||
Wide awake | 8 | 1 | |
Doze | 2 | 29 | |
Sensitivity is 88.9% | |||
Specificity is 93.5% | |||
Positive predictive value is 80% | |||
System Accuracy is 92.5% |
Publisher’s Note: MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations. |
© 2022 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https://creativecommons.org/licenses/by/4.0/).
Share and Cite
Chang, R.C.-H.; Wang, C.-Y.; Chen, W.-T.; Chiu, C.-D. Drowsiness Detection System Based on PERCLOS and Facial Physiological Signal. Sensors 2022, 22, 5380. https://doi.org/10.3390/s22145380
Chang RC-H, Wang C-Y, Chen W-T, Chiu C-D. Drowsiness Detection System Based on PERCLOS and Facial Physiological Signal. Sensors. 2022; 22(14):5380. https://doi.org/10.3390/s22145380
Chicago/Turabian StyleChang, Robert Chen-Hao, Chia-Yu Wang, Wei-Ting Chen, and Cheng-Di Chiu. 2022. "Drowsiness Detection System Based on PERCLOS and Facial Physiological Signal" Sensors 22, no. 14: 5380. https://doi.org/10.3390/s22145380
APA StyleChang, R. C.-H., Wang, C.-Y., Chen, W.-T., & Chiu, C.-D. (2022). Drowsiness Detection System Based on PERCLOS and Facial Physiological Signal. Sensors, 22(14), 5380. https://doi.org/10.3390/s22145380