Machine Learning Models and Videos of Facial Regions for Estimating Heart Rate: A Review on Patents, Datasets, and Literature
Abstract
:1. Introduction
2. Materials and Methods
2.1. Planning
2.2. Defining the Scope
2.3. Searching the Literature
2.4. Assessing the Evidence Base
2.5. Synthesizing and Analyzing
3. Results And Discussion
3.1. Patent landscape
3.2. Datasets
3.3. Article Landscape
3.3.1. Bibliometric Analysis
3.3.2. Use Cases
3.3.3. Models
3.3.4. Comparison of Metrics
4. Conclusions
Author Contributions
Funding
Acknowledgments
Conflicts of Interest
References
- Rahman, H.; Ahmed, M.U.; Barua, S.; Begum, S. Non-contact-based driver’s cognitive load classification using physiological and vehicular parameters. Biomed. Signal Process. Control 2020, 55, 101634. [Google Scholar] [CrossRef]
- Schneider, C.; Hanakam, F.; Wiewelhove, T.; Döweling, A.; Kellmann, M.; Meyer, T.; Pfeiffer, M.; Ferrauti, A. Heart rate monitoring in team sports—A conceptual framework for contextualizing heart rate measures for training and recovery prescription. Front. Physiol. 2018, 9, 639. [Google Scholar] [CrossRef] [PubMed] [Green Version]
- Sharma, R.; Goel, D.; Srivastav, M.; Dhasmana, R. Differences in Heart Rate and Galvanic Skin Response among Nurses Working in Critical and Non-Critical Care Units. J. Clin. Diagn. Res. 2018, 12, CC09–CC12. [Google Scholar] [CrossRef]
- Bertilsson, J.; Niehorster, D.C.; Fredriksson, P.; Dahl, M.; Granér, S.; Fredriksson, O.; Mårtensson, J.; Magnusson, M.; Fransson, P.A.; Nyström, M. Towards systematic and objective evaluation of police officer performance in stressful situations. Police Pract. Res. 2020, 21, 655–669. [Google Scholar] [CrossRef] [Green Version]
- Siddharth, S.; Trivedi, M.M. On Assessing Driver Awareness of Situational Criticalities: Multi-modal Bio-Sensing and Vision-Based Analysis, Evaluations, and Insights. Brain Sci. 2020, 10, 46. [Google Scholar] [CrossRef] [Green Version]
- Yu, W.; Ding, S.; Yue, Z.; Yang, S. Emotion Recognition from Facial Expressions and Contactless Heart Rate Using Knowledge Graph. In Proceedings of the 2020 IEEE International Conference on Knowledge Graph (ICKG), Nanjing, China, 9–11 August 2020; pp. 64–69. [Google Scholar] [CrossRef]
- Young, H.A.; Benton, D. Heart-rate variability: A biomarker to study the influence of nutrition on physiological and psychological health? Behav. Pharmacol. 2018, 29, 140. [Google Scholar] [CrossRef] [Green Version]
- Colasante, T.; Malti, T. Resting heart rate, guilt, and sympathy: A developmental psychophysiological study of physical aggression. Psychophysiology 2017, 54, 1770–1781. [Google Scholar] [CrossRef]
- Du, G.; Long, S.; Yuan, H. Non-Contact Emotion Recognition Combining Heart Rate and Facial Expression for Interactive Gaming Environments. IEEE Access 2020, 8, 11896–11906. [Google Scholar] [CrossRef]
- Hsu, G.S.J.; Xie, R.C.; Ambikapathi, A.; Chou, K.J. A deep learning framework for heart rate estimation from facial videos. Neurocomputing 2020, 417, 155–166. [Google Scholar] [CrossRef]
- Szankin, M.; Kwasniewska, A.; Sirlapu, T.; Wang, M.; Ruminski, J.; Nicolas, R.; Bartscherer, M. Long Distance Vital Signs Monitoring with Person Identification for Smart Home Solutions. In Proceedings of the 2018 40th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC), Honolulu, HI, USA, 18–21 July 2018; Volume 2018, pp. 1558–1561. [Google Scholar] [CrossRef]
- Song, R.; Zhang, S.; Li, C.; Zhang, Y.; Cheng, J.; Chen, X. Heart rate estimation from facial videos using a spatiotemporal representation with convolutional neural networks. IEEE Trans. Instrum. Meas. 2020, 69, 7411–7421. [Google Scholar] [CrossRef]
- Martis, R.J.; Acharya, U.R.; Adeli, H. Current methods in electrocardiogram characterization. Comput. Biol. Med. 2014, 48, 133–149. [Google Scholar] [CrossRef] [PubMed]
- Allen, J. Photoplethysmography and its application in clinical physiological measurement. Physiol. Meas. 2007, 28, R1–R39. [Google Scholar] [CrossRef] [PubMed] [Green Version]
- Qiu, Y.; Liu, Y.; Arteaga-Falconi, J.; Dong, H.; Saddik, A.E. EVM-CNN: Real-Time Contactless Heart Rate Estimation From Facial Video. IEEE Trans. Multimed. 2019, 21, 1778–1787. [Google Scholar] [CrossRef]
- Huang, B.; Chang, C.M.; Lin, C.L.; Chen, W.; Juang, C.F.; Wu, X. Visual Heart Rate Estimation from Facial Video Based on CNN. In Proceedings of the 2020 15th IEEE Conference on Industrial Electronics and Applications (ICIEA), Kristiansand, Norway, 9–13 November 2020; pp. 1658–1662. [Google Scholar] [CrossRef]
- Nowara, E.; McDuff, D. Combating the Impact of Video Compression on Non-Contact Vital Sign Measurement Using Supervised Learning. In Proceedings of the 2019 IEEE/CVF International Conference on Computer Vision Workshop (ICCVW), Seoul, Korea, 27–28 October 2019; pp. 1706–1712. [Google Scholar] [CrossRef]
- Yazdani, S.; Fallet, S.; Vesin, J.M. A Novel Short-Term Event Extraction Algorithm for Biomedical Signals. IEEE Trans. Biomed. Eng. 2018, 65, 754–762. [Google Scholar] [CrossRef]
- Goodfellow, I.; Bengio, Y.; Courville, A.; Bengio, Y. Deep Learning; MIT Press: Cambridge, UK, 2016; Volume 1. [Google Scholar]
- Aggarwal, A.; Mittal, M.; Battineni, G. Generative adversarial network: An overview of theory and applications. Int. J. Inf. Manag. Data Insights 2021, 1, 100004. [Google Scholar] [CrossRef]
- Hsu, G.; Ambikapathi, A.; Chen, M. Deep learning with time-frequency representation for pulse estimation from facial videos. In Proceedings of the 2017 IEEE International Joint Conference on Biometrics (IJCB), Denver, CO, USA, 1–4 October 2017; pp. 383–389. [Google Scholar] [CrossRef]
- Scherpf, M.; Ernst, H.; Malberg, H.; Schmidt, M. DeepPerfusion: Camera-based Blood Volume Pulse Extraction Using a 3D Convolutional Neural Network. In Proceedings of the 2020 Computing in Cardiology, Rimini, Italy, 13–16 September 2020; pp. 1–4. [Google Scholar] [CrossRef]
- Page, M.J.; McKenzie, J.E.; Bossuyt, P.M.; Boutron, I.; Hoffmann, T.C.; Mulrow, C.D.; Shamseer, L.; Tetzlaff, J.M.; Akl, E.A.; Brennan, S.E.; et al. The PRISMA 2020 statement: An updated guideline for reporting systematic reviews. BMJ 2021, 88, 105906. [Google Scholar]
- Booth, A.; Sutton, A.; Papaioannou, D. Systematic Approaches to a Successful Literature Review; SAGE Publications Ltd.: Thousand Oaks, CA, USA, 2016. [Google Scholar]
- Grames, E.M.; Stillman, A.N.; Tingley, M.W.; Elphick, C.S. An automated approach to identifying search terms for systematic reviews using keyword co-occurrence networks. Methods Ecol. Evol. 2019, 10, 1645–1654. [Google Scholar] [CrossRef]
- Codes, E.M. Derwent World Patents Index (DWPI). 2018. Available online: https://www.jaici.or.jp/newstn/pdf/dwpi_database_information.pdf (accessed on 27 March 2022).
- Zhang, Z.; Girard, J.; Wu, Y.; Zhang, X.; Liu, P.; Ciftci, U.; Canavan, S.; Reale, M.; Horowitz, A.; Yang, H.; et al. Multimodal spontaneous emotion corpus for human behavior analysis. In Proceedings of the IEEE Conference On Computer Vision And Pattern Recognition, Las Vegas, NV, USA, 27–30 June 2016; pp. 3438–3446. Available online: https://openaccess.thecvf.com/content_cvpr_2016/papers/Zhang_Multimodal_Spontaneous_Emotion_CVPR_2016_paper.pdf (accessed on 27 March 2022).
- Niu, X.; Shan, S.; Han, H.; Chen, X. RhythmNet: End-to-End Heart Rate Estimation From Face via Spatial-Temporal Representation. IEEE Trans. Image Process. 2020, 29, 2409–2423. [Google Scholar] [CrossRef] [Green Version]
- Koelstra, S.; Mühl, C.; Soleymani, M.; Lee, J.; Yazdani, A.; Ebrahimi, T.; Pun, T.; Nijholt, A.; Patras, I. DEAP: A Database for Emotion Analysis Using Physiological Signals. IEEE Trans. Affect. Comput. 2012, 3, 18–31. [Google Scholar] [CrossRef] [Green Version]
- Heusch, G.; Anjos, A.; Marcel, S. A reproducible study on remote heart rate measurement. arXiv 2017, arXiv:1709.00962. [Google Scholar]
- Soleymani, M.; Lichtenauer, J.; Pun, T.; Pantic, M. A multimodal database for affect recognition and implicit tagging. IEEE Trans. Affect. Comput. 2012, 3, 42–55. [Google Scholar] [CrossRef] [Green Version]
- Spetlik, R.; Cech, J.; Franc, V.; Matas, J. Visual Heart Rate Estimation with Convolutional Neural Network. In Proceedings of the British Machine Vision Conference, Newcastle, UK, 3–6 September 2018. [Google Scholar]
- Hoffman, W.; Lakens, D. Public Benchmark Dataset for Testing rPPG Algorithm Performance; Technical Report; 4TU.Centre for Research Data: Delft, The Netherlands, 2019. [Google Scholar]
- Nowara, E.M.; Marks, T.K.; Mansour, H.; Veeraraghavan, A. Near-Infrared Imaging Photoplethysmography During Driving. IEEE Trans. Intell. Transp. Syst. 2020, 23, 3589–3600. [Google Scholar] [CrossRef]
- Pai, A.; Veeraraghavan, A.; Sabharwal, A. HRVCam: Robust camera-based measurement of heart rate variability. J. Biomed. Opt. 2021, 26, 022707. [Google Scholar] [CrossRef] [PubMed]
- Svoren, H.; Thambawita, V.; Halvorsen, P.; Jakobsen, P.; Garcia-Ceja, E.; Noori, F.; Hammer, H.; Lux, M.; Riegler, M.; Hicks, S. Toadstool: A dataset for training emotional intelligent machines playing Super Mario Bros. In Proceedings of the 11th ACM Multimedia Systems Conference, Istanbul, Turkey, 8–11 June 2020; pp. 309–314. [Google Scholar] [CrossRef]
- Bobbia, S.; Macwan, R.; Benezeth, Y.; Mansouri, A.; Dubois, J. Unsupervised skin tissue segmentation for remote photoplethysmography. Pattern Recognit. Lett. 2019, 124, 82–90. [Google Scholar] [CrossRef]
- Niu, X.; Zhao, X.; Han, H.; Das, A.; Dantcheva, A.; Shan, S.; Chen, X. Robust remote heart rate estimation from face utilizing spatial-temporal attention. In Proceedings of the 2019 14th IEEE International Conference on Automatic Face & Gesture Recognition (FG 2019), Lille, France, 14–18 May 2019; IEEE: Piscataway, NJ, USA, 2019; pp. 1–8. [Google Scholar]
- Zou, J.; Li, Z.; Yan, P. Automatic Monitoring of Driver’s Physiological Parameters Based on Microarray Camera. In Proceedings of the 2019 IEEE Eurasia Conference on Biomedical Engineering, Healthcare and Sustainability (ECBIOS), Okinawa, Japan, 31 May–3 June 2019; IEEE: Piscataway, NJ, USA, 2019; pp. 15–18. [Google Scholar]
- Kopeliovich, M.; Mironenko, Y.; Petrushan, M. Architectural Tricks for Deep Learning in Remote Photoplethysmography. In Proceedings of the IEEE/CVF International Conference on Computer Vision Workshops, Seoul, Korea, 28 October 2019; pp. 1688–1696. [Google Scholar]
- Hsiao, C.C.; Zheng, W.D.; Lee, R.G.; Lin, R. Emotion Inference of Game Users with Heart Rate Wristbands and Artificial Neural Networks. In Proceedings of the 2018 International Symposium on Computer, Consumer and Control (IS3C), Taichung, Taiwan, 6–8 December 2018; IEEE: Piscataway, NJ, USA, 2018; pp. 326–329. [Google Scholar]
- McDuff, D. Deep Super Resolution for Recovering Physiological Information from Videos. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, Salt Lake City, UT, USA, 18–22 June 2018; pp. 1448–14487. [Google Scholar] [CrossRef]
- Chakraborty, P.R.; Tjondronegoro, D.W.; Zhang, L.; Chandran, V. Towards Generic Modelling of Viewer Interest Using Facial Expression and Heart Rate Features. IEEE Access 2018, 6, 62490–62502. [Google Scholar] [CrossRef]
- Estepp, J.R.; Blackford, E.B.; Meier, C.M. Recovering pulse rate during motion artifact with a multi-imager array for non-contact imaging photoplethysmography. In Proceedings of the 2014 IEEE International Conference on Systems, Man, and Cybernetics (SMC), San Diego, CA, USA, 5–8 October 2014; IEEE: Piscataway, NJ, USA, 2014; pp. 1462–1469. [Google Scholar]
Identification Number | Title | Assignee |
---|---|---|
WO2019202305A1 | System for vital sign detection from a video stream | ClinicCo Ltd. (London, UK) |
CN110738155A | Face recognition method and device, computer equipment and storage medium | Hangzhou First People’s Hospital (Hangzhou, China) |
CN110909717A | Moving object vital sign detection method | Nanjing University of Science and Technology (Nanjing, China) |
CN111259787A | Unlocking method and device, computer equipment and storage medium | Hangzhou First People’s Hospital (Hangzhou, China) |
CN111260634A | Facial blood flow distribution extraction method and system | Tianjin Polytechnic University (Tianjin, CN) |
CN111797794A | Facial dynamic blood flow distribution detection method | People’s Public Security University of China (Bejing, China) |
US20200155040A1 | Systems and methods for determining subject positioning and vital signs | Hill Rom Services Inc. (Batesville, US) |
Datasets | Number of Videos | Subjects | Presence of Infrared Videos |
---|---|---|---|
MMSE-HR | 102 | 40 | No |
VIPL-HR | 3230 | 107 | Yes |
Deap—Part 01 | 120 | 14–16 | No |
Deap—Part 02 | 40 | 32 | No |
COHFACE | 160 | 40 | No |
MAHNOB | 3741 | 27 | Yes |
ECG-Fitness Dataset | 207 | 17 | No |
r-pp. | 21 | 03 | No |
MR-NIRP | 180 | 18 | Yes |
Imaging Photoplethysmography Dataset | 60 | 12 | No |
Toadstool | 10 | 10 | No |
UBFC-Rpp. | 42 | 42 | No |
Ref. | Title | Publication Year |
---|---|---|
[6] | Emotion recognition from facial expressions and contactless heart rate using knowledge graph. | 2020 |
[36] | Toadstool: A dataset for training emotional intelligent machines playing Super Mario Bros. | 2020 |
[10] | A deep learning framework for heart rate estimation from facial videos. | 2020 |
[1] | Non-contact-based driver’s cognitive load classification using physiological and vehicular parameters. | 2020 |
[9] | Non-Contact Emotion Recognition Combining Heart Rate and Facial Expression for Interactive Gaming Environments. | 2020 |
[16] | Visual Heart Rate Estimation from Facial Video Based on CNN. | 2020 |
[5] | On assessing driver awareness of situational criticalities: Multi-modal bio-sensing and vision-based analysis, evaluations, and insights. | 2020 |
[22] | DeepPerfusion: Camera-based Blood Volume Pulse Extraction Using a 3D Convolutional Neural Network. | 2020 |
[12] | Heart Rate Estimation from Facial Videos Using a Spatiotemporal Representation with Convolutional Neural Networks. | 2020 |
[38] | Robust remote heart rate estimation from face utilizing spatial-temporal attention. | 2019 |
[39] | Automatic Monitoring of Driver’s Physiological Parameters Based on Microarray Camera. | 2019 |
[17] | Combating the impact of video compression on non-contact vital sign measurement using supervised learning. | 2019 |
[40] | Architectural tricks for deep learning in remote photoplethysmography. | 2019 |
[41] | Emotion inference of game users with heart rate wristbands and artificial neural networks. | 2019 |
[15] | EVM-CNN: Real-Time Contactless Heart Rate Estimation from Facial Video. | 2019 |
[11] | Long Distance Vital Signs Monitoring with Person Identification for Smart Home Solutions. | 2018 |
[18] | A Novel Short-Term Event Extraction Algorithm for Biomedical Signals. | 2018 |
[21] | Deep learning with time-frequency representation for pulse estimation from facial videos. | 2018 |
[42] | Deep super resolution for recovering physiological information from videos. | 2018 |
[43] | Towards Generic Modelling of Viewer Interest Using Facial Expression and Heart Rate Features. | 2018 |
Metric | Refs. |
---|---|
Mean Error | [10,15,21,38] |
Standard Deviation | [6,10,12,15,16,21,38] |
Root Mean Squared Error | [6,10,11,12,15,16,21,38,42] |
Mean Error Rate | [6,10,12,15,21,38] |
Pearson Correlation | [10,12,15,21,38,42] |
Mean Absolute Error | [12,16,17,38,40,42] |
Mean Absolute Percentage Error (MAPE) | [16] |
Pearson Product Moment Correlation | [16] |
Signal-to-Noise Ratio | [17,42] |
Coverage at ±3 bpm | [40] |
Publisher’s Note: MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations. |
© 2022 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https://creativecommons.org/licenses/by/4.0/).
Share and Cite
Pagano, T.P.; Santos, V.R.; Bonfim, Y.d.S.; Paranhos, J.V.D.; Ortega, L.L.; Sá, P.H.M.; Nascimento, L.F.S.; Winkler, I.; Nascimento, E.G.S. Machine Learning Models and Videos of Facial Regions for Estimating Heart Rate: A Review on Patents, Datasets, and Literature. Electronics 2022, 11, 1473. https://doi.org/10.3390/electronics11091473
Pagano TP, Santos VR, Bonfim YdS, Paranhos JVD, Ortega LL, Sá PHM, Nascimento LFS, Winkler I, Nascimento EGS. Machine Learning Models and Videos of Facial Regions for Estimating Heart Rate: A Review on Patents, Datasets, and Literature. Electronics. 2022; 11(9):1473. https://doi.org/10.3390/electronics11091473
Chicago/Turabian StylePagano, Tiago Palma, Victor Rocha Santos, Yasmin da Silva Bonfim, José Vinícius Dantas Paranhos, Lucas Lemos Ortega, Paulo Henrique Miranda Sá, Lian Filipe Santana Nascimento, Ingrid Winkler, and Erick Giovani Sperandio Nascimento. 2022. "Machine Learning Models and Videos of Facial Regions for Estimating Heart Rate: A Review on Patents, Datasets, and Literature" Electronics 11, no. 9: 1473. https://doi.org/10.3390/electronics11091473
APA StylePagano, T. P., Santos, V. R., Bonfim, Y. d. S., Paranhos, J. V. D., Ortega, L. L., Sá, P. H. M., Nascimento, L. F. S., Winkler, I., & Nascimento, E. G. S. (2022). Machine Learning Models and Videos of Facial Regions for Estimating Heart Rate: A Review on Patents, Datasets, and Literature. Electronics, 11(9), 1473. https://doi.org/10.3390/electronics11091473