Remote Heart Rate Estimation by Pulse Signal Reconstruction Based on Structural Sparse Representation
Abstract
:1. Introduction
- Based on the observation, we adopt the Airpls algorithm to eliminate the trend variation. The experimental results show the superiority of de-trending for the heart rate estimation.
- We find the fact that the heart rates from different sub-regions are consistent and propose the SSR by constraining the consistency of the sparse representation for different sub-regions.
- The experimental results on the two benchmark datasets show that SSR2RPS significantly outperforms the state-of-the-art methods.
2. Related Works
2.1. Video-Based rPPG Measurement
2.2. Sparse Representation
3. Framework
3.1. Face Key Points Detection and ROI Segmentation
3.2. Extraction of Chrominance Signal
3.3. De-Trending Filter
3.4. Reconstruction of the Heart Rate Signals
3.5. Heart Rate Signal Calculation
3.6. Algorithm
Algorithm 1: Remote Heart Rate Estimation by Pulse Signal Reconstruction Based on Structural Sparse Representation. |
Input: A video sequence with l frames. : combined dictionary. : 0.5.
|
Output:. |
4. Experimental Results
4.1. Datasets and Evaluation Metrics
4.2. Comparison of Methods
4.2.1. Performance on UBFC Dataset
4.2.2. Performance on COHFACE Dataset
4.3. Effect of Baseline Elimination
4.4. Parameter Setting
5. Conclusions
Author Contributions
Funding
Data Availability Statement
Conflicts of Interest
References
- Jo, E.; Lewis, K.; Directo, D.; Kim, M.J.; Dolezal, B.A. Validation of biofeedback wearables for photoplethysmographic heart rate tracking. J. Sport Sci. Med. 2016, 15, 540. [Google Scholar]
- Spierer, D.K.; Rosen, Z.; Litman, L.L.; Fujii, K. Validation of photoplethysmography as a method to detect heart rate during rest and exercise. J. Med. Eng. Technol. 2015, 39, 264–271. [Google Scholar] [CrossRef] [PubMed]
- Diao, J.A.; Marwaha, J.S.; Kvedar, J.C. Video-based physiologic monitoring: Promising applications for the ICU and beyond. NPJ Digit. Med. 2022, 5, 1–2. [Google Scholar] [CrossRef] [PubMed]
- Pankaj; Kumar, A.; Komaragiri, R.; Kumar, M. Reference signal less Fourier analysis based motion artifact removal algorithm for wearable photoplethysmography devices to estimate heart rate during physical exercises. Comput. Biol. Med. 2022, 141, 105081. [Google Scholar] [CrossRef]
- Magdalena Nowara, E.; Marks, T.K.; Mansour, H.; Veeraraghavan, A. SparsePPG: Towards driver monitoring using camera-based vital signs estimation in near-infrared. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, Salt Lake City, UT, USA, 18–23 June 2018; pp. 1272–1281. [Google Scholar]
- Lee, H.; Ko, H.; Chung, H.; Nam, Y.; Hong, S.; Lee, J. Real-time realizable mobile imaging photoplethysmography. Sci. Rep. 2022, 12, 1–14. [Google Scholar] [CrossRef]
- Jorge, J.; Villarroel, M.; Tomlinson, H.; Gibson, O.; Darbyshire, J.L.; Ede, J.; Harford, M.; Young, J.D.; Tarassenko, L.; Watkinson, P. Non-contact physiological monitoring of post-operative patients in the intensive care unit. Nat. Partn. J. Digit. Med. 2022, 5, 1–11. [Google Scholar]
- Wang, W.; Den Brinker, A.C.; Stuijk, S.; De Haan, G. Algorithmic principles of remote PPG. IEEE Trans. Biomed. Eng. 2016, 64, 1479–1491. [Google Scholar] [CrossRef] [Green Version]
- De Haan, G.; Jeanne, V. Robust pulse rate from chrominance-based rPPG. IEEE Trans. Biomed. Eng. 2013, 60, 2878–2886. [Google Scholar] [CrossRef]
- Chen, W.; McDuff, D. Deepphys: Video-based physiological measurement using convolutional attention networks. In Proceedings of the European Conference on Computer Vision, Munich, Germany, 8–14 September 2018; pp. 349–365. [Google Scholar]
- Kang, J.; Yang, S.; Zhang, W. TransPPG: Two-stream Transformer for Remote Heart Rate Estimate. arXiv 2022, arXiv:2201.10873. [Google Scholar]
- Niu, X.; Yu, Z.; Han, H.; Li, X.; Shan, S.; Zhao, G. Video-based remote physiological measurement via cross-verified feature disentangling. In Proceedings of the European Conference on Computer Vision, Glasgow, UK, 23–28 August 2020; pp. 295–310. [Google Scholar]
- Hill, B.L.; Liu, X.; McDuff, D. Beat-to-beat cardiac pulse rate measurement from video. In Proceedings of the IEEE International Conference on Computer Vision, Montreal, QC, Canada, 10–17 October 2021; pp. 2739–2742. [Google Scholar]
- Qiu, Y.; Liu, Y.; Arteaga-Falconi, J.; Dong, H.; El Saddik, A. EVM-CNN: Real-time contactless heart rate estimation from facial video. IEEE Trans. Multimed. 2018, 21, 1778–1787. [Google Scholar] [CrossRef]
- Li, L.; Chen, C.; Pan, L.; Zhang, J.; Xiang, Y. Video is All You Need: Attacking PPG-based Biometric Authentication. arXiv 2022, arXiv:2203.00928. [Google Scholar]
- Verkruysse, W.; Svaasand, L.O.; Nelson, J.S. Remote plethysmographic imaging using ambient light. Optics Express 2008, 16, 21434–21445. [Google Scholar] [CrossRef] [PubMed] [Green Version]
- Wang, W.; Stuijk, S.; De Haan, G. A novel algorithm for remote photoplethysmography: Spatial subspace rotation. IEEE Trans. Biomed. Eng. 2015, 63, 1974–1984. [Google Scholar] [CrossRef] [PubMed]
- Zhang, X.; Xia, Z.; Dai, J.; Liu, L.; Jiang, X.; Feng, X. Heart rate estimation via self-adaptive region selection and multiregion-fusion 1D CNN. J. Electron. Imaging 2022, 31, 023006. [Google Scholar]
- Cai, K.; Yue, H.; Li, B.; Chen, W.; Huang, W. Combining chrominance features and fast ICA for noncontact imaging photoplethysmography. IEEE Access 2020, 8, 50171–50179. [Google Scholar] [CrossRef]
- Pilz, C.S.; Zaunseder, S.; Krajewski, J.; Blazek, V. Local group invariance for heart rate estimation from face videos in the wild. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, Salt Lake City, UT, USA, 18–23 June 2018; pp. 1254–1262. [Google Scholar]
- Yang, Z.; Wang, H.; Lu, F. Assessment of Deep Learning-based Heart Rate Estimation using Remote Photoplethysmography under Different Illuminations. arXiv 2022, arXiv:2107.13193. [Google Scholar] [CrossRef]
- Schrumpf, F.; Frenzel, P.; Aust, C.; Osterhoff, G.; Fuchs, M. Assessment of Non-Invasive Blood Pressure Prediction from PPG and rPPG Signals Using Deep Learning. Sensors 2021, 21, 6022. [Google Scholar] [CrossRef]
- Hu, C.; Zhang, K.Y.; Yao, T.; Ding, S.; Li, J.; Huang, F.; Ma, L. An End-to-end Efficient Framework for Remote Physiological Signal Sensing. In Proceedings of the IEEE International Conference on Computer Vision, IEEE, Montreal, QC, Canada, 10–17 October 2021; pp. 2378–2384. [Google Scholar]
- Gao, H.; Wu, X.; Geng, J.; Lv, Y. Remote Heart Rate Estimation by Signal Quality Attention Network. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, New Orleans, LA, USA, 19–24 June 2022; pp. 2122–2129. [Google Scholar]
- Li, T.; Chen, W. Bathtub ECG as a Potential Alternative to Light Stress Test in Daily Life. Electronics 2022, 11, 1310. [Google Scholar] [CrossRef]
- Pagano, T.P.; Santos, V.R.; Bonfim, Y.d.S.; Paranhos, J.V.D.; Ortega, L.L.; Sá, P.H.M.; Nascimento, L.F.S.; Winkler, I.; Nascimento, E.G.S. Machine Learning Models and Videos of Facial Regions for Estimating Heart Rate: A Review on Patents, Datasets, and Literature. Electronics 2022, 11, 1473. [Google Scholar] [CrossRef]
- Mallat, S.G.; Zhang, Z. Matching pursuits with time-frequency dictionaries. IEEE Trans. Signal Process. 1993, 41, 3397–3415. [Google Scholar] [CrossRef] [Green Version]
- Pati, Y.C.; Rezaiifar, R.; Krishnaprasad, P.S. Orthogonal matching pursuit: Recursive function approximation with applications to wavelet decomposition. In Proceedings of the 27th Asilomar Conference on Signals, Systems and Computers, Pacific Grove, CA, USA, 1–3 November 1993; pp. 40–44. [Google Scholar]
- Liu, S.; Lyu, N.; Wang, H. The implementation of the improved OMP for AIC reconstruction based on parallel index selection. IEEE Trans. Very Large Scale Integr. Syst. 2017, 26, 319–328. [Google Scholar]
- Zhang, Z. Photoplethysmography-based heart rate monitoring in physical activities via joint sparse spectrum reconstruction. IEEE Trans. Biomed. Eng. 2015, 62, 1902–1910. [Google Scholar] [CrossRef] [PubMed] [Green Version]
- Liu, X.; Yang, X.; Jin, J.; Wong, A. Detecting pulse wave from unstable facial videos recorded from consumer-level cameras: A disturbance-adaptive orthogonal matching pursuit. IEEE Trans. Biomed. Eng. 2020, 67, 3352–3362. [Google Scholar] [CrossRef] [PubMed]
- Dabhi, M.K.; Pancholi, B.K. Face detection system based on Viola-Jones algorithm. Int. J. Sci. Res. 2016, 5, 62–64. [Google Scholar]
- Wong, K.L.; Chin, J.W.; Chan, T.T.; Odinaev, I.; Suhartono, K.; Tianqu, K.; So, R.H. Optimising rPPG Signal Extraction by Exploiting Facial Surface Orientation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, New Orleans, LA, USA, 19–24 June 2022; pp. 2165–2171. [Google Scholar]
- Kwon, S.; Kim, J.; Lee, D.; Park, K. ROI analysis for remote photoplethysmography on facial video. In Proceedings of the 2015 37th Annual International Conference of the IEEE Engineering in Medicine and Biology Society, Milan, Italy, 25–29 August 2015; pp. 4938–4941. [Google Scholar]
- Guo, J.; Deng, J.; Lattas, A.; Zafeiriou, S. Sample and Computation Redistribution for Efficient Face Detection. arXiv 2021, arXiv:2105.04714. [Google Scholar]
- Zhang, Z.M.; Chen, S.; Liang, Y.Z. Baseline correction using adaptive iteratively reweighted penalized least squares. Analyst 2010, 135, 1138–1146. [Google Scholar] [CrossRef]
- Černá, D.; Rebollo-Neira, L. Construction of wavelet dictionaries for ECG modeling. MethodsX 2021, 8, 101314. [Google Scholar] [CrossRef]
- Rajaei, A.; Fattaheian-Dehkordi, S.; Fotuhi-Firuzabad, M.; Moeini-Aghtaie, M. Decentralized transactive energy management of multi-microgrid distribution systems based on ADMM. Int. J. Electr. Power Energy Syst. 2021, 132, 107126. [Google Scholar] [CrossRef]
- Zhao, H.; Liu, H.; Jin, Y.; Dang, X.; Deng, W. Feature extraction for data-driven remaining useful life prediction of rolling bearings. IEEE Trans. Instrum. Meas. 2021, 70, 1–10. [Google Scholar]
- Bobbia, S.; Macwan, R.; Benezeth, Y.; Mansouri, A.; Dubois, J. Unsupervised skin tissue segmentation for remote photoplethysmography. Pattern Recognit. Lett. 2019, 124, 82–90. [Google Scholar] [CrossRef]
- Heusch, G.; Anjos, A.; Marcel, S. A reproducible study on remote heart rate measurement. arXiv 2017, arXiv:1709.00962. [Google Scholar]
- Poh, M.Z.; McDuff, D.J.; Picard, R.W. Advancements in noncontact, multiparameter physiological measurements using a webcam. IEEE Trans. Biomed. Eng. 2010, 58, 7–11. [Google Scholar] [CrossRef] [PubMed]
- Lewandowska, M.; Rumiński, J.; Kocejko, T.; Nowak, J. Measuring pulse rate with a webcam—A non-contact method for evaluating cardiac activity. In Proceedings of the 2011 Federated Conference on Computer Science and Information Systems, Szczecin, Poland, 18–21 September 2011; pp. 405–410. [Google Scholar]
Methods | (bpm) | (bpm) | (bpm) | |
---|---|---|---|---|
CHROM [9] | 5.92 | 6.37 | 9.10 | 0.91 |
ICA [42] | 24.83 | 26.78 | 32.59 | 0.37 |
PCA [43] | 12.46 | 18.36 | 22.32 | 0.31 |
POS [8] | 6.37 | 6.52 | 10.52 | 0.86 |
DAOMP [31] | 6.68 | 7.34 | 14.50 | 0.87 |
LGI [20] | 9.23 | 10.29 | 16.61 | 0.65 |
SSR2RPS | 1.70 | 2.57 | 4.69 | 0.97 |
Methods | Good Condition 0 | Good Condition 1 | ||||||
(bpm) | (bpm) | (bpm) | (bpm) | (bpm) | (bpm) | |||
CHROM [9] | 5.67 | 6.37 | 8.43 | 0.87 | 4.35 | 5.93 | 8.68 | 0.87 |
ICA [42] | 11.99 | 16.48 | 26.05 | 0.36 | 12.99 | 17.48 | 29.54 | 0.30 |
PCA [43] | 3.97 | 12.33 | 14.37 | 0.45 | 4.18 | 7.42 | 9.38 | 0.43 |
POS [8] | 4.77 | 7.36 | 11.76 | 0.69 | 5.63 | 8.36 | 13.82 | 0.76 |
DAOMP [31] | 3.65 | 5.20 | 9.58 | 0.89 | 3.77 | 6.37 | 11.26 | 0.83 |
LGI [20] | 8.29 | 12.46 | 13.64 | 0.62 | 7.87 | 11.74 | 14.54 | 0.63 |
SSR2RPS | 3.25 | 3.43 | 4.11 | 0.91 | 3.53 | 3.54 | 4.63 | 0.90 |
Methods | Nature Condition 2 | Nature Condition 3 | ||||||
(bpm) | (bpm) | (bpm) | (bpm) | (bpm) | (bpm) | |||
CHROM [9] | 4.65 | 6.80 | 7.00 | 0.77 | 3.65 | 6.70 | 10.26 | 0.84 |
ICA [42] | 9.99 | 14.48 | 20.19 | 0.24 | 12.04 | 17.48 | 23.54 | 0.22 |
PCA [43] | 12.24 | 17.33 | 23.37 | 0.28 | 9.97 | 15.33 | 17.37 | 0.22 |
POS [8] | 6.63 | 11.36 | 18.82 | 0.70 | 7.13 | 9.36 | 16.82 | 0.74 |
DAOMP [31] | 3.77 | 7.34 | 13.63 | 0.76 | 3.07 | 7.84 | 12.02 | 0.79 |
LGI [20] | 6.31 | 10.97 | 13.23 | 0.59 | 7.13 | 11.72 | 14.17 | 0.68 |
SSR2RPS | 4.27 | 4.68 | 5.31 | 0.88 | 4.26 | 4.75 | 5.76 | 0.85 |
Methods | (bpm) | (bpm) | (bpm) | |
---|---|---|---|---|
No elimination of trends | 4.08 | 4.56 | 9.16 | 0.88 |
Linear de-trending | 4.87 | 5.32 | 10.28 | 0.86 |
Fourth-order polynomial de-trending | 3.27 | 3.73 | 7.02 | 0.93 |
Fifth-order polynomial de-trending | 3.41 | 3.79 | 7.00 | 0.93 |
Airpls de-trending | 1.70 | 2.57 | 4.69 | 0.97 |
Publisher’s Note: MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations. |
© 2022 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https://creativecommons.org/licenses/by/4.0/).
Share and Cite
Han, J.; Ou, W.; Xiong, J.; Feng, S. Remote Heart Rate Estimation by Pulse Signal Reconstruction Based on Structural Sparse Representation. Electronics 2022, 11, 3738. https://doi.org/10.3390/electronics11223738
Han J, Ou W, Xiong J, Feng S. Remote Heart Rate Estimation by Pulse Signal Reconstruction Based on Structural Sparse Representation. Electronics. 2022; 11(22):3738. https://doi.org/10.3390/electronics11223738
Chicago/Turabian StyleHan, Jie, Weihua Ou, Jiahao Xiong, and Shihua Feng. 2022. "Remote Heart Rate Estimation by Pulse Signal Reconstruction Based on Structural Sparse Representation" Electronics 11, no. 22: 3738. https://doi.org/10.3390/electronics11223738
APA StyleHan, J., Ou, W., Xiong, J., & Feng, S. (2022). Remote Heart Rate Estimation by Pulse Signal Reconstruction Based on Structural Sparse Representation. Electronics, 11(22), 3738. https://doi.org/10.3390/electronics11223738