# Enhanced Contactless Vital Sign Estimation from Real-Time Multimodal 3D Image Data

^{1}

^{2}

^{*}

## Abstract

**:**

## 1. Introduction

_{2}), respiration rate, body temperature, etc. is an important basic task in biomedical metrology. Conventional devices for such tasks mostly take contact-based measurement approaches. However, contact measurement has several disadvantages. Above all, contact with the body and skin raises the risk of skin irritation and germ contamination. Moreover, contact-based devices significantly limit the freedom of body movement of the patients, and hence, it could lead to severe discomforts. Therefore, the contactless estimation of vital signs using image sensors has continuously gained importance because of its advantages regarding hygiene and patient-friendliness.

_{2}conversion are to be empirically determined.

## 2. Multimodal 3D Imaging System

^{2}and, thus, within the eye safety limits [25]. The high-speed cameras of the stereo-vision setup and both the NIR cameras are equipped with band-pass optical filters with respective central wavelengths and a full-width at half-maximum (FWHM) of 50 nm. The GOBO projector contains a band-pass filter at 850 nm with a 50 nm FWHM in order to avoid possible spectral crosstalk between pattern projection and NIR cameras.

## 3. Algorithms

#### 3.1. Face Analysis

#### 3.2. 3D Face Tracking

#### 3.3. Vital Sign Estimation

_{2}value, can be calculated by calculating the AC and DC components at both 780 and 940 nm. In order to improve the stability of oxygen saturation estimation, the RR values should be averaged over a certain time window.

## 4. Results

#### 4.1. Body Temperature and Respiration Rate

#### 4.2. Heart Rate

#### 4.3. Oxygen Saturation

_{2}values and calculated RR values within 65 s. Both these variations are qualitatively consistent with each other in the course of time, and the Pearson moment correlation coefficient between them is 0.867. In Figure 12b, the result of the linear regression between RR and SpO

_{2}values is shown. In general, a linear relation can be found and the prediction of SpO

_{2}values based on RR is possible, though the data appear to be somewhat noisy and the coefficient of determination R

^{2}of the linear statistical model is 0.7521. Based on SpO

_{2}values estimated from the RR values using this linear model, the limits of agreement were determined according to the Bland–Altman method and are shown in Figure 12c. Here, the mean difference was zero, because the same ground truth values used for the fitting of the linear RR-to-SpO

_{2}conversion model were also used for the determination of LoAs, and the 95% confidence LoAs were −4.26% and 4.26%. Based on Figure 12a and the LoAs, it can be claimed that changes in the SpO

_{2}values by more than 5% could be detected with high reliability.

## 5. Summary and Discussion

## Author Contributions

## Funding

## Acknowledgments

## Conflicts of Interest

## References

- Verkruysse, W.; Svaasand, L.O.; Nelson, J.S. Remote plethysmographic imaging using ambient light. Opt. Express
**2008**, 16, 21434–21445. [Google Scholar] [CrossRef] [PubMed][Green Version] - De Haan, G.; Jeanne, V. Robust pulse rate from chrominance-based rPPG. IEEE Trans. Biomed. Eng.
**2013**, 60, 2878–2886. [Google Scholar] [CrossRef] [PubMed] - Xu, S.; Sun, L.; Rohde, G.K. Robust efficient estimation of heart rate pulse from video. Biomed. Opt. Express
**2014**, 5, 1124–1135. [Google Scholar] [CrossRef] [PubMed][Green Version] - Rapczynski, M.; Werner, P.; Saxen, F.; Al-Hamadi, A. How the Region of Interest Impacts Contact Free Heart Rate Estimation Algorithms. In Proceedings of the 25th IEEE International Conference on Image Processing (ICIP), Athens, Greece, 7–10 October 2018; pp. 2027–2031. [Google Scholar]
- Tarassenko, L.; Villarroel, M.; Guazzi, A.; Jorge, J.; Clifton, D.A.; Pugh, C. Non-contact video-based vital sign monitoring using ambient light and auto-regressive models. Physiol. Meas.
**2014**, 35, 831–907. [Google Scholar] [CrossRef] [PubMed] - Kumar, M.; Veeraraghavan, A.; Sabharwal, A. DistancePPG: Robust non-contact vital signs monitoring using a camera. Biomed. Opt. Express
**2015**, 6, 1565–1588. [Google Scholar] [CrossRef] [PubMed][Green Version] - Guazzi, A.R.; Villarroel, M.; Jorge, J.; Daly, J.; Frise, M.C.; Robbins, P.A.; Tarassenko, L. Non-contact measurement of oxygen saturation with an RGB camera. Biomed. Opt. Express
**2015**, 6, 3320–3338. [Google Scholar] [CrossRef] [PubMed][Green Version] - Scully, C.G.; Lee, J.; Meyer, J.; Gorbach, A.M.; Granquist-Fraser, D.; Mendelson, Y.; Chon, K.H. Physiological Parameter Monitoring from Optical Recordings with a Mobile Phone. IEEE Trans. Biomed. Eng.
**2012**, 59, 303–306. [Google Scholar] [CrossRef][Green Version] - Bal, U. Non-contact estimation of heart rate and oxygen saturation using ambient light. Biomed. Opt. Express
**2015**, 6, 86–97. [Google Scholar] [CrossRef][Green Version] - Rosa, A.; Betini, R. Noncontact SpO
_{2}Measurement Using Eulerian Video Magnification. IEEE Trans. Instrum. Meas.**2020**, 69, 2120–2130. [Google Scholar] [CrossRef] - Van Gastel, M.; Stuijk, S.; De Haan, G. Robust respiration detection from remote photoplethysmography. Biomed. Opt. Express
**2016**, 7, 4941–4957. [Google Scholar] [CrossRef][Green Version] - Fiedler, M.-A.; Rapczynski, M.; Al-Hamadi, A. Fusion-Based Approach for Respiratory Rate Recognition from Facial Video Images. IEEE Access
**2020**, 8, 130036–130047. [Google Scholar] [CrossRef] - Pereira, C.B.; Yu, X.; Czaplik, M.; Rossaint, R.; Blazek, V.; Leonhardt, S. Remote monitoring of breathing dynamics using infrared thermography. Biomed. Opt. Express
**2015**, 6, 4378–4394. [Google Scholar] [CrossRef] [PubMed] - Cho, Y.; Julier, S.J.; Marquardt, N.; Bianchi-Berthouze, N. Robust tracking of respiratory rate in high-dynamic range scenes using mobile thermal imaging. Biomed. Opt. Express
**2017**, 8, 4480–4503. [Google Scholar] [CrossRef] [PubMed][Green Version] - Rapczynski, M.; Zhang, C.; Al-Hamadi, A.; Notni, G. A Multi-Spectral Database for NIR Heart Rate Estimation. In Proceedings of the 2018 25th IEEE International Conference on Image Processing (ICIP), Athens, Greece, 7–10 October 2018; pp. 2022–2026. [Google Scholar]
- Kim, J.; Yu, S.; Kim, I.-J.; Lee, S. 3D Multi-Spectrum Sensor System with Face Recognition. Sensors
**2013**, 13, 12804–12829. [Google Scholar] [CrossRef] [PubMed][Green Version] - Chane, C.S.; Mansouri, A.; Marzani, F.; Boochs, F. Integration of 3D and multispectral data for cultural heritage applications: Survey and perspectives. Image Vis. Comput.
**2013**, 31, 91–102. [Google Scholar] [CrossRef][Green Version] - Chromy, A.; Klima, O. A 3D Scan Model and Thermal Image Data Fusion Algorithms for 3D Thermography in Medicine. J. Healthc. Eng.
**2017**, 2017, 5134021. [Google Scholar] [CrossRef][Green Version] - Heist, S.; Zhang, C.; Reichwald, K.; Kühmstedt, P.; Notni, G.; Tünnermann, A. 5D hyperspectral imaging: Fast and accurate measurement of surface shape and spectral characteristics using structured light. Opt. Express
**2018**, 26, 23366–23379. [Google Scholar] [CrossRef] - Landmann, M.; Heist, S.; Dietrich, P.; Lutzke, P.; Gebhart, I.; Templin, J.; Kühmstedt, P.; Tünnermann, A.; Notni, G. High-speed 3D thermography. Opt. Lasers Eng.
**2019**, 121, 448–455. [Google Scholar] [CrossRef] - Heist, S.; Lutzke, P.; Schmidt, I.; Dietrich, P.; Kühmstedt, P.; Tünnermann, A.; Notni, G. High-speed three-dimensional shape measurement using GOBO projection. Opt. Lasers Eng.
**2016**, 87, 90–96. [Google Scholar] [CrossRef] - Heist, S.; Dietrich, P.; Landmann, M.; Kühmstedt, P.; Notni, G.; Tünnermann, A. GOBO projection for 3D measurements at highest frame rates: A performance analysis. Light
**2018**, 7, 71. [Google Scholar] [CrossRef] - Heist, S.; Kühmstedt, P.; Tünnermann, A.; Notni, G. Theoretical considerations on aperiodic sinusoidal fringes in comparison to phase-shifted sinusoidal fringes for high-speed three-dimensional shape measurement. Appl. Opt.
**2015**, 54, 10541–10551. [Google Scholar] [CrossRef] [PubMed] - Dietrich, P.; Heist, S.; Landmann, M.; Kühmstedt, P.; Notni, G. BICOS—An Algorithm for Fast Real-Time Correspondence Search for Statistical Pattern Projection-Based Active Stereo Sensors. Appl. Sci.
**2019**, 9, 3330. [Google Scholar] [CrossRef][Green Version] - International Commission on Non-Ionizing Radiation Protection. Guidelines of limits of exposure to broad-band incoherent optical radiation (0.38 to 3 µm). Health Phys.
**1997**, 73, 539–554. [Google Scholar] - Rosenberger, M.; Zhang, C.; Zhang, Y.; Notni, G. 3D high-resolution multimodal imaging system for real-time applications. In Proceedings of the SPIE 11397, Dimensional Optical Metrology and Inspection for Practical Applications IX, 27, Anaheim, CA, USA, 4–8 May 2020; p. 1139704. [Google Scholar]
- Viola, P.; Jones, M.J. Robust Real-Time Face Detection. Int. J. Comput. Vis.
**2004**, 57, 137–154. [Google Scholar] [CrossRef] - Baltrušaitis, T.; Robinson, P.; Morency, L.-P. OpenFace: An open source facial behavior analysis toolkit. In Proceedings of the 2016 IEEE Winter Conference on Applications of Computer Vision (WACV), Lake Placid, NY, USA, 7–10 March 2016. [Google Scholar]
- Ren, S.; Cao, X.; Wei, Y.; Sun, J. Face Alignment at 3000 FPS via Regressing Local Binary Features. In Proceedings of the 2014 IEEE Conference on Computer Vision and Pattern Recognition, Columbus, OH, USA, 23–28 June 2014; pp. 1685–1692. [Google Scholar]
- Shi, J.; Tomasi, C. Good Features to Track. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Seattle, WA, USA, 21–23 June 1994; pp. 593–600. [Google Scholar]
- Lucas, B.D.; Kanade, T. An Iterative Image Registration Technique with an Application to Stereo Vision. In Proceedings of the 1981 DARPA Image Understanding Workshop, Washington, DC, USA, 23 April 1981; pp. 121–130. [Google Scholar]
- Kalman, R.E. A New Approach to Linear Filtering and Prediction Problems. Trans. ASME J. Basic Eng.
**1960**, 82, 35–45. [Google Scholar] [CrossRef][Green Version] - Wu, H.-Y.; Rubinstein, M.; Shih, E.; Guttag, J.V.; Durand, F.; Freeman, W.T. Eulerian Video Magnification for Revealing Subtle Changes in the World. ACM Trans. Graph.
**2012**, 31, 65. [Google Scholar] [CrossRef] - Bennett, S.L.; Goubran, R.; Knoefel, F. Adaptive eulerian video magnification methods to extract heart rate from thermal video. In Proceedings of the 2016 IEEE International Symposium on Medical Measurements and Applications (MeMeA), Benevento, Italy, 15–18 May 2016; pp. 1–5. [Google Scholar]
- Dosso, Y.S.; Bekele, A.; Green, J.R. Eulerian Magnification of Multi-Modal RGB-D Video for Heart Rate Estimation. In Proceedings of the 2018 IEEE International Symposium on Medical Measurements and Applications (MeMeA), Rome, Italy, 11–13 June 2018; pp. 1–6. [Google Scholar]
- Ordóñez, C.; Cabo, C.; Menéndez, A.; Bello, A. Detection of human vital signs in hazardous environments by means of video magnification. PLoS ONE
**2018**, 13, e0195290. [Google Scholar] [CrossRef][Green Version] - Andelson, E.H.; Anderson, C.H.; Bergen, J.R.; Burt, P.J.; Odgen, J.M. Pyramid methods in image processing. RCA Eng.
**1984**, 29, 33–41. [Google Scholar] - Bland, J.M.; Altman, D.G. Statistical methods for assessing agreement between two methods of clinical measurement. Lancet
**1986**, 1, 307–310. [Google Scholar] [CrossRef] - Munkholm, S.; Krøgholt, T.; Ebbesen, F.; Szecsi, P.B.; Kristensen, S.R. The smartphone camera as a potential method for transcutaneous bilirubin measurement. PLoS ONE
**2018**, 13, e0197938. [Google Scholar] [CrossRef] - Zhang, G.; Shan, C.; Kirenko, I.; Long, X.; Aarts, R.M. Hybrid Optical Unobtrusive Blood Pressure Measurements. Sensors
**2017**, 17, 1541. [Google Scholar] [CrossRef] [PubMed][Green Version]

**Figure 1.**Multimodal 3D imaging system. 1/2: High-speed cameras at 850 nm, 3: High-speed GOes Before Optics (GOBO) projector at 850 nm, 4: Color camera, 5: NIR camera at 780 nm, 6: NIR camera at 940 nm, 7: Thermal camera, 8: Light emitting diode (LED) array at 780 and 940 nm.

**Figure 4.**(

**a**) Selection of region of interest (ROI) and feature points on the basis of face analysis in 2D color image, (

**b**) transformation of ROI and feature points into 3D space.

**Figure 7.**Extraction of primary photoplethysmographic (PPG) signal for heart rate estimation and enhanced NIR grey value signals for oxygen saturation estimation from 3D and NIR image data.

**Figure 9.**Estimation of respiration rate from the nostrils’ ROI: (

**a**) Temporal temperature signal, (

**b**) power density spectrum calculated from the respiration signal.

**Figure 10.**Estimation of one heart rate: (

**a**) Final PPG signal, (

**b**) power density spectrum calculated from the final PPG signal.

**Figure 11.**Continuous estimation of heart rate: (

**a**) Ground truth values and estimated values of heart rate within 80 s, (

**b**) difference between estimated heart rate and ground truth.

**Figure 12.**Continuous estimation of oxygen saturation: (

**a**) Ground truth SpO

_{2}values and ratio of ratios (RR) values within 65 s, (

**b**) linear regression between RR and SpO

_{2}values, (

**c**) difference between estimated SpO

_{2}and ground truth.

Publisher’s Note: MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations. |

© 2020 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (http://creativecommons.org/licenses/by/4.0/).

## Share and Cite

**MDPI and ACS Style**

Zhang, C.; Gebhart, I.; Kühmstedt, P.; Rosenberger, M.; Notni, G. Enhanced Contactless Vital Sign Estimation from Real-Time Multimodal 3D Image Data. *J. Imaging* **2020**, *6*, 123.
https://doi.org/10.3390/jimaging6110123

**AMA Style**

Zhang C, Gebhart I, Kühmstedt P, Rosenberger M, Notni G. Enhanced Contactless Vital Sign Estimation from Real-Time Multimodal 3D Image Data. *Journal of Imaging*. 2020; 6(11):123.
https://doi.org/10.3390/jimaging6110123

**Chicago/Turabian Style**

Zhang, Chen, Ingo Gebhart, Peter Kühmstedt, Maik Rosenberger, and Gunther Notni. 2020. "Enhanced Contactless Vital Sign Estimation from Real-Time Multimodal 3D Image Data" *Journal of Imaging* 6, no. 11: 123.
https://doi.org/10.3390/jimaging6110123