Next Article in Journal
Computational Diagnostic Techniques for Electrocardiogram Signal Analysis
Next Article in Special Issue
Smart Sensors for Healthcare and Medical Applications
Previous Article in Journal
Interdigital Capacitor-Based Passive LC Resonant Sensor for Improved Moisture Sensing
Previous Article in Special Issue
Tracking and Characterization of Spinal Cord-Injured Patients by Means of RGB-D Sensors
 
 
Font Type:
Arial Georgia Verdana
Font Size:
Aa Aa Aa
Line Spacing:
Column Width:
Background:
Article

Non-Contact Respiration Monitoring and Body Movements Detection for Sleep Using Thermal Imaging

by
Prasara Jakkaew
1,2,*,† and
Takao Onoye
1,†
1
Information Systems Synthesis Laboratory, Department of Information Systems Engineering, Graduate School of Information Science and Technology, Osaka University, 1-5 Yamadaoka, Suita, Osaka 565-0871, Japan
2
School of Information Technology, Mae Fah Luang University, 333-1 Thasud, Muang, Chiang Rai 57100, Thailand
*
Author to whom correspondence should be addressed.
These authors contributed equally to this work.
Sensors 2020, 20(21), 6307; https://doi.org/10.3390/s20216307
Submission received: 15 September 2020 / Revised: 22 October 2020 / Accepted: 2 November 2020 / Published: 5 November 2020
(This article belongs to the Special Issue Smart Sensors for Healthcare and Medical Applications)

Abstract

:
Monitoring of respiration and body movements during sleep is a part of screening sleep disorders related to health status. Nowadays, thermal-based methods are presented to monitor the sleeping person without any sensors attached to the body to protect privacy. A non-contact respiration monitoring based on thermal videos requires visible facial landmarks like nostril and mouth. The limitation of these techniques is the failure of face detection while sleeping with a fixed camera position. This study presents the non-contact respiration monitoring approach that does not require facial landmark visibility under the natural sleep environment, which implies an uncontrolled sleep posture, darkness, and subjects covered with a blanket. The automatic region of interest (ROI) extraction by temperature detection and breathing motion detection is based on image processing integrated to obtain the respiration signals. A signal processing technique was used to estimate respiration and body movements information from a sequence of thermal video. The proposed approach has been tested on 16 volunteers, for which video recordings were carried out by themselves. The participants were also asked to wear the Go Direct respiratory belt for capturing reference data. The result revealed that our proposed measuring respiratory rate obtains root mean square error (RMSE) of 1.82 ± 0.75 bpm. The advantage of this approach lies in its simplicity and accessibility to serve users who require monitoring the respiration during sleep without direct contact by themselves.

1. Introduction

The respiratory rate is one of the critical vital signs that indicate health problems. The respiratory system is the gas exchange process to take in oxygen and expel carbon dioxide, as breathing moves air in and out of the lungs. The frequency of breaths is defined as a respiratory rate (RR) that is usually measured by counting the number of breaths a person takes per minute. A clinical staff member can count the number of times the chest moves up and down for a full minute. In general, the regular respiratory rate for healthy individuals is 12–20 bpm [1]. A change as little as three to five bpm may indicate a change in the patient’s condition [2]. A patient suffering a severe adverse event on the general wards, such as a cardiac arrest or ICU admission, shows an increase in RR of up until 24 h before the severe events with high specificity [3]. For that reason, respiration monitoring should be performed continuously for a long time without impressing the patient’s burden.
Owing to the fact that humans spend almost 30% of the time in sleeping, the respiration monitoring during sleep, which accurately reflects one’s health condition is an appropriate and reasonable option. It is generally known that poor sleep significantly affects work productivity, cognitive performance, and overall life quality. Sleep monitoring can detect sleep disorders associated with cardiovascular disease, including heart failure, hypertension, and increased arrhythmia [4]. The breathing patterns during sleeping are utilized to identify the sleep disorder as sleep apnea, including obstructive sleep apnea (OSA), central sleep apnea, and complex sleep apnea syndrome. Sleep apnea is a cessation of the airflow that occurs when breathing repeatedly stops and starts during sleep, resulting in decreased oxygen flow to the brain and the rest of the body. Sleep apnea is generally characterized by cessation of breath for at least 10 s during sleep [5]. The well-known index used to indicate the severity of sleep apnea is Apnea–Hypopnoea Index (AHI), which counts the number of apnea events per hour. Harvard Medical School [6] classifies the severity of OSA as:
  • None/Minimal: AHI < 5 per hour
  • Mild: AHI ≥ 5, but < 15 per hour
  • Moderate: AHI ≥ 15, but < 30 per hour
  • Severe: AHI ≥ 30 per hour
Besides, sleep monitoring can detect periodic limb movement disorder (PLMD), which is repetitive cramping or jerking of the legs during sleep. Patients with PLMD may suffer from daytime sleepiness, daytime fatigue, trouble falling asleep at night, and difficulty staying asleep throughout the night [7]. Usually, patients with PLMD are unaware of their leg movements unless their bed partner tells them. It is also reported that movements are repetitive and rhythmic and occur every 20–40 s [8].
It has been demanded for long years to develop a novel technology to monitor respiratory and body movements without disturbing human sleeping. In recent years, non-contact respiration monitoring solutions have been proposed, like camera imaging [9], thermal imaging [10], and microwave Doppler radar [11]. As for camera imaging, a simple camera like a CCD camera, webcam, digital camera, or smartphone camera is used to detect the respiratory rate from the breathing motion, whereas for thermal imaging, a thermal camera or infrared camera is used to detect the temperature changes due to breaths. Respiratory rate often relies on visual observation of chest movement at periodic intervals [1]; a small movement that is hard to see with the naked eye. A motion magnification technique was proposed to magnify a baby’s respiratory chest movements from a digital video camera [12]. Several researchers have analyzed video and image sequences to detect breath motions and extract vital signs in sleeping positions. For example, Nakajima et al. and Frigola et al. apply an optical flow technique to image data from a CCD camera and a remote TV camera, respectively [13,14]. Wiesner et al. monitor respiratory by tracking the motion of a fiducial marker placed on the patient’s abdomen with a single webcam [15]. However, there are still some limitations of disturbing the natural sleep environment, such as visible light and privacy issues.
Thermal imaging is a rapidly evolving technology, now turning up in hospitals, airports, and even smartphones. Respiratory can be monitored through thermal imaging [16,17,18,19,20,21]. Thermal imaging cameras rely on microelectromechanical sensors to produce an image from heat; the human body stands out from the surrounding field because it gives off more heat. The thermal image-based method has an advantage under conditions of varying illumination and can reduce privacy issues. Previously, several approaches have been proposed to monitor respiration with a thermal camera by detecting the temperature change around the nostrils [18,21,22,23,24,25,26,27,28] or the airflow [19,20,29] in seated positions. They set the nose or the mouth as the region of interest (ROI) that can be defined manually or automatically by using anatomical features integrated with tracking algorithms [18,21,22,23,25]. They performed by simulated breathing following scenarios that the researcher designed, i.e., regular breathing, fast breathing, and hold breathing [18,21,24,25,26]. The excellent result showed when they took the experiments in a controlled room like temperature, humidity, and light. However, the nose detection during sleep is still unsuccessful at all monitoring times.
In the sleeping position, the thermal-based method is an effective technique to measure the nasal airflow patterns [16] and has been utilized to detect sleep apnea [30,31], analyze sleep activity [32], classify body posture [33] during sleep to assist the diagnosis of sleep disorders or evaluation of the quality of sleep. The studies using thermal imaging to monitor the respiration in sleeping position are reviewed in Table 1. Usman et al. [30] adopted thermal imaging to detect sleep apnea and study in a variety of breathing patterns. They used the Kanade–Lucas–Tomasi tracking algorithm to track the manual selected nose region. The result has shown that 16% of a subject’s head position did not allow correct identification of the region of interest at nostrils. Therefore, this method was only possible with minor head movement without changing position. The automatic ROI selection was used to locate the nostrils, the tip of the nose, and mouth area [31,34,35]. That ROI requires a tracking algorithm and works well without large head movement under a controlled environment. Abbas et al. [16] developed respiration monitoring for neonatal intensive care units by manually select the ROI around the nostrils of infant. Most techniques work well when the nose is clearly visible in the image. The measurement was not feasible when the nose is outside the camera’s field of view, a blanket blocks the nose, or the subject has large head movements. Recent works from Pereira et al. and Lorato et al. [36,37] detected the respiration signal without the use of anatomical features. They selected the ROI containing the respiration information by using the Signal Quality Index to analyze the ROIs. However, they took an experiment in a controlled environment with a short period that is not a real environment. Moreover, the motion artifacts are still a significant drawback of the proposed algorithm. It was suitable for monitoring infants in neonatal care who did not have large movements.
The present study aims to develop the measuring system capable of non-contact monitoring of respiration and body movements in natural sleep environments using a thermal video. The natural sleep environments imply an uncontrolled sleep posture, darkness, and covered subjects with a blanket. In the study, the different approaches based on temperature detection and motion detection were investigated to extract the respiration signal, and then the suitable approach is selected. Our main contributions of this paper are to present: (1) the respiration monitoring based on ROI detection combined with breathing motion, which does not require facial landmarks’ visibility; (2) body movement detection to estimate the numbers of movement during sleeping that affect sleep quality. We validated the proposed approach by comparing it with the signal obtained from the respiratory belt.

2. Proposed Method

In this section, the proposed method for respiration monitoring and body movements detection is described. An overview of the proposed method is depicted in Figure 1. The input of the proposed method is the thermal video obtained under darkness light. The Gaussian filter is applied to the input images as the pre-processing so as to remove noises from the input. The main part of the proposed method is composed of respiration monitoring and body movements detection, each of which utilizes image processing and signal processing techniques in order. Details of these processes will be written below.

2.1. Respiration Monitoring

The respiration monitoring method contains an automatic detection of ROIs by finding the highest temperature point and the largest portion of the high-temperature area and a breathing motion detection. The respiration signals extracted by automatic ROI detection and breathing motion detection are integrated. Then the signal processing is applied to calculate the respiratory rate.

2.1.1. Automatic ROI Detection

We employ the ROI detection to limit the observation area, extracting important information to raise the accuracy of the respiratory estimation. Determining a suitable ROI position with proper size is also important. In the sleep monitoring environment, it is not easy to detect a face or nostrils as an ROI because of the uncontrolled sleep posture, and the fixed camera position may mean that they could make a face that does not appear in the camera view on some occasions. Besides, when the subject changes the sleep posture, ROI should be updated to the new location for which some research applied a tracking algorithm, as summarized in Section 1. The tracking algorithm works well with an apparent object, but sometimes fails to track the nose or mouth in a sleeping posture. In this work, we propose an ROI detection on the thermal image in a sleeping position that does not require a tracking algorithm. Two different ROI detections are considered: (1) The highest temperature point detection and (2) The largest portion of high-temperature area detection.
(1)
The highest temperature point is detected by using minMaxLoc. The minMaxLoc function is one of the OpenCV [38] libraries that returns minimum and maximum intensities found in an image with their (x,y) coordinates. It is assumed that the maximum pixel intensities of the thermal image refer to a human’s heat signature that is not covered by a blanket. The maximum pixel intensities found in the image correspond to the highest temperature of the body. We set the pixel to the center of the observation area. Then we draw a rectangle around the pixel, with the size of the square N × N pixels depending on original frame resolution. In [39], the authors compared the ROI size of 10 × 10 , 25 × 25 , 50 × 50 , 100 × 100 , and 150 × 150 pixels. They found that the size of the ROIs for respiratory rate estimation is usually smaller than that for heart rate estimation. Therefore, in this study, we consider the three different R O I h sizes as 10 × 10 , 25 × 25 , and 50 × 50 , as shown in Figure 2. The result in empirical research has shown that the 50 × 50 pixels provided the highest accuracy in accordance with the original frame resolution of 640 × 480.
(2)
The largest portion of the high-temperature area is detected by using the thresholding method. Thresholding is the method of segmenting the object from the background. The threshold image g ( x , y ) can be defined as (1) [40]:
g ( x , y ) = 1 i f I ( x , y ) T R 0 i f I ( x , y ) < T R
This operation replaces each pixel values in an image with a white pixel if the pixel intensity I ( x , y ) is greater than or equal to the threshold value ( T R ), assigned to a black pixel if the pixel intensity is less than T R . In this work, the white pixels are considered to represent the human skin area.
To determine the threshold value T R , we coordinated empirical research with varying the value among 128, 144, 160, 176, 192, 208 and 224. It was confirmed that the T R value 176 is the one that yielded the best results in all the performed tests.
Figure 3a shows the segmentation of the input image with the thresholding method. Then, we used the findContours function to find the location of white regions that return the outlines corresponding to each of the white blobs on the binary image. The bounding box is drawn around those contours (see Figure 3b). Finally, we find the most prominent contour and bounding box around that contour, as shown in Figure 3c. In this study, we selected the biggest box as R O I t .
Then R O I h and R O I t are cropped for extract signals S h ¯ ( f ) and S t ¯ ( f ) by computing the average of pixel values (2) within each ROI of each frame, where S ( x , y , f ) is the pixel values of the thermal image at pixel ( x , y ) in the video frame f, N * is the vector of pixel coordinates in R O I h or R O I t , and n * is its number.
S * ¯ ( f ) = 1 n * x , y N * S ( x , y , f )

2.1.2. Breathing Motion Detection

Breathing motion detection applies, in the background, a subtraction method for detecting the motion by calculating the difference between the current frame and previous frame. Specifically, absolute difference for all pixels between the current frame I ( x , y , f ) and the frame one second before I ( x , y , f s ) is calculated (3):
B ( x , y , f ) = | I ( x , y , f ) I ( x , y , f s ) | ,
where s is the frame rate.
Then, we extract portions of moved area by using thresholding, erosion, and dilation operations. Parameters used in these operations are 5 for thresholding pixel value difference is less than 5 and 5 × 5 kernel for opening (i.e., erosion and dilation). Next, bounding boxes are determined by finding contours with filtering out small movement as noise. Finally, the number of bounding boxes are counted as the metric of breathing motion (BM).

2.1.3. Respiration Signal Analysis

There are three respiration signals extracted by ROIs detection based on temperature detection and breathing motion detection. The following are the steps to estimate the respiration signals.
(1)
The respiration can be extracted by detecting the chest movements, the breaths airflow, and the temperature change around the nostrils. However, the detection in a specific method cannot be guaranteed in the sleep monitoring because of the fixed camera position, and an independent subject posture may make the region out from the camera view. In such a case, an alternative method for respiration detection is required. It is reasonable to assume that the respiration can be detected by blending the temperature change of ROIs and breathing motion. Therefore, we combine three signals by employing the root mean square (RMS) to calculate the average of the respiration signals as (4).
R e s p i r a t i o n   s i g n a l s = R O I t 2 + R O I h 2 + B M 2
(2)
The 3rd order of Butterworth bandpass filter [41] with a lower cutoff frequency of 0.05 Hz and a higher cutoff frequency of 1.5 Hz was applied. The frequency bound is equivalent to 3–90 bpm, based on the typical RR for an adult person (12–20 bpm) and monitoring the abnormal RR that is less than 12 bpm and higher than 20 bpm.
(3)
The Savitzky–Golay (SG) filter is a least-square polynomial filter that reduces noises while retaining the shape and height of waveform peaks [42]. Here, the SG filter was used to smooth the signal after the bandpass filter. The SG filter’s output increased the precision of the data without distorting the signal tendency. There are two parameters of the SG filter, including window length and the filter order, which closely relates to the performance of the filter. In this study, we tested the parameters and selected the optimal values to get the best-filtered signal, i.e., the window length of 51 and the polynomial order at 3rd were used. The result of SG filter still includes the small peaks, and thus a moving average is calculated to detect only the desired peaks and ignore small ones.
(4)
The fusion signal in Figure 4a was smoothed by the SG filter and moving average (see Figure 4b), and then the number of peaks is counted. Figure 4c depicted the peaks detection of the experiment signal, followed by the peaks detection of the reference signal in Figure 4d, which are assumed to correspond to the number of breaths. The findpeaks function is used with adjusting the width as 10 based on empirical research.
(5)
The number of peaks is calculated as breaths per minute (bpm) for each 60 s slice of input video (1020 samples at 17 fps) and was compared with the reference RR. For performance comparison, the accuracy of the RR estimation was tested using the RMSE defined as (5)
R M S E = 1 N i = 1 N x i e x p x i r e f 2
where N is the total number of the slices, and x i e x p and x i r e f represent the experimented and reference RR values obtained for slice.

2.2. Body Movements Detection

This process aims to determine the more significant action than the respiration representing the big movement like limb movement, head movement, and full position change during sleep. First, an absolute difference image B ( x , y , f ) between adjust two thermal images ( I ( x , y , f ) and I ( x , y , f 1 ) ) is obtained by B ( x , y , f ) = | I ( x , y , f ) I ( x , y , f 1 ) | . Then, we binarize the difference image by thresholding method.
In the same manner as the breathing motion detection, we extract portions of the moved area by erosion and dilation operations. While using 35 for thresholding in order to detect large body movements, the same parameters are used for erosion and dilation. Then, we apply the findContour function to examine whether those are a portion of moved area. If any contour is found, the body movement signal is set to 1. An example output of the body movement is shown in Figure 5, where the left panel is the movement detection, and the right panel plots the movement signal.

3. Experimental Results

This section analyzes the signal gathered in two experiments. Experiment (1) the respiration monitoring, and experiment (2) the body movements detection during sleep. The results were compared with the reference signals obtained by the Go Direct Respiration Belt.

3.1. Experimental Setup

We assessed the performance of our proposed non-contact monitoring of respiration and body movement detection under natural sleep environments. During the experiments, the thermal videos were captured using a portable thermal camera (Seek Thermal Compact PRO for iPhone) attached to a smartphone and fixed on a tripod in front of a participant located at approximately 100 cm. The camera was set at the proper position so that the upper body of a participant was apparent in the camera’s field of view. The Seek Thermal Compact PRO is a highly portable thermal imaging camera with a wide, 32-degree field of view. This thermal camera has a resolution of 640 × 480 pixels and detects infrared wavelengths in the spectral range of 7.5 to 14 Microns. The camera’s emissivity was set to 0.97, as this is suitable for human skin temperature measurement [43]. Besides, the videos were recorded at 17 frames per second (fps). The Go Direct Respiration Belt [44] was used as a reference to collect human respiratory effort and respiratory rate from a force sensor and an adjustable nylon strap around the chest during respiration. The measuring parameters were set to 10 samples/s, and the duration is approximately 5400 s.
The data were collected on different days, from multiple camera positions with volunteers wearing different clothing. The experiments were conducted on real-life conditions, and volunteers were invited to record in their room while they were sleeping. They placed a respiratory belt around their ribs and mounted a thermal camera on a tripod by themselves before they go to bed. Figure 6 illustrates an environment setup. Sixteen healthy volunteers with ages between 25 years old and 37 years old ( 29.88 ± 3.26 years old), ten females and six males, with height between 151 cm and 180 cm ( 162.63 ± 7.37 cm), weight between 47 kg and 78 kg ( 57.38 ± 9.28 kg) and body mass index (BMI) between 18.65 kg/m 2 and 27.64 kg/m 2 ( 21.64 ± 2.98 kg/m 2 ) volunteered for this experiment. Table 2 shows the participants’ data.

3.2. Respiratory Rate Estimation and Body Movements Detection

Table 3 contains the results obtained for all subjects, including respiratory rate estimation and body movement detection. The respiratory rate estimated by our proposed method was compared with the reference signal obtained by the respiratory belt. The RMSE was calculated by considering all the breaths in each signal collected during the experiment minute-by-minute. The average respiratory rate in the overall subjects is 14.78 ± 1.93 bpm for the reference signal and 14.47 ± 0.60 bpm for the proposed approach. The standard deviation of RMSE for the respiratory rate of all subjects is 0.75 bpm, and the average is 1.82 bpm. The small RMSE indicates that the proposed approach is robust for subjects variation. As for body movements detection, we counted the number of movement, the number of frames including body movement, and the total duration of body movements as summarized in Table 3.
The histogram of the number of body movements in every 40 s is also calculated to check the symptoms of PLMD. A small movement like a limb or head movement while sleeping takes a short duration, while significant movements or position changes can take a long time. Therefore, we calculated the degree of body movements by dividing the movement period with the number of movements, which is assumed to be closely related to sleep quality. Figure 7 shows the respiratory rate and body movements of S01–S16. The blue bar represents an occurred body movement. Several times in the event, in the beginning, refer to difficulty falling asleep like subject 4 and subject 8. The reference and experiment’s respiratory rates were plotted ‘x’ and ‘o’, respectively. The blue column represents the histogram of body movements every 40 s. From this table, we can confirm that there was no regular and repetitive body movements for all subjects during experiments, which is the typical phenomenon of PLMD.
Our proposed approach provides respiration and body movements monitoring, which no existing thermal image based sleep monitoring system presents. We believe that body movement detection would characterize abnormal movements and behaviors during sleep, and is more comfortable for the users and completely unobtrusive.

4. Conclusions

This paper proposed an approach for non-contact respiration monitoring and body movements detection in a natural sleep environment using a thermal camera. The thermal camera can handle many viewing angles, which makes installation in the bedroom easy. We have to overcome specific challenges to acquire non-contact respiration data from participants in their natural sleep environment when the lights were turned off and they were covered by a blanket. The thermal video sleep monitoring can be performed in a dark environment to settle privacy concerns. The participants were asked to set up the system and perform a recording by themselves at their home. This approach aims to use for screening the irregular respiration before going to the hospital.
The proposed approach consists of automatically computing the ROIs that it can use to acquire the respiration signal and detecting the body movements of the participant by employing an image processing on the continuous thermal image. The signals were obtained in each frame process with normalizing and smoothing signals. Then, we computed the number of breathing and counted the number of body movements. The approach has been validated using a respiratory belt as a reference signal. We evaluate respiration monitoring performance and body movements detection in different rooms with 16 participants who have independent sleep postures. Our results show that the proposed approach successfully estimated that the RR was obtaining an RMSE of 1.82 ± 0.75 bpm. The performed experiments confirmed that a thermal camera is easy to use for respiration monitoring and body movements during sleeping within various environments. In future work, we will focus on monitoring a patient who has irregular breathing. Another element of our work that could be improved is the automatic optimization of the thresholding value. The other limitations of the proposed method, such as a variation of room temperature, the type of bed cover, blanket, and night sweats (neck or face) in subjects, rapid eye movement (REM) stage, and heart rate, provide ideas for addressing these issues in future studies.

Author Contributions

Conceptualization, T.O. and P.J.; methodology, P.J.; software, P.J.; validation, P.J.; formal analysis, P.J.; investigation, P.J.; resources, T.O.; data curation, P.J.; writing—original draft preparation, P.J.; writing—review and editing, T.O. and P.J.; visualization, P.J.; supervision, T.O.; project administration, P.J.; funding acquisition, T.O. All authors have read and agreed to the published version of the manuscript.

Funding

This research received no external funding.

Conflicts of Interest

The authors declare no conflict of interest.

References

  1. Wheatley, I. Respiratory rate 3: How to take an accurate measurement. Available online: https://www.nursingtimes.net/clinical-archive/respiratory-clinical-archive/respiratory-rate-3-how-to-take-an-accurate-measurement-25-06-2018/ (accessed on 30 January 2020).
  2. Sheppard, M.; Wright, M.W. Principles and Practice of High Dependency Nursing; Elsevier Health Sciences: Amsterdam, The Netherlands, 2006. [Google Scholar]
  3. Cretikos, M.; Chen, J.; Hillman, K.; Bellomo, R.; Finfer, S.; Flabouris, A.; Investigators, M.S. The objective medical emergency team activation criteria: A case–control study. Resuscitation 2007, 73, 62–72. [Google Scholar] [CrossRef] [PubMed]
  4. Somers, V. American Heart Association Council for High Blood Pressure Research Professional Education Committee, Council on Clinical Cardiology. American Heart Association Stroke Council. American Heart Association Council on Cardiovascular Nursing. American College of Cardiology Foundation. Sleep apnea and cardiovascular disease: An American Heart Association/American College Of Cardiology Foundation Scientific Statement from the American Heart Association Council for High Blood Pressure Research Professional Education Committee, Council on Clinical Cardiology, Stroke Council, and Council On Cardiovascular Nursing. In collaboration with the National Heart, Lung, and Blood Institute National Center on Sleep Disorders Research (National Institutes of Health). Circulation 2008, 118, 1080–1111. [Google Scholar] [PubMed]
  5. Meoli, A.L.; Casey, K.R.; Clark, R.W.; Coleman, J.A.; Fayle, R.W.; Troell, R.J.; Iber, C.; Clinical Practice Review Committee. Hypopnea in sleep-disordered breathing in adults. Sleep 2001, 24, 469–470. [Google Scholar] [PubMed]
  6. Harvard Medical School, H.U. Understanding the Results: Sleep Apnea. Available online: http://healthysleep.med.harvard.edu/sleep-apnea/diagnosing-osa/understanding-results (accessed on 20 February 2020).
  7. Zagaria, M.A.E. Periodic Limb Movement Disorder, Restless Legs Syndrome, and Pain. US Pharm. 2015, 40, 19–21. [Google Scholar]
  8. Madhushri, P.; Ahmed, B.; Penzel, T.; Jovanov, E. Periodic leg movement (PLM) monitoring using a distributed body sensor network. In Proceedings of the 2015 37th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC), Milan, Italy, 25–29 August 2015; pp. 1837–1840. [Google Scholar]
  9. Koolen, N.; Decroupet, O.; Dereymaeker, A.; Jansen, K.; Vervisch, J.; Matic, V.; Vanrumste, B.; Naulaers, G.; Van Huffel, S.; De Vos, M. Automated Respiration Detection from Neonatal Video Data. In Proceedings of the International Conference on Pattern Recognition Applications and Methods (ICPRAM-2015), Lisbon, Portugal, 10–12 January 2015; pp. 164–169. [Google Scholar]
  10. AL-Khalidi, F.Q.; Saatchi, R.; Burke, D.; Elphick, H. Tracking human face features in thermal images for respiration monitoring. In Proceedings of the ACS/IEEE International Conference on Computer Systems and Applications-AICCSA 2010, Hammamet, Tunisia, 16–19 May 2010; pp. 1–6. [Google Scholar] [CrossRef] [Green Version]
  11. Lee, Y.S.; Pathirana, P.N.; Steinfort, C.L.; Caelli, T. Monitoring and analysis of respiratory patterns using microwave doppler radar. IEEE J. Transl. Eng. Health Med. 2014, 2, 1–12. [Google Scholar] [CrossRef] [PubMed]
  12. Al-Naji, A.; Chahl, J. Remote respiratory monitoring system based on developing motion magnification technique. Biomed. Signal Process. Control 2016, 29, 1–10. [Google Scholar] [CrossRef]
  13. Nakajima, K.; Matsumoto, Y.; Tamura, T. A monitor for posture changes and respiration in bed using real time image sequence analysis. In Proceedings of the 22nd Annual International Conference of the IEEE Engineering in Medicine and Biology Society (Cat. No. 00CH37143), Chicago, IL, USA, 23–28 July 2000; Volume 1, pp. 51–54. [Google Scholar]
  14. Frigola, M.; Amat, J.; Pagès, J. Vision based respiratory monitoring system. In Proceedings of the 10th Mediterranean Conference on Control and Automation (MED 2002), Lisbon, Portugal, 9–12 July 2002; pp. 9–13. [Google Scholar]
  15. Wiesner, S.; Yaniv, Z. Monitoring patient respiration using a single optical camera. In Proceedings of the 2007 29th Annual International Conference of the IEEE Engineering in Medicine and Biology Society, Lyon, France, 22–26 August 2007; pp. 2740–2743. [Google Scholar]
  16. Abbas, A.K.; Heimann, K.; Jergus, K.; Orlikowsky, T.; Leonhardt, S. Neonatal non-contact respiratory monitoring based on real-time infrared thermography. Biomed. Eng. Online 2011, 10, 93. [Google Scholar] [CrossRef] [Green Version]
  17. Fei, J.; Pavlidis, I. Thermistor at a distance: Unobtrusive measurement of breathing. IEEE Trans. Biomed. Eng. 2009, 57, 988–998. [Google Scholar]
  18. Lewis, G.F.; Gatto, R.G.; Porges, S.W. A novel method for extracting respiration rate and relative tidal volume from infrared thermography. Psychophysiology 2011, 48, 877–887. [Google Scholar] [CrossRef] [Green Version]
  19. Murthy, R.; Pavlidis, I. Noncontact measurement of breathing function. IEEE Eng. Med. Biol. Mag. 2006, 25, 57–67. [Google Scholar] [CrossRef] [PubMed]
  20. Murthy, R.; Pavlidis, I.; Tsiamyrtzis, P. Touchless monitoring of breathing function. In Proceedings of the 26th Annual International Conference of the IEEE Engineering in Medicine and Biology Society, San Francisco, CA, USA, 1–5 September 2004; Volume 1, pp. 1196–1199. [Google Scholar]
  21. Pereira, C.B.; Yu, X.; Czaplik, M.; Rossaint, R.; Blazek, V.; Leonhardt, S. Remote monitoring of breathing dynamics using infrared thermography. Biomed. Opt. Express 2015, 6, 4378–4394. [Google Scholar] [CrossRef] [PubMed]
  22. Alkali, A.H.; Saatchi, R.; Elphick, H.; Burke, D. Facial tracking in thermal images for real-time noncontact respiration rate monitoring. In Proceedings of the 2013 European Modelling Symposium, Manchester, UK, 20–22 November 2013; pp. 265–270. [Google Scholar]
  23. Alkali, A.H.; Saatchi, R.; Elphick, H.; Burke, D. Thermal image processing for real-time non-contact respiration rate monitoring. IET Circuits Devices Syst. 2017, 11, 142–148. [Google Scholar] [CrossRef]
  24. Bennett, S.L.; Goubran, R.; Knoefel, F. The detection of breathing behavior using Eulerian-enhanced thermal video. In Proceedings of the 2015 37th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC), Milan, Italy, 25–29 August 2015; pp. 7474–7477. [Google Scholar] [CrossRef]
  25. Pereira, C.B.; Yu, X.; Czaplik, M.; Blazek, V.; Venema, B.; Leonhardt, S. Estimation of breathing rate in thermal imaging videos: A pilot study on healthy human subjects. J. Clin. Monit. Comput. 2017, 31, 1241–1254. [Google Scholar] [CrossRef]
  26. Kwasniewska, A.; Szankin, M.; Ruminski, J.; Kaczmarek, M. Evaluating Accuracy of Respiratory Rate Estimation from Super Resolved Thermal Imagery. In Proceedings of the 2019 41st Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC), Berlin, Germany, 23–27 July 2019; pp. 2744–2747. [Google Scholar]
  27. Jagadev, P.; Giri, L.I. Non-contact monitoring of human respiration using infrared thermography and machine learning. Infrared Phys. Technol. 2020, 104, 103117. [Google Scholar] [CrossRef]
  28. Al-khalidi, D.F.; Saatchi, R.; Elphick, H.; Burke, D. Tracing the Region of Interest in Thermal Human Face for Respiration Monitoring. Int. J. Comput. Appl. 2015, 119, 42–46. [Google Scholar] [CrossRef]
  29. Fei, J.; Pavlidis, I. Analysis of breathing air flow patterns in thermal imaging. In Proceedings of the 2006 International Conference of the IEEE Engineering in Medicine and Biology Society, New York, NY, USA, 30 August–3 September 2006; pp. 946–952. [Google Scholar]
  30. Usman, M.; Evans, R.; Saatchi, R.; Kingshott, R.; Elphick, H. Non-invasive respiration monitoring by thermal imaging to detect sleep apnoea. submitted.
  31. Fei, J.; Pavlidis, I.; Murthy, J. Thermal vision for sleep apnea monitoring. In Proceedings of the International Conference on Medical Image Computing and Computer-Assisted Intervention, London, UK, 20–24 September 2009; Springer: Berlin/Heidelberg, Germany, 2009; pp. 1084–1091. [Google Scholar]
  32. Seba, A.; Istrate, D.; Guettari, T.; Ugon, A.; Pinna, A.; Garda, P. Thermal-Signature-Based Sleep Analysis Sensor. Informatics 2017, 4, 37. [Google Scholar] [CrossRef] [Green Version]
  33. Chen, Z.; Wang, Y. Sleep monitoring using an infrared thermal array sensor. In Sensors and Smart Structures Technologies for Civil, Mechanical, and Aerospace Systems; International Society for Optics and Photonics: Bellingham, WA, USA, 2019; Volume 10970, p. 109701D. [Google Scholar]
  34. Al-Kalidi, F.; Elphick, H.; Saatchi, R.; Burke, D. Respiratory rate measurement in children using a thermal camera. Int. J. Sci. Eng. Res. 2015, 6, 1748–1756. [Google Scholar]
  35. Hu, M.; Zhai, G.; Li, D.; Fan, Y.; Duan, H.; Zhu, W.; Yang, X. Combination of near-infrared and thermal imaging techniques for the remote and simultaneous measurements of breathing and heart rates under sleep situation. PLoS ONE 2018, 13, e0190466. [Google Scholar] [CrossRef] [Green Version]
  36. Pereira, C.B.; Yu, X.; Goos, T.; Reiss, I.; Orlikowsky, T.; Heimann, K.; Venema, B.; Blazek, V.; Leonhardt, S.; Teichmann, D. Noncontact monitoring of respiratory rate in newborn infants using thermal imaging. IEEE Trans. Biomed. Eng. 2018, 66, 1105–1114. [Google Scholar] [CrossRef]
  37. Lorato, I.; Bakkes, T.; Stuijk, S.; Meftah, M.; De Haan, G. Unobtrusive respiratory flow monitoring using a thermopile array: A feasibility study. Appl. Sci. 2019, 9, 2449. [Google Scholar] [CrossRef] [Green Version]
  38. Opencv Dev Team. Operations on Arrays. 2019. Available online: https://docs.opencv.org/3.4/d2/de8/group__core__array.html (accessed on 20 February 2020).
  39. Tarassenko, L.; Villarroel, M.; Guazzi, A.; Jorge, J.; Clifton, D.A.; Pugh, C. Non-contact video-based vital sign monitoring using ambient light and auto-regressive models. Physiol. Meas. 2014. [Google Scholar] [CrossRef]
  40. Kumbhar, P.G.; Holambe, S.N. A Review of Image Thresholding Techniques. Int. J. Adv. Res. Comput. Sci. Softw. Eng. 2015, 5, 160–163. [Google Scholar]
  41. Virtanen, P.; Gommers, R.; Oliphant, T.E.; Haberland, M.; Reddy, T.; Cournapeau, D.; Burovski, E.; Peterson, P.; Weckesser, W.; Bright, J.; et al. SciPy 1.0: Fundamental Algorithms for Scientific Computing in Python. Nat. Methods 2020, 17, 261–272. [Google Scholar] [CrossRef] [Green Version]
  42. Schafer, R.W. What is a Savitzky-Golay filter? [lecture notes]. IEEE Signal Process. Mag. 2011, 28, 111–117. [Google Scholar] [CrossRef]
  43. Sanchez-Marin, F.J.; Calixto-Carrera, S.; Villaseñor-Mora, C. Novel approach to assess the emissivity of the human skin. J. Biomed. Opt. 2009, 14, 024006. [Google Scholar] [CrossRef]
  44. Vernier. Go Direct® Respiration Belt. Available online: https://www.vernier.com/manuals/gdx-rb/ (accessed on 20 March 2020).
Figure 1. Proposed method.
Figure 1. Proposed method.
Sensors 20 06307 g001
Figure 2. The sample of three different R O I h sizes as 10 × 10 , 25 × 25 , and 50 × 50 (red point indicates the highest temperature point).
Figure 2. The sample of three different R O I h sizes as 10 × 10 , 25 × 25 , and 50 × 50 (red point indicates the highest temperature point).
Sensors 20 06307 g002
Figure 3. (a) All contours, (b) bounding rectangles around all contours, and (c) bounding rectangle around the most prominent contour.
Figure 3. (a) All contours, (b) bounding rectangles around all contours, and (c) bounding rectangle around the most prominent contour.
Sensors 20 06307 g003
Figure 4. (a) Sample of fusion signal, (b) filtered and smoothed signals, (c) peak detection of experiment signal, and (d) peak detection of the reference signal.
Figure 4. (a) Sample of fusion signal, (b) filtered and smoothed signals, (c) peak detection of experiment signal, and (d) peak detection of the reference signal.
Sensors 20 06307 g004
Figure 5. Sample output of the body movements detection.
Figure 5. Sample output of the body movements detection.
Sensors 20 06307 g005
Figure 6. Data collection setup.
Figure 6. Data collection setup.
Sensors 20 06307 g006
Figure 7. The result of the RMSE (respiratory rate) and body movements of S01–S16.
Figure 7. The result of the RMSE (respiratory rate) and body movements of S01–S16.
Sensors 20 06307 g007
Table 1. A research review of the thermal imaging-based method for respiration monitoring of sleeping position.
Table 1. A research review of the thermal imaging-based method for respiration monitoring of sleeping position.
ROI Localization
AuthorsSubjectsExp DurationControlled EnvSimulated
Breathing
Selection/
Detection
AreaTracking
Usman et al. [30]Adult5 minYesYesMNostrilsYes
Fei et al. [31]Adult60 minYesNoA-SNostrilsYes
Al-Khalidi et al. [34]Children2 minYesNoA-STip of the noseYes
Hu et al. [35]Adult10 minYesYesA-SNose, mouthYes
Abbas et al. [16]Infant2 minYesNoMNostrilsNo
Pereira et al. [36]Infant5 minNoNoA-DN/ANo
Lorato et al. [37]Adult2 minYesYesA-DN/ANo
Our proposedAdult60–90 minNoNoA-DN/ANo
M: Manually, A-S: Automatically Selection, A-D: Automatically Detection.
Table 2. Participants Data.
Table 2. Participants Data.
SubjectsGenderAge (years)Height (cm)Weight (kg)BMI (kg/m 2 )
S01F281625621.34
S02F361675218.65
S03F311625019.05
S04F291635319.95
S05F321585421.63
S06M251617027.01
S07F311514720.61
S08F291605019.53
S09M301685519.49
S10F281595822.94
S11M281807523.15
S12M261695820.31
S13M271685920.90
S14M291687827.64
S15F321535623.92
S16F371534720.08
Table 3. The result of respiratory rate estimation and body movements detection.
Table 3. The result of respiratory rate estimation and body movements detection.
SubjectsRespiratory Rate (bpm)Body Movements
Duration (s)ReferenceExperimentRMSE#Movements#FramesDuration (s)Degree
S015371.0512.7114.051.561426915.691.12
S025397.5413.6914.221.111519911.650.78
S035379.3716.7514.782.207633.690.53
S045192.3112.2313.372.003564237.861.08
S055212.7817.6214.393.32920011.721.30
S065200.5116.4514.482.23921412.531.39
S075332.3914.3814.361.471621812.800.80
S083495.3914.6514.291.181574943.482.90
S095407.2212.1714.602.68030.170.00
S104520.2614.9114.790.755915.331.07
S115346.7013.1213.761.251641724.421.53
S125361.4513.2515.372.3571408.201.17
S135399.7418.6115.992.795201.170.23
S145380.5215.1514.321.491653531.141.95
S155315.2316.3214.401.991222513.191.10
S164287.1214.4314.410.72625014.512.42
Mean5100.0014.7814.471.82 1.21
STD537.631.930.600.75 0.74
Publisher’s Note: MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Share and Cite

MDPI and ACS Style

Jakkaew, P.; Onoye, T. Non-Contact Respiration Monitoring and Body Movements Detection for Sleep Using Thermal Imaging. Sensors 2020, 20, 6307. https://doi.org/10.3390/s20216307

AMA Style

Jakkaew P, Onoye T. Non-Contact Respiration Monitoring and Body Movements Detection for Sleep Using Thermal Imaging. Sensors. 2020; 20(21):6307. https://doi.org/10.3390/s20216307

Chicago/Turabian Style

Jakkaew, Prasara, and Takao Onoye. 2020. "Non-Contact Respiration Monitoring and Body Movements Detection for Sleep Using Thermal Imaging" Sensors 20, no. 21: 6307. https://doi.org/10.3390/s20216307

Note that from the first issue of 2016, this journal uses article numbers instead of page numbers. See further details here.

Article Metrics

Back to TopTop