Life Signs Detector Using a Drone in Disaster Zones

Abstract

In the aftermath of a disaster, such as earthquake, flood, or avalanche, ground search for survivors is usually hampered by unstable surfaces and difficult terrain. Drones now play an important role in these situations, allowing rescuers to locate survivors and allocate resources to saving those who can be helped. The aim of this study was to explore the utility of a drone equipped for human life detection with a novel computer vision system. The proposed system uses image sequences captured by a drone camera to remotely detect the cardiopulmonary motion caused by periodic chest movement of survivors. The results of eight human subjects and one mannequin in different poses shows that motion detection on the body surface of the survivors is likely to be useful to detect life signs without any physical contact. The results presented in this study may lead to a new approach to life detection and remote life sensing assessment of survivors.

1. Introduction

In the case of disaster relief, the search operation for survivors is a challenge for lifesavers in conditions when contact with a survivor is a difficult, potentially unsafe, and very stressful task, leading to cognitive and physical fatigue. Recent advances in unmanned aerial vehicle (UAV), now synonymously, “drone” technologies will be suitable for this purpose and can help in the effort to search for living survivors in the aftermath of a disaster [1,2,3]. Current ground-based options for detecting humans, such as rescue robots, are expensive, require invasive equipment, and require a team of human operators to move around inside the search and rescue operation [4,5]. Furthermore, accessing a disaster zone is more difficult for rescue robots than for rescue workers and rescue dogs due to limited mobility of wheeled and tracked platforms. To address these challenges, we are developing a highly autonomous drone to assist first responders in their search to find survivors in disaster zones.

The task of observing and analysing humans from an aerial robot in outdoor environments has been of interest to the computer vision community for the last decade. Techniques to address the problem of estimating human pose and trajectory from the sky have been suggested. For example, Perera et al. [6,7] proposed an approach to estimation of the gait sequence and movement trajectory of human subjects from video captured by a drone. However, their proposed approach was limited to a fully visible subject and a limited range of poses such as standing or walking. Another study by Lygouras et al. [8] used UAV videos for the identification and location of open water swimmers. The proposed system detected the swimmers’ presence by applying a deep learning architecture and it was capable of accurately detecting swimmers without apparent human intervention. However, the proposed system may fail in some environmental conditions and no life detection was attempted in that study. Other techniques for detecting human subjects in aerial videos have been proposed for search and rescue operations in disaster scenarios [2,9,10,11,12]. The main focus of these studies was to determine humans’ locations and identify various humans poses on the ground without any detection of life signs of the subjects. Other techniques using aerial thermal videos have been suggested. For example, a study by Kang et al. [13] proposed a technique for detecting and tracking human subjects from the air based on thermal images captured by a UAV. A study by Portmann et al. [14] used aerial thermal images captured by a UAV for human detection and tracking. Another study by Rudol and Doherty [15] proposed a technique based on a combination of thermal and colour cameras to detect humans on standard hardware onboard an autonomous UAV in outdoor environments. A combination of thermal and colour cameras attached to a UAV was also used by Rivera et al. [16] to detect humans with geolocation of the survivors. These studies, however, have several disadvantages, such as short-range detection, low resolution, high cost, and motion artefacts. Human life signs detection was not established for all previous studies. Thermal cameras only detect live subjects where there is thermal contrast against the background, thus, there is a lack of ability to specifically detect the temperature of deceased subjects and limited ability to detect living subjects in warm environments, against warm backgrounds, or in the presence of insulated clothing when thermal cameras are used. A notable system proposed by Al-Kaff et al. [17] was developed to detect humans lying on the ground in unconstrained poses using the colour and depth data obtained from sensors on board the UAV. Although this system was efficient for detecting humans in many poses, it was limited to detecting human forms without any aspect of life signs detection and was limited to close range. Recent studies by Al-Naji et al. [18,19] proposed a robust technique for monitoring human vital signs (heart rate and breathing rate remotely from the video taken by a drone). The technique used imaging photoplethysmography (iPPG) based on skin colour analysis of the facial region at a distance of 3 meters. However, the technique relied on skin colour analysis more than motion analysis which means it is difficult to apply when the region of interest (ROI) is unclear due to occlusion by clothing, debris, blood, or when the subject is lying face down. In addition, the technique was limited to one pose where the subject was standing in front of the UAV. The experiments were further limited in not controlling for human forms without life signs. Therefore, the main contribution of the current study is to propose a new life signs detector system based on a drone observer that can efficiently detect living and non-living human subjects at a distances limited by optics, in these experiments, 4–8 meters, and at different poses intended for disaster zones.

This paper is organised as follows: Section 2 presents the methods and materials of the proposed life detector system, including the subjects and ethics considerations, experimental setup, data collection, system framework, and data analysis. The MATLAB graphical user interface (MathWorks, NSW, Australia) and the experimental results of the proposed life signs detection system with different poses are presented in Section 3. Section 4 discusses the experimental results, limitations, and directions of future work. Finally, concluding remarks are outlined in Section 5.

2. Methods and Materials

2.1. Human Ethics Considerations

The research procedure described in this study was carried out in accordance with the ethical standards of the Declaration of Helsinki (ethical principles for medical research involving human subjects), and approved by the University of South Australia, Human Research Ethics Committee (HREC) (Mawson Lakes, South Australia, Protocol number: 0000035185). A written description of the experimental procedures was provided to the participants who signed an informed consent form before commencing the experiment.

This study is the first of its type and was performed with a range of living subjects and a mannequin to emulate a deceased person. Studies with the deceased require a high degree of perceived societal benefit that may be fitting for mature development and certification from products resulting from work on this topic. We have thus focused on a high diversity of appearance of living human subjects rather than on incorporating deceased persons.

2.2. Experimental Setup and Data Acquisition

The video data was captured using a GoPro Hero 4 black action camera (GoPro Inc., San Mateo, CA, USA) with replacement lens (5.4 mm, 10MP, infrared IR cut filter) to reduce the fish-eye effect and narrow the field of view mounted on a three-axis 3D Robotics Solo gimbal. To reduce the vibration of the video camera installed on the drone, a standard gimbal was used to minimise the camera vibrations and stabilise the footage. The camera was attached to the drone with a three-axis gimbal (3DR Solo gimbal, 3DR Inc., Berkeley, CA, USA). The gimbal handles camera attitude and stabilisation autonomously and records stabilised videos within 0.1 degree of pointing accuracy. The vibration of the motors transferred to the camera is minimised by the gimbal.

The videos were sampled at 25 frames per second and the resolution was 3840 × 2160 pixels with sensitivity of the image sensor at 400 for the International Standards Organization (ISO). The videos were captured at an altitude of between 4 and 8 metres from the subject as shown in Figure 1.

The experiment was performed using eight human subjects (four males: South Asian, Middle Eastern, two Caucasians, and four females: one Sub-Saharan African, two Middle Eastern, one Asian) aged from 20 to 40 years and one full-body male mannequin. All subjects were asked to lie down on the ground in different poses and to breathe normally. In addition to the human subjects, a full-body male mannequin with a height of 1.96 m, fully clothed, wearing a black wig was used for data collection to simulate a deceased subject. The mannequin had a realistic face and both the arms and head of the mannequin were adjustable to different angles to create different poses. The videos were acquired during three days in daylight and in relatively low wind conditions for one minute for each subject and repeated at different poses to obtain sufficient video data for experimental purposes. However, we only used 30 s from each video to reduce the execution time which yielded the same detection accuracy compared to longer video recording times. The human subject and the mannequin lying on the ground next to each other was also recorded by the drone. The data acquisition of eight human subjects and one mannequin in different poses is shown in Figure 2.

2.3. System Framework and Data Analysis

The schematic diagram of the proposed system used as a life signs detector from video data captured by a drone is demonstrated in Figure 3.

The motion of the chest wall due to the cardiopulmonary activity for live subjects directly causes variations of reflected intensity values in the image sequences. To detect this motion, several image and video processing techniques are applied in this work to automatically analyse and extract luminance values of interest from video data. Firstly, each video is converted to image sequences and saved as joint photographic experts group (JPEG) format files on the personal computer (PC) for further analysis. The drone camera captured the video in the red, green and blue (RGB) colour model, so, in order to get the image intensity data as the luminance component, the MATLAB’s built-in function, ‘rgb2ycbcr’, was used to convert the RGB colour model into the YCbCr colour model using the following equation:

$$\left[\begin{array}{c}Y\\ C_b\\ C_r\end{array}\right]=\left[\begin{array}{ccc}65.481& 128.553& 24.966\\ -39.797& -74.203 & 112\\ 112& -93.786 & -18.2214\end{array}\right]\left[\begin{array}{c}R\\ G\\ B\end{array}\right]+\left[\begin{array}{c}16\\ 128\\ 128\end{array}\right],$$

(1)

where $Y$ component represents the luminance values that ranged from 16 to 235, while $C_b$ and $C_r$ components are the chrominance values that ranged from 16 to 240 [20,21].

Secondly, the chest region where the cardiopulmonary motion is most pronounced was automatically selected using a body joint estimation approach, called OpenPose [22]. OpenPose is an efficient and robust approach to the challenges associated with unconstrained or multiple subjects that produces high-quality parses of body poses at low computational cost [22]. OpenPose approach can estimate 135 key-points on a single image (18 body joints for each subject). However, we only used the torso joints (neck, left shoulder, right shoulder, left hip, and right hip) to choose the chest region. After determining the detected joints that indicate upper body pose, the torso area was divided into upper and lower torso regions. We considered the upper torso region as our ROI as illustrated in Figure 4.

Thirdly, the luminance pixel values over the image sequences of the selected ROI from the Y component were averaged as follows:

$$I_{Y_{avg}}\left(t\right)=\frac{{\sum }_{x,y\in ROI}{I}_{\left(x,y,t\right)}}{\left|ROI\right|},$$

(2)

where $I_{\left(x,y,t\right)}$ is the luminance pixel value at image location ($x,y$) over time ($t$) from image sequences, and $\left|ROI\right|$ is the size of the selected ROI.

To reduce the motion artefact noise from the $I_{Y_{avg}}\left(t\right)$ signal, resulting from drone movement during videoing, a wavelet signal denoising method using an empirical Bayesian method with a Cauchy prior and a smoothing signal method using a moving average filter were used. MATLAB’s built-in function, called ‘wdenoise’, was firstly used to denoise the signal as shown in Figure 5b, with wavelet Daubechies (dbN) family and a level-dependent denoising method of universal soft threshold that estimates the variance of the noise based on the wavelet coefficients at each resolution level. MATLAB’s built-in function, called ‘smooth’, was secondly used to smooth the denoised signal as shown in Figure 5c.

Lastly, to identify the periodicity of peaks, their locations, and the number of peaks in the smoothed signal, peak detection based on the wavelet transform was used. Continuous wavelet transformation (CWT) is defined as the scalar multiplication of the input signal, $I_{Y_{avg}}\left(t\right)$, and the scaled, shifted versions of the wavelet mother function, ψ. Mathematically, a CWT on a signal, $I_{Y_{avg}}\left(t\right),$ at point $\left(x,y\right)$ is described as [23,24]:

$$W_{\left(x,y\right)}={\int }_{-\infty }^{+\infty }{I}_{Y_{avg}}\left(t\right) {\psi }_{x,y}\left(t\right) dt, {\psi }_{x,y}\left(t\right)=\frac{1}{\sqrt{x}}\psi \left(\frac{t-y}{x}\right),$$

(3)

where $I_{Y_{avg}}\left(t\right)$ is a signal after denoising and smoothing and ${\psi }_{x,y}\left(t\right)$ is $\psi \left(t\right)$ translated by scale $x$ and shifted by $y$. The outcome of CWT coefficients contain patterns of peaks and periodicity which can be used to detect the number of peaks, their locations, and their strengths in $I_{Y_{avg}}\left(t\right)$ of similar size to ${\psi }_{x,y}\left(t\right)$c. Varying x in ${\psi }_{x,y}\left(t\right)$ yields different width wavelets, and so all peaks in $I_{Y_{avg}}\left(t\right)$, regardless of their width, can be detected. According to the patterns of peaks and their periodicity, our proposed system can detect life signs of the target subject.

3. Experimentation and Results

A graphical user interface (GUI) panel was carried out by using the MATLAB program—R2019a (MathWorks, NSW, Australia) with the Microsoft Windows 10 operating system. This panel allows the user to load video data captured by the drone, automatically select the ROI where chest motion is most apparent, and execute the algorithm. The proposed GUI provides an easy tool to see video information, the selected ROI, input signal, denoised/smoothed signal, and number of peaks. It also enables the user to detect the life sign of the target subject (alive or deceased) as illustrated in Figure 6 and Figure 7, respectively. To detect the chest motion on the human subject, our proposed life sign detector system extracts the signal from the selected ROI and applies denoising/smoothing methods on the selected signal and then calculates the number of peaks and their periodicity for each image. After determining the number of peaks for each signal, our proposed system could determine whether the target subject was alive or dead. Figure 6 shows the GUI main panel of the proposed life detector system for a human subject.

Our proposed system considered the mannequin to be a deceased subject since the mannequin had no motion of the chest region, thus, the input signal has no peaks on the selected ROI. Even if some peaks generated by drone motion and other motion artefacts may fall in the signal, the periodicity of the peaks is still nonuniform and our system will consider the mannequin to be a deceased subject. Figure 7 shows the GUI main panel of the proposed life detector system for the deceased subject (mannequin).

The experimental results were carried out for all subjects at four different poses (back position, side position facing the camera, side position non-facing the camera, and stomach position) as shown in Figure 8.

Table 1. The experimental results of eight human subjects and one mannequin at different poses.

Subject	Poses	No. of Peaks	The Proposed Life Signs Detector System
			ALIVE	DECEASED
Subject 1	Pose 1	13	✓
	Pose 2	13	✓
	Pose 3	12	✓
	Pose 4	12	✓
Subject 2	Pose 1	9	✓
	Pose 2	9	✓
	Pose 3	9	✓
	Pose 4	9	✓
Subject 3	Pose 1	11	✓
	Pose 2	11	✓
	Pose 3	11	✓
	Pose 4	12	✓
Subject 4	Pose 1	11	✓
	Pose 2	10	✓
	Pose 3	11	✓
	Pose 4	11	✓
Subject 5	Pose 1	8	✓
	Pose 2	8	✓
	Pose 3	8	✓
	Pose 4	8	✓
Subject 6	Pose 1	7	✓
	Pose 2	8	✓
	Pose 3	8	✓
	Pose 4	8	✓
Subject 7	Pose 1	11	✓
	Pose 2	10	✓
	Pose 3	10	✓
	Pose 4	11	✓
Subject 8	Pose 1	10	✓
	Pose 2	10	✓
	Pose 3	10	✓
	Pose 4	9	✓
Mannequin	Pose 1	1		✓
	Pose 2	1		✓
	Pose 3	0		✓
	Pose 4	0		✓

demonstrates the life detection results of eight human subjects and one mannequin at the poses shown in Figure 8 based on motion caused by cardiopulmonary activity.

The results from Table 1 show that motion detection from the chest region could be effectively and efficiently used to calculate the number of motion peaks for each image and thus to determine whether the subject is alive or deceased for all eight human subjects and one mannequin at different poses.

To detect life signs for multiple subjects, we recorded a video for both the human subject and the mannequin lying on the ground next to each other. The proposed system succeeded in recognising the life signs of these subjects and in determining which was alive and which was not, as shown in Figure 9.

4. Discussion

With the data set available, we have demonstrated that motion-based breathing detection from an aerial platform is possible with high certainty under the conditions tested. The technique could be included in mapping software to enhance drone generated maps of disaster scenes with life sign annotations although substantially more automation and integration would be required.

The system we have proposed successfully distinguished live subjects from a mannequin in daylight, in calm conditions, and using a powerful PC equipped with MATLAB software and a human operator. Each of these conditions is a limitation. Although the participants comprised a reasonable subset of humanity, with South Asian, East Asian, African, Middle Eastern, and European individuals, there were combinations of genders and appearances not tested. The mannequin was somewhat realistic; however, it is apparent to humans that it is not a human from around 8 m, the same distance as the longest ranges tested. Since the experiments were carried out within the altitude of 4–8 m, the size of upper-torso area (ROI) contained enough details to extract the body movements caused by the breathing with a high detection accuracy. The videos recorded above 8 m height had small ROIs and did not have enough image details at our recorded resolution (3840 × 2160 pixels). Therefore, the data recording was limited to the above altitude range for a reasonably sized ROI. Whilst the OpenPose method performed well, there is no doubt that field conditions and partial occlusion of subjects would create challenging situations.

The limit of the range of the camera is more driven by the stability of the gimbal than optics or dynamics of the platform. Although ranges of 4–8 m were tested, it is likely given the clean waveforms in Figure 4 that longer focal length optics would deliver detection at multiples of this range. This possibility would be due to the performance of the gimbal, that might also become far worse under less calm conditions.

Since the means of detection was focused on the cardiopulmonary motion in an ROI, it is also possible that the periodic chest movement alone might be a cue for finding occluded or camouflaged subjects. Such an approach would require significant dwell time on each scene imaged, but it might yield a new capability.

Several researches [25,26,27,28,29,30] have utilised video magnification techniques with human subjects to reveal physiological signs on an appropriate ROI when the cardiopulmonary motion is difficult to see using a vision system. These studies may provide further insight on the technology for future work directions.

5. Conclusions

Globally, recovery of survivors in a disaster zone is a time critical, high-priority activity. In this paper, we have demonstrated for the first time a technology using standard colour cameras, fitted by default to consumer drones, that can detect the presence or absence of life signs from humans and human-shaped objects lying in many poses on the ground. Under the test conditions, the obtained results at distances of 4–8 metres from the subject demonstrated the robustness of the proposed system for detecting life signs at different poses with an accuracy of 100%. We noted that the proposed system is able to efficiently detect the life signs of multiple subjects, showing promise as a future tool for search and rescue operations. Future work will focus on more realistic scenarios and more automation of the technique to ease operational application of the technology. Development of a software simulator for a breathing human in a cluttered simulated scene is a must for achieving a reliable and tested capability due to a large number of variables and scenarios.