1. Introduction
There are different search and rescue techniques, and the choice of which one to use during a mission depends on the terrain. Rescuers use models—such as the ring model [
1,
2], the bike wheel model [
3], and the mobility model [
4]. In addition, numerous field techniques, such as line formation and a rapid triplet search [
5], employing dogs or performing linear searches by quads [
6], are commonly used. Recent years brought an unprecedented interest in using manned [
7] and unmanned [
8] aerial technologies to assist search and rescue missions.
With the advent of unmanned aerial vehicles (UAVs), commonly known as drones, numerous search and rescue teams around the world have employed unmanned systems to assist searches for persons lost in the wilderness. Although the potential of UAVs to facilitate search and rescue missions has been noticed in the 2000s [
9,
10,
11], the first operational applications were reported much later. Though it is impossible to list all successful searches for missing people conducted with the support of drones, a few of them are mentioned herein.
Van Tilburg [
12] was the first to report on the operational use of a portable UAV in a search and rescue mission, during which a human body was spotted by an analyst who viewed a signal recorded by a drone-based camera. The BBC described a story of a climber missing in the Himalayas, who was noticed by a mountaineer piloting a drone [
13]. Moreover, the BBC reported on the rescue of a 75-year-old man lost in Norfolk (UK) who was spotted by a police analyst who inspected drone-acquired imagery [
14]. Recently, Polsat News presented a story of a 63-year-old Polish woman rescued by the police after the drone pilot noticed her in aerial imagery [
15]. One of the UAV manufacturers, DJI, presented a web map service where all uses of DJI drones for locating missing persons were placed in an interactive map [
16]. The DJI database and the individual cases reported on in the literature (and mass media publications) provide unequivocal proof that drones may save lives.
The majority (if not all) of the cases were associated with a human analyst viewing the screen or images. However, over the years, scientists have developed algorithms and methods for human detection in an unsupervised fashion, using artificial neural networks [
17,
18,
19,
20], statistical clustering [
21,
22,
23], pictorial structures [
24,
25], histograms of oriented gradients [
26,
27], Haar classifiers [
28,
29], body-part detectors [
30,
31], and scale-invariant feature transform [
32]. However, no successful operational use of such algorithms has been reported on, thus far, in the scientific literature. In this context, it is worth referring to the book by Robinson [
33], who managed to use microbiological software to successfully detect people in aerial images; however, as the author said, “even that was after the fact”.
Some human detection methods have been embedded into search and rescue software. For instance, ResQuad software processes a histogram of an image and detects color anomalies [
21]. A similar approach is employed in the commercially available software Loc8 [
34], which identifies anomalous pixel clusters, given a priori information on the color range to be detected [
35]. A different approach is utilized in the current version of the SARUAV software program [
36,
37], which uses deep learning techniques to detect persons in aerial imagery.
On 29 June 2021, one of the operational users of the SARUAV software, Bieszczady Mountain Rescue Service (Poland) (Grupa Bieszczadzka Górskiego Ochotniczego Pogotowia Ratunkowego—GOPR), used the software to search for a 65-year-old man in Beskid Niski (SE Poland). The SARUAV system was employed nearly 24 h after the man left his house in the village of Cergowa [
38]. Within a few hours from its launch in the field, the SARUAV software detected a lost person who was reached and saved by rescuers [
39,
40]. The objective of this paper is to report on the aforementioned mission, which, according to the authors’ best knowledge, is the first successful application of the automated human detection system in a real search and rescue mission that resulted in saving a person’s life.
2. Methods
This section describes terrestrial and aerial search methods used on 29 June 2021, by Bieszczady Mountain Rescue Service, while searching for a missing man in Beskid Niski. In Poland, Mountain Rescue Services are legally responsible for search and rescue activities in mountainous terrains, while the police is the leading unit responsible for searching for lost persons in the entire country. The mission on 29 June 2021, was conducted by Bieszczady Mountain Rescue Service, with the police, the volunteer fire brigades, and other volunteer search units.
2.1. Terrestrial Search Methods
Commonly, having interviewed the relatives of lost individuals, a lost person is assigned to one of the categories of missing people. This allows the rescuers to determine the area in which a lost person is most likely to be present. As a result, the search region can be spatially limited. The profile of a missing person, and the resulting limitation of terrain to be checked in the first place, is often produced based on the International Search and Rescue Incident Database (ISRID) [
41]. Such a limitation (in regard to the area) can be carried out using models, i.e., the ring model [
1], the mobility model [
4], or the bike wheel model [
3]. Most of them use the point last seen (PLS), the last known point (LKP), and the initial planning point (IPP). During the reported mission, the bike wheel model was utilized, which, apart from the hub (lower circle) and the rim (bigger circle), focuses on linear terrain features (e.g., roads, paths, streams), and so called “hot points”.
Roads are often searched using the method known as SPD (abbreviation from Polish phrases: “systematyczne przeszukanie dróg” or “szczegółowe przeszukanie dróg”). It is elaborated by Chrustek [
42] and used in Poland. The method forms a systematic way of using quad vehicles in searching for a lost person along roads. The quad patrol travels only within a pre-selected sector, treating crossroads as spatial references, which enable marking side roads. This helps the patrol to identify uninspected roads for further checks. In addition, the quad team should occasionally stop the vehicle and its engine to call a lost person. Linear features (not only roads, but also streams, balks, or purlieu) are also inspected by 2- or 3-person walking patrols. It is a tedious task since rescuers who visually search terrain can easily be distracted due to the sameness of such an activity. Unintentionally, rescuers may omit a missing person who, for instance, may be hidden behind obstacles.
Moreover, terrestrial searches include methods that employ rescue dogs. One novel approach in this field is known as mantrailing. Its idea differs from tracking, in which a dog directly follows the footsteps of a missing person. In mantrailing, however, a specially trained canine does not precisely travel along a person’s route, it detects drafts of scents spread by the wind or other environmental factors [
43].
2.2. Aerial Search Methods
Bieszczady Mountain Rescue Service owns a few UAVs. During their mission on 29 June 2021, in Beskid Niski, the DJI Matrice 300 platform was utilized. It was equipped with an XT-S thermal sensor and ZH20 digital RGB camera manufactured by DJI. The resolution of the RGB aerial images was 4056 × 3040, with a 4.5 mm f/2.8 lens.
To use the SARUAV software, RGB imagery must be acquired according to the recommendations formulated by the software manufacturer (
Figure 1). Namely, flights should be conducted in the daylight, over open terrain in the wilderness. It is not recommended to fly over densely built-up areas, because the number of objects resembling persons is high in villages or cities. In addition, nadir and near-nadir JPG photographs are required, taken during a photogrammetric mission.
The photogrammetric approach, employing nadir or near-nadir imagery, taken with sufficient side and front overlap, is used to: (1) observe every place from multiple camera positions, to increase the chances for capturing a person; (2) compute object coordinates with the highest possible accuracy, having a nadir-looking camera, without a need for producing orthomosaics. Nadir images acquired from different altitudes above terrain lead to different coverage, which influences geometry and the computation of the object’s location (
Figure 2). Moreover, the lower the flight altitude is, the lower the ground sampling distance (GSD) becomes (improving conditions for visual assessment), which facilitates human detection and expedites the verification of hits generated by the detection system. In addition, oblique photographs do not allow determining terrain coordinates in a straightforward way, since geometry, in respect to the center of projection, becomes more complicated, as presented in
Figure 2. It means that a camera should point vertically down, towards the terrain.
The JPG compression enables working on imagery, the size of which is significantly smaller than the corresponding raw images, leading to faster image processing and wider availability. Previous studies confirmed the successful human detection in high-resolution JPG photographs [
22,
44,
45,
46]. In addition, RGB cameras enable recording JPG images and offer valuable material in cases when thermovision fails (e.g., during hot days). They are cheaper than thermal sensors and may complement thermovision because consumer-grade RGB cameras have high resolution, and RGB imagery provides a natural picture for the analysis by human eyes.
Recommended parameters of the photogrammetric flight are the following: (1) side overlap of approximately 60–70%; (2) front overlap of approximately 80%; (3) altitude so that GSD does not exceed 2 cm/px (3 cm/px at worst); (4) platform horizontal velocity of approximately 2.5–3.0 m/s. The considerable side and front overlaps mean that every point is photographed from a number of different camera locations, increasing the probability of capturing a person who may be obscured by other land cover objects (
Figure 3). The small GSD values guarantee a sufficient level of detail and help the detector to differentiate between persons and similar objects. Moreover, small GSD values facilitate the human-assisted verification of hits provided by the system. The low platform horizontal velocity positively influences the sharpness of aerial imagery. In addition, the location of take-off site must be recorded for each flight. This makes the system compatible with various platforms, which record altitudes using different vertical reference systems. New models of digital cameras must be consulted with the SARUAV manufacturer prior to the software use.
2.3. Methods of Image Analysis
The automated search for persons in the wilderness was carried out using SARUAV software manufactured by SARUAV, Ltd. The computer program is a successor of the early prototype of the SARUAV system, elaborated at the University of Wrocław (Poland). The prototype used the nested k-means algorithm for detecting persons in close-range aerial imagery [
22], later developed by initial image pre-processing using the saturation boost [
45]. The early prototype was tested in the field, but was found to be computationally demanding and required improvement in its accuracy [
46]. Thus, the first commercial version of the software uses different, non-statistical detection approaches by SARUAV Ltd.
The commercial version of the SARUAV software [
36,
37] performs computations fast, i.e., 100 images are processed within approximately 2 min. In the current version, which is operationally used by a few rescue units in Europe, the calculations are carried out in the field on a computationally efficient laptop, without need for sending imagery to remote computers. Computations are performed on a graphics processing unit (GPU), based on NVIDIA CUDA/cuDNN. From a user perspective, SARUAV is a computer program that is dedicated specifically to rescuers. It leads a rescuer through the process of locating a missing person and offers: spatial modeling (for UAV-assisted and other missions), person detection (for UAV-assisted searches), and planning tools (for any search and rescue mission). Its graphical user interface was developed in cooperation with rescuers who participated in the eye-tracking research and the survey. Recent tests of the performance of the SARUAV software show that the detection rate, understood as a percentage of persons detected on at least one photo (out of several images of the same place, acquired in a photogrammetric mission), ranges between 86% and 100% (
Table 1).
Figure 4,
Figure 5 and
Figure 6 provide data that enable verifying the performance statistics presented in
Table 1.
For the purpose of image analysis, the SARUAV system uses the original convolutional neural network (CNN) algorithm. Effective detection is possible due to the extensive amount of JPG images that were used for the CNN learning. The imagery was taken during various seasons of the year (different colors of land cover, including white when snow cover was present) and different periods of the day (different lighting and air transparency). Imagery was acquired in lowland, upland, and in the mountains, in the areas of various land cover. Persons who were photographed wore clothes of dissimilar colors, including vivid ones and camouflage colors. Their sizes were different, i.e., from children in the preschool age, through middle-size adults to stocky men. Photographs used for CNN learning were taken by different cameras and platforms (rotary-wing and fixed-wing UAVs), operating from dissimilar altitudes. This provides a wealth of diverse information and is evidence of the potential of the algorithm.
Detecting objects that resemble persons in imagery requires splitting a photo into segments. It can be done using a number of approaches, including different overlaps [
22,
47] and different ways of declaring window size (in metric units, in pixels). Although, in the SARUAV prototype, the segmentation into 10 × 10 m windows with 2-m stripes in their bottom and right sides was employed [
22], in the herein used commercial version of the SARUAV software, the segmentation into 640 × 640 pixels was utilized. In the context of aerial imagery, the two approaches (meters vs. pixels) have some advantages and disadvantages when GSD varies significantly between individual photos taken during a photogrammetric flight. The former approach takes into account the actual GSD of a given image and, therefore, the window size in pixels is always adjusted to keep a constant metric size. It is a virtue when terrain anomalies are sought using purely statistical methods in which the size of a person matters [
22]; however, it leads to uneven grids when using neural networks. In CNNs, which are used in the operational version of the SARUAV software, the initial resolution of image fragments is set constant [
20].
The computer program is run on a laptop, the minimum (recommended) specification of which includes: graphics card NVIDIA GeForce GTX 1060 6 GB (NVIDIA GeForce RTX 2060 6 GB), processor Intel i5 or AMD Ryzen 5 (Intel i7 or AMD Ryzen 7), 8 GB RAM (16 GB RAM), at least 10 GB for the province-size terrain, Windows 10. The key computations are performed on a GPU.
The operational procedure assumes performing a UAV mission according to the aforementioned recommendations, downloading a set of nadir or near-nadir JPG images taken during a single photogrammetric flight to the laptop, and running the person detection tool. Depending on the number of images to be processed, after a few minutes, the output is available. The effect of the calculations is a map of terrain with yellow pins, superimposed in places where the system automatically detects objects resembling persons. The coordinates of these pins are juxtaposed in the list in the vicinity of the map, and below each location, filenames of images in which the algorithm detected a person are attached. Below the list of places with the associated photos, the screen is provided, where an analyst may visually verify hits suggested by the system. The analyst confirms these hits quickly in order to filter out false positives (
Figure 7). Such an analysis usually takes from a few to a dozen minutes. The final report for rescuers can be generated in the next step.
3. Results
Figure 8 presents the timeline along which search and rescue activities, undertaken by search units on 28 and 29 June 2021, while searching for the 65-year-old man in the vicinity of the village of Cergowa in Beskid Niski in SE Poland (
Figure 9), are denoted. The presented description of results is chronological and follows the timeline in question. All times provided in this paper are local times.
In the afternoon of 28 June 2021, a 65-year-old man, a citizen of the village of Cergowa, left his house. The man was seriously ill, endured a stroke the previous day, and suffered from Alzheimer’s disease. He also had problems with walking and, according to his family, was unable to stand up after falling down. At 23:30 on 28 June, 2021, the man’s family reported that they had not been in contact with the man since he left his house in the afternoon. The man was last seen by his neighbor in a meadow at the position (49.56342N, 21.71102E), which was later designated by rescuers as LKP. Moreover, the neighbor reported that the missing man was heading towards his house.
At 23:50 on 28 June 2021, the rescuers of Bieszczady Mountain Rescue Service left their emergency station in the town of Dukla, shortly after receiving notification about the man being lost (
Figure 9). They arrived at the village of Cergowa at midnight, where all search units (police, mountain rescue service, volunteer fire brigade) organized a meeting point. At the same time, the rescuers of Bieszczady Mountain Rescue Service called their central station in the town of Sanok (
Figure 9) to prepare the UAV equipment. At 00:10 on 29 June 2021, the IPP was established: (49.56428
N, 21.69753
E).
Moreover, at 00:10 on 29 June 2021, terrestrial searches were initiated. They included foot (2- or 3-person) and quad (the SPD method) patrols carried out by rescuers of Bieszczady Mountain Rescue Service and the volunteer fire brigade. In the first phase of searches rescuers checked the vicinity of a house and linear features (roads, streams, balks, and purlieu) between IPP and LKP. In addition, a rescuer with his rescue dog began to work (mantrailing between the house and potential routes between IPP and LKP). Since the terrestrial searches were not successful, the need for the aerial monitoring became evident.
At 02:00 on 29 June 2021, the UAV team from the central station of Bieszczady Mountain Rescue Service, based in Sanok, was asked to arrive. Around 03:00 on 29 June 2021, the UAV was used for the first time during the search (
Table 2). The drone equipped with thermal camera was utilized to check potential places along meadow roads, and thermal imagery was viewed by an analyst. The missing man was not found using thermovision, though visual monitoring lasted a few hours.
The terrestrial and aerial searches were temporarily stopped by the police at 06:00 on 29 June 2021. The reason for suspending the search and rescue activities was the need to reorganize resources and delineate new sectors for the subsequent searches.
The action was resumed by the police at 12:00 on 29 June 2021. The rescuers searched the newly delineated sectors. The UAV team performed five photogrammetric missions using the ZH20 RGB camera (
Table 2). The coverage of each mission is presented in
Figure 10. In total, 910 nadir and near-nadir JPG photographs were taken, covering the area of 95–100 ha. At 13:50 on 29 June 2021, the SARUAV software was launched for the first time during the search and rescue mission. The system processed images from four flights (RGB2–RGB5), and objects automatically detected by SARUAV were subsequently verified by human analysts in order to filter out false positive hits. The performance statistics of the system, including the number of all hits, are juxtaposed in
Table 3. Meanwhile, a duty officer from the police force in Dukla was informed about utilizing new resources, such as STORAT (the association of rescue dog guards) and Legion Gerarda (volunteer search and rescue group).
At 16:20 on 29 June 2021, the searches were suspended due to unfavorable weather conditions (heat wave, thunderstorms). In addition, after a prolonged search activity in the complex terrain (high grass, valleys, dense forest), there was the need for regeneration of personnel. Since the visual analysis of hits provided by the SARUAV system is not very demanding due to properly designed graphical user interface, the analysts who utilized the SARUAV software kept analyzing the imagery acquired by the UAV team and processed by the SARUAV system.
At 18:21 on 29 June 2021, a rescuer of Bieszczady Mountain Rescue Service reported that the SARUAV software detected the missing man. The true positive hit was confirmed in the software by the analyst (
Figure 11). Moreover, the rescuer provided the coordinates of the place where the lost person was identified by the system: (49.5635
N, 21.7010
E). The information was immediately passed on to the police and the rescuers from the volunteer fire brigade. Soon (at 18:25), the rescuers headed towards the location estimated by the SARUAV system. At 18:30 on 29 June 2021, they arrived at the indicated place, found the missing man in the field, and initiated evacuation. Subsequently (at 18:50), the man was transported to an ambulance of the Medical Emergency Service. The entire search and rescue action was completed at 20:00 on 29 June 2021.
Figure 11 presents the graphical user interface of the SARUAV software, with the successful detection of the lost person. The missing man was detected in RGB flight 3, during which 121 nadir and near-nadir JPG images were acquired. The processing time (automatic detection in SARUAV) was 1 min 50 s, and the verification of hits by the analyst took 2 min 15 s (
Table 3). The missing man was successfully detected in three images out of three photographs in which he was captured.
In the entire search and rescue mission, 17 rescuers of the Bieszczady Mountain Rescue Service were involved. They used specialized cars, quads, a rescue dog, and UAVs. Moreover, rescuers from the police (commander and 2–3 patrols), volunteer fire brigade (3–4 units), and other volunteer brigades (2 units) very significantly contributed to the searches.
If the SARUAV software had not been applied for searching the area of 95–100 ha, the same terrain would have been checked by 10 groups of three rescuers within 2 h or more (the practice and theory behind the rapid triplet search [
5] suggest that the most convenient area of terrain to be searched was of 10–12 ha, which, on average, can be appropriately checked by a single three-person group within 2–2.5 h). The land cover was highly diverse, with meadows, dense bushes, and forests. In addition, there were built-up areas, as well as arable land. The relief was also complicated, with flat areas and mountainous topography cut by deep valleys. Due to the difficulty of terrain, it was possible to overlook the missing man when walking 2 m apart. In addition, the skills of the rescue dog were limited by high grass and high temperature. Hence, the environmental conditions made terrestrial searches difficult. Therefore, aerial searches using UAVs and the SARUAV software were the appropriate methods to facilitate and quicken the mission.
4. Discussion
It is rather impossible to unequivocally judge how this specific search incident would proceed and with what effect it would end without the use of UAV, assisted by SARUAV software. However, it is evident that these technologies provided significant support and possibly quickened the mission. Fortunately, the missing man was located on time, and found alive. He required only minor medical assistance and evacuation.
Figure 8 shows that the time elapsed from leaving the house to the successful detection was greater than 24 h. According to Koester and Stooksbury [
48], who studied the behavior of missing people suffering from Alzheimer’s disease in Virginia (USA), survivability after 24 h was of 54%. It means that 46% of patients who spent more than 24 h in the wilderness were found dead on arrival of search units. It can be inferred that life of the 65-year-old man lost in the vicinity of the village of Cergowa in Poland (also the Alzheimer’s disease patient), who was successfully rescued with the use of unmanned technologies, was at serious risk after 24 h of searching. Moreover, the missing man reported post-stroke effects and problems with mobility, which additionally complicated his situation after being exposed to the wilderness. Therefore, every minute of the search and rescue mission was essential for its overall success.
We believe that quickening the detection by applying the UAV together with the SARUAV software was the key factor that increased the probability of survival. This is particularly evident because, at a certain stage of searching, other methods were found not successful, required new resources and the regeneration of personnel. Little work burden and low risk for rescuers when using UAV with SARUAV allowed the terrestrial work to efficiently continue, without the need for allocating new resources, bearing new costs, or waiting for personnel recuperation.
The fundamental role of time in search and rescue activities was highlighted by Adams et al. [
49], who emphasized it in their work: “Search is a time-critical event: when search and rescue missions may become futile”. Although they did not only focus on Alzheimer’s disease patients, but conducted a much wider study, their Kaplan–Meier estimator predicted the survival status as a function of search time, and determined the cut-off point of 51 h (in accordance with Koester and Stooksbury [
48], mortality among Alzheimer’s disease patients grows to 62% if a person remains un-found for longer than 48 h).
Factors that contributed to the prolonged health stability of the missing man were the weather and clothing. The weather conditions in Beskid Niski on 28 and 29 June 2021, were favorable. The following measures were recorded in the central station of Bieszczady Mountain Rescue Service in Sanok (approximately 37 km from Cergowa): temperature 29
C, wind speed 5 km/h, good visibility, cloudiness 3/8, no precipitation (28 June 2021, 14:30 local time); temperature 17
C, no wind, good visibility, cloudiness 2/8, no precipitation (29 June 2021, 7:30 local time); temperature 27
C, no wind, good visibility, cloudiness 4/8, no precipitation (29 June 2021, 14:30 local time). The lost person wore trousers and a loose shirt (probably one layer). According to Tikuisis [
50], a standard healthy man who wears one 1-millimeter loose clothing, exposed to wind of 5 km/h and air temperature of 0
C, can survive approximately 36 h. In our study, the missing man had standard posture, but was seriously ill and was exposed to much higher summer temperatures. Thus, the described case cannot be directly compared with model-predicted survival times elaborated by Tikuisis [
50].
The uniqueness of the case described in the paper lies in the fact that the lost person was detected automatically in aerial images. To date, other operational searches with the use of UAVs were successful due to human observers viewing the screen [
12,
13,
14,
15,
16]. Our work presents a step forward in performing, facilitating, and quickening searches for missing people using drones. It confirms that near-real-time image analytics with SARUAV software not only decreases search time (leading to the increase in the probability of survival), but also decreases work burden imposed on the search and rescue personnel (automation of the process of human detection and the graphical user interface designed for rescuers).
Although the paper describes a single case study, it is possible to formulate lessons learned from the search and rescue activities conduced with the use of a UAV photography and the SARUAV software. Firstly, the analysis should be carried out as quickly as possible after acquiring imagery. In the reported case, the analysis was suspended (
Table 2), which resulted in an unnecessary delay (the time from the first launch of the system to the successful detection was 4 h 31 min, however, the analysis of a set of images collected in one flight, in which the person was located, took only 5 min and 48 s)—it is apparent from
Figure 8 and
Table 3. Such an approach, in which the analysis begins just after a UAV mission, allows rescuers to promptly verify hits generated by the system in the field. Secondly, the study confirms that the analysis of aerial imagery should be carried out with the use of dedicated software, since visual inspection of images is often ineffective. This finding is in agreement with the study of gaze behavior of spotters in the air-to-ground detection exercise [
51]. Thirdly, UAV flight regions should be delineated by a planist and imported from the planist’s computer. This will prevent flying beyond sectors of interest, stop overlapping flight regions and prevent omitting areas between sectors. In the presented case, RGB flights 1 and 2 overlapped and, in addition, there was a spatially considerable gap between RGB flight 3, RGB flights 1/2 and RGB flights 4/5 (
Figure 10). Such situations would have been easily eliminated if the planist-produced sectors were imported. Fourthly, a UAV pilot and a SARUAV analyst should operate from a stand that is isolated from third parties. Such a recommendation fits in the process of assigning roles in search and rescue activities with the use of drones [
52].