Enhancing Search and Rescue Missions with UAV Thermal Video Tracking
Abstract
1. Introduction
- We introduce an object detection and tracking pipeline for analyzing UAV thermal videos and identifying people lost in the wilderness. The object detector is trained to recognize targets in individual thermal frames characterized by dense vegetation typical of forest environments. The tracking module processes sequences of frames, suppresses sparse false alarms, and temporarily tracks detected people masked by vegetation.
- We enhance object tracking performance by introducing a Camera Motion Compensation module that utilizes UAV telemetry data. This addition achieves better camera alignment compared to CV-based motion estimation algorithms, which struggle with low-texture thermal images. The Camera Motion Compensation introduces a minimal overhead with negligible impact on computational efficiency.
- The proposed architecture is evaluated on a dataset of UAV thermal videos specifically constructed to represent realistic WSAR scenarios. The dataset includes 213 sequences, totaling more than 39 minutes of video footage, each lasting 5 to 30 seconds. The performance is assessed at various framerates to identify a trade-off between detection accuracy and computational requirements. The dataset variant obtained by extracting thermal video frames at 15 Frames Per Second (FPS) contains 35,714 images. The resulting configuration supports real-time execution and can be used to guide search operations towards locations with a high probability of human presence.
2. Materials and Methods
2.1. Target Detection and Tracking Pipeline
2.1.1. Target Detection
2.1.2. Target Tracking
2.2. Pose-Based Camera Motion Compensation (PB-CMC)
2.2.1. Backprojection of Principal Point
2.2.2. Projection onto the Current Frame
2.2.3. Motion Vector Estimation
2.3. Datasets
2.3.1. Drone Thermal Video (DTV) Dataset
2.3.2. AOS Dataset
2.4. Evaluation Procedure
2.4.1. Training
2.4.2. Video Object Tracking Evaluation
3. Results
3.1. Evaluation of Object Detection and Tracking Results
Framerate Analysis and Contribution of BT Modules
3.2. Contribution of the Pose-Based Camera Motion Compensation
3.2.1. Assessment of Robustness to Pose Error
4. Discussion
5. Conclusions
Author Contributions
Funding
Data Availability Statement
Acknowledgments
Conflicts of Interest
Abbreviations
AOS | Airborne Optical Sectioning |
BT | BoostTrack++ |
CB-CMC | Content-Based Camera Motion Compensation |
CNN | Convolutional Neural Network |
CMC | Camera Motion Compensation |
CV | Computer Vision |
DEM | Digital Elevation Model |
DL | Deep Learning |
DLO | Detecting Likely Objects |
DTV | Drone Thermal Video |
DUO | Detecting Unlikely Objects |
ECC | Enhanced Correlation Coefficient |
FPS | Frames Per Second |
IoU | Intersection over Union |
MOT | Multi-Object Tracking |
PB-CMC | Pose-Based Camera Motion Compensation |
RTK | Real-Time Kinematic |
SfM | Structure-from-Motion |
UAV | Unmanned Aerial Vehicle |
WSAR | Wilderness Search and Rescue |
References
- Weldon, W.T.; Hupy, J. Investigating Methods for Integrating Unmanned Aerial Systems in Search and Rescue Operations. Drones 2020, 4, 38. [Google Scholar] [CrossRef]
- Vincent-Lambert, C.; Pretorius, A.; Tonder, B.V. Use of Unmanned Aerial Vehicles in Wilderness Search and Rescue Operations: A Scoping Review. Wilderness Environ. Med. 2023, 34, 580–588. [Google Scholar] [CrossRef] [PubMed]
- Karaca, Y.; Cicek, M.; Tatli, O.; Sahin, A.; Pasli, S.; Beser, M.F.; Turedi, S. The potential use of unmanned aircraft systems (drones) in mountain search and rescue operations. Am. J. Emerg. Med. 2018, 36, 583–588. [Google Scholar] [CrossRef]
- van Veelen, M.J.; Roveri, G.; Voegele, A.; Cappello, T.D.; Masè, M.; Falla, M.; Regli, I.B.; Mejia-Aguilar, A.; Mayrgündter, S.; Strapazzon, G. Drones reduce the treatment-free interval in search and rescue operations with telemedical support—A randomized controlled trial. Am. J. Emerg. Med. 2023, 66, 40–44. [Google Scholar] [CrossRef]
- Sambolek, S.; Ivasic-Kos, M. Automatic Person Detection in Search and Rescue Operations Using Deep CNN Detectors. IEEE Access 2021, 9, 37905–37922. [Google Scholar] [CrossRef]
- Niedzielski, T.; Jurecka, M.; Miziński, B.; Pawul, W.; Motyl, T. First Successful Rescue of a Lost Person Using the Human Detection System: A Case Study from Beskid Niski (SE Poland). Remote Sens. 2021, 13, 4903. [Google Scholar] [CrossRef]
- Schedl, D.C.; Kurmi, I.; Bimber, O. Search and rescue with airborne optical sectioning. Nat. Mach. Intell. 2020, 2, 783–790. [Google Scholar] [CrossRef]
- Doherty, P.; Rudol, P. A UAV Search and Rescue Scenario with Human Body Detection and Geolocalization. In Proceedings of the AI 2007: Advances in Artificial Intelligence, Gold Coast, Australia, 2–6 December 2007; pp. 1–13. [Google Scholar]
- Park, K.Y.; Hwang, S.Y. An improved Haar-like feature for efficient object detection. Pattern Recognit. Lett. 2014, 42, 148–153. [Google Scholar] [CrossRef]
- Hoai, D.K.; Van Phuong, N. Anomaly color detection on UAV images for search and rescue works. In Proceedings of the 2017 9th International Conference on Knowledge and Systems Engineering (KSE), Hue, Vietnam, 19–21 October 2017; pp. 287–291. [Google Scholar] [CrossRef]
- Proft, J.; Suarez, J.; Murphy, R. Spectral anomaly detection with machine learning for wilderness search and rescue. In Proceedings of the 2015 IEEE MIT Undergraduate Research Technology Conference (URTC), Cambridge, MA, USA, 7–8 November 2015; pp. 1–3. [Google Scholar] [CrossRef]
- Reed, I.; Yu, X. Adaptive multiple-band CFAR detection of an optical pattern with unknown spectral distribution. IEEE Trans. Acoust. Speech Signal Process. 1990, 38, 1760–1770. [Google Scholar] [CrossRef]
- Rasmussen, N.D.; Morse, B.S.; Goodrich, M.A.; Eggett, D. Fused visible and infrared video for use in Wilderness Search and Rescue. In Proceedings of the 2009 Workshop on Applications of Computer Vision (WACV), Snowbird, UT, USA, 7–8 December 2009; pp. 1–8. [Google Scholar] [CrossRef]
- Vempati, A.S.; Agamennoni, G.; Stastny, T.; Siegwart, R. Victim Detection from a Fixed-Wing UAV: Experimental Results. In Proceedings of the Advances in Visual Computing, Las Vegas, NV, USA, 14–16 December 2015; Bebis, G., Boyle, R., Parvin, B., Koracin, D., Pavlidis, I., Feris, R., McGraw, T., Elendt, M., Kopper, R., Ragan, E., et al., Eds.; Springer: Cham, Switzerland, 2015; pp. 432–443. [Google Scholar]
- Bejiga, M.B.; Zeggada, A.; Nouffidj, A.; Melgani, F. A Convolutional Neural Network Approach for Assisting Avalanche Search and Rescue Operations with UAV Imagery. Remote Sens. 2017, 9, 100. [Google Scholar] [CrossRef]
- Kurmi, I.; Schedl, D.C.; Bimber, O. Combined person classification with airborne optical sectioning. Sci. Rep. 2022, 12, 3804. [Google Scholar] [CrossRef]
- Schedl, D.C.; Kurmi, I.; Bimber, O. An autonomous drone for search and rescue in forests using airborne optical sectioning. Sci. Robot. 2021, 6, eabg1188. [Google Scholar] [CrossRef] [PubMed]
- Yeom, S. Thermal Image Tracking for Search and Rescue Missions with a Drone. Drones 2024, 8, 53. [Google Scholar] [CrossRef]
- Bewley, A.; Ge, Z.; Ott, L.; Ramos, F.; Upcroft, B. Simple online and realtime tracking. In Proceedings of the 2016 IEEE International Conference on Image Processing (ICIP), Phoenix, AZ, USA, 25–28 September 2016; pp. 3464–3468. [Google Scholar] [CrossRef]
- Yi, K.; Luo, K.; Luo, X.; Huang, J.; Wu, H.; Hu, R.; Hao, W. UCMCTrack: Multi-Object Tracking with Uniform Camera Motion Compensation. Proc. AAAI Conf. Artif. Intell. 2024, 38, 6702–6710. [Google Scholar] [CrossRef]
- Wojke, N.; Bewley, A.; Paulus, D. Simple Online and Realtime Tracking with a Deep Association Metric. arXiv 2017, arXiv:1703.07402. [Google Scholar] [CrossRef]
- Wang, Y.H.; Hsieh, J.W.; Chen, P.Y.; Chang, M.C.; So, H.H.; Li, X. SMILEtrack: SiMIlarity LEarning for Occlusion-Aware Multiple Object Tracking. Proc. AAAI Conf. Artif. Intell. 2024, 38, 5740–5748. [Google Scholar] [CrossRef]
- Stanojević, V.; Todorović, B. BoostTrack++: Using tracklet information to detect more objects in multiple object tracking. arXiv 2024, arXiv:2408.13003. [Google Scholar]
- Hassan, S.; Mujtaba, G.; Rajput, A.; Fatima, N. Multi-object tracking: A systematic literature review. Multimed. Tools Appl. 2024, 83, 43439–43492. [Google Scholar] [CrossRef]
- Li, Z.; Dong, Y.; Shen, L.; Liu, Y.; Pei, Y.; Yang, H.; Zheng, L.; Ma, J. Development and challenges of object detection: A survey. Neurocomputing 2024, 598, 128102. [Google Scholar] [CrossRef]
- Yaseen, M. What is YOLOv8: An In-Depth Exploration of the Internal Features of the Next-Generation Object Detector. arXiv 2024, arXiv:2408.15857. [Google Scholar]
- Tian, Y.; Ye, Q.; Doermann, D. YOLOv12: Attention-Centric Real-Time Object Detectors. arXiv 2025, arXiv:2502.12524. [Google Scholar]
- Ge, Z.; Liu, S.; Wang, F.; Li, Z.; Sun, J. YOLOX: Exceeding YOLO Series in 2021. arXiv 2021, arXiv:2107.08430. [Google Scholar] [CrossRef]
- Redmon, J.; Farhadi, A. YOLOv3: An Incremental Improvement. arXiv 2018, arXiv:1804.02767. [Google Scholar] [CrossRef]
- Milan, A.; Leal-Taixé, L.; Reid, I.; Roth, S.; Schindler, K. MOT16: A Benchmark for Multi-Object Tracking. arXiv 2016, arXiv:1603.00831. [Google Scholar] [CrossRef]
- Dendorfer, P.; Rezatofighi, H.; Milan, A.; Shi, J.; Cremers, D.; Reid, I.; Roth, S.; Schindler, K.; Leal-Taixé, L. MOT20: A benchmark for multi object tracking in crowded scenes. arXiv 2020, arXiv:2003.09003. [Google Scholar] [CrossRef]
- Evangelidis, G.D.; Psarakis, E.Z. Parametric Image Alignment Using Enhanced Correlation Coefficient Maximization. IEEE Trans. Pattern Anal. Mach. Intell. 2008, 30, 1858–1865. [Google Scholar] [CrossRef] [PubMed]
- Schonberger, J.L.; Frahm, J.M. Structure-From-Motion Revisited. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Las Vegas, NV, USA, 27–30 June 2016. [Google Scholar]
- Zeybek, M. Accuracy assessment of direct georeferencing UAV images with onboard global navigation satellite system and comparison of CORS/RTK surveying methods. Meas. Sci. Technol. 2021, 32, 065402. [Google Scholar] [CrossRef]
- Schedl, D.C.; Kurmi, I.; Bimber, O. Data: Search and Rescue with Airborne Optical Sectioning (Dataset), Version v2. Zenodo. 2020. [CrossRef]
- Kurmi, I.; Schedl, D.C.; Bimber, O. Data: Autonomous Drones for Search and Rescue in Forests (Dataset), Version v1. Zenodo. 2020. [CrossRef]
- Tavasci, L.; Nex, F.; Gandolfi, S. Reliability of Real-Time Kinematic (RTK) Positioning for Low-Cost Drones’ Navigation across Global Navigation Satellite System (GNSS) Critical Environments. Sensors 2024, 24, 6096. [Google Scholar] [CrossRef] [PubMed]
- Lin, T.Y.; Maire, M.; Belongie, S.; Bourdev, L.; Girshick, R.; Hays, J.; Perona, P.; Ramanan, D.; Zitnick, C.L.; Dollár, P. Microsoft COCO: Common Objects in Context. arXiv 2015, arXiv:1405.0312. [Google Scholar] [CrossRef]
- Kunisada, Y.; Premachandra, C. High Precision Location Estimation in Mountainous Areas Using GPS. Sensors 2022, 22, 1149. [Google Scholar] [CrossRef] [PubMed]
Framerate | Frames | Annotations | Empty Frames |
---|---|---|---|
1 FPS | 2481 | 1647 | 834 |
3 FPS | 7225 | 4880 | 2345 |
5 FPS | 11,981 | 8103 | 3878 |
10 FPS | 23,856 | 16,175 | 7681 |
15 FPS | 35,714 | 24,228 | 11,486 |
Model | Precision | Recall | F1 | FPS |
---|---|---|---|---|
YOLOv8 + BT | 70.9% | 72.2% | 71.1% | 31.5 |
YOLOv12 + BT | 74.6% | 63.4% | 67.9% | 28.3 |
YOLOX + BT | 90.3% | 73.4% | 80.2% | 29.3 |
Model | BT | DLO | DUO | Precision | Recall | F1 | FPS |
---|---|---|---|---|---|---|---|
YOLOX | ✗ | ✗ | ✗ | 87.4% | 74.3% | 79.7% | 107.6 |
✓ | ✗ | ✗ | 90.3% | 73.4% | 80.2% | 29.3 | |
✓ | ✓ | ✗ | 75.2% | 79.7% | 77.2% | 28.7 | |
✓ | ✓ | ✓ | 67.4% | 80.1% | 72.9% | 28.3 |
CMC Method | Precision | Recall | F1 | FPS |
---|---|---|---|---|
PB-CMC (ours) | 89.0% | 82.2% | 85.5% | 35.2 |
CB-CMC (original) | 89.1% | 75.4% | 81.7% | 14.5 |
No CMC | 72.5% | 76.1% | 82.3% | 35.6 |
Noise | Precision | Recall | F1 | Avg Error | Max Error |
---|---|---|---|---|---|
10 cm | 89.0% | 82.2% | 85.5% | 12.8 cm | 36.4 cm |
20 cm | 89.0% | 81.4% | 85.0% | 25.6 cm | 72.9 cm |
30 cm | 89.0% | 80.3% | 84.5% | 38.4 cm | 109.3 cm |
50 cm | 89.4% | 74.2% | 81.1% | 64.0 cm | 182.2 cm |
Disclaimer/Publisher’s Note: The statements, opinions and data contained in all publications are solely those of the individual author(s) and contributor(s) and not of MDPI and/or the editor(s). MDPI and/or the editor(s) disclaim responsibility for any injury to people or property resulting from any ideas, methods, instructions or products referred to in the content. |
© 2025 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https://creativecommons.org/licenses/by/4.0/).
Share and Cite
Fraternali, P.; Morandini, L.; Motta, R. Enhancing Search and Rescue Missions with UAV Thermal Video Tracking. Remote Sens. 2025, 17, 3032. https://doi.org/10.3390/rs17173032
Fraternali P, Morandini L, Motta R. Enhancing Search and Rescue Missions with UAV Thermal Video Tracking. Remote Sensing. 2025; 17(17):3032. https://doi.org/10.3390/rs17173032
Chicago/Turabian StyleFraternali, Piero, Luca Morandini, and Riccardo Motta. 2025. "Enhancing Search and Rescue Missions with UAV Thermal Video Tracking" Remote Sensing 17, no. 17: 3032. https://doi.org/10.3390/rs17173032
APA StyleFraternali, P., Morandini, L., & Motta, R. (2025). Enhancing Search and Rescue Missions with UAV Thermal Video Tracking. Remote Sensing, 17(17), 3032. https://doi.org/10.3390/rs17173032