A Thermal Imaging Flame-Detection Model for Firefighting Robot Based on YOLOv4-F Model
Abstract
:1. Introduction
2. Related Work
3. Methodology
3.1. YOLOv4-Tiny Flame Detection Model
3.2. YOLOv4-F Flame Detection Model
4. Development of Experimental Platform
4.1. Detection Dataset Platform
4.2. Experimental Test Platform
5. Results and Discussion
5.1. Model Training Result Analysis
5.2. Thermal Imaging Flame Image Detection
6. Conclusions
Author Contributions
Funding
Institutional Review Board Statement
Informed Consent Statement
Data Availability Statement
Conflicts of Interest
References
- Yu, N.; Chen, Y. Video flame detection method based on TwoStream convolutional neural network. In Proceedings of the 2019 IEEE 8th Joint International Information Technology and Artificial Intelligence Conference (ITAIC), Chongqing, China, 24–26 May 2019; pp. 482–486. [Google Scholar]
- Gao, S.; Zhang, Z.; Zhao, Z.; Jamali, M.M. Vision and infra-red sensor based fire fighting robot. In Proceedings of the 2018 IEEE 61st International Midwest Symposium on Circuits and Systems (MWSCAS), Windsor, ON, Canada, 5–8 August 2018; pp. 873–876. [Google Scholar]
- Ryu, J.; Kwak, D. A Study on a Complex Flame and Smoke Detection Method Using Computer Vision Detection and Convolutional Neural Network. Fire 2022, 5, 108. [Google Scholar] [CrossRef]
- Guede-Fernández, F.; Martins, L.; de Almeida, R.V.; Gamboa, H.; Vieira, P. A Deep Learning Based Object Identification System for Forest Fire Detection. Fire 2021, 4, 75. [Google Scholar] [CrossRef]
- Barmpoutis, P.; Dimitropoulos, K.; Kaza, K.; Grammalidis, N. Fire detection from images using faster R-CNN and multidimensional texture analysis. In Proceedings of the ICASSP 2019–2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Brighton, UK, 12–17 May 2019; pp. 8301–8305. [Google Scholar]
- Sharma, J.; Granmo, O.-C.; Goodwin, M.; Fidje, J.T. Deep convolutional neural networks for fire detection in images. In International Conference on Engineering Applications of Neural Networks; Springer: Berlin/Heidelberg, Germany, 2017; pp. 183–193. [Google Scholar] [CrossRef]
- Shamsoshoara, A.; Afghah, F.; Razi, A.; Zheng, L.; Fulé, P.Z.; Blasch, E. Aerial imagery pile burn detection using deep learning: The FLAME dataset. Comput. Netw. 2021, 193, 108001. [Google Scholar] [CrossRef]
- Umar, M.M.; Silva, L.C.D.; Bakar, M.S.A.; Petra, M.I. State of the art of smoke and fire detection using image processing. Int. J. Signal Imaging Syst. Eng. 2017, 10, 22–30. [Google Scholar] [CrossRef]
- Dunnings, A.J.; Breckon, T.P. Experimentally defined convolutional neural network architecture variants for non-temporal real-time fire detection. In Proceedings of the 2018 25th IEEE international conference on image processing (ICIP), Athens, Greece, 7–10 October 2018; pp. 1558–1562. [Google Scholar]
- Suresh, J. Fire-fighting robot. In Proceedings of the 2017 International Conference on Computational Intelligence in Data Science (ICCIDS), Chennai, India, 2–3 June 2017; pp. 1–4. [Google Scholar]
- Ramasubramanian, S.; Muthukumaraswamy, S.A.; Sasikala, A. Fire detection using artificial intelligence for fire-fighting robots. In Proceedings of the 2020 4th International Conference on Intelligent Computing and Control Systems (ICICCS), Madurai, India, 13–15 May 2020; pp. 180–185. [Google Scholar]
- Li, S.; Feng, C.; Niu, Y.; Shi, L.; Wu, Z.; Song, H. A fire reconnaissance robot based on SLAM position, thermal imaging technologies, and AR display. Sensors 2019, 19, 5036. [Google Scholar] [CrossRef] [Green Version]
- Jiang, Y.; Pang, D.; Li, C. A deep learning approach for fast detection and classification of concrete damage. Autom. Constr. 2021, 128, 103785. [Google Scholar] [CrossRef]
- Matthes, J.; Waibel, P.; Vogelbacher, M.; Gehrmann, H.J.; Keller, H.B. A new camera-based method for measuring the flame stability of non-oscillating and oscillating combustions. Exp. Therm. Fluid Sci. 2019, 105, 27–34. [Google Scholar] [CrossRef]
- Nadarajan Assari Syamala, S.R.; Santhosh, A.; Ramesh, R. Analysis of Forest Fire in Australia using Visible Infrared Imaging Radiometer Suite. In Proceedings of the 2021 11th International Conference on Cloud Computing, Data Science & Engineering (Confluence), Kunming, China, 12 March 2021; pp. 482–486. [Google Scholar]
- Qin, C.; Zhang, M.; He, W.; Guan, C.; Sun, W.; Zhou, H. A New Real-Time Fire Detection Method Based On Infrared Image. In Proceedings of the 2019 IEEE 7th International Conference on Computer Science and Network Technology (ICCSNT)2019, Dalian, China, 19–20 October 2019; pp. 476–479. [Google Scholar] [CrossRef]
- Gong, F.; Li, C.; Gong, W.; Li, X.; Yuan, X.; Ma, Y.; Song, T. A Real-Time Fire Detection Method from Video with Multifeature Fusion. Comput. Intell. Neurosci. 2019, 2019, 1939171. [Google Scholar] [CrossRef]
- Jeong, S.Y.; Kim, W.H. Thermal Imaging Fire Detection Algorithm with Minimal False Detection. KSII Trans. Internet Inf. Syst. 2020. [Google Scholar] [CrossRef]
- Umoh, U.; Udo, E.; Emmanuel, N. Support Vector Machine-Based Fire Outbreak Detection System. Int. J. Soft Comput. Artif. Intell. Appl. 2019, 8, 1–18. [Google Scholar] [CrossRef]
- Wang, Q.; Yang, C.; Duan, S.; Wei, S. Research on Fire Prediction Algorithm Based on Thermal Infrared Image. In Proceedings of the 2021 7th International Conference on Computing and Artificial Intelligence, Tianjin, China, 23–26 April 2021; Association for Computing Machinery: Tianjin, China, 2021; pp. 216–221. [Google Scholar]
- Wang, K.; Zhang, Y.; Wang, J.; Zhang, Q.; Chen, B.; Liu, D. Fire detection in infrared video surveillance based on convolutional neural network and SVM. In Proceedings of the 2018 IEEE 3rd International Conference on Signal and Image Processing (ICSIP), Shenzhen, China, 13–15 July 2018; pp. 162–167. [Google Scholar]
- Agrawal, G.; Mishra, R.; Ransingh, A.; Chakravarty, S. Flame Temperature Prediction Using Machine Learning Model. In Proceedings of the 2020 IEEE India Council International Subsections Conference (INDISCON), Visakhapatnam, India, 3–4 October 2020; pp. 157–162. [Google Scholar]
- Xiao, Y.; Tian, Z.; Yu, J.; Zhang, Y.; Liu, S.; Du, S.; Lan, X. A review of object detection based on deep learning. Multimed. Tools Appl. 2020, 79, 23729–23791. [Google Scholar] [CrossRef]
- Gong, M.; Wang, D.; Zhao, X.; Guo, H.; Luo, D.; Song, M. A review of non-maximum suppression algorithms for deep learning target detection. In Proceedings of the Seventh Symposium on Novel Photoelectronic Detection Technology and Applications, Kunming, China, 12 March 2021; pp. 821–828. [Google Scholar]
- Jiang, X.; Wang, C.; Fu, Q. Development and application of deep convolutional neural network in target detection. In AIP Conference Proceedings; AIP Publishing LLC: Xi’an, China, 2018; p. 040036. [Google Scholar]
- Ding, Y.; Zhao, X.; Zhang, Z.; Cai, W.; Yang, N. Multiscale Graph Sample and Aggregate Network With Context-Aware Learning for Hyperspectral Image Classification. IEEE J. Sel. Top. Appl. Earth Obs. Remote Sens. 2021, 14, 4561–4572. [Google Scholar] [CrossRef]
- Ding, Y.; Zhao, X.; Zhang, Z.; Cai, W.; Yang, N.; Zhan, Y. Semi-Supervised Locality Preserving Dense Graph Neural Network With ARMA Filters and Context-Aware Learning for Hyperspectral Image Classification. IEEE Trans. Geosci. Remote Sens. 2022, 60, 1–12. [Google Scholar] [CrossRef]
- Ding, Y.; Zhang, Z.; Zhao, X.; Cai, Y.; Li, S.; Deng, B.; Cai, W. Self-Supervised Locality Preserving Low-Pass Graph Convolutional Embedding for Large-Scale Hyperspectral Image Clustering. IEEE Trans. Geosci. Remote Sens. 2022, 60, 1–16. [Google Scholar] [CrossRef]
- Parico, A.I.B.; Ahamed, T. Real time pear fruit detection and counting using YOLOv4 models and deep SORT. Sensors 2021, 21, 4803. [Google Scholar] [CrossRef]
- Yuan, N.; Kang, B.H.; Xu, S.; Yang, W.; Ji, R. Research on image target detection and recognition based on deep learning. In Proceedings of the 2018 International Conference on Information Systems and Computer Aided Education (ICISCAE), Changchun, China, 6–8 July 2018; pp. 158–163. [Google Scholar]
- Girshick, R.; Donahue, J.; Darrell, T.; Malik, J. Rich feature hierarchies for accurate object detection and semantic segmentation. In Proceedings of the IEEE conference on computer vision and pattern recognition, Columbus, OH, USA, 23–28 June 2014; pp. 580–587. [Google Scholar]
- Girshick, R. Fast r-cnn. In Proceedings of the IEEE international conference on computer vision, Santiago, Chile, 7–13 December 2015; pp. 1440–1448. [Google Scholar]
- Zhang, B.; Zhang, Y.; Pan, Q. Irregular Target Object Detection Based on Faster R-CNN. In Proceedings of the IOP Conference Series: Earth and Environmental Science, Xi’an, China, 15–16 December 2018; p. 042111. [Google Scholar]
- Ren, S.; He, K.; Girshick, R.; Sun, J. Faster r-cnn: Towards real-time object detection with region proposal networks. arXiv 2015, arXiv:1506.01497v2. [Google Scholar] [CrossRef] [Green Version]
- Dai, J.; Li, Y.; He, K.; Sun, J. R-fcn: Object detection via region-based fully convolutional networks. Adv. Neural Inf. Process. Syst. 2016, 29. [Google Scholar] [CrossRef]
- He, K.; Gkioxari, G.; Dollár, P.; Girshick, R. Mask r-cnn. In Proceedings of the IEEE international conference on computer vision, Venice, Italy, 22–29 October 2017; pp. 2961–2969. [Google Scholar]
- Pang, J.; Chen, K.; Shi, J.; Feng, H.; Ouyang, W.; Lin, D. Libra r-cnn: Towards balanced learning for object detection. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, Long Beach, CA, USA, 15–20 June 2019; pp. 821–830. [Google Scholar]
- Fu, X.; Li, K.; Liu, J.; Li, K.; Zeng, Z.; Chen, C. A two-stage attention aware method for train bearing shed oil inspection based on convolutional neural networks. Neurocomputing 2020, 380, 212–224. [Google Scholar] [CrossRef]
- Guo, J.; Hou, Z.; Xie, X.; Yao, S.; Wang, Q.; Jin, X. Faster R-CNN Based Indoor Flame Detection for Firefighting Robot. In Proceedings of the 2019 IEEE Symposium Series on Computational Intelligence (SSCI), Xiamen, China, 6–9 December 2019; pp. 1390–1395. [Google Scholar]
- Redmon, J.; Divvala, S.; Girshick, R.; Farhadi, A. You only look once: Unified, real-time object detection. In Proceedings of the IEEE conference on computer vision and pattern recognition, Las Vegas, NV, USA, 27–30 June 2016; pp. 779–788. [Google Scholar]
- Redmon, J.; Farhadi, A. YOLO9000: Better, faster, stronger. In Proceedings of the IEEE conference on computer vision and pattern recognition, Honolulu, HI, USA, 21–26 July 2017; pp. 7263–7271. [Google Scholar]
- Redmon, J.; Farhadi, A. Yolov3: An incremental improvement. arXiv 2018, arXiv:1804.02767. [Google Scholar] [CrossRef]
- Liu, W.; Anguelov, D.; Erhan, D.; Szegedy, C.; Reed, S.; Fu, C.-Y.; Berg, A.C. Ssd: Single shot multibox detector. In Proceedings of the European conference on computer vision, Amsterdam, The Netherlands, 11–14 October 2016; pp. 21–37. [Google Scholar]
- Lin, T.-Y.; Goyal, P.; Girshick, R.; He, K.; Dollár, P. Focal loss for dense object detection. In Proceedings of the IEEE international conference on computer vision, Venice, Italy, 22–29 October 2017; pp. 2980–2988. [Google Scholar]
- Cao, Z.; Liao, T.; Song, W.; Chen, Z.; Li, C. Detecting the shuttlecock for a badminton robot: A YOLO based approach. Expert Syst. Appl. 2021, 164, 113833. [Google Scholar] [CrossRef]
- Li, P.; Zhao, W. Image fire detection algorithms based on convolutional neural networks. Case Stud. Therm. Eng. 2020, 19, 100625. [Google Scholar] [CrossRef]
- Bochkovskiy, A.; Wang, C.-Y.; Liao, H.-Y.M. Yolov4: Optimal speed and accuracy of object detection. arXiv 2020, arXiv:2004.10934. [Google Scholar] [CrossRef]
- Wang, Y.; Hua, C.; Ding, W.; Wu, R. Real-time detection of flame and smoke using an improved YOLOv4 network. Signal Image Video Process. 2022, 16, 1109–1116. [Google Scholar] [CrossRef]
- Zhao, Z.; Han, J.; Song, L. YOLO-highway: An improved highway center marking detection model for unmanned aerial vehicle autonomous flight. Math. Probl. Eng. 2021, 2021, 1205153. [Google Scholar] [CrossRef]
- Misra, D. Mish: A self regularized non-monotonic neural activation function. arXiv 2019, arXiv:1908.08681. [Google Scholar] [CrossRef]
- Russakovsky, O.; Deng, J.; Su, H.; Krause, J.; Satheesh, S.; Ma, S.; Huang, Z.; Karpathy, A.; Khosla, A.; Bernstein, M. Imagenet large scale visual recognition challenge. Int. J. Comput. Vis. 2015, 115, 211–252. [Google Scholar] [CrossRef] [Green Version]
- Yang, D.; Zhong, X.; Gu, D.; Peng, X.; Hu, H. Unsupervised framework for depth estimation and camera motion prediction from video. Neurocomputing 2020, 385, 169–185. [Google Scholar] [CrossRef]
- Cheng, Q.; Li, H.; Wu, Q.; Ngan, K.N. Hybrid-loss supervision for deep neural network. Neurocomputing 2020, 388, 78–89. [Google Scholar] [CrossRef]
- Li, S.; Li, Y.; Li, Y.; Li, M.; Xu, X. YOLO-FIRI: Improved YOLOv5 for Infrared Image Object Detection. IEEE Access 2021, 9, 141861–141875. [Google Scholar] [CrossRef]
- Jia, X.; Zhu, C.; Li, M.; Tang, W.; Zhou, W. LLVIP: A visible-infrared paired dataset for low-light vision. In Proceedings of the IEEE/CVF International Conference on Computer Vision, Montreal, BC, Canada, 11–17 October 2021; pp. 3496–3504. [Google Scholar]
- Xu, Z.; Zhuang, J.; Liu, Q.; Zhou, J.; Peng, S. Benchmarking a large-scale FIR dataset for on-road pedestrian detection. Infrared Phys. Technol. 2019, 96, 199–208. [Google Scholar] [CrossRef]
- Dunnings, A.; Breckon, T. Fire Image Data Set for Dunnings 2018 study-PNG still image set. Durh. Univ. 2018. [Google Scholar] [CrossRef]
- Foggia, P.; Saggese, A.; Vento, M. Real-time fire detection for video-surveillance applications using a combination of experts based on color, shape, and motion. IEEE Trans. Circuits Syst. Video Technol. 2015, 25, 1545–1556. [Google Scholar] [CrossRef]
- Dimitropoulos, K.; Barmpoutis, P.; Grammalidis, N. Spatio-temporal flame modeling and dynamic texture analysis for automatic video-based fire detection. IEEE Trans. Circuits Syst. Video Technol. 2014, 25, 339–351. [Google Scholar] [CrossRef]
- Zeng, Y.; Zhang, L.; Zhao, J.; Lan, J.; Li, B. JRL-YOLO: A novel jump-join repetitious learning structure for real-time dangerous object detection. Comput. Intell. Neurosci. 2021, 2021, 5536152 . [Google Scholar] [CrossRef]
Hardware Specifications | Configuration Information |
---|---|
Processor | Intel(R) Xeon(R) CPU E5-2630 v2 @ 2.60GHz |
Graphics card | NVIDIA Quadro K2000 4G |
RAM | 48G |
Operating system | Ubuntu 16.04 |
Width | Height | Channels | Momentum | Batch |
---|---|---|---|---|
608 | 608 | 3 | 0.9 | 64 |
subdivision | learning_rate | max_batches | steps | scales |
32 | 0.001 | 4000 | 3200, 3600 | 0.1, 0.1 |
Detection Model | Precision | Recall | F1-Score | Average IOU |
---|---|---|---|---|
YOLOv4-tiny | 0.82 | 0.75 | 0.79 | 58.43% |
YOLOv5-s | 0.85 | 0.79 | 0.82 | 60.54% |
YOLOv7-tiny | 0.89 | 0.85 | 0.85 | 64.64% |
YOLOv4-F | 0.95 | 0.96 | 0.96 | 72.63% |
Detection Model | [email protected] (%) | [email protected] (%) | [email protected] (%) | [email protected] (%) | [email protected] (%) |
---|---|---|---|---|---|
YOLOv4-tiny | 90.53 | 75.1 | 41.07 | 8.9 | 0.08 |
YOLOv5-s | 91.23 | 74.64 | 47.89 | 8.5 | 0.07 |
YOLOv7-tiny | 92.56 | 84.38 | 50.56 | 10.62 | 0.75 |
YOLOv4-F | 96.54 | 85.83 | 56.75 | 19.81 | 0.82 |
Large Scale Image | Large Scale Image 1 | Large Scale Image 2 | Large Scale Image 3 | Large Scale Image 4 | Large Scale Image 5 | Large Scale Image 6 |
---|---|---|---|---|---|---|
Original image of infrared flame | ||||||
YOLOv4-F flame detection | ||||||
YOLOv4-tiny flame detection | ||||||
YOLOv5-s flame detection | ||||||
YOLOv7-tiny flame detection |
Medium Scale Image | Medium Scale Image 1 | Medium Scale Image 2 | Medium Scale Image 3 | Medium Scale Image 4 | Medium Scale Image 5 | Medium Scale Image 6 |
---|---|---|---|---|---|---|
Original image of infrared flame | ||||||
YOLOv4-F flame detection | ||||||
YOLOv4-tiny flame detection | ||||||
YOLOv5-s flame detection | ||||||
YOLOv7-tiny flame detection |
Small Scale Image | Small Scale Image 1 | Small Scale Image 2 | Small Scale Image 3 | Small Scale Image 4 | Small Scale Image 5 | Small Scale Image 6 |
---|---|---|---|---|---|---|
Original image of infrared flame | ||||||
YOLOv4-F flame detection | ||||||
YOLOv4-tiny flame detection | ||||||
YOLOv5-s flame detection | ||||||
YOLOv7-tiny flame detection |
Publisher’s Note: MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations. |
© 2022 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https://creativecommons.org/licenses/by/4.0/).
Share and Cite
Li, S.; Wang, Y.; Feng, C.; Zhang, D.; Li, H.; Huang, W.; Shi, L. A Thermal Imaging Flame-Detection Model for Firefighting Robot Based on YOLOv4-F Model. Fire 2022, 5, 172. https://doi.org/10.3390/fire5050172
Li S, Wang Y, Feng C, Zhang D, Li H, Huang W, Shi L. A Thermal Imaging Flame-Detection Model for Firefighting Robot Based on YOLOv4-F Model. Fire. 2022; 5(5):172. https://doi.org/10.3390/fire5050172
Chicago/Turabian StyleLi, Sen, Yeheng Wang, Chunyong Feng, Dan Zhang, Huaizhou Li, Wei Huang, and Long Shi. 2022. "A Thermal Imaging Flame-Detection Model for Firefighting Robot Based on YOLOv4-F Model" Fire 5, no. 5: 172. https://doi.org/10.3390/fire5050172
APA StyleLi, S., Wang, Y., Feng, C., Zhang, D., Li, H., Huang, W., & Shi, L. (2022). A Thermal Imaging Flame-Detection Model for Firefighting Robot Based on YOLOv4-F Model. Fire, 5(5), 172. https://doi.org/10.3390/fire5050172