Real-Time Obstacle Detection Method in the Driving Process of Driverless Rail Locomotives Based on DeblurGANv2 and Improved YOLOv4
Abstract
:1. Introduction
2. Methods
Algorithm 1: pseudo code of the proposed algorithm [33] |
1. Input: A picture with obstacle targets. |
2. Execute the algorithm in following order to get the desired result. |
3. begin |
4. Blur judgment |
5. do |
6. Use formula (2) to judge whether the image is blurred or not. |
7. Using DeblurGANv2 to deblur the blurred image. |
8. end |
9. Improved YOLOv4 algorithm |
10. do |
11. Replace the backbone network and reduce the number of |
channels in the Neck and Head parts. |
12. Introduce the SANet attention mechanism. |
13. the K-means++ algorithm is adopted to cluster prior frames. |
14. the Focal loss function was introduced to increase the loss |
weight of small target samples. |
15. end |
16. Training network |
17. do |
18. Build dataset. |
19. Training parameter configuration. |
20. Weights: use pre-trained VOC dataset. |
21. Train the network and generate model weights. |
22. end |
23. end |
24. Output: obstacles were detected through the Bounding box around. |
2.1. Image Deblurring via DeblurGANv2
2.2. MobileNetv2 as the YOLOv4 Backbone Network
2.3. SA Attention Mechanism
2.4. Optimal Design of Prior Frame
2.5. Optimization of Loss Function
2.6. Improved Network Structure
3. Results and Discussion
3.1. Create a Dataset
3.2. Test Parameter Configuration
3.3. Model Training and Evaluation Index
3.4. Analysis of Experimental Results
3.4.1. Experimental Results
3.4.2. Ablation Experiment
3.4.3. Comparative Experiments on Different Attention Mechanisms
3.4.4. Comparative Experiment for Different Algorithms
4. Conclusions
Author Contributions
Funding
Institutional Review Board Statement
Informed Consent Statement
Data Availability Statement
Acknowledgments
Conflicts of Interest
References
- Han, J.; Xing, W.; Yang, L.; Zhen, W.; Yun’an, C.; Lei, C. Driverless technology of underground locomotive in coal mine. J. China Coal Soc. 2020, 45, 2104–2115. [Google Scholar]
- Yangyang, C.; Zhenlong, H.; Zhiwei, L. Development trend and key technology of coal mine transportation robot in China. Coal Sci. Technol. 2020, 48, 233–242. [Google Scholar]
- Shirong, G.; Eryi, H.; Wenliang, P. Classification system and key technology of coal mine robot. J. China Coal Soc. 2020, 45, 455–463. [Google Scholar]
- Goodfellow, I.J.; Pouget-Abadie, J.; Mirza, M.; Xu, B.; Warde-Farley, D.; Ozair, S.; Courville, A.; Bengio, Y. Generative adversarial nets. In Proceedings of International Conference on Neural Information Processing Systems; MIT Press: Kuching, Malaysia, 2014; pp. 2672–2680. [Google Scholar]
- Radford, A.; Metz, L.; Chintala, S. Unsupervised representation learning with deep convolutional generative adversarial networks. arXiv 2015, arXiv:1511.06434. [Google Scholar]
- Ledig, C.; Theis, L.; Huszár, F.; Caballero, J.; Cunningham, A.; Acosta, A.; Aitken, A.; Tejani, A.; Totz, J.; Wang, Z.; et al. Photo-realistic single image super-resolution using a generative adversarial network. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Honolulu, HI, USA, 21–26 July 2017; pp. 4681–4690. [Google Scholar]
- Mirza, M.; Osindero, S. Conditional generative adversarial nets. arXiv preprint 2014, arXiv:1411.1784. [Google Scholar] [CrossRef]
- Kupyn, O.; Budzan, V.; Mykhailych, M.; Mishkin, D.; Matas, J. DeblurGAN: Blind motion deblurring using conditional adversarial networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Salt Lake City, UT, USA, 18–22 June 2018; pp. 8183–8192. [Google Scholar] [CrossRef] [Green Version]
- Kupyn, O.; Martyniuk, T.; Wu, J.; Wang, Z. Deblurgan-v2: Eblurring (orders-of-magnitude) faster and better. In Proceedings of the IEEE/CVF International Conference on Computer Vision, Seoul, Republic of Korea, 27 October–2 November 2019; pp. 8878–8887. [Google Scholar]
- Souani, C.; Faiedh, H.; Besbes, K. Efficient algorithm for automatic road sign recognition and its hardware implementation. J. Real Time Image Process. 2014, 9, 79–93. [Google Scholar] [CrossRef]
- Maldonado, B.S.; Lafuente, A.S.; Gil, J.P.; Gomez-Moreno, H.; Lopez-Ferreras, F. Road-sign detection and recognition based on support vector machines. IEEE Trans. Intell. Transp. Syst. 2007, 8, 264–278. [Google Scholar] [CrossRef] [Green Version]
- Girshick, R.; Donahue, J.; Darrell, T.; Malik, J. Rich Feature Hierarchies for Accurate Object Detection and Semantic Segmentation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Washington, DC, USA, 23–28 June 2014; pp. 580–587. [Google Scholar]
- Tian, Y.; Yang, G.; Wang, Z.; Wang, H.; Li, E.; Liang, Z. Apple detection during different growth stages in orchards using the improved YOLO-V3 model. Comput. Electron. Agric. 2019, 157, 417–426. [Google Scholar] [CrossRef]
- Hendry, H.; Chen, R. Automatic License Plate Recognition via sliding-window darknet-YOLO deep learning. Image Vis. Comput. 2019, 87, 47–56. [Google Scholar] [CrossRef]
- Bochkovskiy, A.; Wang, C.Y.; Lia, H.-Y.M.O. YOLOv4: Optimal speed and accuracy of object detection. arXiv 2014, arXiv:2004.10934. [Google Scholar]
- Tan, M.; Pang, R.; Le, Q.V. Efficientdet: Scalable and efficient object detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Washington, DC, USA, 16–18 June 2020; pp. 10781–10790. [Google Scholar]
- Liu, W.; Anguelov, D.; Erhan, D.; Szegedy, C.; Reed, S.; Fu, C.-Y.; Berg, A.C. SSD: Single shot multibox detector. In Proceedings of European Conference on Computer Vision; Springer: Berlin, Germany, 2016; pp. 21–37. [Google Scholar]
- Girshick, R. Fast R-CNN. In IEEE International Conference on Computer Vision; IEEE Press: Washington, DC, USA, 2015; pp. 1440–1448. [Google Scholar]
- Ren, S.Q.; He, K.M.; Girshick, R.; Girshick, R.; Sun, J. Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks. IEEE Trans. Pattern Anal. Mach. Intell. 2017, 39, 1137–1149. [Google Scholar] [CrossRef] [Green Version]
- He, D.; Ren, R.; Li, K.; Zou, Z.; Ma, R.; Qin, Y.; Yang, W. Urban rail transit obstacle detection based on Improved R-CNN. Measurement 2022, 196, 111277. [Google Scholar] [CrossRef]
- He, D.; Qiu, Y.; Miao, J.; Zou, Z.; Li, K.; Ren, C.; Shen, G. Improved Mask R-CNN for obstacle detection of rail transit. Measurement 2022, 190, 110728. [Google Scholar] [CrossRef]
- He, D.; Li, K.; Chen, Y.; Miao, J.; Li, X.; Shan, S.; Ren, R. Obstacle detection in dangerous railway track areas by a convolutional neural network. Meas. Sci. Technol. 2021, 32, 105401. [Google Scholar] [CrossRef]
- He, D.; Zou, Z.; Chen, Y.; Liu, B.; Yao, X.; Shan, S. Obstacle detection of rail transit based on deep learning. Measurement 2021, 176, 109241. [Google Scholar] [CrossRef]
- He, D.; Zou, Z.; Chen, Y.; Liu, B.; Miao, J. Rail Transit Obstacle Detection Based on Improved CNN. IEEE Trans. Instrum. Meas. 2021, 70, 2515114. [Google Scholar] [CrossRef]
- Wang, W.; Wang, S.; Guo, Y.; Zhao, Y. Obstacle detection method of unmanned electric locomotive in coal mine based on YOLOv3–4L. J. Electron. Imaging 2022, 31, 023032. [Google Scholar] [CrossRef]
- Chen, Y.; Lu, C.; Wang, Z. Detection of foreign object intrusion in railway region of interest based on lightweight network. J. Jilin Univ. 2021, 52, 2405–2418. [Google Scholar]
- Han, L.; Zheng, P.; Li, H.; Jiangfan, C.; Zexi, H.; Zutao, Z. A novel early warning strategy for right-turning blind zone based on vulnerable road users detection. Neural Comput. Applic 2022, 34, 6187–6206. [Google Scholar] [CrossRef]
- Dong, C.; Pang, C.; Li, Z.; Zeng, X.; Hu, X. PG-YOLO: A Novel Lightweight Object Detection Method for Edge Devices in Industrial Internet of Things. IEEE Access 2022, 10, 123736–123745. [Google Scholar] [CrossRef]
- Hao, S.; Zhang, X.; Ma, X. Foreign object detection in coal mine conveyor belt based on CBAM-YOLOv5. J. China Coal Soc. 2022, 47, 4147–4156. [Google Scholar]
- Sandler, M.; Howard, A.; Zhu, M.; Zhmoginov, A.; Chen, L.-C. Mobilenetv2: Inverted residuals and linear bottlenecks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, San Juan, PR, USA, 17–19 June 2018; pp. 4510–4520. [Google Scholar] [CrossRef] [Green Version]
- Zhang, Q.-L.; Yang, Y.-B. SA-Net: Shuffle Attention for Deep Convolutional Neural Networks. In Proceedings of the IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Singapore, 6–11 June 2021; pp. 2235–2239. [Google Scholar] [CrossRef]
- Lin, T.Y.; Goyal, P.; Girshick, R.; He, K.; Dollár, P. Focal loss for dense object detection. IEEE Trans. Pattern Anal. Mach. Intell. 2020, 42, 318–327. [Google Scholar] [CrossRef] [PubMed] [Green Version]
- Farid, A.; Hussain, F.; Khan, K.; Shahzad, M.; Khan, U.; Mahmood, Z. A Fast and Accurate Real-Time Vehicle Detection Method Using Deep Learning for Unconstrained Environments. Appl. Sci. 2023, 13, 3059. [Google Scholar] [CrossRef]
- Szegedy, C.; Ioffe, S.; Vanhoucke, V.; Alemi, A.A. Inception-v4, inception-resnet and the impact of residual connections on learning. In Proceedings of the Thirty-First AAAI Conference on Artificial Intelligence, San Francisco, CA, USA, 4–9 February 2017. [Google Scholar]
- Zhou, L.; Min, W.; Lin, D.; Han, Q.; Liu, R. Detecting Motion Blurred Vehicle Logo in IoV Using Filter-DeblurGAN and VL-YOLO. IEEE Trans. Veh. Technol. 2020, 69, 3604–3614. [Google Scholar] [CrossRef]
- Woo, S.; Park, J.; Lee, J.Y.; Kweon, I.S. CBAM: Convolutional block attention module. In Proceedings of the 15th European Conference; Springer: Munich, Germany, 2018; pp. 3–19. [Google Scholar]
- Esteves, R.M.; Hacker, T.; Rong, C. Competitive K-Means, a New Accurate and Distributed K-Means Algorithm for Large Datasets. In Proceedings of the IEEE 5th International Conference on Cloud Computing Technology and Science, Bristol, UK, 2–5 December 2013; pp. 17–24. [Google Scholar]
- Wang, W.; Wang, S.; Guo, Y.; Zhao, Y.; Tong, J.; Yang, T. Detection method of obstacles in the dangerous area of electric locomotive driving based on MSE-YOLOv4-Tiny. Meas. Sci. Technol. 2022, 33, 115403. [Google Scholar] [CrossRef]
- Bargoti, S.; Underwood, J. Deep fruit detection in orchards. In Proceedings of the IEEE International Conference on Robotics and Automation (ICRA), Singapore, 29 May–3 June 2017; pp. 3626–3633. [Google Scholar]
- Ding, W.; Taylor, G. Automatic moth detection from trap images for pest management. Comput. Electron. Agric. 2016, 123, 17–28. [Google Scholar] [CrossRef] [Green Version]
- Ge, P.; Guo, L.; He, D.; Huang, L. Light-weighted vehicle detection network based on improved YOLOv3-tiny. Int. J. Distrib. Sens. Netw. 2022, 18, 15501329221080665. [Google Scholar] [CrossRef]
- Li, X.; Pan, J.; Xie, F.; Zeng, J.; Li, Q.; Huang, X.; Liu, D.; Wang, X.; Huang, X. Fast and accurate green pepper detection in complex backgrounds via an improved Yolov4-tiny model. Comput. Electron. 2021, 191, 106–115. [Google Scholar] [CrossRef]
Input Size | Type | Output Channel Number | Stride |
---|---|---|---|
416 × 416 × 3 | Conv 3 × 3 | 32 | 2 |
208 × 208 × 32 | Block1 | 16 | 1 |
208 × 208 × 16 | Block2 | 24 | 2 |
104 × 104 × 24 | Block1 | 24 | 1 |
104 × 104 × 24 | Block2 | 32 | 2 |
52 × 52 × 32 | Block1 | 32 | 1 |
52 × 52 × 32 | Block1 | 32 | 1 |
52 × 52 × 32 | Block1 | 64 | 1 |
52 × 52 × 64 | Block1 | 64 | 1 |
52 × 52 × 64 | Block1 | 64 | 1 |
52 × 52 × 64 | Block1 | 64 | 1 |
52 × 52 × 64 | Block2 | 96 | 2 |
26 × 26 × 96 | Block1 | 96 | 1 |
26 × 26 × 96 | Block1 | 96 | 1 |
26 × 26 × 96 | Block1 | 96 | 1 |
26 × 26 × 96 | Block1 | 96 | 1 |
26 × 26 × 96 | Block2 | 160 | 2 |
13 × 13 × 160 | Block1 | 160 | 1 |
13 × 13 × 160 | Block1 | 160 | 1 |
13 × 13 × 160 | Block1 | 320 | 1 |
Feature Map | Receptive Field | Anchor |
---|---|---|
13 × 13 | Big | (74,57), (114,204), (126,109) |
26 × 26 | Medium | (45,31), (50,85), (62,148) |
52 × 52 | Small | (20,29), (23,15), (30,54) |
Dataset | Detection Model | Average Precision (AP)/% | mAP/% | ||
---|---|---|---|---|---|
E-L | People | Stone | |||
1 | Improved YOLOv4 proposed | 99.16 | 98.65 | 96.24 | 98.02 |
2 | Improved YOLOv4 proposed | 96.27 | 88.12 | 80.46 | 88.28 |
3 | Improved YOLOv4 proposed | 98.83 | 97.56 | 95.18 | 97.19 |
Network | YOLOv4 and Its Improvements | Average Precision (AP)/% | mAP/% | FPS | ||
---|---|---|---|---|---|---|
E-L | People | Stone | ||||
A | YOLOv4 + K-means clustering | 98.51 | 97.67 | 95.83 | 97.34 | 41 |
B | A + Mobilenetv2 | 98.33 | 99.20 | 88.26 | 95.26 | 72 |
C | B + SA module | 99.32 | 98.41 | 92.15 | 96.63 | 68 |
D | C + K-means++ clustering | 99.22 | 98.86 | 94.38 | 97.49 | 68 |
E | D + Focal loss function | 99.16 | 98.65 | 96.24 | 98.02 | 68 |
Model | Average Precision (AP)/% | mAP/% | FPS | ||
---|---|---|---|---|---|
E-L | People | Stone | |||
Faster R-CNN [19] | 99.48 | 92.53 | 91.24 | 94.42 | 10 |
YOLOv3 [13] | 99.87 | 98.84 | 90.78 | 96.50 | 39 |
YOLOv3-tiny [41] | 98.43 | 98.46 | 81.83 | 92.91 | 104 |
YOLOv4 [15] | 98.51 | 97.67 | 95.83 | 97.34 | 41 |
YOLOv4-tiny [42] | 98.49 | 97.61 | 85.48 | 93.86 | 109 |
Improved YOLOv4 | 99.16 | 98.65 | 96.24 | 98.02 | 68 |
Disclaimer/Publisher’s Note: The statements, opinions and data contained in all publications are solely those of the individual author(s) and contributor(s) and not of MDPI and/or the editor(s). MDPI and/or the editor(s) disclaim responsibility for any injury to people or property resulting from any ideas, methods, instructions or products referred to in the content. |
© 2023 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https://creativecommons.org/licenses/by/4.0/).
Share and Cite
Wang, W.; Wang, S.; Zhao, Y.; Tong, J.; Yang, T.; Li, D. Real-Time Obstacle Detection Method in the Driving Process of Driverless Rail Locomotives Based on DeblurGANv2 and Improved YOLOv4. Appl. Sci. 2023, 13, 3861. https://doi.org/10.3390/app13063861
Wang W, Wang S, Zhao Y, Tong J, Yang T, Li D. Real-Time Obstacle Detection Method in the Driving Process of Driverless Rail Locomotives Based on DeblurGANv2 and Improved YOLOv4. Applied Sciences. 2023; 13(6):3861. https://doi.org/10.3390/app13063861
Chicago/Turabian StyleWang, Wenshan, Shuang Wang, Yanqiu Zhao, Jiale Tong, Tun Yang, and Deyong Li. 2023. "Real-Time Obstacle Detection Method in the Driving Process of Driverless Rail Locomotives Based on DeblurGANv2 and Improved YOLOv4" Applied Sciences 13, no. 6: 3861. https://doi.org/10.3390/app13063861
APA StyleWang, W., Wang, S., Zhao, Y., Tong, J., Yang, T., & Li, D. (2023). Real-Time Obstacle Detection Method in the Driving Process of Driverless Rail Locomotives Based on DeblurGANv2 and Improved YOLOv4. Applied Sciences, 13(6), 3861. https://doi.org/10.3390/app13063861