A Drone Based Transmission Line Components Inspection System with Deep Learning Technique
Abstract
:1. Introduction
1.1. Handcrafted Feature-Based Detection (Do not Include Any Learning)
1.2. Handcrafted Feature-Based Detection (Include Learning)
1.3. Convolutional Neural Network-Based (CNN) Detection
Convolutional Neural Network (CNN)
- Unlike earlier studies that use handcraft features, this study explores the robustness of the CNN features and uses them for the task of multi-type HV transmission line components detection in a highly cluttered environment.
- Successful implementation of an embedded system for real-time processing of drone videos with CNN-based transmission line components detection framework. Unlike previous CNN-based transmission line component detection frameworks where the CNN-based detector is just focused to detect one type of insulator and GPU resources are left unused, this article successfully demonstrated the feasibility of using real-time transmission line component detector that is trained to detect nine different transmission line components with above 90% recall. Moreover, the CNN-based detector is optimized to fully utilize the GPU resources and exhibit real-time processing capabilities.
- A light-weight and robust power line detection algorithm is also proposed in this paper.
- A novel defect analysis method is proposed that can detect multiple defects in transmission line components, such as broken sheds in insulators, balisor fading, broken wires, rust in sag adjustors, splits in insulator, etc. Other than broken shed defects, these transmission line component defect analyzing methods have never been covered in previous studies.
- A complete transmission line component inspection system is presented and its robustness and real-time performance are evaluated.
2. Proposed System
2.1. CNN Based Robust Transmission Line Components Detector
2.2. Power Line Detection
Algorithm 1 Power Line Detection. |
Input: = input image, = 2D edge filter, = For thresholding edge image, = threshold for selecting only longer edges, thrmse = threshold for goodness of fit of the line and points on the line |
Output: = list of end points of line |
Preprocessing: |
Main algorithm: |
fordo |
get bounding rectangle around contour in |
get length of contour in pixels from |
If |
Append |
Append |
Find end points of contours from |
End of if |
End of for |
for do |
for do |
End of for |
Append if ( AND is curve up) |
End of for |
Return |
2.3. Defect Analyzer
2.3.1. Broken Shed or Disk in PorSTI-W and PorSTI-R
2.3.2. Balisor Color Fading Defect
2.3.3. Corrosion Defect in Sag Adjusters
2.3.4. Broken Power Line/Wire-Rope Detection
2.3.5. Broken Vibration Damper
2.3.6. Splits and Puncture Detection in Polymer Insulator
3. Experimental Results and Discussion
3.1. Database Acquisition
3.2. Components Detection
3.3. Power Line Detection
3.4. Defect Analysis
4. Conclusions and Future Directions
Author Contributions
Funding
Conflicts of Interest
References
- Mei, X.; Lu, T.; Wu, X.; Zhang, B. Insulator surface dirt image detection technology based on improved watershed algorithm. In Proceedings of the Asia-Pacific Power and Energy Engineering Conference, Shanghai, China, 27–29 March 2012. [Google Scholar] [CrossRef]
- Junfeng, L.; Min, L.; Qinruo, W. A Novel Insulator Detection Method for Aerial Images. In Proceedings of the 9th International Conference on Automation Engineering (ICCAE), Sydney, Australia, 18–21 February 2017. [Google Scholar] [CrossRef]
- Ali, Z.; Park, U. Real-time Safety Monitoring Vision System for Linemen in Buckets Using Spatio-temporal Inference. Int. J. Control. Autom. Syst. 2020, in press. [Google Scholar] [CrossRef]
- Occupational Safety and Health Administration (Fatality Inspection Data), United States Department of Labor Website. Available online: https://www.osha.gov/dep/fatcat/fy17_federal-state_summaries.csv (accessed on 11 June 2020).
- Siddiqui, Z.A.; Park, U.; Lee, S.-W.; Jung, N.-J.; Choi, M.; Lim, C.; Seo, J.-H. Robust Powerline Equipment Inspection System Based on a Convolutional Neural Network. Sensors 2018, 18, 3837. [Google Scholar] [CrossRef] [PubMed]
- Yang, T.W.; Yin, H.; Ruan, Q.Q.; Han, J.; Qi, J.T.; Yong, Q.; Wang, Z.T.; Sun, Z.Q. Overhead Power Line Detection from UAV Video Images. In Proceedings of the 19th International Conference on Mechatronics and Machine Vision in Practice (M2VIP), Auckland, New Zealand, 28–30 November 2012; Available online: https://ieeexplore.ieee.org/stamp/stamp.jsp?tp=&arnumber=6484570&isnumber=6484553 (accessed on 11 June 2020).
- Yetgin, O.E.; Senturk, Z.; Gerek, O.N. A comparison of line detection methods for power line avoidance in aircrafts. In Proceedings of the 2015 9th International Conference on Electrical and Electronics Engineering (ELECO), Bursa, Turkey, 26–28 November 2015. [Google Scholar] [CrossRef]
- Liu, Y.; Mejias, L.; Li, Z. Fast power line detection and localization using steerable filter for active uav guidance. ISPRS - Int. Arch. Photogramm. Remote. Sens. Spat. Inf. Sci. 2012, 491–496. [Google Scholar] [CrossRef]
- Correa, A.C.; Mondragon, I.F.; Ortiz, F.A.P. Towards visual based navigation with power line detection. In Advances in Visual Computing, ISVC 2014, Lecture Notes in Computer Science; Bebis, G., Boyle, R., Parvin, B., Koracin, D., McMahan, R., Jerald, J., Zhang, H., Drucker, S.M., Kambhamettu, C., Choubassi, M.E., et al., Eds.; Springer: Cham, Switzerland, 2014; Volume 8887, pp. 827–836. [Google Scholar] [CrossRef]
- Sampedro, C.; Martínez, C.; Chauhan, A.; Campoy, P. A supervised approach to electric tower detection and classification for power line inspection. In Proceedings of the 2014 International Joint Conference on Neural Networks (IJCNN), Beijing, China, 6–11 July 2014. [Google Scholar] [CrossRef]
- Han, B.; Wang, X. Learning for Tower Detection of Power Line Inspection. DEStech Trans. Comput. Sci. Eng. 2017. [Google Scholar] [CrossRef]
- Martínez, C.; Sampedro, C.; Chauhan, A.; Campoy, P. Towards autonomous detection and tracking of electric towers for aerial power line inspection. In Proceedings of the 2014 International Conference on Unmanned Aircraft Systems (ICUAS), Orlando, FL, USA, 27–30 May 2014. [Google Scholar] [CrossRef]
- Lin, J.; Han, J.; Chen, F. Defects Detection of Glass Insulator Based on Color Image. Power Syst. Technol. 2011, 35, 127–133. Available online: http://en.cnki.com.cn/Article_en/CJFDTOTAL-DWJS201101023.htm (accessed on 11 June 2020).
- Huang, X.; Zhang, Z. A method to extract insulator image from aerial image of helicopter patrol. Power Syst. Technol. 2010, 34, 194–197. Available online: http://en.cnki.com.cn/Article_en/CJFDTotal-DWJS201001038.htm (accessed on 11 June 2020).
- Li, W.; Ye, G.; Huang, F.; Wang, S.; Chang, W. Recognition of insulator based on developed MPEG-7 texture feature. In Proceedings of the IEEE 3rd Intern. Congress on Image and Signal Processing (ICISP), Yantai, China, 16–18 October 2010. [Google Scholar] [CrossRef]
- Zhang, X.; An, J.; Chen, F. A Simple Method of Tempered Glass Insulator Recognition from Airborne Image. In Proceedings of the 2010 International Conference on Optoelectronics and Image Processing (ICOIP), Haikou, China, 11–12 November 2010; Volume 1, pp. 127–130. [Google Scholar]
- Prasad, S.; Reddy, B.S. Digital image processing techniques for estimating power released from the corona discharges. IEEE Trans. Dielectr. Electr. Insul. 2017, 24, 75–82. [Google Scholar] [CrossRef]
- Han, Y.; Liu, Z.; Lee, D.-J.; Liu, W.; Chen, J.; Han, Z. Computer vision–based automatic rod-insulator defect detection in high-speed railway catenary system. Int. J. Adv. Robot. Syst. 2018, 15, 1–15. [Google Scholar] [CrossRef]
- Zhai, Y.; Cheng, H.; Chen, R.; Yang, Q.; Li, X. Multi-Saliency Aggregation-Based Approach for Insulator Flashover Fault Detection Using Aerial Images. Energies 2018, 11, 340. [Google Scholar] [CrossRef]
- Yu, Y.; Cao, H.; Wang, Z.; Li, Y.; Li, K.; Xie, S. Texture-and-Shape Based Active Contour Model for Insulator Segmentation. IEEE Access 2019, 7, 78706–78714. [Google Scholar] [CrossRef]
- Zhao, J.; Liu, X.; Sun, J.; Lei, L. Detecting insulators in the image of overhead transmission lines. In Intelligent Computing Technology, ICIC 2012, Lecture Notes in Computer Science; Huang, D.S., Jiang, C., Bevilacqua, V., Figueroa, J.C., Eds.; Springer: Berlin/Heidelberg, Germany, 2012; Volume 7389. [Google Scholar] [CrossRef]
- Liao, S.; An, J. A Robust Insulator Detection Algorithm Based on Local Features and Spatial Orders for Aerial Images. IEEE Geosci. Remote. Sens. Lett. 2015, 12, 963–967. [Google Scholar] [CrossRef]
- Zhai, Y.; Wu, Y.; Chen, H.; Zhao, X. A Method of Insulator Detection from Aerial Images. Sensors Transducers 2014, 177, 7–13. Available online: http://www.sensorsportal.com/HTML/DIGEST/august_2014/Vol_177/P_2159.pdf (accessed on 11 June 2020).
- Zuo, D.; Hu, H.; Qian, R.; Liu, Z. An insulator defect detection algorithm based on computer vision. In Proceedings of the 2017 IEEE International Conference on Information and Automation (ICIA), Macau, China, 18–20 July 2017. [Google Scholar] [CrossRef]
- Li, B.; Wu, D.; Cong, Y.; Xia, Y.; Tang, Y. A Method of Insulator Detection from Video Sequence. In Proceedings of the International Symposium on Information Science and Engineering (ISISE), Shanghai, China, 20–22 December 2012. [Google Scholar] [CrossRef]
- Wang, X.; Zhang, Y. Insulator identification from aerial images using Support Vector Machine with background suppression. In Proceedings of the 2016 International Conference on Unmanned Aircraft Systems (ICUAS), Arlington, VA, USA, 7–10 June 2016. [Google Scholar] [CrossRef]
- Jabid, T.; Uddin, Z. Rotation invariant power line insulator detection using local directional pattern and support vector machine. In Proceedings of the 2016 International Conference on Innovations in Science, Engineering and Technology (ICISET), Dhaka, Bangladesh, 28–29 October 2016. [Google Scholar] [CrossRef]
- Oberweger, M.; Wendel, A.; Bischof, H. Visual recognition and fault detection for power line insulators. In Proceedings of the 19th Computer Vision Winter Workshop, Krtiny, Czech Republic, 3–5 February 2014; Kukelov, Z., Heller, J., Eds.; Available online: http://cmp.felk.cvut.cz/cvww2014/papers/19/19.pdf (accessed on 11 June 2020).
- Redmon, J.; Farhadi, A. YOLO9000: Better, Faster, Stronger. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Honolulu, HI, USA, 21–26 July 2017. [Google Scholar] [CrossRef]
- Ren, S.; He, K.; Ross, G.; Sun, J. Faster R-CNN: Towards real-time object detection with region proposal networks. In Proceedings of the 28th International Conference on Neural Information Processing Systems – Volume 1, Montreal, ON, Canada, 7–12 December 2015. [Google Scholar] [CrossRef]
- Liu, W.; Anguelov, D.; Erhan, D.; Szegedy, C.; Reed, S.; Fu, C.-Y.; Berg, A.C. SSD: Single shot multibox detector. In Computer Vision - ECCV 2016, Lecture Notes in Computer Science; Leibe, B., Matas, J., Sebe, N., Welling, M., Eds.; Springer: Cham, Switzerland, 2016; Volume 9905, pp. 21–37. [Google Scholar]
- Razavian, A.S.; Azizpour, H.; Sullivan, J.; Carlsson, S. CNN features off-the-shelf: An astounding baseline for recognition. In Proceedings of the IEEE on Computer Vision and Pattern Recognition Workshops, Columbus, OH, USA, 23–28 June 2014. [Google Scholar] [CrossRef]
- Zhao, Z.; Zhen, Z.; Zhang, L.; Qi, Y.; Kong, Y.; Zhang, K. Insulator Detection Method in Inspection Image Based on Improved Faster R-CNN. Energies 2019, 12, 1204. [Google Scholar] [CrossRef]
- Alka, N.U.; Jiya, J.D.; Anenne, E.C.; Haruna, Y.S.; Hamid, M.A. An effective algorithm for power line detection from UAV images. Int. J. Adv. Res. Electr. Electron. Instrum. Eng. 2018, 7, 3607–3616. [Google Scholar]
- Von Gioi, R.; Jakubowicz, J.; Morel, J.-M.; Randall, G. LSD: A Fast Line Segment Detector with a False Detection Control. IEEE Trans. Pattern Anal. Mach. Intell. 2008, 32, 722–732. [Google Scholar] [CrossRef] [PubMed]
- Ma, Q.; Goshi, D.S.; Shih, Y.-C.; Sun, M.-T. An Algorithm for Power Line Detection and Warning Based on a Millimeter-Wave Radar Video. IEEE Trans. Image Process. 2011, 20, 3534–3543. [Google Scholar] [CrossRef]
- Lecuna, R.; Castro, P.; Manana, M.; Laso, A.; Domingo, R.; Arroyo, A.; Martinez, R. Non-contact temperature measurement method for dynamic rating of overhead power lines. Electr. Power Syst. Res. 2020, 185, 106392. [Google Scholar] [CrossRef]
- Phillips, A.J.; Engelbrecht, C.S.; Major, J.M.; Lynch, R.C. Leakage Current Sensor for Suspension Type Insulator. US Patent US20160018455A1, 27 December 2012. [Google Scholar]
- Wydra, M.; Kacejko, P. Power system state estimation using wire temperature measurements for model accuracy enhancement. In Proceedings of the IEEE PES Innovative Smart Grid Technologies Conference Europe (ISGT-Europe), Ljubljana, Slovenia, 9–12 October 2016. [Google Scholar] [CrossRef]
- Qin, X.; Wu, G.; Lei, J.; Fan, F.; Ye, X.; Mei, Q. A Novel Method of Autonomous Inspection for Transmission Line based on Cable Inspection Robot LiDAR Data. Sensors 2018, 18, 596. [Google Scholar] [CrossRef]
- Qin, X.; Wu, G.; Lei, J.; Fan, F.; Ye, X. Detecting Inspection Objects of Power Line from Cable Inspection Robot LiDAR Data. Sensors 2018, 18, 1284. [Google Scholar] [CrossRef]
- Redmon, J. Darknet: Open Source Neural Networks in C. Available online: http://pjreddie.com/darknet/ (accessed on 30 May 2020).
- Electric Power Research Institute. Field Guide: Visual Inspection of Polymer Insulators. EPRI 2014. Available online: https://www.epri.com/#/pages/product/3002003720/ (accessed on 11 June 2020).
- Rosli, H.; Othman, N.A.; Jamail, N.A.M.; Ismail, M.N. Potential and Electric Field Characteristics of Broken Porcelain Insulator. Int. J. Electr. Comput. Eng. (IJECE) 2017, 7, 3114. [Google Scholar] [CrossRef]
- HAVAD Industrial Limited (Warning Sphere). Available online: https://www.warningspheres.com/pdf/warning-sphere.pdf (accessed on 30 May 2020).
- Dolan, S.E.; Ziehm, M.W.; Kuhns, E.C.; Lombardo, M.J.R. Electroceramic Coating of a Wire for Use in a Bundled Power Transmission Cable. US Patent US9953747B2, 6 February 2017. [Google Scholar]
- Liu, X.; Jiang, H.; Chen, J.; Chen, J.; Zhuang, S.; Miao, X. Insulator detection in aerial images based on faster regions with convolutional neural network. In Proceedings of the 2018 IEEE 14th International Conference on Control and Automation (ICCA), Anchorage, AK, USA, 12–15 June 2018. [Google Scholar] [CrossRef]
- Miao, X.; Liu, X.; Chen, J.; Zhuang, S.; Fan, J.; Jiang, H. Insulator Detection in Aerial Images for Transmission Line Inspection Using Single Shot Multibox Detector. IEEE Access 2019, 7, 9945–9956. [Google Scholar] [CrossRef]
- Tao, X.; Zhang, D.; Wang, Z.; Liu, X.; Zhang, H.; Xu, D. Detection of Power Line Insulator Defects Using Aerial Images Analyzed With Convolutional Neural Networks. IEEE Trans. Syst. Man, Cybern. Syst. 2020, 50, 1486–1498. [Google Scholar] [CrossRef]
- Han, J.; Yang, Z.; Zhang, Q.; Chen, C.; Li, H.; Lai, S.; Hu, G.; Xu, C.; Xu, H.; Wang, D.; et al. A Method of Insulator Faults Detection in Aerial Images for High-Voltage Transmission Lines Inspection. Appl. Sci. 2019, 9, 2009. [Google Scholar] [CrossRef]
- Everingham, M.; Van Gool, L.; Williams, C.K.I.; Winn, J.; Zisserman, A. The Pascal Visual Object Classes (VOC) Challenge. Int. J. Comput. Vis. 2009, 88, 303–338. [Google Scholar] [CrossRef]
- Wu, Q.; An, J. An Active Contour Model Based on Texture Distribution for Extracting Inhomogeneous Insulators from Aerial Images. IEEE Trans. Geosci. Remote. Sens. 2013, 52, 3613–3626. [Google Scholar] [CrossRef]
- Github. Available online: https://github.com/primetang/LSD-OpenCV-MATLAB (accessed on 3 March 2020).
- Github. Available online: https://github.com/mtamburrano/LBD_Descriptor/blob/master/EDLineDetector.cpp (accessed on 3 March 2020).
- Zhai, Y.; Zhang, M.; Guo, F.; Wang, D.; Wang, J. Fault detection of insulator based on saliency and adaptive morphology. Multimedia Tools Appl. 2016, 76, 12051–12064. [Google Scholar] [CrossRef]
- Zhai, Y.; Chen, R.; Yang, Q.; Li, X.; Zhao, Z. Insulator Fault Detection Based on Spatial Morphological Features of Aerial Images. IEEE Access 2018, 6, 35316–35326. [Google Scholar] [CrossRef]
- Cheng, H.; Zhai, Y.; Chen, R.; Wang, D.; Dong, Z.; Wang, Y. Self-Shattering Defect Detection of Glass Insulators Based on Spatial Features. Energies 2019, 12, 543. [Google Scholar] [CrossRef]
Components Type | YOLO V3 | YOLO V3 (Multi-Scaling Removed) | |||||
---|---|---|---|---|---|---|---|
#Train Samples | #Test Samples | Total #Samples | Precision (%) | Recall (%) | Precision (%) | Recall (%) | |
Transmission-tower | 4002 | 1458 | 5460 | 80.86 | 84.03 | 81.81 | 85.46 |
Spacer | 2692 | 464 | 3156 | 78.87 | 86.93 | 81.9 | 92.96 |
Balisor | 316 | 82 | 398 | 100.00 | 100.00 | 100.00 | 100.00 |
Lightning-arrester | 2982 | 454 | 3436 | 83.91 | 89.42 | 84.93 | 90.75 |
PorSTI-W+PorSTI-R | 7404 | 990 | 8394 | 91.87 | 97.07 | 93.42 | 97.47 |
Insulator (polymer) | 800 | 48 | 848 | 92.23 | 95.36 | 93.35 | 96.21 |
Damper-weight | 4088 | 352 | 4440 | 77.19 | 75.00 | 79.83 | 81.45 |
Sag adjuster | 1830 | 334 | 2164 | 71.85 | 86.64 | 75.45 | 87.2 |
Avg. | 24,114 | 4182 | 28,296 | 84.60 | 89.31 | 86.34 | 91.44 |
Method | Evaluation Dataset Size | Recall (%) | Precision (%) | Avg. Precision (%) |
---|---|---|---|---|
Wu and An [52] | 50 | 86.47 | 85.59 | 86.03 |
Liao and An [22] | 100 | 91.00 | 87.00 | 89.00 |
Oberweger et al. [28] | 375 | 98.00 | 33.00 | 65.50 |
Jabid [27] | 500 | 94.24 | 89.54 | 91.89 |
Zhao et al. [33] | 380 | 75.00 | 85.00 | 80.00 |
Liu et al. [47] | 500 | 87.53 | 94.40 | 90.96 |
Miao et al. [48] | 200 | 90.00 | 93.75 | 91.87 |
Tao et al. [49] | 385 | 96.60 | 90.40 | 93.50 |
Han et al. [50] | 1356 | 87.36 | 89.96 | 88.66 |
Proposed | 990 | 97.47 | 93.42 | 95.45 |
Method | Precision @ 80% Recall | Speed (ms) |
---|---|---|
EDLines [7] | 52.33 | 78.25 |
LSD [35] | 35.12 | 415.00 |
Proposed | 90.56 | 30.21 |
Method | # of Test Samples | Precision (%) | Recall (%) | Processing Time (s) | Training Time and GPU | |
---|---|---|---|---|---|---|
Broken shed | ResNet [49] | 60 | 91.00 | 95.80 | 0.149 | 16 h on GTX-1080 |
VGG-16 [49] | 41.50 | 62.90 | 0.080 | |||
[50] | 90 | 91.70 | 95.00 | 0.127 | 28 h on Titan X | |
[55] | 90 | 67.20 | 43.57 | 0.525 | Not required | |
[56] | 90 | 77.77 | 52.10 | 0.667 | Not required | |
[57] | 67 | 92.54 | 92.87 | 0.580 | Not required | |
proposed | 75 | 87.50 | 93.33 | 0.065 | Not required | |
Balisor fading | Proposed | 75 | 76.19 | 100.00 | 0.102 | Not required |
Rust in sag adjusters | proposed | 76 | 70.27 | 92.31 | 0.073 | Not required |
Splits in PolSTI | proposed | 547 | 38.84 | 93.75 | 0.153 | Not required |
© 2020 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (http://creativecommons.org/licenses/by/4.0/).
Share and Cite
Siddiqui, Z.A.; Park, U. A Drone Based Transmission Line Components Inspection System with Deep Learning Technique. Energies 2020, 13, 3348. https://doi.org/10.3390/en13133348
Siddiqui ZA, Park U. A Drone Based Transmission Line Components Inspection System with Deep Learning Technique. Energies. 2020; 13(13):3348. https://doi.org/10.3390/en13133348
Chicago/Turabian StyleSiddiqui, Zahid Ali, and Unsang Park. 2020. "A Drone Based Transmission Line Components Inspection System with Deep Learning Technique" Energies 13, no. 13: 3348. https://doi.org/10.3390/en13133348
APA StyleSiddiqui, Z. A., & Park, U. (2020). A Drone Based Transmission Line Components Inspection System with Deep Learning Technique. Energies, 13(13), 3348. https://doi.org/10.3390/en13133348