Bolt Installation Defect Detection Based on a Multi-Sensor Method
Abstract
:1. Introduction
2. Related Work and Contributions
2.1. State of the Art
2.2. Main Contributions
- The torque and angle sensors inside the screwdriver and the visual sensor described are used for the comprehensive detection and judgment of bolt installation defects. Torque and angle sensors can detect bolt torque defects and whether bolts have slipped. Vision sensors can detect the incorrect and missing installation of bolts.
- Using the YOLO v3 network, the recognition rate of visual detection is up to 99.75%, and the average confidence of the output is 0.947.
- Simulation experiments are carried out for several single-type sensors which are prone to mis-inspection. The results show that the detection method based on multiple sensors can output accurate detection results in the case of bolt missing or incorrect installation, torque defects and whether bolts have slipped, and has great advantages over the detection method based on a single type of sensor.
3. Overview of Detection Methods
- When the inspection results of both modules are normal assembly, the final inspection result is normal assembly of the bolts.
- When the inspection result of the screwdriver module is a torque defect and the inspection result of the visual module is normal assembly, the final inspection result is that the bolt has a torque defect.
- When the inspection result of the screwdriver module is normal assembly and the inspection result of the visual module is incorrect installation, the final inspection result is incorrect installation.
- When the detection result of the screwdriver module is that the torque is too low, the angle is too large and the time is out, and the detection result of the visual module is that the bolt is missing installation, the final detection result is missing installation.
- When the inspection result of the screwdriver module is too large and the inspection result of the visual module is normal assembly, the final inspection result is the bolt slip.
4. Visual Monitoring Based on YOLO v3
4.1. Algorithm Overview
4.2. Network Training
4.2.1. Experimental Data Preparation
4.2.2. Model Training and Experimental Environments
4.2.3. Analysis Results
5. Multiple-Type Sensor Detection Experiment
5.1. Detection Subject
5.2. Experimentation
5.2.1. Normal Bolt Installation
5.2.2. Incorrect Bolt Installation
5.2.3. Missing Bolt Installation
5.2.4. Torque Defects and Bolt Slips
6. Conclusions
Author Contributions
Funding
Institutional Review Board Statement
Informed Consent Statement
Data Availability Statement
Conflicts of Interest
References
- Zhao, Z.; Qi, H.; Qi, Y.; Zhang, K.; Zhai, Y.; Zhao, W. Detection Method Based on Automatic Visual Shape Clustering for Pin- Missing Defect in Transmission Lines. IEEE Trans. Instrum. Meas. 2020, 69, 6080–6091. [Google Scholar] [CrossRef]
- Han, Y.; Han, J.; Ni, Z.; Wang, W.; Jiang, H. Instance Segmentation of Transmission Line Images Based on an Improved D-SOLO Network. In Proceedings of the 2021 IEEE 3rd International Conference on Power Data Science, Harbin, China, 26 December 2021; pp. 40–46. [Google Scholar]
- Tan, L.; Tang, T.; Yuan, D. An Ensemble Learning Aided Computer Vision Method with Advanced Color Enhancement for Corroded Bolt Detection in Tunnels. Sensors 2022, 22, 9715. [Google Scholar] [CrossRef] [PubMed]
- Tang, M.; Meng, C.; Wu, H.; Zhu, H.; Yi, J.; Tang, J.; Wang, Y. Fault Detection for Wind Turbine Blade Bolts Based on GSG Combined with CS-Light GBM. Sensors 2022, 22, 6763. [Google Scholar] [CrossRef] [PubMed]
- Girshick, R.; Donahue, J.; Darrell, T.; Malik, J. Rich feature hierarchies for accurate object detection and semantic segmentation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Columbus, OH, USA, 23–28 June 2014; pp. 580–587. [Google Scholar]
- Girshick, R. Fast R-CNN. In Proceedings of the IEEE International Conference on Computer Vision, Santiago, Chile, 7–13 December 2015; pp. 1440–1448. [Google Scholar]
- Liu, W.; Anguelov, D.; Erhan, D.; Szegedy, C.; Reed, S.; Fu, C.-Y.; Berg, A.C. Ssd: Single Shot Multibox Detector. In Proceedings of the European Conference on Computer Vision, Amsterdam, The Netherlands, 8–16 October 2016; Springer: Cham, Switzerland, 2016; pp. 21–37. [Google Scholar]
- Redmon, J.; Divvala, S.; Girshick, R.; Farhadi, A. You only look once: Unified, real-time object detection. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, NV, USA, 27–30 June 2016; pp. 779–788. [Google Scholar]
- Redmon, J.; Farhadi, A. YOLO 9000: Better, faster, stronger. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Hawaii, HI, USA, 21–26 July 2017; pp. 7263–7271. [Google Scholar]
- Redmon, J.; Farhadi, A. Yolov3: An incremental improvement. In Proceedings of the Computer Vision and Pattern Recognition, Salt Lake City, UT, USA, 18–23 June 2018; pp. 3523–3541. [Google Scholar]
- Zhang, X.; Zhang, L.; Li, D. Transmission Line Abnormal Target Detection Based on Machine Learning Yolo V3. In Proceedings of the 2019 International Conference on Advanced Mechatronic Systems (ICAMechS), Kusatsu, Japan, 26–28 August 2019; pp. 344–348. [Google Scholar]
- Xu, J.; Zhang, C.; Liu, Z.; Pei, Y. A Review on Significant Technologies Related to the Robot-Guided Intelligent Bolt Assembly under Complex or Uncertain Working Conditions. IEEE Access 2019, 7, 136752–136776. [Google Scholar] [CrossRef]
- Yao, J.; Cai, D.; Zhang, H.; Wang, H.; Wu, D.; Zhao, Y. Task-oriented design method and research on force compliant experiment of six-axis wrist force sensor. Mechatronics 2016, 35, 109–121. [Google Scholar] [CrossRef]
- Xiao, H.; Tan, J.X. The application of robot bolt runner in engine assembly. Equip. Manuf. Technol. 2015, 9, 70–74. [Google Scholar]
- Jiang, D.; Shi, Q.F.; Fei, Q.G.; Wu, S.Q. Stiffness identification of fixed bolted-joint interface. J. Solid Rocket Technol. 2014, 37, 688–693. [Google Scholar]
- Jiang, D.; Wu, S.Q.; Shi, Q.F.; Fei, Q.G. Parameter identification of bolted-joint based on the model with thin-layer elements with isotropic constitutive relationship. J. Vib. Shock. 2014, 33, 35–40. [Google Scholar]
- Tan, Z.; Fei, Q.; Wu, H.; Zhang, H.; Jiang, D. Thermal adaptive technique for connecting composite material and high-temperature alloy bolt. J. Southeast Univ. Nat. Sci. Ed. 2017, 47, 337–342. [Google Scholar]
- Wang, M.; Jiang, D. Anti-loosening Experiment of Composite Bolted Structures under High Temperature and Vibration Circumstance. J. Vib. Meas. Diagn. 2018, 38, 1169–1175+1292. [Google Scholar]
- Zhu, J.; Gao, Q.X. Application of two kinds of identification and detection technology in vehicle model error prevention. J. Automob. Ind. 2016, 20, 27–29. [Google Scholar]
- Liang, S.; Xiao, Y.; Ma, N. The Utility Model Relates to a Device for Preventing Bolts from Leaking during Assembly Manufacturing. CN Patent CN 210524387U, 15 August 2019. [Google Scholar]
- Liu, L.; Zhou, F.; He, Y. Automated Visual Inspection System for Bogie Block Key Under Complex Freight Train Environment. IEEE Trans. Instrum. Meas. 2015, 65, 2–14. [Google Scholar] [CrossRef]
- Feng, H.; Jiang, Z.; Xie, F. Automatic Fastener Classification and Defect Detectionin Vision-Based Railway Inspection Systems. IEEE Trans. Instrum. Meas. 2014, 63, 877–888. [Google Scholar] [CrossRef]
- Zhong, J.; Liu, Z.; Han, Z.; Han, Y.; Zhang, W. A CNN-based defect inspection method for catenary split pins in high-speed railway. IEEE Trans. Instrum. Meas. 2018, 68, 2849–2860. [Google Scholar] [CrossRef]
- Yang, J.; Tao, W.; Liu, M.; Zhang, Y.; Zhang, H.; Zhao, H. An efficient direction field-based method for the detection of fasteners on high-speed railways. Sensors 2011, 11, 7364–7381. [Google Scholar] [CrossRef]
- Marino, F.; Distante, A.; Mazzeo, P.L.; Stella, E. A real-time visual inspection system for railway maintenance: Automatic hexagonal-headed bolts detection. IEEE Trans. Syst. Man. Cybern. C. Appl. Rev. 2007, 37, 419–428. [Google Scholar] [CrossRef]
- Aytekin, Ç.; Rezaeitabar, Y.; Dogru, S.; Ulusoy, İ. Railway Fastener Inspection by Real-Time Machine Vision. IEEE Trans. Syst. Man Cybern. Systems 2015, 45, 1101–1107. [Google Scholar] [CrossRef]
- Li, Q.; Ren, S. A Real-Time Visual Inspection System for Discrete Surface Defects of Rail Heads. IEEE Trans. Instrum. Meas. 2012, 61, 2189–2199. [Google Scholar] [CrossRef]
- Resendiz, E.; Molina, L.F.; Hart, J.M.; Edwards, J.R.; Sawadisavi, S.; Ahuja, N.; Barkan, C.P.L. Development of a machine vision system for inspection of railway track components. In Proceedings of the 12th World Conference on Transport Research (WCTR), Lisbon, Portugal, 11–15 July 2010. [Google Scholar]
- Kang, G.; Gao, S.; Yu, L.; Zhang, D.; Wei, X.; Zhan, D. Contact Wire Support Defect Detection Using Deep Bayesian Segmentation Neural Networks and Prior Geometric Knowledge. IEEE Access 2019, 7, 173366–173376. [Google Scholar] [CrossRef]
- Soukup, D.; Huber-Mörk, R. Convolutional neural networks for steel surface defect detection from photometric stereo images. In Proceedings of the International Symposium on Visual Computing, Las Vegas, NV, USA, 8–10 December 2014; Springer: Cham, Switzerland, 2014; pp. 668–677. [Google Scholar]
- Nguyen, T.-C.; Huynh, T.-C.; Ryu, J.-Y.; Park, J.-H.; Kim, J.-T. Bolt-loosening identification of bolt connections by vision image based technique. In Proceedings of the Nondestructive Characterization and Monitoring of Advanced Materials, Aerospace, and Civil Infra-structure 2016, Las Vegas, NV, USA, 21–24 March 2016; pp. 227–243. [Google Scholar]
- Wang, C.; Wang, N.; Ho, S.-C.; Chen, X.; Song, G. Design of a New Vision-Based Method for the Bolts Looseness Detection in Flange Connections. IEEE Trans. Ind. Electron. 2020, 67, 1366–1375. [Google Scholar] [CrossRef]
- Xiao, L.; Wu, B.; Hu, Y. Missing Small Fastener Detection Using Deep Learning. IEEE Trans. Instrum. Meas. 2021, 70, 3502209. [Google Scholar] [CrossRef]
- Liu, L.; Zhao, J.; Chen, Z.; Zhao, B.; Ji, Y. A New Bolt Defect Identification Method Incorporating Attention Mechanism and Wide Residual Networks. Sensors 2022, 22, 7416. [Google Scholar] [CrossRef] [PubMed]
- Zhang, Y.; Sun, X.; Loh, K.J.; Su, W.; Xue, Z.; Zhao, X. Autonomous bolt loosening detection using deep learning. Struct. Health Monit. 2019, 19, 105–122. [Google Scholar] [CrossRef]
- Zhang, Y.; Yuen, K.-V. Bolt damage identification based on orientation-aware center point estimation network. Struct. Health Monit. 2021, 21, 438–450. [Google Scholar] [CrossRef]
- Sun, Y.; Li, M.; Dong, R.; Chen, W.; Jiang, D. Vision-Based Detection of Bolt Loosening Using YOLOv5. Sensors 2022, 22, 5184. [Google Scholar] [CrossRef]
- Pal, J.; Sikdar, S.; Banerjee, S. A deep-learning approach for health monitoring of a steel frame structure with bolted connections. Struct. Control. Health Monit. 2021, 29, e2873. [Google Scholar] [CrossRef]
- Zhao, X.; Zhang, Y.; Wang, N. Bolt loosening angle detection technology using deep learning. Struct. Control. Health Monit. 2019, 26, e2292. [Google Scholar] [CrossRef]
- Pham, H.C.; Ta, Q.-B.; Kim, J.-T.; Ho, D.-D.; Tran, X.-L.; Huynh, T.-C. Bolt-Loosening Monitoring Framework Using an Image Based Deep Learning and Graphical Model. Sensors 2020, 20, 3382. [Google Scholar] [CrossRef]
- Qi, Y.; Wu, X.; Zhao, Z.; Shi, B.; Nie, L. Faster R-CNN Aerial Photographic Transmission Line Bolt Defect Detection Embedded with Dual Attention Mechanism. Chin. J. Image Graph. 2021, 26, 2594–2604. [Google Scholar]
- Li, X.-X.; Li, D.; Ren, W.-X.; Zhang, J.-S. Loosening Identification of Multi-Bolt Connections Based on Wavelet Transform and ResNet-50 Convolutional Neural Network. Sensors 2022, 22, 6825. [Google Scholar] [CrossRef]
- Chen, J.; Liu, Z.; Wang, H.; Núñez, A.; Han, Z. Automatic defect detection of fasteners on the catenary support device using deep convolutional neural network. IEEE Trans. Instrum. Meas. 2017, 67, 257–269. [Google Scholar] [CrossRef]
Contrast Term | Torque Sensor | Range Sensor | Visual Sensor |
---|---|---|---|
Detection content | Incorrect and missing installation, torque defect | Missing installation | Incorrect and missing installation |
Conditions of use | No harsh condition | No harsh condition | Needs steady light |
Robustness | Stronger | Stronger | Susceptible to light conditions |
Characteristic Map Size | Receptive Field | Anchor Box |
---|---|---|
13 × 13 | Large | (116 × 90) (156 × 198) (373 × 326) |
26 × 26 | Medium | (30 × 61) (62 × 45) (59 × 119) |
52 × 52 | Small | (10 × 13) (16 × 30) (33 × 23) |
Parameter | Numerical Value |
---|---|
Weight_decay | 0.0005 |
Batch_ Size | 8 |
Nms_ Iou | 0.3 |
Confidence | 0.5 |
Learning rate | 0.0001 |
Disclaimer/Publisher’s Note: The statements, opinions and data contained in all publications are solely those of the individual author(s) and contributor(s) and not of MDPI and/or the editor(s). MDPI and/or the editor(s) disclaim responsibility for any injury to people or property resulting from any ideas, methods, instructions or products referred to in the content. |
© 2023 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https://creativecommons.org/licenses/by/4.0/).
Share and Cite
An, S.; Xiao, M.; Wang, D.; Qin, Y.; Fu, B. Bolt Installation Defect Detection Based on a Multi-Sensor Method. Sensors 2023, 23, 4386. https://doi.org/10.3390/s23094386
An S, Xiao M, Wang D, Qin Y, Fu B. Bolt Installation Defect Detection Based on a Multi-Sensor Method. Sensors. 2023; 23(9):4386. https://doi.org/10.3390/s23094386
Chicago/Turabian StyleAn, Shizhao, Muzheng Xiao, Da Wang, Yan Qin, and Bo Fu. 2023. "Bolt Installation Defect Detection Based on a Multi-Sensor Method" Sensors 23, no. 9: 4386. https://doi.org/10.3390/s23094386