Real-Time High-Performance Laser Welding Defect Detection by Combining ACGAN-Based Data Enhancement and Multi-Model Fusion
Abstract
:1. Introduction
- (1)
- A hybrid welding fault diagnosis scheme based on ACGAN [43,44] (auxiliary classifier generative adversarial networks) and CNN [45,46] model has been proposed. Fake data are generated by the ACGAN generator using real data, and the CNN classifier is trained with both fake data and real data. Test samples are then input into the trained CNN model for fault diagnosis and prediction.
- (2)
- Secondly, in order to increase the difference between categories and improve the recognition performance of the classifier, a data filtering and purification scheme based on ensemble learning is proposed. Multiple support vector machines (SVMs) [47,48] are used to learn different features of defect states and make integrated classification judgments. This integrated classifier filters out the bad data generated by the generator. The filtered data and the original training data are then put into the CNN model for training.
- (3)
- Finally, under different amounts of training data, the ability of different models to identify welding defects is tested. Through experimental comparison with other classical classification models, the superiority of the ACGAN-SVM-CNN detection scheme has been proved.
2. Auxiliary Classifier Generative Adversarial Networks (ACGAN)
2.1. Structure Principle
2.2. The ACGAN Training Process
3. Algorithmic Design and Experimental Analysis
3.1. Data Acquirement Method and Data Description
- “Qualified” means that no weld defects have been found and that they meet the technological requirements.
- “Defocus 3 mm” refers to the defocusing distance over 3 mm. The focus plane above the workpiece is positive defocus, while the focus plane below the workpiece is negative defocus. The defocusing distance of excessively large absolute value leads to the overly low power density acting on the workpiece, making it difficult to reach the purpose of welding.
- “Defocus −3 mm” represents defocusing distances of less than −3 mm.
- “Deformation” means that as the welding current increases, the width of the weld increases, and splashes occur gradually, resulting in oxidative deformation and roughness on the surface of the weld product.
- “Cracks” refer to high temperature cracks. In the process of laser welding, due to the small heat input of laser, the welding deformation and welding stress are small, thus generally, high temperature cracks will not occur.
- “Repetition” means to weld again based on the existing welded seam.
- “Lack of Weld” indicates that there are some missing welding points. ‘Lack of Weld’ is a widespread operation error.
- “Drift” indicates the welding position suddenly drifted.
- “Tilt” represents the base metal’s tilt during welding, so that defocusing distance has been changed.
- “Watermarks” indicates there is water on the surface of the base metal.
3.2. Design of ACGAN Model
3.3. ACGAN-SVM-CNN Defect Detection Fusion Algorithm
3.4. Data Filtering and Purification Strategy
3.5. Comparative Experiment and Result Analysis
4. Conclusions
Author Contributions
Funding
Institutional Review Board Statement
Informed Consent Statement
Data Availability Statement
Conflicts of Interest
References
- Mira-Aguiar, T.; Galvo, I.; Leito, C.; Rodrigues, D.M. Analysis of weld defects in similar and dissimilar resistance seam welding of aluminium, zinc and galvanised steel. Sci. Technol. Weld. Join. 2015, 20, 409–417. [Google Scholar] [CrossRef]
- Pańcikiewicz, K.; Świerczyńska, A.; Hućko, P.; Tumidajewicz, M. Laser Dissimilar Welding of AISI 430F and AISI 304 Stainless Steels. Materials 2020, 13, 4540. [Google Scholar] [CrossRef] [PubMed]
- Sahul, M.; Tomčíková, E.; Sahul, M.; Pašák, M.; Ludrovcová, B.; Hodulova, E. Effect of Disk Laser Beam Offset on the Microstructure and Mechanical Properties of Copper—AISI 304 Stainless Steel Dissimilar Metals Joints. Metals 2020, 10, 1294. [Google Scholar] [CrossRef]
- Landowski, M.; Świerczyńska, A.; Rogalski, G.; Fydrych, D. Autogenous Fiber Laser Welding of 316L Austenitic and 2304 Lean Duplex Stainless Steels. Materials 2020, 13, 2930. [Google Scholar] [CrossRef] [PubMed]
- You, D.Y.; Gao, X.D.; Katayama, S. Review of laser welding monitoring. Sci. Technol. Weld. Join. 2014, 19, 181–201. [Google Scholar] [CrossRef]
- Bebiano, D.; Alfaro, S. A Weld Defects Detection System Based on a Spectrometer. Sensors 2009, 9, 2851–2861. [Google Scholar] [CrossRef] [Green Version]
- Huang, W.; Kovacevic, R. A Laser-Based Vision System for Weld Quality Inspection. Sensors 2011, 11, 506–521. [Google Scholar] [CrossRef]
- Hong, Y.; Chang, B.; Peng, G.; Yuan, Z.; Hou, X.; Xue, B.; Du, D. In-Process Monitoring of Lack of Fusion in Ultra-Thin Sheets Edge Welding Using Machine Vision. Sensors 2018, 18, 2411. [Google Scholar] [CrossRef] [Green Version]
- Zhao, Z.B.; Qi, H.Y.; Qi, Y.C.; Zhang, K.; Zhai, Y.J.; Zhao, W.Q. Detection Method Based on Automatic Visual Shape Clustering for Pin-Missing Defect in Transmission Lines. IEEE Trans. Inst. Meas. 2020, 69, 6080–6091. [Google Scholar] [CrossRef] [Green Version]
- Zhou, X.E.; Wang, Y.N.; Zhu, Q.; Mao, J.X.; Xiao, C.Y.; Lu, X.; Zhang, H. A Surface Defect Detection Framework for Glass Bottle Bottom Using Visual Attention Model and Wavelet Transform. IEEE Trans. Ind. Inform. 2020, 16, 2189–2201. [Google Scholar] [CrossRef]
- Li, T.H.; Pang, X.H.; Jia, B.Y.; Xia, Y.W.; Zeng, S.M.; Liu, H.L.; Tian, H.; Lin, F.; Wang, D. Detection and Diagnosis of Defect in GIS Based on X-ray Digital Imaging Technology. Energies 2020, 13, 661. [Google Scholar] [CrossRef] [Green Version]
- Moon, J.S.; Yoo, M. Development of a Seismic Detection Technology for High-Speed Trains Using Signal Analysis Techniques. Sensors 2020, 20, 3708. [Google Scholar] [CrossRef] [PubMed]
- Gao, X.D.; Huan, Y.J.; Liu, G.Q. A kind of Laser Welding Defocus Amount Determining Device and Assay Method. C.N. Patent 201710768378, 31 August 2017. [Google Scholar]
- Bardin, F.; Cobo, A.; Collin, O.; López-Higuera, J.M.; Pascal, A.; Dubois, T.; Nylen, P.; Jonsson, P.; Högström, M.; Jones, J.D.C. Closed-loop power and focus control of laser welding for full-penetration monitoring. Appl. Optics. 2005, 44, 13–21. [Google Scholar] [CrossRef] [PubMed]
- Huang, S.H.; Huang, Y.J.; Hsieh, C.H.; Chen, H.Z.; Chui, H.C. Visual-assisted laser microwelding of carbon microfiber on metal plates. Opt. Laser Technol. 2018, 108, 368–371. [Google Scholar] [CrossRef]
- Zhang, Y.X.; Gao, X.D.; You, D.Y.; Katayama, S. Online monitoring of welding status based on a DBN model during laser welding. Engineering 2019, 5, 671–678. [Google Scholar] [CrossRef]
- Purtonen, T.; Kalliosaari, A.; Salminen, A. Monitoring and adaptive control of laser processes. Phys. Procedia 2014, 56, 1218–1231. [Google Scholar] [CrossRef] [Green Version]
- Shevchik, S.A.; Le, T.Q.; Farahani, F.V.; Neige, F.; Meylan, B.; Zanoli, S.; Wasmer, K. Laser Welding Quality Monitoring via Graph Support Vector Machine With Data Adaptive Kernel. IEEE Access 2019, 7, 93108–93122. [Google Scholar] [CrossRef]
- Krajewski, A.; Klekot, G.; Cybulak, M.; Kolodziejczak, P. A Novel Method of Supporting the Laser Welding Process with Mechanical Acoustic Vibrations. Materials 2020, 13, 4179. [Google Scholar] [CrossRef]
- Park, Y.W.; Park, H.; Rhee, S.; Kang, M. Real time estimation of CO2 laser weld quality for automotive industry. Opt. Laser Technol. 2002, 34, 135–142. [Google Scholar] [CrossRef]
- Zhang, Y.X.; Gao, X.D.; You, D.Y.; Zhang, N.F.; Gao, P.P. Welding defects detection based on deep learning with multiple optical sensors during disk laser welding of thick plates. J. Manuf. Syst. 2019, 51, 87–94. [Google Scholar] [CrossRef]
- Jager, M.; Humbert, S.; Hamprecht, F.A. Sputter tracking for the automatic monitoring of industrial laser-welding processes. IEEE Trans. Ind. Electron. 2008, 55, 2177–2184. [Google Scholar] [CrossRef]
- Hipp, D.; Mahrle, A.; Beyer, E.; Jäckel, S.; Hertel, M.; Füssel, U. Thermal Efficiency Analysis for Laser-Assisted Plasma Arc Welding of AISI 304 Stainless Steel. Materials 2019, 12, 1460. [Google Scholar] [CrossRef] [PubMed] [Green Version]
- Bélanger, E.; Bernier, M.; Faucher, D.; Vallée, R. High-power and widely tunable all-fiber Raman laser. J. Lightwave Technol. 2008, 26, 1696–1701. [Google Scholar] [CrossRef]
- Zhang, Y.X.; Gao, X.D.; You, D.Y.; Zhang, N.F. Data-driven detection of laser welding defects based on real-time spectrometer signals. IEEE Sens. J. 2019, 19, 9364–9373. [Google Scholar] [CrossRef]
- Deng, F.Q.; Huang, Y.S.; Lu, S.; Chen, Y.Y.; Chen, J.; Feng, H.; Zhang, J.M.; Yang, Y.; Hu, J.J.; Lam, T.L. A Multi-Sensor Data Fusion System for Laser Welding Process Monitoring. IEEE Access. 2020, 8, 147349–147357. [Google Scholar] [CrossRef]
- Lei, T.; Huang, Y.; Wang, H.; Rong, Y.M. Automatic weld seam tracking of tube-to-tubesheet TIG welding robot with multiple Sensors. J. Manuf. Process. 2021, 63, 60–69. [Google Scholar] [CrossRef]
- Gang, T.; Takahashi, Y.; Wu, L.X. Intelligent pattern recognition and diagnosis of ultrasonic inspection of welding defects based on neural network and information fusion. Sci. Technol. Weld. Join. 2002, 7, 314–320. [Google Scholar] [CrossRef]
- Ma, S.J.; Cai, W.; Liu, W.K.; Shang, Z.W.; Liu, G. A Lighted Deep Convolutional Neural Network Based Fault Diagnosis of Rotating Machinery. Sensors 2019, 19, 2381. [Google Scholar] [CrossRef] [Green Version]
- Cao, P.; Zhang, S.L.; Tang, J. Preprocessing-free gear fault diagnosis using small datasets with deep convolutional neural network-based transfer learning. IEEE Access. 2018, 6, 26241–26253. [Google Scholar] [CrossRef]
- Lu, Q.D.; Yang, R.; Zhong, M.Y.; Wang, Y.Q. An improved fault diagnosis method of rotating machinery using sensitive features and RLS-BP neural network. IEEE Trans. Instrum. Meas. 2019, 69, 1585–1593. [Google Scholar] [CrossRef]
- Yan, H.; Xu, Y.X.; Cai, F.Y.; Zhang, H.; Zhao, W.D.; Gerada, C. PWM-VSI Fault Diagnosis for a PMSM Drive Based on the Fuzzy Logic Approach. IEEE Trans. Power Electr. 2019, 34, 759–768. [Google Scholar] [CrossRef]
- Zhu, H.L.; Wang, H.Z.; Kang, D.H.; Zhang, L.Z.; Lu, L.X.; Yao, J.X.; Hu, Y. Study of joint temporal-spatial distribution of array output for large-scale photovoltaic plant and its fault diagnosis application. Sol. Energy. 2019, 181, 137–147. [Google Scholar] [CrossRef]
- Liu, Z.; Zhang, M.; Liu, F.; Zhang, B. Multidimensional Feature Fusion and Ensemble Learning-Based Fault Diagnosis for the Braking System of Heavy-Haul Train. IEEE Trans. Ind. Inform. 2021, 17, 41–51. [Google Scholar] [CrossRef]
- Wang, Z.R.; Huang, H.A.; Wang, Y.R. Fault diagnosis of planetary gearbox using multi-criteria feature selection and heterogeneous ensemble learning classification. Measurement 2021, 173, 108654. [Google Scholar] [CrossRef]
- Li, Z.C.; Tian, L.; Jiang, Q.C.; Yan, X.F. Fault Diagnostic Method Based on Deep Learning and Multimodel Feature Fusion for Complex Industrial Processes. Ind. Eng. Chem. Res. 2020, 59, 18061–18069. [Google Scholar] [CrossRef]
- Bal, P.R.; Kumar, S. WR-ELM: Weighted Regularization Extreme Learning Machine for Imbalance Learning in Software Fault Prediction. IEEE Trans. Reliab. 2020, 69, 1355–1375. [Google Scholar] [CrossRef]
- Goodfellow, I.J.; Pouget-Abadie, J.; Mirza, M.; Xu, B.; Warde-Farley, D.; Ozair, S.; Courville, A.; Bengio, Y. Generative adversarial networks. NIPS 2014, 3, 2672–2680. [Google Scholar] [CrossRef]
- Waheed, A.; Goyal, M.; Gupta, D.; Khanna, A.; Al-Turjman, F.; Pinheiro, P.R. CovidGAN: Data Augmentation Using Auxiliary Classifier GAN for Improved Covid-19 Detection. IEEE Access 2020, 8, 91916–91923. [Google Scholar] [CrossRef]
- Fahimi, F.; Dosen, S.; Ang, K.K.; Mrachacz-kersting, N.; Guan, C. Generative Adversarial Networks-Based Data Augmentation for Brain-Computer Interface. IEEE Trans. Neur. Net. Lear. 2020, 32, 4039–4051. [Google Scholar] [CrossRef]
- Guo, R.Y.; Liu, H.; Xie, G.; Zhang, Y.M. Weld Defect Detection From Imbalanced Radiographic Images Based on Contrast Enhancement Conditional Generative Adversarial Network and Transfer Learning. IEEE Sens. J. 2021, 21, 10844–10853. [Google Scholar] [CrossRef]
- Jiang, X.Y.; Ge, Z.Q. Data Augmentation Classifier for Imbalanced Fault Classification. IEEE Trans. Autom. Sci. Eng. 2021, 18, 1206–1217. [Google Scholar] [CrossRef]
- Odena, A.; Olah, C.; Shlens, J. Conditional Image Synthesis with Auxiliary Classifier GANs. In Proceedings of the 34th International Conference on Machine Learning, Sydney, Australia, 6–11 August 2017. [Google Scholar]
- Wu, Z.; Chen, H.; Lei, Y. Unidimensional ACGAN Applied to Link Establishment Behaviors Recognition of a Short-Wave Radio Station. Sensors 2020, 20, 4270. [Google Scholar] [CrossRef]
- Wang, Z.; Jing, J. Pixel-Wise Fabric Defect Detection by CNNs Without Labeled Training Data. IEEE Access 2020, 8, 161317–161325. [Google Scholar] [CrossRef]
- Yuan, C.; Chen, W.S.; Hao, H.; Kong, Q.Z. Near real-time bolt-loosening detection using mask and region-based convolutional neural network. Struct. Control Health Monit. 2021, 28, e2741. [Google Scholar] [CrossRef]
- Li, X.; Ling, S.H.; Su, S. A Hybrid Feature Selection and Extraction Methods for Sleep Apnea Detection Using Bio-Signals. Sensors 2020, 20, 4323. [Google Scholar] [CrossRef] [PubMed]
- Zeng, R.D.; Lu, Y.S.; Long, S.; Wang, C.; Bai, J.Y. Cardiotocography Signal Abnormality Classification Using Time-frequency Features and Ensemble Cost-sensitive SVM Classifier. Comput. Biol. Med. 2021, 130, 104218. [Google Scholar] [CrossRef]
- Webb, G.I.; Zheng, Z. Multistrategy ensemble learning: Reducing error by combining ensemble learning techniques. IEEE Trans. Knowl. Data En. 2004, 16, 980–991. [Google Scholar] [CrossRef] [Green Version]
- Wang, X.Z.; Xing, H.J.; Li, Y.; Hua, Q.; Dong, C.R.; Pedrycz, W. A Study on Relationship Between Generalization Abilities and Fuzziness of Base Classifiers in Ensemble Learning. IEEE Trans. Fuzzy Syst. 2015, 23, 1638–1654. [Google Scholar] [CrossRef]
Parameters | Specifications |
---|---|
RAM | 256 G |
CPU | Intel(R) Xeon(R) CPU E5-2680 v2 @2.80 GHZ |
GPU | GeForce GTX TITAN X |
OS | Windows 10 |
Operation | Kernel | Strides | Feature Maps | BN? | Dropout | Nonlinearity |
---|---|---|---|---|---|---|
input (128 × 512 × 1 × 1) | ||||||
Linear | N/A | N/A | 128 × 256 × 1 × 1 | × | 0.0 | Leaky ReLu |
Linear | N/A | N/A | 128 × 16,640 × 1 × 1 | × | 0.0 | N/A |
Upsample | scale factor = 2 | 128 × 128 × 130 × 4 | ||||
Convolution | 3 × 2 | 1 × 1 | 128 × 128 × 130 × 3 | √ | 0.0 | Leaky ReLu |
Convolution | 3 × 1 | 1 × 1 | 128 × 64 × 130 × 3 | √ | 0.0 | Leaky ReLu |
Convolution | 3 × 1 | 1 × 1 | 128 × 1 × 130 × 3 | × | 0.0 | Sigmoid |
Operation | Kernel | Strides | Feature Maps | BN? | Dropout | Nonlinearity |
---|---|---|---|---|---|---|
input (1 × 130 × 3) | ||||||
Convolution | 3 × 3 | 2 | 128 × 16 × 65 × 2 | × | 0.2 | Leaky ReLu |
Convolution | 3 × 3 | 2 | 128 × 32 × 33 × 1 | √ | 0.2 | Leaky ReLu |
Convolution | 3 × 3 | 2 | 128 × 32 × 17 × 1 | √ | 0.2 | Leaky ReLu |
Convolution | 3 × 3 | 2 | 128 × 64 × 9 × 1 | √ | 0.2 | Leaky ReLu |
Convolution | 3 × 3 | 2 | 128 × 128 × 5 × 1 | √ | 0.2 | Leaky ReLu |
Class | Qua (%) | Def3 (%) | Def-3 (%) | Defor (%) | Cra (%) | Rep (%) | LoW (%) | Dri (%) | Tilt (%) | W (%) | |
---|---|---|---|---|---|---|---|---|---|---|---|
Metrics | |||||||||||
Precision | 100.0 | 95.87 | 99.17 | 99.24 | 95.20 | 97.83 | 98.29 | 99.22 | 99.22 | 99.16 | |
Recall | 98.80 | 95.87 | 98.35 | 97.74 | 97.54 | 99.26 | 99.14 | 99.22 | 99.22 | 98.33 | |
F1 | 99.40 | 95.87 | 98.75 | 98.48 | 96.35 | 98.54 | 98.71 | 99.22 | 99.22 | 98.74 |
Training (min) | Testing (ms/Sample) | |
---|---|---|
ACGAN-SVM-CNN | 21.61 | 0.76 |
ACGAN-CNN | 16.19 | 0.71 |
CNN | 6.81 | 0.69 |
SVM | 4.53 | 1.78 |
ADABOOST | 5.21 | 2.26 |
100% | 90% | 80% | 75% | 70% | |
---|---|---|---|---|---|
ACGAN-SVM-CNN | 98.37% | 98.02% | 96.78% | 92.02% | 91.15% |
ACGAN-CNN | 97.86% | 97.33% | 96.62% | 92.18% | 90.04% |
CNN | 96.83% | 96.17% | 95.82% | 86.75% | 86.33% |
ACGAN discriminator | 85.13% | 76.54% | 64.85% | 60.37% | 53.26% |
SVM | 83.35% | 81.47% | 80.26% | 65.31% | 64.25% |
ADABOOST | 81.65% | 80.25% | 71.35% | 66.27% | 63.93% |
Publisher’s Note: MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations. |
© 2021 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https://creativecommons.org/licenses/by/4.0/).
Share and Cite
Fan, K.; Peng, P.; Zhou, H.; Wang, L.; Guo, Z. Real-Time High-Performance Laser Welding Defect Detection by Combining ACGAN-Based Data Enhancement and Multi-Model Fusion. Sensors 2021, 21, 7304. https://doi.org/10.3390/s21217304
Fan K, Peng P, Zhou H, Wang L, Guo Z. Real-Time High-Performance Laser Welding Defect Detection by Combining ACGAN-Based Data Enhancement and Multi-Model Fusion. Sensors. 2021; 21(21):7304. https://doi.org/10.3390/s21217304
Chicago/Turabian StyleFan, Kui, Peng Peng, Hongping Zhou, Lulu Wang, and Zhongyi Guo. 2021. "Real-Time High-Performance Laser Welding Defect Detection by Combining ACGAN-Based Data Enhancement and Multi-Model Fusion" Sensors 21, no. 21: 7304. https://doi.org/10.3390/s21217304