Next Article in Journal
Effects of Concurrent and Terminal Visual Feedback on Ankle Co-Contraction in Older Adults during Standing Balance
Next Article in Special Issue
Neural Architecture Search for 1D CNNs—Different Approaches Tests and Measurements
Previous Article in Journal
Optical OFDM Error Floor Estimation by Means of OTDR Enhanced by Front-End Optical Preamplifier
Previous Article in Special Issue
An Improvement of the Fire Detection and Classification Method Using YOLOv3 for Surveillance Systems
 
 
Font Type:
Arial Georgia Verdana
Font Size:
Aa Aa Aa
Line Spacing:
Column Width:
Background:
Article

Real-Time High-Performance Laser Welding Defect Detection by Combining ACGAN-Based Data Enhancement and Multi-Model Fusion

1
School of Computer and Information, Hefei University of Technology, Hefei 230009, China
2
Biomedical Device Innovation Center, Shenzhen Technology University, Shenzhen 518118, China
*
Authors to whom correspondence should be addressed.
Sensors 2021, 21(21), 7304; https://doi.org/10.3390/s21217304
Submission received: 21 September 2021 / Revised: 11 October 2021 / Accepted: 30 October 2021 / Published: 2 November 2021
(This article belongs to the Special Issue Artificial Intelligence for Smart Sensing, Test and Measurement)

Abstract

:
Most of the existing laser welding process monitoring technologies focus on the detection of post-engineering defects, but in the mass production of electronic equipment, such as laser welding metal plates, the real-time identification of defect detection has more important practical significance. The data set of laser welding process is often difficult to build and there is not enough experimental data, which hinder the applications of the data-driven laser welding defect detection method. In this paper, an intelligent welding defect diagnosis method based on auxiliary classifier generative adversarial networks (ACGAN) has been proposed. Firstly, a ten-class dataset consisting of 6467 samples, was constructed, which originate from the optical and thermal sensory parameters in the welding process. A new structured ACGAN network model is proposed to generate fake data similar to the true defect feature distributions. In addition, in order to make the difference between different defects categories more obvious after data expansion, a data filtering and data purification scheme was proposed based on ensemble learning and an SVM (support vector machine), which is used to filter the bad generated data. In the experiments, the classification accuracy can reach 96.83% and 85.13%, for the CNN (convolutional neural network) algorithm model and ACGAN model, respectively. However, the accuracy can further improve to 97.86% and 98.37% for the fusion models of ACGAN-CNN and ACGAN-SVM-CNN models, respectively. The results show that ACGAN can not only be used as an algorithm model for classification, but also be used to achieve superior real-time classification and recognition through data enhancement and multi-model fusion.

1. Introduction

In the manufacturing process of metal products, laser welding, as an important processing technology, has been widely used in many fields of industry. In the laser welding process, there will be severe thermal conversion effects, which require extremely high precision for the welding process parameter and fixing workpiece, and small weld deviation can lead to serious welding defects, which affects the quality of the welded products [1]. Moreover, the difference of welding materials and welded joints will also affect the quality of welding [2,3,4]. Therefore, rapid and effective defect detection of welded products is particularly important in the process of mass production. Existing laser welding defect detection schemes are mainly focused on the detection of post-welding defects. Conventional nondestructive testing methods are widely used but they have some limitations [5,6,7]. For example, visual testing can easily miss detection [8,9,10]; radiographic testing has radiation risk and its equipment cost is high [11]; magnetic particle testing and penetration testing can locate the defect position but cannot specifically display its shape; and eddy current detection and other signal detection technologies can not directly reflect the defect shape [12].
Because the actual process of production and manufacturing is complex and changeable, different instruments and methods are needed during defect detection in different processes. In most cases, a single device or method cannot accurately complete the detection task, however, it is impossible to integrate all the corresponding instruments and methods into one single system due to the large workload and high complexity of detecting every possible problem in each production process [13]. In the process of laser welding, the above-mentioned problems still exist. Thus, due to the complexity of both processes and defects, the judgement system is difficult to establish.
In the existing researches, four kinds of sensors are mainly used to monitor the laser welding process, such as visual [14,15,16], acoustic [17,18,19], optical [20,21] and thermal [22,23] sensors. These sensors are used to capture the changes of characteristic quantities during the welding process [24,25,26,27] and establish a correlation between the changing trends of characteristic quantities and the final welding effect. In the actual production line, it is difficult to realize real-time visual testing because of the need to capture and process a large amount of image data. In addition, the strong light during laser welding can lead to image saturation and thus reduce the accuracy of the system. Moreover, due to the unavoidable background noise in the production line, it is not suitable to integrate acoustic sensors into the laser welding process monitoring system to detect defects.
With the rapid developments and maturity of deep learning theory, intelligent fault diagnosis technology has become a research hotspot and development direction in the field of artificial intelligence. At present, the intelligent fault diagnosis algorithms mainly include artificial neural network [28,29,30,31], fuzzy logic [32,33], etc., which developed from single strategy classification prediction to multi-strategy fusion [34,35,36]. In addition, in some actual experiments, it is found that sample data is difficult to obtain, so that it is difficult for the deep learning method to fully learn the differences between different categories of data. To solve this problem, a data enhancement method is commonly used to expand the training data set. In 2020, Bal et al. [37] explored an effective machine learning method, the extreme learning machine, to predict the number of software faults, which could predict the software fault type in time when the software fault data was unbalanced. Generative adversarial networks (GAN) proposed by Goodfellow et al. [38] have been widely used in image processing and natural language processing, in which the generator and discriminator compete with each other, and fake data obtained from the generator and real data can be used to train the discriminator. By adversarial learning mechanism and adding new samples, the discriminator ability and the generation ability can be improved simultaneously, so as to improve the learning and generalization ability of the neural network. In 2020, Waheed et al. [39] proposed Covid-GAN which can produce composite images and can be used to enhance CNN (convolutional neural network) detection performance. In a brain computer interface (BCI) system, the performance of a classifier depends on the quality and quantity of training data to a great extent. Fahimi et al. [40] proposed a framework based on the deep convolutional generative adversarial networks for generating artificial electroencephalogram to augment the training set in order to improve the performance of a BCI classifier. In 2021, Guo et al. [41] developed a welding defect detection method using a generative adversarial network combined with transfer learning which is proposed to solve the data imbalance and improve the accuracy of defect detection. Jiang et al. [42] proposed a data selection strategy based on data filtering and data purification in model training, which combines supervised learning and the data generation process to obtain an end-to-end model.
The main contributions of this paper are summarized as follows.
(1)
A hybrid welding fault diagnosis scheme based on ACGAN [43,44] (auxiliary classifier generative adversarial networks) and CNN [45,46] model has been proposed. Fake data are generated by the ACGAN generator using real data, and the CNN classifier is trained with both fake data and real data. Test samples are then input into the trained CNN model for fault diagnosis and prediction.
(2)
Secondly, in order to increase the difference between categories and improve the recognition performance of the classifier, a data filtering and purification scheme based on ensemble learning is proposed. Multiple support vector machines (SVMs) [47,48] are used to learn different features of defect states and make integrated classification judgments. This integrated classifier filters out the bad data generated by the generator. The filtered data and the original training data are then put into the CNN model for training.
(3)
Finally, under different amounts of training data, the ability of different models to identify welding defects is tested. Through experimental comparison with other classical classification models, the superiority of the ACGAN-SVM-CNN detection scheme has been proved.
The rest of this paper is organized as follows. Section 2 briefly introduces the basic theory of ACGAN. Section 3 describes the proposed hybrid detection approach, and the effectiveness and superiority of the proposed method have been investigated and proved by the comparative experiments of different models with different amounts of original data. Finally, conclusions are drawn in Section 4.

2. Auxiliary Classifier Generative Adversarial Networks (ACGAN)

GAN is an unsupervised deep learning model, from which other types of network structures can be derived, such as conditional generative adversarial nets (CGAN) and semi-supervised learning with generative adversarial networks (SGAN). CGAN improves the quality of generated data by combining tag information, while SGAN improves the data quality by reconstructing tag information. On the basis of these two networks, ACGAN extends its advantages and adds a category classification network to the system.

2.1. Structure Principle

The ACGAN is improved by the GAN model with supervised mechanism. The main difference between ACGAN and GAN is the added label information of auxiliary training. As shown in Figure 1, ACGAN is composed of a generator and a discriminator, whose output contains judgment information of not only the true or false data, but also the data category. Meanwhile, ACGAN considers the diversity of samples better than other varieties of GAN models.
ACGAN can generate higher quality samples using auxiliary classification tags. The ‘ c ’ in Figure 1 represents the class label of the corresponding data. The loss function of AGAN contains two parts, as shown in Equations (1) and (2), respectively.
L s = E x P d a t a [ log D ( x ) ] + E z P z [ log ( 1 D ( G ( z ) ) ) ]
L c = E c P d a t a [ log D ( c ) ] + E c P z [ log ( 1 D ( G ( c ) ) ) ]
In the above formulas, L s represents the cost function for data authenticity and L c represents the cost function for data classification accuracy. E represents the operation to find the mathematical expectation, x / c P d a t a ensures x / c obeys the original data distribution, z / c P z ensures z / c obeys the Gaussian distribution and z represents random noise. Since the discriminator should distinguish the generated data from the real data as much as possible, for classifying the data effectively, the training goal of discriminator D is to make L c + L s as maximum as possible. Meanwhile, it is expected that the data generated by the generator will be recognized as real data by the discriminator and effectively classified, which means that the goal of generator G training is to make L c L s as large as possible.

2.2. The ACGAN Training Process

The training process of ACGAN is basically the same as that of GAN. Based on the theory of zero-sum game, the discriminator and generator are trained alternately to achieve the final optimization effect.
In the ACGAN model, the first step is to train the discriminator with real data and fake data generated by the generator. When the discriminator training is completed, the parameters of the discriminator can be kept unchanged temporarily, at which the parameters of the discriminator will not be updated, and only the parameters in the generator will be updated according to the loss of feedback from the discriminator. The generator will generate fake data which is closer to the real data distribution. The updated generator will generate a new generation of fake data, and then train the discriminator with the new generation of fake data and real data. The iteration training repeats as above. The specific training steps are described as follows:
Input the randomly generated noise z vector with Gaussian normal distribution into generator G , and then generate fake data G ( z ) .
Mark the fake data X f a k e = G ( z ) generated by generator G as 0 and the corresponding category label c is attached. Real data X r e a l is marked as 1 and the corresponding category label c is also attached. The real and fake data are input into discriminator D together in batches, and the network terminal outputs the distinguished result through softmax classifier. The objective function of the optimized discriminator is as follows:
L D = log B r e a l + log ( 1 B f a k e ) + log C r e a l + log C f a k e
In Equation (3), C r e a l is the category probability of multi-classification output when real data are input into the discriminator, and C f a k e is the category probability of multi-classification output when fake data are input into the discriminator. B f a k e is the real/fake (1/0) binary output when the fake data are input into the discriminator, and B r e a l is the real/fake (1/0) binary output when the real data are input into the discriminator.
Keep the parameters of discriminator D unchanged, input the random noise vector z into generator G to generate fake data X f a k e = G ( z ) , and attach the corresponding category label c . Fake data and real data are labeled as 1 together and input to discriminator D . When discriminator D determines that fake data X f a k e is false (the output label is 0), it means that the fake data X f a k e fails to deceive discriminator D successfully. In order to make the fake data generated by generator G successfully deceive discriminator D , it is necessary to maximize the objective function L G of the generator. The definition of L G can be expressed as follow.
L G = log B f a k e + log C r e a l + log C f a k e
Repeat steps (1)–(3) to train the discriminator D and generator G iteratively until Nash equilibrium is reached, in which the true and fake resolution of the discriminator D are 50%. This indicates that the effect of the data generated by the generator is closest to the distribution of the original data, and the trained discriminator can be used for defect detection and classification.

3. Algorithmic Design and Experimental Analysis

In this section, we describe the ACGAN structure in detail and propose a data filtering and data purification strategy based on ensemble learning. By comparing different models and training data settings with different amounts of data, the superiority of the proposed welding defect detection scheme is demonstrated.

3.1. Data Acquirement Method and Data Description

The scanning laser welding machine mainly includes an SPI (Southampton Photonics Inc., Southampton, UK) pulsed fiber laser, a galvanometer scanner, a flat field lens and two vibrating mirrors. The galvanometer scanner can reflect laser light to the desired position by turning the vibrating mirrors to change the laser path. The front-end signal acquisition consists of two different photodiode sensors, a thermometer sensor and some optical elements. The overall schematic structure of the data acquisition system is shown in Figure 2. The first photodiode sensor is installed behind a 45-degree prism to obtain plasma intensity. After the other 45-degree prism, the second photodiode sensor is set up to capture light intensity information. Pyrometer sensors at the end can monitor temperature changes during welding.
In the process of the data collection experiment, 130 sampling times are set for each standard workpiece welding process, and the values of plasma intensity, light intensity and temperature are collected at each sampling time. In this way, we collected and constructed a data set with 6467 data, which contains 10 welding defects. The standard part is SUS304 stainless steel plate (Jiangsu Weigang alloy products Co., Ltd., Wuxi, China) with the thickness of about 0.3 mm and a radius of 22 mm. CW (continuous wave) optical fiber laser (Suzhou Chuangxuan Laser Technology Co., Ltd., Suzhou, China) was used in the experiment. The laser output power is 80 W, and the welding speed is 50 mm/s.
After the data are collected, they need to be standardized. Here we convert the data to a standard dataset with a mean of 0 and a variance of 1. The ten kinds of welding defects are “Qualified”, “Defocus 3 mm”, “Defocus −3 mm”, “Deformation”, “Cracks”, “Repetition”, “Lack of Weld”, “Drift”, “Tilt” and “Watermarks”, respectively. Additionally, the concrete data amounts have been shown in Figure 3. In order to avoid the mode collapse during training, we try to keep the same amount of data for each category when collecting sample data. The descriptions of these categories are as follows:
  • “Qualified” means that no weld defects have been found and that they meet the technological requirements.
  • “Defocus 3 mm” refers to the defocusing distance over 3 mm. The focus plane above the workpiece is positive defocus, while the focus plane below the workpiece is negative defocus. The defocusing distance of excessively large absolute value leads to the overly low power density acting on the workpiece, making it difficult to reach the purpose of welding.
  • “Defocus −3 mm” represents defocusing distances of less than −3 mm.
  • “Deformation” means that as the welding current increases, the width of the weld increases, and splashes occur gradually, resulting in oxidative deformation and roughness on the surface of the weld product.
  • “Cracks” refer to high temperature cracks. In the process of laser welding, due to the small heat input of laser, the welding deformation and welding stress are small, thus generally, high temperature cracks will not occur.
  • “Repetition” means to weld again based on the existing welded seam.
  • “Lack of Weld” indicates that there are some missing welding points. ‘Lack of Weld’ is a widespread operation error.
  • “Drift” indicates the welding position suddenly drifted.
  • “Tilt” represents the base metal’s tilt during welding, so that defocusing distance has been changed.
  • “Watermarks” indicates there is water on the surface of the base metal.
Figure 3. The composition of the data set.
Figure 3. The composition of the data set.
Sensors 21 07304 g003

3.2. Design of ACGAN Model

In the field of deep learning, the dependence of neural network algorithm on the amount of data is well known. As a prior knowledge, the larger the capacity of the original data, the more sufficient the information about the target it contains, which is beneficial to the training of deep neural network. However, in some experimental scenarios where it is difficult to measure or the experimental measurement data is very complicated, it is hard to collect the data. In order to solve this problem, we can use the data enhancement ability of ACGAN to generate fake data and expand the data capacity for training deep neural network.
The Pytorch deep learning framework (Pytoch is the python version of torch (version 1.6.0), which is an open-source neural network framework by Facebook. The pytoch used in this article is downloaded from the Internet.) is used in all of the experiments. The Anaconda3 virtual environment is used for management. The Python version is 3.6.12, the optimizer is “Adam”, and the learning rate is 0.001. The detailed configurations of the experimental environment can be found in Table 1.
During the experiment, we designed a model structure of ACGAN for defect detection of laser welding data. The structure of ACGAN is shown in Table 2 and Table 3. In designing the structure of generator and discriminator, Leaky ReLu is selected as the activation function of each layer through comparative experiments. Such a choice also solves the problem of neuron death. Dropout is used after each convolution layer, which can eliminate the joint adaptability between neurons and enhance the generalization ability of neural network. Using the structure described above, generator G and discriminator D can generally be optimized after 120 training epochs.

3.3. ACGAN-SVM-CNN Defect Detection Fusion Algorithm

The training difficulty of GAN is usually greater than that of a general neural network. When D and G are entangled with each other, the generator generated low-quality and high-quality data. Therefore, the fake data X f a k e generated by the generator of the ACGAN algorithm model are not all optimal data. In order to eliminate some inferior fake data, we proposed to use original training data X t r a i n to train a filtering model based on SVM algorithm, so as to screen the generated fake data, and combined the filtered data X f i l t e r e d _ d a t a and training data X t r a i n together for the training of CNN algorithm model. In order to compare the performance of the fusion algorithm, the data of the test set was derived from real data, and the test set was always kept unchanged. The fused algorithm model is shown in Figure 4.

3.4. Data Filtering and Purification Strategy

Our goal is to select data which can approximate the original data distribution characteristics from the generated data. Using the ACGAN model designed in this paper to generate 2010 pieces of data (201 pieces of data are generated for each welding defect), and the classification accuracy of ACGAN discriminator is 85.13%. For the generated fake data, it is not always very close to the original data distribution, and those inferior data need to be filtered out. In this issue, we propose to use the concept of ensemble learning to filter data. We use the original real data (6467 × 1 × 130 × 3) to train three SVM models, which can be used to filter the fake data. Three SVM classifiers are trained by using the three features in the original data, so that each feature data can be trained to obtain a classifier. In the ensemble learning theory, a strong learner can be formed by training several individual learners and following certain combination strategies [49,50]. We collected three physical quantities (plasma intensity, light intensity and temperature) in the welding process to determine if there is a welding defect or not. These classifiers are then used to identify whether each feature in the generated fake data corresponds to the type of welding defect. The features of the generated fake data are input into the corresponding SVM classifier for classification. If two or more classifiers give correct classification results, the fake data can be retained. The process is shown in Figure 5.
Through the above filtering method, 2010 pieces of data were filtered and 1147 pieces of data were obtained. In this way, the original data were expanded from 6467 pieces to 7614 pieces. By eliminating the fake data which does not match the original data distribution, the differences between categories will be more prominent, which lays a good foundation for the subsequent training of the classification model.

3.5. Comparative Experiment and Result Analysis

In order to compare with the traditional CNN method, we divide the original data into two parts, 5174 pieces are taken as the general training set and 1293 pieces are taken as the general test set. When dividing data sets, the balance of each category of data is fully considered. The fake data filtered by ACGAN- SVM ensemble model and 5174 pieces of original data are combined as an enhanced training set. The enhanced training set is sent to the CNN model for training, and then the general test set is used to test the accuracy of CNN.
The changing curve of loss in the training process and the changing curve of accuracy in the test set are shown in Figure 6. Finally, the classification accuracy of our proposed ensemble model can reach 98.37%.
Figure 7 shows the confusion matrix obtained from 10-fold cross-validation on the enhanced dataset. The analysis of the confusion matrix shows that the CNN classifier at the final training site has a good classification effect. However, there are some deficiencies in the distinction between ”Defocus 3 mm“ and “Cracks”. This indicates that our classifier is not very good at learning the difference between these two types of welding defects and needs further improvement in future research.
Through Figure 7, it can be seen that the classification prediction result of “Defocus 3 mm” is relatively worse. The model sometimes classifies “Defocus 3 mm” into “Cracks” by mistake, which indicates that the two welding defect states have high similarity. As shown in Table 4, we use precision, recall and F1 (F1 score) to measure the detection performance of the model in different categories. The abbreviations “Qua”, “Def3”, “Def-3”, “Defor”, “Cra”, “Rep”, “LoW”, “Dri”, “Tilt” and “W” in the table represent “Qualified”, “Defocus 3 mm”, “Defocus −3 mm”, “Deformation”, “Cracks”, “Repetition”, “Lack of Weld”, “Drift”, “Tilt” and “Watermarks”, respectively. It can be seen from Table 4 that our proposed fusion model has the best performance in detecting the “Qualified” state, but the recognition of “Defocus 3 mm” category is relatively inferior.
The diversity of samples is enriched through data enhancement, and the difference between classes is ensured through filtering and purification of the generated data. In order to compare the time consumption of models with different complexity, we take the training set and test set of the same volume for the timing test (5174 pieces of training data and 1293 pieces of test data), and the results are shown in Table 5. The ACGAN-SVM-CNN can reach an average speed of 0.76 ms per sample, which meets the real-time requirement for industrial production. Because of the complexity of training ACGAN, the training time is relatively long.
In order to further compare the proposed method of ACGAN-SVM-CNN in this paper with other classical methods in machine learning, we adjust the number of original data. We compare the classification accuracy of various algorithm models through reducing the amount of original data by 10%, 20%, 25% and 30%. Table 6 shows the comparison results of classification accuracies of various methods in the case of different amounts of original data. By comparing the experimental results under different data volumes, our proposed detection model has better discrimination ability in detecting defect categories. When the amount of data is reduced to only 75% of the original data, the classification accuracies of all classifiers will decline greatly. In the process of reducing the amount of original data, the accuracy of the proposed fusion model (ACGAN-SVM-CNN) also decays, but it is the slowest one, which reflects the resistance of the model to data loss.

4. Conclusions

In this paper, we have proposed an intelligent diagnosis method (ACGAN-SVM-CNN) for detecting laser welding defects. This detection scheme combines and makes full use of the data enhancement ability of ACGAN and the classification ability of CNN. The trained ACGAN model learns the potential space corresponding to all kinds of welding defect data, and generates fake data for data enhancement. In addition, we propose an ensemble learning-based data filtering and data purification method to filter out the bad ones from generated fake data, which makes the difference between the class characteristics of the enhanced dataset more obvious. The filtered enhanced dataset can train models with higher classification recognition rate. This method has been compared with other existing detection models, in which the amount of original training data has been adjusted differently. The experimental results show that the ACGAN-SVM-CNN scheme can detect the categories of welding defects better when the amount of original training data is decreasing. In our further research, we will develop more effective multi-model fusion strategies and model parameter sharing schemes to improve the training speed and recognition ability of the fusion models.
This method has a good reference significance for the industrial sector where it is difficult to obtain process data, and can be easily extended to deal with the fault or defect detection problems of some key machine components.

Author Contributions

Conceptualization, K.F. and Z.G.; methodology, K.F., P.P. and H.Z.; software, K.F.; investigation, H.Z., P.P. and L.W.; writing—original draft preparation, K.F.; writing—review and editing, P.P. and Z.G.; supervision, Z.G.; funding acquisition, Z.G. All authors have read and agreed to the published version of the manuscript.

Funding

This research was supported by National Natural Science Foundation of China (grant number 61775050), Fundamental Research Funds for the Central Universities (grant number PA2019 GDZC0098), International science and technology cooperation project of the Shenzhen Science and Technology Commission (GJHZ20200731095804014).

Institutional Review Board Statement

Not applicable.

Informed Consent Statement

Not applicable.

Data Availability Statement

The data presented in this study are available on request from the corresponding author.

Conflicts of Interest

The authors declare no conflict of interest.

References

  1. Mira-Aguiar, T.; Galvo, I.; Leito, C.; Rodrigues, D.M. Analysis of weld defects in similar and dissimilar resistance seam welding of aluminium, zinc and galvanised steel. Sci. Technol. Weld. Join. 2015, 20, 409–417. [Google Scholar] [CrossRef]
  2. Pańcikiewicz, K.; Świerczyńska, A.; Hućko, P.; Tumidajewicz, M. Laser Dissimilar Welding of AISI 430F and AISI 304 Stainless Steels. Materials 2020, 13, 4540. [Google Scholar] [CrossRef] [PubMed]
  3. Sahul, M.; Tomčíková, E.; Sahul, M.; Pašák, M.; Ludrovcová, B.; Hodulova, E. Effect of Disk Laser Beam Offset on the Microstructure and Mechanical Properties of Copper—AISI 304 Stainless Steel Dissimilar Metals Joints. Metals 2020, 10, 1294. [Google Scholar] [CrossRef]
  4. Landowski, M.; Świerczyńska, A.; Rogalski, G.; Fydrych, D. Autogenous Fiber Laser Welding of 316L Austenitic and 2304 Lean Duplex Stainless Steels. Materials 2020, 13, 2930. [Google Scholar] [CrossRef] [PubMed]
  5. You, D.Y.; Gao, X.D.; Katayama, S. Review of laser welding monitoring. Sci. Technol. Weld. Join. 2014, 19, 181–201. [Google Scholar] [CrossRef]
  6. Bebiano, D.; Alfaro, S. A Weld Defects Detection System Based on a Spectrometer. Sensors 2009, 9, 2851–2861. [Google Scholar] [CrossRef] [Green Version]
  7. Huang, W.; Kovacevic, R. A Laser-Based Vision System for Weld Quality Inspection. Sensors 2011, 11, 506–521. [Google Scholar] [CrossRef]
  8. Hong, Y.; Chang, B.; Peng, G.; Yuan, Z.; Hou, X.; Xue, B.; Du, D. In-Process Monitoring of Lack of Fusion in Ultra-Thin Sheets Edge Welding Using Machine Vision. Sensors 2018, 18, 2411. [Google Scholar] [CrossRef] [Green Version]
  9. Zhao, Z.B.; Qi, H.Y.; Qi, Y.C.; Zhang, K.; Zhai, Y.J.; Zhao, W.Q. Detection Method Based on Automatic Visual Shape Clustering for Pin-Missing Defect in Transmission Lines. IEEE Trans. Inst. Meas. 2020, 69, 6080–6091. [Google Scholar] [CrossRef] [Green Version]
  10. Zhou, X.E.; Wang, Y.N.; Zhu, Q.; Mao, J.X.; Xiao, C.Y.; Lu, X.; Zhang, H. A Surface Defect Detection Framework for Glass Bottle Bottom Using Visual Attention Model and Wavelet Transform. IEEE Trans. Ind. Inform. 2020, 16, 2189–2201. [Google Scholar] [CrossRef]
  11. Li, T.H.; Pang, X.H.; Jia, B.Y.; Xia, Y.W.; Zeng, S.M.; Liu, H.L.; Tian, H.; Lin, F.; Wang, D. Detection and Diagnosis of Defect in GIS Based on X-ray Digital Imaging Technology. Energies 2020, 13, 661. [Google Scholar] [CrossRef] [Green Version]
  12. Moon, J.S.; Yoo, M. Development of a Seismic Detection Technology for High-Speed Trains Using Signal Analysis Techniques. Sensors 2020, 20, 3708. [Google Scholar] [CrossRef] [PubMed]
  13. Gao, X.D.; Huan, Y.J.; Liu, G.Q. A kind of Laser Welding Defocus Amount Determining Device and Assay Method. C.N. Patent 201710768378, 31 August 2017. [Google Scholar]
  14. Bardin, F.; Cobo, A.; Collin, O.; López-Higuera, J.M.; Pascal, A.; Dubois, T.; Nylen, P.; Jonsson, P.; Högström, M.; Jones, J.D.C. Closed-loop power and focus control of laser welding for full-penetration monitoring. Appl. Optics. 2005, 44, 13–21. [Google Scholar] [CrossRef] [PubMed]
  15. Huang, S.H.; Huang, Y.J.; Hsieh, C.H.; Chen, H.Z.; Chui, H.C. Visual-assisted laser microwelding of carbon microfiber on metal plates. Opt. Laser Technol. 2018, 108, 368–371. [Google Scholar] [CrossRef]
  16. Zhang, Y.X.; Gao, X.D.; You, D.Y.; Katayama, S. Online monitoring of welding status based on a DBN model during laser welding. Engineering 2019, 5, 671–678. [Google Scholar] [CrossRef]
  17. Purtonen, T.; Kalliosaari, A.; Salminen, A. Monitoring and adaptive control of laser processes. Phys. Procedia 2014, 56, 1218–1231. [Google Scholar] [CrossRef] [Green Version]
  18. Shevchik, S.A.; Le, T.Q.; Farahani, F.V.; Neige, F.; Meylan, B.; Zanoli, S.; Wasmer, K. Laser Welding Quality Monitoring via Graph Support Vector Machine With Data Adaptive Kernel. IEEE Access 2019, 7, 93108–93122. [Google Scholar] [CrossRef]
  19. Krajewski, A.; Klekot, G.; Cybulak, M.; Kolodziejczak, P. A Novel Method of Supporting the Laser Welding Process with Mechanical Acoustic Vibrations. Materials 2020, 13, 4179. [Google Scholar] [CrossRef]
  20. Park, Y.W.; Park, H.; Rhee, S.; Kang, M. Real time estimation of CO2 laser weld quality for automotive industry. Opt. Laser Technol. 2002, 34, 135–142. [Google Scholar] [CrossRef]
  21. Zhang, Y.X.; Gao, X.D.; You, D.Y.; Zhang, N.F.; Gao, P.P. Welding defects detection based on deep learning with multiple optical sensors during disk laser welding of thick plates. J. Manuf. Syst. 2019, 51, 87–94. [Google Scholar] [CrossRef]
  22. Jager, M.; Humbert, S.; Hamprecht, F.A. Sputter tracking for the automatic monitoring of industrial laser-welding processes. IEEE Trans. Ind. Electron. 2008, 55, 2177–2184. [Google Scholar] [CrossRef]
  23. Hipp, D.; Mahrle, A.; Beyer, E.; Jäckel, S.; Hertel, M.; Füssel, U. Thermal Efficiency Analysis for Laser-Assisted Plasma Arc Welding of AISI 304 Stainless Steel. Materials 2019, 12, 1460. [Google Scholar] [CrossRef] [PubMed] [Green Version]
  24. Bélanger, E.; Bernier, M.; Faucher, D.; Vallée, R. High-power and widely tunable all-fiber Raman laser. J. Lightwave Technol. 2008, 26, 1696–1701. [Google Scholar] [CrossRef]
  25. Zhang, Y.X.; Gao, X.D.; You, D.Y.; Zhang, N.F. Data-driven detection of laser welding defects based on real-time spectrometer signals. IEEE Sens. J. 2019, 19, 9364–9373. [Google Scholar] [CrossRef]
  26. Deng, F.Q.; Huang, Y.S.; Lu, S.; Chen, Y.Y.; Chen, J.; Feng, H.; Zhang, J.M.; Yang, Y.; Hu, J.J.; Lam, T.L. A Multi-Sensor Data Fusion System for Laser Welding Process Monitoring. IEEE Access. 2020, 8, 147349–147357. [Google Scholar] [CrossRef]
  27. Lei, T.; Huang, Y.; Wang, H.; Rong, Y.M. Automatic weld seam tracking of tube-to-tubesheet TIG welding robot with multiple Sensors. J. Manuf. Process. 2021, 63, 60–69. [Google Scholar] [CrossRef]
  28. Gang, T.; Takahashi, Y.; Wu, L.X. Intelligent pattern recognition and diagnosis of ultrasonic inspection of welding defects based on neural network and information fusion. Sci. Technol. Weld. Join. 2002, 7, 314–320. [Google Scholar] [CrossRef]
  29. Ma, S.J.; Cai, W.; Liu, W.K.; Shang, Z.W.; Liu, G. A Lighted Deep Convolutional Neural Network Based Fault Diagnosis of Rotating Machinery. Sensors 2019, 19, 2381. [Google Scholar] [CrossRef] [Green Version]
  30. Cao, P.; Zhang, S.L.; Tang, J. Preprocessing-free gear fault diagnosis using small datasets with deep convolutional neural network-based transfer learning. IEEE Access. 2018, 6, 26241–26253. [Google Scholar] [CrossRef]
  31. Lu, Q.D.; Yang, R.; Zhong, M.Y.; Wang, Y.Q. An improved fault diagnosis method of rotating machinery using sensitive features and RLS-BP neural network. IEEE Trans. Instrum. Meas. 2019, 69, 1585–1593. [Google Scholar] [CrossRef]
  32. Yan, H.; Xu, Y.X.; Cai, F.Y.; Zhang, H.; Zhao, W.D.; Gerada, C. PWM-VSI Fault Diagnosis for a PMSM Drive Based on the Fuzzy Logic Approach. IEEE Trans. Power Electr. 2019, 34, 759–768. [Google Scholar] [CrossRef]
  33. Zhu, H.L.; Wang, H.Z.; Kang, D.H.; Zhang, L.Z.; Lu, L.X.; Yao, J.X.; Hu, Y. Study of joint temporal-spatial distribution of array output for large-scale photovoltaic plant and its fault diagnosis application. Sol. Energy. 2019, 181, 137–147. [Google Scholar] [CrossRef]
  34. Liu, Z.; Zhang, M.; Liu, F.; Zhang, B. Multidimensional Feature Fusion and Ensemble Learning-Based Fault Diagnosis for the Braking System of Heavy-Haul Train. IEEE Trans. Ind. Inform. 2021, 17, 41–51. [Google Scholar] [CrossRef]
  35. Wang, Z.R.; Huang, H.A.; Wang, Y.R. Fault diagnosis of planetary gearbox using multi-criteria feature selection and heterogeneous ensemble learning classification. Measurement 2021, 173, 108654. [Google Scholar] [CrossRef]
  36. Li, Z.C.; Tian, L.; Jiang, Q.C.; Yan, X.F. Fault Diagnostic Method Based on Deep Learning and Multimodel Feature Fusion for Complex Industrial Processes. Ind. Eng. Chem. Res. 2020, 59, 18061–18069. [Google Scholar] [CrossRef]
  37. Bal, P.R.; Kumar, S. WR-ELM: Weighted Regularization Extreme Learning Machine for Imbalance Learning in Software Fault Prediction. IEEE Trans. Reliab. 2020, 69, 1355–1375. [Google Scholar] [CrossRef]
  38. Goodfellow, I.J.; Pouget-Abadie, J.; Mirza, M.; Xu, B.; Warde-Farley, D.; Ozair, S.; Courville, A.; Bengio, Y. Generative adversarial networks. NIPS 2014, 3, 2672–2680. [Google Scholar] [CrossRef]
  39. Waheed, A.; Goyal, M.; Gupta, D.; Khanna, A.; Al-Turjman, F.; Pinheiro, P.R. CovidGAN: Data Augmentation Using Auxiliary Classifier GAN for Improved Covid-19 Detection. IEEE Access 2020, 8, 91916–91923. [Google Scholar] [CrossRef]
  40. Fahimi, F.; Dosen, S.; Ang, K.K.; Mrachacz-kersting, N.; Guan, C. Generative Adversarial Networks-Based Data Augmentation for Brain-Computer Interface. IEEE Trans. Neur. Net. Lear. 2020, 32, 4039–4051. [Google Scholar] [CrossRef]
  41. Guo, R.Y.; Liu, H.; Xie, G.; Zhang, Y.M. Weld Defect Detection From Imbalanced Radiographic Images Based on Contrast Enhancement Conditional Generative Adversarial Network and Transfer Learning. IEEE Sens. J. 2021, 21, 10844–10853. [Google Scholar] [CrossRef]
  42. Jiang, X.Y.; Ge, Z.Q. Data Augmentation Classifier for Imbalanced Fault Classification. IEEE Trans. Autom. Sci. Eng. 2021, 18, 1206–1217. [Google Scholar] [CrossRef]
  43. Odena, A.; Olah, C.; Shlens, J. Conditional Image Synthesis with Auxiliary Classifier GANs. In Proceedings of the 34th International Conference on Machine Learning, Sydney, Australia, 6–11 August 2017. [Google Scholar]
  44. Wu, Z.; Chen, H.; Lei, Y. Unidimensional ACGAN Applied to Link Establishment Behaviors Recognition of a Short-Wave Radio Station. Sensors 2020, 20, 4270. [Google Scholar] [CrossRef]
  45. Wang, Z.; Jing, J. Pixel-Wise Fabric Defect Detection by CNNs Without Labeled Training Data. IEEE Access 2020, 8, 161317–161325. [Google Scholar] [CrossRef]
  46. Yuan, C.; Chen, W.S.; Hao, H.; Kong, Q.Z. Near real-time bolt-loosening detection using mask and region-based convolutional neural network. Struct. Control Health Monit. 2021, 28, e2741. [Google Scholar] [CrossRef]
  47. Li, X.; Ling, S.H.; Su, S. A Hybrid Feature Selection and Extraction Methods for Sleep Apnea Detection Using Bio-Signals. Sensors 2020, 20, 4323. [Google Scholar] [CrossRef] [PubMed]
  48. Zeng, R.D.; Lu, Y.S.; Long, S.; Wang, C.; Bai, J.Y. Cardiotocography Signal Abnormality Classification Using Time-frequency Features and Ensemble Cost-sensitive SVM Classifier. Comput. Biol. Med. 2021, 130, 104218. [Google Scholar] [CrossRef]
  49. Webb, G.I.; Zheng, Z. Multistrategy ensemble learning: Reducing error by combining ensemble learning techniques. IEEE Trans. Knowl. Data En. 2004, 16, 980–991. [Google Scholar] [CrossRef] [Green Version]
  50. Wang, X.Z.; Xing, H.J.; Li, Y.; Hua, Q.; Dong, C.R.; Pedrycz, W. A Study on Relationship Between Generalization Abilities and Fuzziness of Base Classifiers in Ensemble Learning. IEEE Trans. Fuzzy Syst. 2015, 23, 1638–1654. [Google Scholar] [CrossRef]
Figure 1. Main Structure of ACGAN.
Figure 1. Main Structure of ACGAN.
Sensors 21 07304 g001
Figure 2. Structure diagram of data acquisition system.
Figure 2. Structure diagram of data acquisition system.
Sensors 21 07304 g002
Figure 4. ACGAN-SVM-CNN.
Figure 4. ACGAN-SVM-CNN.
Sensors 21 07304 g004
Figure 5. Integrated decision model.
Figure 5. Integrated decision model.
Sensors 21 07304 g005
Figure 6. Loss and accuracy curve of ACGAN-SVM-CNN.
Figure 6. Loss and accuracy curve of ACGAN-SVM-CNN.
Sensors 21 07304 g006
Figure 7. Confusion matrix of ACGAN-SVM-CNN integrated model for welding defect identification. (a) Confusion matrix of the classification result; (b) confusion probability matrix of the classification result.
Figure 7. Confusion matrix of ACGAN-SVM-CNN integrated model for welding defect identification. (a) Confusion matrix of the classification result; (b) confusion probability matrix of the classification result.
Sensors 21 07304 g007
Table 1. Server configuration information.
Table 1. Server configuration information.
ParametersSpecifications
RAM256 G
CPUIntel(R) Xeon(R) CPU E5-2680 v2 @2.80 GHZ
GPUGeForce GTX TITAN X
OSWindows 10
Table 2. Structure of the generator.
Table 2. Structure of the generator.
OperationKernelStridesFeature MapsBN?DropoutNonlinearity
input (128 × 512 × 1 × 1)
LinearN/AN/A128 × 256 × 1 × 1×0.0Leaky ReLu
LinearN/AN/A128 × 16,640 × 1 × 1×0.0N/A
Upsamplescale factor = 2128 × 128 × 130 × 4
Convolution3 × 21 × 1128 × 128 × 130 × 30.0Leaky ReLu
Convolution3 × 11 × 1128 × 64 × 130 × 30.0Leaky ReLu
Convolution3 × 11 × 1128 × 1 × 130 × 3×0.0Sigmoid
Table 3. Structure of the discriminator.
Table 3. Structure of the discriminator.
OperationKernelStridesFeature MapsBN?DropoutNonlinearity
input (1 × 130 × 3)
Convolution3 × 32128 × 16 × 65 × 2×0.2Leaky ReLu
Convolution3 × 32128 × 32 × 33 × 10.2Leaky ReLu
Convolution3 × 32128 × 32 × 17 × 10.2Leaky ReLu
Convolution3 × 32128 × 64 × 9 × 10.2Leaky ReLu
Convolution3 × 32128 × 128 × 5 × 10.2Leaky ReLu
Table 4. Performance metrics of ACGAN-SVM-CNN integrated model in each category.
Table 4. Performance metrics of ACGAN-SVM-CNN integrated model in each category.
ClassQua
(%)
Def3
(%)
Def-3
(%)
Defor
(%)
Cra
(%)
Rep
(%)
LoW
(%)
Dri
(%)
Tilt
(%)
W
(%)
Metrics
Precision100.095.8799.1799.2495.2097.8398.2999.2299.2299.16
Recall98.8095.8798.3597.7497.5499.2699.1499.2299.2298.33
F199.4095.8798.7598.4896.3598.5498.7199.2299.2298.74
Table 5. Training and testing time of different models.
Table 5. Training and testing time of different models.
Training
(min)
Testing
(ms/Sample)
ACGAN-SVM-CNN21.610.76
ACGAN-CNN16.190.71
CNN6.810.69
SVM4.531.78
ADABOOST5.212.26
Table 6. Performance of different models under different amount of training data.
Table 6. Performance of different models under different amount of training data.
100%90%80%75%70%
ACGAN-SVM-CNN98.37%98.02%96.78%92.02%91.15%
ACGAN-CNN97.86%97.33%96.62%92.18%90.04%
CNN96.83%96.17%95.82%86.75%86.33%
ACGAN discriminator85.13%76.54%64.85%60.37%53.26%
SVM83.35%81.47%80.26%65.31%64.25%
ADABOOST81.65%80.25%71.35%66.27%63.93%
Publisher’s Note: MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Share and Cite

MDPI and ACS Style

Fan, K.; Peng, P.; Zhou, H.; Wang, L.; Guo, Z. Real-Time High-Performance Laser Welding Defect Detection by Combining ACGAN-Based Data Enhancement and Multi-Model Fusion. Sensors 2021, 21, 7304. https://doi.org/10.3390/s21217304

AMA Style

Fan K, Peng P, Zhou H, Wang L, Guo Z. Real-Time High-Performance Laser Welding Defect Detection by Combining ACGAN-Based Data Enhancement and Multi-Model Fusion. Sensors. 2021; 21(21):7304. https://doi.org/10.3390/s21217304

Chicago/Turabian Style

Fan, Kui, Peng Peng, Hongping Zhou, Lulu Wang, and Zhongyi Guo. 2021. "Real-Time High-Performance Laser Welding Defect Detection by Combining ACGAN-Based Data Enhancement and Multi-Model Fusion" Sensors 21, no. 21: 7304. https://doi.org/10.3390/s21217304

Note that from the first issue of 2016, this journal uses article numbers instead of page numbers. See further details here.

Article Metrics

Back to TopTop