Next Article in Journal
Empirical Mode Decomposition as a Novel Approach to Study Heart Rate Variability in Congestive Heart Failure Assessment
Next Article in Special Issue
Application of Continuous Wavelet Transform and Convolutional Neural Network in Decoding Motor Imagery Brain-Computer Interface
Previous Article in Journal
Slope Entropy: A New Time Series Complexity Estimator Based on Both Symbolic Patterns and Amplitude Information
Previous Article in Special Issue
Invariant Image-Based Currency Denomination Recognition Using Local Entropy and Range Filters
 
 
Font Type:
Arial Georgia Verdana
Font Size:
Aa Aa Aa
Line Spacing:
Column Width:
Background:
Article

Using Entropy for Welds Segmentation and Evaluation

Faculty of Electrical Engineering and Information Technology, Slovak University of Technology in Bratislava, 841 04 Bratislava, Slovakia
*
Author to whom correspondence should be addressed.
Entropy 2019, 21(12), 1168; https://doi.org/10.3390/e21121168
Submission received: 24 October 2019 / Revised: 23 November 2019 / Accepted: 26 November 2019 / Published: 28 November 2019
(This article belongs to the Special Issue Entropy in Image Analysis II)

Abstract

:
In this paper, a methodology based on weld segmentation using entropy and evaluation by conventional and convolution neural networks to evaluate quality of welds is developed. Compared to conventional neural networks, there is no use of image preprocessing (weld segmentation based on entropy) or data representation for the convolution neural networks in our experiments. The experiments are performed on 6422 weld image samples and the performance results of both types of neural network are compared to the conventional methods. In all experiments, neural networks implemented and trained using the proposed approach delivered excellent results with a success rate of nearly 100%. The best results were achieved using convolution neural networks which provided excellent results and with almost no pre-processing of image data required.

1. Introduction

The Fourth Industrial Revolution (Industry 4.0) has opened space for research and development of new manufacturing methods, systems and equipment based on innovations such as computing intelligence, autonomous robots, big data, augmented reality, process simulation, quality management systems, etc. [1].
Weld evaluation is very important quality control process in many manufacturing processes. Without this technological process, it would be almost impossible to produce welded constructions with current efficiency—whether we are talking about time, price, or material consumption. It is therefore necessary to welds be inspected to meet the specified quality level. In order to detect the possible presence of different weld defects, proper sensing, monitoring and inspection methods are necessary for quality control. Very effective and non-destructive method for weld evaluation is visual inspection. Inspection process using this method can be in certain level automated and done by computer systems [2,3].
Visual inspection of a weld is an important non-destructive method for weld quality diagnostics that enables to check welded joint and its various parameters. This examination is carried out as a first examination and able to detect various defects [4].
In this paper, we focus on indirect visual evaluation due to which the evaluation process can be automated. Indirect inspection can be applied also in places that are not directly accessible, for example the inner surface of a pipeline, the interior of pressure vessels, car body cavities etc. It also eliminates errors of human judgment and removes errors caused by workers for such reasons as e.g., fatigue, inattention or lack of experience.
The improved beamlet transformation for weld toe detection described in [5,6] considers images which are corrupted by noise. The authors aim at detecting edge borders of welds. The dynamic thresholding is performed in one of the beamlet algorithm steps. The algorithm predicts the directional characteristics of the weld allows to filtrate unsuitable edges. Using this method, it is possible to directly extract weld seam edges from highly noisy welding images without any pre-processing or post-processing steps.
In [7], the authors work with pipeline weld images with a very low contrast and corrupted by noise; this causes problems to conventional edge detectors. At first, the image is noise-filtered using a morphological operation of opening and closing. Next, the improved algorithm of fuzzy edge detection is applied. Multi-level fuzzy image improvement is based on interactive searching of optimal threshold level and multi-directional edge detector which convolution kernel is 5 × 5 with 8 directions based on gradient searching. The result of the algorithm is compared with detectors as Sobel, canny FED and fast FED.
Edge detection and histogram projection are used in [8], where histogram projections of tested welds are compared with a specified similarity threshold used to evaluate quality of the tested welds. The loaded image pattern has the same specifications (width and position) as the tested image. Always one vertical line from the pattern and the tested images is compared. Line histograms of pattern and tested images are computed, the correlation degree of two histograms is computed using the Tukey HSD difference. A lower correlation degree than the specified correlation threshold indicates edge defects in this part of the examined image. The procedure is repeated over the entire width of image.
Evaluation of metal cans welds is dealt with in [9]. Can’s weld defects may not be directly related to welding (they can be brought about by rest of glue, dust, etc.). Therefore, authors use probability evaluation of two evaluation methods; the Column Gray-Level Accumulation Inspection represents histogram projection in general. The histogram projections of the pattern and the tested weld are compared. The comparison of first derivation for making better results is also performed. This method can detect defects of wider surface. The overall evaluation is done using Dampster-Shafer theory of evidence.
In another work [10], the above authors deal with edge detection based on pixel intensity difference of the foreground and the background. The background pixels’ intensity occurs with a maximum probability and the distribution of the background pixels fits the Gauss distribution.
The weld visual inspection process performed through image processing on the image sequence to improve data accuracy is presented in [11]. The Convolution Neural Network (CNN) as an image processing technique can determine the feature automatically to classify the variation of each weld defect pattern. A classification using CNN consists of two stages: image extraction using image convolution, and image classification using neural network. The proposed evaluation system has obtained classification for four different types of weld defects with validation accuracy of 95.83%.
A technique for automatic endpoint detection of weld seam removal in a robotic abrasive belt grinding process using a vision system based on deep learning is demonstrated in [12]. The paper presents results of the first investigative stage of semantic segmentation of weld seam removal states using encoder-decoder convolutional neural networks (EDCNN). The prediction system based on semantic segmentation is able to monitor weld profile geometry evolution taking into account the varying belt grinding parameters during machining which allows further process optimization.
Utilizing computing intelligent using support vector machine (SVM) is presented in [13,14]. Authors developed real-time monitoring system to automatically evaluate the welding quality during high-power disk laser welding. Fifteen features were extracted from images of laser-induced metal vapor during welding. To detect the optimal feature subset for SVM, a feature selection method based on the SFFS algorithm was applied. An accuracy of 98.11% by 10-fold cross validation was achieved for the SVM classifier generated by the ten selected features. The authors declare the method has the potential to be applied in the real-time monitoring of high-power laser welding.
The authors of [15,16,17,18] deal with the development of a system for automatic weld evaluation using new information technologies based on cloud computing and single-board computer in the context of Industry 4.0. The proposed approach is based on using a visual system for weld recognition, and a neural network cloud computing for real-time weld evaluation, both implemented on a single-board low-cost computer. The proposed evaluation system was successfully verified on welding samples corresponding to a real welding process. The system considerably contributes to the weld diagnostics in industrial processes of small- and medium-sized enterprises. In [18], the same authors use a single-board computer able to communicate with an Android smartphone which is a very good interface for a worker or his shift manager. The basic result of this paper is a proposal of a weld quality evaluation system that consists of a single-board computer in combination with Android smartphone.
This paper deals with development of a software system for visual weld quality evaluation based on weld segmentation using entropy and evaluation by conventional and convolution neural networks. The evaluation of the performance results is compared to the conventional methods (weld segmentation based on entropy and evaluation using conventional neural networks with and without weld segmentation). Most experiments of proposed method apply on weld metal, however, one experiment with convolution neural networks applies also on weld adjected zones. 6422 real and adjusted laboratory samples of welds are used for experiments. The paper is organized in five sections: Section 2 deals with preparation of input data for the neural network. Section 3 describes configuration of used neural networks and their training process. In Section 4 the results of experiments are presented. In Section 5 we discuss the results.

2. Preparation of Input Data for the Neural Network

The input data for the proposed diagnostic system were represented in the form of grayscale laboratory samples of metal sheet welds in JPEG format. The samples were pre-classified as OK (correct) and NOK (incorrect) (Figure 1 and Figure 2). Defective weld samples (NOK) include samples of various surface defects such as irregular weld bead, excess weld metal, craters, undercut, etc. Welds images are captured under the same illumination and have the same resolution 263 × 300 pixels. The total number of evaluated sample images was 6422.
However, for several reasons the image resolution 263 × 300 pixels is not suitable for a conventional neural network due to the necessity of large amount of allocated memory (about gigabytes for thousands of frames even in a relatively low resolution) and time-consuming network training time.
Several suitable options for data processing that eliminate the above problems are presented next. At first, the background weld segmentation is described. Segmentation provides two outputs - the weld mask and the segmented weld itself. Three transformations of the weld mask into a one-dimensional feature vector are described further. Feature vectors are useful as inputs for the multilayer perceptron (MLP)/radial basis function (RBF) neural networks. Finally, the size of the segmented/unsegmented weld image is reduced when applied in the conventional neural network (if CNN is applied, no size reduction is needed).

2.1. Weld Segmentation

The sample images depict the weld itself and the background—metal sheet. The background does not affect the evaluation of the weld and is masked from the images by the proposed algorithm. The simplified flowchart of the algorithm is shown in Figure 3.
After reading the images, local entropy of each pixel is computed according to [19]:
i = 1 K j = 1 K p i j log 2 p i j ,
where p i j represents the probability function for the pixel [ i , j ] .
This value contains information about the complexity/unevenness around the pixel. The neighbourhood radius was set to 8 pixels. To compute the entropy, the filters.rank.entropy function from the Python library scikit-image was used. The resulting local entropy matrix effectively finds the edges and texture complexity in the image. The results of filtering can be seen in Figure 4.
As the entropy resolution values were too detailed for our application, the blur filtering was applied. The anisotropic blur filter from the imager library was implemented, which removes noise/unimportant details while preserving edges better than other types of blur filters. The blur filter with an amplitude of 250 was applied (Figure 5).
The next step is thresholding. In the image matrix, the value 1 (white) represents weld pixels, the value 0 (black) represents background. Thresholding was implemented using the function threshold from the imager library. The optimal threshold value was computed automatically using the kmeans method (Figure 6).
The thresholding result may have some imperfections—small blobs and unfilled areas. Unfilled areas are removed using the inverted output of the function bucketfill (imager library). It is applied on the background of the weld and it finds all pixels of the background. The remaining the pixels are filled with value 1 (white) (Figure 7a).
Very small blobs were removed using the function clean (imager library). This function reduces objects size using morphological erosion, and then increases it. This causes, that very small objects are removed and the shape of larger object is simplified (Figure 7b).
However, larger blobs were not removed in the previous step. To find the largest object in the image, the function split_connected (imager library) was used (Figure 8).
The segmentation result—the mask and the masked weld can be seen in Figure 9.

2.2. Vector of Sums of Subfields in the Mask

The first representation of the mask is a vector which entries are sums of subfields. For input images of resolution 263 × 300 pixels, was selected a subfield of 50 × 50 pixels, which corresponds to 36 values. The function for vector calculation is shown in the Algorithm 1.
The function ceiling rounds a number to the next higher integer. Using division of the index ( i , j ) by the size of the subfield, and subsequently the function ceiling, we obtained i n d I / i n d J for the selected index i / j . The function as.vector retypes the resulting two-dimensional array into a vector by writing the matrix elements column-wise into a vector. Example of retyping can be understood from Figure 10 and Figure 11.
Graphs for OK and NOK welds (Figure 12) can be compared in Figure 13: the OK mask graph has every third value (representing the subfields in the image center) maximal. Values of the NOK weld graph are distributed into more columns and the values do not achieve maximum values. The main drawback of this representation is that it can be used only for images with the same size. The benefit is a multiple reduction of input data (number of mask pixels in our case has been reduced 502-times).
Algorithm 1. Computing of subfields sums of the mask
procedure MaskToSums(img, size)
  xLen ←length(img[ ,1])
  yLen ←length(img[1, ])
  nRows ← ceiling(xLen/size)
  nCols ← ceiling(yLen/size)
  res ← matrix(0, nRows, nCols)
  for i in 1:xLen do
    for j in 1:yLen do
     if img[i,j] == TRUE then
       indI ← ceiling(i/size)
       indJ ← ceiling(j/size)
       res[indI, indJ] ++
     end if
    end for
  end for
  return as.vector(res)
end procedure

2.3. Histogram Projection of the Mask

A histogram projection is a vector containing sums of columns and rows of the input image matrix (Figure 14). In the case of an image mask, these are amounts representing numbers of white pixels. Thus, the length of the vector corresponds to the vector of the height and width of the image.
In the graphs (Figure 15 and Figure 16) showing the histogram projection of the mask, the difference between correct and wrong welds is visible. The projection of the correct weld mask is more even, the sums by columns have an even increase and slope, and the sums per line have small variations. On the other hand, the histogram projection of the wrong weld mask has a lot of irregularities. The disadvantage of this representation consists in that it cannot be used for input images of different resolutions. The resulting projection vector is much larger than other representations. The advantage is easy implementation and calculation.

2.4. Vector of Polar Coordinates of the Mask Boundary

A next representation of a weld mask in this paper is the vector of polar coordinates of the mask boundary. To transform weld masks, an algorithm has been proposed and implemented. Its main steps are described below.
The first step is to find the x ,   y coordinates of the mask boundary using the function boundary (imager library). Then, coordinates of the center of the object [ c x ,   c y ] are calculated according to:
c x = m a x ( x ) m i n ( x ) 2 + m i n ( x ) ,
c y = m a x ( y ) m i n ( y ) 2 + m i n ( y ) ,
In the next step, the position of the object is normalized (the center is moved to the position [ 0 , 0 ] ) according to the found coordinates. Then, for each boundary point, the coordinates are converted from Cartesian to polar [ r , α ] (i.e., distance from center, angle). According to the Pythagorean theorem, the distance is calculated as follows:
r = x 2 + x 2 ,
Calculation of the angle is realized by Algorithm 2:
Algorithm 2. Calculation of angle from Cartesian coordinates
procedure Angle(x, y)
  z ← x + 1i * y
  a ← 90 - arg(z) / π * 180
  return round(a mod 360)
end procedure
If the resulting number of coordinates is less than 360, the missing angle values are completed and the corresponding distances are calculated from the surrounding values by linear interpolation using the na_approx function (zoo library). The result is a vector with 360 elements, which indices correspond to the angle values in degrees, and the value is the distance r. The resulting graphs of OK and NOK weld masks (Figure 17) are in Figure 18 and Figure 19.
The representation in the form of polar coordinates for the OK weld visibly differs from the NOK one. The big jumps and variations on the graph are caused by large irregularities in the weld shape. The advantage of such representation is that it can be used for any input mask resolution. The disadvantage is a complicated calculation. Generally, mask representations contain information only about the shape of the weld, which can be considered as a disadvantage because texture information is important input data for the neural network.

2.5. Data Preparation for Neural Network

Weld images and feature vectors were stored in two data structures of type list. The first list represented welds classified as NOK (incorrect); the second list welds classified as OK (correct). For neural networks, it was necessary to combine data, i.e., to transform and randomly mix them. For MLP and RBF networks, each input vector has to have assigned a classification value 0 (incorrect) or 1 (correct). Then, the vectors were merged together and with randomly mixed elements. Next, the L2-normalization was applied to the data. Finally, 85% of training and 15% of test samples were selected randomly. For convolution neural networks, the images were 5-times reduced, then the data type was converted to a three-dimensional array data structure. In the arrays, the dimensions were transposed to represent to correspond to the following structure: [ n u m b e r   o f   i m a g e s l e n g t h h e i g h t ] . The vector of zeros with the same length as the first dimension corresponded to the first array (array of NOK welds). The vector of ones corresponded to the second array (array of OK welds). The arrays and vectors were merged into a common list and their elements were mixed randomly. Then, 77% of training samples, 15% of test samples and 8% of validation samples were selected.

3. Configuration and Training of Neural Networks

Several neural network architectures were configured for comparison and testing. Their parameters were changed during the experiments and the experiment results were compared and evaluated. Both RBF and MLP networks were configured in The Stuttgart Neural Network Simulator for R language - RSNNS library, the MLP networks were configured in the Keras library, and the convolution networks were configured in the Keras and the MXNet libraries.

3.1. RBF Network

To implement the RBF network, the RSNNS library was chosen (just in this one the RBF network template is available). Three RBF networks were configured using the function rbf (RSNN library). The set parameters were the number of units in the hidden layer and the number of epochs, the initial parameters had default values. The best configurations were chosen experimentally. Configuration details are in Figure 20, Figure 21 and Figure 22.

3.2. MLP Network

Experiments with training and testing of MLP networks showed, that a one-layer architecture is sufficient for our data representation. The performance of the network was very good and the difference from multiple hidden layers was negligible. To keep the objectivity, MLP networks had the same configuration in both libraries. The sigmoid activation function and the randomize weights initialization functions were used. For the NN training, the error backpropagation algorithm with learning parameter 0,1 was used.
The implementation in the RSNNS library uses the mlp function for configuration and training. Configuration details are in Figure 23, Figure 24 and Figure 25.
The implementation of the MLP network in the Keras library required a detailed list of layers in the code. Two layer_dense layers were used; the first one defines the hidden layer with the ReLU activation function, and the second one defines the output layer with the size 2 (two output categories) using the softmax activation function (Figure 26).

3.3. Convolution Neural Network

For an objective comparison of the Keras and MXNet libraries, the same convolution network architecture in both libraries was used at first, however in the MXNet library, training such a neural network was too slow. Thus, we designed our own architecture with a better learning time performance. The discussion about the results is provided in the next Section 4.
The architecture of the convolution network 1 is shown in Figure 27 and visualized in Figure 28. The architecture includes a list of all layers and the size of output structures for both NN. Two pairs of convolution and pooling layers were used, the convolution being applied twice before the first pooling layer. The input image size was 56 × 60. The number of convolution filters was 32 at the beginning, in further convolution filters it rose to 64. A dropout was used between some layers to prevent overtraining of the neural network by deactivating a certain percentage of randomly selected neurons. At the end, the flatten layer was used to convert the resulting structure into a one-dimensional vector used as an input for a simple MLP network with one hidden layer containing 256 neurons.
Parameters of individual layers are shown in the diagram in Figure 28. For example, the convolution layer (red) contains a list of 3 × 3 - filter size, 3 × 3 - stride, 32 - number of filters.
The architecture of the convolution network 2 is visualized in Figure 29. Two pairs of convolution and pooling layers were used, however in this case a double convolution occurs only in the second layer. There is also a difference in the design of the convolution, where the parameter stride (step of the filter) is 3,3. Dropout was used only in two places.

4. Results

This chapter presents results of code profiling, weld segmentation and evaluation of neural networks.

4.1. Code Profiling

Profiling was done using the profvis library at the level of the code line. The output is an interactive visualization using memory listing in MB and computing time in ms for each code line. The example can be seen in Figure 30.
Profiling was performed on a desktop computer with parameters listed in Table 1 (the graphic card was not used).

4.2. Results of Data Preparation and Segmentation

Segmentation was successful for all tested weld samples. For some NOK defective welds which consisted of several parts or contained droplets, only the largest continuous weld surface was segmented, which was considered to be a correct segmentation for proposed methodology. Segmentation examples are shown in Figure 31.
The segmentation time is an important indicator in comparison of results. Results of profiling different parts of the segmentation process can be seen in Figure 32. Code profiling was carried out using a computer with the technical specification shown in Table 1.
Segmentation was performed by concatenating the outputs from functions load.image, grayscale, entropyFilter, createMask, and segmentWeld. Almost all functions in this section of the program were performed very quickly (within 30 ms) except for the entropyFilter function, which took an average of 158 ms to be completed. This function is the most important part of the segmentation algorithm; the time was acceptable. The average time to complete the whole segmentation was 194 ms. The average amount of memory allocated was 74.76 MB. For MLP and RBF networks, the next step was to transform masks into feature vectors. The profiling results of functions performing three types of transformations can be seen in Figure 33.
The results show that these functions are optimal, taking up minimal memory and time. The mean values for computing the vector of sums of subfields in the mask are 16 ms and 0.1 MB; for the histogram projection vector, it is less than 10 ms and less than 0.1 MB (estimation of profiling tool, real values are immeasurably small). Values for the polar coordinates vector are 18 ms and 7.56 MB. Presented results are also shown in Table 2.

4.3. Criteria for Evaluation of Neural Network Results

As the main criterion for results evaluation the confusion matrix was chosen. The main diagonal of the confusion matrix contains the numbers of correctly classified samples, the antidiagonal contains the numbers of incorrectly classified samples; the smaller values in the antidiagonal, the more successful the prediction model. In a binary classification this matrix contains four values (Figure 34): TP—true positive; FP—false positive; FN—false negative; TN—true negative.
The accuracy was computed from the confusion matrix and is expressed as the ratio of correctly classified samples to all samples, see Equation (5) [20].
A c c u r a c y = T P + T N a l l   s a m p l e s ,
Accuracy is an objective criterion only if the FN and FP values are similar.
A more objective criterion for comparing results is the F-score. The F-score is calculated as the harmonic average of the precision and the recall (sensitivity) values [20], the best score corresponds to F-score = 1:
P r e c i s i o n = T P T P + F P ,
R e c a l l = T P T P + F N ,
F - s c o r e = 2 R e c a l l P r e c i s i o n T P + F N   R e c a l l + P r e c i s i o n ,
To visualize the success of neural network classification, the ROC (Receiver operating characteristics) curve was chosen. It shows the recall (sensitivity) value depending on the value 1-specificity at the variable threshold [20] (Figure 35):
S p e c i f i c i t y = T N T N + F P ,
The ROC curve for the best possible classifier is rectangular with the vertex [0,1].

4.4. Results of Neural Network Classificaton

We configured and tested neural networks for all data representations (in total 15 experiments). For a better clarity, the experiments results are labelled using labels from Table 3.
The first tests were carried out for RBF and MLP networks with input data formats according to Table 3. Resulting confusion matrices for RBF networks are as follows:
r b f r s n s u m 01 = [ 502 14 15 433 ] , r b f r s n h p r 02 =   [ 434 0 83 447 ] , r b f r s n p o l 03 = [ 435 0 82 447 ] ,
From the matrices (10) it is evident that the RBF network performed bad when classifying NOK welds—they are often classified as OK. ROC curves of trained RBF networks are depicted in Figure 36.
ROC curves for MLP networks are depicted Figure 37 and Resulting confusion matrices are as follows:
m l p r s n s u m 04 = [ 516 1 1 446 ] , m l p r s n h p r 05 = [ 5017 0 0 447 ] , m l p r s n p o l 06 = [ 514 1 3 446 ] ,
m l p k e r s u m 07 = [ 517 15 17 446 ] , m l p k e r h p r 08 = [ 511 2 23 459 ] , m l p k e r p o l 09 = [ 522 13 12 448 ] ,
The results show that the MLP implementation in the RSNNS library was more successful compared with the Keras library. The networks had no problem to classify correct (OK) or incorrect (NOK) welds. FP and FN values were approximately similar. The resulting calculated accuracy and F-scores shown in Table 4 describe the performance of the trained neural networks.
The results show that MLP networks are much more successful. Using default RBF initialization weights the RBF network less successful. From a practical point of view, MLP networks are more suitable for weld evaluation.
It was hard to compare the results for MLP networks, they provided similar results for all data representations. The RBF network achieved significantly better results in the vector of sums of subfields in the mask data representation.
It was found out, that using the same network configuration in the two libraries yields slightly different results. The implementation in the RSNNS library was almost 100% successful and therefore it was considered as the best candidate for practical use.
Training profiling for RSNN library was done next. Although training in the Keras library allocated less memory, the training time was several times longer than in case of the RSNNS library. Using vector of sums of subfields in the mask, the MLP network training time in RSNNS took less than one second, while using the Keras library was tens of seconds. The list of training profiling results is shown in Table 5.
Comparison of convolution neural nets was again based on the confusion matrices, ROC curves, accuracy and F-scores. The input of the networks were just images of welds without any filtration and masked welds without background (black background). Confusion matrices are as follows:
c n n k e r o r i 10 = [ 534 1 0 460 ] , c n n m x n o r i 12 =   [ 559 8 1 431 ] , c n n m x n o r i 14 = [ 498 0 0 460 ] ,
c n n k e r s e g 11 = [ 534 0 0 461 ] ,   c n n m x n s e g 13 =   [ 558 0 2 439 ] , c n n m x n s e g 15 = [ 498 0 0 460 ] ,
Classification error in convolution neural networks was minimal, therefore the ROC curve was evaluated as ideal for all experiments with indistinguishable differences. For all neural nets, the ROC curve was the same (Figure 38).
The resulting accuracy and F-scores along with the number of epochs needed to train the networks are listed in Table 6.
For convolution networks, changes of accuracy after each epoch for both training (blue line) and validation data (green line) are shown in Figure 39. The charts show that training with non-segmented weld images started at a lower accuracy and the learning was slower (Figure 40).
The progress of training for the Keras library was more uniform, without steps. The graphs can be seen in Figure 41 and Figure 42.
The success rate for all networks was higher than 99%. The decisive factor for comparison were the code profiling results shown in Table 7.
It can be concluded, that the network with the architecture shown in Figure 29 in Section 3.3 implemented using the MXNet library was the fastest. With a training time 12.170 ms and a 100% success also for non-segmented data it is considered the best choice for practical use.
Although the MLP network (mlp-rsn-sum04) was similarly successful and several times faster in training, the preparation of the representation in the form of the vector of sums of subfields in the mask took considerably more time. The number of training samples was approximately 5400, the average time to obtain a mask of one sample was 164 ms, and the vector calculation was 16 ms, in total 972 ms.

4.5. Profiling Single Weld Diagnostics

In practice, neural network training is not a frequent process. Usually, the network is trained once and then implemented for prediction. Therefore, at the end we decided to evaluate the prediction of one weld for the most successful models. The provided results represent the average of five independent tests. The list can be seen in Table 8 along with the average image preparation time and memory required to prepare the weld input image for the specific diagnostic model.
The diagnostic profiling results confirmed that the best solution was the classification of the weld using the convolution net with the architecture shown in Figure 29 in Section 3.3. The average image loading time and its 5× reduction took only 14 ms on average, and evaluation time was 14 ms.

5. Discussion

The aim of this paper was to develop a neural network based methodology to evaluate quality of welds. Several types of neural networks implemented in several software libraries were compared with respect to performance. It was necessary to prepare the data (images of welds) into a format suitable for neural network processing. For some types of networks (convolution) the input data preparation was minimal (segmentation or no segmentation), while for other networks (MLP, RBF), a sophisticated data preprocessing was required (filtering, equalizing and segmenting the image based on entropy). Each library required its own input data format which also had to be taken into account during programming. The main result of the paper is confirmation, that the convolutional neural networks can be used for weld quality evaluation without using image preprocessing and in case of using no segmentation, they can be used for evaluation not only weld metal but also adjected zones.
Neural networks were configured experimentally to achieve the best performance and the obtained results were compared. In all cases, neural networks implemented and trained using the proposed approach delivered excellent results with a success rate of nearly 100%. Thus, we can recommend any of the tested libraries to solve the weld quality evaluation problem. The best results were achieved using convolution neural networks which provided excellent results and with almost no pre-processing of image data required. The longer training time of these networks is acceptable in practical usage.
In summary, based on achieved experimental results, convolution neural networks have shown to be a promising approach for weld evaluation and will be applied in the future research dealing with evaluation of images in the real welding processes. The convolutional neural networks can be used for weld quality evaluation without using image preprocessing.

Author Contributions

O.H. proposed the idea in this paper and prepared data; O.H., E.K., P.D. and J.C. designed the experiments, E.K. and P.D. performed the experiments; O.H., and E.K. analyzed the data; O.H. wrote the paper; E.K., P.D. and J.C. edited and reviewed the paper; All authors read and approved the final manuscript.

Funding

This research was supported by the Slovak research and Development Agency under the contract no. APVV-17-0190, by the Scientific Grant Agency of the Ministry of Education, Science, Research and Sport of the Slovak Republic under the grant VEGA 1/0819/17, and by the Cultural and Educational Grant Agency of the Ministry of Education, Science, Research and Sport of the Slovak Republic, KEGA 038STU-4/2018.

Acknowledgments

We would like to thank to Alena Kostuňová for helping with programming the implementation.

Conflicts of Interest

The authors declare no conflict of interest.

References

  1. Akşit, M. The Role of Computer Science and Software Technology in Organizing Universities for Industry 4.0 and beyond. In Proceedings of the 2018 Federated Conference on Computer Science and Information Systems, FedCSIS 2018, Poznań, Poland, 9–12 September 2018. [Google Scholar]
  2. Dahal, S.; Kim, T.; Ahn, K. Indirect prediction of welding fume diffusion inside a room using computational fluid dynamics. Atmosphere 2016, 7, 74. [Google Scholar] [CrossRef]
  3. Huang, W.; Kovacevic, R. A laser-based vision system for weld quality inspection. Sensors 2011, 11, 506–521. [Google Scholar] [CrossRef] [PubMed]
  4. Noruk, J. Visual weld inspection enters the new millennium. Sens. Rev. 2001, 21, 278–282. [Google Scholar] [CrossRef]
  5. Deng, S.; Jiang, L.; Jiao, X.; Xue, L.; Deng, X. Image processing of weld seam based on beamlet transform. Hanjie Xuebao/Trans. China Weld. Inst. 2009, 30, 68–72. [Google Scholar]
  6. Deng, S.; Jiang, L.; Jiao, X.; Xue, L.; Cao, Y. Weld seam edge extraction algorithm based on Beamlet Transform. In Proceedings of the 1st International Congress on Image and Signal Processing, CISP 2008, Hainan, China, 27–30 May 2008. [Google Scholar]
  7. Zhang, X.; Yin, Z.; Xiong, Y. Edge detection of the low contrast welded joint image corrupted by noise. In Proceedings of the 8th International Conference on Electronic Measurement and Instruments, ICEMI 2007, Xi’an, China, 16–18 August 2007. [Google Scholar]
  8. Hou, X.; Liu, H. Welding image edge detection and identification research based on canny operator. In Proceedings of the 2012 International Conference on Computer Science and Service Systems, CSSS 2012, Nanjing, China, 11–13 August 2012. [Google Scholar]
  9. Shen, Z.; Sun, J. Welding seam defect detection for canisters based on computer vision. In Proceedings of the 6th International Congress on Image and Signal Processing, CISP 2013, Hangzhou, China, 16–18 December 2013. [Google Scholar]
  10. Liao, Z.; Sun, J. Image segmentation in weld defect detection based on modified background subtraction. In Proceedings of the 6th International Congress on Image and Signal Processing, CISP 2013, Hangzhou, China, 16–18 December 2013. [Google Scholar]
  11. Khumaidi, A.; Yuniarno, E.M.; Purnomo, M.H. Welding defect classification based on convolution neural network (CNN) and Gaussian Kernel. In Proceedings of the 2017 International Seminar on Intelligent Technology and Its Application: Strengthening the Link between University Research and Industry to Support ASEAN Energy Sector, ISITIA 2017, Surabaya, Indonesia, 28–29 August 2017. [Google Scholar]
  12. Pandiyan, V.; Murugan, P.; Tjahjowidodo, T.; Caesarendra, W.; Manyar, O.M.; Then, D.J.H. In-process virtual verification of weld seam removal in robotic abrasive belt grinding process using deep learning. Robot. Comput. Integr. Manuf. 2019, 57, 477–487. [Google Scholar] [CrossRef]
  13. Chen, J.; Wang, T.; Gao, X.; Wei, L. Real-time monitoring of high-power disk laser welding based on support vector machine. Comput. Ind. 2018, 94, 75–81. [Google Scholar] [CrossRef]
  14. Wang, T.; Chen, J.; Gao, X.; Qin, Y. Real-time monitoring for disk laser welding based on feature selection and SVM. Appl. Sci. 2017, 7, 884. [Google Scholar] [CrossRef]
  15. Haffner, O.; Kucera, E.; Kozak, S.; Stark, E. Proposal of system for automatic weld evaluation. In Proceedings of the 21st International Conference on Process Control, PC 2017, Štrbské Pleso, Slovakia, 6–9 June 2017. [Google Scholar]
  16. Haffner, O.; Kučera, E.; Kozák, Š. Weld segmentation for diagnostic and evaluation method. In Proceedings of the 2016 Cybernetics and Informatics, K and I 2016—Proceedings of the the 28th International Conference, Levoca, Slovakia, 2–5 February 2016. [Google Scholar]
  17. Haffner, O.; Kučera, E.; Kozák, Š.; Stark, E. Application of Pattern Recognition for a Welding Process. In Proceedings of the Communiation Papers of the 2017 Federated Conference on Computer Science and Information Systems, FedCSIS 2017, Prague, Czech Republic, 3–6 September 2017. [Google Scholar]
  18. Haffner, O.; Kučera, E.; Bachurikova, M. Proposal of weld inspection system with single-board computer and Android smartphone. In Proceedings of the 2016 Cybernetics and Informatics, K and I 2016—Proceedings of the the 28th International Conference, Levoca, Slovakia, 2–5 February 2016. [Google Scholar]
  19. Gajowniczek, K.; Ząbkowski, T.; Orłowski, A. Comparison of decision trees with Rényi and Tsallis entropy applied for imbalanced churn dataset. In Proceedings of the 2015 Federated Conference on Computer Science and Information Systems, FedCSIS 2015, Łódź, Poland, 13–16 September 2015. [Google Scholar]
  20. Sokolova, M.; Lapalme, G. A systematic analysis of performance measures for classification tasks. Inf. Process. Manag. 2009, 45, 427–437. [Google Scholar] [CrossRef]
Figure 1. Laboratory sample of an OK weld.
Figure 1. Laboratory sample of an OK weld.
Entropy 21 01168 g001
Figure 2. Laboratory sample of NOK weld.
Figure 2. Laboratory sample of NOK weld.
Entropy 21 01168 g002
Figure 3. The simplified flowchart of the segmentation algorithm.
Figure 3. The simplified flowchart of the segmentation algorithm.
Entropy 21 01168 g003
Figure 4. Step 1—local entropy computing.
Figure 4. Step 1—local entropy computing.
Entropy 21 01168 g004
Figure 5. Step 2—blur filtering.
Figure 5. Step 2—blur filtering.
Entropy 21 01168 g005
Figure 6. Step 3—thresholding.
Figure 6. Step 3—thresholding.
Entropy 21 01168 g006
Figure 7. Step 4—filling holes (a) and morphological simplification (b).
Figure 7. Step 4—filling holes (a) and morphological simplification (b).
Entropy 21 01168 g007
Figure 8. Step 5—Finding the largest object.
Figure 8. Step 5—Finding the largest object.
Entropy 21 01168 g008
Figure 9. Results of segmentation.
Figure 9. Results of segmentation.
Entropy 21 01168 g009
Figure 10. Two-dimensional array of sums.
Figure 10. Two-dimensional array of sums.
Entropy 21 01168 g010
Figure 11. Resulting vector of sums.
Figure 11. Resulting vector of sums.
Entropy 21 01168 g011
Figure 12. Weld masks.
Figure 12. Weld masks.
Entropy 21 01168 g012
Figure 13. Graphs of vector of sums of subfields in the mask for OK (a) and NOK (b) weld.
Figure 13. Graphs of vector of sums of subfields in the mask for OK (a) and NOK (b) weld.
Entropy 21 01168 g013
Figure 14. Weld masks for histogram projection.
Figure 14. Weld masks for histogram projection.
Entropy 21 01168 g014
Figure 15. Graph of histogram projection of an OK weld.
Figure 15. Graph of histogram projection of an OK weld.
Entropy 21 01168 g015
Figure 16. Graph of histogram projection of a NOK weld.
Figure 16. Graph of histogram projection of a NOK weld.
Entropy 21 01168 g016
Figure 17. Mask of OK and NOK weld.
Figure 17. Mask of OK and NOK weld.
Entropy 21 01168 g017
Figure 18. Graph of polar coordinates vector of an OK weld mask.
Figure 18. Graph of polar coordinates vector of an OK weld mask.
Entropy 21 01168 g018
Figure 19. Graph of polar coordinates vector of a NOK weld mask.
Figure 19. Graph of polar coordinates vector of a NOK weld mask.
Entropy 21 01168 g019
Figure 20. Settings for RBF network—for the vector of sums of subfields in the mask.
Figure 20. Settings for RBF network—for the vector of sums of subfields in the mask.
Entropy 21 01168 g020
Figure 21. Settings for RBF network—for the histogram projection vector.
Figure 21. Settings for RBF network—for the histogram projection vector.
Entropy 21 01168 g021
Figure 22. Settings for RBF network—for the polar coordinates vector.
Figure 22. Settings for RBF network—for the polar coordinates vector.
Entropy 21 01168 g022
Figure 23. Settings for MLP network—for vector of sums of subfields in the mask.
Figure 23. Settings for MLP network—for vector of sums of subfields in the mask.
Entropy 21 01168 g023
Figure 24. Settings for MLP network—for histogram projection vector.
Figure 24. Settings for MLP network—for histogram projection vector.
Entropy 21 01168 g024
Figure 25. Settings for MLP network - for the polar coordinates vector.
Figure 25. Settings for MLP network - for the polar coordinates vector.
Entropy 21 01168 g025
Figure 26. MLP network architecture—for vector of sums of subfields in the mask.
Figure 26. MLP network architecture—for vector of sums of subfields in the mask.
Entropy 21 01168 g026
Figure 27. Architecture of the convolution neural network 1.
Figure 27. Architecture of the convolution neural network 1.
Entropy 21 01168 g027
Figure 28. Architecture visualization of the convolution neural network 1.
Figure 28. Architecture visualization of the convolution neural network 1.
Entropy 21 01168 g028
Figure 29. Architecture visualization of the Convolution neural network 2.
Figure 29. Architecture visualization of the Convolution neural network 2.
Entropy 21 01168 g029
Figure 30. Example of profiling output using profvis.
Figure 30. Example of profiling output using profvis.
Entropy 21 01168 g030
Figure 31. Examples of weld segmentation results (ap).
Figure 31. Examples of weld segmentation results (ap).
Entropy 21 01168 g031aEntropy 21 01168 g031b
Figure 32. Results of segmentation process profiling.
Figure 32. Results of segmentation process profiling.
Entropy 21 01168 g032
Figure 33. The profiling results of data transformation.
Figure 33. The profiling results of data transformation.
Entropy 21 01168 g033
Figure 34. Confusion matrix.
Figure 34. Confusion matrix.
Entropy 21 01168 g034
Figure 35. ROC curves: excellent (blue); good (green); worthless (red).
Figure 35. ROC curves: excellent (blue); good (green); worthless (red).
Entropy 21 01168 g035
Figure 36. ROC curves for experiments with RBF networks.
Figure 36. ROC curves for experiments with RBF networks.
Entropy 21 01168 g036aEntropy 21 01168 g036b
Figure 37. ROC curves for experiments with MLP networks.
Figure 37. ROC curves for experiments with MLP networks.
Entropy 21 01168 g037aEntropy 21 01168 g037b
Figure 38. ROC curve for all convolution nets.
Figure 38. ROC curve for all convolution nets.
Entropy 21 01168 g038
Figure 39. Progress of accuracy for cnn-ker-ori10.
Figure 39. Progress of accuracy for cnn-ker-ori10.
Entropy 21 01168 g039
Figure 40. Progress of accuracy during epochs for cnn-ker-seg11.
Figure 40. Progress of accuracy during epochs for cnn-ker-seg11.
Entropy 21 01168 g040
Figure 41. Progress of accuracy during epochs for cnn-mxn-ori14.
Figure 41. Progress of accuracy during epochs for cnn-mxn-ori14.
Entropy 21 01168 g041
Figure 42. Progress of accuracy during epochs for cnn-mxn-seg15.
Figure 42. Progress of accuracy during epochs for cnn-mxn-seg15.
Entropy 21 01168 g042
Table 1. Technical specifications of PC.
Table 1. Technical specifications of PC.
Operating SystemWindows 7 Professional 64-bit
ProcessorIntel Core i7-2600 CPU @ 3,40 GHz
Memory16 GB DDR3
DiscSamsung SSD 850 EVO 500 GB
Table 2. Algorithms results for transform masks into feature vectors.
Table 2. Algorithms results for transform masks into feature vectors.
Data InterpretationTime [ms]Memory [MB]
the vector of sums of subfields in the mask160.1
histogram projection vector100.1
polar coordinates vector187.56
Table 3. Labels of neural network experiment.
Table 3. Labels of neural network experiment.
Test LabelNetwork TypeLibraryData Format
rbf-rsn-sum01RBFRSNNSSubfields sum
rbf-rsn-hpr02RBFRSNNSHistogram projection
rbf-rsn-pol03RBFRSNNSPolar coordinates
mlp-rsn-sum04MLPRSNNSSubfields sum
mlp-rsn-hpr05MLPRSNNSHistogram projection
mlp-rsn-pol06MLPRSNNSPolar coordinates
mlp-ker-sum07MLPKerasSubfields sum
mlp-ker-hpr08MLPKerasHistogram projection
mlp-ker-pol09MLPKerasPolar coordinates
cnn-ker-ori10CNN 1KerasOriginal
cnn-ker-seg11CNN 1KerasSegmented
cnn-mxn-ori12CNN 1MXNetOriginal
cnn-mxn-seg13CNN 1MXNetSegmented
cnn-mxn-ori14CNN 2MXNetOriginal
cnn-mxn-seg15CNN 2MXNetSegmented
Table 4. Accuracy a F-score for RBF and MLP networks.
Table 4. Accuracy a F-score for RBF and MLP networks.
Test LabelAccuracyF-Score
rbf-rsn-sum010.96990.9719
rbf-rsn-hpr020.91390.9127
rbf-rsn-pol030.91490.9139
mlp-rsn-sum040.99790.9981
mlp-rsn-hpr051.00001.0000
mlp-rsn-pol060.99590.9961
mlp-ker-sum070.96780.9700
mlp-ker-hpr080.97610.9761
mlp-ker-pol090.97660.9766
Table 5. Profiling of RBF and MLP networks training.
Table 5. Profiling of RBF and MLP networks training.
Test Label.Time [ms]Memory [MB]
rbf-rsn-sum016660687.6
rbf-rsn-hpr0242,530775.6
rbf-rsn-pol0332,080752.3
mlp-rsn-sum04850769.8
mlp-rsn-hpr059890653.7
mlp-rsn-pol0617,270672.0
mlp-ker-sum0752,830485.2
mlp-ker-hpr0845,660410.4
mlp-ker-pol0946,420401.9
Table 6. Accuracy and F-scores for convolution neural network experiments.
Table 6. Accuracy and F-scores for convolution neural network experiments.
Test LabelEpochsAccuracyF-Score
cnn-ker-ori1050.99900.9991
cnn-ker-seg1141.00001.0000
cnn-mxn-ori1260.99100.9920
cnn-mxn-seg1330.99800.9982
cnn-mxn-ori1441.00001.0000
cnn-mxn-seg1541.00001.0000
Table 7. Code profiling results for designed convolution neural networks.
Table 7. Code profiling results for designed convolution neural networks.
Test LabelEpochsTime [ms]Memory [MB]
cnn-ker-ori10538,610186.9
cnn-ker-seg11430,660180.0
cnn-mxn-ori126119,6304.7
cnn-mxn-seg13382,5802.6
cnn-mxn-ori14412,170157.9
cnn-mxn-seg15411,8503.7
Table 8. Profiling results for single weld diagnostics.
Table 8. Profiling results for single weld diagnostics.
Test LabelImage Time Preparation [ms]Diagnostic Time [ms]Memory [MB]
mlp-rsn-sum04210200.2
mlp-rsn-hpr051942403.0
mlp-rsn-pol061981051.8
cnn-mxn-ori1414140.5
cnn-mxn-seg1519440.5

Share and Cite

MDPI and ACS Style

Haffner, O.; Kučera, E.; Drahoš, P.; Cigánek, J. Using Entropy for Welds Segmentation and Evaluation. Entropy 2019, 21, 1168. https://doi.org/10.3390/e21121168

AMA Style

Haffner O, Kučera E, Drahoš P, Cigánek J. Using Entropy for Welds Segmentation and Evaluation. Entropy. 2019; 21(12):1168. https://doi.org/10.3390/e21121168

Chicago/Turabian Style

Haffner, Oto, Erik Kučera, Peter Drahoš, and Ján Cigánek. 2019. "Using Entropy for Welds Segmentation and Evaluation" Entropy 21, no. 12: 1168. https://doi.org/10.3390/e21121168

Note that from the first issue of 2016, this journal uses article numbers instead of page numbers. See further details here.

Article Metrics

Back to TopTop