Machine Learning-Augmented Micro-Defect Detection on Plastic Straw

Abstract

Plastic straws are well-known tools to assist human beings in drinking fluid, but most of them have micro-defects including black spot defects, head problems, pressure tube defects, and sealing wrinkles. The manual detection of these defects has drawbacks such as low efficiency, a high false detection rate, and excessive labor. This paper proposed machine vision-based detection with self-adaption and high-accuracy characteristics. A serial synthesis of algorithms including homomorphic filtering, Nobuyuki Otsu, and morphological opening operations is proposed to obtain plastic straws with binary images with good performance, and it was further found that the convolutional neural network can be designed to realize the real-time recognition of black spot defects, where the corner detection algorithm demonstrates the linear fitting of the edge point of the straw with the effective detection of sealing wrinkle defects. We also demonstrated that the multi-threshold classification algorithm is used to detect defects effectively for head problems and pressure tube defects. The detection system based on machine vision successfully overcomes shortcomings of manual inspection, which has high inspection efficiency and adaptively detects multiple defects with 96.85% accuracy. This research can effectively help straw companies achieve high-quality automated production and promotes the application of machine vision in plastic straw defects with the aid of machine learning.

1. Introduction

The plastic straw is widely used in dairy products, carbonated drinks, and freshly made drinking fields, among which the U-shaped plastic straw and telescopic straw are widely used. In the production process of U-shaped plastic straws, black spot defects, head problems, and pressure tube defects are three common defects. Black spot defects are manifested as black spots on the U-shaped plastic straw and are a kind of micro/nano defect. The head problems are that bending folds connecting long and short phases appear in the long or short phases. The pressure tube defects are the straws that are connected and extruded. Sealing wrinkle defects is a common defect of telescopic straws, and the reason is that the indentation on the neck turns the original arc into a straight line, and many times, this defect is micro/nanoscale. Traditional plastic straws and telescopic straw manufacturers use manual vision to detect the products on the assembly line and screen the defective products. However, traditional manual inspection has some shortcomings, such as strong subjectivity, low efficiency, and lagging behind the production. It is easy to ignore the micro/nano-defects such as black spots and sealing wrinkles of the straw and cause false detection by manual inspection. The straw production speed is fast and the quantity is large, but the manual inspection efficiency is low. This requires a lot of manpower and material resources to meet production, resulting in a huge waste of energy. It cannot meet the production requirements of large quantities and high standards in modern enterprises. Therefore, modern advanced methods such as machine vision, machine learning [1,2], deep learning, and smart sensor [3] must be introduced.

Machine vision is a modern technology which consists of a hardware part and a software part. The hardware part mainly includes the computer and the camera, and the software part mainly refers to image processing and analysis software, which often uses machine learning and deep learning methods. The working line of machine vision is generally to take photos with the camera and transfer the photos to the computer, and the software will process and analyze the pictures to conclude. With the development of technology, machine vision has the advantages of real time, high efficiency, high accuracy, adaptability, and non-contact. These advantages make defect and quality inspection technology based on machine vision with the aid of machine learning widely used in various fields, mainly including manufacturing [4,5,6], food [7,8,9], pharmaceutical [10,11,12], and agriculture [13,14,15]. In addition, image noise reduction [16], image segmentation [17], feature extraction [18], Principal Component Analysis (PCA) [19], support vector machine (SVM) [20], and convolutional neural networks (CNN) [21] algorithms are often used in machine vision to improve the detection effect. Xu et al. [22] proposed a machine learning method to detect plastic straw defects and used sparse rules and small-scale factors to optimize the network. Finally, the detection accuracy of straw defects reached 96.19% under the condition of compressing the network volume. Sun et al. [23] proposed a welding defect detection and classification algorithm for a thin-walled metal tank based on machine vision. The algorithm extracts the defect regions by constructing the image sequence background and then classifies and detects different types of welding defects according to the defect features. Liu et al. [24] proposed an online learning strategy based on machine vision to locate the position of railway track fasteners and used a deep convolution neural network algorithm based on machine vision to detect fastener defects. Harn et al. [25] proposed a new method based on machine vision and load cells to detect egg density and the real-time monitoring of egg grading and freshness. The method uses machine vision to measure the external physical characteristics of the eggs, evaluate the volume of the eggs, measure the quality of the eggs through the weight sensor, and then calculate the density of the eggs and evaluate the freshness of the eggs. Hou et al. [26] proposed an algorithm based on machine vision and machine learning to detect pills’ spherical, abnormal size, and color defects. Tong et al. [27] designed a machine vision system for estimating the quality of seedlings based on the leaf area of seedlings and proposed a decision-making method that combines the center of the cross-border leaf area and an improved watershed segmentation method of overlapping leaf images to assist the machine vision system in solving the problem of leaf overlap. Finally, the system successfully identified the quality of tomato, cucumber, eggplant, and pepper seedlings. From the above literature, machine vision has been widely used in the field of weld, railway and other metal products surface defect detection, but there is no research on the surface defects of plastic straw products.

In this paper, we proposed a machine vision-based inspection system with self-adaption and high accuracy that successfully automatically detects four defects of plastic straws: black spot defects, head problems, pressure tube defects, and sealing wrinkles. It overcomes the shortcomings of low efficiency and low accuracy of manual inspection in the field of straw inspection, and when detecting the same number of straws, it effectively saves manpower and material resources, realizes energy saving, improves production efficiency and product quality, helps companies reduce labor costs, and realizes the automated production of plastic straws. At the same time, this research also expands the application range of machine vision and offers certain reference significance for machine vision in the field of plastic straw defect detection. The rest of this article is structured as follows. In Section 2, the machine vision inspection system for straw defects with high accuracy is built. The image processing and defect detection methods and experimental results are presented in Section 3. Finally, the paper is concluded in Section 4.

2. Materials and Methods

Figure 1a depicts a system schematic of defect detection through machine vision. The camera is used instead of human eyes to observe objects, and the computer is used instead of human brains to judge whether there are defects. In actual use, it is necessary to adjust the location, number of cameras and defect discrimination algorithm according to specific situations. As shown in Figure 1b, a machine vision-based detection system is established to detect defects of U-shaped plastic straws and telescopic straws. In the straw production line, the packed straws are driven by a motor and move forward continuously through multiple rollers. In the production of traditional straws, quality inspectors are required to test whether each straw meets the quality requirements, which is far from meeting the requirements of modern production. In the detection system based on machine vision, to suppress the diffuse reflected light on the surface of the straw, the Chinese OPT-FC series LED light, which is a coaxial planar LED lamp with both coaxial light and shadowless light, is selected as the system’s illuminating light source, and the forward photo method based on reflected light is used in the system. The German Basler’s ACE series model ACA1300-30gc CCD cameras and the PENTAX C418DX lens are used to replace the human eyes to take photos, and the Chinese Advantech 610L industrial computer is used to replace the human brain to analyze photos.

To ensure that both the front and back of each straw are captured, depending on the type of straw, the shooting range and frequency of the upper and lower cameras are adjusted to 5 frames per second when it is a U-shaped straw, each containing two complete U-shaped straws; and 4 frames per second when it is a telescopic straw, each containing three complete telescopic straws on the assembly line. Then, the taken straw photo is transferred to the host computer and undergoes a series of image processing, and finally, the machine vision system software analyzes which type of straw is in the photo and judges whether it is defective, which type of defect it is, and whether it meets the quality requirements. Considering that the current straw production process is mature, the defect rate is low, and the packaged straws are inspected; when a defective straw is detected, the system alarms and manually removes the defective straw.

3. Results and Analysis

3.1. Introduction of Straw Defects and Binarization Method

Two kinds of straws are shown in Figure 2: a U-shaped plastic straw and telescopic straw. Photographs U-shaped plastic straws taken by mobile phones are shown in Figure 2a, and there are three defects in U-shaped plastic straws: aside from the no defect image (Figure 2b), there were also black spot defects (Figure 2c), head problems (Figure 2d), and pressure tube defects (Figure 2e). One defect was found in telescopic straws: aside from the no defect image (Figure 2f), there were also sealing wrinkle defects (Figure 2g). The black spot defects are caused by the appearance of a black spot on the U-shaped plastic straw that should have no abnormal color; this kind of defect is very small. The analysis shows that the effective defective pixel amount to detect black spots is low, and the gray-scale contrast of the defect areas is low. Few pixels in the black spot region make it difficult to identify defects effectively. The head problem of the U-shaped plastic straw is manifested in bending folds connecting long and short phases in the long or short phases and can be in any position. The defects of pressure tube defects are that the straws are connected and squeezed, which should not be connected. The geometric characteristics of head problems and pressure tube defects are more obvious, but it is difficult to characterize due to the changeable geometric shape, which makes it difficult to effectively detect the defects. According to the sealing wrinkle defects image, the defect is that the indentation on the neck turns the original arc into a straight line. Although the outline of the image can be outlined by edge detection, it is difficult to define and detect defects. It can be concluded from the figure that due to the limitation of human visual ability, some straw defects such as Figure 2c are not directly detectable by humans, so the use of a computer vision recognition method can improve the recognition efficiency and increase the recognition accuracy.

Although the detection system of the straw uses a coaxial light source to suppress diffuse emission, the image of the telescopic straw still has uneven illumination due to the reflective characteristics of the plastic straw material. This makes the image gray value range wider, as shown in Figure 2h, and there is no obvious double-peak area. Therefore, it is necessary to preprocess the straw image first to obtain the high-quality binary image of the straw. The OTSU algorithm is also called the maximum between-class variance method [28], which divides the data into two parts through a constant threshold so that the data variance between two parts is the largest and the data value variance between each part is the smallest. In addition, the segmentation threshold $t^{\ast }$ is calculated as the following Equation (1), and the calculation of each part is shown in Equations (2)–(6).

$$t^{\ast }=Argmax\left(w_1\left(t\right)u_1^2\left(t\right)+w_2\left(t\right)u_2^2\left(t\right)\right)$$

(1)

$$w_1\left(t\right)={\sum }_{i=0}^t{p}_i$$

(2)

$$w_2\left(t\right)={\sum }_{i=t+1}^{L-1}{p}_i$$

(3)

$$u_1\left(t\right)={\sum }_{i=0}^{t}i{p}_i∕w_1\left(t\right)$$

(4)

$$u_2\left(t\right)={\sum }_{i=t+1}^{L-1}i{p}_i∕w_2\left(t\right)$$

(5)

$$p_i=\frac{n_i}{n}$$

(6)

In these equations, $t^{\ast }$ represents the value of t when traversing all t to obtain the maximum value for Equation (1), where $0\le t<L$, L is the maximum value of image gray, n is the number of image pixels, $n_i$ is the number of pixels whose gray value is i, and then $p_i$ is the probability of the occurrence of pixels with a gray value of i in the image. According to the calculation equation, $w_1\left(t\right)$ is the probability of pixels with gray values between 0 and t in the image, $w_2\left(t\right)$ is the probability of pixels with gray values between (t + 1) and (L − 1) in the image, $u_1\left(t\right)$ is the average gray value of pixels with gray values between 0 and t, and $u_2\left(t\right)$ is the average gray value of pixels with gray values between (t + 1) and (L − 1). An image with obvious double peaks in the gray value distribution can obtain a better binary image through the OTSU algorithm, but it cannot segment the image well under uneven lighting conditions, and it is also sensitive to noise [29]. To solve these two problems, homomorphic filtering and morphological opening operations are introduced into the OTSU algorithm. The homomorphic filtering algorithm converts the image to the logarithmic domain for operation, which can effectively improve the contrast of the low gray value area. [30] The homomorphic filtering algorithm can eliminate the uneven illumination of the image [31] and form an effective dual-peak gray distribution. The homomorphic filtering algorithm involves logarithmic transform and Discrete Fourier Transform (DFT), where $H\left(u,v\right)$ represents the filtering type, as well as Inverse Discrete Fourier Transform (IDFT) and exponential transform, which is denoted in Formula 7.

$$f\left(x,y\right)\to lnf\to DFT\to H\left(u,v\right)\to {\left(DFT\right)}^{-1}\to exp\to g\left(x,y\right)$$

(7)

The homomorphic filtering was performed on the image. The processed image is shown in Figure 2i, and the gray-scale distribution is shown in Figure 2j. The morphological opening operation [32,33] can remove isolated dots, burrs, and bridges, while the overall position and shape remain unchanged; the principle is to corrode the binary image first and then expand it [34]. When the binary image of a straw is obtained from the grayscale image of the straw, the above three algorithms have a good complementary effect. So, a serial synthesis algorithm of homomorphic filtering, OTSU, and morphological open operation is proposed to suppress diffuse reflection and obtain high-quality binary images of the straw.

3.2. Defect Detection for Sealing Wrinkles, Head Problems, and Pressure Tube Defects

The left edge of the straw is formed by a straight line and a circular arc, and the right edge is a straight line due to the sealing wrinkle. Therefore, the left and right contours of the straw can be fitted with a straight line, the size of the loss function can be compared to determine whether there is a defect, and the sealing wrinkle defects detection flow chart is summarized as shown in Figure 3a. The serial synthesis algorithm of homomorphic filtering, OTSU, and morphological open operation was used to process the straw and obtain a high-quality binary image. Then, four corners need to be extracted to determine the range of the edge points on the left and right sides of the straw. This paper proposes a corner detection algorithm based on the Susan operator. The idea of the algorithm is as follows: set a circular sliding window with a radius of r and an area of S, and traverse all the points on the edge contour with the center of the circle. Count the number of pixels with a pixel value of 1 in the sliding window, which is denoted as $N_1$. Set a threshold σ; when $N_1/S\le \mathsf{\sigma }$, it is judged as a suspected corner. All the suspected corner points obtained are divided into two categories according to the size in the y-direction, while the maximum and minimum values are screened in the x-direction, respectively, and the two largest and smallest pixels in the x-direction in each category are selected to obtain coordinates of the four corners. In this paper, $r=30$, $\mathsf{\sigma }=0.35$, and the corner obtained are shown in the corner position. As shown in Figure 3b,c, compared with the effect of the Harris operator [35] and Susan operator [36], this algorithm reduces the influence of noise through the binary image input and ensures the detection of four corner points through non-maximum and non-minimum value suppression, which meets the requirements of this research. The least-square method is a commonly used method of fitting [37], which was used to fit the straight line between the detected corner points on the left and right edge contour points in this research. The root means square error (RMSE) was calculated to determine if the straw has a sealing wrinkle defect. The RMSE of a straight line on the left is 50, and the RMSE of a straight line on the right is 10, so the threshold $\mathsf{\gamma }=30$ was set in this paper. If the RMSE is less than this threshold, it is considered that there is a sealing wrinkle, and the accuracy reaches 96%.

The inspection process of head problems and pressure tube defects of U-shaped plastic straws is shown in Figure 3d. Because the picture taken by the camera contains multiple U-shaped straws, the area where a single complete straw is located needs to be separated first. The pixel values of the image background area are quite different from those of the U-shaped plastic straw area. The serial synthesis algorithm of homomorphic filtering, OTSU, and morphological open operation was used to process the U-shaped plastic straw with head problem and pressure tube defects, and then, a high-quality binary image was obtained as shown in Figure 3e,f. Binary connected regions labeling algorithms have two types: four-domain region labeling and eight-domain region labeling [38]. An eight-domain area marking algorithm was used to label binary connected regions, which considers four diagonal domains with higher accuracy rates. The red area displays the connected regions of the background area, and the other color area of the image represents the connected regions of the straw foreground area. The area between a complete straw and a part of the straw is different, so the connected area of the complete straw is extracted from the areas of each connected area. Geometric features of the connected domain from the complete straw can be extracted to construct the pattern vector, because the geometric difference among the straw head problem, the pressure tube defect, and the perfect straw is significant. The multi-threshold classification algorithm which is presented in Figure 3g is used for defect detection; three features are extracted from three areas. The area A is obtained by counting the pixels of the defect-connected domain area; the area R is found through counting the pixels of the circumscribed rectangle of the connected domain of the defect area; and the ellipse minor axis length C is detected by ellipse fitting of the defect area. To construct feature vectors, suppose the pattern vector is X; then, $\mathrm{X}={\left[\mathrm{A}\mathrm{R}\mathrm{C}\right]}^{\mathrm{T}}$. The two pattern classes, $w_1$ and $w_2$, represent the head problems and pressure tube defects, respectively. Three geometric features of A, R, and C are employed as computer templates. Then, we construct the threshold vector $\mathrm{T}={\left[\mathrm{AT}\mathrm{RT}\mathrm{CT}\right]}^{\mathrm{T}},$ which is set as $T_1={\left[\mathrm{26,000}\mathrm{80,000}100\right]}^{\mathrm{T}}$, $T_2={\left[\mathrm{26,000}\mathrm{90,000}135\right]}^{\mathrm{T}}$. In the test section, the pattern vector component is compared with the threshold vector component sequentially to identify the type of defect. As shown in

Table 1. Recognition result of the multi-threshold method.

Defect	Number of Samples	Correct Number of Samples	Recognition Rate (%)
Head problems	100	98	98
Pressure tube defects	100	100	100

, the multi-threshold classification algorithm can detect these two types of defects based on the above statistical geometric features, and its accuracy is more than 98%. Due to the diverse head problems, their geometric features are not obvious compared with the pressure tube defects, which cause a slightly lower accuracy rate.

3.3. Defect Detection for Black Spot Defects

After judging that the connected domain of the complete straw matches the threshold vector of the perfect straw, the black spot defect of the straw is further determined. The black spot defects have little effect on the geometric characteristics of the connected area of the straw. Therefore, unlike the detection of head problems and pressure tube defects, it is difficult to effectively detect the black spot defect of the straw through the geometric characteristics of the connected area of the straw. The mask algorithm is used to obtain the grayscale image of a single straw through the rectangle containing the connected area of the straw, and the effect is shown in the image of the straw. The serial synthesis algorithm of homomorphic filtering, OTSU, and morphological open operation was used to process the U-shaped plastic straw with a black spot defect, and then, a high-quality binary image was obtained. The binary image and the single-straw grayscale image are image-fused to obtain a grayscale image of a single straw with black background, and the effect is shown in the gray image on black background of Figure 4a. Considering that the black spot defects area is small and inside the straw area, and the grayscale image of the straw is not the same as the normal part of the inside of the straw, the image should be enhanced first. After comparing the noise reduction effects of mean filtering [39], no filtering, and Gaussian filtering [33] operations, mean filtering is selected as the noise reduction method in this research, and the straw after means filtering is shown in the gray image after mean filtering. By the mean filtering processing, the pixels of the black spot defect areas will increase, and the color of the normal area of the straw will be smoother and more uniform. These effects make the edge between the black spot and the straw increase, and the false edge reduces, which is caused by the uneven grayscale inside the straw. The adaptive threshold segmentation algorithm was used to extract the edge contours that exist in the image and then filter the possible black spot edge contours based on the contour area without losing the true black spot edge contour. The noise inside of the straw is likely to be misjudged as the contour of the black spot edge. The mask algorithm is used to obtain the possible black spot positions through the obtained possible black spot contours on the grayscale image of a single straw, as shown in possible black spot positions on the straw. Comparing the black spot defect images obtained by Gaussian filtering and no filtering as shown in Figure 4b,c, the effect of mean filtering is better. The position of the black spot defects is located; there are not only defective areas in the image but also pseudo-black spot defects areas due to noise. The real black spot defects in the image after adaptive threshold segmentation will be detected.

The black spot defects are very small in the image straw area, and the accuracy of the defect detection using the classifier after the feature extraction of the original image is not good, so these possible black spot defects are segmented to detect black spot defects. The self-attention mechanism [40] is improved from the attention mechanism, which can capture the global information to obtain a larger sense space, calculate the response of a certain position in the image as the weighted sum of all global position features, and strengthen the capture of the long-term dependence across regions in the image [41,42]. The working principle of the self-attention mechanism is shown in Figure 4d. Using the self-attention mechanism can highlight the black spot defect features in the global image, which is conducive to further defect feature extraction. As an excellent classifier, CNN [21] has a good application in classification and recognition. Different from the traditional method of manually extracting features and then classifying, CNN has a strong ability to automatically extract features and could find the feature rules in samples. CNN is widely used in image classification and target recognition. As shown in Figure 4e, a CNN usually consists of the convolutional layers, the pooling layers, and the fully connected layers. The convolution unit in CNN just pays attention to the neighborhood area each time and performs local area operations while ignoring the influence of other areas in the global image on the current area. To highlight the global black spot defects, the self-attention mechanism is introduced before the CNN input as a supplement to the framework. The self-attention module calculates the response of a certain position in the picture and strengthens the black spot defects in the global image capture, effectively compensating for the characteristics of local convolution operation in CNN.

The Relu activation function is used in traditional CNN, and its formula is $f\left(x\right)=\max \left(0,x\right)$. It is known that when the input is negative, the gradient of the Relu function is zero, which causes its weight to be unable to be updated, and it will remain silent during the rest of the training process. In this case, the learning speed of the network may become very slow and even make the network directly invalid input. The gradient of the Relu function is discontinuous, which may cause uncertainty in the training result [43]. To overcome the above shortcomings of the Relu function, the Mish activation function [44] was used, and the formula is as shown in Equations (8) and (9).

$$f\left(x\right)=x\ast \tanh \left(\ln \left(1+e^x\right)\right)$$

(8)

$$\tanh \left(x\right)=\frac{e^x-e^{-x}}{e^x+e^{-x}}$$

(9)

Compared with the Relu function, the Mish function guarantees the smoothness of each point, which makes the gradient descent effect better than Relu. When the value is negative, a smaller negative gradient is allowed to flow in to ensure that the information will not be interrupted, so that the network has better accuracy and generalization ability. In addition, the lower bound can also improve the regularization effect of the network. Based on the above analysis, the self-attention mechanism and Mish function can well make up for some of the shortcomings of traditional CNN. The CNN with the self-attention mechanism and Mish function is proposed to detect real black spot defects. Overall, 557 samples were collected from U-shaped plastic straws, the number of samples with black spot defects was 274, and the number of samples with fake black spot defects was 283. The samples are not large enough for CNN. To avoid over-fitting, the 10-fold cross-validation method [45] was adopted. The termination condition of CNN iteration is set to 2500 times, and the learning rate is 0.005. After 2500 iterations of training, the cost is reduced to 0.243, the accuracy of the model’s training set reaches 96.74%, and the test set accuracy reaches 96.85%, as shown in Figure 5. The train accuracy and test accuracy, the CNN with the Mish function, and the self-attention mechanism are higher than the detection results using classical CNN.

4. Conclusions

In summary, straw defect detection mainly relies on manual inspection in industrial areas. Since the defects are micro-sized and thus unrecognized by human vision and human fatigue, manual detection is unable to meet the high precision and demanding work requirements of the manufacturing industry. In this regard, the machine vision-based straw defect detection system proposed in the article can improve the recognition accuracy and rate. It is mainly reflected in the following three aspects. Firstly, a suitable lighting scheme for straw detection is proposed to achieve image acquisition and analysis of different straw defects; secondly, a homomorphic filtering algorithm is used to reconstruct images and improve image recognizability for problems such as low contrast and uneven illumination of images acquired by the image acquisition system. Lastly, for the different types of defects of telescopic straws and U-shaped straws, various defect detection methods are adopted to further improve the detection accuracy.

Specific experimental results are shown as follows: compared to manual detection, the machine vision-based straw defects detection system with self-adaption and high accuracy is tried to detect black spot defects, head problems, pressure tube defects, and sealing wrinkle defects in the production process of plastic straws in an energy-saving manner. The serial synthesis algorithm of homomorphic filtering and morphological open operation is proposed to assist in suppressing diffuse reflection and obtaining high-quality binary images. The CNN with a self-attention mechanism and Mish function is proposed to detect the black spot defects of plastic straws, and the accuracy is over 96.74%, which is higher than the accuracy rate of the classic CNN algorithm. The multi-threshold classification algorithm is used to detect the head problems and pressure tube defects of plastic straws with an accuracy rate of over 98%. A corner detection algorithm is proposed to assist linear fitting, which is used to detect sealing wrinkle defects with an accuracy rate of 96%. The detection system based on machine vision adaptively detects multiple defects in a variety of straws, and its accuracy and detection speed exceed manual detection; these advantages make it save a lot of manpower and material resources for the enterprise and realize energy savings. This can effectively help straw companies achieve high-quality automated production. This research promotes the application of machine vision in the field of plastic straw defect detection.