Baru-Net: Surface Defects Detection of Highly Reflective Chrome-Plated Appearance Parts

Abstract

Chrome plating parts with highly reflective characteristics are often used as appearance parts and must undergo strict defect detection to ensure quality. The defect detection method based on machine vision is the best choice. But due to the characteristic of high reflection, image acquisition is hard. For diverse defect appearances, it is difficult to use traditional algorithm for feature extraction. In this paper, a reasonable lighting scheme was designed to collect images effectively, and artificial defect images were made to expand the dataset to improve the deficiency of defect samples. A network, Baru-Net (Bis-Attention Rule), based on Unet architecture, the CBAM module and the ASPP module, was designed, and a block-step training strategy was proposed. With hyperparameter debugging, the semantic segmentation and classification of defects were carried out, and an accuracy rate of 98.3% achieved. Finally, QT realized the call to the weight model so that the AI model could be integrated into the automatic detection system.

1. Introduction

Chrome plating is a common process for appearance parts because of its excellent features, such as corrosion resistance, high brightness and specular reflection. With improvements in living standards, people’s requirements for the appearance of products are more and more strict, and slight product defects in the appearance can cause surface corrosion and sales difficulties, so it is necessary to detect all the appearance parts before they are assembled into the product. At present, appearance defect inspection is mainly performed manually. Through investigation, it was found that the appearance parts are often transported to a fixed detection area to wait for manual surface defect inspection. The delivery will delay production and reduce manufacturing efficiency. Moreover, multiple factors influence the reliability of manual surface defects detection, such as the fluctuation of the environment, eye strain of the workers, caused by long-term work, and a lack of experience of the workers, which makes it easy to cause missed detection or false detection. On the other hand, prolonged eye observation on highly reflective surfaces may also cause damage to workers’ eye health. Therefore, it is very important to improve the automation level of chrome plating appearance parts detection based on the defects detection algorithm.

The detection of highly reflective surface defects has attracted the attention of many scholars. Traditional feature extraction and AI technology are trendily adopted [1]. Neogi [2] adopted a global adaptive percentile threshold of gradient image to identify steel strips’ surface defects. Shao [3] proposed the entropy, grid gray gradient, and Hu invariant moment matching and morphological classification to detect highly reflective roller surfaces. Feng [4] proposed an adaptive image chromaticity information adjusting method, an adaptive threshold segmentation and a Haar-like feature extraction algorithm to detect surface defects in the high-reflective-metal surface. He [5] skillfully used a different image, in which the defect features were highlighted, and found out the rail surface defects through adaptive threshold binarization. YUN [6] used a discrete wavelet transform and double, adaptive, and local threshold to achieve defect detection in steel wire rods.

The above research was based on traditional computer vision technology, which uses image features to distinguish defects and obtain good experimental results. The traditional method has strong advantages for the cases where the defect samples are minimal and the defect morphology is simple, but the traditional method has the following common problems:

• High requirements for lighting technology. It is necessary to ensure the stability of lighting and avoid the influence of ambient light to obtain reliable digital image quality to meet the threshold condition.
• Image prepossessing and feature extraction algorithms are complex; several thresholds are commonly needed in these sequential algorithms, which influence the reliability and robustness of detection.

Therefore, the traditional method of defect feature extraction is often difficult for tasks with unstable ambient light or various defects’ morphology. Therefore, deep learning models with automatic feature extraction ability have been applied to such defect detection.

Guo [7] used CNN to classify resistance welding spots with very complex and diverse features and obtained a 99.01% accuracy rate, which is far above human work. Ş. Öztürk [8] introduced end-to-end training of a two-stage network with several extensions of the training process; the result shows that a 100% detection rate was achieved on the surface defect dataset DAGM and KolektorSDD. Je-Kang Park [9] used CNN to inspect surface defects and achieve an inspection speed thousands of times higher than that of manual inspection with close accuracy. Zhifen Zhang [10] combined multiple hierarchical features into one feature to locate defect details and achieve the potential for real-time detection. R. Baskaran [11] adopted MobileNet to detect steel frame structure defects based on Transfer Learning, and built an Android application incorporated with the model proposed, achieving an accuracy rate of 91% for scratch. Z. Ma [12] proposed a novel and lightweight detection method based on attention mechanism to inspect aluminum strip defects based on the YOLOv4 framework. The above two pieces of research focused on lightweight network design and network scale compression work to deploy defect detection to mobile phones or embedded systems.

It can be seen from the above research that deep learning has good feasibility and a strong potential for complex surface defect detection. In this paper, a reasonable lighting scheme was adopted to acquire the right images. We introduced a network called Baru-Net. It was based on the Unet architecture, a lightweight attention module CBAM (Convolutional Block Attention Module) and an ASPP (Atrous Spatial Pyramid Pooling) module. The attention module and ASPP can improve the segmentation accuracy and performance of the network and achieve effective segmentation and classification of surface defects of chrome-plated appearance parts.

2. High Reflective Chrome-Plated Surface Image Acquisition

The acquisition of proper images is the prerequisite for detecting highly reflective appearance parts, and reasonable lighting will significantly improve the detection effect [13,14]. Figure 1 and Figure 2 show the comparison of two lighting schemes. In Figure 1a, the traditional ring shaped light source causes specular reflection, resulting in distorted image collecting, as shown in Figure 1b.

In Figure 2a, two LED strip backlight sources were used and placed on symmetrical sides of the camera. The beam of light and the measured surface were at a certain angle, so that the reflected light could enter the camera for imaging. We fused these two separate images from the two light sources based on OpenCV, as shown in Figure 2b. The fused images overcame the specular reflection problem and effectively detected defects.

3. Identification and Location of Highly Reflective Surface Defects Based on Baru-Net

3.1. Network Structure

The proposed network Baru-Net was built on Unet architecture and consists of a CBAM module and an ASPP module, as shown in Figure 3.

3.1.1. Unet Architecture

Unet is a classic semantic segmentation network widely used in defect segmentation tasks; it builds on encoder–decoder end-to-end architecture [15]. Convolution and downsampling are used as encoding methods to obtain the feature map, which can expand the receptive field and integrate context information. Upsampling and deconvolution are used to decode the feature map and output a pixel-level classification result with the same size as the original image to achieve semantic segmentation. In upsampling methods, bilinear interpolation is a simple and effective up-sampling method, which has the advantages of fast calculation speed, small calculation, wide application, a strong ability to retain image details, fast running speed and simple operation, so bilinear interpolation was chosen in this paper.

Unet adopts a skip connection to reduce the information loss in convolution and downsampling by concatenating the feature map in the upsampling process and the feature map with the same size in the downsampling. Therefore, Unet can combine low resolution information (for object class recognition) with high resolution information (for precise segmentation and positioning).

3.1.2. CBAM Module

Baru-Net added an attention mechanism and an ASPP module based on Unet architecture. The Unet encoder has four stages. At each stage, two 3 × 3 convolutional layers are used to obtain the feature map, which is then input into the CBAM module. The output feature map of CBAM is processed by the pooling layer and sent to the next stage. The final stage feature map is put into the ASPP module. CBAM is a lightweight attention module, and it includes two sub-modules, as shown in Figure 4a; one is the CAM (Channel Attention Module), and the other is the SAM (Spatial Attention Module), which allows CBAM to judge essential parts of an image in both channel and spatial dimensions.

The CAM (as shown in Figure 4b) performs global average pooling (AvgPool) and global maximum pooling (MaxPool) on the incoming feature map (C₁ × W₁ × H₁) to obtain two C₁ × 1 × 1 feature maps, then puts these two C₁ × 1 × 1 feature maps into a shared fully connected layer (Shared MLP) and adds the two output result maps. Then, Sigmoid is used to obtain a weight between 0 and 1, which is the channel attention value. Finally, the channel attention value is multiplied by each channel of the original map F to obtain a new feature map F′, which has the same size as F (as shown in Figure 4a). CAM keeps the channels’ dimensions unchanged, compresses the spatial dimensions and extracts the channel attention value, usually for multi-classification detection [16].

The SAM is the complement to the CAM, as shown in Figure 4c. An average pooling and a maximum pooling are applied to the input feature map F′ along the channel of each feature point. $F_{Max}^s\in R^{1\times H\times W}$ and $F_{avg}^s\in R^{1\times H\times W}$ are the average pooling feature and the maximum pooling feature of the channels. Then, these two maps are concatenated together and achieve a 2 × W₁ × H₁ feature map. A standard convolution layer with a channel count of 1 is used for concatenation and convolution, reducing the feature map channels to 1 × W₁ × H₁. Then, Sigmoid is called to obtain a 2D feature map with weights between 0 and 1, which is the spatial attention value. Multiply each element in the spatial attention map by the corresponding element in the feature map, and then add weights to obtain a new weighted feature map F″ which has the same channels size of C₁ × H₁ × W₁ as the import feature map.

So SAM calculates the spatial attention value based on F′ and multiplies it by F′ to obtain F″. Then, F″ is added to the input feature to perform adaptive feature refinement and obtain the output feature of CBAM, as shown in Figure 4a. CBAM is a lightweight and general-purpose module that does not generate a large amount of redundant computing or parameters. While saving computational overhead, it analyzes which locations of the feature map need to be emphasized or suppressed, thereby quickly determining the highly focused area of the input image and achieving significant performance improvements [17].

3.1.3. ASPP Module

Segmentation requires a large receptive field. The larger convolution kernel or larger pool step size can enlarge the receptive field, but the former requires too much calculation, while the latter will cause resolution loss. ASPP extends the convolution kernel by adding (spreading rate-1) zero between the elements of the convolution kernel to generate convolution kernels with different sampling rates so as to realize free multi-scale feature extraction, as shown in Figure 5. It has the advantages of both large convolution kernel and a large pooling step.

In this paper, the result of encoding with feature map channels of C₄ × H₄ × W₄ was put into the ASPP module for further processing. Three atrous convolutions with spreading rates of 6, 12, and 18 were used to extract multi-scale feature extraction. The extracted feature map remained the same size, and the results were concatenated along the channel with the results generated by ordinary convolution and adaptive average pooling, which were used to obtain image-level global features. The concatenated feature map had a channel size of 5C₄ × H₄ × W₄, in order to use the feature map as the input for the next layer, performing a convolution operation on the feature map to reduce its size back to C₄ × H₄ × W₄. ASPP had a larger receptive field without losing too much multi-resolution and without additional calculation. Therefore, ASPP is very suitable for segmentation and detection tasks.

3.2. Experiments Setup

3.2.1. Dataset Making

Labelme is used to label the collected images, as shown in Figure 6. A json file will be generated for each image, which can be converted into a label image with four categories, image background, hump, pit and scratch, through a Python script.

In deep learning, the dataset scale influences the accuracy, generalization ability and robustness of the model. But defective samples are limited in industry. In order to expand the dataset, Photoshop was used to cut and copy the defect areas from the original images and paste them into the non-defect image to complete the artificial defect image so as to solve the problem of insufficient defect samples, as shown in Figure 7. Each artificial defect image contained only one type of defect. No combination of multiple defects was set. Then, digital image processing algorithms such as random rotation, cropping, shifting and deformation were used to enhance the dataset further.

The dataset included three parts: original image, artificial defect image and data enhanced image. There were 100 original images, 300 artificial defect images and 600 data-enhanced images, respectively, totaling 1000. Then, the final dataset was divided into train set, validation set and test set, and the ratio was 6:2:2.

3.2.2. Model Training and Evaluation

The model training was carried out on NVIDIA GeForce RTX3060 GPU, the initial learning rate was set to 0.0005 and the iteration number epoch was set to 50. Other hyperparameters are shown in

Table 1. Training hyperparameters setting.

Hyperparameters	Batch Size	Width	Height	Loss	Optimizer
Parameter value	4	96	96	Cross Entropy Loss	Adam

.

Accuracy and Macro-F1 are adopted as the evaluation indexes of the model, as shown in Formulas (1)–(3).

$$\mathrm{A}\mathrm{c}\mathrm{c}\mathrm{u}\mathrm{r}\mathrm{a}\mathrm{c}\mathrm{y}=\frac{\mathrm{T}\mathrm{P}+\mathrm{T}\mathrm{N}}{\mathrm{T}\mathrm{P}+\mathrm{T}\mathrm{N}+\mathrm{F}\mathrm{P}+\mathrm{F}\mathrm{N}}$$

(1)

$$\mathrm{F}1-{\mathrm{s}\mathrm{c}\mathrm{o}\mathrm{r}\mathrm{e}}_i=\frac{2{\mathrm{T}\mathrm{P}}_i}{2{\mathrm{T}\mathrm{P}}_i+{\mathrm{F}\mathrm{P}}_i+{\mathrm{F}\mathrm{N}}_i}$$

(2)

$$\mathrm{M}\mathrm{a}\mathrm{r}\mathrm{c}\mathrm{o}-\mathrm{F}1=\frac{\mathrm{F}1-{\mathrm{s}\mathrm{c}\mathrm{o}\mathrm{r}\mathrm{e}}_1+\mathrm{F}1-{\mathrm{s}\mathrm{c}\mathrm{o}\mathrm{r}\mathrm{e}}_2+\mathrm{F}1-{\mathrm{s}\mathrm{c}\mathrm{o}\mathrm{r}\mathrm{e}}_3+\mathrm{F}1-{\mathrm{s}\mathrm{c}\mathrm{o}\mathrm{r}\mathrm{e}}_4}{4}$$

(3)

where TP_i, FP_i and FN_i represent the true positive, false positive and false negative of each category, respectively.

Class imbalance generally affects the convergence of the model and the generalization ability on the test set. A special training strategy was adopted in this paper to deal with the class imbalance caused by the size of defects that were much smaller than the background area. We cut the original image into several block images with defects and block background images without defects. As shown in Figure 8, the image was divided according to the preset size. The defective block was put into stage 1 training, and the optimal model was saved as the pre-training model for stage 2 training. The background block image was put into stage 2 training based on the pre-training model, so that the model could segment the defects well.

To verify the proposed training method, the Macro_F1 of the conventional training method (method of putting all samples into model training) and the proposed method are compared and shown in Figure 9. The Macro_F1 of the proposed training method showed more suitability for the situation in this paper.

To verify whether ASPP and CBAM modules can improve the network performance, Unet, Unet + CBAM, Unet + ASPP and Baru-Net (Unet + CBAM + ASPP) were compared. As shown in Figure 10, CBAM or ASPP modules could separately improve the validation accuracy and reduce loss; the simultaneous use of CBAM and ASPP could further enhance network performance.

In a word, Baru-Net extracts image features in the down-sampling based on Unet. Due to the use of CBAM, a lightweight attention module that can enhance useful features and suppress noise in both spatial and channel scales, Baru-Net finally obtains more accurate detailed features. The introduced ASPP module could expand the receptive field and extract features at multiple scales.

To verify the performance of the Baru-Net model, compare it with the commonly used models such as UNet, UNet++ [18], AttentionUNet [19] and Res_UNet [20], as shown in

Table 2. Accuracy of different network models.

Model	Accuracy (%)	Macro-F1 (%)
UNet++	96.1	89.9
UNet	91.4	89.4
AttentionUNet	97.8	90.1
Res_UNet	89.3	88.4
Baru-Net	98.3	91.3

; both the Macro_F1 score and the accuracy of Baru-Net are significantly improved compared with the other models.

3.3. Defect Identification and Experimental Result Analysis

Baru-Net was used to detect the surface defects of chrome-plating appearance parts, as shown in Figure 11. The results show that the proposed model could accurately classify and segment chrome-plated surface defects.

In order to prove the validity of the artificial defect dataset, we trained Baru-Net on the datasets with and without artificial defects, respectively. Segmentation and identification results of Figure 11a are shown in Figure 12. The model trained on the original dataset could not detect the target well, as shown in Figure 12a. The error detection rate of the model trained on the enhanced dataset was significantly reduced, but the contours of defects were inaccurate, and stains and dust were wrongly identified as defects, as shown in Figure 12b. The model trained on the dataset with artificial defects obtained the best detection results, as shown in Figure 11d. It can be seen that in the case of defect detection, it is of practical significance to use artificial defect images to make up for the shortage in defect samples.

We also compared the proposed method with the traditional method based on OpenCV digital image processing algorithm to verify the effectiveness of the proposed method. The results show that OpenCV was insufficient in extracting small and hump defects, as shown in the blue and red wireframe in Figure 13a. Moreover, it is challenging to set thresholds to distinguish and identify different defects.

Meanwhile, the proposed method was compared with the Yolov8 model training under the same dataset as Baru-net. By comparing Figure 14a,c and 14b,d, the results show that yolov8 had false or missing detection. In contrast, Baru-net had significantly better detection accuracy for stains and small-size defects than Yolov8. This was because the original Yolov8 model was trained based on a large dataset, and the dataset in this paper is a small sample dataset, which could not meet the detection requirements even if transfer training was carried out. Conversely, the proposed Baru-net was based on the Unet network and inherited the advantages of Unet, which just needed a small amount of data to achieve high-quality segmentation, and was suitable for dealing with the problem of small datasets in industry.

3.4. Model Deployment and Practical Application

We used OpenVINO to deploy the trained model weight for the automatic detection system developed based on QT. The implementation processes were as follows:

• Transform the trained weight file into ONNX type and OpenVINO intermediate IR file.
• Configure OpenVINO, OpenCV and libtorch environments in QT.
• Call the API of Pytorch in OpenVINO, load the converted ONNX and IR files into the OpenVINO interface, and encapsulate this process into a reasoning and prediction class.
• Use OpenCV to read the image to be predicted, and put it into the inference port for inference and prediction.
• Convert the prediction result into the image type used in QT, and the sorting mechanism will be triggered according to the defect detection result.

The automatic detection system was developed based on QT, and a stepper motor drove the conveyor belt; the defect detection was accurate and efficient, and the detection speed of a single piece reached about 800 ms, which met industrial inspection requirements. However, the charging and blanking mechanism needs to be further developed, and as for special-shaped parts, the detection path can be designed with the cooperation of the robot.

4. Conclusions

The following conclusions can be drawn based on the results and discussion.

(1)

Firstly, we proposed a network for surface defect detection in a highly reflective chrome plating work-piece that combined dual attention mechanisms and semantic segmentation to detect three kinds of defects. Secondly, a fusion of images from different light angles was used to avoid the effects of high reflectivity. In addition, the dataset was enhanced by creating artificial defect images to solve the problem of insufficient datasets. Finally, a step-by-step training strategy could solve the problem of category imbalance caused by the defect size being too minor compared to the background.

(2)

The model achieved a detection accuracy of 98.3 IEEE and a detection speed of around 800 ms on a single GPU. The Baru-Net and dual-angle light source approach could be applied to the industrial scene of small-sized samples and highly reflective surface.

(3)

In the future, the dataset will be further expanded in terms of the image number and defect type to ensure better generalization. The method based on transfer learning can also be implemented to improve the model performance. In addition, for the detection of special-shaped parts, we will further optimize the inspection device so that it can meet practical needs.