A Review of the Image Classification Models Used for the Prediction of Bed Defects in the Selective Laser Sintering Process ^†

Abstract

Defects formed during the spreading of powder, known as powder bed defects, are a major issue in additive manufacturing processes. Deep learning (DL)-based image classification models can be utilised to detect defects caused by the powder spreading process. The aim of this research was to review and compare the performance of the EfficientNet_v2 deep learning image classification model against the commonly used VGG-16 model on a selective laser sintering powder bed defects (SLS PBDs) dataset. It was observed that the EfficientNet_v2 model achieved higher performance than the commonly used VGG-16 model, with a defect prediction accuracy of 97.54% and model sensitivity of 96.3%.

1. Introduction

The quality of the final desired product manufactured via selective laser sintering (SLS) is determined not only by a sufficient level of fusion between the powder layers but also the quality of the powder bed spread by the recoater blade. This is because the defects become an integral part of the product which can lead to a decrease in part quality [1].

Deep learning (DL) image recognition models are capable of analysing the physical world by measuring the differences in colour shades and estimating the spatial relations between different physical objects [2]. However, creating a DL model from scratch requires an extensive, labelled dataset and is computationally intensive. To reduce computational costs, a pre-trained model can be used, leveraging the progress it has made on a different task. This is called transfer learning [3].

The aim of this research was to compare the performance of DL models for the prediction of defects in the selective laser sintering powder bed Defects (SLS PBDs) dataset captured and published by Westphal et al. [4]. The objectives were as follows:

• To replicate and apply the commonly used VGG-16 on the SLS PBDs dataset [4].
• To build and test an Efficientnet_v2 [5] model on the same dataset.
• To compare the accuracy and sensitivity of the VGG-16 and Efficientnet_v2 models.
• Based on the comparison, identify any improvements in using the Efficientnet_v2 model for defect detection.

2. Method

Two pre-trained models, VGG-16 and EfficientNet_v2 network architectures, were used in this research. The SLS PBDs dataset [4] was used to train, validate and test the two aforementioned architectures.

2.1. Dataset and Pre-Processing

The powder bed images were captured with a resolution of 640 × 480 pixels. This generated 9426 powder bed images. Some images were captured under poor lighting conditions, resulting in 912 images being manually removed by the authors of the dataset. The dataset was manually labelled into two categories for binary classification: OK and DEF [4]. The OK class with 7808 images was populated with images that had no PBDs, and the DEF class with 706 images contained images where the powder bed had defects such as clumps, foreign bodies, etc. Figure 1a shows an image before and Figure 1b shows an image after pre-processing.

Figure 1. (a) Image before pre-processing and (b) image after pre-processing [4] (the image published by Westphal and Seitz is licensed under CC BY 4.0 DEED).

After labelling, a dataset imbalance was observed. The level of class imbalance can be calculated by using the imbalance ratio (IR) [4]. For the initial dataset, the calculated IR was 11.06, while the ideal ratio is 1. Therefore, random undersampling (RUS) and random oversampling (ROS) methods were used to reduce the dataset imbalance. Using RUS, 5808 images were removed from the OK class. Using ROS, 1294 images were randomly duplicated and added to the DEF class [4]. The balanced dataset contained a total of 4000 images: 2000 in the OK class and 2000 in the DEF class. The resulting IR ratio was 1 in this instance. The dataset was then split using a train/validation/test ratio of 50:25:25. This was carried out so that model validation and testing could be completed on unseen data.

2.2. Modelling, Hyperparameters and Performance Metrics

In this research, VGG-16 and Efficientnet_v2 DL models were evaluated. The model training process was divided into two stages. The first stage was an initial training stage using the pre-trained models. During the second stage, both models were fine-tuned to improve performance. To fine-tune the models, they were trained again using a learning rate reduced by a factor of 50, and all model layers were unfrozen. This process has the effect of further improving model performance. Table 1 shows the values set for the hyperparameters of the DL models used in this study. These values were taken from the original VGG-16 defect detection paper written by Westphal et al. [4]. This was carried out to ensure that a fair comparison could be made between the VGG-16 and EfficientNet_v2 models.

Table 1. Model hyperparameters [4].

Cost Function	Learning Rate	Optimiser	Epochs	Batch Size	Lr Decay	Early Stoppage
Binary cross-entropy	0.001	Adam ß1 = 0.9 ß2 = 0.999	120	16	Patience = 5	Patience = 20

3. Results

3.1. Model Accuracy and Sensitivity

Figure 2 depicts an accuracy graph for VGG-16 and EfficientNet_v2 model training during the fine-tuning process.

Figure 2. Accuracy plot for the VGG-16 and EfficientNet_v2 models.

3.2. Confusion Matrix

Table 2 shows a confusion matrix for the VGG-16 and Efficientnet_v2 models.

Table 2. Confusion matrix.

Predicted Values
VGG-16			EfficientNet_v2
	OK	DEF		OK	DEF
OK	460	20	OK	479	7
DEF	38	478	DEF	19	491

4. Discussion

Figure 2 shows that there was a large amount of volatility in the test accuracy of the VGG-16 model at the beginning epochs of model training. This could possibly be due to a large model size relative to the amount of data to train the model on [6]. On the other hand, comparing training and testing accuracy for the EfficientNet_v2 model, it was observed that the test accuracy was significantly higher than the training accuracy. This could be because the EfficientNet_v2 model uses dropout [5], while the VGG-16 model does not. Dropout disables neurons during training to make it artificially harder for the network, which can increase model performance [5]. This artificial increase in training difficulty would cause a decrease in training accuracy when compared to the testing accuracy. From the model’s performance on the test dataset, it was observed that the VGG-16 model had an accuracy of 95.8%. In comparison, the EfficientNet_v2 model achieved a higher accuracy of 97.54%. Model sensitivity is a particularly important performance metric in defect detection due to the large importance placed on avoiding false negative predictions. The sensitivity of the VGG-16 model was 93.9% compared to the sensitivity of 96.3% achieved by EfficientNet_v2. From Table 2, it can be observed that 20 erroneous false negative predictions were made by VGG-16, while for EfficientNet_v2, this was only 7.

5. Conclusions

This research aimed to test the performance of the proposed Efficientnet_v2 model against the commonly used VGG-16 model on the SLS PBDs dataset. The EfficientNet_v2 model and the VGG-16 model were trained, validated and tested using the hyperparameters published in the original research by Westphal et al. [4]. The proposed EfficientNet_v2 model achieved greater performance with a classification accuracy of 97.54% and model sensitivity of 96.3% when compared to the VGG-16 performance results. Also, the EfficientNet_v2 model made fewer erroneous false negative predictions when compared to VGG-16.