Research on AI-Driven Classification Possibilities of Ball-Burnished Regular Relief Patterns Using Mixed Symmetrical 2D Image Datasets Derived from 3D-Scanned Topography and Photo Camera

Abstract

The present research is related to the application of artificial intelligence (AI) approaches for classifying surface textures, specifically regular reliefs patterns formed by ball burnishing operations. A two-stage methodology is employed, starting with the creation of regular reliefs (RRs) on test parts by ball burnishing, followed by 3D topography scanning with Alicona device and data preprocessing with Gwyddion, and Blender software, where the acquired 3D topographies are converted into a set of 2D images, using various virtual camera movements and lighting to simulate the symmetrical fluctuations around the tool-path of the real camera. Four pre-trained convolutional neural networks (DenseNet121, EfficientNetB0, MobileNetV2, and VGG16) are used as a base for transfer learning and tested for their generalization performance on different combinations of synthetic and real image datasets. The models were evaluated by using confusion matrices and four additional metrics. The results show that the pretrained VGG16 model generalizes the best regular reliefs textures (96%), in comparison with the other models, if it is subjected to transfer learning via feature extraction, using mixed dataset, which consist of 34,037 images in following proportions: non-textured synthetic (87%), textured synthetic (8%), and real captured (5%) images of such a regular relief.

1. Introduction

Ball burnishing (BB) is a highly effective surface finishing process that offers significant improvements in surface roughness, hardness, tribological properties, corrosion resistance, and fatigue strength of the various machine parts [1,2,3,4,5,6,7,8,9,10,11,12]. The ability to induce compressive residual stress and refine microstructures further enhances the mechanical properties of the material. The classical BB process [13] is versatile and can be optimized by adding several additional parameters into the operation’s kinematic to achieve more complex surface topography characteristics. Among them, the so called “vibratory BB” (or VBB) [14,15] and ultrasonic vibration assisted BB (or VABB) [16,17] have found a wider spread in the practice. Applying additional ultrasonic vibrations, generated by piezo actuators integrated in the burnishing tool, leads to further decreasing of the roughness heights and allows smaller deforming force magnitudes to be used [16,18]. All of them are applied to different types of parts, having outer and inner rotational [13,14,19], planar [20,21,22,23,24] and even surfaces with more intricate shapes [2,25,26]. Each type of surface shape utilizes different kinematics and parameters such as the chosen type of BB process, the magnitude of used deforming force, speed and feedrate, the number of passes of the deforming tool [27], the type of lubrication [28,29] and the materials involved in the process. The deforming force most often is applied mechanically by using springs and/or screw-nut mechanisms but also it can be maintained hydrostatically (i.e., the so-called “Low-Plasticity BB” [6]). Balls with a certain diameter, usually made of hardened steel, are used as a deforming element, but sometimes they could also be made of ceramics, especially in the hydrostatic burnishing applications. The ball-tool rolls over the burnished surface in all types of the abovementioned BB processes.

The main difference between ultrasonic VABB and VBB is the enforced oscillation parameters of the burnishing tool. While in ultrasonic VABB the frequency of oscillations can reach 40 kHz and their amplitude is up to several micrometers, in VBB, the amplitudes of the additional reciprocate movement of the ball tool can vary between 0.5 and 12 mm, and the frequency usually does not exceed 60 Hz. While BB and VABB are used mainly to smooth the processed surface, VBB has been developed to create specific plastically deformed patterned textures [14]. Initially, they were called “regular micro-reliefs,” (RMRs) and have been formed onto the surfaces with different shapes, using a combination of manually operated turning (or milling) machines and specially designed burnishing tools [30,31,32]. These tools have been autonomously driven to provide an additional reciprocating movement of the deforming element (i.e., the ball tool) with amplitudes of several millimeters and frequencies of a few tens of hertz in addition to the classical kinematics of the BB-processes.

Nowadays, with the mass spread of Computer-Numerical-Control (CNC)-driven processing machines in the industry, the needed kinematics of the BB operations are provided programmatically [25,33,34,35,36]. By generating suitable Numerical-Control (NC) instructions that drive the machine axes appropriately, the ball tool can perform the complex toolpath needed, using only the machine’s axes (there is no need for additional drives, transmissions, etc.). This allows the design of the burnishing tools to be simplified and their size to be significantly downscaled [30,34,37,38,39,40]. Additionally, now they can perform toolpaths with arbitrary complexity, which widely enhances the applicability of that finishing operation in comparison with the initial vibratory-based ball burnishing process. Therefore, the shape and size of the pattern cells can now also be complicated and enhanced in comparison with the classical partially and fully covered RMRs, which are limited only to almost “hexagonal” or “tetragonal” shaped patterns [32]. There is a Russian state standard [41] (issued in 1981 in former USSR, but which is still active) that determines some parameters and characteristics of such RMRs types, obtained by using classical VBB process on rotational surfaces, according to the classification proposed in [32]. However, as of now, it could be considered obsolete regarding the abovementioned approaches to form RRs by BB-operations on different types of surfaces, using CNC-driven machine tools. Therefore, the classification issue of the resulting cells shapes and sizes of the formed RRs patterns using nowadays approaches for BB is current. The classification of such textures is not a simple task; however, since the cells’ shape and size depend not only on the CNC-driven toolpaths, but also on the plastically deformed material relocation beneath and around the intentionally produced “traces”, which remain on the BB-processed surface [32]. Therefore, only the graphical representation of the ball toolpath is not enough for acquiring the proper notion of the real shapes and sizes of the cells of the pattern, which will form onto the real surface.

Surface texture classification approaches are essential in the manufacturing industry for quality control and efficiency assessment of the production processes. It evolved from manual checks to automated systems using machine vision [42] and AI [43]. The main methods include traditional image processing and deep learning, each with pros and cons. Traditional techniques include histogram and statistical analysis [44] for quick, real-time texture info, and Gray Level Co-occurrence Matrix (GLCM) [45] to capture spatial pixel relationships. Fourier and wavelet transforms analyze frequency patterns and multi-scale textures [46], with wavelets decomposing images for detailed analysis. AI-based methods feature CNNs that teach hierarchical features [47], excelling with complex textures and lighting. Recurrent Neural Networks (RNNs) are used for temporal texture changes [43]. Some hybrid methods [48] combine traditional image techniques with models like Support Vector Machines (SVM) [49] and Artificial Neural Networks (ANN) [43,46,50] for better accuracy. Deep learning advances include architectures like Faster R-CNN [51] and YOLOv4 [52] for faster, precise detection, essential for real-time defect detection. Feature extraction methods like LETRIST [53] improve classification, especially for steel surface defects. Techniques, such as data augmentation, transfer learning, and ensemble methods enhance model performance by leveraging large, pre-trained datasets and improving generalization across different object categories [54,55,56]. These approaches are already widely applied in fields as autonomous vehicles [57,58], medical imaging [59,60,61,62], quality assessment of parts or assemblies [63,64], and security activities, demonstrating significant improvements in both accuracy and efficiency over traditional classification methods [65,66].

Also, there are lots of non-contact 3D surface scanning methods and devices developed as for now, which enable the accurate digital reconstruction of surface topographies, which are crucial for activities, such as quality assurance, reverse engineering, process control, etc. They offer some main advantages, such as higher spatial resolution, faster scanning speeds, improved topography accuracy, etc. Applying them to machine parts, especially those with shiny or reflective surfaces (which is the case with RRs obtained after BB), sometimes could pose a significant challenge for 3D scanning, due to scattering of visible, structured or laser light on such surfaces, which can create misleading topography features, thus complicating the accurate extraction of surface’s details. Nevertheless, many of these drawbacks of non-contact methods can be successfully overcome by applying additional software processing and filtering techniques of the (raw) scanned topographic data.

This gives us reason to research the possibilities to combine the advantages of the methods discussed above to define a new approach for automatic classification of RRs topographies obtained after CNC-driven BB-operations. The research is based on using synthetic and real captured 2D images datasets of RRs, and combinations between them. The synthetic 2D images are derived from RR’s 3D scanned topographies, which are additionally processed in Gwyddion and Blender software to get closer to these that are captured by a physical photo camera. Then, four pre-trained CNN models, i.e., VGGNet (Visual Geometry Group at the University of Oxford) [67,68], EfficientNetB0 (Microsoft) [69], MobileNet, developed by Google [70], and DenseNet (Cornell University and Tsinghua University) [71,72], which are additionally fine-trained with the synthetic and real RRs images datasets are evaluated to determine the recognition performances of different RRs types formed by CNC-driven BB-operations.

The goal of the present research is to determine which of the four pretrained CNN models gives the most correct recognition score, using 2D synthetic and real images datasets of RRs (or combination of them) for fine-tuning.

The paper is structured into several sections and subsections, which include an introduction, materials and methods, results and discussion, and conclusions. The Material and methods section consists of three subsections: 1. Formation of RRs by BB, 3D scanning, preprocessing with Gwyddion, and creation of artificial 2D images in Blender. 2. Selection, training, and evaluation of CNN models (DenseNet121, EfficientNetB0, MobileNetV2, VGG16), using datasets of synthetic, real captured 2D images and combinations of them; 3. Comparative evaluation of the four fine-tuned pre-trained CNN-models for automatic classification of real RRs, captured by physical photo-camera. The results and discussion section presents the performance metrics of the CNN models with various datasets and combinations of them, including confusion matrices, accuracy, precision, recall, and F1 scores. The discussion section addresses the effectiveness of synthetic vs. real images for training and recognition. In the conclusion section, the findings are summarized, emphasizing the potential of the tested CNNs for classifying properly RRs, based on synthetic datasets and with real images.

2. Materials and Methods

The different steps of the stages as algorithm diagram is presented in Figure 1. The research methodology includes two main stages, starting from the first one where RRs with distinguishable shapes and sizes are formed by BB operation, 3D scanning of their topographies, and generating 3D models, which are importable into Gwyddion (ver. 2.64) and Blender (ver. 4.2.3 LTS) software products, where 2D “synthetic” images are derived. At the second stage, the captured by the camera and generated symmetrically 2D images are organized into suitable datasets, which are used for the application of transfer learning via feature extraction and testing of four pre-trained CNN models (i.e., DenseNet121, EfficientNetB0, MobileNetV2, and VGG16), which can classify the RRs into predefined classes, based on the RRs textures types (see Figure 2a). In that stage, the trained CNNs are tested with real 2D images of RRs that are captured with a real photographic camera to determine which of them generalize the best the textures. A more detailed description of the methods and materials used is given below.

2.1. Stage 1. Forming RRs by BB and Acquiring Their Artificial 2D Images Description

At that stage four main steps are involved:

2.1.1. Forming Suitable RRs Patterns by BB Operation

The formed RRs should be with distinguishable cell patterns, regarding their shapes and sizes to define the different classes for the CNN models recognition. In the current research, eight different toolpaths are created, using the BB -process by the methodology described in [73], by which eight different patterns of RRs onto the top planar surface of a sheet that is made of austenitic stainless steel AISI 304 are formed. The dimensions of the sample plate are 110 × 60 × 4 mm, and those of the areas with formed RRs patterns are approximately 10 × 10 mm each (see Figure 2a). The BB operations for the eight RRs patterns were conducted with the following regime parameters: the deforming force is 750 N, and the feedrate is 500 mm/min. The amplitude of the near-sine wave toolpaths generated varies from 0.5 mm for patterns 1, 3, 5, and 7 to 2.5 mm for patterns 2, 4, 6, and 8 along the X-axis (see Figure 2a). Along the Y-axis, it has the following values: 0.785 mm for patterns 5 and 6, 1.57 mm for patterns 1, 2, 7, and 8, and 3.14 mm for patterns 3 and 4.

They were formed using a HAAS (Philadelphia, PA, USA) TM-1 CNC milling machine and a specially developed deforming tool [74], which is spring-based and using ball tool with diameter of 6 mm, made of hardened steel AISI-440. Mobil (ExxonMobil, Spring, TX, USA) DTE 25 oil was also used as a lubricant during the BB.

2.1.2. Optical 3D Scanning of the RRs Patterns

Each of the eight RR topographies were scanned by using an Alicona Focus X optical microscope (Bruker Corp., Billerica, MA, USA), as shown in Figure 2b with the following approximate settings: size of scanned area—11.4938 × 11.4938 mm, and contains 7945 × 7945 points, which means that the sampling distance between adjacent measurements along the horizontal and vertical directions is 0.00145 mm. The objective of type 800 WD17 was used. The contrast was set to 0.617362, and the exposure time was 1.057 ms/point. Before each scanning of RR’s areas, the test surface was cleaned using isopropyl alcohol to remove dust and other chemicals and/or mechanical residues. The obtained 3D scanned topographies in this way are exported and saved to Alcona’s default file format “al3d”.

Additionally, from the scanned 3D topographies by Alicona Focus X optical microscope, textures of the RRs are also derived to be used as surface textures in the Blender generated 2D images of the RR’s topographies.

2.1.3. Pre-Processing of the Raw 3D Scanned Topography, Using Gwyddion Software

In the present research, Gwyddion software [75] was used because it is a modular program for scanning probe microscopy (SPM) data visualization and analysis and is free and open-source software, covered by the General Public License (GNU). It incorporates a wide range of data processing capabilities, encompassing all standard statistical analysis, data leveling and correction, filtering, and grains processing functions. Gwyddion also supports many SPM data formats, including Alcona’s default file format “al3d”, which makes it very suitable for pre-processing the 3D scanned topographies of RR, because there is no need to reformat it from other third-party formats.

After the “al3d” file with 3D scanned topography is imported into Gwyddion it appears on 2D window as an image (see pos. 1 in Figure 3) which is sized by X, Y and Z. The height of the raw RRs scanned topography often has minimum values that are different from zero, which leads to difficulties in the actual heights of the relief’s estimation, so that shifting of the minimum height value to zero is performed as first processing operation (pos. 2). At the next step (pos. 3) the background is removed, using mean plane (or sometimes polynomial surface) subtraction, which eliminates slopes and/or other macro deviations in the shape of the topography due to errors in manufacturing and/or positioning the sample on the microscope for scanning. In the present research, the background remover is set as a polynomial of third degree. On the next step (pos. 4), the RR’s boundaries, where the relief cells’ shapes differ significantly from the interior of the burnished area, are removed by cropping the data. After that, RR’s relief areas have dimensions of approximately 9.5 × 9.5 mm and contain cells with homogeneous shape (pos. 5).

Sometimes, during the optical scanning process, some peaks or dales whose height sharply differs from the neighboring pixels could also be captured by the microscope. They are not part of the real burnished topography and are caused by incidental reflections from mechanical defects or surface contamination, which, however, could lead to 2D image interference at later steps of the process. That is why, if such defects occur, they should be filtered from the RRs topography by using the “Remove Spots tool” incorporated in Gwyddion (pos. 6). In the current research, a hyperbolic flatten method was used for creating data in place of the found “spots,” which uses information from outer boundary pixels of the selected area to interpolate the data inside the area. If it is necessary to filter the noise from certain regions or for the entire image of the scanned 3D topography, Gwyddion has a wide set of basic and convolution filters that can be applied (pos. 7) [75]. As a noise smoothing filter, a Gaussian filter with a 15 px size is used in the present work. Using Gwyddion’s tool for 3D view (pos. 8), it is possible to review and assess the scanned 3D topography, processed in that way before exporting it as STL file to further steps of the process.

2.1.4. Creating 2D “ARTIFICIAL” Images of RRs from Their 3D Topographies in Blender

The dataset was generated using 3D topographical models derived from confocal microscopy scans, which are pre-processed in Gwyddion. These topography representations provided high-resolution surface data, which were imported into Blender [76] for further refinement. Extraneous details from the scanned models were removed to isolate the core relief features, ensuring that only relevant topographical structures were retained. As a result of this preprocessing, the resulting 3D topographical models of the reliefs take on either a square or a rectangular shape (see Figure 4 and Figure 5), which requires modification to some elements of the imaging setup, regarding the virtual camera positions and movements.

The real RRs images capturing setup (see Figure 1) consists of a camera with high resolution (see Figure 1), which is mounted on a stand so that it can be placed over the top of the milling machine table to capture images of the processed RRs. As it might be expected, the camera lens will not always be in the same position and pointed at the same angle to the plane of the machine table. Over the course of the control cycles, the relative position of the camera and the captured RRs topographies will fluctuate symmetrically around the initial position to some extent. To account for this, the virtual camera in Blender is set to perform certain movements and tilts following certain trajecto-ries to simulate these symmetrical fluctuations. The automatically captured in this way, 2D images in Blender, which are used in training and testing the CNN model at next stage, will contain the possible symmetrical fluctuations of the real setup. Thus, the model will be more robust against these symmetrical positional fluctuations of the real capturing set-up of RRs images.

For capturing 2D image sets, four strategies were used in the present study, depending on the camera position and its motion trajectories around the 3D topographies. They are as follows:

• Moving camera, placed normally against the RRs topography (see Figure 4a). In that strategy, the camera is perpendicular to the RRs plane and moves along to the zig-zag type of trajectory, capturing image at interval of 0.5 mm. In that way, the shift fluctuations in the XY-plane were simulated.
• Moving the camera against the RRs topography, when it is fixed to single point (see Figure 4b). The strategy is like the previous one as crawling over the topography, but camera is fixed at the center of the RR and tilting symmetrically during following the zig-zag trajectory. This strategy was used to capture images for RRs with No. 2, 4, and 7 showed in Figure 2a.
• The camera followed a circular trajectory around a fixed point at the geometrical center of the RRs area (see Figure 5a), and captured images, ensuring multiple symmetric angular perspectives. For one full round of the camera circular orbit 100 images were captured, which means that a 2D image was gathered every 3.6 degrees. The radius of the circular orbit was set to 3 mm, which means that the virtual camera’s offset distance (i.e., the arc length) between two consecutive synthetic images was 0.19 mm approximately.
• The camera rotated in elliptical orbit around a fixed point at the geometrical center of the RRs area (see Figure 5b) above the 3D topography and captured images like in the previous strategy. Again, this strategy was applied only for RRs with № 2, 4, and 7 showed in Figure 2a to skip the areas where the burnished traces did not contain cells from their patterns.

All 2D image datasets were standardized with a resolution of 796 × 796 pixels, ensuring sufficient topography details for CNN model training. The camera was positioned at H = 300 mm above the RRs topography plane, and the focal length was set to 1.8 m, to maintain a consistent field of view across all captures in the four different strategies.

The synthetic images database generated has two different components, where the first part consisted of textures models that were built-in Blender, while the second part used textures that were derived from the microscope topographies scanning process (i.e., exported as files in “tga” file format [77]). The rendering engines and lighting conditions were one and the same for both parts.

Three types of rendering engines available in Blender were used for 2D image generation:

• Workbench, this is a fast-rendering engine without photorealistic effects.
• Eevee, this is a lightweight ray-tracing engine, offering a balance between performance and realism.
• Cycles, this is a physically accurate ray-tracing engine producing high-fidelity images, albeit at a higher computational cost.

Two different types of lighting were used, depending on the rendering engine:

• High Dynamic Range Imaging (HDRI) lighting environments were used to introduce realistic illumination and reflections, simulating real-world optical conditions [78]. The used HDRi add-ones are as follows: Hanger Exterior Cloudy, Qwantani Moonrise, Qwantani Night, Lakeside Sunrise.
• The following built-in Blender’s Studio Lights options for lighting were employed: Default, basic.sl, rim.sl, studio.sl.

For all the lightings the following rotation angles of the source around the object were used: 10, 90, and 165 degrees. In addition, the MatCap check_rim_light.exr [79] was also used to achieve a high contrast between the peaks and dalles of the RRs. The imaging setup was repeated for each of the eight 3D topography models.

The second part of the database uses the same camera movements but is rendered only using the following rendering engines: EEVEE and Cycles. When the resulting 2D synthetic images should contain the scanned textures derived from the Alicona microscope, these rendering engines have to be used because they provide more photo-realistic results.

2.1.5. Real Images Capturing Setup Description

At this stage, the trained CNN model was used to perform automatic classification operations of real-formed RRs, which were captured by a photo camera installed above the work zone of the CNC-milling machine. The testing setup (see Figure 1) consists of a full frame shutter and electronic exposure time-controlled camera, type “a2A2600-64ucPRO,” with an objective, type “C11-1220-12M-P” (Basler, Ahrensburg, Germany), and a polarizing filter, type “UX II CIR-PL SLIM 46MM” (HOYA, KentFaith, Shenzhen, China), mounted in front of the objective. They were used for capturing images from the top surface of the test part where the RRs were formed without glaring. The camera and its elements were mounted on the adjustable combi-boom stand (Videndum Media Solutions Spa, Cassola, Italy), which allowed a versatile positioning at suitable distance of the camera over the top side of the milling machine table.

The images taken with the camera were cropped so that only regions containing real formed RR were retained. The color images were saved in PNG format. Before loading into the model, the images were resized to dimensions 80 × 80 pixels.

The total number of camera images that were captured by the camera in the present research was as follows: Pattern 1—255 images; Pattern 2—269 images; Pattern 3—274 images; Pattern 4—255 images; Pattern 5—244 images; Pattern 6—260 images; Pattern 7—252 images; Pattern 8—256 images. From the images captured 70 % were used for training of the model, 20 % of them were used for the evaluation of the model performance and the last 10% of the images were used for testing the models. For the feature extraction as well as the gradual unfreezing transfer learning strategies the pre-trained CNN models were compiled as mentioned in Section 2.2.3 and were used for automatic recognition of real captured RRs images by the camera (see Figure 1).

For illustration, two sample images for each type of RRs texture included in dataset for CNN training are shown in

Table 1. Samples of unaugmented images that were used in datasets for the CNN-models training.

	RR Texture 1	RR Texture 2	RR Texture 3	RR Texture 4	RR Texture 5	RR Texture 6	RR Texture 7	RR Texture 8
Generated by Blender
Derived from the Alicona scanning microscope
Derived from the physical photo camera

. The table presents the rendered in Blender and captured by physical camera images. Details about the number of images derived and times needed for that for the different types of datasets, used in evaluated CNN models training, are summarized in

Table 2. Information regarding image database creation.

Type of the Image DB Used for CNN—Model Fine Tunning	The Number of All Rendered And/Or Captured Images	Rendered Images in Blender Software	Rendered with Textures Captured by Alicona Microscope	Captured Images by the Physical Photo-Camera	Overall Time for Dataset Preparation, [min]	Average Time for a Single Image Extraction, [s]
Only Blender rendered images	34,037	34,037 (100%)	-	-	621	1.09
Only images rendered with textures from Alicona microscope	3232	-	3232 (100%)	-	512	9.50
Only captured images by the physical photo-camera	2065	-	-	2065 (100%)	452	13.13
Mixture of Blender rendered images + textures from Alicona microscope	37,269	34,037 (91%)	3232 (9%)	-	1133	1.82
Mixture of Blender rendered images + textures from Alicona microscope + physical camera captured images	39,334	34,037 (87%)	3232 (8%)	2065 (5%)	1585	2.42

.

During the fine-tuning part of the study to enhance the generalization capabilities of the model and simulate real-world imaging imperfections, two distinct augmentation strategies were applied during training. These were implemented, using custom aug-mentation pipelines and integrated with the TensorFlow tf.data.Dataset API. In the first augmentation method, Gaussian blur was applied to each image in the dataset. For each image, a random standard deviation between 0.5 and 2.5 was sampled from a uniform distribution. This value was used in a Gaussian filter, applied independently to each image channel via the Gaussian filter function from the Scipy library. The blurred image was obtained because of convolving the original image with the Gaussian kernel. This form of augmentation introduces optical distortions that emulate defocus or motion blur, in order to further simulate real imaging conditions from which the model would have to classify image. By applying such a filter, the models should be encouraged to learn RRs topography features more robustly.

The second method combined Gaussian blur with additive Gaussian noise. Initially, the same Gaussian blur operation was applied. Subsequently, each pixel in the blurred image received additive noise following a Gaussian distribution with a mean of 0 and variance of σ, where σ is a random value between 2 and 12. The resulting image was clipped to ensure pixel values remained within the valid range between 0 and 255. This composite transformation simulates more severe image degradation, such as low-light conditions or sensor-related noise, thereby forcing the model to develop a stronger invariance to this type of quality degradation.

The combined Gaussian blur and noise augmentation was applied to the Blender rendered images with built-in textures as there were fewer camera artifacts in this dataset, while the blur only augmentation was applied to the microscope texture derived images as the textures already capture surface artifacts that were present on the physical object to be imaged with the real camera.

Both augmentation strategies were applied to training, validation, and test datasets using parallelized map operations with TensorFlow’s py_function. Augmentations were performed dynamically during the training process, thus maximizing variability across epochs and avoiding dataset redundancy. After the initial evaluation of the feature extraction transfer learning strategy based on the training of the models on the unaugmented artificial datasets two CNN models that show best scores were chosen for fine-tuning and further refining using the gradual unfreezing strategy. The augmented dataset was only used for fine-tuning of the CNN models in that strategy.

2.2. Stage 2. Training and Testing the CNN Models

In the present comparative analysis, the methodology utilizes the feature extraction capabilities inherent in four pre-trained image classification CNN architectures, pre-trained on the ImageNet dataset. They have the following characteristics:

2.2.1. Description of the Evaluated CNN Architectures

Four different pre-trained CNN models, i.e., DenseNet121, EfficientNetB0, MobileNetV2, and VGG16 were subjected to evaluation. Each model was originally trained on the ImageNet dataset, containing millions of labeled images across 1000 categories. This prior training enables the networks to extract a hierarchy of visual features, ranging from low-level edges and textures in the initial layers to more complex, abstract patterns in the deeper layers. These models were loaded using the TensorFlow.Keras (ver. 2.6) API within the Spyder 5.5.6 environment.

The four models, selected for the initial task of transfer learning via feature extraction, for the task of classification of real RRs images of ball-burnished surfaces, are as follows:

• VGG16—this model of architecture utilizes a simple and uniform architecture with stacked 3 × 3 convolutional filters. Its depth allows the learning of highly abstract, discriminative features critical for identifying subtle surface variations, serving as a robust baseline for evaluating newer architectures.
• DenseNet121—features dense connections in which each layer receives input from all preceding layers and contributes to all subsequent ones. This design promotes gradient flow and feature reuse, enhancing learning efficiency and detail preservation while maintaining a compact parameter footprint.
• EfficientNetB0—employs compound scaling to uniformly adjust network depth, width, and resolution. This leads to an optimal balance between accuracy and computational efficiency, making it ideal for scenarios with limited processing power.
• MobileNetV2—is designed for deployment in mobile and embedded systems. This model uses depth wise separable convolutions to significantly reduce computational complexity while maintaining competitive performance.

The models chosen in the study were modified for the task of classifying the eight distinct BB RR’s by using models that had been pre-trained on the ImageNet dataset, used them as a backbone and added a Global Average Pooling layer, used to reduce spatial dimensions to a fixed-length feature vector and a Densely connected classification head, including a softmax layer with eight output units corresponding to the texture classes.

2.2.2. Transfer Learning Strategies Used

The study was conducted using two different transfer learning strategies, which were as follows:

Feature Extraction Strategy

In this approach, the pre-trained base model was frozen, and only the custom classification head was trained. Early stopping, based on validation metrics, was employed to prevent overfitting. This is a simpler strategy that is useful when applying transfer learning to pre-trained models by leveraging the features that the models have learned during their original training to be able to classify new objects into new classes.

Gradual Unfreezing Strategy

This strategy was used for fine-tuning the models that were chosen, based on the results of the feature extraction strategy. This strategy was applied in a multi-phased approach which allows progressive adaptation of the pre-trained weights to the target domain. The implementation for both models chosen is in three phases, but due to the differences in the model used as a backbone, the fine tuning differs in the second phase. For both models in the first phase only the classifier head was trained.

For DenseNet121, the second phase selectively unfroze only the batch normalization layers, which allowed the internal statistics to adapt without altering convolutional feature detectors, while for the VGG16 based model, the second phase consisted of unfreezing the last block of the model. The final phase for both models involved unfreezing model blocks, where in the case of DenseNet121 only the last block was unfrozen, while for the VGG16 model the last two blocks were unfrozen.

A smaller learning rate of 1 × 10⁻⁵ was used for fine-tuning during the second and third phase of the gradual unfreezing of the VGG16 based model as well as the third phase of the DenseNet121 model.

2.2.3. Model Compilation

The models studied were compiled using the Adam optimizer with different learning rates for the different transfer learning strategies. During the application of the feature extraction transfer learning strategy the models were compiled with an initial learning rate of 0.001. During the use of the gradual unfreezing transfer learning strategy the models were recompiled with a reduced learning rate of 1 × 10⁻⁵ to preserve previously learned representations while fine-tuning deeper layers. The loss function used for all the models was Sparse Categorical Crossentropy, suitable for multi-class classification with integer labels.

2.2.4. Training Settings

The training of the different model used a batch size of 64 and an Early stopping callback set to monitor the validation loss of the model, with a patience of 9 epochs. The best weights for the model were restored.

The number of epochs differed on the base of the applied transfer learning strategy: for the feature extraction strategy the models were trained for 100 epochs on the respective datasets. For the Gradual unfreezing transfer learning the models were trained for 50 epochs during each phase—a total of 150 epochs.

During the initial feature extraction-only transfer learning the four models were trained in the following ways: training only on the non-textured Blender images, training only on the textured images, training on both the textured and non-textured images, training on 70 % of the available camera images, training on a mix of all the available training images—non-textured, textured, and camera.

When using the gradual unfreezing transfer learning strategy, the models were trained on the augmented non-textured Blender images, the augmented textures as well as on the mixed augmented texture and non-textured Blender images.

For all the different datasets the tested dataset was split into 70% training, 20% validation, and 10% testing subsets with the same random seed in order to ensure adequate and reproducible data for learning while retaining reliable sets for performance evaluation and testing. To achieve symmetrical and comparable results, the performance of the models when tested on the real images dataset, used the same testing subset as the models that were trained on the real camera images dataset.

The PC used for both the Blender image rendering and training of the CNN has following configuration: OS: Windows 10 (10.0.19045), CPU: AMD Ryzen 5 5500, Cores: 12, RAM: 32 GB, GPU: NVIDIA GeForce GTX 970, 4096.0MB VRAM, CUDA Version: 64_113. The software platforms and tools used were, as follows: Python Version: 3.9.15; tensorflowgpu: 2.6.0; numpy: 1.23.5; scikit-learn: 1.6.1; scipy: 1.13.1; matplotlib: 3.9.1; pandas: 2.2.3.

2.2.5. Evaluation of the CNN-Models

The performance of the CNN-models was evaluated using accuracy, macro-precision, macro-recall, and macro-F1 score metrics [79], as follows:

• Accuracy (${ACC}_A$) is the percentage of correctly classified images with respect to the total number of classified images. The accuracy is calculated as follows:

  $${ACC}_A={\displaystyle \frac{{TP}_A}{N_A}},$$

  (1)

  where ${TP}_A$ (i.e., true positive) is the number of correctly classified texture types, N is the total number of assessed textures from type A.
• Precision measures of how much of the predictions of the trained model regarding a given texture are true, with respect to all the predictions. It is calculated as follows:

  $${PR}_A={\displaystyle \frac{{TP}_A}{{TP}_A+{FP}_A}}$$

  (2)

  where ${TP}_A$ is a measure of how many of the predictions of the trained model regarding a given texture type A are true, ${FP}_A$ is a measure of how many of the predictions of the trained model regarding a given texture type A are misclassified as another texture type.
• Macro-precision ${PR}_{mac}$ is a metric that averages the precision of the model for each pattern over all the patterns. It is calculated as follows:

  $${PR}_{mac}={\displaystyle \frac{\sum _i^{N_A}{PR}_i^A}{N_A}}$$

  (3)

  where ${Pr}_i$ is the precision of the given RR’s pattern type; $N_A$ is the number of patterns types (i.e., $N_A=8$ in present case).
• The recall ${RCL}_A$ is a measure of how many images that have a certain texture A, were correctly identified as being of their correct texture’s class, and is calculated as follows:

  $${RCL}_A={\displaystyle \frac{{TP}_A}{{TP}_A+{FN}_A}}$$

  (4)

  where ${FN}_A$ (or “false negative”) is the number of textures that are classified as a given texture class A, but they are misclassified actually in that class by the model.
• Macro-recall is a metric that averages the recall of the model for each pattern over all of the patterns, and it is calculated, as follows:

  $${RCL}_{mac}={\displaystyle \frac{\sum _i^{N_A}{RCL}_i^A}{N_A}}$$

  (5)

The F1-score is the harmonic average of the precision and recall, which calculates as follows:

$${F1}_A^{score}={\displaystyle \frac{2}{\frac{1}{{PR}_A}+\frac{1}{{RCL}_A}}}$$

(6)

Macro F1-score is a metric that averages all the F1-scores of the model for each pattern’s type over number of all of the patterns.

$${F1}_{mac}^{score}={\displaystyle \frac{\sum _i^{N_A}{F1}_{A_i}^{score}}{N_A}}$$

(7)

All trained CNN models classify RRs images into eight classes (i.e., RRs distinguished each other’s patterns), allowing seven incorrect and just one correct classification. It can classify accurately or misclassify the test images, captured by the real camera. Therefore, the CM can be used as a tool for the total number (and the percentage) calculation of the images that were classified correctly or incorrectly by the given fine trained CNN model (see Appendix A). Each row of the CM corresponds to the actual RRs pattern class instances. For example, the first row of CM will show how many instances of class “RRs Pattern 1” were predicted as RRs Pattern 1, RRs Pattern 2, etc. Each column of CM represents the predicted by the trained model class instances. In the present case, the first column of the CM will show how many RRs patterns instances were predicted as class RRs Pattern 1, regardless of their actual class. The diagonal elements of CM indicate correct predictions, where the predicted RRs patterns match the actual RRs class. Off-diagonal elements of the CM represent misclassifications, where the predicted RRs class does not match the actual class. The integers in the CM cells show the number of images that are recognized in a given class. Also, they are presented in the CM cells as a percentage of the total number of images for the given class. The higher the percentage calculated in each cell, the darker blue its color is.

3. Results and Discussion

3.1. Performance Assessment of the Evaluated CNN Models When Tested with RRs Images from the Same Dataset with Which They Have Been Finely Trained

These assessments are conducted only on resulting confusion matrices for the four evaluated CNN models, which are given in Appendix A.1, Appendix A.2 and Appendix A.3. As can be seen from Appendix A.1, where the four CNN models underwent transfer learning based on the feature extraction strategy, using a subset of the non-textured Blender images dataset, and were tested with images of the same subset. Three of them (i.e., DenseNet121, MobileNetV2, and VGG16) performed very well, unlike the EfficientNetB0 model, which incorrectly recognized all tested images from the different patterns only as RRs pattern 4. When trained with subsets of textured Blender images (see Appendix A.2) and real captured images of RRs (see Appendix A.3) the performing results of the four models were very close for DenseNet121, MobileNetV2, and VGG16 models, whereas the EfficientNetB0 model again demonstrated wrong classification of the different RRs patterns only as pattern 1 (see Figure A2b) or only as pattern 2 (see Figure A3b). This suggests that EfficientNetB0 struggles to generalize its decision boundaries effectively under the feature extraction paradigm, potentially caused by overfitting synthetic images due to its heavy regularization and deeper architecture.

This means that three of the four evaluated models show good ability to properly classify the RRs formed by BB in the pre-defined eight patterns classes, when tested with the same type of datasets by which they had been finely tuned. This gives us reason to exclude the EfficientNetB0 model from further assessments as unsuitable for RRs classification.

3.2. Performance Assessment of the Three Evaluated CNN Models When Finely Trained with Different Datasets and Tested with Real Captured RRs Images

The results for the performance of the three tested CNN models, using the macro metrics, given in Section 2.2.4, are shown in

Table 3. Summarized results from the performance metrics of the investigated CNN-models using the feature extraction transfer learning strategy.

Type of the Image Dataset Used for CNN—Model Fine Trained by Transfer Learning Strategy	Metrics	CNN-Model DenseNet121	CNN-Model MobileNetV2	CNN-Model VGG16
Rendered non-textured images in Blender	Accuracy	29%	24%	29%
	Precision	26%	21%	21%
	Recall	33%	26%	29%
	F1-score	29%	23%	24%
Rendered with textured images from Alicona microscope	Accuracy	44%	26%	31%
	Precision	30%	29%	25%
	Recall	44%	28%	29%
	F1-score	36%	28%	27%
Rendered in Blender non-textured + textured images from Alicona microscope	Accuracy	38%	31%	32%
	Precision	32%	36%	30%
	Recall	41%	31%	32%
	F1-score	36%	33%	31%
Rendered in Blender non-textured + textured images from Alicona microscope + physical camera captured images mix	Accuracy	91%	65%	96%
	Precision	92%	77%	96%
	Recall	91%	63%	96%
	F1-score	91%	69%	96%
Captured images from the Physical Photo-Camera	Accuracy	98%	98%	99%
	Precision	98%	98%	99%
	Recall	98%	98%	99%
	F1-score	98%	98%	99%

and

Table 4. Summarized results from the performance metrics of the investigated CNN-models using the gradual unfreeze transfer learning strategy.

Type of the Image Dataset Used for CNN—Model (Fine-Tuned by Gradual Unfreezing Transfer Learning Strategy)	Metrics (Macro-)	CNN-Model DenseNet121	CNN-Model VGG16
Augmented rendered non-textured images in Blender	Accuracy	44%	31%
	Precision	46%	33%
	Recall	44%	31%
	F1-score	45%	32%
Augmented rendered with textured images from Alicona microscope	Accuracy	43%	33%
	Precision	49%	36%
	Recall	42%	33%
	F1-score	45%	35%
Augmented rendered in Blender non-textured + augmented textured images from Alicona microscope	Accuracy	40%	30%
	Precision	44%	39%
	Recall	40%	30%
	F1-score	42%	34%
Augmented rendered in Blender non-textured + augmented textured images from Alicona microscope + physical camera captured images	Accuracy	92%	95%
	Precision	92%	94%
	Recall	92%	95%
	F1-score	92%	94%

. The corresponding confusion matrices of models’ performance are also given in Table 4 and Appendix B.1, Appendix B.2 and Appendix B.3.

Analysis of the result when utilizing the feature extraction strategy is shown in Table 3. The results from the table reveal certain trends in macro-precision and macro-recall across different models. DenseNet121 consistently exhibits high macro-precision across nearly all datasets, indicating its effectiveness in minimizing false positives across all classes. In contrast, MobileNetV2 demonstrates a notable macro-precision of 36% on the non-textured + textured dataset, despite achieving only 31% accuracy. This discrepancy suggests a precision skew, likely due to the model’s selective correct predictions in high-certainty classes.

Macro-recall trends align with the findings from analyzing macro-precision, as DenseNet121 also leads to high macro-recall performance across most datasets. This stability in metric performance indicates a consistent ability to cover all classes effectively. VGG16, on the other hand, demonstrates a balanced relationship between precision and recall, which would be attributed to its generalization capabilities, especially when real images are incorporated into the training process.

Based on this analysis, the models DenseNet121 and VGG16 were chosen to be tested with the gradual unfreezing transfer learning strategy. The results of the transfer training using augmented synthetic and mixed datasets are shown in Table 4.

When analyzing the gradual unfreezing strategy, as illustrated in Table 4, the results indicate that macro-metric have slightly improved in comparison with the feature extraction strategy consistently across all synthetic datasets. For instance, DenseNet121 achieved F1-scores of 45% on augmented non-textured dataset, 45% on augmented textured dataset, and 42% on the combined augmented nontextured + textured datasets, outperforming the VGG16 in all synthetic images’ datasets context. This performance advantage is likely attributable to DenseNet121’s dense connectivity, which facilitates superior gradient propagation and feature reuse—an essential factor when dealing with domain-shift noise. VGG16, while slightly trailing in synthetic contexts, still showed improved F1-scores of 32%, 35%, and 34%, respectively, underlining its robustness when trained with gradual layer adaptation strategy.

Notably, when real camera images were included in the training dataset along with symmetrically augmented synthetic datasets, both models achieved high F1-scores, with DenseNet121 reaching 92% and VGG16 achieving 94%. These high values confirm the crucial role of real image data in anchoring the model’s feature representations to the target domain. The application of gradual unfreezing thus appears to enhance macro-precision, macro-recall, and macro-F1 across the board, confirming that fine-tuning deeper convolutional layers with a decreased learning rate, enables the networks to adjust their internal feature representations more effectively toward the real domain.

Using augmented synthetic datasets along with the gradual unfreezing transfer learning strategy seemed to yield measurable performance improvements, particularly in macro-recall and macro-F1 score metrics. The augmentation process likely contributes to regularizing the learning experience by diversifying the feature space, thus enhancing the robustness of CNNs to real-world imperfections, including motion blur, poor focus of the physical camera, and photo sensor noise. The greater depth of learning as well as the augmentation-induced noise and/or blur that serve to mimic camera imperfections, allow the model to extract more generalizable patterns. Nevertheless, the current methodology cannot entirely bridge the domain gap as shown by the inclusion of real images when the accuracy of the models ranges from 92% to 96%, as also reflected in both macro-accuracy and macro-F1 scores significant increase when the dataset includes symmetric mixture of synthetic images, derived from Gwyddion, Alicona textured images and real photo images of RRs (see Table 4).

3.3. Evaluation of the Methodologies Employed for the Preparation of RR’s Image Datasets in Relation to the Duration of Time Required

This comparative evaluation is based on comparing the times needed for single image extraction from the different datasets, exhibited in Table 2 using the approach explained in Section 2.1.2 and Section 2.1.4. As can be seen from the table, the Blender based approach has the shortest time for single image extraction (approx. 1.09 s/image), relative to the number of synthetically generated images. The second result of 9.5 s/image demonstrates the approach, based on the textured images dataset derived by Alicona’s microscope software. Here, the increase in the image extraction time can be explained with the higher resolution of the topography scanned textured 2D image, which requires more computing power for the extraction operation in comparison with the non-textured approach. As expected, the slowest approach is the extracting of images derived from the real camera, which takes 13.13 s per single image, because the extraction process is related to some manual labor by the operator.

In cases of mixture datasets, the average times per single image extraction are shortened significantly (up to 2.42 s/image) due to the comparatively large proportion of the synthetic images in comparison with the real captured ones in the mixed dataset (see Table 2). The obtained results show that even a comparatively small percentage (5%) of added real images of RRs along with the synthetic extracted ones, included in the mixed dataset for training the tested four CNN models significantly increases their efficiency of properly recognizing the types of RRs (up to 96% for VGG16 model), according to the efficiency metrics calculated results (see Table 3 and Table 4) and the confusion matrices obtained (see Appendix B.5). Therefore, the proposed approach demonstrates some advantages in terms of shortening overall time to fine tuning of two of the researched pre-trained CNNs, using fast generated synthetic 2D images from 3D scanned real topographies of different RRs patterns.

4. Conclusions

The AI technologies, when integrated with synthetic image generation of RRs patterns formed by BB-operations, demonstrate promising capabilities for different reliefs textures classification. The proposed workflow is based on two-stage methodology, beginning with the creation of RRs using BB-operations, followed by 3D scanning and preprocessing surface topographies via Gwyddion and Blender to deriving 2D synthetic images that simulate real-world symmetrical fluctuations of the real photo camera. Four pretrained CNNs models (i.e., DenseNet121, EfficientNetB0, MobileNetV2 and VGG16) are subjected to comparative assessments to evaluate their abilities to properly classify eight different RRs in eight distinguished classes. The transfer learning strategies employed, including feature extraction and gradual unfreezing, are effective for this classification task. The gradual unfreezing strategy allows for a more refined adaptation of the pre-trained models to the irregularities of the synthetic and real datasets. This approach not only enhances the internal feature representations but also leads to improved performance metrics when tested with augmented data subsets. The experimental results demonstrate that DenseNet121 and VGG16, when fine-tuned using gradual unfreezing, perform exceptionally well—with models even achieving accuracy rates up to 96% (VGG16) when comparatively small percentage of real captured images are included in the training dataset.

One central conclusion is that the quality of the textures used for models training plays a pivotal role in the performance of the CNN models. Specifically, synthetic datasets rendered with Blender alone yielded faster image extractions and comparable classification accuracies, while the inclusion of a minor percentage of real images improved the models’ capacity to generalize to the target domain. This finding underscores the need for an optimal balance in dataset composition, where even a small proportion of real image data intermingled with synthetic data can bridge the domain gap, leading to a notable enhancement in macro-precision, macro-recall, and overall F1-scores.

The study finds out that the hybrid approach of combining rapid Blender-generated images with more nuanced textured images extracted from high-resolution scans, accelerates the overall training process and reduces manual intervention. As seen from Table 2 the time efficiency of mixed synthetic and real image extracted dataset (2.42 s/image) compared to real captured image extraction (13.13 s/image) is an important advantage of the proposed hybrid approach.

The future work will focus on further refining these mixed datasets and exploring more sophisticated data augmentation techniques, including more sophisticated symmetrical toolpaths and orbits of the virtual camera to close the remaining gaps between synthetic and real domains, thereby robustly supporting practical implementations in quality control and surface analysis applications in manufacturing of RRs by BB-operations.

Overall, the main contribution of the present work is to provide a framework for the integration of synthetic and real data in RRs surface texture automatic classification, based on the exploration and comparative analysis of different transfer learning strategies applied to four pre-trained CNN models to determine those that best generalize the tested RRs topography’s textures. The combined contributions of enhanced data synthesis, strategic transfer learning adaptations, and effective data augmentation collectively push forward current approaches in automating and optimizing manufacturing quality control processes.