Discovery and Classification of Defects on Facing Brick Specimens Using a Convolutional Neural Network

Abstract

In recent years, visual automatic non-destructive testing using machine vision algorithms has been widely used in industry. This approach for detecting, classifying, and segmenting defects in building materials and structures can be effectively implemented using convolutional neural networks. Using intelligent systems in the initial stages of manufacturing can eliminate defective building materials, prevent the spread of defective products, and detect the cause of specific damage. In this article, the solution to the problem of building elements flaw detection using the computer vision method was considered. Using the YOLOv5s convolutional neural network for the detection and classification of various defects of the structure, the appearance of finished products of facing bricks that take place at the production stage is shown during technological processing, packaging, transportation, or storage. The algorithm allows for the detection of foreign inclusions, broken corners, cracks, and color unevenness, including the presence of rust spots. To train the detector, our own empirical database of images of facing brick samples was obtained. The set of training data for the neural network algorithm for discovering defects and classifying images was expanded by using our own augmentation algorithm. The results show that the developed YOLOv5s model has a high accuracy in solving the problems of defect detection: mAP0.50 = 87% and mAP0.50:0.95 = 72%. It should be noted that the use of synthetic data obtained by augmentation makes it possible to achieve a good generalizing ability from the algorithm, it has the potential to expand visual variability and practical applicability in various shooting conditions.

1. Introduction

In the current situation, enterprises of the construction complex are striving for automation and intellectualization of routine work, creating a common digital space. Flexible and intelligent systems are being created that carry out automated management of the life cycle of building materials and structures, which can significantly improve the quality of production and eliminate most defective products [1,2]. Modern methods of artificial intelligence (AI) play a significant role in increasing the level of automation of the construction industry. AI algorithms make it possible to implement a model that imitates the cognitive functions of an expert, whose “eyes” are computer vision algorithms [3]. Due to computing power, this technology is able to store, process, systematize, visualize, and analyze unstructured information, including in real time [4,5,6]. The implemented flaw detection technologies based on computer vision algorithms show a high level of reliability and stability, which corresponds to, and in some cases even exceeds, the expert one. Intelligent algorithms prevent the spread of defective products, help determine the cause of a particular damage, and give a timely management response based on the analytical findings of the expert system [5,7].

To detect defects in building materials, structures, and buildings made of innovative supplies, many different methods are used, including artificial intelligence [8]. At the same time, the types of materials, structures, as well as the results obtained are quite diverse. A fairly common method for detecting damage and destruction of various building materials and structures is the acoustic emission method [9]. It is used in scenarios heavily polluted by residual noise and helps to predict and characterize the type of failure. Rolled steel defects were detected using advanced models YOLOv5s-GCE [10], adaptive bounding box annotation and lightweight MobilenetV2 [11], MSFT-YOLO based on a single-stage detector [12], and ACA-Net (adaptive convolution and anchor) [13]. All models showed an improvement in accuracy over the original model. In addition, a random forest and artificial neural network with an error of less than 0.6% was used to detect cracks and determine their severity in steel beams [14]. A comparison of different time-of-flight (ToF) algorithms and a ToF self-diagnostic approach based on a Defect Peak Tracking Model (DPTM) has been successfully applied in [15] to detect and analyze pipeline defects. Additionally, the use of computer vision in detecting defects in railway tracks has already been studied [16,17]. Machine learning methods have been successfully applied in order to detect and monitor surface defects of photovoltaic panels (PV) [18,19]. The improved MobilenetV1-YOLOv4 network has been successfully used to detect insulation defects and thereby improve the security of power lines [20]. The work [21] describes the use of a model for monitoring the state of a conveyor belt structure using machine learning and connecting to the Internet of things, as well as an ultra-high-frequency RFID sensor [21]. The YOLO-RFF method, based on an extended field of sensations and feature fusion, has proven itself in the accurate and high-speed detection of various industrial defects [22]. The ResNeXt + YOLO v3, Inception v3 + YOLO v3, and YOLO v3 models, which have an integrated structure, were studied in order to categorize and locate defects in ceramic substrates to reduce the human resource costs associated with manual re-inspection [23].

Machine learning methods have shown their high accuracy in detecting and monitoring defects in buildings (multiscale feature fusion method: 3ScaleNetwork deep convolutional neural network, local binary pattern, simple linear iterative clustering) [24], retaining walls (machine vision combined with drones) [25], bridges (improved YOLOv3 network, combining high- and low-resolution element images) [26], and pavements (model BV-YOLOv5S (BiFPN Varifocal Loss-YOLOv5S)) [27]. There are quite a variety of methods for detecting and monitoring defects on the surface and inside concrete [28,29,30,31]. Compared to manual instrumentation methods such as ultrasonic [31], non-destructive non-contact inspection methods using thermal imaging and AI are preferable to control the depth of cracks in concrete [29,30].

Of great importance for this study was the already available information on the research and application of various methods for detecting, measuring, and monitoring defects in brick and brick masonry structures [32,33,34,35,36,37]. A sensor technology based on smart bricks—piezoresistive brick sensors for a structural health monitoring system—has been used to monitor deformations and early detection of cracks in shear brickwork [32]. Non-destructive electro-acoustic methods (resonant pulse method or ultrasonic pulse method) were applied in order to discover and analyze defects in the internal structure of bricks. The most successful model of the resonant pulse method had a success rate of up to 85% [33]. Using a break of ordinary quartz optical fibers (6–12 pieces) attached to the surface of the masonry with epoxy adhesive to detect and monitor structural cracks in brickwork, with a detection probability tending to one, allowed us to determine the residual life of structural elements of a building exposed to natural and man-made anomalous phenomena [34]. Automatic brick segmentation and image crack quantification using machine learning was implemented using various AI methods. The accuracy of checking the brick detection model was 96.86% [35]. The DeepLabV3 model for efficient automatic crack segmentation in masonry and actual measurement of crack length in masonry has outperformed previous state-of-the-art crack segmentation models according to the authors of [37]. The Convolutional Neural Network (CNN) model showed better results in detecting cracks in brickwork images taken from the Internet (81.0% accuracy) than in laboratory images (61.5%). The features were built from grayscale image fragments that were focused on dark areas indicating the appearance of cracks [36].

The work [38] presents an improved SSD (Single Shot MultiBox Detector) algorithm, which is used to detect cracks in the workpiece. The algorithm can effectively detect small cracks; the image detection accuracy is over 80%. In recent years, systems for recognizing pavement cracks based on computer vision have become widespread. Thus, in this study [39], a technology for autonomous identification of road cracks is proposed. It is noted that the heterogeneity of the intensity of cracks, and the complexity of the background, complicate the detection. The researchers concluded that an SSD is best suited for this task. In [40], Faster R-CNN and Mask R-CNN are compared and analyzed. The results show that the joint training strategy is very efficient and can ensure that both Faster R-CNN and Mask R-CNN perform the crack detection task with over 90% accuracy when trained with more than 130 images, and it can outperform YOLOv3. However, the collaborative learning strategy causes a decrease in the performance of the bounding box found by Mask R-CNN. In [41], for the detection of cracks in concrete structures, the trained AlexNet model, the prediction accuracy was 99.9%. AlexNet neural networks used a new non-linearity activation function, ReLU, which avoided the gradient disappearing problem. By analyzing the considered architectures of neural networks, it can be noted that the YOLO class operates in a wide range and has high robustness and, when using the SSD network, both large and small objects are determined in one network run; a series of architectures based on R-CNN is not always applicable when detecting objects in real time.

Based on the results of the literature review, it can be summarized that AI methods used throughout the entire life cycle of construction projects are increasingly relevant and popular in assessing the properties, defects, and residual life of building products and buildings, as well as their structures [42,43,44]. AI is needed where it is necessary to accelerate and stabilize the processes of intellectual activity.

YOLO (You Only Look Once) is a neural network architecture designed to detect objects in an image. YOLOv5 “refers to the One-stage detector architecture—an approach that predicts the coordinates of a certain number of Bounding boxes with the results of classification and the probability of finding an object” [45]. Accordingly, “YOLOv5s, YOLOv5m, YOLOv5l and YOLOv5x mean small (s), medium (m), large (l), and very large (x) models. Each model has its pros and cons, but ultimately the differences lie in their complexity, performance, and overall accuracy” [46].

A lack of research has been found on the identification and monitoring of various defects in facing bricks that occur at various stages of its manufacture. The scientific novelty of the research lies in:

-

application of the developed augmentation algorithm to increase the accumulated database of images of facing bricks [47];

-

development of an algorithm that allows the detection of foreign inclusions, broken corners, cracks, and uneven coloring, including the presence of rusty spots on facing bricks;

-

creation, optimization, and testing of the computer vision model based on the CNN YOLOv5s.

In this paper, we consider the solution to the problems from the field of building flaw detection using the computer vision method. The main goal was to use the YOLOv5s convolutional neural network to discover and classify various violations of the structure and appearance of finished products of facing bricks that occurred in the production stage during processing, packaging, transportation, or storage. The tasks of this study are as follows:

-

creating a database of images of facing brick samples;

-

substantiation and description of the selected YOLOv5s detector;

-

carrying out the process of augmentation to expand the training data set;

-

implementation, optimization, debugging, and testing of the algorithm using the CNN YOLOv5s;

-

determination of the quality of the metrics of the implemented model.

The practical value of this study lies in the development of an applied algorithm capable of acting as the basis for intelligent online control of defects in conveyor production products.

2. Materials and Methods

2.1. Markup and Image Augmentation

For the development of a neural network algorithm for detecting and classifying defects, the stage of preparing a training data set is important, on the qualitative and quantitative characteristics of which further accuracy depends. The initial data for this study were 16 images of facing brick samples with various types of defects. For photography, a Canon EOS 60D camera (Tokyo, Japan) with a 22.3 × 14.9 mm matrix and an image resolution of 18 megapixels was used.

2.1.1. Image Markup

Image annotation is one of the key steps in creating an effective computer vision system. This process converts the information into a format that can be understood by the image analysis algorithm. During markup, the original image is supplemented with metadata about the location and class of the defect if it is fixed by an expert technologist.

Figure 1 shows an image of a product annotated with the Image Labeler, where red Bounding Boxes are the “cracks” class, purple are the “rust” class, green are the “broken corners” class, and blue are the “inclusion” class. The Bounding Box stores the coordinates of an object and is thus used to determine the location of an object in space.

A MAT file with bounding box coordinates will be the result of this step in the form of an exported labeled dataset.

2.1.2. Image Augmentation

In the case when there is little data for training and it is necessary to achieve a good generalization ability from the algorithm, researchers usually resort to data manipulation. The use of synthetic data obtained in the process of augmentation improves the quality of the work of CNN and reduces the amount of necessary “real” data by simulating various shooting conditions using image processing methods [7].

However, when a high-quality set of initial data appears, it is necessary to mark up an additional set of images. To automatically recalculate the bounding box for each new image, a custom MATLAB script was proposed that performs the following steps (Figure 2).

• Adding original photos without changes to the training set.
• Image representation horizontally and vertically.
• Shift of the original image by a random value along the Ox and Oy axes.
• Rotation of the image at angles: 90°, 180°, and 270°.
• Application of filters to change brightness, contrast, and saturation.

Image augmentation made it possible to expand the dataset of 16 photographs to 400.

2.2. Development of an Intelligent Algorithm Based on the YOLOv5s Convolutional Neural Network

In this study, the lightest YOLOv5s network is selected for reasons of recognition accuracy and speed. Figure 3 shows the structure of YOLOv5s. “It consists of: the input layer (Input), the backbone network (backbone), the bottleneck layer network (neck), and the detection layer (output). The backbone and neck are composed of Focus, CBL (Convolutional, Batch normalization, Leaky Relu), CSP (Cross Stage Partial), and SPP (Space Pyramid Pooling)” [48]. The “backbone part focuses on extracting feature information from input images, the neck part combines the extracted feature information and generates feature maps at three scales, and the output part detects features from these generated feature maps” [49].

The implementation of YOLOv5s CNN is based, among other things, on the choice of parameters for its training, the main parameters are presented in

Table 1. Basic training parameters of the YOLOv5s CNN.

Num	Parameter	Value
1	Number of images in the training set	300 (60%)
2	Number of images in the validation set	100 (20%)
3	Number of images in the test set	100 (20%)
4	BatchSize	10
5	Number of epochs	500
6	Number of iterations	5000
7	Learning rate	0.01
8	Solver	Stochastic gradient descent

.

The model was trained on NVIDIA Tesla T4 accelerators (Santa Clara, CA, USA) and took 96 min. For software that works in real time (which is assumed in the future), it is necessary to have the best ratio of speed of work to recognition accuracy. For the efficient use of computing resources (memory) in the training of the CNN, BatchSize = 10 is used. The Number of epochs = 500 parameter, since the error changes slightly with further training. The values of the Learning rate and Solver parameters are used by default.

3. Results and Discussion

3.1. Description of Facing Brick

Facing the facades of buildings plays not only the role of decor, but also performs several structural functions: improving heat and sound insulation and protecting walls from environmental influences, thereby increasing the calendar duration of the functioning of structural elements and the building as a whole. A variety of different building materials are presented on the market, and they differ in their pricing policy and various properties, which makes it possible to satisfy the needs of any consumer.

Among the variety of facing materials, it is worth highlighting brick, which is the oldest building material that is constantly being improved, offering a huge selection of textures and decorative possibilities. Facing bricks are universal materials and are used not only for the facade of the house. They are actively used for the construction of fences, the improvement of buildings on the site, and the construction of partitions in the house that do not require finishing.

The indicators of the appearance of facing bricks directly affect their physical and mechanical properties. The finished product has dimensions of 515 mm × 105 mm × 38 mm. Quality is determined by the absence of violations of the structure and appearance of finished products.

Facing bricks may have various defects that occur both during transportation and storage and during their manufacture. EN 771-1:2011 [50] and EN 771-2:2011 [51] regulate possible brick defects, as well as their number. If the quantity of defective bricks is within the normative requirements, then such a batch is considered to be of high quality. However, even factories with a good reputation are not immune to the appearance of defective batches of goods. To determine the quality of products, it is necessary to carry out a visual inspection method that will save money on dismantling and re-purchasing building materials. Most of the defects are visually distinguishable, which makes it impossible for the manufacturer or dealer to sell such bricks.

Figure 4 shows defects on a facing brick sample: foreign inclusions, broken corners, cracks, and color unevenness, including the presence of rust spots. This markup was carried out manually by a specialist technologist. It takes from one to three minutes to mark one photo. The process of detecting foreign inclusions is the longest due to their small size.

Such defects may occur in a case of violation of production technology, improper transportation, or non-compliance with storage conditions. In our case, foreign inclusions are a manufacturing defect—broken corners, cracks—and are the result of mechanical action during the transportation of the sample; the uneven coloring, including the presence of rust spots, is a result of non-compliance with storage conditions. In order to speed up the process of visual inspection and identify defects of a different nature on facing brick samples, we proposed the development of an intelligent algorithm that simulates the cognitive functions of an expert technologist, whose “eyes” are a computer vision algorithm based on a CNN.

3.2. Implementation, Optimization, Debugging, and Testing of the Algorithm Using the CNN YOLOv5s

The YOLOv5s loss function consists of three parts (Figure 5):

-

box_loss—loss of regression of the bounding box (root mean square error);

-

obj_loss—confidence in the presence of an object is a loss of objectivity;

-

cls_loss—classification loss (Cross Entropy).

Mean Average Precision (mAP) is used to evaluate computer vision models; to calculate it, you must first determine the Precision (P), Recall (R), and Average Precision (AP) indicators.

Calculation of P and R values is carried out according to the formulas:

$$\mathrm{Precision} \left(\mathrm{P}\right)=\mathrm{TP} / \left(\mathrm{TP}+\mathrm{FP}\right)$$

(1)

$$\mathrm{Recall} \left(\mathrm{R}\right)=\mathrm{TP} / \left(\mathrm{TP}+\mathrm{FN}\right),$$

(2)

“here true positive (TP) is the correct detection made by the model; false positive (FP) is incorrect detection made by the detector; false negative (FN) is a true result missed (not detected) by the detector” [7].

An auxiliary metric Intersection over Union (IoU) is used to determine AP. IoU shows how much our predicted boundary overlaps with the true (real object boundary). Figure 6 schematically shows the calculation of IoU where the intersection area is divided by the area of the union of the true and predicted frames.

The area of intersection of the regions in Figure 6 is divided by the area of the union between the true and predicted frames. The IoU metric, also known as the Jaccard index, is a number from zero to one that shows how much two objects (reference and predicted by the neural network) have the same internal “volume”.

AP@α is an area under Precision–Recall Curve (AUC–PR) that is estimated at threshold α IoU. The formula for the calculation is as follows:

$$\mathrm{AP}@\mathsf{\alpha }={\displaystyle \underset{0}{\overset{1}{\int }}\mathrm{P}\left(\mathrm{R}\right)}\hspace{0.17em}dr$$

(3)

Mean Average Precision (mAP) is equal to the average of Average Precision over all classes in the model.

Table 2. Model Quality Estimates.

Num	Parameter	Instances	mAP0.50	mAP0.50:0.95
1	all	825	0.87	0.72
2	broken corners	98	0.84	0.68
3	cracks	106	0.89	0.84
4	inclusion	61	0.86	0.60
5	rust	560	0.88	0.77

estimates mAP for the test set, averaged over IoU ∈ [0.50:0.95] (standard COCO metric) and mAP0.50 (PASCAL VOC metric).

It is worth noting that the “broken corners” and “inclusion” classes are defined less precisely than the “cracks” and “rust” classes. This is due to the geometric parameters of the defects themselves: cracks and rust spots are most noticeable to the expert at the marking stage and, hence, to the detector. Smaller defects of the “inclusion” type can be missed by a specialist when marking, and therefore will not be taken into account in the process of training a neural network, which entails a decrease in accuracy when this class is detected. Careful markup of these objects is required, and was performed. The effect of interference was taken into account in the process of data augmentation by changing the brightness, contrast, and saturation of images. Figure 7 shows the Confusion Matrix, which is a classification performance metric.

Figure 8 shows the results of the developed algorithm. The model shows bounding boxes indicating the predicted feature class and how confident the network is in its prediction.

The results show that the detector detects and classifies defects in photographs subjected to changes during the augmentation process, namely, changing lighting conditions and shooting angles: Figure 8a,c is shifted along the OX axis, Figure 8e is changed brightness. Thus, various survey factors are taken into account in the learning process, and they do not reduce the efficiency of the detection algorithm.

The results obtained correspond to the needs of the problems of detection and classification of defects of various natures in the facing building material and are of practical value.

Thus, the expansion of the training set through the use of the author’s augmentation algorithm of the own empirical database of facing brick images based on simulating various shooting conditions, the selection of parameters, training, and optimization of the detector based on the YOLOv5s CNN, made it possible to obtain an algorithm that has a high accuracy in the tasks of building flaw detection. The values of the mAP0.50 and mAP0.50:0.95 metrics for the developed model were 87% and 72%, which is not inferior to the average accuracy (AP) of the model of a one-stage crack detection network consisting of the EfficientNetB0 backbone network and a detector (76.4%) [28], and AlexNet proposed in [52] (89.31%) outperforms the FPN model proposed for detecting and segmenting cracks on the surface of marble slabs (71.35%) [53], as well as the YOLACT model for detecting cracks in concrete with ResNet-50 main architectures and ResNet-10 (37.39% and 36.05%) [54].

Comparison of the obtained results with the results of other studies showed a rather high accuracy of the developed model aimed at detecting and classifying defects. The model can be improved by expanding the training data set and improving the augmentation algorithm by introducing additional lighting conditions and shooting angles, as well as a combination of various noises and effects. These transformations contribute to the expansion of variations of possible representations of the original image and, thus, have a positive effect on the training of the detector. Expanding the sample leads to more efficient training of the detector, improving its degree of recognition of new objects.

The obtained results should also be discussed in terms of scientific novelty and practical significance. The scientific novelty of this study and, accordingly, its scientific result will be, primarily, the created empirical base, which will allow using the new methodology to determine defects in facing materials, namely in bricks. The novelty lies in the fact that, earlier in construction practice, this approach was based on a physical examination of the condition of facing materials and the detection of defects either with the help of magnifying glasses or with the naked eye. Our approach is methodologically new, as it is based on the introduction of the artificial intelligence method and its verification by laboratory studies of real brick samples. Pilot testing of this method is possible in two ways. These two ways are approbation at the stage of brick production, that is, at construction industry enterprises, and approbation at the stage of erection of buildings and structures in masonry. Thus, in view of the fact that the physical experiment under conditions of simulation of real operating conditions, on the samples selected in real conditions, showed good convergence with the proposed methods of artificial intelligence, the proposed method can be considered effective and accurate. At the same time, the practical significance of this study should be noted. Such a study makes it possible to detect defects in facing products at the production stage with greater efficiency, accuracy, and the least labor costs. At the stage of construction and operation, this method will allow monitoring of the technical condition and construction control over the construction of buildings and structures. Finally, a certain phenomenological base has been laid for future scientific research, based on an integrated approach that combines a physical experiment with the method of artificial intelligence. Thus, summing up the results of this study, it can be noted that from the point of view of information technology, the existing ideas about the methods of artificial intelligence in the construction industry and construction have been developed. From the point of view of construction, a new methodology has been developed for managing the life cycle at the stage of production, construction, operation, and monitoring of facing products and structures of buildings, as well as structures made from them.

The use of machine learning algorithms is becoming an increasingly relevant method for solving both scientific problems and business needs. The results obtained in this study can be considered as a flexible and scalable tool that does not require special operating conditions. The developed algorithm has a high speed, high accuracy, and reliability of results. However, it should be noted that data processing by artificial intelligence methods is an auxiliary tool for making a decision, and it is not the final verdict on the quality of a building material.

4. Conclusions

Based on the results obtained, the following conclusions have been formulated.

(1)

An empirical database of images of facing bricks has been created, which has various violations of the structure and appearance which have arisen at the production stage during technological processing, packaging, transportation, and storage.

(2)

We developed, optimized, and tested a computer vision model based on the YOLOv5s convolutional neural network. By applying a custom augmentation algorithm, synthetic images were created for the training dataset. The application of the developed algorithm in practice became possible with the help of a model trained on its own set of data obtained by simulating the shooting conditions, the angle of rotation, object deformation, and light distortion by image processing methods.

(3)

The developed algorithm detects defects both in images taken under normal shooting conditions and in the presence of color/light distortions. In the work, it was taken into account that defects that are small in size can be missed by a specialist when marking and, therefore, will not be taken into account in the process of training a neural network. Careful markup of these objects is required, which was performed in this study, which made it possible to achieve an accuracy of mAP0.50 = 87% for the “inclusion” class.

(4)

The developed algorithm has a high accuracy in the problems of detection and classification of defects: mAP0.50 = 87% and mAP0.50:0.95 = 72%.

(5)

Behind the conclusion about the quality of products based on intelligent control, there is always a specialist who can adequately respond outside the machine protocol, taking into account the real factors of production, transportation, and storage of products.

The prospect of improving the model is to expand the visual variability in the generation of images in the process of augmenting the original data set, as well as to adapt software tools for intelligent online control of surface defects in conveyor production products.

The developed defect detection and classification model can also be applied to other facing materials and products, such as stone, clinker facade tiles, facade panels, and plaster for more complete control of the degree of defectiveness of each product. The developed model is quite versatile and can be used for most building materials and products that are subject to increased requirements for appearance.

5. Patents

Beskopylny, A.N.; Mailyan, L.R.; Stel’makh, S.A.; Shcherban’, E.M.; Razveeva, I.F.; Beskopylny, N.A.; Dotsenko, N.A.; El’shaeva, D.M. The program for determining the mechanical properties of highly functional lightweight fiber-reinforced concrete based on artificial intelligence methods. Russian Federation Computer program 2022668999, 14 October 2022. Available online: https://www.fips.ru/registers-doc-view/fips_servlet?DB=EVM&DocNumber=2022668999&TypeFile=html (accessed on 9 March 2023).

Beskopylny, A.N.; Stel’makh, S.A.; Shcherban’, E.M.; Razveeva, I.F.; Kozhakin, A.N.; Beskopylny, N.A.; Onore, G.S. Image Augmentation Program. Russian Federation Computer program 2022685192, 21 December 2022. Available online: https://www.fips.ru/registers-doc-view/fips_servlet?DB=EVM&DocNumber=2022685192&TypeFile=html (accessed on 9 March 2023).