Detecting Cracks in Aerated Concrete Samples Using a Convolutional Neural Network

Abstract

The creation and training of artificial neural networks with a given accuracy makes it possible to identify patterns and hidden relationships between physical and technological parameters in the production of unique building materials, predict mechanical properties, and solve the problem of detecting, classifying, and segmenting existing defects. The detection of defects of various kinds on elements of building materials at the primary stages of production can improve the quality of construction and identify the cause of particular damage. The technology for detecting cracks in building material samples is of great importance in building monitoring, in pre-venting the spread of defective material. In this paper, we consider the use of the YOLOv4 convolutional neural network for crack detection on building material samples. This was based on the creation of its own empirical database of images of samples of aerated concrete. The number of images was increased by applying our own augmentation algorithm. Optimization of the parameters of the intellectual model based on the YOLOv4 convolutional neural network was performed. Experimental results show that the YOLOv4 model developed in this article has high precision in defect detection problems: AP@50 = 85% and AP@75 = 68%. It should be noted that the model was trained on its own set of data obtained by simulating various shooting conditions, rotation angles, object deformations, and light distortions through image processing methods, which made it possible to apply the developed algorithm in practice.

1. Introduction

At present, construction companies are introducing new technologies to automate various processes at all stages of construction, the purposes of which are to reduce the accident rate, improve the performance of facilities under construction, reduce labor costs, reduce construction time, and as a result, obtain economic benefits [1]. However, despite the involvement of representatives of the construction industry in the process of applying new technologies, there is a problem of the rather low level of production automation [2,3]. The most important role in solving this problem can be played by modern methods of artificial intelligence (AI). AI tools will allow one to process, systematize, and analyze a large amount of accumulated information, which is not structured. Particular attention is paid to the issues of digital modernization of the life cycle of building materials, which allows improvements in the quality level and enhances their characteristics. The creation and training of artificial neural networks with a given precision allows one to identify patterns and hidden relationships between technical and technological parameters in the manufacturing of unique building materials, predict mechanical properties, and also solve the problem of detecting, classifying, segmenting existing defects [2,4,5,6]. The detection of defects of various natures on samples of building materials at the primary stages of production will improve the quality of construction and identify the cause of particular damage.

There are many methods for monitoring and detecting defects in building products and structures, such as geotechnical monitoring [7], the geodetic method and cracking meter [8], the use of built-in sensors (ultrasonic, electromagnetic, piezoceramic) [9,10,11,12,13,14], the method of electromechanical impedance [15], the finite element method [16], the instrumental method of neural networks with the addition of pulses [17], and the acoustic emission method [18,19,20]. At the same time, it is important to detect defects and damage to buildings already in operation in a timely manner, including those caused by earthquakes, using artificial intelligence methods [21,22,23,24], using a radar interferometer [25]. For monitoring and detecting damage and defects and predicting the service life of structures, in particular bridges of differing designs and architecture, artificial intelligence methods have also been widely studied and applied [26,27,28,29]. To prevent the premature destruction of products and structures, various methods for the early detection of defects on construction objects based on the algorithm [30] have already been studied and put into practice, using convolutional neural networks on images obtained via laser fusion in a powder layer [31], based on deep learning [32,33], using a generative adversarial network [34]. In [2,35,36,37,38,39,40], the authors investigated the possibility of using deep learning methods to detect defects in buildings and structures, as well as various building materials. Thus, in [2], the possibility of using a convolutional neural network and transfer learning to detect cracks in stone walls was studied. Based on the results of the research, a neural network was developed that allows the identification of cracks on the surface of brickwork with an accuracy of up to 95.3%. Moreover, in [38], the authors considered the possibility of unmanned aerial vehicles (UAVs) equipped with vision sensors to search for cracks in various structures of bridge structures. Based on 384 crack photographs, a convolutional neural network was developed, and the results of field tests proved the effectiveness of using this network and UAVs to find and quantify cracks in structures. The authors [39] also studied the possibility of identifying cracks in concrete structures using UAV technology and hybrid image processing. This scheme turned out to be effective for finding and measuring cracks with an opening width of more than 0.1 mm, and the maximum error was 7.3%. In [40], the possibility of using a convolutional neural network to automatically search for various defects in buildings and structures from images was evaluated. According to the results of the study, the reliability and ability to accurately localize the defect was proven.

In the works [41,42,43,44], the effectiveness of the application of deep learning methods in the engineering industry was studied. In [41], a probabilistic neural network algorithm was developed for detecting defects in the welding process. Weld defect texture images were considered as input parameters for building the model, and in terms of the output, the developed FAST-PNN model provides fairly accurate data on detected defects (burns, pores, cracks, and so on). Thus, in studies [42,43], the authors developed neural networks that make it possible to find defects in welds with an accuracy of more than 93%. In [44], a 2D convolutional neural network was developed to control the quality of 3D ball joints, which allows one to find defective products with an accuracy of 97%. An interesting study is [45], where using deep learning methods, a smartphone application was developed to monitor the performance of buildings in real time. The resulting model has high precision. However, this model can be improved by using large datasets. In [46], a model was developed for the real-time monitoring of concrete structures, consisting of the Efficient–NetB0 backbone network and detectors. The model was trained on a data set with concrete surface defects, and the test results showed a high precision of defect detection—74.6% for cracks and 89.9% for open bars.

The analysis showed the growing relevance and popularity of using artificial intelligence methods in the construction industry at all stages of construction, from determining the mechanical properties of materials to expert systems that provide recommendations for assessing the reliability of building structures of buildings and structures on various grounds. Particular attention is paid to AI methods specifically when searching for defects in building structures and materials. This is justified by the desire of builders to exclude the influence of the human factor, as well as to reduce time and labor costs. In this paper, we consider the use of the YOLOv4 convolutional neural network for crack detection on building material samples.

The YOLOv4 model has optimal speed (due to the fact that the model is a single-stage detector) and high object-detection precision. The scientific novelty of the research lies in the following:

−

creation of their own empirical base of images of building material samples [47] and increasing the number of images by applying their own augmentation algorithm [48];

−

optimization of the parameters of the intellectual model based on the convolutional neural network YOLOv4 for its further use in production to track defective products in real time.

To achieve this goal, it is necessary to solve the following tasks:

−

collection of an empirical database of images of aerated concrete samples;

−

substantiation of the chosen detection method;

−

conducting the augmentation process to expand the sample;

−

implementation and optimization of the algorithm using the convolutional neural network YOLOv4.

The theoretical significance of the study lies in the expansion of ideas about the possibilities of using artificial intelligence methods in the construction industry. The practical significance of the work lies in the development of an applied and scalable algorithm that allows the production and quality control of building materials, products, and structures to reject low-quality goods in time.

2. Materials and Methods

2.1. Characteristics of the Analyzed Building Material

In construction, various building materials and products from them are used. The main building materials in industrial and civil construction are cement, concrete, brick, stone, wood, lime, sand, and glass. Materials scientists are constantly improving these materials, giving them new characteristics necessary for specific building needs. Therefore, aerated concrete is a universal material related to cellular concrete, used for the construction of both non-load-bearing and load-bearing walls. For many years, builders have chosen it as a reliable, easy-to-use, and process material for various jobs. Due to its unique structure, aerated concrete has a number of special physical and technical properties. This is a high quality, warm, and environmentally friendly material. It has all the advantages of concrete, but it is easy to process and use. Therefore, it is often used to build walls of a complex shape. A feature of aerated concrete blocks is the precision of their dimensions and the correctness of their shape. Because of this, laying them is very easy and fast. The roughness of the surfaces of the blocks facilitates their subsequent processing. An important feature of the material is its excellent sound and heat insulation properties, fire safety, and environmental friendliness [49,50].

The external characteristics of aerated concrete are an important source of information in determining its strength characteristics. The quality of aerated concrete determines the uniformity of distribution, the equality of volume, and the closeness of the pores, as well as the absence of cracks. Defects in aerated concrete blocks occur when the production technology is violated, when one or another number of components in the mixture for blocks is incorrectly supplied, and technological failures in the operation of the autoclave. There are cases of defective batches of blocks at factories with a good reputation and consistently high quality of manufactured materials and products. Therefore, careful checking of suppliers or dealers is recommended when purchasing building materials, as such manufacturing defects can cause too much damage for the builder. Vertical, horizontal, and microcracks in aerated concrete blocks indicate the poor quality of the building material. In the future, when using defective block stone, shrinkage cracks may occur.

Figure 1 shows a crack in an aerated concrete sample. Identification of damage of this kind is one of the main criteria for the visual inspection of blocks in production and transportation.

Figure 1 shows crack-damage resulting from the improper transportation of products.

To automate the process of detecting defects on aerated concrete samples and minimizing the influence of the human factor, developing an intelligent algorithm that allows the detection of defective products is proposed.

2.2. Augmentation and Data Markup

A dataset designed to train a neural network is crucial in the application of artificial intelligence methods. The initial data set in this study was 15 images of aerated concrete samples, in which there were cracks of various geometric parameters.

2.2.1. Image Markup

The first step in preparing images for input to a convolutional neural network is the marking of objects in the images. Image annotation is an integral part of the development of an artificial intelligence model, and it is one of the main tasks in computer vision technology. Annotated images are needed as input for training neural networks (Figure 2). In this study, Image Labeler was used to label regions of interest.

There are several labeling methods for solving the detection problem: selecting a crack with a complete bounding box and dividing the image into patches [2], as shown in Figure 2a,b, respectively. The marking method performed by highlighting a crack with a fully bounding box was considered at the testing stage of the developed model. Based on the results obtained, it was concluded that in this case, the crack occupies a small area in the region of interest and when rotated, the bounding box in some cases goes beyond the image boundaries, which is an additional load when training the neural network and entails unsatisfactory detection results—low detection precision at long time costs. Therefore, the second method was chosen for marking up images.

The result of this stage is an exported labeled dataset—a MAT-file containing the coordinates of the bounding boxes.

2.2.2. Image Augmentation

Often, in engineering or scientific problems, there is a lack of data due to the complexity of obtaining/processing/saving them, which is why it is necessary to replace the missing data with modifications of the existing ones. Augmentation, or generating new data based on existing ones, allows one to solve some of the problems with the training sample in handy ways. For example, in the case of photographic data, as in this study, shooting conditions, camera features, rotation angles, object deformations, and many other distortions are successfully simulated via image processing methods. By modeling deformations in this way, it is easy to achieve an increase in the quality of the model and increase its resistance to various noises in the input data.

To expand the data set, a script was developed in the MATLAB language. The following steps were performed during the operation of the algorithm (Figure 3):

• Original photo addition without changes to the training set.
• Display (vertical and/or horizontal).
• Random image shift along the Ox and Oy axes.
• Image rotation 90°, 180°, 270°.
• Change brightness, contrast, and saturation.

The advantage of using one’s own algorithm is that for the new image obtained during the augmentation process, the bounding box is automatically recalculated.

In total, thanks to the augmentation process, the data set of 15 photographs was expanded to 4000 images of aerated concrete samples with cracks.

2.3. Development of an Intelligent Algorithm Based on the YOLOv4 Convolutional Neural Network

One of the most popular technologies for solving the detection problem is the YOLO (You Only Look Once) State-of-the-Art model. It is a real-time object detection algorithm that is a deep convolutional neural network.

YOLOv4 is a single-stage detector that prioritizes output speed, which is important for our study (Figure 4).

The YOLOv4 architecture consists of the following elements:

• Input, represented as our set of training images that will be fed into the network.
• Backbone and Neck, which perform feature extraction and aggregation. The Backbone is based on a pre-trained deep learning network built using CSP-DarkNet-53 as the base network.

CSPDarknet53 is a convolutional neural network based on DarkNet-53. The CSPNet strategy splits the base-level feature maps into two parts and then combines them through a hierarchy between stages. The use of a split-and-merge strategy provides a more gradient flow through the network [51].

The Neck block adds additional layers between the Backbone block and the Head block to provide “richer” spatial and semantic information. The Neck in the YOLOv4 architecture uses a combination of spatial pyramid pooling (SPP) and path aggregation network (PAN) modules. The SPP module combines feature maps of different scales by performing several subsampling operations with different window sizes (1 × 1, 5 × 5, 9 × 9, and 13 × 13) and is used to expand the perceptual area and extract the most significant semantic features. PAN uses upsampling and downsampling operations to establish upstream and downstream paths to combine low-level and high-level features.

The Neck combines feature maps from different layers of the backbone network and sends them as input to the Head.

3.

In the YOLOv4 network, single-stage object detectors, such as in YOLOv3, are used as detection heads.

4.

Result, which is an image with the specified bounding box coordinates.

An important step in the implementation of the YOLOv4 architecture convolutional neural network is the selection of parameters during its training. The main parameters are presented in

Table 1. Parameters for training the YOLOv4 convolutional neural network.

Num	Parameter	Value
1	Number of photos in the training set	2800 (70%)
2	Number of photos in the validation set	800 (20%)
3	Number of photos in the test set	400 (10%)
4	MiniBatchSize	28
5	Number of epochs	30
6	Number of iterations	3000
7	Learning rate	0.001
8	Solver	Adam solver

.

The calculation was carried out on a high-performance VDK cluster based on two-processor servers with an installed Intel Xeon E5-2683 v3 processor (14 cores, 2 GHz, 35 Mb cache); the total number of cores was 128, and the amount of RAM was 256 Gb.

3. Results and Discussion

The model training graph is shown in Figure 5, where the ordinate shows the loss values, and the abscissa shows the various iterations.

The loss function gives the error between the predicted and actual bounding box. The smaller the loss function, the better the trained network detects objects.

To demonstrate the effectiveness of the developed algorithm for detecting cracks in aerated concrete samples, in this experiment, precision (P), recall (R), and average precision (AP) are used as model estimates.

Precision and recall can be calculated using the formulas:

$$\mathrm{Precision}\left(P\right)=\frac{TP}{TP+FP}$$

(1)

$$\mathrm{Recall}\left(R\right)=\frac{TP}{TP+FN}$$

(2)

Here, true positive (TP) is the correct detection made by the model;

False positive (FP) is incorrect detection made by the detector;

False negative (FN) is a true result missed (not detected) by the detector.

To define AP, an auxiliary metric called Intersection over Union (IoU) is needed. IoU is a good way to measure the degree of overlap between two bounding boxes—a predicted model and a real one.

Schematically, the IoU calculation is shown in Figure 6, where the intersection area is divided by the union area between the true and predicted frames.

AP@α is the area under Precision–Recall Curve (AUC–PR) estimated at threshold α IoU. The formula for the calculation is as follows:

$$AP@\alpha ={\displaystyle \underset{0}{\overset{1}{\int }}P\left(R\right)dr}$$

(3)

AP@50 and AP@75 values were calculated showing AP values calculated at IoU = 0.50 and IoU = 0.75, respectively: AP@50 = 85% and AP@75 = 68%.

Table 2. Model quality estimates.

Number	Parameter	IoU = 0.50	IoU = 0.75
1	Precision	88%	71%
2	Recall	70%	61%
3	AP	85%	68%

presents the main metrics characterizing the quality of the resulting model with the threshold values IoU = 0.50 and IoU = 0.75.

Figure 7a or Figure 7c shows the input images, Figure 7b or Figure 7d shows the results of crack detection on aerated concrete samples using the YOLOv4 convolutional neural network.

The results obtained meet the needs of the problems of detecting cracks in a building material and have practical applied value.

Thus, creating our own empirical database of images of aerated concrete samples [47], training the model on our own data set obtained by simulating various shooting conditions, the rotation angle of object deformation, and light distortion using image processing methods, using our own augmentation algorithm to increase the number of images, and therefore, the expansion of the sample [48] made it possible to optimize the parameters of an intelligent model based on the YOLOv4 convolutional neural network, which has high precision in defect detection problems. The obtained precision of the model corresponds to the improved model (AP@50 = 84.06% and AP@75 = 67.85%) in [52]. It is also not inferior to the model for detecting cracks on the concrete surface proposed in [46] (76.4%) and exceeds the precision of the model for detecting and segmenting cracks on a surface with a random texture similar to marble (71.35%) [32]. This comparative analysis indicates the prospects of the developed model in the problems of defect detection, taking into account its subsequent improvement by expanding the training data set. Expanding the training sample by introducing additional shooting angles and lighting conditions, as well as a combination of various noises and effects, is planned. The resulting changes will have a positive effect on the training of the detector, expanding the range of possible representations of the original image. A detector trained on an expanded sample is more ready to recognize new objects. The developed algorithm in the future is planned to be used in production for tracking defective products in real time.

Thus, the developed mechanism is a proven methodology for detecting defects in aerated concrete, which helps to reduce the share of manual labor and time spent in aerated concrete production. Given the complexity of aerated concrete technology and the number of factors affecting the quality of aerated concrete products, artificial intelligence methods are especially relevant in the production of this building material. It is assumed that the developed concept can be extended to other simpler building materials, such as foam concrete and polystyrene concrete, the quality of which depends on fewer factors, but also requires a detailed analysis and the complete control of each product. Therefore, the developed concept is universal and can be applied to any building materials with a similar structure. Thus, the prospects for the development of the study lie in extending the proposed methodology to other building materials, for which the appearance is a quality projection due to the inadmissibility of cracks, cavities, and other structural defects. For materials with a porous structure, such as aerated concrete, not only the absence of defects in the form of cracks and cavities is important, but also the porous nature of the structure, which is the degree of dispersion of the pores, the location of the pores relative to each other, the microporosity of the interpore partitions, and other factors that worsen not only the external decorative appearance of the product, but also lead to a decrease in the quality and consumer properties of such material. Thus, the developed concept has value for the production process due to the increased automation of aerated concrete production, reduced defects in the construction of facilities, and reduced labor, time, and energy costs in the production of aerated concrete and also contributes to the development of smart technologies for the introduction of artificial intelligence methods in the production of construction materials and practical construction. The scientific novelty of the proposed method lies in the creation of an empirical base, the accumulation of photographic images of aerated concrete structures, the search for smart algorithms for determining structural defects, and thus, preparing the groundwork for future research based on the results already achieved.

4. Conclusions

The results obtained in the study allowed us to draw the following conclusions.

Creating our own empirical database of images of aerated concrete samples, as well as training the model on our own data set obtained by simulating various shooting conditions, and increasing the number of images by using our own augmentation algorithm made it possible to optimize the parameters of an intelligent model based on the YOLOv4 convolutional neural network. The experimental results showed that the developed YOLOv4 model has high precision in defect detection problems: AP@50 = 85% and AP@75 = 68%. The obtained precision values indicate the prospects of the developed model in the problems of flaw detection, taking into account its subsequent improvement by expanding the training set. Expanding the training data set by introducing additional shooting angles and lighting conditions, as well as a combination of various noises and effects, is planned, which will have a positive impact on detector training, expanding the range of possible representations of the original image. The developed algorithm in the future is planned to be used in production for tracking defective products in real time.

It is assumed that the developed defect detection model can be extended to other building materials, such as foam concrete and polystyrene concrete, the quality of which depends on fewer factors, but also requires a detailed analysis and complete control of each product. Therefore, the developed concept is universal and can be applied to any building materials for which the appearance is a quality projection due to the inadmissibility of cracks, cavities, and other structure defects and damage.

5. Patents

Beskopylny, A.N.; Mailyan, L.R.; Stel’makh, S.A.; Shcherban’, E.M.; Razveeva, I.F.; Beskopylny, N.A.; Dotsenko, N.A.; El’shaeva, D.M. The program for determining the mechanical properties of highly functional lightweight fiber-reinforced concrete based on artificial intelligence methods. Russian Federation Computer program 2022668999, 14 October 2022. Available online: https://www.fips.ru/registers-doc-view/fips_servlet?DB=EVM&DocNumber=2022668999&TypeFile=html (accessed on 7 January 2023).

Beskopylny, A.N.; Stel’makh, S.A.; Shcherban’, E.M.; Razveeva, I.F.; Kozhakin, A.N.; Beskopylny, N.A.; Onore, G.S. Image Augmentation Program. Russian Federation Computer program 2022685192, 21 December 2022. Available online: https://www.fips.ru/registers-doc-view/fips_servlet?DB=EVM&DocNumber=2022685192&TypeFile=html (accessed on 7 January 2023).

Beskopylny, A.N.; Mailyan, L.R.; Stel’makh, S.A.; Shcherban’, E.M.; Razveeva, I.F.; Kozhakin, A.N.; Beskopylny, N.A.; El’shaeva, D.M.; Tyutina, A.D.; Onore, G.S. Photographic images of the structure of aerated concrete. Russian Federation Database 2022623622, 22 December 2022. Available online: https://www.fips.ru/registers-doc-view/fips_servlet?DB=DB&DocNumber=2022623622&TypeFile=html (accessed on 7 January 2023).