Concrete Crack Identiﬁcation and Image Mosaic Based on Image Processing

Featured Application: This article introduces a crack size extraction and image stitching technology based on image processing that can safely and e ﬃ ciently obtain the crack information of concrete structures. This work can provide assistance for those who are engaged in the crack detection of concrete bridges. Abstract: Crack assessment is an essential process in bridge detection. In general, most non-contact crack detection techniques are not suitable for widespread use. The reason for this is that they all need to position the ruler at the inspection site in advance or calibrate the camera unit pixel size at a certain distance in a very intricate process. However, the object distance method in this paper can complete the calculation using only the crack image and the working distance, which are provided by an acquisition system equipped with a camera and laser range ﬁnder. First, the object distance method and the scale method are compared by calculating the crack width, and the results show that the object distance method is the more accurate method. Then, a double edge pixel statistical method is proposed to calculate the crack length, which solves the problem of redundant and missing pixels. In addition, the conventional mosaic algorithm is improved to realize an image mosaic for the more e ﬃ cient splicing of crack images. Finally, a series of laboratory tests were conducted to verify the proposed approach. The experiments showed that the precision of crack length extraction can reach 92%, and the improved algorithm stitching precision can reach 98%.


Introduction
Concrete is one of the most widely used engineering materials. Owing to its low cost, good durability, ease of shaping and other characteristics, concrete plays a role in widespread applications in construction engineering, traffic engineering, water conservancy engineering, and other infrastructure construction efforts. However, with the coming maintenance period of the engineering structures built in the early years, concrete structures inevitably suffer damage and destruction [1,2] under the long-term actions of freezing-thawing cycles and loading. This degradation first manifests as structural cracks whose size and direction are important factors to evaluate structural reliability. After a crack occurs in a structure, moisture enters the concrete beam through the crack, causing the steel to rust. The expansion of water into ice saggravate the development of cracks when the external temperature is low. The direction of the crack allows structural engineers and relevant management personnel to evaluate the structural condition of the crack and predict its development. Therefore, the safe operation of engineering structures is often evaluated based on cracks.
At present, cracks on concrete structures are mainly inspected by the manual visual observation of the surface. For example, in bridge crack detection, a large machine such as a test vehicle is usually used to position the inspector underneath the bridge, and the cracks are identified by the naked eye and measured by an optical microscope. It is precisely because of this measurement characteristic that the detection accuracy of cracks largely depends on the proficiency of the detection personnel. For many structures, the inspector cannot go to the testing location. Therefore, this manual detection method is inefficient, costly, and has security risks. With the rapid development of signal processing theory and modern computer technology, machine vision [3][4][5][6][7][8] has played an increasingly important role in crack detection. Image processing technology can be used to safely and efficiently extract crack information. Crack information extraction mainly includes the identification of the crack size [9][10][11][12] and the final stitching [13][14][15][16][17] of the crack image.
Abdel-Qader et al. [18] compared four edge detection algorithms, the fast Haar transform (FHT), the fast Fourier transform, and the Sobel and Canny algorithms, and their results showed that the FHT had the best performance in crack detection. Zou et al. [19] developed CrackTree, a fully automatic method for detecting crack curves from a pavement image, and proposed a new geodetic shadow-removal algorithm for eliminating road shadows and enhancing the contrast of cracks. Li et al. [20] proposed the FoSA, or F* (pronounced F star) seed growing approach, for pavement crack line detection, which is based on crack seed growing strategies and solved the problem of automatic selection of the start and end points of the F* program. This method has a strong anti-speckle-noise capability for extracting crack lines from pavement images. PraSanna et al. [21] designed a spatial demodulation robust multi-feature classifier for the crack detection of bridges, which is a detection method based on machine learning and the integration of multiple features. Experiments have shown that the classification accuracy of this method is high. Talab et al. [22] used multiple filters to process crack images and proposed a new image processing method to detect cracks. Shi [23] investigated CrackForest, a road crack detection framework based on a random structure forest. They used a new crack descriptor to more effectively identify cracks in a noisy environment. Kim et al. [24] presented a crack identification process using a hybrid image processing strategy with an unmanned aerial vehicle (UAV) that utilized two sets of optimal parameters, P w (optimal parameters minimizing estimation errors in crack width) and P l (optimal parameters minimizing estimation errors in crack length), to accurately detect the crack width and minimize the loss of length. Jeon et al. [25] introduced the applications of a UAV in crack identification with a deep learning algorithm. The algorithm was adopted to effectively classify and locate images, and the identified cracks were automatically visualized on an inspection map using location matching.
Though these methods have certain advantages in various fields, some problems remain. First, when acquiring images, experimenters need to calibrate the camera's unit pixel size [26,27] in a very cumbersome way, such as the scale method [28] of positioning rulers at the crack position, which is not applicable to all bridge types. Relatively speaking, the object distance method does not require cumbersome calibration, the operation is simple, and it is suitable for almost all bridge types. However, there is no uniform standard for the calculation of crack size. Second, most current methods for calculating crack length refine the crack into a single pixel skeleton along the direction of the crack and then calculate it by counting the number of pixels. However, these methods heavily rely on the choice of threshold for image binarization. When the threshold value is large, the ends of the cracks and some small cracks are often ignored, which can result in missing pixels, making the calculated value smaller than the actual length. When the threshold value is selected to be small, the crack is wide, and a large amount of noise is present, the problem of pixel redundancy occurs, resulting in a calculated result that is larger than the actual value. Third, most image stitching algorithms [29,30] tend to apply point-to-point matching when feature points are matched. There is no correlation between feature points, which often causes mismatching. Then, the mismatched points should be eliminated, which results in low computational efficiency. To the authors' knowledge, the associated researchers did not check the stitching accuracy after they finished image stitching.
In this paper, a crack information extraction method is proposed for length calculation and image mosaic, and a data acquisition system with a camera and a laser range finder was built to provide the crack image and working distance needed for data processing. Some main aspects of the work are as follows: • The object distance method and the scale method were compared for the aim of estimating the crack width, and the results showed that the object distance method was more efficient than the ruler method.

•
The method of "double edge skeleton method" was adopted after analyzing the disadvantages of common crack length calculation methods. First, the edge information of the crack was preserved as much as possible by setting the appropriate threshold. Then, the number of pixels in the edge of the crack was calculated by region. Third, redundant pixels with wide cracks and large amounts of noise were removed, and the missing pixels of narrow cracks were added. This method greatly improved the calculation accuracy.

•
The scale-invariant feature transform (SIFT) algorithm was used and improved to correlate and compare feature points in the image mosaic. The improved algorithm could achieve the priority matching of feature points, thereby improving the splicing efficiency. After the image mosaic was created, the accuracy was verified by comparing the mosaic image with the actual structure.

Image Binarization
Crack width is obtained as the distance between the edge pixels nearest to the center skeleton pixel, as shown in Figure 1. In order to obtain crack size, the crack edge must be detected. In the broad sense, the sudden change of an object is called the edge. In image processing, the edge is often detected by abrupt changes in the grey value. Edge detection must meet two conditions: it must effectively suppress noise and locate the edge as accurately as possible. Image binarization [31,32] is one of the most commonly used image processing methods. In the binarization process, a specified threshold is set for the pixel value of the greyscale image. When the greyscale pixel value is lower than the corresponding threshold, the binarization result is zero (black). In contrast, the binarization result is one (white) when the pixel value is higher than the threshold [24]. Therefore, image binarization is the primary task of calculating crack width. For the acquired image, after the correction and the obtaining of the grayscale image, binarization can be performed.
The Canny edge detection algorithm is the most widely used edge detection algorithm. The algorithm adds non-maximum suppression and double threshold improvement based on the first order differentiator. Non-maximum suppression can eliminate the nonlocal maximum in the first order differentiator, and the double threshold can reduce the missed detection rate of the edge. However, the Canny operator requires manual threshold setting. A threshold setting that is too high can result in edge loss, and one that is too low cannot detect the true edge, thus reducing detection efficiency. However, the Otsu method [33] can avoid this situation. In this paper, the combination of the Otsu method and Canny operator [34,35] was used for edge detection. The results are shown in Figure 2. It can be seen that this method could accurately identify the edges.

Calculation Principle of the Two Methods
There are two methods for estimating crack size. One is the scale method, which is calculated according to the principle of proportional conversion. This method requires a square of a known size to be attached to the inspection site, and the size of the crack is converted by the square size. The calculation principle is shown in Figure 3. The other is object distance method, which is calculated according to the optical triangle similarity theory. This method only needs to know the working distance and camera focal length during image acquisition, as shown in Figure 4. The object distance method and the scale method were compared in this paper to achieve a more accurate estimation of the crack width. The width information calculated by the two methods was then converted to metric units using the following equation: where W r is the real crack width in metric units (mm), W PC is the obtained crack width in pixels, W PS is the dimension of the scale in pixel, l is the dimension of the scale, D P is the pixel resolution of the imaging system, a is the pixel size, D is the distance in mm from the object plane to the camera chip, and L f is the focal length of the camera in mm. The calculation principle diagram of the pixel resolution D P is shown in Figure 4.  The comparison process of the two methods accuracy is shown in Figure 5.

Morphological Screening of Crack Images
Crack length estimation [36] is challenging due to irregularities in crack development. Traditional manual measurements have large errors, and thus, image processing technology has obvious advantages in length estimation. Crack length is generally calculated by counting the number of pixels. As seen from the previous section, although the Canny edge detection algorithm combined with the Otsu method eliminates most of the noise, there is still a small amount of isolated noise that is difficult to eliminate. Therefore, in order to estimate the crack length, it is necessary to eliminate the noise as much as possible. This can be done by iterating through all the pixels, looking for all connected fields, as shown in Figure 6, and marking them with their circumscribed rectangles. One can then assume that the size of the circumscribed rectangle of the p th connected field is M p × N p and S p is the area of the connected field (i.e., the number of pixels). When the pixel is black, x i,j = 0, and when the pixel is white, x i,j = 1. The area S p is defined as: One can then set a threshold A for the area of the connected domain [37], according to experience. If S p < A, the connected domain is removed; otherwise it is reserved. The image after noise removal is shown in Figure 7.

Crack Length Estimation Based on Double Edge Pixel Statistics
The central skeleton method [38] is currently the most commonly used method. This method refines the crack into a single-pixel skeleton, and estimates the crack length by counting the number of skeleton pixels. However, the pixel missing and redundancy problems caused by this method are often difficult to compensate (see Figure 8). In order to solve the problem of missing and redundant central skeleton pixels, this paper proposes double edge skeleton method. By counting the number of pixels by region, the missing pixels can be complemented while eliminating the effects of redundant pixels, as shown in Figure 9.  According to Equation (4), it is assumed that the total area of all connected domains is S T ; then, the crack length is defined as: where L is the total crack length in mm; n and m are the area numbers of the redundancy and deficiency parts, respectively; S Ri and S Dj are the areas of the ith redundant region and the jth deficient region, respectively; and D P is the pixel resolution of the imaging system.

Method of Mosaicing
In the process of acquiring images, a crack may be scattered in multiple images due to various conditions, which increases the difficulty of obtaining crack information. Therefore, it is necessary to splice multiple crack images into a complete crack image to obtain more comprehensive information about the crack. In the process of image stitching, it is necessary to adjust the size and angle of the original second image and then stitch it with the original first image. The precondition for stitching two adjacent images is that they must share an overlapping area. The larger the proportion of the overlapping area is, the higher the stitching precision. The main task of splicing is to determine the overlap degree of adjacent images in the directions of the height and width and then remove the overlapping area to obtain a complete panoramic image [39]. The splicing process is shown in Figure 10. Image pre-processing primarily corrects the distortion of the original image and suppresses the noise points to prepare for the next step of image registration. Distortion correction can eliminate obvious geometric distortion in the original image, and noise point suppression can avoid the occurrence of some mismatches to the greatest extent. Image registration [40] is mainly used to extract the matching information in the original image and to perform the spatial alignment and greyscale fusion of multiple images taken in the same scene at different moments with different light intensities, different sensors and different angles. Among the constituent steps, feature matching is the basis of image registration. Feature matching refers to the finding of similar parts of two images to be registered based on feature points or greyscale to find the image similar to the search image.
A commonly used feature matching method is the SIFT [41][42][43][44][45] method. The essence of the SIFT algorithm is to find feature points in different scale spaces and calculate the direction of the feature points. These feature points are some very prominent points that do not change due to factors such as the lighting, image size, and noise. However, in the process of matching key points, the SIFT algorithm often produces mismatching, and the mismatching points need to be eliminated later. Therefore, based on the SIFT algorithm, this paper incorporated the feature point correlation comparison and priority matching algorithm.
The first step is coarse adjustment, which plays a role in positioning. First, traverse all the feature points on the left and then find the corresponding points of the feature points in the right image. Each feature point may have many corresponding points in the right image. Since the feature point association comparison process is complicated, three feature points are taken as an example. Take any of the three feature points in the left figure and record them as P 1,l , P 2,l , P 3,l (see Figure 11a). These three feature points form three sets in the points corresponding to Figure 11b, recorded as P 11 , P 12 . . . P 1i . . . P 1m ; P 21 , P 22 . . . P 2 j . . . P 2n ; and P 31 , P 32 . . . P 3k . . . P 3r . There are m × n × r combinations that include a point from each of these three sets. Suppose that the distance between any two points P 1 and P 2 is P 1 P 2 . The distances of the three feature points in the left image and the distances of the three points in all combinations of the right image are associated. The relationship between the distances of the three points in the left figure and the distances of the three points corresponding to the right picture is denoted by ξ, as shown in Equation (5). Let l = 1 and find the i, j, k that minimize ξ; if the result is not unique, let l = l + 1 until the result is unique. At this time, P 1i , P 2 j and P 3k are the points corresponding to the three feature points in the left figure. The right image is scaled by the proportionality factor α, rotated appropriately, and then moves in the direction of the left figure so that the three points coincide, thus completing the coarse adjustment. ξ = P 1,l P 2,l − αP 1i P 2 j + P 2,l P 3,l − αP 2 j P 3k + P 1,l P 3,l − αP 1i P 3k , where α is the proportionality factor which can be obtained using the Gaussian Pyramid. The second step is to fine tune the alignment. After the first step of feature point positioning, the image mosaic has basically been completed. However, due to the difference in the number of matching points, although the feature points have been correlated, some stitching errors will inevitably occur. An insufficient or excessive number of matching points may also adversely affect stitching accuracy. The second step is performed to avoid these conditions, and the process is as follows. Adjust the coefficient distRatio that controls the number of matching points in the Match.m file to obtain different numbers of matching points. As the number of matching points changes, the stitching result is adjusted accordingly. By adjusting this factor, the stitched image can be fine-tuned.
Taking Figure 12 as an example, Figure 12a,b, which are to be stitched, are undistorted images. After processing by the SIFT algorithm, Figure 12a yields 6501 detected key points, and Figure 12b yields 5450 detected key points. Then, by assigning two values to distRatio, the numbers of matching points obtained are 2139 and 1936 respectively. The matching results are shown in Figure 13. It can be seen that although there are more matching points in Figure 13a, there is a phenomenon of matching confusion, but Figure 13b does not show this effect. The results of the stitching are shown in Figure 14. Obviously, Figure 14b has a better stitching effect.

Effect Detection of Mosaicing
To evaluate the stitching effect, this paper evaluated the accuracy by comparing the ratio of the distance on the stitched image to the actual distance. In the stitched image, the point p 2 of the stitching is selected (see Figure 15a), points p 1 and p 3 are selected on the left and right sides, respectively, and the pixel distance between the three points is measured. The ratio between the three distances is then calculated, as shown in Equation (6). These three points on the beam are found, and a steel ruler is used to measure the actual distance between the three points (see Figure 15b). Then, the ratio of the three points in the same way can be calculate, as shown in Equation (7). The two sets of ratios to obtain the stitching error of each part can be compared. The average stitching error ω is defined as:

Crack Image Acquisition System and Test Model
Laboratory testing was conducted in this study to effectively obtain the size information of the crack. The experimental model is two concrete test beams with dimensions of 150 × 300 × 2200 mm in the Wind Tunnel Laboratory in the Northeast Forestry University. By controlling the loading force, the two beams can be loaded to contain cracks almost entirely in the width. The test used an acA1300-30 µm Basler industrial black and white charge-coupled device (CCD) camera (see Figure 16). The camera does not require an external power supply and uses only a data cable connected to the computer's USB3.0 interface, which enables convenient operation. The camera sensor chip size is 4.9 × 3.6 mm with 1.3 million pixels, the horizontal/vertical resolution is 1296 × 966 px, and the horizontal/vertical pixel size is 3.75 × 3.75 µm. To enact the procedure, one must adjust the tripod to level the camera and rotate the camera to hold the camera's optical axis perpendicular to the plane of the test beam to capture an image. Then, one must use an HCIYET HT-310 Hongcheng laser range finder to measure the distance between the camera and the subject. The crack image acquisition system is shown in Figure 17.

Crack Width
In this paper, six cracks of different brightness, shape and width were selected for characterization. Six measuring points were selected for each crack, so a total of 36 sets of crack width data were obtained. The crack width was calculated by the scale method and the object distance method, and the accuracy of the two methods was compared. The crack width estimation results are shown in Table 1:  (20) According to the data in Table 1, the error comparison curves of the object distance method and the scale method were obtained (for several cracks with the same width but different errors, the error was averaged), as shown in Figure 18. The identification accuracy of the two methods increased with increasing crack width. When the crack width was less than 0.2 mm, the accuracy of the object distance method was obviously higher than that of the scale method. A detailed comparison of the errors is shown in Table 2. The comparison results showed that when the crack width was greater than 0.20 mm, the average error of both methods was within 8.2%. When the width was less than 0.20 mm, the error of the scale method was larger and the average value was more than 30%. When the crack width was between 0.10 and 0.20 mm, the average accuracy of the object distance method could reach 90%; when the crack width was 0.10 mm or less, the accuracy of the object distance method could not meet the measurement requirements. In summary, the object distance method had a high accuracy and could accurately identify cracks with a width above 0.10 mm.

Crack Length
In this paper, five cracks of different brightness, shape and width were selected for length estimation. A cotton thread was laid along the crack for comparison with the calculated results. The calculation errors are shown in Table 3. The proposed method could accurately estimate the length of the crack with a maximum error of 8.3%.

Crack Mosaicing
This experiment aimed to achieve the research goal by stitching several groups of crack images. Due to the length limit of this article, only five cracks with different brightness, shape and width are selected for illustration (see Figure 19). The stitching results are shown in Figure 20, where the number represents the crack width at that particular pixel.

Accuracy Validation
To evaluate the stitching effect, the stitching accuracy of the above five images was verified, and the results are shown in Table 4. It can be seen that the crack splicing accuracy could reach 98%. Therefore, the algorithm can accurately splice the cracks of disperse brightness, shape and width and retain complete crack information.

Discussion and Future Work
It can be seen from the calculation results of the crack width that the calculation accuracy of the two methods increased as the crack width increased. However, as the crack width decreased, the trends of the accuracies of the two methods differed. The accuracy of the optical similarity method represented by the object distance method was much higher than that of the scale conversion method represented by the scale method, possibly because the calibration plate inevitably had a certain error, and the square border of the calibration plate itself occupied a certain width. When the crack width was small, the error caused by this width could not be ignored. However, there was no such problem with optical similarity rules. In general, for the proposed crack length calculation method, the calculation accuracy should increase as the crack length increases when the crack quality is good. However, this rule was not shown in this experiment because the calculation of the crack length was based on image processing technology. The image processing technology was not only related to the actual length of the crack but also to the clarity of the image, the influence of noise, the width and shape of the crack, and other factors. Due to these factors, when the number of cracks was small, the calculation results showed no regularity. Additionally, by improving the splicing program and verifying the splicing accuracy, the splicing effect was quantified in this paper, which could directly reflect the development of the cracks.
In recent years, crack detection methods based on UAVs have attracted widespread attention due to their stability and high efficiency. The cloud platform of the UAV can ensure that the camera is completely horizontal during the shooting process. A drone image acquisition system equipped with a laser range finder can simultaneously acquire the crack image and the corresponding focal length and working distance in the front view of the camera. However, to obtain a complete crack, the use of the object distance method requires the controlling of the flying speed of the UAV and the shooting time interval, which is undoubtedly inefficient. Therefore, in the future, the range finder should be replaced with two laser emitters with a pitch of 10 cm mounted on both sides of the camera so that the emitted laser light is always within the shooting range. Then, the UAV can calculate the crack size using a video by taking the emitted laser as a ruler, which will greatly improve the detection efficiency. Since a laser spot has a certain size, improving measurement accuracy is still a difficult aspect of crack detection.

Conclusions
In this paper, the sizes and trends of cracks were studied by image recognition. First, the crack width was computed using the scale method and the object distance method, and the accuracies of the two methods were compared. Next, a double edge pixel statistical algorithm was proposed to calculate the crack length. Finally, an improved algorithm based on SIFT feature matching was proposed to solve the problem of crack image mosaic. The results of the experimental evaluation can be summarized as follows: 1.
The object distance method can accurately measure cracks with a width of more than 0.10 mm, and its accuracy can reach 90%, much higher than the scale method.

2.
Accuracy can be improved by calculating the length of the crack using the double edge skeleton method. Compared with the measured results, the accuracy of crack length was up to 92%. 3.
The improved splicing algorithm can accurately and quickly acquire the complete shape of the crack, and the splicing precision can reach 98%.
Consequently, the results show that the double edge skeleton method and the improved SIFT splicing algorithm can effectively and reliably identify cracks.

Conflicts of Interest:
The authors declare no conflict of interest.