Benchmarking Image Processing Algorithms for Unmanned Aerial System-Assisted Crack Detection in Concrete Structures

Abstract

This paper summarizes the results of traditional image processing algorithms for detection of defects in concrete using images taken by Unmanned Aerial Systems (UASs). Such algorithms are useful for improving the accuracy of crack detection during autonomous inspection of bridges and other structures, and they have yet to be compared and evaluated on a dataset of concrete images taken by UAS. The authors created a generic image processing algorithm for crack detection, which included the major steps of filter design, edge detection, image enhancement, and segmentation, designed to uniformly compare different edge detectors. Edge detection was carried out by six filters in the spatial (Roberts, Prewitt, Sobel, and Laplacian of Gaussian) and frequency (Butterworth and Gaussian) domains. These algorithms were applied to fifty images each of defected and sound concrete. Performances of the six filters were compared in terms of accuracy, precision, minimum detectable crack width, computational time, and noise-to-signal ratio. In general, frequency domain techniques were slower than spatial domain methods because of the computational intensity of the Fourier and inverse Fourier transformations used to move between spatial and frequency domains. Frequency domain methods also produced noisier images than spatial domain methods. Crack detection in the spatial domain using the Laplacian of Gaussian filter proved to be the fastest, most accurate, and most precise method, and it resulted in the finest detectable crack width. The Laplacian of Gaussian filter in spatial domain is recommended for future applications of real-time crack detection using UAS.

1. Introduction

The United States is home to more than 600,000 bridges, more than one-third of which include a concrete superstructure or wear surface [1]. These bridges require a variety of periodic inspections in accordance with federal regulations. The most common inspection type is routine inspection, wherein the inspector scans the bridge deck to identify surface degradation or surface cracking. Such inspections are costly, time-consuming, and labor-intensive [2,3]. Autonomous inspection could be a cost-effective solution to these problems if the accuracy of human inspection can be matched [3,4,5,6,7,8,9]. Image-based inspection of infrastructure for concrete delamination [10,11,12,13], cracks [14,15,16,17], and spalls [18,19] using unmanned aerial systems (UASs) have been proven effective based on previous literature [20].

Image-based autonomous inspections still require human inspectors to review images. The number of images collected depends on a number of factors, but it is commonly in the several thousands. For instance, SDNET2018 [21], with more than 56,000 labeled images of concrete structures, covers three small lab-made bridge decks, walls of a building, and several paved sidewalks, which are significantly smaller than common inspected infrastructures in practice. Manual identification of flaws in such large image sets is time consuming and prone to inaccuracy because of inspector fatigue or human error [22,23,24,25,26]. Image processing algorithms can improve the accuracy and efficiency of autonomous inspections by either (a) enhancing images to improve ease of human detection of defects or (b) autonomously identifying defects. Additionally, edge detectors are used in combination with more contemporary techniques such as deep learning convolutional neural networks for UAS applications [27], reducing false positive cases by 20 times compared to sole use of edge detectors [28].

Cracks in a two-dimensional (2D) image are classified as edges, and, thus, existing edge detection algorithms are likely candidates for crack identification. 2D images are represented mathematically by matrices (one matrix, in the case of greyscale images, or three matrices in the case of red/green/blue color images). An ideal edge is defined as a discontinuity in the greyscale intensity field. Crack detection algorithms can emphasize edges by applying filters in either the spatial or frequency domain. Even though use of edge detectors for crack detection goes back to the early 2000s [29], these methods have been used in the past and are still being used in recent studies because of their simplicity and pixel-based detection of cracks [15,28,30,31,32,33,34,35,36,37,38,39,40,41,42]. Even in emerging applications of supervised machine learning methods, edge detectors are still considered in practice since they do not require expensive annotated training datasets. Using edge detectors for crack detection is computationally fast, making it an appealing option for real-time crack detection during UAS inspections; they have been implemented in ground-based robotic inspections in the past [4,18,38]. Edge detectors can have different sizes, shapes, and values. There are limited guidelines for researchers and practitioners in choosing a proper edge detector for their applications, especially for UAS collected data.

Save two noteworthy exceptions, most research focuses on developing new methods for crack detection rather than comparing the performance of existing methods. Abdel-Qader et al. [29] compared the performance of the fast Haar transform, Fourier transform, Sobel filter, and Canny filter for crack detection in 25 images of defected concrete and 25 images of sound concrete. The fast Haar transform was the most accurate method, with an overall accuracy of 86%, followed by the Canny filter (76%), Sobel filter (68%), and the Fourier transform (64%). Processing time was not considered, and no actual definition of accuracy was presented. Mohan and Poobal [43] reviewed a number of edge detection techniques for visual, thermal, and ultrasonic images, but the information presented was from several studies that considered vastly different data sets, and so the results were not directly comparable. This paper presents the results of a direct comparison of four common edge detection methods in the spatial domain (Roberts, Prewitt, Sobel, and Laplacian of Gaussian) and two in the frequency domain (Butterworth and Gaussian) by applying them to a dataset of 50 sound and 50 defected images of concrete collected from UAS. The goal of this study is to determine the most efficient edge detectors to use in aiding UAS condition assessments of concrete structures when all the other parameters are kept the same. Prior to implementation, as an emerging technology in bridge inspections, UAS need a rigorous investigation as part of a process model to determine whether they can be used in lieu of hands-on inspections. While this is not a part of this study, such an approach would involve investigating the reliability and effectiveness of UAS inspections through a generic decision making tool that includes a multivariable analysis of the inspection accuracy, cost, time, and hazard.

2. Analytical Program

Figure 1 shows a generic image analysis algorithm developed for this study. The generic algorithm included three main steps: Edge detection, edge image enhancement, and segmentation. Edge detection in the spatial domain involved greyscale conversion and application of a filter. Edge detection in the frequency domain required additional steps to transform the image from the spatial domain to the frequency domain before application of the filter. The inverse operation to transform the filtered image back to the spatial domain was also an additional step. This section includes the particulars of each step in the generic image processing algorithm.

2.1. Greyscale Conversion

Edge detection algorithms perform best with greyscale images [44], so the first step in the image analysis procedure was greyscale conversion of color images. The original color image comprised a matrix of pixels, each with a defined red, green, and blue intensity. Greyscale conversion followed Equation (1), where $\mathrm{I}\left(\mathrm{x},\mathrm{y}\right)$ is the grayscale intensity of pixel $\left(\mathrm{x},\mathrm{y}\right)$, and $\mathrm{R}\left(\mathrm{x},\mathrm{y}\right)$, $\mathrm{G}\left(\mathrm{x},\mathrm{y}\right)$, and $\mathrm{B}\left(\mathrm{x},\mathrm{y}\right)$ are the red, green, and blue pixel intensities of the same, respectively.

$$I\left(x,y\right)=0.2989R\left(x,y\right)+0.5870G\left(x,y\right)+0.1140B\left(x,y\right).$$

(1)

2.2. Edge Detection in the Spatial Domain

In general, edge detection in images requires filtering by one of several common methods, which are discussed in detail below. Filters are applied as a small matrix of values (called a kernel) through a mathematical operation known as convolution. In general form, the convoluted image $O$ is the sum of the element-by-element products of the image intensity matrix $I$ and the kernel $K$ in every position in which $K$ fits fully inside $I$. Equation (2) describes this in plainer terms for image size $\mathrm{M}\times \mathrm{N}$ and kernel size $\mathrm{m}\times \mathrm{n}$.

$$O\left(i,j\right)={\sum }_{k=1}^m{\sum }_{\ell =1}^{n}I\left(i+k-1,j+\ell -1\right)K\left(k,\ell \right),$$

(2)

The convoluted image $O$ will be of size $\left(\mathrm{M}-\mathrm{m}+1\right)\times \left(\mathrm{N}-\mathrm{n}+1\right)$. The kernel typically includes both $\mathrm{x}$ and $\mathrm{y}$ components; the convoluted images ${\mathrm{E}}_{\mathrm{x}}$ and ${\mathrm{E}}_{\mathrm{y}}$ obtained from the $\mathrm{x}$ and $\mathrm{y}$ components of the filter emphasize vertical and horizontal edges, respectively. The final edge image $\mathrm{E}$ is the square root of the sum of the squared component images, i.e.,

$$O\left(i,j\right)={\sum }_{k=1}^m{\sum }_{\ell =1}^{n}I\left(i+k-1,j+\ell -1\right)K\left(k,\ell \right).$$

(3)

Common edge detecting filters in the spatial domain include Roberts, Prewitt, and Sobel. Equations (4)–(6) give the $\mathrm{x}$ and $\mathrm{y}$ kernels for the Roberts (${\mathrm{R}}_{\mathrm{x}}$ and ${\mathrm{R}}_{\mathrm{y}}$), Prewitt (${\mathrm{P}}_{\mathrm{x}}$ and ${\mathrm{P}}_{\mathrm{y}}$), and Sobel (${\mathrm{S}}_{\mathrm{x}}$ and ${\mathrm{S}}_{\mathrm{y}}$) filters. These filters compute the gradient between neighboring pixels in the $\mathrm{x}$ and $\mathrm{y}$ directions and intensify areas of high gradient (i.e., edges). Filters are constructed such that the components are of opposite sign and the sum of all components is zero. The Roberts filter (Equation (4)) is a compact kernel, which could lead to very fast processing times. The Prewitt (Equation (5)) and Sobel (Equation (6)) filters use larger $3\times 3$ kernels and are therefore more powerful but require extended computation times. The Prewitt is a first-order filter (the largest magnitude component is one); the second-order Sobel filter will likely produce an image with more intensified edges.

$$R_x=\left[\begin{array}{cc}1& 0\\ 0& -1\end{array}\right], R_y=\left[\begin{array}{cc}0& 1\\ -1& 0\end{array}\right],$$

(4)

$$O{P}_x=\left[\begin{array}{ccc}-1& 0& 1\\ -1& 0& 1\\ -1& 0& 1\end{array}\right],P_y=\left[\begin{array}{ccc}1& 1& 1\\ 0& 0& 0\\ -1& -1& -1\end{array}\right],$$

(5)

$$S_x=\left[\begin{array}{ccc}-1& 0& 1\\ -2& 0& 2\\ -1& 0& 1\end{array}\right],S_y=\left[\begin{array}{ccc}1& 2& 1\\ 0& 0& 0\\ -1& -2& -1\end{array}\right],$$

(6)

Another popular edge detection method in the spatial domain is the Laplacian of Gaussian (LoG) function. When applied to an image with intensities $\mathrm{I}\left(\mathrm{x},\mathrm{y}\right)$, the Laplacian operator ${\nabla }^2=\frac{{\partial }^2\mathrm{I}}{\partial {\mathrm{x}}^2}+\frac{{\partial }^2\mathrm{I}}{\partial {\mathrm{y}}^2}$ emphasizes both edges and noise or artifact. The influence of noise can be reduced by first applying the Gaussian smoothing filter given by Equation (7), where $\mathrm{x}$ and $\mathrm{y}$ are the spatial coordinates within the Gaussian kernel and $\mathsf{\sigma }$ is the standard deviation

$$G\left(x,y\right)= \frac{1}{\sqrt{2\pi {\sigma }^2}}exp\left(-\frac{x^2+y^2}{2{\sigma }^2}\right).$$

(7)

Equation (8) gives the Laplacian of the Gaussian, which can be pre-allocated for a given filter size m × n and standard deviation $\mathsf{\sigma }$.

$$LoG={\nabla }^2\left(G\left(x,y\right)\right)=\frac{x^2+y^2-2{\sigma }^2}{4{\sigma }^4}exp\left(-\frac{x^2+y^2}{2{\sigma }^2}\right).$$

(8)

Iterative optimization of the parameters $\mathrm{m}$, $\mathrm{n}$, and $\mathsf{\sigma }$ is possible on an image-by-image basis, but it is convenient to predefine both the size and standard deviation. For the purposes of this study, the LoG kernel was defined as a square matrix with size equal to 0.5% of the maximum image dimension, and the standard deviation was defined as one-fourth the maximum image dimension. At first glance, it would appear that the larger 13 × 13 LoG filter would be more computationally intensive than the smaller Roberts, Prewitt, and Sobel filters discussed previously. However, the LoG filter does not include $\mathrm{x}$ and $\mathrm{y}$ component kernels. Thus, only one convolution operation (Equation (2)) was required, and there was no need for the component transformation (Equation (3)).

2.3. Edge Detection in the Frequency Domain

Edge detection in the frequency domain requires transformation from the spatial domain to the frequency domain. This is quickly accomplished using the fast Fourier transform (FFT), which transforms the greyscale image intensities $\mathrm{I}\left(\mathrm{x},\mathrm{y}\right)$ into the frequency components $\mathrm{F}\left(\mathrm{u},\mathrm{v}\right)$. Unlike in the spatial domain, where the filter kernel is of arbitrary size, the filter kernel in the frequency domain is the same size as the image. The edge image $\mathrm{E}\left(\mathrm{u},\mathrm{v}\right)$ in the frequency domain is the element-by-element product of the filter kernel $\mathrm{K}\left(\mathrm{u},\mathrm{v}\right)$ and the frequency domain image $\mathrm{F}\left(\mathrm{u},\mathrm{v}\right)$, i.e.,

$$E\left(u,v\right)=K\left(u,v\right)\odot F\left(u,v\right),$$

(9)

where $\odot$ denotes element-wise multiplication. Inverse fast Fourier transformation (iFFT) of the frequency domain edge image $\mathrm{E}\left(\mathrm{u},\mathrm{v}\right)$ gives the edge image in the spatial domain $\mathrm{E}\left(\mathrm{x},\mathrm{y}\right)$.

The two most common frequency domain edge detection filters include Butterworth [45] and Gaussian [45,46,47] high pass filters. High pass filters attenuate frequencies above some defined cutoff frequency ${\mathrm{D}}_0$. Equation (10) gives the general form of the $\mathrm{n}$th-order Butterworth filter kernel ${\mathrm{K}}_{\mathrm{B}}\left(\mathrm{u},\mathrm{v}\right)$, where $\mathrm{D}\left(\mathrm{u},\mathrm{v}\right)$ is the distance between the pixel $\left(\mathrm{u},\mathrm{v}\right)$ and the origin of the frequency (the center of the M × N image), as defined by Equation (11).

$$K_B\left(u,v\right)=1-\frac{1}{1+{\left[\frac{D\left(u,v\right)}{D_0}\right]}^{2n}} ,$$

(10)

$$D\left(u,v\right)=\sqrt{{\left[u-\left(\frac{M}{2}+1\right)\right]}^2+{\left[v-\left(\frac{N}{2}+1\right)\right]}^2}.$$

(11)

Similarly, Equation (12) gives the general form of the Gaussian high pass filter kernel ${\mathrm{K}}_{\mathrm{G}}\left(\mathrm{u},\mathrm{v}\right)$, where D(u,v) is again the distance between the pixel $\left(\mathrm{u},\mathrm{v}\right)$ and the frequency origin, and $\mathsf{\sigma }$ is the assumed standard deviation of the frequency distribution.

$$K_G\left(u,v\right)=1-e^{\frac{-D^2\left(u,v\right)}{2{\sigma }^2}} .$$

(12)

For the purposes of this study, a fourth order (n = 4)Butterworth filter was constructed with cutoff frequency D₀ = M/10 The Guassian filter was constructed with standard deviation σ = M/10 Figure 2 presents a graphical representation of the Butterworth and Gaussian filters.

2.4. Edge Image Enhancement

Edge images $\mathrm{E}\left(\mathrm{x},\mathrm{y}\right)$ resulting from spatial or frequency domain edge detection filters contain a range of pixel intensities that require scaling. The scaling function given by Equation (13) converts edge image pixel intensities $\mathrm{E}\left(\mathrm{x},\mathrm{y}\right)$ to linearly scaled edge image pixel intensities ${\mathrm{E}}_{\mathrm{sc}}\left(\mathrm{x},\mathrm{y}\right)$ such that $0\le {\mathrm{I}}_{\mathrm{sc}}\left(\mathrm{x},\mathrm{y}\right)\le 1$.

$$E_{sc}\left(x,y\right)=\left[E\left(x,y\right)-min\left(E\right)\right]\left[\frac{1}{max\left(E\right)-min\left(E\right)}\right].$$

(13)

The scaled edge image ${\mathrm{E}}_{\mathrm{sc}}\left(\mathrm{x},\mathrm{y}\right)$ requires contrast adjustment to improve edge clarity. Equation (14) transforms the scaled edge image ${\mathrm{E}}_{\mathrm{sc}}\left(\mathrm{x},\mathrm{y}\right)$ into the enhanced edge image ${\mathrm{E}}_{\mathrm{e}}\left(\mathrm{x},\mathrm{y}\right)$, where ${\mathsf{\mu }}_{{\mathrm{E}}_{\mathrm{sc}}}$ and ${\mathsf{\sigma }}_{{\mathrm{E}}_{\mathrm{sc}}}$ are the mean and standard deviation of the scaled edge image pixel intensities, respectively.

$$E_e\left(x,y\right)=\left[E_{sc}\left(x,y\right)-min\left(E_{sc}\right)\right]\left[\frac{2{\sigma }_{E_{sc}}}{max\left(E_{sc}\right)-min\left(E_{sc}\right)}\right]+{\mu }_{E_{sc}} .$$

(14)

2.5. Segmentation

Segmentation was the final step in the proposed image analysis algorithm. This process converts the edge image to the binary image, in which pixels belonging to a crack take an intensity value of one, and the remaining pixels take an intensity value of zero. Selection of an appropriate threshold intensity—above which a pixel is classified as a crack and below which it is not—is critical. If the threshold intensity is too high, cracks go undetected. If it is too low, the image becomes noisy, and it is difficult to differentiate cracks from noise. This work considered two threshold operations for segmentation: pixel threshold and area threshold.

The pixel threshold operation follows Equation (15), where ${\mathrm{B}}_1\left(\mathrm{x},\mathrm{y}\right)$ is the first-level binary image, and ${\mathrm{T}}_1$ is the pixel threshold value.

$$B_1\left(x,y\right)=\{\begin{array}{c}0, E_e\left(x,y\right)<T_1\\ 1, E_e\left(x,y\right)\ge T_1\end{array}.$$

(15)

${\mathrm{T}}_1$ can be selected using Otsu’s method [47] or other intuitive/adaptive approaches [31]. In this study, ${\mathrm{T}}_1$ was selected based on the statistical properties of pixel intensities in the enhanced edge image ${\mathrm{E}}_{\mathrm{e}}\left(\mathrm{x},\mathrm{y}\right)$. Equation (16) defines ${\mathrm{T}}_1$, where ${\mathsf{\mu }}_{{\mathrm{E}}_{\mathrm{e}}}$ and ${\mathsf{\sigma }}_{{\mathrm{E}}_{\mathrm{e}}}$ are the mean and standard deviation of the enhanced edge image pixel intensities.

$$T_1= {\mu }_{E_e}+3{\sigma }_{E_e}.$$

(16)

Similarly, the area threshold operation follows Equation (17), where $B_2\left(x,y\right)$ is the second-level binary image and $T_2$ is the area threshold value.

$$B_2\left(x,y\right)=\{\begin{array}{c}0, B_1\left(x,y\right)<T_2\\ 1, B_1\left(x,y\right)\ge T_2\end{array}.$$

(17)

Equation (18) defines ${\mathrm{T}}_2$ according to the area of each connected component ${\mathrm{A}}_{\mathrm{cc}}$, where ${\mathsf{\sigma }}_{{\mathrm{A}}_{\mathrm{cc}}}$ is the standard deviation of the areas of connected components in ${\mathrm{B}}_1$.

$$T_2={\sigma }_{A_{cc}} .$$

(18)

The area of connected components ${\mathrm{A}}_{\mathrm{cc}}$ is determined according to eight-neighbor connectivity, which considers pixel connectivity in the vertical, horizontal, or diagonal directions, such that pixel $\left(\mathrm{x},\mathrm{y}\right)$ is connected to all pixels $\left(\mathrm{x}\pm 1,\mathrm{y}\pm 1\right)$. ${\mathrm{A}}_{\mathrm{cc}}$ could alternatively be defined according to four-neighbor connectivity, which is a stricter definition that only considers connectivity in the vertical and horizontal directions, such that pixel $\left(\mathrm{x},\mathrm{y}\right)$ is connected to pixels $\left(\mathrm{x}\pm 1,\mathrm{y}\right)$ and $\left(\mathrm{x},\mathrm{y}\pm 1\right)$. For the purposes of this research, the more relaxed eight-neighbor definition of connectivity was adopted.

The second-level binary image ${\mathrm{B}}_2$ is the final product of the proposed crack detection algorithm.

3. Experimental Program

In order to test the crack detection algorithm discussed above, the researchers gathered 50 images of sound concrete and 50 images of cracked concrete from several previously tested concrete panels at the Systems, Materials, and Structural Health Laboratory (SMASH Lab) at Utah State University. Images were taken with a 12 MP digital camera with focal length of 35 mm. The distance between the lens and the surface was approximately 0.3 m given the ability of the UAS to hold its position. The surface illumination, as verified by a Digi-Sense data logging light meter with NIST traceable calibration, was 1500–5000 lx. The image resolution was 2592 × 4608 px, and the approximate field size was $1.0\times 1.2$ m. RGB images were saved in JPEG format. Image processing was performed in MATLAB on a 64-bit operating system with 32 GB memory and 3.40 GHz processor. Figure 3 shows representative images of defected and sound concrete. Images were processed in six iterations, corresponding to the four spatial domain edge detectors and two frequency domain edge detectors.

Following image processing, an inspector reviewed the second-level binary images resulting from each of the six iterations in random order and classified each image as cracked or sound. The inspector reviewed only the second-level binary images; they did not review the original images or images from intermediate steps in the crack detection algorithm. The same inspector reviewed all of the images. The team then compared the results of each inspection to the ground truth, i.e., the known classification of each image as defected or sound based on physical inspection of the concrete surface aided by a crack microscope. The team then recorded the number of true positives (TPs), true negatives (TNs), false positives (FPs), and false negatives (FNs) for each iteration of the crack detection algorithm. A TP was a defected image in which the inspector accurately identified the defect. A TN was a sound image that the inspector accurately identified as sound. An FP was a sound image within which the inspector inaccurately identified a defect. An FN was a defected image that the inspector inaccurately identified as sound. A hit required the inspector to identify at least half of the actual crack length in a defected image. FP occurred when the inspector identified a crack in the noise or artifact of the second-level binary image. The performances of each approach of crack detection were evaluated in terms of accuracy, precision, processing time, and missed crack width (MCW). Accuracy, Ac., and precision, Pr., were calculated according to the following equations:

$$Ac.= \frac{TP+TN}{TP+FP+TN+FN} ;$$

(19)

$$Pr= \frac{TP}{TP+FP}$$

(20)

To obtain the processing time, each method of crack detection was run ten times on the same desktop and the same dataset, and the mean of the processing times was reported as each method’s processing time. In order to find MCW, the missed cracks by each crack detection method were identified and then measured using a crack microscope with 0.02 mm resolution. The algorithms were also compared in terms of the pixel intensity range in the enhanced edge images and the noise-to-signal ratio ($\mathrm{N}/\mathrm{S}$). A wider range of pixel intensities suggested a sharper contrast between defects and sound regions. $\mathrm{N}/\mathrm{S}$ described the level of noise or artifact in the image and was defined as the ratio of lit pixels (ones) to the total number of pixels in the second-level binary image ${\mathrm{B}}_2$. $\mathrm{N}/\mathrm{S}$ was only computed for the sound dataset because any lit pixels were known to be noise and not defects. A lower $\mathrm{N}/\mathrm{S}$ was greatly preferred because defects became more difficult to resolve when the image was noisy.

4. Results

Table 1. Performance of different edge detectors in the proposed crack detection algorithm.

Domain	Edge Detector	TPR1 (%)	TNR2 (%)	FPR3 (%)	FNR4 (%)	Ac.5 (%)	Pr.6 (%)	MCW7 (mm)	Time (s)
Spatial	Roberts	64	90	10	36	77	86	0.4	1.67
Spatial	Prewitt	82	82	18	18	82	82	0.2	1.4
Spatial	Sobel	86	84	16	14	85	84	0.2	1.4
Spatial	Laplacian of Gaussian (LoG)	98	86	14	2	92	88	0.1	1.18
Frequency	Butterworth	80	86	14	20	83	85	0.2	1.81
Frequency	Gaussian	80	88	12	20	84	87	0.2	1.92

¹ Ture Positive Rate, ² Ture Negative Rate, ³ False Positive Rate, ⁴ False Negative Rate, ⁵ Accuracy, ⁶ Precision, and ⁷ Missed Crack Width.

summarizes the results of the six iterations of the proposed crack detection algorithm. The TP rate (TPR), TN rate (TNR), FP rate (FPR), and FN rate (FNR) in the table are the percentages of TP, TN, FP, and FN reports. Note that the reported metrics in this paper were inclusive to the defined parameters in experimental and analytical procedures, and the authors were not suggesting that one could get similar results in practice with less-controlled situations.

In order to gain perspective about the numbers in this table, the accuracy of visual inspections should be determined. Washer et al. investigated the quality of element-level bridge inspection data. A required accuracy was arbitrary in visual inspections because the achieved accuracy was tied to the inspection process itself and was unaffected by the requirement [48]. Different inspectors responded very differently when inspecting the same structure or defect. The coefficients of variation of a rated bridge deck by different inspectors in visual inspections were between 57% and 96%. Detection of concrete deck cracks can be considered a “typical” decision in bridge inspections, and considering a normal distribution for all the calls, it is acceptable to have 67.5% of correct calls, i.e., ± one standard deviation, [23,26,48]. This meant each of the edge detectors, except for Roberts, had surpassed the acceptable accuracy of the visual inspections. A bridge inspector reviewed the images in this study and was able to label all of the images correctly into sound and defected images (100% accuracy). Unlike the edge detectors, the inspector did not localize the pixels associated with cracks in the images, since it was very time-consuming, to provide a pixel-segmented ground truth [28]. This gave the edge detectors an advantage over human inspectors.

4.1. Spatial Domain, Roberts Filter

Crack detection in the spatial domain using the Roberts filter resulted in the lowest number of TPs (32), but it also resulted in the lowest FP (5). Thus, while the Roberts filter was the least accurate (77%), its precision (86%) was among the highest. The minimum detectable crack width was 0.4 mm and was the largest of the six edge detectors evaluated. The processing time (1.67 s per image) was near the median of the six methods evaluated. Figure 4 shows representative enhanced edge, first-level binary, and second-level binary images from spatial domain edge detection of an image from the defected set (Figure 3a) using the Roberts filter.

4.2. Spatial Domain, Prewitt Filter

Crack detection in the spatial domain using the Prewitt filter resulted in the second lowest number of TP (41) and the highest FP (9). The Prewitt filter was the second least accurate and the least precise of the six methods evaluated. The minimum detectable crack width was 0.2 mm, which was comparable to four of the six methods. The processing time (1.40 s per image) was among the shortest. Figure 5 shows representative enhanced edge, first-level binary, and second-level binary images from spatial domain edge detection of an image from the defected set using the Prewitt filter.

4.3. Spatial Domain, Sobel Filter

Crack detection in the spatial domain using the Sobel filter resulted in the second highest number of TPs (43) and the second highest FP (8). Thus, while the Sobel filter was among the most accurate (85%), it was also among the least precise (84%). The minimum detectable crack width was 0.2 mm, and the processing time (1.4 s per image) was among the shortest. Figure 6 shows representative enhanced edge, first-level binary, and second-level binary images from spatial domain edge detection of an image from the defected set using the Sobel filter.

4.4. Spatial Domain, Laplacian of Gaussian (LoG) Filter

Crack detection in the spatial domain using the LoG filter resulted in the highest number of TPs (49), with only one miss in 50 defected images. The FP (7) was near the median for the six methods evaluated. Nevertheless, the LoG filter was the most accurate (92%) and the most precise (88%). Furthermore, the LoG method had the narrowest minimum detectable crack width (0.1 mm) and the shortest processing time (1.18 s per image). Figure 7 shows representative enhanced edge, first-level binary, and second-level binary images from spatial domain edge detection of an image from the defected set using the LoG filter.

4.5. Frequency Domain, Butterworth Filter

Crack detection in the frequency domain using the Butterworth filter resulted in the median number of TPs (40) and the median FP (7). The accuracy (83%) and precision (85%) were also near the median of the six methods evaluated. The minimum detectable crack width was again 0.2 mm, and the processing time (1.81 s per image) was the second longest of the six methods. Figure 8 shows representative enhanced edge, first-level binary, and second-level binary images from frequency domain edge detection of an image from the defected set using the Butterworth filter.

4.6. Frequency Domain, Gaussian Filter

Crack detection in the frequency domain using the Gaussian filter resulted in the median number of TPs (40) and the second lowest FP (12%). The accuracy (84%) was also near the median value, but the precision (87%) was the second highest. The minimum detectable crack width was again 0.2 mm. The processing time (1.92 s per image) was the longest of the six methods evaluated. Figure 9 shows representative enhanced edge, first-level binary, and second-level binary images from frequency domain edge detection of an image from the defected set using the Gaussian filter.

4.7. Comparison

Table 2. The average range and threshold value for each method in defected and sound datasets.

Edge Detector	Defected Dataset				Sound Dataset
	J1	J2	T1	T2	J1	J2	T1	T2	Average N/S (%)
Roberts	0.204	0.251	0.70	25	0.21	0.25	0.64	21	0.41
Prewitt	0.232	0.290	0.66	76	0.23	0.29	0.59	53	0.32
Sobel	0.230	0.291	0.67	75	0.23	0.29	0.59	53	0.33
LoG	0.534	0.590	0.71	58	0.62	0.69	0.63	32	0.90
Butterworth	0.581	0.631	0.89	10	0.57	0.64	0.93	6	1.74
Gaussian	0.594	0.640	0.89	8	0.58	0.64	0.93	5	1.76

presents a comparison of the range of pixel intensities in the enhanced edge image ${\mathrm{E}}_{\mathrm{e}}$, the pixel thresholds ${\mathrm{T}}_1$ and ${\mathrm{T}}_2$ used for construction of the first- and second-level binary images ${\mathrm{B}}_1$ and ${\mathrm{B}}_2$, and the noise-to-signal ratio $\mathrm{N}/\mathrm{S}$ observed in sound dataset using the six edge detection methods. Figure 10 presents a direct comparison of superimposed second-level binary images to the original image from analysis of the image in Figure 3a, a member of the defected dataset. Similarly, Figure 11 shows a direct comparison of the second-level binary images from analysis of the image in Figure 3b, a member of the sound dataset (only LoG and Gaussian filter results were illustrated for brevity).

Edge detection in the spatial domain using the LoG filter was the fastest of the six crack detection methods evaluated. Even though the difference between computational times was not very significant for one image (0.74 s between LoG and Gaussian), and considering it could take more than 1000 images to cover all areas of an infrastructure, using LoG would be roughly 10 min faster over all images than Gaussian, which was definitely significant. For example, current inspection procedures may only take 10 min of total overall inspection time for a routine inspection [9,49]. The effect of this time difference on an automated or semi-automated inspection is yet unknown, as these are not yet possible and could have very different processes; however, it could result in less on-site time or changes to the inspection procedure.

Frequency domain methods were expected to be the fastest because the element-wise product (Equation (9)) required far fewer floating-point operations than the iterative convolution operation (Equation (2)). However, the computational intensity of the Fourier and inverse Fourier transformations used to move between the spatial and frequency domains greatly increased processing time. The frequency domain methods took an average of 1.87 s per image, while the spatial domain methods took an average of 1.41 s per image. The LoG filter was expected to be computationally efficient compared to the other methods despite its comparatively large size ($13\times 13)$. The computational efficiency of this method resulted from the fact that LoG used only one kernel, as opposed to $\mathrm{x}$ and $\mathrm{y}$ component kernels in the other spatial domain methods. This reduced the number of convolution operations (Equation (2)) from two to one, which obviated the use of Equation (3). Computational efficiencies of the other spatial domain methods did not follow the expected trend. It was expected that processing time would increase with kernel size, and that the $3\times 3$ Prewitt and Sobel filters would require longer computational times than the $2\times 2$ Roberts filter. In fact, the opposite was true. Processing time for the Roberts filter was 20% longer than for the Prewitt or Sobel. The reader will recall the output image from Equation (2) is of dimension $\left(\mathrm{M}-\mathrm{m}+1\right)\times \left(\mathrm{N}-\mathrm{n}+1\right)$ for image size $\mathrm{M}\times \mathrm{N}$ and kernel size $\mathrm{m}\times \mathrm{n}$. Thus, a smaller kernel produced a larger edge image. This explained, at least in part, the increased computational time for the smaller Roberts filter. The LoG filter was also both the most accurate and the most precise of the six methods tested. The LoG method resulted in 98% TPs with only one miss among the fifty images in the defected dataset. The next most accurate method recorded seven misses. The remaining methods all recorded ten or more misses. Thus, the accuracy of LoG (92%) was significantly higher than the other five methods (77%–85%). The precision of LoG (88%), which also considered FP, was much closer to that of the other five methods (82%–87%). The LoG method recorded seven false positives in the 50 images in the sound dataset. The Roberts filter, with 18 misses, was by far the least accurate (77%). However, with only five false positives, Roberts was among the most precise (86%). Prewitt was the least precise with nine misses, nine false positives, and 82% precision.

The LoG filter resolved the finest cracks with an MCW of 0.1 mm. Most of the other methods were only able to resolve cracks 0.2 mm or wider. The Roberts filter could only detect cracks 0.4 mm or wider. Considering the image size used in this study, one pixel was equivalent to 0.2 mm. Thus, the LoG filter was useful in detecting cracks that were about one pixel wide, while Roberts could only resolve cracks that were two pixels wide.

The contrast adjustment ranges, segmentation thresholds ${\mathrm{T}}_1$ and ${\mathrm{T}}_2$, and noise-to-signal ratios $\mathrm{N}/\mathrm{S}$ listed in Table 1 gave some context to the performance metrics discussed above. The contrast adjustment ranges, J₁ and J₂, represented the range of pixel intensities in the enhanced edge image ${\mathrm{E}}_{\mathrm{e}}$. A wider range of contrast values (J₂–J₁) corresponded to more intensified edges within the image. Thus, cracks should be more easily detected when the contrast adjustment range was large. The Roberts filter, which performed poorly according to the performance metrics discussed above, exhibited the smallest range. The LoG filter, which arguably exhibited the best performance, had one of the widest contrast adjustment ranges. Furthermore, the contrast adjustment range for the LoG filter was quite different between the defected and sound datasets. This resulted from a large number of pixels with high intensities in the defected image.

The noise-to-signal ratio $\mathrm{N}/\mathrm{S}$ was evaluated only for the sound dataset for the simple reason that the noise in sound images was more well-defined. In the perfectly ideal case, no pixels should be lit in the second-level binary image from the sound dataset. Thus, any lit pixels were noise by default. In the defected dataset, distinction between signal and noise was ill-defined. In general, spatial domain methods exhibited lower $\mathrm{N}/\mathrm{S}$ than frequency domain methods. The lowest $\mathrm{N}/\mathrm{S}$ ratios were observed for the Prewitt and Sobel filters, with $\mathrm{N}/\mathrm{S}$ ratios of 0.32 and 0.33, respectively. The Roberts filter exhibited only slightly more noise ($\mathrm{N}/\mathrm{S}=0.41$). In comparison, the LoG filter produced a fairly noisy edge image ($\mathrm{N}/\mathrm{S}=0.90)$.

Increased noise in the frequency domain manifested as an increase in the standard deviation ${\sigma }_{E_e}$ of the pixel intensities of the enhanced edge image $E_e$. Following Equation (18), this caused an increase in the pixel threshold $T_1$. While pixel thresholds were higher in the frequency domain, the area thresholds were lower. This resulted from reduced continuity of cracks in the frequency domain.

It was expected that the LoG method, which was the most successful in terms of accuracy, precision, MCW, and processing time, would also exhibit the least noise. Instead, the noise-to-signal ratio in the LoG images was among the highest observed. This can be explained in part by the images shown in Figure 10 and Figure 11. The presence of cracks in even the noisiest of images in Figure 10e and f was clear. Similarly, even in the sound images with the highest $N/S$ (Figure 11e and Figure 11f), it was easy to see that no cracks were present. Despite the large number of lit pixels, no pattern of connectivity was apparent, thus, the inspector could reasonably conclude that he or she was observing noise and not a defect. These images represented only a single data point for each method from defected and sound datasets. However, they suggested that the level of noise in the binary image was not the only factor affecting the inspector’s ability to detect cracks. Continuity of cracks in the binary image was also important, especially considering that the inspector needed to identify at least half of the crack in order to register a hit.

The value of area threshold ${\mathrm{T}}_2$ gave some idea of the continuity of cracks in the defected images. ${\mathrm{T}}_2$ was defined in Equation (18) as the standard deviation ${\mathsf{\sigma }}_{{\mathrm{A}}_{\mathrm{cc}}}$ of the areas of connected components ${\mathrm{A}}_{\mathrm{cc}}$. When the continuity of cracks in the binary image was poor (i.e., the cracks were discontinuous), ${\mathsf{\sigma }}_{{\mathrm{A}}_{\mathrm{cc}}}$ was small. Conversely, when the cracks in the binary image were highly continuous, ${\mathsf{\sigma }}_{{\mathrm{A}}_{\mathrm{cc}}}$ increased. Thus, higher values of ${\mathrm{T}}_2$ implied a higher degree of continuity of cracks in the binary image. Additionally, when the cracks were highly continuous in binary images from the defected dataset, the value of ${\mathrm{T}}_2$ would be much higher for the defected dataset than for the sound dataset. Such was the case for the Prewitt, Sobel, and LoG filters. The same was also true, but to a lesser degree, for the Butterworth and Gaussian filters. The values of ${\mathrm{T}}_2$ for the sound and defected datasets using the Roberts filter were similar. This suggested poor continuity of cracks in the binary images, which was confirmed in Figure 10a. Considering that the Roberts filter was among the worst methods tested here, this result was not at all surprising. The cracks in the rest of the binary images from the defected dataset (Figure 10b–f) were visibly more continuous.

The results presented here have some significant implications for future work in the realm of automated detection without human inspectors. For all of the evaluated methods, the pixel segmentation threshold ${\mathrm{T}}_1$ was higher for the defected dataset than for the sound dataset. The same was true for the area segmentation threshold ${\mathrm{T}}_2$. For the LoG method, the contrast adjustment ranges were also much different for the defected dataset than for the sound dataset. Future research can consider these differences as indicators of the likelihood that a particular processed image includes a defect.

5. Conclusions

The literature contains few investigations comparing different edge detection algorithms for accuracy, none of which are on UAS captured images. This study investigated a generic image-processing algorithm designed to objectively compare different edge detection algorithms on detection of defects in concrete. The algorithm involved edge detection, edge image enhancement, and segmentation. Edge detection was completed in the spatial domain using Roberts, Prewitt, Sobel, and LoG filters, and in the frequency domain using Butterworth and Gaussian filters. Fifty images of defected concrete and 50 of sound concrete were analyzed by the proposed algorithm in six iterations (making use of the six aforementioned edge detection strategies). An inspector reviewed the resulting binary images from each iteration to determine if the detector found a crack. The inspection results were compared to the ground truth, and the six edge detection methods were compared based on accuracy, precision, minimum detectable crack width, and processing time per image. Edge detection in the spatial domain using the LoG filter yielded the highest accuracy (92%) and precision (88%), the finest minimum detectable crack width, and the fastest processing time (1.18 s per image). All but one of the remaining methods (edge detection in the spatial domain using the Roberts filter) yielded greater than 80% accuracy, and they were able to detect cracks as fine as 0.2 mm. While crack detection in the spatial domain using the Roberts filter yielded the lowest accuracy (77%), it also yielded the fewest false positives (10%), and its precision (86%) was among the highest. In general, the processing time was longer for crack detection in the frequency domain (1.8–1.9 s per image) than in the spatial domain (1.2–1.7 s per image). Additionally, the second-level binary images (the final product of the image processing algorithm) were much noisier in the frequency domain. According to these results, crack detection in the spatial domain using the LoG filter yields the best and fastest results for detecting defects in concrete structures. Therefore, the proposed crack detection algorithm using LoG filter is recommended for real-time crack detection of concrete structures using UASs.