Multivariate Analysis of Concrete Image Using Thermography and Edge Detection

Abstract

With the growing demand for structural health monitoring system applications, data imaging is an ideal method for performing regular routine maintenance inspections. Image analysis can provide invaluable information about the health conditions of a structure’s existing infrastructure by recording and analyzing exterior damages. Therefore, it is desirable to have an automated approach that reports defects on images reliably and robustly. This paper presents a multivariate analysis approach for images, specifically for assessing substantial damage (such as cracks). The image analysis provides graph representations that are related to the image, such as the histogram. In addition, image-processing techniques such as grayscale are also implemented, which enhance the object’s information present in the image. In addition, this study uses image segmentation and a neural network, for transforming an image to analyze it more easily and as a classifier, respectively. Initially, each concrete structure image is preprocessed to highlight the crack. A neural network is used to calculate and categorize the visual characteristics of each region, and it shows an accuracy for classification of 98%. Experimental results show that thermal image extraction yields better histogram and cumulative distribution function features. The system can promote the development of various thermal image applications, such as nonphysical visual recognition and fault detection analysis.

1. Introduction

The issue of concrete infrastructure deterioration has become a global concern; maintenance or rehabilitation for infrastructure stability is necessary. In addition, unexpected expansion, external forces such as wind [1], and incremental loads contribute to structural aging, which increases maintenance, repair, or replacement costs [2,3,4]. Further, numerous variables can cause concrete infrastructure deterioration, such as mechanical stress, fatigue [5], and chemical and environmental conditions [6]. In the past, assessing infrastructure’s condition was done using human subjectiveness. However, when human subjectiveness is used to perform crack analysis, detection, and evaluation as a manual approach, the frequent outcome is time-consuming and error-prone, especially in large datasets. Psychophysical measurements are used to assess image quality in the perception of visual information by a human observer, which constrains the performance of human vision.

Unmanned techniques have been implemented to improve the accuracy and efficiency of inspection. Digital image processing is a widely used and standard method for detecting and evaluating potential cracks in structures [7,8,9]. An algorithm for concrete compressive strength detection using image processing and artificial neural networks (ANNs) was presented in Reference [10]. The presented method used five specific parameters, concrete samples, and acquired images. Multitarget multiclass defects in concrete structures found in civil infrastructures were classified using a deep multi-deformation-aware attention learning architecture, comprising a multiscale assembly of attention and fine-grained feature-induced attention modules [11]. An image-processing technique to provide concrete regression analysis has been employed [12]; the technique can be an auxiliary tool for destructive and nondestructive testing methods. Reference [13] used a convolutional neural network (CNN) to recognize the orientation of crack segments in concretes. In [14], an enhancement algorithm using the swarm optimization technique with adaptive cumulative distribution function (CDF) based on histogram equalization (HE) was proposed for iteratively optimized contrast enhancement and HE, which resulted in a combined function that converted the input images into quality images. An imaging-based detection technique for mapping crack damages in concrete structures has also been introduced [15]; this technique employed the gray-level co-occurrence matrix texture analysis approach integrated with ANN as a classifier to obtain surface damage information. Reference [16] presented an automated crack assessment and quantitative growth monitoring employing dual CNNs. An artificial-intelligence-empowered pipeline for image-based inspection of concretes was illustrated in [17]; this innovative approach comprised anomaly detection, extraction, and defect classification of regions from large image datasets. A review and model comparison using a machine learning (ML) approach for crack detection on a selective scale are presented in [18]. Additionally, Reference [19] used image processing to conduct an experimental study on microscopic mobbing characteristics of pedestrians in built corridors to provide the real-time characteristics of the structures with pedestrian interaction. The fundamental purpose of this research is to provide an image analysis that presents the graphical characteristics linked to images, such as histograms, which can be used for further study. Additionally, images in grayscale offer information on identifying the many types of objects that are included in images and good description of the surfaces of the objects that are present. Thermal imaging was also considered, which provides a more significant image than images taken from an ordinary camera. The image analysis algorithms were utilized for classification, defect detection, and segmentation, and neural network was used for pattern recognition. Both were used for image segmentation and pattern recognition with minimal human intervention.

To solve the issues mentioned, we focus on establishing sufficient information on images to analyze concrete conditions. In addition, to achieve the research objective, an image segmentation approach is implemented to enhance acquired images to simplify the study of the concrete conditions, and an ML technique is employed for detecting concrete defects. In the proposed method, the acquired images are initially preprocessed to remove unwanted information. Then, edge detection (ED) and filtering are used to highlight only the defect region on concrete images. The remainder of this paper is structured as follows: Section 2 addresses the related works. The proposed method is discussed and described in Section 3. Section 4 presents simulations, evaluation results, and interpretations. Finally, Section 5 concludes this study.

2. Review of Related Literatures

2.1. Image-Processing Analysis

Image processing is a technique for improving an image or extracting relevant information from it. Digital image-processing methods enable the alteration of digital images using computers. Fundamental image parameters include resolution, contrasts, dynamic range, and the signal-to-noise ratio. Details of these parameters are provided in Appendix A.

The representation of an image can take several forms, which can vary in regards to their color space (including aspects such as hue, saturation, or value) and even in graphical schemes. This representation conveys information, such as color, coded information, temperature mapping, and how an image is digitally preserved. Cumulative histograms and the cumulative distribution function (CDF) are the two common graphical representations of digital image processing. The detailed description of these two graphical representations is presented in Appendix A. Figure 1 shows the corresponding histogram of a sample image.

Figure 2 shows the histogram and CDF comparison between two images taken from an ordinary and a thermal camera, respectively. The image histogram was plotted using pixel values and the number of pixels. The histogram of the sample image taken from the ordinary camera exhibits a bimodal distribution. One peak represents the object pixels, whereas the other peak represents the background. Meanwhile, the histogram of the thermal image shows a good distribution of pixels over the entire intensity range. The histogram also shows most pixel values clustered in a small area; the top half of the intensity values are occupied by only a few pixels. The more the pixels are evenly distributed over the entire intensity range, the more easily the image can be transformed. The CDF is more linear in the thermal image, producing a more enhanced image. Thus, the representation of a thermal image in a different domain from the extracted features such as the histogram and CDF is helpful for postprocessing, e.g., in pattern selection, which can be used for classifying and assessing an image.

Using an image edge detection (ED) method, the object boundaries can be established on a per-image basis. These methods can be helpful for examining individual pixels and their nearby segments to determine portions of an image that have strong contrast. The method of identifying edges in image processing is known as ED [20]. Image segmentation is a necessary step in image analysis. The segmentation process separates an image into its components or objects that have the same texture or color. In this study, only Sobel edge detection is used in the simulation process. A brief discussion of edge detection is presented in Appendix A. Figure 3 shows the simulated results using the different ED techniques compared between an image dataset taken from an ordinary camera and a thermal camera. It shows that the output from the thermal camera is better than that from the ordinary camera.

2.2. Related Implementation

Below are some related works for concrete analysis that use image-processing techniques. Reference [21] described an online image-processing-based technique for rapidly and non-invasively detecting cracks in pressed-panel goods; however, this method used a standard camera, and the dataset was relatively small. An algorithm developed in [22] presented a feature detection approach that uses the Sobel operator to filter and denoise concrete images before implementing the Otsu method for thresholding segmentation for crack edge identification. Another implementation that uses the ED technique was presented in [23]; it analyzes crack identification for bridges. The presented work compared the crack detection results using the fast Fourier transform, Sobel filter, and Canny filter. A comparison of performance using deep CNNs and EDs for image-based crack detection in concrete structures yielded an 86% accuracy rate in the network that correctly detected the cracked images [24]. Finally, Reference [25] presented a CNN application for ground-penetrating radar images that automatically recognized, located, measured, and provided a three-dimensional reconstruction of concealed cracks. In addition, three distinct CNNs were constructed to automate the tasks mentioned above: recognition, location, and feature extraction.

3. Methodology

Below are presented the detailed technical specifications, algorithm, and setup used in this paper.

Table 1. Thermal camera technical specifications.

Brand/Company Name	FLIR E8
Field of View	45° × 34°
Object Temperature Range	−20 °C to 250 °C
Image Frequency	9 Hz
Thermal Sensitivity	<0.06 °C
Accuracy	±2 °C
Thermal Palettes	Iron, Rainbow, Grayscale
File Format	Radiometric JPG
On-board Digital Camera	640 × 480

under Experimental Setup provides the technical specifications of the thermal camera used in this study. The specifications for the physical setup in data acquisition are shown in Figure 4.

Experimental Setup.

Data Acquisition Setup.

3.1. Image Acquisition

The acquired dataset consists of 2700 thermal concrete images from various structures of universities in Daegu City, Republic of Korea, between November 2019 and September 2021. The proposed approach was simulated using the MATLAB platform. Detailed descriptions of the algorithm and experimental setup are discussed below.

3.2. Algorithm

The following sections describe each step of the proposed work. Pre-image processing was implemented to improve the raw image. All concrete images were then enhanced using different image-processing techniques as shown in Figure 5. Finally, a convolutional neural network approach was applied for automatic image classification to assess the accuracy of classification on the testing image, which is shown in Figure 6.

Below are the details of each processing block and the MATLAB syntaxes used in this study.

Step 1: Loading and Reading of Images.

Initially, the raw thermal concrete images were to be imported from the dataset folder. First, the MATLAB function “dir” prepared the listing of the files and folders in the current folder for the task. Then, the “imread” function was implemented at this stage.

Step 2: Resizing Images.

To visualize the change in the images, two MATLAB functions were implemented. The “imresize” function provided the image in the size necessary for the required task. Then, the “imshow” function displayed the image for visual verification. In this study, the base size of all sample images was set to a 720 × 576 pixel value (4:3 aspect ratio).

Step 3: Image Segmentation.

Image segmentation is used to transform an image representation into something more meaningful and easier to analyze. This study applied image segmentation to concrete images; only those with cracks were extracted. Furthermore, the clustering method called k-means clustering was performed for the segmentation process with a k value of 4.

Step 4: Grayscale-Level Image.

The resulting image from step 3 was converted to a grayscale image using the MATLAB function “rgb2gray,” which removed the hue and saturation content information while maintaining the luminance.

Step 5: ED Techniques.

This step presented the relative performance of ED techniques as SED. The ED technique was implemented and tested using a sample crack image. The objective was to produce a clean edge map by extracting the principal edge features of the image.

Step 6: Morphologic Noise Reduction

Aiming to detect concrete surface cracks, the Otsu algorithm is processed based on differential images. The Otsu method selects a threshold that reduces the intraclass variance of the black and white pixels that have been thresholded.

Figure 7 shows the different threshold iteration levels used in the sample image. The inverted image is depicted in Figure 8.

Step 7: Median Filtering of Grayscale Level.

In this study, median filtering was used to minimize noise while preserving the edges of the sample images. The median filter works by moving an image pixel by pixel and changing each value with the median value of the adjacent pixel. The median is calculated by first numerically sorting all pixel values in the window and then replacing the pixel under consideration with the middle (median) value. Further, as shown in Figure 9, a noise reduction algorithm was implemented with an appropriate filter iteration to enhance the sample images being tested, preventing any unnecessary data from being included in the noise reduction process.

As seen in Figure 10, the original image was further filtered to remove any unwanted noise in the target object of the image. Additionally, a clearer version of the filtered output image is shown in Figure 11.

3.3. Image Classifier

Following feature extraction and image enhancement, the proposed approach employs CNN [26,27] as a feature extractor and a support vector machine (SVM) as a classifier to categorize images. The results of this module section can then be used to pinpoint faults. This module section of the classification’s full description is as follows. The employed CNN can be used as a feature extractor and a classifier in various real-world applications [26]. This study used the Keras sequential model, which included convolutional, activation, and max-pooling layers. The first convolutional layer comprises 32 filters with 3 × 3 pixel dimensions. Following filtering, a ratio of 2 was employed to facilitate the max-pooling procedure.

The convolutional layer’s main job is to detect the local connections of features from the prior layer. The feature map output is subsequently transmitted to the activation layer, which is the ReLU. In vision systems, the max-pooling approach is used for two reasons: (1) to reject non-maximal values, which reduces layer calculation time; and (2) to execute down-sampling operations on 2 × 2 subregions to minimize the dimensions of the intermediate feature vectors. Then, the filters are piled together, and fully connected layers are used to compute the class scores. The output of the fully connected layers is used as the input feature vectors for the SVM classifier in the proposed model. At the final stage of the proposed defect detection, an SVM classifier was used instead of CNN’s softmax layer to find a hyperplane that divides the most significant fraction of a labeled dataset into subgroups appropriate for binary classification. The training data comprise pairs of training samples (x1, y1),...(xi, yi), where xi is the observation or input feature for the ith sample, and yi is the associated class label (x1, 0). The discriminant function that transforms an input feature space xi into a class label yi is the SVM classifier. Since a radial basis function was used as the kernel in SVM, cross-validation was performed to obtain the optimal kernel values. Moreover, the experimental setup was simulated via MATLAB environment for both the ground truth/actual and predicted labels; positive and negative values were assigned to defective and nondefective input images, respectively.

A confusion matrix is a representation of the performance of any classification model on a dataset. For reference, the two possible predicated classes are “yes” and “no”. In this paper, predicting non-defective concrete images means “yes”, while “no” would indicate cracked images. The rows of confusion matrix correspond to the predicted class (Output Class), while the columns correspond to the true class (Target Class). The diagonal cells represent correctly classified observations, and the off-diagonal cells are incorrectly classified observations. The matrix shows both the number of observations and the corresponding observed percentage equivalent to the total dataset number in each cell. Additionally, the row at the bottom of the confusion matrix provides the percentage equivalent of all datasets belonging to each class that are correctly and incorrectly classified. On the other hand, the column of the confusion matrix shows all the predicted percentages that belong to each class that are correctly and incorrectly classified. Lastly, the cell at the bottom right of the confusion matrix shows the overall accuracy of classification. Below are the definitions of each type of cell in a confusion matrix.

True Negative (TN): Predicted “no”; means that cracked images are classified correctly as “defective”.

False Positive (FP): Predicted “yes”; means that cracked images are classified inaccurately as “non-defective”.

False Negative (FN): Predicted “no”; means that non-defective concrete images are classified as “defective”.

True Positive (TP): Predicted “yes”; means that non-defective concrete images are classified correctly as “non-defective”.

Below are equations for the confusion matrix interpretation.

$$Accuracy=\frac{\mathrm{TP}+\mathrm{TN}}{\mathrm{TP}+\mathrm{TN}+\mathrm{FP}+\mathrm{FN}}$$

(1)

$$Precision=\frac{\mathrm{TP}}{\mathrm{TP}+\mathrm{FP}}$$

(2)

$$Sensitivity=\frac{\mathrm{TP}}{\mathrm{TP}+\mathrm{FN}}$$

(3)

$$Specificity=\frac{\mathrm{TN}}{\mathrm{TN}+\mathrm{FP}}$$

(4)

$$Negative Predictive Value=\frac{\mathrm{TN}}{\mathrm{TN}+\mathrm{FN}}$$

(5)

For this objective, two ML-based classifiers (SVM and ANN) were employed. A challenging aspect of developing ML-based classifiers is determining their parameters. In this study, the classifier parameters, such as the SVM kernel type, number of ANN layers, and number of neuron-nodes per layer, were correctly chosen based on earlier work [28] and subsequent experimental results.

4. Results and Discussion

Figure 12 illustrated the sample results of each step in the proposed image-processing method, for images taken from both ordinary and thermal cameras. It shows that using a thermal camera for concrete crack image acquisition could provide a better result when the proposed method was implemented. Images taken from both the ordinary camera and the thermal camera were subjected to different digital image processing. The figure illustrates that the thermal-imaging technique of this proposed algorithm is appropriate and can provide additional support for digital image analysis, because the outcome of each stage for the thermal image is much better than that of the image taken from the ordinary camera.

Figure 13 shows some sample outputs of corresponding inputs, illustrating images taken from both ordinary and thermal cameras. As seen in this figure, by assessing the perceptual quality of the sample input images, various features could be estimated to represent subjective qualities, whose characteristics could be considered desirable or not. Images taken from the thermal camera provided better edge visibility, contrast, and brightness.

4.1. Analysis of Five Tonal Zone of Histogram

Five tonal zones of the histogram are illustrated in Figure 14; these zones are (from 1 to 5) blacks, shadows, midtones, highlights, and whites, corresponding to the tonal range of 0–255.

1. Blacks—this segment is completely black, with no details captured. It also has a very narrow tonal range located on the far left side of the histogram. When a histogram reaches the far left of the chart (tonal range of 0), it means that shadows have been clipped on the image.

2. Shadows—shadows are frequently mistaken for blacks, especially when a darker shadow appears; however, this segment has a slightly wider tonal range than blacks. Shadows have some details and can be lightened to some extent. Usually, image noise appears in this area.

3. Midtones—this segment has the most tonal range and contains the most pixels. Even if there is stretch or shift in any direction, the tones will most likely remain intact.

4. Highlights—this segment has the same property as shadows, but it is in the brighter part of the image and contains some visible details. It can be cautiously adjusted to the far right toward clipping.

5. Whites—whites have a similar characteristic to blacks, but this segment is pure white with no details. If a histogram reaches the far right, it means that more information in the brightest portion has been clipped.

Figure 15 shows the tonal zone representation of the image taken from the ordinary camera; a small-scale shadows region appears in the histogram, which indicates that the input image is close to a black segment. In addition, more information is provided in the midtone section, resulting in an asymmetrical histogram. Finally, a small portion of highlights exists in this image, which is likely the same as the shadow contents.

Figure 16 depicts a symmetrical histogram if shifted further to the far left portion of the image. If shifted to the far left portion, the sample image might have a better contrast quality. In addition, it shows that most pixels fall between the midtone and highlight portions of the image.

4.2. Image Quality Metrics

As shown in

Table 2. Image quality metrics.

	Ordinary Camera	Thermal Camera
images
MSE	23,432.0312	240.7319
PSNR	4.4327	24.3155
SNR	4.4053	24.2881
MS-SSIM	0.0321	0.7885
BRISQUE	43.4582	45.0100
NIQE	15.1913	13.2069
PIQE	63.0411	88.7166

, image quality assessment is a difficult task, yet the choice of technique is fundamental for evaluating image quality. Techniques belonging to objective fidelity, such as mean square error (MSE) and peak SNR (PSNR) assessment, and subjective fidelity, which corresponds to the human visual system, such as the multiscale structural similarity (MS-SSIM) index, are widely used. In addition, no-reference algorithms use statistical features of the input image to evaluate image quality, such as blind referenceless image spatial quality evaluators (BRISQUE), natural image quality evaluators (NIQE), and perception-based image quality evaluators (PIQE). The reference image was set using the masking technique for MSE, PSNR, SNR, and MS-SSIM only and was compared with the final output of the proposed image-processing method using the thermal and ordinary cameras.

The experiment results of the CNN and proposed method, which is the CNN–SVM, were compared.

Table 3. Validation results of the training set based on sensitivity, specificity, and accuracy.

Classifier Method	Percent Sensitivity	Percent Specificity	Percent Accuracy
CNN	98%	94.96	93.96
CNN-SVM	99%	95.3%	98%

displays the results of the proposed fault identification scheme for the training dataset. Further, the CNN–SVM method alone outperformed the CNN method for both the training and testing datasets (Table 3 and

Table 4. Validation results of the testing set based on sensitivity, specificity, and accuracy.

Classifier Method	Percent Sensitivity	Percent Specificity	Percent Accuracy
CNN	94.25	90%	90%
CNN-SVM	97.55%	93.65%	93.96%

).

The correlation value indicates whether the provided dataset is on the best-fit line. In most cases, a positive correlation should be close to +1, and a negative correlation should be close to −1. Moreover, the regression value should be between 0 and 1, with values closer to 1 indicating a model that better fits the dataset.

Table 5. Correlation and Regression Values of CNN and CNN–SVM.

Coefficient	CNN	CNN–SVM
Correlation	0.9321	0.9981
Regression	0.9629	0.9995

shows that the correlation and regression values provided the best model.

5. Conclusions

Images of structural sites are frequently used to document construction scenes. The ability to automatically detect material regions in these images can be utilized to automate construction applications, such as monitoring and surface quality assessment. Existing studies necessitated the use of acceptable material classification thresholds. However, they did not elucidate how to determine these levels. This paper presents an algorithm for using thermal imaging technology for concrete image analysis that utilizes different image-processing techniques, with the aim of representing the sample images in an easier-to-interpret domain.

Furthermore, noise reduction techniques were implemented with an appropriate filter iteration to enhance the sample images being tested, preventing any unnecessary data from being included in the noise reduction process. Concrete regions in a sample image can be recognized without the need for manually specifying thresholds. Additionally, in this study, we propose an automated model for detecting concrete regions in images of structural sites. As such, we trained a classifier on 2700 samples.

Moreover, the proposed approach not only uses ordinary images but also uses thermal images. The thermal imaging technology algorithm extracts characteristics from thermal images to simplify their representation into a more manageable area for analysis. In addition to an image histogram, a thermal image provides considerable information during image processing. Experimental results demonstrated that the improved thermal images provided better histogram and CDF features—further, the proposed method employed CNN to improve image classification, with a 98% accuracy. Lastly, the correlation and regression values provided the best model of the proposed concept in the dataset used. The proposed method may encourage the development of various thermal image applications, such as nonphysical visual recognition and fault detection analyses. In the future, many properties of these thermal images could help neural networks in categorization applications.

In the future, we plan to deal with different external factors such as various lightning conditions, high surface roughness, and differences in the concrete surface, and to provide different comparative analyses on how these factors affect the equipment technical specifications and setup. We will also consider sensitivity testing in different models.