Image-Based Corrosion Detection in Ancillary Structures

Abstract

Ancillary structures are essential for highways’ safe operationality but are mainly prone to environmental corrosion. The traditional way of inspecting ancillary structures is manned inspection, which is laborious, time-consuming, and unsafe for inspectors. In this paper, a novel image processing technique was developed for autonomous corrosion detection of in-service ancillary structures. The authors successfully leveraged corrosion features in the YCbCr color space as an alternative to the conventional red–green–blue (RGB) color space. The proposed method included a preprocessing operation including contrast adjustment, histogram equalization, adaptive histogram equalization, and optimum value determination of brightness. The effect of preprocessing was evaluated against a semantically segmented ground truth as a set of pixel-level annotated images. The false detection rate was higher in Otsu than in the global threshold method; therefore, the preprocessed images were converted to binary using the global threshold value. Finally, an average accuracy and true positive rate of 90% and 70%, respectively, were achieved for corrosion prediction in the YCbCr color space.

1. Introduction

Most ancillary structures, such as high-mast light towers, cantilevered sign structures, overhead traffic signals, and luminaires, are continuously exposed to wind-related fatigue cracks, vibration issues, missing bolts, and loosened nuts [1]. However, corrosion can be considered as the most common defect in them. In ancillary structures, corrosion is caused by factors such as exposure to weather elements, chemicals, and salt compounds. Traditional inspection methods, such as visual and physical nondestructive evaluations (NDEs), can be time-consuming, expensive, complex, dangerous, and even impossible for inaccessible areas. The outcome of the manned evaluations is typically subjective and inconsistent. It is possible that accuracy depends on the inspector’s experience and location’s accessibility for inspection [2]; however, visual inspections remain the most common method for corrosion damage assessment. Maintenance and protection of traffic (MPT) safety legislation demands make it further challenging for stakeholders to perform in-service ancillary structure physical and visual inspections [3]. Therefore, developing noncontact methods augmented with artificial intelligence (AI) as an alternative to conventional defect detection is a necessity.

Corrosion can be defined as the reaction of a metal to the surrounding corrosive environment due to the change in its properties, consequently resulting in functional deterioration of the metallic object [4]. The service life of a steel structure can be reduced by external or internal surface corrosion [5]. The United States records an estimated annual corrosion damage cost of approximately USD 10.15 billion for steel bridges [6]. Figure 1a,b illustrates images of an existing ancillary structure in Grand Forks, ND, exhibiting significant corrosion and without visible corrosion, respectively.

Corrosion significantly affects the overall cost of steel structure maintenance [7], and if neglected, can lead to section loss and eventual failure. For instance, the corroded ancillary structure shown in Figure 1a was replaced due to severe corrosion to avoid continuous deterioration and subsequent failure. A total of 42% failures of structural elements occur due to corrosion of steel structures [8]. Moreover, continuous contact with water or electrolytes could cause internal corrosion in the case of tubular members [3].

The hands-on inspection technique has certain drawbacks [9] that create the need for computerized digital image recognition, a feasible alternative regarding safety, efficacy, consistency, and accuracy. Despite the advantages of the currently practiced NDE technique in the inspection of steel structures, the automated optical technique is the most preferred due to its simplicity in use and interpretation [6,10,11]. This usually involves the onsite acquisition of digital images of the structure, followed by an offsite analysis using image processing techniques for corrosion detection [6,12]; however, there have been some attempts for real-time processing [13,14].

Researchers have proposed various image processing techniques to detect steel structure corrosion [7,8,15]. The texture analysis technique characterizes the pixels of images for classification problems. Pixels associated with corrosion in visual images of ancillary structures have a rougher texture and a distinct color compared to noncorroded or sound pixels. Here, color and rough texture are considered features for pixel classification. Past research studies have investigated corrosion using RGB (red, green, blue) color space without considering the presence of undesired objects in the image’s background [12,16]. Most studies have either evaluated corrosion without corrosion quantification from ground truth or have used images collected under controlled environments [6,7,8], as these image-based algorithms are usually affected by undesirable illumination presence of the background.

The state’s Department of Transportation (DOT) commonly performs corrosion detection inspections using visual and physical methods. The primary goal of this research is to develop an automated adaptive image processing-based algorithm to detect the corrosion of in-service ancillary structures under ambient environmental conditions. In addition, actual onsite conditions of ancillary structures, such as background and natural lighting, were considered while processing the data.

2. Overview of Corrosion Detecting Sensors

There are numerous nondestructive techniques for corrosion detection. Fiber Bragg grating (FBG) sensors are used to evaluate coated steel’s corrosion behavior [17]. The performance of electrochemical tests and FBG sensors was compared. Two types of coating—polymetric and wire arc-sprayed Al-Zn (aluminum-zinc) coatings—were used to verify the FBG sensor’s performance. Finally, it was shown that FBG sensors perform well for both detecting corrosion and crack initiation. In past studies, researchers found ultrasonic sensors to be one of the most effective devices for corrosion detection [18,19], but they required expertise to identify the critical location of corrosion. Again, an optical microscope was used to detect the hidden corrosion of depths 0.02 to 0.40 mm in steel plates [18].

Moreover, the traffic lane may need to be closed as these methods need contact with the investigating structural element. Corrosion identification using image processing methods can address both limitations. Digital cameras collect the digital data necessary for image processing methods. However, the sensor type and number specified for data collection are not constant in all inspections. Visual and thermal cameras are the most widely used sensors for corrosion detection and structure evaluation due to their availability, even though other sensor types can perform corrosion assessment and evaluation [12]. In addition, these sensors can be mounted on an unmanned aerial vehicle/system (UAV/UAS) [9], which does not need the closure of a traffic lane, as well as auxiliary arrangements such as a ladder and other detection instruments.

3. Color Spaces and Image Processing Techniques

3.1. Color Spaces

The human eye or camera sensor can detect light incidents when reflected off a material. The beam of light that reaches the eye or camera is the interactive result between the distribution of the light source’s spectral power and the target’s spectral reflectance. The formation of color into a signal can be described by the following Equation (1) [20].

$$f^c\left(x\right)={{\displaystyle \int }}_{\omega }^{ }E\left(\gamma ,x\right){\rho }^c\left(\gamma \right)d\gamma$$

(1)

Here, f^c (x) is the measured observation value; E(γ,x) is reflected energy from the surface, where γ is the wavelength and x is the spatial coordinate of the image; and ρ^c is the camera sensitivity, where c∈ {R, G, B} and ω is the visible spectrum. R, G, and B are red, green, and blue, respectively. Understanding color space is necessary to develop digital image processing models and properly develop color image processing methods. According to colorimetry, each color is a combination of three color coordinates. Commission Internationale de l’éclairage (CIE) introduced color space by mathematical formulas [21]. Again, color is defined as the portion of EM (electromagnetic) radiation seen by humans [20]. The wavelength band ranges from approximately 380 nm to 740 nm. The color images contain more detail and information than grayscale images [22]. Some of these details are chromaticity and luminosity. Chromaticity refers to color classification based on its deviation from white light, also known as hue and purity. The luminosity parameter measures the hue’s degree of brightness or darkness. Color spaces can be of different types: RGB (red, green, blue), HSV (hue, saturation, value), HSI (hue, saturation, intensity), L*a*b (luminosity, red or green component, yellow or blue component), YCbCr (luminosity and combination of chrominance component). In this study, two color spaces, RGB and YCbCr, were considered. RGB is the conventional color space for visual imaging. On the other hand, it is revealed that human vision is more sensitive to the luminance component; therefore, it becomes more efficient to represent images in YCbCr form. A concise explanation of these two color spaces is mentioned in the following sections.

3.1.1. RGB Color Space

RGB color spaces are primarily composed of three color bands. The intensity value ranges from 0–255 for each band. A combination of all color bands at the same intensity results in white color (255,255,255); however, the absence of all color components results in black color (0,0,0). The brightness of the same color can be modified by manipulating the color components using a constant multiplier or quotient [23]. The origin of the RGB color space is represented in a 3D cube in Figure 2, with values ranging from 0–255.

3.1.2. YCbCr Color Space

The Y component of the YCbCr color transformation space depicts the luminance component, a measure of light intensity. The Cb and Cr components represent the chrominance component, which measures the green component with respect to the blue and red component intensity differences [24]. The most important feature of the YCbCr model is that it imitates human vision, which responds to light intensity changes more than hue changes [25]. Therefore, the Y component has a range of [16–235], while the Cb and Cr components have a range of [16–240] [25]. Figure 3 illustrates the transformation of captured images from RGB to YCbCr color space. As can be seen in Figure 3, pixels associated with corrosion are more pronounced in the YCbCr color space. The mathematical equation for this transformation will be introduced later in this paper.

3.2. Image Processing Techniques

Routine corrosion detection is vital to ensure ancillary structure integrity. However, conventional physical and visual methods are used with inspection aids such as ladders or scaffolders to gain access to difficult parts of the structure. Image acquisition using UAS combined with image-based corrosion detection algorithms is an alternative solution that can increase accuracy and robustness without lane closures [9]. Moreover, image processing techniques using an automated inspection system are timesaving and less tedious than visual inspections [26,27].

Collecting quality image data is a crucial step for obtaining reliable evaluation results. Image data using UAS can be collected using hand-held or mounted sensors or other data collection platforms. In texture analysis, a rough area was categorized as a corroded region performed on the images captured by a camera with a resolution between 12 MP–18 MP [7,28]. Again, UAS can be integrated with image processing techniques for large steel structure corrosion detection with intelligent obstacle avoidance, positioning, stable hovering, and other flight characteristics [29]. Non-contact-based thin steel film electrical resistance (TFER) is another technique developed to detect corrosion by measuring the change in the sensing element’s electrical resistance [30]. The k-means clustering algorithm was used to extract the region of interest (ROI) from the images of pipelines in a subsea environment by grouping the corroded pixels of different intensities [31]; however, the mean reported accuracy was 90%. To improve the result of conventional image processing, different preprocessing steps such as the change of color space, applying filters, and morphological operators play significant roles [6,32,33]. The transformation of color space might help to segregate the background from the foreground with the same color [32]; however, the L*a*b color space proved better among the 14 color spaces used for conversion. Again, the use of Fourier transforms together with image processing steps was found to be most effective in cyan (C), magenta (M), yellow (Y), or CMY color spaces, with about 99% accuracy for corrosion detection [33]. Some researchers have focused on developing a relationship between the statistical features of sound and defective images [26,34]. A rust defect detection method (RUDR) was designed to analyze corrosion detection images comprising data preparation, analysis, statistical modeling, testing, and validation. The images were preprocessed prior to the main processing. The preprocessing step evaluated the presence or absence of defects in each digital image. The image was further processed and evaluated if found to be defective [26].

On the other hand, the combination of k-means clustering and statistical parameters was adopted for corrosion detection. Mean, median, skewness, and textural parameters in the HSI color space were used to segment microscopic corrosion images with 10 pixels by 10 pixels, yielding 85% accuracy [34]. The similarity index is also used for corrosion recognition using color and texture features extracted from the images [35]; the images with a similarity index less than one are defined as defective. Finally, pixels are compared with the corrosion color spectrum. Pitting corrosion had also been detected in pipelines using a support vector machine (SVM)-based machine learning model, yielding an accuracy rate of 92% [8]. On the other hand, 67–90% corrosion detection accuracy was obtained from the fully trained artificial neural network from the images transformed into four different color spaces [6].

Past research shows that images used for corrosion detection were free of artifacts and collected under controlled lighting conditions. Therefore, artifacts such as background were not excluded from the dataset, unlike other studies where the datasets are clean and do not depict the actual onsite condition of structures. At the time of preprocessing, the brightness of the images was modified accordingly. The model’s performance was validated using images captured by a camera mounted on the UAV. Removal of the background is an essential step in preprocessing. In most studies [6,7,8], model performance was evaluated by determining only the conventional performance metrics. In this study, image quality metrics were also considered as model performance evaluators. Furthermore, the proposed method will enable the inspector to exclude the image’s background, making it a possible and effective method for real-time inspection.

4. Dataset Generation and Processing

4.1. Image Data Acquisition

A cellphone camera was used to capture 300 images from four in-service traffic poles located in Grand Forks, North Dakota, on 17 May 2021, between the hours of 10 a.m. to noon. The poles had an average height of 7 m and a cantilever arm length of 6–7 m. This mobile phone had phase detection autofocus (PDAF) technology designed to mimic human eyes by pairing masked pixels on the image sensor. Examples of images collected are shown in Figure 4. The specifications of the camera used are summarized in

Table 1. Camera specifications.

Type of the Device	Samsung Galaxy M30	UAS Camera
Resolution	13 Megapixels	7860 × 4320 Megapixels
Aperture size	f-stop; f/1.9.	f-stop; f/2.8–f/11

. The sizes of the images are 2322 pixels by 4128 pixels. The images were collected for differing degrees of corroded and sound regions showing different background scenes in the image. In addition, the data were collected considering the traffic to ensure the safety of inspectors.

Another set of image datasets was collected in Fargo using a UAS equipped with a visual camera to verify the model’s performance with respect to image quality. The ambient weather condition for temperature, humidity, and wind speed during data collection is shown in

Table 2. Ambient weather conditions during data collection.

Date	Time	Temperature	Humidity	Wind Speed
17 May 2021	9:53 a.m.	74 °F	38%	6 mph
	10:53 a.m.	77 °F	34%	6 mph
	11:53 a.m.	81 °F	29%	13 mph
20 July 2022	10.53 a.m.	82 °F	60%	14 mph
	11.53 a.m.	83 °F	54%	18 mph
	1.53 p.m.	86 °F	41%	16 mph

[36].

4.2. Data Annotation

Images were annotated using an image labeler app in MATLAB R2020a image processing toolbox. All computations in this study were performed on a desktop computer with a 64-bit operating system, 16 GB memory, and a 2.9 GHz processor running an Intel ^® Core™ i7-10700 CPU. The annotated images were used as ground truth for benchmarking the input images. As shown in Figure 5, all images were labeled into two classes: corroded and non-corroded. A total of 95,539,808 pixels of these categories were annotated for each image. Depending upon the severity of corrosion, a range of 2.5 to 64.5% of the pixels of the images were annotated as corrosion with red, blue, and pink colors. The different colors represent the different severity of corrosion. The remaining portion was considered as background pixels. Since the determination of the corrosion scale was way beyond this paper’s scope, all images were converted to binary images (pixels with and without corrosion) and used as ground truth for further analysis. This semantic segmentation method is known as a dense prediction, as it predicts the class of each input pixel with respect to the assigned class.

4.3. Scene Constraints and Preprocessing

In developing a robust and viable algorithm for corrosion detection, existing environmental constraints play a crucial role. For instance, sunlight reflection could cause some portions of the region of interest to have high brightness, while other parts could be under the shadow; illumination plays a vital role in image quality. The illumination of all the images was modified by developing an algorithm to increase the proposed algorithm’s success. All the original images were preprocessed by converting them to the YCbCr color space to control the luminance. In addition, varying objects on the image background could challenge the ease of corrosion detection. Though the corroded part had prominent color, sometimes the background color created complexity. The existing site constraints and surrounding objects were captured in the background during data collection. Since these were field data, the researchers had no control over these constraints appearing on the image’s background. Dey [37] discussed the effect of uneven illumination and concluded that noise due to external interference and imbalanced illumination is a typical image-related artifact during image acquisition. In addition, the uneven distribution of light could have been caused by the presence of large objects in the image’s background [38]. They also mentioned that the presence of other objects’ shadows makes it challenging to optically segment and isolate the object of interest. The constraints resulting from illumination variations and the presence of undesired objects in the background make conventional segmentation techniques ineffective in corrosion detection for small objects. Moreover, an unsupervised segmentation technique can produce unreliable results in the presence of noise components, shadows, and reflections from light sources.

A preprocessing algorithm was developed in this study to increase the quality of the collected images for corrosion detection. Some image enhancement steps followed the preprocessing operation. In this step, contrast adjustment and histogram equalizations were used to distribute the pixels uniformly or eliminate noise from the image. First, the contrast was adjusted to improve the output image quality, resulting in 1% of the pixel values being saturated at high and low intensities. Here, saturation implies remapping pixel values in a low-contrast grayscale image to fill the entire intensity range [0, 255]. After the contrast adjustment, the input image’s histogram was equalized using Equation (2) to ensure uniform distribution of pixels’ intensities [16].

$$c\left(I\right)=1/N{{\displaystyle \sum }}_{i=0}^I\mathrm{h}\left(\mathrm{i}\right)$$

(2)

where N is the number of pixels in the image. Here, c(I) is the cumulative distribution from the integration of the histogram of the original image, h(I).

Finally, all the pixel values were redistributed using adaptive histogram equalization, which differs from histogram equalization. In the adaptive method, several histograms were computed for the images. Finally, the output image shows improved contrast and a more visible edge than the input images. Figure 6 shows corroded images before and after preprocessing. Corrosion is seen to be more visible in the preprocessed images than the raw images, since the presence of shadow or uneven illumination have been minimized in the final image.

The change in brightness as a result of the preprocessing is computed from Equations (3)–(5) [39]. If R′, G′, B′ is the normalized value of the R, G, B image for the standard RGB color space (sRGB), the relative luminance(Y) can be defined as Equation (3).

$$Y=0.2126\times sR+0.7152\times sG+0.0722\times sB$$

(3)

Here, sR can be determined from Equation (4).

$$sR=\{\begin{array}{c}\frac{ {\mathrm{R}}^{\prime }}{12.92} if {\mathrm{R}}^{\prime }\le 0.03928\\ {\left(\frac{ {\mathrm{R}}^{\prime }+0.055}{1.055}\right)}^{2.4}Otherwise \end{array}$$

(4)

Similarly, sG and sB values can be determined from Equation (4) by replacing R′ with G′ and B, after which the brightness (L*) of the image can be calculated from the following Equation (5).

$${\mathrm{L}}^{*}=\{\begin{array}{c}Y\times 903.3 Y\le 0.008856\\ Y^{\raisebox{1ex}{$1$}\!\left/ \!\raisebox{-1ex}{$3$}\right.}\times 116-16 Y>0.008856\end{array}$$

(5)

Since the brightness of images plays a significant role in detection accuracy, the performance of the model was verified by testing on another dataset. The dataset consists of images without corrosion and UAV payload images. The images without defects will be mentioned as sound images in all the text later. Moreover, image quality has an enormous impact on the result of image processing techniques. Quality metrics such as the Blind/Referenceless Image Spatial Quality Evaluator (BRISQUE), Natural Image Quality Evaluator (NIQE), and Perception-based Image Quality Evaluator (PIQE) have been determined for the images [40,41,42]. In this case, sound images are used as reference images. The quality metrics were used to detect possible poor images after the preprocessing.

4.4. Artifact Reduction from Images

Except for the issues related to illumination, the presence of undesirable objects as background could affect the performance of the corrosion detection model. A computationally light framework for the user was developed to subtract the background from each image. At first, the boundary was drawn by a polygon around the background. Afterward, a binary mask was generated, where nonzero pixels were for the background and zero pixels were for the region of interest. In the next step, the background was subtracted from the input image. By completing this step, the region of interest was separated from the background. Finally, the input image was overlaid on the binary image (Figure 7). After overlaying, the background color turned yellow by default. The novelty of this algorithm is that in real-time, the inspector can subtract the background to explore their desired inspection location. Color space transformation is the next step implemented on these images.

4.5. Color Space Transformation

The proposed approach transformed the captured images in RGB color space into YCbCr color space. Afterward, the transformed images were split into three different components (channels) based on luminance and chrominance. Equation (6) shows the transformation matrix [25]. Corrosion was expected to be more pronounced in the Cb component, and therefore segmented more conveniently. Finally, the color threshold app of MATLAB image toolbox was used for the thresholding to extract the foreground (object) from the background. Figure 8 and Figure 9 show images in YCbCr color space and corrosion detection of images shown in Figure 4.

$$\left[\begin{array}{c}Y\\ Cb\\ Cr\end{array}\right]=\left[\begin{array}{c}16\\ 128\\ 128\end{array}\right]+\left[\begin{array}{c}0\cdot 279 0\cdot 5040\cdot 98\\ -0\cdot 148-0\cdot 2910\cdot 439\\ 0\cdot 439-0\cdot 368-0.071\end{array}\right]\left[\begin{array}{c}R\\ B\\ G\end{array}\right]$$

(6)

4.6. Binarization

4.6.1. Otsu Threshold Method

In this process, each pixel in the image is placed in one of two classes. One class contains pixels within the region of interest; the other contains pixels not in the region of interest. In this step, corroded pixels are labeled as one and the remainder as 0. Binarization is an essential step in image processing. The binary image is used afterwards to build a set of parameters for classification. Successful binarization depends on the selection of the threshold value. This method can determine an optimum threshold by minimizing the weighted sum of variance between two classes: background and foreground. The colors in grayscale vary from 0 to 255. The binarization process can be described by Equations (7)–(10) [43]. At any threshold t, the variance between two classes (σ²) is evaluated by Equation (7):

$${\mathsf{\sigma }}^2\left(\mathrm{t}\right)={ \mathsf{\omega }}_{\mathrm{bg}}\left(\mathrm{t}\right){\mathsf{\sigma }}^2{}_{\mathrm{bg}}\left(\mathrm{t}\right)+{ \mathsf{\omega }}_{\mathrm{fg}}\left(\mathrm{t}\right){\mathsf{\sigma }}^2{}_{\mathrm{fg}}\left(\mathrm{t}\right)$$

(7)

$${\mathsf{\omega }}_{\mathrm{bg}}\left(\mathrm{t}\right)={\mathrm{P}}_{\mathrm{bg}}\left(\mathrm{t}\right){\mathrm{P}}_{\mathrm{all}}$$

(8)

$${\mathsf{\omega }}_{\mathrm{fg}}\left(\mathrm{t}\right)={\mathrm{P}}_{\mathrm{fg}}\left(\mathrm{t}\right){\mathrm{P}}_{\mathrm{all}}$$

(9)

The variance can be calculated using the formula below:

$${\mathsf{\sigma }}^2\left(\mathrm{t}\right)={\displaystyle \sum }{\left({\mathrm{x}}_i-{\mathrm{x}}_{\mathrm{mean}}\right)}^2/\mathrm{N}-1$$

(10)

where x_{_i} is the value of the pixel at i in the group (b_g or f_g), x_{_mean} is the mean of pixel values in the group (b_g or f_g), and N is the number of pixels.

4.6.2. Global Threshold Method

A threshold value should be selected in a way that would suppress the background while leaving the corroded pixels visible. A low threshold value will create noisy pixels that can distort the features, introduce extra features, and increase the complexity of the analysis. The Otsu method selects the threshold value automatically, whereas in the global threshold method, one arbitrary number is used for the whole image. In this study, we developed an iterative algorithm by evaluating the relationship between the threshold value and the performance metrics rather than choosing an arbitrary threshold value. The model was evaluated with the performance evaluation metrics: (i) accuracy (ACC), (ii) positive predictive value/precision (PPV), (iii) F1-value, (iv) sensitivity/recall/true positive rate (TPR), (v) specificity/true negative rate (TNR), (vi) false-positive rate (FPR), (vii) false-negative rate (FNR), (vii) intersection over union (IOU).

The metrics are calculated by the following Equations (11)–(15):

$$ACC=\frac{\mathrm{TP}+\mathrm{TN}}{\mathrm{TP}+\mathrm{FP}+\mathrm{TN}+\mathrm{FN}}$$

(11)

$$F1-Score=\frac{2\mathrm{TP}}{2\mathrm{TP}+\mathrm{FP}+\mathrm{FN}}$$

(12)

$$TPR=\frac{\mathrm{TP}}{\mathrm{TP}+\mathrm{FN}}$$

(13)

$$TNR=\frac{\mathrm{TN}}{\mathrm{TN}+\mathrm{FP}}$$

(14)

$$IOU=\frac{\mathrm{TP}}{\mathrm{TP}+\mathrm{FP}+\mathrm{FN}}$$

(15)

Here, TP means correct detection of the corroded pixel and TN means correct detection of sound pixels. On the other hand, FN and FP are opposite to TP and TN, respectively. ACC is the rate of correct corrosion detection for the total detections. The PPV and TPR provide helpful information about the prediction outcome. TPR gives the fraction of actual corrosion that the model correctly detects. Lower TPR rates indicate a high number of false negatives. All the performance metric parameters are determined with respect to the ground truth. Figure 10 shows the representative relationship between the threshold value and performance metric elements. The point where all the curves for each performance metric element meet can be termed an intersection point. The intersection points for all other images were also determined and used as the base threshold value for binarization.

4.7. Morphological Operations

The output of the proposed image processing method is a binary image that includes one or more connected components segmented as corrosion (Figure 11). However, most semantic segmentation techniques suffer from residual misclassification, generally manifested as more minor discontinuities in the binary image. Morphological operations can manipulate the shape and size of connected components in binary images. They are commonly used in texture analysis, noise elimination, and boundary extraction [44]. Morphological operation is applied using a small template called structuring element (se). The structural element is applied to all possible locations of the input image and generates an output image with the same size as the input. The operation’s output produces a new binary image with a non-zero-pixel value at that location if the operation is successful. There are different structural elements, such as diamond shapes, lines, disk shapes, circles, and spheres. Conventionally, opening and closing operations are performed on binary images. This study performed opening and closing operations with disk-shape structural elements on the RGB images after the illumination modification. Since the dimensionality of the RGB image is greater than that of the structural element, this morphological operation worked along each channel. In this step, any artifacts from processed images were removed. After binarization, a line-shaped structural element with a length of 20 at an angle of 180 degrees denoised the binary image more than the disk shape. Therefore, it is better to apply the morphological command after defining the boundary of the region of interest. In this study, all the detected edges bridged other regions for extracting the target area before implementing the morphological operations (Figure 12).

If morphological opening is ${\gamma }_s\left(A\right)$ and morphological closing is ${\beta }_s\left(A\right)$, then from Equations (16) and (17) [45]:

$${\gamma }_s\left(A\right)=\left(A \ominus \$\right)\oplus {\$}^{\prime }$$

(16)

$${\beta }_s\left(A\right)=\left(A \oplus \$\right) \ominus {\$}^{\prime }$$

(17)

Here, A is the binary image, and $ is the structural element of the desired shape and size. In this work, $ is a line-shaped structural element with a length of 20. ⊖ and ⊕ operators are used for erosion and dilation, respectively. $′ is the transpose of $.

Dilation also transforms the images with a change in size. This operation stretches or thickens the region of interest by bridging the gaps between pixels [41]. Dilation can remove negative impulsive noises. The dilation of A by the structuring element $ is defined by Equation (18) [45]. Here, Ac denotes the complement of A.

$$A\oplus \$={\left(A^c\ominus \$\right)}^c$$

(18)

5. Results and Discussion

5.1. Brightness and Image Quality

The novelty of the proposed method is the brightness modification of the test images at the preprocessing stage. This step is followed by background removal, color space transformation, and corroded pixel detection rate determination. From the results presented in

Table 3. Comparison of brightness values of original and preprocessed sound and UAV images.

Image	Original Brightness (cd)	Preprocessed Brightness (cd)	Brightness Change
Sound_1	61.40	56.10	8.63
Sound_2	59.60	54.69	8.24
Sound_3	64.54	57.78	10.47
Sound_4	64.54	57.78	10.47
Sound_5	59.95	54.69	8.76
Sound_6	62.62	59.29	5.32
Sound_7	62.80	59.61	5.09
Sound_8	63.56	60.57	4.71
UAV_1	50.18	53.12	−5.85
UAV_2	56.23	55.34	1.57

, it is found that in comparing the processed and preprocessed images, the brightness change is not beyond 10%. This implies that the proposed model did not change the brightness of the images unnecessarily to change the image quality.

The same sound images were used for quality check. From the reported values of quality metrics in Figure 13, it was found that after preprocessing, all values decreased. The average reduction in BRISQUE is 21.5%, while for NIQE and PIQE quality metrics, the reduction rates are 10.53% and 37.53%, respectively. This implies that the quality of the images improved with the brightness modification.

A comparison of images in the original and preprocessed conditions is presented in

Table 4. Comparison of images in original and preprocessed condition.

Image	Original	Preprocessed	Image	Original	Preprocessed
1			5
2			6
3			7
4			8

. In Table 4 and

Table 5. Comparison of brightness values of original and preprocessed images.

Image	Brightness (cd)		Brightness Change (%)	Change in TPR (%)
	Original	Preprocessed
1	61.63	57.27	7.08	3
2	54.33	52.46	3.45	9
3	58.35	59.71	−2.32	9
4	57.33	55.34	3.48	20
5	50.64	58.15	−14.81	−6
6	53.61	54.17	−1.042	12
7	58.56	57.53	1.77	7
8	44.24	51.54	−16.49	−20

, all the brightness values in candela(cd) of different images are presented. Further investigation was performed on the influence of preprocessing on the TPR and TNR values. From the result, it is reported that brightness change did not significantly affect TNR values. On the other hand, a change in brightness above 10% reduced the correct detection rate up to 20%. The brightness of all the images except images 5 and 8 changed by less than 10%; the corroded zone segregated from the sound part more clearly. For example, in image 3, the right corner portion of the image was overexposed to the sun; therefore, the corroded part became blurry. After preprocessing with a 2.32% brightness increment, all the corroded parts became prominent, which finally led to a 9% increment in TPR value. The highest increment in TPR value happened for image 4, which is 20% with 3.5% brightness modification. The reason behind this significant increment can be justified by the binary image presented in

Table 6. Comparison of a binary image with and without preprocessing.

Ground Truth	Preprocessed	Without Preprocessing

.

The preprocessing steps included the conversion to the YCbCr color space, which was found to be better for the separation of a corroded part from the background. Moreover, proper extraction of the corroded region from the sound part helped in denoising. On the other hand, the TPR value decreased by 20%, with a 16.5% increment in brightness for image 8. From Table 4, it is clear that after the preprocessing steps, color modification of the sound portion took place, and thus due to the similarity in color, the model predicted these as false positives. The change in brightness after preprocessing is determined with respect to the original image; a negative sign implies an increment after preprocessing. On the other hand, a negative sign for TPR changes means a decrement in correct detection with respect to the original image.

5.2. Optimum Threshold and Morphological Operators

In this study, both the Otsu and global threshold methods were used for image binarization prior to preprocessing. For the global thresholding method, the threshold value was not selected randomly. The value of the performance metrics for 0 to 1 threshold value with increment 0.01 have been determined for each image (Figure 10). For all images, there was no common intersection point for the plotted curves. The summation of the true positive and true negative was determined for each threshold value (Figure 14). From the comparison of Figure 10 and Figure 14, it can be deduced that at the intersection of the curves, as the TPR value increased, the TNR decreased significantly. For example, most of the performance metrics curves reached it’s peak at threshold value 0.6. However, in Figure 14, it is depicted that the correct prediction rate of the sound pixels decreased by around 0.6. The selected threshold value ranged from 0.27–0.52 for the images. Beyond this range, the summation of TPR and TNR drastically dropped. In general, the Otsu and global threshold methods work better on the images with bimodal histograms. Since the tested images are with different scales of corrosion, the pixel intensity distribution would not be bimodal. From Figure 14, it is evident that TPR+TNR-values change even with a 0.01 increment in the threshold value. All the performance metrics related to each threshold method are reported in

Table 7. Comparison of Otsu and global threshold.

Image No.	Otsu/Global Values	TPR (%)	TNR (%)	Accuracy (%)	F1-SCORE
1	Otsu = 0.47	100	48	49	8
	Global = 0.27	71	98	96	58
2	Otsu = 0.6	41	11	30	83
	Global = 0.52	77	74	76	80
3	Otsu = 0.4	56	98	80	71
	Global = 0.41	41	99	74	57
4	Otsu = 0.53	78	57	62	48
	Global = 0.45	51	69	66	40
5	Otsu = 0.53	99	31	48	48
	Global = 0.45	93	53	89	81
6	Otsu = 0.33	60	68	44	37
	Global = 0.46	33	95	44	37
7	Otsu = 0.6	95	43	53	49
	Global = 0.27	40	95	85	49
8	Otsu = 0.33	77	92	85	83
	Global = 0.35	80	91	86	84

. From the result, it is found that Otsu reduced the overall accuracy of the model by 20%; therefore, global threshold values were adopted for the binarization of the preprocessed images. After extracting the connecting edges, a dilation operator was implemented with line-shaped structural element size 20- and 180-degree angles. This is the optimum value for both size and angle. False-positive detection was higher with the increasing size.

5.3. Performance Metrics

In Figure 15, all the relevant parameters of performance metrics are presented for both the original and preprocessed images. From the results presented in Figure 15, TPR and accuracy are seen to have increased for all represented images except for images 5 and 8. The rate of increment for each image varies from 10 to 20%. The average TPR value for all preprocessed images was about 66%, whereas this value was 56% for the images without preprocessing. Among the investigated images, the corroded areas with uniform illumination and without undesired objects were categorized with an average true positive rate of 70%, whereas for uneven images, it is around 60%. The reduction in corrosion detectability in images 5 and 8 could be due to the loss of some features in the image during illumination modification, such as the increment in brightness. The reported brightness increment is 14–16% for these two images, which is beyond the optimum value. On the contrary, the proposed algorithm increased the accuracy and true positive rates by 7–10% for the preprocessed images. The average TNR value for preprocessed images is 90%, whereas it is 84% for images without preprocessing. These values indicate that the model successfully detected the sound pixels. The overall rate of accuracy for the preprocessed images was 90%, whereas for the original images it was about 77%. Similarly, the F1 score considerably increased from 60% for the unprocessed images to 67% for the preprocessed images. The mean IOU increased from 46% to 57.3% after preprocessing of images.

5.4. Efficacy of Proposed Method

5.4.1. Comparision with the Previous Method

The authors adopted the methodology proposed in the previous study in [15]. The researchers in [15] acknowledged that they had intentionally ignored the background for the sake of easiness of model development. On the other hand, a background removal algorithm was incorporated by the authors in this study. After comparing two methodologies, it has been revealed that the proposed methodology detected 23% more corroded pixels than the method mentioned in [15] when the image had background. However, the proposed method showed its superiority, with 11% more correction detection, even with the images with very negligible background. The results are presented in

Table 8. Comparative results with previous method.

Previous Method	Proposed Method
TPR:70.49%	TPR:93%
TPR:69.49%	TPR:80%

.

5.4.2. Robustness of Model Irrespective to Dataset

The robustness of the proposed model was evaluated qualitatively on several images collected by UAS from in-service ancillary structures. From visual scrutiny, it was confirmed that the model was capable of segmenting the corroded pixels from the background at large. In addition to this, it is worth mentioning that these test images were not used during model development as they were collected after the model was developed. Results are presented in

Table 9. Robustness evaluation of proposed model on external images.

Original Image	Binary Image by Proposed Model

.

Though the proposed model proved its supremacy with respect to the previously used method and external datasets, it had some limitations. The effect of different lighting conditions was not considered for developing the brightness modification algorithm. As a result, the generic modification of brightness did not work as expected for all the images except for the overexposed images. Background removal was another cruical step in the proposed preprocessing steps. As a first step of background removal, a boundary was drawn by a polygon around the background. Since it is drawn manually, leaving out any background pixel can cause a false prediction. Like the other image processing method, the threshold value for binarization needs to be defined, which might hinder the autonomy of the model to some extent.

6. Conclusions

This study presents a novel image-based technique to detect corrosion in ancillary structures. The model was evaluated on sets of images with and without preprocessing. The maximum 20% TPR value improved after the preprocessing steps, followed by background removal and color space transformation. However, the brightness modification should be within the optimum range. Among the investigated images, the corroded areas with uniform illumination and without undesired objects were categorized with an average true positive rate of 70%, whereas for uneven images it was around 60%. Regarding the threshold method, optimization of the threshold value with respect to the performance metrics showed better results than Otsu.

The efficacy of the developed model was evaluated in two different ways: comparison with previous method and tested on external images. In comparison with the previous method, the proposed model detected 23% more corroded pixels correctly from the images with background. Qualitative results of the proposed model on the external images are also promising.

The study revealed that several factors, such as the presence of background objects, preprocessing, color space used for processing, image quality, and thresholding values, affect the performance of any conventional image processing model. Several of these shortcomings have been investigated and addressed through the proposed methodology. The results from this study show significant promise for the future adoption of autonomous unmanned aerial systems and artificial intelligence methods for image-based corrosion monitoring and detection in ancillary structures. Limitations of the proposed image-based algorithms include the selection of the optimum threshold value for binarization, generic brightness modification, and the need for background removal. Deep learning semantic segmentation models in combination with image processing methods could be considered for future work to develop a robust corrosion detection model to combat this limitation.