Pavement Crack Detection from Hyperspectral Images Using a Novel Asphalt Crack Index †

: Detection of road pavement cracks is important and needed at an early stage to repair the road and extend its lifetime for maintaining city roads. Cracks are hard to detect from images taken with visible spectrum cameras due to noise and ambiguity with background textures besides the lack of distinct features in cracks. Hyperspectral images are sensitive to surface material changes and their potential for road crack detection is explored here. The key observation is that road cracks reveal the interior material that is different from the worn surface material. A novel asphalt crack index is introduced here as an additional clue that is sensitive to the spectra in the range 450–550 nm. The crack index is computed and found to be strongly correlated with the appearance of fresh asphalt cracks. The new index is then used to differentiate cracks from road surfaces. Several experiments have been made, which conﬁrmed that the proposed index is effective for crack detection. The recall-precision analysis showed an increase in the associated F1-score by an average of 21.37% compared to the VIS2 metric in the literature (a metric used to classify pavement condition from hyperspectral data).


Introduction
The conservation of city infrastructure requires the maintenance of road pavement as an important asset. This requires continuous monitoring of road pavement cracks for making later decisions such as sealing them. This task is ideal for automation to save money and human efforts.
Detection of road cracks from images is difficult since cracks are dark, have only primitive features and are hard to distinguish from road texture [1]. As a result, state of the art road crack detection systems from visual camera information often miss real cracks and deliver high false positive rates as reported in [2,3].
Indeed, the high false positive rate can be attributed to the lack of sufficient clues for road cracks. Hyperspectral imaging (HSI), though relatively expensive and developed mainly for satellite and scientific imaging, is now becoming affordable and can be exploited for city road monitoring. Hyper Spectral Cameras (HSC) have been used to identify changes of surface materials having a unique spectral signature [4][5][6]. The potential of HSC for crack detection will be explored in this paper, with the aim to add physical clues to improve crack detection.
HSI has been used previously to classify road conditions from satellite images [6][7][8][9][10][11][12][13]. The research was intended to classify road conditions in general and the spatial resolution can not detect road cracks or defects. Only a few papers have considered the detection of pavement cracks based on hyperspectral data [1,[14][15][16]. In these cases, HSC were fitted on drones of low altitude flights to have higher spatial resolutions to enable observing cracks.
The previous studies considered using descriptors of the spectrum such as the VIS2 (intensity difference between 830 and 490 nm measurements) and Short Wave Infra Red (SWIR) (Intensity difference between 2120 and 2340 nm). Spectral reflectance is a measurement of how materials reflect and absorb radiation according to the wavelength in the visible and near infrared regions.
The metrics measure the rise and decay of the spectral reflectance at specific wavelength regions. These metrics have also been linked [17] to the Pavement Condition Index (PCI, a standard metric by American Society for Testing Materials (ASTM) D6433 and D5340, used to indicate the condition of road pavement) and is usually computed using visual surveys [15].
In this paper, a new Asphalt Crack Index (ACI) is proposed to describe the spectra of road pavement and, in particular, assist the search for cracks. New roads are mainly composed of fresh asphalt (and newly formed cracks as well), while deteriorated roads show aged asphalt [11,18]. The difference in spectral response will be used to distinguish cracks from normal un-cracked road surface material.
A novel index is proposed to discriminate cracks. The use of fewer bands of the hyperspectral data may suffice for road condition monitoring as reported in [12]. They showed that the discriminating ability is concentrated within a few bands. However, the design of a spectral index or descriptor for classification can be viewed as a compromise between simplicity and exploiting the rich spectral information of the data. This paper is structured as follows: The next section outlines the relevant work in road condition analysis using hyper-spectral imaging. Then, the new ACI and classification algorithm is described in detail in Section 3. The experiments and results on real road conditions are described in Section 4. The application details are discussed in Section 5, then conclusions are drawn in Section 6.

HSI for General Applications
Hyper-Spectral Imaging (HSI) is used to characterize surface materials and has been popular in remote sensing and analysis of natural resources and several other forestry and agricultural applications [19,20]. Some applications have used HSI as a tool for detecting forgery in artwork [21] and for crack detection in paintings [22].
Spectral Angle Mapper (SAM) [24] is a classification algorithm that determines the spectral similarity between two spectra by calculating the angle between the spectra and treating them as vectors in a space with dimensions similar to the number of bands. SAM compares spectra of any kind: spectra from imagery towards spectral reference libraries developed in laboratories, spectra from different hyperspectral sensors and spectra from field spectrometers towards spectra from imagery.
The Spectral Feature Fitting (SFF) algorithm compares the fit of image spectra to reference spectra using a least-squares technique. Some algorithms classify hyperspectral images using information measurement such as Spectral Divergence Information (SDI) [29]. Other algorithms use a combination of approaches such as an integration of SAM and SDI [30].
The frequency domain has been exploited to derive spectral analysis. A spectral similarity measure was suggested [31] using the magnitude values of the first few low-frequency components for spectral signature. Harmonic analysis was also used to describe the spectral reflectance and recognize objects [32].
Deep learning and convolutional neural networks have been used in a purely learning approach to identify features in both spectral and spatial domains. In some cases, classification exploits a training set, such as the work reported by [33][34][35][36][37][38][39][40]. The work in this category is interesting, especially for unsupervised classification, since there is a limited set of labelled data for training in general. It is also challenging due to the huge computational complexity of deep learning added to the complexity of the spectral cube typical for HSI [41]. Feature mining has also been reported to learn discriminative features from datasets through feature selection and reduction methods [42] and to improve the spectral and spatial data [43].

HSI for Road Monitoring Applications
There is currently an increasing interest in the application of HSI with UAV to monitor the conditions of city roads, see [19,44] for a review of UAV based sensors. However, the use of HSI for road condition monitoring had been reported in a few studies using several spectral descriptors. Approaches to associate the spectra of asphalt roads with road conditions and age in general (not for crack detection) have been reported in [6,10,12,15,45]. From their studies, we may conclude that road condition can be monitored through observing two spectral bands namely VIS2 (in the VNIR region) and SWIR for longer wavelengths. Since we are interested in the use of VNIR cameras, discussion will be limited to the detection using spectral bands below 950 nm. The reflectance in the VNIR region increases when the road ages [10].
Conventional spectral classification techniques such as SAM and SFF have not been used for pavement condition inspection to the best of our knowledge. It seems that the methods work when there is a significant spectral difference between materials. Other techniques have been used for pavement inspection using VIS2 and SWIR [6,15].
The VIS2 ratio is a simple spectral descriptor that was proposed to measure the rise of spectral response curves for road materials in the VNIR spectral range [15]. The ratio was used as the metric for pavement condition defined as follows [6]: where I is the intensity of light and λ is the wavelength in nanometres, nm. It should be noted that the VIS2 metric is termed as a ratio in the literature, while it is mathematically a difference. Therefore, we preferred to define it as two metrics, namely the difference and the ratio. A spectral index should be accurate, easy to compute and encapsulate all the precious information obtained from the camera spectra. This was highlighted in [12] where they discussed whether hyperspectral imaging with a huge number of spectral bands is really required rather than using a limited number of bands, in a multispectral imaging fashion and at a much reduced cost. However, the issue of compromise between the spatial and spectral scale in hyperspectral imaging is critical and has been the subject of extensive research in remote sensing [12,46].

The Asphalt Crack Index
Roads are constantly subject to wear by vehicles' tyres and therefore it has a brighter colour due to the partial appearance of aggregates used to build pavement layers and the wear of darker asphalt. Newly developed cracks will show up the road interior material, which is different from the pavement surface material. This difference in material can be easily detected by a hyper-spectral camera in a specific spectral band. The new crack shows darker material due to its asphalt content, which is rich in heavy hydrocarbons and oil. The darkening of cracks in images may also result due to the cavity shading as well and this will be illustrated in the experimental section. This condition acts as a strong physical clue to discriminate road cracks from other edges.
The regions used for pavement inspection are shown in Figure 2 showcasing the typical response curve from an HSC. It is known that the range of 450-900 nm is sensitive to the iron oxides and iron hydroxides [47]. The camera used is working in the Visible and Near Infra Red (VNIR) spectral range. That means the spectral range that can be of interest for road material change detection, and hence cracks, is the range similar to that used by VIS2 metric and showing metal and iron oxide in particular. The response in the VNIR range changes significantly for road based on age and wear as reported in several studies (e.g., [6,11,12]). The hypothesis of this paper is that cracks show different material, and therefore, present a different spectral response. The main idea is explained schematically in Figure 3. The crack shows the internal pavement material, which is different from the road surface due to surface wear. The surface usually reflects bright light because of partial appearance of gravel pigments and loss of the bonding asphalt rich in hydrocarbon and oil. In the previous work, only the intensity at two spectral bands was used to derive the VIS2 ratio. The new Asphalt Crack Index (ACI) relies on the observation from Figure 4, that the slope is different between cracks and normal surfaces in the range 450-550 nm. In this paper, the approximate line that represents the spectrum in the region between 450 and 550 nm is used. The slope angle, φ 1 is computed in the range 450-550 nm relative to the horizontal axis from the following equation: The reason for selecting the range (450-550 nm) is the observation of the slope in this region. The boundary of 450 nm is decided by the camera specification for the least spectral measurement and the upper boundary of 550 nm has been decided manually from observing the spectral response curves. The spectral range is a subset of the total iron oxide range 450-900 nm. There is an assumption that the surfaces are clean because the spectral response depends on the surface condition.
The rate of intensity darkening in this spectral zone is claimed to be an indication of material change rather than the shading effect and this will be proved experimentally. The ACI histogram is shown in Figure 5 for a sample crack image. The index can be represented in radians or degrees and its range is 0-1.571 rad or 0-90 degrees. The threshold value for the ACI is 1.53 rad and this has been identified manually and is confirmed to give best results.
The images are then classified according to the algorithm described in Algorithm 1. The process of computing is shown in Figure 6 indicating the input image, computed ACI and the thresholded image.

Name
Original ACI Overlayed image ACI classified Process Figure 6. The HSI processing.

Experiments
The objective of the experiments is to explore the ability of the ACI to detect cracks from HSI and quantify the performance compared with the VIS2 metric.

Conditions and Metrics
The full spectral range for hyperspectral imaging is 350-2500 nm, which extends beyond the human vision range (400-700 nm). The hyperspectral camera used in this study is a Cubert (model S185) measuring in the range 450-950 nm across 125 channels and fitted with a lens of 23 mm focal length. Table 1 shows a summary of hyperspectral camera specifications. The images were then processed and classified using the ENVI software package [49].
Experiments were made in the city of Leeds, northern areas in St. Helens lane and Adel wood grove, LS16 8JJ, UK, on the 20 April 2020 at 16:00, 25 June 2020 at 16:25 and 6 August at 16:00 in 2020, and the weather was mostly cloudy. In the optical system used, one pixel represents (0.5 × 0.5) mm area at a one-metre distance. The spectral range of the camera enables the measurement of the iron oxides sensitive spectra as shown in Figure 2. Ground truth crack images were obtained by hand annotating the test images. The test images and results are available in supplementary material with this paper. Recall-precision analysis is used with the following definitions: where TP is the True Positives, FN is the False Negatives and FP is the False Positives. The F1 score can be computed from both recall and precision using the following formula: The Pratt Figure of Merit, PFOM, [50] can quantitatively measure the edge detection performance by considering the differences between a reference and a test subject; the reference subject is the step function in edge detection. It is defined by: where i, I A , I I , d i , a are respectively the edge pixel index, detected edges, the ideal edges, the distance between the actual and the ideal edges, and a design constant used to penalize displaced edges.

Separability
Several experiments were conducted using the camera to capture real crack images in paved roads and three sample images are shown in Figure 7. The location of cracks are known in these experiments and both 'normal surface' and 'crack' pixels were randomly selected from identified areas.
It is observed in Figure 7 of the spectral response that this particular region can be approximated by a straight line. This justification is proved empirically through the experiments.
The images are shown on the left-hand side of Figure 7 and two square windows marked in the image, one for normal road surface and another for a crack region. The right-hand side curve shows the spectral response of both normal surface and crack reflection. The area used to capture the spectrum is fixed during this experiment to one pixel size.
It can be observed in Figure 7, that the overall reflection on spectra from cracks is lower than in flat surfaces. It can also be observed that the slope of the spectrum in the 450-550 nm range is radically different from normal to crack surfaces. It can also be observed that the slope angle is lower for the crack regions than the surface regions of the pavements in most of the cases. The VIS2 ratio was computed for the same sample set and is shown in Figure 8 as a ratio Equation (2). The VIS2 difference Equation (1) is also computed and shown in Figure 8.
The histograms of this angle computed for cracks and normal surfaces have very little overlap as shown in the right-hand side of Figure 8. However, for any given sample, the angle of slope of a crack is always lower than that of the road surface. This implies that a single threshold can be used to discriminate cracks from normal surfaces. It is observed that the histograms are almost 80% overlapping in Figure 8 for the VIS2 ratio and difference compared to less than 20% overlap for the proposed slope angle. The experiment was made to check the ability of ACI to detect material changes not related to deep cracks and the results are summarised in Figure 9. The case A9 shows surface contours of fresh asphalt overlayed around a manhole, and the ACI can detect the material shown in the red colour area. The A15 case shows spots of recent asphalt on pavement, and it is easily detected by the ACI also shown in red colour. The ability to detect potholes of fresh asphalt content on the road is shown in case A17, where the pothole is detected in red. The case for A20 shows a pothole filled with fresh asphalt, and this is again detected and shown in red. The detection of fresh asphalt in these cases confirms that the ACI can detect material changes from fresh asphalt to slightly aged asphalt.
The ACI is measuring the rate at which asphalt is getting dark in a small spectral region. It is not measuring the asphalt darkness itself, which may be due to local shading across the crack depth. In some cases where the crack is old, the ACI did not detect any change of material since the exposed asphalt gets aged.

Name
Original ACI image ACI classified Overlayed A9 A15 A17 A20 Figure 9. Sample images showing the change of asphalt material A9 and A15, and example for pothole detection A17 and A20.

Comparisons
The crack images were classified using SAM and SFF methods, and the results are shown in Figure 10 and the reference sample was manually selected from the crack region. The output of the algorithms does not represent the crack shown on the right. The poor results because the spectral difference between cracks and road normal surface is not enough to trigger sufficient response. This is believed to be the reason behind the use of the VIS2 metric to classify road pavement conditions and grading it. Hyperspectral images were classified based on the ACI, and are displayed in Figure 11 showing the classified image in the left column and overlayed on the original image in the right column. The first row shows the orginal image and the image of ACI on the right. The second row shows the classified image using the ACI computed in the range 450-500 nm with 50 nm difference only. The third row shows the result using the ACI computed from 450-550 nm with a 100 nm difference. The fourth row shows the result for the VIS2 metric. It can be observed that the VIS2 classification shows more false positive pixels than the real cracks shown on the original image. The result in the third row for 100 nm shows that the result is better than the 50 nm case in the second row in terms of increased recall and better precision.
Sample classified images are given in Figure 12, showing the original, ground truth, VIS2 and ACI-classified images. It is clear that the VIS2 results show more false positive pixels and the ACI on the right column is closer in performance to the ground truth images. This observation is quantified in Figure 13 showing a comparison of the F1 score for both the ACI and VIS2 and it shows that the F1 score is higher for the ACI in all cases. Table 2 shows the recall, precision and F1 score for 50 test images. It is clear that the ACI shows much better performance.
The results of precision-recall analysis are summarised in Figure 14, where each point in the graph represents the results of a single test image. Both the results of the ACI and VIS2 are included for comparison. The ACI outperforms the VIS2 metric classification. The two sets are not disjoint in this graph, but the previous table can be consulted for a case by case comparison showing that only in five images VIS2 outperformed the ACI in terms of the F1 score.
The statistics of the tested 50 test images are summarised in Table 3. It can be seen that the F1 score increased by 21.37%, which is a significant improvement. The precision-recall sensitivity to changes in the threshold value is shown in Figures 15 and 16 for the ACI and VIS2, respectively, computed for a sample image and a similar trend was observed in other images. The F1 score has a peak at an optimum threshold, which is 1.53 rad (ACI) and 5000 for the VIS2 case. The effect of the threshold on the precision-recall analysis is also shown in Figure 17. It is clear that the ACI performs better than the VIS2.

Discussion
The idea of using spectral information as an added clue to detect cracks reliably proved to be effective compared to the existing method of VIS2 and the ACI helped in this detection process. The added clues helped to find road pavement cracks that are hard to find in visible spectrum imaging because of the ambiguity with background textures.
It is clear that the geometrical properties of cracks alone do not provide useful clues for crack detection, simply because cracks have no distinct shapes. The spectral domain here was exploited with the assumption that cracks are newly developed to show different distinguishable surfaces in the hyperspectral image. The main assumption is that crack surfaces are free from debris and dust and are at least slightly worn roads or older roads. This means the method will work better in older roads where new cracks form. This is, however, precisely the kind of asset where finding these cracks is essential as early detection allows for a prompt fix that stops further deterioration that may result in costly reparation.
The results showed significant improvement in the precision of crack detection, which confirmed the concept of crack verification from the ACI. Spatial resolution affects the ability to detect thin cracks from a distance and in our experiments the camera was used at a close distance to the road (one metre). Thin cracks in our experiments refer to 5 pixels width or length (2.5 mm in our system) and it gets harder to detect any thinner/shorter cracks.
Camera optics can be modified to help detect cracks from aerial images with sufficient spatial resolution. Conventional spectral classification algorithms are not suited to detect the different grades of asphalt due to aging, since the differences are not spectrally significant. The ACI index improves on the concept of VIS2 through measuring the gradient of change of intensity with respect to wavelength.
The results of ACI showed an improvement over the VIS2 metric but the overall F1 score may still be not high enough for commercial application. The ACI may suffer false positives for surface stains of fresh asphalt. In the future we will aim to combine this method with other crack detection methods to serve as added clues for detection and for discriminating false positives.
VIS2 had been used previously to classify road pavement condition and reported to give good results, but it seems this metric was more suited for long-range imaging. The ACI here outperformed VIS2 for close range imaging and this should be of interest for inspection in low altitude drone scenarios. HSC are expensive, and it would be useful to make the utmost utilization of their data in the simplest way without sacrificing useful information. There should be a compromise between the exploitability of spectral data and simplicity, which may dictate using only a few spectral data or just a multispectral camera working in the spectral region of interest.

Conclusions
The potential of hyper-spectral imaging is exploited in this paper to discover cracks in road pavement. The basic measurement concept is that a newly developed crack shows the road interior, which has fresh asphalt content relative to the worn asphalt road surface. When cracks are observed with an HSC, the spectral reflection caused by different materials will be a good clue to detect cracks. A novel ACI index is proposed to indicate the change of material through a formula measuring the intensity change in the spectral region of 450-550 nm. The ACI is the slope angle of the spectral line in this region that utilizes more information from the spectral response than the previous VIS2 indicator.
The new technique had been tested with a hyperspectral camera in the field and was found effective, and the ACI is confirmed by experiment to outperform the VIS2 metric. The F1 score was increased using the ACI by 21.37% relative to the case of the VIS2 metric. Exploiting more spectral information is useful to improve the clues used to find material changes and hence associate this with cracks.