The Crack Diffusion Model: An Innovative Diffusion-Based Method for Pavement Crack Detection

Abstract

Pavement crack detection is of significant importance in ensuring road safety and smooth traffic flow. However, pavement cracks come in various shapes and forms which exhibit spatial continuity, and algorithms need to adapt to different types of cracks while preserving their continuity. To address these challenges, an innovative crack detection framework, CrackDiff, based on the generative diffusion model, is proposed. It leverages the learning capabilities of the generative diffusion model for the data distribution and latent spatial relationships of cracks across different sample timesteps and generates more accurate and continuous crack segmentation results. CrackDiff uses crack images as guidance for the diffusion model and employs a multi-task UNet architecture to predict mask and noise simultaneously at each sampling step, enhancing the robustness of generations. Compared to other models, CrackDiff generates more accurate and stable results. Through experiments on the Crack500 and DeepCrack pavement datasets, CrackDiff achieves the best performance (F1 = 0.818 and mIoU = 0.841 on Crack500, and F1 = 0.841 and mIoU = 0.862 on DeepCrack).

1. Introduction

Pavement crack detection plays a crucial role in the maintenance and management of road infrastructure worldwide. Cracks on road surfaces are common indicators of pavement distress, often resulting from various factors such as traffic load, environmental conditions, and material deterioration. These cracks, if left unattended, can lead to a multitude of issues, including compromised road safety, increased risk of accidents, and hindered traffic flow. Traditional methods of crack detection primarily rely on manual visual inspection, where engineers or inspectors visually examine the surfaces of structures or materials to identify cracks and defects. This approach relies on human expertise, demanding significant time and labor efforts.

In recent years, numerous research efforts have been dedicated to developing automated pavement crack detection algorithms and systems, utilizing various imaging modalities such as visible light photography [1], infrared thermography [2], and LiDAR scanning [3]. These technologies offer the potential for faster, more accurate, and cost-effective crack detection compared to manual inspection methods. Methods utilizing visible light images for pavement crack detection offer the advantages of cost-effectiveness, enhanced detection speed, and automation. Image-based crack detection methods mainly include threshold segmentation [4], edge detection [5,6,7], traditional machine learning [8,9,10], and deep learning techniques [11,12,13,14,15,16,17,18]. Deep learning models, particularly convolutional neural networks (CNNs) [19], have exhibited significant promise in the field of crack detection because they can automatically learn image features, leading to highly accurate crack detection. Influenced by materials and environment, pavement cracks exhibit various shapes and features, but they typically present spatial continuity. However, limited by the receptive field of the kernel of CNNs, deep learning methods based on CNNs often produce fragmented and discontinuous detection results. There are mainly two reasons for producing such results: 1. Cracks exhibit continuity in space, while CNN-based methods are unable to learn this underlying spatial relationship between pixels. 2. Cracks exhibit significant scale differences between length and width dimensions. Moreover, annotating ground truth for cracks can be challenging as it is difficult for humans to accurately delineate crack contours, which results in a decline in CNN performance.

In recent years, generative models have garnered substantial popularity in the domain of deep learning. Deep generative models are essentially designed to seek and express the probability distribution of (multivariate) data in some way. These models, by capturing the data generation process, possess a certain level of robustness and probabilistic inference capability. They can handle missing and unlabeled data, learn high-level feature representations, and, as a result, find wide application in tasks such as visual object recognition, information retrieval, classification, and regression. Particularly, diffusion models [20] have emerged as a cutting-edge generative model, surpassing traditional models like Generative Adversarial Networks (GANs) [21] and Variational Autoencoders (VAEs) [22] in image generation tasks. Diffusion models learn through the processes of adding noise and sampling, and they possess two major advantages in feature representation. Firstly, diffusion models can predict pixel-level noise at each diffusion step, thereby characterizing the joint probability distribution of all pixels. This implies that they can learn broader spatial relationships, capturing more details and structural information during the image generation process. Secondly, the sampling process of diffusion models progresses from noise to image and from coarse to fine, showcasing their strong learning ability for both shallow and deep image features. This sampling process helps the model gradually understand the structure and content of images, leading to the generation of higher-quality images. Therefore, this paper aims to leverage the capability of the diffusion model for data distribution to learn the latent spatial relationships of pavement cracks and address the issues present in the CNN models mentioned above and, finally, generate accurate and continuous crack segmentation results.

However, currently, diffusion models are primarily used in the field of image generation. Unlike discriminative models, which can easily compute the correlation between predicted results and ground truth, there are only a few generative tasks that have well-defined ground truth, such as image super-resolution [23]. Currently, there is no research on incorporating the learning capability of diffusion models into crack detection. Addressing this issue, this paper proposes the Crack Diffusion Model (CrackDiff), a pavement crack detection model based on the diffusion model. It is designed to learn the spatial distribution characteristics of pavement cracks, generating accurate and continuous crack segmentation results.

CrackDiff uses the framework of the diffusion denoising probability model (DDPM) [20], where the crack image is embedded and input into a multi-task UNet [24] network alongside the noise image. In this network, one branch is responsible for predicting the image’s segmentation result, while the other branch predicts the noise at the current step, as illustrated in Figure 1. With this design, CrackDiff is capable of accurately learning the distribution and spatial relationships of pavement cracks. During the initial phases of the sampling process, CrackDiff can learn the approximate outline of the cracks. In the later stages of sampling, CrackDiff is capable of generating detailed contours of the cracks. Finally, it can generate highly confident segmentation results.

The primary contributions of this paper include:

• This study introduces a novel framework for pavement crack detection based on the diffusion model, CrackDiff, which is capable of learning both surface and deep features related to the distribution and spatial relationships of cracks, leading to accurate and continuous crack segmentation results.
• This study proposes a diffusion model structure based on multi-task UNet, which enhances the guidance effect on the original image by predicting image segmentation results, resulting in robust crack segmentation.
• Through experiments conducted on the Crack500 and DeepCrack datasets, CrackDiff achieves state-of-the-art results in pavement crack detection.

The subsequent structure of this paper is outlined as follows: In Section 2, a concise overview of the existing research on crack detection and diffusion models is presented. Section 3 elaborates on the principles of the diffusion model, CrackDiff, and the specific architecture of the network. Section 4 presents the experimental settings and qualitative and quantitative analyses of the performance of CrackDiff and demonstrates its advantages through ablation experiments. Section 5 discusses the limitations of CrackDiff, emphasizes existing challenges, and proposes potential directions for future research. Finally, Section 6 provides a summary of the paper’s conclusions.

2. Related Works

2.1. Pavement Crack Detection

Image-based pavement crack detection methods mainly include threshold segmentation, edge detection, traditional machine learning, and deep learning techniques. Threshold segmentation methods classify pixels in a given image into object and background classes based on pixel thresholds [4]. These methods are computationally efficient and fast, but they are highly sensitive to noise. The edge is the main feature of the cracks in the image. Edge detection methods use edge detection operators to identify these edges, such as Sobel, Canny, Prewitt, etc. [5,6,7]. However, the choice of parameters in these algorithms can significantly impact the detection of cracks, and finding the optimal parameters for different images, especially those with complex backgrounds, can be challenging. Traditional machine learning methods involve manually extracting crack features from images and then using techniques like support vector machines [8,9] or random forests [10] for feature classification. These methods offer better accuracy compared to traditional image processing techniques but rely on manually extracted crack features.

Deep learning is currently the most popular method. Semantic segmentation models are widely used in pavement crack detection because they can classify images pixel by pixel, allowing for quantitative measurement of road crack severity and density while simultaneously expressing the size and location of the targets. Various image segmentation models, including FCN [25], UNet [24], DeepLabv3+ [26], and SegNet [27], have been applied extensively in pavement crack detection research [11,12,13,14,15]. To capture the multi-scale spatial features of cracks, techniques such as spatial pyramids [16,28,29], dilated convolutions [30], and residual connections [31] have been incorporated into network designs. Additionally, attention mechanisms have been widely applied due to their advantages in representing long-range spatial dependencies and inter-channel correlations [17,32,33,34], especially transformer-based approaches [18,35,36,37,38].

Despite the integration of dilated convolutions, attention mechanisms, and various other techniques into CNN-based models, the performance of crack detection based on CNNs is still limited by the local receptive field. This limitation hinders the ability to identify long cracks and capture the entire image background, leading to discontinuous detection results and susceptibility to noise interference. Additionally, due to the difficulty in accurately labeling ground truth and data imbalance, the network may easily converge to treating all pixels as background.

Some studies have already applied the feature learning capability of generative models to crack detection. CrackGAN [39] proposed a crack-patch-only (CPO) supervised GAN network for generating crack-GT images. Deep convolutional GAN [40] is employed to generate a crack image dataset. Conditional GAN [41] is used to first extract the road and then detect cracks. However, due to the simultaneous training of both the generator and discriminator, GANs are challenging to balance, leading to training instability.

2.2. Diffusion Model

The diffusion model is a type of probabilistic generative model that can be divided into two main stages: the forward noise-adding stage and the backward denoising stage. In the forward noise stage, original images are progressively corrupted by adding Gaussian noise to the original data. In the backward denoising stage, the generative model’s task is to learn the noise-adding in the forward process and recover the original input data from the noisy data. Currently, diffusion models can be categorized into three main types, namely Denoising Diffusion Probabilistic Models (DDPMs) [20], Score-based Generative Models [42], and Generative Models based on Stochastic Differential Equations [43]. CrackDiff is founded on the architecture of the DDPM.

The inspiration for the DDPM comes from non-equilibrium thermodynamics, and the training process involves two stages: the forward diffusion with noise process and the reverse denoising process, which will be explained in the next section. To accelerate the sampling speed, for traditional forward Markov processes, Denoising Diffusion Implicit Models (DDIMs) [44] has demonstrated that using non-Markovian processes can still yield good generative results and improve the sampling speed. The IDDPM [45] reduced the number of sampling steps by adding cosine noise during the forward noise injection process and introducing learnable variance during the reverse denoising process. In terms of model structure, D2C [46] uses the idea of contrastive representation learning for the diffusion decoder model and improves the quality of representations through contrastive self-supervised learning. Peebles [47] replaced the commonly used UNet network for reverse image generation with transformer networks, achieving better training results. In the realm of conditional diffusion probabilistic models, the IDDPM [45] incorporates class information as a conditional embedding in the timestamp embedding, generating images of a given class. SRDiff [48] combines diffusion generative models to make predictions for super-high-resolution images (SR) during the reverse process. It uses the low-resolution information encoded by an encoder as conditional noise to progressively denoise high-resolution images and generate super-resolution images. SegDiff [49] adds original image embeddings to the network and generates segmentation results for the original image.

Apart from the applications mentioned above, diffusion models have been successfully applied to various complex visual problems, such as image-to-image translation [50] and image blending [51], demonstrating the excellent capability of diffusion models in capturing latent patterns and relationships between samples.

3. Methodology

In this section, the foundational diffusion model will be introduced initially, followed by an exploration of the enhanced diffusion model utilized for pavement crack detection. The details of the network and other pertinent information employed in this context will also be presented.

3.1. Diffusion Model

The diffusion model includes two processes: the forward noise-adding process and the backward denoising process. Both the forward and backward processes are parameterized Markov chains, where the backward process can be employed for data generation.

The forward process is also referred for the diffusion process. It gradually adds Gaussian noise to data until the data become random noise. For the original ground truth image (segmentation mask) ${\mathbf{GT}}_0$∼$q\left({\mathbf{GT}}_0\right)$, the diffusion process consists of a total of T steps. In each step, Gaussian noise is added to the data obtained in the previous step, as shown in Equation (1):

$$q\left({\mathbf{GT}}_t\mid {\mathbf{GT}}_{t-1}\right)=\mathcal{N}\left({\mathbf{GT}}_t;\sqrt{1-{\sigma }_T^2}{\mathbf{GT}}_{t-1},{\sigma }_T^2\mathbf{I}\right)$$

(1)

${\left\{{\sigma }_T^2\right\}}_{t=1}^T$ represents the variance of noise used at each step. It is manually specified and falls within the range of 0 to 1. As each step adds random noise which is unrelated to the previous step, generating ${\mathbf{GT}}_t$ at an arbitrary timestamp t from ${\mathbf{GT}}_0$ is a Markov process:

$$q\left({\mathbf{GT}}_t\mid {\mathbf{GT}}_0\right)=\prod _{i=1}^{t}q\left({\mathbf{GT}}_i\mid {\mathbf{GT}}_{i-1}\right)$$

(2)

With the reparameterization technique (similar to VAE), the formula to calculate ${\mathbf{GT}}_t$ without iteration is:

$$q\left({\mathbf{GT}}_t\mid {\mathbf{GT}}_0\right)=\mathcal{N}({\mathbf{GT}}_t;\sqrt{{\overline{\alpha }}_t}{\mathbf{GT}}_0,\left(1-{\overline{\alpha }}_t\right)\mathbf{I})$$

(3)

where ${\alpha }_t=1-{\sigma }_T^2$ and $\overline{{\alpha }_t}={\prod }_{i=1}^t{\alpha }_i$.

In the denoising process, given a random noise ${\mathbf{GT}}_T$∼$\mathcal{N}(0,\mathbf{I})$, a noise-free predictive segmentation image can be systematically generated if the distribution of each step $q({\mathbf{GT}}_{t-1}\mid {\mathbf{GT}}_t)$ is known. That is the purpose our network needs to learn. Based on Bayes’ theorem and Equation (3), the following result is derived:

$$\begin{array}{cc}\hfill q\left({\mathbf{GT}}_{t-1}\mid {\mathbf{GT}}_t,{\mathbf{GT}}_0\right)& =q\left({\mathbf{GT}}_t\mid {\mathbf{GT}}_{t-1},{\mathbf{GT}}_0\right)\frac{q\left({\mathbf{GT}}_{t-1}\mid {\mathbf{GT}}_0\right)}{q\left({\mathbf{GT}}_t\mid {\mathbf{GT}}_0\right)}\hfill \\ \hfill & =\mathcal{N}\left({\mathbf{GT}}_{t-1};{\tilde{\mathit{\mu }}}_t\left({\mathbf{GT}}_t,{\mathbf{GT}}_0\right),{\widetilde{{\sigma }_t}}^2\mathbf{I}\right)\hfill \end{array}$$

(4)

where ${\tilde{\mathit{\mu }}}_t$ and $\widetilde{{\sigma }_t}$ are calculated using the reparameterization technique, as shown below.

$${\tilde{\mathit{\mu }}}_t\left({\mathbf{GT}}_t,t\right)=\frac{1}{\sqrt{{\alpha }_t}}{\mathbf{GT}}_t-\frac{{\sigma }_t^2}{\sqrt{{\alpha }_t(1-{\overline{\alpha }}_t)}}{ϵ}_{\theta }\left({\mathbf{GT}}_t,t\right)$$

(5)

$$\widetilde{{\sigma }_t}=\sqrt{\frac{1-{\overline{\alpha }}_{t-1}}{1-{\overline{\alpha }}_t}}·{\sigma }_t$$

(6)

${ϵ}_{\theta }\left({\mathbf{GT}}_t,t\right)$ is the noise added at timestep t and learned by the neural network. $\widetilde{{\sigma }_t}$ can be computed through the initial variance without the need for learning.

Then, ${\mathbf{GT}}_{t-1}$ can calculate using Equations (4) and (5).

$${\mathbf{GT}}_{t-1}={\tilde{\mathit{\mu }}}_t\left({\mathbf{GT}}_t,t\right)+\widetilde{{\sigma }_t}\eta$$

(7)

$\eta$ is a random noise sampled from a standard normal distribution. ${\mathbf{GT}}_0$ can be generated step by step using Equation (7).

3.2. Crack Diffusion Model

This section will introduce the Crack Diffusion Model (CrackDiff). The main differences between CrackDiff and the diffusion model are model input and output. The input of noise estimation network ${ϵ}_{\theta }$ at the training stage is the combination of embedding derived from both the ground truth at the current step and the crack image. The input at the sample (test) stage is the combination of embedding derived from the current estimation and the crack image, as shown in Figure 1.

The embedding of the ground truth or the estimation is denoted as $Em{b}_{\mathbf{GT}}$ and the embedding of the crack image is denoted as $Em{b}_I$. These features are summed and passed to ${ϵ}_{\theta }$’s encoder network $\mathbb{E}$.

${ϵ}_{\theta }$ is structured as a multi-task UNet, comprising an encoding branch and two decoding branches. The decoding branches are responsible for predicting the segmentation (mask) and ${ϵ}_{\theta }({\mathbf{GT}}_t,t)$ (noise), respectively. The encoder part ${ϵ}_{\theta }$ is denoted as ${ϵ}_{{\theta }_{encode}}$, the two decoder part of ${ϵ}_{\theta }$ is denoted as ${ϵ}_{{\theta }_{decode}}^{mask}$ and ${ϵ}_{{\theta }_{decode}}^{noise}$, and the network ${ϵ}_{\theta }$ can be formulated as follows:

$$\begin{array}{c}\hfill {ϵ}_{{\theta }_{encode}}\left({\mathbf{GT}}_t,I,t\right)=\mathbb{E}(Em{b}_{\mathbf{GT}}\left({\mathbf{GT}}_t\right)+Em{b}_I\left(I\right), Em{b}_t\left(t\right))\end{array}$$

(8)

$$\begin{array}{c}\hfill {ϵ}_{{\theta }_{decode}}^{mask}\left({\mathbf{GT}}_t,I,t\right)={\mathbb{D}}_{mask}(\mathbb{E},Em{b}_t\left(t\right))\end{array}$$

(9)

$$\begin{array}{c}\hfill {ϵ}_{{\theta }_{decode}}^{noise}\left({\mathbf{GT}}_t,I,t\right)={\mathbb{D}}_{noise}(\mathcal{E},Em{b}_t\left(t\right),{\mathbb{D}}_{mask})\end{array}$$

(10)

The mask decoder network ${\mathbb{D}}_{mask}$ is the same architecture as UNet. The noise decoder network ${\mathbb{D}}_{noise}$ takes both the encoder feature map and mask decoder feature map as inputs. The current step index t is encoded and added to both the encoder and decoder parts. The details of the network architecture will be explained in the following section.

The variance parameter ${\sigma }_t^2$ in the diffusion process is a linearly increasing constant, the same as the DDPM [20], as shown below.

$${\sigma }_t^2=\frac{{10}^{-4}(T-t)+2\ast {10}^{-2}(t-1)}{T-1}$$

(11)

Algorithm 1 depicts the training procedure. The input training dataset comprises images of cracks alongside their corresponding ground truth segmentation images. The hyperparameter total diffusion steps (T) are predefined. In the training procedure, a random number t is sampled from a uniform distribution as the current timestep, and a random noise $ϵ$ is sampled from a standard normal distribution. ${\sigma }_t,{\alpha }_t,\overline{{\alpha }_t}$, and ${\mathbf{GT}}_t$ are sequentially calculated according to Equations (3) and (11). The loss function will be introduced in next section.

Algorithm 1 Training Algorithm

Input: images and masks $D={\left\{(I_i,{\mathbf{GT}}_i)\right\}}_i^N$, total diffusion steps T  repeat   Sample $t\sim \{1,\dots ,T\},ϵ\sim N(0,I),(I_i,{\mathbf{GT}}_i)\sim D$   Calculate ${\sigma }_t,{\alpha }_t,\overline{{\alpha }_t},{\mathbf{GT}}_t$ according to Equations (3) and (11)   Calculate loss $L_t$ and gradient ${\nabla }_{\theta }$ according to Equation (12)   Backward  until iteration stop

The inference procedure is depicted in Algorithm 2. It takes a random noise ${\mathbf{GT}}_T$ sampled from a standard normal distribution as input and a crack image as guidance. The total diffusion steps (T) are the same as in training procedure. From T to 1, the network predicts the mean and variance of noise in the current timestep to denoise ${\mathbf{GT}}_t$ and calculate the estimation ${\mathbf{GT}}_{t-1}$ according to Equation (7).

Algorithm 2 Inference Algorithm

Input: image I, total diffusion steps T  Sample ${\mathbf{GT}}_T\sim N(0,I)$  for $t=T,T-1,\dots ,1$ do   Sample $\eta \sim N(0,I)$   Calculate ${\sigma }_t,{\alpha }_t,\overline{{\alpha }_t},{\tilde{\sigma }}_t$   Calculate noise prediction ${ϵ}_{{\theta }_{decode}}^{noise}({\mathbf{GT}}_t,I,t)$   Calculate ${\mathbf{GT}}_{t-1}$ according to Equation (7)  end for  return ${\mathbf{GT}}_0$

3.3. Structure of the Denoising Network

The network structure used for predicting ground truth and noise at each step is illustrated in Figure 2a. The network takes the crack image, the ground truth image at timestep t, and the time step t as input. The output consists of the segmentation image and the noise image at timestep t.

The crack image encoder is constructed using Residual in Residual Dense Blocks (RRDBs) [52], as shown in Figure 2b. It consists of multiple 3 × 3 convolutional layers with residual connections to the higher-level modules. The output dimensions of the convolutional layers are indicated in parentheses. The ground truth image at time step ${\mathbf{GT}}_t$ is encoded using a single layer of 3 × 3 convolution. The structure of the encoder layers of timestep t and the residual block is depicted in Figure 3. First, a lookup table for t is defined for map t into a feature vector, which is then encoded through a series of fully connected layers. The encoded crack image and ${\mathbf{GT}}_t$ are used as the initial inputs to the network, and the encoded timestep t is added to each residual module. The attention block in the residual block is a self-attention block proposed in a non-local neural network [53].

The encoder part of the network consists of four sets of modules. Each set contains two residual modules and a downsampling module, with the feature dimensions after each set marked in the figure. The residual module, as depicted in Figure 3, includes group normalization, Silu activation, and 2D convolution layers. The residual blocks receive time embeddings through a linear layer, and there is an attention layer after each residual block, including multiple attention heads. The downsampling module is a 3 × 3 convolution layer with a stride of 2.

The decoder part of the network comprises two branches, each consisting of four sets of modules. Similar to the encoder, each module contains upsampling modules and three residual modules. The residual modules are the same as in the encoder, and the upsampling module includes a nearest-neighbor interpolation function and a 2D convolution layer. Each layer in the encoder has a skip connection to the left branch and the right branch of the decoder, and at the same time, each layer in the decoder’s left branch has a skip connection to the right branch.

The loss function of the multi-task network includes two parts: noise-predicted loss and mask-predicted loss, as shown in Equation (12).

$$\begin{array}{cc}\hfill L_t=& {\mathbb{E}}_{{\mathbf{GT}}_0,ϵ,I,t}[Los{s}_I({\mathbf{GT}}_0,{ϵ}_{{\theta }_{decode}}^{mask}(\sqrt{{\overline{\alpha }}_t}{\mathbf{GT}}_0+\sqrt{1-{\overline{\alpha }}_t}ϵ,I_i,t))\hfill \\ \hfill & +\parallel ϵ-{ϵ}_{{\theta }_{decode}}^{noise}(\sqrt{{\overline{\alpha }}_t}{\mathbf{GT}}_0+\sqrt{1-{\overline{\alpha }}_t}ϵ,I_i,t){\parallel }^2]\hfill \end{array}$$

(12)

The image loss function $Los{s}_I$ used in the article is Focal Tversky Loss, with $\alpha =0.7,\beta =0.3,\gamma =0.75,\delta =1$, as shown in Equation (13).

$$Los{s}_I={(1-\frac{TP+\delta }{TP+\alpha \ast FP+\beta \ast FN+\delta })}^{\gamma }$$

(13)

4. Experiments

4.1. Experimental Settings

In order to test the performance of CrackDiff, this study conducted experiments on two public pavement crack datasets, Crack500 [1] and DeepCrack [12]. These two datasets offer diverse representations of road crack images, encompassing various pavement materials, lighting conditions, and weather scenarios, thereby enabling a more comprehensive training and evaluation of models, enhancing their generalization capabilities. Additionally, Crack500 and DeepCrack are widely used datasets in the field of pavement crack detection and have been extensively applied in academic research. Therefore, utilizing these two datasets enables better comparison and validation with existing studies, thereby enhancing the credibility and reproducibility of the research.

The Crack500 dataset comprises 500 pavement images, each approximately 2000 × 1500 pixels in size, captured by cell phones across the main campus of Temple University. The predominant material of the pavement is asphalt. These images were subdivided into 16 non-overlapping regions, with only those regions containing over 1000 pixels of cracks retained. After removing images with significant annotation errors and resizing all images to 448 × 448 pixels, 1896 training images, 348 validation images, and 400 test images are randomly selected for our experiment.

The DeepCrack dataset comprises 537 pavement images meticulously annotated for segmentation, with each image measuring 544 × 384 pixels. The pavement predominantly consists of asphalt and concrete materials. The dataset is partitioned into 300 training images, 137 validation images, and 100 test images for comprehensive evaluation and analysis.

Our method is compared with widely used models UNet [24], FCN [25], DeepLabV3+ [26], improved models PSPNet [28], HRNet [31], transformer-based model SegFormer-B5 [35], customized models for crack detection DeepCrack [12], MFPANet [17], CrackFormer [36], CT-crackseg [38], and diffusion-based model SegDiff [49]. The UNet architecture is the same as the backbone of the CrackDiff network with only the crack predict decoder. Precision, recall, F1 score, and mIoU (Mean Intersection over Union) are commonly used evaluation metrics in crack detection and image segmentation tasks. The calculation of precision and recall is influenced by threshold settings. The F1 score is the harmonic mean of precision and recall, providing a comprehensive evaluation of the model’s performance. It does not fluctuate significantly with changes in probability thresholds, thus offering a more objective assessment. The mIoU calculates the accuracy of pixel positioning, and its value also remains relatively stable when probability thresholds change. It provides a comprehensive assessment of the model’s segmentation performance for cracks and backgrounds, rather than simply evaluating overall pixel-level accuracy.

$$F1=\frac{2\times precision\times recall}{precision+recall}$$

(14)

$$mIoU=\frac{1}{2}(\frac{TP}{TP+FP+FN}+\frac{TN}{TN+FN+FP})$$

(15)

During the inference process, as the initial values are sampled from a standard normal distribution, each inference result exhibits a certain level of variability. Therefore, this study conducts multiple inferences and averages the results. The variability and impact of multiple inferences will be discussed in the analysis section. In this study, inference is performed 10 times.

The AdamW optimizer is employed for updating the parameters of the network, where the initial learning rate is 1 × 10⁻⁴. The diffusion step T is 500 and the training epoch is 500. The size of the input images is 256 × 256. No data augmentation algorithms were employed on the input data. The algorithms in this study, as well as the comparative algorithms, were implemented on two RTX 4090 GPUs.

4.2. Experiment Results

The comparison of the crack segmentation results of CrackDiff and other methods on Crack500 is shown in

Table 1. Comparison between CrackDiff and alternative methods on the Crack500 dataset.

Method	Precision	Recall	F1	mIoU
UNet [24]	76.00	87.53	80.20	83.00
FCN [25]	77.58	84.17	79.67	82.64
DeepLabV3+ [26]	77.99	85.88	80.72	83.39
PSPNet [28]	76.01	73.45	72.93	78.01
HRNet [31]	85.04	74.83	78.49	81.56
MFPANet [17]	75.73	86.88	80.92	82.83
SegFormer [35]	85.89	78.00	80.84	83.32
CrackFormer [36]	84.06	79.19	81.55	82.47
CT-crackseg [38]	68.51	76.37	72.14	74.89
DeepCrack [12]	81.17	78.97	78.21	81.66
SegDiff [49]	80.47	79.63	80.05	80.90
CrackDiff (ours)	81.34	84.10	81.78	84.09

and Figure 4. CrackDiff achieved slightly lower precision and recall metrics compared to other models, but it obtained the best scores in terms of the F1 and mIoU metrics. The comparison of crack segmentation results with DeepCrack is shown in

Table 2. Comparison between CrackDiff and alternative methods on the DeepCrack dataset.

Method	Precision	Recall	F1	mIoU
UNet [24]	79.76	88.08	81.82	84.62
FCN [25]	78.68	82.08	77.90	82.02
DeepLabV3+ [26]	68.93	88.56	74.98	80.10
PSPNet [28]	63.93	48.23	51.46	66.84
HRNet [31]	89.58	68.51	78.42	82.13
MFPANet [17]	81.82	85.67	83.70	84.62
SegFormer [35]	91.79	71.09	79.42	82.56
CrackFormer [36]	88.32	78.11	82.90	85.46
CT-crackseg [38]	87.43	79.87	83.48	85.83
DeepCrack [12]	81.56	84.02	80.49	83.85
SegDiff [49]	85.60	68.10	75.85	84.07
CrackDiff (ours)	91.93	79.45	84.06	86.24

and Figure 5. CrackDiff obtains the best scores for precision, F1, and mIoU.

The highest F1 and mIoU scores demonstrate that CrackDiff excels in comprehensive performance and strikes a good balance between precision and recall. In the crack segmentation task, which involves an imbalance between positive (cracks) and negative (background) samples, it performs well in handling both the positive and negative classes. The mIoU score indicates that CrackDiff performs well in terms of both localization and classification accuracy.

In Figure 4 and Figure 5, the characteristics of the model that extend beyond the scope of most metrics can be observed. Given the complexity of ground truth annotations in the task of crack segmentation, many cracks cannot be accurately labeled, and their contours are not well represented in the ground truth. Comparing the prediction of CrackDiff with the original image, CrackDiff demonstrates high precision, as evidenced by the sharper boundaries in its segmentation results and its ability to predict less prominent cracks even better than ground truth. This, to some extent, explains why CrackDiff may not exhibit the best precision or recall but achieves the highest F1 and mIoU scores.

Figure 6 shows the confusion matrices for the detection results. From the results, we can observe that pavement crack detection is an imbalanced task, with over 90% of pixels belonging to the background, while crack pixels only account for around 5%. CrackDiff misclassified approximately 2% of the pixels. On DeepCrack, there are very few background pixels classified as cracks, which explains why CrackDiff has the highest precision on DeepCrack.

Furthermore, the formation of cracks often exhibits a certain level of continuity, meaning there is a spatial intrinsic connection between them. In CrackDiff’s prediction results, the cracks demonstrate better continuity and there are few isolated predictions, which indicates that CrackDiff has learned both the shallow and deep features of the cracks.

4.3. Model Analysis

4.3.1. Visualization of Diffusion Steps

Figure 7 depicts CrackDiff’s prediction results at different timesteps T during the diffusion process, along with the predictions from the crack prediction branch. In the figure, blue represents 0, and red represents 1. The results show that during the diffusion process, the pixel values in the crack region gradually tend toward 1, while those in the background region gradually tend toward 0. In the initial stages of the diffusion process, it primarily outlines the approximate contour of the crack. In the later 100 steps, the diffusion results exhibit significant changes, yielding fine-grained crack segmentation results.

The trend of the crack prediction branch aligns with the diffusion results. In the early stages, the model can predict the approximate contour of the crack. As the diffusion steps progress, the crack prediction results become more refined, and the boundaries become sharper.

4.3.2. Ablation Study

To validate the effectiveness of the diffusion model and multi-task architecture in the CrackDiff model, this study compared it with models that only use the backbone for segmentation and models that perform segmentation without a multi-task structure. The results are presented in

Table 3. Results of ablation study.

	Method	Precision	Recall	F1	mIoU
Crack500	Backbone	76.00	87.53	80.20	83.00
	Single task	80.47	79.63	80.05	80.90
	CrackDiff	81.34	84.10	81.78	84.09
DeepCrack	Backbone	79.76	88.08	81.82	84.62
	Single task	85.60	68.10	75.85	84.07
	Crackdiff	91.93	79.45	84.06	86.24

.

From Table 3, it can be observed that compared to using the UNet architecture backbone directly for crack segmentation, the diffusion model enhances precision, F1, and mIoU, with a slight decrease in recall. When compared to the single-task diffusion model that only predicts noise, the multi-task diffusion model shows improvements in all metrics, indicating that the task of predicting the ground truth effectively enhances the guidance effect on the original input images.

4.3.3. Impact of the Number of Generations

In the experimental settings, it was noted that the sampling of the diffusion model from a standard distribution introduces variability in the generated segmentation results. To mitigate the impact of this variability, this study employed the method of multiple sampling (10 times) and taking the mean. This section discusses the influence of averaging the results of multiple samplings.

The variation in metrics with the number of result accumulations is shown in Figure 8. From the figure, it can be observed that as the number of samplings increases, precision gradually decreases, while recall, F1, and mIoU all show improvement. The results from two samplings are significantly better than those from one sampling, and as the number of samplings increases, the improvement becomes less pronounced. When 10 samplings are performed, F1 and mIoU no longer show improvement. Therefore, by increasing the number of samplings appropriately, the accuracy of the results can be improved.

4.3.4. Robustness of CrackDiff

To validate that multi-tasking effectively enhances the stability of the diffusion model, this study analyzed the metrics for each sampling result (without accumulation) for both multi-task and single-task structures. The results are presented in Figure 9. From the figure, it can be observed that the sampling results for the single-task structure exhibit greater variability in all metrics compared to the multi-task structure. This indicates that the sampling results generated by the multi-task structure have less variability.

For a quantitative depiction of the model’s stability, the standard deviation of each metric was computed, as shown in

Table 4. Standard deviations for multiple generations. The smaller, the better.

	Precision	Recall	F1	mIoU
Single	0.514	0.63	0.508	0.289
Multi	0.112	0.084	0.095	0.067

. The table results align with the results in the figure, where the standard deviation of each metric for the multi-task structure is significantly smaller than that of the single-task structure. The standard deviation of accuracy for multiple samplings is 0.112%, and the standard deviations for recall, F1, and mIoU are all less than 0.1%. This indicates that CrackDiff produces reliable results.

4.3.5. Model Generalization

Model generalization refers to the ability of a model to perform effectively on unseen data beyond the training dataset. It is a critical aspect of model evaluation, ensuring that the model’s performance extends to real-world scenarios. In this section, we use two different pavement datasets that were not used in the previous experiment to evaluate our model generalization.

The CrackForest-Dataset (CFD) [10] comprises 118 pavement images captured using a mobile phone, with each image having a resolution of 480 × 320 pixels. These images contain a notable presence of noisy pixels, including oil spots and water stains. CrackTree [54] consists of 260 pavement images captured using an area-array camera, boasting a higher resolution of 800 × 600 pixels. However, a significant portion of the images in CrackTree exhibit shadows, which poses challenges for accurate crack detection.

The test results for CrackDiff (trained on Crack500) on representative images from CFD are shown in Figure 10. The test results for CrackDiff (trained on Crack500) on representative images from CrackTree are shown in Figure 11. In Crack500, there are no data affected by road markings and shadows, so these test images challenge the model’s generalization. From the results, it is evident that CrackDiff performs better overall on the CFD dataset, showing similar results to Crack500. Road markings have a minor impact on CrackDiff. However, its performance is poorer on the CrackTree dataset, indicating that shadows have a significant effect on CrackDiff. Cracks in shadows are challenging to detect, and some shadows may be falsely detected.

5. Discussion

In this section, we will discuss the limitations of the model. The first is the model’s inference time. Due to the sampling process of the diffusion model starting with random noise and gradually generating images, this step cannot be parallelized, resulting in slower inference speeds.

Table 5. GPU inference time(s) for CrackDiff and comparison methods.

Method	Crack500	DeepCrack
UNet [24]	15.01	4.15
DeepLabV3+ [26]	16.19	4.55
MFPANet [17]	30.83	8.60
DeepCrack [12]	18.76	5.40
SegFormer [35]	26.48	7.37
PSPNet [28]	7.90	2.66
CrackDiff	892.57	243.88

illustrates the comparison of inference time between the proposed method and the comparison methods. CrackDiff does indeed require significantly more time compared to the other methods. Currently, accelerating the sampling in diffusion models, such as the DDIM and IDDPM, is a key research focus. In future research, we will investigate the performance of these faster diffusion models in pavement crack detection tasks.

Another limitation of CrackDiff is inherent to 2D visible light images themselves. A significant limitation of visible light images is their sensitivity to lighting conditions. For instance, in model generalization analysis, it becomes challenging for models to detect cracks within shadows. Additionally, variations in pavement textures can affect the accuracy of crack detection algorithms. To address these issues, some methods involving the use or fusion of other sensors have been proposed, such as employing infrared thermography [2] and LiDAR scanning [3]. Compared to 2D cameras, 3D laser scanning can capture the depth information for the cracks, providing additional metrics for assessing structural conditions. The detection system based on infrared thermography utilizes the contrasting reflectivity and surface temperature characteristics between cracks and intact pavement, resulting in a pronounced contrast in the infrared images of the two targets. These approaches can capture more useful information even under poor lighting conditions. We will also explore how to leverage them to enhance the effectiveness of pavement crack detection.

6. Conclusions

This paper introduces a diffusion model structure, CrackDiff, which is the first application of the diffusion model for pavement crack detection. In this architecture, crack images are employed as a guide for the sampling process, and a multi-task UNet framework is utilized to predict crack truth and noise simultaneously. Robust crack segmentation results are obtained by sampling random noise. Compared to the other models, CrackDiff excels in capturing both shallow and deep image features. Through comparisons with advanced algorithms on the Crack500 and DeepCrack datasets, CrackDiff achieves superior performance.

The paper also proposes insights regarding conditional diffusion models. In comparison to single-task network structures, multi-task structures exhibit stronger robustness and can produce results that align with conditions. However, the diffusion model faces challenges such as slow inference speed and difficulty in detecting cracks within shadows. Further research is needed to enhance the inference speed of diffusion models and explore broader applications.

The code will be made publicly accessible. Please refer to https://github.com/zhysess/CrackDiff for the address (accessed on April 2024).