MSP U-Net: Crack Segmentation for Low-Resolution Images Based on Multi-Scale Parallel Attention U-Net

Abstract

As the expected lifespans of structures and road approaches, as well as the importance of road maintenance, increase globally, safety inspections have emerged as a crucial task. Nonetheless, the existing crack detection models focus on multi-scale feature loss and performance degradation in learning various types of cracks. We propose the Multi-Scale Parallel Attention U-Net (MSP U-Net) as a network designed for low-resolution images that considers the irregular characteristics of cracks. MSP U-Net applies a large receptive field flock to an attention U-Net, minimizing feature loss across multiple scales. Using the Crack500 dataset, our network achieved a mean intersection of union (mIoU) of 0.7752, outperforming the existing methods on low-resolution images.

1. Introduction

As traffic structures and paved roads worldwide are approaching their expected lifespans, cracks tend to form and expand over time, compromising traffic safety. Consequently, safety inspections for road maintenance are conducted annually across the globe [1]. These inspections are crucial in ensuring safe transportation and traffic conditions. Research in this domain can be divided into two methodologies. The first methodology is based on traditional image processing techniques [2]. Because this approach considers only the features of images—without considering the characteristics of target objects, such as edge detectors—it typically exhibits limitations in detecting targets with irregular features in various environments (e.g., lighting, angles, and backgrounds). The second methodology is based on deep learning technology, which offers the advantage of generalizability to various environments and irregular objects. Deep learning is a stable methodology that considers not only the features of images but also those of detection targets. The core of this technology is the convolutional neural network (CNN), which has consistently outperformed traditional computer vision techniques. With the increasing use of graphics processing units (GPUs), the development of CNNs has also accelerated. Since the emergence of AlexNet [3], which won the ImageNet LSVRC-2010 contest, a wide range of CNN-based architectures have been researched and applied to tasks such as road crack detection, including VGGNet [4], introduced in 2014; ResNet [5], introduced in 2015; and DenseNet [6], introduced in 2017.

Although the early neural-network-based models for crack detection have correctly outlined the research trajectory, they exhibited clear drawbacks. First, these models were insufficiently adaptable to diverse data characteristics, such as brightness and illumination, and were highly sensitive to image resolution. Recently, researchers have developed segmentation algorithms that leverage deep learning with advantages in various environments.

Lau et al. [7] employed a pre-trained ResNet-34 from ImageNet as an encoder and constructed a decoder using transpose convolutional layers, similar to the conventional U-Net, achieving an F-1 Score of 0.7327. They demonstrated distinct advantages over other methods by applying spatial and channel feature extraction techniques in the form of Spatial and Channel Squeeze and Excitation (SCSE) modules [8]. Nguyen et al. [9] applied preprocessing techniques suitable for deep-learning-based networks rather than digital image processing techniques used in traditional computer vision tasks, showing good accuracy and model stability. Yu et al. [10] proposed a semantic segmentation network called RUC-Net that combines ResNet-18 with U-Net. They applied focal loss functions to address class imbalance issues, reduced sample weights, and demonstrated better detection performance than classical segmentation algorithms. Wang et al. [11] proposed U-Net with DenseNet121 as an encoder based on a Pyramid Attention Network (PAN). They applied feature pyramid attention modules between the encoder and decoder to extract high-level features for classification and segmentation. The decoder module achieved good performance by accurately combining features from both low and high dimensions using the Global Attention Upsample (GAU) module. Di et al. [12] designed a U-Net-based architecture using a ResNet-50 encoder. Through transfer learning, the proposed architecture outperformed the existing methods using a common dataset. Choi et al. [13] proposed a network called SDDNet by applying a purely deep learning method. This network can perform real-time crack detection while ignoring backgrounds that resemble crack surfaces. Despite an 88-fold reduction in parameters compared to the latest models at the time of publication, SDDNet achieved better performance and faster processing speed than said models. Kang et al. [14] proposed STRNet, which excels at segmenting concrete cracks at the pixel level. The network was trained using a large-scale dataset and tested on 545 images. The performance-recorded Precision, Recall, F-1 Score, and mIoU were 91.7%, 92.7%, 92.2%, and 92.6%, respectively. The proposed network has the advantage of processing relatively large input images (1280 × 720 and 1024 × 512) in real-time while maintaining excellent performance. Chun et al. [15] proposed a road surface crack detection technique using fully convolutional neural networks (FCNNs). They segmented images obtained while driving on actual roads, with a total of 6756 images, and classified pavement damage into six classes. Because the dataset was initially insufficient, it was stabilized using K-fold cross-validation. Furthermore, they applied a semi-supervised learning-based method using pseudo-label datasets, outperforming conventional supervised-learning-based networks.

Although the aforementioned studies have been somewhat successful in improving crack detection performance, they still exhibited suboptimal accuracy when handling low-resolution images. One of the main reasons for this is the loss of features in certain layers of the encoder in an autoencoder-based structure. Drawing inspiration from this, we propose a network that achieves improved detection performance with low-resolution images. The contributions of the proposed network are summarized as follows:

First, we focused on the loss of features occurring in certain network layers as a major cause of degradation in detection performance. To mitigate this feature loss, we propose an attention gate with a larger receptive field block that preserves the diversity of the receptive field. Second, we resized high-resolution images to a lower resolution as part of preprocessing and used them as input data for training, demonstrating satisfactory crack detection performance even in low-resolution settings. Note that the high-resolution images range from 512 × 512 to 1024 × 1024, while the low-resolution images are set to 128 × 128.

In this paper, we briefly describe related studies on the methods used and present the results of experiments where our approach is applied. Our research excels at detecting fine cracks, particularly in low-resolution images, as demonstrated in the experimental results.

2. Related Work

In this section, we introduce the methods primarily used in our research and the related studies.

2.1. Residual Block

The residual block, which is widely used in neural networks, is based on the concept of residual learning utilized in ResNet [5]. This concept involves a skip connection, wherein the result, after passing through the activation function following the convolutional layers, is added to the result of the previous layer’s operation. Figure 1 illustrates the skip connection of a residual block.

This alleviates the vanishing gradient problem that occurs when a backpropagation algorithm is used in deep networks.

2.2. Attention Network

Attention [16] was introduced in the field of natural language processing with the presentation of transformer architecture. The input sequence is divided into a Query, Key, and Value to calculate the relevance of each element with respect to every other element. This procedure is referred to as self-attention. The following formula represents the objective function for linear operations with the elements of the input sequence: Query, Key, Value, and dimensions d_k and V.

$$\mathrm{A}\mathrm{t}\mathrm{t}\mathrm{e}\mathrm{n}\mathrm{t}\mathrm{i}\mathrm{o}\mathrm{n}\left(\mathrm{Q},\mathrm{K},\mathrm{V}\right)=softmax\left({\displaystyle \frac{Q{K}^T}{\sqrt{d_k}}}\right)$$

(1)

As an extension of this concept, Multi-Head Attention (MHA) is designed to extract various features and compute their relevance to the input sequence. Through this process, the relationship between the input and output can be determined. Finally, positional encoding is performed to restore the order of the previously divided input sequence, allowing the network to determine the sequence information of the input and relative positions of the feature elements. The formula for MHA calculation is as follows:

$$\mathrm{M}\mathrm{H}\mathrm{A}\left(Q, K, V\right)=Concat\left({head}_1, \cdots , {head}_h\right)W^O$$

(2)

$${head}_i=\mathrm{A}\mathrm{t}\mathrm{t}\mathrm{e}\mathrm{n}\mathrm{t}\mathrm{i}\mathrm{o}\mathrm{n}(\mathrm{Q}{W}_i^Q,\mathrm{K}{W}_i^K,\mathrm{V}{W}_i^V)$$

(3)

In recent years, attention mechanisms have demonstrated excellent performance in various fields, including segmentation. By mimicking the human perception of visual information, the attention mechanism selects important and useful information from large amounts of data. This is a reasonable approach for detecting cracks involving unspecified and diverse variables.

Lin et al. [17] introduced a full-attention U-Net, which utilizes an attention mechanism and skip connections. The distinguishing feature of this network is the application of attention blocks to all layers by incorporating attention networks in the decoder as well as the encoder. Although this approach exhibits slightly lower performance compared to the case where an attention network is only used in the decoder, it achieves higher performance when noise removal strategies are applied.

Wang et al. [18] proposed the SegCrack model, which adopts a hierarchical transformer architecture to output multistage features and alleviates class imbalances by applying an ascending path, lateral connections, and the OHEM strategy. By subsampling advantageous pixel features for losses, it enhances crack detection performance, achieving an F-1 Score and mIoU of 96.05% and 92.63%, respectively, outperforming CNN-based networks. Sun et al. [19] introduced the DMA-Net, which incorporates a multi-scale attention module into the decoder component of the DeepLabv3+ model, thereby improving the integration of multi-scale features present in irregularly shaped cracks. They conducted extensive experiments using three datasets, demonstrating the practicality of the proposed method even with smartphone-captured pavement images.

Research utilizing vision transformers (ViTs), an extended form of the attention mechanism, has also been actively pursued. Tao et al. [20] proposed a convolution-transformer network for crack segmentation. This network—which included extended residual blocks, boundary-aware modules, and MobileViT blocks—was experimentally demonstrated to outperform previously developed methods.

Dong et al. [21] proposed a patch-based semantic segmentation method that trains by obtaining image-level labels instead of pixel-level annotations by dividing images into patch units, creating synthetic labels through identification/location algorithms, and performing CRF processing to reduce method complexity. Moon et al. [22] proposed PCTC-Net. This parallel dual-encoder network, which integrates pre-conv-based transformers and CNNs (PCTC-Net), addresses data-intensive problems with relatively small datasets, demonstrating higher generalizability than the DTC-Net state-of-the-art (SOTA) model.

Oktay et al. [23] proposed the integration of an attention mechanism into a U-Net architecture. In pancreatic segmentation tasks conducted in medical imaging, the segmentation performance was improved compared to the original U-Net architecture, demonstrating the network’s ability to focus on image features.

3. Proposed Method

3.1. Proposed Architecture

The proposed model represents a deep learning technique that can accurately detect and segment cracks of various sizes from images. Figure 2 illustrates the convolution block implemented in the proposed architecture. In the encoder, the convolutional layers pass through, generate filters, and apply the batch normalization and ReLU activation functions.

Figure 3 presents an overview of the proposed network. Initially, 2 × 2 max pooling is performed to reduce the spatial dimensions. Subsequently, the 3 × 3 CNN layers pass sequentially based on the filters of the convolutional layers. Then, a bottleneck layer with 3 × 3 convolutional layers containing 1024 filters is applied. In the decoder part, upsampling is performed to restore the reduced spatial dimensions, and weights are applied through the attention block. Similarly, in the decoder, after passing through the 3 × 3 CNN layers, the final output is generated through the convolutional layers and a sigmoid activation function.

3.2. Attention Gate, Bigger Receptive Field Block

This layer is integrated into the decoder component of the standard U-Net architecture. By applying the attention mechanism through an attention gate, greater focus is placed on important information via weighting operations at each encoder stage. This is applied along with upsampling to combine the feature maps of the higher levels, ensuring more accurate and detailed segmentation. Figure 4 depicts the attention gate of the proposed network.

We designed the attention gate as the core module for crack detection in low-resolution images. This module generates weights for attention by passing them through two 1 × 1 convolutional layers and a bigger receptive field block (BRFB). The f_prev layer takes the feature from the immediately preceding layer as input. g_skip and f_skip take features from the corresponding layers in the encoder as inputs through skip connections. The difference between the two layers lies in the receptive field sizes of the extracted features. To consider irregular crack features, we minimized the feature loss by bringing features from both small and large receptive fields into the encoder layers and performing summation operations. This allowed us to focus on features that required attention. Within the module, all three layers generate feature maps with coefficient channels, with each coefficient equivalent to the number of channels in the corresponding encoder layer. Details of the larger receptive field block are depicted in Figure 5.

Features introduced via the skip connection in the f_skip layer have a limited receptive field compared with those of the previous layer. Like other U-net, we adopted a method using 7 × 7, 5 × 5, and 3 × 3 layers to extract wide context information. This layer recognizes more complex patterns and extracts features from a wider receptive field. By considering the irregularities and varying sizes of cracks, the proposed architecture is capable of learning deeper features. The computed feature maps are summed, followed by the application of the ReLU activation function. The objective functions of the proposed attention gate can be expressed as follows:

$$f_{prev}\left(x\right)=r\left(W_x·x+b\right)$$

(4)

$$f_{skip}\left(y\right)=r\left(W_y·y+b\right)$$

(5)

$$g_{skip}\left(z\right)=r\left(W_z·z+b\right)$$

(6)

$$\alpha =sig\left(r\left(Wx+Wy+Wz+b_{x, y, z}\right)\ast (W_z+b_{ z})\right)$$

(7)

where x represents the features of the previous layer, y and z represent features corresponding to the encoder layers with the same values, and α denotes the attention weight computation result. The ReLU activation function ensures that only positive values remain, whereas negative values are set to 0. After performing a 1 × 1 convolution and batch normalization, the sigmoid activation function is applied to obtain the final attention weights. These weights, ranging from 0 to 1, indicate the importance of each pixel. Finally, the attention weights are multiplied by the operation results of the skip connection layer. Our experimental results demonstrate improved edge and detailed crack detection in the decoder layers. Furthermore, the proposed attention block minimizes feature loss, allowing for more accurate target localization. Overall, we successfully achieved multi-feature fusion by combining diverse receptive field features among layers, thereby enhancing crack detection performance.

3.3. Loss Function

We adopted Binary Cross-Entropy (BCE) as the loss function for training. This function selects the class that best describes the input and adjusts the parameters to values closer to the ground-truth probabilities. The formula for BCE is as follows:

$$\mathrm{B}\mathrm{C}\mathrm{E}\left(y,\widehat{y}\right)=-{\displaystyle \frac{1}{N}}\sum _{i=1}^N\left[y_i\log \left({\widehat{y}}_i\right)+(1-y_i)\log (1-{\widehat{y}}_i)\right]$$

(8)

Here, y represents the actual binary label values of the input data, and $\widehat{\mathrm{y}}$ denotes the probability values between 0 and 1, representing the model’s actual predictions. N indicates the number of samples, and the total loss is calculated by summing the calculations for the actual labels of 1 and 0. This metric penalizes situations wherein the model misclassifies all scenarios of true-positive and false-positive predictions, thereby encouraging the model to generate calibrated probabilities. Moreover, this function is differentiable, making it suitable for training neural networks using gradient-based optimization algorithms such as stochastic gradient descent. In addition, it is specialized for binary classification problems, making it appropriate for cases in which each instance of crack detection belongs to one of two classes.

4. Analysis of Experimental Result

The Crack500 [24] dataset consists of 3368 crack images divided into subsets of 1896, 348, and 1124 images for training, validation, and testing, respectively. Among these data, 352 images were duplicates; nonetheless, all 3368 images, including the duplicates, were used. This dataset was captured using a mobile phone camera on the main campus of Temple University. Each image was divided into 16 nonoverlapping patches, and only regions containing more than 1000 crack pixels were retained. The input images were resized using the nearest neighbor method, and size-based training was conducted based on the resized images. Detailed information regarding the dataset is presented in Figure 6.

The following settings were adopted throughout the experiments. Training was conducted for 100 epochs with an initial learning rate of 2 × 10⁻³. A dynamic learning rate adjustment metric was applied to improve the learning rate. The batch size for training was set to 16, and Adam [24] was used as the initialization function.

Table 1. Experimental hardware.

CPU	Memory	GPU
AMD EPYC 2713	512 GB	RTX A6000
OS	Python	Tensorflow
Ubuntu 20.04	3.8	2.5.0

provides information on the hardware used in the experiments.

The evaluation metrics were based on a confusion matrix. We adopted the F-1 score, which is the harmonic mean of precision and recall, along with the mean intersection over union (mIoU). The F-1 score considers both positive and negative model samples, thereby complementing issues that may arise when considering only precision or recall.

$$Precision:{\displaystyle \frac{TP}{(TP+FP)}}$$

(9)

$$Recall:{\displaystyle \frac{TP}{(TP+FN)}}$$

(10)

$$F-1 Score:2\times {\displaystyle \frac{Precision\times Recall}{Precision+Recall}}$$

(11)

The mIoU is a metric that represents the proportion of overlapping areas between predicted and ground-truth regions, evaluating how accurately objects are detected. The following formula is used for the IoU metric:

$${\displaystyle \frac{1}{N}}{\sum }_{i=1}^N{IoU}_i$$

(12)

The confusion matrix for the training results, shown in Figure 7, indicates that while the segmentation of areas without cracks was highly accurate, there was room for improvement in accurately segmenting the crack areas. In particular, incorrect classification was observed in fine cracks, as well as in distinguishing cracks from dark areas such as shadows.

Looking at the results in

Table 2. Training results by input size.

Input Image Size	Precision	Recall	F-1 Score	mIoU
64 × 64	0.7092	0.6832	0.6959	0.7495
128 × 128	0.7561	0.7315	0.7436	0.7752
256 × 256	0.7352	0.6926	0.7133	0.7170
512 × 512	0.7310	0.6883	0.7090	0.5962

, our proposed network demonstrated higher performance when low-resolution images from the Crack500 dataset were used as the input. The results show an approximate difference of 0.17 in terms of mIoU between the model trained with 128 × 128 input images and that trained with 512 × 512 input images. Overall, the model with low-resolution input images exhibited a higher mIoU. Figure 8 presents prediction results with respect to input image size. Here, both test images were predicted to be similar to the ground truth at 64 × 64 and 128 × 128 resolutions. However, when handling higher-resolution images, the model did not accurately represent crack boundaries and tended to misidentify non-crack areas as cracks. To obtain objective results, comparative experiments were conducted using several models. First, we compared the prediction results across FCN-8s [25], U-Net [26], and our network, with the results shown in Figure 9 and Figure 10.

The comparison of the prediction results among the three models demonstrated the proposed network’s relative strength in detailed detection. Although the network did not perfectly capture the connectivity present within fine cracks, it demonstrated a more sensitive response than the other models.

Table 3. Comparison with other studies (Crack500).

Method	Precision	Recall	F-1 Score	mIoU
U-Net By Lau [7]	0.7426	0.7285	0.7327	0.5782
U-Net By Nguyen [9]	0.6954	0.6744	0.6895	0.5261
U-Net By Yu [10]	0.6988	0.7619	0.7290	0.5736
U-Net By A Di [12]	0.8534	0.6813	0.7327	0.6248
FCN-8s [25]	0.7031	0.6172	0.6574	0.6898
U-Net [26]	0.7207	0.6401	0.6779	0.6852
Attention U-Net By [27]	-	-	0.7706	0.6311
Ours	0.7561	0.7315	0.7436	0.7752

summarizes the comparative results between our network and those proposed by Lau et al. [7], Nguyen et al. [9], Yu et al. [10], Di et al. [12], and Katsamenis et al. [27].

From these results, we can observe that the proposed method significantly outperformed the other networks in terms of mIoU. Specifically, the proposed method achieved an F-1 score of 0.7436 and mIoU of 0.7752. Notably, it exhibited higher performance when the BRFB module was used, demonstrating the effectiveness of the BRFB in conjunction with the attention gate designed to mitigate feature loss. The substantial difference in mIoU, reaching over 0.2 in some cases, confirms a considerable overlap between the predicted areas and actual regions of interest across all test images.

Table 4. Comparison using BRFB module (Crack500).

Method	Precision	Recall	F-1 Score	mIoU
Without BRFB (Attention U-Net)	0.7195	0.7033	0.7113	0.7183
With BRFB	0.7561	0.7315	0.7436	0.7752

shows the results based on whether the BRFB module is used. This result shows that the use of the BRFB module yields statistically significant results in the metrics and demonstrates its effectiveness in aiding crack detection.

Table 5. Comparison of training result by batch size (input size is 128 × 128).

Batch Size	Precision	Recall	F-1 Score	mIoU
4	0.6884	0.6746	0.6814	0.7398
8	0.7218	0.6448	0.6811	0.7404
16	0.7561	0.7315	0.7436	0.7752
32	0.7128	0.7087	0.7107	0.7585
64	0.7199	0.6472	0.6816	0.7406

shows the results of training with different batch sizes, with the best performance observed for a batch size of 16. The network incorporating the BRBF module ultimately achieved the highest mIoU, demonstrating its ability to effectively learn and segment key features even for low-resolution images.

5. Conclusions

In this study, we designed a module that enhances the attention block to minimize feature loss in the existing autoencoder-style architectures. Through the combination of a skip connection, which reduces the loss in both the encoder and decoder, and the BRFB, which learns features from diverse receptive fields, we achieved effective extraction of features from low-resolution images while minimizing loss and accurately locating features. We conducted a comparative analysis of the existing models using the Crack500 dataset. Notably, the proposed architecture achieved the highest mIoU metric compared to methods using the same dataset, demonstrating the superior performance of our approach. Furthermore, in the comparative experiments with high-resolution images, our method exhibited better performance when low-resolution images were used as input. However, the proposed method has limitations when handling very fine or extensive cracks. Although this combination reduces feature loss, it might not entirely address the loss of some crucial feature points. In the future research, the following directions should be pursued:

First, we intend to evaluate our model’s generalizability by training it on larger and more diverse datasets. We plan to collect publicly available datasets from various sources and merge them to construct a more diverse overall dataset of crack images. This will allow the model to be trained on a variety of crack patterns and environments.

Second, we intend to optimize the network architecture and training process to further enhance performance. Specifically, we aim to find ways to reduce the number of training parameters without compromising performance.

Furthermore, recent advances in semantic segmentation have led to the introduction of a series of new loss functions. It may be possible to incorporate these factors into the proposed network to further improve its performance.

This work is based on the author’s undergraduate thesis submitted to Chosun University [28].