SAM-Guided Concrete Bridge Damage Segmentation with Mamba–ResNet Hierarchical Fusion Network

Abstract

Automated damage segmentation for concrete bridges is a fundamental task in infrastructure maintenance, yet existing systems often depend heavily on large annotated datasets, which are costly and time-consuming to produce. This paper presents an innovative framework for concrete bridge damage segmentation, leveraging the Segment Anything Model (SAM) to reduce the reliance on extensive annotated data while enhancing segmentation accuracy and efficiency. Firstly, a SAM-guided mask generation network is introduced, which utilizes the SAM’s segmentation capabilities to generate supplementary supervision labels for damage segmentation. Then, a novel point-prompting strategy, incorporating saliency information, is proposed to refine SAM’s prompts, ensuring accurate mask generation for complex damage patterns. Next, a trainable semantic segmentation network is designed, integrating MambaVision and ResNet as dual backbones to capture multi-level features from concrete bridge damages. To fuse these features effectively, a Hierarchical Attention Fusion (HAF) mechanism is introduced. Finally, a Polarized Self-Attention (PSA) decoder is employed to improve segmentation precision. Experiments on a dataset of 10,000 concrete bridge images with box-level annotations achieved state-of-the-art performance, with an MIoU of 60.13%, PA of 74.02%, and MDice of 75.40%, outperforming existing segmentation models. In summary, this study improves the accuracy of concrete bridge damage segmentation through a series of innovative methods and strategies, and the concrete bridge damage segmentation algorithm opens up new horizons and directions.

1. Introduction

With the growing problem of infrastructure aging, particularly in concrete bridges, the detection and assessment of bridge damages have become crucial for ensuring traffic safety [1]. Concrete bridges are subject to various forms of deterioration such as cracks and spall, which directly affect their structural integrity and service life [2]. Therefore, timely and accurate damage detection is essential to prevent catastrophic failures and to extend the lifespan of these vital infrastructures [3,4]. Traditional methods for damage detection predominantly rely on manual inspection or hand-labeling of damaged areas [5,6]. While these approaches are still in use, they suffer from several significant limitations, including low efficiency, high labor costs, and dependence on the experience and skill of the inspectors. Moreover, manual detection is prone to human error, and the results often exhibit inconsistencies, making the process unreliable for large-scale assessments [7]. As a result, there is a pressing need for automated damage detection systems that can provide both efficiency and accuracy [8].

Deep learning techniques have shown great promise in overcoming these challenges. However, training these models requires large quantities of high-quality labeled data, which remains a significant bottleneck. The process of labeling damages is time-consuming and requires expert knowledge, especially due to the diversity of damage types and the intricacies of different concrete surfaces. This results in a scarcity of labeled data and a slow labeling process, ultimately hindering the widespread application of automated damage detection systems [9]. Consequently, improving data annotation methods and efficiency remains one of the primary challenges for advancing automated concrete damage detection technologies.

While significant progress has been made in the application of deep learning techniques, such as convolutional neural networks (CNNs) [10], for concrete damage segmentation, these methods still face several challenges, particularly concerning the dependence on large amounts of labeled data. These techniques have shown remarkable success in accurately detecting and segmenting damages, such as cracks and delamination, on concrete surfaces, making them highly valuable for automated inspection systems [11]. However, the performance of these models is still highly dependent on the availability of substantial amounts of high-quality annotated data. The major issue lies in the data annotation process itself. To train deep learning models effectively, large annotated datasets are essential. For concrete bridge damage detection, this means the need for meticulous and time-consuming labeling, often requiring expert knowledge to accurately identify and mark damages. Consequently, obtaining a sufficient volume of high-quality labeled data remains a significant obstacle to the widespread application of these techniques [12]. Even when a considerable amount of annotated data is available, the process of labeling is still slow and inefficient, especially for large-scale bridge inspection tasks. The complexity and time required for accurate labeling significantly hinder the scaling of data collection efforts, which in turn limits the expansion of datasets and the continuous optimization of deep learning models. This persistent bottleneck in annotation efficiency continues to impede the progress of automated concrete damage detection systems, highlighting the urgent need for alternative methods that can improve both data annotation efficiency and quality [13].

In order to address the challenges of limited labeled data and low annotation efficiency, the Segment Anything Model (SAM) [14] has emerged as a powerful image segmentation technique. The SAM leverages self-supervised learning and pre-trained models to perform efficient and accurate damage segmentation with minimal labeled data, providing a promising solution to the data bottleneck in automated concrete damage detection. The SAM algorithm operates by utilizing a guided segmentation approach that significantly reduces the dependency on large amounts of manually annotated data [15]. Unlike traditional deep learning methods that require substantial datasets for training, the SAM can generate high-quality segmentation results with only a small amount of annotated data. This approach dramatically cuts down the cost and time required for data annotation, making it a more scalable option for real-world applications [16].

In this paper, we propose a novel training framework for concrete bridge damage segmentation that leverages the Segment Anything Model (SAM) to reduce the reliance on extensive annotated data while improving segmentation efficiency and accuracy. Specifically, we introduce a SAM-guided mask generation network, which harnesses the segmentation mask generation capability of SAM. These masks are subsequently utilized as supplementary supervision labels to train a damage segmentation network. To ensure that SAM generates reliable and precise masks, we propose an Optimizing Prompting Strategy with saliency information. This strategy enhances the quality of the prompts provided to the SAM, thereby improving the accuracy of the generated masks. Following this, we present a trainable semantic segmentation network that employs a multi-level feature extraction architecture. This architecture utilizes distinct backbone networks to extract semantic features of varying levels from concrete bridge damages. To effectively fuse the features obtained at different levels, we propose a Hierarchical Attention Fusion (HAF) mechanism. HAF enables the integration of multi-level features to enhance the overall segmentation performance. Additionally, we utilize a Polarized Self-Attention (PSA) decoder as the feature decoder, which predicts precise segmentation results for concrete bridge damages. The contributions of this study can be summarized as follows:

1.

We propose a SAM-guided mask generation network that leverages the segmentation mask generation capabilities of SAM to provide supplementary supervision labels for training concrete bridge damage segmentation networks. This innovation significantly reduces reliance on large annotated datasets, addressing a critical limitation in existing automated damage detection systems.

2.

A novel point-prompting strategy is introduced, which employs saliency information to refine the prompts provided to SAM. This ensures the generation of reliable and accurate segmentation masks, even for complex damage patterns, thereby enhancing the overall segmentation quality.

3.

We design a trainable semantic segmentation network featuring a multi-level feature extraction framework. By integrating MambaVision and ResNet as dual backbone networks, the system captures hierarchical semantic features from concrete bridge damages, simultaneously extracting high-level global semantic features and robust local feature representations.

4.

To effectively integrate multi-level features and enhance segmentation performance, we introduce a Hierarchical Attention Fusion (HAF) mechanism. This includes a Boosting Textures Module (BTM) designed to strengthen the representation of discriminative texture features. Combined with the Polarized Self-Attention (PSA) decoder, our approach achieves high-precision damage segmentation results, demonstrating its robustness and applicability in practical engineering scenarios.

The rest of the paper is structured as follows: Section 2 presents related work focusing on semantic segmentation algorithms with applications to concrete bridge damage segmentation and the application of the Segment Anything Model. Section 3 describes the SAM-guided concrete damage segmentation mask generation and the Mamba–ResNet hierarchical fusion network. Section 4 details the experimental procedure and the experimental results of the proposed method. Section 5 provides a discussion of our study. Finally, Section 6 summarizes the conclusions of this paper.

2. Related Works

2.1. Semantic Segmentation Algorithm with Application to Concrete Bridge Damage Segmentation

Concrete bridge surface damage recognition typically relies on classification and object detection methods. These approaches classify regions in images into predefined categories or detect specific damages within candidate boxes. Such methods have been widely applied in the rapid detection and localization of concrete structures. In contrast, semantic segmentation plays a crucial role in concrete bridge surface damage recognition because it allows for pixel-level classification of the damages. This fine-grained approach is vital for obtaining quantitative parameters (such as the exact area of damage on the bridge surface), which classification or object detection methods cannot directly provide [17,18].

While concrete bridge surface damage classification methods classify image patches, and object detection methods focus on classifying and regressing candidate boxes of various sizes within the image, pixel-level semantic segmentation requires classifying every pixel in the image. Rubio et al. [19] proposed a pixel-level recognition method for concrete bridge stratification and reinforcement based on Fully Convolutional Networks [20], using a pre-trained VGG network as the feature extractor. This method achieved average accuracies of 89.7% for stratification and 78.4% for exposed reinforcement. Shi et al. [21] introduced a VGG-Unet-based semantic segmentation method for bridge reinforcement corrosion and rubber bearing cracks, where the Background Data Drop Rate (BDDR) was used to reduce the number of background pixels and control the proportion of damaged pixels in the dataset. Deng et al. [22] incorporated the atrous spatial pyramid pooling (ASPP) [23] module into LinkNet and used pre-trained ResNet34 as the feature extractor. The model also employed a weighted–balanced joint intersection over union (IoU) loss function to achieve precise segmentation on highly imbalanced small datasets. Their experimental results showed that the method achieved a mean IoU (mIoU) of 61.95%, outperforming other models. Narazaki et al. [24] developed a visual-based automatic structural state assessment system that can identify and locate concrete bridge damages. To address the issue of limited image data, they used a synthetic environment to randomly generate target structures and damage scenarios. The model achieved an IoU of 87.9% on synthetic images and 73.9% on real images. Zou et al. [25] adopted the idea of Seg-Net [26] and chose a labeled pooling layer during the down-sampling stage to avoid information loss during up-sampling. Yang et al. [27] combined the feature pyramid network (FPN) [28] architecture with semantic segmentation networks to detect multi-scale cracks, achieving good results on public datasets. Choi and Cha [29] proposed a novel deep learning structure called the Semantic Damaged Dam Age Detection Network (SDDNet) for real-time segmentation of structural surface cracks, and created a new crack dataset called Crack200. [30] separated the crack regions from point cloud images captured by a depth camera and directly quantified the crack volume. Zhang et al. [31] modified the Mask R-CNN [32] network to address issues with the mask branch’s inability to accurately predict crack details. Additionally, Li et al. [33] proposed a pixel-based adaptive weighted cross-entropy (WCE) loss function combined with Jaccard distance to analyze crack images, which facilitated high-quality pixel-level road crack detection. However, convolutional neural networks (CNNs) lack a global understanding of images and cannot establish dependencies between features [34]. Vaswani et al. [35] introduced the Vision Transformer (ViT), applying self-attention to image segmentation tasks for the first time. Liu et al. [36] proposed the Swin Transformer, based on sliding windows and hierarchical structures, achieving better results in semantic segmentation tasks than most CNNs. As Transformer-based networks have gained widespread application, more and more researchers are utilizing Transformers for damage segmentation tasks, particularly for concrete crack segmentation. Xu et al. [37] combined Transformers with skip connection strategies for high-resolution road crack recognition. Shamsabadi et al. [38] proposed a method based on ViT, while Chen et al. [39] pointed out that compared to CNNs, Transformer-based image recognition networks suffer from data dependency and the loss of local features due to the lack of inductive biases, such as positional encoding, translation invariance, and hierarchical structure.

2.2. Application of Segment Anything Model

The Segment Anything Model (SAM), introduced by Meta AI, is a versatile segmentation framework designed for universal application. Leveraging prompt-based learning and pre-trained on over a billion masks, SAM achieves high generalization across tasks without task-specific fine-tuning [14]. Its Vision Transformer backbone ensures robust performance, making it a cornerstone for interactive and zero-shot segmentation.

Zhou et al. [40] leveraged the Segment Anything Model (SAM), known for its rich prior knowledge and strong generalization ability, by integrating a lightweight and learnable crack-adaptive layer with a sparse prompt generation method based on high-frequency components of road images. This framework enabled efficient road crack segmentation. However, in scenarios with subtle crack patterns, both this framework and the Mask2Former model struggled to achieve accurate segmentation. Additionally, the inference speed of the framework was slightly slower compared to the CNN-based PIDNet model. Wang et al. [41] proposed a SAM-based Image Enhancement (SAM-IE) approach, which utilizes the masks and stability scores generated by SAM to enhance the diagnostic performance of medical image classification models. This method does not require extensive modifications or excessive prompts for SAM, making it a practical enhancement strategy. Further, Zhou et al. [42] introduced a novel SAM-based framework named AoP-SAM, specifically designed to improve segmentation accuracy of the pubic symphysis (PS) and fetal head (FH) in intrapartum ultrasound images, demonstrating SAM’s adaptability to specialized medical imaging applications.

Building upon the findings in the related works section, the following key insights can be drawn:

• Focus on Single-Class Segmentation: Research on the semantic segmentation of concrete bridge damages leveraging deep learning has predominantly concentrated on single-class segmentation. While effective in isolating specific damage types, this approach often overlooks the complexity and diversity of real-world scenarios where multiple damage types co-exist. As a result, the exploration of multi-class segmentation remains underdeveloped, leaving a critical gap in addressing practical applications.
• Challenges in Multi-Class Segmentation: Multi-class segmentation faces considerable hurdles, with two primary challenges being the scarcity of annotated datasets and the inherent difficulty of the annotation process. The annotation of multi-class datasets demands significant expertise, time, and resources, particularly in scenarios involving subtle or overlapping damage features. These factors have constrained the advancement of robust multi-class segmentation techniques tailored for bridge damage analysis.
• Limitations of SAM in Bridge Damage Segmentation: While the Segment Anything Model (SAM) represents a promising tool for mask generation in general segmentation tasks, its application to bridge damage imagery is not without limitations. The generic nature of SAM’s mask generation requires adaptation to account for the unique visual characteristics of bridge damages, such as irregular shapes, complex textures, and diverse environmental conditions. Therefore, further refinements are necessary to improve the effectiveness of SAM in this specialized domain.

3. Method

3.1. Overall Architecture

The overall framework of our proposed structure is shown in Figure 1. We propose a SAM-guided mask generation framework for damage segmentation in concrete bridges. Let the input image be $X\in {\mathbb{R}}^{H\times W\times C}$, where H, W, and C represent the height, width, and channels of the image. The initial bounding box annotation is defined as $Y_B\in {\mathbb{R}}^4$, where the four values represent the coordinates of the bounding box. We propose a Prompt Optimization Generation module ${\mathcal{T}}_{POG}$ that refines the initial bounding box $Y_B$ into an optimized prompt $Y_B^{\prime }\in {\mathbb{R}}^4$. The optimized prompt $Y_B^{\prime }$ is then used as input to the SAM model ${\mathcal{F}}_{SAM}$, which generates a segmentation mask $Y_M\in {\mathbb{R}}^{H\times W}$ as the output:

$$\begin{array}{c}\hfill Y_M={\mathcal{F}}_{SAM}(X,Y_B^{\prime })\end{array}$$

(1)

where $Y_B^{\prime }={\mathcal{T}}_{POG}\left(Y_B\right)$. Here, ${\mathcal{F}}_{SAM}$ denotes the segmentation model, which is composed of a frozen encoder–decoder architecture. Next, we employ a trainable encoder–decoder network ${\mathcal{F}}_{ED}$ to process the segmentation mask $Y_M$. The network ${\mathcal{F}}_{ED}$, consisting of encoder ${\mathcal{E}}_{ED}$ and decoder ${\mathcal{D}}_{ED}$, produces the predicted segmentation mask $Y_P\in {\mathbb{R}}^{H\times W}$ as follows:

$$\begin{array}{c}\hfill Y_P={\mathcal{D}}_{ED}\left({\mathcal{E}}_{ED}\left(X,Y_M;{\theta }_{ED}\right)\right)\end{array}$$

(2)

where ${\theta }_{ED}$ represents the parameters of the encoder–decoder network ${\mathcal{F}}_{ED}$, which are learned during training. The objective of the training process is to minimize the segmentation loss ${\mathcal{L}}_{seg}$, which measures the difference between the predicted segmentation $Y_P$ and the ground-truth segmentation mask $Y_M$. Specifically, we aim to minimize the following loss:

$$\begin{array}{c}\hfill {\mathcal{L}}_{seg}=\mathcal{L}\left(Y_P,Y_M\right)\end{array}$$

(3)

where $\mathcal{L}$ is a suitable loss function. During training, the parameters ${\theta }_{ED}$ of the encoder–decoder network are updated, while the SAM model’s encoder–decoder components are kept frozen. After training, the learned encoder–decoder network ${\mathcal{F}}_{ED}$ is used for inference on new, unseen images X, generating segmentation predictions $Y_P$.

3.2. SAM-Guided Concrete Damage Segmentation Mask Generation

3.2.1. Segment Anything Model

As shown in Figure 2, the Segment Anything Model (SAM) [14] segmentation framework represents a state-of-the-art approach to generalizable object segmentation. Its architecture is structured around three key components: a Transformer-based image encoder, a mask decoder, and a prompt embedding module.

1.

Transformer Image Encoder: The image encoder in SAM leverages a Vision Transformer (ViT) [43] architecture to encode the input image into a high-dimensional latent representation. Formally, let $I\in {\mathbb{R}}^{H\times W\times C}$ denote the input image, where H,W,C represent height, width, and the number of channels, respectively. The image is partitioned into patches, and each patch is projected into a token embedding $X\in {\mathbb{R}}^{N\times d}$, where $N=HW/P^2$ is the number of patches and d is the embedding dimension. These tokens are then processed through a series of transformer layers, each comprising multi-head self-attention and feed-forward networks. The resulting feature map $F\in {\mathbb{R}}^{N\times d}$ encodes the spatial and contextual information critical for segmentation tasks.

2.

Mask Decoder: The mask decoder refines the encoded image features to produce segmentation masks. Using the encoded feature map F as input, along with a set of learnable query embeddings $Q\in {\mathbb{R}}^{M\times d}$ (where M is the number of masks), the decoder computes attention weights to localize the object of interest. The mask prediction is then given by $M=\sigma \left(Q{F}^T\right)$, where $\sigma$ is the sigmoid activation applied element-wise to generate pixel-wise probabilities. This formulation allows the SAM to generate multiple masks simultaneously while maintaining scalability.

3.

Prompt Embedding: The prompt embedding module allows the SAM to adapt to various inputs, including points, bounding boxes, and textual descriptions. For a point-based prompt, the embedding is represented as a learnable vector $p\in {\mathbb{R}}^d$, added to the corresponding spatial location in the encoded feature map. For bounding boxes, the embeddings are parameterized as the positional encodings of the box’s corners, integrated into the transformer layers. This versatility enables SAM to function effectively across diverse user-provided guidance signals.

3.2.2. Optimizing Prompting Strategy with Saliency Information

In the field of image segmentation, the SAM algorithm has significantly reduced the need for large-scale manually annotated data by leveraging prompting techniques. Unlike traditional deep learning models that require vast amounts of labeled data for training, SAM allows for high-quality segmentation with minimal input through various prompting strategies, such as bounding boxes or points. This ability to generate accurate segmentation results with minimal user intervention not only reduces the time and cost associated with data labeling but also presents a more scalable solution for real-world applications. When only bounding boxes are used as prompts, the SAM struggles to accurately segment narrow or elongated objects. Cracks, as shown in Figure 3a, are typically thin and irregular, and a bounding box prompt often leads to imprecise segmentation that includes parts of the background. This issue arises because a bounding box provides a coarse estimate of the object’s position, which is insufficient to guide the model in refining the segmentation of such intricate structures. While bounding box prompts serve as an initial guide, they are not always detailed enough to achieve the fine-grained segmentation required for challenging cases like cracks, where precision is crucial [44,45].

The SAM primarily relies on the Transformer architecture and large-scale pre-training data to achieve generalized segmentation capabilities for arbitrary objects. Its segmentation results are highly dependent on the provided prompts, with common prompt types including point prompts and box prompts. The combination of saliency maps generated from point and box prompts can leverage their complementary advantages. Box prompts help the SAM constrain the approximate target region, reducing background interference, while point prompts provide fine-grained local information, guiding the model to segment crack regions more accurately. This approach is particularly effective for elongated targets, as cracks exhibit weak local features, making box prompts alone prone to misclassification. In contrast, point prompts can provide additional information at critical locations along the crack, facilitating the learning of its complete structure by the SAM. Moreover, existing studies have demonstrated that integrating multiple prompts can significantly improve the SAM’s segmentation performance in complex scenarios. For instance, in medical image segmentation tasks, researchers have found that using both point and box prompts enhances model stability, thereby improving accuracy [46]. Similarly, in concrete bridge damage detection, this combined strategy strengthens the SAM’s ability to adapt to intricate damage patterns such as cracks, ultimately improving the quality of segmentation results.

As shown in Figure 3b, to enhance segmentation accuracy, we propose the incorporation of point prompts in addition to bounding box cues. In order to accurately obtain prompt points, we propose a strategy for optimizing prompts using saliency information. This process involves four key steps: (1) computing the saliency map, (2) thresholding, (3) connected component analysis, and (4) computing the centroid of the region. By refining the segmentation prompts with these saliency-driven steps, we aim to improve both the precision and accuracy of the segmentation masks generated by the SAM. The steps are shown below:

(1) Compute the Saliency Map: The first step involves computing the saliency map for the input image, which highlights regions of interest based on their visual importance. The spatial contrast-based saliency map S can be formulated mathematically as follows:

$$\begin{array}{c}\hfill S(x,y)=\sum _{i,j\in N(x,y)}|I(x,y)-I(x+i,y+j)|\end{array}$$

(4)

where $I\left(x,y\right)$ is the intensity of the pixel at position $\left(x,y\right)$ in the image, and $N\left(x,y\right)$ is a local neighborhood of size $k\times k$ centered at the pixel $(x,y)$. The summation is taken over all neighboring pixels $(x+i,y+j$) in the local neighborhood. This measure computes the contrast between a pixel and its neighbors by evaluating the absolute differences in pixel intensities.

(2) Thresholding: After normalizing the saliency map $\mathrm{S}$ to produce $S_{norm}$ with values in the range $\left[0,1\right]$, we proceed to identify the most salient regions by applying a threshold. To achieve this, we first calculate the maximum value of the normalized saliency map, $S_{max}$ which is given by

$$\begin{array}{c}\hfill S_{max}=max\left(S_{norm}\right)\end{array}$$

(5)

This value represents the highest saliency level in the image. We define the threshold T as half of the maximum saliency value:

$$\begin{array}{c}\hfill T=0.5\times S_{max}\end{array}$$

(6)

Using this threshold, we create a binary map B by comparing each pixel in the normalized saliency map $S_{norm}(x,y)$ to T. Specifically, for each pixel $(x,y)$, if the saliency value $S_{norm}(x,y)$ is greater than or equal to T, it is considered a salient pixel and assigned a value of 255 in the binary map:

$$\begin{array}{c}\hfill B(x,y)=\left\{\begin{array}{cc}255\hfill & S_{\mathrm{norm}}(x,y)\ge T\hfill \\ 0\hfill & S_{\mathrm{norm}}(x,y)<T\hfill \end{array}\right.\end{array}$$

(7)

(3) Connected Component Analysis: After thresholding, connected component analysis is performed to identify distinct, contiguous regions in the image. This step is crucial as it allows the model to isolate individual objects or regions of interest that have been highlighted by the saliency map. To identify regions of interest in the binary map B, the first step is to find all connected components. A connected component is a set of pixels in B that are connected to each other and share the same binary value, in this case, 255. These connected components correspond to distinct regions of interest in the binary map. Once the connected components are identified, we proceed to compute the area of each component. For a given connected component ${\mathcal{R}}_i$, the area $A_i$ is defined as the total number of pixels within that component. Mathematically, this is expressed as

$$\begin{array}{c}\hfill A_i=\sum _{(x,y)\in {\mathcal{R}}_i}1\end{array}$$

(8)

where the summation iterates over all pixels $(x,y)$ that belong to the region ${\mathcal{R}}_i$, and the result gives the total count of pixels.

(4) Compute Centroid of the Region: The final step is to compute the centroid of each connected region, which provides a precise point location that represents the “center” of the object or region of interest. This centroid can be used as a point-based prompt to guide the segmentation process, improving the accuracy of the segmentation masks. By using the centroid as a reference, the model can more effectively refine the segmentation boundaries and focus its attention on the most salient areas [4]. For each valid region ${\mathcal{R}}_i$, we compute its centroid $c_i=(x_c,y_c)$, which represents the center of mass of the region. The centroid coordinates are calculated as the weighted average of the pixel positions within the region, given by

$$\begin{array}{c}\hfill c_i=(x_c,y_c)={\displaystyle \frac{1}{A_i}}\sum _{(x,y)\in {\mathcal{R}}_i}\left[\begin{array}{c}x\\ y\end{array}\right]\end{array}$$

(9)

where $A_i$ is the area of the region ${\mathcal{R}}_i$, and the summation iterates over all pixels $(x_c,y_c)$ in the region. This compact form allows simultaneous computation of both the x and y coordinates of the centroid.

3.3. Mamba–ResNet Hierarchical Fusion Network for Concrete Damage Segmentation

The overall architecture of our proposed method is shown in Figure 4. In the task of concrete bridge damage segmentation, traditional single-encoder architectures often struggle to effectively balance the capture of global semantic information and local structural details. To overcome this limitation, we propose the Mamba–ResNet hierarchical fusion network, which utilizes a dual-branch design with MambaVision [47] as the main backbone and ResNet [48] as the auxiliary backbone. This dual-backbone architecture ensures a complementary feature extraction process, enhancing the model’s ability to handle complex structural variations in damaged bridges. As shown in Figure 4, MambaVision, as the main backbone, leverages its self-attention mechanism to excel in capturing long-range dependencies and extracting high-level semantic features. This is particularly useful for understanding the broader context in bridge damage, where large spatial relationships are essential for distinguishing various types of damage from the surrounding background. ResNet, acting as the auxiliary backbone, provides robust local feature representations through its hierarchical convolutional structure. This allows it to focus on capturing fine-grained details, such as the precise boundaries of cracks and surface degradation, which are crucial for accurate damage segmentation. The fusion of these two backbones allows the model to simultaneously benefit from both global context understanding and local feature precision, ensuring comprehensive damage segmentation. Bridge damage typically appears in different scales, from small cracks to large-area spalling or corrosion. A single-backbone network often struggles to capture both fine-scale details and large-scale structural patterns simultaneously. By using MambaVision as the main backbone, the network can establish a global understanding of damage distribution, while ResNet as the auxiliary backbone refines the segmentation results by enhancing local details at various scales. This hierarchical fusion ensures that the model can adapt to both local irregularities and global structural changes, improving segmentation robustness in real-world applications.

To integrate the multi-level features extracted by MambaVision and ResNet, we introduce a Hierarchical Attention Fusion (HAF) module. To improve the segmentation of concrete bridge damages, we integrated the Polarized Self-Attention (PSA) mechanism into the decoder. PSA enhances the model’s ability to distinguish texture features by effectively capturing both global patterns and local details. This is particularly important for damage segmentation, where subtle texture variations often define the boundary between damages and background, and traditional methods struggle with such fine-grained distinctions [49,50].

3.3.1. MambaVision Backbone

Recent developments in deep learning have revealed fundamental differences between CNN, Transformers, and state-space models like Mamba in modeling long-range dependencies. CNNs, known for their hierarchical feature extraction, primarily focus on local spatial patterns. Their reliance on stacked convolutional layers limits their ability to capture global context efficiently, as deeper layers are required to expand the receptive field. While dilated convolutions and non-local blocks have been introduced to alleviate this limitation, they still struggle with fully modeling distant dependencies [51]. Transformers effectively capture global dependencies through self-attention mechanisms, allowing each token to attend to all others in the input. This enables robust contextual modeling across an entire image. However, the computational complexity of self-attention scales quadratically with the number of tokens, posing challenges for high-resolution dense prediction tasks such as damage segmentation [43]. Various adaptations, such as windowed attention and sparse attention, attempt to address this, but they often compromise on capturing full global dependencies [52]. Mamba, a state-space sequence model (SSM), presents an alternative approach by encoding long-range interactions through structured state-space modeling [53]. Unlike CNNs, which require multiple layers to aggregate global information, Mamba inherently maintains long-range dependencies within a single operation. Unlike Transformers, which rely on explicit pairwise attention, Mamba propagates information sequentially in a manner that allows for efficient modeling of distant relationships without quadratic complexity. This structured approach ensures that long-range dependencies are captured effectively, making it particularly suited for segmentation tasks that require preserving global structural integrity.

In order to address the issues of insufficient feature extraction and inadequate exploration of deep contextual information in concrete bridge damage segmentation, we chose MambaVision as the backbone network for feature extraction. MambaVision has demonstrated significant advantages in capturing fine-grained features and preserving long-range dependencies, which are essential for accurately identifying subtle damage patterns in concrete structures. Recent studies have shown that deep contextual information plays a crucial role in improving segmentation performance by providing a more comprehensive understanding of the spatial relationships between different regions of the bridge surface [54]. Therefore, utilizing MambaVision allows for enhanced feature extraction and better integration of spatial context, ultimately improving the accuracy and reliability of damage detection in concrete bridges. As shown in Figure 5, the MambaVision architecture is a hierarchical network that consists of four distinct stages. The first two stages are primarily dedicated to feature extraction using CNN-based blocks, while the third and fourth stages focus on more complex feature extraction using the Mamba-based model.

The input image, sized $H\times W\times 3$, is first divided into overlapping patches of size ${\displaystyle \frac{H}{4}}\times {\displaystyle \frac{W}{4}}\times C$, and these patches are then projected into a C-dimensional embedding space. The stem module, consisting of two sequential $3\times 3$ convolutional layers with stride 2, performs the initial feature extraction. This step is essential for reducing the image resolution while preserving essential information, enabling a more efficient analysis. The structure then progresses into stages 1 and stages 2, which rely on CNN blocks for rapid extraction of localized features. These stages use a down-sampling mechanism via batch-normalized convolutional layers with stride 2, which progressively reduces the image resolution. The CNN block formulation is represented by the following equation:

$$\begin{array}{c}\hfill \widehat{z}=GELU\left(BN\left({\mathrm{Conv}}_{3\times 3}\left(z\right)\right)\right)\end{array}$$

(10)

$$\begin{array}{c}\hfill z=BN\left({\mathrm{Conv}}_{3\times 3}\left(\widehat{z}\right)\right)+z\end{array}$$

(11)

where GELU refers to the Gaussian error linear unit activation function, and BN is batch normalization [55].

As the architecture progresses into stages 3 and 4, it integrates the MambaVision Mixer and Transformer blocks, which enhance the model’s ability to capture both spatial and sequential dependencies. The design ensures that each of these components works in concert, balancing local feature extraction with broader context modeling.

A distinctive feature of the Mamba architecture is its use of the Structured State Space Model (SSM), which plays a central role in overcoming the limitations of traditional convolutional and Transformer-based networks. The SSM is designed to model long-range dependencies in visual data, allowing the network to process sequential input in an efficient and context-aware manner. This is particularly important for visual tasks where global context and distant spatial relationships need to be captured. Specifically, the input $x\left(t\right)$ is processed through a series of hidden states $h\left(t\right)\in {\mathbb{R}}^M$, with the following formulation:

$$\begin{array}{c}\hfill h^{\prime }\left(t\right)=Ah\left(t\right)+Bx\left(t\right)\end{array}$$

(12)

$$\begin{array}{c}\hfill y\left(t\right)=Ch\left(t\right)\end{array}$$

(13)

where A, B, and C are learnable parameters. To improve computational efficiency, these continuous parameters are discretized as follows:

$$\begin{array}{c}\hfill \overline{A}=exp(\Delta A)\end{array}$$

(14)

$$\begin{array}{c}\hfill \overline{B}={(\Delta A)}^{-1}(exp(\Delta A)-I)(\Delta B)\end{array}$$

(15)

$$\begin{array}{c}\hfill \overline{C}=C\end{array}$$

(16)

This discretization enables the model to perform efficient computations in sequential tasks while preserving the ability to model long-range dependencies. The output can then be computed using a convolutional kernel K, which aggregates the processed input $x\left(t\right)$:

$$\begin{array}{c}\hfill y=x\ast K\end{array}$$

(17)

$$\begin{array}{c}\hfill K=[CB,CAB,\dots ,C{A}^{T-1}B]\end{array}$$

(18)

This structure allows the network to integrate both local and global features efficiently, significantly enhancing its representational capacity.

As shown in Figure 6, the MambaVision block introduces a dual-branch design, combining the strengths of selective scan operations (Scan) and regular convolutional paths.

The input $X_{in}$ undergoes two parallel processing streams. In the first branch, $X_1$ is computed using a selective scan operation applied to features processed by a linear transformation and convolution, mathematically described as:

$$\begin{array}{c}\hfill X_1=Scan\left(\sigma \left(Conv\left(Linear(C,{\displaystyle \frac{C}{2}})\left(X_{in}\right)\right)\right)\right)\end{array}$$

(19)

where $\sigma$ represents the activation function (e.g., Sigmoid Linear Unit, SiLU), and $Linear(C,{\displaystyle \frac{C}{2}})$ denotes a linear transformation reducing the input dimensions. The second branch computes $X_2$ by applying a similar convolution and activation process without the Scan operation:

$$\begin{array}{c}\hfill X_2=\sigma \left(Conv\left(Linear(C,{\displaystyle \frac{C}{2}})\left(X_{in}\right)\right)\right)\end{array}$$

(20)

The outputs from both branches are concatenated and projected back into the embedding space C using a final linear transformation:

$$\begin{array}{c}\hfill X_{out}=Linear\left({\displaystyle \frac{C}{2}},C\right)\left(Concat(X_1,X_2)\right)\end{array}$$

(21)

This design ensures that both local features (captured by the convolutional path) and global dependencies (modeled by the Scan operation) are effectively integrated into the final feature representation. By incorporating a symmetric path without SSM in the second branch, the MambaVision block mitigates any loss of critical local information, striking a balance between sequential and spatial modeling.

In the final stages of MambaVision, Transformer blocks are integrated to recover lost global context and capture long-range spatial dependencies. These blocks employ a multi-head self-attention mechanism, which is essential for modeling the complex relationships between distant parts of the image. The attention mechanism is formulated as

$$\begin{array}{c}\hfill Attention\left(Q,K,V\right)=Softmax\left({\displaystyle \frac{Q{K}^T}{\sqrt{d_h}}}\right)V\end{array}$$

(22)

where Q, K, and V are the query, key, and value matrices, respectively, and $d_h$ is the dimension of the attention heads. This mechanism enables the model to focus on relevant parts of the image, even if they are far apart spatially, allowing it to effectively capture dependencies across large regions of the image.

3.3.2. Hierarchical Attention Fusion

Integrating multi-level features from different network layers is a widely used approach to improve feature representation in vision tasks [56]. Two common methods for such integration are feature concatenation and element-wise addition. Feature concatenation combines features by stacking them along the channel dimension, allowing the model to preserve all information from both levels. On the other hand, element-wise addition fuses features by summing corresponding elements, effectively reducing redundancy and maintaining compactness. These methods, while effective in many applications, struggle to capture the intricate texture details required for segmenting damages in concrete bridges.

In the context of concrete bridge damage segmentation, these general methods often fail to effectively highlight the subtle texture variations that distinguish damages from background noise. This limitation arises due to the lack of an adaptive mechanism to emphasize critical texture features and suppress irrelevant information. To address this challenge, we propose the Hierarchical Attention Fusion (HAF) framework. As shown in Figure 7, the proposed HAF framework employs an attention mechanism to integrate features from different levels of MambaVision and ResNet. By utilizing attention, the HAF adaptively assigns importance weights to features, enhancing their discriminative power for damage segmentation.

For a given pair of input features X and Y with dimensions $C\times H\times W$, HAF operates through parallel pathways. Each pathway first processes its input using the BTM, refining the texture details and emphasizing the regions critical for damage identification. Mathematically, the attention-weighted feature maps $F_X$ and $F_Y$ can be represented as

$$\begin{array}{c}\hfill F_X={\mathrm{Conv}}_{1\times 1}\left(ReLU\left(BTM\left(X\right)\right)\right)\end{array}$$

(23)

$$\begin{array}{c}\hfill F_Y={\mathrm{Conv}}_{1\times 1}\left(ReLU\left(BTM\left(Y\right)\right)\right)\end{array}$$

(24)

where $BTM$ represents the Boosting Textures Module, ${\mathrm{Conv}}_{1\times 1}$ is a $1\times 1$ convolution layer, an $\sigma$ denotes the sigmoid activation function. The expression for the output of HAF is as follows:

$$\begin{array}{c}\hfill F_{HAF}=X·\sigma (F_X+F_Y)+Y·\sigma (F_X+F_Y)\end{array}$$

(25)

where · denotes element-by-element multiplication, and $\sigma$ denotes the activation function.

The BTM, a core component of HAF, is specifically designed to enhance the representation of discriminative texture features. As detailed in Figure 7, the BTM captures multi-scale texture patterns through a combination of $3\times 3$ convolutions and dilated convolutions with dilation rates $d=3,5,7$. These multi-scale outputs are concatenated along the channel dimension:

$$\begin{array}{c}\hfill F_{BTM}=Concat({\mathrm{Conv}}_{3\times 3}\left(F\right),{\mathrm{Conv}}_{3\times 3}^{d=3}\left(F\right),{\mathrm{Conv}}_{3\times 3}^{d=5}\left(F\right),{\mathrm{Conv}}_{3\times 3}^{d=7}\left(F\right))\end{array}$$

(26)

The resulting feature map is then refined through a $1\times 1$ convolution to reduce dimensionality:

$$\begin{array}{c}\hfill F={\mathrm{Conv}}_{1\times 1}\left(F_{BTM}\right)\end{array}$$

(27)

Neuroscience studies have revealed that the human visual system employs a series of receptive fields with varying sizes to process visual information effectively. Specifically, it has been demonstrated that these receptive fields are organized to emphasize the area near the fovea of the retina, which is particularly sensitive to subtle spatial displacements [57]. In the context of damage segmentation in concrete bridges, this design principle allows the network to simulate the multi-scale processing mechanism of the human visual system. By employing convolutions with dilation rates $d=3,5,7$, the network can simultaneously capture fine-grained details, such as micro-cracks, and broader contextual patterns, such as spalling or extensive cracking.

3.3.3. Polarized Self-Attention

In segmentation networks, the decoder plays a pivotal role in recovering high-resolution spatial details from low-resolution feature maps generated by the encoder. Traditional decoders often rely on up-sampling operations such as transposed convolutions, bilinear interpolation, or nearest-neighbor interpolation to reconstruct spatial details [58]. While effective in generating higher-resolution outputs, these methods primarily focus on local context and often fail to model global dependencies, which are critical for understanding complex spatial relationships. To address this, researchers have proposed feature fusion techniques that aggregate information across different levels of the network. For instance, skip connections, as seen in U-Net, combine high-resolution features from earlier encoder layers with the up-sampled outputs of the decoder, thereby enriching the spatial detail [59]. Pyramid-based approaches, such as the Pyramid Scene Parsing Network (PSPNet), use multi-scale pooling to capture context at various resolutions [60]. Similarly, atrous spatial pyramid pooling (ASPP) effectively integrates features at multiple scales, improving the segmentation of objects with varying sizes [61]. Despite their effectiveness, these methods are limited in their ability to dynamically adapt to diverse inputs, often treating all features equally without emphasizing the most salient ones.

Attention mechanisms offer a solution by enabling selective focus on critical regions while suppressing irrelevant information. This capability is particularly valuable for tasks like concrete bridge damage segmentation, where anomalies such as cracks, spalling, and other forms of damage vary significantly in appearance and scale. By incorporating Polarized Self-Attention into the decoder, the network gains the ability to model both local and global dependencies, enhancing precision and robustness in feature reconstruction.

As illustrated in Figure 8, PSA consists of two key components: channel-only self-attention and spatial-only self-attention. In the channel dimension, PSA retains half of the original features’ dimensions, while in the spatial dimension, it maintains the full dimensions of the original features. This design reduces the information loss caused by dimensionality reduction.

In the channel attention component, we first apply a 1 × 1 convolution to the input feature $X\in {\mathbb{R}}^{{\displaystyle \frac{C}{2}}\times H\times W}$, transforming it into two feature maps, $Q\in {\mathbb{R}}^{{\displaystyle \frac{C}{2}}\times H\times W}$ and $V\in {\mathbb{R}}^{C\times H\times W}$. Here, Q’s channel dimension is compressed to half, whereas V’s channel dimension is preserved at a higher level. To strengthen the information content of Q, the Softmax function is applied across the channel dimension:

$$\begin{array}{c}\hfill Q_c^{\prime }=Softmax\left(Q_c\right),\forall c\in \left[1,{\displaystyle \frac{C}{2}}\right]\end{array}$$

(28)

Next, a matrix multiplication is performed between Q and K to aggregate channel attention:

$$\begin{array}{c}\hfill A=Q^{\prime }·K^T\end{array}$$

(29)

where K is another transformation of the input features via $1\times 1$ convolution. Subsequently, we use a $1\times 1$ convolution and normalization layer to project the aggregated features back to C-dimensional space:

$$\begin{array}{c}\hfill F_{channel}=Norm\left(Conv1x1(A·V)\right)\end{array}$$

(30)

Finally, a sigmoid function is applied to restrict all parameters within the range [0, 1]:

$$\begin{array}{c}\hfill F_{channel}^{out}=Sigmoid\left(F_{channel}\right)\end{array}$$

(31)

In the spatial attention component, we similarly apply a 1×1 convolution to transform the input feature X into two spatial maps, $Q\in {\mathbb{R}}^{C\times H\times W}$ and $V\in {\mathbb{R}}^{C\times H\times W}$. To compress the spatial dimension of Q, we use global average pooling:

$$\begin{array}{c}\hfill Q^{\prime }=GAP\left(Q\right)\end{array}$$

(32)

The spatial dimension of V remains intact as $H\times W$. Softmax is then applied to enhance the information content of Q:

$$\begin{array}{c}\hfill Q_{h,w}^{\prime }=Softmax\left(Q_{h,w}\right),\forall (h,w)\in [1,H]\times [1,W]\end{array}$$

(33)

The attention map is obtained by performing matrix multiplication between Q and K:

$$\begin{array}{c}\hfill A_{spatial}=Q^{\prime }·K^T\end{array}$$

(34)

Subsequently, feature transformation and a sigmoid function are applied:

$$\begin{array}{c}\hfill F_{spatial}^{out}=Sigmoid\left(Transform(A_{spatial}·V)\right)\end{array}$$

(35)

The final output feature $F_{PSA}$ is obtained by combining both channel attention and spatial attention outputs:

$$\begin{array}{c}\hfill F_{PSA}=F_{channel}^{out}+F_{spatial}^{out}\end{array}$$

(36)

3.4. Loss Function

In this study, the proposed method for segmenting concrete bridge damages leverages a combination of Cross-Entropy Loss (CE Loss) and Dice Loss to optimize the segmentation performance.

CE Loss, commonly used in multi-class and binary classification tasks, measures the dissimilarity between the predicted probabilities and the ground-truth labels. It effectively penalizes incorrect predictions, ensuring that the model learns to output probabilities close to the true label distributions. For damage segmentation, CE Loss helps in handling class imbalances by emphasizing the need for accurate pixel-wise classification. The loss function is defined as follows:

$$\begin{array}{c}\hfill {\mathcal{L}}_{CE}=-{\displaystyle \frac{1}{N}}\sum _{i=1}^N\left[y_{i}log\left({\widehat{y}}_i\right)+\left(1-y_i\right)log\left(1-{\widehat{y}}_i\right)\right]\end{array}$$

(37)

where $y_i$ and ${\widehat{y}}_i$ represent the true and predicted probabilities for pixel i, and N is the total number of pixels.

Dice Loss, on the other hand, directly optimizes for the overlap between the predicted and ground-truth masks by focusing on maximizing their Dice Similarity Coefficient (DSC). It is particularly effective in addressing issues of class imbalance, as it equally emphasizes both large and small regions within the mask. Dice Loss is given by

$$\begin{array}{c}\hfill {\mathcal{L}}_{Dice}=1-{\displaystyle \frac{2{\sum }_{i=1}^N{y}_i{\widehat{y}}_i}{{\sum }_{i=1}^N{y}_i+{\sum }_{i=1}^N{\widehat{y}}_i}}\end{array}$$

(38)

The combined loss function, a weighted sum of CE Loss and Dice Loss, ensures robust optimization by leveraging the strengths of both approaches:

$$\begin{array}{c}\hfill \mathcal{L}=\alpha {\mathcal{L}}_{CE}+\beta {\mathcal{L}}_{Dice}\end{array}$$

(39)

where $\alpha$ and $\beta$ are hyperparameters that balance the contributions of each loss term. By combining these loss functions, the model achieves enhanced segmentation accuracy, as CE Loss ensures precise pixel-wise predictions while Dice Loss enforces shape and region-level consistency.

4. Experiments and Results

4.1. Experimental Environment

4.1.1. Datasets

Since there is no publicly available dataset for concrete bridge damage detection, we constructed our own dataset by collecting 10,000 concrete bridge inspection images from various bridge inspection reports, as shown in Figure 9. These images contain six types of damage, including cracks, spalling, corrosion, rebar exposure, stains, and holes, as well as background images of the bridge surface. Among these, 7354 images depict different types of damage, while 2644 images represent background scenes.

In order to create the box-level annotations for these six types of damage, we annotated the damage regions with bounding boxes, as illustrated in Figure 10. The distribution of images and instance-level annotations for each type of damage are provided in

Table 1. Statistical information on concrete bridge damage dataset.

	Spall	Rebar	Corrosion	Crack	Hole	Speckle	Total
Number of boxes	6921	9901	2486	2735	1260	1092	30,458
Number of images	3365	2711	2486	1287	633	557	7354

.

The annotation process was conducted under the supervision of a domain expert with extensive experience in machine vision and structural damage identification. The expert first guided the selection of bridge damage images, which were compiled into a curated damage image sample library. Next, the expert distributed the damage sample images to three independent annotators, who manually labeled the damage regions using a bounding-box-based approach. After the initial annotation, the labeled images were reviewed by the expert. Any incorrect or inconsistent annotations were reassigned to another annotator for re-labeling. This iterative annotation and validation process was repeated until all annotations met the predefined accuracy and consistency standards. Each image in the dataset is uniquely identified by a numerical file name ranging from 1 to 10,000, ensuring clear and consistent labeling. The original image files are in JPG format, and the corresponding annotation files are stored in XML format. The XML files contain detailed information about the annotations, including the bounding box coordinates and damage categories for each image. As shown in Figure 11, the XML file structure includes metadata, the labeled damage regions, and additional attributes such as image dimensions and class labels. The standardized naming and file structure facilitate efficient management and use of the dataset.

In our dataset, the distribution of damage center points is predominantly concentrated near the center of the image, with a relatively small proportion of damage regions appearing at the edges. This is illustrated in Figure 12, which shows the normalized label center points and the width-to-height ratios of the bounding boxes for the annotated damage regions. Concrete bridge damage typically occupies a small portion of the image, suggesting that networks designed for small-object detection could achieve better performance in detecting such damage types.

Since the task at hand is semantic segmentation, we further created pixel-level annotations for 1000 images to evaluate the proposed method. A sample of these pixel-level annotations is shown in Figure 13.

4.1.2. Experimental Settings

The model training was carried out over a total of 100 epochs, with each epoch consisting of a complete pass through the training dataset. The batch size was set to 16, meaning that the model parameters were updated after processing 16 samples. This batch size strikes a balance between computational efficiency and stability of gradient estimates. Training was conducted using a Stochastic Gradient Descent (SGD) optimizer, widely known for its efficiency in large-scale machine learning tasks. The optimizer incorporated a momentum term of 0.9 to accelerate convergence and smooth the optimization process by considering past gradients, which helps in reducing oscillations. The initial learning rate ${\eta }_0$ was set to ${10}^{-3}$, providing a good balance between rapid convergence and stability during training. As training progressed, the learning rate decayed exponentially according to the following schedule:

$$\begin{array}{c}\hfill {\eta }_t={\eta }_0·e^{-\lambda t}\end{array}$$

(40)

where t is the epoch index and $\lambda$ is the decay factor, set to 0.01. This exponential decay allowed for finer adjustments to the learning rate as training advanced.

The model was optimized using a combined loss function, which consisted of both Cross-Entropy Loss and Dice Loss, addressing the specific challenges of bridge disease segmentation in concrete structures. Cross-Entropy Loss measures the discrepancy between predicted probabilities and true class labels, ensuring accurate classification performance, while Dice Loss quantifies the overlap between predicted and true segmentation masks, which is particularly valuable when dealing with imbalanced datasets. Both losses were weighted equally, with coefficients $\alpha =0.5$ and $\beta =0.5$, to balance their respective contributions during training.

All experiments were conducted using the PyTorch 2.1 framework. The training and evaluation processes were performed on four NVIDIA RTXA6000 GPUs (Nvidia Corporation, Santa Clara, CA, USA), leveraging their high computational efficiency. Mixed precision training was employed to optimize memory usage and speed up computation.

4.1.3. Evaluation Metrics

To evaluate the performance of the proposed segmentation method for concrete bridge damage detection, four widely used metrics were adopted: intersection over union (IoU), pixel accuracy (PA), and Dice Similarity Coefficient (Dice). These metrics comprehensively measure the accuracy of pixel-level predictions, ensuring both localization and classification performance are assessed.

IoU measures the overlap between the predicted segmentation and the ground truth, defined as the ratio of their intersection over their union:

$$\begin{array}{c}\hfill IoU={\displaystyle \frac{TP}{TP+FP+FN}}\end{array}$$

(41)

where TP, FP, and FN represent the number of true positive, false positive, and false negative pixels, respectively. IoU provides an overall measure of the region-level overlap.

PA indicates the number of pixels with correct prediction categories as a percentage of the total number of pixels:

$$\begin{array}{c}\hfill PA={\displaystyle \frac{TP+TN}{TP+TN+FP+FN}}\end{array}$$

(42)

Dice is a region-based metric that quantifies the harmonic mean of precision and recall, providing a balanced measure of segmentation accuracy:

$$\begin{array}{c}\hfill Dice={\displaystyle \frac{2·TP}{2·TP+FP+FN}}\end{array}$$

(43)

Dice is particularly sensitive to imbalanced datasets and small damage regions, making it an essential metric for assessing damage segmentation tasks.

In addition to segmentation accuracy, computational efficiency is a crucial aspect in evaluating the practicality of deep learning models for real-world deployment in bridge inspection tasks. Therefore, we report the model’s performance in terms of frames per second (FPS) and GPU memory usage to provide a comprehensive assessment of its applicability.

4.2. Performance Comparison of Different Models

To evaluate the effectiveness of the proposed Mamba–ResNet hierarchical fusion network in addressing the segmentation of concrete bridge damage, we conducted comprehensive experiments comparing its performance against several state-of-the-art models. The selected baseline models represent a diverse set of architectures and methodologies, including both transformer-based and convolutional neural network (CNN)-based designs, to ensure a thorough performance assessment. Specifically, we included DeepLabV3+, a widely used semantic segmentation framework known for its atrous spatial pyramid pooling [62]; SegFormer, which combines a lightweight design with Transformer-based representation [63]; HRViT, leveraging high-resolution feature fusion with Vision Transformers [64]; ABCNet, which uses spatial path and contextual path architecture [65]; PIDNet, which uses a novel three-branch network architecture [66]; SeaFormer, emphasizing efficiency in lightweight segmentation tasks [67]; Mamba–UNet, a Mamba-based U-Net architecture [68]; and SCTNet, which extracts rich semantic information by learning from Transformer and CNN semantic information alignment [69]. The inclusion of these models provides a comprehensive benchmarking landscape, highlighting the strengths and limitations of our approach.

As shown in

Table 2. Results of comparison with different methods.

Method	Backbone	MIoU (%)	PA (%)	MDice (%)	Model Parameters (M)	FPS	IoU (%)
							Rebar	Spall	Corrosion	Crack	Hole	Speckle	Background
DeepLabV3+	ResNet50	54.45	69.84	70.91	42.0	30.2	48.05	60.86	50.52	41.49	20.57	62.77	96.94
SegFormer	Mit B0	56.45	71.33	72.96	84.7	20.3	50.82	62.25	51.55	45.16	23.55	64.91	96.95
HRViT	HRViT-B3	57.11	72.47	73.88	28.6	50.6	51.05	63.27	52.57	46.37	23.73	65.84	96.97
ABCNet	ResNet101	58.31	72.55	74.68	14.6	40.2	51.08	64.13	56.75	48.37	24.05	66.89	96.92
PIDNet	-	58.20	72.35	73.72	36.9	31.1	51.12	64.06	56.72	48.16	21.77	68.65	96.95
SeaFormer	MiT-B4	58.70	73.48	73.35	17.1	62.8	50.86	63.67	57.21	48.90	23.82	69.54	96.95
Semi-Mamba	VMamba	59.09	73.21	74.8	65.5	25.6	52.23	64.34	57.34	48.73	25.88	68.23	96.93
SCTNet	-	57.90	72.51	74.03	17.4	61.2	49.86	63.62	57.18	48.23	23.88	65.63	96.91
OUR	MambaVision	60.13	74.02	75.40	106.7	15.3	53.55	64.74	57.24	49.75	26.92	71.76	96.98

, our proposed method achieves superior performance, with MIoU, PA, and MDice scores of 60.13%, 74.02%, and 75.40%, respectively. Compared to the second-best-performing model, Semi-Mamba, our approach improves MIoU by 1.04%, PA by 0.81%, and MDice by 0.6%. These improvements highlight the efficacy of our hierarchical fusion strategy in feature extraction and integration. Notably, Semi-Mamba also incorporates the SSM method from Mamba for backbone feature extraction, which aligns with our approach in utilizing advanced feature extraction techniques. However, the performance gains in our proposed method underscore the advantages of leveraging ViT methods for backbone network feature extraction over traditional CNN approaches. From the individual class IoU results, it is evident that our model consistently achieves the best results across various damage categories. The segmentation precision for the hole class, in particular, shows a remarkable improvement, attributable to our multi-layer feature extraction and Hierarchical Attention Fusion (HAF) methods. These techniques enhance the network’s ability to capture intricate damage textures and efficiently merge features, thereby boosting the overall segmentation accuracy.

4.3. Ablation Studies

4.3.1. Ablation Study on Overall Framework

To assess the contribution of each component within the proposed Mamba–ResNet hierarchical fusion network for concrete bridge damage segmentation, an ablation study was conducted. The experimental configurations tested are as follows:

1.

This configuration corresponds to the baseline network, which excludes all the components proposed in this study, including MambaVision, Optimizing Prompting Strategy (OPS), Hierarchical Attention Fusion (HAF), Boosting Textures Module (BTM), and Polarized Self-Attention (PSA). This baseline serves as a reference to evaluate the network’s performance without the optimization and enhancement mechanisms introduced in this work. In the baseline network, the backbone is ResNet, and all feature fusion is performed using simple feature map concatenation.

2.

In this configuration, OPS is used for training the baseline network.

3.

This configuration replaces the baseline backbone with MambaVision as the main network backbone, without employing OPS for training.

4.

Building on the previous configuration, this setup uses MambaVision as the main backbone and incorporates OPS for network training.

5.

This configuration includes the OPS and HAF, but excludes the BTM within HAF.

6.

In this setup, the Boosting Textures Module (BTM) is introduced in addition to OPS and HAF, enabling the evaluation of the role of texture enhancement within the feature fusion framework.

7.

This configuration removes BTM from HAF and includes the Polarized Self-Attention (PSA).

8.

In this configuration, only the PSA module is introduced, allowing for the isolated evaluation of the PSA’s effect on segmentation performance.

9.

This represents the full model configuration proposed in this study.

As shown in

Table 3. Results of the overall framework ablation studies. ✓ represents the use of this experimental setup.

Exp.	Baseline	MambaVision	OPS	HAF	BTM	PSA	MIOU (%)	PA (%)	MDice (%)
#1	✓						53.28	68.23	70.22
#2	✓		✓				54.67	69.93	71.03
#3	✓	✓					56.85	71.67	73.31
#4	✓	✓	✓				57.69	72.79	73.68
#5	✓	✓	✓	✓			58.04	73.10	73.91
#6	✓	✓	✓	✓	✓		59.21	73.58	74.57
#7	✓	✓	✓	✓		✓	58.42	73.21	74.11
#8	✓	✓	✓			✓	58.31	73.25	74.08
#9	✓	✓	✓	✓	✓	✓	60.13	74.02	75.40

, the ablation study presents the results of evaluating the overall framework of the proposed method. Exp. #1 and #2 demonstrate that adding OPS to the baseline model results in a performance improvement, with MIoU increasing by 1.39%. Exp. #3 and #4 further confirm that incorporating OPS into the MambaVision backbone also enhances the network’s performance, with MIoU increasing by 0.84%. Exp. #4 and #5 show that adding HAF without BTM also increases the network’s accuracy, with MIoU improving by 0.35%. The comparison between Exp. #2 and #3 shows a substantial improvement in performance when using MambaVision as the backbone, with MIoU increasing by 3.57%, proving the effectiveness of the hierarchical feature extraction network proposed in this work. Exp. #5 and #6 indicate that incorporating BTM further enhances accuracy, with MIoU improving by 1.17%, validating the effectiveness of BTM in capturing the texture features specific to concrete bridge damages. Exp. #7 and #8 show that HAF provides a slight improvement to the network, with MIoU increasing by 0.11%. Finally, the comparison of Exp. #4 and #8 reveals that using PSA as the decoding network, even without HAF, enhances segmentation performance, with MIoU improving by 0.62%. Overall, the proposed method demonstrates significant improvements over the baseline, with MIoU, PA, and MDice increasing by 6.85%, 5.79%, and 5.18%, respectively.

4.3.2. Ablation Study on Different SAM Prompting Strategies

To rigorously analyze the impact of different SAM prompting strategies on concrete bridge damage segmentation, we conducted an ablation study comparing three prompting approaches: point, box, and box + point. The three prompting strategies were evaluated using standard segmentation metrics including MIOU, MPA, and MDice. The experimental results are presented in

Table 4. Results of the different SAM prompting strategies. ✓ represents the use of this experimental setup.

Point	Box	box + Point	MIOU	MPA	MDice
✓			57.28	72.34	73.56
	✓		58.67	73.11	74.22
✓	✓	✓	60.13	74.02	75.40

.

The results indicate that the box and point prompting strategy outperforms the other approaches with the highest MIOU of 60.13%, MPA of 74.02%, and MDice of 75.40%. The box-only strategy shows better performance than the point-only method with improvements of approximately 1.39% in MIOU, 0.77% in MPA, and 0.66% in MDice. These findings confirm that integrating both types of prompts offers complementary benefits by constraining the target region and providing essential fine-grained details, which is particularly beneficial for complex and elongated damage patterns.

4.3.3. Ablation Study on Different Backbone Networks

The goal of this section is to investigate the impact of different backbone networks on the segmentation results. Specifically, we aim to assess how the choice of backbone network affects the performance of the proposed method for concrete bridge damage segmentation. The approach employs a dual-backbone architecture, where the primary backbone, based on MambaVision, captures global contextual features, while the secondary backbone, using ResNet, extracts local features.

To explore the role of the main backbone, we replace MambaVision with several alternative backbone networks, including MobileViT, a lightweight hybrid model combining CNNs and ViTs to balance efficiency and accuracy [70]; Swin Transformer, which uses hierarchical window-based attention to model local and global features at multiple scales [52]; ConvFormer, which integrates convolutional operations into Transformer structures to enhance locality modeling [71]; and CaiT, a Transformer model designed to improve attention mechanisms for capturing fine-grained details [72]. These ablation experiments are designed to demonstrate the effectiveness of MambaVision in comparison to other popular architectures.

As shown in

Table 5. Results of ablation study on main backbone.

Backbone Networks	MIOU (%)	PA (%)	MDice (%)
MobileViT	58.26	73.57	73.91
Swin Transformer	59.21	73.78	74.21
ConvForme	58.13	73.21	74.34
CaiT	58.78	73.65	74.21
MambaVision	60.13	74.02	75.40

, the experimental results using different backbone networks highlight the superior performance of the MambaVision backbone. It achieves the best results in terms of MIoU, PA, and MDice, demonstrating MambaVision’s significant advantage in capturing fine-grained features and maintaining long-range dependencies. Compared to the Swin Transformer, the MambaVision backbone improves the MIoU, PA, and MDice by 0.92%, 0.24%, and 1.19%, respectively.

In order to explore the individual contributions of different backbone networks to the performance of concrete bridge damage segmentation, we designed an ablation study to compare the performance of the proposed Mamba–ResNet hierarchical fusion network with two separate backbone configurations: MambaVision and ResNet.

As shown in

Table 6. Results of the ablation study on the individual contributions of different backbones. ✓ represents the use of this experimental setup.

ResNet	MambaVision	MIOU	MPA	MDice
✓		54.33	69.67	71.24
	✓	58.42	73.36	74.11
✓	✓	60.13	74.02	75.40

, the experimental results indicate that the ResNet-only configuration achieves an mIoU of 54.33%, an mPA of 69.67%, and an mDice of 71.24%, while the MambaVision-only configuration yields an mIoU of 58.42%, an mPA of 73.36%, and an mDice of 74.11%. Moreover, the fusion of ResNet and MambaVision produces the highest performance with an mIoU of 60.13%, an mPA of 74.02%, and an mDice of 75.40%. By using MambaVision as the main backbone, the network can establish a global understanding of damage distribution, while ResNet as the auxiliary backbone refines the segmentation results by enhancing local details at various scales. This hierarchical fusion ensures that the model can adapt to both local irregularities and global structural changes, improving segmentation robustness in real-world applications.

4.3.4. Ablation Study on Different Attention Module Decoders

To evaluate the impact of different attention mechanisms on segmentation performance, we conducted an ablation study by integrating five distinct attention modules into the decoder architecture of our proposed Mamba–ResNet hierarchical fusion network. Specifically, we compared the performance of the following attention mechanisms: Squeeze-and-Excitation (SE) [73], Convolutional Block Attention Module (CBAM) [74], Selective Kernel Attention (SKA) [75], Efficient Multi-scale Attention (EMA) [76], and Global Attention Module (GAM) [77].

As shown in

Table 7. Results of ablation study on attention module decoders.

Attention Module	MIOU (%)	PA (%)	MDice (%)
SE	59.32	73.68	74.77
CBAM	59.51	73.79	74.83
SKA	59.59	73.85	74.95
EMA	60.02	73.92	75.21
GAM	59.48	73.72	74.78
SPA	60.13	74.02	75.40

, the results of experiments using different attention modules for the decoder demonstrate that the SPA decoder achieves the best performance in terms of MIOU, PA, and MDice. This confirms that SPA effectively models both local and global dependencies, offering a more balanced performance compared to other attention mechanisms. Compared to EMA, SPA improves the MIoU, PA, and MDice by 0.11%, 0.10%, and 0.19%.

4.3.5. Ablation Study on Loss Function

In this section, we compare our proposed CE + Dice Loss with three commonly used loss functions, Mean Absolute Error (MAE), Cross-Entropy (CE), and Dice Loss, to demonstrate the advantages of our approach for concrete bridge damage segmentation.

As shown in

Table 8. Results of ablation study on loss function.

Loss Function	MIOU (%)	PA (%)	MDice (%)
$L_{mae}$	58.11	72.56	73.62
$L_{CE}$	59.52	73.73	74.76
$L_{Dice}$	59.89	73.95	75.32
$L_{CE}+L_{Dice}$	60.13	74.02	75.40

, our proposed CE + Dice Loss combines the advantages of CE and Dice to deal with category imbalance, improve the detection of small damage regions, and maintain robustness to noise. Our ablation experimental results show that CE + Dice outperforms other loss functions in MIOU, PA, and MDice.

4.3.6. Hyperparametric Analysis

To analyze the impact of the hyperparameters on the model’s accuracy, we designed an ablation experiment. The learning rate (Lr) is one of the most critical hyperparameters in training machine learning models, influencing the final performance of the model. In this experiment, we systematically varied the learning rate while keeping all other hyperparameters constant. This approach allowed us to evaluate how different learning rates affected the model’s ability to detect and segment concrete bridge defects. Specifically, we tested a range of learning rates from 0.0001 to 0.01, with each experiment conducted under identical conditions to ensure the results were solely due to the change in learning rate.

The experimental results are presented in

Table 9. Results of hyperparametric ablation experiments.

Lr	MIOU (%)	MPA (%)	MDice (%)
0.0001	58.27	73.35	73.64
0.0005	59.32	73.88	74.36
0.001	60.13	74.02	75.40
0.005	58.77	73.72	74.41
0.01	58.68	73.62	74.12

. The table summarizes the performance of the model under different learning rates, showing the impact on both the segmentation accuracy and the overall model efficiency. As seen in the table, the model with a learning rate of 0.001 achieved the highest accuracy, with a significant improvement in defect segmentation quality compared to the other rates.

4.4. Visualization Results

In this section, we present representative segmentation results to demonstrate the effectiveness of the proposed method. We selected several representative cases to perform comparisons with other state-of-the-art networks and analyze the results of ablation experiments. The visual comparisons serve as a critical evaluation tool for highlighting the superiority of the SAM-guided Mamba–ResNet hierarchical fusion network in concrete bridge damage segmentation.

Figure 14 presents a comparison of various methods evaluated in Section 4.2, including DeepLabV3+, SegFormer, HRViT, ABCNet, PIDNet, SeaFormer, Semi-Mamba, and SCTNet. As observed in the first and second rows of the figure, our proposed method demonstrates excellent segmentation results for single-class damage, particularly for the crack category, where the crack contours are closest to the ground truth. In the third row, it is evident that our method excels in segmenting small objects, with other networks struggling to accurately segment smaller features, such as rebar, leading to segmentation failures and fragmentations. The fourth row highlights our method’s superior ability to identify regions in the concrete background that resemble damage. While some methods incorrectly classify damaged background regions as cracks, our approach accurately differentiates these areas. Finally, in the fifth row, the dense damage segmentation results show that although our method, like others, occasionally misclassifies some background regions, it still produces the most accurate segmentation for actual damage areas. This demonstrates the robustness of our approach in handling complex and densely damaged concrete structures.

Figure 15 illustrates the ablation study results presented in Section 4.3.1, comparing several configurations: baseline, baseline + OPS, baseline + OPS + MambaVision, baseline + OPS + MambaVision + HAF, and baseline + OPS + MambaVision + HAF + PSA. As shown in the first row for crack segmentation, without OPS, the baseline method struggles to accurately segment the background surrounding the cracks. However, after adding OPS, the crack segmentation results improve significantly, with the network able to better delineate the crack contours. In the second row, for multi-class damage segmentation, the baseline model without OPS mistakenly recognizes complex background areas as damage categories. After integrating OPS, the contours of the damage become more aligned with the ground truth, although some discrepancies still remain. Adding MambaVision and HAF further refines the segmentation, with the contours becoming progressively clearer. The final model, with all components (OPS, MambaVision, HAF, PSA), shows a segmentation result that closely matches the ground truth, with minimal difference. The comparison in the second row clearly demonstrates that OPS provides a substantial improvement in segmentation performance. This is primarily due to the enhanced quality of labels generated by the SAM, which leads to more accurate and reliable segmentation results, bringing the network’s performance closer to the true segmentation.

4.5. Diversity Validation

To comprehensively evaluate the diversity and generalization capability of the proposed Mamba–ResNet hierarchical fusion network, we conducted extensive experiments on the publicly available CrackLS315 dataset. This dataset consists of a variety of real-world concrete crack images, making it a suitable benchmark for validating the robustness of our approach. We compared our model against several state-of-the-art segmentation networks, including DeepLabV3+ [62], SegFormer [63], HRVIT [64], ABCNet [65], PIDNet [66], SeaFormer [67], Semi-Mamba [68], and SCTNet [69]. These models represent a range of architectures, from transformer-based designs to CNN-based and hybrid approaches, ensuring a fair and comprehensive evaluation. The evaluation metrics used in this experiment were MIOU, MPA, and MDice.

As shown in

Table 10. Results of comparison with different methods on CrackLS315 dataset.

Method	Backbone	MIoU (%)	MPA (%)	MDice (%)	Model Parameters (M)	FPS
DeepLabV3+	ResNet50	65.89	75.44	74.88	42.0	30.2
SegFormer	Mit B0	66.22	76.33	75.13	84.7	20.3
HRViT	HRViT-b3	67.22	80.62	76.12	28.6	50.6
ABCNet	ResNet101	66.41	76.21	75.78	14.6	40.2
PIDNet	-	67.89	80.89	76.69	36.9	31.1
SeaFormer	MiT-B4	68.06	81.86	76.85	17.1	62.8
Semi-Mamba	VMamba	67.21	80.58	76.26	65.5	25.6
SCTNet	-	66.32	80.51	76.01	17.4	61.2
OUR	MambaVision	69.01	82.19	77.73	106.7	15.3

, our proposed method achieved an MIoU of 69.01%, an MPA of 82.19%, and an MDice of 77.73%, outperforming all the compared networks. Notably, compared with traditional CNN-based models like DeepLabV3+ and Transformer-based models such as HRViT and SeaFormer, our model exhibits significant improvements. These improvements highlight the efficacy of our hierarchical fusion strategy in feature extraction and integration. The results demonstrate that our method effectively combines local and global contextual information, resulting in enhanced segmentation performance across diverse scenarios.

Figure 16 illustrates the experimental crack segmentation visualization results on the CrackLS315 dataset. This visual evidence further confirms the superiority of our approach in accurately delineating crack regions, even in cases of extremely fine cracks and challenging conditions.

5. Discussion

The proposed framework has demonstrated significant potential in concrete bridge damage segmentation. Beyond this specific application, the underlying principles of our architecture suggest its adaptability to a broader range of infrastructure damage detection tasks. Given that our approach combines hierarchical feature fusion with attention mechanisms for effective feature extraction of surface defects to remove other interfering backgrounds, it can be extended to the detection of structural defects in other infrastructure domains, such as pavement collapse [78], tunnel cracks [79], and dam surface erosion [80]. One key advantage of our framework is its ability to leverage foundational models like SAM, which enables robust feature extraction even in scenarios where labeled training data are scarce. This property suggests promising applicability in scenarios where high-quality annotations are difficult to obtain, such as corrosion detection in steel structures or defect identification in historical buildings, where manual labeling remains a significant challenge [81,82]. Furthermore, the effectiveness of our method in segmenting bridge damages indicates that it could be generalized to other multi-scale damage detection tasks. For example, in railway track inspection [83], where damages can appear at varying resolutions and orientations, our hierarchical fusion strategy could enhance segmentation accuracy. Similarly, in offshore wind turbine maintenance, where visual inspection is often constrained by harsh environmental conditions, SAM-guided segmentation could facilitate automated damage detection using drone imagery [84].

In the pursuit of enhancing concrete bridge damage segmentation, future research should explore the integration of few-shot learning and unsupervised learning methodologies. These approaches have the potential to significantly reduce the dependency on extensive labeled datasets while maintaining robust segmentation performance. Few-shot learning aims to enable models to generalize from limited annotated samples, which is particularly beneficial for damage segmentation where acquiring labeled data is labor-intensive and costly. Unsupervised learning approaches, such as self-supervised representation learning and contrastive learning, offer a promising direction for bridge damage segmentation. These methods enable models to extract meaningful features from large-scale unlabeled datasets, reducing reliance on manually annotated damage images. The fusion of few-shot learning and unsupervised learning could lead to a more data-efficient and adaptable segmentation framework. A hybrid pipeline could leverage self-supervised pre-training for feature extraction, followed by fine-tuning with few-shot learning strategies on specific damage categories. This integration may reduce annotation costs while maintaining high segmentation precision, particularly in real-world scenarios where annotated datasets are scarce. Future studies should explore optimal network architectures and training paradigms that effectively combine these learning paradigms to maximize segmentation performance.

The integration of Siamese neural networks (SNNs) into our proposed framework holds significant potential. SNNs have been widely used in image similarity tasks due to their ability to learn discriminative feature embeddings for paired input comparisons [85]. By incorporating SNNs, our framework could enhance damage detection accuracy by improving feature differentiation between damaged and undamaged regions. Moreover, SNNs could improve the adaptability of our framework to unseen damage patterns by learning fine-grained similarities across different bridge structures. This would be particularly beneficial for real-world applications where pre-existing datasets may not fully capture the diversity of damage manifestations. Additionally, the combination of SNNs and our proposed hierarchical fusion network could reduce false positives by emphasizing structural consistency in segmentation predictions.

Concrete bridges often suffer from large-area spalling, which not only compromises structural integrity but also complicates damage detection due to the similarity between damage and background, as well as the presence of complex textures. These factors pose significant challenges for current deep learning-based segmentation methods, limiting their ability to distinguish defects from surrounding environments and leading to reduced accuracy and robustness. To address these challenges, future research should focus on integrating prior knowledge fusion with graph neural networks (GNNs) [86] to enhance defect segmentation performance. Specifically, prior knowledge should incorporate spatial relationships and co-occurrence patterns of defects, as certain types of defects often appear in specific locations or in relation to other defects due to structural stress distribution and material degradation processes. By encoding these spatial and co-occurrence constraints into the learning model, segmentation accuracy can be significantly improved, as the model learns to predict defect locations with higher contextual awareness. Moreover, GNNs can effectively model these spatial dependencies and relational structures, allowing the network to capture the underlying geometric and topological patterns of defect distribution [87]. By constructing a graph representation of the defects, GNNs enable the model to refine segmentation results by leveraging the correlations between different defect types. Additionally, integrating multi-scale attention mechanisms with GNNs can further enhance segmentation by dynamically focusing on critical defect areas while suppressing background noise.

We plan to represent each segmented image as a graph $G=(V,E)$, where each node $v_i\in V$ corresponds to a damage region (e.g., cracks, spalling), and each edge $e_{ij}\in E$ encodes spatial adjacency or structural dependency between regions. The feature vector of each node ${\mathbf{h}}_i^{\left(0\right)}$ is initialized using region-level CNN embeddings extracted from our current segmentation backbone. We will apply a standard GCN layer, defined as

$$\begin{array}{c}\hfill {\mathbf{H}}^{(l+1)}=\sigma \left({\tilde{D}}^{-1/2}\tilde{A}{\tilde{D}}^{-1/2}{\mathbf{H}}^{\left(l\right)}{\mathbf{W}}^{\left(l\right)}\right)\end{array}$$

(44)

where $\tilde{A}=A+I$ is the adjacency matrix with self-connections, $\tilde{D}$ is the degree matrix of $\tilde{A}$, $\mathbf{H}\left(l\right)$ is the node feature matrix at layer l, and ${\mathbf{W}}^{\left(l\right)}$ are learnable weights. The activation function $\sigma (·)$ can be ReLU or LeakyReLU. We will train this GNN integration model based on the image dataset generated from the framework proposed in this paper and evaluate it using metrics such as mIoU. The contribution of the GNN module in the overall performance will also be evaluated through ablation experiments.

6. Conclusions

In this paper, we have introduced a novel training framework for concrete bridge damage segmentation that significantly reduces the dependency on extensive annotated data while enhancing segmentation efficiency and accuracy. Our approach leverages the Segment Anything Model (SAM) to generate high-quality segmentation masks, which are then used as supplementary supervision labels to train a damage segmentation network. To ensure the reliability and precision of the masks generated by the SAM, we proposed an Optimizing Prompting Strategy with saliency information, which improves the quality of the prompts provided to SAM, thereby enhancing the accuracy of the generated masks. Furthermore, we presented a trainable semantic segmentation network that employs a multi-level feature extraction architecture. In this architecture, MambaVision serves as the primary backbone network, enabling the extraction of rich and diverse semantic features from concrete bridge damages at varying levels. To effectively integrate these multi-level features and improve segmentation performance, we introduced a Hierarchical Attention Fusion (HAF) mechanism. The HAF mechanism includes a Texture Enhancement Module (TEM), which is designed to strengthen the representation of discriminative texture features, thereby enhancing the network’s ability to capture fine-grained details in concrete bridge damages. Additionally, we utilized a Polarized Self-Attention (PSA) decoder as the feature decoder, which predicts precise segmentation results for concrete bridge damages. Our experimental results demonstrate that the proposed framework achieves state-of-the-art performance in concrete bridge damage segmentation, outperforming existing methods in terms of both accuracy and efficiency. The integration of SAM-guided mask generation, the Optimizing Prompting Strategy, and the Hierarchical Attention Fusion mechanism with its Texture Enhancement Module and the robust feature extraction capabilities of MambaVision collectively contribute to the robustness and effectiveness of our approach.