Controlled Benchmarking and Component-Aware Ablation for Railway Viaduct Structural and Damage Segmentation

Abstract

Automated damage inspection of railway viaducts requires pixel-level identification of structural components and surface damage such as cracking and rebar exposure. A common assumption in bridge inspection is that damage segmentation improves when component information is provided alongside the image. This study tests that assumption on the Tokaido synthetic viaduct dataset using controlled comparisons between segmentation models with and without component information. Both damage and structural component segmentation are evaluated across multiple architectures, and the trained component model is assessed on real viaduct photographs against a baseline model requiring no task-specific training. Under the original random split, explicit component conditioning does not produce a measurable improvement in damage segmentation: all tested strategies remain within 0.008 mean Intersection-over-Union (mIoU) of a baseline without component input, and this null result persists even when component predictions are reliable. Under a leakage-controlled scene-disjoint split, however, the same component-aware variants show a small positive trend (up to $+0.019$ mIoU over three seeds), so the effect of component conditioning depends on the evaluation protocol. The best unconditioned model reaches 0.569 mIoU for damage segmentation; for real-photo component segmentation, the trained model reaches 0.424 mIoU compared with 0.250 mIoU for the training-free baseline. These results show that multitask benefits reported in bridge inspection do not automatically translate into gains from explicit use of component information on synthetic viaduct data, where damage occurs almost exclusively on columns yet is too sparse for structural element identity to yield more than a marginal localisation gain. The multi-architecture benchmark and the measured real-photo structural transfer gap provide reference baselines for subsequent work on component-aware and transfer-robust inspection.

1. Introduction

Railway viaduct inspection requires pixel-level information about both structural components and visible surface damage. Manual visual inspection remains costly, time-consuming, and dependent on subjective interpretation, especially when elevated structures must be assessed repeatedly across large transport networks [1,2,3]. Semantic segmentation offers a direct route from inspection imagery to spatially explicit component and damage maps, but its usefulness depends on whether models can identify small damage regions and transfer beyond the data used for training. These requirements make viaduct inspection a stringent test case for modern segmentation methods.

Recent bridge-inspection research has established two important directions. First, bridge element parsing and defect segmentation can be learned jointly, and multitask models are often reported to improve element or defect metrics relative to single-task baselines [2,3]. Second, synthetic environments, such as the Tokaido synthetic viaduct benchmark and related synthetic inspection datasets [4,5,6], can reduce annotation cost and support repeatable experiments. These studies show that component information, synthetic data, and deep segmentation architectures can support structural inspection. They do not, however, resolve the issue of whether explicit component conditioning improves damage localisation under a controlled viaduct benchmark.

The distinction between multitask learning and explicit component conditioning is central to this study. A multitask network may improve damage segmentation because shared features regularise the encoder, because component supervision improves structural representation, or because the damage branch directly receives component information. Prior bridge-inspection work has mostly evaluated the combined effect of feature sharing and task interaction [2,3]. The practical question remains narrower: if a model already has access to an image, does adding predicted component information help it localise cracks, concrete damage, or exposed reinforcement? This question is testable only when architecture, training recipe, dataset split, and evaluation protocol are controlled.

Architecture choice adds a second unresolved factor. General semantic-segmentation research has moved from convolutional encoder–decoder models such as U-Net and DeepLabV3+ toward hierarchical transformers and modernised convolutional architectures, including Swin Transformer, SegFormer, and ConvNeXt [7,8,9,10,11,12,13,14]. Inspection imagery differs from generic scene parsing because damage classes are sparse, thin, and texture-driven. A model that performs well on broad segmentation benchmarks may therefore fail under strong foreground–background imbalance. A controlled comparison across convolutional neural network (CNN), ConvNeXt, Vision Transformer (ViT), SegFormer, and Swin-style architectures is needed before claims about the benefits of component-aware damage localisation can be interpreted.

Synthetic-to-real transfer also remains a limiting condition for real-life deployment. The Tokaido dataset provides controlled synthetic viaduct imagery with component and damage annotations, while dacl10k and related bridge datasets demonstrate the diversity and difficulty of real inspection photographs [4,15]. Foundation segmentation models provide a complementary reference point because GroundingDINO and the Segment Anything Model (SAM) can produce zero-shot masks without task-specific training [16,17]. Comparing a trained synthetic-data model against such a training-free baseline helps separate ordinary segmentation accuracy from real-photo transfer performance.

This study evaluates railway viaduct component and damage segmentation under a controlled benchmark design. Seven segmentation architectures are trained for structural component and damage segmentation on the Tokaido synthetic dataset, using fixed splits, repeated seeds, and a common evaluation protocol. A separate component-aware ablation study tests damage-only, parallel-head, hard-mask, detached soft-conditioning, and end-to-end soft-conditioning variants. The design isolates whether component predictions improve damage localisation beyond a damage-only baseline. The best damage model reaches $0.569$ foreground mIoU on the synthetic test split, while the component-aware variants remain within $0.008$ mIoU of the unconditioned baseline. Under this random split, the component-aware variants provide no measurable damage segmentation gain; under a leakage-controlled scene-disjoint split, however, the same variants show a small positive trend, so the effect of component conditioning is protocol-dependent.

The paper makes three contributions. First, the study provides a controlled CNN–Transformer benchmark for viaduct structural component and damage segmentation on the Tokaido dataset. Second, the component-aware ablation shows that the effect of reliable component predictions on damage localisation is protocol-dependent: absent under the original random split, with a small positive trend under a leakage-controlled scene-disjoint split. Third, the real-photo component-transfer evaluation compares a component segmentation model trained on synthetic data with a GroundingDINO + SAM zero-shot pipeline (GroundedSAM), where the trained model reaches $0.424$ mIoU and the zero-shot baseline reaches $0.250$ mIoU on real viaduct photographs. Together, these results position the benefits of component-aware conditioning as an empirical question rather than an assumed benefit, and they provide baseline evidence for future transfer-robust inspection models.

2. Related Work

Research on automated bridge and viaduct inspection draws from two distinct but interacting bodies of literature: generic semantic segmentation, which supplies the backbone architectures and training methods, and inspection-specific segmentation, which defines the tasks, datasets, and domain challenges. Progress in generic segmentation has been rapid and architecturally diverse, while progress in inspection-specific methods has been constrained by small real-world datasets, heterogeneous evaluation protocols, and narrow damage taxonomies. This section reviews both areas with respect to the three unresolved questions that motivate the present study: how convolutional neural network (CNN) and transformer architectures compare on the same viaduct benchmark, whether explicit component-label conditioning improves damage localisation, and how large the synthetic-to-real transfer gap is against a training-free baseline.

2.1. Semantic Segmentation Architectures for Structural Inspection

The encoder–decoder paradigm, established by U-Net [7] and its densely connected extension UNet++ [18], defined the spatial segmentation baseline: a contracting encoder captures context and a symmetric decoder recovers spatial resolution through skip connections. DeepLabv3+ [8] offered a complementary approach via atrous spatial pyramid pooling, capturing multi-scale context without the resolution loss of pooling. UPerNet [19] paired a feature pyramid network with a pyramid pooling module over any hierarchical backbone, and it is the primary decoder used with Swin Transformer and ConvNeXt in the present benchmark. These convolutional baselines are well validated for bridge crack and damage segmentation tasks [20,21,22,23,24,25,26] and represent the CNN family in the controlled comparison.

Vision Transformer architectures introduced global self-attention and altered the accuracy–efficiency tradeoff [27,28,29]. Swin Transformer [9] achieved 48.1 mIoU on ADE20K with the UPerNet decoder by using shifted window attention to build hierarchical feature maps. SegFormer [10] combined a mix-transformer hierarchical encoder with a lightweight all-multilayer-perceptron (MLP) decoder, reaching 51.0/51.8 mIoU (single-/multi-scale). ConvNeXt [11] demonstrated that a convolutional network redesigned with transformer principles—large kernels, inverted bottleneck, and layer normalisation—matches Swin Transformer at comparable scale (44.4 vs. 44.5 mIoU). This result showed that self-attention is not strictly necessary for competitive dense segmentation and motivates treating CNN and transformer families as peers under controlled comparison rather than as successive generations.

Applied to inspection-specific tasks, the advantage of transformers over CNNs is inconsistent [30,31]. TransUNet improves over U-Net and DeepLabv3+ by at least 3.8% mIoU on concrete crack datasets [32], suggesting that global attention helps localise thin crack features. However, the CCSNet ablation [21] reveals that the transformer branch alone drops to 52.61% mIoU on a three-class concrete damage task, well below the CNN branch at 76.13%; only their combination reaches 82.74%. MixSegNet [33] and ETCS-Net [34] report similar patterns: hybrid CNN-Transformer designs outperform either family in isolation for crack and multiclass damage segmentation [35,36,37,38,39,40]. Because existing inspection papers use heterogeneous datasets and training protocols, the relative ranking of architecture families on a common viaduct benchmark remains an open empirical question. Using the Tokaido benchmark, Roy et al. [41] demonstrate the superior performance of transformer-based architectures, showing that the SegFormer (MiT-B5) model significantly outperforms conventional U-Net and Attention U-Net baselines in both structural component and damage detection tasks.

2.2. Bridge Component Segmentation and Multitask Inspection

Bridge element parsing originated as component classification [1] and evolved into pixel-level component segmentation [42,43,44], enabling spatial attribution of damage to specific structural elements. The dominant framework became multitask learning [45], in which a shared encoder is jointly supervised on component and damage targets, motivated by the expectation that component identity provides a structural prior for damage localisation.

The reported multitask evidence is quantitatively consistent. Zhang et al. [2] compare twelve multitask design variants on a 145-image Virginia DOT bridge dataset. The best variant improves bridge element mIoU from 82.89% to 85.48% and corrosion mIoU from 80.85% to 83.31% over single-task baselines while also reducing total training time. AECIF-Net [3] replaces simple feature sharing with bidirectional cross-task co-attention between task-specific relearning subnets. The model achieves 92.11% element mIoU and 87.16% defect mIoU on SBCIV, improving over naive multitask learning by 2.51% and 1.90% respectively. Yao et al. [6] validate joint component and crack segmentation on high-rise building images, achieving component F-scores above 92% and a crack F-score 73.51% under a synthetic-trained DeepLabv3+ model.

The studies above do not establish whether the reported performance gain obtained by multitask training arises from shared-encoder regularisation, from richer structural feature representations, or from explicit routing of component information into the damage branch. All three studies train a shared encoder jointly; none isolate the conditioning signal by providing component predictions as an explicit input to a damage-only encoder. Because the present study asks precisely this question, the multitask literature provides the motivating context but not the answer.

2.3. Synthetic Inspection Data and Domain Transfer

Manually annotating real inspection images is expensive and inconsistent: lighting, viewpoint, surface texture, and damage morphology vary widely, and expert labellers are scarce. The choice of imaging modality further affects segmentation performance [46,47]. Synthetic rendering offers a scalable alternative. The Tokaido dataset [4] renders Tokaido Shinkansen viaducts from 3D CG models under varied lighting, weather, and damage states, producing pixel-accurate component and damage annotations without manual labelling. Zarski et al. [5] apply a comparable approach to post-earthquake building inspection, achieving component mIoU 91.54% on synthetic test data. Crack and exposed-rebar IoU remain near 30% because label overlap and class imbalance are not resolved by the synthetic rendering alone, and the authors explicitly note that real-world fine-tuning is required. Yao et al. [6] generate 2000 photorealistic building images using a physics-based crack plugin, reaching component F-scores above 92% and crack F-score 73.51%, with the synthetic model successfully identifying over 96% of real field images. Rahman et al. [48] target bridge damage transfer directly but do not report standardised mIoU transfer metrics. Addressing the domain shift inherent in heritage structures, Fallahy and Rezazadeh [49] introduce MARBLE-DA, a label-free unsupervised domain-adaptation framework that combines Sobel-edge preprocessing with adaptive batch normalisation to enable robust crack detection in masonry without requiring target-domain annotations.

These studies share a common limitation: the magnitude of the synthetic-to-real performance drop is reported inconsistently, measured on private test sets, or not reported at all. dacl10k [15,50], a 9920-image real bridge-inspection benchmark with 12 damage and six component classes, provides the standardised evaluation domain needed to measure transfer consistently, but dacl10k has not previously been used as the target for Tokaido-trained models. The dacl10k taxonomy covers 12 damage classes with a different labelling convention from Tokaido’s two-class damage scheme (concrete surface damage; ExposedRebar); mapping the two taxonomies would require annotation re-labelling beyond the scope of this study. The 16 hand-annotated real viaduct photographs used here provide a smaller but taxonomy-compatible transfer target. Promptable foundation models offer a complementary reference point. GroundingDINO + SAM [16,17] produces text-prompted segmentation without task-specific training and is used here as a zero-shot lower bound. Parameter-efficient foundation-model adaptation is also viable: low-rank adaptation (LoRA) of SAM achieves mask average precision ($A{P}_{\mathrm{mask}}$) $=0.703$ and mIoU $=0.868$ on a masonry crack dataset [51], demonstrating that fine-tuning can close the gap to task-specific segmentation. Fine-tuned adaptation and zero-shot deployment differ fundamentally in data requirements, however, and the zero-shot protocol is the appropriate baseline for measuring how much task-specific synthetic training contributes.

Collectively, the existing literature establishes that hybrid CNN-Transformer designs outperform single-architecture baselines for damage segmentation, that multitask bridge inspection is broadly beneficial, and that synthetic environments can support inspection-relevant segmentation. Three questions remain unresolved under controlled conditions: whether architecture family determines performance on a standardised viaduct benchmark, whether explicit component-label conditioning provides damage segmentation gains beyond shared-encoder supervision, and what the synthetic-to-real component segmentation gap is relative to a training-free zero-shot baseline on real viaduct photographs.

3. Methodology

This section defines the segmentation tasks, implemented model families, component-aware conditioning variants, and training configuration used throughout the benchmark reported in the present paper. The study is both a controlled architecture benchmark and an ablation study design: standard segmentation architectures are compared under the same conditions, while the component-aware variants constitute the conditioning interventions used to test whether structural component information helps damage localisation.

3.1. Task Formulation and Label Spaces

Each input image is a red–green–blue (RGB) viaduct rendering $x\in {\mathbb{R}}^{3\times H\times W}$, resized and cropped to $H\times W=512\times 896$ before being passed to the network. Two dense prediction tasks are considered. The structural component task predicts one of five labels at each pixel: nonbridge, slab, beam, column, or nonstructural. The last class, nonstructural, aggregates minor structural elements (rail, sleeper, and other non-load-bearing parts); the merging uses zero-based indices after label conversion (see Section 3.2). The damage task predicts one of three labels: nondamage, concrete, or ExposedRebar. The single-task models output one logit tensor $z\in {\mathbb{R}}^{C\times H\times W}$ for the selected task, with $C=5$ for component segmentation and $C=3$ for damage segmentation.

Performance is optimised and monitored using foreground mean Intersection-over-Union (mIoU), denoted ${\mathrm{mIoU}}_{\mathrm{main}}$, where Intersection-over-Union (IoU) is computed per class before averaging. This metric excludes the dominant background class, nonbridge for component segmentation and nondamage for damage segmentation, so that model selection is driven by the structural and damage classes that determine inspection usefulness. The same foreground convention is used by the validation scheduler and by the checkpoint criterion.

3.2. Dataset

Both tasks are trained and evaluated on the Tokaido dataset [4], a synthetic photorealistic collection of 7575 viaduct images rendered from three-dimensional bridge models using the open-source Blender3D modelling tool (version 4.3.2; Blender Foundation, Amsterdam, The Netherlands). A viaduct is a multi-span elevated bridge structure, typically used to carry railways or roads over valleys or urban areas. The images are representative of UAV-acquired aerial imagery and come with pixel-accurate ground-truth annotations stored as unsigned-integer label images. Each sample includes segmentation masks covering structural object types—slab (the flat horizontal deck that forms the roadway or rail bed), beam (longitudinal load-bearing members that span between supports), column (vertical pillars that carry the beam loads to the ground), rail and sleeper (railway track components), and other minor elements—as well as two damage categories: concrete surface damage (cracking and spalling visible on the surface) and exposed rebar (severe deterioration where steel reinforcing bars have become visible due to cover loss or corrosion).

A validity flag stored in the file-list CSV is used to exclude low-quality or incomplete samples prior to any split; files whose label images are missing from disk are additionally filtered out. Figure 1 shows representative examples of the input images and their ground-truth annotations.

Figure 1. Representative samples from the Tokaido synthetic dataset (test split). Each row shows the RGB image (left), component ground-truth mask (centre; slab = red, beam = green, column = blue, nonstructural = yellow-orange, nonbridge = dark grey), and damage ground-truth mask (right; concrete = red, ExposedRebar = green, and nondamage = dark grey). Rows 1–3 are selected to show abundant damage pixels from both foreground classes; rows 4–6 cover all structural component classes. Outside the selected rows, damage pixels are typically sparse and localised on structural surfaces, reflecting the severe class imbalance addressed by the imbalance-aware loss functions.

Class Indexing and Label Merging

Raw label images store class indices as unsigned integers starting at 1; the preprocessing pipeline subtracts 1 to obtain zero-based indices in the range 0–7 before merging. For the component task, zero-based source classes 5, 6, and 7—corresponding to raw classes 6 (rail), 7 (sleeper), and 8 (other minor elements)—are merged into the nonstructural class at zero-based index 4. Raw class 5 already maps to nonstructural and is not a merge source. The merge reduces the eight raw component labels to five training classes.

3.3. Preprocessing and Augmentation

All images follow a common preprocessing pipeline before tensor conversion. For training, the image is first scaled with LongestMaxSize so that the longer side matches the configured training size and is then padded with zeros using PadIfNeeded until the $512\times 896$ canvas is obtained. A random $512\times 896$ crop is then extracted. Training augmentation consists of horizontal flipping with probability $0.5$, random brightness–contrast perturbation with probability $0.3$, and Gaussian blur with probability $0.15$, followed by normalisation and ToTensorV2. For non-augmented evaluation transforms, the same longest-side scaling, zero padding, normalisation, and tensor conversion are used without random crop or photometric augmentation. The same resize–pad–normalise convention is used for both component and damage inputs, ensuring that task differences arise from labels and model heads rather than from preprocessing.

3.4. Single-Task Segmentation Architectures

The single-task benchmark evaluates seven segmentation architectures under the same task interface. Three models use segmentation_models_pytorch decoders with an ImageNet-pretrained timm-efficientnet-b3 encoder: U-Net [7], UNet++ [18], and DeepLabV3+ [8]. U-Net provides the basic encoder–decoder baseline with skip connections; UNet++ adds nested skip pathways to reduce the semantic gap between encoder and decoder features; and DeepLabV3+ replaces the symmetric decoder with atrous spatial pyramid pooling and a lightweight refinement decoder. Their verified trainable parameter counts are 13.16 million, 13.63 million, and 11.68 million, respectively.

The remaining four models use custom wrappers around timm encoders and decoder heads. ViT-Seg uses vit_base_patch16_224 [52] with a UPerNet-style decoder [19]; intermediate transformer blocks are reshaped into spatial feature maps and fused at one-quarter input resolution. ConvNeXt-Seg uses convnext_base [11] as a four-stage feature encoder with the same UPerNet-style decoder. SegFormer uses a Pyramid Vision Transformer v2 (PVT v2) pvt_v2_b2 hierarchical encoder with a SegFormer-style multilayer-perceptron (MLP) decoder [10]; each stage is projected to a common channel dimension, upsampled to one-quarter resolution, concatenated, and fused before the segmentation head. Swin-UPerNet uses swin_base_patch4_window7_224 [9] with the UPerNet-style decoder. The verified trainable parameter counts are 93.12 million for ViT-Seg, 93.37 million for ConvNeXt-Seg, 25.97 million for SegFormer, and 92.12 million for Swin-UPerNet.

UPerNet Decoder

The shared UPerNet decoder used by ViT-Seg, ConvNeXt-Seg, and Swin-UPerNet consists of (i) per-level lateral $1\times 1$ convolutions that project all feature maps to a common $d=256$-dimensional space, (ii) a feature pyramid network (FPN) top-down pathway that adds upsampled deeper features to shallower ones, (iii) $3\times 3$ convolutional refinement on each level, and (iv) a final fusion step that upsamples all levels to a common resolution, concatenates them, and applies a further $3\times 3$ convolution. The segmentation head consists of a Conv-BN-ReLU block followed by a $1\times 1$ convolution that produces the C-channel logit map, which is upsampled to the original image resolution using bilinear interpolation.

3.5. Component-Aware Damage Segmentation

The component-aware multitask model uses a shared convnext_base encoder and task-specific UPerNet-style decoder branches. The component branch produces a feature map $h^c$ at one-quarter input resolution and component logits $z^c\in {\mathbb{R}}^{5\times H\times W}$; the damage branch produces a damage feature map $h^d$ and damage logits $z^d\in {\mathbb{R}}^{3\times H\times W}$. When conditioning is active, the damage feature map is concatenated with a component-derived conditioning tensor $q^c$ and passed through a single Conv-BN-ReLU fusion block before the damage head. The input channel count is $256+5=261$ for the hard-mask variant (one-hot over five component classes) and $256+256=512$ for the soft variants (full feature map); in both cases the output is projected back to 256 channels. Figure 2 shows the shared encoder, task branches, and optional conditioning route.

Figure 2. Component-aware damage segmentation architecture. The shared ConvNeXt encoder $E_{\theta }$ produces multi-scale features F fed to both branches. The component branch (top) computes feature map $h^c$ and logits $z^c$; the damage branch (bottom) fuses $h^d$ with the conditioning tensor $q^c$ before the damage head. The five conditioning modes differ in how $q^c$ is formed (see legend); in the damage-only and parallel-head modes, the dashed conditioning path is absent. The subscript $\theta$ denotes trainable parameters.

Five conditioning configurations are evaluated, corresponding to distinct hypotheses about the role of component context (Table 1):

Table 1. Summary of component-aware ablation variants. indicates the feature is present; × indicates it is absent.

Variant	Component Head	Fusion	Conditioning Signal	Gradient to Component Branch
damage-only	×	×	—	not applicable
parallel-heads	✔	×	—	component loss only
hard-mask	✔	✔	one-hot ($argmax{z}^c$)	none from damage †
soft-detach	✔	✔	$\mathrm{sg} \left({\mathbf{F}}_{\mathrm{comp}}\right)$	none from damage †
soft-full	✔	✔	${\mathbf{F}}_{\mathrm{comp}}$	yes

^† No gradient from the damage loss reaches the component branch; the component branch continues to be trained via the component segmentation loss.

• Damage-only: The component branch is absent; no fusion. Single-task damage baseline (93.37 M parameters).
• Parallel-heads: Both decoders are present and trained jointly on a combined loss, but the fusion module is absent. Tests whether shared multitask supervision alone benefits damage segmentation quality (99.17 M parameters).
• Hard-mask: $q^c$ is a discrete one-hot map derived from $argmax{z}^c$ at resolution $H/4\times W/4$, passed through the fusion module without gradient (non-differentiable argmax). A one-hot encoding assigns each pixel a binary vector with a single 1 in the position of the winning class and 0s everywhere else, discarding all probability information. Because argmax is not differentiable, no gradient flows through this conditioning path (99.78 M parameters).
• Soft-detach: $q^c=\mathrm{sg}\left({\mathbf{F}}_{\mathrm{comp}}\right)$, where $\mathrm{sg}(·)$ denotes the stop-gradient operator (implemented via .detach() in PyTorch), which passes the value forward normally but blocks gradient flow during backpropagation. The continuous component feature map is used as conditioning signal, but no gradient flows from the damage loss through the component path. This isolates the effect of the conditioning signal from gradient coupling (100.35 M parameters).
• Soft-full: $q^c={\mathbf{F}}_{\mathrm{comp}}$, without stop-gradient. The damage loss backpropagates through the fusion module into the component decoder and the shared encoder (100.35 M parameters).

A sixth variant, component-only, trains only the component branch in isolation to establish structural segmentation performance independently of the damage objective; it is not part of the conditioning ablation.

A ground-truth mask oracle is additionally evaluated as a perfect-information upper bound for the mask-conditioning mechanism: the damage branch is conditioned on the one-hot ground-truth component mask (nearest-neighbour downsampled to decoder resolution) at both training and test time, with the predicted component head disabled so that the upper bound is not confounded with auxiliary component supervision. An attention-based fusion variant is also evaluated, replacing the Conv-BN-ReLU fusion block with an attention gate over the component conditioning tensor, to test whether a mainstream attention-fusion mechanism changes the conditioning outcome.

The causal comparisons implied by these variants are part of the method definition. Damage-only versus parallel-heads tests whether auxiliary component supervision helps the shared representation when no component signal is received by the damage decoder. Parallel-heads versus hard-mask tests whether coarse predicted component identity helps damage segmentation. Parallel-heads versus soft-detach tests whether feature-level component information helps independently of extra gradient coupling. Soft-detach versus soft-full tests whether allowing for the damage objective to update the component path through the conditioning branch changes the damage solution.

3.6. Training Objective and Optimisation

Single-task models in the main benchmark are trained with one of four configured losses: cross-entropy plus Dice loss, weighted cross-entropy plus Dice loss, Tversky loss [53], and focal Tversky loss [54]. The preliminary loss ablation additionally evaluates plain cross-entropy (CE) and focal cross-entropy [55], giving six loss variants in that study. Other IoU-surrogate objectives such as the Lovász-Softmax loss were not evaluated [56]. Dice loss is used in multiclass mode from logits [57], and the Dice term receives weight $0.5$ in the composite cross-entropy–Dice objectives. The Tversky parameters are $\alpha =0.3$ and $\beta =0.7$. For focal Tversky, the verified focal exponent is $\gamma =2.0$ for the structural task and $\gamma =1.5$ for the damage and joint component-aware tasks. Weighted losses use per-class weights derived from training-set pixel frequencies. Let $f_c$ denote the fraction of training pixels assigned to class c. The per-class weight is

$$w_c=clip\left({\displaystyle \frac{{\tilde{w}}_c}{\overline{\tilde{w}}}},0.25,5.0\right),{\tilde{w}}_c={\displaystyle \frac{1}{\sqrt{f_c+\epsilon }}},$$

where $\epsilon ={10}^{-12}$ and $\overline{\tilde{w}}$ is the mean of ${\tilde{w}}_c$ over all classes. The square-root inverse downweights frequent classes without extreme amplification of rare ones; the $[0.25,5.0]$ clip prevents any single class from dominating the loss.

For joint component-aware training, the component and damage branches use the same configured loss type and hyperparameters, each with its own class weights. When both branches are present, the total objective is

$$\mathcal{L}={\mathcal{L}}_d(z^d,y^d)+{\lambda }_c{\mathcal{L}}_c(z^c,y^c),$$

where ${\mathcal{L}}_d$ and ${\mathcal{L}}_c$ are the damage and component segmentation losses, $y^d$ and $y^c$ are the corresponding label maps, and ${\lambda }_c=1.0$. In component-only mode, the damage term is absent and only ${\mathcal{L}}_c$ is optimised. In damage-only mode, the component term is absent and only ${\mathcal{L}}_d$ is optimised.

Checkpointing and Best-Model Selection

Validation is performed on the held-out validation split after every training epoch. The model state is saved whenever the foreground validation ${\mathrm{mIoU}}_{\mathrm{main}}$ strictly improves; after training, the best checkpoint is reloaded for final evaluation on the test set.

All models are trained with AdamW [58] and weight decay $0.01$. The learning rate is $3\times {10}^{-4}$ for the segmentation_models_pytorch (SMP)-based CNN single-task architectures and $1\times {10}^{-4}$ for the custom timm-based single-task architectures and joint component-aware variants. A ReduceLROnPlateau scheduler monitors validation ${\mathrm{mIoU}}_{\mathrm{main}}$ in maximisation mode with factor $0.5$, patience 3, and minimum learning rate ${10}^{-6}$. Single-task models are configured for 40 epochs and joint component-aware models for 60 epochs. The configured seeds are 1, 2, and 3. The original-split conditioning ablation is additionally extended to seeds 4 and 5 for the paired equivalence analysis (Section 5.4). Batch size is 4 for U-Net, UNet++, DeepLabV3+, ConvNeXt-Seg, SegFormer, and the joint component-aware variants and 2 for ViT-Seg and Swin-UPerNet. Mixed precision is enabled when supported by the training device.

4. Experiments

This section describes the dataset split, evaluation metrics, and experimental protocols. The benchmark covers seven architectures across both the damage and component segmentation tasks, each trained with four loss variants and three random seeds, yielding 168 nominal benchmark runs. The experiment log additionally contains repeated-attempt runs for a few configurations (notably the ConvNeXt-Seg and DeepLabV3+ damage cells); these reused-seed attempts are excluded from all reported tables, which use the three distinct seeds (1–3) per cell. Together with 86 ablation runs and 15 joint-ablation runs, the experiment log contains 295 completed training runs in total. The 86 ablation runs comprise four training-protocol factors (loss, encoder, resolution, and pretraining) each evaluated on both the damage and component tasks with 2–3 seeds per variant (68 runs), plus an 18-run cross-family loss confirmation that verified the loss ranking from U-Net/EfficientNet-B3 generalises to ConvNeXt-Seg and ViT-Seg before committing the recipe to the full benchmark; the 15 joint-ablation runs are five conditioning variants × 3 seeds. All experiments were executed on a single NVIDIA RTX A6000 (48 GB) graphics processing unit (GPU) (NVIDIA Corporation, Santa Clara, CA, USA) using the Compute Unified Device Architecture (CUDA) toolkit (version 12.4), with PyTorch (torch 2.6.0, torchvision 0.21.0), segmentation_models_pytorch 0.5.0, and timm 1.0.25 on Python 3.13. Residual GPU nondeterminism may prevent bit-exact reproduction of individual runs.

4.1. Dataset Split

All valid annotated images—those passing the dataset’s built-in validity filter—are partitioned into train, validation, and test sets at a 70/15/15 ratio using a fixed random seed (seed 123). The same split is applied identically to both tasks and to all experiment variants.

Because Tokaido renders multiple camera frames from each underlying three-dimensional viaduct scene, this random split places frames of the same scene in different subsets. To quantify and control the resulting leakage, we additionally construct a scene-disjoint (grouped) split: the 175 distinct scenes, identified by the case field of the file names, are partitioned at the same 70/15/15 ratio so that no scene contributes frames to more than one subset. Under the random split, all 175 component scenes and $95.4\%$ of damage scenes appear in more than one subset; the scene-disjoint split removes this overlap by construction. The component-aware ablation is reported under both protocols (Section 5.4).

4.2. Evaluation Metrics

All segmentation performance is reported using mean Intersection-over-Union (mIoU) over foreground classes (${\mathrm{mIoU}}_{\mathrm{main}}$), which excludes the dominant background class (nondamage for damage segmentation; nonbridge for component segmentation). The full-class mean (${\mathrm{mIoU}}_{\mathrm{all}}$), macro Dice coefficient, precision, recall, and per-class Intersection-over-Union (IoU) are reported as secondary metrics. All metrics are computed from a global confusion matrix accumulated over the entire test split; results across seeds are reported as mean ± standard deviation. Classes with zero ground-truth support in the evaluated split yield undefined values and are excluded from macro averages. Inference throughput in frames per second (FPS) is measured on the same CUDA-compatible GPU hardware used for training.

4.3. Ablation Study Design

Four training-protocol factors are ablated in isolation using U-Net with EfficientNet-B3 as the reference model, varying one factor at a time with all others fixed. Most factors use two seeds per variant; the pretraining factor uses three. The optimal setting from each factor is carried forward to the main benchmark. The four factors are: (1) loss function, (2) encoder family, (3) input resolution, and (4) ImageNet pretraining.

4.4. Oracle-Filter Analysis Design

The null conditioning result admits two interpretations: either component predictions are too noisy to carry useful information to the damage branch (hypothesis H1), or component layout is genuinely uninformative for damage (hypothesis H2). To distinguish between these hypotheses, a filter oracle evaluates damage segmentation on a subset of test images where component predictions are demonstrably reliable.

For each seed, the soft-detach model’s per-image foreground component mIoU (${\mathrm{mIoU}}_{\mathrm{main}}^{\mathrm{comp}}$, where the superscript comp denotes the component task) is computed on each test image. Images are partitioned into a reliable component subset (${\mathrm{mIoU}}_{\mathrm{main}}^{\mathrm{comp}}\ge \tau$) and its complement (the unreliable component subset) at two confidence thresholds, $\tau \in \{0.7,0.8\}$. Damage mIoU is then computed separately for each subset using pooled confusion matrices to match the training evaluation convention. The soft-detach and damage-only models are compared on the reliable component subset: if H1 were correct, the conditioned model should outperform the unconditioned model when component predictions are accurate.

4.5. Synthetic-to-Real Transfer and Zero-Shot Baseline

The three single-task ConvNeXt-Seg component checkpoints (seeds 1–3) from the main benchmark are evaluated on 16 real viaduct photographs with hand-annotated component masks, using the same resize–pad–normalise preprocessing pipeline and no test-time augmentation. Because real photographs are available for the component task only, damage transfer is not evaluated on real images; the reported domain gap is therefore specific to structural component segmentation. The real-image component annotations were created manually using the Image Labeler application in MATLAB R2025b (The MathWorks, Inc., Natick, MA, USA). Each image was annotated at the pixel level by a single annotator following the same structural component definitions as the synthetic Tokaido dataset (e.g., slab, beam, column, and nonstructural/background classes). Polygon-based manual delineation was used to generate the semantic segmentation masks, with visual inspection and correction performed before export. No automatic pre-segmentation or AI-assisted annotation was employed. Because the real-image dataset was intended as an exploratory benchmark rather than a fully validated reference dataset, no second independent annotation or inter-annotator agreement study was conducted.

To establish a training-free lower bound, the GroundingDINO + SAM pipeline (GroundedSAM) [16,17] is applied to the same 16 photographs in a zero-shot setting. GroundingDINO (version 0.4.0) receives text prompts describing each structural class and outputs bounding boxes; SAM generates pixel-level masks for each box. SAM ViT-H is used for the prompt ablation and main zero-shot comparison, and SAM ViT-H, ViT-L, and ViT-B are evaluated in the additional sensitivity sweep (SAM checkpoints sam_vit_h_4b8939, sam_vit_l_0b3195, and sam_vit_b_01ec64). Five prompt variants are evaluated and the best-performing prompt set is used for the comparative results reported in Section 5.6. As an additional sensitivity check, the best prompt set is re-evaluated with SAM ViT-H, ViT-L, and ViT-B under two GroundingDINO confidence settings: box/text thresholds $(0.35,0.25)$ and $(0.25,0.20)$. For each backbone/threshold pair, we also evaluate two bounding-box merge settings (no merge and non-maximum suppression) and two mask postprocessing settings (none and morphological closing with a $3\times 3$ kernel), giving 24 zero-shot structural runs.

5. Results

This section first presents the completed ablation studies that establish the best training recipe and then reports damage segmentation performance on the synthetic Tokaido test split for the available single-task baselines, examines whether component-aware joint modelling improves damage localisation, and finally assesses how well component segmentation transfers to real photographs.

Unless noted otherwise, values reported in the form $a\pm b$ in this section denote mean ± one standard deviation over repeated runs with different random seeds.

5.1. Ablation Studies

The four recipe-selection ablations (loss, encoder, resolution, and pretraining) use U-Net with EfficientNet-B3 as the controlled baseline model, varying one factor at a time while keeping all other settings fixed (512×896 resolution, ImageNet-pretrained encoder, and ce_dice loss). Most factors use two seeds per variant; the pretraining factor uses three. This single-architecture design is standard practice: the ablation findings establish the best training recipe, which is then applied uniformly to all seven architectures in the main benchmark—the ablation results themselves are not part of the architecture comparison.

5.1.1. Loss Function

Table 2 compares six loss functions on the damage task. Weighted cross-entropy plus Dice (weighted_ce_dice) achieves the highest foreground mIoU ($0.535$), followed by Tversky ($0.526$) and unweighted cross-entropy plus Dice (ce_dice, $0.523$). Focal variants together with plain cross-entropy (ce$=0.491$) rank the lowest (focal_tversky $=0.498$; focal_ce$=0.467$), likely because the focal exponent suppresses the already-dominant nondamage class while simultaneously reducing the gradient signal on the rare ExposedRebar class. Based on these results, weighted_ce_dice is adopted as the default loss for all subsequent single-task experiments. The $0.068$ gap between the best and the worst loss confirms that imbalance management—not architecture choice—is the primary tuning lever for this task.

Table 2. Ablation: loss comparison (damage).

Variant	mIoUmain	mIoUall	Dice	Precision
weighted_ce_dice	0.535	0.688	0.797	0.816
tversky	0.526	0.681	0.792	0.793
ce_dice	0.523	0.679	0.790	0.846
focal_tversky	0.498	0.662	0.775	0.768
ce	0.491	0.659	0.771	0.852
focal_ce	0.467	0.642	0.756	0.838

5.1.2. Class-Balanced Sampling

Beyond loss weighting, we test whether resampling improves the rare ExposedRebar class. Two samplers—class-balanced sampling and rare-class oversampling—are compared against the weighted-loss recipe over three seeds. Class-balanced sampling reaches $0.5615\pm 0.0006$ damage ${\mathrm{mIoU}}_{\mathrm{main}}$ with ExposedRebar IoU $0.523$, and rare-class oversampling reaches $0.5626\pm 0.0047$ with ExposedRebar IoU $0.529$; both lie within seed-level variation of the weighted-loss damage baseline (≈0.560 mIoU; ExposedRebar IoU $\approx 0.53$). Resampling therefore does not yield a meaningful gain over loss weighting for the rare class, and the weighted-loss recipe is retained.

5.1.3. Encoder Backbone

Table 3 shows a clear capacity gradient across SMP-compatible encoders. EfficientNet-B5 attains the highest foreground mIoU ($0.547$), followed by B3 ($0.526$), B0 ($0.488$), and ResNet-50 ($0.418$). The $0.129$ gap between EfficientNet-B5 and ResNet-50 confirms that encoder width and depth matter substantially for this task. EfficientNet-B3 is retained as the default in the main benchmark to balance accuracy and computational cost; B5 is noted as the stronger option when inference latency and memory are not a constraint. The consistent ranking across encoders confirms that representational capacity is the dominant factor, independent of decoder design or loss function.

Table 3. Ablation: encoder comparison (damage).

Variant	mIoUmain	mIoUall	Dice	Precision
efficientnet_b5	0.547	0.696	0.803	0.850
efficientnet_b3	0.526	0.681	0.791	0.842
efficientnet_b0	0.488	0.656	0.769	0.823
resnet50	0.418	0.609	0.724	0.780

5.1.4. Input Resolution

Table 4 reveals a monotonic improvement with resolution: foreground mIoU increases from $0.398$ at $256\times 448$ to $0.549$ at $640\times 1120$, a gain of $0.151$ over the tested range. The sharpest step occurs between $384\times 672$ ($0.464$) and $512\times 896$ ($0.536$), suggesting that crack and rebar textures are partially lost below the $512\times 896$ threshold. The standard benchmark resolution is $512\times 896$. This sensitivity underscores the fact that fine-scale crack and rebar features require sufficient spatial detail, a baseline condition that the full benchmark satisfies.

Table 4. Ablation: resolution comparison (damage).

Variant	mIoUmain	mIoUall	Dice	Precision
640 × 1120	0.549	0.697	0.805	0.857
512 × 896	0.536	0.689	0.797	0.839
384 × 672	0.464	0.640	0.754	0.822
256 × 448	0.398	0.595	0.711	0.763

5.1.5. ImageNet Pretraining

Table 5 quantifies the impact of encoder initialisation. Pretrained weights yield foreground mIoU $0.519$ versus $0.264$ from random initialisation—a relative drop of almost $50\%$. Figure 3 confirms visually that the scratch model produces fragmented, noisy predictions with poor class separation, while the pretrained model already localises damage reliably. Pretrained encoders are used throughout all experiments.

Table 5. Ablation: pretrained vs. scratch (damage).

Variant	mIoUmain	mIoUall	Dice	Precision
pretrained	0.519	0.677	0.787	0.826
scratch	0.264	0.504	0.605	0.638

Figure 3. ImageNet pretraining versus random initialisation (U-Net, EfficientNet-B3, $512\times 896$). Columns: input image, ground-truth, pretrained prediction, and scratch prediction. Classes: concrete = red, ExposedRebar = green, and nondamage = dark grey. Training from scratch produces fragmented, noisy outputs compared to the pretrained model, reflecting the near-$50\%$ foreground mIoU gap ($0.519$ vs. $0.264$).

5.2. Damage Segmentation on the Synthetic Test Split

All seven architectures are trained using the recipe identified in the ablation studies: weighted_ce_dice loss, EfficientNet-B3 encoder for the three SMP-based models (U-Net, UNet++, and DeepLabV3+), $512\times 896$ input resolution, and ImageNet-pretrained encoder weights. Table 6 reports mean test metrics per architecture (weighted CE + Dice loss; seeds 1–3). SegFormer achieves the highest foreground ${\mathrm{mIoU}}_{\mathrm{main}}$ ($0.569\pm 0.013$) and macro Dice ($0.815$), closely followed by ConvNeXt-Seg ($0.564\pm 0.009$; $0.812$). The two lightweight EfficientNet-based decoders, UNet++ and UNet, form a second tier in the range $0.533$–$0.538$. DeepLabV3+ is notably weaker ($0.475$) despite using the same EfficientNet-B3 encoder, reflecting the limitations of its atrous spatial pyramid pooling (ASPP)-based decoder for fine-grained damage textures. ViT-Seg underperforms all CNN models ($0.362$) in spite of its large parameter count, consistent with the known sensitivity of plain ViT architectures to dataset size and high-resolution fine-scale features. Swin-UPerNet fails to converge meaningfully on the damage task (${\mathrm{mIoU}}_{\mathrm{main}}=0.021\pm 0.011$): in all 12 runs, the model collapses to predicting the nondamage class almost exclusively, with ExposedRebar IoU $\approx 0.000$. This behaviour is consistent with the known instability of the Swin-UPerNet training pipeline under strong class imbalance, even with weighted losses.

Table 6. Damage segmentation benchmark on the Tokaido synthetic test split (weighted CE + Dice loss mean ± standard deviation over three distinct seeds (1–3); n denotes the number of independent seeds per cell). Params denotes trainable parameter count in millions (M); Prec and Rec denote precision and recall. ^† Swin-UPerNet collapses to predicting the nondamage class in all runs; ExposedRebar IoU ≈ 0.

Model	Encoder	Params	n	mIoUmain	mIoUall	Dice	Prec	Rec	FPS
SegFormer	PVT-V2-B2	26.0 M	3	$0.569\pm 0.013$	0.710	0.815	0.817	0.814	19.5
ConvNeXt-Seg	ConvNeXt-B	93.4 M	3	$0.564\pm 0.009$	0.707	0.812	0.824	0.801	18.9
UNet++	EfficientNet-B3	13.6 M	3	$0.538\pm 0.011$	0.690	0.798	0.838	0.766	17.8
UNet	EfficientNet-B3	13.2 M	3	$0.533\pm 0.008$	0.686	0.795	0.816	0.777	19.6
DeepLabV3+	EfficientNet-B3	11.7 M	3	$0.475\pm 0.012$	0.647	0.761	0.785	0.741	20.3
ViT-Seg	ViT-B/16	93.1 M	3	$0.362\pm 0.013$	0.571	0.686	0.743	0.645	14.1
Swin-UPerNet †	Swin-B	92.1 M	3	$0.021\pm 0.011$	0.341	0.357	0.376	0.352	13.7

Table 7 reports the mean per-class IoU for each architecture, over seeds 1–3 (weighted CE + Dice; run counts match Table 6). The nondamage class is near-perfectly solved (≈0.99) across all converging models; the remaining challenge lies entirely in the two foreground classes. Concrete is consistently the easier foreground class ($0.359$–$0.605$), while ExposedRebar is harder ($0.366$–$0.535$) because rebar pixels are sparse and visually thin. The gap between concrete and ExposedRebar is small for strong models (SegFormer: $0.602$ vs. $0.535$) and collapses for ViT-Seg ($0.359$ vs. $0.366$), where neither class is well localised. Swin-UPerNet effectively predicts zero ExposedRebar IoU in all runs, confirming its collapse to the majority class.

Table 7. Per-class IoU breakdown on the Tokaido synthetic test split (weighted CE + Dice loss; mean ± standard deviation over three distinct seeds (1–3)). Nondamage standard deviation is <0.002 for all models except Swin-UPerNet ($0.003$) and is omitted for brevity. ^† Swin-UPerNet collapses to the nondamage class in all runs.

Model	mIoUmain	Nondamage	Concrete	ExposedRebar
SegFormer	$0.569\pm 0.013$	0.994	$0.602\pm 0.018$	$0.535\pm 0.010$
ConvNeXt-Seg	$0.564\pm 0.009$	0.994	$0.605\pm 0.012$	$0.522\pm 0.006$
UNet++	$0.538\pm 0.011$	0.993	$0.570\pm 0.007$	$0.507\pm 0.016$
UNet	$0.533\pm 0.008$	0.993	$0.556\pm 0.006$	$0.510\pm 0.014$
DeepLabV3+	$0.475\pm 0.012$	0.992	$0.512\pm 0.019$	$0.438\pm 0.011$
ViT-Seg	$0.362\pm 0.013$	0.990	$0.359\pm 0.021$	$0.366\pm 0.008$
Swin-UPerNet †	$0.021\pm 0.011$	0.982	$0.042\pm 0.021$	$0.000\pm 0.000$

Swin-UPerNet Remediation and Capacity Context

To test whether the Swin-UPerNet collapse reflects optimisation rather than architecture, nine remediation runs were trained with separately tuned class weighting and recall-oriented losses (aggressive weighting, recall–Tversky, and focal-recall-Tversky). None recovered: the best damage ${\mathrm{mIoU}}_{\mathrm{main}}$ across all nine runs was $0.0082$, and no run exceeded $0.10$. Figure 4 contrasts the final damage performance of all nine runs with the band reached by the six converging architectures (a) and the validation-mIoU training dynamics per strategy (b): no strategy or seed exceeds $0.01$ at any epoch, far below the $0.36$–$0.57$ converging-model band, so none escapes the majority-class solution. The gradient-norm traces (Figure 5) remain non-trivial throughout, indicating a stuck majority-class optimum rather than a vanishing-gradient failure. Because the same implementation and recipe reach $0.832$ component ${\mathrm{mIoU}}_{\mathrm{main}}$ (Table 8), the failure is specific to the highly imbalanced damage task rather than a general inability to learn structural features. We note that Swin-UPerNet and ViT-Seg use a smaller batch size (2 versus 4) than the other models; the same implementation’s success on the component task argues against a general implementation failure, and the failed remediation runs show that loss and learning-rate retuning do not recover the model, so the collapse is best attributed to the extreme class imbalance of the damage task.

Figure 4. Swin-UPerNet damage-task remediation (three loss strategies × three seeds; nine runs). (a) Final damage ${\mathrm{mIoU}}_{\mathrm{main}}$ of every run, grouped by remediation strategy (individual seeds shown as points), against the $0.36$–$0.57$ band reached by the six converging architectures (Table 6) and the collapse zone ($\mathrm{mIoU}<0.01$). (b) Validation-mIoU training dynamics; one mean ± range band per strategy across its three seeds. No strategy or seed exceeds $0.01$ (best $0.0082$); every run collapses to the majority nondamage class, far below the converging models.

Figure 5. Gradient-norm traces for the Swin-UPerNet remediation runs, where each colour denotes one remediation strategy (see the in-plot legend) and the shaded band shows the mean ± range over that strategy’s three seeds. The norms remain non-trivial throughout training—ruling out a vanishing-gradient cause—while the model nonetheless stays at the majority-class solution. These traces are available only for the remediation reruns, not for the original collapsed benchmark runs.

Table 8. Component segmentation benchmark on the Tokaido synthetic test split (weighted CE + Dice loss; mean ± standard deviation over seeds 1–3). Params denotes trainable parameter count in millions (M); Prec and Rec denote precision and recall. ${\mathrm{mIoU}}_{\mathrm{main}}$ excludes the nonbridge background class.

Model	Encoder	Params	n	mIoUmain	mIoUall	Dice	Prec	Rec	FPS
ConvNeXt-Seg	ConvNeXt-B	93.4 M	3	$0.957\pm 0.001$	0.964	0.981	0.979	0.984	19.5
SegFormer	PVT-V2-B2	26.0 M	3	$0.951\pm 0.001$	0.959	0.979	0.976	0.981	24.6
UNet++	EfficientNet-B3	13.6 M	3	$0.948\pm 0.005$	0.957	0.978	0.976	0.980	24.5
UNet	EfficientNet-B3	13.2 M	3	$0.947\pm 0.001$	0.956	0.977	0.975	0.980	27.6
DeepLabV3+	EfficientNet-B3	11.7 M	3	$0.946\pm 0.002$	0.955	0.977	0.975	0.979	27.6
ViT-Seg	ViT-B/16	93.1 M	3	$0.937\pm 0.001$	0.947	0.973	0.972	0.974	16.8
Swin-UPerNet	Swin-B	92.1 M	3	$0.832\pm 0.014$	0.857	0.922	0.918	0.926	16.4

The strongest individual checkpoint is SegFormer with Tversky loss (seed 3), reaching ${\mathrm{mIoU}}_{\mathrm{main}}=0.583$, ${\mathrm{mIoU}}_{\mathrm{all}}=0.720$, macro Dice $=0.823$, and $19.2$ FPS, with class-wise IoUs of $0.994$ (nondamage), $0.607$ (concrete), and $0.559$ (ExposedRebar). The best weighted CE + Dice run overall is SegFormer seed 3 at foreground mIoU $0.577$, followed closely by SegFormer seed 2 ($0.576$) and ConvNeXt-Seg seed 3 ($0.574$).

Qualitative results are shown in Figure 6.

Figure 6. Qualitative damage segmentation results for ConvNeXt-Seg (weighted CE + Dice; seed 3). Images are selected from the synthetic test split to show substantial damage from both foreground classes. Columns: input image, ground-truth mask, predicted mask, and prediction–ground-truth (GT) overlay. Classes: concrete = red, ExposedRebar = green, and nondamage = dark grey. The model closely follows the ground truth for both damage classes while maintaining near-perfect nondamage accuracy.

5.3. Component Segmentation on the Synthetic Test Split

Table 8 reports component segmentation metrics on the synthetic test split for all seven architectures (weighted CE + Dice loss; seeds 1–3). Unlike the damage task, all CNN-based and transformer-based models converge successfully on the component task, achieving foreground ${\mathrm{mIoU}}_{\mathrm{main}}$ between $0.937$ and $0.957$. ConvNeXt-Seg leads at $0.957\pm 0.001$, followed by SegFormer ($0.951\pm 0.001$); the three EfficientNet-based decoders cluster tightly in the $0.946$–$0.948$ range. ViT-Seg reaches $0.937\pm 0.0003$, and Swin-UPerNet, while far behind on damage, still attains $0.832\pm 0.014$ on components—indicating that its failure on the damage task is specific to the highly imbalanced damage label distribution rather than a general inability to learn structural features.

These high component scores motivate the joint-architecture ablation in Section 5.4: a component branch trained on this task provides reliable structural-layout estimates (${\mathrm{mIoU}}_{\mathrm{main}}^{\mathrm{comp}}\approx 0.92$–$0.96$), making the question of whether such estimates help damage detection well-posed and testable.

5.4. Component-Aware Joint Architecture Ablation

We explicitly test the hypothesis that component conditioning improves damage localisation beyond what a shared encoder and auxiliary component supervision already provide. All five conditioning variants (damage-only, parallel-heads, hard-mask, soft-detach, and soft-full; see Section 3.5) use the ConvNeXt-Seg architecture (ConvNeXt-Base encoder; UPerNet decoder), which achieved the highest component mIoU among single-task models. Table 9 reports the test-set damage and component metrics for all five variants, averaged over three seeds on the original random split. Under this split, the table does not support the initial hypothesis: all five variants remain within a narrow damage-performance band, even though prior bridge-inspection studies report gains from cross-task interaction in other settings [2,3]. As shown below, however, this outcome is sensitive to scene leakage in the split, so we report it alongside a leakage-controlled scene-disjoint protocol rather than as a universal counterexample.

Table 9. Component-aware ablation results on the Tokaido synthetic test set (mean ± standard deviation over seeds 1–3). Superscripts dmg and comp denote damage and component tasks, respectively. ^† Variant has no component head; component metrics are not applicable.

Variant	${\mathbf{mIoU}}_{\mathbf{main}}^{\mathbf{dmg}}$	Concrete IoU	ExposedRebar IoU	${\mathbf{mIoU}}_{\mathbf{main}}^{\mathbf{comp}}$
damage-only †	$0.5600\pm 0.0060$	$0.5882$	$0.5319$	—
parallel-heads	$0.5535\pm 0.0055$	$0.5884$	$0.5187$	$0.9229\pm 0.0065$
hard-mask	$0.5574\pm 0.0045$	$0.5936$	$0.5212$	$0.9262\pm 0.0036$
soft-detach	$0.5609\pm 0.0086$	$0.5976$	$0.5242$	$0.9231\pm 0.0021$
soft-full	$0.5545\pm 0.0112$	$0.5909$	$0.5181$	$0.9270\pm 0.0027$

Qualitative predictions across the five variants are shown in Figure 7, and component segmentation results are illustrated in Figure 8. Two observations stand out. First, all five variants attain very similar damage foreground mIoU, ranging from $0.553$ to $0.561$—a spread of $0.008$. The best value, soft-detach at $0.5609\pm 0.0086$, exceeds the damage-only baseline ($0.5600\pm 0.0060$) by only $0.0009$, which is an order of magnitude smaller than the per-variant seed standard deviations ($0.0045$–$0.0112$). Extending the comparison to five paired seeds and applying a paired bootstrap with a two-one-sided-tests (TOST) equivalence procedure against a $\pm 0.010$ mIoU margin, only hard-mask conditioning is statistically equivalent to the damage-only baseline ($\Delta =-0.0007$); soft-detach ($\Delta =-0.0011$), soft-full ($\Delta =-0.0064$), and parallel-heads ($\Delta =-0.0070$, significantly below the baseline) remain inconclusive, so no conditioning variant shows a statistically supported damage gain over the baseline on this split. An attention-based fusion variant, added to broaden the tested fusion family beyond concatenation-based conditioning, likewise shows no gain on this split ($0.5510\pm 0.0074$, $\Delta =-0.0090$ versus damage-only). Second, the component branch delivers consistently strong component segmentation across all four variants that include it, with ${\mathrm{mIoU}}_{\mathrm{main}}^{\mathrm{comp}}$ between $0.916$ and $0.930$. Joint training therefore achieves high-quality component segmentation, and the null result cannot be explained by failure of the component task.

Figure 7. Qualitative comparison of damage predictions across all five joint-model ablation variants (seed 1) on representative synthetic test images. Columns: ground-truth damage mask, damage-only, parallel-heads, hard-mask, soft-detach, and soft-full. All five variants produce visually similar outputs, consistent with the near-identical quantitative scores in Table 9. Classes: concrete = red, ExposedRebar = green, and nondamage = dark grey.

Figure 8. Qualitative component segmentation results for the joint soft-detach model (seed 1) on representative synthetic test images. Columns: input image, ground-truth component mask, predicted component mask, and prediction–ground-truth (GT) overlay. Classes: slab = red, beam = green, column = blue, nonstructural = yellow-orange, and nonbridge = dark grey. The model accurately delineates all major structural elements, achieving ${\mathrm{mIoU}}_{\mathrm{main}}^{\mathrm{comp}}\approx 0.92$ on the test split.

5.4.1. Scene-Disjoint (Leakage-Controlled) Protocol

The random split distributes multiple camera frames of each viaduct scene across the train, validation, and test sets: all 175 component scenes ($100\%$) and $167/175$ damage scenes ($95.4\%$) appear in more than one subset, and 142 test scenes are also present in training. We therefore re-run the full ablation under the scene-disjoint split of Section 4.1 (three seeds). Under this protocol the ordering reverses: every component-aware variant has a higher mean damage mIoU than the damage-only baseline ($0.5853\pm 0.0056$), with soft-detach at $0.6046\pm 0.0004$ ($\Delta =+0.0193$) and parallel-heads at $0.6029\pm 0.0021$ ($\Delta =+0.0176$); for these two variants, every seed exceeds every damage-only seed, and the same direction holds for the rare ExposedRebar class (soft-detach $0.5917$ versus $0.5721$). These grouped-split runs use three seeds without a paired significance analysis, so we report the direction and magnitude of the trend but make no significance claim; a paired analysis matching the random-split procedure is required to confirm it. The absolute scores are not comparable across protocols because the two test sets contain different scenes, and component segmentation remains strong under the scene-disjoint split (${\mathrm{mIoU}}_{\mathrm{main}}^{\mathrm{comp}}$ between $0.916$ and $0.934$), so the trend is not an artefact of degraded component branches.

5.4.2. Perfect-Mask Upper Bound and Damage Component Dependence

To bound the conditioning effect independently of component-branch accuracy, a ground-truth mask oracle (Section 3.5) conditions the damage branch on the one-hot ground-truth component mask. On the random split it reaches $0.5702\pm 0.0052$ damage mIoU, $+0.0102$ over the damage-only baseline and above the best predicted-mask variant: even perfect component masks yield only a small gain on this split. A pixel-level dependence analysis explains why this gain is bounded. Damage is almost entirely confined to columns—on the scene-disjoint test split, $5.9\%$ of column pixels are concrete damage and $0.7\%$ are ExposedRebar, while slab, beam, nonbridge, and nonstructural pixels are effectively all nondamage—yet damage is sparse even within columns, so the normalised mutual information between component and damage labels is low ($0.071$ on the grouped split; $0.061$ on the random split). Component identity is thus a strong indicator of where damage can occur but a weak indicator of whether a given pixel is damaged.

Relative to the damage-only variant ($19.5$ ms per image, $51.4$ FPS), adding the component branch increases damage inference cost to $26.9$–$28.0$ ms per image ($35.8$–$37.2$ FPS), i.e., a 38–$44\%$ latency increase without a measurable accuracy gain.

In practical terms, the component-aware variants first estimate which structural component each pixel belongs to (for example, slab, beam, or column) and then allow the damage branch to use that information. On the random split these variants do not outperform the damage-only baseline, indicating that under that protocol damage localisation is driven mainly by local surface appearance cues—cracking, spalling texture, and exposed rebar—that a damage-only model already captures. The small positive trend under the scene-disjoint protocol, together with the concentration of damage on columns, is consistent with component context contributing mainly when the model must generalise to unseen scenes; we treat this as a plausible interpretation rather than an established mechanism, given the three-seed grouped estimate. Read in the context of the bridge-inspection literature, the result is therefore best understood as a protocol- and dataset-dependent effect rather than a universal absence of benefit.

5.4.3. Oracle-Filter Analysis

To test whether imperfect component predictions are masking a genuine conditioning benefit, the test set is re-evaluated on subsets where the soft-detach component branch is reliable. At threshold $\tau =0.8$, the reliable component subset contains 138–140 images depending on seed. On this subset, soft-detach achieves ${\mathrm{mIoU}}_{\mathrm{main}}^{\mathrm{dmg}}=0.522\pm 0.016$, compared with $0.526\pm 0.010$ for damage-only and $0.523\pm 0.017$ for parallel-heads. The same pattern holds at the looser threshold $\tau =0.7$ (soft-detach $0.539\pm 0.010$, damage-only $0.544\pm 0.019$, and soft-detach minus damage-only difference $=-0.005$). On the random split, the null result therefore persists even when conditioning quality is explicitly filtered, arguing against unreliable component predictions as the explanation for the lack of improvement. Even with reliable component predictions, we find no statistically meaningful improvement in damage localisation.

Notably, all three variants show the same damage-score drop on the reliable component subset relative to its complement ($\Delta [\mathrm{reliable}-\mathrm{unreliable}]\approx -0.05$ to $-0.06$). Because this drop is identical across the conditioned and unconditioned variants, it is a subset-composition effect rather than a consequence of conditioning. We caution specifically against attributing it to damage quantity: the reliable and unreliable component subsets contain very similar amounts of damage (mean foreground-damage fraction $1.7\%$ versus $1.8\%$ at $\tau =0.8$, with essentially zero correlation between component-prediction quality and damage fraction, $r=-0.01$). The two subsets are instead distinguished by component layout—reliable component scenes are frontal, single-component (typically column) views that the model segments almost perfectly, whereas unreliable component scenes are oblique, multi-component views—not by how much damage they contain. Readers should therefore not interpret the lower subset-level damage scores as evidence that conditioned models perform worse on easier images. Figure 9 shows representative examples: scenes from both subsets carry comparably sparse damage while differing markedly in component-layout complexity.

Figure 9. Representative scenes with reliable versus unreliable component predictions for the soft-detach model, i.e., the reliable component subset (${\mathrm{mIoU}}_{\mathrm{main}}^{\mathrm{comp}}\ge 0.80$) and its complement (qualitative examples, not a statistical sample). Columns: input; component ground truth; component prediction; and damage ground truth. Component colours—slab = red, beam = green, column = blue, nonstructural = orange, and nonbridge = dark grey; damage colours—concrete = red, ExposedRebar = green, and nondamage = dark grey. Each row gives its component foreground mIoU and damage-pixel fraction. Top (reliable): frontal, single-component scenes segmented almost perfectly. Bottom (unreliable): oblique, multi-component scenes the model fails to segment. Both groups carry comparably sparse damage (subset means $1.7\%$ vs. $1.8\%$; correlation between component-prediction quality and damage fraction $\approx 0$), so the split reflects component-layout complexity, not damage content.

5.5. Component Segmentation on Real Photographs

To assess synthetic-to-real transfer of the component model, the three single-task ConvNeXt-Seg component checkpoints from seeds 1–3 are evaluated on the 16 real viaduct photographs with hand-annotated component masks. Inference uses the same resize–pad–normalise pipeline as the synthetic evaluation; results are reported in Table 10 as global pooled IoU (matching the training metric).

Table 10. Component segmentation on real photographs (ConvNeXt-Seg, seeds 1–3, and weighted CE + Dice; same three single-task checkpoints as Table 8; global pooled IoU over 16 images). Rows “Seed 1”–“Seed 3” are individual checkpoints; the final ${\mathrm{mIoU}}_{\mathrm{main}}$ entry is mean ± standard deviation across seeds. The nonbridge class is omitted from ${\mathrm{mIoU}}_{\mathrm{main}}$ because its support is small and arises primarily from resize–pad preprocessing in the evaluation pipeline.

	Nonbridge	Slab	Beam	Column	Nonstructural	${\mathbf{mIoU}}_{\mathbf{main}}$
Seed 1	0.010	0.456	0.409	0.598	0.036	0.374
Seed 2	0.011	0.727	0.447	0.755	0.056	0.496
Seed 3	0.009	0.505	0.424	0.643	0.034	0.402
Mean	0.010	0.562	0.427	0.665	0.042	$0.424\pm 0.064$

The same three single-task ConvNeXt-Seg checkpoints achieve $0.957\pm 0.001$ on the synthetic test split (Table 8), giving an absolute domain gap of $0.533$ ($0.957\pm 0.001\to 0.424\pm 0.064$). Because the real-photo evaluation reuses the exact benchmark checkpoints, this gap is a matched model comparison for the component task rather than a cross-model comparison. Three patterns emerge from the per-class breakdown.

Column generalises best (mean IoU $=0.665$; standard deviation $=0.066$): columns have a distinctive elongated vertical geometry and uniform material appearance that transfer from the photorealistic Tokaido renders.

Slab is the highest-variance class (IoU range $0.46$–$0.73$; standard deviation $=0.118$), indicating that seed-to-seed variation in the synthetic optimum translates to large differences in cross-domain performance for this class. Slab occupancy varies substantially across real images, amplifying inter-seed variability.

Beam achieves a moderate IoU (mean $=0.427$; standard deviation $=0.016$), consistent across seeds. Beam appearances vary considerably with bridge type and viewing angle, making generalisation harder than for columns.

Nonstructural is near zero (mean IoU $=0.042$), as expected: this class aggregates heterogeneous elements (railings, cables, and signage) that are underrepresented in training and visually diverse in the field.

Nonbridge IoU is trivially low (≈0.01), reflecting the small amount of padding-induced background introduced by resize–pad preprocessing. This class is uninformative in the real evaluation.

Overall, the component model demonstrates meaningful cross-domain generalisation for structurally salient classes (column and slab) while showing limited transfer for heterogeneous or underrepresented classes. Closing the domain gap would require domain-adaptive training or fine-tuning on annotated real imagery.

5.6. Zero-Shot GroundedSAM Baseline

To provide a training-free lower bound for real-image structural segmentation, we evaluate the GroundingDINO + SAM pipeline [16,17] (“GroundedSAM”) in a zero-shot setting on the same 16 real viaduct photographs used in Section 5.5. The real photographs carry component annotations only; no real-image damage annotations are available, so this evaluation covers the structural task exclusively. This experiment should not be read as a full evaluation of SAM-based inspection methods. It isolates what an off-the-shelf prompt-driven pipeline can deliver without task-specific adaptation. Trainable or refined SAM-based frameworks can be materially stronger in structural-inspection settings [59,60]; the present comparison is therefore intended only as a zero-shot lower bound.

Zero-shot means that the model is applied directly to the target task without any task-specific training or fine-tuning: it generalises purely from its large-scale pretraining on diverse image–text data. The GroundedSAM pipeline operates in two stages. First, GroundingDINO [17] is a vision–language model that takes a natural-language text prompt as input (e.g., “pillar. concrete pillar. support column”) and outputs bounding boxes in the image that spatially correspond to the described concept, together with a confidence score. Second, the Segment Anything Model (SAM) [16] takes each bounding box as a spatial prompt and produces a pixel-precise segmentation mask for the corresponding region. The two stages are therefore complementary: GroundingDINO localises “what” in the image matches the text, and SAM delineates “where” exactly the boundary lies. SAM ViT-H (the highest-capacity SAM variant) is used for the prompt ablation and main zero-shot comparison; the sensitivity sweep below additionally evaluates SAM ViT-L and ViT-B under the same best prompt set. No task-specific training is performed; the only design choices are the text prompts that describe each foreground class.

5.6.1. Prompt Sensitivity

Because GroundingDINO is sensitive to prompt wording, five prompt variants are tested for the structural task (Table 11 and Table 12). The baseline prompts use extended bridge-specific phrases; the remaining variants progressively simplify or replace the vocabulary.

Table 11. Text prompts used for each structural class across the five GroundedSAM prompt variants (v0–v4; note that these are distinct from the five joint-model conditioning variants in Table 9). Each cell lists the period-separated phrases passed to GroundingDINO for that class. Nonstructural (class 4) is not directly prompted; it receives all pixels not assigned to classes 1–3 after priority-order merging.

Variant	Slab (1)	Beam (2)	Column (3)	Nonstructural (4)
v0—baseline	bridge deck slab. concrete slab. bridge floor.	bridge girder. bridge beam. concrete beam.	bridge column. bridge pier. concrete column.	railway rail. railway sleeper. bridge railing.
v1—descriptive geom.	flat concrete deck. horizontal concrete surface. concrete floor plate.	concrete T-beam. prestressed concrete girder. longitudinal girder under deck.	vertical concrete pier. cylindrical column. rectangular concrete pillar.	metal handrail. steel guard rail. bridge fence.
v2—single terms	slab.	beam.	column.	railing.
v3—common vocab. (best)	concrete floor. flat concrete ceiling. concrete soffit.	structural beam. concrete support beam. I-beam.	pillar. concrete pillar. support column.	railing. fence. barrier.
v4—combined	bridge deck slab. flat concrete deck. concrete floor plate. bridge floor.	bridge beam. bridge girder. concrete beam. prestressed concrete girder. T-beam.	bridge column. concrete pier. bridge pier. vertical concrete pillar.	bridge railing. metal handrail. steel fence. guardrail.

Table 12. Structural-task prompt ablation for GroundedSAM on 16 real photographs (${\mathrm{mIoU}}_{\mathrm{main}}$ excludes nonbridge).

Variant	Slab	Beam	Column	Nonstructural	${\mathbf{mIoU}}_{\mathbf{main}}$
v0—baseline (bridge-specific phrases)	0.291	0.126	0.223	0.142	0.195
v1—descriptive geometry	0.184	0.018	0.090	0.133	0.106
v2—single terms (slab/beam/…)	0.372	0.089	0.310	0.105	0.219
v3—common vocabulary (best)	0.338	0.133	0.491	0.038	0.250
v4—combined extended phrases	0.131	0.147	0.106	0.088	0.118

Bold indicates the best (highest) value in each column.

The results reveal nuanced prompt sensitivity (Table 12). The common-vocabulary variant v3 achieves the highest overall ${\mathrm{mIoU}}_{\mathrm{main}}$ ($0.250$), followed by v2 single terms ($0.219$) and the bridge-specific baseline v0 ($0.195$). The descriptive-geometry variant v1 ($0.106$) and the combined extended phrases v4 ($0.118$) are substantially weaker, suggesting that verbose multi-component descriptions impair GroundingDINO’s grounding even when the individual terms are semantically accurate. Prompt conciseness appears to matter as much as vocabulary choice: the two weakest variants are precisely those with the longest, most composite prompt strings. The common-vocabulary variant v3 is used as the best structural prompt set in the benchmark results below. Figure 10 shows qualitative predictions for all tested variants.

Figure 10. Qualitative prompt ablation for GroundedSAM structural segmentation on real viaduct photographs. Columns: input image, component ground-truth, and zero-shot predictions for prompt variants v0 (bridge jargon), v2 (single terms), v3 (common vocabulary, best), and v4 (combined extended phrases). Classes: slab = red, beam = green, column = blue, nonstructural = yellow-orange, and nonbridge = dark grey. Variants v1 (descriptive geometry) and v4 (combined extended phrases) produce the weakest structural differentiation; v3 (common vocabulary) achieves the best overall mIoU (Table 12).

5.6.2. SAM-Backbone, Box-Merge, and Postprocessing Sensitivity

To test whether the zero-shot lower bound is sensitive to SAM backbone capacity, detector confidence settings, bounding-box merge strategy, or mask postprocessing, the best prompt set (v3) is re-evaluated over a 24-run grid: SAM ViT-H, ViT-L, and ViT-B; two GroundingDINO threshold pairs; no box merge versus non-maximum suppression; and no mask postprocessing versus morphological closing (Table 13). The strongest real-image score in this grid remains the default ViT-H setting with no box merging, ${\mathrm{mIoU}}_{\mathrm{main}}=0.240$. ViT-L is close under the same setting ($0.234$), whereas ViT-B is weaker (best $0.181$). Non-maximum suppression does not improve any backbone/threshold pair and generally lowers performance; morphological closing changes scores by at most $0.001$. Thus, none of the requested SAM-backbone, box-strategy, threshold, or postprocessing alternatives improves the zero-shot structural lower bound. Because this table is produced by a separate production rerun, we use it for within-sweep comparison of these axes; the main zero-shot benchmark value remains the prompt-ablation result reported in Table 12.

Table 13. GroundedSAM SAM-backbone, detector-threshold, box-merge, and postprocessing sensitivity for the best structural prompt set (v3), computed as a separate production rerun. Real-image scores are computed on the 16 annotated photographs; synthetic scores are reported only as a reference for the same zero-shot run. ${\mathrm{mIoU}}_{\mathrm{main}}$ excludes nonbridge.

SAM	Box Thr.	Text Thr.	Box Merge	Mask Postproc.	Real ${\mathbf{mIoU}}_{\mathbf{main}}$	Real Image-Wise Mean	Synthetic ${\mathbf{mIoU}}_{\mathbf{main}}$
ViT-H	0.35	0.25	none	none	0.240	0.171	0.115
ViT-H	0.35	0.25	none	close	0.240	0.171	0.115
ViT-H	0.35	0.25	NMS	none	0.202	0.142	0.125
ViT-H	0.35	0.25	NMS	close	0.202	0.142	0.125
ViT-H	0.25	0.20	none	none	0.196	0.164	0.132
ViT-H	0.25	0.20	none	close	0.196	0.164	0.132
ViT-H	0.25	0.20	NMS	none	0.163	0.122	0.122
ViT-H	0.25	0.20	NMS	close	0.163	0.122	0.122
ViT-L	0.35	0.25	none	none	0.234	0.190	0.133
ViT-L	0.35	0.25	none	close	0.234	0.190	0.133
ViT-L	0.35	0.25	NMS	none	0.227	0.186	0.120
ViT-L	0.35	0.25	NMS	close	0.227	0.186	0.120
ViT-L	0.25	0.20	none	none	0.221	0.160	0.147
ViT-L	0.25	0.20	none	close	0.221	0.160	0.147
ViT-L	0.25	0.20	NMS	none	0.205	0.163	0.126
ViT-L	0.25	0.20	NMS	close	0.205	0.163	0.126
ViT-B	0.35	0.25	none	none	0.143	0.159	0.151
ViT-B	0.35	0.25	none	close	0.143	0.159	0.151
ViT-B	0.35	0.25	NMS	none	0.139	0.155	0.151
ViT-B	0.35	0.25	NMS	close	0.139	0.155	0.151
ViT-B	0.25	0.20	none	none	0.180	0.162	0.162
ViT-B	0.25	0.20	none	close	0.181	0.162	0.162
ViT-B	0.25	0.20	NMS	none	0.113	0.137	0.131
ViT-B	0.25	0.20	NMS	close	0.113	0.138	0.131

5.6.3. Zero-Shot Benchmark Results

Table 14 reports GroundedSAM performance on the structural task alongside the trained ConvNeXt-Seg component model from Section 5.5. Real-damage annotations are not available for the 16 real photographs; the GroundedSAM evaluation therefore covers the structural task only.

Table 14. GroundedSAM zero-shot structural evaluation on 16 real photographs (prompt variant v3, common vocabulary). The trained-model row is the mean over three seeds from Table 10. ${\mathrm{mIoU}}_{\mathrm{main}}$ excludes nonbridge.

Model	Slab	Beam	Column	Nonstructural	${\mathbf{mIoU}}_{\mathbf{main}}$
Structural task
GroundedSAM (zero-shot, v3)	0.338	0.133	0.491	0.038	0.250
ConvNeXt-Seg (trained, mean)	0.562	0.427	0.665	0.042	0.424

GroundedSAM achieves ${\mathrm{mIoU}}_{\mathrm{main}}=0.250$ on structural components without any task-specific training, establishing a zero-shot lower bound for real-image structural segmentation. These numbers should therefore be interpreted narrowly: they quantify the performance of a prompt-based zero-shot GroundingDINO + SAM workflow on our real-image subset, not the ceiling of SAM-derived methods for infrastructure inspection. Figure 11 provides a qualitative comparison of both models on real photographs. The trained component model outperforms GroundedSAM on all structural classes, with the largest advantage on column ($0.665$ vs. $0.491$) and slab ($0.562$ vs. $0.338$). The overall trained-vs-zero-shot gap is $0.174$ in ${\mathrm{mIoU}}_{\mathrm{main}}$.

Figure 11. Structural component segmentation on real viaduct photographs (trained ConvNeXt-Seg; seed 1). Columns: input image, component ground-truth, and trained prediction. Classes: slab = red, beam = green, column = blue, nonstructural = yellow-orange, and nonbridge = dark grey. Real photographs carry component annotations only; damage annotations are unavailable for this image set.

6. Discussion

The benchmark separates two questions that are often conflated in inspection segmentation: which architecture is the strongest for a given task, and whether structural component information is useful for damage localisation. The architecture comparison shows that these answers are task-dependent. SegFormer gives the strongest damage segmentation result, while ConvNeXt-Seg gives the strongest component segmentation result. This difference is consistent with the visual character of the two tasks. Damage segmentation is dominated by sparse, thin, texture-level foreground classes, where the SegFormer-style multi-scale representation is competitive despite its moderate parameter count. Component segmentation is closer to scene parsing: slabs, beams, and columns occupy larger coherent regions, and the ConvNeXt-Seg model captures this structural layout more effectively on the synthetic test split. The practical implication is that a single architecture ranking is not sufficient for inspection systems that must solve both component and damage segmentation.

The component-aware ablation provides the paper’s central methodological result: the value of component conditioning depends on the evaluation protocol. Under the original random split, adding a component branch, feeding a hard predicted component mask or detached component features into the damage segmentation module, or allowing for full end-to-end feature conditioning did not produce a measurable damage gain over the damage-only baseline. The observed spread across variants is smaller than the run-to-run variation, a five-seed paired analysis finds no variant significantly better than the baseline, and even a ground-truth-mask upper bound adds only $0.0102$ foreground mIoU. Under a leakage-controlled scene-disjoint split, by contrast, the same variants show a small positive trend (soft-detach $+0.019$ foreground mIoU over three seeds) that still requires paired confirmation. The oracle-filter analysis makes the interpretation more specific: even on images where component predictions are reliable, the conditioned model does not outperform the unconditioned model. This argues against the explanation that component-aware conditioning failed simply because the component branch was inaccurate. A pixel-level dependence analysis shows why any gain is bounded yet nonzero: damage is concentrated almost entirely on columns but is sparse even there, so component identity strongly constrains where damage can occur while carrying little information about whether a given pixel is damaged.

This protocol-dependent result should be read as dataset- and label-specific. It does not contradict bridge-inspection studies where joint component and damage learning improves performance, because those studies do not necessarily isolate explicit component conditioning from shared supervision and may involve damage categories that correlate more strongly with structural role. The present design tests a narrower claim: whether a damage segmentation model that already receives the original image as input benefits from being provided with predicted component information under a controlled Tokaido benchmark. For this claim, the evidence is protocol-dependent: negative under the random split and a small positive trend under the scene-disjoint split. The tested fusion module is intentionally simple—a single local Conv-BN-ReLU block that injects masks or feature maps at decoder resolution—so any conclusion about damage segmentation gains is specific to this controlled local-fusion design, not to richer conditioning architectures such as cross-attention or spatial routing. On the cost side, the component-aware variants increase inference latency by 38–$44\%$ relative to the damage-only model for at most a small damage-accuracy change, so the added cost is not justified by damage accuracy on the random split and remains to be confirmed under the scene-disjoint protocol.

6.1. Deployment Guidance

The tabulated accuracy, throughput, and parameter counts support concrete model choices. For accuracy-critical damage segmentation we recommend SegFormer, which attains the highest damage ${\mathrm{mIoU}}_{\mathrm{main}}$ ($0.569$) at competitive throughput ($19.5$ FPS). When component segmentation is also required, ConvNeXt-Seg is preferred for its strongest component ${\mathrm{mIoU}}_{\mathrm{main}}$ ($0.957$). The EfficientNet-B3 decoders (U-Net, UNet++) remain attractive when a lighter convolutional stack is desired. Regarding the component branch, the 38–$44\%$ latency increase is not justified by damage accuracy under the random split; the small positive trend under the scene-disjoint protocol would need paired confirmation before the added cost could be recommended for damage localisation alone.

The real-photograph component experiment shows a different but equally important boundary. The trained ConvNeXt-Seg component model outperforms the zero-shot GroundedSAM baseline on the 16 annotated real photographs, indicating that synthetic task-specific training transfers useful structural information. At the same time, the drop from synthetic component performance to real-photo component performance is large, and the weakest real-photo class is the heterogeneous nonstructural category. The zero-shot result should therefore be interpreted as a lower bound for prompt-based segmentation without adaptation, not as a ceiling for foundation-model approaches. The additional 24-run GroundedSAM sensitivity sweep did not improve this lower bound: ViT-H remained best in the sweep ($0.240$), ViT-L was close ($0.234$), ViT-B was weaker, NMS did not improve any backbone/threshold pair, and morphological closing had negligible effect. Adapted SAM-style models or fine-tuned component segmenters may close part of this gap, but the current experiment shows that synthetic training alone does not remove the domain shift.

6.2. ProgressRelative to Prior Work

The present benchmark substantially extends the earlier study [61], which evaluated only U-Net and DeepLabV3+ on the same Tokaido dataset at a lower resolution of $320\times 160$ pixels using a custom Keras/TensorFlow implementation without repeated-seed evaluation or systematic loss ablation. Table 15 summarises the key quantitative improvements.

Table 15. Summary of progress relative to the prior study [61]. All metrics are foreground mIoU (background excluded) unless otherwise noted. Prior damage figures are per-class IoU averaged over the two foreground damage classes (Cracks and Reinforcement) from Table 3 of that paper.

Aspect	Prior Work	Present Work
Input resolution	$320\times 160$	$512\times 896$
Architectures evaluated	2	7
Loss functions compared	ad-hoc	4 (systematic ablation)
Repeated seeds	no	3 per config
Component mIoU (synthetic test)	76–87% (val, 4 cls)	94.6–95.7% (test, 5 cls)
Damage mIoU (synthetic test)	$\sim 0.46$ (2 arch)	$0.569$ (SegFormer mean)
Component mIoU (real photos)	$\sim 0.18$ (U-Net, 4 cls)	$0.424$ (trained, 5 cls, gap $\approx 0.53$)
Zero-shot structural baseline	—	$0.250$ (GroundedSAM)
Joint component-aware ablation	no	yes (5 variants)

The two studies differ in resolution, class count, evaluation split, framework, and architecture scope, so no single factor can be isolated as the cause of any difference; the values in Table 15 are therefore descriptive rather than causal. On the component task, the previous study reported validation mIoU of 76% (U-Net) and 87% (DeepLabV3+) over four classes at $320\times 160$. The present benchmark, evaluated on the held-out test set at $512\times 896$ with five classes and repeated seeds, reports ${\mathrm{mIoU}}_{\mathrm{main}}$ of $0.947$ (U-Net) and $0.946$ (DeepLabV3+), with the best architecture (ConvNeXt-Seg) reaching $0.957$. On the damage task, the previous study reported a per-class IoU of 48% for Cracks and 44% for Reinforcement (U-Net, Tversky loss); the best model here (SegFormer, Tversky loss) achieves ${\mathrm{mIoU}}_{\mathrm{main}}^{\mathrm{dmg}}=0.583$, with the mean over three seeds at $0.569$. The metrics are not directly comparable (per-class IoU versus foreground mIoU; validation versus test), so these figures serve as historical context only. On real-image transfer, the prior study reported a four-class foreground mean of approximately 18%; the present ${\mathrm{mIoU}}_{\mathrm{main}}^{\mathrm{comp}}=0.424$ on the same 16 photographs reflects simultaneous changes in class set, resolution, and training protocol, making direct comparison difficult.

The main limitation on the conclusions of the present study is scope. The conditioning conclusion is restricted to the Tokaido synthetic damage labels and to the tested ConvNeXt-based conditioning mechanisms. It should not be generalised to all bridge datasets or all component-aware designs, especially where damage type is physically tied to structural role. The real-photo evaluation is also restricted to component segmentation on 16 annotated photographs; the masks were produced manually by a single annotator and were not independently validated through inter-annotator agreement. No real-image damage masks are available in this study, so real-world damage transfer cannot be assessed. Future work should therefore focus on two targeted extensions: real-image fine-tuning or domain adaptation for component segmentation, and conditioning tests on datasets where component labels and damage labels have stronger physical coupling. These extensions would test whether the reported protocol-dependent result follows from the properties of the Tokaido dataset or is a broader limitation of explicit component conditioning for fine-scale damage segmentation.

7. Conclusions

This study evaluated structural component and damage segmentation for railway viaduct inspection under a controlled synthetic-data benchmark. Seven segmentation architectures were compared on the Tokaido dataset, and a separate component-aware ablation tested whether explicit component information improves damage localisation. The strongest damage segmentation model was SegFormer, with foreground ${\mathrm{mIoU}}_{\mathrm{main}}=0.569\pm 0.013$, while the strongest component segmentation model was ConvNeXt-Seg, with foreground ${\mathrm{mIoU}}_{\mathrm{main}}=0.957\pm 0.001$. These task-dependent rankings show that architecture choice cannot be reduced to a single best model across structural and damage segmentation.

The central result is that the value of explicit component conditioning depends on the evaluation protocol. Under the original random split, conditioning did not improve damage segmentation: across the five ConvNeXt-based variants the full range of foreground damage mIoU was $0.008$, a five-seed paired analysis found no variant significantly better than the damage-only baseline, and a ground-truth-mask upper bound added only $0.0102$ mIoU. Under a leakage-controlled scene-disjoint split, the same variants showed a small positive trend (up to $+0.019$ mIoU over three seeds) that a paired analysis still needs to confirm. The oracle-filter analysis on the random split reached the same conclusion on images where component predictions were reliable: conditioned and unconditioned damage models remained statistically indistinguishable. This evidence supports a narrow but important conclusion: for the Tokaido damage labels tested here, any component-conditioning benefit is small and sensitive to whether the evaluation controls for scene leakage. It does not imply that component-aware inspection is generally ineffective; rather, the benefit depends on the evaluation protocol and on whether the dataset encodes a meaningful relationship between structural element type and damage occurrence.

The synthetic-to-real evaluation showed both the value and the limit of task-specific training. A ConvNeXt-Seg component model trained on synthetic Tokaido imagery reached ${\mathrm{mIoU}}_{\mathrm{main}}=0.424\pm 0.064$ on 16 real viaduct photographs, compared with $0.250$ for the zero-shot GroundedSAM baseline. At the same time, the drop from $0.957\pm 0.001$ on synthetic component segmentation to $0.424\pm 0.064$ on real photographs quantifies a substantial domain gap. Future work should therefore prioritise real-image adaptation, larger real-photo test sets, and datasets where component labels and damage labels have stronger physical coupling. The benchmark and ablation results reported here provide reference baselines for that next stage while showing that component-aware conditioning should be treated as an empirical design choice rather than an assumed source of improvement.

8. Future Plans

In this section, we will discuss possible future research activities on two time scales, i.e., short term and medium term.

Short-Term Research Directions:

• One immediate priority is expanding the real-world viaduct dataset. At present, the real-photo evaluation relies on just 16 manually annotated photographs, which limits how confidently we can draw conclusions. Broadening this dataset and incorporating publicly available bridge-inspection benchmarks would allow for a more thorough assessment of how well models trained on synthetic data transfer to real-world conditions. Beyond the component masks, this expansion should add real damage annotations and an inter-annotator agreement study with a second independent annotator to validate label quality.
• Closely related is the challenge of synthetic-to-real adaptation in component segmentation. The noticeable performance drop when moving from synthetic to real imagery suggests that transfer-learning and domain-adaptation techniques deserve closer attention. Fine-tuning on real inspection images is a natural next step, with the goal of making component segmentation more reliable in practical settings.
• Finally, once component models have been adapted to real imagery, it will be worth revisiting the role of explicit component conditioning. The current study found no measurable benefit from this approach under the original random split on the synthetic Tokaido benchmark, while the leakage-controlled scene-disjoint split showed a small positive trend; it remains an open question whether this protocol-dependent result stems from the synthetic training domain itself. Repeating the ablation with real-data-adapted models should help clarify this.

Medium-Term Research Directions:

• Looking further ahead, the component-aware hypothesis should be tested across a broader range of bridge-inspection datasets—ones that differ in structural layout, damage distribution, and inspection conditions—which will require aligning structural component taxonomies across datasets, for instance, toward dacl10k. This would help determine whether the protocol-dependent pattern of gains on Tokaido reflects something specific to that dataset or points to a more general limitation of the approach.
• It would also be valuable to focus on inspection scenarios where damage and structural role are more tightly coupled—for instance, corrosion concentrated on steel members or deterioration patterns tied to particular bridge components. Such cases may be where component-level information proves most useful and studying them could reveal whether stronger physical relationships between structural elements and damage mechanisms change the picture.
• Finally, the conditioning mechanism itself deserves more scrutiny. The local Conv-BN-ReLU fusion used in this work is relatively straightforward, and it is worth exploring whether richer alternatives—such as cross-attention or feature-routing approaches—yield different results. This would help distinguish between limitations of the conditioning architecture and limitations of the conditioning signal.