Bridge Surface Defect Detection via Heterogeneous Feature Fusion and Multi-Scale Enhancement

Abstract

Automated bridge surface defect detection is essential for improving the efficiency and objectivity of infrastructure inspection under complex field imaging conditions. This study proposes a Multi-Scale Detection Transformer (MS-DETR), an RT-DETR-based detector that integrates HeteroFusionNet, a multi-objective scale-aware integration network (MOSAIN), and a global attention two-dimensional module (GATM). HeteroFusionNet combines shared shallow feature extraction with heterogeneous dual-branch deep modelling to reduce redundant computation and enhance complementary local and global representations. MOSAIN injects high-resolution shallow details into the P3 feature path to improve small-defect recognition, whereas GATM adapts high-level attention encoding to dense two-dimensional visual features. On the self-built bridge defect dataset, MS-DETR achieved a mAP50 of 65.4% and an F1-score of 0.64, with 14.33 M parameters, 39.1 GFLOPs, and 62.3 FPS on an RTX 4090 GPU. On the RDD (China) road defect dataset and the VisDrone2019 dataset, MS-DETR achieved mAP50 values of 88.9% and an AP50 of 0.487, respectively. These results demonstrate that MS-DETR achieves competitive detection accuracy while maintaining a favorable balance between model complexity and inference speed under the evaluated experimental settings.

1. Introduction

Bridges are vital infrastructure in modern transportation systems and road networks. As of the end of 2024, China has a total of 1.1084 million road bridges and 101.9758 million linear meters. China has become the country with the most bridges in the world [1]. However, with the increase in service life and the long-term alternating effects of various adverse factors such as fatigue load, extreme climate, material aging, and geological deformation, bridge structures will inevitably suffer various forms of damage, such as surface cracks, concrete spalling, and steel corrosion. Among them, surface defects such as cracks are often the most intuitive indicators of potential degradation or major damage to the bridge structure [2]. If these defects are not discovered and dealt with early, they will not only destroy the aesthetic appearance of the bridge, but also seriously weaken the strength and durability of the structure, and may even lead to catastrophic accidents such as bridge collapse, causing huge casualties and property losses. Therefore, regular and accurate inspection and health assessment of bridges are of extremely important practical significance for ensuring the safety of bridge operations, preventing major accidents, and optimizing bridge maintenance management decisions.

For a long time, traditional bridge safety inspections have mainly relied on detailed manual visual inspections by experienced inspectors [3,4]. However, this manual inspection approach has many inherent limitations. First, manual inspection requires substantial time and cost, and inspectors must receive extensive professional training. Second, due to the need to use bridge inspection vehicles or set up scaffolding, the inspection process usually requires temporary traffic closure, which not only brings inconvenience to the public, but also exposes inspection personnel to higher safety risks. In addition, it is difficult to accurately obtain precise quantitative information such as crack length, depth, and width by relying solely on visual observation, and the detection results are easily affected by human fatigue and subjective experience.

To overcome the shortcomings of traditional manual inspection, automated inspection technology based on computer vision has received widespread attention from academia and industry. In particular, the development of unmanned aerial vehicle (UAV) technology provides a safer, faster and more cost-effective alternative for bridge inspection. UAVs can flexibly carry high-definition cameras, easily reach dangerous or high-altitude areas that are difficult to access manually, and obtain high-resolution bridge surface images. Combined with early computer vision technology, researchers have developed a variety of image processing methods based on traditional machine learning (such as support vector machines, random forests, etc.) to extract crack features. Although these methods improve detection efficiency to a certain extent, they usually rely on manually designed feature extraction methods, have poor generalization capabilities, and are difficult to adapt to complex real environments. In recent years, the rapid advancement of deep learning technology has brought breakthrough progress to bridge damage detection. Various convolutional neural networks (CNN, such as Faster R-CNN, YOLO series) and emerging visual transformer (ViT) technology can automatically extract deep-level abstract features from massive images, greatly improving the classification, positioning and segmentation accuracy of defects such as cracks and spalling on the bridge surface [5,6].

In view of the challenges faced by deep learning models in bridge surface defect detection, including strong background interference, loss of small-scale defect features, high computational cost, and limited inference efficiency, this study proposes an automated bridge defect detection algorithm named Multi-Scale Detection Transformer (MS-DETR) based on RT-DETR. The proposed method introduces three main components into the RT-DETR framework: the HeteroFusionNet heterogeneous dual-branch backbone, the MOSAIN small-object enhanced feature fusion network, and the GATM high-level feature encoding module. HeteroFusionNet reduces redundant computation while enhancing the complementary representation of local texture and global structural semantics through shared shallow feature extraction and heterogeneous dual-branch deep modelling. MOSAIN injects shallow high-resolution detail information into the mid-level feature path to improve the recognition of small-scale defects such as fine cracks and exposed rebar. GATM enhances high-level contextual modelling using a feature encoding strategy that is more compatible with two-dimensional dense visual features. Through these improvements, MS-DETR aims to improve multi-scale defect representation and reduce model complexity while maintaining competitive detection accuracy, thereby providing an efficient visual detection method for bridge surface defect inspection.

The primary task of this study is to detect and localize four categories of bridge-surface defects using bounding boxes: cracks, corrosion, exposed reinforcement, and concrete spalling. The RDD (China) pavement-distress dataset and the VisDrone2019 dataset are used only for auxiliary cross-domain evaluations. Their results are reported separately and are not combined with the bridge-defect results. This study does not estimate physical crack width, defect depth, or component-specific severity. Accordingly, we do not claim that pavement cracks and bridge-surface cracks are equivalent in terms of geometric characteristics, engineering significance, or inspection requirements.

2. Related Work

2.1. RT-DETR Architecture

RT-DETR (Real-Time DEtection TRansformer) is an end-to-end real-time object detector mainly composed of three components: a backbone network (Backbone), an efficient hybrid encoder (Efficient Hybrid Encoder), and a decoder (Decoder), as illustrated in Figure 1. This model introduces the Transformer architecture into real-time object detection tasks and achieves a favorable balance between detection speed and accuracy through an efficient encoder and an IoU-aware query selection mechanism [7].

Figure 1. Architecture of RT-DETR.

The backbone network of RT-DETR is responsible for extracting multi-scale features from the input image and passing the features to subsequent encoders and decoders. Commonly used backbone networks include convolutional network structures such as ResNet and HGNet. This study uses ResNet-18 as the backbone network of the RT-DETR baseline model for subsequent model improvement and comparative analysis.

RT-DETR’s efficient hybrid encoder is used for interactive modeling and semantic enhancement of multi-scale features, mainly including the intra-scale feature interaction module AIFI and the cross-scale feature fusion module CCFM. AIFI only performs self-attention modeling on high-level semantic features to reduce redundant calculations brought by low-level high-resolution features while maintaining the ability to model global contextual relationships. This design helps improve the quality of high-level semantic feature expression and reduces the computational overhead in real-time detection tasks.

Cross-scale feature fusion module CCFM is used to integrate feature information at different scales, thereby enhancing the model’s adaptability to scale changes and small objects. This module complements shallow detail information with deep semantic information through adjacent scale feature fusion, allowing the detector to pay attention to target edges, local textures and contextual relationships at the same time. Because CCFM adopts a relatively compact convolutional fusion structure, it can improve multi-scale detection capabilities while controlling additional computational complexity.

2.2. Deep Learning for Bridge Defect Detection

2.2.1. Deep Learning-Based Image Classification of Bridge Defects

With the development of deep learning, convolutional neural networks (CNN) have replaced traditional manual feature extraction methods and become the mainstream standard in the field of computer vision because of their ability to automatically extract hierarchical features from images. In the early research on bridge surface defect identification, scholars mainly focused on the binary classification problem (that is, distinguishing whether there are cracks) [8]. For example, Mostafa Aliyari et al. [9] proposed a bridge crack detection method based on UAV images and convolutional neural network (CNN) transfer learning to accurately classify whether there are cracks in bridge images. Zhang et al. [10] proposed a 1D-CNN-LSTM deep learning detection framework based on image frequency domain feature conversion, which achieved rapid and high-precision automatic identification of cracks in concrete bridge decks. Subsequently, the research gradually expanded to multi-category and multi-label defect classification (such as simultaneous identification of cracks, spalling, whitening, exposed rebar, etc.). The researchers introduced InceptionV3, ResNet and the emerging Visual Transformer (ViT) technology, which greatly improved the model’s ability to identify multiple defects in complex backgrounds. For example, Mustafa Abubakr et al. [11] proposed a multi-objective defect classification method for reinforced concrete bridges based on deep learning (Xception and Vanilla convolutional neural network), achieving high-precision identification of five types of common defects such as cracks and spalling, and confirmed that the Xception model (94.95% accuracy) has better classification performance than the basic network. Matthieu Pâques et al. [12] proposed a multiclass classification framework for bridge components and defects based on the visual transformer architecture and category balancing technology, achieving efficient feature learning for noisy large-scale datasets. However, image classification can determine whether an image contains a defect but cannot provide its precise location.

2.2.2. Object Detection-Based Localization of Bridge Defects

To obtain the specific location of the defect and conduct quantitative assessment, the object detection algorithm based on bounding box (bounding box) is widely used in bridge detection. The current deep learning object detection is mainly divided into two-stage, one-stage and Transformer-based methods:

Dual-stage object detection: The two-stage algorithm represented by R-CNN and Faster R-CNN series performs well in detection accuracy. Li et al. [5] combined the Faster R-CNN algorithm based on VGG-16 transfer learning with unmanned aerial vehicle (UAV) technology to automatically detect horizontal, longitudinal and network cracks on the surface of the bridge tower. The overall precision (Precision) of the model reached 92.03%, and the recall (Recall) reached 96.26%.

To meet the needs of real-time detection and reduce computing costs, single-stage algorithms such as the YOLO (You Only Look Once) series and SSD treat object detection as a regression problem and have achieved significant breakthroughs in detection speed. Xia et al. [13] proposed an improved algorithm based on YOLOv8 for bridge crack detection in complex environments. This study introduced the SPPF_UniRepLk module into the backbone network to enhance feature extraction capabilities, and added a global channel spatial attention mechanism (GCSA) to capture global dependencies in the feature map. Finally, it reached a mAP50-95 of 72.1% in the test, achieving extremely high detection accuracy while ensuring detection speed. Lu et al. [14] proposed an improved SSD object detection network (ISSD) that integrates depth-separable deformation convolution, inception and feature recalibration modules, achieving simultaneous optimization of computational efficiency and irregular feature extraction capabilities.

In recent years, the transformer architecture has shown great potential in capturing long-distance dependencies. Wan et al. [15] proposed a new bridge damage detection model (BR-DETR) based on Detection Transformers (DETR). This model replaces standard convolution with deformable convolution (Deformable Conv2D), adds convolutional project attention (Convolutional Project Attention) after the self-attention layer, and uses a local enhancement feedforward network (LeFF) to extract local information. Experiments have proven that the detection performance of this model is significantly better than the traditional CNN and original DETR models in complex bridge damage scenes. Yang et al. [6] proposed an enhanced YOLO object detection algorithm that combines the visual Transformer (C3MaxViT) and the anchor-free mechanism, which achieves effective modeling of global feature dependencies and decoupling of classification and regression tasks, significantly improving damage detection accuracy in complex real-life scenes (mAP50 increased by 8.1%).

2.2.3. Multi-Scale and Multi-Branch Transformer-Based Detectors

Multi-scale feature modelling and multi-branch network design have been widely investigated to improve object detection performance under substantial target-scale variations. Ji et al. [16] proposed MS-DETR, a real-time multi-scale detection transformer for PCB defect detection, demonstrating the effectiveness of multi-scale transformer representations in industrial defect inspection. Lu et al. [17] developed a novel multi-scale transformer for aerial object detection, with particular emphasis on dense targets and scale variations in UAV imagery. Zhang et al. [18] investigated the efficient utilization of multi-scale features in transformer-based object detectors and discussed the redundancy and information-selection problems associated with multi-scale feature representation.

Multi-scale transformer architectures have also been explored in other application domains. Xu et al. [19] proposed DMS-DETR, which combines dynamic multi-scale detection with progressive path attention for personal protective equipment detection. Although not developed for visual defect detection, Xi et al. [20] introduced a multi-scale transformer-based model for network intrusion detection, further demonstrating the broad applicability of multi-scale feature modelling in complex recognition tasks. These studies indicate that multi-scale representation learning, branch-specific feature extraction, and progressive feature propagation are established strategies for improving the recognition of targets with different scales and structures.

However, the above methods mainly focus on PCB defects, aerial objects, general object detection, PPE detection, or network traffic, and they do not directly address the specific challenges of bridge surface defect detection. Bridge images commonly contain large-area spalling, slender cracks, exposed rebar, and irregular corrosion regions, which exhibit substantial differences in scale, morphology, texture, and boundary characteristics. In addition, bridge inspection images often contain strong background interference and weak defect–background contrast. Therefore, the use of a multi-scale or multi-branch structure alone should not be regarded as the primary novelty of the present study. Instead, this work focuses on the task-specific integration of HeteroFusionNet, MOSAIN, and GATM. HeteroFusionNet enhances heterogeneous local and global feature representation, MOSAIN introduces high-resolution detail information into the P3 feature path, and GATM improves high-level two-dimensional feature encoding. The proposed design is intended to address the particular scale variation, texture ambiguity, and computational constraints encountered in bridge surface defect detection.

3. Method

To improve the detection accuracy and inference efficiency in complex bridge defect scenarios, this study develops an improved detector, MS-DETR, based on the RT-DETR framework. The overall architecture of MS-DETR is illustrated in Figure 2 and consists of three main components: the heterogeneous dual-branch backbone network HeteroFusionNet, the multi-scale feature-enhanced neck network, and the high-level feature modeling module based on GATM. First, the input image is passed through a shared HGStem to extract generic shallow features, thereby reducing redundant computation in the high-resolution stage. Subsequently, the features are fed into two deep branches with different structures: one branch consists of HGBlock and DWConv, focusing on hierarchical semantic aggregation and lightweight feature extraction; the other branch consists of C2f and Conv, focusing on local texture modeling and detail information retention. The two branches interact through dynamic alignment fusion modules at multiple scale stages, thereby alleviating the spatial misalignment and semantic differences between heterogeneous features.

Figure 2. Architecture of the MS-DETR model.

After the backbone network outputs multi-scale features, this study further constructs a feature fusion path for small-object defects. This path introduces shallow high-resolution details through SPDConv, and combines modules such as upsampling, concatenation, RepC3, and CSPOmniKernel to achieve cross-scale information integration. This design enables the model to simultaneously utilize shallow texture, mid-level structure and high-level semantic information, thereby enhancing the representation of small or slender defects such as cracks and exposed rebar. Finally, the fused multi-scale features are sent to the detection head to complete defect category prediction and bounding box regression. Overall, MS-DETR improves defect identification capabilities while controlling the number of model parameters and computational overhead through the collaborative design of “shared shallow feature extraction, heterogeneous deep modeling, dynamic feature alignment and multi-scale enhanced fusion”.

3.1. HeteroFusionNet

Although existing bridge defect detection methods have made significant progress in object detection accuracy, they still have several common limitations. First, although many detection methods based on single-path convolution backbones have strong local texture modeling capabilities, they are still insufficient in characterizing high-level structural semantics and long-range dependencies in complex backgrounds, which easily affects the recognition effect of slender cracks, weak texture damage, and complex morphological defects [13,14]. Second, some methods improve feature expression capabilities by introducing an attention mechanism or Transformer structure. Although the global modeling effect is improved, it is often accompanied by an increase in the number of parameters, computational cost, and inference latency, thus weakening its deployment efficiency in actual bridge inspection scenarios [6,15]. In addition, bridge defect images usually have problems such as large scale changes, strong illumination disturbance, and obvious motion blur, which further exacerbates the difficulty of feature extraction and cross-layer semantic modeling, making the robustness of existing methods under complex working conditions still to be improved.

In response to the above problems, this study proposes a heterogeneous fusion detection network HeteroFusionNet. The core idea of this method is to maintain the efficiency of the overall detection framework and balance local detail modeling, global structure perception and cross-branch feature collaboration through the method of “shared shallow feature extraction + heterogeneous dual-branch deep modeling + dynamic alignment fusion”. Specifically, this study retains only one shared shallow Stem in the backbone network to reduce redundant computing overhead in the high-resolution stage; then constructs two heterogeneous branches with different modeling preferences to extract complementary multi-level features; and finally introduces a Dynamic Align Fusion (DAF) module at multiple pyramid levels to achieve adaptive calibration and high-quality fusion of cross-branch features.

HeteroFusionNet performs a dual-branch reconstruction based on shared shallow modules. Specifically, the input image first extracts underlying general visual features through the shared HGStem, and then branches into two heterogeneous branches for deep representation learning. Among them, the first branch consists of C2f module and standard convolution alternately, focusing on strengthening local texture modeling capabilities and context information reuse; the second branch consists of HGBlock and depth separable convolution (DWConv), focusing more on hierarchical feature aggregation and global structure information extraction. In the three stages of P3, P4 and P5 of the feature pyramid, the outputs of the two branches are interacted and fused step by step through the dynamic alignment fusion module DAF. Finally, the fused multi-scale features are uniformly sent to the neck network (Neck) and decoder (Decoder) of RT-DETR to complete the classification and positioning of bridge defect targets.

Unlike conventional dual-backbone architectures, dual-backbone structure that configures a separate shallow feature extraction module for each branch, this method only retains one shared Stem. This design avoids parameter redundancy and computational waste caused by repeated dense convolution operations in high-resolution shallow stages, while retaining the representational differences in deep heterogeneous branches, allowing the network to learn more complementary multi-level feature representations at lower computational costs. Therefore, the proposed structure not only balances detection accuracy, but also has better inference efficiency.

The detailed stage-wise configuration of HeteroFusionNet is summarized in Table 1. The model adopts the l-scale configuration with depth and width multipliers of 1.0. For an input image of 640 × 640 pixels, the shared HGStem produces a P2/4 feature map with a spatial resolution of 160 × 160 and 32 output channels. The two branches then perform three successive downsampling operations with a stride of 2, producing P3/8, P4/16, and P5/32 feature maps with spatial resolutions of 80 × 80, 40 × 40, and 20 × 20, respectively.

Table 1. Stage-wise configuration of HeteroFusionNet.

Stage	Feature Level	Spatial Size	Branch A	Output Channels	Branch B	Output Channels	DAF Output
Shared stem	P2/4	160 × 160	HGStem (cm = 16, c2 = 32)	32	Shared HGStem	32	–
Shallow stage	P2/4	160 × 160	C2f (n = 1)	32	HGBlock (cm = 16, c2 = 32, k = 3, n = 3)	32	–
Stage 1	P3/8	80 × 80	Conv (3 × 3, s = 2) + C2f (n = 1)	64	DWConv (3 × 3, s = 2) + HGBlock (cm = 32, c2 = 64, k = 3, n = 3)	64	64
Stage 2	P4/16	40 × 40	Conv (3 × 3, s = 2) + C2f (n = 1)	128	DWConv (3 × 3, s = 2) +3 sequential HGBlock (cm = 64, c2 = 128, k = 5, n = 3)	128	128
Stage 3	P5/32	20 × 20	Conv (3 × 3, s = 2) + C2f (n = 3)	256	DWConv (3 × 3, s = 2) + HGBlock (cm = 128, c2 = 256, k = 5, n = 3)	256	256

In multi-branch detection networks, shallow feature maps usually have larger spatial resolution. Directly building an independent front-end feature extraction module for each branch, the number of parameters, floating point operations (FLOPs), and inference time will be significantly increased. Based on this, this study constructs a heterogeneous dual-branch backbone structure based on shared HGStem. Given an input image X, the network first extracts underlying common features by sharing Stem:

$$F_0=H_{stem}{(X)}_{\omega }$$

(1)

where H_stem(⋅) denotes the nonlinear mapping operation of shared H_stem. Subsequently, using F₀ as the common input of two heterogeneous branches, parallel feature extraction is performed at different stages. For the stage-s feature stage, denoted as s ∈ {P3, P4, P5}, the outputs of the two branches can be expressed as

$$F_s^{(1)}=B_s^{(1)}(F_{s-1}),F_s^{(2)}=B_s^{(2)}(F_{s-1})$$

(2)

where $B_s^{\left(1\right)}$ denotes the stage-s feature extraction stage composed of C2f and Conv, $B_s^{\left(2\right)}$ denotes the stage-s feature extraction stage composed of HGBlock and DWConv, $s \in \{\mathrm{P}3,\mathrm{P}4,\mathrm{P}5\}$.

This design mainly has two advantages. First, shared Stem can effectively reuse low-level visual priors such as textures, edges, and contours, fundamentally reducing the additional computational overhead caused by repeated shallow feature extraction. Second, since the two branches have different topological structures and feature modeling preferences, they can respectively focus on fine-grained local information and high-level structural semantics, thereby achieving complementary feature learning at a higher level and enhancing the network’s comprehensive representation capabilities for complex bridge defect targets.

The specific combination of the two branches was selected according to the characteristics of bridge surface defects. Branch A combines C2f with standard convolution. C2f facilitates feature reuse and information propagation, while standard convolution provides effective spatial and channel interaction. This branch is mainly intended to preserve local texture, edge, and boundary information, which is important for detecting thin cracks, exposed rebar, corrosion boundaries, and irregular spalling contours. Branch B combines HGBlock with depthwise separable convolution. HGBlock enhances hierarchical feature aggregation, whereas depthwise separable convolution reduces the computational cost of spatial feature extraction and downsampling. This branch is therefore designed to capture more contextual and structural information under complex backgrounds, illumination variations, and changes in defect scale.

Accordingly, the two branches provide complementary rather than identical feature representations. The C2f–standard convolution branch emphasizes fine-grained local details, while the HGBlock–depthwise separable convolution branch focuses on hierarchical contextual information and computational efficiency. The effectiveness of this specific operator assignment is further examined through controlled branch-composition ablation experiments in Section 4.4.1.

Since the two branches use different feature-extraction operators, their outputs may differ in receptive field, spatial response, and semantic distribution. Therefore, an adaptive fusion mechanism is required before the multi-scale features are passed to the subsequent neck network.

Although the heterogeneous dual-branch structure can significantly enrich feature expression, due to the differences in network depth, receptive field, and feature distribution between the two branches, cross-branch feature fusion often faces problems such as spatial response misalignment, inconsistent semantic focus, and distribution shifts. If static hard fusion strategies such as element-wise addition or channel concatenation are used, these inconsistent information will be directly passed to the subsequent network, thereby affecting the discriminative ability of the fused features. To this end, this study designed a dynamic alignment fusion module (Dynamic Align Fusion, DAF) to achieve dynamic spatial calibration and adaptive fusion of heterogeneous branch features.

For the two input features F1 and F₂ of the same pyramid stage, DAF first uses two 1 × 1 convolutions to map them to a unified channel space:

$${\widehat{F}}_1={\phi }_1(F_1),{\widehat{F}}_2={\phi }_2(F_2)$$

(3)

where ${\mathit{\varphi }}_1(\cdot )$ and ${\mathit{\varphi }}_2(\cdot )$ denotes the channel-aligned convolution operation of the two-way features respectively. Subsequently, the aligned two-way features are concatenated in the channel dimension, and the dynamic spatial alignment weight is generated through the convolution layer and the Sigmoid activation function:

$$W=\sigma (\psi ([{\widehat{F}}_1;{\widehat{F}}_2]))$$

(4)

where $[\cdot ;\cdot ]$ denotes channel concatenation operation,$\psi (\cdot )$ denotes feature aggregation convolution mapping,$\mathsf{\sigma }(\cdot )$ is the Sigmoid function. Next, the generated global weight W is equally divided into two parts along the channel dimension:

$$W_1,W_2=Split(W)$$

(5)

Based on the above dynamic weights, pixel-level adaptive recalibration is performed on the two-way features to obtain:

$${\tilde{F}}_1={\widehat{F}}_1\odot W_1,{\tilde{F}}_2={\widehat{F}}_2\odot W_2$$

(6)

where $\odot$ denotes element-wise multiplication. On this basis, in order to further adjust the contribution ratio of dual-channel features at the global level, the module introduces two sets of channel-level learnable scaling parameters. P₁ and P₂, and finally passed $1 \times 1$ Convolution output final fusion feature:

$$F_{out}={\phi }_f({\tilde{F}}_1\odot P_1+{\tilde{F}}_2\odot P_2)$$

(7)

where ${ \mathit{\varphi }}_f(\cdot )$ denotes the final mapping convolution used for feature smoothing and dimensionality reduction.

Compared with static traditional fusion methods, the DAF module does not simply superimpose features, but dynamically generates spatial alignment weights based on the contextual content of the features, and combines it with learnable channel scaling parameters to achieve adaptive fusion of cross-branch features. This mechanism effectively alleviates the spatial mismatch and semantic offset between heterogeneous branches, and greatly improves the robustness and discriminative ability of fused features.

To fully exploit and utilize the complementary information of heterogeneous branches at different semantic levels, this study does not adopt the conventional strategy of only performing one-time fusion at the end of the backbone network. Instead, it embeds DAF modules in the three scale stages of P3, P4 and P5 to construct a multi-stage step-by-step fusion paradigm. Assume that the two branch outputs of the stage-s feature stage are respectively $F_s^{\left(1\right)}$ and $F_s^{\left(2\right)}$, then its fusion interaction process can be formalized as:

$$F_s=DAF(F_s^{(1)},F_s^{(2)})$$

(8)

This step-by-step feature interaction mechanism enables continuous information exchange and feature calibration across different spatial scales. Consequently, the subsequent neck network receives multi-scale features with improved consistency and more complete semantic representations, thereby providing a robust feature foundation for object detection in complex scenes. The overall architecture of HeteroFusionNet, including the shared HGStem, heterogeneous dual branches, and multi-stage DAF modules, is illustrated in Figure 3.

Figure 3. Architecture of HeteroFusionNet.

In summary, the advantages of the method proposed in this study are mainly reflected in the following three dimensions. First, the shared HGStem mechanism significantly reduces the redundant computing overhead in the shallow high-resolution stage, allowing the multi-branch structure to maintain strong feature representation capabilities while taking into account excellent inference efficiency. Second, the heterogeneous dual-branch architecture achieves complementary and joint modeling of local fine-grained texture and high-level structural semantics by introducing differentiated feature extraction operators. Third, the DAF module successfully overcomes the inherent spatial differences and semantic conflict bottlenecks in the cross-branch feature fusion process through innovative dynamic spatial alignment and adaptive channel weighting strategies, and achieves high-quality multi-scale feature collaboration.

3.2. Multi-Objective Scale-Aware Integration Network

For bridge defect images, small-scale defect targets usually occupy only a few pixels, and their edge contours, local textures, and geometric shapes are prone to rapidly attenuating during the continuous downsampling process, resulting in a lack of sufficient discriminative information in deep features [21,22]. Existing research usually enhances small-object representation capabilities by introducing higher-resolution shallow detection layers, but this approach is often accompanied by a significant increase in computational costs, especially during dense convolution on high-resolution feature maps, candidate-box generation, and subsequent decoding processes, which will significantly weaken the efficiency advantage of the real-time detection framework [23]. Similar problems have been reflected in existing bridge defect detection research, that is, although shallow information is crucial for the identification of small defects, directly improving the high-resolution prediction branch usually brings additional inference burden and deployment cost.

To alleviate the above problems, this study proposes a Multi-Objective Scale Aware Integration Network (MOSAIN). Different from the solution of directly adding a P2 detection head, MOSAIN does not change the original P3-P5 three-scale prediction framework of RT-DETR. Instead, it adopts the design idea of “shallow detail sinking + cross-scale multi-source fusion + high-order feature integration” to inject the high-resolution detail information sensitive to small objects in the shallow P2 into the P3 path, and uses the CSPOmniKernel module after the fusion node to further enhance its local detail modeling and global context awareness capabilities. In this way, the model can improve the detection capabilities of small objects with a small additional computational cost while keeping the overall decoding structure unchanged.

Let the multi-scale feature set output by the backbone network be {P2, P3, P4, P5}. In standard RT-DETR, the detection head only uses three scales: P3, P4 and P5 for prediction. This study introduces the MOSAIN module on the P3 path to reconstruct enhanced feature representation for small-object detection. The overall process can be expressed as

$$P^{\prime }3=FSOEP(P2,P3,P4)$$

(9)

where FSOEP(⋅) denotes the overall mapping function of MOSAIN, including operations such as shallow detail injection, multi-source feature fusion, and high-order feature integration. The obtained enhanced feature P3′ will be input into the Transformer decoder together with subsequent scale features to complete target prediction.

In this structure, shallow P2 features are first used to supplement the local detail information of small objects that is easily lost in deep networks. Considering that P2 has a higher spatial resolution, this study uses SPDConv to downscale it instead of directly compressing it through step convolution or pooling. Specifically, given the input feature P2 ∈ R^C×H×W, its detail injection process is defined as

$$\widehat{P}2=C(SPD(P2))$$

(10)

where SPD(⋅) denotes the deterministic rearrangement operation from space to channel, C(⋅) denotes the subsequent nonlinear convolution mapping, and P2 is the enhanced feature that is completely consistent with the spatial dimension of P3 after transformation.

Compared with traditional downsampling methods, SPDConv first rearranges local spatial neighborhoods to the channel dimension, and then integrates cross-channel information through convolution. Therefore, it can complete scale transformation while retaining high-frequency detailed information in shallow layers as much as possible. This is particularly critical for small-object detection, because the discriminative cues for tiny defect targets are usually concentrated in edge responses, weak textures and local geometric shapes. With this space-to-channel rearrangement mechanism, MOSAIN can improve the sensitivity of P3 features to small-object details without introducing a high-resolution detection layer.

After completing the encoding of shallow details, MOSAIN Multi-source heterogeneous features are further aggregated into the P3 path. Specifically, the enhanced shallow features $\widehat{P}2$, the current layer feature P3 and the semantic context obtained by upsampling the higher layer feature P4 are jointly fused in the channel dimension to form a multi-source concatenation feature:

$$F_{cat}=[\widehat{P}2;P3;U(P_4)]$$

(11)

where $U(\cdot )$ denotes the upsampling operation, $[\cdot ;\cdot ]$ denotes the concatenation of channel dimensions.

This design enables the enhanced P3 to have three types of complementary information sources at the same time: First, P2 retains richer local details of small objects to make up for the lack of high-frequency information in deep features; second, P3 maintains the structural expression ability of the original mesoscale features; third, U(P4) introduces a higher-level global semantic context, which helps to enhance the model’s understanding of the relationship between small-object categories and the surrounding environment in complex scenes. Therefore, MOSAIN does not simply add a shallow branch, but establishes a cross-layer joint modeling mechanism for small-object detection within the P3 scale.

However, directly using the concatenated F_cat as output may still cause problems such as feature redundancy, semantic conflicts, and inconsistent information distribution. To this end, this study introduces the CSPOmniKernel module after the fusion node to further integrate and enhance multi-source features. Assuming that the input concatenation feature is F_cat, the output can be expressed as

$$F_{out}={\phi }_2([\mathcal{O}(F_a);F_b])$$

(12)

where F_a and F_b are obtained by proportionally splitting F_cat along the channel dimension. The two sub-features are then fed into the complex modeling enhancement branch and the identity shortcut branch, respectively. O(·) denotes the core OmniKernel operator, and φ₂(·) denotes the output aggregation convolution.

Inside the OmniKernel operator O(·), the features first pass through a 1 × 1 convolution for initial channel interaction and are then fed into multiple depthwise separable convolution branches with different receptive fields, including 1 × 1, 1 × k, k × 1, and k × k large-kernel branches, to capture information ranging from local fine-grained patterns to large-scale global context. More importantly, the module incorporates frequency-domain modulation and spatial-channel recalibration mechanisms to strengthen high-frequency boundary signals and suppress low-frequency background noise in the frequency domain. The final response is summarized below:

$$O(x)={\phi }_o(x+K_{1\times 1}(x)+K_{1\times k}(x)+K_{k\times 1}(x)+K_{k\times k}(x)+G(x))$$

(13)

where $K_{\ast }(\cdot )$ refers to deep convolution kernels with different spatial receptive field sizes, G() denotes the global modulation branch that integrates frequency domain transformation and channel attention, ${\mathit{\varphi }}_o(\cdot )$ map for output.

The purpose of introducing CSPOmniKernel is to further enhance feature quality after multi-source feature fusion. First, the multi-scale large-core convolution structure can expand the receptive field, thereby improving the model’s spatial recognition ability of small objects in complex backgrounds. Second, the frequency domain enhancement branch supplements the representation ability of traditional spatial domain convolution from the perspective of frequency response, helping to improve the stability of weak target responses and suppress background noise interference. Finally, the CSP-based split structure allows only some channels to participate in more complex feature enhancement operations, which improves representation capabilities while controlling additional computing overhead.

In summary, without changing the original three-scale prediction framework of RT-DETR, MOSAIN effectively enhances the feature representation capability of the P3 path for small objects through shallow high-resolution feature injection, multi-source cross-scale fusion, and high-order feature integration. Compared with directly adding a P2 detection layer, the proposed method avoids the additional computational and post-processing costs associated with a high-resolution prediction branch. Meanwhile, it preserves detailed shallow-layer information and employs CSPOmniKernel to strengthen the collaborative modeling of local and global features. Therefore, MOSAIN improves small-object representation while controlling the additional computational overhead, thereby providing a computationally efficient basis for future deployment-oriented evaluation. The overall architecture of MOSAIN is illustrated in Figure 4.

Figure 4. Architecture of the Multi-Objective Scale-Aware Integration Network.

3.3. Global Attention Two-Dimensional Module

Although RT-DETR achieves a good balance between detection accuracy and inference efficiency, its high-level feature encoding stage still mainly relies on the original AIFI module to complete global context modeling. This module essentially follows the serialization modeling paradigm of the standard Transformer encoder, that is, first flattening the two-dimensional feature map into a one-dimensional Token sequence, and then combining multi-head self-attention and explicit position encoding for global interaction. Although this type of design can effectively establish long-range spatial dependence, it still has two shortcomings in modeling dense visual features. First, the sequence-based feature organization method is not completely consistent with the two-dimensional topological structure output by the convolutional backbone network, which easily weakens the spatial local continuity during the encoding process [24]. Second, traditional linear feedforward network (FFN) and sequence dimension-oriented normalization methods are more suitable for general visual Transformer, but are not optimal for two-dimensional dense features in object detection tasks. In addition, although standard self-attention has good global modeling capabilities, its implementation lacks structural compatibility when it is subsequently expanded to low-bit, energy-efficient inference. For tasks such as bridge defect detection that require global semantic perception and emphasize efficient deployment, the above problems will limit the adaptability and scalability of high-level feature encoding to a certain extent.

To this end, this study proposes a new high-level feature encoding layer Global Attention Two-dimensional Module, denoted as GATM. Different from the original AIFI, this module uses two-dimensional feature maps as the core representation form while maintaining the global attention modeling ability, and implements global correlation modeling through the BinaryAttention framework. At the same time, this study replaces the original linear FFN with a convolutional feed-forward network, and adopts channels-first 2D layer normalization (2D LayerNorm) to improve the coding layer’s ability to adapt to dense visual features.

Let the input feature be X ∈ R^B×C×H×W, where B, C, H and W represent the batch size, number of channels, height and width respectively. The proposed coding layer consists of an attention branch A(⋅) and a convolutional feedforward branch F(⋅), and combines residual connections and two-dimensional layer normalization to achieve feature update. Its overall forward propagation process can be expressed as

$$\widehat{X}=Nor{m}_1(X+A(X))$$

(14)

$$Y=Nor{m}_2(\widehat{X}+F(\widehat{X}))$$

(15)

where Norm₁ and Norm₂ represent 2D LayerNorm, $\widehat{ X}$ is the intermediate feature after the attention branch output, and Y is the final output of the encoding layer.

In the attention branch, the input features are first flattened into Token sequences. Set spatial scale $N = H \times W$, the features are rearranged as $X_f \in R^{B\times N\times C}$. Query, key and value tensors are then generated via linear mapping:

$$Q,K,V=Proj(X_f)$$

(16)

where $\mathrm{Q}, \mathrm{K}, \mathrm{V} \in R^{B\times h\times N\times d}$, h denotes the number of attention heads, and d = C/h is the feature dimension of each head. The attention weights are calculated using the scaled dot-product attention mechanism:

$$\mathrm{S}=\mathrm{Softmax}({\mathrm{QK}}^{\wedge }\mathrm{T}/\mathrm{qrt}(\mathrm{d}))$$

(17)

The feature aggregation output is $\mathit{Z} = \mathit{SV}$. After completing the multi-head concatenation, it is restored to the two-dimensional spatial topology through linear projection:

$$A(X)=Reshape(Pro{j}_o(Z))$$

(18)

It should be noted that although this branch still performs global Token interaction, its underlying implementation has been migrated to the BinaryAttention framework. Therefore, while maintaining the current global modeling capabilities, it has the structural foundation to further expand to a low-bit attention mechanism.

In the feedforward branch, this study abandons the linear structure of AIFI and uses convolutional FFN. Given input $\widehat{\mathrm{X}}$, the calculation process is

$$F(\widehat{X})=W_2(\sigma (W_1(\widehat{X})))$$

(19)

where ${ W}_1$ and $W_2$ denotes 1 × 1 Point-by-point convolution, $\mathsf{\sigma }(·)$ Activation function for GELU. 1 × 1 Convolution not only retains the ability to interact with cross-channel information, but also directly acts on two-dimensional feature maps, effectively bridging the field gap in feature organization forms between Transformer and convolutional backbone networks. To further adapt to this structure, this study adopts Channels-first 2D LayerNorm. Different from the traditional normalization that acts on the sequence dimension, this strategy directly normalizes spatial features in the channel dimension, which greatly improves the training stability and feature expression quality of the model in dense prediction tasks.

In summary, GATM offers two main advantages over the original AIFI module. First, it retains the global self-attention mechanism for modeling long-range spatial dependencies, thereby enhancing high-level semantic representations. Second, the convolutional FFN and channels-first 2D LayerNorm make the encoding process more suitable for the two-dimensional visual features used in object detection. The detailed architecture of GATM is illustrated in Figure 5.

Figure 5. Architecture of the Global Attention Two-Dimensional Module.

4. Experimental Results and Analysis

4.1. Experimental Environment and Hyperparameters

All experiments in this study were completed under a unified hardware platform, software environment and training strategy. The specific settings are shown in Table 2.

Table 2. Hardware, software, and training configurations.

Hardware environment	CPU	Intel(R) Xeon(R) Platinum 8255C
	RAM	256 GB
	GPU	RTX4090(24 GB)
Software environment	Operating system	Ubuntu 18.04
	Programming language	Python3.8
	Deep learning frameworks	PyTorch1.11.0
Training hyperparameters	LearningRate	0.01
	ImageSize	640 × 640
	Momentum	0.937
	Optimizer	SGD
	BatchSize	64

4.2. Datasets

4.2.1. Self-Built Bridge Defect Dataset

In this study, 1200 original bridge-defect images were collected from multiple bridges in China to construct a bridge surface defect detection dataset. All original images have a resolution of 1080 × 720 pixels. The dataset contains 4536 manually annotated defect instances from four common bridge surface defect categories: corrosion, crack, exposed rebar, and concrete spalling. All defect regions were annotated using Labelme and subsequently converted into YOLO-format bounding-box labels.

To prevent data leakage, the 1200 original images were first divided at the image level into training, validation, and test subsets at a ratio of 7:2:1, corresponding to 840, 240, and 120 original images, respectively. Data augmentation was then performed independently within each subset. The augmentation operations included rotation, flipping, shearing, geometric deformation, scaling, blurring, noise injection, and brightness adjustment. Each original image generated five samples in total, including the original image and four augmented variants. Consequently, the training, validation, and test subsets contained 4200, 1200, and 600 images, respectively. This procedure ensured that augmented variants derived from the same source image were assigned to the same subset and were not distributed across different subsets.

Table 3 reports the category distribution and object-scale statistics of the augmented dataset. The dataset contains 22,680 annotated defect instances after augmentation. The four categories are moderately imbalanced: exposed rebar and spalling account for 34.79% and 34.17% of all instances, respectively, whereas corrosion and crack account for 16.60% and 14.44%. Object scale was determined using the normalized bounding-box area, $r=(w\times h)/(W\times H)$, where $\mathrm{w}$ and $\mathrm{h}$ denote the bounding-box width and height and $\mathrm{W}$ and $\mathrm{H}$ denote the image width and height. Instances with $r<0.01$, $0.01\le r<0.09$, and $\mathrm{r}\ge 0.09$ were categorized as small, medium, and large objects, respectively. Overall, small, medium, and large defects account for 39.6%, 32.2%, and 28.2% of all instances, respectively. Cracks and corrosion contain relatively high proportions of small instances, whereas spalling is predominantly represented by large instances. These characteristics make the dataset suitable for evaluating detection performance across heterogeneous defect scales.

Table 3. Category distribution and object-scale statistics of the self-built bridge defect dataset.

Defect Category	Instances After Augmentation	Proportion	Small	Medium	Large
Corrosion	3765	16.60%	2060 (54.7%)	1210 (32.1%)	495 (13.1%)
Crack	3275	14.44%	1980 (60.5%)	835 (25.5%)	460 (14.0%)
Exposed rebar	7890	34.79%	3710 (47.0%)	3320 (42.1%)	860 (10.9%)
Spalling	7750	34.17%	1225 (15.8%)	1940 (25.0%)	4585 (59.2%)
Total	22,680	100%	8975 (39.6%)	7305 (32.2%)	6400 (28.2%)

4.2.2. RDD (China)

This study also verifies the cross-scenario generalization ability of the model on the RDD (China) road defect dataset. This dataset contains 4378 Chinese road images, including 1977 images taken by unmanned aerial vehicles (UAV) and 2401 images taken by vehicle-mounted cameras, containing a total of 7718 defect annotation boxes. The dataset covers five types of road defects: transverse cracks (D10), longitudinal cracks (D00), potholes (D40), network cracks (D20) and road repair (Repair). The experiment divided the training set and validation set according to the ratio of 8:2, and the images were scaled according to the input size set by each model during the training and inference stages [25]. Representative examples of the five road-defect categories are shown in Figure 6.

Figure 6. Examples of different road-defect categories in the RDD dataset: (a) longitudinal crack (D00); (b) transverse crack (D10); (c) pothole (D40); (d) network crack (D20); and (e) repaired area (Repair).

All images were resized to the input resolution specified for each model during training and inference. It should be emphasized that the input size of 640 × 640 pixels used for MS-DETR refers to the network input after preprocessing and should not be interpreted as a validated minimum native-image resolution. Since no controlled resolution-sensitivity experiment was conducted in the present study, the minimum source-image resolution required for reliable identification of longitudinal and transverse cracks has not yet been established. Determining such a threshold would require systematically downsampling the same source images to multiple resolutions while keeping the model and evaluation protocol unchanged. Therefore, the minimum operational image resolution remains to be investigated in future work.

4.2.3. VisDrone2019

VisDrone2019 is a UAV scene object detection dataset built by the AISKYEYE team of the Machine Learning and Data Mining Laboratory of Tianjin University. The dataset contains 6471 training images, 548 validation images, and 3190 test images, covering a variety of scenes in 14 cities in China, including urban and rural environments, sparse and dense targets, sunny and cloudy days, day and night, and other conditions. Compared with conventional object detection datasets, VisDrone images have a higher proportion of small objects and dense targets, and complex background interference. Therefore, they are often used to evaluate the multi-scale detection capabilities and generalization performance of detection algorithms in drone scenes [26].

4.3. Evaluation Metrics

To objectively evaluate model performance, this study uses five metrics: GFLOPs, parameters, mAP50, F1-score, and FPS. GFLOPs (giga floating-point operations) quantifies the computational cost of a model; parameters measure the model size and complexity; and FPS measures inference speed. mAP (mean average precision) evaluates detection accuracy. In this study, mAP is reported at an IoU threshold of 0.5 (mAP50), where a predicted bounding box is considered correct when its intersection over union (IoU) with the ground-truth box is greater than 0.5. The calculation of mAP is shown in Equation (20). The F1-score, calculated from precision and recall, provides a balanced measure of detection performance, as shown in Equation (21).

$$mAP={\displaystyle \frac{\sum P_A}{N}}$$

(20)

$$F1-\mathrm{score}=2\times {\displaystyle \frac{Precision\times Recall}{Precision+Recall}}$$

(21)

In Equation (20), N denotes the number of defect categories, and AP_i denotes the average precision of the i-th category, namely the area under the precision-recall curve. Precision is the proportion of correctly detected positive samples among all samples predicted as positive, whereas recall is the proportion of correctly detected positive samples among all ground-truth positive samples. The F1-score is the harmonic mean of precision and recall. The formulas for precision and recall are shown in Equations (22) and (23).

$$Precision={\displaystyle \frac{TP}{TP+FP}}$$

(22)

$$Recall={\displaystyle \frac{TP}{TP+FN}}$$

(23)

where TP (true positives) denotes correctly detected defect instances, FP (false positives) denotes false detections, and FN (false negatives) denotes missed defect instances.

4.4. Ablation Study

To evaluate the specific contribution of each improved module to model performance, this study uses RT-DETR as the baseline model and conducts ablation experiments under the same dataset, training strategy and experimental environment. The experiment examines the impact of GATM, MOSAIN and HeteroFusionNet on detection accuracy, model size, computational complexity and inference speed respectively. Evaluation indicators include the number of parameters (Params), floating point operations (FLOPs), mAP50 and FPS. The results of the ablation experiment are shown in Table 4.

Table 4. Ablation study of GATM, MOSAIN, and HeteroFusionNet.

GATM	MOSAIN	HeteroFusionNet	Backbone	Para (M)	FLOPs (G)	mAP50 (%)	FPS
			ResNet-18	20.18	58.6	60.0	64.4
√			ResNet-18	19.87	57.2	61.6	92.4
	√		ResNet-18	20.59	65.5	62.2	60.2
		√	HeteroFusionNet	14.08	34.4	61.2	48.6
√	√		ResNet-18	20.49	65.4	63.9	79.7
√		√	HeteroFusionNet	13.98	34.2	62.4	68.1
	√	√	HeteroFusionNet	14.33	38.9	64.7	58.6
√	√	√	HeteroFusionNet	14.33	39.1	65.4	62.3

Note: “√” indicates that the corresponding module is included in the model configuration, whereas a blank cell indicates that the module is not included.

It can be seen from the single module experimental results that all three modules can improve the performance of the baseline model in different aspects. After the introduction of GATM, the model’s mAP50 increased from 60.0% to 61.6%, while the number of parameters decreased from 20.18 M to 19.87 M, FLOPs decreased from 58.6 G to 57.2 G, and FPS increased from 64.4 to 92.4. This shows that GATM reduces computational overhead while enhancing high-level feature expression and significantly improves inference efficiency. After the introduction of MOSAIN, the model mAP50 increased to 62.2%, achieving the largest accuracy gain in a single module. Although its number of parameters and FLOPs increased to 20.59 M and 65.5 G respectively, and FPS dropped to 60.2, this result shows that MOSAIN has a strong enhancement effect on small objects and complex defect characteristics. After replacing the original ResNet-18 backbone with the proposed heterogeneous backbone and incorporating DAF, the number of model parameters decreased to 14.08 M, while the computational cost decreased to 34.4 GFLOPs and mAP50 increased to 61.2%. This result reflects the joint effect of the backbone redesign and the DAF-based heterogeneous feature-fusion strategy.

The dual-module results further clarify the relationships among the three modifications. Relative to the RT-DETR baseline, GATM, MOSAIN, and HeteroFusionNet provide individual gains of 1.6, 2.2, and 1.2 percentage points in mAP50, respectively. The GATM + MOSAIN combination achieves 63.9% mAP50, which is 0.1 percentage points above the simple additive reference. The MOSAIN + HeteroFusionNet combination achieves 64.7% mAP50, exceeding the corresponding additive reference by 1.3 percentage points and indicating a strong complementary effect between small-object enhancement and the lightweight heterogeneous backbone. In contrast, GATM + HeteroFusionNet achieves 62.4% mAP50, which is 0.4 percentage points below the simple additive reference of 62.8%. Although this combination still outperforms either individual variant, the sub-additive gain suggests partial overlap between the high-level feature enhancement introduced by GATM and the backbone-level representation change introduced by HeteroFusionNet. Therefore, the three modifications should not be interpreted as uniformly synergistic; instead, they provide different accuracy–efficiency trade-offs under the adopted experimental setting.

When the three modules of GATM, MOSAIN and HeteroFusionNet are introduced at the same time, the model achieves the best comprehensive performance. The mAP50 of the final model reaches 65.4%, which is 5.4 percentage points higher than the 60.0% of the original RT-DETR; the number of parameters is reduced from 20.18 M to 14.33 M, a reduction of approximately 29.0%; FLOPs are reduced from 58.6 G to 39.1 G, a reduction of approximately 33.3%. Meanwhile, the complete model achieves an inference speed of 62.3 FPS on an RTX 4090 GPU at an input resolution of 640 × 640, indicating competitive computational efficiency under the evaluated desktop-GPU setting. These results indicate that the three modules proposed in this study are not simply superimposed, but form a good balance between feature enhancement, lightweight modeling and inference efficiency.

4.4.1. Ablation of the HeteroFusionNet Fusion Strategy

To isolate the contribution of the Dynamic Align Fusion (DAF) module, the shared HGStem, the two heterogeneous branches, all fusion stages, and the downstream detection components were kept unchanged. Only the cross-branch fusion operation was replaced. Element-wise addition and direct concatenation were used as static fusion baselines. For the concatenation variant, a 1 × 1 projection layer was added after concatenation to match the output channel dimension of DAF.

The results in Table 5 demonstrate the independent contribution of the DAF module under an otherwise identical heterogeneous feature-fusion architecture. Element-wise addition achieved a mAP50 of 60.3%, while direct concatenation with a 1 × 1 projection slightly improved the result to 60.5%. In contrast, DAF achieved the highest mAP50 of 61.2%, corresponding to improvements of 0.9 and 0.7 percentage points over element-wise addition and direct concatenation, respectively.

Table 5. Ablation study of fusion strategies in HeteroFusionNet.

Fusion Strategy	mAP50 (%)	Para (M)	GFLOPs	FPS
Element-wise addition	60.3	10.84	29.1	55.8
Direct concatenation + 1 × 1 projection	60.5	11.01	29.4	54.1
DAF	61.2	14.08	34.4	48.6

These results indicate that the performance gain of HeteroFusionNet is not solely attributable to the use of two heterogeneous feature branches. The adaptive spatial alignment and channel-wise weighting in DAF enable more effective interaction between feature maps with different spatial responses and semantic distributions. This improves the consistency and discriminability of the fused representation. The improvement is achieved at the cost of 3.24 M and 3.07 M additional parameters compared with element-wise addition and direct concatenation, respectively. The corresponding computational cost increases by 5.3 and 5.0 GFLOPs, while the inference speed decreases by 7.2 and 5.5 FPS. Thus, DAF provides a measurable accuracy improvement with a moderate computational overhead.

The second experiment examined whether the specific combination of C2f with standard convolution and HGBlock with depthwise separable convolution was preferable to other possible branch configurations. The spatial resolutions, channel dimensions, downsampling positions, DAF modules, and downstream detection components were kept unchanged in all variants. Only the operator combination used in the two branches was modified.

The proposed configuration achieves the highest mAP50 of 61.2%. Compared with the homogeneous C2f configuration, it improves mAP50 by 0.2 percentage points while reducing the number of parameters by 1.65 M and the computational cost by 3.5 GFLOPs. This indicates that replacing the second C2f branch with an HGBlock-based branch provides additional hierarchical feature representation while reducing computational redundancy.

Compared with the homogeneous HGBlock configuration, the proposed design improves mAP50 by 1.0 percentage point. Although the proposed configuration introduces 0.86 M additional parameters and 0.3 GFLOPs, the increase in computational cost is relatively limited compared with the accuracy improvement. This result suggests that the C2f branch is more effective for preserving local texture and boundary information required by bridge-defect detection.

The operator-swapped configuration achieves an mAP50 of 60.5%, which is 0.7 percentage points lower than the proposed configuration, despite having the same number of parameters and GFLOPs. This comparison is particularly important because it shows that the performance gain does not simply result from using two different types of operators. Instead, the assignment of the operators to the two branches also affects the detection performance. Combining C2f with standard convolution in Branch A and HGBlock with depthwise separable convolution in Branch B provides a more suitable balance between local-detail preservation, hierarchical feature aggregation, and computational efficiency.

Overall, the results in Table 5 and Table 6 demonstrate that both the fusion operation and the branch composition influence the performance of HeteroFusionNet. The proposed DAF module improves the interaction between heterogeneous features, while the selected branch combination provides complementary representations for different bridge defects. In particular, the C2f–standard convolution branch is more suitable for retaining fine-grained information from thin cracks, exposed rebar, and defect boundaries, whereas the HGBlock–depthwise separable convolution branch provides additional hierarchical and contextual information for corrosion, spalling, and defects under complex backgrounds. Therefore, the proposed HeteroFusionNet is not a simple duplication of two backbones, but a controlled combination of heterogeneous feature extraction and adaptive feature fusion.

Table 6. Ablation study of branch configurations in HeteroFusionNet.

Variant	Branch A	Branch B	mAP50 (%)	Para (M)	FLOPs (G)
Homogeneous C2f branches	C2f + Conv	C2f + DWConv	61.0	15.73	37.9
Homogeneous HGBlock branches	HGBlock + Conv	HGBlock + DWConv	60.2	13.22	34.1
Operator-swapped heterogeneous branches	C2f + DWConv	HGBlock + Conv	60.5	14.08	34.4
Proposed configuration	C2f + Conv	HGBlock + DWConv	61.2	14.08	34.4

4.4.2. Ablation of the MOSAIN Downsampling Operator

To verify whether SPDConv is responsible for the detail-preservation capability of MOSAIN, the downsampling operation applied to the shallow P2 feature was replaced with a standard strided convolution and a pooling-based alternative. The output spatial resolution and channel dimension were kept identical across all variants. All other components of MOSAIN, including multi-source feature fusion and CSPOmniKernel, remained unchanged. Because all original images have a resolution of 1080 × 720 pixels, small defects were defined as annotated bounding boxes whose areas were smaller than 1% of the original image area:

$$A_{\mathrm{s}\mathrm{m}\mathrm{a}\mathrm{l}\mathrm{l}}<0.01\times 1080\times 720=7776{ \mathrm{p}\mathrm{i}\mathrm{x}\mathrm{e}\mathrm{l}\mathrm{s}}^2.$$

(24)

The comparative results obtained using this definition are summarized in Table 7.

Table 7. Ablation study of downsampling strategies in MOSAIN.

Downsampling Strategy	mAP50 (All) (%)	mAP50 (Small Defects) (%)	Recall (Small Defects) (%)	Para (M)	GFLOPs	FPS
3 × 3 stride-2 convolution + projection	61.3	16.3	36.7	20.45	63.3	63.5
2 × 2 max pooling + projection	60.9	16.1	36.1	20.42	62.8	62.3
SPDConv	62.2	17.3	38.5	20.59	65.5	60.2

The SPDConv variant achieved the highest overall mAP50 of 62.2% and the highest small-defect mAP50 of 17.3%. Compared with the standard stride-2 convolution, SPDConv improved the overall mAP50, small-defect mAP50, and small-defect recall by 0.9, 1.0, and 1.8 percentage points, respectively. Compared with max pooling, the corresponding improvements were 1.3, 1.2, and 2.4 percentage points.

These results provide quantitative evidence that the space-to-channel rearrangement of SPDConv is beneficial for preserving fine edges, weak textures, and local geometric cues during shallow-feature downsampling. The improvement is particularly evident for small defects, which are more vulnerable to information loss caused by conventional downsampling. Compared with the stride-convolution baseline, SPDConv introduces an additional computational cost of 2.2 GFLOPs and reduces the inference speed by 3.3 FPS. Compared with max pooling, the corresponding differences are 2.7 GFLOPs and 2.1 FPS. This represents a non-negligible but acceptable efficiency trade-off for the improved small-defect detection capability.

4.4.3. Ablation of the GATM Feed-Forward Network and Normalization

A 2 × 2 factorial ablation was conducted to disentangle the effects of the convolutional feed-forward network and the two-dimensional layer normalization. The attention branch, number of attention heads, embedding dimension, residual connections, and training protocol were kept unchanged. Only the FFN type and normalization strategy were varied. The results of the factorial ablation study are summarized in Table 8.

Table 8. Factorial ablation of Conv-FFN and 2D LayerNorm in GATM.

FFN Type	Normalization	mAP50 (%)	Para (M)	GFLOPs	FPS
Linear FFN	Standard sequence-wise LayerNorm	59.9	19.87	57.2	95.1
Conv-FFN	Standard sequence-wise LayerNorm	60.2	19.87	57.2	92.5
Linear FFN	Channels-first 2D LayerNorm	60.4	19.87	57.2	88.9
Conv-FFN	Channels-first 2D LayerNorm	61.6	19.87	57.2	92.4

The influence of Conv-FFN can be evaluated by comparing the first two rows under standard sequence-wise LayerNorm and the last two rows under channels-first 2D LayerNorm. Under standard LayerNorm, replacing Linear FFN with Conv-FFN increased mAP50 from 59.9% to 60.2%, corresponding to a gain of 0.3 percentage points. When channels-first 2D LayerNorm was used, the same replacement increased mAP50 from 60.4% to 61.6%, resulting in a larger improvement of 1.2 percentage points.

The contribution of the normalization strategy can be analyzed by comparing the first and third rows under Linear FFN and the second and fourth rows under Conv-FFN. Replacing standard sequence-wise LayerNorm with channels-first 2D LayerNorm improved mAP50 by 0.5 percentage points with Linear FFN and by 1.4 percentage points with Conv-FFN. These results indicate that channels-first 2D LayerNorm is more effective at preserving the spatial and channel-wise statistics of the two-dimensional feature maps used by GATM.

Among the four variants, the combination of Conv-FFN and channels-first 2D LayerNorm achieved the highest mAP50 of 61.6%. The improvement brought by Conv-FFN was larger when it was combined with 2D LayerNorm, while the benefit of 2D LayerNorm was also more evident when Conv-FFN was adopted. This consistent performance pattern suggests a positive interaction between the two components. Conv-FFN enhances local spatial feature transformation through convolutional operations, whereas channels-first 2D LayerNorm normalizes the feature responses directly in their spatial representation. Their combination therefore provides more effective modeling of local structural patterns and spatially varying defect features.

All four variants have the same number of parameters and nearly identical computational complexity, with approximately 19.87 M parameters and 57.2 GFLOPs. Therefore, the observed accuracy differences cannot be attributed to increased model capacity. The differences in FPS are relatively small and mainly reflect the implementation and runtime characteristics of the corresponding operators. Overall, the factorial ablation confirms that both Conv-FFN and channels-first 2D LayerNorm contribute to the performance of GATM, while their combination provides the most effective feature transformation and normalization strategy.

4.5. Comparison with Other Models

To comprehensively evaluate the comprehensive performance of MS-DETR in complex bridge defect detection scenarios, this study selects a variety of representative object detection models for comparative experiments. The comparison methods cover the two-stage detector Faster R-CNN [27], the single-stage detectors FCOS [28], RetinaNet [29], CenterNet [30] and EfficientNet [31], the YOLO series models YOLOv5, YOLOv8, YOLOv10 [32] and Hyper-YOLO [33], and the detection models TOOD [34], DDQ [35], DINO [36] and RT-DETR based on Transformer or dynamic label allocation mechanism. All models were tested on the same computing equipment and software environment, and the evaluation metrics included mAP50, number of parameters, inference speed, and F1-score. The specific results are shown in Table 9.

Table 9. Comparison of MS-DETR with representative object detectors on the self-built bridge defect dataset.

Methods	Backbone	Size	mAP50 (%)	Para (M)	FPS	F1-Score
Faster R-CNN	ResNet-50	(640, 640)	56.3	41.38	32.1	0.58
FCOS	ResNet-50	(640, 640)	53.1	32.29	29	0.53
TOOD	ResNet-50	(640, 640)	55.5	32.03	23.4	0.56
DDQ	ResNet-50	(640, 640)	59.2	48.32	12.5	0.57
ResNet-50	ResNet-50	(640, 640)	36.3	14.21	113.7	0.42
RetinaNet	ResNet50	(640, 640)	55.0	19.83	38.6	0.49
EfficientNet	EfficientNet-b3	(640, 640)	52.4	18.40	34.1	0.54
DINO	ResNet-50	(640, 640)	40.6	47.55	15.7	0.45
YOLOv10	DarkNet53	(640, 640)	56.1	7.24	90.3	0.55
YOLOv8	DarkNet53	(640, 640)	58.6	11.15	123.5	0.60
YOLOv5	DarkNet53	(640, 640)	57.3	9.14	110.7	0.59
Hyper-YOLO	MANet	(640, 640)	57.9	14.84	82.5	0.58
RT-DETR	ResNet18	(640, 640)	60.0	20.18	64.4	0.61
MS-DETR (Ours)	HeteroFusionNet	(640, 640)	65.4	14.33	62.3	0.64

As shown in Table 9, MS-DETR achieved the best performance in terms of detection accuracy, with mAP50 reaching 65.4% and F1-score reaching 0.64. Compared with Faster R-CNN, the mAP50 of MS-DETR is improved by 9.1 percentage points and the F1-score is improved by 0.06; compared with DDQ, the mAP50 is improved by 6.2 percentage points. Compared with the baseline model RT-DETR, the mAP50 of MS-DETR increased from 60.0% to 65.4%, an increase of 5.4 percentage points, while the F1-score increased from 0.61 to 0.64. The above results show that MS-DETR has stronger target representation capabilities in bridge images with complex backgrounds, scale changes, and large differences in defect morphology.

Model size and inference speed are important factors affecting the potential deployment of visual defect detection models. The parameter numbers of Faster R-CNN, DDQ, and DINO are 41.38 M, 48.32 M, and 47.55 M, respectively, indicating relatively high model complexity. In comparison, MS-DETR contains 14.33 M parameters, which is approximately 29.0% lower than the 20.18 M parameters of the original RT-DETR baseline. Although models such as YOLOv5, YOLOv8, and YOLOv10 achieve higher inference speeds or smaller parameter sizes, their mAP50 and F1-scores on the self-built bridge defect dataset are lower than those of MS-DETR. Under the RTX4090 experimental environment and 640 × 640 input size, MS-DETR achieved 62.3 FPS, suggesting good computational efficiency in the tested GPU setting.

In summary, MS-DETR achieves a favorable balance among detection accuracy, model complexity, and inference speed under the adopted experimental setting. Compared with the evaluated two-stage, one-stage, YOLO-series, and Transformer-based detectors, MS-DETR achieves competitive mAP50 and F1-score values while maintaining 14.33 M parameters and an inference speed of 62.3 FPS on an RTX 4090 GPU. These results demonstrate that the integration of HeteroFusionNet, MOSAIN, and GATM improves the representation of complex bridge defects while maintaining competitive computational efficiency. A visual comparison of the model parameters, mAP50, and inference speed across the evaluated detectors is presented in Figure 7.

Figure 7. Comparison of parameters, mAP50, and inference speed across object detectors.

4.6. Auxiliary Cross-Domain Evaluation

4.6.1. Pavement-Distress Evaluation on the RDD (China) Dataset

To further verify the applicability of MS-DETR in different structural damage detection tasks, this study conducted supplementary experiments on the RDD road defect dataset. This dataset contains a variety of road surface defects. The image background is usually interfered by factors such as asphalt texture, pavement markings, shadows, and local wear. There are certain differences in target morphology and imaging background from bridge defect images. Therefore, experiments on this dataset can be used to evaluate the stability of the model in the cross-scenario defect detection task. To ensure the objectivity of the comparison, this study selects a variety of mainstream object detection algorithms and tests them on the same dataset and the same software and hardware environment. The experimental results are shown in Table 10.

Table 10. Comparison of MS-DETR with representative object detectors on the RDD (China) dataset.

Models	Backbone	Size	mAP50 (%)	Para (M)	FPS	F1-Score
Faster R-CNN	ResNet50	600	76.5	41.38	11	0.60
FCOS	ResNet50	640	72.3	32.29	29	0.61
SSD	VGG	300	73.6	26.28	76	0.61
CenterNet	DarkNet53	512	80.1	32.66	57	0.72
RetinaNet	ResNet50	600	79.4	55.41	35	0.63
YOLOv3-tiny	DarkNet53	640	76.1	12.13	222	0.71
YOLOv3	DarkNet53	640	84.6	103.69	59	0.85
YOLOv5s	DarkNet53	640	85.2	7.02	136	0.83
YOLOv6s	DarkNet53	640	85.5	16.29	108	0.81
YOLOv7	DarkNet53	640	86.0	6.02	138	0.82
YOLOv8-s	DarkNet53	640	86.8	11.16	120	0.83
RT-DETR	ResNet18	640	87.1	20.18	64	0.84
MS-DETR (Ours)	HeteroFusionNet	640	88.9	14.33	62.3	0.86

As can be seen from Table 10, MS-DETR achieved an mAP50 of 88.9% and an F1-score of 0.86 on the RDD dataset, both of which are the highest values among all compared models. Compared with Faster R-CNN, the mAP50 of MS-DETR increased from 76.5% to 88.9%, an increase of 12.4 percentage points; compared with YOLOv8-s, the mAP50 increased by 2.1 percentage points. Compared with the baseline model RT-DETR, the mAP50 of MS-DETR is improved by 1.8 percentage points, and the F1-score is improved from 0.84 to 0.86. The above results show that the improvement strategy proposed in this study can not only improve the bridge defect detection performance, but also maintain good detection accuracy in road defect detection tasks.

From the perspective of computational efficiency, MS-DETR contains 14.33 M parameters, approximately 29.0% fewer than the 20.18 M parameters of the original RT-DETR baseline. Although several compact detectors achieve higher FPS or use fewer parameters, their detection accuracy and F1-score are lower under the present evaluation. MS-DETR achieved 62.3 FPS on an RTX 4090 at an input resolution of 640 × 640, indicating a favorable accuracy–efficiency trade-off in the tested desktop-GPU setting.

In summary, the experimental results on the RDD dataset indicate that MS-DETR is not limited to the self-built bridge-defect dataset. When applied to road-defect images containing complex pavement textures and strong background interference, the model maintains competitive mAP50 and F1-score values together with relatively low model complexity. These findings provide additional evidence of the cross-dataset applicability of HeteroFusionNet, MOSAIN, and GATM and support their potential use in visual defect-detection tasks involving different infrastructure scenarios. A visual comparison of the mAP50 and F1-score values achieved by the evaluated object detectors on the RDD dataset is presented in Figure 8.

Figure 8. Comparison of mAP50 and F1-score across object detectors on the RDD dataset.

4.6.2. UAV-Scene Evaluation on the VisDrone2019 Dataset

To further assess the performance of MS-DETR under dense UAV-object detection conditions, comparative experiments were conducted on the VisDrone2019 dataset. VisDrone contains numerous small objects, dense object distributions, severe occlusion, and complex urban backgrounds, thereby providing a challenging benchmark for evaluating multi-scale object detection. Since the model was trained and evaluated using the VisDrone training and validation splits, this experiment is regarded as an additional dataset evaluation rather than a zero-shot out-of-distribution test. Table 11 compares MS-DETR with representative two-stage, one-stage, and transformer-based detectors under a common input resolution of 640 × 640.

Table 11. Comparison of MS-DETR with representative object detectors on the VisDrone2019 dataset.

Model	Input Shape	GFLOPs	Params	AP50	APs	APm	APl
Faster-RCNN	(640, 640)	208 G	41.39 M	0.329	0.095	0.309	0.429
Cascade-RCNN	(640, 640)	236 G	69.29 M	0.326	0.099	0.309	0.406
ATSS	(640, 640)	110 G	38.91 M	0.338	0.100	0.317	0.485
TOOD-R50	(640, 640)	199 G	32.04 M	0.339	0.102	0.317	0.403
DINO	(640, 640)	274 G	47.56 M	0.445	0.150	0.371	0.503
DDQ	(640, 640)	-	-	0.463	0.159	0.390	0.526
YOLOX-Tiny [37]	(640, 640)	7.57 G	5.03 M	0.278	0.076	0.221	0.278
GFL [38]	(640, 640)	206 G	32.27 M	0.321	0.094	0.300	0.409
RTMDet-Tiny [39]	(640, 640)	8.03 G	4.876 M	0.312	0.077	0.288	0.445
D-FINE-N [40]	(640, 640)	24.85 G	10.18 M	0.394	0.128	0.331	0.468
DEIM-FINE-N [41]	(640, 640)	24.85 G	10.18 M	0.384	0.122	0.321	0.397
MS-DETR	(640, 640)	39.1 G	14.33 M	0.487	0.191	0.407	0.553

As shown in Table 11, MS-DETR achieved the highest AP50 of 0.487 among the compared methods. This value is 2.4 percentage points higher than that of DDQ and 4.2 percentage points higher than that of DINO. In addition, MS-DETR obtained APs, APm, and APl values of 0.191, 0.407, and 0.553, respectively. These results indicate that the proposed model provides competitive detection performance across different object scales under the adopted VisDrone evaluation protocol. A normalized visual comparison of the computational complexity, model size, and detection performance of the evaluated object detectors is presented in Figure 9.

Figure 9. Performance comparison of different object detectors on VisDrone.

However, the absolute small-object performance remains limited. To investigate this issue in greater detail, a size-stratified and category-wise analysis was conducted on the VisDrone validation set. According to the area-based evaluation protocol, objects with annotated bounding-box areas smaller than ${32}^2$. pixels were defined as small objects. The validation set contains 26,586 small-object instances, accounting for 68.6% of all valid annotations. The overall APs and ARs are 0.191 and 0.286, respectively, indicating that extremely small-object detection remains the main performance bottleneck.

As shown in Table 12, the performance degradation is not uniform across categories. Awning-tricycle, bicycle, truck, tricycle, and van exhibit the lowest AP values. In particular, bicycle and van contain 1053 and 937 small instances, respectively, but still achieve AP values below 0.17. This result suggests that the difficulty cannot be explained solely by limited sample availability. Instead, severe scale reduction, dense object overlap, frequent occlusion, weak appearance cues, and visual similarity among vehicle categories jointly affect detection performance. The confusion matrix further indicates that bicycles are frequently missed or confused with motorbikes, awning-tricycles are confused with tricycles, and vans and trucks are confused with cars. In contrast, cars achieve a substantially higher APs of 0.435, showing that category separability and contextual appearance also influence small-object detection performance.

Table 12. Category-wise performance for small objects on the VisDrone validation set.

Category	Small Instances	APs	ARs
Awning-tricycle	305	0.072	0.165
Bicycle	1053	0.077	0.173
Truck	289	0.105	0.198
Tricycle	621	0.144	0.264
Van	937	0.161	0.21
People	4631	0.193	0.329
Pedestrian	7746	0.231	0.332
Bus	116	0.235	0.271
Motor	3978	0.26	0.393
Car	6910	0.435	0.524
Overall	26,586 (68.6%)	0.191	0.286

Therefore, although MS-DETR achieves the highest small-object AP among the compared methods, the AP value of 0.191 should be interpreted as a relative improvement rather than evidence that extremely small-object detection has been fully resolved. Future improvements may focus on preserving higher-resolution features, reducing the information loss caused by image resizing, and improving discrimination among visually similar objects in densely populated scenes.

Although MOSAIN is designed to preserve multi-scale detail and GATM strengthens high-level feature representation, the present category-wise analysis does not independently isolate their respective contributions on VisDrone. The remaining failures are consistent with the limitation that feature enhancement cannot fully restore discriminative cues lost through extreme resizing, sparse pixel coverage, and severe occlusion. Thus, the results indicate room for improving the interaction between multi-scale detail preservation and semantic feature propagation, rather than demonstrating that either module alone is responsible for the observed errors.

4.7. Visualization Analysis

4.7.1. Detection Result Visualization

To intuitively evaluate the identification capabilities of different detection models in complex bridge defect scenarios, this study conducts a qualitative visual comparison of Faster R-CNN, DINO, YOLOv10, RT-DETR and the MS-DETR proposed in this study, as shown in Figure 10. The figure shows the prediction results of each model from left to right. The selected samples cover typical bridge defects such as spalling, exposed rebar, corrosion, and crack, and include challenging detection scenarios such as coexistence of multi-scale targets, slender structures, dense small objects, and weak texture backgrounds. Overall, MS-DETR performs relatively stably in terms of target positioning integrity, fine-grained defect identification, and false detection suppression. It is especially suitable for detecting slender cracks, densely exposed rebar, and corrosion areas with blurred boundaries.

Figure 10. Detection results of Faster R-CNN, DINO, YOLOv10, RT-DETR, and MS-DETR under different bridge-defect scenarios. The columns from left to right correspond to the five evaluated models. The rows show: (a) coexistence of spalling and multiple slender exposed-rebar defects; (b) corrosion with blurred boundaries and complex background textures; (c) a slender and continuously extending crack; and (d) dense coexistence of large-area spalling and multiple small exposed-rebar defects.

The first group of samples contains both spalling and multiple slender exposed rebars, which is a typical multi-defect coexistence scenario. Faster R-CNN and DINO can detect the main spalling areas, but the positioning of the exposed rebar is not stable enough, and some prediction boxes have problems with scattered responses or incomplete coverage. YOLOv10 and RT-DETR improved on multi-object recognition, but still had fluctuations in confidence and box localization of local elongated objects. In contrast, MS-DETR can give more consistent detection results for multiple exposed rebar instances while maintaining the complete positioning of the spalling area, indicating its stronger adaptability in joint characterization of multi-scale defects.

The second group of samples mainly shows the detection performance for corrosion defects. This type of defect often has blurred borders, irregular texture distribution, and can be easily confused with the concrete background or exposed rebar. It can be seen from the visualization results that Faster R-CNN, DINO and YOLOv10, while covering the main corrosion area, are still accompanied by varying degrees of redundant responses, and some areas are misjudged as other defect categories. The prediction results of RT-DETR and MS-DETR are more concentrated, and the prediction box of MS-DETR is more concise and closer to the ground-truth corrosion region, indicating that the model has better category discrimination ability and false detection suppression ability for complex texture backgrounds.

The third group of samples is a typical crack detection scenario. Compared with spalling and corrosion, cracks usually have the characteristics of slenderness, low contrast, and continuous extension, which place higher requirements on the model’s long-range structural modeling capabilities. Faster R-CNN has obvious frame fragmentation in this scene, accompanied by low-confidence responses; although DINO can cover part of the crack area, there is still category confusion; RT-DETR has a strong response to the main body of the crack, but the prediction results are still divided into multiple local boxes. In contrast, MS-DETR can form a more continuous and complete positioning result along the main crack trajectory, and better maintain the geometric extension characteristics of the crack.

The fourth set of samples further demonstrates the complex scenario of dense coexistence of spalling and exposed rebar. In this type of image, the spalling area is large, and the multiple exposed rebar targets are compactly distributed and small in scale, which can easily lead to missed detections, false detections, or category confusion. Faster R-CNN, DINO and YOLOv10 can all identify primary spalling regions, but the detection of multiple exposed rebars still suffers from insufficient local confidence or unstable response. RT-DETR has better identified major defect targets, but the response to some small-scale exposed rebar is still weak. In contrast, MS-DETR is more stable in positioning spalling and maintains good consistency and integrity for densely exposed rebar targets, indicating that it has stronger robustness in dense small-object detection scenarios.

Based on the above visualization results, MS-DETR shows a relatively stable comprehensive detection capability in multiple types of bridge defect scenarios. Its advantages are mainly reflected in three aspects: first, in multi-scale scenes where spalling and exposed rebar coexist, MS-DETR can balance the joint detection of large-area defects and small-scale slender targets; second, in scenes with blurred boundaries and complex textures such as corrosion, the model can reduce background interference and category confusion; third, on continuous slender structural targets such as crack, the model can obtain more complete spatial positioning results. These qualitative results are basically consistent with the aforementioned quantitative experimental results, further demonstrating the effectiveness of MS-DETR in multi-scale feature modeling, detail preservation, and complex background robustness.

4.7.2. Heatmap Visualization

To further analyze the ability of different detection models to focus on bridge surface defect regions, this study uses heat maps to visually compare Faster R-CNN, DINO, YOLOv10, RT-DETR and the MS-DETR proposed in this study, as shown in Figure 11. The red and yellow areas in the heat map represent the discriminative areas where the model has a higher response, and the blue area denotes the background area where the response is weaker. A variety of typical bridge defect samples are selected in the figure, including local fine cracks, long cracks, cross cracks, and concrete spalling/corrosion areas, to verify the feature focusing ability of the model under different scales, shapes, and background interference conditions.

Figure 11. Heatmap visualization results of Faster R-CNN, DINO, YOLOv10, RT-DETR, and MS-DETR for different bridge-defect samples. The columns from left to right show the original image and the visualization results of the five evaluated models. The rows correspond to: (a) small and discrete vertical cracks; (b) a long and continuously extending crack; (c) intersecting or cross-shaped cracks; and (d) concrete spalling and corrosion with complex background textures. The heatmap colors represent normalized feature-response intensity: red indicates the highest response, yellow and green indicate intermediate-to-high responses, and blue indicates the lowest response or background region.

It can be seen from the visualization results that Faster R-CNN and YOLOv10 can respond to areas near the defect in some samples, but their high response areas are often scattered and easily cover a large background range. For example, in samples with small cracks, Faster R-CNN’s attention shows obvious lateral diffusion and fails to accurately distinguish multiple discrete defects; YOLOv10 also has the problem of wider response areas in samples with long cracks and cross-cracks, indicating that its feature activation is easily affected by changes in texture and illumination around cracks. DINO and RT-DETR have a relatively stronger ability to capture crack contours and can form a continuous response along the crack trend to a certain extent, but edge noise and non-defect region activation will still appear in complex backgrounds or local damaged areas.

In contrast, MS-DETR exhibited a more concentrated defect response in all four types of samples. For small, discrete vertical cracks, MS-DETR can accurately focus the high response area on the main body of the crack, and the background suppression is more obvious; for long cracks and cross cracks, the heat map of MS-DETR forms a continuous and clear high response zone along the crack direction, which can more completely retain the elongated structural characteristics of the crack; for concrete spalling and corrosion samples, the response of MS-DETR is mainly concentrated in the core damage area, rather than spreading to the surrounding complete concrete background in a large area. The above results show that MS-DETR can more effectively model the multi-scale morphological characteristics of bridge defects and enhance the model’s discriminative attention ability to real defect areas.

Overall, the heat map visualization results verify the effectiveness of MS-DETR from the perspective of interpretability. Compared with the contrasting models, MS-DETR exhibits a more stable attention pattern in small-object defect localization, continuous modeling of slender cracks, and suppression of complex background interference. This shows that the proposed model not only improves detection performance, but also improves feature interpretability and location reliability in the process of bridge defect identification.

4.8. Robustness Analysis

To assess the sensitivity of the proposed model to image-quality variations that may occur during bridge inspection, a controlled corruption robustness analysis was conducted on the validation set. The analysis used the same 100 images and 282 annotated defect instances as the clean validation setting. No additional training or fine-tuning was performed. Four label-preserving image corruptions were applied at two severity levels: low illumination, local shadow, motion blur, and Gaussian noise. The bounding-box annotations were retained unchanged because these perturbations only affected image appearance. The performance decrease was calculated as

$$\Delta M=M_{\mathrm{clean}}-M_{\mathrm{corrupted}}$$

(25)

where a positive value indicates a reduction relative to the clean-image baseline.

As shown in Table 13, the proposed model achieved an mAP50 of 0.654, an mAP50:95 of 0.466, and an F1-score of 0.64 on the clean validation images. Under mild corruptions, performance remained relatively stable, although the effects varied across corruption types. Mild low illumination, motion blur, and Gaussian noise reduced mAP50 by 0.010, 0.049, and 0.008, respectively, whereas mild local shadow resulted in a slight increase of 0.001. This marginal increase is attributed to finite-sample variation rather than an actual improvement in model performance.

Table 13. Robustness of MS-DETR under controlled image corruptions.

Condition	mAP50	mAP50:95	F1	ΔmAP50
Clean	0.654	0.466	0.64	—
Low illumination, mild	0.644	0.451	0.63	0.01
Low illumination, severe	0.571	0.399	0.58	0.083
Local shadow, mild	0.655	0.476	0.64	−0.001
Local shadow, severe	0.645	0.47	0.64	0.009
Motion blur, mild	0.605	0.434	0.62	0.049
Motion blur, severe	0.55	0.372	0.58	0.104
Gaussian noise, mild	0.646	0.457	0.63	0.008
Gaussian noise, severe	0.609	0.405	0.59	0.045

Performance degradation became more apparent as corruption severity increased. Severe low illumination reduced mAP50 to 0.571, corresponding to a decrease of 0.083. Severe Gaussian noise reduced mAP50 to 0.609, with a decrease of 0.045. The largest degradation was observed under severe motion blur, where ${\mathrm{mAP}}_{50}$ decreased to 0.550 and mAP50:95 decreased to 0.372. This result suggests that image motion and loss of boundary sharpness have a stronger adverse effect on defect localization than the other simulated appearance variations considered in this study.

Figure 12 presents the quantitative comparison across corruption conditions. Figure 13 provides representative qualitative examples. The predicted bounding boxes generally remain consistent with the ground-truth regions under clean, shadowed, and mildly degraded conditions. However, severe low illumination, motion blur, and noise can reduce prediction confidence, cause localization offsets, or produce missed and redundant detections, particularly for defects with fine boundaries or weak texture contrast, such as cracks and corrosion regions.

Figure 12. Quantitative robustness of MS-DETR under different controlled image corruptions.

Figure 13. Qualitative detection results of MS-DETR under different controlled image corruptions.

This experiment should be interpreted as a controlled corruption robustness analysis, rather than a complete evaluation of generalization to unseen bridge structural types. Because the corrupted images were derived from the same validation images, the analysis isolates the influence of image degradation while keeping scene content and annotations fixed. It does not represent zero-shot performance on new bridge types, acquisition viewpoints, background materials, or environmental conditions that are absent from training. Future work will therefore collect independent field images from additional bridge structures and inspection conditions to conduct a stricter cross-bridge out-of-distribution evaluation.

5. Discussion

The error cases shown in Figure 14 were further analyzed from the perspective of the model mechanism. The observed failures are not attributed to a single module alone, because the current experiments evaluate the modules mainly through overall detection metrics rather than through sample-level causal intervention. Instead, the following interpretation is based on the error visualizations, the module ablation results, the branch-composition experiments, and the robustness analysis under controlled image corruptions.

Figure 14. Representative error cases of MS-DETR in complex bridge defect scenes.

The most evident error pattern is the confusion among concrete spalling, corrosion, and exposed rebar. This confusion does not necessarily indicate that the heterogeneous dual-branch architecture itself produces semantic ambiguity. In fact, the branch-composition ablation shows that the proposed C2f–standard convolution and HGBlock–depthwise separable convolution configuration achieves an mAP50 of 61.2%, compared with 61.0%, 60.2%, and 60.5% for the homogeneous C2f, homogeneous HGBlock, and operator-swapped configurations, respectively. These results suggest that the heterogeneous architecture provides complementary representations on average. However, residual category confusion can still occur when different defects appear within the same local region. For example, exposed rebar may be surrounded by rust or located inside a spalled concrete area. Corrosion and exposed rebar may also share similar colour and elongated texture cues, whereas the boundary of spalling may contain sharp edges and dark regions that resemble crack responses. Under these conditions, the local features extracted by the two branches may contain similar class-related responses, making it difficult for the detection head to assign an exclusive category.

DAF is designed to reduce the spatial and semantic differences between the two heterogeneous branches, but its dynamic weighting mechanism cannot completely eliminate ambiguity when the input image itself does not contain sufficiently discriminative category information. In particular, near the boundary between corrosion, exposed rebar, and spalling, the two branches may focus on different parts of the same damaged region. The resulting alignment weights may preserve the dominant shared response while weakening a less salient class-specific cue. Therefore, the remaining category confusion may be partly related to residual feature-alignment bias, but it should not be interpreted as evidence that DAF systematically degrades category discrimination. The fusion-strategy ablation demonstrates that DAF performs better than static addition and concatenation under the controlled experimental setting. Its limitation is more likely to appear in visually overlapping or spatially nested defect regions.

A second important failure pattern is the missed detection or fragmented localization of small defects. MOSAIN improves the transmission of high-resolution information to the detection path, and its single-module ablation increases the overall mAP50 from 60.0% to 62.2%. Therefore, the missed small defects do not indicate that MOSAIN fails to provide useful detail information. Nevertheless, high-resolution feature injection cannot fully recover information that has already been lost because of insufficient image resolution, weak defect–background contrast, motion blur, severe illumination variation, or partial occlusion. When a small corrosion spot or exposed-rebar segment occupies only a limited number of pixels, the feature response may remain weak even after multi-scale enhancement. This explains why some small defects are detected with low confidence or are merged into nearby large spalling regions.

The robustness experiments provide further evidence for this interpretation. Severe motion blur and severe low illumination produce the largest mAP50 reductions, with absolute decreases of 0.104 and 0.083 compared with the clean validation condition, respectively. These disturbances directly reduce the spatial sharpness and local contrast of small defects, suggesting that part of the missed-detection problem originates from insufficient visual evidence in the input rather than from MOSAIN alone. At the same time, residual alignment errors in DAF may further affect small defects because slight spatial shifts are more significant when the target occupies only a small feature region. Thus, the current results suggest that missed small defects may result from the joint influence of limited input evidence, incomplete high-resolution information preservation, and residual cross-branch alignment error. The available experiments do not support attributing these errors exclusively to either MOSAIN or DAF.

These findings indicate several directions for future improvement. For category confusion, class-aware feature disentanglement, boundary-aware supervision, and context modeling of spatially co-occurring defects could help distinguish corrosion, exposed rebar, and spalling. For small-defect detection, higher-resolution feature paths, image tiling or local cropping, scale-aware training objectives, and hard-example mining may provide additional detail information. In addition, a deformable or uncertainty-aware alignment mechanism could be introduced into DAF to reduce the influence of local spatial misalignment under blur, low illumination, and irregular defect boundaries. These improvements may enhance the reliability of the detector in ambiguous regions, while retaining the role of the current model as a decision-support tool rather than a replacement for engineering inspection judgement.

The reported 62.3 FPS was measured on an RTX 4090 GPU at an input resolution of 640 × 640 and should be interpreted as a desktop-GPU reference rather than as evidence of real-time performance on UAV-mounted or embedded hardware. Although MS-DETR contains 14.33 M parameters and requires 39.1 GFLOPs, actual deployment performance depends on the target platform, runtime environment, input pipeline, operator support, memory constraints, and device power mode.

A deployment-oriented workflow will first export MS-DETR to ONNX and establish an FP16 TensorRT baseline with batch-one inference. INT8 post-training quantization using a representative calibration set containing bridge images with different illumination conditions, defect scales, and backgrounds will then be evaluated. If quantization causes an unacceptable degradation in small-defect detection, quantization-aware training will be considered. In addition, structured channel pruning of the convolutional branches and convolutional FFN can reduce dense tensor dimensions and should be followed by fine-tuning. The GATM-related operations must also be verified for ONNX/TensorRT compatibility; unsupported operators would require an appropriate implementation or plugin.

No Jetson-series or UAV-mounted inference experiment was conducted in this work. Accordingly, we do not claim embedded real-time capability. Future experiments will benchmark FP32/FP16/INT8 variants on target hardware using batch-one inference and will report mAP50, F1-score, end-to-end latency, throughput, peak memory usage, and power consumption under documented software versions and power modes.

6. Conclusions and Future Work

This study proposed MS-DETR, an RT-DETR-based detector for bridge surface defect detection. The proposed framework integrates HeteroFusionNet, MOSAIN, and GATM to improve heterogeneous feature representation, high-resolution detail preservation, and high-level contextual modelling. Compared with the original RT-DETR under the same experimental conditions, MS-DETR improved mAP₅₀ by 5.4 percentage points while reducing the parameter count and computational cost by approximately 29.0% and 33.3%, respectively. These results indicate that the proposed architecture provides a more favorable accuracy–complexity trade-off for bridge surface defect detection.

In addition, this study evaluated MS-DETR on the RDD road-defect dataset and the VisDrone2019 drone-scene dataset. The proposed model maintained competitive detection performance across different visual scenes, suggesting that its multi-scale feature-enhancement strategy is applicable to the evaluated datasets. Experimental results show that the proposed model maintained competitive detection performance in different visual scenes, indicating that its multi-scale feature enhancement mechanism suggests applicability across the evaluated datasets. The visualization results further show that MS-DETR can focus more closely on defect regions such as cracks, corrosion, exposed rebar, and concrete spalling, and reduce the interference of complex backgrounds on detection results.

Although the proposed method achieved promising results, several limitations remain. First, this study mainly focused on defect category recognition and bounding-box localization, but did not further estimate engineering quantitative indicators such as crack width, defect area, or spalling extent. Second, the inference speed was evaluated on a desktop GPU platform, which demonstrates computational efficiency under controlled experimental conditions but does not directly verify performance on field inspection equipment or edge devices. Third, the robustness of the model under practical disturbances, such as illumination variation, motion blur, perspective changes, surface occlusion, and different bridge components, still requires further validation.

Future research can be carried out from the following aspects. First, the bridge defect dataset should be further expanded by adding images from different bridge types, components, shooting distances, and environmental conditions to improve cross-scenario robustness. Second, object detection can be combined with instance segmentation, crack geometry measurement, and defect severity assessment, so that the model can provide quantitative inspection outputs with greater engineering significance. Third, model pruning, quantization, and edge–device deployment should be investigated to evaluate the feasibility of practical field application. Through these improvements, MS-DETR may provide more reliable technical support for intelligent bridge inspection, rapid defect screening, and structural health assessment.