Towards Practical Object Detection with Limited Data: A Feature Distillation Framework

Abstract

Underwater structural surface defect detection—such as identifying cavities and spalling—faces significant challenges due to complex environments, scarce annotated data, and the reliance of modern detectors on large-scale datasets. While current approaches often combine large-data training with fine-tuning or image enhancement, they still require extensive underwater samples and are typically too computationally heavy for resource-constrained robotic platforms. To address these issues, we introduce a defect detection model based on feature distillation, which achieves high detection accuracy with limited samples. We tackle three key challenges: enhancing sample diversity under data scarcity, selecting and training a baseline model that balances accuracy and efficiency, and improving lightweight model performance using augmented samples under computational constraints. By integrating a feature distillation mechanism with a sample augmentation strategy, we develop a compact detection strategy and framework that delivers notable performance gains in limited data, offering a practical and efficient solution for real-world underwater inspection.

1. Introduction

The maintenance and inspection of underwater infrastructure—such as pipelines, ship hulls, and offshore platforms—are essential for maritime safety, environmental protection, and economic sustainability [1,2,3,4]. Continuously exposed to harsh marine conditions, these structures undergo surface degradation due to corrosion, biofouling, and mechanical damage, including cavities and spalling [5,6,7]. Timely and accurate defect detection is crucial to preventing catastrophic failures. Traditional methods relying on human divers or remotely operated vehicles (ROVs) entail high costs, prolonged operations, and safety risks [8,9,10]. Consequently, vision-based autonomous inspection systems deployed on autonomous underwater vehicles (AUVs) or lightweight ROVs are increasingly needed to enhance efficiency and reliability [10,11,12,13].

In recent years, deep learning-based object detection has achieved significant success in various computer vision tasks, including underwater target recognition [14,15], and has been widely adopted in vision-based autonomous inspection systems. Public datasets for underwater structural surface defects are scarce, and in situ data collection and annotation are costly and labor-intensive [16]. Although image enhancement [17] and transfer learning from general datasets [18] have been proposed to alleviate data scarcity, these strategies still depend heavily on substantial source data or extensive fine-tuning. Moreover, high-accuracy detection models typically feature complex architectures with millions of parameters, leading to considerable computational costs [19]. Such computational demands render these models impractical for deployment on the embedded platforms used in AUVs, which operate under tight constraints on power, memory, and processing capability [20].

Collectively, existing approaches face two persistent challenges: a heavy dependence on large-scale annotated data and high computational complexity that hinders real-world deployment. To address these dual constraints, we propose a synergistic learning framework that integrates style transfer with feature distillation. Although our framework builds upon existing components, its core innovation lies in the synergistic design: by transforming style-transferred data into a feature-level knowledge reservoir for distillation, we establish a closed-loop paradigm where data generation facilitates knowledge transfer, and knowledge transfer, in turn, compensates for data scarcity. Our approach systematically alleviates the fundamental tension between limited samples and efficiency requirements without increasing deployment overhead. Specifically, our contributions are summarized as follows:

(1)

To mitigate defect sample scarcity and low diversity, we utilize an unsupervised, unpaired style transfer model to adapt images from a large-scale general dataset (e.g., COCO [21]) to the style of the target underwater environment. This approach enriches training data and enhances feature variability without requiring extensive real defect annotations.

(2)

To balance the dual requirements of detection accuracy and operational efficiency, we select and adapt a suitable baseline model to serve as the teacher and the student model, thereby establishing a practical foundation for deployment on embedded systems with limited computational resources.

(3)

To improve model performance under limited-data conditions, we propose a multi-level feature distillation mechanism, through which robust feature extraction ability from a pre-trained large teacher model (trained on a relevant large-scale dataset) is transferred to the lightweight student network, compensating for the information shortage caused by limited real samples.

2. Related Works

Underwater object detection (UOD) must address severe image degradation—such as low contrast, color distortion, haze-like blurring, and detail loss [15,22]—which causes a pronounced domain shift and significant performance drop when terrestrial-trained models are deployed underwater. To tackle these challenges, researchers have developed approaches that can be broadly categorized into three evolving paradigms.

2.1. Image Enhancement/Restoration-Based Pre-Processing Paradigm

Motivated by the potential link between visual quality and detection performance, this paradigm adopts a sequential approach of “improving input quality before detection,” aiming to boost detector accuracy by restoring degraded underwater images to a more natural appearance.

Early methods largely employed physics-based restoration algorithms (e.g., underwater dark channel priors) or histogram equalization. With the advent of deep learning, data-driven enhancement techniques using Generative Adversarial Networks (GANs) and Convolutional Neural Networks (CNNs), such as FUnIE-GAN, have gained prominence [15,22]. However, subsequent systematic studies [20] have shown that enhancement methods optimized for human visual perception do not always correlate with improved detector performance. Some can even reduce accuracy by introducing artifacts or suppressing critical features. To mitigate this issue, recent efforts have explored lightweight coupling and joint optimization between enhancement and detection modules. For instance, UnitModule [18] serves as an embeddable enhancement unit trained end-to-end with a detector to learn transformations favorable to detection. Wang et al. [23] introduced a reinforcement learning framework in which the detector’s performance guides the selection of enhancement algorithms. Nevertheless, these approaches still treat enhancement largely as a separate preprocessing stage, falling short of fully integrating low-level visual features with high-level detection semantics for truly bidirectional optimization within the deep network.

2.2. Network Architecture Adaptation Paradigm

Departing from the conventional preprocessing paradigm, these methods integrate image enhancement with object detection by restructuring the core architecture of detection models to improve their handling of underwater degradation. The central objective is to enhance the model’s inherent capacity for extracting robust semantic features from visually complex data, establishing this direction as one of the most actively studied paradigms in contemporary underwater vision research.

Initial efforts often incorporated channel, spatial, or hybrid attention mechanisms—such as the CT module [24] and mDFLAM [25]—to direct focus toward salient target regions while reducing interference from background noise and optical distortions [14,17,26]. As understanding of multi-scale underwater targets grew, research shifted toward improving feature representation through enhanced Feature Pyramid Networks (FPNs). Recent innovations include progressive dynamic aggregation modules [27], cross-scale differential units [28] and CNN–Transformer hybrid architectures [28,29] that combine convolutional local feature extraction with Transformer-based global context modeling. Another line of work seeks to explicitly embed underwater priors into the model. For example, Fu et al. [30] introduced a “heavily degraded prior” learned via a Residual Feature Transference Module, which transfers feature knowledge from the clear-image domain to the underwater model. Similarly, Ref. [31] applied end-to-end color space conversion to implicitly correct color distortion caused by light absorption.

2.3. Data and Training Strategy Co-Optimization Paradigm

In contrast to prior paradigms that focus primarily on model-level enhancements, the data-centric paradigm shifts attention toward data provisioning and training optimization, aiming to alleviate the scarcity of high-quality underwater annotations and to improve learning from limited or challenging samples. At its core lies the premise that dependence on any single image domain—whether raw or synthetically enhanced—remains inadequate; stronger performance is instead achieved through the synergistic integration of multiple complementary data sources or forms.

For instance, dual-branch architectures such as GCC-Net [19] and DJL-Net [32] use gated fusion mechanisms to integrate features from degraded images—which preserve rich texture details—with those from enhanced images that improve target visibility, yielding a more comprehensive feature representation. Separately, to address the challenge of “hard samples” resulting from blurry and variable underwater target appearances, researchers have refined training objectives. Boosting R-CNN [33], for example, incorporates prediction uncertainty from the Region Proposal Network to dynamically reweight training samples, thereby prioritizing difficult cases during learning. Further, to reduce dependency on costly and often inconsistent manual bounding box annotations, collaborative weakly supervised frameworks have been introduced. Cai et al. [34] proposed a method where two detectors mutually supervise each other and iteratively select high-confidence samples, effectively leveraging noisy labeled data and demonstrating promise in annotation-scarce underwater inspection scenarios.

Despite progress in these directions, current UOD methods remain constrained by their dependence on large-scale annotated datasets—a critical bottleneck in structural defect inspection where samples are inherently scarce. Moreover, few existing approaches simultaneously address the interrelated challenges of limited-sample generalization, computational complexity, and adaptation under extreme data scarcity. To overcome these limitations, we propose a strategy integrating pseudo-sample generation through style transfer with teacher-student knowledge distillation. This unified framework enables accurate defect detection with only minimal real annotations while meeting real-time operational constraints, offering a practical and efficient solution for data-limited underwater inspection.

3. Proposed Method

Focusing on surface defects of underwater structures as the target category, we employ a style-transfer approach to generate a large volume of in-distribution samples. Furthermore, we construct a teacher-student framework based on feature distillation, which guides the student model in both feature extraction and detection capabilities. This integrated strategy ultimately enhances the detection performance of compact models in limited data environments.

3.1. Model

We target surface defects—such as cavities and spalling—and incorporate them into an object detection pipeline, forming the feature distillation-based limited-sample surface defect detection model shown in Figure 1. To enable defect detection under limited data, the proposed model comprises three components: style transfer for sample generation, YOLOv10 [35] as the baseline detection network, and feature distillation for compact model optimization.

3.1.1. Sample Generation

The sample generation component is designed to increase both the volume and diversity of training data. In underwater structural surface defect detection, images containing defects such as cavities or spalling constitute valid samples. Such samples are difficult to obtain, and conventional data augmentation techniques (e.g., flipping, rotation, translation, scaling) offer limited improvement in feature variability. Recognizing that low-level image features are largely generic, we apply style transfer to align the distribution of the COCO dataset with that of the operational environment dataset, thereby enriching the training set and substantially enhancing sample diversity.

We utilize the Contrastive Unpaired Translation (CUT) model [36], an unpaired image-to-image translation framework based on contrastive learning, to adapt images from the COCO dataset (source domain) to the visual style of the real operational environment (target domain). As illustrated in Figure 2, the CUT model performs unpaired style transfer in a patch-wise manner via contrastive learning.

We define the source image as $\mathcal{X}\in {\mathbb{R}}^{H\times W\times C}$ and the target domain as $\mathcal{Y}\in {\mathbb{R}}^{H\times W\times C}$, with corresponding datasets $\mathrm{X}=\left\{x\in \mathcal{X}\right\}$ and $\mathrm{Y}=\left\{y\in \mathcal{Y}\right\}$. The generator $G$, composed of an encoder $G_{enc}$ and a decoder $G_{dec}$, produces the output $\widehat{y}=G_{dec}\left(G_{enc}\left(x\right)\right)$. The model is optimized with the following three loss components:

(1)

Adversarial Loss:

To align the distribution of generated images $G\left(x\right)$ with the target domain $Y$, To align the distribution of generated images:

$${\mathcal{L}}_{GAN}\left(G,D,X,Y\right)={\mathbb{E}}_{y\sim Y}\left[\log D\left(y\right)\right]+{\mathbb{E}}_{x\sim X}\left[\log (1-D(G\left(x\right)))\right],$$

(1)

where $D$ denotes the discriminator and the operator $\mathbb{E}\left[·\right]$ represents the mathematical expectation.

(2)

Structural Similarity Loss:

To preserve structural content, a patch-based Noise Contrastive Estimation (NCE) loss is applied across multiple layers. For a query path feature $v$, a positive sample $v^{+}$ (from corresponding location in source image), and a set of negative samples ${\left\{v_n^{-}\right\}}_{n=1}^N$ (from other spatial locations), the contrastive loss for one patch is:

$$\mathcal{l}\left(v,v^{+},v^{-}\right)=-\log {\displaystyle \frac{\exp (v·v^{+}/\tau )}{\exp (v·v^{+}/\tau )+{\sum }_{n=1}^N\exp (v·v_n^{-}/\tau )}},$$

(2)

where the temperature parameter $\tau$, which controls the sharpness of the output distribution in the contrastive loss, is set to 0.07 following the convention established in the CUT framework [36]. This value represents a widely adopted empirical setting in feature-based contrastive learning, and we retain it in our implementation for consistency and comparability.

Extending to multiple feature layers, let $L$ be the total number of selected layers, $S_l$ the number of spatial positions in layer $l$, and $H_l$ a two-layer MLP projection head for layer $l$. The ${\mathcal{L}}_{PatchNCE}$ is:

$${\mathcal{L}}_{PatchNCE}\left(G,H,X\right)={\mathbb{E}}_{x~X}\left[{\sum }_{l=1}^L{\sum }_{s=1}^{S_l}\mathcal{l}({\widehat{\mathcal{z}}}_s^l,{\mathcal{z}}_s^l,{\mathcal{z}}_{S/s}^l)\right],$$

(3)

where ${\mathcal{z}}_s^l\in {\mathbb{R}}^{C_l}$ denotes the source feature at position $s$ in layer $l$, ${\widehat{\mathcal{z}}}_s^l$ is the corresponding generated feature, and ${\mathcal{z}}_{S/s}^l\in {\mathbb{R}}^{(S_l-1)\times C_l}$ the denotes features at all other positions.

(3)

Identity Consistency Loss:

To prevent unnecessary alterations to images already in the target domain, the ${\mathcal{L}}_{Identity}$ is applied:

$${\mathcal{L}}_{Identity}\left(G,Y\right)={\mathbb{E}}_{y~Y}\left[{‖G\left(y\right)-y‖}_1\right],$$

(4)

where $y$ represents an image drawn from the target domain $Y$, $G\left(y\right)$ is the image produced by generator $G$ when y is fed as input, and ${‖·‖}_1$ stands for the $L_1$-norm, i.e., the mean absolute pixel-wise difference.

(4)

Total Objective

The complete training objective combines the above losses with balancing coefficients ${\lambda }_X$ and ${\lambda }_Y$:

$${\mathcal{L}}_{total}={\mathcal{L}}_{GAN}\left(G,D,X,Y\right)+{\lambda }_X{\mathcal{L}}_{PatchNCE}\left(G,H,X\right)+{\lambda }_Y{\mathcal{L}}_{PacthNCE}(G,Y),$$

(5)

where ${\lambda }_X$ and ${\lambda }_Y$ control the identity loss weight for the source and target domains, respectively. Setting ${\lambda }_X=1$, ${\lambda }_Y=1$ yields the full CUT model, while ${\lambda }_X=1$, ${\lambda }_Y=0$ corresponds to its lightweight variant, FastCUT. In our work, we adopt the full configuration with ${\lambda }_X=1$, ${\lambda }_Y=1$ to ensure robust style alignment and feature preservation during translation.

3.1.2. Baseline Model

Real-time object detection, which classifies and localizes targets with minimal latency, is essential for efficient and accurate inspection of underwater structural surface defects. Given its strong real-time performance, high detection accuracy, effectiveness in small-object detection, and generalization capability, we adopt YOLOv10 [35] (as shown in Figure 3) as the baseline model for our limited-sample detection framework.

YOLOv10, introduced by Tsinghua University in 2024, builds upon the YOLO series and delivers notable improvements in speed, robustness, and efficiency. Its overall loss function combines three carefully designed components:

$${\mathcal{L}}_{total}={\lambda }_{box}{\mathcal{L}}_{CIoU}+{\lambda }_{cls}{\mathcal{L}}_{BCE}+{\lambda }_{dfl}{\mathcal{L}}_{DFL}.$$

(6)

Here, ${\lambda }_{box}$, ${\lambda }_{cls}$, and ${\lambda }_{dfl}$ denote the balancing coefficients for the bounding-box regression, classification, and distribution focal loss terms, respectively. Following the official implementation [35], these coefficients are empirically set as ${\lambda }_{box}=7.5$, ${\lambda }_{cls}=0.5$, and ${\lambda }_{dfl}=1.5$ to ensure stable and balanced gradient contributions from each loss component during training.

(1)

Bounding Box Regression Loss:

The Complete Intersection over Union (CIoU) loss improves localization by incorporating center distance and aspect ratio consistency:

$${\mathcal{L}}_{CIoU}=1-\mathrm{I}\mathrm{o}\mathrm{U}+{\displaystyle \frac{{\rho }^2\left(b,b^{gt}\right)}{c^2}}+\alpha .$$

(7)

Here, $\mathrm{I}\mathrm{o}\mathrm{U}$ represents the intersection over union between the predicted and ground-truth boxes. ${\rho }^2\left(b,b^{gt}\right)$ is the squared Euclidean distance between the centers of the predicted box $b$ and ground-truth box $b^{gt}$. $c$ denotes the diagonal length of the smallest enclosing box covering both predicted and ground-truth boxes. $v$ measures the aspect ratio consistency, defined as $v={\displaystyle \frac{4}{{\pi }^2}}{\left(\arctan {\displaystyle \frac{{\omega }^{gt}}{h^{gt}}}-\arctan {\displaystyle \frac{\omega }{h}}\right)}^2$, $\alpha ={\displaystyle \frac{v}{\left(1-\mathrm{I}\mathrm{o}\mathrm{U}\right)+v}}$ is a trade-off parameter.

(2)

Classification Loss:

Binary Cross-Entropy (BCE) loss is utilized for classification to handle multi-class labeling independently and efficiently:

$${\mathcal{L}}_{BCE}=-{\sum }_{c=1}^C\left[y_c\log \left(p_c\right)+\left(1-y_c\right)\log (1-p_c)\right],$$

(8)

where $C$ is the number of classes, $y_c$ is the ground-truth label (0 or 1) for class $c$, and $p_c$ is the predicted probability for class $c$.

(3)

Distribution Focal Loss:

To refine localization—especially for small objects—DFL sharpens the predicted coordinate distribution around ground truth:

$${\mathcal{L}}_{DFL}=-{\sum }_i(\left(y_{i+1}-y\right)\log \left(P_i\right)+\left(y-y_i\right)\log \left(P_{i+1}\right)),$$

(9)

where $y$ is the continuous ground-truth coordinate, $y_i$ and $y_{i+1}$ are discrete neighboring quantization points, and $P_i$ and $P_{i+1}$ are the corresponding predicted probabilities. This formulation enables more precise regression by learning a flexible distribution over possible locations.

3.1.3. Feature Distillation

Under conditions of data scarcity, Masked Generative Distillation (MGD) [37] provides an effective mechanism for transferring robust knowledge from a teacher model, even when the teacher is pre-trained on potentially noisy data. MGD operates through a mask-and-reconstruct paradigm: the student is required to reconstruct the teacher’s feature map based on its own randomly masked activations. This design compels the student to learn contextual relationships among features and to internalize the teacher’s feature-generation logic. Framed as a self-supervised task, the process encourages the student to explore its own feature space, thereby expanding its representational capacity. Furthermore, because the reconstruction is anchored in the student’s own features, the method prioritizes the transfer of structural patterns from the teacher, which enhances robustness against label noise and domain mismatch. As shown in Figure 4, MGD trains the student to recover the teacher’s feature maps from masked versions of its own activations, promoting comprehensive spatial feature utilization and strengthening the model’s overall expressive power.

Formally, let $T^l\in {\mathbb{R}}^{C\times H\times W}$ and $S^l\in {\mathbb{R}}^{C\times H\times W}$ denote the $l$-th layer feature maps of the teacher and student models, respectively. A random binary mask $M_{i,j}^l$ is generated for each spatial position $(i,j)$:

$$M_{i,j}^l=\left\{\begin{array}{c}\begin{array}{cc}0,& if{R}_{i,j}^l<\lambda \end{array}\\ \begin{array}{cc}1,& otherwise\end{array}\end{array}\right.,$$

(10)

where $R_{i,j}^l$ denotes a random number uniformly sampled from (0, 1), $i$ and $j$ index the spatial coordinates of the feature map, and $\lambda$ corresponds to the masking ratio—i.e., the proportion of the $l$-th feature map to be masked. In our work, we adopt a uniform masking ratio $\lambda =0.5$ across all distillation layers, motivated by three principal considerations. First, a fixed $\lambda$ maintains consistent learning difficulty and training stability across layers, reducing potential optimization variance introduced by varying feature complexities. Second, randomly masking roughly half of the spatial positions acts as a form of dropout-like regularization, which encourages the model to capture contextual dependencies among features rather than merely replicating local patterns, thereby improving generalization. Finally, using a uniform $\lambda$ simplifies the hyperparameter design and enhances the reproducibility and practical usability of the method. This design is particularly well-suited for underwater defect detection, as the spatially unbiased masking ensures that defect features appearing at any location contribute equally during training, ultimately strengthening the model’s robustness in localizing unseen defects.

The masked student feature is then transformed through a feature mapping function $G$ to reconstruct the teacher’s feature map, formulated as:

$$G\left(f_{align}\right(S^l)\odot M^l)\to T^l,$$

(11)

where $f_{align}$ denotes an optional alignment layer that adapts the student feature’s dimensions to match those of the teacher, in our work, this layer specifically consists of a 1 × 1 convolution and a normalization operation, aiming to further align the number of feature channels on the premise of consistent feature spatial dimensions, as well as to reduce the discrepancy in feature distributions. ⊙ represents element-wise multiplication, $S^l$ refers to the student feature at layer l, $M^l$ is the corresponding binary mask, and $T^l$ indicates the target teacher feature from the same layer.

The feature mapping layer $G$ is composed of two convolutional layers $W_1^l$, $W_2^l$, and a ReLU activation layer. Given an input feature map $F$, the transformation is defined as:

$$G\left(F\right)=W_2^l(ReLU\left(W_1^l\left(F\right)\right).$$

(12)

Finally, the feature distillation loss ${\mathcal{L}}_{dis}$ is defined as the mean squared error between the teacher’s features and the reconstructed features across all distillation layers:

$${\mathcal{L}}_{dis}\left(S,T\right)={\sum }_{l=1}^L{\sum }_{k=1}^C{\sum }_{i=1}^H{\sum }_{j=1}^W{(T_{k,i,j}^l-G(f_{align}\left(S_{k,i,j}^l\right)·M_{i,j}^l\left)\right)}^2,$$

(13)

where $L$ denotes the total number of distillation layers, and $C$, $H$, $W$ represent the channel number, height, and width of the feature maps, respectively. $S$ and $T$ refer to features from the student and teacher models, respectively.

3.2. Training

3.2.1. Style Transfer via CUT

To construct a large-scale dataset that aligns visually with the target operational environment, we employ the unsupervised, unpaired image-to-image translation model CUT. In our framework, we specifically leverage a CUT model pre-trained on general image datasets to perform zero-shot style transfer. This design is motivated by the intended role of style transfer within our pipeline: our goal is to align the distribution of mid-to-low-level visual features (e.g., texture, tone, and local structure) between the source and the target domain, rather than producing photorealistic underwater images. The pre-trained CUT model exhibits strong cross-domain style generalization, enabling it to supply the teacher model with stylistically consistent training samples. Even when domain discrepancies exist, the teacher model can still learn semantically meaningful mid-level features from the stylized data that remain effective for detection. Furthermore, the subsequent feature distillation stage is designed to specifically transfer these mid-to-low-level features, which are inherently more robust to style variations than high-level semantic representations. Experimental results in Section 4 confirm that this approach substantially boosts the student model’s performance, demonstrating that zero-shot transfer with a pre-trained model offers a practical and effective trade-off under constraints of limited data and low computational overhead.

3.2.2. Teacher Model Pre-Training

We select YOLOv10x—the highest-accuracy variant in the YOLOv10 family (see

Table 1. Comparison of Performance and Parameters Across Different YOLOv10 Variants.

Variant	Parameters (M)	GFLOPs (640 × 640)	mAP@[0.5~0.95] (%)	Primary Target Scenario
YOLOv10-N	2.3–2.7	6–8	37.0–38.5	Extreme-edge deployment: mobile devices, embedded systems, low-power real-time detection
YOLOv10-S	7.0–8.0	20–25	43.0–44.5	Lightweight real-time detection: mobile apps, UAVs, surveillance camera video streams
YOLOv10-M	20.0–22.0	55–65	46.5–48.0	General-purpose real-time detection: industrial inspection, autonomous driving perception, generic object recognition
YOLOv10-B	28.0–31.0	85–95	48.5–50.0	High-performance student model: high-accuracy real-time systems, strong student in knowledge distillation
YOLOv10-L	55.0–60.0	160–170	50.0–51.5	High-accuracy server-side inference: cloud-based analysis, offline detection, medium-scale teacher model
YOLOv10-X	90.0–95.0	230–240	51.0–52.0	Ultimate-accuracy teacher model: research benchmarking, high-value image analysis, strong teacher in distillation

)—as the teacher architecture. It is first initialized with publicly released pre-trained weights (https://github.com/THU-MIG/yolov10/releases/download/v1.1/yolov10x.pt, (accessed on 30 June 2025)) and then fine-tuned on the style-transferred COCO dataset, which mimics the visual distribution of the target underwater environment. This two-stage process—initial pre-training followed by domain-adaptive fine-tuning—can be viewed as an implicit form of feature distillation, yielding a robust teacher model suitable for guiding the student. Furthermore, we set the batch size to 4, the input image size to 640 × 640, the number of epochs to 500, the IoU threshold to 0.7, the learning rate to 0.01, and the weight decay to 0.0005.

We acknowledge that training the teacher model on style-transferred COCO data introduces label noise due to bounding-box misalignment. Our framework addresses this issue through the following mechanisms: the teacher model is fine-tuned primarily to adapt its feature extractor to the target domain’s visual style rather than pursuing high detection accuracy on noisy labels; only the intermediate feature maps of the teacher model are distilled via Masked Generative Distillation (MGD), which capture structural and semantic patterns that are insensitive to localization errors; and the consistent performance improvement of the distilled student model confirms that the teacher provides transferable and beneficial feature representations. This design embodies the synergistic principle of our framework: leveraging noisy yet stylistically aligned data enables the teacher model to learn domain-adaptive features, which are then distilled to enhance the data-efficient student model.

3.2.3. Training for Distillation Model

To meet deployment constraints on embedded hardware, a compact YOLOv10 variant is chosen as the student model. The distillation follows a three-stage procedure: the student is first trained directly on the limited real underwater dataset; both the pre-trained teacher and the initialized student are then loaded; and finally, Masked Generation Distillation (MGD) is applied to transfer feature-level knowledge from teacher to student.

During distillation, the training configuration mirrors that of teacher pre-training, except the number of epochs is reduced to 200. To align features effectively, we select intermediate layers [2,4,6,8,12,15] from the YOLOv10 backbone (see Figure 1) for distillation. These lower- and middle-level features are chosen because higher-level semantic representations may differ substantially between the style-transferred COCO data and the real underwater dataset, despite sharing the same detection task and visual style.

4. Experiments and Analysis

We validate the baseline model and the proposed training strategy using real-world data collected by a custom underwater inspection robot. This section describes the datasets and presents comparative experiments along with corresponding analyses, focusing on YOLOv10-based detection and performance improvements through feature distillation.

4.1. Datasets

Our experiments employ two datasets: a style-transferred dataset for training the teacher model and an operational environment dataset for training and evaluating the student model.

4.1.1. Style-Transferred Dataset

We transform the large-scale COCO dataset using the CUT model to align its visual style with the target underwater environment. While the original COCO annotations are retained, the resulting bounding boxes are not pixel-wise aligned with objects in the stylized images. This deliberate label-style mismatch serves a specific purpose: it encourages the model to learn more discriminative foreground-background representations, reinforcing our strategy of distilling primarily low- and mid-level features that are less sensitive to precise object boundaries.

4.1.2. Operational Environment Dataset

This dataset contains field-collected images of underwater structural surfaces, specifically captured by our custom-built inspection robot during scanning operations at the Bahe Second Bridge in Xi’an. The images are manually annotated using LabelImg, covering four common types of surface defects: spalling, pinholes, attachments, and construction holes. The dataset comprises a total of 304 annotated images, which are randomly divided into training, validation, and test sets in a 7:2:1 ratio, resulting in 212 training, 60 validation, and 32 test images. Representative samples from the dataset are shown in Figure 5.

It should be noted that the experimental data used in this work are collected from the pier surface of the Bahe Second Bridge for performance analysis and validation of the proposed framework. When deploying the system in other underwater environments, defect characteristics may vary. Therefore, retraining or fine-tuning the model with environment-specific data is recommended to ensure optimal detection performance.

4.2. Component Selection, Analysis, and Validation

Given that the overall model configuration, the selection of the distillation model, and the choice of distillation layers all critically influence model performance, this section systematically conducts multiple sets of ablation experiments and corresponding analyses.

4.2.1. Motivation for CUT Adoption

We select CUT over alternative unpaired image-to-image translation methods due to its stronger alignment with the requirements of data-efficient underwater defect detection. CUT achieves higher training efficiency and stability through patch-wise contrastive learning, eliminating the need for cycle-consistency constraints and thereby reducing both mode-collapse risks and training time. Moreover, its multilayer feature-contrast mechanism better preserves structural content in generated images—an essential characteristic for maintaining the geometric integrity of defect regions. Furthermore, the lightweight variant FastCUT supports efficient zero-shot style transfer without requiring retraining on target-domain images, which fits naturally within our resource-conscious, few-sample learning framework.

4.2.2. Overview and Selection Rationale for YOLOv10 Variants

The YOLOv10 series comprises multiple variants (N, S, M, B, L, X) spanning from highly lightweight to high-precision architectures. Their core distinctions lie in network depth, width, parameter count, and computational complexity, enabling tailored deployment across scenarios ranging from edge devices to server-side inference. Key architectural parameters and baseline performance of each variant on the COCO dataset are summarized in Table 1.

This family reflects a systematic trade-off between performance and complexity. For constrained edge deployment, the YOLOv10-N/S variants emphasize low latency and minimal memory via shallow-narrow designs, making them suitable for tasks with simple objects or strict real-time requirements on low-power hardware. In general-purpose real-time detection, YOLOv10-M/B achieve a practical balance; increased depth and width support richer hierarchical feature learning, with YOLOv10-B often serving as a “sweet-spot” student model in distillation-delivering robust feature extraction without prohibitive complexity. For high-accuracy server/cloud inference, YOLOv10-L/X are capacity-optimized; their substantially greater depth and width capture fine-grained patterns and complex details, positioning YOLOv10-X as a powerful teacher that provides dense, high-quality knowledge for distillation, albeit at the cost of inference speed and model size. This structured differentiation offers a clear rationale for variant selection based on specific task constraints and performance objectives.

4.2.3. Analysis of the Distillation Design

(1)

Selection of the Distillation Model

To evaluate the impact of different distillation strategies on the performance of the proposed model, we conduct a comparative analysis of three established distillation methods: Channel-wise Distillation (CWD) [38], Masked Generative Distillation (MGD) [37], and Mimic [39]. The performance of the proposed model integrated with each of these strategies is presented in

Table 2. Comparison of the Impact of Different Distillation Strategies on Detection Performance.

Distillation Strategies	mAP@0.5 (%)	mAP@[0.5~0.95] (%)
CWD	82.5	53.1
MGD	83.2	53.5
mimic	81.1	51.0

. In these experiments, the style transfer module was consistently implemented with CUT, and distillation was consistently applied to layers [2,4,6,8,12,15]. The results were obtained with the teacher model being the best-performing YOLOv10-X checkpoint after 500 training epochs and the student model being the optimal YOLOv10-B checkpoint obtained after 400 epochs.

As shown in Table 2, MGD proves to be the most suitable feature distillation strategy for our specific scenario and dataset, confirming its effectiveness for the present task.

(2)

Selection of Distillation Layers

From a theoretical standpoint, we posit that distilling low- to mid-level features can enhance the student model’s detection performance. To validate this hypothesis, we perform a visualization analysis on feature maps from layers [2,4,6,8,10,12,15,18,20,22] of both teacher and student models (as shown in Figure 6). The results indicate that in layers 2, 4, 6, and 8, the teacher exhibits richer and more detailed activations. In layers 10 and 12, the features of both models show strong internal consistency. At layer 15, the teacher continues to present more informative representations. In the higher layers (18, 20, and 22), substantial semantic discrepancies emerge between the two models, implying that distilling high-level features risks introducing interference due to excessive task-specific specialization and representational divergence. Based on these observations, we select layers 2, 4, 6, 8, 12, and 15 as distillation targets, thereby balancing the transfer of structural information while mitigating potential noise from higher-level semantic mismatches.

Additionally, we take YOLOv10-B as an example to visually demonstrate how feature distillation influences model performance. Figure 7 presents the training curves (over 400 epochs) of YOLOv10-B when trained only on real operational data, showing the evolution of different loss terms and the mAP. Figure 8 displays the corresponding curves (over 200 epochs) during the distillation phase, where YOLOv10-B acts as the student model guided by the high-accuracy YOLOv10-X teacher. In the pre-training stage, all losses and mAP gradually converge to stable values. In contrast, during distillation training-since both the optimized teacher model and the initialized student model are loaded from pre-trained checkpoints-the loss and mAP start from relatively favorable initial states. The distillation process then fine-tunes the student’s parameters, driving further performance improvement toward a more optimized equilibrium.

4.2.4. Effectiveness of Individual Components

To evaluate the impact of the proposed style transfer and feature distillation methods on overall detection performance, we design three comparative experimental setups: style transfer only, feature distillation only, and both style transfer and feature distillation. Specifically, style transfer is implemented using the CUT model; the teacher model is YOLOv10-X and the student model is YOLOv10-B; feature distillation follows the MGD strategy and is applied to layers [2,4,6,8,12,15]. The entire distillation framework is trained for 200 epochs. The results are summarized in

Table 3. Comparison of the Impact of Each Component on Detection Performance.

Components	Style-Transfer	Feature Distillation	mAP@0.5 (%)	mAP@[0.5~0.95] (%)
	–	–	79.3	49.7
	√	–	70.5	45.7
	–	√	80.8	51.9
	√	√	83.4	53.7

, where the symbol “√” indicates that the corresponding module is enabled, and “–” indicates that it is disabled.

It can be observed that under the configuration using style transfer only, the detection performance of the model on the target domain shows a decline compared to the baseline. This result stems from the fact that while style transfer reduces inter-domain discrepancies, the correlation of high-level features between the source and target domains remains relatively low. In the feature distillation only configuration, the student model achieves improved detection accuracy by absorbing mid- and low-level knowledge from the teacher model, validating the effectiveness of knowledge transfer. When both style transfer and feature distillation are applied, the model attains the best performance. This indicates that the two techniques are complementary: style transfer achieves domain adaptation at the data level, while feature distillation enhances knowledge at the representation level. Their synergistic interaction more comprehensively improves the generalization and detection capability of the lightweight model in the target domain.

4.3. Pre- vs. Post-Distillation Performance Analysis

The comparative experiments evaluate the rationality of the baseline model and the effectiveness of feature distillation from both quantitative and qualitative perspectives. In the quantitative analysis stage, we compare the performance of multiple YOLOv10 framework architectures before and after feature distillation. In the qualitative stage, we randomly select test samples for detailed interpretation and discussion.

4.3.1. Quantitative Analysis

To evaluate the detection performance of different YOLOv10 variants on real-world operational data, we conducted multi-epoch training experiments using the field-collected dataset and employed 5-fold cross-validation to ensure statistical reliability under limited sample conditions. The results show that the average performance of each variant aligns closely with previously reported values, with low variance across folds (

Table 4. Comparing the Performance of YOLOv10 in Different Frameworks.

Frameworks	Input Resolution	Epoch	mAP@0.5 (%)	mAP@[0.5~0.95] (%)
YOLOv10-N	640 × 640	100	55.8 ± 0.3	33.1 ± 0.2
		200	70.3 ± 0.2	46.0 ± 0.1
		300	76.6 ± 0.1	49.5 ± 0.2
		400	79.5 ± 0.3	50.2 ± 0.3
		500	79.5 ± 0.2	49.5 ± 0.1
YOLOv10-S	640 × 640	100	64.8 ± 0.3	38.4 ± 0.3
		200	77.0 ± 0.3	50.0 ± 0.1
		300	79.5 ± 0.1	52.9 ± 0.2
		400	80.4 ± 0.3	51.5 ± 0.2
		500	80.2 ± 0.2	52.0 ± 0.1
YOLOv10-M	640 × 640	100	71.5 ± 0.3	45.1 ± 0.3
		200	79.4 ± 0.3	50.4 ± 0.1
		300	80.7 ± 0.2	52.5 ± 0.3
		400	80.7 ± 0.1	52.5 ± 0.1
		500	81.4 ± 0.3	52.6 ± 0.2
YOLOv10-B	640 × 640	100	70.4 ± 0.1	43.1 ± 0.3
		200	79.2 ± 0.2	49.7 ± 0.1
		300	80.9 ± 0.2	52.0 ± 0.2
		400	82.5 ± 0.3	52.5 ± 0.1
		500	81.6 ± 0.1	52.7 ± 0.1
YOLOv10-L	640 × 640	100	68.7 ± 0.3	40.1 ± 0.3
		200	80.4 ± 0.1	51.3 ± 0.2
		300	82.4 ± 0.2	52.8 ± 0.1
		400	81.7 ± 0.2	52.5 ± 0.1
		500	81.6 ± 0.1	52.4 ± 0.1
YOLOv10-X	640 × 640	100	72.1 ± 0.2	45.6 ± 0.2
		200	78.0 ± 0.1	50.0 ± 0.2
		300	79.0 ± 0.1	51.2 ± 0.1
		400	80.8 ± 0.2	51.6 ± 0.2
		500	82.4 ± 0.1	53.4 ± 0.1

), confirming the robustness of our findings. Specifically, YOLOv10-N achieved its best performance at 400 epochs (mAP@0.5 = 79.5%, mAP@0.5:0.95 = 50.2%), demonstrating its suitability for lightweight deployment despite capacity constraints. YOLOv10-S also peaked at 400 epochs (mAP@0.5 = 80.4%, mAP@0.5:0.95 = 51.5%), showing a measurable gain over the N-variant. YOLOv10-M required 500 epochs to reach its optimum (mAP@0.5 = 81.4%, mAP@0.5:0.95 = 52.6%), indicating that its deeper architecture benefits from extended training. YOLOv10-B attained the best results at 400 epochs (mAP@0.5 = 82.5%, mAP@0.5:0.95 = 52.5%), highlighting its efficient balance between capacity and convergence speed. YOLOv10-L saturated early at 300 epochs (mAP@0.5 = 82.4%, mAP@0.5:0.95 = 52.8%), reflecting its strong representational ability on the current data scale. YOLOv10-X consistently outperformed all other variants (mAP@0.5 = 82.4%, mAP@0.5:0.95 = 53.5%), validating its role as a high-capacity teacher model. These results not only follow the expected scalability trend of the YOLOv10 family—where larger models generally achieve higher accuracy—but also provide empirical support for our model-selection strategy: YOLOv10-X serves as a powerful teacher, while YOLOv10-B offers an effective trade-off between performance and complexity as a student model.

Based on the comparative results, we select YOLOv10-N (Epoch = 400), YOLOv10-S (400), YOLOv10-M (500), YOLOv10-B (400), YOLOv10-L (300), and YOLOv10-X (500) as student models for subsequent feature distillation experiments. In these experiments, the teacher model is the high-capacity YOLOv10-X pre-trained on the style-transferred dataset, while each student model is first pre-trained to its optimal state on the actual operational environment dataset. Using the operational environment data as input, we distill low- and mid-level features from the teacher to each student over 200 epochs of distillation training, after which performance is evaluated on the validation set.

When the baseline performance from Table 4 is compared with the distillation outcomes in

Table 5. Performance Comparison of YOLOv10 with Feature Distillation Across Different Frameworks.

Teacher-Model	Student-Model	Epoch	mAP@0.5 (%)	mAP@[0.5~0.95] (%)
YOLOv10-X	YOLOv10-N	200	79.5 ± 0.3	50.2 ± 0.1
	YOLOv10-S	200	81.2 ± 0.1	52.2 ± 0.2
	YOLOv10-M	200	81.8 ± 0.2	53.2 ± 0.1
	YOLOv10-B	200	83.4 ± 0.2	53.6 ± 0.2
	YOLOv10-L	200	82.6 ± 0.2	52.9 ± 0.1
	YOLOv10-X	200	83.1 ± 0.1	53.9 ± 0.1

, distinct patterns emerge across model scales. Due to its highly compact architecture, YOLOv10-N exhibits a fundamental limitation in representational capacity; consequently, even under guidance from a high-capacity teacher, its performance does not improve significantly, indicating that effective knowledge transfer requires a student model with sufficient inherent expressivity. In contrast, all other variants demonstrate measurable gains after distillation, confirming that feature-level knowledge can be successfully transferred when the student possesses adequate baseline capability. Among them, YOLOv10-B and YOLOv10-X achieve the most balanced post-distillation results: YOLOv10-B reaches mAP@0.5 = 83.2% and mAP@0.5:0.95 = 53.2%, while YOLOv10-X attains mAP@0.5 = 83.1% and mAP@0.5:0.95 = 53.9%. Considering the trade-off between model size, inference efficiency, and accuracy, YOLOv10-B presents a particularly favorable profile for deployment on resource-constrained platforms such as underwater inspection robots—its performance closely approaches that of the largest variant while requiring substantially fewer parameters and less computation. These findings support two main conclusions: (1) the YOLOv10 family provides a scalable detection backbone suitable for the target operational data, as evidenced by the consistent performance progression in Table 4; and (2) feature distillation offers a viable pathway to enhance few-shot learning in this domain, as shown by the systematic improvements in Table 5, where the process refines the student’s parameters through structured feature-level guidance without overriding its intrinsic capacity—a dynamic visually reflected in the training curves of YOLOv10-B (Figure 7 and Figure 8). In practical terms, while YOLOv10-B is recommended for embedded deployment, other variants remain applicable under specific hardware constraints; for instance, in scenarios with extreme limitations on memory or computation, YOLOv10-N could still meet basic functional requirements, highlighting that framework selection can be adaptively tailored based on a comprehensive evaluation of storage, computational resources, and detection-performance thresholds.

4.3.2. Qualitative Analysis

To visually evaluate the detection performance of the proposed feature distillation-based few-shot surface defect detection method in actual operational environments, we divide the collected real-world data and reserve 32 images as a test set, which has no overlap with the training or validation data. As shown in Figure 9, we randomly select four images from the test set as model inputs (labeled as “Test image”) and display the test results of the relatively optimal models for each YOLOv10 framework (as summarized in Table 4) in rows corresponding to each input.

The qualitative results depicted in Figure 9 reveal distinct detection behaviors across YOLOv10 variants when processing representative defect samples. In image (1), YOLOv10-N incorrectly merges two distinct large-area spalling defects into a single bounding box, while YOLOv10-S fails to localize both defects entirely. YOLOv10-L produces redundant detections for the left-side spalling, indicating over-localization. Only YOLOv10-X achieves precise, separate detection of all defects. For image (2), all models except YOLOv10-X exhibit missed detections, underscoring the latter’s superior recall in challenging scenes. Image (3) further highlights inconsistency in YOLOv10-L, which misses one defect present in the scene. In contrast, image (4)—characterized by more pronounced defect features—is accurately detected by all models, suggesting that well-defined structural damage is reliably identifiable even by lighter architectures.

Collectively, these observations confirm that while all YOLOv10-based frameworks satisfy basic engineering detection requirements, the large-capacity YOLOv10-X delivers the most robust and consistent performance across varied defect morphologies and spatial arrangements. This justifies its selection as the teacher model in our distillation framework.

Building upon YOLOv10 as the baseline and YOLOv10-X as the teacher, we construct a teacher-student framework optimized through feature distillation. Since YOLOv10-B exhibits the most favorable trade-off between capacity and efficiency post-distillation, we use it as a representative case and visually compare its detection outputs before and after distillation on selected test samples (as shown in Figure 10). In images (1)–(3), the pre-distillation model fails to detect several defects (highlighted with green ellipses), all of which are correctly identified after distillation. Moreover, in image (4), the pre-distillation version mistakenly classifies fine surface textures as spalling (marked with purple ellipses), an error that is eliminated following distillation. These visual comparisons demonstrate that feature distillation not only improves recall by reducing missed detections but also enhances precision by suppressing false positives induced by ambiguous textures. Thus, the distillation process effectively refines the student model’s feature representations, leading to more accurate and reliable defect localization in complex underwater surface inspection scenarios.

5. Conclusions

This study focuses on detecting underwater structural surface defects such as spalling and cavities. To address the limited quantity and low diversity of real defect samples, we apply unpaired image style translation using the CUT model, generating a large-scale synthetic dataset that matches the visual distribution of the target underwater environment. YOLOv10 is adopted as the baseline detector to balance accuracy and efficiency. To comply with the storage and computational limits of robotic platforms, a lightweight YOLOv10 variant is selected as the student model, while a larger, high-accuracy teacher model is pre-trained on the synthetic dataset. A teacher-student framework is then constructed and optimized via feature distillation, which enhances the detection performance of the compact student model and ensures its deployability on embedded hardware. While the proposed method demonstrates promising results in the tested underwater environment, we acknowledge that its generalization capability across diverse underwater sites (e.g., different water qualities, lighting conditions, or geographical regions) remains to be fully validated. The current study focuses on establishing the efficacy of the style-transfer-and-distillation framework under controlled experimental conditions. In future work, we plan to extend the evaluation to multi-site datasets and explore domain-adaptive strategies to further enhance the model’s robustness and practical applicability in real-world underwater inspection scenarios.