Dual-Modal Vision–Sonar Object Detection for Underwater Robots Based on Deep Learning

Abstract

Applying state-of-the-art RGB object detectors (e.g., YOLOv8) to underwater scenes often yields unstable performance due to scattering, absorption, illumination deficiency, and bandwidth-limited transmission that severely corrupt image contrast and details. Forward-looking sonar (FLS) remains informative in turbid or low-visibility water, yet its low resolution and weak semantics make conventional fusion architectures costly and difficult to deploy on resource-constrained robots. This paper proposes a paired-sample-free RGB–FLS joint training paradigm based on parameter sharing, where RGB and FLS images from different datasets are jointly used during training without any frame-level pairing or architectural modification. The resulting model preserves the original detector parameter scale and inference cost, and requires only RGB input at test time. Experiments on the SeaClear and Marine Debris FLS datasets under six representative underwater degradation factors (contrast loss, blur, resolution reduction, color cast, and JPEG compression) show consistent robustness gains over RGB-only training. In particular, under severe low-contrast corruption, the proposed training strategy improves mAP50 by more than 14 percentage points compared with the RGB-only baseline. These results indicate that sonar-domain supervision functions as an auxiliary structural constraint during optimization, rather than a conventional multi-source data enlargement. By forcing a shared-parameter detector to fit a texture-poor, geometry-dominant sonar domain, the learned representation is biased away from color/texture shortcuts and becomes more stable under adverse underwater degradations, without increasing deployment complexity.

1. Introduction

Underwater robots, including remotely operated vehicles (ROVs) and autonomous underwater vehicles (AUVs), are increasingly employed for marine inspection, debris removal, and environmental monitoring tasks. Reliable perception of underwater targets is a fundamental requirement for these applications. However, optical imaging in underwater environments is severely affected by light absorption, scattering, and color distortion, which lead to reduced contrast, blurred details, and unstable visual appearance. As a consequence, object detection models that perform well in terrestrial scenes often exhibit significant performance degradation when applied directly to underwater imagery. In contrast, sonar sensors are capable of providing stable perception in turbid or low-visibility waters, as their imaging process does not rely on ambient light conditions. Nevertheless, sonar images typically suffer from low spatial resolution, strong noise, and limited semantic information, which makes them difficult to directly integrate into conventional deep learning-based visual detection frameworks. Owing to the complementary characteristics of optical and acoustic sensing, multimodal perception approaches that combine camera and sonar data have attracted increasing attention in underwater target detection research. Integrating optical and acoustic data allows the respective strengths to be combined, thereby improving the accuracy of underwater target detection [1]. Kim et al. [2] designed an optical-multibeam sonar fusion system that aligns the two sensors through geometric calibration and fuses color and distance information at the pixel level to enhance understanding of underwater scenes. Li et al. [3] proposed the RGB–Sonar tracking benchmark RGBS50 and the SCANet network for tracking tasks. They introduced a spatial cross-attention module to mitigate spatial misalignment between modalities and generated “sonar-style” images via simulation to compensate for the scarcity of real RGB–sonar paired data.

These studies demonstrate that optical and sonar systems exhibit inherent complementarity at the information level, and multimodal interaction can enhance underwater target perception capabilities to a certain extent. However, in practical underwater operations, existing multimodal solutions still face several prominent challenges. First, most fusion methods rely on strictly synchronized, well-calibrated RGB–sonar paired data, which are costly to collect and requires complex experimental platforms, making deployment challenging for underwater robotic systems operating long-term in open waters. To circumvent this, some studies resort to generating “pseudo-sonar images” through simulation, but discrepancies between the real and simulated domains may compromise model generalization capabilities [3]. Secondly, to achieve cross-modal feature interaction, many networks incorporate dual-branch backbones, cross-modal attention modules, or multi-level fusion modules in their architecture. This leads to a significant increase in parameters and computational overhead, making real-time deployment on underwater robots with limited computing power challenging [2]. Additionally, systematic robustness evaluations under typical underwater degradation scenarios remain relatively lacking, making it difficult to answer the critical question: “To what extent does multimodal information mitigate the performance loss caused by visual degradation?”

To enhance the accuracy and stability of underwater target detection, this paper proposes an alternative approach that leverages sonar information to improve visual detection performance without relying on sample-level registration or increasing network parameters. While maintaining the original network architecture, a joint training strategy is adopted: without introducing an additional fusion branch or requiring paired data, the approach blends RGB and sonar image inputs into the network model through “sample-level mixing + shared parameters.” This eliminates the need for spatial or temporal pairing between modalities, enabling the visual detector to learn structural features embedded in sonar data during training and achieve cross-modal representation transfer. During inference, the detector operates on single-domain RGB images with the same computational overhead as a monomodal detector. By leveraging optical and sonar images from different datasets through a unified training strategy, the proposed scheme enhances the model’s detection robustness in degraded scenarios. Importantly, although the implementation resembles union-style sampling, the sonar domain is introduced as a texture-suppressed structural constraint to discourage color-dependent solutions, rather than to improve average performance across multiple visual domains.

The main contributions of this paper are three-fold:

(1)

We propose a paired-sample-free RGB–FLS joint training paradigm with shared parameters, enabling cross-modal supervision without any architectural modification or paired-frame alignment.

(2)

We perform a systematic robustness evaluation under multiple underwater degradation factors (contrast loss, blur, resolution reduction, color cast, and compression) and show consistent robustness gains while preserving clean-domain accuracy on SeaClear (or after lightweight fine-tuning under domain shift).

(3)

We provide additional cross-dataset evidence and an ablation-style analysis using SUIM and its corrupted variants (grayscale/blur/edge) to isolate whether the robustness gain is sonar-specific or can be explained by generic degradation exposure, and we further demonstrate when Union training is beneficial and when a lightweight RGB fine-tuning stage is necessary under significant domain shift.

The structure of this paper is as follows: Section 2 reviews relevant research on underwater object detection and multimodal learning; Section 3 presents the proposed joint training framework; Section 4 details dataset processing and implementation specifics; Section 5 analyzes experimental results.

2. Related Work

2.1. Underwater Optical Target Detection

Optical imaging provides rich texture and color information and is widely used in underwater applications such as coral reef monitoring [4], ship hull inspection [5], motion estimation [6], and underwater archeological surveys [7]. However, underwater optical images are severely affected by light absorption, scattering, turbidity, and noise, leading to significant performance degradation when deep learning-based object detectors trained on natural images are directly applied to underwater environments [8].

To mitigate the impact of underwater image degradation, extensive studies have focused on underwater image enhancement and detection-oriented network optimization. Cong et al. [9] systematically reviewed deep learning-based underwater image enhancement methods, highlighting the importance of data synthesis, evaluation metrics, and loss design. In terms of detection models, lightweight and task-specific architectures have been widely explored. Li et al. [10] proposed SCSE-YOLO by introducing channel–spatial attention mechanisms to improve underwater debris detection accuracy. Aryaza et al. [11] accelerated YOLO inference through channel pruning while maintaining competitive detection performance. Cui et al. [12] presented a lightweight UTNet model for real-time marine debris detection, providing valuable engineering insights for vision systems deployed on underwater robots. Jian et al. [13] further summarized existing underwater object detection methods and public datasets, pointing out that complex lighting conditions, suspended particles, and limited data scale remain key challenges.

Despite these efforts, optical-only detection methods remain fundamentally constrained by image quality and often exhibit unstable performance under severe underwater degradation, such as low contrast and turbidity.

2.2. Sonar-Based Detection and Multimodal Perception

Compared with optical sensors, forward-looking sonar maintains stable perception capability in low-visibility underwater environments. Consequently, sonar-based detection, segmentation, and contour extraction have attracted increasing attention. However, sonar imagery typically suffers from low spatial resolution, high noise levels, and blurred boundaries, which pose challenges for accurate target detection [14]. To facilitate research in this domain, several public sonar datasets and benchmark methods for marine debris and multi-class underwater object detection have been established, providing a reproducible platforms for sonar-based perception research [15].

To exploit the complementary characteristics of optical and acoustic sensing, multimodal and acoustic–optical fusion approaches have been actively investigated. Gong et al. [1] proposed an acoustic–optical image fusion method based on an improved MobileNetV2 network for underwater target classification. Kim et al. [2] developed an underwater optical–sonar image fusion system that integrates enhanced optical color information with sonar-derived distance cues through geometric calibration. Li et al. [3] established the RGB–Sonar tracking benchmark RGBS50 and proposed SCANet, which introduces a spatial cross-attention mechanism to mitigate alignment inconsistencies between RGB and sonar modalities.

In addition, several studies have explored feature-level or architecture-level fusion strategies. Ge et al. [16] proposed the YOLOv7-Fusion model using intermediate fusion to enhance multimodal feature interaction, while Liu et al. [17] improved underwater small-object detection through lightweight reparameterized modules and multi-scale feature fusion. Negahdaripour et al. [18] and Babaee et al. [19] investigated explicit optical–sonar feature fusion for motion estimation and 3D object modeling, demonstrating improved robustness in high-turbidity environments.

2.3. Limitations of Existing Multimodal Approaches

Although existing multimodal methods demonstrate performance gains over monomodal detectors, most of them rely on strictly paired RGB–sonar data and introduce dual-branch architectures or complex fusion modules [2,3,16]. These designs significantly increase model parameters, computational overhead, and system complexity, resulting in high data acquisition costs and limited feasibility for real-time deployment on resource-constrained underwater robotic platforms. Moreover, most approaches require simultaneous multimodal inputs during inference, further increasing deployment and maintenance costs.

These limitations motivate the exploration of alternative multimodal learning strategies that can leverage sonar information to enhance visual detection robustness without requiring paired data or increasing network complexity.

2.4. Domain Generalization and Multi-Source Training

Beyond architecture-level multimodal fusion, an alternative line of research studies domain generalization and multi-source training, where a single model is trained on multiple source domains to improve robustness on unseen target conditions. Typical strategies include learning domain-invariant representations, style randomization/augmentation, and multi-source risk minimization. Our method is conceptually related to the literature in that it also optimizes a unified model on heterogeneous data sources. However, RGB and forward-looking sonar differ fundamentally in sensing physics and information content: RGB images are texture- and color-rich, while FLS images are echo- and intensity-dominant with little fine texture. Therefore, our joint training should be viewed as cross-modal structural regularization rather than conventional multi-style augmentation among visually similar domains. This distinction motivates our robustness evaluation and the ablation-style comparison against non-sonar corruptions in Section 4.5. For completeness and reproducibility, Section 3 briefly summarizes the YOLOv8 detection pipeline and its standard optimization objective. These descriptions are not presented as novel contributions, but as a necessary baseline to clearly define the proposed cross-modal training objective and evaluation protocol. In other words, FLS is not treated as another “style” of RGB, but as a physics-driven modality that removes texture cues and thus constrains the optimization trajectory of the shared backbone.

3. RGB-FLS Joint Training Framework and Implementation Details

3.1. Contribution Clarification and Positioning

This work does not propose a new detector architecture and does not modify any YOLOv8 components. Instead, we contribute a paired-sample-free RGB–FLS joint training paradigm that leverages cross-modal supervision through strict parameter sharing.

Conventional multi-source or data-mixing training aims to improve average performance across multiple source domains by enlarging training diversity. In contrast, the sonar domain in our framework is intentionally introduced as an auxiliary structural constraint: forward-looking sonar images are texture-poor and semantically weak, but preserve stable geometric and intensity structures under adverse underwater conditions. By forcing a single detector to minimize detection losses on both RGB and FLS samples, the shared backbone is discouraged from relying on color/texture shortcuts that are unstable underwater and is biased toward modality-invariant structural cues (e.g., contours, spatial layouts, and intensity discontinuities).

Accordingly, while the implementation resembles union-style sampling, the methodological role of sonar data is to constrain the solution space of the shared representation rather than to jointly optimize multiple visual domains. We evaluate two variants: Union (mixed RGB/FLS training with shared parameters) and Union + FT (Union followed by short RGB-only fine-tuning to recover the clean-RGB decision boundary under strong domain shift). Both variants keep the same architecture and parameter count; fine-tuning updates shared weights only.

3.2. Problem Modeling

Underwater debris detection is treated as a bounding box-based object detection task, where the RGB and sonar images are denoted as Iv and Is, respectively. The object detection network, F(·;θ), outputs bounding boxes with corresponding class labels and confidence scores, as shown in Equation (1).

$$\left\{\right(b_i,c_i,s_i\left)\right\}=F(I;\theta )$$

(1)

Here, b_i represents the coordinates of the i-th detection box, c_i is the class label, and s_i is the confidence score. The input I can come from either the RGB or FLS domain. This work employs a unified detection network, with cross-modal joint training via shared parameters θ. Inference is performed using only RGB images, with no additional computational cost.

This formulation is included for reproducibility and to clearly define the shared-parameter training objective. Importantly, the detector architecture is kept unchanged throughout this work.

3.3. YOLOv8n Backbone as a Baseline Detector

We adopt YOLOv8n (Ultralytics) as the baseline detector. This subsection is included only for reproducibility and to clarify that our method introduces no architectural modification and no additional fusion branch. YOLOv8n consists of a backbone for hierarchical feature extraction, a neck for multi-scale feature aggregation, and detection heads at three resolutions (Figure 1). As a representative unchanged component, the SPPF block is used for lightweight context aggregation with negligible overhead (Figure 2). All YOLOv8n modules, including SPPF, remain identical to the official implementation throughout this work. Readers familiar with YOLO-style detectors may skip this subsection and proceed to Section 3.4.

Figure 1. Overall structure of the YOLOv8 model.

Figure 2. SPPF block in YOLOv8n for lightweight multi-scale context aggregation. This component is kept unchanged in our framework, illustrating that no additional cross-modal fusion modules are introduced.

3.4. RGB-FLS Joint Training Strategy

RGB and FLS modalities differ fundamentally in sensing physics, resolution, noise characteristics, and semantic density. Forward-looking sonar images lack color and fine textures and are dominated by geometric echoes and intensity patterns. Therefore, introducing FLS into training is not intended to enlarge visual styles as in conventional multi-source training; instead, it acts as an auxiliary structural regularizer that constrains the shared representation to rely less on color/texture shortcuts and more on modality-invariant structural cues.

The joint training strategy employs the aforementioned network model, which can be understood as a regularization mechanism that operates in a “multi-domain during training, single-domain during inference” manner. Although the training pipeline resembles union-style sampling, the auxiliary FLS data serve as a structural regularizer to suppress color/texture-biased solutions and promote structure-driven representations.

During joint training, the same set of convolutional kernels operates on both the RGB domain and the FLS domain. If features become overly dependent on color, high-noise, low-resolution FLS samples will incur significant losses and be corrected through gradients. This causes the backbone to favor reinforcing geometric structural features such as light–dark boundaries, object contours, overall shapes, and relative positions, while reducing reliance on unstable colors and high-frequency textures. The Neck’s multi-scale fusion further propagates this structural information to both high- and low-resolution branches: high-resolution feature maps become more sensitive to small object contours and local intensity jumps, while low-resolution feature maps enhance modeling of object clusters and scene layout.

Within the Detection Head, the classification and regression branches share these multiscale structural features: for FLS samples, the classification branch must distinguish objects from background based on echo shape and spatial distribution despite lacking color and fine texture; the regression branch predicts bounding boxes under blurred edge conditions by leveraging intensity differences between continuous strong echo regions and background. This reliance on “geometric structure + intensity patterns” is similarly transferred to the RGB domain through shared parameters. Consequently, even in low-contrast, green-biased, or compressed/degraded scenarios where color and detail are severely compromised, the Detection Head can still make relatively stable classification and regression predictions based on object contours and spatial layout.

To investigate the feasibility of enhancing visual detection robustness using sonar modalities without increasing inference overhead, this paper designs three training configurations within the unified YOLOv8n detection framework to compare single-domain training versus joint training:

• RGB-only Single-Domain Baseline: Trained exclusively on the SeaClear RGB dataset, denoted as D^v. Throughout training, YOLOv8n inputs consistently originate from the camera image’s RGB domain. The resulting model is labeled RGB-only.
• FLS-only Single-Domain Baseline: Trained solely on the Marine Debris FLS dataset, denoted as D^s. The resulting model is labeled FLS-only to evaluate detection performance under pure sonar modality.
• RGB-FLS Joint Training—Union: Employing the YOLOv8n network, RGB and FLS samples are blended to form a joint training dataset.

$${\mathrm{D}}^{\mathrm{u}}={\mathrm{D}}^{\mathrm{v}}\cup {\mathrm{D}}^{\mathrm{s}}$$

(2)

In each iteration, a batch is randomly sampled from Du, containing samples from both RGB and FLS. Regardless of whether the input originates from the RGB or FLS domain, the same detection network with shared parameters is employed, as formulated in Equation (3).

$${\mathrm{ŷ}}^v=f_{\mathsf{\theta }}\left(D^v\right){\mathrm{ŷ}}^s=f_{\mathsf{\theta }}\left(D^s\right)$$

(3)

The detection network is $f_{\theta }(\cdot )$, where $\theta$ is the training parameter shared across different modal samples. The detection loss consists of three components: bounding box regression loss, object loss, and classification loss. Weight coefficients provided by Ultralytics are used:

$$L(x,y)=L_{bbox}(\mathrm{ŷ},y)+L_{cls}(\mathrm{ŷ},y)+L_{obj}(\mathrm{ŷ},y)$$

(4)

$L_{\mathrm{b}\mathrm{b}\mathrm{o}\mathrm{x}}$ denotes the bounding box regression loss, $L_{\mathrm{c}\mathrm{l}\mathrm{s}}$ represents the class classification loss, and $L_{\mathrm{o}\mathrm{b}\mathrm{j}}$ signifies the object confidence loss.

By simultaneously minimizing detection losses for both RGB and FLS data through unified parameters, implicit feature alignment between the two modalities is achieved. The jointly trained model is denoted as Union. During inference validation, only RGB images are input to the model. Since the model architecture remains unchanged, inference time and deployment complexity align with those of the RGB-only model.

In addition, we adopt an optional lightweight RGB fine-tuning stage after joint training. Specifically, we first perform Union training with mixed RGB and FLS samples, and then fine-tune the resulting model on clean RGB data only for a small number of epochs. This two-stage scheme helps recover the RGB decision boundary while preserving robustness gained from sonar-domain supervision. In our implementation, fine-tuning is performed on the RGB training split for 50 epochs with a learning rate of 1 × 10^-4, while keeping the architecture unchanged (no additional parameters). Unless otherwise specified, all layers remain trainable.

3.5. Evaluation Metrics and Training Details

The evaluation metrics employ mAP50 and mAP50-95 to assess model performance across varying degradation scenarios. mAP50 calculates the average AP (Area Under the Precision–Recall curve) across all classes at an IoU threshold of 0.5 for bounding boxes relative to ground truth boxes, then averages this value across categories. mAP50-95 represents the average accuracy within the IoU range from 50% to 95%. AP is calculated at 10 thresholds (IoU = 0.50, 0.55, …, 0.95) and averaged. AP is derived from Precision and Recall are defined as Equations (5) and (6), respectively.

$$\mathrm{Precision}={\displaystyle \frac{TP}{TP+FP}}$$

(5)

$$\mathrm{Recall}={\displaystyle \frac{TP}{TP+FN}}$$

(6)

Among these, TP represents the number of correctly detected targets, FP denotes the number of false positives, and FN indicates the number of missed detections. Precision primarily reflects the model’s ability to control false alarms, while Recall measures the model’s capability to detect true targets.

During training, YOLOv8’s built-in data augmentation strategies—including scaling, cropping, flipping, and color perturbation—were enabled to enhance the model’s generalization capabilities. All input images were resized to 640 × 640 resolution. Based on evaluation results, the optimal weight file best.pt was selected as the final model for subsequent assessments.

4. Experimental Procedure and Results

4.1. Processing of Underwater Debris Datasets

The SeaClear [20] dataset is constructed for underwater debris detection tasks, comprising multiple segments of underwater videos and static images captured in coastal environments such as ports and docks. This paper selected RGB images and their annotations from the dataset, focusing on three categories: bottle, can, and tire. After converting them to YOLO format, the data was split in an 8:1:1 ratio. The training set comprises 3739 images, while the validation set contains 467 images corresponding to 718 annotated instances. The original images have a resolution of 1920 × 1080, with annotations provided as bounding boxes. Figure 3 shows representative SeaClear training samples with different category labels, illustrating the diversity of target appearances and backgrounds rather than any paired multimodal correspondence. This paper employs the training set for model training, the validation set for performance evaluation, and subsequent experiments on robustness to image degradation.

Figure 3. Example samples from the SeaClear training set. The labels shown on the subfigures indicate different debris categories (e.g., bottle/can/tire) and are used only to illustrate category diversity; the subfigures do not imply temporal correspondence or paired multi-sensor acquisition.

Using the SeaClear validation set, we constructed six degraded validation sets: Gaussian blur, motion blur, reduced resolution, reduced contrast, green color cast, and image compression. Each set maintains the same number of images and annotations as the original validation set, differing only in degraded image quality, to evaluate robustness.

The Marine Debris FLS [21] dataset was collected using forward-looking sonar in offshore environments and similarly contains various marine debris and engineering targets. Three categories—bottle, can, and tire—were selected. After conversion to YOLO format, the training set comprises 844 sonar images, while the validation set contains 105 sonar images corresponding to 134 annotated instances. Figure 4 shows a sample from the Marine Debris FLS training set.

Figure 4. Marine Debris FLS training sample set.

4.2. Construction of Underwater Image Degradation Validation Sets

Based on the SeaClear validation set, six categories of artificial degradation versions were constructed, with each category further subdivided into four degradation levels according to intensity.

• Resolution Reduction: Downsampling and resampling the original image to simulate detail loss caused by long-distance and low-resolution imaging;
• Gaussian Blur: Applying Gaussian blurring with varying standard deviations to the original image, simulating blurring due to water scattering and unfocused imaging;
• Motion blur: Blurs the original image using a one-dimensional convolution kernel in random directions to simulate streaking caused by relative motion between the ROV and the target object;
• Low contrast: Reduces overall image contrast by compressing the pixel dynamic range, simulating scenes with insufficient lighting or turbidity;
• Green Color Cast: Enhances the green channel while suppressing the red and blue channels;
• JPEG Compression: Compresses the image through encoding and decoding to simulate the degradation in image quality during low-bitrate transmission.

Figure 5 shows a comparison between the degraded image and the original image, demonstrating the enhanced robustness of the jointly trained model through the use of lower-quality images.

Figure 5. Comparison between the original image and the degraded image.

4.3. Performance on the SeaClear and FLS Validation Sets

On the SeaClear and FLS validation sets, the detection performance of RGB-only, FLS-only, and Union models was evaluated, with results shown in Table 1. It can be observed that on the SeaClear validation set, Union improved mAP50 and mAP50-95 by 1.0 and 0.5 percentage points, respectively. On the FLS validation set, Union’s performance metrics were slightly lower than those of FLS-only. This indicates that when the deployment scenario relies exclusively on sonar inputs, modality-specialized optimization can yield slightly higher peak performance, which is consistent with the trade-offs inherent in shared-parameter learning. The primary benefit of the Union model, achieved through joint training, lies in enhanced robustness under degraded RGB conditions. Performance data suggests that joint training does not significantly compromise detection accuracy on RGB data and may provide slight improvements depending on the dataset.

Table 1. Performance comparison between jointly trained models and single-domain trained models.

Model	Precision	Recall	mAP50	mAP50-95
RGB-only	0.945	0.898	0.944	0.741
RGB-Union	0.964	0.904	0.954	0.746
FLS-only	0.936	0.924	0.953	0.768
FLS-Union	0.906	0.909	0.945	0.750

The slightly higher mAP of the FLS-only model on the sonar validation set compared to Union stems from the performance trade-offs inherent in parameter sharing. During joint training, the same set of network parameters must simultaneously accommodate the high resolution and rich textures of RGB data while processing the low-resolution, high-noise features of the sonar domain. This inevitably introduces trade-offs. When serving two modalities simultaneously, peak performance on a single modality may slightly decrease, but overall robustness improves. Since inference primarily occurs on RGB images, this minor loss in sonar performance is acceptable. The robustness gains achieved across multiple degraded scenarios are more critical for enhancing the reliability of actual underwater operations.

Practical guideline. If the robot operates primarily with an RGB camera and the main concern is robustness under unpredictable visibility (e.g., turbidity, low illumination, low-bitrate transmission), Union is recommended because it improves degraded-RGB reliability without any inference overhead. If the system relies on sonar-only perception (no RGB camera) or aims to maximize peak performance on the sonar domain, an FLS-only model can be preferred. When both modalities exist but the auxiliary sonar set is substantially mismatched to the RGB domain, Union + FT provides a compromise by recovering clean-RGB accuracy while retaining robustness benefits from cross-modal supervision.

To further examine whether the observed robustness gain generalizes beyond the SeaClear dataset, we additionally conducted cross-dataset experiments on the SUIM benchmark and its degraded variants. Compared with the RGB-only baseline trained on SUIM, the proposed Union strategy combined with a lightweight RGB fine-tuning stage achieves comparable performance on clean RGB images, while consistently improving detection accuracy on sonar-domain inputs and multiple degraded RGB conditions (e.g., blur and grayscale corruption).

4.4. Robustness Analysis Under Six Types of Deterioration Conditions

The six-category degradation validation set constructed using the SeaClear validation set specifically includes: Gaussian, Motion, Downsampling, Low-Contrast, Green color cast, and JPEG compression. Both RGB-only and Union models were tested, with Union demonstrating significant advantages. Figure 6 illustrates the changes in mAP50-95 metrics for Union and RGB-only models under varying degradation intensities. Under five degradation types—resolution reduction, Gaussian blur, green cast, JPEG compression artifacts, and motion blur—the Union model consistently outperformed the RGB-only model on mAP50, achieving average gains of 1–3 percentage points. Notably, the average improvement reached 2.3 percentage points in the JPEG compression scenario. This demonstrates that joint training helps mitigate the impact of detail loss caused by low-bitrate transmission.

Figure 6. Performance comparison under six degradation conditions: (a) Performance Comparison Under Reducing Contrast; (b) Performance Comparison Under Motion Blur; (c) Performance Comparison Under Resolution reduction; (d) Performance Comparison Under Gaussian Blurring; (e) Performance Comparison Under Green Color Bias; (f) Performance Comparison Under JPEG Image Compression.

The difference between the two models is most pronounced in low-contrast degraded scenarios: the RGB-only model’s average mAP50 is significantly lower than the Union model’s, with the Union model achieving over a 14-percentage-point improvement over RGB-only. Correspondingly, the mAP50-95 also shows an increase of over 10 percentage points. Low-contrast degradation simulates adverse conditions such as insufficient lighting and murky water. These results demonstrate that joint training with sonar domain samples effectively provides structural prior knowledge, enhancing the network’s ability to represent target contours and locations in low-contrast and dark scenarios.

This performance improvement aligns with a common finding in multimodal learning: networks trained simultaneously on multiple imaging styles or modalities tend to learn “structural” features that remain relatively stable across domains. This enables them to maintain strong generalization capabilities even when encountering unseen degraded scenarios. In this paper’s joint training setup, YOLOv8n must simultaneously perform garbage detection tasks in both the RGB visual image domain and the sonar domain using the same set of parameters. Compared to models trained solely in the RGB domain, this process effectively introduces “structural priors” from the sonar modality into the feature space: sonar imaging does not rely on visible light and is insensitive to color and texture variations, but it is more sensitive to an object’s geometric shape, spatial position, and relative brightness. Consequently, during optimization, the network is compelled to seek discriminative features applicable to both imaging modalities—such as object contours, intensity differences between subjects and backgrounds, and stable spatial layouts—while reducing reliance on subtle textures or specific color distributions inherent to RGB images. This “bias toward structural features” in learning explains why Union models outperform RGB-only models under degraded conditions like reduced resolution, Gaussian blur, and motion blur: when details are compressed or obscured, it is precisely this low-frequency geometric structural information that often persists.

In low-contrast and green-shifted scenes, the Union model demonstrates a more pronounced advantage. Low-contrast degradation weakens the luminance difference between objects and backgrounds, complicating discrimination based on local texture and color. Green shift, meanwhile, significantly alters the relative intensity distribution across RGB channels, undermining the color priors learned by the model within a single domain. In contrast, sonar imagery provides a “color-agnostic” complementary perspective during training. Its imaging results are primarily determined by acoustic reflection intensity and geometric occlusion relationships, exhibiting minimal sensitivity to illumination and color shifts. During joint training, this modality’s insensitivity to color and illumination variations encourages the network to reduce reliance on absolute color values in feature space, instead focusing on shape, shadow distribution, and overall brightness gradients. Consequently, Union models maintain higher detection rates and more stable bounding box regression accuracy when confronted with degradations like low contrast, color shifts, and JPEG compression that severely distort color distributions. Similar phenomena are observed in indoor scene recognition and multimodal classification tasks: incorporating additional modalities or semantic channels often significantly enhances model robustness across diverse image corruption conditions.

Table 2 presents averaged performance metrics across degradation levels. After averaging all degradation types and intensity levels, RGB-only achieves mAP50 and mAP50-95 of 0.827 and 0.631, respectively, while Union achieves mAP50 and mAP50-95 of 0.861 and 0.656, representing improvements of 3.4 and 2.5 percentage points, respectively, alongside gains in precision and recall.

Table 2. Model performance comparison under six degrees of degeneration.

Degradation	Model	mAP50	mAP50-95
Resolution reduction	RGB-only	0.943	0.745
Resolution reduction	Union	0.953	0.747
Gaussian Blur	RGB-only	0.843	0.643
Gaussian Blur	Union	0.846	0.647
Green Color Cast	RGB-only	0.920	0.714
Green Color Cast	Union	0.932	0.723
JPEG	RGB-only	0.894	0.677
JPEG	Union	0.916	0.695
Low Contrast	RGB-only	0.451	0.300
Low Contrast	Union	0.597	0.406
Motion Blur	RGB-only	0.909	0.705
Motion Blur	Union	0.926	0.719

4.5. Ablation Study: Is Sonar Simply Another Degradation Modality?

This section is designed as an ablation-style study to test whether the robustness gain of Union training is sonar-specific or can be explained by generic exposure to degraded visual styles. We evaluate on SUIM because it provides a different data distribution and includes multiple corruption variants (grayscale/blur/edge) that partially suppress color or texture cues. Notably, the auxiliary sonar-like set in SUIM introduces a stronger domain shift than MFLS/SeaClear; therefore, we report a two-stage variant (Union + FT) that first learns cross-modal structural supervision and then restores clean-RGB accuracy via short RGB-only fine-tuning. We note that grayscale/blur corruptions here are used as evaluation-time probes to suppress color or texture cues, providing a practical approximation to test whether the robustness gain is structure-related.

To further assess the generalizability of the proposed parameter-sharing RGB–FLS joint training strategy across datasets and modalities, we conducted an additional evaluation on the SUIM benchmark with externally generated sonar and corruption variants. Two pipelines were compared: (1) an RGB-only baseline trained on SUIM-RGB (denoted RGB-only) and (2) a two-stage Union pipeline (denoted Union + FT) trained by first mixing SUIM-RGB with the sonar auxiliary set (Union pre-training) and then applying a lightweight RGB fine-tuning stage on clean SUIM-RGB to recover the RGB decision boundary while retaining robustness learned from sonar supervision. All models were evaluated using the same class taxonomy (7 classes) and the same validation split.

Table 3 reports Precision, Recall, mAP50, and mAP50-95 under clean RGB, sonar-domain validation, and two representative corruption sets (blur and gray). Figure 7 and Figure 8 visualize the corresponding mAP trends. In this additional benchmark, Union + FT preserves clean-RGB accuracy on SUIM (mAP50: 0.288 vs. 0.292; mAP50-95: 0.194 vs. 0.195) while improving the sonar-domain evaluation (mAP50: 0.215 vs. 0.179; mAP50-95: 0.141 vs. 0.114). On corrupted RGB inputs, the gains are modest but consistent (e.g., grayscale mAP50: 0.254 vs. 0.243; blur mAP50-95: 0.114 vs. 0.108), which supports the interpretation that cross-modal supervision encourages more structure-driven representations rather than relying purely on color/texture cues.

Table 3. Cross-dataset evaluation on SUIM (RGB-only vs. Union + FT).

Condition	Model	Precision	Recall	mAP50	mAP50-95
Clean RGB (SUIM-RGB)	RGB-only-1002	0.530	0.275	0.292	0.195
Clean RGB (SUIM-RGB)	Union + FT (sonar-aux6)	0.488	0.305	0.288	0.194
Sonar domain (SUIM-Sonar)	RGB-only-1002	0.352	0.194	0.179	0.114
Sonar domain (SUIM-Sonar)	Union + FT (sonar-aux6)	0.439	0.214	0.215	0.141
Blur corruption	RGB-only-1002	0.402	0.135	0.173	0.108
Blur corruption	Union + FT (sonar-aux6)	0.391	0.139	0.173	0.114
Gray corruption	RGB-only-1002	0.444	0.243	0.243	0.164
Gray corruption	Union + FT (sonar-aux6)	0.465	0.255	0.254	0.169
Clean RGB (SUIM-RGB)	RGB-only-1002	0.530	0.275	0.292	0.195
Clean RGB (SUIM-RGB)	Union + FT (sonar-aux6)	0.488	0.305	0.288	0.194

Figure 7. mAP50 comparison on SUIM across clean RGB, sonar domain, and corrupted RGB variants (RGB-only-1002 vs. Union + FT).

Figure 8. mAP50-95 comparison on SUIM across clean RGB, sonar domain, and corrupted RGB variants (RGB-only-1002 vs. Union + FT).

5. Conclusions and Discussion

Underwater RGB object detection is highly sensitive to visibility degradation, where scattering, low illumination, and compression artifacts can suppress contrast and fine details and cause substantial performance drops. To improve reliability without increasing inference overhead or introducing paired multimodal acquisition requirements, this work presents a paired-sample-free RGB–FLS joint training strategy with shared parameters. By optimizing a single detector on heterogeneous RGB and FLS samples during training, the model is encouraged to reduce reliance on unstable color/texture cues and to learn more modality-invariant structural representations. At inference time, the detector remains an RGB-only model with unchanged architecture and computational cost.

Experiments on SeaClear and Marine Debris FLS demonstrate that the proposed strategy consistently improves robustness under multiple degradation factors, with the most pronounced gains observed in severe low-contrast scenarios. On clean RGB validation data, the joint training does not significantly degrade performance and can provide small improvements depending on the dataset. On the sonar-only validation set, a sonar-specialized model can achieve slightly higher peak performance than the shared-parameter model, reflecting an expected trade-off of weight sharing across modalities.

Practical guideline. The proposed Union training is best suited for deployments where an RGB camera is available and robustness under unpredictable underwater visibility is critical (e.g., turbidity, illumination variation, bandwidth-limited transmission). If the system operates exclusively on sonar inputs or prioritizes peak sonar-domain performance, sonar-only training may be preferred. When the auxiliary sonar distribution introduces substantial domain shift relative to the RGB domain, a short RGB-only fine-tuning stage after Union training (Union + FT) can recover clean-RGB accuracy while retaining robustness benefits from sonar-domain supervision.

This work has several limitations. First, the contribution is at the training-paradigm level rather than proposing a new detector architecture; further improvements may be obtained by incorporating domain-aware sampling or regularization to better control modality trade-offs. Second, the current evidence for “structural bias” is primarily behavioral (robustness under corruptions); deeper analysis (e.g., feature-space alignment or frequency sensitivity) is a promising direction. Finally, the proposed strategy has not yet been validated in closed-loop experiments on real underwater robotic platforms. Future work will explore domain-balanced training, more direct interpretability analyses of cross-modal representation transfer, and online evaluation in real missions.