A Multi-Supervised Network for Real-Time and Accurate Semantic Segmentation in Underwater Scenes

Abstract

Real-time semantic segmentation is a core perception capability for underwater robots and autonomous underwater vehicles (AUVs), yet it remains challenging because underwater imagery often exhibits low contrast, blurred boundaries, and strong appearance degradation under strict onboard computation budgets. This paper proposes MSNet, a multi-supervised two-pathway network that decouples feature learning into a semantic branch for context modeling and a detail branch for preserving high-resolution spatial information. MSNet introduces three complementary supervisory signals: (i) low-frequency semantic supervision derived from smoothed labels to encourage body semantics, (ii) high-frequency detail supervision derived from edge-enhanced labels to improve boundary localization, and (iii) category representation supervision implemented by a Category Representation Enhancement Module (CREM) to strengthen class discrimination at the deepest stage. To prevent auxiliary supervision from amplifying cross-resolution misalignment during fusion, we embed a Bilateral Flow-based Alignment Module (BFAM) into multi-stage feature fusion. Experiments on the SUIM benchmark show that MSNet achieves 79.83% mIoU and 86.57% F-score at 55 FPS with 6.2 M parameters on an RTX 3060 GPU, outperforming mainstream encoder–decoder and two-pathway algorithms. Compared with SFNet and BiSeNet V3, MSNet improves mIoU by 1.52% and 1.89%, and runs 9 FPS faster than SFNet. Ablation studies verify the effectiveness and complementarity of the proposed supervision and alignment strategies, indicating MSNet offers a practical accuracy–speed trade-off for marine engineering applications.

1. Introduction

Real-time underwater semantic segmentation is a fundamental task in the visual system of underwater robots and AUVs [1]. By providing pixel-level class predictions, semantic segmentation provides accurate environmental semantic information for marine engineering applications such as resource exploration, ecological monitoring, seabed mapping, and marine debris removal [2]. However, underwater platforms are typically constrained in computation and memory, which makes deploying heavy segmentation models difficult. Meanwhile, underwater images often suffer from degraded image quality, including low contrast, blurred boundaries, and strong appearance degradation [3]. This leads to a persistent accuracy–speed trade-off: preserving high-resolution details increases computation, whereas capturing multi-scale context requires deeper networks. Therefore, the key to practical real-time underwater segmentation is to extract and fuse semantic and detail features efficiently [4].

Existing real-time underwater segmentation approaches are adapted from general segmentation models and can be broadly grouped into encoder–decoder architectures and two-pathway architectures.

(1) Encoder–decoder architectures. Originating from U-Net [5], the U-shaped structure recovers spatial resolution via upsampling and skip connections, and has been widely adopted for underwater segmentation [6,7,8]. Early real-time designs such as ENet [9] and SegNet [10] improve efficiency by using bottleneck blocks in both encoder and decoder, and by reusing max-pooling indices to guide upsampling. Li et al. [11] integrate SegNet with stereo matching to build an unsupervised framework, which improves robustness in noisy or textureless regions and reduces reliance on dense underwater annotations. SwiftNet [12] proposes a real-time U-shaped design that adds an auxiliary encoder sharing weights with the main encoder. For underwater fish segmentation, Zhang et al. [13] compensate for early loss of fine details using multi-input processing in the first four encoder layers, and reduce parameters and FLOPs via multi-scale residual factorized convolutions. DeepLab V3+ [14]-based encoder–decoder variants can also achieve a favorable accuracy–speed trade-off when combined with lightweight backbones for underwater imagery [15]. Transformer-based methods [16] further improve global context modeling and have been increasingly explored in underwater scenes. SegFormer [17] and TopFormer [18] employ a hierarchical transformer encoder to output multi-scale features while controlling inference overhead. Chen et al. [19] adapt SegFormer to underwater segmentation by replacing the backbone with a Swin Transformer [20], leveraging shifted-window attention for context modeling and a pyramid-style decoder for fusion.

However, U-shaped encoder–decoder models typically rely on repeated upsampling and dense skip-feature fusion to recover details, which increases memory access and may propagate low-level noise. Directly injecting them into the decoder can blur boundaries or introduce false positives, especially for small objects in degraded underwater imagery. In addition, encoder–decoder fusion still suffers from cross-resolution feature misalignment due to the inconsistent distribution of information [21], and upsampling is often insufficient to correct it. As illustrated in Figure 1a, recent algorithms such as SFNet [22] and PP-LiteSeg [23] improve fusion by introducing dedicated alignment strategies (e.g., a flow alignment module, FAM). These strategies effectively mitigate misalignment and yield higher segmentation accuracy under real-time constraints.

Figure 1. Framework diagrams of the mainstream segmentation algorithms and MSNet. The dashed lines indicate components that are used only during training.

(2) Two-pathway (bilateral) architectures. Two-pathway real-time segmentation originates from BiSeNet [24] and has been progressively optimized by subsequent designs such as BiSeNet V2 [25], STDC [26], and DDRNet [27] (Figure 1b–d). These models employ two parallel branches to extract semantic context information and high-resolution spatial cues separately, and fuse them to obtain an accuracy–speed balance. BiSeNet V2 strengthens the bilateral design with a stem module and improves semantic representation via an auxiliary loss. BiSeNet V3 [28] introduces feature refinement and coordinate attention to enhance context modeling, and incorporates edge detection to strengthen boundary representation. For underwater blur interference, Zhang et al. [29] propose a multi-scale information enhancement module based on BiSeNet that aggregates local spatial and channel cues. Another underwater adaptation based on BiSeNet V2 [30] designs swap-attention by treating features from the two pathways as the key and query to capture cross-branch global dependencies. Beyond the BiSeNet family [31,32], Fast-SCNN [33] merges shallow convolutions and splits only in deeper layers with a global feature extractor. DDRNet [27] shares a merged shallow backbone and splits deeper layers into both branches to reduce computation, while adding auxiliary supervision on the spatial branch. STDC [26] builds high-quality features by concatenating multiple receptive-field maps and uses an extra layer as a detail head to enhance edge prediction. STDC-MA [34] further integrates hierarchical multi-scale attention and employs deformable convolution to correct pixel shifts. PSS-net [35] targets low-light underwater segmentation using separate paths, with pixel-aware loss for foreground boundaries and an improved dice loss to alleviate background imbalance. In addition to convolutional networks, Swin Transformer has also shown promising performance for underwater segmentation [36]. Nevertheless, two-pathway designs are mainly optimized by architectural separation and generic auxiliary segmentation losses, and they usually do not explicitly decouple supervision into low-frequency semantics and high-frequency details. Under underwater degradations, this can limit class discrimination for visually similar categories and make the fusion between specialized branches more sensitive to cross-resolution misalignment.

Beyond the final segmentation loss, both paradigms commonly introduce auxiliary losses, including semantic supervision and detail supervision, and discard the auxiliary heads at inference to improve accuracy without sacrificing runtime speed. These auxiliary objectives encourage the network to learn stronger semantic bodies and sharper spatial details (e.g., edges and corners). However, degraded underwater scenes often exhibit large-scale variations with densely distributed small targets, where supervision becomes less effective due to sparse pixels and weak visual cues. Although the mainstream methods in Figure 1 incorporate auxiliary losses to partially alleviate this issue, they often (i) under-utilize the effectiveness of supervised losses, and (ii) overlook category representation supervision that can strengthen class discrimination under underwater degradation. Moreover, stronger auxiliary supervision can further polarize low-level features toward details and high-level features toward semantics, which aggravates cross-resolution fusion misalignment and weakens the expected benefit of multi-stage fusion.

To address these issues, we propose MSNet (Figure 1e), a multi-supervised two-pathway network for real-time underwater semantic segmentation. MSNet explicitly supervises low-frequency semantic bodies and high-frequency spatial details, and further enhances category-aware representation via a Category Representation Enhancement Module. The low-frequency semantic loss suppresses pixel-level noise and enforces coherent body regions, while the high-frequency detail loss explicitly strengthens boundary pixels that are easily washed out by blur. In addition, we introduce a Bilateral Flow-based Alignment Module to align cross-resolution features during multi-stage fusion, mitigating misalignment amplified by stronger supervision. In contrast, existing bilateral designs such as BiSeNet and STDC typically rely on a single auxiliary segmentation loss or a generic detail head, which may not sufficiently disentangle these degradation-specific cues under strict real-time constraints. Overall, MSNet improves semantic consistency, boundary localization, and class discrimination while preserving real-time efficiency, which makes it suitable for underwater deployment in marine engineering scenarios.

2. Research Methods

Section 2.1 briefly introduces the overall framework of the proposed MSNet. Section 2.2, Section 2.3 and Section 2.4 provide detailed descriptions of the multi-supervised loss, the bilateral flow-based alignment module, and the loss function, respectively.

2.1. Overall Framework of MSNet

We propose MSNet, a multi-supervised network for real-time and accurate underwater scene segmentation. As shown in Figure 2, MSNet starts with high-resolution, low-channel feature representations and progressively forms a compact feature hierarchy. The network follows a two-pathway design with a semantic branch and a detail branch: the two branches share early backbone stages (Stage 1 to Stage 3), and the detail branch is extended from the endpoint of Stage 3. The semantic branch extracts multi-scale features across six stages and captures deep contextual information at 1/64 resolution. In contrast, the detail branch maintains feature maps at 1/8 resolution to balance spatial fidelity and runtime cost. To prevent loss of fine details, the detail branch avoids additional downsampling in later stages. This bilateral design enables MSNet to jointly preserve both large receptive fields and high-resolution features.

Figure 2. An overview of the multi-supervised network. The dashed lines indicate components that are used only during training.

MSNet introduces three complementary supervisory signals to maximize the effectiveness of auxiliary supervision: low-frequency semantic loss, high-frequency detail loss, and category representation loss. These losses are connected to the backbone via lightweight segmentation heads. Each head consists of a 3 × 3 Conv-BN-ReLU block followed by a 1 × 1 convolution to output a logit map. The semantic supervision is obtained by smoothing the ground truth mask, while the detail supervision is generated by subtracting the smoothed mask from the original ground truth and applying edge extraction. In addition, we introduce a CREM to strengthen category-aware representation. The CREM leverages the ground truth as supervision to reinforce class discrimination in the backbone. All auxiliary heads are used only during training and are removed at inference, thus preserving real-time efficiency [25].

After incorporating stronger and more specialized supervision, the semantic and detail optimization objectives become more focused, which can further amplify cross-resolution feature misalignment during fusion. To address this issue, we propose a Bilateral Flow-based Alignment Module to maintain consistent representations and enable reliable integration. Specifically, the BFAM is applied to all fusion steps, including two semantic-to-detail fusions, two detail-to-semantic fusions, and a final fusion across both branches. Finally, the fused features are used to produce the segmentation prediction and compute the overall training objective.

2.2. Multi-Supervised Loss

This section details the three supervisory signals used in MSNet: low-frequency semantic loss, high-frequency detail loss, and category representation loss.

2.2.1. Semantic Branch and Low-Frequency Semantic Loss

The semantic branch is designed to capture global context and body-consistent semantics. However, underwater objects are often blurred. Noise within the object significantly complicates feature extraction. To stabilize training and encourage coherent body regions, we construct a smoothed semantic target by applying a filtering operation to the ground truth mask GT_seg. This produces a more tolerant supervision signal that suppresses noise and promotes consistent semantic representations. The supervision of the low-frequency semantic loss is shown in Equation (1).

$$G{T}_{\mathrm{semantic}}=P\cdot G{T}_{\mathrm{seg}}$$

(1)

where P is a channel-wise Gaussian filter applied to GT_seg with kernel size 7 × 7 and σ = 2.0 (Figure 3c). We choose a compact 7 × 7 kernel to keep the semantic target smoothing local and to avoid over-blurring thin structures after resizing inputs to 256 × 192. A moderate σ = 2.0 provides sufficient attenuation of pixel-level label noise and blur-induced ambiguity inside underwater objects, while still preserving the coarse object extent needed by the semantic branch. As shown in the orange dashed box in Figure 2, semantic supervision is applied to the outputs of stages 2–5 through lightweight segmentation heads. During training, the low-frequency semantic loss is computed by comparing each auxiliary prediction with the smoothed semantic target defined in Equation (1).

Figure 3. Illustration of the detail loss supervision framework.

2.2.2. Detail Branch and High-Frequency Detail Loss

In contrast, the detail loss is designed to rectify finer spatial details, such as edges and corners, which are critical for boundary localization in underwater imagery. Boundary pixels are typically sparse and difficult to classify because they contain limited appearance information but exhibit complex shapes. To provide stronger and more targeted supervision for these hard pixels, we explicitly derive a boundary-oriented target for the detail branch. Specifically, because high-frequency details are intentionally attenuated when constructing the smoothed semantic targets, we first subtract the smoothed semantic supervision from the original ground truth to recover the previously blurred high-frequency components (Figure 3d). We then apply an edge operator (e.g., Sobel, Canny, or Laplacian) to extract a clean boundary target, as defined in Equation (2).

$$G{T}_{\mathrm{detail}}=C\cdot (G{T}_{\mathrm{seg}}-G{T}_{\mathrm{semantic}})$$

(2)

where C represents the edge extraction operation. We instantiate C as a Laplacian operator (ksize = 3) on the residual map, take the absolute response, and normalize and binarize it with a threshold 0.1 to obtain GT_detail. This two-step design produces clearer and more complete boundaries than directly applying edge detection to the raw ground truth (Figure 3e,f). The detail loss is connected to each component of the detail branch via segmentation heads. During training, the detail heads predict a high-frequency boundary logit map representing object outlines. The loss is computed by comparing the prediction with GT_detail.

2.2.3. CREM and Category Representation Loss

To enhance category discrimination for underwater instances, we propose the Category Representation Enhancement Module, as illustrated in Figure 4. We first transform the output of the semantic branch at Stage 6 (the deepest feature) into a set of category embeddings, where each embedding corresponds to one class. These category embeddings are directly supervised by the ground truth to inject explicit category information. In parallel, we project the same stage 6 features into pixel-level representations. Based on the pixel representations and category embeddings, we compute k category representation vectors (prototypes) for each class by aggregating the pixel-level features belonging to that category. Pixel-category correlation is measured by dot-product similarity between each pixel embedding and the class prototypes. The correlations are normalized with softmax to obtain aggregation weights. Using these weights, we form a category-aware contextual feature as a weighted sum of the prototypes, so that each pixel receives context aligned with its most relevant category representations. This contextual representation is then concatenated with the output from stage 6 of the semantic branch, supplementing the semantic features with category information. A 1 × 1 convolution is applied for channel compression to produce the CREM output.

Figure 4. Detailed design of the CREM. The dashed lines indicate components that are used only during training.

We apply the CREM after stage 6 for two practical reasons. First, the semantic supervision has already improved feature consistency, which makes category representation learning more reliable. Second, stage 6 features are computed at 1/64 resolution, so the CREM introduces only a modest runtime overhead, which is important for real-time deployment on underwater platforms. The weights in the CREM are computed based on pixel-category correlations, which allow for the discrimination of context pixels from the same or different object classes, thus augmenting the category information pertaining to the target object. By supervising category embeddings and constructing ground-truth-guided class prototypes, the CREM encourages intra-class compactness and enlarges inter-class margins, which helps reduce confusion under noise or class overlap.

2.3. Bilateral Flow-Based Alignment Module

After introducing the multi-supervised losses, the two branches become more specialized: the semantic branch emphasizes context, while the detail branch focuses on boundaries (Figure 5, blue box). This specialization can amplify cross-resolution misalignment during fusion, resulting in inconsistent information distribution in feature maps at different scales [21]. In practice, simple upsampling followed by element-wise addition is often insufficient to correct these shifts. To address this issue under real-time constraints, we insert a Bilateral Flow-based Alignment Module into each fusion stage to align adjacent features and enable efficient multi-scale aggregation.

Figure 5. Illustration of the BFAM.

The BFAM architecture is shown in Figure 5. For low-to-high resolution fusion, the BFAM predicts a 2D flow field (two channels for x/y offsets) from the high-resolution feature and the upsampled low-resolution feature using a lightweight 3 × 3 convolution. The low-resolution feature is then warped by bilinear sampling using the learned offsets to match the high-resolution grid. For high-to-low fusion (detail-to-semantic alignment), the flow field is predicted from the low-resolution feature and the downsampled high-resolution feature, and the high-resolution feature is warped to the low-resolution grid (Figure 5b). For adjacent stages with a 2× scale gap, offsets are scaled consistently to maintain a unified coordinate convention.

The BFAM promotes coherent information transfer between deep and shallow layers. MSNet applies the BFAM at multiple fusion locations, including two semantic-to-detail fusions, two detail-to-semantic fusions, and the final branch-level fusion. This design mitigates misalignment that would otherwise be amplified by stronger auxiliary supervision.

2.4. Loss Function

Introducing multi-level supervision increases the number of optimization targets. Following the principle of jointly supervising intermediate representations and the final prediction, we explicitly model low-frequency semantics, high-frequency boundaries, and category-aware representation, and jointly optimize them with the final segmentation loss in a unified objective. The overall training loss is defined in Equation (3).

$$L={\lambda }_s{L}_{\mathrm{semantic}}+{\lambda }_d{L}_{\mathrm{detail}}+{\lambda }_c{L}_{\mathrm{category}}+{\lambda }_f{L}_{\mathrm{final}}$$

(3)

Low-frequency semantic loss. Let $\left\{p_l^s\right\}$ denote the predictions of the semantic auxiliary heads attached to stages $l\in \{2,3,4,5\}$. Using the smoothed supervision GT_Semantic, we define

$$L_{\mathrm{semantic}}={\displaystyle \frac{1}{\left|H_s\right|}}{\displaystyle \sum _{l\in H_s}\mathrm{CE}(p_l^s, G{T}_{\mathrm{semantic}})}$$

(4)

where CE( ) is the pixel-wise cross-entropy, $H_s=\{2, 3, 4, 5\}$ and GT_Semantic is defined in Equation (1).

High-frequency detail loss. Let $\left\{p_m^d\right\}$ be the boundary predictions from the detail heads, supervised by GT_detail defined in Equation (2). Because boundary pixels are sparse, we adopt a class-balanced binary cross-entropy. The detail loss is defined in Equations (5) and (6).

$$L_{\mathrm{detail}}={\displaystyle \frac{1}{\left|H_d\right|}}{\displaystyle \sum _{m\in H_d}{\mathrm{BCE}}_w(p_m^d, G{T}_{\mathrm{detail}})}$$

(5)

$${\mathrm{BCE}}_w(p, \mathrm{y})=-\alpha y\log p-\beta (1-y)\log (1-p)$$

(6)

where α and β are set according to the positive/negative pixel ratio to mitigate imbalance.

Category representation loss. For the CREM, we impose category-aware supervision to the CREM output $p^c$. The ground truth label GT_seg is downsampled to match the CREM resolution, and the loss is computed by cross-entropy as defined in Equation (7).

$$L_{\mathrm{category}}=\mathrm{CE}(p^c, G{T}_{\mathrm{seg}})$$

(7)

Final segmentation loss. The final prediction $p^f$ is supervised by the original ground truth using cross-entropy, as defined in Equation (8).

$$L_{\mathrm{final}}=\mathrm{CE}(p^f, G{T}_{\mathrm{seg}})$$

(8)

Loss weights and inference. We use ${\lambda }_s={\lambda }_d={\lambda }_c=0.4$ for the auxiliary losses and ${\lambda }_f=1$ for the final segmentation loss, following common weighting settings in segmentation with auxiliary supervision [25,37]. During inference, all auxiliary heads are removed, so the runtime overhead is limited to the main network path and preserves real-time efficiency.

3. Experiments

This section describes the experimental settings, validates each component through ablation studies, and compares MSNet with mainstream semantic segmentation methods on underwater data.

3.1. Experimental Settings

Runtime environment. All experiments were conducted in PyTorch 1.7. and Python 3.8 on Ubuntu 20.04. The hardware platform includes an Intel i5-10400F CPU, 16 GB RAM, and an NVIDIA GeForce RTX 3060 GPU. CUDA 11.0 and cuDNN 8.0.4 have also been installed for GPU acceleration.

Dataset and Evaluation. We evaluate MSNet on the typical underwater scene dataset SUIM [38], which provides pixel-level annotations for underwater scenes. SUIM contains over 1500 underwater images and covers eight core categories: fish, reefs, plants, wrecks/ruins, humans, robots, sea-floor/sand, and waterbody background. Segmentation accuracy is assessed using mean Intersection over Union (mIoU) and F-score. MIoU quantifies the overlap degree between the predicted segmentation mask and the label by calculating the mean intersection over union of all categories, and it is the core criterion for evaluating the segmentation accuracy. F-score is computed from precision (P) and recall (R), as defined in Equation (9).

$$F={\displaystyle \frac{2\times P\times R}{P+R}}$$

(9)

Real-time performance is measured in frames per second (FPS). To make the real-time benchmarking reproducible, FPS is measured with batch size 1 at an input resolution of 256 × 192 on the RTX 3060 GPU. We report the pure network inference throughput and exclude data loading and disk I/O time. Before timing, we run 50 warm-up iterations to stabilize CUDA kernel selection and GPU clocks. We then measure 300 consecutive forward passes and compute the average latency per image; CUDA synchronization is enforced before and after each timed iteration. Unless otherwise stated, all results are obtained in FP32 using native PyTorch inference; mixed precision (FP16) and TensorRT acceleration are not used. We enable cuDNN autotuning for stable performance (torch.backends.cudnn.benchmark = True). The reported FPS is computed as the reciprocal of the average per-image latency.

Training Details. All models are trained with the Adam optimizer [39] with an initial learning rate of 0.0004. The batch size is 8, and training runs for 400 epochs. Input images are resized to 256 × 192, with random data augmentation including random flipping, translation, and rotation.

3.2. Ablation Studies

3.2.1. Ablation Studies on Multi-Supervised Loss

We first evaluate the three proposed supervisory signals—low-frequency semantic loss, high-frequency detail loss, and category representation loss. The two-pathway architecture in Figure 2 is chosen as the baseline, which reaches 74% mIoU at 68 FPS. The baseline removes all auxiliary heads and the BFAM, and uses simple upsampling followed by element-wise addition for fusion.

Low-frequency semantic loss. Equation (1) constructs semantic supervision by smoothing the ground truth, which suppresses noise and stabilizes body-aware semantics. Different smoothing filters are compared, including mean, Gaussian, and median filters. As shown in Table 1, all smoothed targets improve over the baseline, and the Gaussian filter with kernel size 7 × 7 and σ = 2.0 yields the best performance (75.49% mIoU). Therefore, Gaussian smoothing is used in subsequent experiments.

Table 1. Ablation results of different smoothing operations.

Smoothing Filters	k = 5	k = 7	k = 9
Gaussian filters (σ = 1.0)	74.72	75.36	73.83
Gaussian filters (σ = 2.0)	74.81	75.49	73.91
Gaussian filters (σ = 3.0)	74.84	75.44	73.93
Mean filters	74.88	75.38	73.97
Median filters	74.26	74.65	73.75

High-frequency detail loss. As illustrated in Figure 6a, directly supervising boundary prediction with the original ground truth is ineffective and slightly degrades accuracy, because dominant body regions weaken the learning signal for sparse boundaries. We therefore subtract the smoothed semantic supervision from the ground truth to isolate high-frequency components, and apply edge extraction to obtain clearer boundary targets. Among Sobel, Canny, and Laplacian operators, Laplacian performs best (75.87% mIoU) and is adopted in MSNet. These gains are observed on SUIM under typical visibility; in extremely low-contrast or highly turbid scenes, excessive smoothing may over-blur small objects and the benefit may diminish.

Figure 6. Ablation studies on semantic loss, detail loss, and category representation loss.

CREM and category representation loss. We further strengthen class discrimination using the CREM. As shown in Figure 6b, supervising stage 6 features alone yields limited improvement by 0.79% mIoU. Adding the CREM enhances category-aware representation with only a small runtime impact due to the CREM operating at 1/64 resolution, and applying category-level supervision on CREM output further improves accuracy to 75.79% mIoU.

Table 2 shows that each loss term is beneficial when used alone, and combining two losses yields further gains. In particular, when semantic loss and category representation loss act together, mIoU increases by 2.73%, because the more consistent semantic features provide a better foundation for category-aware enhancement. Enabling all three losses achieves the highest accuracy, reaching 77.79% mIoU with a modest reduction in speed by 4 FPS. Overall, jointly modeling low-frequency semantics, high-frequency boundaries, and category representation improves underwater segmentation under real-time constraints.

Table 2. Ablation studies on multi-supervised loss. ✓ indicates that the module is included, while ✗ indicates that the module is excluded.

Semantic Loss	Detail Loss	Category Representation Loss	mIoU (%)	FPS
✗	✗	✗	74.0	68
✓	✗	✗	75.49	68
✗	✓	✗	75.87	68
✗	✗	✓	75.78	64
✓	✓	✗	76.61	68
✓	✗	✓	76.73	64
✗	✓	✓	76.57	64
✓	✓	✓	77.79	64

To illustrate the effect of multi-supervised losses, Figure 7 visualizes the learned feature responses. In the detail branch, the features guided by the detail loss preserve richer spatial cues and produce sharper edges and corners, which is particularly beneficial for underwater targets with blurred boundaries. In the semantic branch, the original responses are relatively coarse. The semantic loss encourages the network to capture more complete object information, including the body region and the overall outline. After introducing the CREM and category representation loss, features become more category-aware, which improves discrimination in low-contrast underwater scenes.

Figure 7. Visualization of features improved by multi-supervised loss.

3.2.2. Ablation Studies on the Bilateral Flow-Based Alignment Module

We evaluate the BFAM as an alignment module at three fusion locations: high-to-low fusion in the semantic branch, low-to-high fusion in the detail branch, and the final branch-level fusion. We also compare the BFAM with a deformable convolutional module [40] as an alternative alignment module. Table 3 shows that both alignment designs improve accuracy, while the BFAM consistently achieves higher mIoU with comparable overhead. The largest gain is observed at the final fusion (1.33% mIoU and 2.12% mIoU), where misalignment is most pronounced. Applying BFAM at all fusion stages yields the best alignment quality and improves mIoU to 77.35% with acceptable runtime overhead. Compared with DCN, the BFAM increases mIoU by 1.44%, and its FPS only decreases by 2.

Table 3. Ablation studies on the BFAM.

Method	High-to-Low	Low-to-High	Final	mIoU (%)	FPS
Baseline	✗	✗	✗	74.0	68
Deformable convolutional module [40]	✓	✗	✗	74.98	66
	✗	✓	✗	75.06	66
	✗	✗	✓	75.33	67
	✓	✓	✓	75.91	62
BFAM	✓	✗	✗	75.81	65
	✗	✓	✗	75.87	65
	✗	✗	✓	76.12	66
	✓	✓	✓	77.35	60

3.2.3. Overall Ablation Studies

Finally, Table 4 summarizes the overall ablation results. Each strategy is effective when applied individually. Multi-supervised loss improves the mIoU and F-score by 3.79% and 0.81%, respectively. By mitigating cross-resolution inconsistencies during fusion, the BFAM alone improves the mIoU by 3.35% and F-score by 1.37%. When both strategies are enabled, multi-supervised loss strengthens semantic cues, boundary details, and category representations but also amplifies feature misalignment, whereas the BFAM corrects this misalignment during fusion. Consequently, MSNet achieves the best accuracy of 79.83% mIoU and 86.57% F-score. Although the speed decreases to 55 FPS, it still satisfies real-time requirements for underwater segmentation. These results indicate that the two strategies are complementary and most effective when combined. Notably, although auxiliary supervision and bilateral architectures have been explored in prior real-time models, the proposed multi-supervised loss together with the BFAM yields substantial and complementary gains, which explains why MSNet outperforms existing solutions in Table 5.

Table 4. Overall ablation studies.

Multi-Supervised Loss	BFAM	mIoU (%)	F-Score (%)	FPS
✗	✗	74.0	84.14	68
✓	✗	77.79	84.95	64
✗	✓	77.35	85.51	60
✓	✓	79.83	86.57	55

3.3. Comparison

We compare MSNet with mainstream segmentation models, including encoder–decoder architectures (U-Net, DeepLab V3, SFNet, and TopFormer) and two-pathway architectures (BiSeNet, STDC, and DDRNet). To ensure a fair and reproducible comparison, we follow a unified evaluation protocol for MSNet and all baselines. Specifically, we use the same SUIM training/test split, the same input resolution (256 × 192), and the same training schedule (Adam optimizer with an initial learning rate of 0.0004, batch size 8, and 400 epochs), together with the same data augmentation policy. All models are trained from scratch. During inference, we adopt a single-scale test setting (no multi-scale or flip testing). FPS is measured on the same RTX 3060 GPU under the same input resolution and averaged with batch size 1.

As summarized in Table 5, two-pathway designs generally provide a better accuracy–speed trade-off for real-time underwater segmentation, and the number of network parameters is significantly smaller. SFNet achieves the best accuracy (78.31% mIoU and 85.95% F-score) among encoder–decoder methods and runs at 46 FPS. It surpasses U-Net and DeepLab V3 by 0.32% and 1.06% mIoU, respectively. BiSeNet V3 is the strongest baseline among two-pathway architectures, reaching 77.94% mIoU and 85.80% F-score at 57 FPS and 13 M parameters. Built on a bilateral architecture, MSNet further improves representation learning through multi-supervision and mitigates cross-resolution misalignment via the BFAM, achieving the best overall performance of 79.83% mIoU and 86.57% F-score. This corresponds to gains of 1.52% mIoU over SFNet and 1.89% over BiSeNet V3, while maintaining 55 FPS and 6.2 M parameters, which meets real-time and deployment requirements for underwater platforms.

Table 5. Comparison of MSNet and other mainstream segmentation algorithms.

	Method	mIoU (%)	F-Score (%)	Parameters (M)	FPS
Encoder–Decoder architectures	UNet	77.99	85.82	31.0	32
	SegNet	75.84	84.93	15.0	20
	Deeplab V3	77.25	85.51	41.3	33
	SegFormer	78.05	85.83	27.5	27
	TopFormer	78.08	85.74	5.1	59
	SFNet	78.31	85.95	12.9	46
Two-pathway architectures	BiSeNet V2	77.19	85.49	4.3	60
	STDC	77.46	85.60	8.4	67
	DDRNet	77.81	85.74	5.7	63
	BiSeNet V3	77.94	85.80	13.0	57
	MSNet	79.83	86.57	6.2	55

Table 5. Comparison of MSNet and other mainstream segmentation algorithms.

	Method	mIoU (%)	F-Score (%)	Parameters (M)	FPS
Encoder–Decoder architectures	UNet	77.99	85.82	31.0	32
	SegNet	75.84	84.93	15.0	20
	Deeplab V3	77.25	85.51	41.3	33
	SegFormer	78.05	85.83	27.5	27
	TopFormer	78.08	85.74	5.1	59
	SFNet	78.31	85.95	12.9	46
Two-pathway architectures	BiSeNet V2	77.19	85.49	4.3	60
	STDC	77.46	85.60	8.4	67
	DDRNet	77.81	85.74	5.7	63
	BiSeNet V3	77.94	85.80	13.0	57
	MSNet	79.83	86.57	6.2	55

Figure 8 further reports class-wise performance on five major SUIM categories: HD (Human divers), WR (Wrecks/ruins), RO (Robots/instruments), RI (Reefs and invertebrates), and FV (Fish and vertebrates). The proposed method achieves 80.78%, 85.59%, 68.18%, 85.20%, and 79.40% mIoU on these classes, respectively. MSNet consistently outperforms SFNet and BiSeNet V3 across these categories, reflecting improved boundary localization and category discrimination under low-contrast underwater conditions.

Figure 8. Precision of SUIM major categories.

We further diagnose MSNet by considering category imbalance. Some classes in SUIM appear far less frequently than others (e.g., RO appears in 101 images, while RI and FV appear in 1028 and 1030 images, respectively). This imbalance tends to amplify under-segmentation and boundary ambiguity for rare or small-extent objects. Consistent with this observation, RO yields the lowest mIoU in Figure 8, whereas more frequent and spatially larger categories such as WR/RI achieve higher IoU. These results suggest that MSNet improves category discrimination under underwater degradation, while remaining sensitive to the scarcity and scale of certain object classes.

Figure 9 compares qualitative predictions. Unlike generic segmentation scenarios, low contrast underwater scenes contain large-scale variation and many densely distributed small targets, which makes both boundary localization and small-object segmentation challenging. MSNet produces the most accurate segmentation for both large structures (e.g., wreck-like objects) and densely packed small-scale targets (e.g., fish), demonstrating the effectiveness of the proposed multi-supervised learning and alignment design.

Figure 9. Prediction results of different models.

Although MSNet generally produces more complete masks for both large structures and densely packed small targets, failure cases still occur in challenging underwater conditions. For example, in scenes with highly cluttered small-object distributions (e.g., dense fish schools), MSNet may still miss a portion of isolated instances or produce fragmented masks due to severe scale variation and weak local contrast. In low-illumination or heavy color-cast scenes around large structures (e.g., wreck-like regions), the boundary between object surfaces and surrounding background can become indistinct, which may lead to local boundary leakage or scattered false positives near textured regions. These cases are consistent with typical underwater degradations, and indicate that further robustness improvements remain valuable for onboard deployment.

4. Discussion

This paper proposes MSNet for real-time underwater semantic segmentation and demonstrates a strong accuracy–speed trade-off on SUIM. The method benefits from a multi-supervised loss that provides complementary learning signals for semantic regions and spatial details, while the CREM enhances category-aware discrimination under underwater degradation. Meanwhile, the BFAM mitigates cross-resolution feature misalignment during fusion, which is crucial when stronger supervision emphasizes fine structures. Overall, MSNet achieves high segmentation quality while keeping model size and inference cost suitable for real-time deployment. In addition to FPS, the reported throughput corresponds to an average latency of 18.2 ms per frame for MSNet on the RTX 3060 at 256 × 192. Profiling on embedded platforms (e.g., Jetson Orin/Xavier) will be included in future work; nevertheless, the compact model size (6.2 M parameters) and the clarified inference protocol support the feasibility of onboard deployment.

MSNet still has limitations. These limitations can be quantified from the class-wise results in Figure 8: among the five major SUIM categories, the per-class mIoU ranges from 68.18% to 85.59%, and the RO (Robots/Instruments) category achieves only 68.18% mIoU, which is 11.22–17.41 percentage points lower than the other four categories (HD 80.78%, WR 85.59%, RI 85.20%, and FV 79.40%). This gap indicates that small, reflective instruments and visually similar structures remain challenging under low contrast and blur, where boundaries are ambiguous and inter-class appearance overlap is more severe. In addition, while MSNet is trained and evaluated on SUIM, cross-domain generalization under changes in water types, illumination, and sensors is not explicitly quantified in this work, and performance may drop under domain shift. Due to limited computational resources, in this research, we focus on real-time segmentation under general underwater scenes and do not add an additional robustness benchmark with synthetic noise or contrast perturbations; we discuss this limitation here and will include such stress tests and broader cross-domain evaluations in future work.

Future work will focus on improving robustness under diverse underwater conditions by incorporating visibility-aware training and domain generalization/adaptation strategies, and further optimizing the network for embedded platforms through (i) knowledge distillation from a stronger teacher model to maintain accuracy after compression, and (ii) structured pruning/channel slimming to reduce memory bandwidth and latency, together with deployment-oriented acceleration (e.g., TensorRT/ONNX export and operator fusion) when available, while maintaining boundary accuracy and real-time performance. In future work, we will report end-to-end latency and power measurements on embedded platforms (e.g., Jetson Orin/Xavier) and evaluate operational stability on continuous video streams (e.g., FPS jitter and temporal consistency) under camera motion. Furthermore, to improve robustness in extremely low-visibility and noisy conditions and to better support cross-domain and new-class transfer, future work will explore domain generalization/adaptation (e.g., style and visibility perturbation, synthetic turbidity/contrast augmentation, and test-time adaptation) and few-shot or continual learning for efficiently incorporating new categories with limited annotations.

5. Conclusions

This paper presented MSNet, a multi-supervised two-pathway network for real-time semantic segmentation in underwater scenes. MSNet introduces multi-supervised learning to decouple optimization targets: smoothed labels guide low-frequency semantics, boundary-focused labels enhance high-frequency details, and the CREM provides category-aware representation to improve discrimination under underwater degradation. To counteract the misalignment problem amplified by stronger auxiliary supervision, the BFAM is integrated into cross-resolution fusion to align adjacent features during aggregation.

Experiments on the SUIM benchmark demonstrate that MSNet achieves 79.83% mIoU and 86.57% F-score at 55 FPS, outperforming mainstream encoder–decoder and two-pathway architectures. Ablation studies confirm that the proposed losses are complementary and that the BFAM effectively mitigates misalignment, enabling a favorable accuracy–speed balance for deployment on underwater platforms. Future work will investigate robustness under more diverse water types and visibility conditions, and will target embedded real-time deployment via concrete model-level and deployment-level optimizations, including quantization (FP16/INT8), knowledge distillation, and structured pruning, as well as compiler/engine acceleration such as TensorRT/ONNX when applicable.