USV-Seg: A Vision-Language Framework for Guided Segmentation of USV with Physical Constraint Optimization

Abstract

Unmanned Surface Vehicles (USVs) play a critical role in maritime monitoring, environmental protection, and emergency response, necessitating accurate scene understanding in complex aquatic environments. Conventional semantic segmentation methods often fail to capture global context and lack physical boundary consistency, limiting real-world performance. This paper proposes USV-Seg, a unified segmentation framework integrating a vision-language model, the Segment Anything Model (SAM), DINOv2-based visual features, and a physically constrained refinement module. We design a task-specific <Describe> Token to enable fine-grained semantic reasoning of navigation scenes, considering USV-to-shore distance, landform complexity, and water surface texture. A mask selection algorithm based on multi-layer Intersection-over-Prediction (IoP) heads improves segmentation precision across sky, water, and obstacle regions. A boundary-aware correction module refines outputs using estimated sky-water and land-water boundaries, enhancing robustness and realism. Unlike prior works that simply apply vision-language or geometric post-processing in isolation, USV-Seg integrates structured scene reasoning and scene-aware boundary constraints into a unified and physically consistent framework. Experiments on a real-world USV dataset demonstrate that USV-Seg outperforms state-of-the-art methods, achieving 96.30% mIoU in challenging near-shore scenarios.

1. Introduction

Unmanned Surface Vehicles (USVs) are gaining increasing prominence across a wide range of applications, including maritime monitoring, environmental surveillance, emergency response, and autonomous navigation [1,2,3]. Their expanding role in marine scenarios has spurred the development of advanced perception technologies, with image segmentation emerging as a critical task for scene understanding and navigation decision-making [4,5,6]. Accurate segmentation of semantic regions—such as sky, water surfaces, shoreline, and obstacles—is essential for ensuring the robustness and safety of USV operations, particularly in dynamic and visually complex environments.

Over the past decade, significant progress has been made in USV image segmentation. Traditional approaches based on edge detection, thresholding, or region-growing techniques [7] provided simple solutions but were highly sensitive to illumination changes, water reflections, and wave-induced patterns, limiting their effectiveness in real-world maritime scenes. The advent of deep learning revolutionized this field: fully convolutional networks (FCNs) [8] and U-Net [9] architectures significantly improved segmentation accuracy by learning rich hierarchical features and capturing fine details. Later studies introduced post-processing with conditional random fields (CRFs) [10] or superpixel enhancements [11] to further improve boundary delineation and robustness.

Recent trends have explored multimodal and promptable segmentation models that leverage vision-language learning [12,13] and self-supervised representation learning [14,15] to enable zero-shot generalization, few-shot learning, and reduced annotation burdens. In parallel, researchers have begun incorporating domain-specific physical constraints into the segmentation process [16,17], using LiDAR, multibeam sonar, and epipolar geometry to enforce geometric plausibility and improve shoreline detection. These advances have led to improved performance in specific aspects of the segmentation task.

Nevertheless, current methods remain inadequate in several respects when deployed in realistic maritime environments. Many vision-only models lack scene-level reasoning, treating each pixel independently without considering the broader navigational context. Geometric constraints are often applied only as post-processing steps, rather than being integrated into the model’s core reasoning pipeline. Furthermore, the computational demands of large deep networks frequently exceed the processing capacity of embedded USV hardware, making them unsuitable for real-time deployment. Finally, generalization to diverse and visually ambiguous maritime scenes—such as those with adverse weather, dynamic waves, or complex shorelines—remains an unresolved challenge.

To summarize, existing segmentation methods for USV navigation face the following key limitations:

• Poor generalization to diverse maritime scenes: Current models trained on limited datasets often fail when applied to unseen conditions, such as varying illumination, sea states, or shoreline geometries.
• Lack of physically plausible boundaries: Many approaches produce inconsistent or implausible boundaries, misclassifying water, sky, and land in visually ambiguous regions.
• Insufficient semantic reasoning: Existing models rarely incorporate structured scene-level knowledge, such as the expected spatial relationship between shoreline, water surface, and sky.
• computational complexity: State-of-the-art deep models are often too resource-intensive for real-time inference on USV platforms.

To address these challenges, we propose USV-Seg, a novel semantic segmentation framework tailored to the unique demands of USV navigation. USV-Seg integrates vision-language understanding, multimodal promptable segmentation, and domain-specific physical constraints into a unified and lightweight model that is both accurate and real-time capable. Our framework bridges the gap between semantic reasoning and physical plausibility, producing scene-consistent and robust segmentations suitable for real-world deployment.

2. Related Work

2.1. Traditional Image Segmentation for USVs

Early image segmentation approaches for USVs primarily relied on conventional image processing techniques, such as edge detection and threshold-based segmentation. As reviewed by Albukhnefis [7], these included edge detection algorithms, threshold-based segmentation, and region-growing methods. While effective in controlled environments, these techniques demonstrated limited performance in complex maritime scenarios due to their susceptibility to illumination variations, water surface reflections, and dynamic wave patterns. Other notable approaches included histogram-based water detection [18], line-segment-based detection with temporal stability filtering [19], and gradient-based sky-water separation [20]. These early methods laid the groundwork for later developments but were often brittle and lacked robustness under dynamic sea states and non-uniform lighting conditions [21,22,23].

2.2. Deep Learning-Based Segmentation

The advent of deep learning revolutionized image segmentation for USVs, with Fully Convolutional Networks (FCN) [8] establishing the foundation for modern approaches. Ronneberger [24] demonstrated that FCN-based architectures could achieve over 95% accuracy in water/non-water classification tasks, a significant improvement over traditional methods. The introduction of U-Net [9] with its distinctive encoder-decoder structure and skip connections further enhanced performance, particularly in capturing fine details of water edges and small obstacles.

Subsequent work adapted other mainstream architectures to maritime scenes, such as DeepLabv3+ [25,26], PSPNet [27], and HRNet [28], which leveraged atrous convolutions and pyramid pooling to better model multi-scale context. GAN-based methods were also introduced for improving segmentation quality under adverse weather and low-light conditions [29]. Transformer-based architectures like SegFormer [30] and Swin-UNet [31] were recently explored for their ability to capture long-range dependencies and improve boundary sharpness. Recent domain adaptation techniques [32] addressed dataset bias by aligning feature distributions across different maritime datasets. Xue [11] proposed a SLIC algorithm-enhanced deep semantic segmentation model that improved obstacle edge detection accuracy by 12% compared to standard U-Net. Zhan [10] developed an adaptive semantic segmentation method incorporating Conditional Random Fields (CRF) for post-processing, achieving excellent performance with limited training data through self-training with refined pseudo-labels.

2.3. Physical Constraints in Segmentation

Recognizing that pure vision-based approaches sometimes produce physically implausible results, researchers have increasingly incorporated domain-specific physical constraints into segmentation frameworks. Jaszcz [16] developed a comprehensive system combining USV-mounted LiDAR and multibeam echosounder (MBES) data with satellite imagery to generate accurate shoreline segmentation masks. Several studies [33,34] have focused specifically on water-shoreline (WSL) detection, a critical task for USV navigation. Zou [19] proposed a three-stage method using line segment detection, water region instability analysis, and epipolar constraints that demonstrated robust performance for both straight and curved shorelines. Halicki [17] further improved precision by incorporating LiDAR point cloud data from USV-mounted sensors, enabling sub-meter accuracy in shoreline extraction. Additional approaches integrated elevation models [16], polarimetric imaging [35], and bathymetric priors [36] to enforce geometric plausibility. Recent probabilistic graphical models [37,38] have also been employed to enforce higher-order spatial consistency across segmentation boundaries.

2.4. Multimodal and Prompt-Based Segmentation

Recent advances in multimodal learning have opened new possibilities for maritime image segmentation. The Segment Anything Model (SAM) [12] introduced a promptable segmentation framework that enables zero-shot generalization to new maritime environments. Wang [13] extended this approach to multimodal image segmentation through latent space token generation and fused mask prompting mechanisms. Self-supervised learning methods like DINOv2 [14] have demonstrated remarkable feature learning capabilities without extensive labeled data. Ayzenberg [15] successfully adapted these techniques for few-shot image segmentation, significantly reducing annotation requirements while maintaining high accuracy. Other prompt-based approaches such as CLIPSeg [39], LSeg [40], and DenseCLIP [41] have demonstrated impressive flexibility in transferring semantic knowledge to novel maritime domains. Multimodal sensor fusion combining RGB, LiDAR, SAR, and hyperspectral imagery has been explored to enhance segmentation reliability under extreme conditions [42,43,44]. Recent work has also proposed integrating temporal cues for video-based segmentation of USV navigation scenes [45,46].

2.5. Remaining Challenges

While USV image segmentation has made notable progress, current methods still face key limitations in complex maritime environments. Despite high accuracy in controlled settings, they struggle with illumination changes, wave dynamics, and adverse weather. Generalization across diverse water bodies is poor due to limited training datasets. Moreover, balancing accuracy and real-time performance remains a challenge for resource-constrained USV platforms, especially when handling high-resolution or multi-sensor data. A more fundamental issue lies in the limited incorporation of domain-specific physical constraints—such as water-sky and water-shoreline relationships—into segmentation models. This often leads to boundary misclassification in visually ambiguous scenes. While some approaches incorporate LiDAR or radar, they typically require costly sensor setups and lack the flexibility of vision-only solutions. Developing lightweight, physically consistent, and semantically aware segmentation methods remains an open research direction.

3. Algorithm

3.1. Algorithm Structure

The proposed USV-Seg system consists of five main modules: a pre-trained MiniCPM vision-language model, the Segment Anything Model (SAM), the DINOv2 model, a mask selection module, and a physical constraint-enhanced segmentation module. The vision-language model (MiniCPM-o 2.6) [47] is responsible for interpreting both the input image and user queries. We introduce a special segmentation control token, <Describe>, into the MiniCPM-o 2.6 vocabulary to indicate segmentation. During forward propagation, the <Describe> token is dynamically updated to store specific information related to USV segmentation targets.

In our implementation, the MiniCPM visual encoder is kept frozen during both training and inference to reduce computational cost and prevent catastrophic forgetting of pre-trained knowledge. The structured prompt template (see Section 3.2) is automatically constructed in JSON-like format with placeholders filled by scene recognition modules to enable reproducible and consistent input to MiniCPM.

The mask proposal generator processes the input image and outputs multiple binary masks representing candidate regions. The mask selection module, guided by the <Describe> token, selects the optimal combination of masks to form an initial segmentation result. Finally, the physical constraint-enhanced segmentation module further refines the result to ensure consistency with real-world physical structures. The boundary-aware module is implemented as a non-differentiable, modular post-processing stage applied only during inference and does not participate in backpropagation.

As shown in Figure 1, only the mask selection module contains trainable parameters. The Segment Anything Model (SAM), CLIP model, and MiniCPM visual encoder are all frozen during training. This parameter control strategy enables the entire system to run efficiently on a single RTX 3090 GPU. To facilitate downstream navigation decisions, we treat both land regions (e.g., shoreline, buildings) and water surface obstacles (e.g., floating debris, ships) as non-navigable areas. These regions are grouped into a unified "obstacle" category in the final segmentation mask to ensure the safe path planning of the USV system.

3.2. <Describe> Token Generation

A vision-language model-based scene understanding framework is employed to perform fine-grained reasoning on the geographic features and physical properties present in complex USV environments. Specifically, the system uses a visual encoder to extract high-level semantic features from images, and a text generation model to produce structured scene descriptions. These outputs are transformed into <Describe> Tokens, which serve as control inputs to guide the segmentation network.

USV navigation scene localization is determined based on the spatial distribution of land and water within the image. The model infers the relative distance between the USV and the shoreline by identifying nearshore objects (e.g., docks, vegetation), medium-range landmarks (e.g., lighthouses, buoys), and far-background features (e.g., horizon lines, mountain silhouettes). The scene is then categorized into one of three types: “close to shore,” “moderate distance,” or “far from shore.”

To support reproducibility and systematic generation of the <Describe> token, the prompt template is implemented as a structured JSON-like string with predefined keys for each semantic dimension. The placeholders in the template are automatically filled based on the scene analysis module’s predictions, then serialized and fed to MiniCPM for token embedding. Compared to conventional free-text prompts, this structured format ensures explicit encoding of multi-dimensional scene semantics, reduces ambiguity, and facilitates consistent interaction with the vision-language model.

3.3. Mask Selection

The mask selection process is based on a multi-class recognition algorithm that incorporates multi-layer IoP (Intersection over Prediction) heads. It comprises two key components: a Fusion Module and a 3-layer IoP Head. In the Fusion Module, each mask embedding interacts with a special segmentation control <Describe> token, through multiple layers of self-attention and cross-attention, resulting in a set of fused mask features. These features are then passed to the IoP Head, where each mask is assigned a prediction score across three target categories: sky, water, and obstacle. These scores are used to determine the final multi-class masks during inference.

3.3.1. Model Overview

As shown in Figure 2, the input to the module consists of candidate mask embeddings ${\left\{m_k\right\}}_{k=1}^K$, where each $m_k$ is a high-dimensional vector (e.g., 256 or 512 dimensions) representing the k-th candidate mask. The <Describe> token serves as a task-specific control token that steers the network toward segmentation-relevant information. The goal of the Fusion Module is to integrate contextual information between the candidate mask embeddings and the <Describe> token to produce a set of fused mask features, denoted as ${\left\{f_k\right\}}_{k=1}^K$.

Self-attention enables mutual information exchange among the mask embeddings, capturing complementary or competitive relations to avoid redundancy or conflicts. Cross-attention allows the <Describe> token to modulate each mask embedding with task-relevant semantic cues. If the model also contains learnable category tokens (e.g., "sky", "water", "obstacle"), the cross-attention mechanism guides these semantic flows to corresponding masks.

The attention outputs are passed through one or more multilayer perceptrons (MLPs) to generate unified feature vectors of fixed size (e.g., $f_k\in {\mathbb{R}}^{256}$). The final output of the Fusion Module is denoted as $F_a$, a matrix consisting of K fused feature vectors: see Equation (1).

$$F_a={\left[\begin{array}{cccc}{f}_1& f_2& \dots & f_K\end{array}\right]}^{\top }\in {\mathbb{R}}^{K\times d}$$

(1)

These fused features $f_k$ serve as general-purpose representations of the candidate masks. To perform multi-class, multi-instance segmentation (i.e., multiple masks per category), the model must estimate the purity or accuracy of each mask for each target class. This is done through the 3-layer IoP Head, which maps the $K\times d$ feature matrix to a $K\times 3$ prediction matrix.

First, all $f_k$ vectors are passed through a shared-parameter MLP to extract features suitable for IoP prediction. These are then split into three parallel branches, each corresponding to one of the target categories: Sky, Water, and Obstacle. Each branch outputs a scalar ${\widehat{p}}_k^c\in [0,1]$, where $c\in 1,2,3$ denotes the class index. In our implementation, the Obstacle category encompasses both land regions (shoreline, buildings) and waterborne obstacles (floating objects, vessels). This unified category design reflects their shared semantic role as non-navigable zones in USV navigation. The result is a $3\times K$ score matrix P, where:

$$P_{c,k}={\widehat{p}}_k^c$$

(2)

$P_{c,k}$ indicating the predicted IoP value of the k-th mask with respect to class c.

3.3.2. Training

During training, each image is annotated with pixel-level or mask-level labels across the three classes: sky, water, and obstacle. Once the network predicts IoP scores ${\widehat{p}}_k^c$, these are regressed against ground-truth IoP values $g_k^c$ using a weighted mean squared error (MSE) loss:

$${\mathcal{L}}_{\mathrm{IoP}}=\sum _{c=1}^3{\lambda }_c·{\displaystyle \frac{1}{K}}\sum _{k=1}^K\mathrm{MSE}\left({\widehat{p}}_k^c,g_k^c\right)$$

(3)

where ${\lambda }_c$ denotes the class-specific weight used to balance class imbalance (e.g., obstacle regions are typically underrepresented).

Through backpropagation, the network gradually learns to predict accurate IoP scores for each mask across all classes, enabling robust multi-category segmentation.

3.4. Physical Constraint Enhanced Segmentation

To improve the accuracy and real-world applicability of semantic segmentation for USV images, we propose a boundary-aware physical constraint framework. Unlike traditional CRF-based post-processing that applies general smoothness priors, our method explicitly models sky-water and land-water boundaries, adapts dynamically to the inferred scene type, and provides physically interpretable correction rules at both local and global levels.

3.4.1. Boundary Line Definition and Extraction

Let $M_{\mathrm{sky}},M_{\mathrm{water}},M_{\mathrm{land}}\in {\{0,1\}}^{H\times W}$ denote the predicted binary masks for sky, water, and land, respectively. Let $L_{\mathrm{sky}}$ and $L_{\mathrm{water}}$ denote the sky and water boundaries, respectively, which partition the image into three regions: sky, water, and land.

Sky boundary line $L_{\mathrm{sky}}$ is the lowest edge of each connected sky region which is extracted to form a boundary line. To estimate the lower edge of the sky region, we compute the bottom-most pixel of each sky-connected component along the horizontal axis. For each column x, the vertical position $y_{\mathrm{sky}}\left(x\right)$ corresponds to the largest y such that the pixel belongs to the sky mask $M_{\mathrm{sky}}$. The sky boundary line is thus defined as:

$$L_{\mathrm{sky}}=\left\{(x,y_{\mathrm{sky}}\left(x\right))|y_{\mathrm{sky}}\left(x\right)=max\{y\mid M_{\mathrm{sky}}(x,y)=1\}\right\}$$

(4)

The water boundary line $L_{\mathrm{water}}$ serves as an important physical constraint that separates the water surface from the sky or adjacent land regions. To delineate the upper edge of the water surface, we extract the topmost water pixels across the image. Specifically, for each column x, we identify the first pixel y from top to bottom such that $M_{\mathrm{water}}(x,y)=1$ and all pixels above are non-water. The set of these edge pixels is:

$$\begin{array}{cc}\hfill {\mathcal{E}}_{\mathrm{water}}& =\left\{(x,y)|M_{\mathrm{water}}(x,y)=1,\forall y^{\prime }<y,M_{\mathrm{water}}(x,y^{\prime })=0\right\}\hfill \end{array}$$

(5)

To ensure robustness against noise and local irregularities, we apply the RANSAC algorithm to fit a straight line to the points in ${\mathcal{E}}_{\mathrm{water}}$. The resulting line is modeled as:

$$\begin{array}{cc}\hfill L_{\mathrm{water}}^{\mathrm{fit}}\left(x\right)& =kx+b\hfill \end{array}$$

(6)

The fitted waterline is accepted if the majority of the candidate points satisfy the residual condition:

$${ϵ}_i=\left|y_i-(k{x}_i+b)\right|<\delta$$

(7)

where $\delta$ is a predefined tolerance threshold.

3.4.2. Scene-Aware Geometric Constraints on Water Boundary

Based on the relative distance between the USV and the shoreline—determined by the vision-language model through semantic classification (e.g., <Near Shore>, <Moderate Distance>, <Far from Shore>)—we impose different geometric constraints on the fitted water boundary line $L_{\mathrm{water}}^{\mathrm{fit}}$. This scene-aware modeling approach enables the segmentation framework to adapt to diverse spatial configurations, thereby improving accuracy and robustness in complex maritime environments.

In nearshore scenarios (Type C), where the USV is close to the shoreline, the visible shoreline may exhibit significant curvature due to landform irregularities and perspective distortion. To accommodate such cases, we apply a point-wise deviation constraint, which ensures that each detected water boundary point remains close to the fitted line within a small tolerance:

$$|y_i-L_{\mathrm{water}}^{\mathrm{fit}}\left(x_i\right)|<{\delta }_c$$

(8)

This constraint preserves local variations while maintaining overall boundary coherence.

For moderate-distance scenes (Type M), where the shoreline appears approximately linear, we employ a relaxed global constraint based on the average deviation of boundary points from the fitted line. This approach provides a balance between local tolerance and global shape consistency:

$${\displaystyle \frac{1}{N}}\sum _i|y_i-L_{\mathrm{water}}^{\mathrm{fit}}\left(x_i\right)|<{\delta }_m$$

(9)

Such a formulation is effective in scenes where small disturbances (e.g., ripples or uneven terrain) may affect boundary detection.

In farshore conditions (Type F), where the USV is located far from the landmass, the visible water horizon typically appears as a straight line. Under this assumption, we impose a strict linearity constraint by minimizing the root mean square error (RMSE) between the fitted line and the actual water boundary points:

$$\mathrm{RMSE}({\mathcal{E}}_{\mathrm{water}},L_{\mathrm{water}}^{\mathrm{fit}})\approx 0$$

(10)

This ensures that the segmentation output reflects the expected geometric simplicity of the distant horizon.

By dynamically adapting the geometric constraints according to the inferred scene type, our method enhances the reliability of water boundary estimation, particularly in visually ambiguous or geometrically challenging regions.

3.4.3. Boundary-Guided Mask Correction Rules

To further enhance segmentation reliability and eliminate physically implausible outputs, we introduce a set of boundary-guided post-processing rules that leverage the estimated sky and water boundaries. These rules correct potential inconsistencies arising from visual ambiguities or imperfect mask selection, ensuring that final segmentation results conform to real-world scene geometry.

Rule 1: Remove Non-Sky Labels Above the Sky Boundary

Due to over-segmentation or misclassification, land or water labels may occasionally appear above the estimated sky boundary. Such labels are physically invalid, as the sky region is expected to occupy the uppermost portion of the image. Therefore, for any pixel located above the sky boundary $y<y_{\mathrm{sky}}\left(x\right)$, if it is mistakenly labeled as land or water, its label is reassigned as sky:

$$M_{\mathrm{sky}}(x,y)\leftarrow 1,\mathrm{if}y<y_{\mathrm{sky}}\left(x\right)\wedge \left[M_{\mathrm{land}}(x,y)=1\vee M_{\mathrm{water}}(x,y)=1\right]$$

(11)

This rule eliminates upward label contamination and enforces vertical consistency in the spatial layout of sky-related regions.

Rule 2: Reject Water Labels Above the Water Boundary

Water regions are physically constrained to lie below the water surface boundary. However, due to reflection artifacts or texture confusion (e.g., with clouds), water may be incorrectly labeled above the horizon. To address this, we remove all water labels that are located significantly above the fitted water boundary line, with a tolerance margin $\delta$:

$$M_{\mathrm{water}}(x,y)\leftarrow 0,\mathrm{if}y<L_{\mathrm{water}}^{\mathrm{fit}}\left(x\right)-\delta$$

(12)

This constraint suppresses implausible high-position water predictions and helps disambiguate water-sky confusion, especially in bright scenes.

Rule 3: Reject Sky Labels Below the Water Boundary

Similarly, sky labels that appear below the water boundary are also considered physically incorrect, as they often arise from misclassified bright water patches or occluded reflections. Therefore, we enforce a rule that removes sky labels below the water boundary, allowing a small tolerance $\delta$ to accommodate fitting noise:

$$M_{\mathrm{sky}}(x,y)\leftarrow 0,\mathrm{if}y>L_{\mathrm{water}}^{\mathrm{fit}}\left(x\right)+\delta$$

(13)

This rule sharpens the vertical separation between sky and non-sky regions and improves robustness in horizon detection scenarios.

These boundary-aware correction rules collectively reinforce semantic consistency across vertical image structures and ensure that segmentation outcomes respect scene physics and horizon geometry. They are particularly effective in refining predictions near ambiguous regions such as horizon lines, cloud edges, or reflective water surfaces.

3.4.4. Global Scene Consistency via Area Proportions

While local boundary constraints help ensure pixel-level plausibility, a global understanding of scene composition is also crucial for achieving semantically coherent segmentation. To this end, we introduce a global consistency check based on the pixel-level area proportions of each semantic class (i.e., sky, water, and land). This check ensures that the overall segmentation result aligns with expected spatial distributions under different navigation scenarios.

Let the normalized area proportion of each semantic class be defined as:

$${\rho }_{\mathrm{cls}}={\displaystyle \frac{1}{HW}}\sum _{x=0}^{W-1}\sum _{y=0}^{H-1}\mathbb{1}[M_{\mathrm{cls}}(x,y)=1],\mathrm{cls}\in \{\mathrm{sky},\mathrm{water},\mathrm{land}\}$$

(14)

where H and W denote the height and width of the input image, respectively, and $\mathbb{1}[·]$ is the indicator function that counts whether a pixel belongs to the target class.

To evaluate scene-level plausibility, we predefine the expected class area distribution vector ${\overrightarrow{\rho }}^{\left(s\right)}$ for each scene type $s\in \{\mathrm{nearshore},\mathrm{moderate},\mathrm{farshore}\}$. For example, in farshore scenes, we expect a large portion of the image to be sky and water, with minimal land presence; conversely, nearshore scenes may contain more land pixels.

We then compute an L2-based consistency score between the observed class distribution ${\overrightarrow{\rho }}_{\mathrm{obs}}$ and the scene-specific expectation ${\overrightarrow{\rho }}^{\left(s\right)}$ as follows:

$${\mathrm{Score}}_{\mathrm{consistency}}=1-{\parallel {\overrightarrow{\rho }}_{\mathrm{obs}}-{\overrightarrow{\rho }}^{\left(s\right)}\parallel }_2$$

(15)

This score lies in the range $[0,1]$, where higher values indicate better alignment with the expected scene composition. A low score reveals global inconsistency in the segmentation, which may result from misclassification or poor mask fusion in ambiguous regions.

In such cases, the system can trigger fallback mechanisms, such as uncertainty-aware refinement modules, confidence thresholding, or secondary prompt-based verification using the vision-language model. This global validation step complements local correction rules and reinforces holistic scene understanding, particularly in visually complex or transitional areas.

3.4.5. Integration and Deployment

This framework is implemented as a modular post-processing component that operates on the output of existing segmentation networks. Specifically, it takes as input the raw segmentation masks produced by models such as Segment Anything Model (SAM), DeepLabV3+, or SegFormer, and applies physically-informed corrections based on geometric boundaries and global scene consistency.

4. Experiments

4.1. Dataset Preparation and Training Details

To comprehensively evaluate the effectiveness and robustness of the proposed method in supporting multi-class semantic segmentation for Unmanned Surface Vehicles (USVs) under complex water environments, we constructed a diverse image dataset captured using real-world USV platforms. The dataset covers a wide range of aquatic scenarios, including far-from-shore scenes, medium-distance scenes, and near-shore complex scenes.

In far-from-shore scenarios, the images primarily consist of expansive water surfaces with clearly defined boundaries between sky and water. Medium-distance scenarios contain a balanced proportion of water and land with moderately complex backgrounds. Near-shore complex scenarios feature intricate shoreline contours with various interfering elements such as vegetation, buildings, and reflections. Each image is annotated at the pixel level, dividing pixels into three categories: sky, water, and non-navigable regions (including land and waterborne obstacles). A total of approximately 30,000 images were collected, with the distribution of scene categories summarized in

Table 1. Scene Category Statistics.

Scene Category	Description	Number of Images	Proportion
Far-from-shore	Open waters, clear sky-water boundary	9000	30%
Medium-distance	Balanced water-land ratio, moderate background complexity	10,500	35%
Near-shore complex	Irregular shoreline, multiple interfering elements	10,500	35%

.

To facilitate model training and evaluation, the dataset is split into training, validation, and testing sets, with approximately 20,000, 5000, and 5000 images respectively. Each scene category is proportionally represented across the three subsets. For example, the 9000 images in the “far-from-shore” scene category are divided according to the specified ratio into 6000 training images, 1500 validation images, and 1500 test images. The other scene categories are similarly partitioned following the same ratio.

The proposed USV-Seg framework is implemented using PyTorch v2.6.0 and trained on a single NVIDIA RTX 3090 GPU. We use a batch size of 16, an initial learning rate of $1\times {10}^{-4}$ with a cosine annealing schedule, and the AdamW optimizer with weight decay set to $1\times {10}^{-4}$. The model is trained for 100 epochs.

4.2. Describe Token Generation

We conducted experiments to evaluate the generation of Describe Tokens using the proposed vision-language model across different scene types. The model processes input images through a visual encoder to extract high-level semantic features, which are then used by a text generation module to construct structured scene descriptions. Each Describe Token contains three semantic dimensions: USV scene localization, landform complexity analysis, and water surface texture evaluation.

For each dimension, human-annotated ground truth labels were created to serve as references for accuracy evaluation. The classification accuracy of the generated Describe Tokens under each scene type is shown in

Table 2. Describe Token Classification Accuracy(%) Across Scene Types.

Scene Type	Scene Localization	Landform Classification	Water Texture
Far-from-shore	99.25	99.10	98.96
Medium-distance	99.57	99.24	99.12
Near-shore complex	98.76	98.20	98.05

.

As shown in Table 2, the proposed Describe Token generation method based on the vision-language model demonstrates stable and high classification accuracy across all three semantic dimensions, validating its effectiveness and robustness under various complex water scenes.

4.3. Quantitative Evaluation

To evaluate the performance of the proposed segmentation method under diverse maritime conditions, we conducted experiments across three scene types categorized by the distance between the USV and the shoreline. These include simple far-from-shore environments, moderately complex medium-distance scenes, and near-shore cases characterized by irregular shorelines and reflection-induced interference. We compare our method with six state-of-the-art segmentation models: GRES [48], OV-Seg [49], X-Decoder [50], SEEM [51], SC-CLIP [52], and LISA [53].

Table 3. mIoU Results (%) of Different Algorithms on USV Water Segmentation Tasks.

Method	Far-from-Shore	Medium-Distance	Near-Shore Complex
GRES [48]	93.75	92.39	87.18
OV-Seg [49]	94.95	90.19	87.17
X-Decoder [50]	94.69	91.64	87.92
SEEM [51]	94.14	93.64	85.30
SC-CLIP [52]	93.73	90.94	86.26
LISA [53]	94.88	92.16	85.18
USV-Seg	98.17	97.83	96.30

presents the mean Intersection over Union (mIoU) results across all scene types.

As shown in Table 3, our method consistently outperforms the baselines across all scene types. In particular, it achieves outstanding performance in complex near-shore scenarios where elevated landforms and strong reflections often challenge conventional methods. The proposed method USV-Seg maintains mIoU scores above 95% even in the most difficult scenes, demonstrating its robustness and superior accuracy.

4.4. Ablation Study

We conducted ablation studies to assess the contribution of each component in the proposed USV-Seg framework. In particular, we examined the impact of removing the Describe Token, the boundary-aware correction module, and the three individual semantic dimensions of the Describe Token.

Table 4. Ablation Study on Near-shore Complex Scenes (mIoU, %).

Configuration	mIoU
Full USV-Seg	96.30
w/o Describe Token	92.15
w/o Boundary-aware Correction	91.78
Describe Token: only Scene Localization	93.21
Describe Token: only Landform Complexity	93.05
Describe Token: only Water Texture	92.89

summarizes the results on the near-shore complex scene subset.

As shown in Table 4, removing the Describe Token or the boundary-aware correction results in a notable mIoU drop of over 4%, confirming that both modules are critical and complementary. The Describe Token provides semantic priors to resolve ambiguities in reflective and irregular scenes, while the boundary-aware correction enforces geometric consistency to suppress leakage and over-segmentation.

We also evaluated each semantic dimension of the Describe Token in isolation. Scene localization contributes the most (93.21%), as it provides coarse positional awareness. Landform complexity (93.05%) helps distinguish irregular shorelines, and water texture (92.89%) aids in handling reflective or cluttered surfaces. None of the dimensions alone match the full token, confirming the advantage of integrating all three.

These results highlight the necessity of combining high-level semantic priors with low-level geometric constraints, as well as the complementary nature of the three semantic dimensions, for robust segmentation in challenging maritime environments.

4.5. Qualitative Evaluation

Figure 3 presents a visual comparison of segmentation results under different scene types, including far-from-shore, medium-distance, and near-shore complex conditions. The figure includes the original image, results from SEEM [51], SC-CLIP [52], LISA [53], the proposed USV-Seg, and ground truth annotations.

As visually evident, most methods perform reasonably well under simple and moderately complex scenes. However, our method delivers superior boundary precision and mask quality, producing tighter adherence to object contours. In more challenging scenes with strong reflections and intricate shoreline structures, our approach remains highly robust and accurately distinguishes between navigable and non-navigable regions. In contrast, other methods tend to exhibit segmentation gaps or boundary misclassification.

5. Conclusions

This paper presents a unified semantic segmentation framework for Unmanned Surface Vehicles (USVs) navigating in complex aquatic environments. To address key challenges such as segmentation inaccuracy, boundary ambiguity, and lack of physical consistency, we integrate a vision-language model with a physically constrained refinement strategy. A core contribution of this work is the design of a structured Describe Token mechanism, which leverages prompt-guided reasoning to infer scene semantics—such as shoreline distance, landform complexity, and water surface texture—without requiring large-scale annotations. Experimental results demonstrate high accuracy in Describe Token classification across diverse water scenarios, confirming the model’s capability for semantic scene understanding.

In addition, a boundary-aware correction module is introduced to refine segmentation results using sky-water and land-water constraints, enhancing geometric plausibility. Extensive evaluations show that the proposed framework significantly outperforms state-of-the-art methods, achieving up to 96.30% mIoU in near-shore complex scenes. Ablation studies further confirm the effectiveness and complementary nature of the key modules.

We note that the proposed method may exhibit reduced performance in highly challenging scenarios, such as under complex illumination, very close proximity to irregular shorelines, severe reflections, large waves, camera tilt, or nighttime conditions—where even human observers struggle to delineate clear boundaries. Addressing these limitations will be part of our future work, through the exploration of temporal modeling, multi-sensor fusion, uncertainty-aware reasoning, and robust adaptation to embedded hardware platforms.

Overall, our approach offers a robust, interpretable, and extensible solution for maritime segmentation tasks, with promising potential for real-world USV navigation and monitoring applications.