Geo-RVF: A Multi-Task Lightweight Perception System Based on Radar–Vision Fusion for USVs

Abstract

Visual perception in Unmanned Surface Vehicles (USVs) suffers from drastic lighting changes and missing texture features. These factors lead to depth scale drift and motion estimation bias. Moreover, existing multi-modal fusion models are computationally complex and unfit for resource-limited edge devices. To address these problems, a lightweight Radar–Vision Fusion (Geo-RVF) algorithm is proposed. To supplement spatial information, point clouds are projected to build sparse depth maps. A probability consistency-based depth correction module is designed to suppress water noise. This helps extract accurate geometric anchors to guide visual depth propagation. Subsequently, a Recurrent Autoregressive Network (RAN) fuses radar and image features in the temporal dimension. This resolves dynamic positional deviations caused by texture degradation and distant small targets. After real-time optimization, Geo-RVF achieves 23 FPS on the Jetson Orin NX. On a collected dataset, the method attains a mean average precision (mAP) 50–90 of 44.2% and a mean intersection over union (mIoU) of 99%, outperforming HybridNets and Achelous.

1. Introduction

Unmanned Surface Vehicles (USVs) play an increasingly important role in marine research and resource management [1]. For USV autonomous navigation, environmental perception is the prerequisite for obstacle avoidance and path planning. Visual perception is currently the primary method due to its high resolution, rich semantics, and low cost. To facilitate 3D motion control, 2D images are typically converted into a Bird’s-Eye View (BEV). BEV eliminates perspective effects and provides a unified spatial scale. However, constructing the BEV space relies heavily on precise depth estimation, which remains a major challenge for visual sensors in complex water environments.

Although images contain rich semantics, they lack absolute physical scale constraints when mapped to BEV. This leads to severe depth scale problems. In dynamic water scenes (characterized by strong reflections, waves, and textureless areas), depth estimation models often degrade, causing significant scale drift. Furthermore, pure vision has limitations in dynamic perception. Inferring speed from multi-frame analysis is computationally slow and easily affected by hull vibration. Due to missing depth information and uncertain motion estimation, pure vision cannot meet USV safety requirements. Therefore, sensors with physical ranging capabilities must be introduced.

Sensor fusion must balance performance, hardware cost, and computational efficiency. LiDAR provides dense point clouds but is too expensive and power-hungry for small USVs. In contrast, millimeter-wave radar works effectively in all weather conditions, penetrating rain and fog [2]. Although its point clouds are sparse, it provides Doppler velocity, which complements high-resolution visual semantics. Therefore, combining cameras with radar is the most practical choice for USVs [3].

Deep learning has significantly advanced multi-modal fusion, particularly in the automotive autonomous driving sector. However, transferring these mainstream solutions directly to USVs presents a critical bottleneck. Although methods like Centerfusion [4] attempt to improve 3D perception by associating radar with visual center points, they still struggle to construct a unified and dense representation. Conversely, state-of-the-art Transformer-based detectors [5] and BEV frameworks [6] offer comprehensive fusion but rely on complex feature interactions. While highly accurate, they demand massive memory and computational resources. Such hardware is readily available in automobiles but is strictly prohibited on small USVs, which operate under severe payload and battery constraints using low-power edge computing platforms. Furthermore, although pure 2D visual detectors operate extremely fast, they lack functions such as depth estimation, radar fusion, and 3D BEV transformation, and therefore cannot meet the requirements for safe unmanned surface vessel navigation. Consequently, there is an urgent, unmet need for a perception system that can achieve the robustness of 3D BEV fusion while strictly adhering to the computational limitations of USV edge devices. To address this gap, we propose Geo-RVF, a lightweight perception system for edge devices. By compressing a complete multi-modal 3D BEV pipeline into just 10.3 GFLOPs, our approach bridges the gap between functionally limited pure vision models and computationally prohibitive multi-modal frameworks, achieving a truly lightweight status for edge devices. Our algorithm uses radar point clouds as reliable geometric priors and dynamically corrects visual depth and velocity deviations, ensuring 3D precision with reduced computational overhead.

To implement this, Geo-RVF establishes a pipeline from depth inference to state correction. The key is fusing camera and radar data to generate spatially accurate BEV features. We designed a depth correction module based on probability consistency. This maps radar points as sparse “geometric anchors” to guide visual depth scaling. Subsequently, a Recurrent Autoregressive Network (RAN) fuses radar Doppler velocity with visual features. This addresses position deviations caused by distant small targets or texture degradation. Finally, the system is optimized for the Jetson Orin NX, achieving high-precision real-time perception.

The contributions of this study are as follows:

• A lightweight radar-vision fusion architecture for edge devices. Based on Lift, Splat, Shoot (LSS) [7] and YOLO-AFM (YOLO with Adaptive Feature Module) [8], we incorporate radar point clouds. By combining early geometric priors with late constraints, we enhance perception robustness in complex lighting environments.
• A depth scale correction mechanism based on radar geometric anchors. This overcomes the scale drift of monocular depth estimation. It uses radar as a physical constraint, combining adaptive matching and spatial weight transfer for accurate BEV mapping.
• A motion modeling and tracking method. We introduce a RAN to model temporal states and use an Extended Kalman Filter (EKF) to fuse radar information. This strategy effectively alleviates the lag and jitter found in pure visual speed estimation.

2. Related Work

2.1. Radar–Camera Fusion for Object Detection

Due to the complementary nature of radar and cameras, fusing these sensors is a mainstream solution for 3D object detection. Existing methods generally fall into two categories: Region of Interest (ROI)-based fusion and end-to-end fusion.

ROI-based fusion typically generates proposals from one modality and refines them with features from the other. Early methods projected radar points onto the image plane to generate candidate regions [9]. However, projection discards the depth dimension, leading to a loss of 3D spatial information [10]. To address radar sparsity, recent studies generate proposals from images first. Centerfusion [4] is a representative method. It uses pillar expansion to address the lack of radar height information and associates radar detections with image frustums to regress velocity and depth. Similarly, CRAFT [11] proposes a soft association mechanism in the polar coordinate system. It uses consecutive cross-attention layers to adaptively exchange spatial context, handling spatial misalignment between sensors.

End-to-end methods integrate features from both modalities into a unified framework. This enables comprehensive feature interaction. Methods based on 3D proposal prediction incorporate a radar processing branch into the detection framework. Li et al. [12] addressed hull shaking deviations using motion-compensated projection and distance association modules, achieving 50.1% AP. FMA-FCOS [13] improves detection performance in adverse weather and occlusion via radar feature enhancement, gated fusion, and adaptive feature pyramids. Additionally, Hwang et al. [14] focused on robust detection of distant small objects in adverse weather for autonomous driving. They proposed spatial partition projection and weighted fusion modules. With the rise of BEV perception, transforming image features into a unified BEV space for fusion has become a mainstream approach. RCM-Fusion [15] is built on the BEVFormer [16] architecture and uses radar positions to construct optimized queries. Xu et al. [17] proposed RCFusionNet with a radar-guided two-stage fusion strategy. It enhances robustness by interacting with radar data during BEV feature generation and fusion.

2.2. Depth Estimation

2.2.1. Monocular Depth Estimation

In recent years, monocular depth estimation has widely adopted deep learning. Mainstream works formulate the problem as pixel depth regression [18,19] or depth distribution classification [20]. Early works treated depth estimation as a regression problem, using LiDAR depth labels as supervision for training on large-scale datasets. Eigen et al. [21] proposed a multi-scale CNN that first regresses a coarse depth map and then refines it. Subsequently, fully convolutional networks (FCNs) and U-Net became standard architectures [22,23]. They use skip connections to recover spatial resolution lost during downsampling. Although accurate, these supervised methods face severe generalization issues [24]. Models trained on specific datasets (e.g., NYUv2, KITTI) struggle to estimate depth in unseen scenes.

To reduce reliance on ground truth data, self-supervised learning has become a major research direction. These methods treat depth estimation as an image synthesis problem, optimizing via reprojection loss [25]. Methods proposed by Garg et al. [26] and Godard et al. [27] use rectified stereo supervision. They estimate disparity via left-right consistency loss and convert it to depth. While effective, stereo-based training requires synchronized stereo cameras, limiting data availability. Zhou et al. [25] proposed a framework to learn depth and ego-motion from monocular video sequences using a pose network. Monodepth2 [28] significantly improved baselines by introducing minimum reprojection loss for occlusion and masking strategies for moving objects. However, monocular video methods suffer from scale ambiguity and sensitivity to dynamic objects [29].

Although these methods achieve high performance on public benchmarks, generalizing to diverse environments remains a key challenge. Consequently, generative adversarial networks (GANs) are increasingly used to align synthesized and real feature distributions [30]. Recent research also explores online learning and meta-learning to adapt pre-trained models to new scenes during inference [31], addressing generalization issues in real-world applications.

2.2.2. Point Cloud–Vision Depth Estimation

Monocular depth estimation excels at recovering scene structure but lacks metric scale. To recover precise metric depth, integrating active range sensors (e.g., LiDAR or radar) is promising. Existing methods generally fall into two categories: dense LiDAR-based depth completion and sparse radar–camera fusion.

Mainstream depth completion projects LiDAR points onto the image plane to create sparse depth maps, which are then completed by encoder–decoder networks [32]. While feasible, this discards inherent 3D geometric information. To preserve 3D geometry, recent studies extract features directly from 3D space before fusion. For instance, point-based branches [33] or GNN [34] propagate features in 3D space. However, these methods rely on high-quality, dense LiDAR data and struggle with extremely sparse radar data.

Compared to LiDAR, millimeter-wave radar is cheaper and more robust in harsh environments. Early methods, such as that proposed by Lin et al. [35], directly projected radar points, resulting in sparse and blurry images. To address this, techniques like DORN [36] expand radar points into pillars or use ordinal regression to model depth distribution. Since radar echoes contain noise, simple fusion often leads to artifacts. Recent research focuses on learning reliable associations between radar points and image pixels. Long et al. [37] use Doppler velocity and optical flow to filter dynamic outliers. Instead of hard projection, Singh et al. [38] propose confidence scores for radar–pixel correspondence to generate semi-dense depth maps, followed by a gated fusion network.

Architectures combining multi-modal features have also evolved. Traditional methods simply concatenate radar and image features [39]. However, directly fusing heterogeneous data can confuse the learning process, causing aliasing artifacts and blurred boundaries [40]. An emerging trend separates shape prediction from scale recovery. Methods like RadarCam-Depth [41] use image branches to predict high-fidelity, scale-free depth maps and use radar data solely to learn metric scales. Similarly, GET-UP [42] introduces a point cloud upsampling module [43] to densify radar points based on visual features, enhancing geometric consistency. These methods demonstrate that preserving the unique characteristics of each modality improves performance. However, none of the above methods account for radar detection errors, leading to suboptimal depth estimation accuracy.

3. Methods

As illustrated in Figure 1, the proposed Geo-RVF framework operates as an end-to-end Multi-Task Learning (MTL) system, taking images, radar point clouds, and calibration parameters as inputs. First, the camera image is fed into a shared visual backbone (an improved CSPDarknet) to extract multi-scale hierarchical features. To minimize computational redundancy, these shared features then branch into three parallel task heads: (1) a 2D Object Detection head, (2) a Drivable Area Segmentation head, and (3) a Depth Estimation head.

Figure 1. Overall framework of the perception algorithm. The system takes images and radar point clouds as inputs. A backbone network generates 2D detection, segmentation, and depth features in parallel. Radar point clouds are used as geometric anchors to probabilistically correct visual depth. The final output is a Bird’-Eye View (BEV) image.

Simultaneously, sparse radar point clouds are processed and clustered. During the depth inference stage, target-associated radar point clouds serve as spatial geometric anchors to constrain and correct the visual depth probabilities. Subsequently, the fused multi-modal features are transformed into the unified BEV space using the LSS paradigm. Finally, a RAN module is constructed to jointly model target appearance and temporal motion states, and an Extended Kalman Filter (EKF) is employed to fuse radar radial Doppler velocity with visual displacement vectors.

3.1. Geometrically Consistent Depth Estimation

3.1.1. Visual Discrete Depth Estimation

Based on semantic features from the multi-scale fusion network, the depth branch recovers dense depth information on the image plane. Since the fused features integrate radar spatial hints and camera semantics, the depth network can more sharply capture distance differences between target contours and the background. In the depth estimation branch, fused features are fed into a depth network encoder. This module extracts distance-sensitive features through convolutional layers, reduces channel redundancy, and outputs a single-channel depth map via a 1 × 1 convolutional head. A Sigmoid function then constrains the output to a normalized interval, mapping it to the depth range of interest $[D_{max},D_{min}]$:

$$D_{pred}=\sigma (f_{depth}(I_{fused}))\cdot (D_{max}-D_{min})+D_{min}$$

(1)

Although this method incorporates radar features, relying solely on convolutional networks remains a visual depth estimation task. It is prone to scale drift in open waters. Therefore, this output serves only as an initial estimate. It requires correction in the next stage using radar physical measurements as geometric anchors.

3.1.2. Visual Depth Estimation Correction

To recover precise distance scale, inspired by LSS probabilistic concepts, we designed a depth correction mechanism based on radar geometric anchors, as illustrated in Figure 2. It does not forcibly overwrite visual predictions. Instead, it optimizes depth estimation by matching probability distributions. For successfully associated radar targets, the radar measurement $d_{radar}$ is not perfectly accurate. Measurements are affected by hull swaying and target pose changes. Therefore, the system models it as a Gaussian probability distribution along the optical ray:

$$P_{radar}(d)=\mathcal{N}(d_{radar},{\sigma }_{radar}^2),$$

(2)

where the variance ${\sigma }_{radar}^2$ is adaptively adjusted based on target distance and association confidence. This constructs a sparse but reliable depth probability distribution.

Figure 2. Illustration of Radar Depth Probabilistic Modeling and Correction.

In water scenes, the actual scale of objects is primarily manifested through the waterline contact. To effectively propagate radar scale information to unmatched visual depths, the algorithm focuses on the contact region of each target. During training, the depth value $D_{pred}$ regressed by the visual network is also converted into a probability distribution $D_{pred}$. We minimize cross-entropy to align the visual depth distribution with the radar distribution:

$$L_{scale}=-{\displaystyle \sum _{k=1}^K{P}_{radar}(d)\log (P_{vis}(d))}$$

(3)

By minimizing this loss, radar physical scale information is incorporated as a soft constraint into the depth regression network. Leveraging the LSS paradigm, the network preserves detailed features when visual textures are clear. When scale drift occurs, radar anchors pull the depth prediction back to the correct range. This achieves a geometrically unified BEV construction.

3.1.3. Multi-Task Joint Loss Optimization

To avoid any single task dominating the entire learning process, we dynamically balance the multi-task loss by adopting homoscedastic task uncertainty [44]. The overall loss function $L_{total}$ is redefined as:

$$L_{total}={\displaystyle \sum _{t\in det,seg,depth,scale}\left(\frac{1}{2\exp (s_t)}{L}_t+\frac{1}{2}{s}_t\right)},$$

(4)

where $L_t$ represents the loss value for each specific task (detection, segmentation, initial visual depth, and radar-guided scale correction). For each task, we introduce $s_t$, which is a learnable logarithmic variance parameter. To ensure stability during the early training stage, all $s_t$ are initialized to 0.

This formulation ensures that no single task dominates the entire network. The dynamic weights for each task are actually $1/(2\exp (s_t))$. During backpropagation, if the loss value $L_t$ of a specific task is significantly large, the network will adaptively learn a larger variance $s_t$ to minimize the overall objective. This exponentially increasing denominator significantly reduces the effective weight of that task, thereby preventing its gradient from disrupting the shared multi-scale features. Meanwhile, the additive term 1/2 $s_t$ serves as a regularization penalty to prevent the network from simply minimizing the loss by infinitely increasing $s_t$. As we confirmed by our training metrics, this self-regulating mechanism can standardize the gradient scale and enable all parallel tasks to reach consensus harmoniously.

3.2. Radar–Visual Fusion Object Tracking

3.2.1. Object Detection

Radar and LiDAR share similar data forms but differ significantly in resolution and precision. Therefore, directly transferring state-of-the-art LiDAR architectures is difficult. We adopt a multi-stage hybrid fusion strategy to handle radar sparsity and noise effectively. To overcome sparsity and the lack of semantics, the detection branch uses early fusion. Radar points projected onto the image plane are treated as key locations $(u,v)$ to generate a heatmap $H(x,y)$. Each projected point is modeled as a pulse signal and convolved with a 2D Gaussian kernel to create a continuous density map:

$$H(x,y)={\displaystyle \sum _{i=1}^N\exp }\left(-{\displaystyle \frac{{(x-u_i)}^2+{(y-v_i)}^2}{2{\sigma }_i^2}}\right),$$

(5)

where ${\sigma }_{i_i}=\alpha \cdot d_{i_i}+\beta$ is an adaptive standard deviation linearly related to the radar measurement distance $d_i$ (where $\alpha$ and $\beta$ are hyperparameter). The heatmap is concatenated with the RGB image to form a four-channel tensor $I_{fused}\in {ℝ}^{H\times W\times 4}$. This design handles visual interference such as sun glint and wave foam. By introducing the radar channel, the network perceives physical echoes during feature extraction, effectively suppressing background noise.

The backbone uses an improved CSPDarknet as implemented in YOLOv11 to extract multi-level features. To fuse radar geometry with camera semantics, we designed a multi-scale feature fusion network (shown in Figure 1). This module includes an Adaptive Feature Module (AFM). The AFM first extracts local features via 3 × 3 convolution. It then splits into two paths: one remains unchanged, while the other generates channel attention weights via global average pooling and 1 × 1 convolution. The channels are then weighted and fused. This dynamically suppresses water clutter and enhances target regions containing radar echoes. The fused features enter the detection head, which outputs bounding boxes, class confidence, and appearance vectors.

Since detection results lack velocity, the system performs cross-modal data association within a single frame before tracking. Point cloud clusters are projected onto the image plane. We construct multi-dimensional vectors based on position deviation, size ratio, and motion state. A dynamically weighted similarity function calculates matching costs. Through this process, radar Doppler velocity is assigned to visual targets, providing complete input for subsequent temporal modeling.

3.2.2. Object Tracking

After detection, the system must solve cross-frame data association. We introduce a RAN to model target appearance and motion temporally, as shown in Figure 3. Using hidden states containing historical information, the system matches new detections with existing trajectories.

Figure 3. RAN Framework.

The model receives input vectors $x_{t:t-k}(x_t\in {ℝ}^N)$ from time step 1 to k. It estimates the conditional probability distribution $p(x_t\mid x_{1:t-1})$ of the current input $x_t$. This process can be expressed as the weighted sum of historical inputs plus Gaussian noise:

$$x_t={\displaystyle \sum _{k=1}^K{\alpha }_t^k}{x}_{t-k}+{\epsilon }_t,$$

(6)

where ${\epsilon }_t$ is Gaussian noise. To determine the importance of different historical moments, weights ${\alpha }_t$ are dynamically generated via an attention mechanism:

$${\alpha }_t^k={\displaystyle \frac{\exp (q_t\cdot k_{t-k})}{{\displaystyle \sum _{k^{\prime }=1}^K\exp }(q_t\cdot k_{t-k^{\prime }})}}$$

(7)

We construct an attention layer using the current query $q_t$ and historical keys $k_{t-k}$. Assuming the noise term ${\epsilon }_t^i$ follows a normal distribution with zero mean and standard deviation:

$$p(x_t\mid x_{1:t-1})=\mathcal{N}(x_t\mid {\mu }_t,{\Sigma }_t)$$

(8)

Therefore, the predicted mean and variance are:

$${\mu }_t={\displaystyle \sum _{k=1}^K{\alpha }_t^k}{x}_{t-k},\hspace{1em}{\Sigma }_t=\mathrm{diag}({\sigma }_t^2)$$

(9)

In the RAN model, weights ${\alpha }_t$ and standard deviation ${\sigma }_t$ change over time. This allows the model to adjust its estimation strategy as input features change.

To capture long-term trends and estimate ${\alpha }_t$ and ${\sigma }_t$, a gated recurrent unit (GRU) is embedded. It maintains a hidden state $h\in {ℝ}^d$ as an internal parameter. At each step $t$, the GRU updates based on input $x_t$ and the previous state $h^{t-1}$. Based on the current hidden state, the parameters of the autoregressive model are estimated via a mapping function $f(\cdot )$:

$$({\widehat{\alpha }}_t,{\widehat{\sigma }}_t)=f(h_{t-1}),$$

(10)

where $f(\cdot )$ is a fully connected layer containing specific element-wise transformations:

$${\alpha }_t^k={\displaystyle \frac{\exp ({\widehat{\alpha }}_t^k)}{{\displaystyle \sum _{k^{\prime }=1}^K\exp }({\widehat{\alpha }}_t^{k^{\prime }})}},\hspace{1em}{\sigma }_t=\ln (1+\exp ({\widehat{\sigma }}_t))$$

(11)

The Softmax function ensures weights ${\alpha }_k^t$ sum to 1, while Softplus ensures the standard deviation ${\sigma }_t$ is greater than 0. This guarantees accurate estimation of the conditional probability distribution $p(x_t\mid x_{1:t-1})$.

Given a set of detections, the system must determine the relationship between each new object and existing trajectories. Let $i$ be the trajectory index and $j$ be the current detection. We extract appearance features ${\left\{b_i^j\right\}}_{i=1}^k$ and dynamic motion features ${\left\{{\varphi }_i^j\right\}}_{i=1}^k$ from the bounding box. For trajectory $i$, let the historical boxes and appearance features be ${\varphi }_{1:k-1}^i$ and $b_{1:k-1}^i$. The association score $s_{i,l}$ between trajectory $i$ and object $j$ is defined as the joint conditional probability:

$$s_{i,j}=p({\varphi }_i^k\mid {\varphi }_{1:k-1}^i)\cdot p({\mathrm{b}}_i^k\mid {\mathrm{b}}_{1:k-1}^i)$$

(12)

Both appearance and bounding box features are modeled using RAN, assuming they are independent. If the score $s_{i,j}$ exceeds a threshold, object $j$ is associated with the highest-scoring trajectory $i$. Otherwise, object $j$ is not associated and is marked as lost.

Since appearance and motion differ significantly, we use two sets of hidden states $h_{\varphi }^t,h_b^t$ and external parameters ${\epsilon }_{\varphi }^t,{\epsilon }_b^t$. During tracking, when a new detection is associated, both memory types update. The probabilities ${\alpha }_{\varphi }^t,{\alpha }_b^t$ and ${\sigma }_{\varphi }^t,{\sigma }_b^t$ are derived independently and then combined. When association fails, only the motion prediction updates, while appearance parameters remain unchanged.

3.3. BEV Mapping and Fusion Velocity Update

3.3.1. Compensation-Based BEV Mapping

After obtaining the corrected depth distribution $D_{refind}\in {ℝ}^{D\times H\times W}$ and semantic features $F_{pv}\in {ℝ}^{C\times H\times W}$, the core task is to transform perspective features into a vessel-centric BEV, as shown in Figure 4. This facilitates subsequent motion planning. However, USVs face 6-DOF motion challenges on water. Specifically, roll and pitch cause severe geometric distortion in projection methods that assume flat ground. To address this, we designed an LSS pipeline that includes attitude compensation.

Figure 4. Schematic diagram of BEV transformation.

First, pixels are lifted into frustum coordinates $F_{fr}\in {ℝ}^{N\times C\times D\times H\times W}$ by calculating the outer product of semantic features and depth distribution. Subsequently, to eliminate the effects of hull vibration, the system introduces real-time attitude correction during back-projection. Using real-time inertial measurement unit (IMU) attitude data, we construct a rotation matrix $R_{imu}$. Combining this with camera intrinsics $K$ and extrinsics $T_{ex}$, the frustum features $F_{fr}$ are mapped to the BEV plane:

$$P_{BEV}=R_{imu}(\varphi ,\theta ,\psi )\cdot T_{ex}\cdot K^{-1}\cdot F_{fr},$$

(13)

where $\varphi ,\theta$ and $\psi$ represent roll, pitch, and yaw angles, respectively. This explicitly cancels camera optical axis shifts caused by waves, ensuring the spatial stability of the BEV feature map.

3.3.2. Radar-Guided EKF State Update

In water scenes, velocity estimation relies directly on positional accuracy in the BEV space. However, due to scale drift in monocular depth and BEV projection, velocity estimation suffers from jitter and lag. In contrast, radar provides instantaneous radial Doppler velocity but lacks tangential information. Therefore, an EKF with a flexible observation model is proposed. The state vector of the i-th target at time k is defined as ${\phi }_k^i={[x_k^i,y_k^i,v_k^i,{\theta }_k^i]}^T$. Here, denote the position in the BEV plane, resultant velocity, and heading angle, respectively. Assuming the target follows a constant velocity model over a short period, the equation is:

$$\left\{\begin{array}{c}{\widehat{x}}_{k|k-1}=F_k{\widehat{x}}_{k-1}+w_k\\ z_k=h({\widehat{x}}_k)+{ϵ}_k\end{array}\right.,$$

(14)

where $F_k$ is the state transition matrix, and $w_k$ and ${ϵ}_k$ represent process noise. To handle missed detections in vision or radar, the observation model and function are dynamically switched based on the radar association status.

When a target is captured only by vision, the observation vector contains only position information $z_k={\left[x_k^{vis},y_k^{vis}\right]}^T$. In this case, $h(\cdot )$ is linear, and the filter degenerates into a standard Kalman Filter. It uses historical trajectories to smooth visual position jitter. If radar successfully captures the target, the observation vector extends to $z_k={\left[x_k^{vis},y_k^{vis},v_k^{radar}\right]}^T$. Since radial velocity is a non-linear function of resultant velocity and relative angle in the state vector:

$$h(X)=\left[\begin{array}{c}x\\ y\\ v\cdot \cos (\varphi -\arctan ({\displaystyle \frac{x}{y}})\end{array}\right]$$

(15)

At this point, the Jacobian matrix is calculated to complete the EKF update. Using radar physical measurements to directly constrain the state space eliminates cumulative errors from visual differentiation. This allows velocity estimation to converge rapidly within a few frames. Combined with target association from the RAN module, the system achieves robust state tracking of dynamic water targets.

4. Experiments and Analysis

4.1. Experimental Settings

4.1.1. Datasets

The training phase primarily utilizes the WaterScenes [45] dataset. Unlike other marine datasets that provide only image data, WaterScenes synchronously collects and annotates visual and radar information. This supports systematic research on radar–vision fusion algorithms.

Furthermore, to improve algorithm robustness in complex water environments and address data scarcity in extreme conditions, a dynamic water surface simulation environment was built using Gazebo within the ROS framework (Figure 5). A simulated dataset was collected and annotated in this environment. It contains 62 object instances covering 11 typical water surface classes, including sailboats, yachts, and buoys.

Figure 5. Simulation Environment: (a) Simulated water-surface environment; (b) RGB image; (c) USV model; (d) radar point cloud.

The platform supports dynamic adjustments of environmental factors like light intensity and wave height. This reproduces various maritime navigation scenes. Combining simulated and real data ensures diversity and improves model generalization in complex environments.

The WaterScenes dataset serves as the primary source for model training and the quantitative evaluation of perception performance (including object detection, semantic segmentation, and depth estimation). The Gazebo-based simulated dataset is utilized to supplement complex multi-target port scenarios, thereby enhancing the model’s robustness during the training process. Finally, data collected from the real USVs platform is exclusively applied for qualitative visual verification and real-time deployment assessment in real-world environments.

Specifically, the dataset used in our experiments contains a total of 58,540 synchronized image and radar frames. The original image resolution is 1920 × 1080 pixels. The entire dataset was divided into training and validation sets at a ratio of 8:2, resulting in 46,832 samples for model training and 11,708 samples for the validation set. All quantitative evaluations (Table 1, Table 2 and Table 3) were conducted on this validation set, which effectively serves as our test set. Regarding the radar data, the employed millimeter-wave radar provides multi-dimensional point clouds. For each reflected target, the radar point contains essential physical measurements, including 3D spatial coordinates, radial Doppler velocity, and Radar Cross Section. These features are crucial for our subsequent depth correction and velocity state updates.

Table 1. Comparison of perception algorithm performance. (Evaluated on the WaterScenes dataset).

Method	Scenes	Parameters (M)	FLOPs (G)	OD			SS	FPS
				mAP50	mAP 50–90	AR 50–91	mIoU
Faster R-CNN	C	41.7	93.6	74.4	39.7	46.1	-	15.2
YOLOv5-T	C	5.6	13.4	67.8	33.9	41.3	-	90.5
YOLOv8-N	C	3.2	9.3	70.1	36.2	42.4	-	112.8
DETR	C	41.9	84.1	74.3	42.5	46.6	-	25.3
Mask R-CNN	C	44.5	183.2	73.5	40.3	47.5	-	12.4
YOLOP	C	7.9	18.6	68.3	37.5	43.6	98.4	85.7
HybridNets	C	12.8	15.7	69.2	38.5	44.1	98.7	30.2
Achelous	R + C	7.2	6.0	76.5	43.8	46.5	99.3	53.9
Ours	R + C	9.5	10.3	77.4	44.2	47.4	99.6	51.6

Table 2. Ablation study of radar constraints. (Evaluated on the WaterScenes dataset).

Method	RC	PM	PD	D-MAE (m)	BEV IoU (%)
Vision-only	✗	✗	✗	4.31	54.1
+HR	✓	✗	✗	1.94	64.2
+PR	✓	✓	✗	1.22	70.6
Ours	✓	✓	✓	1.09	78.4

RC = Radar Constraint, PM = Probabilistic Modeling, PD = Point Diffusion; D-MAE = Depth MAE, HR = Hard Radar, PR = Prob Radar.

Table 3. RAN ablation experiment. (Evaluated on the WaterScenes dataset).

Association	AR	MR	AMR	Trajectory Discretization	Max Frames	MOTA (%)
KF	✗	✗	✗	29	63	69.4
AR	✓	✗	✓	21	82	75.6
MR	✗	✓	✓	15	89	78.1
AMR	✓	✓	✗	11	97	81.3
Ours	✓	✓	✓	7	112	84.6

AR = Appearance RAN, MR = Motion RAN, AMR = Appearance + Motion RAN.

4.1.2. Baseline Models

To evaluate the proposed improvements, two baseline systems were constructed for comparison. Both baselines and Geo-RVF share the same visual backbone, detection framework, and training data. They differ only in multi-modal fusion strategies and motion modeling methods.

The Vision Baseline operates as a pure vision USV perception system, utilizing only visual inputs without any radar information. It implements object detection and segmentation based on YOLO-AFM and constructs BEV representations via a monocular depth estimation model, where target velocity is estimated by differentiating adjacent BEV positions.

The Vision + EKF Baseline augments the purely visual framework with an EKF to explicitly model motion states. Although this system still relies exclusively on visual observations, it is designed to verify the improvements in motion estimation stability provided by traditional filtering techniques when direct physical velocity constraints are absent.

The relevance of establishing these two specific baseline models is to serve as systematic ablation benchmarks. By comparing Geo-RVF against the ‘Vision Baseline’, we can quantitatively demonstrate the necessity of radar geometric priors in resolving visual depth scale drift. Furthermore, comparing Geo-RVF against the ‘Vision + EKF Baseline’ isolates and validates the contribution of our proposed RAN module and radar Doppler velocity in achieving stable temporal tracking compared to traditional purely kinematic filtering.

The proposed method further introduces radar data to achieve cross-modal depth scale correction and radar-aware motion estimation. Figure 6 illustrates key training metrics, including loss, detection accuracy, and segmentation accuracy. These metrics indicate effective model convergence. After applying pruning techniques, high detection accuracy is maintained, achieving 82.5% in mean average precision (mAP) 50–90.

Figure 6. Loss and accuracy changes during visual model training. (Evaluated on the WaterScenes and simulated datasets).

4.1.3. Experimental Details

All models were implemented using PyTorch2.2.1 and CUDA12.7. Experiments used the AdamW optimizer with an initial learning rate of 1 × 10⁻⁴. Weight decay was set to 0.5. Cosine annealing was used as the learning rate scheduler, decaying to 0.1 starting from 40% of the training process. During both training and inference, to balance perception accuracy and the limited computational resources of edge devices, the input images for all models were uniformly resized to 640 × 640 pixels. The number of epochs was set to 100 with a batch size of 16. Training took 33 h on an RTX 3090 (NVIDIA, Santa Clara, CA, USA). Data was uniformly split into training and validation sets. All comparison methods were trained using the same dataset and strategy. During inference, all methods used identical input resolutions and post-processing pipelines. The system was deployed and tested on a Jetson Orin NX (NVIDIA, Santa Clara, CA, USA) to evaluate real-time performance.

4.2. Perception Performance Evaluation and Analysis

To verify the effectiveness of the method in water surface perception, state-of-the-art and classic models suitable for aquatic environments were selected for comparison. Performance was evaluated based on object detection, semantic segmentation, and detection efficiency. As shown in Table 1, Geo-RVF achieved optimal performance in object detection.

Regarding detection performance, traditional two-stage methods (e.g., Faster R-CNN [46], Mask R-CNN [47]) offer high accuracy but require significant computational resources, making them unsuitable for real-time edge deployment. Lightweight one-stage methods (e.g., YOLOv5-T, YOLOv8-N) offer fast inference but suffer from reduced accuracy in complex water scenes due to lighting changes and scale uncertainty. Joint detection and segmentation models (YOLOP [48], HybridNets [49]) perform stably in segmentation but are limited by errors inherent to pure visual input.

In contrast, both Achelous [50] and the proposed method incorporate radar data, significantly outperforming pure vision solutions. This validates the effectiveness of radar geometric priors in water environments. The proposed method achieved the best results in mAP₅₀, mAP_50–90, and AR_50–90 metrics, while maintaining leading performance in segmentation accuracy. While a pure 2D vision model like YOLOv8-N achieves a higher FPS, it inherently lacks depth estimation, radar fusion, and 3D BEV transformation capabilities. In the specific domain of 3D multi-modal BEV perception, mainstream frameworks (e.g., BEVFormer or TransFusion) typically require massive computational resources (often exceeding 100 GFLOPs), making them unviable for USV edge devices. Although our Geo-RVF has slightly higher FLOPs than the highly optimized Achelous, this marginal increase in computation is deliberately traded for superior perception accuracy and enhanced robustness in depth correction. By compressing a complete multi-modal 3D BEV pipeline into just 10.3 GFLOPs and maintaining an inference speed of over 50 FPS (23 FPS on an edge device), Geo-RVF demonstrably achieves a lightweight status while providing a stable and reliable foundation for subsequent depth correction and BEV construction.

4.2.1. Depth Estimation and BEV Quality Analysis

Accurate depth is the foundation of BEV construction. Depth estimation errors were analyzed across different distance intervals. As shown in Figure 7, the number of radar points detected on targets decreases with distance. Beyond 80 m, fewer than five points match the bounding box. Pure vision methods exhibit significant scale drift for distant targets. Introducing radar constraints significantly improved depth accuracy in target regions, maintaining stable spatial positioning across all distances. While hard radar constraints limit error growth, they risk propagation errors when point cloud matching fails. In contrast, the proposed depth correction based on probability consistency maintained a low and stable error level across the full range.

Figure 7. Visualization of point cloud projection. (Qualitative results from the real USVs platform).

In Figure 9a, baseline methods correct depth near contact points, but this constraint acts only locally. This often causes depth mutations at target boundaries, disrupting the continuity of the depth field. Geo-RVF uses radar measurements as soft constraints and diffuses them along the water surface. Consequently, the method maintains low depth errors even in distant water regions.

To evaluate the spatial structural validity of the BEV representation, the generated BEV water area was compared with a real map (Figure 9b). The shoreline trend in the generated BEV aligns with the satellite map, showing no obvious deformation or structural breaks.

4.2.2. Robustness in Complex Lighting Scenarios

To verify the stability of this system under complex lighting conditions, this study conducted a qualitative comparison between the pure visual benchmark and Geo-RVF. As shown in Figure 8, pure visual perception relies entirely on texture and gradient features, but when pixels are overexposed due to sunlight glare or underexposed at night due to insufficient light, these features are easily damaged or completely disappear. Therefore, models relying solely on vision will suffer from severe missed detections.

Figure 8. Complex Lighting Comparison Chart.

In contrast, by introducing radar spatial anchors and using an adaptive feature module to fuse multimodal features, our radar visual fusion system effectively utilizes radar echoes to penetrate visual interference. Even when interfered with by strong light when crossing an archway, the radar can provide highly reliable spatial cues. As shown in Figure 8, our fusion system successfully restored the bounding boxes under these unfavorable conditions and maintained accurate navigation area segmentation, demonstrating adaptability to diffuse lighting environments.

4.3. Dynamic Tracking Performance and Ablation Study

4.3.1. Velocity Estimation Accuracy in Motion Scenarios

Three typical motion scenarios were set up for a target boat navigating open waters. The velocity estimation differences between the pure vision scheme and the fusion scheme were compared using root mean square error (RMSE) (Figure 9c). Baseline methods showed high variance. Specifically, baselines rely on differentiating target positions between adjacent frames, which causes lag during acceleration or deceleration. By fusing radar Doppler velocity, the system reflected real-time speed changes with an RMSE of only 0.12 m/s. In turning scenarios, baseline methods occasionally lost targets, whereas the radar method significantly improved stability by directly constraining the motion state via radial velocity.

Figure 9. (a) Visualization comparison of depth estimation; (b) BEV visualization comparison. To intuitively evaluate spatial correctness, the generated BEV drivable areas (blue masks) are overlaid onto corresponding global maps. An ego-vessel coordinate system is also overlaid to illustrate the viewpoint transformation. (c) Velocity estimation comparison in three scenarios. (Qualitative results from the real USVs platform).

Results indicate that after introducing radar information, the system can directly correct motion states using Doppler velocity. RMSE decreased from approximately 2.5–2.8 m/s to 0.1–0.2 m/s, enhancing perception stability in dynamic real-world environments.

4.3.2. Ablation Experiments

Table 2 shows that BEV mapping accuracy benefits from point cloud priors, outperforming pure radar-guided depth estimation. However, using 3D point cloud hard constraints proved ineffective and unreasonable. To address this, a depth correction method based on probability consistency modeling was proposed. By fusing radar ranging and visual features at the probability distribution level, depth estimation for objects improved significantly. Furthermore, combining this with a contact point diffusion strategy allowed depth constraints to propagate smoothly across the water surface, ensuring geometric consistency while avoiding local depth mutations.

To evaluate the dynamic tracking performance of the RAN module in Table 3, we constructed a multi-object tracking evaluation set. This subset consists of five consecutive video sequences selected from the validation set, totaling 64 unique dynamic targets. The selected scenarios include target intersections, partial occlusions, and severe scale changes during relative motion. This is to test the robustness of the algorithm in terms of identity switching and trajectory fragmentation. To verify the effectiveness of the RAN, an ablation analysis was performed on the association module, as shown in Table 3. Introducing Appearance-RAN reduced trajectory interruptions by eight times, increased the max tracking length to 82 frames, and improved Multi-Object Tracking Accuracy (MOTA) by 6.2%. This confirms the module effectively models geometric contours and scale changes in the BEV space, improving robustness under short-term occlusion. Introducing only the Motion-RAN module improved stability under complex motion conditions. Trajectory interruptions dropped to 16, max tracking length rose to 89 frames, and MOTA increased to 78.1%. This module achieved temporal modeling of acceleration and turning, aligning state prediction with physical motion constraints.

When both Appearance-RAN and Motion-RAN were introduced without the autoregressive fusion strategy, performance improved but remained limited in trajectory continuity. In contrast, the proposed joint autoregressive association model synergistically modeled appearance and motion. This reduced trajectory interruptions to seven, extended stable tracking to 112 frames, and raised MOTA to 84.6%. This demonstrates that the autoregressive fusion mechanism effectively accumulates historical association information and suppresses uncertainty propagation.

4.4. Real-Ship Deployment and Performance Verification

Figure 10a shows the USVs platform used for water surface perception. The platform measures 2 × 5 m with a maximum speed of 2 m/s. It integrates multi-source sensors, including an RGB camera (SG2-IMX390C-5200-G2A, Sensing World, Shenzhen, China; FOV = 118° × 62°), Airy LiDAR (RoboSense, Shenzhen, China), and a positioning module (FSS-NAV680-SA, FORSENSE, Shanghai, China; IMU, real-time kinematic (RTK) and Global Positioning System (GPS)). All perception and fusion algorithms run on an Nvidia Orin NX edge computing platform (8 GB RAM, 70 TOPS). A sensor fusion strategy combines data from the camera, LiDAR, GPS, and IMU to enhance object detection. This framework facilitates the transition of algorithms to real-world USVs platforms.

Figure 10. Algorithm Operation Diagram. (a) USV platform and perception sensors; (b) detection and segmentation; (c) depth estimation; (d) BEV. The controllable viewing area in (b) is geometrically transformed into the top-down view space in (d). In this figure, the ships detected on the right side of (b) are accurately mapped to the white void area (which is non-navigable) in the upper-right corner of the bounding box map (d). (Qualitative results from the real USVs platform).

On the software level, the system includes sensor drivers, time synchronization mechanisms, and coordinate transformation modules. Without optimization, the system occupied 2.6 GB memory, 30% CPU, and 75% GPU, with an output frequency of 12 FPS. After TensorRT acceleration optimization, GPU utilization increased by 8%, and the frame rate rose to 22 FPS, ensuring responsiveness under dynamic conditions.

5. Conclusions

To address core challenges in USV navigation, such as visual depth drift, motion estimation lag, and limited edge computing resources, a lightweight radar–vision fusion perception system named Geo-RVF is proposed. The system enhances perception robustness through dual geometric and temporal physical constraints. To resolve scale uncertainty in open waters, a probabilistic depth correction mechanism based on radar geometric anchors is designed. By utilizing the target-water contact region, sparse radar ranging is converted into soft constraints. This effectively suppresses water clutter interference and corrects scale deviations in BEV mapping. Furthermore, a tracking framework combining RAN and EKF is constructed. The long-term memory of RAN resolves association difficulties caused by texture degradation. Additionally, radar Doppler velocity is introduced to correct visual observation lag in real time, significantly reducing velocity estimation errors. Experiments demonstrate that the TensorRT-optimized Geo-RVF achieves real-time inference of 23 FPS on the Jetson Orin NX. Its detection accuracy of 44.2% mAP and segmentation performance of 99% mean intersection over union (mIoU) outperform existing mainstream lightweight models. Future work will focus on integrating infrared thermal sensors to address night perception limitations and exploring geometric self-supervised learning paradigms to further improve generalization and autonomous decision making in extreme sea conditions.