Research on Multi-USV Collision Avoidance Based on Priority-Driven and Expert-Guided Deep Reinforcement Learning

Abstract

Deep reinforcement learning (DRL) has demonstrated considerable potential for autonomous collision avoidance in unmanned surface vessels (USVs). However, its application in complex multi-agent maritime environments is often limited by challenges such as convergence issues and high computational costs. To address these issues, this paper proposes an expert-guided DRL algorithm that integrates a Dual-Priority Experience Replay (DPER) mechanism with a Hybrid Reciprocal Velocity Obstacles (HRVO) expert module. Specifically, the DPER mechanism prioritizes high-value experiences by considering both temporal-difference (TD) error and collision avoidance quality. The TD error prioritization selects experiences with large TD errors, which typically correspond to critical state transitions with significant prediction discrepancies, thus accelerating value function updates and enhancing learning efficiency. At the same time, the collision avoidance quality prioritization reinforces successful evasive actions, preventing them from being overshadowed by a large volume of ordinary experiences. To further improve algorithm performance, this study integrates a COLREGs-compliant HRVO expert module, which guides early-stage policy exploration while ensuring compliance with regulatory constraints. The expert mechanism is incorporated into the Soft Actor-Critic (SAC) algorithm and validated in multi-vessel collision avoidance scenarios using maritime simulations. The experimental results demonstrate that, compared to traditional DRL baselines, the proposed algorithm reduces training time by 60.37% and, in comparison to rule-based algorithms, achieves shorter navigation times and lower rudder frequencies.

1. Introduction

1.1. Related Works

The efficacy of collision avoidance in multi-USV cooperative scenarios is critically challenged by dynamic obstacle behaviors, complex sea conditions, and stringent maritime navigation regulations [1,2]. Conventional rule-based methods frequently yield oscillatory or deadlock-prone trajectories, while purely data-driven approaches are characterized by slow convergence and inadequate regulatory compliance [3]. Furthermore, many reactive strategies are inadequate in achieving stable and efficient maneuvering, often resulting in path oscillations, velocity fluctuations, and an inherent trade-off between safety and efficiency. Specifically, conservative policies, while reducing the risk of collision, tend to cause detours and delays; conversely, aggressive behaviors may increase the likelihood of collisions [4]. These limitations underscore the urgent need to develop adaptive, regulation-aware obstacle avoidance strategies tailored to complex, multi-agent maritime environments.

The development of vessel tracking and control has evolved significantly in recent decades, transitioning from classical control strategies to learning-based approaches. For example, Proportional-Integral-Derivative (PID) controllers were first widely used in current control class methods [5]. Later Vadim Utkin et al. proposed Sliding Mode Control (SMC), which improved robustness to model uncertainty and external disturbances through variable structure control, but suffered from chattering effects [6]. The Line of Sight (LOS) guidance method provided a geometric solution based on heading deviation, yet its performance degraded under strong lateral disturbances [7,8]. With the rise of prediction-based methods, Model Predictive Control (MPC) enabled more flexible trajectory tracking by optimizing actions over a finite horizon while incorporating constraints [9]. However, its high computational cost limits real-time applications. Recently, the emergence of DRL provides new approaches for tracking and controlling unmanned vessels. DRL continuously optimizes the corresponding decision-making strategies through continuous learning in the constant interaction of the intelligence with the environment [10,11]. Among DRL algorithms, the Deep Deterministic Policy Gradient (DDPG) algorithm introduced deterministic actor-critic learning for continuous control tasks such as vessel navigation [12]. Later, Proximal Policy Optimization (PPO) addressed the training instability of DDPG by bounding policy updates, thus improving training stability and sample efficiency, making it suitable for complex marine scenarios [13]. Despite these limitations, DRL offers new opportunities for enhancing autonomous collision avoidance in USVs, particularly in dynamic and uncertain maritime environments.

Typical obstacle avoidance algorithms for USVs include the Artificial Potential Field (APF), Dynamic Window Approach (DWA), and Velocity Obstacle (VO) methods. Khatib proposed APF algorithm to generate real-time paths by simulating attractive and repulsive virtual forces [14]. The DWA evaluates candidate trajectories based on safety and efficiency metrics to make real-time avoidance decisions [15], but lacks adaptability to dynamic obstacles. The VO method detects potential collisions by predicting velocity obstacles and selecting safe speeds accordingly [16,17]. However, it suffers from inefficient paths and poor compliance with maritime navigation rules, limiting its real-world applicability [18]. To address the oscillation issue in VO, the Reciprocal Velocity Obstacle (RVO) algorithm introduces mutual responsibility between agents, allowing cooperative avoidance among multiple vessels. This narrows the feasible velocity space and produces smoother, more realistic maneuvers [19]. These methods, while effective in structured scenarios, often fail to comply with the COLREGs, which are essential for safe and lawful maritime navigation [20,21].

While DRL has demonstrated significant potential in addressing complex decision-making problems and has been increasingly integrated with rule-based collision avoidance strategies, several fundamental limitations remain in existing hybrid approaches [22]. Methods combining artificial potential fields with DRL often suffer from local minima and oscillatory behaviors, limiting their robustness in dense multi-vessel encounters. Classical HRVO- or VO-based planners, although effective in generating collision-free velocities under geometric constraints, are typically employed as deterministic controllers and lack long-term optimization and adaptability to complex maritime scenarios. Prioritized experience replay-based DRL methods, such as PER-SAC or PER-DDPG, primarily rely on TD-error to improve learning efficiency, but they do not explicitly emphasize safety-critical avoidance behaviors, causing rare yet crucial evasive experiences to be underrepresented during training. In addition, many COLREGs-aware reinforcement learning approaches mainly incorporate maritime rules through reward shaping or penalty terms, which cannot fully prevent unsafe or non-compliant exploration during early training stages. As a result, despite notable progress, existing methods still face challenges in balancing safety, learning efficiency, and regulatory compliance in dynamic multi-agent maritime environments.

1.2. Novelty and Contributions

To address the limitations of existing hybrid collision avoidance methods in terms of convergence efficiency, safe exploration, and robustness in multi-vessel interactions, this paper proposes an expert-guided DPER-SAC framework. The core idea of this framework is to reconstruct the reinforcement learning process for complex maritime collision avoidance tasks by integrating structured expert priors and task-aware experience replay.

Unlike APF-DRL or reward-shaping methods that indirectly influence policy learning—often leading to decision oscillations or local deadlocks—this paper integrates COLREGs-compliant velocity preferences directly into an HRVO-based expert module. This approach imposes regulatory consistency constraints on the candidate velocity space during the action generation phase. Rather than relying on static reward penalties, we introduce encounter-dependent risk weights to dynamically calibrate COLREGs compliance based on real-time environmental risk and regulatory priorities. This mechanism allows the agent to flexibly adapt its maneuvers to diverse interaction scenarios, effectively mitigating the conflicts and sub-optimal behaviors inherent in rigid reward-shaping frameworks.

Traditional rule-based collision avoidance algorithms, such as VO and HRVO, are often implemented as fixed controllers. However, these methods can lead to overly conservative maneuvers and potential deadlock situations. To overcome these limitations, this paper incorporates HRVO as a decaying expert prior to providing safe and structured guidance during the initial stages of reinforcement learning. This mechanism effectively restricts the search space and avoids the inefficiencies of random exploration. As training progresses, the SAC framework’s long-term reward optimization takes precedence, gradually phasing out the expert’s influence. This transition enables the agent to autonomously develop and execute optimal collision avoidance strategies in complex multi-vessel interactions.

Furthermore, the proposed DPER mechanism effectively addresses the inherent limitations of traditional PER in complex maritime collision avoidance tasks by integrating both TD-error and collision avoidance quality into the prioritization framework. While conventional PER methods rely primarily on TD-error to identify high-learning-potential experiences, successful collision avoidance in multi-vessel scenarios is often a sparse event. Consequently, these critical avoidance behaviors are frequently overshadowed by a vast volume of redundant, non-critical data, leading to a training bias toward common behaviors at the expense of rare, high-value samples. To mitigate this, DPER extends the PER framework by introducing collision avoidance quality as a secondary priority metric. By explicitly prioritizing high-quality avoidance samples directly aligned with task objectives, DPER ensures that these pivotal experiences are sufficiently reinforced, thereby accelerating policy convergence and enhancing robustness.

2. Dynamics of Unmanned Surface Vessel

2.1. USV Model

In this study, a dual-coordinate system was employed to model the motion of an USV. The selection of an appropriate coordinate system ensures accurate representation of the vessel’s orientation and position. As illustrated in Figure 1, the Earth’s inertial reference frame (O_E-X_EY_EZ_E) was used, with X_E and Y_E aligned to the geographical directions, while Z_E is oriented along the Earth’s gravitational axis. The USV’s local coordinate system (O_B-X_BY_BZ_B) was defined as follows: X_B corresponds to the forward direction, Y_B represents the right side, and Z_B aligns with the vertical direction.

2.2. Vessel Dynamics Characteristics

2.2.1. Dynamic Modeling of Motion Systems

This study is based on the principles of vessel motion mechanics and fluid dynamics, with the aim of developing three degrees of freedom model for USV. The focus is on the horizontal, vertical, and longitudinal movements of the vessel within the water. The dynamic equation governing the system is as follows:

$$\left\{\begin{array}{l}\dot{u}=f_u\left(u,v,r,\delta ,{\tau }_{env}\right)\\ \dot{v}=f_v\left(u,v,r\right)\\ \dot{r}=f_r\left(r,\delta ,{\tau }_{env}\right)\end{array}\right.$$

(1)

where $u,v$ and $r$ denote the surge velocity, sway velocity, and yaw rate. $\dot{u}$, $\dot{v}$ and $\dot{r}$ represent their corresponding time derivatives. $\delta$ denotes the rudder and ${\tau }_{env}$ denotes the effective environmental force. The functions $f_u$, $f_v$, and $f_r$ describe the respective dynamic variations.

Traditional Nomoto models often overlook critical factors like the angle of attack and the interactions with environmental forces. To overcome these limitations, modified models incorporating environmental moment coefficients, ${\tau }_{env,N}$, have been proposed to more accurately capture the impact of wind, waves, and currents on vessel dynamics. The improved Nomoto model can be expressed as follows:

$$T\dot{r}+r=K\delta +{\tau }_{env}$$

(2)

where $T$ is the time constant of the yaw rate response, $K$ is the rudder gain coefficient.

In the horizontal motion of USVs, hydrodynamic forces in the Z direction significantly influence the vessel’s speed dynamics. During turning maneuvers, fluid–structure interactions generate nonlinear resistance forces that affect horizontal acceleration. This resistance is described by the following equation:

$$\dot{v}={\displaystyle \frac{v - c\cdot u\cdot r}{d}}$$

(3)

where c is the velocity-dependent coupling coefficient that characterizes the interaction between surge velocity and yaw rate, and d denotes the horizontal damping coefficient associated with hydrodynamic resistance during sway motion.

2.2.2. Kinematic Characteristics

The field of vessel dynamics focuses on the geometric properties of vessels, including their position, velocity, and direction. A vessel’s state of motion evolves over time. The variables $\dot{x}$ and $\dot{y}$ represent the real-time updates of the vessel’s position, while $v$ and $\dot{x}$ describe the relation vessel between the vessel’s velocity and the rate of change in its position within the vessel’s local coordinate system, as compared to the positional coordinates of a fixed reference frame on Earth. The following equation represents the dynamic equation of the vessel:

$$\left\{\begin{array}{c}\dot{x}=usin\left(\psi \right)+vcos\left(\psi \right)\\ \dot{y}=ucos\left(\psi \right)-vsin\left(\psi \right)\\ \dot{\psi }=r\end{array}\right.$$

(4)

where $\dot{x}$ denotes the rate of movement in the eastward direction, $\dot{y}$ denotes the rate of movement in the northward direction, and $\dot{\psi }$ denotes the rate of change in the heading.

3. Proposed Deep Reinforcement Learning Method for Collision Avoidance

3.1. COLREGs-Aware Velocity Selection with HRVO

3.1.1. The COLREGs Rules

The COLREGs (Rules 13–15) outline standardized protocols for vessel encounters, categorizing them into three primary types: overtaking, head-on, and crossing situations. As depicted in Figure 2, this study utilizes these rules to design scenario-specific avoidance behaviors for autonomous surface vessels. These regulations are essential for preventing accidents in congested maritime environments, where multiple vessels often interact within limited spaces. By adhering to COLREGs, the USV can make decisions that are not only collision-free but also legally compliant, thereby enhancing both safety and efficiency in maritime operations.

3.1.2. Hybrid Reciprocal Velocity Obstacles Method

The VO model offers a geometric framework for predicting potential collisions by identifying the set of velocities that would result in contact based on relative motion. As illustrated in Figure 3, the VO space encompasses all velocity vectors that would lead to a collision with a moving obstacle. However, this approach does not consider the intentions of other agents, which can limit its effectiveness in dynamic, multi-agent environments. Although the VO model provides a simple and intuitive formulation, it places full avoidance responsibility on the ego agent, often leading to oscillatory or overly conservative behaviors.

To address the limitations of conventional VO methods, this study proposes an enhanced HRVO formulation that integrates both reciprocal coordination and maritime regulatory constraints. Departing from traditional methods with rigid assumptions, the HRVO framework introduces a dynamic responsibility allocation mechanism. This approach flexibly adjusts the avoidance responsibilities among interacting agents based on their observed intentions, combining the efficiency of reciprocal cooperation with the robustness required to manage non-cooperative behaviors. Furthermore, velocity selection is guided by COLREGs-based preferences, enabling the expert module to generate candidate actions that are not only collision-free but also compliant with navigational rules.

The HRVO algorithm combines the reciprocal principles of the VO method with speed-space avoidance techniques, offering a more efficient solution for multi-agent collision avoidance. It adopts a hybrid responsibility allocation mechanism to effectively manage obstacle avoidance among multiple intelligent agents in complex environments. The mathematical formulation of the HRVO algorithm is presented as follows:

$${HRVO}_{\mathrm{U}\mathrm{S}\mathrm{V}|\mathrm{A}}\left(\tau \right)=\left\{v_{USV}\mid \exists t\in \left[0,\tau \right]:D+\left(v_{USV}-v_{\mathrm{A}}+wu\right)t<r_{\mathrm{U}\mathrm{S}\mathrm{V}}+r_{\mathrm{A}}\right\}$$

(5)

where ${ v}_{\mathrm{U}\mathrm{S}\mathrm{V}}-v_{\mathrm{A}}$ represents the relative velocity between agents, $\tau$ denotes the time horizon for collision prediction, $D$ is the current distance between the two agents, and $r_{USV} ,{ r}_A$ are the radii of the safety regions of agent USV and A, $w \in [0, 1]$ is the dynamic responsibility weight. When $w=0.5$, the model degenerates into the RVO algorithm, and when $w=1,$ it degenerates into the standard VO algorithm.

Furthermore, the HRVO framework enables velocity selection to be optimized according to specific task objectives and environmental constraints. A representative cost-based velocity selection model is defined as follows:

$$v_{\mathrm{new}}=\mathrm{a}\mathrm{r}\mathrm{g}\underset{v\in \mathrm{H}\mathrm{R}\mathrm{V}\mathrm{O}}{min}({\parallel v-v_{\mathrm{pref}}\parallel }^2+\lambda \cdot C_{\mathrm{COLREGs}}(v)+\gamma \cdot C_{\mathrm{dynamic}}(v))$$

(6)

where $v_{\mathrm{new}}$ is the selected goal-directed velocity, $v_{\mathrm{pref}}$ denotes the preferred velocity towards the target. $C_{\mathrm{COLREGs}}\left(v\right)$ measures compliance with maritime collision regulations. $C_{\mathrm{dynamic}}\left(v\right)$ penalizes high acceleration or sharp maneuvers. Coefficients $\lambda$ and $\gamma$ control the trade-off between regulation adherence, safety, and motion smoothness.

As illustrated in Figure 4, an adaptive HRVO model is employed for multi-agent collision avoidance. In Figure 4a, the USV is simplified as a reference point for efficiency, while target vessels are represented by elliptical envelopes. The USV is simplified as a reference point for efficiency, while target vessels are represented by elliptical envelopes. Compared to conventional circular models, the elliptical abstraction based on semi-major axis a and semi-minor axis b provides a tighter fit to the vessel’s physical profile. By integrating real-time heading and dimensions, this approach defines a dynamically rotating safety buffer, ensuring that velocity-space transformations rigorously account for the obstacle’s spatial occupancy and orientation-dependent constraints.

Figure 4b illustrates the construction of velocity obstacles under the HRVO framework. In this representation, each obstacle (A, B, C) is modeled as a dynamic agent with known positions ($p_{\mathrm{A} }$, $p_{B }$, $p_{\mathrm{C} }$) and velocities ($v_{\mathrm{A} }$, $v_B$, $v_C$), while the USV is assigned a current velocity V_U. For each obstacle, a velocity obstacle cone is constructed in velocity space, defining the set of velocities that would result in a collision within a finite time horizon.

Unlike traditional VO methods, the apex of each HRVO cone is not located at the origin but is shifted to a reciprocal velocity point that reflects the shared responsibility between the USV and the obstacle. This reciprocal velocity apex is computed as follows:

$$v^{\prime }=w\cdot v_{\mathrm{o}\mathrm{b}\mathrm{s}\mathrm{t}\mathrm{a}\mathrm{c}\mathrm{l}\mathrm{e}}+\left(1-w\right)\cdot v_{\mathrm{U}\mathrm{S}\mathrm{V}}$$

(7)

where $v^{\prime }$ denotes the shifted velocity apex used for HRVO cone construction. $v_{USV}$ and $v_{obstacle}$ are the velocities of the USV and the obstacle. w ∈ [0, 1] represents the dynamic responsibility weight. This weighted combination reflects the assumed reciprocal responsibility of both agents in executing avoidance maneuvers.

The collision avoidance velocity space is defined by aggregating the velocity obstacle cones of surrounding dynamic agents. Compared to RVO, HRVO adopts a hybrid approach by incorporating adaptive weighting and trajectory optimization. The integration of these mechanisms enables the framework to respond more effectively to non-cooperative behaviors and dynamic environmental disturbances. In practice, HRVO not only considers relative velocities but also optimizes trajectories by accounting for the evolving intentions of target vessels. When a target vessel demonstrates no avoidance intent, the responsibility weight $w$ is adaptively increased, shifting the primary collision avoidance burden to the agent. Moreover, to accommodate vessel inertia, this weight adjustment occurs gradually rather than instantaneously. By ensuring that $w$ evolves incrementally, the system facilitates smoother, more kinematically feasible maneuvers, avoiding abrupt course changes. This approach effectively balances immediate safety needs with the physical limitations of maritime motion, ensuring robust operational stability in complex scenarios.

While HRVO serves as a safety-oriented expert prior, it is prone to path sub-optimality and systemic deadlocks, particularly in dense environments. Moreover, HRVO’s focus on immediate safety frequently results in inefficient trajectories. These limitations were empirically observed during training, with vessels either encountering standstills in crowded scenarios or exhibiting excessive maneuvering in sparse ones. To address these issues, the SAC framework utilizes its stochastic exploration and experience replay mechanisms to intervene upon detecting deadlock, increasing the SAC control weight to shorten the duration of the deadlock state. Additionally, during subsequent autonomous control, the agent autonomously re-learns from these high TD-error experiences.

In summary, the HRVO framework provides a robust and interpretable foundation for motion planning in dynamic marine environments. By incorporating COLREGs-compliant velocity selection into the expert model, it supplies structured prior knowledge that not only guides the learning process but also alleviates the generation of rule-violating or overly conservative velocities in DRL.

3.2. Dual-Priority Experience Replay Mechanism

Experience replay is a critical component in reinforcement learning, contributing to both training stability and learning efficiency. However, conventional replay buffers typically employ uniform sampling strategies that disregard the varying significance of individual transitions. This oversight can impede the learning process, particularly in complex, multi-agent environments such as multi-USV scenarios.

To address this limitation, we propose a DPER mechanism that integrates two complementary prioritization strategies: one based on TD error and the other on obstacle avoidance quality. The TD error component emphasizes transitions with high learning potential by prioritizing those that exhibit significant discrepancies between predicted and actual rewards, thereby facilitating more effective value function updates. In parallel, the quality-based component assigns higher importance to transitions involving successful obstacle avoidance, which are evaluated through metrics such as minimum distance to obstacles, safe maneuvering speed, and trajectory alignment. During the sampling process, these two priority scores are combined using a weighted formulation to derive the final probability of selection. This dual-priority scheme ensures that both transitions critical for policy optimization and those demonstrating high-quality behavioral outcomes are systematically emphasized, thereby enhancing sample efficiency, promoting regulatory compliance, and improving the overall robustness of the learned policy.

To represent experience transitions, this study employs a quintuple structure defined as $\left(s_t,a_t,r_t,s_{t+1}\right)$, where $s_t$ is the state, $a_t$ is the action, $r_t$ is the reward and $s_{t+1}$ is the next state.

The TD error ${ \delta }_t$ is which measures the discrepancy between predicted and actual rewards, is calculated as

$${\delta }_t=r_t+\gamma \cdot {max}_{a^{\prime }}{Q}_{target}\left(s_{t+1},a^{\prime }\right)-Q_{current}\left(s_t,a_t\right)$$

(8)

where $\gamma \in [0,1]$ denotes the discount factor, ${\mathrm{Q}}_{\mathrm{t}\mathrm{a}\mathrm{r}\mathrm{g}\mathrm{e}\mathrm{t}}$ represents the target $\mathrm{Q}$ network, and ${\mathrm{Q}}_{\mathrm{c}\mathrm{u}\mathrm{r}\mathrm{r}\mathrm{e}\mathrm{n}\mathrm{t}}$ denotes the current $\mathrm{Q}$ network.

The corresponding TD-based priority for sample $i$ is given by

$$P\left(i\right)={\left(TD\left(i\right)+ϵ\right)}^{\alpha }$$

(9)

where $P\left(i\right)$ represents the priority of the i-th experience, $ϵ$ is a constant to prevent division by zero errors, and $\alpha$ determines the level of prioritization.

In addition to TD error, obstacle avoidance performance is used as a complementary prioritization criterion. A Boolean flag $f_{success}\in \left\{\mathrm{0,1}\right\}$ indicates whether the navigation successfully avoided a collision. If successful, a normalized quality score $q_i\in \left[\mathrm{0,1}\right]$ is computed to quantify the effectiveness of the maneuver. The overall formulation is defined as follows:

$$q_i={\eta }_1\cdot d_{min}+{\eta }_2\cdot v_{safe}+{\eta }_3\cdot {\theta }_{align}$$

(10)

where ${\eta }_1$,${\eta }_2$,${\eta }_3$ are the weighting coefficients of each indicator, $d_{min}$ is the minimum distance between the obstacle and the obstacle during obstacle avoidance, $v_{safe}$ is the safe speed during obstacle avoidance, and ${\theta }_{align}$ is the degree of alignment between the heading after obstacle avoidance and the target path.

To enable autonomous obstacle avoidance through expert-guided reinforcement learning, it is essential to gradually reduce reliance on expert knowledge. To this end, a dynamic decay coefficient $\lambda$ is introduced to represent the diminishing influence of expert experience over time. The influence factor $b_t$ is initialized as $b_0$ and decays with training steps $t$, as defined by

$$b_t=b_0\cdot {\lambda }^{t/T}$$

(11)

The dual priority fusion $p_i^{dual}$ formula is

$$p_i^{dual}={\left(TD\left(i\right)+ϵ\right)}^{\alpha }+f_i^{success}\cdot q_i\cdot b_t$$

(12)

The sampling process reflects the mechanism by which experiential transitions are acquired, with the sampling probability directly proportional to the priority level. In this study, the priority exponent α, which controls the degree of prioritization by adjusting the slope of the sampling distribution. The corresponding formula is defined as

$$P\left(t\right)={\displaystyle \frac{p_t^{\alpha }}{{\sum }_i{p}_i^{\alpha }}}$$

(13)

where $P\left(t\right)$ is the normalized sampling probability for time $t$, $p_t$ is the priority of sample $t$.

Since priority-based sampling introduces bias in the distribution of training data, it is necessary to correct this bias by adjusting the importance-sampling weights through a hierarchical update. The corresponding adjustment formula is given as

$$w_t={\left({\displaystyle \frac{1}{N\cdot P\left(t\right)}}\right)}^{\beta }/max\left(w_i\right)$$

(14)

where $w_t$ is the importance-sampling weight for time $t$, N is the buffer size, $\beta \in [0,1]$ is the deviation correction factor. As training progresses, $\beta$ approaches 1 to reduce the bias from prioritized replay.

After computing the importance-sampling weights, a prioritized mini-batch $\mathcal{B}$ is sampled from the replay buffer $\mathcal{D}$ based on the normalized sampling probabilities:

$$\mathcal{B}=\mathrm{S}\mathrm{a}\mathrm{m}\mathrm{p}\mathrm{l}\mathrm{e}(\mathcal{D},P(i\left)\right),i\sim \mathrm{D}\mathrm{i}\mathrm{s}\mathrm{c}\mathrm{r}\mathrm{e}\mathrm{t}\mathrm{e}(p_1,p_2,\dots ,p_N)$$

(15)

where $p_i$ denotes the normalized sampling probability of the $i-\mathrm{t}\mathrm{h}$ transition.

A schematic overview of the proposed DPER mechanism is presented in Figure 5, which integrates TD error and obstacle avoidance quality into a unified experience pool. The agent draws samples from the dual-priority experience pool and updates the parameters of the actor and critic networks. In this architecture, the agent stores experiences $\left(s_t,a_t,r_t,s_{t+1}\right)$ in the dual-priority experience pool. The priority of experiences is calculated based on the TD error and the quality of successful obstacle avoidance. DPER injects experience filtering intelligence into SAC.

By maintaining a balance between expert-guided and suboptimal experiences, the mechanism mitigates the dominance of noisy samples while preserving critical learning signals. This targeted sampling strategy improves both the learning efficiency and policy robustness.

3.3. DPER-SAC Algorithm Architecture

3.3.1. Network Design and Agent Architecture

Recent advances in machine learning have significantly expanded its applicability across diverse domains. These developments are largely driven by the ability of algorithms to adapt to dynamic environments and continuously improve their learning capacity.

In this section, we propose the DPER-SAC algorithm, which integrates a novel dual-priority sampling mechanism into the SAC framework to improve learning efficiency. SAC is based on the maximum entropy reinforcement learning paradigm, aiming to maximize both the expected return and the entropy of the policy, thereby encouraging more comprehensive exploration.

As illustrated in Figure 6, the network architecture adopted in this study comprises three primary components: the actor network, the critic network, and the value network. The actor network, denoted as ${\pi }_{\theta }\left(·\right)$, generates actions based on the current state using a parameterized stochastic policy. The critic network evaluates the selected actions by estimating the Q value and the corresponding state-value. Through interactions with the environment, the agent receives state feedback and transitions to the next state accordingly. The actor network architecture includes both the policy network and its corresponding target network. Based on the current state $s_t$, the actor outputs an action distribution as follows:

$$a_t\sim {\pi }_{\theta }\left(s_t\right)$$

(16)

where ${\pi }_{\theta }$ represents the parameterized policy of the policy network.

Secondly, inspired by the Double DQN architecture, SAC incorporates a dual critic network to mitigate the overestimation of Q values. Specifically, during each update, the minimum value between two Q networks is selected to provide a more conservative target estimate. The critic module in SAC consists of an online critic network and a corresponding target critic network. The critic network outputs the Q value as follows:

$$Q\left(s_t,a_t\right)$$

(17)

The target critic network is used to stabilize the training process, and the output is

$$Q_{target}\left(s_t,a_t\right)$$

(18)

The objective function is defined as the minimization of losses, and its formula is as follows:

$$L\left({\pi }_{\theta },D\right)=-E_{\left(s_t,a_t\right)\sim D}\left[log{\pi }_{\theta }\left(a_t\mid s_t\right)\right]$$

(19)

Concurrently, the loss function of the network of critics is expressed as follows:

$$L\left(\phi ,D\right)=E_{\left(s_t,a_t,r_t,s_{t+1}\right)\sim R}\left[{\left(Q_{\phi }\left(s_t,a_t\right)-r_t-\gamma V\left(s_{t+1}\right)\right)}^2\right]$$

(20)

where R represents the data collected by the strategy in the past, $Q_{\phi }\left(s_t,a_t\right)$ denotes the critic network, and $\gamma$ signifies the discount factor, balancing current and future rewards. $V\left(s_{t+1}\right)$ is the state-value function that estimates the expected return from the next state $s_{t+1}$.

By efficiently storing and leveraging past experiences, the agent can rapidly adapt and optimize its policy in dynamic environments. Moreover, the incorporation of TD error and obstacle avoidance quality as prioritization criteria provides more informative feedback during learning, thereby improving the agent’s practical performance.

3.3.2. Reward Function Design

• The proximity reward and terminal reward

In reinforcement learning for intelligent agents, goal achievement is significantly influenced by the design of proximity and terminal rewards. The proximity reward incentivizes agents to take efficient actions when near the target, thereby improving navigation safety and efficiency. This reward is a function of the distance to the goal, denoted as $d_{goal}$, and is designed to vary with distance, providing differentiated feedback. Such a progressive reward mechanism effectively enhances the exploration capability of the SAC algorithm, as shown in the following formula:

$$r_{proximity}=\left\{\begin{array}{cc}3exp\left(-0.05d_{goal}\right),& d_{goal}<15\\ exp\left(-0.05d_{goal}\right),& 15\le d_{goal}<40\\ 0,& otherwise\end{array}\right.$$

(21)

In this formulation, the proximity reward is highest when the vessel is within 15 m of the target, encouraging optimal navigation near the goal. As the distance increases, the reward decreases exponentially, reflecting reduced proximity.

2.

The heading and distance error reward

The track error is defined as the perpendicular distance between the vessel’s current position and the reference trajectory. Let the vessel’s current position be $(x,y)$, and the endpoints of the reference track segment be $(x_1,y_1)$ and $(x_2,y_2)$. The track error is calculated using the following formula:

$$Track Error={\displaystyle \frac{\left|\left(y_2-y_1\right)x+\left(x_1-x_2\right)y+\left(x_2{y}_1-x_1{y}_2\right)\right|}{\sqrt{{\left(y_2-y_1\right)}^2+{\left(x_2-x_1\right)}^2}}}$$

(22)

The heading error is defined as the angular deviation between the vessel’s current heading and the desired target heading. Let $\theta$ denote the current heading angle and ${\theta }_{goal}$ the target heading angle, both measured in radians. The heading error is computed using the following formula:

$$Heading Error=clip\left(\theta -{\theta }_{goal},-\pi ,\pi \right)$$

(23)

Together, the heading and distance error reward ${ r}_{heading }$ functions enable fine-grained guidance during training, allowing the agent to effectively align its trajectory with the intended path and correct deviations dynamically during navigation.

3.

The obstacle avoidance reward

Collision avoidance rewards in maritime autonomous navigation have traditionally relied on the minimum distance to obstacles. However, this static metric does not capture the dynamic nature of vessel motion. To enhance the agent’s situational awareness, we redefine the reward by incorporating two key maritime safety indices: the Distance to Closest Point of Approach (DCPA) and the Time to Closest Point of Approach (TCPA) thereby enabling a more proactive and multidimensional assessment of collision risks.

The minimum distance reward is calculated as follows:

$$r_{distance}=\left\{\begin{array}{cc}-3exp\left(-0.4d_{obs,min}\right),& d_{obs,min}<4\\ -exp\left(-0.3d_{obs,min}\right),& 4\le d_{obs,min}<15\\ 0,& otherwise\end{array}\right.$$

(24)

where $d_{obs,min}$ denotes the distance between the vessel and the nearest obstacle.

To integrate dynamic risk evaluation, the DCPA and TCPA are introduced to account for relative vessel motion. DCPA measures the time to the closest point of approach, while TCPA reflects the time before vessels reach that point. The dynamic collision risk reward is then calculated as

$$r_{dynamic}=\left\{\begin{array}{cc}-exp\left(-\alpha \cdot DCPA\right),& ifDCPA<threshold\\ -exp\left(-\beta \cdot TCPA\right),& ifTCPA<threshold\\ 0,& otherwise\end{array}\right.$$

(25)

where α and β are constants that adjust the sensitivity of the reward to the DCPA and TCPA, respectively, and the thresholds are set based on the acceptable risk levels for the environment.

The final avoidance reward $r_{avoidance}$ is defined as the sum of both $r_{distance}$ and $r_{dynamic}$. This combined approach enables the agent to make more informed decisions based on both proximity to obstacles and the dynamic risk of collision, improving navigation safety in dynamic and complex maritime environments.

4.

The COLREGs reward

The COLREGs are integrated into the DPER-SAC framework as fundamental constraints to ensure alignment with international maritime safety standards. By embedding these rules into the action selection process, the framework mitigates learning instability and prunes suboptimal trajectories. This integration ensures that the agent’s actions are consistently compliant with maritime navigation rules, thereby enhancing both safety and operational efficiency. The COLREGs reward values are defined as

$$r_{COLREGs}=\left\{\begin{array}{cc}+5.0,& if complying with COLREGs\\ -5.0,& if violating COLREGs\\ 0,& if no relevant encounter\end{array}\right.$$

(26)

5.

Encounter-dependent risk weighting

An encounter-dependent risk weighting scheme is incorporated to account for varying collision risks and the legal priorities defined by the COLREGs. By dynamically adjusting reward weights based on the encounter context, the framework scales the avoidance penalty according to the assessed risk. High-risk proximities lead to higher weights, reinforcing proactive avoidance actions, while lower-risk scenarios apply reduced weights to prevent unnecessary over-maneuvering.

The encounter weight $w_{encounter}$ is determined by the relative distance and velocity between vessels and is calculated differently for various encounter types. The specific calculations are as follows:

$$w_{encounter}=\left\{\begin{array}{ll}{\alpha }_{head-on}\cdot {\displaystyle \frac{1}{d_{rel}}}\cdot {\displaystyle \frac{1}{v_{rel}}},& if Head-on\\ {\alpha }_{overtaking}\cdot {\displaystyle \frac{1}{d_{rel}}}\cdot exp\left(-{\beta }_{overtaking}\cdot {\theta }_{rel}\right),& if Overtaking\\ {\alpha }_{crossing}\cdot {\displaystyle \frac{1}{d_{rel}}}\cdot \left(1-exp\left(-{\gamma }_{crossing}\cdot {\theta }_{rel}\right)\right),& if Crossing\end{array}\right.$$

(27)

where $d_{rel}$ represents the relative distance, $v_{\mathrm{rel}}$ is the relative velocity, and ${\theta }_{rel}$ is the relative heading angle. The coefficients $\alpha$, $\beta$ and $\gamma$ are risk weighting factors specific to each encounter type.

Finally, the total reward $r_{total}$ is calculated by incorporating the encounter-dependent weight $w_{encounter}$ into the overall reward function, which combines proximity, heading, avoidance, and COLREGs compliance

$$r_{total}=w_{encounter}\cdot \left(r_{proximity}+r_{heading}+r_{avoidance}+r_{COLREGs}\right)$$

(28)

The encounter-dependent risk weighting mechanism ensures that the agent prioritizes high-risk encounters while maintaining optimal path planning and compliance with maritime regulations.

3.4. Expert-Guided Deep Reinforcement Learning Strategy

In complex and dynamic maritime environments, the presence of obstacles poses significant challenges to the control and decision-making capabilities of intelligent systems. To address these challenges, this study proposes an expert-guided DRL mechanism for obstacle avoidance, which integrates expert knowledge into the DRL framework.

The core of the proposed Expert-Guided DRL mechanism lies in the dynamic integration of HRVO-based expert guidance with SAC-based autonomous learning. During the early training phase, the agent operates under high-exploration conditions, while safety is maintained through a heavily weighted local obstacle avoidance strategy derived from the HRVO expert. The HRVO system offers reliable safety constraints and effective collision avoidance behaviors, while the SAC agent explores optimal policies within these safe boundaries.

As training progresses and the agent’s policy mature, the influence of HRVO guidance is progressively attenuated via a decay schedule, enabling a smooth transition from expert-guided exploration to fully autonomous control. This adaptive integration is realized through a probabilistic factor that dynamically balances the contribution of HRVO expert actions and SAC-learned policies at each decision step. In high-risk scenarios or when novel obstacle configurations are encountered, the mechanism temporarily increases reliance on HRVO actions to preserve operational safety. The overall expert-guided reinforcement learning framework is depicted in Figure 7.

The computational cost of the proposed method increases with the number of vessels, N, primarily due to the growing interactions between vessels. The complexity is mainly driven by HRVO calculations and SAC updates. As the number of vessels increases, pairwise interactions between vessels grow quadratically, resulting in an increased computational load for HRVO. Simultaneously, as the state space expands, the state-action evaluations in the SAC algorithm become more computationally intensive. To address this issue, the proposed method dynamically adjusts the computational load through the decay of expert guidance and incorporates prioritized sampling in dual experience replay to minimize unnecessary state-action evaluations, prioritizing samples with higher learning value. Additionally, the proposed method models local interactions, limiting each calculation step to consider only vessels within a specific range while disregarding those farther away with a lesser threat, thus reducing the state space and number of interactions, and consequently lowering computational complexity.

This approach not only accelerates convergence and enhances sample efficiency but also ensures the agent inherits robust safety behaviors from the expert while developing autonomous decision-making capabilities. Compared to conventional DRL or purely expert-driven systems, the proposed method achieves a more effective trade-off between safety and exploration, significantly reducing oscillations and mitigating deadlock issues commonly observed in traditional methods.

4. Simulation Analysis

4.1. Environment and Parameters

This study employs the PyBullet (version 3.1.5) simulation platform and the Gym learning framework, integrated with PyTorch (version 1.8.1), to construct a stable baseline environment for DRL. Additionally, a three-dimensional visual simulation environment was developed to support the control and evaluation of USV operations. The simulation scenarios include various obstacle configurations designed to replicate real-world maritime encounters, enabling comprehensive assessments of different obstacle avoidance algorithms. This setup facilitates the evaluation of the effectiveness and robustness of learning-based obstacle avoidance strategies. The training parameters of the proposed DPER-SAC algorithm are summarized in

Table 1. Parameters of the training process.

Parameter	Value
Discount factor γ	0.99
Actor network learning rate	0.0001
Critic network learning rate	0.0001
Replay memory size	100,000
Batch size	256
Action Noise Standard Deviation	0.3
Soft update $\tau$	0.01
Training episodes	1500
Hidden nodes	128

.

4.2. Convergence Verification of Loss and Reward Functions

In DRL algorithms, the reward and loss functions are critical indicators of learning stability and convergence. The reward function provides immediate feedback to the agent, serving as the primary driver for behavior optimization. A steadily increasing and low-variance reward signal typically signifies effective policy learning and convergence toward optimal behavior.

As shown in Figure 8, the training reward trends for three algorithms—TD3 (blue), PPO (orange), and the proposed DPER-SAC (green)—are illustrated, with solid lines representing the average reward per episode. Compared to TD3 and PPO, the DPER-SAC algorithm exhibits markedly superior performance in terms of convergence speed and stability. Specifically, its reward curve rapidly rises and stabilizes at a higher level within the first 200 episodes, maintaining minimal variance throughout training. In contrast, although TD3 eventually reaches a relatively high reward, it experiences frequent sharp drops and requires more episodes to achieve stability. PPO displays a slower and more gradual improvement but converges at a lower reward level, reflecting stable yet less efficient learning behavior.

These comparative results demonstrate that the hybrid architecture of DPER-SAC effectively harnesses expert knowledge to guide early-stage exploration. This enables the agent to concentrate on high-value decision regions while maintaining strong safety constraints. The resulting balance between exploration and stability leads to faster convergence and improved final performance compared to baseline DRL algorithms. Traditional reinforcement learning algorithms for obstacle avoidance typically require 3000 episodes to converge to stable policies. In contrast, expert-guided algorithms using DPER converge within 1189 episodes, achieving a 60.37% reduction in both training time and computational load. This highlights the significant improvement in training efficiency offered by the DPER-SAC algorithm over traditional reinforcement learning methods.

In addition to reward trends, the convergence of loss functions is essential for evaluating the learning dynamics of DRL algorithms. As shown in Figure 9, the training loss curves of the actor and critic networks in the DPER-SAC framework exhibit distinct convergence patterns. The critic loss (blue) initially fluctuates due to environmental adaptation and value estimation instability but gradually decreases and stabilizes by approximately the 180th episode. In contrast, the actor loss (orange) drops sharply to around −17 before gradually rising and converging the 923th episode. These patterns indicate a successful shift from early exploration to policy refinement.

The loss trends confirm the effectiveness of the DPER-SAC architecture in achieving stable policy optimization. The integration of HRVO priors accelerates early learning, narrows the search space, and guides the agent toward long-term strategy optimization. The convergence of both networks further demonstrates the method’s robustness in dynamic environments and its compliance with maritime navigation protocols.

In terms of collision avoidance quality, the DPER-SAC framework demonstrates a decisive advantage. As illustrated in Figure 10, DPER-SAC achieves a mean score of 0.842, representing a 76.8% improvement over the 0.476 achieved by PER-SAC. Notably, DPER-SAC attains peak performance in the early training stages and converges rapidly to a stable equilibrium. In contrast, the PER-SAC baseline exhibits substantial volatility throughout the learning process with a standard deviation of 0.15. The significantly narrower fluctuation range of DPER-SAC underscores the efficacy of the dual-priority mechanism in enhancing training stability and policy robustness.

4.3. Two-Vessel Encounter Scenarios in Simulation

To simulate typical maritime encounters, the vessel configurations used in the experiment are shown in

Table 2. Parameters for two-vessel scenario testing.

Vessel	Initial Position (m)	Initial Velocity (m/s)	Vessel Radius (m)	Safety Distance (m)
Main USV	(−80,−60)	1.80	5.0	-
USV2	(−57.6, 18.3)	1.5	7.5	10.0
USV3	(59.7, −9.8)	2.0	5.0	8.0

. In the simulation tests, to assess the USV’s extreme collision avoidance capabilities in complex environments, the target vessels are modeled in a non-cooperative, predictable mode. Specifically, although some target vessels are required to give way according to COLREGs, this study assumes they take no avoidance actions and maintain their heading and speed. This configuration effectively simulates and validates the hybrid collision avoidance guidance capability of HRVO.

As shown in Figure 11 the simulation results reveal that the DPER-SAC algorithm significantly improves the USV’s performance in avoiding collisions compared to the VO algorithm. The speed comparison between the two algorithms, presented in Figure 11a, demonstrates that the DPER-SAC strategy yields smoother and more consistent velocity transitions.

This smoothness is crucial for ensuring the stability of the USV’s motion during the encounter. Additionally, the angular velocity comparison in Figure 11b further corroborates this finding, showing that the DPER-SAC strategy enables smoother and more controlled turns, whereas the VO algorithm results in more abrupt changes in angular velocity. In Figure 11c, the trajectory plot illustrates the USV’s path as it successfully avoids both obstacles (USV2 and USV3), while adhering to the reference path and reaching the goal. The plot shows that the DPER-SAC algorithm enables the USV to navigate safely and efficiently through the crossing scenario, providing superior collision avoidance and trajectory smoothness when compared to the VO algorithm.

As shown in Figure 12, the comparison of heading angle variations between the VO and DPER-SAC algorithms reveals that the latter offers a smoother and more stable response. The DPER-SAC algorithm adjusts the heading angle consistently, aligning with dynamic constraints, enhancing compliance with COLREGs, and ensuring the USV stays on course. In contrast, the VO algorithm exhibits heading angle oscillations, which can lead to inefficient maneuvers and safety risks. The DPER-SAC algorithm’s ability to balance safety and efficiency while adhering to COLREGs underscores its superior performance in autonomous collision avoidance in dynamic maritime environments.

To further assess the performance of different algorithms in practical simulations,

Table 3. Comparison of Dual-Ship Obstacle Avoidance Using DPER-SAC and VO Algorithms.

Algorithm	Average Navigation Distance (m)	Average Navigation Time (s)	Average Speed (m/s)	Average Path Curvature (deg/Step)	Maximum Single-Step Heading Change (rad)	Minimum Avoidance State Time Steps
DPER-SAC	189.74	97.69	1.92	0.53	0.0574	38 (20.0%)
VO	197.56	127.43	1.53	0.87	0.1432	145 (73.4%)

demonstrates the superiority of the DPER-SAC algorithm across several key metrics. The improved algorithm shows significantly better performance than the VO algorithm, reducing the navigation distance by 7.97 m and the navigation time by 23.4%, indicating faster execution and more efficient path planning. Additionally, the improved algorithm achieves an average speed of 1.92 m/s, which is 25.49% higher than the VO algorithm’s 1.53 m/s, enabling quicker and more effective obstacle avoidance. The average path curvature of the improved algorithm is reduced by 0.31 deg per step compared to the VO algorithm, indicating smoother and more stable navigation, demonstrating its ability to make faster and more accurate collision avoidance decisions.

To ensure statistical rigor, we conducted 200 independent test episodes for each algorithm. The resulting performance distributions are visualized through box plots using normalized metrics, as shown in Figure 13. These plots demonstrate that the DPER-SAC algorithm consistently outperforms the VO algorithm, exhibiting narrower interquartile ranges (IQRs) across most evaluated metrics, indicating enhanced stability and operational reliability. Specifically, DPER-SAC shows significantly lower median values for navigation distance, transit time, path curvature, and heading deviations, while maintaining higher median velocities compared to the VO baseline.

In conclusion, the DPER-SAC algorithm outperforms the VO algorithm in key metrics, such as navigation distance, time, and path efficiency. These improvements highlight its effectiveness in dynamic maritime environments, making it a more efficient and reliable solution for autonomous collision avoidance in two-vessel encounters.

4.4. Multi-Vessel Encounter Scenarios in Simulation

To evaluate the multi-vessel collision avoidance performance, this section presents simulations of scenarios based on the parameters in

Table 4. Parameters for multi-vessel scenario testing.

Vessel	Initial Position (m)	Initial Velocity (m/s)	Vessel Radius (m)	Safety Distance (m)
Main USV	(−80.0, −60.0)	1.80	5.0	-
USV2	(−9.2, −42.6)	1.5	7.5	10.0
USV3	(−101.0, −96.7)	1.6	5.0	8.0
USV4	(49.8, 47.3)	1.5	6.0	9.0
USV5	(−28.1, −64.2)	0.85	5.5	8.0

, which specify the initial positions, velocities, vessel radii, and safety distances for the main vessel and surrounding obstacle vessels. The collision avoidance scenario for multiple USVs also involves target vessels set in a non-cooperative, predictable mode, with no avoidance actions taken.

To assess the multi-vessel collision avoidance performance, Figure 14 presents the simulation results. In Figure 14a, the main USV performs a slight left turn to avoid the overtaking vessel, USV3, and then smoothly turns right to avoid the right-side approaching vessel, following COLREGs. The algorithm efficiently prioritizes obstacles, managing the avoidance of multiple vessels.

Figure 14 presents the complete multi-vessel avoidance trajectory. The VO algorithm, represented by the gray solid line, successfully avoids the obstacle but struggles with multiple vessels, resulting in oscillations and large turning angles. In contrast, the improved algorithm successfully avoids all vessels, demonstrating superior performance in handling complex multi-vessel interactions.

Figure 14b illustrates the head-on encounter scenario, where the main USV adjusts its heading to avoid the approaching obstacle vessel, while minimizing deceleration to maintain smooth navigation. Figure 14c shows the left-side crossing encounter, where the algorithm computes an efficient heading, ensuring the main USV safely reaches its destination.

Finally, Figure 14d provides the complete multi-vessel avoidance trajectory, showing the overall navigation path of the main USV as it interacts with multiple vessels. The comparison of these scenarios with traditional VO methods clearly indicates that the proposed approach provides more stable and efficient maneuvers, ensuring safer navigation in complex, multi-vessel environments.

To evaluate the collision avoidance performance of the DPER-SAC algorithm in multi-vessel encounter scenarios. In Figure 15, the speed profiles of both algorithms are compared. The DPER-SAC algorithm achieves smoother and more consistent speed transitions compared to the VO algorithm, which exhibits abrupt speed changes. This smoother adjustment reduces unnecessary acceleration and deceleration, thus enhancing navigation stability.

In Figure 16, the angular velocity variations for both algorithms are compared. The DPER-SAC algorithm generates more controlled and gradual angular velocity adjustments, whereas the VO algorithm shows noticeable oscillations, which can lead to instability and inefficiency. This further highlights the advantage of the DPER-SAC algorithm in maintaining heading safety.

In Figure 17, the heading angle variations over time for both algorithms are compared. The DPER-SAC algorithm adjusts the heading more smoothly, ensuring the USV follows its intended path without sharp deviations. In contrast, the VO algorithm shows steep fluctuations in heading angle, potentially leading to unpredictable navigation trajectories and posing stability risks.

As shown in

Table 5. Comparison of multi-vessel obstacle avoidance using DPER-SAC and VO algorithms.

Algorithm	Average Navigation Distance (m)	Average Navigation Time (s)	Average Speed (m/s)	Average Path Curvature (deg/Step)	Maximum Single-Step Heading Change (rad)	Minimum Avoidance State Time Steps
DPER-SAC	195.73	98.47	1.89	1.23	0.0556	76 (41.9%)
VO	221.36	145.21	1.43	3.54	0.2183	182 (76.7%)

, a comparison is made between the DPER-SAC algorithm and the VO algorithm across key performance metrics. The DPER-SAC algorithm demonstrates superior performance in several areas, significantly improving path planning and execution efficiency. Compared to the VO algorithm, the DPER-SAC algorithm reduces the navigation distance by 25.63 m and shortens the execution time by 46.74%.

Moreover, the improved algorithm significantly enhances path smoothness, reducing both the average path curvature and the maximum single-step heading change by 65.3% and 74.5%, respectively, compared to the VO algorithm. In addition, the DPER-SAC algorithm demonstrates more efficient decision-making, cutting the avoidance state time steps by 106 steps, while achieving a 32.2% increase in average speed.

Figure 18 presents a statistical summary of 200 simulation trials. In terms of navigation efficiency, DPER-SAC demonstrates significant reductions in both navigation distance and transit time, with lower median values and narrower IQRs. Regarding trajectory smoothness, the algorithm shows substantially reduced variability in path curvature and heading deviations, leading to more fluid and consistent maneuvers.

All experiments in this study were conducted in a controlled simulation environment, and the proposed method has not yet been validated with real-world AIS data or sea trials. Although simulations may not fully capture the intricate complexities of the marine environment, the modeling of vessel dynamics and environmental disturbances in this study ensures that the learned collision avoidance logic is consistent with fundamental physical laws. This consistency provides a reliable basis for the potential transition of the algorithm to real-world applications. Future research will focus on validating the robustness and cross-domain generalization of the algorithm using large-scale AIS datasets and practical sea trial verification.

5. Discussion

While the DPER-SAC framework significantly enhances trajectory smoothness and navigation efficiency in simulations, its performance in high-density maritime environments and large fleets warrants further investigation. In scenarios involving more than 10 vessels, the computational complexity of multi-agent interactions increases, potentially leading to slower decision-making and reduced efficiency in path planning and collision avoidance. Future research should focus on addressing these challenges by optimizing the algorithm for larger fleets, exploring distributed learning approaches, and improving the transition from expert-guided to autonomous control. These advancements are critical for deploying the framework in real-world, large-scale autonomous vessel operations.

6. Conclusions

In this paper, we propose a novel expert-guided framework for dynamic obstacle avoidance in USVs, integrating HRVO-based expert strategies with DPER. This approach effectively combines the stability of traditional methods with the adaptability of reinforcement learning. The proposed method accelerates convergence, improves sample efficiency, and reduces training time by 60.37%, addressing key challenges in dynamic obstacle avoidance tasks.

In summary, the DPER-SAC framework successfully bridges the gap between rule-based stability and reinforcement learning adaptability. The simulation results confirm that by integrating HRVO expert priors with a dual-priority sampling mechanism, the system effectively resolves multi-agent conflicts while optimizing trajectory quality. Beyond performance metrics, this study underscores the practical value of hybrid architectures in mitigating ‘cold-start’ risks for individual autonomous vessels. Furthermore, it provides a scalable foundation for fleet management, ensuring that large-scale autonomous traffic remains both COLREGs-compliant and operationally flexible within high-density maritime corridors.