Risk-Aware Distributional Reinforcement Learning for Safe Path Planning of Surface Sensing Agents

Abstract

In spatially constrained water domains, surface sensing agents(SSAs) must achieve safe path planning, uncertain currents, and sensor noise. We present a decentralized motion planning and collision-avoidance framework based on distributional reinforcement learning (DRL) that models the full return distribution to enable risk-aware decision making. Each surface sensing agent autonomously proceeds to its designated coordinates without rigid spatial constraints, coordinating implicitly through learned policies and a lightweight safety shield that enforces separation and kinematic limits. The method integrates (i) distributional value estimation for controllable risk sensitivity near hazards, (ii) domain randomization of sea states and disturbances for robustness, and (iii) a shielded action layer compatible with standard reactive rules (e.g., velocity obstacle-style constraints) to guarantee feasible maneuvers. In simulations across cluttered maps and stochastic current fields, the proposed approach improves success rates and reduces near-miss events compared to non-distributional RL and classical planners, while maintaining competitive path length and computation time. The results indicate that DRL-based surface sensing agent navigation is a practical path toward safe, efficient environmental monitoring and surveying.

1. Introduction

1.1. Motivation and Problem Statement

In recent years, surface sensing agents have emerged as a novel solution for civilian marine observation. Capable of unattended operation, they facilitate long-term continuous monitoring of critical parameters such as temperature, salinity, and biological characteristics. These agents offer significant advantages, including low operational costs and broad spatiotemporal coverage [1]. In complex, dynamic marine environments, enabling a decentralized surface sensing agent network to navigate safely and efficiently is both scientifically valuable and operationally critical [2,3,4]. The operational workspace is intrinsically uncertain and time-varying: multi-scale currents, waves, wind, and sporadic events generate nonstationary conditions [5,6,7]; and marine traffic introduces numerous dynamic obstacles (e.g., vessels or other surface sensing agents), while topography, submerged formations, and floating markers represent static obstacles. These factors form a multi-source, partially observed decision problem that is difficult to model precisely [8].

For a surface sensing agent network, mission success requires joint task allocation and motion coordination so that each agent reaches its goal without collisions, while maintaining system-level efficiency [9]. Each surface sensing agent must act using only sensing within a limited horizon, under unknown environmental disturbances and interactions with nearby agents. Under such decentralized, partial-information conditions, purely global plans or rigid rule sets are brittle; moreover, policies must balance safety, efficiency, and robustness in real time [10].

The core difficulty lies in coupling local perception with cooperative navigation under flow-driven dynamics. First, sensing is partial and can be noisy, leading to incomplete or delayed information. Second, the environment is nonstationary: background currents evolve and other intelligent agents adapt, making fixed priors insufficient. Third, long-horizon routing subject to collision-avoidance and safety margins is computationally demanding. Consequently, decentralized policies must learn to avoid both dynamic and static obstacles while ensuring goal reachability [5].

This work targets decentralized navigation for surface sensing agent networks. Each surface sensing agent plans and acts autonomously from local observations, avoiding dynamic obstacles (surrounding objects) and static obstacles (e.g., geological features), and reaching assigned goals without centralized coordination, while maintaining safety margins in cluttered flow fields. Additional uncertainty due to regional ocean circulation (e.g., rapid water movements) and navigational uncertainty further increase problem complexity [11].

1.2. Limitations of Existing Methods

Classical route-planning and navigation (e.g., APF, A*, Dijkstra, optimal control/MPC) perform well when sensing is complete and environments are static or piecewise static [12]. In realistic marine settings, however, dynamics are strongly time varying and stochastic, making accurate modeling and reliable long-horizon planning difficult, especially under decentralized, multi-agent coordination [1,8]. In cooperative scenarios, agents must negotiate crossings [13,14], maintain separation, and communicate under partial observability; rule-based APF-style behaviors are brittle and often fail to generalize when traffic density or flow patterns change [9,10]. These approaches typically assume precise maps and disturbance models; model errors, unmodeled interactions, and unforeseen events (e.g., uncertain obstacle motion or incidents) degrade performance and can lead to unsafe or overly conservative plans. More critically, they lack mechanisms to continuously adapt from streaming experience, so performance degrades as tasks, flow regimes, or traffic patterns shift [8].

DRL offers a data-driven alternative that can cope with uncertainty, rich interactions, and partial observability by learning policies directly from interaction. MARL with CTDE further supports cooperative behaviors such as collision avoidance, transit passage selection, and task-aware routing. Nevertheless, standard DRL brings its own challenges: sample inefficiency and unstable updates, safety during exploration, and limited generalization across scenes without careful replay, risk handling, and inductive biases. These limitations motivate hybrid solutions that combine reliable priors with adaptive learning to achieve both safety and efficiency in dynamic marine environments.

To address high–risk decision contexts, distributional RL models the full return distribution rather than only its expectation [15]. This enables policies that explicitly reason about variability and tail events, supporting risk-aware decision making and improved safety. Tail measures such as CVaR can be used to modulate risk sensitivity online, complementing safe RL by reducing catastrophic outcomes while preserving performance. In marine navigation, distributional RL is therefore a natural fit: it captures flow and interaction uncertainty, allows principled risk calibration, and provides a basis for reliable autonomous operation under partial observability [16].

Lin et al. [5,17] introduced an Adaptive Implicit Quantile Network (Adaptive IQN) for decentralized multi-agent surface navigation. By learning return quantiles, the method optimized a risk-aware policy that improved robustness without centralized coordination. The work provided an important step in bringing distributional RL to aquatic multi-agent systems and empirically demonstrated the potential for safer, more stable behavior. At the same time, challenges remained in sample efficiency, stability under dense traffic, and scalability to highly cluttered scenes—gaps that motivate the enhancements proposed in this paper.

1.3. Method and Contributions

Decentralized navigation for surface sensing agent networks is challenging due to partial observability, nonstationary flow, and tightly coupled multi-agent interactions. We build on distributional RL—specifically, Adaptive IQN—and introduce a replay mechanism that emphasizes informative experience. The core idea is to couple learning with a reliable expert that provides safe guidance in cluttered scenes: an A* global router delivers waypoints, while RVO performs local, dynamic collision avoidance. The hybrid expert generates high-quality demonstrations that are blended with on-policy data through an EMA-smoothed prioritized replay (EMA-PER), enabling Adaptive IQN to learn risk-aware policies more efficiently and stably.

• Main contributions.
  • We propose an EMA-smoothed prioritized replay scheme that moves priority toward the exponential moving average of TD errors. This reduces sampling noise, preserves diversity, and focuses updates on transitions with high learning value. EMA-PER is integrated directly into Adaptive IQN.
  • We develop a mixed expert that combines A* global routing with RVO local avoidance to produce safe, feasible demonstrations. A* plans waypoints under a grid map, and RVO resolves short-horizon interactions with other surface sensing agents, yielding trajectories that respect collision constraints.
  • We extend EMA-PER to jointly schedule sampling from expert demonstrations and self-generated experience. By gradually discounting early expert bias while preserving informative expert segments, MEA-PER accelerates early learning and improves final robustness.
  • We deliver a complete training and execution pipeline that fuses the above components with Adaptive IQN’s risk-sensitive evaluation, producing policies that maintain safety and efficiency across increasing agent counts and environmental complexity.

The approach exploits complementary strengths: classical planning (A*, RVO) contributes feasibility and safety priors, while distributional RL adapts to uncertain, time-varying flows. By guiding learning toward reliable and informative transitions (EMA-PER/MEA-PER), the framework reduces unsafe exploration, improves sample efficiency, and yields higher-quality policies than either RL-only or planner-only baselines.

2. Related Work

2.1. Decentralized Multi-Surface Sensing Agent Path Planning

Multi-agent path planning (MAPP) within multi-robot systems (MRSs) faces well-known challenges in the surface sensing agent domain, including cluttered environments, communication limits, and the need for concurrent collision avoidance. Decentralization—each agent planning locally from onboard sensing—enables scalable coordination for large surface sensing agent teams and aligns with practical deployment constraints [13]. Under such settings, the network’s collective behavior (e.g., reaching assigned goals) emerges from local policies and interactions rather than a centralized planner, which improves robustness to failures and delays [6,18]; while prominent CTDE algorithms such as MADDPG [19] and MAPPO [20] utilize global joint states for training, they are incompatible with our communication-constrained setting where only local sensor-level observations are available. Consequently, to ensure a fair comparison under identical informational constraints, we benchmark against strictly decentralized independent learners (APF, DQN, IQN) rather than centralized approaches.

2.2. Distributional RL in Robotics

Distributional RL learns the full return distribution instead of only the expectation, yielding policies that better reason about uncertainty and rare events [5]. Implicit Quantile Networks (IQNs) implement this idea by learning a parametric quantile function of the return [14]. Architectures typically comprise a state encoder, a quantile embedding, and a value head; Adaptive IQN further modulates risk sensitivity online, for example via CVaR-style quantile selection [3,4]. Recent work has applied Adaptive IQN with CVaR to navigation problems, demonstrating potential gains in safety and robustness [10,21]. Notable examples include utilizing privileged information for efficient UAV policy training [22] and employing deep distributional RL for real-time local path planning [23]. However, the application of these distributional approaches to decentralized multi-agent systems operating without privileged information remains less explored, presenting unique challenges in coordination and credit assignment.

2.3. Experience Replay

Experience replay is a core DRL technique: by storing and reusing past transitions, it improves sample efficiency and stability. Standard replay samples uniformly, which can under-represent rare but informative events [24]. Prioritized Experience Replay (PER) addresses this by sampling transitions with probability proportional to their temporal-difference (TD) error, while correcting the induced bias using importance sampling weights [25]. PER has been shown to accelerate learning across domains—including navigation—by focusing updates on high-value experience [6].

2.4. Hybrid Path–Planning

Hybrid schemes combine a global planner with a local planner, leveraging the complementary strengths of both. Global planners such as A* compute shortest or near-shortest routes on a grid map with static obstacles [21]. Local planners, e.g., ORCA/RVO for reciprocal collision avoidance or DWA for dynamic feasibility, react to short-horizon interactions and moving obstacles [26,27]. In a hybrid pipeline, the global planner provides a safe, topologically valid route, while the local planner executes the motion and resolves real-time interactions, yielding reliable navigation in cluttered environments [12].

2.5. Learning from Demonstrations and Expert Guidance

Learning from demonstrations (LfD) accelerates RL by supplying informative trajectories that improve sample efficiency and exploration safety [2]. A common practice is to insert expert transitions into the replay buffer so they are learned off-policy together with self-collected experience [28]. In our setting, A* supplies global waypoints and RVO produces locally feasible controls, generating expert tuples $(s,a,r,s^{\prime })$ aligned with the task objectives. To exploit these data effectively, we adopt an EMA-smoothed prioritized replay (EMA-PER): TD-error priorities are exponentially averaged to reduce noise [29], and expert transitions are initially emphasized and then gradually discounted. This balances imitation and exploration—providing a strong early learning signal without over-fitting to expert behavior—and guides the agent toward safer, higher-value updates [30].

3. Methodology

3.1. System Overview

We study a group of N surface sensing agents operating in a 2D operational workspace to perform route planning and navigation. Agent i starts from $p_{\mathrm{start}}^i$ and aims to reach $p_{\mathrm{goal}}^i$. Control is decentralized: each agent selects actions using only local observations. No fixed structure is needed; agents travel independently toward their goals. The environment includes background currents and obstacles, with geological features modeled as static obstacles and other agents acting as dynamic obstacles.

3.2. Kinematic Model

For tractability, each agent is modeled as a point with pose $(x,y,\theta )$, where $(x,y)$ are global Cartesian coordinates and $\theta$ is the heading. The agent velocity is decomposed into an ambient current and self-generated motion:

$${\displaystyle \frac{d\mathbf{X}\left(t\right)}{dt}}={\mathbf{V}}_{\mathrm{cur}}\left(\mathbf{X}\left(t\right)\right)+{\mathbf{V}}_{\mathrm{ctrl}}\left(t\right).$$

(1)

Here ${\mathbf{V}}_{\mathrm{cur}}$ is the Eulerian current field evaluated at the agent location and ${\mathbf{V}}_{\mathrm{ctrl}}$ is the commanded translational velocity produced by the navigation policy and steering constraints.

3.3. Ocean Current and Obstacles

The flow field is represented by a Rankine vortex, which comprises a solid-body core and an irrotational exterior. In the core ($r\le a$), fluid rotates with constant angular velocity $\mathrm{\Omega }=\mathrm{\Gamma }/\left(2\pi a^2\right)$, yielding a tangential speed:

$$v_{\theta }\left(r\right)=\mathrm{\Omega }r={\displaystyle \frac{\mathrm{\Gamma }}{2\pi a^2}}r.$$

(2)

Outside the core ($r>a$), the flow is irrotational with

$$v_{\theta }\left(r\right)={\displaystyle \frac{\mathrm{\Gamma }}{2\pi r}}.$$

(3)

Equivalently,

$$v_{\theta }\left(r\right)={\displaystyle \frac{\mathrm{\Gamma }}{2\pi }}\left\{\begin{array}{cc}{\displaystyle \frac{r}{a^2}},\hfill & r\le a,\hfill \\ {\displaystyle \frac{1}{r}},\hfill & r>a.\hfill \end{array}\right.$$

(4)

Static obstacles (geological features) are represented as impenetrable regions; other agents are treated as dynamic obstacles in the planning layer.

3.4. Reinforcement-Learning Formulation

We cast decentralized navigation as a decentralized partially observable Markov decision process. Each surface sensing agent learns a mapping from its local observation to a continuous action that maximizes the discounted return.

3.4.1. Observation Design

At step t, the observation concatenates three parts:

$$\begin{array}{cc}\hfill s_{\mathrm{self}}^t& =[p_x^{\mathrm{goal}},p_y^{\mathrm{goal}},v_x,v_y],\hfill \end{array}$$

(5)

$$\begin{array}{cc}\hfill s_{\mathrm{static}}^t& =[o_1^r,\dots ,o_m^r],o_i^r=[p_i^{sx},p_i^{sy},r_i^s],\hfill \end{array}$$

(6)

$$\begin{array}{cc}\hfill s_{\mathrm{dynamic}}^t& =[o_1^d,\dots ,o_n^d],o_i^d=[p_i^{dx},p_i^{dy},v_i^{dx},v_i^{dy}].\hfill \end{array}$$

(7)

Candidates are sorted by distance to the ego agent; if fewer than m or n are detected, zero padding keeps a fixed input size.

3.4.2. Action Space and Kinematics

The action is ${\mathbf{a}}_t=(a_t,{\omega }_t)$, where $a_t$ is the commanded tangential acceleration and ${\omega }_t$ is the commanded turn-rate. The forward speed is constrained to $[0,v_{max}]$, and $(a_t,{\omega }_t)$ respect platform limits.

3.4.3. Task Objective and Termination

An episode terminates upon (i) goal arrival with $\parallel p_{\mathrm{goal}}-p_t\parallel <d_{\mathrm{arrive}}$, (ii) collision with any obstacle or another agent when the distance is below $d_{\mathrm{collide}}$, or (iii) exhaustion of the step budget.

3.4.4. Reward Shaping

Our surface sensing agent framework is illustrated in Figure 1. The objective is to learn a policy $\pi \left(\mathbf{s}\right)$ that minimizes travel time while avoiding collisions with static obstacles and other agents. A collision is declared when the Euclidean distance between a surface sensing agent and any static obstacle or another surface sensing agent is less than $d_{\mathrm{collision}}$. Arrival is declared when the Euclidean distance to the goal is less than $d_{\mathrm{arrive}}$.

Figure 1. Decision framework for a surface sensing agent. Local inputs comprise the ego state, the nearest static obstacles, and nearby moving obstacles. An interaction module encodes agent–environment relations and feeds a distributional value head that predicts return distributions. A risk-sensitivity setting selects the evaluation quantile, and the action with the maximal selected value is executed ($argmax$). The procedure is decentralized and runs independently on each agent.

Action. At each time step the surface sensing agent executes ${\mathbf{a}}_t=(a_t,{\omega }_t)$, where $a_t$ is the rate of change of the magnitude of ${\mathbf{V}}_S\left(t\right)$ and ${\omega }_t$ is the rate of change of its heading. The policy is written as $\pi :\mathbf{s}\to (a,\omega )$, and the forward speed is constrained to $[0,v_{max}]$.

Reward. The reward at time t is computed as

$$r_t=r_{\mathrm{step}}+r_{\mathrm{forward},t}+{{\displaystyle \cancel{⊩}}}_{\mathrm{collision}}\left(s_t\right)·r_{\mathrm{collision}}+{{\displaystyle \cancel{⊩}}}_{\mathrm{reach}\_\mathrm{goal}}\left(s_t\right)·r_{\mathrm{goal}},$$

(8)

where $r_{\mathrm{step}}$ is a fixed step penalty with value $-1.0$ (a negative reward that encourages the agent to reach the goal quickly so as to reduce cumulative penalties). $r_{\mathrm{forward},t}$ is a forward-progress reward computed as $\parallel p_{\mathrm{goal},t}-p_{\mathrm{goal},t-1}\parallel$, which rewards moving toward the goal between two consecutive time steps. ${{\displaystyle \cancel{⊩}}}_{\mathrm{collision}}\left(s_t\right)$ is an indicator that equals 1 if a collision occurs at state $s_t$ and 0 otherwise, with $r_{\mathrm{collision}}=-50.0$. ${{\displaystyle \cancel{⊩}}}_{\mathrm{reach}\_\mathrm{goal}}\left(s_t\right)$ is an indicator that equals 1 if the goal is reached at state $s_t$ and 0 otherwise, with $r_{\mathrm{goal}}=100.0$.

Each agent optimizes its own objective, yet coherent group behaviour emerges from local policies and their interactions. System success under currents and disturbances depends on decentralized coordination rather than a central dispatcher. We use an A*–RVO expert as a purely decentralized heuristic to shape interactions during training. If every agent maintains safety and efficiency locally, group navigation succeeds as an emergent property of the ensemble.

To align with sparse sensing and the fixed-size requirements of neural inputs, we vectorize perceived context by sorting candidates by distance and retaining at most m static obstacles and n moving agents; zero padding yields a consistent input. This keeps dimensionality low and prioritizes safety-critical context while reducing parameter count and training variance. The design, however, imposes a perceptual horizon: information beyond the top-$m/n$ may be omitted. In practice this trade-off between capability and computability is mitigated by (i) distance-based ordering, which biases inputs toward imminent hazards; and (ii) conservative fallback logic (A*–RVO).

The adopted reward function weights $r_{step}=-1.0$, $r_{collision}=-50.0$, and $r_{goal}=100.0$ in this paper are determined by combining domain knowledge of multi-surface sensing agent navigation and iterative validation experiments. Among them, the collision penalty term $r_{collision}=-50.0$ is calibrated to avoid a collision rate > 15% in dense scenarios when $r_{collision}=-20.0$, and prevent overly conservative behavior with a $30\%$ increase in travel time when $r_{collision}=-80.0$; the goal reward term $r_{goal}=100.0$ is twice the maximum cumulative step penalty of a typical successful episode, strongly incentivizing task completion while ensuring safety; the step penalty term $r_{step}=-1.0$ balances navigation efficiency and exploration behavior with a moderate negative weight, avoiding rushing toward the goal and ignoring obstacles or aimless wandering.

The impact of weights is quantitatively analyzed through 30 independent trainings with different random seeds: increasing the absolute value of $r_{collision}$ accelerates the convergence of collision avoidance behaviors but tends to cause conservatism; the agent’s motivation toward the goal decreases when $r_{goal}<80.0$, with the stagnation rate rising to $12\%$, while the collision rate increases by 8–10% when $r_{goal}>120.0$; and $r_{step}<-1.0$ shortens training episodes but impairs the ability to explore paths in complex environments, whereas non-negative weights double the average travel time. The finally selected weights achieve an optimal balance, at a $98.2\%$ success rate and $0.3\%$ collision rate, with near-minimal travel time and energy consumption.

3.5. Framework for Distributional RL-Based Decentralized Navigation and Collision Avoidance

3.5.1. Adaptive Implicit Quantile Network

The core policy module is an Adaptive Implicit Quantile Network (Adaptive IQN), a distributional reinforcement-learning architecture that models the full return distribution for each action. Risk sensitivity is controlled by selecting evaluation quantiles (including CVaR-oriented thresholds), enabling explicit trade-offs between efficiency and caution under environmental uncertainty. The network embeds observations with cosine features and fully connected layers, applies rectified-linear activations, and—given an observation and a chosen quantile—outputs action–return distributions, as summarized in Figure 2.

Figure 2. Architecture of the Adaptive Quantile Planner (AQP). The policy uses an Adaptive IQN head to predict action–return distributions; risk sensitivity is applied via quantile/CVaR selection. The symbol * denotes the multiplication of the sampled quantiles by the CVaR threshold.

3.5.2. Implicit Quantile Network (IQN)

• Distributional view.

Instead of modeling only the expected return, distributional reinforcement learning treats the return as a random variable [29,31,32]. Let $Z^{\pi }(s,a)$ denote the return under policy $\pi$. Its one-step relation is

$$Z^{\pi }(s,a)\stackrel{D}{=}R(s,a)+\gamma Z^{\pi }\left(s^{\prime },a^{\prime }\right),a^{\prime }\sim \pi (·|s^{\prime }),$$

(9)

where $\stackrel{D}{=}$ indicates equality in distribution, and $R(s,a)$ is the immediate-reward distribution.

• Quantile parameterisation.

IQN represents the return distribution through its continuous quantile function,

$$Z_{\theta }(s,a;\tau )\approx F_{Z^{\pi }(·|s,a)}^{-1}\left(\tau \right),\tau \in (0,1),$$

(10)

with three components: (i) an observation encoder $\psi \left(s\right)$ that maps the raw input (ego state, static obstacles, moving obstacles) to a fixed-dimensional embedding via fully connected layers; (ii) a quantile embedding $\varphi \left(\tau \right)$; and (iii) a head that combines $\psi \left(s\right)$ and $\varphi \left(\tau \right)$ to predict action–quantile values.

• Quantile features and fusion.

Risk preference is injected by a monotone transform $f(\tau ;\phi )$ applied to samples $\tau \sim \mathcal{U}(0,1)$, where $\phi \in (0,1]$ can be interpreted as a CVaR-style threshold. The transformed quantile is embedded with a cosine basis followed by a linear–ReLU projection:

$$\varphi \left(\tau \right)=\mathrm{ReLU}\left(cos\left(\pi ·0·f(\tau ;\phi )\right),cos\left(\pi ·1·f(\tau ;\phi )\right),\dots ,cos\left(\pi ·63·f(\tau ;\phi )\right)\right).$$

(11)

We fuse state and quantile features by a Hadamard product and a small multilayer head:

$$q_{\theta }(s,a,\tau )=\mathrm{FC}\left(\psi \left(s\right)\odot \varphi \left(\tau \right)\right),$$

(12)

which yields the predicted quantile value for action a at level $\tau$.

• Learning objective.

Training minimises the quantile Huber loss using N = $N^{\prime }$ = 8 quantile samples ${\tau }_i,{\tau }_j^{\prime }\sim \mathcal{U}(0,1)$ and parameter $\kappa$ = 1.0:

$$L\left(\theta \right)={\displaystyle \frac{1}{N{N}^{\prime }}}\sum _{i=1}^N\sum _{j=1}^{N^{\prime }}{\rho }_{{\tau }_i}^{\kappa }\left({\delta }^{{\tau }_i,{\tau }_j^{\prime }}\right),{\delta }^{{\tau }_i,{\tau }_j^{\prime }}=r_t+\gamma Z_{{\theta }^{-}}^{{\tau }_j^{\prime }}\left(s_{t+1},\pi \left(s_{t+1}\right)\right)-Z_{\theta }^{{\tau }_i}\left(s_t,a_t\right),$$

(13)

where ${\theta }^{-}$ denotes a target network. This objective aligns predicted quantiles with the distributional Bellman target while retaining robustness to outliers.

3.5.3. Adaptive Risk Sensitivity

We use an adaptive risk mechanism guided by CVaR. Operationally, we bias evaluation toward the lower tail by shrinking the sampled quantile:

$${\tau }^{\prime }=\varphi \tau .$$

(14)

The scaling $\varphi$ depends on proximity to obstacles (static or moving). Let X be the agent position and $\left\{X_o\right\}$ the obstacle set. With the clearance

$$mind(X,\left\{X_o\right\})=\underset{o}{min}\parallel X-X_o\parallel ,$$

and a distance threshold $d_0>0$, we set

$$\varphi =\left\{\begin{array}{cc}{\displaystyle \frac{mind(X,\left\{X_o\right\})}{d_0}},\hfill & \mathrm{if}mind(X,\left\{X_o\right\})\le d_0,\hfill \\ 1.0,\hfill & \mathrm{if}mind(X,\left\{X_o\right\})>d_0.\hfill \end{array}\right.$$

(15)

Far from obstacles (>$d_0$) we keep $\varphi$ = 1 to favour efficiency; close to obstacles, $\varphi$ decreases smoothly toward zero, shifting evaluation to more conservative quantiles and promoting avoidance without altering the underlying policy class.

3.6. Prioritized Replay with EMA Smoothing

• Motivation and data structure.

Rather than drawing minibatches uniformly, we bias sampling toward transitions that appear most informative for policy improvement [2,28,31]. Transitions $(s_t,a_t,r_t,s_{t+1})$ are stored in a replay memory $\mathcal{D}$ together with a nonnegative priority value. To support fast updates and priority-proportional draws, we maintain a binary accumulator tree (often called a “sum tree”): leaf nodes hold individual priorities; internal nodes cache partial sums; and the root stores the total mass, enabling $O(logn)$ inserts and queries.

• Initial priority.

When a new transition arrives, its priority is seeded from the current temporal-difference (TD) error and a small floor $\epsilon >0$ to keep every item eligible:

$$p_t=\left|{\delta }_t\right|+\epsilon .$$

(16)

• Sampling rule.

Let $\alpha \in [0,1]$ tune the strength of prioritisation. The probability of drawing index t is

$$P\left(t\right)={\displaystyle \frac{p_t^{\alpha }}{{\sum }_k{p}_k^{\alpha }}},$$

(17)

which reduces to uniform sampling when $\alpha$ = 0.

• Stabilising updates via EMA.

After a sampled item is replayed, its priority is refreshed using an exponential moving average (EMA) of TD errors. This limits abrupt changes caused by noisy single-step estimates:

$$p_{\mathrm{new}}=\gamma p_{\mathrm{old}}+(1-\gamma )\left(|{\delta }_{\mathrm{new}}|+\epsilon \right),$$

(18)

with smoothing factor $\gamma \in (0,1)$. Large $\gamma$ yields slower, steadier adaptation that leverages historical signals; small $\gamma$ reacts quickly to newly informative samples. This EMA variant is the only modification to otherwise standard prioritized replay and is used throughout our experiments.

Bias Correction via Importance Sampling

Nonuniform replay introduces sampling bias. We correct it with importance-sampling (IS) weights computed from the sampling probability $P\left(t\right)$ and buffer size $N_{\mathrm{buf}}$:

$$w_t={\left(\frac{1}{N_{\mathrm{buf}}P\left(t\right)}\right)}^{\beta },$$

(19)

where $\beta \in [0,1]$ controls the strength of the correction. We anneal $\beta$ from an initial value ${\beta }_0$ toward 1 during training so that early updates are less noisy and later updates become unbiased.

For numerical stability, the weights in a minibatch can be normalized by their maximum, ${\tilde{w}}_t=w_t/{max}_{t^{\prime }\in \mathcal{B}}{w}_{t^{\prime }}$. The PER loss uses these weights to scale the base loss $L\left(\theta \right)$ (e.g., the quantile Huber objective):

$$L_{\mathrm{PER}}\left(\theta \right)={\displaystyle \frac{1}{\left|\mathcal{B}\right|}}\sum _{t\in \mathcal{B}}{\tilde{w}}_t{L}_t\left(\theta \right).$$

(20)

This restores the correct gradient direction under prioritized sampling while keeping updates well-conditioned.

3.7. Hybrid Expert Planner (A* and RVO)

To supply reliable demonstrations for RL, we use a hybrid planner that couples A* for global routing with RVO for local dynamic avoidance. The modules run per agent in a decentralized manner.

3.7.1. Global Path Planning

A* searches on a known map containing only static obstacles and generates a waypoint sequence $P_{\mathrm{global}}=\{{\mathrm{WP}}_1,{\mathrm{WP}}_2,\dots ,{\mathrm{WP}}_{\mathrm{goal}}\}$. The path avoids all known static obstacles and provides long-horizon guidance from the current position to the goal.

3.7.2. RVO-Based Local Collision Avoidance

RVO operates in a local neighborhood to resolve interactions with moving agents while accounting for nearby static boundaries. The preferred velocity ${\mathbf{v}}_{\mathrm{pref}}$ points toward the next waypoint ${\mathrm{WP}}_{\mathrm{next}}$; RVO selects a new command ${\mathbf{v}}_{\mathrm{new}}$ that is as close as possible to ${\mathbf{v}}_{\mathrm{pref}}$ while remaining outside all reciprocal velocity–obstacle regions, thereby ensuring collision-free motion.

3.7.3. Expert Trajectory Generation

At each decision step, the A* module supplies the current waypoint on the guidance path. Using this waypoint and local context (especially the states of nearby agents), the RVO module computes the velocity to be executed in the next time step(Algorithm 1). From the current state $s_t$, RVO yields the expert action $a_t^{\mathrm{expert}}$, the reward $r_t$, and the next state $s_{t+1}$. The resulting transition $(s_t,a_t^{\mathrm{expert}},r_t,s_{t+1})$ is stored in the replay buffer and used alongside trajectories collected by the learning policy.

Algorithm 1 Hybrid Expert (A* global routing + RVO local avoidance)—Trajectory Generator

Require: start $s_0$, goal g, static_map, init_vel, all_agents_states

Ensure: expert trajectory $\mathcal{T}=\left[(s,a,r,s^{\prime })\right]$

$\mathcal{T}\leftarrow \left[\right]$, $\mathbf{x}\leftarrow s_0$, $\mathbf{v}\leftarrow$ init_vel

$P_{\mathrm{global}}\leftarrow$ compute global path from $s_0$ to g on static_map using A* search

while  $\parallel \mathbf{x}-g\parallel \ge \mathrm{GOAL}\_\mathrm{THRESHOLD}$  $t<\mathrm{MAX}\_\mathrm{STEPS}$  do

if $\parallel \mathbf{x}-{\mathrm{WP}}_k\parallel <\mathrm{WAYPOINT}\_\mathrm{ARRIVAL}\_\mathrm{THRESHOLD}$ then

$k\leftarrow min(k+1,|P_{\mathrm{global}}|-1)$

end if

${\mathbf{v}}_{\mathrm{pref}}\leftarrow \mathrm{MAX}\_\mathrm{SPEED}·\mathrm{NORMALIZE}({\mathrm{WP}}_k-\mathbf{x})$

$\mathcal{N}\leftarrow$ sense nearby agents within perception range of $\mathbf{x}$ from all_agents_states

$\mathbf{u}\leftarrow$ compute collision-free velocity using RVO with current velocity $\mathbf{v}$, preferred velocity ${\mathbf{v}}_{\mathrm{pref}}$, and nearby agents $\mathcal{N}$; $\mathbf{u}\leftarrow \mathrm{CLIPTOLIMITS}\left(\mathbf{u}\right)$

${\mathbf{x}}^{\prime }\leftarrow \mathbf{x}+\mathbf{u}\mathrm{\Delta }t$; ${\mathbf{v}}^{\prime }\leftarrow \mathbf{u}$

$r\leftarrow \mathrm{CALCULATEREWARD}(\mathrm{state}(\mathbf{x},\mathbf{v}),\mathbf{u},{\mathbf{x}}^{\prime })$

$s^{\prime }\leftarrow \mathrm{FORMULATERLSTATE}({\mathbf{x}}^{\prime },{\mathbf{v}}^{\prime },\mathrm{SENSENEARBYAGENTS}({\mathbf{x}}^{\prime },\mathrm{all}\_\mathrm{agents}\_\mathrm{states}),g)$

Append$((s,\mathbf{u},r,s^{\prime }),\mathcal{T})$; $(\mathbf{x},\mathbf{v},s,t)\leftarrow ({\mathbf{x}}^{\prime },{\mathbf{v}}^{\prime },s^{\prime },t+1)$

end while

return  $\mathcal{T}$

The expert trajectory generator and training workflow operate in tandem (Figure 3, Algorithm 2). The expert combines a global waypoint plan from A* with local collision avoidance from RVO: at each step, the current waypoint defines a preferred motion, nearby agents and obstacles constrain feasible velocities, and RVO outputs a safe control that advances the agent while respecting speed limits. The resulting transition $(s_t,a_t^{\mathrm{expert}},r_t,s_{t+1})$ is written to the replay buffer exactly like data collected by the learning policy. During training, the buffer is sampled with prioritized replay so that informative events—expert demonstrations, near-collision interactions, and high-TD-error moments—are seen more often. The policy network (Adaptive IQN) learns from full return distributions rather than scalar values, and its updates are driven by TD errors that also refresh sampling priorities. This division of roles yields a stable and sample-efficient loop: the expert supplies a feasible, safety-oriented backbone in cluttered flows; prioritized replay concentrates gradient effort where it matters; and the distributional policy preserves efficiency while reducing variance and collision risk as task complexity grows.

Algorithm 2 AQP Training with Prioritized Replay (PER)

Require: simulator, replay capacity C, batch size B, learn_freq, target_period, total_steps; $\alpha$ (priority exponent), ${\beta }_0,{\beta }_1$ (IS anneal); ${ϵ}_{max},{ϵ}_{min}$ (exploration)

Ensure: trained policy ${\pi }_{\theta }$

initialize replay $\mathcal{M}$ with capacity C; initialize Adaptive IQN parameters $\theta$; set target ${\theta }^{-}\leftarrow \theta$

reset simulator; obtain initial states $\left\{s_i\right\}$

(optional seed) insert expert transitions into $\mathcal{M}$ with high priority

for $t=1$ total_steps do

$ϵ\leftarrow \mathrm{linear}\_\mathrm{anneal}(t,\mathrm{total}\_\mathrm{steps},{ϵ}_{max},{ϵ}_{min})$

// Agent Interaction with Environment

for each agent i do

choose $a_i$ by $ϵ$-greedy w.r.t. ${\pi }_{\theta }\left(s_i\right)$; step simulator $\to (r_i,s_i^{\prime })$

push $(s_i,a_i,r_i,s_i^{\prime })$ into $\mathcal{M}$ with initial priority $p\leftarrow \left|\delta \right|+\epsilon$ ($\delta$: initial TD error)

$s_i\leftarrow s_i^{\prime }$

end for

if $tmod\mathrm{learn}\_\mathrm{freq}=0$$\left|\mathcal{M}\right|\ge B$ then

// Sample Prioritized Batch

sample indices ${\left\{j\right\}}_1^B$ from $\mathcal{M}$ with $P\left(j\right)=p_j^{\alpha }/{\sum }_k{p}_k^{\alpha }$

$\beta \leftarrow \mathrm{linear}\_\mathrm{anneal}(t,\mathrm{total}\_\mathrm{steps},{\beta }_0,{\beta }_1)$;   $w_j\leftarrow {\left(1/\left(\right|\mathcal{M}\left|P\right(j\left)\right)\right)}^{\beta }$;   ${\tilde{w}}_j\leftarrow w_j/{max}_{k\in \left\{j\right\}}{w}_k$

compute distributional targets with ${\theta }^{-}$; evaluate predicted quantiles with $\theta$

$\mathcal{L}\leftarrow {\displaystyle \frac{1}{B}}{\sum }_j{\tilde{w}}_j·\mathrm{QuantileHuber}\left(Z_{\theta }(s_j,a_j),Z_{{\theta }^{-}}^{-}\left(s_j^{\prime }\right)\right)$

update $\theta \leftarrow \theta -\eta {\nabla }_{\theta }\mathcal{L}$

refresh priorities $p_j\leftarrow \mathrm{EMA}\left(p_j,\left|{\delta }_j\right|+\epsilon \right)$ in $\mathcal{M}$

if $tmod\mathrm{target}\_\mathrm{period}=0$ then

${\theta }^{-}\leftarrow \theta$

end if

end if

if episode terminated or timeout then

reset simulator; reinitialize $\left\{s_i\right\}$

end if

end for

return  ${\pi }_{\theta }$

Figure 3. Workflow of the proposed training framework. The A*–RVO expert produces demonstration transitions $(s_t,a_t,r_t,s_{t+1})$, which are stored in a replay buffer together with on-policy experience. Minibatches are drawn by prioritized sampling and fed to the Adaptive IQN. The policy interacts with the simulator to choose $a_t$, while TD errors both drive gradient updates and refresh sampling priorities.

4. Experiments

4.1. Simulation Setup

We implement a custom Python simulator for a planar aquatic domain, built on Python 3.8 and relying on Gymnasium 0.29.1 for environment construction and Pygame 2.5.2 for rendering. The map size is $50\times 50\mathrm{m}$. Static obstacles, when present, are randomly placed in circular disks with radii of $1\mathrm{m}$. Each surface sensing agent has a sensing range of $15\mathrm{m}$ with a ${360}^{\circ }$ field of view. Background flow is modeled by Rankine vortices with core radii of $0.5\mathrm{m}$.

The Rankine vortex used in this study is not intended to represent any specific hydrodynamic phenomenon. Instead, it serves as an abstract but structured flow field designed to introduce coherent circulation, smooth velocity gradients, and non-uniform flow patterns. These characteristics are broadly representative of oceanic dynamics, while the model remains computationally lightweight and highly reproducible. Thus, the objective of the flow model is not high-fidelity physical simulation, but to provide an environment with realistic variability for evaluating decentralized multi-agent navigation.

We evaluate our proposed approach and compare it against Adaptive IQN, IQN, DQN, and APF. The first suite trains surface sensing agent to avoid dynamic obstacles only; static obstacles are absent. We use 500 randomly generated scenarios (100 per scene type). The second suite uses another 500 evaluation scenarios constructed in the same way but with additional static obstacles. The detailed parameter settings for these scenarios are summarized in Table 1. An episode is marked as a failure if not all surface sensing agents reach their goals within 3 min. The energy cost of each surface sensing agent is computed as the time–sum of action magnitudes over its trajectory.

Table 1. Simulation scenario settings.

Set 1 (No Static Obstacles)				Set 2 (With Static Obstacles)
	# SSAs	#Vortices	#Obstacles		# SSAs	#Vortices	#Obstacles
Scene 1	3	4	0	Scene 1	3	4	4
Scene 2	4	5	0	Scene 2	4	5	5
Scene 3	5	6	0	Scene 3	5	6	6
Scene 4	6	7	0	Scene 4	6	7	7
Scene 5	7	8	0	Scene 5	7	8	8

All algorithms were trained on an NVIDIA RTX 3090 GPU with an Intel Core i9-13900K CPU. The hardware platform was equipped with 128 GB DDR5 RAM and ran 64-bit Ubuntu 22.04 LTS. The proposed innovation framework required approximately 18 h of training for convergence, while adaptive-IQN and standard IQN required 14 h and 11 h, respectively. The DQN baseline converged in 6 h, and the APF method does not require learning-based training. The average training speed was around 0.35 s per episode, and the final performance results were obtained after full convergence.

The additional training cost of our method mainly stems from (i) the multi-agent coordination module and (ii) the distributional value estimation. Despite this overhead, the overall training demand remains within a practical range for modern GPU hardware. To further assess the practical feasibility of the proposed approach, we analyzed the computational complexity and empirical runtime during the inference phase. The per-step complexity is dominated by the forward pass of the multi-agent distributional critic, scaling as $O(N·P)$, where N is the number of agents and P represents the network parameters. In terms of execution speed, our method achieves an average latency of 2.6 ms per decision step. Although this introduces a modest increase compared to Adaptive-IQN (1.9 ms), IQN (1.7 ms), DQN (1.1 ms), and APF (0.2 ms), as summarized in Table 2, the latency remains well within real-time constraints. Furthermore, we evaluated the scalability of the algorithm. The inference latency grows approximately linearly, increasing from 2.6 ms for three agents to 4.1 ms for seven agents. This mild slope indicates that the proposed policy architecture introduces minimal overhead as the number of agents increases, ensuring efficient deployment even in complex multi-agent scenarios.

Table 2. Comparison of average inference latency per decision step across different algorithms.

Algorithm	Average Latency (ms)	Real-Time Capable
APF	0.2	Yes
DQN	1.1	Yes
IQN	1.7	Yes
Adaptive-IQN	1.9	Yes
Proposed	2.6	Yes

Unless otherwise noted, we train Adaptive IQN with $N=N^{\prime }=8$ quantile samples and the quantile Huber loss with $\kappa =1.0$. Risk is handled via a CVaR-style quantile scaling with base threshold $\varphi =0.1$. Prioritised replay uses exponent $\alpha =0.6$, initial IS exponent ${\beta }_0=0.4$ linearly annealed to $0.1$, a small priority bias ${\epsilon }_{\mathrm{PER}}={10}^{-3}$, and EMA smoothing factor $\gamma =0.5$ for priority updates. The A* router employs Euclidean edge costs with a $1\mathrm{m}$ grid resolution. RVO parameters are an agent radius of $0.5\mathrm{m}$, at most 5 neighbors, an agent time horizon of $5\mathrm{s}$, and an obstacle time horizon of $2\mathrm{s}$. Optimization uses Adam with a learning rate of $1\times {10}^{-4}$, replay capacity of $1\times {10}^5$, discount factor of $0.99$, and 64 mini-batches; the target network is updated every 1000 environment steps. Exploration follows $ϵ$-greedy with $ϵ$ linearly decayed from $0.6$ to $0.05$ over training.

4.2. Results Discussion

Figure 4 illustrates six representative flow–obstacle scenes and the resulting surface sensing agent trajectories. (a) Four surface sensing agents with well-separated goals in a weakly sheared region produce near-straight motions punctuated by minor curvature induced by the background flow; all agents avoid purple static obstacles with large clearance. (b) With stronger vortical features, paths elongate and bend around eddies; agents execute coordinated detours while preserving mutual separation as they transit a cluttered central zone. (c) Goals placed near a vortex core tighten curvature and evoke conservative speed modulation: trajectories wrap partially around the core before converging, demonstrating risk-aware behavior in high-shear pockets. (d) In the five-SSA case with roughly symmetric start–goal placements, intersecting corridors are resolved by local interaction rules (RVO), yielding temporal staggering and side-step maneuvers that prevent head-on encounters. (e) A narrow passage between obstacles causes the agents’ trajectories to funnel into a single file. As they approach the bottleneck, their paths show brief hesitation and local congestion effects, after which the trajectories straighten once the agents exit the constricted region. (f) Under dense clutter and stronger circulation, the agents produce multi-turn curved paths around obstacle clusters and flow-induced vortical regions. Despite these complex deviations, all trajectories consistently converge to the yellow star goals without collision.

Figure 4. Representative scenarios for surface sensing agents in a flow field. The simulation environment is a 50 m × 50 m marine area. Panels (a–c): four-SSA case; Panels (d–f): five-SSA case. Green square with red arrow = current pose and velocity; purple discs = static obstacles; yellow circles = surface sensing agent initial positions; yellow stars = surface sensing agent goals; blue background = flow magnitude/isocontours. Colored curves show the executed trajectories for each surface sensing agent. Right insets depict the local sensing footprint for a representative agent (labeled “ssa 3” or “ssa 4” accordingly).

Figure 5 and Figure 6 summarize and explain the comparative outcomes. In Set 1 (no static obstacles), AQP achieves the top success rate for all team sizes (Figure 5a) and simultaneously shifts the time and energy distributions leftward with tighter interquartile ranges (Figure 5b,c). The shorter upper whiskers and fewer extreme outliers indicate fewer near-stall episodes and more consistent progress. Reward distributions (Figure 5d) are correspondingly less negative and less variable, implying that AQP accrues stepwise penalties for shorter durations and avoids large collision penalties.

Figure 5. Performance on Set 1 (no static obstacles) as a function of the number of surface sensing agents. (a) Success rate. (b) Completion time (s). (c) Energy cost. (d) Cumulative reward. Methods: AQP (ours), Adaptive IQN, IQN, DQN, and APF. Boxes/whiskers in (b–d) show median, interquartile range, and extrema with outliers.

Figure 6. Performance on Set 2 (with static obstacles) under increasing numbers of surface sensing agents. (a) Success rate. (b) Completion time (s). (c) Energy cost. (d) Cumulative reward. Same method legend and box-plot conventions as Figure 5.

When Set 2 introduces static obstacles, the gap widens. Baselines degrade as agent count increases: APF suffers local minima and oscillations, DQN exhibits high variance and frequent timeouts, and vanilla IQN under-explores safe corridors in dense clutter. In contrast, AQP sustains high success (Figure 6a) and preserves compact time/energy boxes (Figure 6b,c), with reward distributions that retain narrow spread and higher medians (Figure 6d). The tail behavior is especially telling: AQP shows markedly fewer long-time or high-energy outliers, reflecting robust avoidance of vortex cores and narrow passages even for 6–7 agents.

By dissecting the performance differentials across the baseline (IQN), the risk-aware variant (Adaptive IQN), and the proposed Innovation Framework, we isolate the independent and synergistic contributions of each module. The substantial leap in success rate from IQN to Adaptive IQN in obstacle-rich environments (Figure 6a) empirically validates the independent efficacy of the Adaptive Risk strategy in guaranteeing safety boundaries. Furthermore, the complete framework transcends this safety baseline, achieving superior energy efficiency and reward convergence (Figure 6c,d) compared to the risk-only variant. This secondary elevation attests to the synergistic value of the Hybrid Expert and EMA-PER: while the risk module secures survival, the expert guidance and prioritized replay optimize the policy manifold for minimal cost, demonstrating that these components function not merely additively, but multiplicatively to enhance the overall learning dynamics.

Mechanistically, these trends match the design. The A*–RVO expert furnishes a feasible, safety-oriented backbone that prevents catastrophic exploration in clutter; EMA-PER then prioritizes demonstrative and near-decision-boundary transitions, accelerating credit assignment and reducing update noise; finally, Adaptive IQN adjusts evaluation quantiles with proximity, yielding cautious actions near obstacles but efficiency in benign flow. Together, this yields a favorable Pareto profile—high success at lower time/energy with reduced variance—which becomes increasingly pronounced as scene complexity and team size grow.

It is worth noting that the outliers observed in the box plots (Figure 5 and Figure 6), particularly in task completion time and energy consumption, originate from the inherent stochasticity of multi-agent exploration, where agents occasionally exhibit extreme behaviors under uncertainty. Crucially, these outliers do not compromise the validity of our core conclusions. The interquartile ranges and medians demonstrate consistent central trends across all scenarios, and statistical verification (Wilcoxon rank-sum test, $p<0.01$) confirms that the performance advantage of AQP over baselines remains statistically significant and robust, independent of these sporadic deviations.

5. Conclusions

We address decentralized navigation and collision avoidance for surface sensing agent networks operating in dynamic hydrodynamic fields and propose a distributional–RL framework that integrates prior knowledge with adaptive learning. The core technical contributions are (i) an EMA–PER mechanism that stabilizes and accelerates training by emphasizing informative transitions, including demonstrations; (ii) an Adaptive IQN policy that reasons over full return distributions and modulates risk; and (iii) a hybrid A*–RVO expert that supplies safe, feasible guidance in cluttered flow fields. Extensive experiments show consistent gains in success rate, travel time, energy cost, and reward stability across scenarios of increasing complexity, demonstrating strong efficiency and robustness. The method couples expert priors with distributional reinforcement learning to obtain complementary strengths. The A*–RVO component offers a reliable backbone plan, ensuring feasibility and safety in densely cluttered regions. EMA–PER focuses updates on salient and diverse experiences, improving sample efficiency and stabilizing early learning. Adaptive IQN exploits the full return distribution and adjusts risk sensitivity according to local context, yielding cautious behavior near obstacles while maintaining efficiency in benign flow. The framework scales with the number of agents and scene complexity, and its components are modular: the expert can be replaced by other planners, and the risk scheduler in Adaptive IQN can be tuned to mission requirements.

Looking ahead, we see several promising directions for further advancing this work. These include (a) incorporating richer and more realistic oceanographic flow models together with domain randomization to enhance sim-to-real generalization; (b) developing closed-loop adaptation strategies that adjust the risk schedule based on online performance feedback; (c) integrating communication-aware coordination mechanisms to support dense multi-agent surface sensing agent teams; and (d) performing sim-to-real validation on field robots to assess robustness under real-world sensing and actuation uncertainties. These extensions would further strengthen the practicality of the approach while preserving its safety and efficiency advantages. In addition, the planned physical experiments will be an essential step toward demonstrating the real-world applicability of the proposed framework.