An Adaptive QAPF Framework with a Discrete CBF-Inspired Safety Filter and Adaptive Reward Shaping for Safe Mobile Robot Navigation

Abstract

Mobile robot navigation remains challenging when fast convergence, collision avoidance and deployability must be satisfied simultaneously. The original Q-learning with Artificial Potential Field (QAPF) paradigm is extended in this paper with three coordinated mechanisms that together yield a reported-horizon convergence reduction of approximately four orders of magnitude (from $\sim 3\times {10}^6$ episodes to $\sim 200$ to 230 episodes under the present protocol) and an internal-ablation collision-rate reduction of approximately one order of magnitude ($6.2\%$ to $0.3\%$), and that open a new capability frontier covering dynamic obstacles, multi-robot coordination, energy-aware velocity modulation and embedded-deployable inference timing. The first mechanism is a potential-based reward-shaping schedule whose unclipped fixed-weight form follows the policy-invariant shaping theorem, while the implemented clipped and time-varying form is used as an empirically stable approximation. Under the present experimental protocol, the reported convergence horizon is reduced from the $\sim 3\times {10}^6$ episodes reported for the original QAPF formulation to approximately 200 to 230 episodes; this comparison is protocol-dependent and is not claimed as a controlled one-to-one runtime speedup. The second mechanism is a discrete Control Barrier Function (CBF)-inspired action filter (thediscrete filter described in this paper is inspired by the continuous-time CBF literature, but does not carry a forward-invariance proof; it is used as an empirical safety mechanism rather than as a formal Control Barrier Function in the formal continuous-time sense) with per episode visit memory by which the held-out collision rate is reduced from $6.2\%$ for QAPF alone to $0.3\%$ while $93.8\%$ task completion is maintained, where this collision-rate comparison is internal to the QAPF ablation because the prior QAPF reference does not report a comparable held-out collision metric. The third mechanism is a set of extensions to dynamic obstacles, two-robot cooperative navigation under a centralized scheme (with an explicit $\mathcal{O}(N^2)$ scaling-cost analysis and three decentralization strategies for fleets beyond the small-N regime), curriculum learning and energy-aware velocity modulation. Disturbance robustness tests, empirical timeout/stagnation detection for unreachable-goal cases, i7 reference inference timing with projected embedded-device latencies, multi-axis generalization over obstacle density and grid size, scalability analysis for centralized multi-robot coordination and a scope comparison against A* and RRT* are added by the revised evaluation. Across 30 independent seeds on held-out static maps, $94.5\pm 2.1\%$ success is achieved by adaptive QAPF while $93.8\pm 2.3\%$ success with $0.3\pm 0.4\%$ collisions is achieved by QAPF+CBF. Under a separate finite robustness suite, $85.0\pm 4.1\%$ success is retained by QAPF+CBF in the combined disturbance regime. The timing study indicates that the 20 Hz real-time threshold is comfortably exceeded by all methods on the measured i7 reference platform and by all projected embedded-device equivalents. The results show that a lightweight and safety-oriented navigation policy for grid-based mobile-robot settings can be provided by APF-guided tabular reinforcement learning when it is paired with a discrete safety filter and a clarified energy and robustness analysis.

1. Introduction

Mobile robots have experienced a significant surge in popularity due to remarkable technological advancements in artificial intelligence, sensor technology and computational capabilities [1,2]. A mobile robot possesses the capability to traverse environments, dynamically adjust its path and maneuver through complex scenarios without human intervention. The primary aim of an intelligent motion control and obstacle avoidance system is to efficiently navigate from any initial state to the goal while actively avoiding any obstacles encountered along the way [3,4,5].

Despite this rapid progress, deploying a mobile robot in a realistic environment requires the simultaneous satisfaction of three conflicting objectives: (i) fast convergence so that the navigation policy is learned within a practical training budget; (ii) provable or near-provable safety so that obstacles are avoided even under sensor noise and actuator imperfections; and (iii) deployability on resource-constrained embedded hardware while preserving robustness to dynamic obstacles, multi-robot coordination and limited battery energy [1,2,5]. None of the dominant paradigms taken in isolation satisfies all three: classical Artificial Potential Field (APF) methods [6] are real-time and map-free but suffer from local-minima trapping in concave regions [3,7]; tabular and deep reinforcement learning [8,9,10,11] are flexible but require long training horizons and provide no formal safety guarantee during learning; and existing APF/RL hybrids such as the QAPF formulation of Orozco-Rosas et al. [12] report convergence horizons of $\sim 3\times {10}^6$ episodes and do not include a safety filter, an energy-aware controller, a multi-robot extension or a robustness/inference-timing analysis.

Our approach combines a potential-field-based collision-avoidance strategy with reinforcement learning to build a mobile-robot system capable of navigating uncharted territory without relying on pre-existing maps. The navigation problem is formulated as a Markov Decision Process (MDP) tailored for a mobile robot moving through obstacle-populated environments [13], while the potential-field component follows the Artificial Potential Field formulation introduced by Khatib [6]. The environment is characterized by attractive and repulsive forces influencing the robot’s movements, forming the potential-field landscape that the agent samples through iterative interaction. On top of this base formulation, three coordinated mechanisms are introduced and analyzed jointly: an adaptive potential-based reward-shaping schedule whose unclipped fixed-weight form is policy-invariant under the theorem of Ng, Harada and Russell [14]; a discrete Control Barrier Function (CBF)-inspired action filter with per episode visit memory [15,16] that turns the safety question from “how often does the agent collide?” to “which action subset should be excluded by a barrier-inspired safety mask?”; and an empirical timeout/stagnation detector that converts indefinite oscillation in unreachable-goal cases into a labeled safe-failure outcome. The same framework is extended to dynamic obstacles, narrow passages, multi-robot cooperative navigation and energy-aware velocity modulation, and is evaluated under sensor and actuator noise, on a measured Intel i7 inference reference with conservative embedded-device latency projections, and across orthogonal generalization axes (held-out maps, obstacle density, grid size and long-horizon stability).

The novelty of the proposed approach does not lie in the general concept of combining Q-learning with APF, which was introduced by Orozco-Rosas et al. [12] under the QAPF name, but rather in a substantially extended framework that closes three concrete methodological gaps of the original formulation. First, the original potential-difference shaping is non-policy-invariant, whereas the proposed unclipped fixed-weight form satisfies the assumptions of Ng, Harada and Russell [14] and the implemented clipped/decayed schedule is presented as an empirically stable approximation rather than as an unconditional guarantee. Second, the original framework has no safety layer, whereas the proposed discrete CBF-inspired filter [15,16], augmented with a per episode visit memory, empirically reduces the held-out collision rate by approximately $20\times$ ($6.2\%\to 0.3\%$) while maintaining $93.8\%$ task completion. Third, the original framework is restricted to static single-robot navigation, whereas the proposed framework is jointly evaluated under dynamic obstacles, narrow passages, multi-robot coordination [17,18], sensor/actuator noise, embedded-device inference timing and out-of-distribution generalization. The classical local-minima failure mode of pure APF [3,7] is mitigated through optimistic Q-initialization and the anti-deadlock progress monitor described in Section 4, and persistently unreachable goals are reported as labeled Timeout-Unreachable or Stagnation-Unreachable outcomes rather than as undefined non-terminations. The detailed quantitative contrast with the closest prior QAPF reference is given at the end of Section 2 (Table 2).

The main contributions of this paper, framed against this comparison, are:

• Adaptive QAPF Framework with Potential-Based Shaping. An adaptive shaping coefficient $\lambda (e)={\lambda }_{min}+({\lambda }_{max}-{\lambda }_{min})exp(-\beta e)$ is introduced around the potential-based shaping form of Ng, Harada and Russell [14], $F(\mathbf{s},{\mathbf{s}}^{\prime })=\gamma \mathsf{\Phi }({\mathbf{s}}^{\prime })-\mathsf{\Phi }(\mathbf{s})$, with $\mathsf{\Phi }(\mathbf{s})=-U(\mathbf{q}(\mathbf{s}))/U_{\Delta ,\mathrm{scale}}$, where $\mathbf{q}(\mathbf{s})$ denotes the continuous/grid position represented by state $\mathbf{s}$ and $U_{\Delta ,\mathrm{scale}}$ normalizes potential differences used for reward shaping. Under the standard assumptions, the unclipped fixed-weight form preserves the optimal policy of the underlying MDP. The implemented clipped and decayed form is therefore presented as an empirically stable approximation rather than as an unconditional policy-invariance guarantee; finite-training performance still depends on exploration, discretization and hyperparameter choices.
• CBF-Inspired Safety Filter with Visit Memory and Empirical Unreachable-Goal Handling. A novel discrete CBF-inspired filter [15,16] is introduced that augments the barrier-function safety test with a per episode visit memory to eliminate oscillation loops, and is paired with an empirical timeout/stagnation detector that converts blocked-goal, sealed-corridor and dead-end concave cases into labeled Timeout-Unreachable and Stagnation-Unreachable outcomes. The held-out collision rate is reduced by the filter from $6.2\%$ (QAPF alone) to $0.3\%$ (QAPF+CBF), which is an approximately $20\times$ reduction relative to the internal QAPF-only ablation, while $93.8\%$ task completion is maintained and persistent no-path behavior is reported as a safe-failure mode rather than as indefinite oscillation. The prior QAPF formulation [12] does not report a comparable held-out collision metric and therefore the safety contrast is internal rather than external.
• Energy-Aware Velocity Modulation. An explicit velocity-modulation law $v(\mathbf{q})=v_{min}+(v_{max}-v_{min})·2/(1+exp(k_E\tilde{g}(\mathbf{q})))$, driven by the normalized gradient magnitude $\tilde{g}(\mathbf{q})=\parallel \nabla U(\mathbf{q})\parallel /G_{\mathrm{scale}}$, is described. The law is constructed so that $v(\mathbf{q})=v_{max}$ when $\tilde{g}(\mathbf{q})=0$ (free space) and smoothly approaches $v_{min}$ in high-gradient obstacle-proximate regions; $G_{\mathrm{scale}}$ normalizes gradient magnitudes, distinct from the potential-difference scale $U_{\Delta ,\mathrm{scale}}$ used by the reward-shaping term. The law is paired with a two-term kinetic and jerk-energy metric $E={\sum }_t{c}_v{v}_t^2+c_a{(v_t-v_{t-1})}^2$.
• Dynamic and Multi-Robot Cooperative Extension. Beyond the static single-robot setting, dynamic obstacles, narrow-passage scenarios and multi-robot cooperative navigation are evaluated; for the cooperative case, an inter-robot virtual repulsion is paired with the per robot CBF-inspired filter, and the centralized scheme is accompanied by an explicit $\mathcal{O}(N^2)$ scaling-cost analysis with three concrete decentralization strategies (k-nearest, communication graph, hierarchical-cluster) [17,18] for fleets beyond the small-N regime.
• Comprehensive Safety-Centric Evaluation Protocol. A held-out evaluation protocol is introduced that jointly reports (i) main safety/efficiency metrics (success rate, collision rate, minimum clearance, Pareto frontier), (ii) robustness under three independent noise channels (observation noise ${\sigma }_{\mathrm{obs}}$, actuator slip $p_{\mathrm{act}}$ and external drift ${\sigma }_{\mathrm{ext}}$) with an accompanying conditional/high-probability inflated-barrier safety statement, (iii) per decision inference latency physically measured on an Intel i7 reference with conservative projections to Jetson/DGX-class platforms, and (iv) multi-axis generalization across held-out maps, $-66.7\%$ to $+66.7\%$ obstacle-density shifts, $\pm 40\%$ grid-size shifts and 1000-episode long-horizon stability.

The remainder of this paper is organized as follows. Section 2 reviews the related work and theoretical background. Section 3 formulates the MDP and simulation assumptions. Section 4 details the QAPF algorithm, the CBF-inspired action filter, the energy-aware velocity modulator and the empirical unreachable-goal detector. Section 5 presents the main comparison, multi-scenario evaluation, robustness analysis, inference-timing study, and ablation, sensitivity, energy, multi-robot, curriculum and generalization studies, followed by the limitations and threats-to-validity discussion. Section 6 concludes. Appendix A provides the algorithm-capability discussion on obstacle shape and concave geometry, empirical unreachable-goal handling, scope comparison against A*/RRT*, 3D-workspace scalability, and kinematic and geometric constraints with pose uncertainty.

2. Literature Review and Background

This section provides a concise review of existing approaches in mobile-robot navigation, obstacle avoidance and reinforcement learning-based path planning. Section 2.1 presents the literature review covering the principal research directions, and Section 2.2 provides the necessary theoretical background on reinforcement learning, potential-based reward shaping and Control Barrier Functions. A comprehensive feature-based comparison is presented in

Table 1. Feature-based comparison across key capability dimensions. ✔ = fully supported, ∼ = partially supported, and ✗ = not supported.

Method	Map-Free Nav.	Dynamic Obs.	Multi-Robot	Safety Filtering	Adaptive Learning	Local-Minima Mitigation	Real-Time Capable
Classical APF [6]	✔	✗	✗	✗	✗	✗	✔
Black-hole APF+RL [3]	✔	✔	✗	✗	∼	✔	✔
Evolutionary APF [19]	✗	✗	✗	✗	✗	∼	✗
Evo. Multi-obj. APF [7]	✔	✔	✔	✗	✗	✗	∼
QAPF (Orozco-Rosas) [12]	✔	✗	✗	✗	✔	∼	✔
Q-Learning [8]	✔	✗	✗	✗	✔	✗	✔
DQN [9]	✔	∼	✗	✗	✔	✔	∼
PPO [10]	✔	✔	✗	✗	✔	✔	∼
SAC [11]	✔	✔	✗	✗	✔	✔	∼
CBF-QP [15]	✗	∼	✗	✔	✗	✗	✔
CBF+RL [16]	✔	✔	✗	✔	✔	✔	∼
QAPF (Ours)	✔	✔	✔	✗	✔	∼	✔
QAPF+CBF (Ours)	✔	✔	✔	✔	✔	∼	✔

.

2.1. Literature Review

2.1.1. Enhanced Artificial Potential Field Methods

The Artificial Potential Field (APF) method of Khatib [6] models the robot’s workspace via an attractive force toward the goal and repulsive forces away from obstacles. The classical formulation suffers from the well-documented local-minima problem [3,7,19]. Yao et al. [3] proposed combining developed black-hole potential fields with reinforcement learning; Montiel et al. [19] applied parallel evolutionary optimization of APF gains; Herrera et al. [7] extended this approach to the multi-robot case with evolutionary multi-objective optimization.

Most closely related to our work, Orozco-Rosas et al. [12] introduced the QAPF learning algorithm with a partially guided Q-learning strategy in which APF biases exploration. Their framework does not use the unclipped potential-based shaping form of Ng, Harada and Russell [14], is restricted to static environments, does not consider CBF-style safety filtering and requires up to $3\times {10}^6$ episodes for convergence. Our work fills these gaps. Earlier improvements to Q-learning-based path planning, such as Low et al. [20], also report substantial gains over vanilla Q-learning on grid environments by reshaping the reward signal; their work, however, does not integrate an APF gradient nor a safety filter, so it remains a single-component baseline of the kind our ablation in Section 5.7 quantifies.

2.1.2. Deep Reinforcement Learning for Autonomous Navigation

Deep RL has transformed mobile-robot navigation [2,5]. Mnih et al. [9] introduced the DQN architecture with experience replay and target networks; Schaul et al. [21] added prioritized experience replay. Policy-gradient methods such as PPO [10] and SAC [11] handle continuous-action control, with TD3 [22] addressing function-approximation overestimation through twin critics and delayed policy updates; transformer architectures have been explored for sequence-level planning [23,24]; multi-agent RL approaches address inter-robot coordination [17,18]. In the broader autonomous-driving context, deep learning-based planners have been surveyed for their safety, generalization and deployability characteristics [25,26]; the QAPF framework targets a complementary map-free, lightweight, tabular regime in which a learned policy must operate alongside a discrete safety filter rather than as part of a heavy end-to-end driving stack.

2.1.3. Collision Avoidance with Dynamic Obstacles

Maw et al. [27] proposed iADA*-RL for multi-threat avoidance. These methods typically require an auxiliary path planner; our approach, by contrast, produces low-collision, reactive trajectories end-to-end via APF-guided Q-learning and the CBF action filter.

2.1.4. Control Barrier Functions

CBFs provide a principled framework for enforcing forward-invariant safe sets [15,28]. Cheng et al. [16] combined model-free RL with a CBF-based safety filter and Gaussian-process dynamics learning. Our contribution is a discrete CBF-inspired filter; appropriate for the grid-world setting and carrying empirical rather than formal guarantees; it is augmented with a per episode visit memory that prevents oscillation and reduces observed collisions to $0.3\%$ in practice (a $\sim 20\times$ reduction relative to QAPF without the filter). Adjacent lines of work formalize safety differently: Shalev-Shwartz et al. [29] propose a responsibility-sensitive contract (RSS) that yields rule-based safety constraints rather than learned ones, while Zhu et al. [30] obtain safe RL behavior through reward shaping designed for autonomous-driving longitudinal control. The discrete CBF filter adopted here is closer in spirit to the latter, a soft, learned-policy-friendly safety layer, but it is bound to the grid-world action space rather than to a continuous accelerator or brake control axis.

2.1.5. Quantitative Headline Comparison Against the Closest Prior QAPF Work

To make this novelty quantitatively concrete,

Table 2. Quantitative comparison with the most relevant prior QAPF formulation [12]. “−” indicates the dimension was not addressed in the prior work. The convergence-horizon comparison is protocol-dependent and should be interpreted as a reported-horizon contrast rather than as a controlled one-to-one runtime speedup.

Capability Dimension	Prior QAPF [12]	This Work	Improvement
Convergence horizon	$\sim 3\times {10}^6$ episodes	∼205 ep (QAPF)/∼230 ep (QAPF+CBF)	Protocol-dependent reported-horizon reduction
Collision rate (held-out)	− (not reported)	$6.2\%$ (QAPF)/$0.3\%$ (QAPF+CBF)	∼20× internal reduction vs. QAPF-only ablation
Success rate (static)	Path-quality metric only	$94.5\%$/$93.8\%$	Qualitative jump
Dynamic obstacles	−	$90.4\%$ SR (QAPF+CBF)	New capability
Narrow passages	−	$85.5\%$ SR (QAPF+CBF)	New capability
Multi-robot cooperation	−	$92.1\%$ joint SR	New capability
Robustness eval. (noise)	−	6 primary regimes (theory + exp.)	New capability
Unreachable-goal handling	−	Timeout/stagnation detector with empirical no-path investigation	New safe-failure capability
Embedded inference timing	−	∼4.9 kHz mean-call throughput on i7; projected embedded latencies	New capability

places our results side-by-side with the most relevant prior QAPF reference [12] along the dimensions on which comparison is meaningful. Three points are made sharply by the table. The reported convergence horizon is substantially shorter under the respective reported protocols ($3\times {10}^6$ to $\sim 200$ episodes), although this comparison is protocol-dependent and is not a controlled one-to-one ablation against the original implementation. Collisions are reduced by approximately $20\times$ by the proposed Control Barrier Function (CBF) layer relative to the same QAPF implementation without the safety layer ($6.2\%$ to $0.3\%$). The capability scope is extended from static single-robot navigation to dynamic, multi-robot, energy-aware, robustness-tested and embedded-deployable navigation. Because the prior QAPF paper did not report the same collision metric, the collision-reduction claim is made against the internal QAPF-only ablation rather than against that prior paper.

2.2. Theoretical Background

2.2.1. Reinforcement Learning and Policy-Invariant Reward Shaping

Q-learning [8] learns the optimal action-value function $Q^{*}(\mathbf{s},a)$ by iteratively applying the temporal-difference update. Convergence requires infinite visitation of all state–action pairs and the Robbins and Monro conditions on the learning-rate schedule [31].

A key theoretical tool for accelerating learning without biasing the optimal policy is the potential-based shaping result of Ng, Harada and Russell [14]. Given a base MDP with reward $r(\mathbf{s},a,{\mathbf{s}}^{\prime })$, an auxiliary shaping function $F(\mathbf{s},{\mathbf{s}}^{\prime })=\gamma \mathsf{\Phi }({\mathbf{s}}^{\prime })-\mathsf{\Phi }(\mathbf{s})$ is defined for any real-valued potential $\mathsf{\Phi }:\mathcal{S}\to \mathbb{R}$; the shaped reward $\tilde{r}=r+F$ then yields exactly the same optimal policy as r. In the proposed framework, $\mathsf{\Phi }(\mathbf{s})=-U(\mathbf{q}(\mathbf{s}))/U_{\Delta ,\mathrm{scale}}$ is taken, where U is the APF total potential evaluated at the robot position encoded by the state, and $U_{\Delta ,\mathrm{scale}}$ is the potential-difference normalization constant used only for reward shaping. This gives the field-aligned signal the same potential-difference structure proven policy-invariant by [14] when it is used in an unclipped fixed-weight form. In the implemented algorithm, clipping and the decaying coefficient $\lambda (e)$ are added for numerical stability and faster finite-training convergence; therefore, the exact policy-invariance guarantee applies to the underlying unclipped fixed-weight form, while the reported clipped scheduled form is evaluated empirically. This distinction is a key difference from the original QAPF formulation [12], which used non-invariant potential-difference shaping.

2.2.2. Control Barrier Functions

A (zeroing) Control Barrier Function [15] is a continuously differentiable $h:\mathcal{X}\to \mathbb{R}$ such that the safe set $\mathcal{C}=\{\mathbf{x}:h(\mathbf{x})\ge 0\}$ is forward-invariant under controls satisfying $\dot{h}(\mathbf{x})\ge -\kappa h(\mathbf{x})$ for some class-$\mathcal{K}$ function $\kappa$. In our discrete grid-world setting the continuous-time condition is relaxed to a discrete inequality $h({\mathbf{q}}^{\prime })\ge {\delta }_{\mathrm{safe}}$, yielding an empirical, not formally forward-invariant, safety guarantee.

3. Problem and System Description

This section formulates the mobile-robot navigation problem as a Markov Decision Process (MDP) and specifies the simulation environment used in the rest of this paper. Section 5.2 states the modeling assumptions on observation, dynamics, obstacle geometry and discrete actions. Section 3.2 presents the MDP formulation, including the discrete state encoding, the action space, the attractive and repulsive potential components and the base reward function. The complete closed-loop architecture is summarized in Figure 1, while the attractive, repulsive and combined potential fields used by the navigation policy are illustrated later in Figure 2.

Note that for notation convention, continuous positions, displacements, obstacle centers, goal positions, forces and noise vectors are written in boldface, e.g., $\mathbf{q}$, ${\mathbf{q}}_f$, $\mathbf{o}$, ${\mathbf{\Delta }}_a$, $\mathbf{F}$ and $\mathit{\eta }$. Matrices are also written in boldface, e.g., ${\mathbf{I}}_2$. Scalar discrete actions, rewards, potentials and state-encoding bins are not bolded.

3.1. System Description

The end-to-end framework is organized as a four-stage closed-loop pipeline summarized in Figure 1. The first stage, Environment and Sensing, models the workspace as a $50\times 50$ discrete grid populated with point obstacles of collision radius ${ϵ}_{\mathrm{coll}}=1.5$ cells (red filled circles), a start cell ${\mathbf{s}}_0$ (green square) and a goal cell ${\mathbf{q}}_f$ (gold star), with each obstacle additionally carrying an APF influence radius ${\rho }_0=3.0$ cells (light-blue ring) inside which the repulsive potential of Equation (4) is non-zero. At every decision step the sensing layer constructs the discrete state $\mathbf{s}$ of Equation (1) from the robot pose, the goal pose and the nearest-obstacle geometry. The second stage, the QAPF policy of Section 4.3, combines the Q-learning update of Equation (6) with the artificial-potential-field guidance of Equations (2)–(4) through the zero-centered hybrid score of Equation (11) and emits a nominal action $a^{★}$. The third stage, the discrete CBF-inspired safety filter of Section 4.5, evaluates the barrier value $h({\mathbf{q}}^{\prime })$ of Equation (12) at every successor cell, intersects the resulting safe-mask with the per episode visit-memory mask, and either passes $a^{★}$ through or substitutes the progress-maximizing safe-and-unforbidden alternative; its output is the executed action $a^{\mathrm{exec}}$. The fourth stage, robot actuation, applies $a^{\mathrm{exec}}$ to the robot dynamics and returns the next state ${\mathbf{s}}^{\prime }$ to the sensing layer, closing the loop. The same pipeline accommodates the dynamic-obstacle, narrow-passage and two-robot cooperative variants of Section 4 without architectural change: only the obstacle-set update rule, the inter-robot virtual repulsion of Equation (17) and the per robot CBF-filter instance differ between configurations. The empirical timeout/stagnation detector of Section 4.7 runs as a parallel supervisor over the same loop and labels persistent no-path behavior as a safe-failure outcome rather than as indefinite oscillation.

3.2. MDP Formulation

1. State ($\mathbf{s}$): The state is encoded as a 7-tuple

$$\mathbf{s}=(x_{\mathrm{bin}},y_{\mathrm{bin}},{\beta }_g,{\beta }_o,d_o,{\dot{d}}_o,{\widehat{d}}_o),$$

(1)

where $x_{\mathrm{bin}},y_{\mathrm{bin}}\in \{0,\dots ,4\}$ are binned position coordinates (five bins per axis, giving 25 position bins), ${\beta }_g,{\beta }_o\in \{0,\dots ,7\}$ are discretized bearings (eight $45°$ sectors) to the goal and nearest obstacle respectively, $d_o\in \{0,1,2,3\}$ is the binned distance to the nearest obstacle, ${\dot{d}}_o\in \{0,1,2\}$ is the discretized approach rate ($0=$ closing, $1=$ stationary, $2=$ receding; obtained by thresholding ${\rho }_{min}({\mathbf{q}}_t)-{\rho }_{min}({\mathbf{q}}_{t-1})$ at $\pm 0.1$ cells) and ${\widehat{d}}_o\in \{0,1,2,3\}$ is the one-step predicted distance bin computed from a constant-velocity extrapolation. The state-space cardinality is $\left|\mathcal{S}\right|=5·5·8·8·4·3·4=\mathrm{76,800}$.

The navigation environment is governed by the APF total potential at position $\mathbf{q}\in {\mathbb{R}}^2$:

$$U(\mathbf{q})=U_{\mathrm{att}}(\mathbf{q})+U_{\mathrm{rep}}(\mathbf{q}),$$

(2)

where $U_{\mathrm{att}}(\mathbf{q})$ is the attractive potential generated by the goal, and $U_{\mathrm{rep}}(\mathbf{q})$ is the repulsive potential generated by the obstacles. Equation (2) decomposes the total potential into an attractive component (Equation (3)) and a repulsive component (Equation (4)), with the attractive component centered at the goal ${\mathbf{q}}_f$,

$$U_{\mathrm{att}}(\mathbf{q})=\frac{1}{2}{k}_{\mathrm{att}}{\parallel \mathbf{q}-{\mathbf{q}}_f\parallel }^2,$$

(3)

where $k_{\mathrm{att}}$ is the attractive-gain coefficient and ${\mathbf{q}}_f\in {\mathbb{R}}^2$ is the goal position, with the repulsive component generating a barrier within the influence distance ${\rho }_0$:

$$U_{\mathrm{rep}}(\mathbf{q})=\left\{\begin{array}{cc}\frac{1}{2}{k}_{\mathrm{rep}}{\left({\displaystyle \frac{1}{\rho (\mathbf{q})}}-{\displaystyle \frac{1}{{\rho }_0}}\right)}^2,\hfill & \mathrm{if}\rho (\mathbf{q})<{\rho }_0\hfill \\ 0,\hfill & \mathrm{otherwise}.\hfill \end{array}\right.$$

(4)

Here $\rho (\mathbf{q})={min}_{\mathbf{o}\in \mathcal{O}}\parallel \mathbf{q}-\mathbf{o}\parallel$ is the Euclidean distance to the nearest obstacle in the obstacle set $\mathcal{O}=\{{\mathbf{o}}_1,\dots ,{\mathbf{o}}_{N_o}\}$.

2. Action (a): $a\in \mathcal{A}=\{\uparrow ,\downarrow ,\leftarrow ,\to \}$. Because $\mathcal{A}$ is discrete and finite, a is a scalar index (not bold).

3. Reward (r): The base reward used by the environment is

$$r(\mathbf{s},a,{\mathbf{s}}^{\prime })=R_{\mathrm{outcome}}-{\lambda }_s-{\lambda }_c\left(1-\frac{\rho ({\mathbf{q}}^{\prime })}{{\rho }_0}\right)\mathbb{1}[\rho ({\mathbf{q}}^{\prime })<{\rho }_0]+{\lambda }_g·\mathrm{clip}\left(d_{\mathrm{goal}}(\mathbf{q})-d_{\mathrm{goal}}({\mathbf{q}}^{\prime }),-1,1\right),$$

(5)

where $R_{\mathrm{outcome}}=+100$ at goal, $-50$ at collision and 0 otherwise; ${\lambda }_s$ is the per step penalty; ${\lambda }_c$ is the proximity-penalty weight; ${\lambda }_g$ is the dense-progress coefficient, and $d_{\mathrm{goal}}(\mathbf{q})=\parallel \mathbf{q}-{\mathbf{q}}_f\parallel$. The QAPF-specific potential-based shaping term of Section 4.3 is added on top of this base reward at the agent level, not the environment level.

3.3. Potential Field Visualization

Figure 2 illustrates the attractive, repulsive and combined potentials that guide the navigation.

4. Methodology

This section presents the methodology of the proposed QAPF framework and its CBF-inspired variant. Section 4.1 and Section 4.2 review the Q-learning update used as the learning backbone and the potential-field force approach used as the guidance signal. Section 4.3 introduces the QAPF learning algorithm with the policy-invariant potential-based shaping form and the adaptive shaping schedule. Section 4.4 describes the zero-centered hybrid Q+APF action-scoring rule. Section 4.5 introduces the discrete CBF-inspired action filter with per episode visit memory. Section 4.6 describes the energy-aware velocity-modulation law. Section 4.7 describes the empirical timeout/stagnation-based unreachable-goal detector. Section 4.8 describes the multi-robot cooperative extension, and Section 4.9 describes the three-dimensional workspace extension. The implementation parameters used by all of these components are summarized in

Table 3. Implementation specifications for the QAPF algorithm (values as implemented).

Parameter	Value/Description
Grid size	$50\times 50$ cells
Max steps per episode	1000
Goal threshold (${ϵ}_{\mathrm{goal}}$)/Collision radius (${ϵ}_{\mathrm{coll}}$)	0.5/1.5 cells
Outcome rewards ($R_{\mathrm{goal}}$, $R_{\mathrm{coll}}$, ${\lambda }_s$, ${\lambda }_c$, ${\lambda }_g$)	$+100$, $-50$, 1, 1, $0.5$
Learning rate $\eta$	0.15 (QAPF/QAPF+CBF); 0.1 (baselines)
Discount factor $\gamma$	0.95
Exploration (${\epsilon }_0$, ${\epsilon }_{min}$, ${\epsilon }_{\mathrm{decay}}$)	0.3, 0.01, 0.995 (QAPF); 0.9, 0.01, 0.995 (baselines)
APF gains ($k_{\mathrm{att}}$, $k_{\mathrm{rep}}$, ${\rho }_0$)	1.0, 100.0, 3.0 cells
Softmax temperature ($T_0$, $T_{min}$, $T_{\mathrm{decay}}$)	2.0, 0.3, 0.995/episode
Optimistic Q-init ($Q_0$)	5.0
Adaptive shaping (${\lambda }_{min}$, ${\lambda }_{max}$, $\beta$)	0.5, 5.0, 0.005
Hybrid guidance weights ($w_g^{\mathrm{train}}$, $w_g^{\mathrm{eval}}$)	1.2, 2.0
$U_{\Delta ,\mathrm{scale}}$ calibration	95th percentile of $|\Delta U|$ over 2000 random steps; used only for reward-shaping normalization
Gradient scale $G_{\mathrm{scale}}$	95th percentile of $\parallel \nabla U\parallel$ over 500 random steps; used only for energy-aware velocity modulation
CBF safety margin (${\delta }_{\mathrm{safe}}$)/visit cap (M)	0.3 cells/3 revisits per cell to action
Empirical unreachable detection	$T_{max}=1000$ steps; $W_{\mathrm{stag}}=15$ steps; $\Delta d_{\mathrm{stag}}=1.0$ cell; $N_{\mathrm{stag}}=3$ consecutive stagnation windows
Max episodes/evaluation logging/convergence extraction	1500/every 50 episodes/convergence episode estimated by interpolation of the smoothed held-out SR curve
Final held-out evaluation episodes	100
Independent seeds	30
Energy-aware module
$k_E$ (Constant/APF-mod/APF-agg)	${10}^{-3}$/0.5/1.5
$v_{max}$, $v_{min}$, $c_v$, $c_a$	1.0, 0.05, 1.0, 0.5
Multi-robot module
$k_{\mathrm{rep}}^{\mathrm{MR}}$, ${\rho }_0^{\mathrm{MR}}$	20.0, 3.0 cells
Curriculum schedule (Section 5.7.5)
Initial density $N_o^{(0)}$/target density $N_o^{(\mathrm{target})}$	5/15
Ramp episodes $E_{\mathrm{ramp}}$/schedule shape	400/linear, monotonic seed

.

4.1. Q-Learning

Q-learning [8] is an off-policy, model-free algorithm that iteratively refines estimates of the expected cumulative discounted reward via

$$Q(\mathbf{s},a)\leftarrow Q(\mathbf{s},a)+\eta \left[r+\gamma \underset{a^{\prime }}{max}Q({\mathbf{s}}^{\prime },a^{\prime })-Q(\mathbf{s},a)\right],$$

(6)

where $\eta \in (0,1]$ is the learning rate, $\gamma \in [0,1)$ is the discount factor and ${\mathbf{s}}^{\prime }$ is the resulting next state. Under standard conditions Q-learning converges to $Q^{*}$ [8,31]. The Q-table is initialized optimistically at $Q_0=5.0$ rather than at zero: this initialization encourages exploration of unvisited state-to-action pairs, because any negative experience of a visited action makes it less attractive than the still-optimistic default.

4.2. Potential Field Force Approach

The force field is the negative gradient of the total potential, $\mathbf{F}(\mathbf{q})=-\nabla U(\mathbf{q})$. On the discrete grid the continuous gradient is approximated by finite differences; the STEP algorithm (Algorithm 1) evaluates the potential at each of the four neighboring cells and encodes the resulting state according to Equation (1).

Algorithm 1 STEP(a): potential-field evaluation

1: Input: action a, current state $\mathbf{s}$, position $\mathbf{q}$ 2: Output: next state ${\mathbf{s}}^{\prime }$, reward r, done flag 3: Compute next position: ${\mathbf{q}}^{\prime }\leftarrow \mathrm{clip}(\mathbf{q}+{\mathbf{\Delta }}_a,\mathbf{0},(G-1)\mathbf{1})$ 4: Calculate $U_{\mathrm{att}}({\mathbf{q}}^{\prime })=\frac{1}{2}{k}_{\mathrm{att}}{\parallel {\mathbf{q}}^{\prime }-{\mathbf{q}}_f\parallel }^2$ and $U_{\mathrm{rep}}({\mathbf{q}}^{\prime })$ per Equation (4) 5: $U_{\mathrm{total}}\leftarrow U_{\mathrm{att}}+U_{\mathrm{rep}}$ 6: Assign base reward r per Equation (5) 7: Encode next state ${\mathbf{s}}^{\prime }$ per Equation (1); check termination 8: return ${\mathbf{s}}^{\prime },r$, done

4.3. QAPF Learning Algorithm

The QAPF algorithm (Algorithm 2) integrates the Q-learning update of Equation (6) with APF guidance through three tightly coupled mechanisms.

The first component, potential-based reward shaping, is described as follows. Instead of adding an arbitrary APF-flavored bonus to the reward, the Ng, Harada and Russell form [14] is adopted:

$$\tilde{r}(\mathbf{s},a,{\mathbf{s}}^{\prime })=r(\mathbf{s},a,{\mathbf{s}}^{\prime })+\lambda (e)·\mathrm{clip}\left({\displaystyle \frac{U(\mathbf{q})-\gamma U({\mathbf{q}}^{\prime })}{U_{\Delta ,\mathrm{scale}}}},-1,+1\right),$$

(7)

which reduces to $F(\mathbf{s},{\mathbf{s}}^{\prime })=\gamma \mathsf{\Phi }({\mathbf{s}}^{\prime })-\mathsf{\Phi }(\mathbf{s})$ with potential $\mathsf{\Phi }(\mathbf{s})=-U(\mathbf{q}(\mathbf{s}))/U_{\Delta ,\mathrm{scale}}$ when the clipping is inactive and $\lambda (e)$ is fixed. For any fixed coefficient and without clipping, the term satisfies the assumptions of [14] and preserves the optimal policy of the base MDP. The implemented version intentionally clips the normalized potential difference and uses the schedule $\lambda (e)$ of Equation (8); these two engineering choices stabilize finite-sample learning but mean that exact policy invariance should not be claimed for the full clipped, time-varying implementation. The policy-invariance statement therefore applies to the underlying unclipped fixed-weight shaping form, while the practical scheduled-and-clipped variant is validated empirically.

The clipping to $[-1,+1]$ stabilizes updates in the presence of heavy-tailed $\Delta U$ (observed raw range $[-69.5,+70.0]$; see Figure 6); $U_{\Delta ,\mathrm{scale}}$ is auto-calibrated as the 95th percentile of $|\Delta U|$ observed during a 2000-step random-walk warm-up. This scale is distinct from the gradient normalization constant $G_{\mathrm{scale}}$ used later by the energy-aware velocity module.

The second component is the adaptive shaping coefficient. The weight $\lambda (e)$ is decayed geometrically from a large initial value to a small residual:

$$\lambda (e)={\lambda }_{min}+({\lambda }_{max}-{\lambda }_{min})exp(-\beta e),$$

(8)

with defaults ${\lambda }_{min}=0.5$, ${\lambda }_{max}=5.0$, and $\beta =0.005$ (tuneable via the sensitivity study, Section 5.7). Early episodes receive a strong APF signal; as e grows, the agent transitions to acting on the autonomous Q-values.

The third component is APF-weighted softmax exploration. During exploration (probability $\epsilon$), an action is sampled from a softmax over the negative APF potential at each candidate successor cell rather than from a uniform distribution:

$$P(a_i\mid \mathbf{s})={\displaystyle \frac{exp\left(-(U({\mathbf{q}}_{a_i}^{\prime })-U_{min})/T\right)}{{\sum }_{j}exp\left(-(U({\mathbf{q}}_{a_j}^{\prime })-U_{min})/T\right)}},$$

(9)

where ${\mathbf{q}}_{a_i}^{\prime }$ is the successor cell reached by candidate action $a_i$, $U_{min}={min}_{j}U({\mathbf{q}}_{a_j}^{\prime })$ is the minimum successor potential subtracted for numerical stability and $T>0$ is the softmax temperature controlling the exploration–exploitation balance, mixed with a $10\%$ uniform floor to maintain full coverage. The temperature T decays geometrically ($T\leftarrow max(T_{min},T·T_{\mathrm{decay}})$ per episode) from $T_0=2.0$ to $T_{min}=0.3$. An anti-deadlock monitor additionally forces exploration (probability raised to at least $0.5$) whenever the robot has not reduced $d_{\mathrm{goal}}$ by more than $1.0$ cell over the preceding 15 steps.

The same progress monitor is also used during evaluation to label empirical unreachable-goal outcomes. A single no-progress window triggers forced exploration, whereas an unreachable label is assigned only if the no-progress condition persists over $N_{\mathrm{stag}}$ consecutive monitoring windows or if the maximum horizon is reached. Thus, ordinary temporary stalls are treated as exploration events, while persistent no-path behavior is reported as a safe-failure mode rather than as indefinite oscillation.

Algorithm 2 QAPF learning algorithm

1:  Initialize $Q(\mathbf{s},a)\leftarrow Q_0=5.0$ for all $\mathbf{s},a$; calibrate $U_{\Delta ,\mathrm{scale}}$ from $|\Delta U|$ (2000-step warm-up) 2:  Set $\eta ,\gamma ,{\epsilon }_0,{\epsilon }_{min},{\epsilon }_{\mathrm{decay}},{\lambda }_{min},{\lambda }_{max},\beta ,T_0,T_{min}$ 3:  for episode $e=1,\dots ,E_{max}$ do 4:     Initialize $\mathbf{s}\leftarrow {\mathbf{s}}_0$ from current training-map seed; reset obstacles and start/goal 5:     Set $t\leftarrow 0$, $done\leftarrow \mathrm{false}$ and $terminal\_status\leftarrow \varnothing$ 6:     Reset stuck-history buffer ${\mathcal{H}}_{d_g}\leftarrow \varnothing$ and persistent-stagnation counter $C_{\mathrm{stag}}\leftarrow 0$ 7:     while not done and $t<T_{max}$ do 8:       Update ${\mathcal{H}}_{d_g}$; determine if stuck (no progress over last $W_{\mathrm{stag}}=15$ steps) 9:       if stuck then $C_{\mathrm{stag}}\leftarrow C_{\mathrm{stag}}+1$ else $C_{\mathrm{stag}}\leftarrow 0$ end if 10:      if (not eval_mode or stuck) and rand <$max(\epsilon ,0.5·\mathbb{1}[\mathrm{stuck}])$ then 11:         Sample a from Equation (9) mixed with $10\%$ uniform 12:      else 13:         Select $a=arg{max}_{a^{\prime }}[Q(\mathbf{s},a^{\prime })-w_g\tilde{U}(a^{\prime })]$              (Equation (11)) 14:      end if 15:      Execute a, obtain ${\mathbf{s}}^{\prime },r$ 16:      Compute shaped reward $\tilde{r}$ via Equation (7) 17:      Update Q using Equation (6) with $\tilde{r}$ 18:      $\mathbf{s}\leftarrow {\mathbf{s}}^{\prime }$ 19:      $t\leftarrow t+1$ 20:      if $C_{\mathrm{stag}}\ge N_{\mathrm{stag}}$ in evaluation mode then set $done\leftarrow \mathrm{true}$ and $terminal\_status$ ← Stagnation-Unreachable end if 21:    end while 22:   if $t\ge T_{max}$ and not goal_reached and not collision then $terminal\_status$ ← Timeout-Unreachable end if 23:    $\epsilon \leftarrow max({\epsilon }_{min},\epsilon ·{\epsilon }_{\mathrm{decay}})$; $T\leftarrow max(T_{min},T·T_{\mathrm{decay}})$ 24: end for 25: return ${\pi }^{*}(\mathbf{s})=arg{max}_a[Q(\mathbf{s},a)-w_g^{\mathrm{eval}}\tilde{U}(a)]$

4.4. Hybrid Q+APF Action Scoring

At decision time, the agent combines learned Q-values with APF-based guidance through the following score. For each candidate action $a_i$ with successor potential $U_i:=U({\mathbf{q}}_{a_i}^{\prime })$, let

$${\tilde{U}}_i={\displaystyle \frac{U_i-\overline{U}}{max\left({max}_j{U}_j-{min}_j{U}_j,{ϵ}_U\right)}},\overline{U}=\frac{1}{\left|\mathcal{A}\right|}\sum _j{U}_j,{ϵ}_U={10}^{-9},$$

(10)

i.e., $\tilde{U}$ defined by Equation (10) is zero-centered and range-normalized. The small constant ${ϵ}_U$ prevents division by zero in flat local fields where all candidate successor cells have identical or numerically indistinguishable potential. This is important: with raw $U_i$, the per action Q-values become confounded with the absolute potential scale, and in obstacle-free regions, all actions receive essentially identical APF terms. Using the zero-centered $\tilde{U}$, only the relative ordering of potentials matters; this is precisely the gradient information that the framework is designed to inject.

The hybrid score is then

$$\mathrm{score}(\mathbf{s},a_i)=Q(\mathbf{s},a_i)-w_g·{\tilde{U}}_i,$$

(11)

with $w_g=w_g^{\mathrm{train}}=1.2$ during training and $w_g=w_g^{\mathrm{eval}}=2.0$ at evaluation. The higher evaluation weight produces more conservative, gradient-following behavior on held-out maps, where the Q-table has not been specifically tuned.

4.5. CBF-Inspired Action Filter with Visit Memory

The QAPF policy is augmented with a discrete CBF-inspired safety filter that additionally contains a per episode visit memory to prevent oscillation loops; a practical issue in which an oscillating robot repeatedly selects the same stuck action. Let the discrete barrier function be

$$h(\mathbf{q})=\rho (\mathbf{q})-{ϵ}_{\mathrm{coll}}=\underset{\mathbf{o}\in \mathcal{O}}{min}\parallel \mathbf{q}-\mathbf{o}\parallel -{ϵ}_{\mathrm{coll}},$$

(12)

so the safe set $\mathcal{C}=\{\mathbf{q}:h(\mathbf{q})\ge {\delta }_{\mathrm{safe}}\}$ is defined via the barrier function $h(\mathbf{q})$ of Equation (12), with ${\delta }_{\mathrm{safe}}=0.3$ cells (an inflated-margin parameter chosen empirically to cover one-step transition uncertainty; its role in robust safety is analyzed in Section 5.5). A visit counter ${\mathcal{V}}_e(\mathbf{q},a)$ tracks, within the current episode e, how many times the agent has executed a from cell $\mathbf{q}$. An action is forbidden at $\mathbf{q}$ if ${\mathcal{V}}_e(\mathbf{q},a)\ge M$ (default $M=3$).

The filter, given the QAPF-nominal action $a^{★}$, proceeds in five phases (Algorithm 3): (1) evaluate h and U at each successor cell; (2) build safe-mask ${\mathcal{M}}_s=\{a:h({\mathbf{q}}_a^{\prime })\ge {\delta }_{\mathrm{safe}}\}$ and forbidden-mask ${\mathcal{M}}_f=\{a:{\mathcal{V}}_e(\mathbf{q},a)\ge M\}$; (3) if $a^{★}\in {\mathcal{M}}_s\setminus {\mathcal{M}}_f$, execute $a^{★}$; (4) else, if ${\mathcal{M}}_s\setminus {\mathcal{M}}_f\ne \varnothing$, execute $arg{min}_{a\in {\mathcal{M}}_s\setminus {\mathcal{M}}_f}U({\mathbf{q}}_a^{\prime })$, the progress-maximizing safe unforbidden action; (5) otherwise, fall back to $arg{min}_{a\in {\mathcal{M}}_s}U$, or, if ${\mathcal{M}}_s=\varnothing$, to $arg{max}_{a}h({\mathbf{q}}_a^{\prime })$.

Unlike the continuous-time CBF-QP formulation of [15], our discrete variant does not carry a formal forward-invariance proof. Nevertheless, combined with the inflated margin ${\delta }_{\mathrm{safe}}$ and the visit memory, it reduces collisions to $0.3\%$ on the held-out evaluation suite across all 30 seeds (a $\sim 20\times$ reduction relative to QAPF without the filter, see Section 5.3) and gracefully degrades under noise (Section 5.5).

Algorithm 3 CBF-inspired action filter with visit memory

1:  Input: nominal action $a^{★}$, position $\mathbf{q}$, visit counter ${\mathcal{V}}_e$ 2:  For each $a\in \mathcal{A}$: ${\mathbf{q}}_a^{\prime }\leftarrow \mathbf{q}+{\Delta }_a$; $h_a\leftarrow h({\mathbf{q}}_a^{\prime })$; $U_a\leftarrow U({\mathbf{q}}_a^{\prime })$ 3:  ${\mathcal{M}}_s\leftarrow \{a:h_a\ge {\delta }_{\mathrm{safe}}\}$;   ${\mathcal{M}}_f\leftarrow \{a:{\mathcal{V}}_e(\mathbf{q},a)\ge M\}$ 4:  if $a^{★}\in {\mathcal{M}}_s$ and $a^{★}\notin {\mathcal{M}}_f$ then 5:     $a^{\mathrm{chosen}}\leftarrow a^{★}$  6:  else if ${\mathcal{M}}_s\setminus {\mathcal{M}}_f\ne \varnothing$ then 7:     $a^{\mathrm{chosen}}\leftarrow arg{min}_{a\in {\mathcal{M}}_s\setminus {\mathcal{M}}_f}{U}_a$  8:  else if ${\mathcal{M}}_s\ne \varnothing$ then 9:     $a^{\mathrm{chosen}}\leftarrow arg{min}_{a\in {\mathcal{M}}_s}{U}_a$ 10: else 11:    $a^{\mathrm{chosen}}\leftarrow arg{max}_{a\in \mathcal{A}}{h}_a$ 12: end if 13: ${\mathcal{V}}_e(\mathbf{q},a^{\mathrm{chosen}})\leftarrow {\mathcal{V}}_e(\mathbf{q},a^{\mathrm{chosen}})+1$  14: return $a^{\mathrm{chosen}}$

The full evaluation procedure that combines the trained QAPF policy with the discrete CBF-inspired filter is summarized in Algorithm 4.

Algorithm 4 QAPF+CBF evaluation wrapper

1:  Input: trained Q-table from Algorithm 2; environment; map seed 2:  Output: terminal status ∈ {Goal, Collision, Timeout-Unreachable, Stagnation-Unreachable} 3:  Initialize $\mathbf{s}\leftarrow {\mathbf{s}}_0$; set $t\leftarrow 0$, $done\leftarrow \mathrm{false}$ and $terminal\_status\leftarrow \varnothing$; reset visit counter ${\mathcal{V}}_e\leftarrow 0$; reset stagnation counter $C_{\mathrm{stag}}\leftarrow 0$ 4:  while not done and $t<T_{max}$ do 5:     Compute QAPF nominal action $a^{★}\leftarrow arg{max}_{a^{\prime }}[Q(\mathbf{s},a^{\prime })-w_g^{\mathrm{eval}}\tilde{U}(a^{\prime })]$          (Equation (11)) 6:     Apply CBF-inspired filter: $a^{\mathrm{exec}}$ ← CBFFilter $(a^{★},\mathbf{q},{\mathcal{V}}_e)$                  (Algorithm 3) 7:     Execute $a^{\mathrm{exec}}$, obtain ${\mathbf{s}}^{\prime }$, r, ${\rho }_{min}({\mathbf{q}}^{\prime })$ 8:     Update stagnation counter $C_{\mathrm{stag}}$ per Equation (16) 9:     if goal reached then terminal_status $\leftarrow$ Goal; done ← true end if 10:    if ${\rho }_{min}({\mathbf{q}}^{\prime })<{ϵ}_{\mathrm{coll}}$ then terminal_status $\leftarrow$ Collision; done ← true end if 11:    if $C_{\mathrm{stag}}\ge N_{\mathrm{stag}}$ then terminal_status ← Stagnation-Unreachable; done ← true end if 12:    $\mathbf{s}\leftarrow {\mathbf{s}}^{\prime }$; $t\leftarrow t+1$ 13: end while 14: if not done and $t\ge T_{max}$ then 15:    terminal_status ← Timeout-Unreachable 16: end if 17: return terminal_status

4.6. Energy-Aware Velocity Modulation

An explicit sigmoidal velocity-modulation law is described that allows the robot to reduce kinetic energy in high-gradient obstacle-proximate regions while retaining approximately full speed in flat/free-space regions. The module is formulated as three equations: a kinetic plus jerk energy per step (Equation (13)), a normalized gradient magnitude (Equation (14)) and a full-speed-at-zero-gradient sigmoidal speed law (Equation (15)):

$$\begin{array}{cc}\hfill \mathrm{E}1(\mathrm{Kinetic}+\mathrm{jerk}\mathrm{energy}\mathrm{per}\mathrm{step}):E(t)& =c_{v}v{(t)}^2+c_a{\left(v(t)-v(t-1)\right)}^2,\hfill \end{array}$$

(13)

$$\begin{array}{cc}\hfill \mathrm{E}2(\mathrm{Normalized}\mathrm{gradient}\mathrm{magnitude}):\tilde{g}(\mathbf{q})& ={\displaystyle \frac{\parallel \nabla U(\mathbf{q})\parallel }{G_{\mathrm{scale}}}},\hfill \end{array}$$

(14)

$$\begin{array}{cc}\hfill \mathrm{E}3(\mathrm{Sigmoidal}\mathrm{speed}\mathrm{law}):v(\mathbf{q})& =v_{min}+(v_{max}-v_{min}){\displaystyle \frac{2}{1+exp(k_E\tilde{g}(\mathbf{q}))}},\hfill \end{array}$$

(15)

where $v_{min}=0.05$ is a hard speed floor preventing degenerate standstill, $v_{max}=1.0$, $c_v=1.0$, $c_a=0.5$ and $k_E$ is the regime-selecting coefficient (constant: $k_E={10}^{-3}$; APF-mod: $k_E=0.5$; APF-agg: $k_E=1.5$). The law satisfies $v(\mathbf{q})=v_{max}$ when $\tilde{g}(\mathbf{q})=0$ and smoothly approaches $v_{min}$ as the normalized gradient increases. The gradient normalization constant $G_{\mathrm{scale}}$ is auto-calibrated as the 95th percentile of $\parallel \nabla U\parallel$ observed during a 500-step random-walk pass over the same obstacle field.

The normalization E2 is essential: because the raw gradient $\parallel \nabla U\parallel$ scales with $k_{\mathrm{rep}}\approx 100$, an unnormalized law would saturate the exponential term at nearly every step and drive $v(\mathbf{q})$ to the speed floor $v_{min}$, producing excessive navigation time. E2 rescales the gradient to a bounded typical range $[0,1.5]$, restoring meaningful velocity modulation. The total episode energy is $E_{\mathrm{ep}}={\sum }_{t=1}^{T}E(t)$, reported both absolutely and as a ratio $E_{\mathrm{ep}}/E_{\mathrm{const}}$ relative to the constant-speed baseline.

4.7. Empirical Unreachable-Goal Detection

Pure APF and APF/RL hybrids share a well-known failure mode in which the gradient field admits a stable non-goal equilibrium [3,7]: the agent oscillates indefinitely without ever terminating. To convert this into a labeled safe-failure outcome rather than an indefinite non-termination, an empirical unreachable-goal detection mechanism is integrated directly at the policy level, in addition to the standard goal-reaching and collision termination criteria. Let $d_t=\parallel {\mathbf{q}}_t-{\mathbf{q}}_f\parallel$ denote the goal distance at step t. A stagnation window is detected when

$$\underset{\tau \in \{t-W_{\mathrm{stag}},\dots ,t\}}{max}{d}_{\tau }-\underset{\tau \in \{t-W_{\mathrm{stag}},\dots ,t\}}{min}{d}_{\tau }<\Delta d_{\mathrm{stag}},$$

(16)

and the robot remains inside a repeated local region during the same window (default $W_{\mathrm{stag}}=15$ steps, $\Delta d_{\mathrm{stag}}=1.0$ cell). A single stagnation window does not yet trigger termination; instead, it raises the exploration probability to $\epsilon \ge 0.5$ and resamples actions from the APF-weighted softmax of Equation (9), which is the same anti-deadlock monitor invoked inside Algorithm 2. Only when the stagnation condition persists over $N_{\mathrm{stag}}=3$ consecutive monitoring windows is the episode terminated and labeled as Stagnation-Unreachable. If the maximum horizon $T_{max}=1000$ is reached without goal arrival or collision, the episode is labeled as Timeout-Unreachable. The two thresholds are chosen so that recoverable temporary stalls are absorbed by exploration boosts, while persistent no-path behavior is reported as a safe-failure mode.

Unlike a graph-theoretic reachability oracle (e.g., A* [32]), this detector is empirical: it does not certify that no feasible path exists in the underlying grid graph. It is, however, sufficient at the policy level to ensure that blocked-goal, sealed-corridor and dead-end concave maps produce a clear terminal status with a bounded detection time, which is the property required by a deployable navigation controller. Empirical detection rates on the no-path test suite are reported in Appendix A.2 and are between $96.5\%$ and $98.7\%$ across the three no-path map families, with a false-unreachable rate of $1.0\pm 0.6\%$ on reachable held-out maps.

4.8. Multi-Robot Cooperative Extension

For the multi-robot setting, each robot k augments its APF repulsive field with a virtual inter-robot repulsive component:

$$U_{\mathrm{rep}}^{\mathrm{MR}}({\mathbf{q}}_k)=\sum _{j\ne k}\frac{1}{2}{k}_{\mathrm{rep}}^{\mathrm{MR}}{\left(\frac{1}{\parallel {\mathbf{q}}_k-{\mathbf{q}}_j\parallel }-\frac{1}{{\rho }_0^{\mathrm{MR}}}\right)}^2\mathbb{1}\left[\parallel {\mathbf{q}}_k-{\mathbf{q}}_j\parallel <{\rho }_0^{\mathrm{MR}}\right],$$

(17)

with $k_{\mathrm{rep}}^{\mathrm{MR}}=20$, ${\rho }_0^{\mathrm{MR}}=3$ cells and ${\mathbf{q}}_j$ being the current position of the other robot(s). The augmented potential $U({\mathbf{q}}_k)+U_{\mathrm{rep}}^{\mathrm{MR}}({\mathbf{q}}_k)$ replaces U in the hybrid score of Equation (11); when combined with the per robot CBF filter of Section 4.5, the joint policy achieves a $92.1\%$ joint success rate (Table 14).

4.9. Three-Dimensional Workspace Extension

The 2D MDP and APF formulation of Section 3 and Section 4 is naturally extended to a three-dimensional workspace in which the robot position becomes $\mathbf{q}\in {\mathbb{R}}^3$. Three families of components are affected by the dimensional change. The continuous-state components, namely the attractive potential $U_{\mathrm{att}}$, the repulsive potential $U_{\mathrm{rep}}$, the gradient-based force $\mathbf{F}=-\nabla U$, the CBF barrier $h(\mathbf{q})$ and the energy module of Equations (13) to (15), are all extended without modification because only a metric and an obstacle geometry are required and both are defined naturally in ${\mathbb{R}}^3$. The discrete state encoding of Equation (1) is generalized by adopting the 3D 10-tuple $(x_{\mathrm{bin}},y_{\mathrm{bin}},z_{\mathrm{bin}},{\beta }_g^{\mathrm{az}},{\beta }_g^{\mathrm{el}},{\beta }_o^{\mathrm{az}},{\beta }_o^{\mathrm{el}},d_o,{\dot{d}}_o,{\widehat{d}}_o)$ in which $z_{\mathrm{bin}}$ is the binned vertical coordinate, ${\beta }_g^{\mathrm{az}}$ and ${\beta }_g^{\mathrm{el}}$ are the azimuth and elevation bearings to the goal, ${\beta }_o^{\mathrm{az}}$ and ${\beta }_o^{\mathrm{el}}$ are the corresponding bearings to the nearest obstacle and the remaining distance and predicted distance bins are retained from the 2D formulation. The action space is extended from $\left|\mathcal{A}\right|=4$ cardinal moves in the plane to $\left|\mathcal{A}\right|=6$ cardinal moves in 3D, with an optional $\left|\mathcal{A}\right|=26$ king-move set for smoother trajectories at higher computational cost. The shaping form of Equation (7) and the CBF filter of Algorithm 3 are unchanged because both depend only on the potential and on the barrier function, both of which are heading- and dimension-invariant. A scalability and complexity analysis of the same 3D extension is provided in Appendix A.4.

4.10. Implementation Specifications

Table 3 summarizes all implementation parameters and reflects the values actually used in the reference implementation.

5. Experiment and Analysis

This section presents the experimental evaluation of the proposed QAPF and QAPF+CBF policies. Section 5.1 describes the experimental setup, simulation parameters and evaluated methods. Section 5.3 reports the main static-obstacle comparison and the multi-scenario evaluation across static, dynamic and narrow-passage settings. Section 5.4 reports the empirical unreachable-goal and stagnation detection study. Section 5.5 reports the robustness analysis under observation noise, actuator slip and external drift, accompanied by a conditional/high-probability inflated-barrier safety statement. Section 5.6 reports the per decision inference timing on an Intel i7 reference and the projected embedded-device latencies. Section 5.7 reports the ablation, sensitivity, energy, multi-robot, curriculum and multi-axis generalization studies. Section 5.8 reports the three-dimensional workspace evaluation, and Section 5.9 discusses the limitations and threats to validity. The main comparative metrics are summarized in Table 5.

All reported metrics are obtained from held-out evaluation suites (seed offsets disjoint from training) rather than from raw training trajectories. Unless otherwise stated, the main comparative, scenario, ablation and generalization experiments use 30 independent seeds with up to 1500 training episodes. Protocol-specific exceptions are stated where they occur: the multi-robot experiment uses 1000 episodes per robot, the robustness study uses five training seeds × 30 noisy evaluation episodes per regime, and the timing study uses repeated per decision timing calls rather than success-rate training seeds. The held-out evaluation additionally records terminal status as goal, collision, Timeout-Unreachable or Stagnation-Unreachable. The latter two labels are used only when the agent neither reaches the goal nor collides, and the maximum horizon or persistent-stagnation criterion is satisfied. This status variable prevents impossible-goal episodes from being conflated with ordinary failed navigation trials. It should also be noted, before the per experiment protocols are detailed, that all simulation experiments reported in this section are run on a $50\times 50\times 1$ voxel grid in which the vertical coordinate is held fixed at $z=0$. The 2D plane is therefore the operational evaluation domain throughout Section 5, and the 3D-on-2D-plane sanity check that exercises the dimensional generalization of QAPF+CBF on the same protocol is reported separately in Section 5.8. A real-aerial validation of the 3D formulation, in which a quadrotor drone is flown through a structured indoor environment, is left for future work and is described in Section 6.

5.1. Experimental Setup and Evaluated Methods

The simulation parameters and experimental configuration used throughout this paper are summarized in

Table 4. Simulation parameters and experimental configuration.

Category	Parameter	Value/Description
Environment	Grid size	$50\times 50$ cells
	Obstacle count	15 (default); variable in scenarios
	Obstacle types	Static, dynamic, narrow-passage
Hardware	Processor	Intel Core i7 (8 cores, $3.6$ $\mathrm{G}$$\mathrm{Hz}$)
	Memory	16 GB DDR4 RAM
	Parallelism	Multiprocessing, 16 CPU threads
Software	Language	Python 3.8
	Numerical	NumPy 1.21.0
	Visualization	Matplotlib 3.4.0
Evaluation	Independent runs	30 for main comparative, scenario, ablation and generalization experiments
	Protocol-specific exceptions	Robustness: 5 training seeds × 30 noisy episodes per regime; timing: repeated per decision calls
	Training episodes	Up to 1500 (main); 1000 (multi-robot)
	Evaluation	Held-out suites (disjoint seed pools)

.

The following algorithms are compared: APF-Only [6] (greedy gradient descent on $U_{\mathrm{total}}$), Standard Q-Learning [8], Efficient Q-Learning (EQL) [13] (optimistic initialization, slow $\epsilon$ decay), Conservative Q-Learning (CQL) [33], Deep Q-Network (DQN) [9], QAPF (ours) and QAPF+CBF (ours).

5.2. Assumptions

The simulation and learning framework adopted in this study is based on the following assumptions:

1.

Nominal state observation: The robot has access to its position and to the positions of all obstacles located within the influence distance ${\rho }_0$. Robustness to bounded observation noise and actuator uncertainty is analyzed in Section 5.5.

2.

Deterministic nominal transitions: Under nominal conditions, executing a discrete action moves the robot deterministically by one grid cell in the selected direction. Robustness to bounded actuator slip and external disturbance is analyzed in Section 5.5.

3.

Known obstacle geometry: Obstacles are represented as point objects with a fixed collision radius of ${ϵ}_{\mathrm{coll}}=1.5$ cells.

4.

Dynamic obstacle model: Dynamic obstacles are assumed to move with constant velocity along fixed linear trajectories and to follow reflective boundary conditions at the workspace limits.

5.

Discrete-action space: The robot selects its motion command from the four cardinal actions $\mathcal{A}=\{\uparrow ,\downarrow ,\leftarrow ,\to \}$.

5.3. Main Results and Multi-Scenario Evaluation

Table 5. Main comparison in the static-obstacle setting with 15 obstacles, reporting means and standard deviations across independent seeds on held-out evaluation maps with the best values per metric shown in bold.

Method	Success Rate (%)	Collision Rate (%)	Min. Clearance (Cells)	Conv. Episode
APF-Only	$72.8\pm 4.1$	$7.9\pm 2.1$	$2.54\pm 0.12$	N/A ‡
Std. QL	$78.3\pm 3.4$	$22.4\pm 3.1$	$3.52\pm 0.35$	$385.0\pm 45.0$
EQL	$86.2\pm 2.8$	$14.3\pm 2.5$	$3.10\pm 0.28$	$320.0\pm 38.0$
CQL	$82.5\pm 3.1$	$18.5\pm 2.8$	$3.35\pm 0.32$	$350.0\pm 42.0$
DQN	$84.7\pm 3.5$	$16.8\pm 3.2$	$3.21\pm 0.42$	$300.0\pm 55.0$
QAPF	$\mathbf{94}.\mathbf{5}\pm \mathbf{2}.\mathbf{1}$	$6.2\pm 1.8$	$2.78\pm 0.15$	$205.0\pm 35.0$
QAPF+CBF	$93.8\pm 2.3$	$\mathbf{0}.\mathbf{3}\pm \mathbf{0}.\mathbf{4}$	$\mathbf{3}.\mathbf{15}\pm \mathbf{0}.\mathbf{18}$	$230.0\pm 40.0$

Metric definitions. SR = percentage of held-out episodes that reach the goal; CR = percentage of episodes terminating with ${\rho }_{min}<{ϵ}_{\mathrm{coll}}$; Min. Clearance = mean closest approach distance ${min}_t{\rho }_{min}({\mathbf{q}}_t)$ in grid cells, interpreted together with CR; Conv. Episode = interpolated plateau-crossing episode from the smoothed held-out SR curve, using the five-percentage-point threshold relative to the final/asymptotic mean. ^‡ APF-Only is a non-learning method and therefore has no convergence-of-learning episode. The ∼50 episodes previously reported for APF-Only correspond to the index at which its empirical success-rate trace stabilizes and is reported only in the running text rather than as a learning-convergence metric in this table. See also Table 17 (Axis A) for the same comparison evaluated under the wider multi-axis generalization protocol of 1000 episodes per cell rather than the 3000-episode held-out suite used here.

presents the primary comparison in the default static-obstacle setting. The metrics reported in the table are defined as follows. The success rate (SR) is computed as the fraction of held-out evaluation episodes terminating in the goal-reached outcome. The collision rate (CR) is the fraction of episodes terminating with ${\rho }_{min}<{ϵ}_{\mathrm{coll}}$. The minimum clearance is the mean over held-out episodes of ${min}_t{\rho }_{min}({\mathbf{q}}_t)$ in grid cells, where collision episodes are still counted through the CR, and the clearance value should therefore be interpreted jointly with the CR rather than as a standalone safety metric. The convergence episode is an estimated plateau-crossing episode obtained by linearly interpolating the smoothed held-out SR curve between logged evaluation checkpoints; it is therefore not constrained to be a multiple of the 50-episode logging interval. The plateau threshold is defined as the first episode at which the smoothed trailing held-out SR is within five absolute percentage points of the final/asymptotic mean.

In the held-out evaluation, the highest success rate of $94.5\pm 2.1\%$ is achieved by the proposed QAPF, and all baseline methods are significantly outperformed. A success rate of $93.8\pm 2.3\%$ with a near-zero collision rate of $0.3\pm 0.4\%$ is achieved by QAPF+CBF, which demonstrates that collisions are reduced to a near-zero level by the CBF safety filter without substantially compromising task completion. Among the baselines, EQL performs best at $86.2\%$ using optimistic Q-value initialization and slower exploration decay, followed by DQN at $84.7\%$, CQL at $82.5\%$ and Std. QL at $78.3\%$. APF-Only does not learn and therefore has no convergence episode in the learning sense. Its empirical success-rate trace is observed to stabilize within approximately 50 evaluation episodes at $72.8\%$ as a result of local-minima trapping; this stabilization index is reported only as descriptive context for the figure and is not comparable to the learning-convergence values listed for the Q-learning baselines. The comparison between QAPF ($6.2\%$ coll.) and QAPF+CBF ($0.3\%$ coll.) is internal to the proposed framework because the prior QAPF reference [12] does not report a comparable held-out collision rate. Accordingly, the approximately $20\times$ reduction is described as an internal-ablation effect rather than as an external comparison against published baselines. Two distinct effects of the CBF-inspired filter are made visible by this internal contrast: (i) the safe-mask ${\mathcal{M}}_s$ pushes the collision rate down by an order of magnitude and (ii) the visit memory ${\mathcal{M}}_f$ keeps the success rate competitive by preventing oscillation deadlocks that would otherwise time out.

Figure 3 shows the learning curves. QAPF reaches its plateau after approximately 205 episodes and QAPF+CBF by approximately 230 episodes. This is substantially shorter than the ∼3 × ${10}^6$ episodes reported by the original QAPF of Orozco-Rosas et al. [12]; however, the comparison is made across different reported protocols and is therefore interpreted as a protocol-dependent convergence-horizon contrast rather than as a controlled one-to-one speedup.

5.3.1. Multi-Scenario Evaluation

Table 6. Multi-scenario evaluation results. Metric definitions match Table 5.

Scenario	Method	Success Rate (%)	Collision Rate (%)	Min. Clearance (Cells)
Static	Std. QL	78.3	22.4	3.52
	EQL	86.2	14.3	3.10
	CQL	82.5	18.5	3.35
	DQN	84.7	16.8	3.21
	QAPF	$\mathbf{94}.\mathbf{5}$	6.2	2.78
	QAPF+CBF	93.8	$\mathbf{0}.\mathbf{3}$	3.15
Dynamic	Std. QL	61.5	31.8	2.85
	EQL	71.8	24.7	2.62
	CQL	67.2	28.3	2.74
	DQN	74.1	22.4	2.55
	QAPF	89.3	9.8	2.18
	QAPF+CBF	$\mathbf{90}.\mathbf{4}$	$\mathbf{0}.\mathbf{5}$	2.82
Narrow	Std. QL	55.8	37.6	3.95
	EQL	64.3	29.2	3.65
	CQL	60.1	34.8	3.78
	DQN	69.4	12.5	3.42
	QAPF	$\mathbf{85}.\mathbf{7}$	9.8	2.38
	QAPF+CBF	85.5	$\mathbf{0}.\mathbf{3}$	3.08

Metric definitions. SR = success rate; CR = collision rate; Min. Clearance = mean closest approach distance in grid cells. These definitions match Table 5; clearance should be read together with the CR because methods can have high mean clearance on non-collision trajectories while still failing in a separate subset of episodes.

reports the results across static, dynamic and narrow-passage scenarios.

Across all three scenarios, the baselines are consistently outperformed by the QAPF variants. In the static scenario, the highest SR ($94.5\%$) is obtained by QAPF and the lowest CR ($0.3\%$) by QAPF+CBF. In the dynamic scenario, both the highest SR ($90.4\%$) and the lowest CR ($0.5\%$) are obtained by QAPF+CBF. In the narrow-passage scenario, QAPF (best SR $85.7\%$) and QAPF+CBF (best CR $0.3\%$) are essentially tied on success while QAPF+CBF wins decisively on safety.

5.3.2. Representative Trajectories, Safety Analysis and Reward Stability

Figure 4 shows representative trajectories from the trained QAPF+CBF agent; the curvilinear paths reflect the zero-centered hybrid score of Equation (11), which produces smooth deflection around obstacles rather than sharp corners. In the figure, static obstacles are indicated by red filled circles drawn at the collision radius ${ϵ}_{\mathrm{coll}}=1.5$ cells; the start state ${\mathbf{s}}_0$ is shown by the green square and the goal position ${\mathbf{q}}_f$ by the gold star; the executed trajectory ${\left\{{\mathbf{q}}_t\right\}}_{t=0}^T$ is plotted as a blue solid curve; the influence radius ${\rho }_0=3.0$ cells, inside which the repulsive potential is non-zero, is rendered as a light-blue shaded region around each obstacle. Abrupt corners in the trajectory would indicate Q-table dominance, which is suppressed at evaluation time by the higher APF-guidance weight $w_g^{\mathrm{eval}}$. Figure 5 visualizes the barrier function $h(\mathbf{q})$ along trajectories with and without the CBF filter: without CBF, h drops below the safety margin on multiple occasions; with CBF, $h\ge {\delta }_{\mathrm{safe}}$ is maintained throughout the trajectory. Figure 6 shows that the raw shaping signal $\Delta U$ spans approximately $[-70,+70]$, while the clipped normalized signal $\widetilde{\Delta U}$ is bounded to $[-1,+1]$, confirming the stabilizing role of the clipping in Equation (7).

Figure 7 presents the Pareto frontier: QAPF+CBF sits closest to the utopia point at $(0.3\%\mathrm{collisions},93.8\%\mathrm{success})$, dominating every baseline along the safety axis while remaining within $0.7$pp of QAPF on success. The QAPF to QAPF+CBF transition trades $0.7$pp of success for an approximately 20-fold reduction in collisions, and the bottom-right baselines (Std QL, EQL, CQL) are clearly Pareto-dominated.

5.4. Empirical Unreachable-Goal and Stagnation Detection

To address cases in which the robot cannot reach the goal, an empirical unreachable-goal investigation is added to the evaluation protocol. This experiment is designed for practical failure recognition rather than for formal graph-theoretic reachability certification. Blocked-goal, sealed-corridor and dead-end concave maps are used as no-path cases. In these settings, a high success rate is not expected because the goal is infeasible under the four-neighbor grid action model. The desired behavior is instead safe termination with an unreachable label and a low collision-before-termination rate. The detailed detector definition and the empirical no-path table are provided in Appendix A.2.

5.5. Robustness Analysis Under Noise and Disturbances

A navigation system that is merely accurate under nominal conditions is of limited practical value if it collapses under the sensor noise and actuator imperfections that characterize real deployments. A dedicated robustness study is therefore conducted with both a theoretical component and a held-out experimental protocol.

5.5.1. Noise Model

Three independent noise channels are injected into the evaluation environment only (policies are trained on clean dynamics). Let ${\mathbf{q}}_t$ denote the robot’s true position at step t, $a_t$ be the nominal action selected by the policy and $a_t^{\mathrm{exec}}$ be the action executed by the actuator:

$$\begin{array}{cccc}\hfill {\tilde{\mathbf{q}}}_t& ={\mathbf{q}}_t+{\mathit{\eta }}_t^{\mathrm{obs}},{\mathit{\eta }}_t^{\mathrm{obs}}\sim \mathcal{N}(\mathbf{0},{\sigma }_{\mathrm{obs}}^2{\mathbf{I}}_2)\hfill & \hfill & (\mathrm{observation}\mathrm{noise}),\hfill \end{array}$$

(18)

$$\begin{array}{cccc}\hfill a_t^{\mathrm{exec}}& =\left\{\begin{array}{cc}{a}_t\hfill & \mathrm{w}.\mathrm{p}.1-p_{\mathrm{act}}\hfill \\ \mathcal{U}(\mathcal{A})\hfill & \mathrm{w}.\mathrm{p}.p_{\mathrm{act}}\hfill \end{array}\right.\hfill & \hfill & (\mathrm{actuator}\mathrm{slip}),\hfill \end{array}$$

(19)

$$\begin{array}{cccc}\hfill {\mathbf{q}}_{t+1}& ={\mathbf{q}}_t+{\mathbf{\Delta }}_{a_t^{\mathrm{exec}}}+{\mathit{\eta }}_t^{\mathrm{ext}},{\mathit{\eta }}_t^{\mathrm{ext}}\sim \mathcal{N}(\mathbf{0},{\sigma }_{\mathrm{ext}}^2{\mathbf{I}}_2)\hfill & \hfill & (\mathrm{external}\mathrm{drift}).\hfill \end{array}$$

(20)

The perturbed observation ${\tilde{\mathbf{q}}}_t$ given by Equation (18) is used to recompute the state-space bins in Equation (1), so the noise actually affects policy output rather than being a no-op. The inflated-margin argument of Equations (21) and (22) is source-agnostic: it depends only on a bounded perturbation $\parallel \eta \parallel \le {\delta }_p$ of the position used to evaluate the barrier h, regardless of whether the perturbation originates from robot self-localization noise, obstacle-position noise or goal-position noise (see Appendix A.5 for the unified treatment). Consequently, the experimental sweep of Table 8, which injects noise on the robot pose used by the policy and operates as a worst-case proxy for diagonal-Gaussian obstacle- and goal-pose uncertainty of comparable or smaller magnitude, because the binned-state features driven directly by $\mathbf{q}$ (the absolute bins $x_{\mathrm{bin}},y_{\mathrm{bin}}$ of Equation (1)) are perturbed by robot-pose noise but not by obstacle- or goal-pose noise alone. The non-Gaussian or correlated obstacle/goal localization error (e.g., systematic SLAM drift) remains an open extension, as discussed in Section 5.9. The actuator slip of Equation (19) is applied to the nominal action, and the external drift of Equation (20) is applied after the nominal transition and before termination re-check. Six primary regimes are reported in

Table 7. Primary noise regimes reported in the robustness study. All noise is i.i.d. per step.

Regime	${\mathit{\sigma }}_{\mathbf{obs}}$ (Cells)	${\mathit{p}}_{\mathbf{act}}$	${\mathit{\sigma }}_{\mathbf{ext}}$ (Cells)	Physical Interpretation
clean	0.0	0.00	0.00	ideal conditions
obs_low	0.3	0.00	0.00	light sensor noise
obs_high	0.8	0.00	0.00	heavy sensor noise
act_low	0.0	0.05	0.00	mild actuator slip
act_high	0.0	0.15	0.00	severe actuator slip
combined	0.3	0.05	0.10	realistic mixed disturbance

.

5.5.2. Theoretical Analysis

For the QAPF+CBF agent under observation noise, a deterministic guarantee can only be stated after replacing the Gaussian perturbation with a bounded or high-probability error radius. Let ${\delta }_p$ be a bound satisfying $Pr(\parallel {\eta }^{\mathrm{obs}}\parallel \le {\delta }_p)\ge p$. For Gaussian observation noise in two dimensions, ${\delta }_p$ may be chosen from the corresponding ${\chi }_2^2$ quantile. Conditional on this high-probability event, the safe set can be inflated by ${\delta }_p$:

$${\tilde{\mathcal{C}}}_p=\{\mathbf{q}:h(\mathbf{q})\ge {\delta }_{\mathrm{safe}}+{\delta }_p\},$$

(21)

where ${\tilde{\mathcal{C}}}_p$ is the inflated high-probability safe set, $h(\mathbf{q})$ is the discrete barrier function of Equation (12), ${\delta }_{\mathrm{safe}}$ is the nominal safety margin and ${\delta }_p$ is the radius of the high-probability error ball at confidence level p. If the filter enforces $h({\tilde{\mathbf{q}}}_t)\ge {\delta }_{\mathrm{safe}}+{\delta }_p$ using the noisy observation, then by the reverse triangle inequality

$$h({\mathbf{q}}_t)=\rho ({\mathbf{q}}_t)-{ϵ}_{\mathrm{coll}}\ge \rho ({\tilde{\mathbf{q}}}_t)-\parallel {\mathbf{q}}_t-{\tilde{\mathbf{q}}}_t\parallel -{ϵ}_{\mathrm{coll}}\ge h({\tilde{\mathbf{q}}}_t)-{\delta }_p\ge {\delta }_{\mathrm{safe}}\ge 0.$$

(22)

Thus, the inflated-margin argument provides a conditional, high-probability safety statement for observation-side perturbations, not an unconditional guarantee for unbounded Gaussian noise: with probability at least p over the realization of ${\eta }^{\mathrm{obs}}$, the filter enforces $h({\mathbf{q}}_t)\ge {\delta }_{\mathrm{safe}}\ge 0$; the complement event of probability at most $1-p$ corresponds to draws of ${\eta }^{\mathrm{obs}}$ that exceed the chosen radius ${\delta }_p$ and for which no safety claim is made. Actuator slip and external drift remain harder because they perturb the realized transition after the action has been selected; they cannot be eliminated by observation-side margin inflation alone.

5.5.3. Experimental Results

Table 8. Robustness study reporting the success rate (%) under each primary noise regime. The protocol of this table is a finite robustness screen and uses noisy evaluation episodes per regime, distinct from the headline static-condition evaluation of Table 5, which uses the same 30 seeds but 100 clean held-out episodes. The clean-regime row of this table is therefore a robustness-screen reference and not a replacement for the headline static-obstacle number reported in Table 5; the two protocols are deliberately kept separate. The best success rate per regime is in bold.

Regime	APF-Only	QAPF	QAPF+CBF
clean	$86.7\pm 6.2$	$85.0\pm 10.8$	$\mathbf{100}.\mathbf{0}\pm \mathbf{0}.\mathbf{0}$
obs_low	$86.7\pm 6.2$	$75.0\pm 4.1$	$\mathbf{100}.\mathbf{0}\pm \mathbf{0}.\mathbf{0}$
obs_high	$86.7\pm 6.2$	$81.7\pm 2.4$	$\mathbf{100}.\mathbf{0}\pm \mathbf{0}.\mathbf{0}$
act_low	$75.0\pm 10.8$	$75.0\pm 7.1$	$\mathbf{83}.\mathbf{3}\pm \mathbf{2}.\mathbf{4}$
act_high	$65.0\pm 10.8$	$63.3\pm 8.5$	$\mathbf{70}.\mathbf{0}\pm \mathbf{0}.\mathbf{0}$
combined	$78.3\pm 2.4$	$68.3\pm 11.8$	$\mathbf{85}.\mathbf{0}\pm \mathbf{4}.\mathbf{1}$

deliberately reports the success rate only. The collision rate and minimum clearance under each noise regime are retained in the raw robustness logs available from the corresponding author upon reasonable request but are not tabulated here because the 150-episode per cell sample produces wide confidence bands on rare-event metrics: a single observed collision in 150 episodes corresponds to an empirical CR of $0.67\%$ with a Wilson $95\%$ upper bound near $3.6\%$, so the per regime CR contrasts at the sub-$1\%$ level are not statistically resolvable at this episode budget. The full SR/CR/minimum-clearance noisy–safety benchmark over a larger episode budget is identified in Section 5.9 as the natural follow-up rather than as a current claim.

The success-rate degradation per noise regime is summarized graphically in Figure 8.

Three observations follow from the results. First, QAPF+CBF shows no observed success-rate degradation under the two observation-noise regimes in this finite suite ($100\%$ SR in obs_low/obs_high); this is consistent with the inflated-safe-set argument, but it should not be interpreted as a universal guarantee for all Gaussian perturbations. Because the robustness suite contains 150 evaluation episodes per cell, a value of $100.0\pm 0.0\%$ means that no failures were observed in this finite sample, not that the true failure probability is zero. For example, a $150/150$ binomial success count corresponds to an approximate $95\%$ Wilson lower confidence bound of about $97.5\%$. Second, actuator slip is the hardest regime for all methods: even at $p_{\mathrm{act}}=0.15$, QAPF+CBF drops to $70\%$ because the filter cannot guarantee execution of the safe action it selects; the slip is downstream of the decision and cannot be filtered. Third, under the realistic combined regime, QAPF+CBF retains $85\%$ success, over 16 points higher than QAPF alone ($68.3\%$) and seven points higher than APF-Only ($78.3\%$), which is the most practically relevant robustness number in this study.

5.6. I7 Inference Timing and Embedded-Device Latency Projection

The per decision inference latency of each method is physically measured on a laptop-class Intel i7 reference and then projected to representative embedded and mobile device classes. The embedded-device values are projection estimates, not direct Jetson/DGX measurements.

5.6.1. Measurement Protocol

For each method–seed pair, the agent is trained under the standard protocol; a fresh evaluation environment is then instantiated, and the following sequence is executed: (i) $n_{\mathrm{warmup}}=50$ untimed select_action calls to populate CPU caches and JIT paths; (ii) $n_{\mathrm{timed}}=2000$ timed calls, each wrapped in time.perf_counter() with microsecond resolution. The median of the 2000 samples is reported in

Table 9. Inference latency: median and mean per decision time ($\mathsf{\mu }$s) physically measured on the Intel i7 reference and projected median latency on embedded/mobile device classes. The Jetson Nano, Jetson Orin NX, Jetson AGX Orin and DGX columns are projection estimates based on Table 10, not direct on-device measurements. The “Hz @ i7 (mean)” column follows the implementation and reports the throughput computed from the mean call latency, not from the median column. Projected embedded-device columns are therefore reported as median-latency values, not as directly comparable mean-throughput measurements. All methods exceed the 20 Hz real-time threshold on the measured i7 reference and under all projected device-class latencies. We emphasize that only the i7 column is a physical measurement; the Jetson Nano, Orin NX, AGX Orin and DGX columns are device-class projections obtained by applying the conservative CPU-side scaling factors of Table 10, and they are not a substitute for direct on-device measurement. Direct physical Jetson-AGX-Orin and Jetson-Nano latency measurements, together with calibration of the Table 10 multipliers against measured ratios, are identified as future work item (i) in Section 6.

Method	i7 Median (µs)	i7 Mean (µs)	Nano	Orin NX	AGX Orin	DGX	Hz @ i7 (Mean)	20 Hz?
APF-Only	158.6	228.2	380.6	174.5	142.7	111.0	4382	✔ †
Std. QL	9.0	14.7	21.6	9.9	8.1	6.3	68,000	✔
DQN	13.4	17.5	32.2	14.7	12.1	9.4	57,000	✔
QAPF	149.4	173.6	358.5	164.3	134.4	104.6	5761	✔
QAPF+CBF	173.7	204.1	417.0	191.1	156.4	121.6	4899	✔

^† APF-Only on Jetson Nano has a projected median latency of $380.6$ $\mathsf{\mu }$$\mathrm{s}$, comfortably below the 50 ms budget of a 20 Hz control loop. Percentile latencies beyond the median are retained in the raw timing logs available from the corresponding author upon reasonable request rather than in the compact manuscript table.

to provide a robust summary against GC-induced outliers. The $p95$ and $p99$ latency values are retained in the raw timing output available from the corresponding author upon reasonable request but are not tabulated in this manuscript to keep the main table compact. The throughput column follows the implementation and is computed from the mean-call latency as ${\mathrm{Hz}}_{\mathrm{ref}}=1/\overline{t}$; therefore, it should not be recomputed as the reciprocal of the median-latency column.

5.6.2. Device Extrapolation

Direct on-device measurements across all target platforms were not performed in this study. Device-class projections are therefore reported, obtained by multiplying the measured i7 reference latency by conservative CPU-side scaling factors (

Table 10. Conservative CPU-side scaling factors used for device-class latency projection. Factor $=\mathrm{median}{\mathrm{latency}}_{\mathrm{device}}/\mathrm{median}{\mathrm{latency}}_{\mathrm{i}7\mathrm{ref}}$. These are projections, not physical on-device measurements.

Platform	Scaling Factor	Rationale
Intel i7 (reference)	$\times 1.0$	Baseline (laptop-class, 8 cores, $3.6$ $\mathrm{G}$$\mathrm{Hz}$)
Jetson Nano	$\times 2.4$	ARM Cortex-A57 @ $1.43$ $\mathrm{G}$$\mathrm{Hz}$, no AVX
Jetson Orin NX	$\times 1.1$	ARM Cortex-A78AE @ $2.0$ $\mathrm{G}$$\mathrm{Hz}$
Jetson AGX Orin	$\times 0.9$	ARM Cortex-A78AE @ $2.2$ $\mathrm{G}$$\mathrm{Hz}$, wider memory
DGX-level GPU (proxy)	$\times 0.7$	CPU-side overhead only; numpy tables do not saturate GPU

). These projections are intended to answer whether the method is plausibly real time on embedded hardware; they do not replace physical Jetson/DGX measurements. DGX is also server-class rather than a mobile-robot platform, so it is included only as an upper-bound compute reference. Real on-device measurements can vary by $\pm 20$ to $50\%$, depending on cache effects, Python/C++ implementation, power mode and memory bandwidth.

5.6.3. Results

The lightweight decision policies are the fastest: Std. QL uses a single Q-table lookup, while the compact DQN baseline uses a short one-hidden-layer forward pass. APF-Only, QAPF and QAPF+CBF all require evaluating the potential at four candidate successor cells, which involves per obstacle distance computations, a $\mathcal{O}(|\mathcal{A}\left|·\right|\mathcal{O}|)$ operation. Despite this overhead, all methods exceed the 20 Hz real-time threshold on the measured i7 reference and under the projected embedded-device latency model. For QAPF+CBF, the implementation reports approximately $4.9$ kHz mean-call throughput on the i7 reference, while the projected median latencies are $417.0$ $\mathsf{\mu }$$\mathrm{s}$ on Jetson Nano and $156.4$ $\mathsf{\mu }$$\mathrm{s}$ on Jetson AGX Orin. These projected median-latency equivalents should not be directly compared with the “Hz @ i7” mean-throughput column.

5.7. Ablation, Sensitivity and Extended Studies

5.7.1. Ablation Study

The contribution of each QAPF component is isolated in

Table 11. Ablation study: isolated contribution of each QAPF component. RL-Only here denotes a strict component-isolation configuration; it removes the QAPF-specific adaptive shaping, APF-weighted softmax exploration and anti-deadlock monitor and is distinct from the Std. QL baseline reported in Table 5. The reported $250\pm 0$ convergence value for RL-Only denotes the convergence-evaluation cap; RL-Only did not reach the convergence criterion within the ablation budget across any seed.

Variant	Success (%)	Collision (%)	Clearance	Conv. Ep
RL-Only	$45.2\pm 5.8$	$48.7\pm 6.3$	$1.75\pm 0.35$	$250\pm 0$
APF-Only	$72.8\pm 4.1$	$7.9\pm 2.1$	$2.54\pm 0.12$	N/A
QAPF-Fixed-$\lambda$	$88.9\pm 3.2$	$12.1\pm 2.8$	$2.56\pm 0.14$	$180\pm 45$
QAPF-Full	$\mathbf{94}.\mathbf{5}\pm \mathbf{2}.\mathbf{1}$	$6.2\pm 1.8$	$2.78\pm 0.15$	$205\pm 35$
QAPF+CBF	$93.8\pm 2.3$	$\mathbf{0}.\mathbf{3}\pm \mathbf{0}.\mathbf{4}$	$\mathbf{3}.\mathbf{15}\pm \mathbf{0}.\mathbf{18}$	$230\pm 40$

and visualized in Figure 9: RL-Only (a strict component-isolation configuration: plain tabular Q-learning on the QAPF state encoding and reward, with all QAPF-specific components removed; this is intentionally distinct from the Std. QL baseline of Table 5, which is tuned as a fair comparison opponent), APF-Only (no learning), QAPF-Fixed-$\lambda$ (no adaptive schedule), the full QAPF-Full and QAPF+CBF.

A clear story is told by the ablation. The adaptive shaping schedule of Equation (8) is observed to improve over the fixed-$\lambda$ variant by approximately $5.6$ percentage points ($94.5\%$ vs. $88.9\%$), and the value of the decay schedule for late-stage policy refinement is therefore demonstrated. A small drop in the success rate ($94.5\%$ to $93.8\%$) is traded by the CBF filter for an approximately $20\times$ reduction in collisions ($6.2\%$ to $0.3\%$), and the minimum clearance is simultaneously improved from $2.78$ to $3.15$ cells, which confirms that the robot is being kept further from obstacles by the filter.

5.7.2. Sensitivity Analysis

Table 12. Diagnostic sensitivity analysis under a separate exploratory sweep configuration (${\lambda }_{min}=1.0$, $T_0=5.0$, $w_g^{\mathrm{train}}=0.30$, $w_g^{\mathrm{eval}}=0.90$). These settings are not the production defaults used in Table 5; the production defaults are listed in Table 3.

${\mathit{\lambda }}_{max}$	$\mathit{\beta }$	Success Rate (%)	Conv. Episode
5	0.005	90.8	225
5	0.01	91.7	218
5	0.02	89.1	210
10	0.005	92.1	245
10	0.01	94.5	198
10	0.02	89.8	176
20	0.005	88.2	258
20	0.01	90.3	231
20	0.02	87.4	219

Reading note. The bolded $({\lambda }_{max}=5,\beta =0.005)$ cell uses the production shaping schedule but not the exploratory exploration settings of this sweep; it is therefore not numerically equal to the QAPF row of Table 5 ($94.5\%/205$ ep), which uses the production exploration settings of Table 3. The numerically matching cell at $({\lambda }_{max}=10,\beta =0.01)$ is a coincidence of two independent runs and should not be interpreted as the production operating point.

reports a diagnostic QAPF sensitivity sweep over the shaping magnitude ${\lambda }_{max}$ and decay rate $\beta$ of Equation (8). This sweep uses a separate exploratory configuration in the code (${\lambda }_{min}=1.0$, $T_0=5.0$, $T_{min}=0.5$, $T_{\mathrm{decay}}=0.99$, $w_g^{\mathrm{train}}=0.30$, $w_g^{\mathrm{eval}}=0.90$) and characterizes behavior around a wider parameter neighborhood; the final QAPF/QAPF+CBF parameters used by make_agent to produce Table 5, Table 6 and Table 11 are those reported in Table 3: ${\lambda }_{min}=0.5$, ${\lambda }_{max}=5.0$, $\beta =0.005$, $T_0=2.0$, $T_{min}=0.3$, $T_{\mathrm{decay}}=0.995$ and APF-guidance weights $w_g^{\mathrm{train}}=1.2$, $w_g^{\mathrm{eval}}=2.0$. The flat success-rate landscape across the center of the table (all SRs within $\pm 3$pp of the maximum) is consistent with a robust setting of the shaping coefficient, which is the operationally important property of a sensitivity analysis. Figure 10 visualizes the same data as a success-rate landscape over $({\lambda }_{max},\beta )$ together with the corresponding convergence-episode landscape.

5.7.3. Energy Modulation Study

The kinetic and jerk energy and navigation time of Equations (13) to (15) are reported in

Table 13. Energy-aware velocity modulation (Equation (13) to (15)). E is the total kinetic plus jerk energy accumulated over the episode; $E/E_{\mathrm{const}}$ is the energy ratio relative to the constant-speed baseline; Nav. Time is the resulting effective navigation time in the simulator. The mean speed $\overline{v}$ is used internally to compute the timing profile but is not tabulated separately.

Regime	${\mathit{k}}_{\mathit{E}}$	SR (%)	E	$\mathit{E}/{\mathit{E}}_{\mathbf{const}}$	Nav. Time
Constant	${10}^{-3}$	94.5	60.05	1.00	14.2
APF-mod	0.5	94.5	49.25	0.82	15.4
APF-agg	1.5	94.5	42.04	0.70	20.8

across the three regimes. The same $94.5\%$ success rate as that of the unmodulated QAPF policy is maintained across all regimes, which confirms the compatibility of the energy module with the QAPF policy. Figure 11 visualizes the resulting velocity profiles and the energy and navigation-time trade-off across the three modes.

APF-mod delivers an $18\%$ energy reduction for an $8.5\%$ increase in navigation time; a favorable trade-off for battery-limited deployments. APF-agg pushes energy savings to $30\%$ but at a $46.5\%$ higher navigation time. The sigmoidal speed law of Equation (15) is observed to produce velocity dips only near obstacles, which concentrates the energy savings where safety is most important. The $94.5\%$ success rate is preserved across all three modes, which is consistent with the static-scenario QAPF row of Table 6: velocity modulation is shown to alter the energy and navigation-time trade-off without affecting task completion.

5.7.4. Multi-Robot Study

Table 14. Two-robot multi-robot study results.

Configuration	Joint Success Rate (%)	Inter-Robot Collisions
Independent	$79.5$	$14.3$
Cooperative	$89.2$	$5.1$
Coop+CBF	$\mathbf{92}.\mathbf{1}$	$\mathbf{0}.\mathbf{5}$

reports three configurations for the same two-robot evaluation setting: independent (no coordination), cooperative (inter-robot APF repulsion per Equation (17)) and Cooperative+CBF. Figure 12 visualizes the joint success rate and inter-robot collision count across the three two-robot configurations. Adding inter-robot APF repulsion alone is shown to be insufficient: residual $5.1$ inter-robot collisions are observed in the Cooperative configuration. When the inter-robot repulsion is paired with the per robot CBF filter, the joint success is driven to $92.1\%$ with only $0.5$ residual inter-robot collisions.

Scalability Discussion

The Cooperative+CBF configuration evaluated above uses a centralized, fully- communicative coordination scheme: at each decision step, every robot k computes the augmented potential $U({\mathbf{q}}_k)+{\sum }_{j\ne k}{U}_{\mathrm{rep}}^{\mathrm{MR}}({\mathbf{q}}_k,{\mathbf{q}}_j)$ over the positions of all other robots. This design is appropriate for the two-robot regime evaluated here (warehouse aisles, lab-scale fleets), but its scaling properties deserve explicit comment for larger deployments. The dominant costs are summarized in

Table 15. Theoretical scaling cost of the centralized cooperative scheme as the number of robots N grows. The table separates the compute/communication scaling from the fleet-level joint-success bound, because both become limiting factors in large multi-robot deployments. The “$N=10$ example” column reports the cost magnitude evaluated at $N=10$ (e.g., $\sim 100$ pair evaluations per decision step at ∼2$\mathsf{\mu }$$\mathrm{s}$ per pair, consistent with the reference-platform timing of Section 5.6).

Cost Dimension	Scaling	$\mathit{N}=10$ Example	Main Bottleneck
Per decision pair evaluations	$\mathcal{O}(N^2)$	90 directed/45 unique pairs	Runtime at large N
Communication bandwidth	$\mathcal{O}(N^2)$	$\sim 90$ state messages per cycle	Wireless contention
Fleet-level joint success	${\prod }_{k=1}^N{\mathrm{SR}}_k$	$0.{938}^{10}\approx 52.7\%$	Joint SR drops rapidly
Single-point-of-failure risk	Centralized coordinator	One coordinator	Resilience requirement

.

The $\mathcal{O}(N^2)$ pair-evaluation and communication costs remain tractable for small fleets but become increasingly restrictive as N grows. More importantly, fleet-level success decreases multiplicatively: even if each robot individually maintains the QAPF+CBF success rate of $93.8\%$, the independent-failure upper bound is $0.{938}^N$, which falls below $50\%$ at $N\approx 11$. Beyond small fleets, three concrete decentralization strategies become necessary: (i) k-nearest-neighbor repulsion (restrict the inter-robot sum to the $k=3$ to 5 closest other robots, dropping cost to $\mathcal{O}(Nk)$ [18]); (ii) communication-graph-aware coordination (explicit communication graph respecting wireless range and bandwidth, decentralized consensus protocol [17,18]); and (iii) hierarchical-cluster coordination (3 to 5-robot clusters with gateway-robot boundary handling, giving approximately $\mathcal{O}(N)$ overall coordination cost).

Limitations of the present evaluation that follow from this analysis. The $92.1\%$ joint SR result in Table 14 should not be read as a claim about N-robot scalability: it is a two-robot evaluation, and as N grows, the joint SR is mathematically bounded above by ${\prod }_k{\mathrm{SR}}_k$ (under independent-failure assumptions), so even individually-strong robots can yield joint SR $\ll 92\%$ for large fleets; specifically, $0.{938}^N<0.5$ already at $N\approx 11$. Fleet-scale validation is treated as the most important extension of the multi-robot story for future work.

5.7.5. Curriculum Learning Study

Table 16. Curriculum learning study comparing fixed-density and progressive-curriculum training schedules.

Training Mode	Success Rate (%)	Convergence Episode
Fixed	$94.5\pm 2.1$	$205\pm 35$
Curriculum	$\mathbf{96}.\mathbf{2}\pm \mathbf{1}.\mathbf{8}$	$\mathbf{120}\pm \mathbf{25}$

compares fixed-density training against a progressive-curriculum schedule that starts at low obstacle density and increases towards the target density across episodes; Figure 13 visualizes the corresponding learning curves.

Two effects are visible. On the asymptotic success rate, fixed-density and curriculum training are statistically indistinguishable: the $1.7$ percentage-point difference ($94.5\%$ vs. $96.2\%$) is comparable to one standard deviation of either estimate. On convergence speed, however, the curriculum schedule is clearly faster ($120\pm 25$ vs. $205\pm 35$ episodes; the gap is well outside one standard deviation), which is the more interesting result for sample-efficiency-constrained deployments. The curriculum benefit is most pronounced when the gap between the easy starting density and the target density is wide, because the agent receives a more informative shaping signal early in training; when the starting density is already close to the target, curriculum and fixed schedules collapse to essentially the same trajectory.

Curriculum Schedule Details

The progressive-curriculum schedule used in this study is parameterized as follows. The initial obstacle density is $N_o^{(0)}=5$, the target density is $N_o^{(\mathrm{target})}=15$ and the ramp duration is $E_{\mathrm{ramp}}=400$ episodes. The density at episode e is set to $N_o(e)=N_o^{(0)}+(N_o^{(\mathrm{target})}-N_o^{(0)})·min(e/E_{\mathrm{ramp}},1)$. After episode $E_{\mathrm{ramp}}$, the density is held fixed at $N_o^{(\mathrm{target})}$ for the remainder of training. The same map seed is used at every density step so that the obstacle layout grows monotonically rather than being resampled at each episode.

5.7.6. Multi-Axis Generalization Study

Generalization is measured by the gap between performance on seen obstacle configurations, which are drawn from the same seed pool used during training and performance on out-of-distribution configurations that the agent never encountered during training. Four orthogonal generalization axes are evaluated, each on 1000 held-out evaluation episodes per cell (30 seeds and approximately 33 episodes per seed). The four axes are listed below. Axis A covers held-out maps under the training distribution ($N_o=15$, $G=50$, disjoint seed pool). Axis B covers out-of-distribution obstacle density ($N_o\in \{5,10,15,20,25\}$, a $-66.7\%$ to $+66.7\%$ shift around the training value $N_o=15$). Axis C covers out-of-distribution grid size ($G\in \{30,50,70\}$, with the relative density held fixed at $15/2500$). Axis D covers long-horizon stability over 1000 consecutive episodes with the per decile success rate.

Table 17. Multi-axis generalization study. All numbers are success rate (%) on 1000 held-out evaluation episodes per cell, 30 seeds. Best per row in bold.

Generalization Axis	Cell	APF-Only	DQN	QAPF	QAPF+CBF
Axis A. Held-out maps under the training distribution
A1 Seen (training pool)	$N_o=15$ $G=50$	$72.5\pm 4.3$	$84.2\pm 3.8$	$\mathbf{94}.\mathbf{1}\pm \mathbf{2}.\mathbf{3}$	$93.4\pm 2.5$
A2 Unseen (disjoint seeds)	$N_o=15$ $G=50$	$71.8\pm 4.8$	$81.5\pm 4.5$	$\mathbf{93}.\mathbf{5}\pm \mathbf{2}.\mathbf{8}$	$92.8\pm 3.1$
Axis B. Out-of-distribution obstacle density (trained on $N_o=15$)
B1 $N_o=5$ (sparse)	$G=50$	$84.6\pm 3.1$	$89.3\pm 2.5$	$\mathbf{96}.\mathbf{8}\pm \mathbf{1}.\mathbf{4}$	$96.5\pm 1.7$
B2 $N_o=10$	$G=50$	$77.9\pm 3.6$	$86.7\pm 3.0$	$\mathbf{95}.\mathbf{4}\pm \mathbf{1}.\mathbf{9}$	$95.0\pm 2.1$
B3 $N_o=15$ (in-dist.)	$G=50$	$71.8\pm 4.8$	$81.5\pm 4.5$	$\mathbf{93}.\mathbf{5}\pm \mathbf{2}.\mathbf{8}$	$92.8\pm 3.1$
B4 $N_o=20$	$G=50$	$63.4\pm 5.5$	$74.1\pm 5.2$	$\mathbf{87}.\mathbf{6}\pm \mathbf{3}.\mathbf{4}$	$87.1\pm 3.6$
B5 $N_o=25$ (dense)	$G=50$	$52.1\pm 6.4$	$63.5\pm 5.9$	$\mathbf{79}.\mathbf{2}\pm \mathbf{4}.\mathbf{1}$	$78.5\pm 4.4$
Axis C. Out-of-distribution grid size (trained on $G=50$, density $15/2500$ held fixed)
C1 $G=30$ (smaller)	$N_o=5$	$70.4\pm 5.1$	$79.2\pm 4.4$	$\mathbf{90}.\mathbf{7}\pm \mathbf{3}.\mathbf{0}$	$90.2\pm 3.2$
C2 $G=50$ (in-dist.)	$N_o=15$	$71.8\pm 4.8$	$81.5\pm 4.5$	$\mathbf{93}.\mathbf{5}\pm \mathbf{2}.\mathbf{8}$	$92.8\pm 3.1$
C3 $G=70$ (larger)	$N_o=29$	$66.2\pm 5.6$	$76.4\pm 4.9$	$\mathbf{89}.\mathbf{0}\pm \mathbf{3}.\mathbf{5}$	$88.4\pm 3.7$

Cross-reference to Table 5. Axis A row A2 ($93.5\pm 2.8\%$ QAPF, $92.8\pm 3.1\%$ QAPF+CBF) is the closest counterpart to the static held-out comparison of Table 5 ($94.5\pm 2.1\%$ and $93.8\pm 2.3\%$ respectively). The two evaluations were run as independent suites: Table 5 uses 30 seeds × 100 held-out episodes (=3000 episodes), whereas the generalization suite uses 30 seeds × approximately 33 episodes per seed (=1000 episodes per cell). The $\sim 1$pp offset is within one standard deviation of either estimate and is consistent with finite-sample variation across the two protocols.

(Axes A to C) shows the four-method comparison aggregated across each axis, while

Table 18. Axis D. Long-horizon stability of QAPF+CBF reported as the success rate (%) per consecutive 100-episode decile across 1000 held-out episodes and 30 seeds. The flat profile (all deciles at or above $93\%$) confirms that no policy drift, replay-buffer corruption, or oscillation accumulation is observed over long deployment horizons.

Decile	D1	D2	D3	D4	D5	D6	D7	D8	D9	D10
SR (%)	$93.6$	$93.4$	$93.8$	$93.5$	$93.9$	$93.7$	$93.5$	$93.8$	$94.0$	$93.6$

(Axis D) reports the per decile stability of QAPF+CBF.

Three findings from the expanded study are worth highlighting. On Axis B, QAPF+CBF is observed to degrade by only $14.3$ percentage points ($92.8\%$ to $78.5\%$) when the obstacle count is scaled up by $66.7\%$ relative to training, whereas 20 points are lost by APF-Only and 18 points by DQN. This robustness to obstacle-density shift is a direct consequence of the relative-feature state encoding (Equation (1)), because $d_o$ and ${\beta }_o$ depend only on the nearest obstacle, and the policy is therefore largely insensitive to how many obstacles are placed in the rest of the field. On Axis C, the policy is observed to transfer cleanly to grid sizes the agent never trained on, with an $88\%$ to $90\%$ SR retained at both $G=30$ and $G=70$, because the bin counts (five position bins per axis) are normalized by grid size during state encoding. On Axis D, a flat $93.4\%$ to $94.0\%$ SR is maintained by QAPF+CBF across all ten deciles of a 1000-episode evaluation, with the slight per decile fluctuation well within the per decile sampling noise. No evidence of long-horizon drift or wear-out is observed in this finite evaluation, and deployment claims for tasks that run for many thousands of decisions per session are supported.

5.8. Three-Dimensional Workspace Evaluation

A simulation-based examination of the 3D extension described in Section 4.9 is reported here so that the dimensional generalization of QAPF+CBF can be evaluated on the same protocol as the 2D study. We state two scope limitations of this evaluation up-front. First, the simulator used here is a $50\times 50\times 1$ voxel grid with the vertical coordinate held at $z=0$; the 2D plane is therefore recovered as a degenerate special case of the 3D formulation, and the experiment is a code-path regression check rather than a non-trivial vertical-trajectory benchmark. Second, no stacked-obstacle, vertical-corridor or fly-over-fly-under scenario is exercised in this paper. A non-trivial 3D evaluation in which the z-axis is genuinely exercised is identified as a future work follow-up (item (viii) in Section 6), and a real-aerial drone validation is the planned vehicle for that follow-up. A real-aerial validation of the same 3D formulation, in which a quadrotor drone is flown through a structured indoor environment, is therefore deferred to future work item (viii) in Section 6. The 3D-on-2D-plane evaluation reproduces the static-obstacle headline metrics within sampling noise: a SR of $93.6\pm 2.6\%$ with a CR of $0.4\pm 0.5\%$ is observed for QAPF+CBF, and a SR of $94.3\pm 2.3\%$ with a CR of $6.4\pm 1.9\%$ is observed for QAPF on the 3D simulator. These numbers are consistent with the 2D static-obstacle results of Table 5 and confirm that the 3D code path does not degrade performance on a problem that the 2D agent solves cleanly. A non-trivial 3D evaluation in which the z-axis is genuinely exercised, for example, through stacked obstacles or vertical corridors, is left as the natural follow-up that the drone-based validation will provide.

5.9. Discussion, Limitations and Threats to Validity

The experimental results consistently support the central claim that adaptive APF-guided Q-learning, combined with a CBF-inspired discrete-action filter with per episode visit memory, produces a safe and efficient navigation policy. The CBF filter delivers an exceptional safety-vs-efficiency trade-off: ablating it lifts the success rate marginally from $93.8\%$ to $94.5\%$ but reintroduces a $\sim 20\times$ higher collision rate ($0.3\%\to 6.2\%$), which is the wrong direction for any safety-critical deployment.

The robustness analysis shows that QAPF+CBF degrades gracefully: it shows no observed degradation under observation noise up to ${\sigma }_{\mathrm{obs}}=0.8$ cells in the finite robustness suite (courtesy of the inflated-margin argument of Section 5.5) and loses at most 30 percentage points under severe actuator slip. Under the realistic combined regime the agent retains $85\%$ success, which we view as evidence of deployability.

Because the robustness table reports the SR rather than the full SR/CR/clearance triplet used in Table 5, it should be interpreted as a finite robustness screen rather than as a complete noisy–safety benchmark. A full noisy-environment safety benchmark reporting the SR, CR and minimum clearance under each disturbance regime is left as a follow-up experiment.

The inference-timing analysis indicates real-time viability on the measured Intel i7 reference and under projected embedded-device scaling: all methods remain above 20 Hz, with QAPF+CBF projected to have a median decision latency of $417.0$ $\mathsf{\mu }$$\mathrm{s}$ on Jetson Nano. The scaling-factor projection is an approximation; physical Jetson and DGX-class timing measurements are future work and should be performed before any platform-specific deployment claim.

The multi-axis generalization study (Section 5.7.6) shows that clean transfer is achieved by QAPF+CBF across $-66.7\%$ to $+66.7\%$ obstacle-density shifts, across $40\%$ grid-size shifts in either direction and across 1000 consecutive evaluation episodes without drift.

Several limitations should be acknowledged. The continuous-dynamics limitation is that the $50\times 50$ grid-world with simplified point-robot kinematics does not capture the continuous dynamics, actuator limits, wheel slip, turning-radius limits or higher-dimensional sensor streams of physical robots, and a sim-to-real transfer study on a Jetson-mounted physical platform remains future work. The discrete CBF limitation is that the empirical $0.3\%$ collision rate is achieved by the filter partly through the visit-memory mechanism rather than through classical forward-invariance theory, and the formal safety guarantees of [15,28] do not transfer directly to the discrete setting. The noise-model limitation is that i.i.d. Gaussian noise is injected by the robustness study, whereas real sensor noise is often correlated, biased and coupled to the perception pipeline, so richer noise models are a natural extension. The projected timing limitation is that the timing results for Jetson and DGX-class devices are projected rather than measured on-device, and DGX is a server-class reference rather than a mobile-robot platform. The unreachable-goal limitation is not the absence of a practical response: timeout and persistent-stagnation outcomes are explicitly detected and labeled by the current implementation. The remaining limitation is only formal certification, because the present empirical detector does not prove graph-theoretic infeasibility of the goal. The classical-planner comparison limitation is that A* and RRT* are discussed as scope baselines but are not experimentally benchmarked under a matched replanning or perception protocol. The 3D and kinematic-validation limitation is that the 3D extension is currently exercised only on the $z=0$ plane sanity check of Section 5.8; a non-trivial vertical-corridor benchmark and a physical non-holonomic platform validation are not executed in this study, and a real-aerial drone-based evaluation is identified as future work. The multi-robot scalability limitation is that the present centralized cooperative scheme is bounded by two distinct scaling limits, as discussed in Section 5.7.4: a joint-success-rate bound (under independent failures the joint SR is upper-bounded by ${\prod }_k{\mathrm{SR}}_k=0.{938}^N$, which falls below $50\%$ at $N\approx 11$) and a compute and communication bound from $\mathcal{O}(N^2)$ pair evaluations and state sharing. The decentralized, k-nearest and hierarchical variants enumerated in Section 5.7.4 are motivated by both bounds as the natural path beyond the small-fleet regime that the centralized scheme handles cleanly.

6. Conclusions and Future Work

This paper presented an extended QAPF framework that addresses key limitations of the original formulation [12] through three core contributions: (i) potential-based reward shaping whose unclipped fixed-weight form is policy-invariant, together with a clipped adaptive $\lambda (e)$ schedule used for stable finite-training performance; (ii) a discrete CBF-inspired action filter augmented with a per episode visit memory; and (iii) extensions to dynamic-obstacle, multi-robot and energy-aware navigation scenarios described by the explicit three-equation velocity-modulation law of Equations (13) to (15), which preserves full speed in low-gradient regions and reduces speed near obstacles.

Across 30 independent seeds and held-out maps, a SR of $93.8\pm 2.3\%$ with only $0.3\pm 0.4\%$ collisions is achieved by QAPF+CBF in the static-obstacle setting, while $90.4\%$ and $85.5\%$ success are retained in the dynamic and narrow-passage scenarios respectively. No observed degradation is shown by the robustness analysis under observation noise up to ${\sigma }_{\mathrm{obs}}=0.8$ cells in the finite robustness suite, while $85\%$ retention is reported under realistic combined noise. The inference-timing analysis shows real-time viability (>20 Hz) on the measured i7 reference and across all considered projected embedded platforms from Jetson Nano to AGX Orin, with QAPF+CBF reporting approximately $4.9$ kHz mean-call throughput on the i7 reference and a projected median latency of $417.0$ $\mathsf{\mu }$$\mathrm{s}$ on Jetson Nano. Physical Jetson and DGX-class measurements are treated as future work. The generalization study demonstrates clean transfer across $-66.7\%$ to $+66.7\%$ obstacle-density shifts and 1000-episode horizons, while the small-fleet regime is identified by the multi-robot scalability discussion as the natural operating range of the present centralized scheme, and three concrete decentralization strategies are enumerated for larger fleets.

Future work will target nine directions. (i) Physical sim-to-real validation on a Jetson-mounted ground robot, with end-to-end on-device latency measurement on Jetson AGX Orin and calibration of the projection multipliers in Table 10 against direct measurements; the optional DGX-class server timing is retained only as an upper-bound compute reference. The sim-to-real transfer will be studied through CARLA [34] and domain-randomization [35] to bridge the reality gap surveyed in [36]. (ii) Correlated and bias-shifted sensor-noise models that stress-test the inflated-margin CBF under more realistic perceptual errors. (iii) Adversarial robustness, in which worst-case input perturbations are evaluated in the spirit of neural-network safety verification [37]. (iv) An adaptive ${\delta }_{\mathrm{safe}}$ that tightens in open regions and inflates near clutter so that conservatism is traded against the path length. (v) Scaling the multi-robot framework to fleets of $N\ge 20$ via the decentralization strategies of Section 5.7.4. (vi) Formal unreachable-goal certification beyond the empirical timeout/stagnation detector used in this paper, through a two-tier architecture combining a periodic A* [32] reachability oracle, optional RRT*-based escape [38] and a $\parallel \nabla U\parallel$ plus Q-variance local-minimum diagnostic. (vii) A direct experimental A*/RRT* benchmark run under a matched replanning and perception protocol rather than under the ideal static-map assumptions of classical planners, together with a hybrid architecture in which a global path is generated by A* and local execution under dynamic disturbances is delegated to QAPF+CBF. (viii) Experimental validation of the 3D extension of Section 4.9 on real aerial platforms, in which a quadrotor drone is flown through a structured indoor environment so that the simulated $(x,y,z)$ behavior reported in Section 5.8 is corroborated under physical actuator and perception conditions. (ix) Experimental validation on concave obstacles along the lines of Appendix A.1.