Risk-Aware Reinforcement Learning with Dynamic Safety Filter for Collision Risk Mitigation in Mobile Robot Navigation

Abstract

Mobile robots face collision risk avoidance challenges in dynamic environments, necessitating that we address the safety and adaptability shortcomings of traditional navigation methods. Traditional methods rely on predefined rules, making it difficult to achieve flexible, safe, and real-time obstacle avoidance in complex, dynamic environments. To address this issue, a risk-aware, dynamic, adaptive regulation barrier policy optimization (RADAR-BPO) method is proposed, combining proximal policy optimization (PPO) with the control barrier function (CBF). RADAR-BPO generates exploratory actions using PPO and constructs a real-time safety filter using the CBF. This method uses quadratic programming to minimize risky actions, thereby ensuring safe obstacle avoidance while maintaining navigation efficiency. Testing of three phased learning environments in the ROS Gazebo simulation environment demonstrated that the proposed method achieves an obstacle avoidance success rate of nearly 90% in complex, dynamic, multi-obstacle environments and improves the overall mission success rate, validating its robustness and effectiveness in complex dynamic scenarios.

1. Introduction

Currently, the convergence of artificial intelligence (AI), the Internet of Things (IoT), and automation technologies is facilitating intelligent transformation across various industries [1,2,3]. Robotic technology represents a key driver of this evolution [4]. Advancements in intelligent algorithms and hardware enable robots to undertake increasingly complex tasks. As a critical subset of robots, mobile robots are being deployed extensively in industry [5,6], logistics [7], and agriculture [8].

Despite their diverse designs and functionalities, mobile robots commonly face challenges in safe obstacle avoidance and efficient navigation during operation [9]. In dynamic and complex environments, they must accurately identify obstacles, plan safe trajectories, and make real-time adjustments [10]. Ensuring task completion necessitates advanced sensor systems for real-time environmental perception [11]. Moreover, using intelligent algorithms to process information, execute dynamic path planning, and adapt to uncertainties is critical, and enhancing these capabilities is key to ensuring efficient and safe operation in complex environments [12,13,14].

Beyond traditional obstacle avoidance, navigating shared environments requires socially aware navigation capabilities [15]. Robots must not only avoid collisions but also adhere to social norms, exhibit predictable behavior, and respect personal space to achieve smooth and comfortable human–robot coexistence [16,17]. This involves understanding and predicting the motivations and intentions of human behavior and has become a research focus in the field of human–robot interaction (HRI) [18,19]. However, although the ultimate goal of social navigation is to achieve natural and comfortable human–machine coexistence, collision risk mitigation remains the most basic and critical safety prerequisite. Any social navigation strategy must be based on absolute collision avoidance in order to further optimize comfort and efficiency [20,21].

While techniques have previously been developed for obstacle avoidance and navigation by mobile robots, significant limitations persist [22,23]. Traditional control methods such as the artificial potential field (APF) method [24] and dynamic window approach (DWA) [25], often reliant on predefined rules and fixed path planning, perform adequately in simple tasks but lack adaptability and flexibility in complex, dynamic environments [26,27,28]. Their limited capacity for autonomous adjustment frequently results in decision errors or inefficiency when handling intricate scenarios [29]. Furthermore, the safety assurance mechanisms of conventional obstacle avoidance strategies remain inadequate, leading to elevated collision risks during operation. These constraints hinder their widespread deployment in complex and variable settings [30,31].

Advances in AI have facilitated the application of reinforcement learning (RL) in mobile robotics [32]. As a machine learning approach, RL demonstrates significant potential for use in autonomous robot decision-making [33]. It addresses key challenges including task execution policy optimization, environmental adaptability enhancement, and autonomous decision-making for complex tasks [34,35]. By learning optimal policies through environmental interaction, RL enables effective decision-making in dynamic and uncertain environments [36].

To enhance the time efficiency for mobile robot navigation in crowded environments, Zhou et al. [37] proposed a social graph-based double dueling deep Q-Network (DQN). This approach employs a social attention mechanism to extract effective graph representations, optimizes the state–action value approximator, and leverages simulated experiences generated by a learned environmental model for further refinement, demonstrating significantly improved success rates in crowded navigation tasks. Li et al. [38] introduced a fused deep deterministic policy gradient (DDPG) method, integrating a multi-branch deep learning network with a time-critical reward function, which effectively enhanced the convergence velocity and navigation performance in complex environments.

On the other hand, the control barrier function (CBF) has garnered attention in regard to obstacle avoidance control for mobile robots [39,40,41]. Originating from control theory, the CBF provides robust control guarantees for safety-critical systems operating under constraints [42]. For mobile robot obstacle avoidance, the CBF enforces strict safety conditions during motion by constructing safety constraints [43,44].

In order to further enhance the obstacle avoidance performance of the CBF, researchers have proposed various improvements and hybrid approaches. Singletary et al. [45] conducted a comparative analysis of the performance of the CBF and artificial potential fields (APFs) in robotic obstacle avoidance, demonstrating that the CBF generates smoother trajectories, effectively mitigates oscillations, and offers enhanced safety guarantees. Jian et al. [46] introduced a dynamic control barrier function, integrating it with model predictive control (MPC) to ensure collision-free trajectories for robots operating in uncertain and dynamic environments.

This paper proposes a novel safe reinforcement learning framework integrating the control barrier function (CBF) and proximal policy optimization (PPO) to reduce collisions in navigation for a mecanum wheel robot. Implemented within the ROS Gazebo simulation environment, the framework leverages CBF-based safety constraints formulated as a quadratic programming problem. This dynamically adjusts potentially unsafe actions generated by the PPO policy, guaranteeing collision mitigation while preserving motion efficiency. Comparative experiments against baseline algorithms (PPO, DQN, and DDPG) are conducted to rigorously evaluate the method’s effectiveness across key safety and navigation performance metrics.

The remainder of this paper is organized as follows: Section 2 introduces the robot model and basic methods; Section 3 presents the main research and methods; Section 4 describes the simulation results and and provides a discussion; and the final section, Section 5, concludes this paper.

2. Preliminary Knowledge

This section establishes the theoretical groundwork for the mobile robot platform, detailing its kinematic model and core implementation approaches. To support the development of subsequent control strategies and navigation algorithms, the unique omnidirectional motion capabilities inherent to the mecanum wheel robot are mathematically characterized. This section then presents fundamental obstacle avoidance techniques utilizing relevant control methodologies.

2.1. Kinematic Model of Robot

The mecanum wheel robot can achieve omnidirectional movement due to its unique wheel hub structure. The passive rollers installed at a 45° angle around each wheel allow the robot to translate or rotate in any direction without changing the orientation of the robot. Its kinematic model defines the relationship between the wheel velocities and the robot’s holistic motion. This representation is illustrated in Figure 1.

2.1.1. Forward and Inverse Kinematics Models

The parameters related to the robot chassis are defined in Figure 1. In the model $W_a$ represents half of the body length (from the front wheel to the center of the rear wheel), and $W_b$ represents half of the body width (the distance between the left and right wheel centers). The robot’s linear velocity is $v_{linear}$, and $v_x$ and $v_y$ are the velocities in the x and y directions, respectively.

In addition $v_n(n=1,2,3,4)$ is the linear velocity of each wheel, and $v_{n\omega }$ is the angular velocity of the chassis corresponding to each wheel, and $V_{nR}$ represents the velocity at which the wheel moves forward. In summary, the forward kinematics model of the robot can be described as

$$\left[\begin{array}{c}{v}_x\\ v_y\\ \omega \end{array}\right]=J{\left[\begin{array}{cccc}{V}_{1R}& V_{2R}& V_{3R}& V_{4R}\end{array}\right]}^T,$$

(1)

$$J={\displaystyle \frac{1}{4}}\left[\begin{array}{cccc}1& 1& 1& 1\\ -1& 1& 1& -1\\ -{\displaystyle \frac{1}{d}}& -{\displaystyle \frac{1}{d}}& {\displaystyle \frac{1}{d}}& {\displaystyle \frac{1}{d}}\end{array}\right],$$

(2)

where $d=W_a+W_b$ represents the characteristic length of the robot chassis. The matrix J establishes the mapping relationship between the wheel velocities and the chassis motion in the body-fixed frame. The corresponding inverse kinematics of the mecanum wheel robot chassis are shown as

$$\left\{\begin{array}{c}{V}_{1R}=v_x-v_y-(W_a+W_b)\omega \hfill \\ V_{2R}=v_x+v_y+(W_a+W_b)\omega \hfill \\ V_{3R}=v_x+v_y-(W_a+W_b)\omega \hfill \\ V_{4R}=v_x-v_y+(W_a+W_b)\omega \hfill \end{array}\right..$$

(3)

Therefore, the velocity of the robot chassis and the velocity of the four wheels can be converted into each other using the forward and inverse kinematics, which facilitates control of the robot’s trajectory. Based on this, a schematic diagram of the robot’s wheel odometry model is shown in Figure 2.

2.1.2. Ideal Model of Robot’s Odometry

Under ideal conditions, the relationship between the robot’s current position, $P_k$, and the previous position, $P_{k-1}$, is expressed by the offset ${\Delta }_k$. The next posture, $P_{k+1}$, is the current posture, $P_k$, plus the offset ${\Delta }_{k+1}$, and the corresponding expression equation is

$$P_{k+1}=P_k+{\Delta }_{k+1}{\Delta }_{t+1},$$

(4)

where $P_k={\left[\begin{array}{ccc}{x}_k& y_k& {\theta }_k\end{array}\right]}^T$, and ${\Delta }_{k+1}={\left[\begin{array}{ccc}{v}_x^{k+1}& v_y^{k+1}& {\omega }_{k+1}\end{array}\right]}^T$. Assuming that the time, ${\Delta }_{t+1}$, between two points is very small, the next position can be expressed as the linear and angular velocity offsets of the current position, and the expanded component form is as follows:

$$\left\{\begin{array}{cc}{x}_{k+1}\hfill & =x_k+v_x^{k+1}\hfill \\ y_{k+1}\hfill & =y_k+v_y^{k+1}\hfill \\ {\theta }_{k+1}\hfill & ={\theta }_k+{\omega }_{k+1}\hfill \end{array}\right..$$

(5)

Therefore, the derived forward and inverse kinematic models, combined with the discrete-time odometry update equations, provide a complete mathematical framework to predict the robot’s chassis motion based on the wheel velocities, which forms the basis for controlling the robot’s trajectory.

2.2. Control Barrier Function

In the process of robot movement and navigation, obstacle avoidance and velocity limiting are indispensable to prevent uncontrollable behavior. This section mainly introduces the control barrier function (CBF), which is often used in the control field as a controller and can also be used to constrain a robot’s actions.

2.2.1. Definition of Safety Set

For a dynamic system, the safety set $\mathcal{D}$ is defined as the set of safe states in the system. A diagram of the safety set is shown in Figure 3, and $\mathcal{D}$ is defined as

$$\begin{array}{cc}\hfill \mathcal{D}& =\{x\mid x\in {\mathbb{R}}^n\wedge h\left(x\right)\ge 0\},\hfill \\ \hfill \partial \mathcal{D}& =\{x\mid x\in {\mathbb{R}}^n\wedge h\left(x\right)=0\},\hfill \\ \hfill \mathrm{Int}\left(\mathcal{D}\right)& =\{x\mid x\in {\mathbb{R}}^n\wedge h\left(x\right)>0\},\hfill \end{array}$$

(6)

where the safe set $\mathcal{D}$ contains the status $x\subset {\mathbb{R}}^n$. When the system state x is safe, it means that it is inside the safety combination $\mathcal{D}$ or just on the boundary, which is $h\left(x\right)\ge 0$. On the contrary, $h\left(x\right)<0$ means that state x is outside the safe set.

2.2.2. Definition of Control Barrier Funciton

Consider a control affine system and the corresponding dynamic system in the nonlinear case, which can be expressed as

$$\dot{x}=F(x,u),$$

(7)

$$\dot{x}=f\left(x\right)+g\left(x\right)u,$$

(8)

where $u\subset {\mathbb{R}}^n$, and F is Lipschitz continuous. The behavior of the system is f when there is no control input, and g represents the control of the system and affects the changes in the system through a control input, u.

For $h\left(x\right)$, $\forall x\in \mathcal{D}$, and $\nabla h\left(x\right)\ne 0$, there exists $\alpha$, such that any state, x, in the set $\mathcal{D}$ satisfies the condition

$$\dot{h}(x,u)=L_{f}h\left(x\right)+L_{g}h\left(x\right)u\ge -\alpha h\left(x\right),$$

(9)

where $h\left(x\right)$ is considered to be a CBF, and $\alpha$ is an extended K function.

2.3. Basic Theory of Reinforcement Learning

Reinforcement learning (RL) is a machine learning approach characterized by an agent learning to optimize its actions through trial-and-error interactions with a dynamic environment, aimed at discovering strategies that maximize the long-term returns. Its core feature is learning through a trial-and-error mechanism, relying on reward signals to guide behavior optimization.

RL usually models a problem as a Markov decision process (MDP). A MDP can be represented by a five-tuple, $S,A,P,R,\gamma$.

S: The state space represents the set of all the possible states of the agent in the environment. The state $x\in S$ represents a specific situation in the environment at a certain moment and contains all the information needed by the agent.

A: The Action Space and the actions that the agent can take during an interaction with the environment are usually represented by $A\left(s\right)$, where $a\in A\left(s\right)$ represents the set of possible actions in state s.

P: The State Transition Function represents the probability of transitioning to state $s_{t+1}$ if an action, a, is executed in state s: that is, $P\left(s_{t+1}\right|s_t,a)$.

R: The reward function can be written as $R(s,a)$ and refers to the immediate reward feedback obtained by the agent when it performs action a in state s.

$\gamma$: The discount factor is used to calculate future cumulative rewards, $\gamma \in [0,1]$. A larger discount factor is suitable for tasks that focus on achieving long-term goals, while a smaller discount factor is more suitable for tasks that emphasize immediate feedback.

A robot can be regarded as an intelligent agent in reinforcement learning. The process of training it to interact with the environment involves RL. The corresponding environmental interaction is shown in Figure 4.

Figure 4 depicts the iterative learning process through which an agent learns to interact with its environment within an Actor–Critic framework. The Actor generates an action mechanism and communicates it to the robot agent. The robot explores the environment and obtains the corresponding rewards and states, which are then passed to the Critic network for value calculation. The Critic updates the Actor with the calculated parameters and outputs a better action strategy.

3. Main Research Content and Methods

This section mainly describes the motion control method based on proximal policy optimization (PPO) and the design of the real-time constraint mechanism using the CBF. On this basis, a novel fusion framework is constructed to implement a strategy for safe reinforcement learning, so as to ensure that the robot maintains safety performance while exerting its exploration advantages and further improve the stability of reinforcement learning.

3.1. Proximal Policy Optimization

The PPO objective function and Actor–Critic architecture are defined as key mechanisms for generating efficient exploratory navigation strategies for omnidirectional mobile robotic platforms. Their core function is to learn adaptive obstacle avoidance strategies for use in complex or dynamic scenarios and output robot velocity commands that balance navigation task completion and safety.

3.1.1. PPO Principle Description

As an Actor–Critic derivative, PPO enforces policy update constraints via a trust-region-inspired mechanism to ensure monotonic improvement. By introducing an objective function clipping mechanism, it achieves efficient learning while ensuring training stability. Its core components and formulas are as follows:

$$\mathrm{Actor}:{\pi }_{\theta }\left(a_t\right|s_t),$$

(10)

$$\mathrm{Critic}:V_{\varphi }\left(s_t\right),$$

(11)

where $\theta$ and $\varphi$ are the parameters of the policy network and the value network, respectively, the current state is $s_t$, and $a_t$ is the generated action.

First, we need to calculate the generalized advantage estimation (GAE), calculate the TD error based on $V_{\varphi }\left(s_t\right)$, and calculate the advantage $\widehat{A}t$ based on the TD error ${\delta }_t$. The specific equation is expressed as

$${\delta }_t=r_t+\gamma V_{\varphi }\left(s_{t+1}\right)-V_{\varphi }\left(s_t\right),$$

(12)

$$\widehat{A}t={\delta }_t+\gamma \lambda {\delta }_{t+1}+\dots +{\left(\gamma \lambda \right)}^{T-t+1}{\delta }_{T-1},$$

(13)

with $\gamma \in [0,1]$ being the discount factor and $\lambda \in [0,1]$ the GAE parameter.

The importance sampling ratio in the policy network update $r_t\left(\theta \right)$ is

$$r_t\left(\theta \right)={\displaystyle \frac{{\pi }_{\theta }\left(a_t\right|s_t)}{{\pi }_{{\theta }_{\mathrm{old}}}\left(a_t\right|s_t)}}.$$

(14)

Next, we need to calculate the clipping objective function, which is also one of the core components of PPO. It can be expressed as

$$L_{clip}^{PPO}\left(\theta \right)={\mathbb{E}}_t\left[\min \left(r_t\left(\theta \right){\widehat{A}}_t,\mathrm{clip}\left(r_t\left(\theta \right),1-ϵ,1+ϵ\right){\widehat{A}}_t\right)\right],$$

(15)

where $ϵ$ is the clipping threshold, and it limits the importance sampling ratio $r_t\left(\theta \right)$ to within the interval $[1-ϵ,1+ϵ]$.

For an action with a positive advantage $({\widehat{A}}_t>0)$, the objective function is clipped to prevent the policy from increasing the probability of the action too aggressively $(r_t\left(\theta \right)>(1+ϵ))$. Conversely, for an action with a negative advantage $({\widehat{A}}_t<0)$, the objective function is clipped to prevent the probability from decreasing too much $(r_t\left(\theta \right)<(1-ϵ))$.

In the Critic network update, the value loss function that needs to be calculated is as follows:

$$L_{vf}^{PPO}\left(\varphi \right)=\mathbb{E}t\left[{\left(V\varphi \left(s_t\right)-R_t\right)}^2\right],$$

(16)

where $R_t={\sum }_{k=0}^T{\gamma }^k{r}_{t+k}$ is the cumulative return. At the same time, PPO introduces an entropy regularization term to enhance the exploration of strategies:

$$L_{ent}^{PPO}\left(\theta \right)=\mathbb{E}t\left[H\left(\theta \left(a\right|s_t)\right)\right],$$

(17)

where $H\left(\theta \left(a\right|s_t)\right)$ is the entropy of the strategy.

Then the total loss function can be calculated as

$$L_{total}^{PPO}(\theta ,\varphi )={\mathbb{E}}_t\left[L_{clip}^{PPO}\left(\theta \right)-c_1{L}_{vf}^{PPO}\left(\varphi \right)+c_2{L}_{ent}^{PPO}\left(\theta \right)\right],$$

(18)

where $c_1$ and $c_2$ are weight parameters.

3.1.2. PPO Algorithm Flow

The PPO algorithm is an efficient policy gradient method. By introducing an objective function clipping mechanism, it can achieve efficient learning while ensuring training stability. The core idea of PPO is to limit the step size of policy updates to avoid performance crashes during training. The algorithm flow is shown in Algorithm 1.

Algorithm 1 Proximal policy optimization strategy.

Input: Initial policy ${\theta }_0$, value ${\varphi }_0$, $ϵ$, $\gamma$, $\lambda$, $c_1$, $c_2$, K, M Output: Optimized ${\theta }^{*}$, ${\varphi }^{*}$ 1: Initialize $\theta \leftarrow {\theta }_0$, $\varphi \leftarrow {\varphi }_0$ 2: for$k=0$  to  K  do 3:     Collect ${\mathcal{D}}_k$ via ${\pi }_{\theta }$ 4:     for each t in ${\mathcal{D}}_k$ do 5:         $R_t\leftarrow {\sum }_{l=t}^T{\gamma }^{l-t}{r}_l$ 6:         ${\widehat{A}}_t\leftarrow {\sum }_{l=0}^{T-t}{\left(\gamma \lambda \right)}^l{\delta }_{t+l}$ 7:     end for 8:     ${\theta }_{old}\leftarrow \theta$ 9:     for episode $=1$ to M do 10:         Sample minibatch $\mathcal{B}$ from ${\mathcal{D}}_k$ 11:         $L_{total}^{PPO}(\theta ,\varphi )\leftarrow 0$ 12:         for each $(s_t,a_t,R_t,{\widehat{A}}_t)$ in $\mathcal{B}$ do 13:            $r_t\left(\theta \right)\leftarrow {\pi }_{\theta }\left(a_t\right|s_t)/{\pi }_{{\theta }_{old}}\left(a_t\right|s_t)$ 14:            $L_{clip}^{PPO}\left(\theta \right)\leftarrow \min (r_t\left(\theta \right){\widehat{A}}_t,\mathrm{clip}(r_t\left(\theta \right),1-ϵ,1+ϵ){\widehat{A}}_t)$ 15:            $L_{vf}^{PPO}\left(\varphi \right)\leftarrow {(V_{\varphi }\left(s_t\right)-R_t)}^2$ 16:            $L_{ent}^{PPO}\left(\theta \right)\leftarrow -\mathbb{E}t\left[H\left(\theta \left(a\right|s_t)\right)\right]$ 17:            $L_{total}^{PPO}(\theta ,\varphi )\leftarrow L_{total}^{PPO}(\theta ,\varphi )-L_{clip}^{PPO}\left(\theta \right)+c_1{L}_{vf}^{PPO}\left(\varphi \right)-c_2{L}_{ent}^{PPO}\left(\theta \right)$ 18:         end for 19:         ${\theta }^{*}\leftarrow \theta -{\eta }_{\theta }{\nabla }_{\theta }{L}_{total}^{PPO}$ 20:         ${\varphi }^{*}\leftarrow \varphi -{\eta }_{\varphi }{\nabla }_{\varphi }{L}_{total}^{PPO}$ 21:     end for 22: end for 23: return ${\theta }^{*}\leftarrow \theta$, ${\varphi }^{*}\leftarrow \varphi$

3.2. Application and Implementation of CBF

The CBF is specifically used for its safety conditions, which are based on the kinematic model of the omnidirectional mobile robot used in this study. The focus is on demonstrating how this function acts as a real-time safety filter, applying online corrections to the original motion output by the PPO algorithm to achieve collision avoidance during navigation. Its quadratic programming formulation explicitly serves a single goal: to minimize the safety corrections to the PPO-determined motion while ensuring that the robot maintains its specific safety distance and stays within its physical motion limits.

3.2.1. Safe Sets and Safe Functions

The set of states, $\mathcal{D}$, in which the robot can safely operate and the barrier function $h\left(p\right)$ are defined as

$$D=\{p\subset {\mathbb{R}}^n:h\left(p\right)\ge 0\},$$

(19)

$$h\left(p\right)=\left|\right|p-p_{obs}{\left|\right|}^2-r_{safe}^2,$$

(20)

where $p=[x,y]$ represents the coordinates of the robot in the global coordinate system, $p_{obs}$ represents the coordinates of the nearest obstacle, and $r_{safe}$ is the safety distance threshold.

3.2.2. Implementation of CBF Safety Constraints

The time derivative of the CBF can be calculated as

$$\begin{array}{cc}\hfill {\displaystyle \frac{dh}{dt}}& ={\displaystyle \frac{\nabla h·dp}{dt}}\hfill \\ & =[{\displaystyle \frac{\partial h}{\partial x}},{\displaystyle \frac{\partial h}{\partial y}}]·{[\dot{x},\dot{y}]}^T,\hfill \end{array}$$

(21)

where ${\displaystyle \frac{\partial h}{\partial x}}=2(x-x_{obs})$, ${\displaystyle \frac{\partial h}{\partial y}}=2(y-y_{obs})$. Then substitute the robot motion model into the equation

$${\displaystyle \frac{dh}{dt}}=2v[(x-x_{obs})cos\theta +(y-y_{obs})sin\theta ].$$

(22)

Since the CBF needs to satisfy this constraint, substitute Equation (22) into Equation (9):

$$2v[(x-x_{obs})cos\theta +(y-y_{obs})sin\theta ]\ge -\alpha \left({(x-x_{obs})}^2+{(y-y_{obs})}^2-r_{safe}^2\right).$$

(23)

Set a relative position vector, $\Delta p$, and the unit vector of the robot’s forward direction, i, as

$$\Delta p={\left[\begin{array}{cc}x-x_{obs},& y-y_{obs}\end{array}\right]}^T,$$

(24)

$$i={\left[\begin{array}{cc}\cos \theta ,& \sin \theta \end{array}\right]}^T.$$

(25)

The corresponding constraints can be transformed into

$$2v·(\Delta p·i)\ge -\alpha (\parallel \Delta p{\parallel }^2-r_{\mathrm{safe}}^2).$$

(26)

In summary, the overall constraint solving framework of the CBF can be written as

$$\begin{array}{cc}\hfill {(v,\omega )}^{*}=\arg \underset{v,\omega \in {\mathbb{R}}^n}{\min }& \left[{(v-a_t\left(v\right))}^2+{(\omega -a_t\left(\omega \right))}^2\right]\hfill \\ \hfill \mathrm{s}.\mathrm{t}.& 2v·(\Delta p·n)\ge -\alpha (\parallel \Delta p{\parallel }^2-r_{safe}^2),\hfill \\ \hfill & 0\le v\le v_{\max },\hfill \\ \hfill & \left|\omega \right|\le {\omega }_{\max },\hfill \end{array}$$

(27)

where $v_{max}$ is the max value of v, and ${\omega }_{max}$ is the max angular velocity of $\omega$.

Figure 5 shows a simple diagram of the CBF obstacle avoidance process, as well as an example of the impact of different $\alpha$ values on the trajectory. At each step, the CBF combines the current velocity and other state variables to determine the next output that meets the constraint requirements and combines this with the result from the obstacle function calculation to continuously move until it reaches the target position.

A CBF framework for safe robot motion is proposed. First, a safe set is used by a barrier function, which represents the set of collision-free trajectories. Then, the time derivative of this barrier function is derived from the robot’s kinematic model and combined with the K function to obtain linear inequalities for the control variables. Finally, a quadratic program is formulated to minimize the modification of the nominal control command while satisfying the CBF safety constraints and the actuator’s restrictions on the linear and angular velocities. The corresponding algorithm for the CBF is shown in Algorithm 2.

Algorithm 2 CBF safety filter.

Input: Robot kinematic model, safety distance threshold $r_{safe}$, maximum velocity $v_{\max }$, maximum angular velocity ${\omega }_{\max }$, CBF parameters $\alpha$ Output: Optimized control inputs $v^{*}$, ${\omega }^{*}$ 1: Initialize robot state $p=[x,y]$, robot orientation $\theta$, velocity v, and angular velocity $\omega$ 2: Calculate the relative position vector $\Delta p={[x-x_{obs},y-y_{obs}]}^T$ to the nearest obstacle 3: Calculate the unit vector of the robot forward direction $i={[\cos \theta ,\sin \theta ]}^T$ 4: Calculate the barrier function $h\left(p\right)={\parallel \Delta p\parallel }^2-r_{safe}^2$ 5: Calculate the time derivative of the barrier function using (22) 6: Set the initial control inputs $v_0$, ${\omega }_0$ 7: for  $k=0$  to  K  do 8:        Calculate the constraint $2v·(\Delta p·i)\ge -\alpha (\parallel \Delta p{\parallel }^2-r_{safe}^2)$ 9:       where $i={[\cos \theta ,\sin \theta ]}^T$ 10:     and $\Delta p={[x-x_{obs},y-y_{obs}]}^T$ 11:   Solve the optimization problem $$\begin{array}{cc}\hfill {(v,\omega )}^{*}=\arg \underset{v,\omega \in {\mathbb{R}}^n}{\min }& \left[{(v-a_t\left(v\right))}^2+{(\omega -a_t\left(\omega \right))}^2\right]\hfill \\ \hfill \mathrm{s}.\mathrm{t}.& 2v·(\Delta p·i)\ge -\alpha (\parallel \Delta p{\parallel }^2-r_{safe}^2),\hfill \\ \hfill & 0\le v\le v_{\max },\hfill \\ \hfill & \left|\omega \right|\le {\omega }_{\max }.\hfill \end{array}$$ 12:     Update control inputs $v^{*}\leftarrow v$, ${\omega }^{*}\leftarrow \omega$ 13:     Update robot state using the kinematic model 14: end for 15: return $v^{*}\leftarrow v$, ${\omega }^{*}\leftarrow \omega$

3.3. Risk-Aware, Dynamic, Adaptive Regulation Barrier Policy Optimization

This section designs a risk-aware, dynamic, adaptive regulation barrier policy optimization (RADAR-BPO) method which combines the efficiency of stable exploration with PPO. The Actor outputs probabilistic actions, and the Critic updates their value and optimizes the Actor based on the information obtained from feedback on the agent’s motion in the environment. In addition, extra security is provided by the CBF, which was added to further optimize the actions output by the PPO algorithm for the actor, filter actions with risky values, and output safe actions. This method deeply integrates the exploration capabilities of PPO and the security of the CBF, so that the robot agent can maximize the value of exploration in different environments. A corresponding block diagram of the complete system is shown in Figure 6, and the algorithm is shown in Algorithm 3.

Figure 6. Complete RADAR-BPO framework for training to interact with environment.

Figure 6. Complete RADAR-BPO framework for training to interact with environment.

Algorithm 3 RADAR-BPO.

Input: Initial policy ${\theta }_0$, value ${\varphi }_0$, safety params Output: Optimized ${\theta }^{*}$, ${\varphi }^{*}$ 1: Initialize $\theta \leftarrow {\theta }_0$, $\varphi \leftarrow {\varphi }_0$ 2: for iteration $k=0$ to K do 3:     Data Collection: 4:     Collect trajectory ${\mathcal{D}}_k$ via policy ${\pi }_{\theta }$ 5:     Policy Optimization: 6:     Compute advantages ${\widehat{A}}_t$ and returns $R_t$ for ${\mathcal{D}}_k$ 7:     ${\theta }_{\mathrm{old}}\leftarrow \theta$ 8:     Update $\theta ,\varphi$ using PPO loss with ${\mathcal{D}}_k$ 9:     for each state $s_t$ in ${\mathcal{D}}_k$ do 10:         Get PPO action $a_t^{\mathrm{PPO}}=(v^{\mathrm{PPO}},{\omega }^{\mathrm{PPO}})$ 11:         Compute safety constraint based on robot state 12:         Safety Filtering: 13:         Solve CBF-QP: $$\begin{array}{cc}\hfill (v^{*},{\omega }^{*})& =\arg \underset{v,\omega }{\min }{\parallel a-a_t^{\mathrm{PPO}}\parallel }^2\hfill \\ \hfill \mathrm{s}.\mathrm{t}.& \mathrm{CBF}\mathrm{constraints}\hfill \end{array}$$ 14:         Execute safe action $a_t^{\mathrm{safe}}=(v^{*},{\omega }^{*})$ 15:     end for 16: end for 17: return ${\theta }^{*}\leftarrow \theta$, ${\varphi }^{*}\leftarrow \varphi$

In addition, this method fully considers information on the robot’s spatial dimensions during the obstacle avoidance process, including its position, orientation, distance from obstacles, etc., and transforms this spatial information into safety constraints using the CBF and embeds it into a reinforcement learning-based decision-making process, thereby achieving safe and efficient navigation in complex dynamic environments.

To achieve an optimal balance between navigation efficiency and safety guarantees, a multi-objective reward function with adaptive components is designed. The mathematical formulation is defined as follows:

$$r_t=r_{\mathrm{h}}+r_{\mathrm{d}}+r_{\mathrm{ob}}+r_{\mathrm{v}},$$

(28)

where $r_{\mathrm{h}}$, $r_{\mathrm{d}}$, $r_{\mathrm{ob}}$, and $r_{\mathrm{v}}$ are defined below.

The heading reward $r_{\mathrm{h}}$ encourages alignment with the target direction:

$$r_{\mathrm{h}}=1+\cos \left({\theta }_{diff}\right),$$

(29)

where ${\theta }_{diff}$ denotes the angular deviation between the robot’s current orientation and the target direction. The distance reward $r_{\mathrm{d}}$ motivates progression toward the target position:

$$r_{\mathrm{d}}=5\exp \left(-2{\displaystyle \frac{d_t}{d_0}}\right),$$

(30)

where $d_t$ is the current Euclidean distance to the target and $d_0$ is the initial distance.

The obstacle penalty $r_{\mathrm{ob}}$ ensures collision risk awareness:

$$r_{\mathrm{ob}}=\left\{\begin{array}{cc}{r}_{risk}\hfill & \mathrm{if}{d}_{\min }<r_{safe}\hfill \\ r_{normal}\hfill & \mathrm{otherwise},\hfill \end{array}\right.$$

(31)

where $d_{\min }$ is the distance to the nearest obstacle.

The velocity reward $r_{\mathrm{v}}$ optimizes the motion efficiency:

$$r_{\mathrm{v}}=\exp \left(-{\displaystyle \frac{{(v_t-v_{des})}^2}{2{\sigma }^2}}\right),$$

(32)

with an adaptive optimal velocity of $v_{des}=0.25\min (1.0,d_t)$ and $\sigma =0.1$. This equation encourages a higher velocity when distant from the target while promoting precision during the final phases.

The reward function integrates the four components in an additive manner. Although this is an equally weighted sum, each term is designed to have comparable magnitudes to prevent any one objective from overly dominating the learning process. For example, $r_{\mathrm{h}}\in [0,2]$, and $r_{\mathrm{d}}$ is scaled by a factor and exponentially decays with distance. $r_{\mathrm{ob}}$ provides a significant but bounded penalty. This design prioritizes simplicity, interpretability, and minimal parameter tuning in the early stages of the algorithm’s implementation, resulting in a reliable benchmark. The $r_{\mathrm{v}}$ component is not a constant reward but a term used to fine-tune the efficiency of the movement.

The RADAR-BPO framework establishes a safety-critical reinforcement learning paradigm through integrated policy optimization and real-time safety assurance. During each training iteration, the agent executes interactions with the environment using its current policy network ${\pi }_{\theta }$, gathering trajectory data that includes environmental states, exploratory actions, and reward signals. This collected experience forms the foundation for subsequent policy refinement.

Following data collection, the algorithm performs proximal policy optimization using the GAE. This involves calculating the temporal difference errors to estimate the action advantages, then updating both the policy and value networks using a specialized objective function. The optimization balances exploration incentives with value approximation while maintaining training stability through gradient clipping.

During action execution, each policy-generated velocity command undergoes safety verification using a CBF filter. This module solves a quadratic optimization problem that minimizes deviations from the original actions while enforcing collision avoidance constraints. The safety verification incorporates the robot’s positional relationship with obstacles and its current heading orientation to dynamically adjust the motion commands.

The resulting safe actions preserve the learning direction of the policy network while ensuring formal safety guarantees through real-time constraint enforcement. This synergistic integration creates an evolution mechanism where policy improvement and safety assurance mutually reinforce each other throughout the learning process. The system maintains continuous compliance with safety boundaries while progressively refining its navigation strategy through environmental interactions.

4. Case Study

To rigorously evaluate the performance and validate the efficacy of the proposed risk-aware, dynamic, adaptive regulation barrier policy optimization (RADAR-BPO) framework for collision risk mitigation in mobile robot navigation, this section presents comprehensive simulation experiments conducted within the ROS Gazebo environment.

4.1. Test Environment Setup

The simulation environments visualized in Figure 7 were constructed within the ROS Gazebo platform to test a robot’s obstacle avoidance navigation. Specifically, Figure 7a presents a dynamic pedestrian environment, Figure 7b demonstrates a multi-obstacle environment, and Figure 7c showcases a complex obstacle environment incorporating both static and dynamic elements.

In Figure 7a, there are two pedestrians walking back and forth at a certain speed. The robot needs to avoid them and reach the target point while the pedestrians are moving. In Figure 7b, there are multiple static cylindrical or cubic obstacles. The robot needs to avoid the static obstacles and successfully navigate to the target point. Figure 7c contains multiple pedestrians and multiple different static obstacles. The robot also needs to complete a navigation task in this complex and changing environment.

4.2. Design of Experimental Test

The motion trajectories of the pedestrians in Figure 7a,c are shown in

Table 1. Trajectory coordinates of pedestrians in environment.

Time (s)	Pedestrian 1		Pedestrian 2
	x (m)	y (m)	x (m)	y (m)
0.0	0.5	2.5	3.0	0.0
3.0	0.5	1.5	3.0	2.0
3.5	0.5	1.5	3.0	2.0
5.5	0.5	2.5	-	-
6.0	0.5	2.5	-	-
6.5	-	-	3.0	0.0
7.0	-	-	3.0	0.0

, and there are two stationary pedestrians in Figure 7c.

In the Gazebo coordinate system (the positive direction on the y-axis is left and the negative direction is right), pedestrian 1 moves 1 m to the right from their initial position (0.5, 2.5) to (0.5, 1.5), turns on the spot, moves 1 m to the left, returns to their starting point (0.5, 2.5), and stays there. At the same time, pedestrian 2 moves 2 m to the left from their initial position (3.0, 0.0) to (3.0, 2.0), turns on the spot, moves 2 m to the right, returns to their starting point (3.0, 0.0), and stays there. Both pedestrians complete a round trip in the y-axis direction. There are two other stationary pedestrians: one is fixed at (1.0, 0.0), and the other is fixed at (2.0, 3.0). The robot starts from the starting point (0.0, 0.0) and needs to traverse the dynamic environment to reach the target point (3.0, 3.0). Its path will be blocked by moving pedestrians and it will need to avoid stationary pedestrian obstacles and other obstacles.

Table 2. Training parameter settings for RADAR-BPO.

Category	Parameter	Symbol	Value/Range
Policy Optimization	Discount factor	$\gamma$	0.99
	GAE parameter	$\lambda$	0.95
	Clipping threshold	$ϵ$	0.15
	Value coefficient	$c_1$	0.5
	Entropy coefficient	$c_2$	0.01
	Policy iterations	K	[150, 200, 300]
	Linear velocity	$v_{\max }$	[0, 0.5]
	Angular velocity	${\omega }_{\max }$	[−1.0, 1.0]
Safety Constraints	Safety distance	$r_{\mathrm{safe}}$	0.5
	Max linear velocity	$v_{\max }$	0.5
	Max angular velocity	${\omega }_{\max }$	1.0
	Barrier parameter	$\alpha$	1.0
Robot Kinematics	Half of body’s length	$W_a$	0.195
	Half of body’s width	$W_b$	0.172
	Characteristic length	$d=W_a+W_b$	0.367
Training Setup	2D lidar angle range	–	[0°, 360°]
	Scanning range	–	[0.08, 10]

lists the key parameter settings used to train the RADAR-BPO navigation algorithm. These parameters cover aspects such as policy optimization, the safety constraints, the robot’s kinematic model, and the training environment configuration. Some parameter values (such as discount factors and GAE parameters) were within typical ranges or needed to be adjusted according to the specific environment.

The algorithm was trained on each environment for a fixed number of episodes: 150 for Env 1 (dynamic pedestrians), 200 for Env 2 (multiple static obstacles), and 300 for Env 3 (multiple complex obstacles). This design with an increasing environment complexity and training time was intended to mimic curriculum learning, allowing the agent to consolidate foundational skills before tackling more difficult tasks. Each training episode terminated when the agent successfully reached the goal, collided with an obstacle, or reached the maximum step limit.

4.3. Test Results and Analysis

To comprehensively evaluate the performance of the proposed RADAR-BPO framework and quantitatively assess its effectiveness in mitigating collision risks while maintaining navigation efficiency, extensive simulations were conducted across the three distinct environments introduced in Section 4.1 (Figure 7). The evaluation employed a three-stage progressive training paradigm: Stage 1 (Figure 7a) utilized the dynamic pedestrian environment (Env 1), Stage 2 (Figure 7b) progressed to the multi-static obstacle environment (Env 2), and Stage 3 (Figure 7c) took place in the complex hybrid environment containing both static obstacles and dynamic pedestrians (Env 3). This staged approach rigorously tested the algorithm to determine its fundamental obstacle avoidance ability.

The core performance metrics included the learning stability, safety performance, and navigation efficiency. Crucially, the cumulative reward curves from throughout the training process were analyzed and compared against those of the baseline PPO algorithm. This direct comparison highlighted the impact of integrating the real-time CBF safety filter within the RADAR-BPO framework. The corresponding segmented training reward curve is shown in Figure 8.

As shown in Figure 8, during the progressive training process implemented across three stages (Env 1, Env 2, and Env 3), the RADAR-BPO algorithm significantly outperformed all the baseline algorithms (PPO, DQN, and DDPG) in terms of the average reward in all the environments. This overall performance advantage was demonstrated by the following: in the relatively simple Env 1 stage, while all the algorithms initially experienced low rewards, RADAR-BPO quickly escaped this low-reward zone and stabilized at a higher level. Entering the more complex Env 2 stage, RADAR-BPO exhibited a burst of performance growth, with its reward values rapidly exceeding 1000 and stabilizing at approximately 1100. In contrast, PPO slowly climbed to around 400, while the DQN and DDPG lagged significantly behind. In the most complex Env 3 stage, the challenging environment caused performance degradation for all the algorithms. However, RADAR-BPO maintained an absolute advantage, stabilizing at around 760 with a smooth, volatility-resistant curve. PPO fluctuated violently below 400 and showed weak growth. The DQN and DDPG’s performance remained sluggish, failing to improve significantly, and the gap between their performance and RADAR-BPO’s was the largest.

Comparing the two curves at the beginning of the Env 3 phase reveals that PPO’s reward values exhibited a larger upward gradient and oscillation amplitude, while the rise in RADAR-BPO’s values was more stable. This phenomenon reveals the different learning modes of the two algorithms: The PPO strategy almost failed after an environment switch, requiring a painstaking learning process to recover from an extremely high collision rate, resulting in an unstable learning process. In contrast, RADAR-BPO benefited from the real-time safety guarantees provided by the CBF. Its strategy retained its core obstacle avoidance capabilities after an environment switch, and its learning process involved stable fine-tuning based on a higher performance baseline to adapt to new dynamic obstacles. Therefore, RADAR-BPO sacrificed a seemingly larger learning amplitude in exchange for a higher, more stable, and safer final performance.

This result clearly shows that the RADAR-BPO framework integrated with real-time CBF security filtering can not only achieve higher task returns in complex dynamic environments (based on the higher success rate and average reward in

Table 3. Algorithm performance comparison.

Environment	Algorithm	Success Count	Collision Count	Average Reward	Collision Rate (%)
Env 1	PPO	27	123	357.23	82.00
	RADAR-BPO	47	103	421.58	68.67
	DDPG	17	133	13.67	88.67
	DQN	0	150	271.50	100.00
Env 2	PPO	134	66	377.01	33.00
	RADAR-BPO	164	36	1070.09	18.00
	DDPG	25	175	58.02	87.50
	DQN	89	111	273.47	55.50
Env 3	PPO	208	92	399.38	30.67
	RADAR-BPO	268	32	762.66	10.67
	DDPG	73	227	19.45	75.67
	DQN	96	204	269.22	68.00

) but also significantly improve the convergence speed of the learning process, the final performance ceiling, and the training stability, effectively solving the core problems of low exploration efficiency, large policy fluctuations, and limited performance of traditional PPO in safety-critical scenarios.

According to the results listed in Table 3, RADAR-BPO outperformed the PPO algorithm in most metrics. In Env 1, RADAR-BPO’s collision rate was 68.67%, lower than PPO’s 82.00%; in Env 3, RADAR-BPO’s collision rate further decreased to 10.67%, compared to PPO’s 30.67%. Furthermore, RADAR-BPO achieved 47, 164, and 268 successes in the three different environments, respectively, all exceeding PPO’s 27, 134, and 208. In terms of the average reward, RADAR-BPO also generally outperformed PPO, with these algorithms achieving 1070.09 and 377.01, respectively, in Env 2. These results demonstrate that RADAR-BPO maintains high task completion efficiency and a high reward yield while reducing the collision rate.

In contrast, the DDPG and DQN algorithms generally underperformed in comparison to RADAR-BPO and PPO. The DDPG’s collision rate was above 75% across all the environments, and its average reward was significantly lower than that of the other algorithms. The DQN had zero successes and a 100% collision rate in Env 1. In Env 2 and Env 3, its collision rates were 55.50% and 68.00%, respectively. While its average reward was higher than that of the DDPG, it was still lower than that of RADAR-BPO and PPO.

To provide an intuitive visualization of the algorithm’s real-time decision-making and safety assurance capabilities in the most challenging scenario, the trajectory evolution of the robot navigating across Env 3 over a critical 9 s interval (from t = 1 s to t = 9 s) is presented and discussed. This sequence illustrates how RADAR-BPO dynamically adjusted the robot’s path to safely avoid both static obstacles and moving pedestrians while progressing towards the target.

In Figure 9, the robot starts from the starting position (0.0, 0.0), initially accelerates to bypass the stationary pedestrian in front, and turns left toward the target direction; when it encounters Pedestrian 1 (0.5, 2.5 → 1.5) moving horizontally to the right, it suddenly turns at a sharp angle to achieve emergency avoidance and simultaneously and smoothly avoids walls and nearby obstacles. Under the combined threat of Pedestrian 2 (3.0, 2.0 → 0.0) moving horizontally and dense obstacles, the robot flexibly adjusts the path to squeeze through and finally arrives at the end point (3.0, 3.0) accurately. The entire path maintains the straight-line efficiency of PPO in open areas and forms a smooth and conservative contour using CBF constraints when approaching obstacles. The collision-free nature of the entire process verifies RADAR-BPO’s seamless integration of exploration and safety in dynamic and dense scenes.

5. Conclusions and Future Work

This paper proposes RADAR-BPO (risk-aware, dynamic, adaptive regulation barrier policy optimization), a novel safe reinforcement learning framework integrating PPO with CBF-based safety filters to mitigate collision risks in mobile robot navigation. The framework leverages PPO for exploratory policy generation while employing the CBF as a real-time safety filter, formulated as a quadratic programming problem, to minimally modify risky actions and ensure collision avoidance. Implemented on a mecanum wheel robot within the ROS Gazebo simulation environment, the method demonstrated significant improvements in safety performance across diverse dynamic and complex scenarios. Comparative experiments against baseline PPO, DQN, and DDPG algorithms confirmed that RADAR-BPO achieves higher success rates, lower collision rates, and superior average rewards while maintaining navigation efficiency, highlighting its effectiveness in balancing exploration with formal safety guarantees.

In future work, the framework could be extended to address more complex real-world challenges. Potential directions include validating the approach on physical robotic platforms to assess its real-time performance and robustness under sensor noise and hardware limitations. Additionally, exploring the use of adaptive or learned CBF parameters to handle heterogeneous obstacle shapes and uncertain dynamics, integrating multi-robot collision avoidance scenarios, and extending the method to unstructured outdoor environments would further enhance this approach’s applicability.