Vehicle Lateral Control Based on Augmented Lagrangian DDPG Algorithm

Abstract

This paper studies the safe trajectory tracking control of intelligent vehicles, which is still an open and challenging problem. A deep reinforcement learning algorithm based on augmented Lagrangian safety constraints is proposed to the lateral control of vehicle trajectory tracking. First, the tracking control of intelligent vehicles is described as a reinforcement learning process based on the Constrained Markov Decision Process (CMDP). The actor-critic neural network based reinforcement learning framework is established and the environment of reinforcement learning is designed to include the vehicle model, tracking model, road model and reward function. Secondly, the augmented Lagrangian Deep Deterministic Policy Gradient (DDPG) method is proposed for updating, in which a replay separation buffer method is used to solve the problem of sample correlation, and a neural network with the same structure is copied to solve the update divergence problem. Finally, a vehicle lateral control approach is obtained, whose effectiveness and advantages over existing results are verified through simulation results.

1. Introduction

The rapid advancement of autonomous driving technology has imposed increasingly stringent requirements on vehicle trajectory tracking control. As a core component of intelligent driving systems for new energy vehicles, trajectory tracking control must balance tracking accuracy, safety, stability, and multi-objective optimization capabilities in complex dynamic environments. Traditional control methods such as model predictive control (MPC), and sliding mode control (SMC) demonstrate advantages including strong interpretability and straightforward implementation. Recent MPC methods achieve sub-cm tracking via adaptive parameter optimization (fuzzy particle swarm optimization) and hierarchical control, effectively managing over-actuated systems (four wheel steer and four wheel drive/agricultural machinery) through real-time quadratic programming in high-dimensional spaces [1,2,3]. Advanced SMC integrates neural disturbance estimation with adaptive feedback, demonstrating lateral error and disturbance rejection under coupled dynamic constraints across variable road conditions [4,5,6]. However, their model-dependent nature fundamentally limits effectiveness in addressing nonlinear system dynamics and multi-constrained optimization challenges. Deep Reinforcement Learning (DRL) has recently emerged as a transformative alternative through end-to-end learning mechanisms that bypass explicit model requirements.

Recent advances in deep reinforcement learning [7,8,9,10] have enhanced intelligent vehicle control through model-integrated end-to-end frameworks [11], expert-guided pretraining [12], and PID hybrid architectures [13]. However, two fundamental limitations persist: (1) Policy instability during early training phases due to random parameter initialization incurs high exploration costs, and (2) Frequent safety violations (boundary breaches/collisions) during exploration reveal critical constraint enforcement deficiencies. To resolve these challenges, we propose a constrained optimization framework modeling lateral control as a CMDP, integrating three key innovations: an augmented Lagrangian safety optimizer, replay-separated experience buffers, and dual-Critic value estimation to prevent overoptimistic Q-values while ensuring stable policy updates.

Deep Reinforcement Learning (DRL) directly maps environmental states to control actions through deep neural networks, with its core advantages manifested in two aspects: (1) leveraging the feature extraction capabilities of deep learning to handle high-dimensional nonlinear systems, and (2) utilizing the trial-and-error mechanism of reinforcement learning to achieve multi-objective dynamic optimization. Its significance stems from the breakthrough solutions offered by Deep Q-Network (DQN) [14] and its derivative algorithms—such as Soft Actor-Critic (SAC) [15], Twin Delayed Deep Deterministic Policy Gradient (TD3) [16], Deep Deterministic Policy Gradient (DDPG) [17], Double Deep Q-Network (Double DQN) [18], and Asynchronous Advantage Actor-Critic (A3C) [19]—in addressing complex decision-making problems. Deep reinforcement learning in vehicle control evolves through two paradigms: model-free and model-based approaches. Model-free methods like DDPG [11] employ end-to-end frameworks to directly map states to control commands, enhancing tracking accuracy in complex scenarios though challenged by inefficient initial exploration. While expert-guided pretraining [12] narrows exploration spaces, it imposes strict requirements on demonstration data quality. Model-based approaches integrate six-DOF dynamics with DDPG for handling tire-road interactions, yet remain vulnerable to parametric uncertainties (e.g., friction coefficient variations) [20] in real-world deployments.

Safe Deep Reinforcement Learning (Safe DRL) establishes a CMDP to transform physical safety boundaries into hard constraints for policy optimization [21]. By incorporating constraint optimization mechanisms, Safe DRL ensures safety during algorithm training. Existing Safe DRL methods primarily employ approaches such as safety shielding layers [22], finite-state controllers [23], semi-supervised reward learning [24], and control barrier functions [25]. However, these solutions often suffer from high implementation complexity and overly conservative constraint conditions. This study innovatively introduces the augmented Lagrangian method to construct a dynamic constraint optimization model. By adaptively adjusting the penalty factor in real-time, it balances the threshold between safety and performance, thereby reducing safety violation rates. This approach provides a novel methodological framework for safe control in complex dynamic environments.

This paper investigates four-wheel independent steering control for vehicles, applying deep reinforcement learning algorithms to address trajectory tracking challenges. The study aims to enhance vehicle stability and accuracy during high-speed maneuvers and complex trajectory scenarios. A DDPG-based approach is employed to optimize the neural network architecture, with specific designs for the Critic and Actor network structures and parameters. A novel replay-separated buffer method is proposed to improve the efficiency of experience replay, accelerating the training process while ensuring robustness. By leveraging the augmented Lagrangian method, the vehicle trajectory tracking problem is modeled as a CMDP, and a new reward function is designed to enhance training efficiency. Experimental results demonstrate that the proposed algorithm achieves significant improvements in key performance metrics, including training safety, tracking precision, and ride comfort.

2. Preliminaries

To address the controller design issues encountered in vehicle trajectory tracking, this study restructures the problem as a Markov decision process and applies deep reinforcement learning techniques to solve it. Training vehicle trajectory tracking controllers using deep reinforcement learning methods can lead to scenarios where the vehicle collides with lane boundaries, prematurely ending the reinforcement learning training process. To mitigate this challenge, we introduce constrained reinforcement learning within lane boundary limitations, formally establishing a Safe Reinforcement Learning (Safe RL) framework with collision-avoidance guarantees.

A Constrained Markov Decision Process (CMDP) ${\mathcal{M}}_{\mathcal{C}}$ is defined by the tuple $(\mathcal{S},\mathcal{A},\mathbb{P},R,C,{\rho }_0,\gamma )$, where $\mathcal{S}$ and $\mathcal{A}$ are the state and action spaces, respectively; $\mathbb{P}:\mathcal{S}\times \mathcal{A}\to P\left(s\right)$ represents the transition probability $\mathbb{P}\left(s^{\prime }\right|s,a)$ from state s to $s^{\prime }$ under action a; $R:\mathcal{S}\times \mathcal{A}\times \mathcal{S}\to \mathbb{R}$ is the reward function, yielding the immediate reward $r(s,a,s^{\prime })$ when transitioning from s to $s^{\prime }$ via action a; C is a set of safety constraints, where each $c_i:\mathcal{S}\times \mathcal{A}\to \mathbb{R}$ is a safety cost function with an associated threshold $b_i$ for $i=1,\dots ,m$; ${\rho }_0:\mathcal{S}\to [0, 1]$ is the initial state distribution; and $\gamma$ is the discount factor for future rewards.

A stationary policy ${\pi }_{\theta }$ is a probability distribution over actions given states, with ${\pi }_{\theta }\left(a\right|s)$ denoting the probability of taking action a in state s. The set of all stationary policies is ${\mathrm{\Pi }}_{\theta }=\{{\pi }_{\theta }:\theta \in {\mathbb{R}}^P\}$. The state-value function for ${\pi }_{\theta }$ is defined as $V_{{\pi }_{\theta }}\left(s\right)={\mathbb{E}}_{{\pi }_{\theta }}\left[{\sum }_{t=0}^{\infty }{\gamma }^t{r}_{t+1}\mid s_0=s\right]$, where ${\mathbb{E}}_{{\pi }_{\theta }}\left[·\right|·]$ denotes the expectation over actions selected by ${\pi }_{\theta }$. The objective of reinforcement learning is to maximize $J_r\left({\pi }_{\theta }\right)={\mathbb{E}}_{s\sim {\rho }_0\left(·\right)}\left[V_{{\pi }_{\theta }}\left(s\right)\right]$. In CMDPs, the expected discounted cost for policy ${\pi }_{\theta }$ is $J_{C_i}\left({\pi }_{\theta }\right)=\mathbb{E}\left[{\sum }_{t=0}^{\infty }{\gamma }^t{c}_i(s_t,a_t)\right]$. The feasible policy set is defined as ${\mathrm{\Pi }}_C={\bigcap }_{i=1}^m\{{\pi }_{\theta }\in {\mathrm{\Pi }}_{\theta }\mid J_{C_i}\left({\pi }_{\theta }\right)\le b_i\}$. The goal of CMDP is to find the optimal policy ${\pi }^{*}=arg{max}_{{\pi }_{\theta }\in {\mathrm{\Pi }}_C}{J}_r\left({\pi }_{\theta }\right)$.

Definition 1.

(Optimization Problem of MDP with Instantaneous Constraints) Consider a Markov Decision Process defined by state transition probabilities $s_{t+1}\sim p_t(·\mid s_t,a_t)$, initial state distribution $s_0\sim p_0$, a bounded reward function $r(s,a)$, and a bounded constraint function $g(s,a)$. The objective is to find the optimal policy ${\pi }^{*}$ that maximizes the expected discounted cumulative reward while satisfying the given constraints. The optimization problem can be formulated as follows:

$$\begin{array}{cc}\hfill \underset{\pi }{max}& J_r\left(\pi \right)=\mathbb{E}\left[\sum _{t=0}^{\infty }{\gamma }^{t}r\left(s_t,a_t\right)\mid \pi \right]\hfill \\ \hfill s.t.& s_{t+1}\sim p\left(·\mid s_t,a_t\right),\forall t\in \{0, 1,\dots \},\hfill \\ \hfill & a_t\sim \pi \left(s_t\right),\forall t\in \{0, 1,\dots \},\hfill \\ \hfill & s_0\sim p_0,\hfill \\ \hfill & \mathbb{E}\left[g\left(s_t,a_t\right)\mid \pi \right]\le 0,\forall t\in \{0, 1,\dots \}.\hfill \end{array}$$

(1)

where the state transition probabilities $s_{t+1}\sim p(·\mid s_t,a_t)$ describe how the system evolves over time, the initial state distribution $s_0\sim p_0$ specifies the starting point of the process, the bounded reward function $r(s,a)$ quantifies the immediate reward obtained from taking action a in state s, and the bounded constraint function $g(s,a)$ ensures that certain constraints are met at each time step. The goal is to maximize the expected discounted cumulative reward $J_r\left(\pi \right)$ while ensuring that the constraints are satisfied.

Although only one set of instantaneous constraints is considered in the bounded constraints of Problem 1, the results of this paper can be extended to the case of multiple sets of instantaneous constraints by associating each constraint with a Lagrange multiplier.

3. Problem Formulation

3.1. The Dynamic Model of the Four-Wheel Steering Vehicle

Vehicle dynamics modeling is the foundation of vehicle trajectory tracking control. The four-wheel independent steering vehicle is a highly coupled, nonlinear, and uncertain system. To simplify the vehicle modeling problem, we assume that the vehicle’s longitudinal dynamics are negligible, focusing solely on lateral and yaw dynamics while ignoring vertical dynamics. For four-wheel independent steering vehicles, the following dynamic equations hold:

$$\begin{array}{c}\hfill m(\dot{v_x}-v_{y}w)=\sum F_x\\ \hfill m(\dot{v_y}+v_{x}w)=\sum F_y\\ \hfill I_z\dot{w}=\sum M_z\end{array}$$

(2)

where m is the vehicle mass, $v_x$ and $v_y$ are the longitudinal and lateral velocities, w is the yaw rate, $I_z$ is the vehicle’s yaw inertia, $F_x$ and $F_y$ are the longitudinal and lateral forces, and $M_z$ is the yaw moment. To simplify the vehicle dynamics, we assume that the longitudinal velocity $v_x$ is constant, i.e., $\dot{v_x}=0$, and $\sum F_x=0$. The lateral force and yaw moment dynamics are as follows:

$$\begin{array}{cc}\hfill m(\dot{v_y}+v_{x}w)=& \sum F_y=F_{y,fl}+F_{y,fr}+F_{y,rl}+F_{y,rr}\hfill \\ \hfill I_z\dot{w}=& \sum M_z\hfill \\ \hfill =& a(F_{y,fl}+F_{y,fr})-b(F_{y,rl}+F_{y,rr})\hfill \\ \hfill & +{\displaystyle \frac{B_f}{2}}(F_{y,fr}-F_{y,fl})+{\displaystyle \frac{B_r}{2}}(F_{y,rr}-F_{y,rl}).\hfill \end{array}$$

(3)

Based on the small angle assumption, the vehicle’s tire model can be simplified as a linear model, i.e., $F_{y,i}=C_i{\alpha }_i$. The lateral slip angles of the vehicle’s tires are expressed as follows:

$$\begin{array}{cc}\hfill {\alpha }_{fl}& ={\delta }_{fl}-{\displaystyle \frac{v_y+aw-\frac{B_f}{2}w}{v_x}}\hfill \\ \hfill {\alpha }_{fr}& ={\delta }_{fr}-{\displaystyle \frac{v_y+aw+\frac{B_f}{2}w}{v_x}}\hfill \\ \hfill {\alpha }_{rl}& ={\delta }_{rl}-{\displaystyle \frac{v_y-bw-\frac{B_r}{2}w}{v_x}}\hfill \\ \hfill {\alpha }_{rr}& ={\delta }_{rr}-{\displaystyle \frac{v_y-bw+\frac{B_r}{2}w}{v_x}}\hfill \end{array}$$

where ${\delta }_{fl}$, ${\delta }_{fr}$, ${\delta }_{rl}$, and ${\delta }_{rr}$ are the steering angles of the vehicle’s front left, front right, rear left, and rear right wheels, respectively, $C_i$ is the tire’s lateral stiffness, a and b are the vehicle’s front and rear axle distances, and $B_f$ and $B_r$ are the vehicle’s front and rear track widths.

Based on the above assumptions, the vehicle dynamics model with 6 degrees of freedom (DOF) as shown in Figure 1 was established, including the lateral motion along the y-axis, the yaw motion around the z-axis, and the rotation of the four wheels.

The state space representation of the 6 DOF model of a four-wheel vehicle is as follows:

$$\begin{array}{c}\hfill \left[\begin{array}{c}\dot{v_y}\\ \dot{w}\end{array}\right]=A\left[\begin{array}{c}{v}_y\\ w\end{array}\right]+B\left[\begin{array}{c}{\delta }_{fl}\\ {\delta }_{fr}\\ {\delta }_{rl}\\ {\delta }_{rr}\end{array}\right]\end{array}$$

(4)

where A is the system matrix:

$$\begin{array}{c}\hfill A=\left[\begin{array}{cc}-{\displaystyle \frac{2(C_f+C_r)}{m{v}_x}}& -v_x-{\displaystyle \frac{2(a{C}_f-b{C}_r)}{m{v}_x}}\\ -{\displaystyle \frac{2(a{C}_f-b{C}_r)}{I_z{v}_x}}& -{\displaystyle \frac{2(a^2{C}_f+b^2{C}_r)+B_f{B}_r(C_f+C_r)/2}{I_z{v}_x}}\end{array}\right]\end{array}$$

and B is the input matrix:

$$\begin{array}{c}\hfill B=\left[\begin{array}{cccc}{\displaystyle \frac{C_f}{m}}& {\displaystyle \frac{C_f}{m}}& {\displaystyle \frac{C_r}{m}}& {\displaystyle \frac{C_r}{m}}\\ {\displaystyle \frac{a{C}_f-B_f{C}_f/2}{I_z}}& {\displaystyle \frac{a{C}_f+B_f{C}_f/2}{I_z}}& {\displaystyle \frac{-b{C}_r-B_r{C}_r/2}{I_z}}& {\displaystyle \frac{-b{C}_r+B_r{C}_r/2}{I_z}}\end{array}\right].\end{array}$$

3.2. The Vehicle Trajectory Error System

The vehicle dynamics model can convert the vehicle’s underlying motion characteristics into a quantitative problem of high-level control objectives, i.e., vehicle trajectory tracking accuracy problem. In the control architecture of a four-wheel steering vehicle, control commands and trajectory tracking performance are dynamically coupled through a multi-level state transmission chain.

Specifically, the four-wheel steering angles ${\delta }_i$ serve as control inputs, first acting on the vehicle’s physical system through the 6 DOF dynamic model: the lateral force of the tires varies with the steering angle, driving the lateral acceleration ${\dot{v}}_y$ and yaw acceleration $\dot{w}$, which in turn update the vehicle’s motion state $(v_y,w)$. Therefore, establishing a vehicle trajectory tracking model based on the vehicle dynamic model effectively addresses the trajectory tracking problem, as demonstrated in Figure 2.

The trajectory tracking error model under single preview-point configuration is derived through kinematic analysis and dynamic coupling. Let L denote the preview distance ahead of the vehicle’s center of gravity. The lateral position error $y_{\mathrm{L}}$, defined as the perpendicular deviation between the vehicle’s Center Of Gravity and the reference path, evolves according to three contributions: (1) longitudinal velocity $v_x$ projected laterally due to heading misalignment ${\epsilon }_{\mathrm{L}}$, (2) direct lateral velocity $-v_y$, and  (3) yaw-induced displacement $-\omega L$. This yields the lateral error dynamics:

$${\dot{y}}_{\mathrm{L}}=v_x{\epsilon }_{\mathrm{L}}-v_y-\omega L.$$

Concurrently, the heading error ${\epsilon }_{\mathrm{L}}$, representing the angular deviation from the path tangent, is governed by the discrepancy between the reference curvature $\rho$ and the vehicle’s yaw rate $\omega$:

$${\dot{\epsilon }}_{\mathrm{L}}=v_x\rho -\omega .$$

The dynamic equation of the vehicle trajectory tracking model with preview distance is as follows:

$$\left[\begin{array}{c}{\dot{y}}_{\mathrm{L}}\\ {\dot{\epsilon }}_{\mathrm{L}}\end{array}\right]=\underset{\mathbf{A}}{\underbrace{\left[\begin{array}{cc}0& v_x\\ 0& 0\end{array}\right]}}\left[\begin{array}{c}{y}_{\mathrm{L}}\\ {\epsilon }_{\mathrm{L}}\end{array}\right]+\underset{\mathbf{B}}{\underbrace{\left[\begin{array}{ccc}-1& -L& 0\\ 0& -1& v_y\end{array}\right]}}\left[\begin{array}{c}{v}_y\\ w\\ \rho \end{array}\right].$$

(5)

The vehicle trajectory tracking model introduces preview distance, which introduces the curvature information of the vehicle trajectory through the preview distance, solving the dynamic tracking problem of the vehicle trajectory.

3.3. Constrained Reinforcement Learning-Based Tracking Control Framework

To address the control problem in the vehicle trajectory tracking system, this paper employs deep reinforcement learning methods, modeling the vehicle trajectory tracking system as a constrained Markov decision process. The vehicle is treated as the environment in the RL problem, while the four-wheel steering controller serves as the agent. Leveraging the periodicity of steering control, each control cycle’s steering actions and resulting vehicle state changes are considered as an agent-environment interaction.

When approaching curved road sections, insufficient lane boundary information acquisition may cause lateral control response delays, leading to overshooting control targets and reduced stability. To ensure timely control updates before entering curvature-changing zones, we select lateral tracking errors $y_{\mathrm{L}}$, heading deviation ${\epsilon }_{\mathrm{L}}$, and trajectory curvature $\rho$ as state space components. The state space $\mathcal{S}$ is defined as: $\mathcal{S}=\{y_{\mathrm{L}},{\epsilon }_{\mathrm{L}},\rho \}$.

The study focuses on four-wheel independent steering vehicles, thus defining the action space as the steering angles of the four wheels: ${\varphi }_{lf}$, ${\varphi }_{lr}$, ${\varphi }_{rf}$, ${\varphi }_{rr}$. The action space $\mathcal{A}$ is defined as: $\mathcal{A}=\{{\varphi }_{lf},{\varphi }_{lr},{\varphi }_{rf},{\varphi }_{rr}\}$. During deep reinforcement learning training, excessively large action ranges in the control policy output degrade training performance, necessitating action range constraints through linear mapping of policy network outputs.

To reduce the occurrence of dangerous behaviors such as overstepping and collisions during training, preventing numerous ineffective training cycles, this paper introduces lane boundary constraints as state constraints. Ensuring that the vehicle trajectory tracking error strictly satisfies the safety boundary constraints, $|y_{\mathrm{L}}| <3\forall t$, essentially requires the control policy to avoid entering dangerous state regions at any time.

First, define the constraint cost function $c:\mathcal{S}\to \{0, 1\}$ to characterize the safety attributes in the state space:

$$\begin{array}{c}\hfill c\left(s_t\right)=\mathbb{I}\left(|y_{\mathrm{L}}| \ge 3\right)=\left\{\begin{array}{cc}1\hfill & |y_{\mathrm{L}}| \ge 3\hfill \\ 0\hfill & \mathrm{others}.\hfill \end{array}\right.\end{array}$$

Based on this, the constraints of the CMDP can be expressed as an upper bound on the expected cumulative constraint cost:

$$\begin{array}{c}\hfill {\mathbb{E}}_{\pi }\left[\sum _{t=0}^{\infty }{\gamma }_c^{t}c\left(s_t\right)\right]\le ϵ\end{array}$$

where ${\gamma }_c\in [0, 1]$ is the constraint discount factor. For strict hard constraint problems, set ${\gamma }_c=1$ and $ϵ=0$, where the constraint condition is equivalent to requiring $c\left(s_t\right)=0$ at all times. Based on Definition 1, the vehicle trajectory tracking problem under safety constraints can be modeled as a CMDP as follows:

$$\begin{array}{cc}\hfill \underset{\pi }{max}{J}_r\left(\pi \right)=& {\mathbb{E}}_{\pi }\left[\sum _{t=0}^T{\gamma }^{t}r(s_t,a_t)|\pi \right]\hfill \\ \hfill \mathrm{s}.\mathrm{t}.& {\mathbb{E}}_{\pi }\left[\sum _{t=0}^{T}c\left(s_t\right)\right]\le 0\hfill \end{array}$$

(6)

where $J_r\left(\pi \right)$ is the expected reward, $r(s_t,a_t)$ is the immediate reward, $c\left(s_t\right)$ is the constraint cost, $\gamma$ is the discount factor, and $\pi$ is the policy. The objective of this optimization problem is to find the optimal policy ${\pi }^{*}$ that maximizes the expected cumulative reward while satisfying the constraints.

4. Main Results

4.1. Augmented Lagrangian Method for Safe RL

The instantaneous constraints in safe reinforcement learning are typically solved using the Lagrange multiplier method, which converts the constrained optimization problem into an unconstrained optimization problem using the Lagrange function, adding the weighted constraints to the objective function to obtain the Lagrange function: $L(\pi ,\lambda )=J_r\left(\pi \right)-\lambda (J_c\left(\pi \right)-b)$. The Lagrange function $L(\pi ,\lambda )$ is used to convert the constrained optimization problem into an unconstrained optimization problem.

$$\underset{\lambda \ge 0}{min}\underset{\pi }{max}L(\pi ,\lambda )=\underset{\lambda \ge 0}{min}\underset{\pi }{max}[J_r\left(\pi \right)-\lambda (J_c\left(\pi \right)-\alpha )].$$

The existing methods exhibit dual theoretical limitations in alternately optimizing policy parameters $\pi$ and Lagrange multipliers $\lambda$. First, the multiplier update mechanism employs a fixed learning rate for gradient ascent, lacking adaptive capability to optimize surface curvature, which leads to unstable convergence and oscillatory divergence with improper learning rates. Second, zero initialization renders multipliers ineffective during early optimization, weakening the constraint violation penalty in the initial training phase, thereby reducing penalty strength during early exploration of safety boundaries. In traditional constrained optimization methods, there is another approach known as the Augmented Lagrangian Method (ALM). This method combines the concepts of the penalty function method and the Lagrange multiplier method to construct an augmented Lagrangian function that includes a quadratic penalty term. For the aforementioned constrained optimization problem, the augmented Lagrangian function can be defined as follows:

$$L(\pi ,\lambda ,\rho )=J_r\left(\pi \right)-\lambda (J_c\left(\pi \right)-\alpha )-{\displaystyle \frac{\rho }{2}}{(J_{\mathrm{c}}\left(\pi \right)-\alpha )}^2.$$

where $\rho >0$ is the penalty factor for the quadratic penalty term ${(J_{\mathrm{c}}\left(\pi \right)-\alpha )}^2$.

Definition 2.

(Augmented Lagrangian MDP Optimization Problem) Using the framework defined in Definition 1, combined with the augmented Lagrangian multiplier method, formalize the constrained optimization problem of the vehicle trajectory tracking problem as the following unconstrained optimization problem:

$$\begin{array}{cc}\hfill \underset{\pi }{max}\underset{\lambda \ge 0}{min}& \mathbb{E}\left[\left.\sum _{t=0}^{\infty }{\gamma }^t\left(r\left(s_t,a_t\right)-{\lambda }_{t}g\left(s_t\right)+{\displaystyle \frac{\rho }{2}}max{\left(0,g\left(s_t,a_t\right)\right)}^2\right)\right|\pi \right]\hfill \\ \hfill s.t.& s_{t+1}\sim p_s\left(·|s_t,a_t\right),\forall t\in \{0,1,\dots \},\hfill \\ \hfill & a_t\sim \pi \left(s_t\right),\forall t\in \{0,1,\dots \},\hfill \\ \hfill & s_0\sim p_0\hfill \end{array}$$

(7)

The constraint function requires that the vehicle’s desired position at each moment does not exceed the lane boundary. Compared with the traditional Lagrange multiplier method, the augmented Lagrangian multiplier method has two main advantages. First, during the iterative optimization process using the augmented Lagrangian function $L(\pi ,\lambda ,\rho )$, the step size for updating $\lambda$ is no longer fixed but is adaptively adjusted by the variable penalty factor $\rho$, which increases progressively during optimization. This mechanism effectively adjusts the learning rate dynamically, where the synergistic effect of the multiplier term and quadratic penalty term enhances algorithmic convergence rate and stability. Second, initializing the penalty factor $\rho$ to a positive value enforces strict “warm-start” penalization of safety constraint violations from the outset of training. This design prioritizes safety considerations during initial learning phases, thereby mitigating the risk of converging to unsafe local optimal solutions.

Transforming the constrained Markov decision process problem into an unconstrained optimization problem using the augmented Lagrangian function enables the solution of constrained optimization problems through the augmented Lagrangian multiplier method. This paper proposes a safe reinforcement learning algorithm based on the augmented Lagrangian function to address the trajectory tracking problem with lane boundary constraints, achieving a better balance between safety and optimal objectives.

4.2. Reward Function Design

In reinforcement learning algorithms, network initialization is random. Consequently, the agent can only interact with the environment based on guidance from the reward function. The reward function is typically a scalar value, where positive values represent rewards and negative values represent penalties. Reinforcement learning algorithms optimize towards maximizing the reward value. The scenario focused on in this paper is lane-keeping for vehicles. This requires that, during driving, the vehicle should stay as close as possible to the center of the lane, and the vehicle’s forward direction should be aligned with the road axis. Additionally, on this basis, the vehicle should aim to increase its speed as much as possible.

Aggressive steering during turns may cause stability loss and uncomfortable lateral acceleration fluctuations. To prevent local optimal solutions, we design an exponential reward function that accelerates reward growth near optimal solutions. The proposed reward function integrates tracking accuracy and comfort. This reward structure systematically integrates trajectory tracking accuracy with steering stability constraints through the following formulation for each timestep t:

$$r_t(s_t,a_t)=\left\{\begin{array}{cc}{r}_t(s_t,a_t)=\underset{\mathrm{Lateral} \mathrm{stability}}{\underbrace{{\delta }_1·e^{-|a_{\mathrm{lat}}|}}}+\underset{\mathrm{Steering} \mathrm{efficiency}}{\underbrace{{\delta }_2·e^{-{\phi }_{\mathrm{steer}}}}}+\underset{\mathrm{Oscillation} \mathrm{penalty}}{\underbrace{{\delta }_3·{\mathcal{P}}_t}},\hfill & \hfill \mathrm{normal} \mathrm{driving};\\ -C,\hfill & \hfill \mathrm{hit} \mathrm{boundary}\end{array}\right.$$

(8)

where:

Lateral Stability Term (${\delta }_1·e^{-|a_{\mathrm{lat}}|}$): Penalizes excessive lateral acceleration $a_{\mathrm{lat}}$ through exponential decay, where ${\delta }_1$ determines the penalty intensity. This term ensures vehicle stability during aggressive maneuvers.

Steering Efficiency Term (${\delta }_2·e^{-{\phi }_{\mathrm{steer}}}$): Encourages minimal steering effort by penalizing the cumulative steering angle ${\phi }_{\mathrm{steer}}={\sum }_{i=1}^4\left|{\varphi }_i\right|$, where ${\varphi }_i$ denotes the steering angle of each wheel. The exponential form prioritizes smooth steering operations.

Oscillation Penalty Term (${\delta }_3·{\mathcal{P}}_t$): Suppresses steering oscillations through a sliding-window mechanism:

$${\mathcal{P}}_t=-\sum _{i=1}^4{\eta }_i\left[{\lambda }_1^{\left(i\right)}{\Delta }_{\mathrm{mean}}^{\left(i\right)}\left(t\right)+{\lambda }_2^{\left(i\right)}{\sigma }^{\left(i\right)}\left(t\right)+{\lambda }_3^{\left(i\right)}{R}^{\left(i\right)}\left(t\right)\right]$$

where:

${\Delta }_{\mathrm{mean}}^{\left(i\right)}$: Mean absolute difference of steering channel i over window ${\mathcal{W}}_t$

${\sigma }^{\left(i\right)}$: Variance of steering actions in channel i

$R^{\left(i\right)}$: Range (max-min) of steering actions in channel i

${\eta }_i$: Channel-specific weights ($\sum {\eta }_i=1$)

${\lambda }_{1:3}^{\left(i\right)}$: Penalty coefficients for temporal variation characteristics

The sliding window ${\mathcal{W}}_t=\{a_{t-N+1},...,a_t\}$ maintains the latest N steering actions ($a_t\in {\mathbb{R}}^4$) using FIFO updating. This multi-timescale penalty simultaneously suppresses high-frequency jitter (via ${\Delta }_{\mathrm{mean}}$), amplitude fluctuations (via $\sigma$), and abrupt changes (via R).

4.3. Vehicle Tracking Control Algorithm Based on Safety Constraints

This study addresses the vehicle trajectory tracking problem, where the vehicle controller regulates the steering angles of all four wheels based on available lane trajectory information (including lane boundaries and trajectory curvature) and vehicle state parameters (such as velocity and acceleration). The objective is to ensure precise and safe lane-following performance. The framework of the proposed trajectory tracking decision control algorithm is illustrated in Figure 3.

The reinforcement learning problem can be solved using the Actor-Critic framework, which combines the advantages of policy gradient methods and temporal difference methods, including an Actor network and a Critic network. The structure of the Actor network consists of four fully connected layers: an input layer, two hidden layers, and an output layer.

The input layer of the Critic network is divided into two parts: one part is the state information of the vehicle simulation environment, and the other part is the action values output by the Actor network. The simulation environment information is processed by one hidden layer, then combined with the action values and input into the second hidden layer. Subsequently, the state information is processed by the output neurons to obtain the state-action reward value.

To enhance the learning efficiency of reinforcement learning, we introduce a prioritized experience replay mechanism and a dual experience pool mechanism within the framework of constrained reinforcement learning algorithms. The prioritized experience replay mechanism allows the agent to preferentially select experience samples with higher importance or value for training. By focusing on more representative situations or critical learning moments, the agent can learn and optimize its strategy more effectively. By prioritizing the replay of key experiences, the agent can learn faster from past experiences, thereby accelerating the learning process. This is particularly useful in situations where rapid adaptation to environmental changes is required.

The actor network generates a set of action values, i.e., the steering angles of the four wheels, and adds random noise before inputting them into the MATLAB (R2024b) simulation software. MATLAB then inputs the next state $s^{\prime }$ into the reward function module based on the actions, storing it in the buffer along with the current state s, reward value r, and action value a. To utilize high-quality data for neural network training and improve training stability, the buffer is divided into a standard error buffer $R_{stand-e}$ and a high-error buffer $R_{high-e}$. The experience buffer is separated by calculating the TD error $TD{E}_t=y_t-\overline{Q}\theta (S_t,a_t)$, with the sampling probability of $R_{high-e}$ increasing as model training progresses.

A certain number of samples are proportionally sampled from the experience replay buffer, with each sample containing $s,a,r,s^{\prime }$. Then, s and $s^{\prime }$ are passed to the Actor network, and $s,a,r,s^{\prime }$ are passed to the Critic network for iterative updates. For the Actor network, it accepts s and $s^{\prime }$, then outputs a along with random noise to MATLAB. Simultaneously, after outputting the next action $a^{\prime }$ to the Critic network, it accepts the state-action value gradient $Q(s,a)$ to $a^{\prime }$ to update the network. For the Critic network, it accepts $s^{\prime }$ and $a^{\prime }$ to calculate $Q_R(s^{\prime },a^{\prime })$. Next, it combines $Q_R(s^{\prime },a^{\prime })$ with r to compute the labels used for network iterative updates. Additionally, s and a are input into the Critic network, and the mean squared error between the output and the labels is used as the loss function.

$$\begin{array}{c}\hfill L_R={\displaystyle \frac{1}{N}}\sum _i(y_i-Q_R(s_i,a_i|{\theta }_R^Q){)}^2\end{array}$$

(9)

The reward Critic network is updated through iteration (9),

$$\begin{array}{c}\hfill L_C={\displaystyle \frac{1}{N}}\sum _i(z_i-Q_C(s_i,a_i|{\theta }_C^Q){)}^2\end{array}$$

(10)

while the loss Critic network is updated through iteration (10).

$$\begin{array}{c}\hfill {\Delta }_{{\theta }^{\mu }}={\displaystyle \frac{1}{N}}\sum _i{\Delta }_{{\theta }^{\mu }}(Q_R(s,\mu \left(s\right|{\theta }^{\mu })|{\theta }_R^Q)-\lambda Q_C(s,\mu \left(s\right|{\theta }^{\mu })|{\theta }_C^Q)){|}_{s=s_i}\end{array}$$

(11)

The sampled policy gradient (11) is used to update the action network, and the Lagrange multiplier is updated using the loss evaluation network. The pseudocode for the safe reinforcement learning algorithm based on the augmented Lagrangian function method is Algorithm 1.

Algorithm 1 Based on the Augmented Lagrangian Method for Safe DDPG Algorithms (DDPGALM).

1: Randomly initialize the network Reward Critic Q-network $Q_R(s,a|{\theta }_R^Q)$, Cost Critic Q-network $Q_C(s,a|{\theta }_C^Q)$ and Actor network $\mu \left(s\right|{\theta }^{\mu })$  2: Initialize the target network ${\theta }_R^{Q^{\prime }}\leftarrow {\theta }_R^Q$,${\theta }_C^{Q^{\prime }}\leftarrow {\theta }_C^Q$,${\theta }^{{\mu }^{\prime }}\leftarrow {\theta }^{\mu }$  3: Initialize parameters Replay buffer $\mathcal{R}$ and Augmented Lagrangian Method $\lambda$  4: for episode k = 0,1,…, do  5:     Initialize a random process $\mathcal{N}$ for action exploration  6:     Get the initial state $s_0\sim p_0$  7:     for t = 1,…,T do  8:         Select an action $a_t=\mu \left(s_t\right|{\theta }^{\mu })$  9:         Execute the action $a_t$, Observe $r_t,c_t,s_{t+1}$ 10:         Store the tuple $(s_t,a_t,r_t,c_t,s_{t+1})$ in the replay buffer $\mathcal{R}$ 11:         Randomly sample N tuples ${\left\{(s_t,a_t,r_t,c_t,s_{t+1})\right\}}_{i=1}^N$ from the replay buffer $\mathcal{R}$ 12:         Define $y_i=r_i+\gamma Q_R^{\prime }(s_{i+1},{\mu }^{\prime }\left(s_{i+1}\right|{\theta }^{{\mu }^{\prime }})|{\theta }^{Q^{\prime }}$,$z_i=c_i+\gamma Q_C^{\prime }(s_{i+1},{\mu }^{\prime }\left(s_{i+1}\right|{\theta }^{{\mu }^{\prime }})|{\theta }_C^{Q^{\prime }})$ 13:         Update the reward evaluation network by minimizing $L_R={\displaystyle \frac{1}{N}}{\sum }_i{(y_i-Q_R(s_i,a_i|{\theta }_R^Q))}^2$, and update the loss evaluation network by minimizing $L_C={\displaystyle \frac{1}{N}}{\sum }_i{(z_i-Q_C(s_i,a_i|{\theta }_C^Q))}^2$. 14:         Update the action network using the sampled policy gradient: $\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}{\Delta }_{{\theta }^{\mu }}={\displaystyle \frac{1}{N}}{\displaystyle \sum _i}{\Delta }_{{\theta }^{\mu }}(Q_R(s,\mu \left(s\right|{\theta }^{\mu })|{\theta }_R^Q)-\lambda Q_C(s,\mu \left(s\right|{\theta }^{\mu })|{\theta }_C^Q)){|}_{s=s_i}$ 15:         Update the Augmented Lagrangian multiplier: $\lambda$: $\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}{\Delta }_{\lambda }={\displaystyle \frac{1}{N}}{\displaystyle \sum _i}[Q_C(s_i,\mu \left(s_i\right|{\theta }^{\mu })-d]$ 16:         Update the target network: $\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em} {\theta }_R^{Q^{\prime }}\leftarrow \tau {\theta }_R^Q+(1-\tau ){\theta }_R^{Q^{\prime }}$ $\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em} {\theta }_C^{Q^{\prime }}\leftarrow \tau {\theta }_C^Q+(1-\tau ){\theta }_C^{Q^{\prime }}$ $\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em} {\theta }^{{\mu }^{\prime }}\leftarrow \tau {\theta }^{\mu }+(1-\tau ){\theta }^{{\mu }^{\prime }}$ 17:     end for 18: end for

The algorithm based on the Augmented Lagrangian Method for Safe DDPG Algorithms addresses the safety constraints in trajectory tracking for four-wheel steering vehicles. Specifically, it aims to mitigate the issue of premature termination during model training caused by collisions with lane edges. By incorporating the Augmented Lagrangian Method, the algorithm ensures that safety constraints are rigorously enforced, thereby enhancing the robustness and reliability of the training process. This approach not only improves the vehicle’s ability to track trajectories accurately but also significantly reduces the risk of unsafe behaviors, such as lane boundary violations, during both training and deployment.

5. Simulation Studies

5.1. Simulation Platform and Model Verification

The vehicle trajectory tracking control algorithm is validated using a co-simulation platform, which consists of three main components: CarSim 2020 (Mechanical Simulation Corporation, Ann Arbor, MI, USA), Simulink R2021b (The MathWorks, Inc., Natick, MA, USA), and Python 3.7. The system architecture incorporates CarSim as a high-fidelity vehicle dynamics solver, embedded into the Simulink environment through S-Function interface, providing multi-degree-of-freedom vehicle models and real-time state feedback, the simulation process in the CarSim software is shown in the Figure 4a. Notably, a dedicated 6-DOF vehicle dynamics model is implemented in Simulink, serving as the primary plant model during deep reinforcement learning training phases. This parallel modeling approach enables comparative validation between the Simulink-based dynamics model and CarSim’s S-Function module, which share similar functional capabilities in providing real-time vehicle state feedback while differing in model fidelity and computational efficiency.

The Simulink platform serves as the control framework, implementing signal routing, data preprocessing, and actuator command conversion. The co-simulation strategy employs the Simulink-built 6-DOF model for algorithm training iterations due to its computational efficiency, while reserving the higher-fidelity CarSim S-Function module for final validation stages. Figure 4b illustrates integrated simulation framework combining Python-based intelligent decision-making with MATLAB/Simulink-CarSim vehicle dynamics modeling. The Python agent, implemented through PyTorch’s deep reinforcement learning toolkit, interfaces with MATLAB via the MatlabEngine package to exchange control actions (wheel_angle) and synchronization signals (pause_flag, pause_time) at 100 Hz frequency. The Simulink environment processes these inputs through a CarSim vehicle model, returning observation states, reward signals, constraint costs, and episode termination flags via shared memory buffers. Tight temporal synchronization (10 ms timestep) is maintained through event-driven pausing mechanisms, enabling deterministic co-simulation while preserving the fidelity of both control computations and vehicle dynamics simulations. This hierarchical modeling architecture ensures both training efficiency through Simulink’s real-time simulation capabilities and validation credibility via CarSim’s professional vehicle dynamics solver.

The Simulink simulation diagram is given as shown in Figure 4c, which includes five main parts. Act module is the strategy actions obtained through reinforcement learning algorithm training; Carsim Model is a high-fidelity simulation software joint simulation module; Dynamics Model is a vehicle dynamics simulation module built in Simulink; Reward module calculates the reward, cost, done, obs and other values of the current environment according to the state obtained in Carsim Model or Dynamics Model; Assertion module is used to control the state communication between Python and Simulink during reinforcement learning training.

Our experimental framework establishes the following environmental assumptions: The simulation assumes dry asphalt road conditions with friction coefficient $\mu =0.85$, excluding terrain irregularities and elevation changes. Vehicle localization integrates LiDAR, vision sensors, and odometry measurements rather than GPS positioning, reflecting modern autonomous system architectures. Environmental disturbances including wind effects and electromagnetic interference are excluded from consideration. The control system implements 10ms actuation latency while maintaining 1ms time resolution in the Simulink-CarSim co-simulation environment to ensure dynamic fidelity. Training trajectories adopt the ISO 3888-1:1999 [26] emergency lane-change standard augmented with curvature perturbations ($\Delta \kappa \sim \mathcal{U}(-0.2, 0.2){\mathrm{m}}^{-1}$) and path length variations ($\Delta s\sim \mathcal{N}(0,0.5\mathrm{m})$), systematically expanding the operational envelope while preserving safety constraints. These explicitly defined parameters delineate the method’s operational domain and enable reproducible validation under controlled autonomous driving conditions.

The double lane change (DLC) maneuver, recognized as a representative scenario for evaluating vehicle stability during high-speed overtaking and obstacle avoidance, serves as a standard test for assessing vehicle handling and stability performance. The vehicle trajectory tracking control system is validated through co-simulation using the integrated Simulink and CarSim platform. To verify the simulation accuracy of both the 6-DOF vehicle dynamics model and CarSim S-Function module, a comparative validation was conducted under dual-lane change maneuvers at 60 km/h and 90 km/h operating conditions. The CarSim platform’s built-in driver controller was employed to generate baseline vehicle responses, with both models receiving identical steering inputs and road signals.

As shown in Figure 5, the comparative analysis of critical state parameters (including lateral velocity, yaw rate) demonstrates that the Simulink-implemented 6-DOF dynamics model achieves a high correlation with CarSim’s reference outputs at 60 km/h and 90 km/h respectively. Quantitative comparisons between simulation platforms demonstrate exceptional agreement across operational scenarios. At 60 km/h, the lateral velocity ($v_y$) achieves an RMSE of $1.990\times {10}^{-4}$ m/s, while yaw rate ($\omega$) maintains $3.376\times {10}^{-5}$ rad/s precision. These metrics scale consistently with velocity, reaching $4.228\times {10}^{-4}$ m/s for $v_y$ and $7.023\times {10}^{-4}$ rad/s for $\omega$ at 90 km/h. The sub-millimeter/second level discrepancies validate the modeling fidelity of our Simulink implementation relative to the CarSim industry standard, confirming the vehicle dynamics replication accuracy essential for reliable reinforcement learning training.

To verify the superiority of the proposed method, different models were used for training, including SAC (Soft Actor-Critic) [15], TD3 (Twin Delayed Deep Deterministic Policy Gradient) [16], DDPG [17], and DDPGALM algorithms in this paper, as shown in

Table 1. Reinforcement learning parameters in the vehicle DLC maneuver.

Parameters	Reinforcement Learning Parameter Values for Different Methods
	SAC	TD3	DDPG	DDPGALM
Sampling time/s	0.01	0.01	0.01	0.01
Batch size	256	256	256	256
Discount factor $\gamma$	0.99	0.99	0.99	0.99
Initial exploration rate	0.2	0.2	0.2	0.2
Final exploration rate	0.01	0.01	0.01	0.01
Critic learning rate	$3\times {10}^{-4}$	$3\times {10}^{-4}$	$1\times {10}^{-3}$	$1\times {10}^{-3}$
Actor learning rate	$3\times {10}^{-4}$	$5\times {10}^{-3}$	$1\times {10}^{-3}$	$1\times {10}^{-3}$
Soft update coefficient	$5\times {10}^{-3}$	$2\times {10}^{-3}$	$1\times {10}^{-2}$	$1\times {10}^{-2}$
Entropy coefficient $\alpha$	0.2	–	–	–
Target policy noise	–	0.2	–	–
Delayed update frequency	–	2	–	–
Initial penalty factor	–	–	–	0.01
Initial Lagrange multiplier	–	–	–	$5\times {10}^{-3}$

.

Our methodology employs safety-constrained offline reinforcement learning to address the adaptation-safety dilemma. The training process utilizes the ISO 3888-1:1999 emergency lane-change trajectory as the baseline, introducing controlled perturbations during policy optimization to enhance generalization while preserving safety constraints. Highway geometry inherently bounds trajectory deviations through physical road boundaries, ensuring operational scenarios remain within the trained parameter space. Experimental validation in next section confirms the policy’s robust adaptation to all tested trajectory variations without safety violations, demonstrating effective mitigation of offline training limitations through systematic environment-aware perturbation design.

5.2. Simulation Results and Analysis

To comprehensively evaluate the performance of the proposed DDPGALM algorithm, comparative simulations were conducted against general deep reinforcement learning algorithms under two distinct speed conditions: 60 km/h (low speed) and 90 km/h (high speed). The selection of these speeds aims to validate the algorithm’s adaptability across diverse driving scenarios. Low-speed scenario (60 km/h) reflect typical urban driving environments, where precise trajectory tracking and smooth control are essential. In contrast, high-speed scenario (90 km/h) impose greater challenges due to increased dynamic complexity, such as higher lateral acceleration and reduced reaction time, thereby testing the robustness and stability of the control strategy.

Figure 6 shows the training process of different reinforcement learning algorithms in different scenarios. At 60 km/h (Figure 6a,b), all algorithms demonstrate comparable training stability with similar episode termination frequencies and reward convergence patterns, attributable to reduced lane boundary constraint activations under lower dynamic complexity. The high-speed scenario (90 km/h, Figure 6c,d) reveals critical differentiation DDPGALM significantly reduces early termination while sustaining stable reward progression, contrasting with general methods that exhibit frequent training interruptions. The off-road rate of DDPGALM, SAC, TD3, and DDPG algorithms is $24.8\%$, $51.4\%$, $64.4\%$, and $71.9\%$ respectively at 60 km/h, and $14.8\%$, $35.7\%$, $75.5\%$, and $43.4\%$ respectively at 90 km/h shown in (Figure 6a,c). This performance divergence stems from DDPGALM’s adaptive learning architecture, which dynamically adjusts exploration strategies and anticipates safety constraints under high-inertia conditions, thereby maintaining coherent policy evolution despite increased dynamic challenges.

As shown in Figure 7, DDPGALM demonstrates remarkable trajectory tracking accuracy in a single episode of the simulation process. At 60 km/h (Figure 7a,b), both algorithms achieve low lateral errors, but DDPGALM exhibits smoother trajectory adjustments, as evidenced by its reduced oscillation in Figure 7b. At 90 km/h (Figure 7c,d), the baseline DRL struggles with accumulating trajectory deviations, particularly during sharp turns, leading to a maximum lateral error of 0.35 m. In contrast, DDPGALM limits the error to 0.15 m, highlighting its enhanced robustness under high-speed perturbations. Our rigorous lateral tracking standard (<0.05 m error threshold) yields $83.24\%$ and $80.86\%$ trajectory completion rates at 60 km/h and 90 km/h respectively, as quantitatively validated in Figure 7.

Figure 8 analyzes steering angle and lateral acceleration. At 60 km/h (Figure 8a,b), the steering angle and lateral acceleration profiles of DDPGALM are smoother, ensuring passenger comfort. At 90 km/h (Figure 8c,d), the general DRL generates abrupt steering adjustments, resulting in unstable lateral acceleration peaks (up to 1.8 m/s²). DDPGALM, however, maintains controlled steering inputs, restricting lateral acceleration below 1.0 m/s².

Figure 9 shows the reward values calculated by the reward function based on the vehicle’s lateral tracking error, wheel steering angles, and lateral acceleration during the trajectory tracking process. The comparative simulations under low- and high-speed scenario demonstrate that DDPGALM outperforms conventional DRL in terms of training efficiency, trajectory tracking precision, and vehicle state stability. The high-speed scenario (90 km/h) particularly underscores the algorithm’s advanced adaptability to dynamic challenges, a critical requirement for autonomous driving systems operating in complex environments. This dual-speed validation framework provides a comprehensive assessment of the algorithm’s scalability and robustness, ensuring its applicability across diverse real-world scenarios.

6. Conclusions

The vehicle trajectory tracking algorithm based on safe reinforcement learning designed in this paper can improve the stability and accuracy of four-wheel steering vehicles during driving. Compared to conventional deep reinforcement learning algorithms, the augmented Lagrangian DDPG algorithm used in this paper models the vehicle trajectory tracking problem as a constrained Markov decision process, characterized by faster training speeds and higher safety. By utilizing a prioritized experience replay buffer, the model’s ability to learn and explore from experience data is enhanced through flexible and efficient use of data. Experiments show that unsafe behaviors, such as collisions with lane edges, are significantly reduced during training. By designing a reasonable reward function, both tracking accuracy and passenger comfort indicators are ensured. However, the algorithm still does not fully guarantee training safety, indicating limitations in handling constrained problems. Future work needs to focus on more complex road scenarios and explore more effective methods for handling constrained problems.