Autonomous Vehicle Drifting Under Dynamically Changing Road Friction Using Adversarial Agents ^†

Abstract

Autonomous vehicle control has undergone remarkable developments in recent years, especially in maneuvering at the limits of traction. These developments promise improved maneuverability and safety, but they also highlight a constant challenge: translating control strategies developed in simulation into robust, real-world applications. The complexity of real-world environments, with their inherent uncertainties and rapid changes, poses significant obstacles for autonomous systems that need to dynamically adapt to unpredictable conditions, such as varying traction. The aim of this research is to investigate the effectiveness of robust adversarial reinforcement learning (RARL) for controlling circular drift maneuvers under dynamic road adhesion changes and uncertainties. The presented simulation results show that agents trained with RARL can enhance agents developed using only standard reinforcement learning techniques, where they were most critically vulnerable, such as sudden significant loss of traction during the drift initiation phase. This could present another step towards the application of more robust autonomous systems.

1. Introduction

Autonomous vehicle (AV) control at the limits of handling has emerged as a crucial frontier for improving safety, stability, and maneuverability in extreme driving scenarios. Recent years have witnessed growing interest in enabling AVs to perform complex dynamic maneuvers (such as drifting) that inherently operate in the nonlinear region of tire dynamics and demand precise, adaptive control. Traditional autonomous driving solutions, often designed for structured environments, can falter under conditions where the dynamics of the environment—like road adhesion—can suddenly change, prompting a shift toward robust, learning-based strategies and advanced control-theoretic frameworks.

A multitude of control strategies have been proposed to tackle these challenges. Classical control methods, such as model predictive control (MPC) [1,2,3] and linear quadratic regulator (LQR) [4], have been successfully applied to path tracking and drift stabilization by leveraging simplified vehicle models and optimization schemes, even under model-based and environmental uncertainties, such as varying road friction. These approaches offer transparency and safety guarantees, and some have demonstrated feasibility in 1:5 scaled or full-scale vehicle experiments [5,6].

Parallel to these developments, data-driven and neural network-based approaches have gained traction for their ability to approximate unmodeled dynamics [7] and learn control policies directly from experience [8]. Notably, hybrid schemes combining traditional control structures with neural components have been developed to track drift references and adapt to varying terrain using online learning and virtual sensing [9]. Several works have reported real-world implementation of such strategies on test vehicles, including production sports cars retrofitted for by-wire control [10].

Reinforcement Learning (RL) could also be a powerful tool for controlling dynamic maneuvers at the limits of handling [11,12,13,14]. However, it can be a significant challenge to train such agents that could perform well in the real world [15,16] due to the challenge of crossing the so-called sim-to-real gap. On the other hand, RL could still have a reasonable advantage for solving this problem since its wide range of applicability and potential for handling uncertainties for autonomous vehicles [17,18] and other fields too [19].

Apart from the above, another promising approach to bridging this sim-to-real gap is Robust Adversarial Reinforcement Learning (RARL) [20]. RARL enhances the resilience of learned policies by introducing adversarial agents during training. These agents are not simple noise injectors but are instead trained to actively perturb the learning process, creating challenging scenarios such as sudden changes in road adhesion or dynamic friction profiles. By confronting the control agent with these tailored disturbances during training, RARL encourages the development of policies that generalize better to unpredictable real-world conditions. The main advantage of RARL against other conventional approaches, like MPC or game theoretic approaches, is that it does not require explicit definitions for the disturbances that must be accounted for in the training environment; rather, this method focuses on finding these disturbances through learning [15].

This paper aims to explore the application of RARL to the specific problem of circular drift maneuvering under dynamic traction conditions. Drifting, particularly at high slip angles, presents a unique set of control challenges due to the nonlinear and rapidly changing dynamics of tire-road interaction. As such, it serves as a compelling test case for evaluating the robustness of control strategies.

2. Methodology

2.1. Vehicle Modeling and Simulation Environment

In this paper, for training both the drift control and adversary agents, a two-wheel planar RWD vehicle model [21] was implemented in a MATLAB/Simulink (R2025a) simulation environment. The model’s continuous state space has three dimensions ($x\in {ℝ}^3$): the longitudinal velocity $v_x$, the lateral velocity $v_y$, and the yaw rate $r$. The $\dot{x}=f\left(x,u\right)$ state transition function can be written with the following equations:

$${\dot{v}}_x={\displaystyle \frac{1}{m}}{F}_x+r{v}_y,$$

(1)

$${\dot{v}}_y={\displaystyle \frac{1}{m}}{F}_y-r{v}_x,$$

(2)

$$\dot{r}={\displaystyle \frac{M_z}{I_z}},$$

(3)

where ${\dot{v}}_x$ is the lateral acceleration, ${\dot{v}}_y$ is the longitudinal acceleration, $\dot{r}$ is the yaw acceleration, $m$ is the vehicle’s mass, $I_z$ is the yaw inertia of the vehicle’s body frame, $F_x$ is the longitudinal force component, $F_y$ is the lateral force component, and $M_z$ is the yaw moment of the vehicle. The vehicle’s side-slip angle is very important from the aspect of drifting, and it is defined for this model as follows:

$$\beta ={\tan }^{-1}\left({\displaystyle \frac{v_y}{v_x}}\right).$$

(4)

The model’s actuators are the current accelerator pedal position $ac{c}_p$ and the steering wheel angle ${\delta }_{steer}$, thus, $u=\left(ac{c}_p,{\delta }_{steer}\right)\in \left(\left[0\%,100\%\right]\times \left[-420°,420°\right]\right)$. The model also considers actuator dynamics, which means that the actuator demands are transferred through an actuator model to calculate a realized accelerator pedal input $ac{c}_a$ and steering wheel angle ${\delta }_{stee{r}_A}$.

The vehicle model is augmented with a combined-slip Magic Formula tire model [22,23] to address the nonlinearities during drifting. The equations of the tire model connect the actuators and the vehicle force components and moments together. Further details and specifications of this vehicle model can be addressed in [15].

2.2. Reinforcement Learning

Reinforcement learning (RL) is a machine learning framework in which agents learn to perform tasks through repeated interactions with a dynamic environment, guided by the objective of maximizing cumulative rewards. In the context of this study, the environment comprises the vehicle and its surrounding conditions. The primary RL agent, referred to as the protagonist, is responsible for controlling the vehicle’s actuators by selecting actions based on the observed state of the environment. Its goal is to learn a control policy that maximizes a reward signal associated with successful execution of a circular drifting maneuver.

In this paper, the protagonist’s state space consists of the vehicle’s state vector $x$, along with the derivate state vector $\dot{x}$:

$$\mathcal{S}=\left\{S\in {ℝ}^6|S=\left(v_x,v_y,r,{\dot{v}}_x,{\dot{v}}_y,\dot{r}\right)\right\}.$$

(5)

The protagonist’s task is to drive the vehicle, with an inference time of $T_S=0.01\mathrm{s}$. The action space of the protagonist is identical to the actuator space of the vehicle model:

$$\mathcal{A}=\left\{A\in \left(\left[0\%,100\%\right]\times \left[-420°,420°\right]\right)|A=\left(ac{c}_p,{\delta }_{steer}\right)\right\}.$$

(6)

The reward function is designed for the protagonist to perform a quasi-equilibrium circular drifting maneuver with a target speed of $v_{x_{targ}}=10\mathrm{m}/\mathrm{s}$, a target side-slip angle of ${\beta }_{targ}=-20°$, and target path curvature ${\kappa }_{targ}=0.08$. From ${\kappa }_{targ}$ and $v_{x_{targ}}$, a yaw rate target $r_{targ}$ was derived as follows:

$$r_{targ}={\kappa }_{targ}\sqrt{v_{x_{targ}}^2+{\left(\tan \left({\beta }_{targ}\right)v_{x_{targ}}\right)}^2}.$$

(7)

The designed reward function $R\left(S,A\right)$ has three components:

$$R\left(S,A\right)=R_{drift}\left(S\right)+R_{steady}\left(S\right)+R_{act}\left(A\right).$$

(8)

The first component $R_{drift}\left(S\right)$ is responsible for encouraging the agent to initiate and stabilize the target drift maneuver based on $r_{targ}$ and ${\beta }_{targ}$ as follows:

$$R_{drift}\left(S\right)=-\sqrt{{\left({\displaystyle \frac{r}{r_{targ}}}-1\right)}^2+{\left({\displaystyle \frac{\beta }{{\beta }_{targ}}}-1\right)}^2}.$$

(9)

The second component $R_{steady}\left(S\right)$ augments the agent’s goal to try to find a yaw equilibrium around the target state to ensure system stability:

$$R_{steady}\left(S\right)=-0.1\left|\dot{r}\right|.$$

(10)

The third component $R_{act}\left(A\right)$ encourages the agent to minimize the excessiveness of the actuator commands:

$$R_{act}\left(A\right)=-0.7\left(\left|ac{c}_p-50\right|+\left|{\delta }_{steer}\right|\right).$$

(11)

The main idea in this paper is to introduce an adversary agent using adversarial RL [20] to enhance robustness under dynamically changing road adhesion conditions. The adversary achieves this by dynamically modifying the road friction coefficient $\mu$ based on the current $S\in \mathcal{S}$ state of the environment. Thus, always the same information is known for both the protagonist and the adversary, which is the vehicle’s velocities and accelerations. The adversary’s exact action space was defined as the following:

$${\mathcal{A}}_{adv}=\left\{A_{adv}\in \left[0.35,1.05\right]|A_{adv}=\mu \right\}.$$

(12)

Thus, the adversary is able to select any value of road friction between 0.35 and 1.05. These represent values between snow-covered roads and commonly traversed dry asphalt. Since the first protagonist was able to ignore oscillating sudden traction changes which were more frequent than 10 Hz, the inference time of the adversary was set to $T_{S_{Adv}}=0.1\mathrm{s}$ to increase the ability to find more practical disturbances. The adversary’s goal is to minimize the protagonist’s reward function, thus, $R_{adv}\left(S,A_{adv}\right)=-R\left(S,A\right)$.

The protagonists and adversaries were trained in an iterative asynchronous manner (see Figure 1). A first protagonist was trained in advance in environments where the road friction coefficient was constant during training episodes, although diverse between them, selected in the beginning of every episode from the interval $\left[0.35,1.05\right]$. Then, the first adversary was trained against the first protagonist. After the first adversary reached a certain average episode reward value, a second (intermediate) protagonist was initialized based on the first protagonist, and trained against the first adversary. This process was repeated until 10 adversaries were trained. Then, initialized from the last intermediate protagonist, a final protagonist was trained on all of the adversaries collectively to reduce overfitting and forgeting due to the iterative manner of the method.

The number of adversaries was determined with early stopping. After each iteration, we evaluated the previous protagonist’s performance on this new adversary. If the episode reward of the protagonist did not decrease compared to the previous iteration, the iterative training process was stopped, and the final collective training step followed.

Each training episode started from the same initial state $S_{init}=\left(8\mathrm{m}/\mathrm{s},0\mathrm{m}/\mathrm{s},0\mathrm{rad}/\mathrm{s},0{\mathrm{m}/\mathrm{s}}^2,0{\mathrm{m}/\mathrm{s}}^2,0{\mathrm{rad}/\mathrm{s}}^2\right)$, which is a constant speed forward movement. Every training episode lasted for $T_t=10\mathrm{s}$. The main point behind this difference between the start speed and $v_{x_{targ}}$ is to also demand from the agent to accelerate the vehicle to a desired speed for drifting, not just to perform a drift maneuver at the current speed.

3. Results

3.1. Collective Training Resulted in More Robust Autonomous Drifting

In Figure 2, the evaluation of first, last, and final protagonists on a 10 s long episode simulation is shown.

Figure 2 shows the longitudinal state variables $v_x$, ${\dot{v}}_x$, and the target $v_{x_{targ}}$ (row (a)), the path curvature $\kappa$ and the target ${\kappa }_{targ}$ (row (b)), the side slip angle $\beta$ and the target ${\beta }_{targ}$ (row (c)), the agent’s current reward $R\left(S,A\right)$, along with the reward components $R_{drift}\left(S\right)$, $R_{act}\left(A\right)$, and $R_{steady}\left(S\right)$ (row (d)), the actuator commands and their realized values (rows (e,f)), and the current adhesion coefficient $\mu$ (row (g)). The bold lines show the mean values at the given timestamp across evaluations on the 10 trained adversary policies (one evaluation per adversary). The shaded areas around the bold lines serve as showing the uncertainty of the controller with respect to the disturbances at these timesteps by picturing the standard deviation of the signals.

3.2. Realizing Stable Autonomous Drifting in Case of Sudden Loss in Traction

Figure 3 shows a scope graph of a 10 s long simulation in a case where the adhesion coefficient drops during the drift initiation phase from $\mu =0.75$ to $\mu =0.35$, stays there for a while, then increases back to $\mu =0.75$. This case is a representation of a real-world event where the car would traverse onto a heavily wet or snow-covered area of the road during a drift initiation phase, then proceed to a dry asphalt surface again. In the figure, the first row shows the changes in the adhesion coefficient $\mu$, the second row shows the side slip angle $\beta$ and the target ${\beta }_{targ}$, and the third row shows the path curvature $\kappa$ and the target ${\kappa }_{targ}$. While they are not shown in the figure, the longitudinal state variables $v_x$, ${\dot{v}}_x$, and the actuator commands $ac{c}_{ped}$ and ${\delta }_{steer}$ in the first protagonist’s case are similar to the ones shown in the left column in Figure 2. This is also true for the final protagonist, but with the right column in Figure 2.

4. Discussion

Figure 2 tells a lot about the importance of the collective training step of the algorithm. As it might be obvious, the first protagonist performs poorly under the influence of the adversaries. The method itself proves its point here that the adversaries aim at the protagonist’s weakness during training. What is more important is that the last protagonist performs even worse in confidence under the different adversaries. This means that the last protagonist is overfitting on the last adversary’s manipulations. This shows the critical importance of the collective training step, since the final protagonist can confidently handle all adversaries with high performance. The last row in Figure 2 also shows that the trained adversaries were indeed utilizing the whole range of adhesion coefficient values during training to find the weak spots of the agent.

Nevertheless, these findings might also highlight an issue that the trained adversaries could correlate with each other. For example, since the 5th intermediate protagonist possibly overfitted on the 4th adversary and might have some knowledge left on the 2nd and 3rd adversaries, the 1st adversary might be barely remembered. Thus, the 5th adversary could be close to or identical to the 1st adversary and still be a problem for the 5th intermediate protagonist. This phenomenon could be addressed, for example, with an extended collective training structure, where each intermediate protagonist is trained on all the previous adversaries. Another solution might be to apply a more careful early stopping method.

The use case presented in Section 3.2 is a case which was found by the first adversary during training. After the first iteration, the first protagonist was evaluated under the influence of the first adversary. It was found that the first protagonist was unable to initiate a successful drift in the case where the road adhesion suddenly dropped under the drift initiation phase. As Figure 3 shows, the side slip angle in the first protagonist’s case is way off the target, and the values can be translated as the vehicle’s spinning out after the road adhesion had dropped. On the other hand, the final protagonist manages to initiate and even stabilize the drift with high performance even in this case. This also means that the training method indeed produces a robust control agent against the cases which the adversaries have presented, even if it occurred during the first iteration. It is also very important that this specific use case could occur in real life with a decent probability, so the fact that the first protagonist failed during this case is critical.

5. Conclusions

This study has demonstrated the effectiveness of Robust Adversarial Reinforcement Learning (RARL) in enhancing the resilience and performance of autonomous drifting controllers under dynamically changing road adhesion conditions. The results confirm that standard reinforcement learning approaches are insufficient for handling rapid and unexpected changes in environmental dynamics, particularly during critical phases such as drift initiation. In contrast, agents trained with adversarial reinforcement learning consistently outperformed their non-robust counterparts, showing improved stability, tracking accuracy, and robustness against unseen scenarios. The final protagonist, trained through a collective adversarial strategy, exhibited strong generalization across all adversarial challenges, validating the benefit of ensemble adversary training to mitigate overfitting. However, the iterative training framework may result in adversary redundancy or policy forgetting, which future research should address through mechanisms like continual learning, diversity-preserving adversarial generation, or curriculum-based training. Future work will aim to improve the applied methodology to address the potential issues discovered, while also testing the final protagonist on real vehicles.