Deep Reinforcement Learning-Based Torque Vectoring Control Considering Economy and Safety

Abstract

This paper presents a novel torque vectoring control (TVC) method for four in-wheel-motor independent-drive electric vehicles that considers both energy-saving and safety performance using deep reinforcement learning (RL). Firstly, the tire model is identified using the Fibonacci tree optimization algorithm, and a hierarchical torque vectoring control scheme is designed based on a nonlinear seven-degree-of-freedom vehicle model. This control structure comprises an active safety control layer and a torque allocation layer based on RL. The active safety control layer provides a torque reference for the torque allocation layer to allocate torque while considering both energy-saving and safety performance. Specifically, a new heuristic random ensembled double Q-learning RL algorithm is proposed to calculate the optimal torque allocation for all driving conditions. Finally, numerical experiments are conducted under different driving conditions to validate the effectiveness of the proposed TVC method. Through comparative studies, we emphasize that the novel TVC method outperforms many existing related control results in improving vehicle safety and energy savings, as well as reducing driver workload.

1. Introduction

The automotive industry has witnessed significant advancements in electric vehicle (EV) technology over the past decade [1,2]. EVs are gaining popularity due to their environmental friendliness and low operating costs. Conventional EVs use a centralized drive system, which transfers the power from the battery to the wheels through a transmission system. However, this system has several limitations such as limited efficiency and reduced stability and control [3]. The four-in-wheel-motor independent-drive electric vehicle (4MIDEV), which places an electric motor in each wheel of the vehicle, has emerged as a promising solution to overcome these limitations [4].

Torque vectoring control (TVC) refers to the ability to distribute the torque to each wheel of the vehicle independently. The torque allocation can be controlled by adjusting the electric motor output torque to each wheel, which is achieved by using an electronic control unit. Several factors affect TVC, including vehicle speed, acceleration, road conditions, and motor efficiency [5]. Hence, it is important to research the TVC of 4MIDEV considering both economy and safety according to the current vehicle state [6].

The TVC is usually classified into holistic and hierarchical structures according to the control framework. The holistic TVC structure typically employs model predictive control (MPC) [7]. However, this approach suffers from certain drawbacks, including a large system size, substantial computational effort, and implementation challenges [8]. Therefore, in practical applications, TVC generally adopts a hierarchical control structure. The active safety control layer generates reference control quantities based on the reference model, while the torque allocation layer generates control commands for each actuator, such as tire longitudinal force and torque.

Many scholars use sliding mode controllers (SMC) to solve the torque allocation problem. Reference [9] presents an SMC with its stability being proven using a Lyapunov function. The Lyapunov control method proposed in the article overcomes the limitations of SMC by improving control accuracy and dynamic tracking performance while effectively reducing chattering and abrupt changes in the control system. In reference [10], a new SMC law is designed to reduce chattering and make state variables converge faster. Reference [11] presents an adaptive and robust controller that combines the driving characteristics of professional drivers to improve vehicle stability and maneuverability and relieve the driver’s workload. Experimental results demonstrate the effectiveness of the proposed controller in reducing the peak steering angle, reducing the driver’s operating load, and addressing the chattering issue in conventional SMC. Reference [12] presents a coordination control strategy for the vehicle stability control system and differential drive-assisted steering, which considers tire sideslip and aims to improve vehicle stability in different driving conditions and presents CarSim simulations and vehicle experiments verifying its effectiveness. Moreover, some researchers have focused on developing fault-tolerant control strategies that can ensure the safety and performance of the 4MIDEV even in the presence of actuator faults. Reference [13] proposes a fault-tolerant control method for 4MIDEV that considers both vehicle safety and motor power consumption.

However, traditional TVC methods often require complex and computationally expensive control algorithms, which limit their effectiveness. To overcome these challenges, reinforcement learning (RL) offers a machine learning approach that enables an agent to learn how to make optimal decisions by interacting with an environment and receiving rewards as feedback [14]. RL has shown significant potential for improving control performance while reducing complexity in many areas, such as robot control [15] and autonomous driving [16]. Reference [17] proposed a direct torque allocation algorithm that employs a deep RL technique to enhance the safety and fuel economy of the vehicle. This paper uses the integrated control framework to train the RL directly. However, the vehicle’s dynamics model is complex, making it difficult to apply RL directly. Additionally, the RL is usually applied to the problem in which objects are difficult to model, or the model is too complex. For stability control, the active safety control layer already has a detailed dynamic model; the required additional yaw moment and longitudinal force can be well solved using model-based control methods such as optimal control or sliding mode control. Therefore, it is not appropriate to use direct RL control in the active safety control layer. Motivated by the above issues, a novel TVC method based on RL for 4MIDEV is proposed in this paper. The contributions of this study are summarized as follows:

• Unlike the reference [18], this paper proposes a TVC method that takes into account both economy and safety. Specifically, the torque allocation layer based on deep RL adaptively adjusts the torque of each wheel according to the current vehicle state.
• An improved heuristic randomized ensembled double Q-learning (REDQ) algorithm is introduced for EV control, which reduces the training complexity of RL compared to existing RL algorithms for direct motor torque control.
• The Fibonacci tree-based tire model identification method is employed, which achieves higher identification accuracy than the genetic algorithm (GA) [19] and the particle swarm optimization (PSO) algorithm [20].

2. The TVC Framework and System Model

2.1. The TVC Framework

Figure 1 illustrates the block diagrams of the TVC system, which is composed of four modules: tire model identification, the vehicle reference model, the active safety control layer, and the RL-based torque allocation layer. The tire mathematical model is obtained through experimental data based on the Fibonacci tree optimization (FTO) algorithm. The active safety control layer is designed to generate longitudinal force $F_x$ and additional yaw moment $M_z$. The torque allocation layer employs the heuristic REDQ algorithm with integrated consideration of both economy and safety to distribute the four-wheel torque. Moreover, the average allocation method gives the RL algorithm heuristic training and reduces its training time.

2.2. Tire Model Identification

The tire is a complex system that interacts with the road surface and the vehicle’s suspension system. The tire’s behavior can significantly affect the vehicle’s dynamics, including acceleration, braking, cornering, and ride comfort. Therefore, accurate modeling of the tire is essential for vehicle control, design, and optimization. A good tire model can provide reliable predictions of the tire forces and moments, which are critical inputs to vehicle dynamic control algorithms. Additionally, tire models can help understand the effects of different tire designs and operating conditions on vehicle performance and handling.

The magic formula tire model is a semi-empirical tire model that has become one of the most widely used tire models in vehicle dynamics and control research. Additionally, the model includes several parameters that can be tuned to match experimental data or represent different tire designs. According to the magic formula, the longitudinal force $F_{xij}^0$, and lateral force $F_{yij}^0$ of the tire model are calculated by:

$$\begin{array}{l}{F}_{xij}^0=D_x\sin [C_x\arctan (B_x{\varphi }_{xij})]+S_{vx}\\ {\varphi }_{xij}=\left(1-E_x\right)({\lambda }_{ij})+(E_x/B_x)\arctan (B_x({\lambda }_{ij}))\\ F_{yij}^0=D_y\sin [C_y\arctan (B_y{\varphi }_{yij})]+S_{vy}\\ {\varphi }_{yij}=\left(1-E_y\right)({\alpha }_{ij})+(E_y/B_y)\arctan (B_y({\alpha }_{ij}))\end{array}$$

(1)

where F_z is the vertical load of the tire and ${\lambda }_{ij}$ and ${\alpha }_{ij}$ tire slip angle or tire longitudinal slip ratio, respectively. $ij\in \left\{fl,fr,rl,rr\right\}$. $B_x=BC{D}_x/(C_x·D_x)$, $C_x=b_0$, $D_x=b_1{F}_z^2+b_2{F}_z$, $BC{D}_x=\left(b_3{F}_z^2+b_{4}F\right)e^{-b_5{F}_z}$, $E_x=b_6{F}_z^2+b_7{F}_z+b_8$, $B_y=BC{D}_y/(C_y·D_y)$, $C_y=a_0$, $D_y=a_1{F}_z^2+a_2{F}_z$, $BC{D}_y=a_3\sin (2\arctan (F_z/a_4))$, $E_y=a_6{F}_z+a_7$.

The raw data for the FTO tire identification algorithm is obtained through a comprehensive tire mechanics test bench. The powertrain drives the simulated road surface at a translational speed of 0.3 m/s from one side of the test bench to the other. The three-dimensional force sensors of the test bench are used to measure the lateral and longitudinal forces. The friction coefficient is 0.8, the slip ratio is 0–0.7, and the lateral slip angle is 0–20 degrees. The test was conducted at three vertical loads: 10 kN, 15 kN, and 20 kN, and the experimental data are shown in Figure 2. Each point in Figure 2 represents a test result. The FTO algorithm is a computational optimization algorithm based on the golden section method and the Fibonacci optimization principle. It optimizes the global search and local search alternately and iteratively constructs the Fibonacci tree structure to find the best solution. The FTO algorithm consists of two stages for each search: a global search and a local search.

In the global search stage, global nodes $N_i$ are randomly generated within the global scope. According to the fitness of each node, global trial nodes $W_a\left(a=1,2,\cdots ,F_i\right)$ are generated as follows:

$$W_a=\left\{\begin{array}{l}{N}_i+\frac{F_{i-1}}{F_i}\cdot \left(B_{ij}-N_i\right),\mathrm{if}Fi{t}_{best}\left(N_i,B_{ij}\right)=N_i\hfill \\ B_{ij}+\frac{F_{i-1}}{F_i}\cdot \left(N_i-B_{ij}\right),\mathrm{otherwise}\hfill \end{array}\right.$$

(2)

F_i is the Fibonacci series, and the general formula is as follows:

$$F_i=\frac{1}{\sqrt{5}}\left[{\left(\frac{1+\sqrt{5}}{2}\right)}^i-{\left(\frac{1-\sqrt{5}}{2}\right)}^i\right]$$

(3)

where $Fi{t}_{best}\left(N_i,B_{ij}\right)$ represents the better adaptability of $N_i$ and $B_{ij}$, which is calculated according to the objective functions (6) and (7).

In the local search stage, the local trial nodes V_b are generated according to the best node of the current node B_i1 and the current node B_ij:

$$\begin{array}{l}{B}_{i1}=Fi{t}_{best}\left(B_{ij}\right)\\ V_b=B_{i1}+\frac{F_{i-1}}{F_i}\left(B_{ij}-B_{i1}\right)\end{array}$$

(4)

Finally, W_a, V_b, and the current node B_ij are sorted according to fitness, and the best F_(i+1) nodes are retained as the next generation nodes B_(i+1)j:

$$B_{\left(i+1\right),j}=Fi{t}_{best}\left(W_a,V_b,B_{i,j}\right)$$

(5)

The algorithm is expressed as Algorithm 1.

Algorithm 1. FTO algorithm

1. Set the depth of Fibonacci tree N and the number of identification parameters n;

2. Randomly generate an initial node B11: randomly generate a global random node N1;

3. Repeat:

4. Generate Fi global trial nodes W1~WFi according to the global random node Ni and the node Bij;

5. Generate Fi−1 local trial nodes V1~VFi−1 according to the best adaptable element Bi1 in the current node and the remaining nodes;

6. Get the next generation node Bi+1j;

7. Update the node set. Incorporate the newly generated trial nodes into the current node set S, calculate the fitness function and sort, and retain the first Fi+1 nodes;

8. Until Fi+1 ≥ FN

9. Output the optimal node;

The objective function for first-level parameter identification is as follows:

$$\begin{array}{l}f{1}_x={\displaystyle \sum _{i=1}^{N_x}}{\{D_x\sin (C_x\mathrm{atan}(B_x{\lambda }_i-E_x(B_x-\arctan (B_x{\lambda }_i))))-F_{xi}^{*}\}}^2\\ f{1}_y={\displaystyle \sum _{i=1}^{N_y}}{\{D_y\sin (C_y\mathrm{atan}(B_y{\alpha }_i-E(B_y-\arctan (B_y{\alpha }_i))))-F_{yi}^{*}\}}^2\end{array}$$

(6)

where, F_xi^*, F_yi,^*λ_i, and α_i are the experimental data of longitudinal force, lateral force, slip rate, and side slip angle under different vertical loads, respectively. N_x = 29 and N_y =20 represent the number of tests.

The objective function for second-level parameter identification is as follows:

$$\begin{array}{ll}f{2}_{x1}={\displaystyle \sum _{j=1}^M{(b_0-C_{xj}^{*})}^2}& f{2}_{y1}={\displaystyle \sum _{j=1}^M{(a_0-C_{yj}^{*})}^2}\\ f{2}_{x2}={\displaystyle \sum _{j=1}^M{[(b_1{F}_{zj}^2+b_2{F}_{zj})-D_{xj}^{*}]}^2}& f{2}_{y2}={\displaystyle \sum _{j=1}^M{[(a_1{F}_{zj}^2+a_2{F}_{zj})-D_{yj}^{*}]}^2}\\ f{2}_{x3}={\displaystyle \sum _{j=1}^M{[(b_3{F}_{zj}^2+b_4{F}_{zj})e^{-b_5{F}_z}-BC{D}_{xj}^{*}]}^2}& f{2}_{y3}={\displaystyle \sum _{j=1}^M{[a_3\sin (2\arctan (F_{zj}/a_4))-BC{D}_{yj}^{*}]}^2}\\ f{2}_{x4}={\displaystyle \sum _{j=1}^M{[(b_6{F}_{zj}^2+b_7{F}_{zj}+b_8)-E_{xj}^{*}]}^2}& f{2}_{y4}={\displaystyle \sum _{j=1}^M{[(a_6{F}_{zj}+a_7)-E_{yj}^{*}]}^2}\\ BC{D}_{xj}^{*}=B_{xj}^{*}·C_{xj}^{*}·D_{xj}^{*}& BC{D}_{yj}^{*}=B_{yj}^{*}·C_{yj}^{*}·D_{yj}^{*}\end{array}$$

(7)

where $B_{xj}^{*},C_{xj}^{*},D_{xj}^{*},E_{xj}^{*},B_{yj}^{*},C_{yj}^{*},D_{yj}^{*},E_{yj}^{*}$ are the first-level parameter values identified by the vertical load applied by group j. After classifying the parameters of the tire model into first-level identification parameters and second-level identification parameters, the hierarchical identification approach is found to not only ensure the accuracy of the tire mathematical model but also improve its identification efficiency compared to the non-hierarchical approach.

The recognition result is shown in Figure 2. To further investigate the performance of the FTO algorithm, a comparative analysis with the commonly used GA and PSO optimization algorithms was conducted, and the results are presented in

Table 1. First-level parameters identification results.

Item	10 kN	15 kN	20 kN
FTO relative residual of longitudinal force	1.40%	1.37%	1.40%
GA relative residual of longitudinal force	4.61%	2.21%	1.73%
PSO relative residual of longitudinal force	3.44%	1.74%	1.66%
FTO relative residual of lateral force	1.48%	1.37%	1.39%
GA relative residual of lateral force	1.78%	1.63%	1.48%
PSO relative residual of lateral force	1.65%	1.69%	1.56%

. The algorithm of GA is from [19], and the algorithm of PSO is from [20]. The depth of the Fibonacci tree is set to 7, while the parameters of the GA are adopted from [19]. The generation gap is set to 0.6, the population size is set to 50, and the crossover probability and mutation probability are set to 0.9 and 0.0097, respectively. Similarly, the parameters of the PSO algorithm are selected from [20], with the population size set to 40 and the inertia weight set to 0.7298, constant $c_1=2$, and constant $c_2=2$.

The relative residual $e$ is expressed as:

$$e=\sqrt{{\displaystyle \sum {(F^{MF}-F^{*})}^2}/{\displaystyle \sum {(F^{*})}^2}}\times 100\%$$

(8)

where $F^{MF}$ is the tire force value identified by the FTO algorithm, and $F^{*}$ is the experimental value of the tire force.

Based on the results presented in Table 1, it is observed that the FTO algorithm outperforms the conventional GA and PSO algorithms in terms of parameter recognition accuracy and relative residual error. The first-level parameter identification has a global relative residual of 1.48%. Moreover, for complex parameter identification problems, it is beneficial to improve the efficiency of parameter identification by classifying the parameters and reducing the number of independent variables of the single identification objective function. After obtaining the identified parameters, we can obtain the mathematical model of the tire, and the longitudinal and lateral forces of the tire can be calculated by substituting different vertical loads.

2.3. Vehicle Reference Model

In vehicle control, achieving both good tracking performance and disturbance rejection is critical for safe and efficient operation. A 2-degree-of-freedom (DOF) reference model is a popular choice for designing controllers that provide these performance objectives. The 2-DOF vehicle model is expressed as:

$$\begin{array}{l}m{v}_x\left(\dot{\beta }+\gamma \right)=2C_f\left({\delta }_f-\beta -a\gamma /v_x\right)+2C_r\left(-\beta +b\gamma /v_x\right)\\ I_z\dot{\gamma }=2a{C}_f\left({\delta }_f-\beta -a/v_x\right)-2b{C}_r\left(-\beta +b/v_x\right)\end{array}$$

(9)

where m is the total mass of the vehicle, $v_x$ is the longitudinal velocity, β is the side slip angle of the vehicle center of gravity (CG), γ is the yaw rate, and C_f and C_r are the front and rear tire cornering stiffness, respectively. a and b are the distance from the front and rear wheel axles to the CG, respectively. ${\delta }_f$ is the steering angle of the front wheels, and $I_z$ is the yaw mass moment of inertia.

Applying the Laplace transform to Equation (9) we can obtain the following transfer function:

$$\frac{\gamma (s)}{{\delta }_f(s)}=G_{\gamma }\frac{{\omega }_n^2\left({\xi }_{1}s+1\right)}{s^2+2{\omega }_n{\xi }_2\cdot s+{\omega }_n^2}$$

(10)

where ${\omega }_n=\frac{2\left(a+b\right)}{v_x}\sqrt{\frac{C_f{C}_{\mathrm{r}}\left(1+K{v}_x^2\right)}{m{I}_z}}$, $G_{\gamma }=\frac{v_x}{\left(a+b\right)\left(1+K{v}_x^2\right)}$, $K=-\frac{m\left(a{C}_f-b{C}_r\right)}{2C_f{C}_r{\left(a+b\right)}^2}$, ${\xi }_1=\frac{m{v}_{x}a}{2C_r\left(a+b\right)}$, ${\xi }_2=\frac{m\left(C_f{a}^2+C_{\mathrm{r}}{b}^2\right)+I_z\left(C_f+C_{\mathrm{r}}\right)}{2\left(a+b\right)\sqrt{m{I}_z{C}_f{C}_{\mathrm{r}}\left(1+K{v}_x^2\right)}}$.

The steady-state value of the yaw rate ${\gamma }^{*}$ can be obtained from Equation (10) as:

$${\gamma }^{*}=G_{\gamma }{\delta }_f$$

(11)

Considering the road adhesion coefficient, γ_d is limited by the following equation:

$$a_y={\gamma }_d{v}_x\le \mu g$$

(12)

where $\mu$ is the road adhesion coefficient, and g is the acceleration of gravity.

Then, the reference yaw rate is expressed as

$${\gamma }_d=\min \left\{\left|{\gamma }^{*}\right|,\left|\frac{\mu g}{v_x}\right|\right\}\cdot \mathrm{sgn}\left({\delta }_f\right)$$

(13)

We note that vehicle safety is guaranteed when the sideslip angle $\beta$ varies within a small range for the normal operating condition of the vehicle. Following related works on vehicle dynamics control [21,22], we conservatively set the desired sideslip angle ${\beta }_d=0$ to maintain vehicle safety under extreme operating conditions.

2.4. Vehicle 7-DOF Dynamic Model

Vehicle models are used to predict vehicles’ behavior under different driving conditions and to design control systems that achieve desired performance objectives. Briefly, 2-DOF vehicle models are widely used due to their simplicity and ease of use, but they have limitations in representing vehicles’ motion under complex driving conditions. Additionally, nonlinear 7-DOF models of vehicles provide a more accurate representation of vehicles’ motion and are essential for advanced control system design, simulation, and testing [23]. Therefore, a nonlinear 7-DOF vehicle dynamics model is introduced. The vehicle dynamics model, as illustrated in Figure 3, serves as a foundation for the TVC in this research.

$$\begin{array}{l}m{\dot{v}}_x=F_{xrl}+F_{xrr}+\left(F_{xfl}+F_{xfr}\right)\cos {\delta }_f-\left(F_{\mathrm{y}fl}+F_{yfr}\right)\sin {\delta }_f+m{v}_y\gamma \\ m{\dot{v}}_y=F_{yrl}+F_{yrr}+\left(F_{xfl}+F_{xfr}\right)\sin {\delta }_f+\left(F_{\mathrm{y}fl}+F_{yfr}\right)\cos {\delta }_f+m{v}_x\gamma \\ I_z\dot{\gamma }=a\left(F_{yfl}+F_{yfr}\right)\cos {\delta }_f-b\left(F_{yrl}-F_{yrr}\right)+\frac{d_w}{2}\left(F_{yfl}-F_{yfr}\right)\sin {\delta }_f+M_z\end{array}$$

(14)

$$I_{\omega ij}{\dot{\omega }}_{ij}=-F_{xij}{R}_{\omega }+T_{tij}$$

(15)

$$\begin{array}{l}{M}_z=F_{xfl}\left(a\sin {\delta }_f-d_w\cos {\delta }_f/2\right)\\ +F_{xf\mathrm{r}}\left(a\sin {\delta }_f+d_w\cos {\delta }_f/2\right)+d_w/2\left(F_{xrr}-F_{xrl}\right)\end{array}$$

(16)

where $v_y$ is the vehicle longitudinal velocity. $d_w$ is the track width. ${\omega }_{ij}$ is the wheel rotation rate, $I_{ij}$ is the wheel moment of inertia, and $T_{ij}$ is the output torque of the in-wheel motor.

The tire slip angle ${\alpha }_{ij}$ is expressed as

$$\begin{array}{l}{\alpha }_{fl}=\arctan \left(\frac{v_y+a\gamma }{v_x-\frac{1}{2}{d}_w\gamma }\right)-{\delta }_f,{\alpha }_{rl}=\arctan \left(\frac{v_y-b\gamma }{v_x-\frac{1}{2}{d}_w\gamma }\right)\hfill \\ {\alpha }_{fr}=\arctan \left(\frac{v_y+a\gamma }{v_x+\frac{1}{2}{d}_w\gamma }\right)-{\delta }_f,{\alpha }_{rr}=\arctan \left(\frac{v_y-b\gamma }{v_x+\frac{1}{2}{d}_w\gamma }\right)\hfill \end{array}$$

(17)

The tire slip ratio ${\lambda }_{ij}$ is calculated by

$${\lambda }_{ij}=\left(v_{ij}-{\omega }_{\mathrm{ij}}{R}_{\omega }\right)/v_{ij}$$

(18)

The expressions of the four-wheel speeds are given as

$$\begin{array}{l}{v}_{fl}=(v_y+a\gamma )\sin {\delta }_f+(v_x-\gamma d_w/2)\cos {\delta }_f,v_{rl}=v_x-\gamma d_w/2\\ v_{fr}=(v_y+a\gamma )\sin {\delta }_f+(v_x+\gamma d_w/2)\cos {\delta }_f,v_{rr}=v_x+\gamma d_w/2\end{array}$$

(19)

The tire force $F_{zij}$ is expressed as

$$\begin{array}{c}{F}_{zfl}=\frac{m}{a+b}\left(\frac{gb}{2}-\frac{a_x{h}_g}{2}-\frac{a_y{h}_{g}b}{d_w}\right)\\ F_{zfr}=\frac{m}{a+b}\left(\frac{gb}{2}-\frac{a_x{h}_g}{2}+\frac{a_y{h}_{g}b}{d_w}\right)\\ F_{zrl}=\frac{m}{a+b}\left(\frac{ga}{2}+\frac{a_x{h}_g}{2}-\frac{a_y{h}_{g}a}{d_w}\right)\\ F_{zrr}=\frac{m}{a+b}\left(\frac{ga}{2}+\frac{a_x{h}_g}{2}+\frac{a_y{h}_{g}a}{d_w}\right)\end{array}$$

(20)

where h_g is the height of the center of gravity.

As the tire force needs to meet the attachment ellipse, $F_{xij}$ and $F_{yij}$ are expressed as:

$$\begin{array}{cc}{F}_{xij}=F_{x0ij}{\psi }_{xij}/{\psi }_{ij}& F_{yij}=F_{y0ij}{\psi }_{xij}/{\psi }_{ij}\end{array}$$

(21)

where ${\psi }_{ij}=\sqrt{{\psi }_{xij}^2+{\psi }_{yij}^2},{\psi }_{xij}=-\frac{{\lambda }_{ij}}{1+{\lambda }_{ij}},{\psi }_{yij}=-\frac{\tan ({\alpha }_{ij})}{1+{\lambda }_{ij}}$.$F_{x0ij}$ and $F_{y0ij}$ are calculated by Equation (1).

3. The TVC Algorithm

This section describes the details of the RL-based TVC algorithm, which includes an active safety control layer and an RL-based torque allocation layer.

3.1. Active Safety Control Layer

The role of the active safety control layer is to calculate the total longitudinal force $F_x$ and additional yaw moment $M_z$ necessary to ensure vehicle stability, which will serve as reference values for the torque allocation layer. If the calculated final torque of each wheel by the torque allocation layer can achieve the total longitudinal force and additional yaw moment reference values, the stability of the vehicle can be ensured.

According to Equations (14)–(20), we can obtain the following nonlinear continuous system.

$$\begin{array}{l}\dot{x}(t)=f(x(t),u(t))\\ y(t)=C\cdot x(t)\end{array}$$

(22)

where $x(t)={\left[v_x(t),v_y(t),\gamma (t)\right]}^{\mathrm{T}}$, $y(t)={\left[v_x(t),\beta (t),\gamma (t)\right]}^{\mathrm{T}}$, $u={\left[F_x(t),M_z(t)\right]}^{\mathrm{T}}$, $C=\left[\begin{array}{ccc}1& 0& 0\\ 0& 1/v_x(t)& 0\\ 0& 0& 1\end{array}\right]$.

Discretizing the Equation (22), we can obtain the difference equation at time k:

$$\begin{array}{l}{v}_x\left(k+1\right)=T_s\left(\frac{1}{m}\left(F_{xrl}+F_{xrr}+\left(F_{xfl}+F_{xfr}\right)\cos {\delta }_f-\left(F_{\mathrm{y}fl}+F_{yfr}\right)\sin {\delta }_f\right)+v_y\left(k\right)\gamma \left(k\right)\right)+v_x\left(k\right)\\ v_y\left(k+1\right)=T_s\left(\frac{1}{m}\left(F_{yrl}+F_{yrr}+\left(F_{xfl}+F_{xfr}\right)\sin {\delta }_f+\left(F_{\mathrm{y}fl}+F_{yfr}\right)\cos {\delta }_f\right)+v_x\left(k\right)\gamma \left(k\right)\right)+v_y\left(k\right)\\ \gamma \left(k+1\right)=T_s\left(\frac{1}{I_z}\left(a\left(F_{yfl}+F_{yfr}\right)\cos {\delta }_f-b\left(F_{yrl}-F_{yrr}\right)+\frac{d_w}{2}\left(F_{yfl}-F_{yfr}\right)\sin {\delta }_f+M_z\right)\right)+\gamma \left(k\right)\\ y(k)=C\cdot x(k)\end{array}$$

(23)

With the active safety control layer defined in Equation (23), we can calculate the $M_z$ and $F_x$ to ensure vehicle safety.

Nonlinear MPC involves the optimization of a nonlinear objective function, which is subject to constraints on the system dynamics and control inputs. The nonlinear MPC controller for the system described in Equation (23) can be designed by formulating an optimal control problem:

$$\underset{X,U}{\min }{\displaystyle \sum _{k=0}^{Np-1}[{(Y-Y_{ref})}^T{W}_Q(Y-Y_{ref})+U^T{W}_{R}U+\Delta U^T{W}_S\Delta U]}$$

(24)

$$\begin{array}{cc}\mathrm{s}.\mathrm{t}.& x_0=x_{in}\end{array}$$

(25)

$$\begin{array}{c}{x}_{k+1}=f_d(x_k,u_k)\\ y(k)=C\cdot x(k)\end{array}$$

(26)

$$x_{\min }\le x_k\le x_{\max }$$

(27)

$$u_{\min }\le u_k\le u_{\max }$$

(28)

where $Y\left(k\right)={\left[y\left(k\right),y\left(k+1\right),\cdots ,y\left(k+N_p\right)\right]}^{\mathrm{T}}$ and $U\left(k\right)={\left[u\left(k\right),u\left(k+1\right),\cdots ,u\left(k+N_c\right)\right]}^{\mathrm{T}}$ are the $N_p$ step state vector and the $N_c$ step input vector, respectively. $\Delta u=u_k-u_{k-1}$. Additionally, $Y_{ref}\left(k\right)={\left[y_{ref}\left(k\right),y_{ref}\left(k+1\right),\cdots ,y_{ref}\left(k+N_p\right)\right]}^{\mathrm{T}}$ where $y_{ref}\left(k\right)=\left[v_{xd}\left(k\right),{\beta }_d\left(k\right),{\gamma }_d\left(k\right)\right]$ is the reference of the state at step k and $f_d$ is the difference equation in Equation (23). $W_Q,W_R,W_S$ are the weighting matrices.

The goal of optimal control is to minimize the objective function given by Equation (24), subject to constraints such as initial conditions Equation (25), discrete nonlinear system dynamics Equation (26), state constraints Equation (27), and control constraints Equation (28). The objective function Equation (24) comprises three main components, namely, the minimization of state-reference trajectory error, system input, and input variation.

To solve the nonlinear programming problem represented by Equations (23)–(27), the widely used sequential quadratic programming algorithm [24] can be employed. Once the optimal input vector is obtained, the first control input $u\left(k\right)$ is then applied to the system, and the process is repeated at the next time step.

3.2. Torque Allocation Layer

Based on Equations (14) and (15), the vehicle longitudinal force $F_x$ and the additional yaw moment $M_z$ are expressed as:

$$F_x=(T_{fl}\cos {\delta }_f+T_{fr}\cos {\delta }_f+T_{rl}+T_{rr})/R_{\omega }$$

(29)

$$\begin{array}{l}{M}_z=\frac{1}{R_{\omega }}((-\frac{d_w}{2}\cos {\delta }_f+a\sin {\delta }_f)T_{fl}\\ +(\frac{d_w}{2}\cos {\delta }_f+a\sin {\delta }_f)T_{fr}-\frac{d_w}{2}{T}_{rl}+\frac{d_w}{2}{T}_{rr})\end{array}$$

(30)

To ensure the stability of the vehicle, the four-wheel torque needs to satisfy Equations (29) and (30). The aim of the torque allocation layer is to distribute the four-wheel torque to achieve $M_z$ and $F_x$. For 4MIDEV, it is a typical over-actuated system with more actuators than degrees of freedom of the system. Therefore, torque allocation is a problem worth researching; it is important to reduce the power consumption of the motor while ensuring safety.

In the torque allocation layer, we considered a combination of safety and power consumption. In particular, the power consumption and safety weights are dynamically adjusted according to the current vehicle state based on RL. To this end, the task of this paper is the torque allocation of the 4MIDEV, which guarantees the safety and power consumption of the EV.

3.2.1. Average Allocation Method

As presented in references [18,25], the most common method of torque allocation is the average allocation method. The average allocation algorithm here serves two purposes. The first is as a base method for comparison. The second is as a heuristic training method for RL in the early stages of training. We have stated this in the revised manuscript. The average allocation methods are shown in the following [26]:

$$\begin{array}{l}{T}_{fl}=\frac{1}{4}\frac{F_x{R}_{\omega }}{\cos {\delta }_f}+\frac{1}{4}\frac{M_z{R}_{\omega }}{-\frac{1}{2}{d}_w\cos {\delta }_f+a\sin {\delta }_f}\hfill \\ T_{fr}=\frac{1}{4}\frac{F_x{R}_{\omega }}{\cos {\delta }_f}+\frac{1}{4}\frac{M_z{R}_{\omega }}{\frac{1}{2}{d}_w\cos {\delta }_f+a\sin {\delta }_f}\hfill \\ \begin{array}{l}{T}_{rl}=\frac{1}{4}{F}_x{R}_{\omega }-\frac{1}{4}\frac{M_z{R}_{\omega }}{\frac{1}{2}{d}_w}\\ T_{rr}=\frac{1}{4}{F}_x{R}_{\omega }+\frac{1}{4}\frac{M_z{R}_{\omega }}{\frac{1}{2}{d}_w}\end{array}\hfill \end{array}$$

(31)

3.2.2. RL-Based Torque Allocation Algorithm

This section describes the details of the RL-based torque allocation algorithm. The state plays a crucial role in the RL algorithm, and in this study, we define the state space using four states: the yaw rate of deviation $e_{\gamma }=\gamma -{\gamma }_{des}$, the stability indicator $\epsilon$, velocity $v$, and velocity deviation $e_v$. Additionally, the action is defined as four-wheel torque.

$$S=\left\{e_{\gamma },\epsilon ,v,e_v\right\},A=\left\{T_{fl},T_{fr},T_{rl},T_{rr}\right\}$$

(32)

where the $\epsilon$ is obtained from the phase plane. In vehicle dynamics, the driving stability and instability regions can be depicted using a phase portrait [27]. The $\epsilon$ is defined as follows.

$$\epsilon =\left|\beta /B_2+\dot{\beta }{B}_1/B_2\right|$$

(33)

where B₁ and B₂ are the parameters associated with the adhesion coefficient. Their corresponding values are specified in reference [28]. This paper aims to improve the economy of EVs while ensuring safety, so the reward function $R$ is expressed as the following four parts:

Firstly, Equations (29) and (30) ensure that the four-wheel torque satisfies the additional yaw moment $M_z$ and longitudinal force $F_x$ from the active safety control layer. Therefore, to ensure vehicle stability, the torque allocation layer needs to satisfy the following equation:

$$A_s{u}_c=B_u$$

(34)

where $A_s=\frac{1}{R}\left[\begin{array}{cccc}\cos {\delta }_f& \cos {\delta }_f& 1& 1\\ -\frac{l_w}{2}\cos {\delta }_f+l_f\sin {\delta }_f& \frac{l_w}{2}\cos {\delta }_f+l_f\sin {\delta }_f& -\frac{l_w}{2}& \frac{l_w}{2}\end{array}\right]$, $B_u={\left[\begin{array}{cc}{F}_x& M_Z\end{array}\right]}^T$, $u_c={\left[\begin{array}{cccc}{T}_{fl}& T_{fr}& T_{rl}& T_{rr}\end{array}\right]}^T$.

Therefore, we constructed $R_{upper}$ to consider vehicle stability in the torque allocation layer from the perspective of the active safety layer:

$$R_{upper}=-{\left(A_s{u}_c-B_u\right)}^T{W}_s\left(A_s{u}_c-B_u\right)$$

(35)

Second, the drive system power consumption is minimized by using the reward function Equation (36).

$$R_{economy}=-W_{economy}{\displaystyle \sum _{ij=\left\{fl,fr,rl,rr\right\}}{P}_{ij}}=-W_{economy}{\displaystyle \sum _{ij=\left\{fl,fr,rl,rr\right\}}\frac{T_{ij}{\omega }_{ij}}{\eta \left(T_{ij},{\omega }_{ij}\right)}}$$

(36)

where $W_{economy}$ is the penalty factors of economy, and ${\omega }_{ij}$ is the speed of motor $ij$. Additionally, the motor efficiency $\eta$ is shown in Figure 4.

Lastly, the stability indicators and driver load are considered in the reward function:

$$\begin{array}{l}{R}_{stability}=-W_{stability}\left({\left(\beta -{\beta }_d\right)}^2+{\left(\gamma -{\gamma }_d\right)}^2\right)\\ R_{driver}=-W_{driver}\left({\dot{\delta }}_{sw}^2+W_{a\_\delta }\cdot a_x^2\right)\end{array}$$

(37)

where $W_{stability}$ and $W_{driver}$ are the penalty factors for safety and driver load, respectively.

$$R=R_{upper}+R_{economy}+R_{stability}+R_{driver}$$

(38)

The safety of torque allocation consideration consists of two parts: $R_{upper}$ and $R_{stability}$. As safety is the primary concern in vehicle operation, we chose penalty factors to ensure that the reward function satisfies the following relationship:

$$\underset{safety}{\underbrace{R_{\mathit{upper}}+R_{\mathit{stability}}}}>R_{economy}>R_{driver}$$

(39)

The goal of the RL is to learn a policy that maximizes the expected cumulative reward over time. To encourage exploration and prevent premature convergence, the entropy term is included in the policy $\pi$:

$${\pi }^{*}=\arg \underset{\pi }{\max }\underset{\tau ~\pi }{\mathbb{E}}\left[{\gamma }_f(r_t+\xi \mathcal{H}(\pi (\cdot |s_{t+1})))\right]$$

(40)

where $\xi$ is a temperature parameter that determines the level of exploration, and the additional entropy term $\mathcal{H}(\cdot )$ encourages exploration and prevents premature convergence to suboptimal policies.

Heuristic REDQ is an improved algorithm of REDQ. REDQ is a deep RL algorithm that combines ideas from both ensemble methods and double Q-learning to improve stability and sample efficiency [29]. In the REDQ algorithm, the Q function is expressed using the Bellman equation:

$$Q^{\pi }(s_t,a_t)=\underset{a_{t+1}~\pi }{\mathbb{E}}[r_t+{\gamma }_f\underset{a_{t+1}}{\max }{Q}^{\pi }(s_{t+1},a_{t+1})+\xi \mathcal{H}(\pi (\cdot |s_{t+1}))]$$

(41)

where $Q^{\pi }(s_t,a_t)$ is the expected long-term reward of taking action $a_t$ based on policy $\pi$ in state $s_t$, $r_t=\mathcal{R}(s_t)$ is the immediate reward based on state $s_t$, ${\gamma }_f\in [0,1]$ is the discount factor, $s_{t+1}$ is the next state after taking action $a_t$ in state $s_t$, and $a_{t+1}$ is the next action.

To improve stability and reduce overestimation, the REDQ algorithm maintains an ensemble of N Q-networks, where each network is represented by a set of weights ${\theta }_i$, and the target value of Q-function is calculated by randomly selecting M Q-networks from N Q-networks:

$$y^{REDQ}=r_t+{\gamma }_f\left(\underset{m\in {\mathcal{I}}_M}{\min }{\tilde{Q}}_m(s_{t+1},a_{t+1},{\tilde{\theta }}_t)+\xi \mathcal{H}(\pi (\cdot |s_{t+1}))\right)$$

(42)

where the set ${\mathcal{I}}_M\subseteq \{1,2,\dots ,N\}$ is M random elements.

The updated rule of evaluating Q-networks’ weights is by minimizing the loss:

$${\mathcal{L}}_{\theta }({\theta }_i)=\underset{(s_t,a_t,r_t,s_{t+1})}{\mathbb{E}}\left[{\left(y^{REDQ}-Q_{{\theta }_i}(s_t,a_t)\right)}^2\right],i=1,2,\dots N$$

(43)

The target Q-networks are updated using a Polyak averaging:

$${\tilde{\theta }}_i\leftarrow \rho {\tilde{\theta }}_i+(1-\rho ){\theta }_i$$

(44)

where $\rho$ is a hyperparameter that determines the smoothing factor.

In addition, the parameter $\varphi$ of the policy ${\pi }_{\varphi }$ is trained by minimizing the loss:

$${\mathcal{L}}_{\varphi }(\varphi )=\underset{(s_t,a_t,r_t,s_{t+1})}{\mathbb{E}}\left[\frac{1}{N}{\displaystyle \sum _{i=1}^N{Q}_{{\theta }_i}(s_t,a_t)}+\xi \mathcal{H}(\pi (a_{\varphi }(s)|s_t))\right]$$

(45)

As the action here is the four-wheeled torque, its action space is four-dimensional, which adds difficulty to the training convergence of the agent. Therefore, a heuristic decay method is introduced here, with the final action being randomly chosen from the set of action strategies $\left\{a_{OP},a_{rule}\right\}$ with distribution $\left\{1-P_{rule},P_{rule}\right\}$. The probability of selecting action $a_{rule}$ is:

$$P_{rule}=\left(1-\tanh \left(\frac{n_r}{\tau }\right)\right)$$

(46)

where $\tanh (x)=\frac{e^{2x}-1}{e^{2x}+1}$, $n_r$ is the cumulative number of runs, and $\tau$ is a constant that denotes the decay speed. The $a_{rule}$ is obtained from Section 4.1, and the $a_{OP}$ is expressed as:

$$a_{OP}(s_t)=\tanh ({\mu }_{\varphi }(s_t)+{\sigma }_{\varphi }(s_t)\odot \phi )T_{\max }$$

(47)

where $\phi ~N(0,I)$ is an independent noise, ${\mu }_{\varphi }(s_t)$ is the mean of the stochastic policy that maps the state $s_t$ to an action, and ${\sigma }_{\varphi }(s_t)$ is a random noise sampled from a probability distribution that encourages exploration. $\odot$ is the Hadamard product, and $T_{\max }$ is the maximum motor torque, which is determined by the external characteristics of the motor.

To reduce overestimation and improve stability, REDQ uses an ensemble of Q-networks to estimate the Q-values and takes the minimum of the target Q-values from all the networks. To further improve sample efficiency, REDQ uses an update-to-data ratio denoted by G, which enables control over the number of times the data is reused. Algorithm 2 shows the detailed steps of the heuristic REDQ.

Algorithm 2. Heuristic REDQ algorithm

1. Initialize an ensemble of Q-networks $Q_{\theta }(s_t,a_t)$ with parameters ${\theta }_i$, Set target parameters

2. Initialize the target Q-networks ${\tilde{Q}}_{\tilde{\theta }}(s_t,a_t)$ with parameters ${\tilde{\theta }}_i\leftarrow {\theta }_i$

3. Initialize the replay buffer

4. For each step t do:

5.  Randomly sample an action $a_t$ from the set of action strategies $\left\{a_{OP},a_{rule}\right\}$ with distribution $\left\{1-P_{rule},P_{rule}\right\}$

6.  Execute the action $a_t$ and observe the next state $s_{t+1}$, reward $r_t$

7.  Store the experience tuple $\left\{s_t,a_t,r_t,s_{t+1}\right\}$ in the replay buffer

8.  for G updates do

9.   Sample a mini-batch experiences $\left\{\left(s_t,a_t,r_t,s_{t+1}\right)\right\}$ from replay buffer

10.   Randomly select m numbers from the set $\left\{1,2,\cdots ,N\right\}$ as a set ${\mathcal{I}}_M$

11.   Based on (42) compute the Q-value estimates $y^{REDQ}$

12.   for  $i=1,2,\cdots ,N$  do

13.    Based on (43), update the parameters ${\theta }_i$ using gradient descent method

14.    Based on (44), Update each target Q-network ${\tilde{Q}}_{\tilde{\theta }}\left(s_t,a_t\right)$

15.   end for

16.  end for

17. end for

18. Return the learned Q-network ensemble.

4. Evaluation Indicators and Simulation Results

4.1. Simulation Environment

To validate the effectiveness of the TVC algorithm, a joint simulation of CarSim, Simulink, and Python is used in this paper. The vehicle model in CarSim is based on the Magic Formula tire model. The vehicle model includes four in-wheel motors, which are controlled by the torque allocation algorithm implemented in Simulink. Python is used for training the RL algorithm. As the trained RL algorithm will not impose an additional computational burden on the TVC system, the main computational consumption of this paper lies in the nonlinear MPC. We have verified the real-time problem of the nonlinear MPC controller in the published paper [26], and the implemented nonlinear MPC algorithm by the ACADO toolkit [24], which has demonstrated its numerical efficiency for EV control in previous studies [5,30]. In this paper, the following five representative methods were selected for comparison and validation.

• RLES. The torque allocation algorithm proposed in this paper. The active safety control layer is a nonlinear MPC controller, and the lower controller is based on a heuristic REDQ deep RL algorithm which integrates considering economy and safety.
• MPC-CO. The torque allocation algorithm proposed in reference [26] which integrates considering economy and safety, where the lower controller is a quadratic planning algorithm.
• LQR-EQ. The active safety control layer is the LQR controller in reference [31], and the torque allocation layer is a common average allocation method in Section 3.2.1. This controller considers vehicle safety only.
• w/o control. There is no additional vehicle lateral control; steering is controlled by the driver.

4.2. Performance Indicators

To evaluate the effectiveness of the proposed torque allocation algorithm, several evaluation indicators are employed in this paper. These indicators include:

• Handling stability

$${\epsilon }_s=-{{\displaystyle \int }}_0^t\left(e_{\beta }^2+W_{\beta \_\gamma }{e}_{\gamma }^2\right)dt$$

(48)

where $e_{\beta }=\beta -{\beta }_r,e_{\gamma }=\gamma -{\gamma }_r$, and $W_{\beta \_\gamma }$ are the weights between $e_{\beta }^2$ and $e_{\gamma }^2$.

2.

Driver workload

$${\epsilon }_{driver}={{\displaystyle \int }}_0^t\left({\dot{\delta }}_{sw}{\left(\tau \right)}^2+W_{a\_\delta }\left(\tau \right)\cdot a_x{\left(\tau \right)}^2\right)d\tau$$

(49)

where $a_x$ is the longitudinal acceleration, ${\delta }_{sw}$ is the driver steering wheel angle, and $W_{a\_\delta }$ is the weight between ${{\dot{\delta }}_{sw}}^2$ and ${a_x^{}}^2$.

3.

Motor load

Frequent variations in motor torque may result in thermal saturation, which will increase the wear and tear on the motor.

$${\epsilon }_{motor}={{\displaystyle \int }}_0^t\left(\Delta T_{fl}{\left(\tau \right)}^2+\Delta T_{fr}{\left(\tau \right)}^2+\Delta T_{rl}{\left(\tau \right)}^2+\Delta T_{rr}{\left(\tau \right)}^2\right)d\tau$$

(50)

4.

Additional yaw moment

The magnitude of $M_z$ can represent the cost of stability control.

$${\epsilon }_{Mz}={{\displaystyle \int }}_0^t{M}_z{\left(\tau \right)}^{2}d\tau$$

(51)

5.

Velocity tracking

Vehicle velocity is important for safety assessment, and ensuring vehicle safety while maintaining speed tracking is the goal of stability control.

$${\epsilon }_v={{\displaystyle \int }}_0^t{\left(v_{ref}\left(\tau \right)-v_x\left(\tau \right)\right)}^{2}d\tau$$

(52)

4.3. Training Performance

To ensure the performance of the controller under extreme conditions. We trained the RL agent in the lower controller on the simulation platform described in Section 3.2.2. The training condition is a low adhesion road with an adhesion coefficient of 0.3, and the vehicle velocity was increased from 60 km/h to 100 km/h in 10 s. The RL sampling time is 0.02 s, and the training time per episode is 10 s. The total number of training episodes is 800. The training results are shown in Figure 5.

The training reward curve of RL shows how the agent’s cumulative reward changes over the course of the training process. We can see that after about 580 episodes of training, the total rewards of the agent begin to converge at −65. During the early stages of training, the reward curve is erratic, with the agent sometimes achieving high rewards and sometimes achieving low rewards. As the agent learns and the training progresses, the reward curve gradually increases and stabilizes, indicating that the agent is becoming more proficient at the task.

4.4. DLC Maneuver on Slippery Road

The effectiveness of the proposed algorithm is tested under DLC conditions on slippery road surfaces. The longitudinal initial velocity is 72 km/h, and the tire–road friction coefficient is set as 0.3.

The vehicle trajectory in Figure 6a shows that the proposed RLES torque allocation algorithm provides better stability and control during the double line change maneuver compared to the MPC CO and LQR EQ controllers. The vehicle is able to maintain a smooth trajectory and quickly change its direction without large deviation or instability. Additionally, the vehicle under w/o control deviated significantly from the desired path. The yaw rate and side slip angle in Figure 6b,c also show improvements with the proposed RLES algorithm. The yaw rate and side slip angle of the RLES and MPC CO controllers can track the reference value throughout the maneuver, indicating good vehicle handling and control. The yaw rate and side slip angle in the LQR EQ and w/o controller show more oscillations and instability, which may cause the vehicle to lose control. The phase trajectory portrait in Figure 7a also proves this point.

The motor power consumption in Figure 7b also shows improvement with the proposed RLES controller. The motor power consumption is reduced, indicating better fuel economy. The other control strategies show higher motor power consumption, which may result in higher energy consumption and lower fuel economy. Compared to w/o control, RLES, MPC CO, and LQR EQ reduce motor power consumption by 80.2%, 78.4%, and 31.4%, respectively.

Table 2. Performance indicators in DLC maneuver on slippery road.

Controller	𝜺s		𝜺driver		𝜺motor		𝜺Mz	𝜺v
RLES	0.4085	−96%	16.28	−81%	20,820	−93%	735,200	1.1 × 10−5
MPC CO	0.5640	−94%	19.58	−77%	23,430	−92%	887,200	1.5 × 10−5
LQR EQ	3.6532	−61%	32.54	−62%	29,680	−90%	892,000	3.5 × 10−4
w/o control	9.260	-	85.46	-	306,300	-	0	2.9 × 102

shows the results of the evaluation indicators for the four controllers. The RLES controller shows advantages for all indicators, especially for ${\epsilon }_{driver}$, ${\epsilon }_{motor}$, and ${\epsilon }_v$.

Overall, the simulation results demonstrate the effectiveness of the torque allocation algorithm in improving vehicle stability, control, and fuel economy during a double-line change maneuver. This is because the RL-based RLES controller can learn the optimal strategy in continuous interaction with the environment and thus output the optimal four-wheel torque according to the current vehicle state.

4.5. DLC Maneuver on Joint Road

To simulate the condition of suddenly encountering an icy road surface at the start of a lane change and analyze the controller performance under extreme conditions, we set the road surface adhesion coefficient as Figure 8. Additionally, the target vehicle speed was 72 km/h, the EV was run in a DLC condition, and the reference trajectory is shown in the reference term in Figure 9a.

The trajectory of a vehicle controlled by RLES has a smaller displacement offset than other controllers as shown in Figure 9a. The vehicle can still complete the DLC maneuver under the w/o controller because of the decrease in vehicle speed, as shown in Figure 10b where the minimum speed of the w/o controller is 51 km/h. The side slip angle of the vehicle controlled by RLES was also well-controlled and remained close to zero, while the LQR EQ and w/o control controllers showed larger deviations from zero, indicating poorer vehicle stability, as shown in Figure 9c. The phase trajectory portrait in Figure 10a shows the relationship between the side slip angle and side slip angle rate, which demonstrates that the proposed RLES torque allocation algorithm provides better stability and control. The phase trajectory of the RLES is within the stability range, while the phase trajectory of the LQR EQ and w/o control controller exceeded the stability boundary and cannot return to the stability point, which may cause the vehicle to lose control.

Figure 11 shows the stability index and power consumption of the four controllers. It can be seen that both the RLES and MPC CO controllers perform significantly better than the LQR EQ and w/o controller. RLES and MPC CO improve the vehicle economy while ensuring vehicle safety. The performance of RLES is better than the MPC CO controller in terms of stability and power consumption, which indicates that the RLES based on RL training is suitable for different working conditions and can adaptively adopt the optimal control strategy according to the current vehicle state.

Table 3. Performance indicators in DLC maneuver on joint road.

Controller	𝜺s		𝜺driver		𝜺motor		𝜺Mz	𝜺v
RLES	0.5574	−94%	16.28	−81%	20,820	−93%	735,200	0.88
MPC CO	0.6368	−93%	19.58	−77%	23,430	−92%	887,200	1.42
LQR EQ	4.2190	−57%	32.54	−62%	29,680	−90%	892,000	2.47
w/o control	9.707	-	85.46	-	306,300	-	0	36.78

summarizes the performance comparison of the four controllers. The simulation results demonstrate that the proposed RLES controller achieves superior performance in terms of power consumption, driver burden, and vehicle safety, making it a promising solution for the torque allocation of 4MIDEV.

4.6. Step Steering Maneuver

To verify the dynamic characteristics of the vehicle, a step steering maneuver was conducted for simulation validation. The vehicle was driving at a speed of 72 km/h on a good road surface with a road adhesion coefficient of 0.75. Within 0.5 s, the steering wheel angle increased from 0 deg to 120 deg and remained unchanged. The simulation results are shown in Figure 12 and Figure 13.

As shown in Figure 12a, the RLES control method shows relative understeering characteristics, which is beneficial for improving driver maneuverability. The yaw rate is shown in Figure 12b, indicating that both the RLES and MPC CO control methods could track the reference value well. Similarly, the side slip angle for the RLES and MPC CO control methods in Figure 12c is better than that of the LQR EQ and w/o control methods, and the RLES control method has the best maneuvering stability. Figure 13a shows the vehicle velocity variation curve in the step steering simulation, and the RLES method has better velocity tracking performance compared to other methods. The vehicle stability is improved under the RLES method while the longitudinal speed tracking performance is guaranteed. As seen in Figure 13b, except for the w/o control method, RLES consumed the least energy during the simulation. The w/o control method consumed the least energy because it does not require torque distribution to generate additional yaw moment by providing differential drive-assisted steering. However, this control method which sacrifices the vehicle stability is actually unsafe.

Table 4. Performance indicators in step steering maneuver.

Controller	𝜺s		𝜺driver		𝜺motor	𝜺Mz	𝜺v
RLES	0.0183	−62%	42.46	−1.7%	9930	1 × 107	0.01895
MPC CO	0.0193	−60%	42.46	−1.7%	9873	1 × 107	0.0273
LQR EQ	0.0274	−43%	43.21	0.1%	4166	170,100	0.04486
w/o control	0.0483	-	43.18	-	132.7	0	2.202

shows the objective evaluation indicators under different controllers. It can be seen from the table that, except for the w/o control method, the RLES controller outperformed the other controllers in terms of vehicle stability, driver burden, and longitudinal speed tracking performance. In conclusion, the controller based on RLES not only ensures vehicle safety but also reduces the energy consumption of the drive system and improves economy under open-loop simulation conditions.

4.7. Driving Cycles

To evaluate the effectiveness of the proposed TVC algorithm in terms of energy savings, a series of tests were conducted under two different driving cycles: New European Driving Cycle (NEDC) and US06. The NEDC case was designed to simulate urban and high-speed driving scenarios, while the US06 driving cycle was designed to replicate a more aggressive driving behavior on the highway. Figure 14a depicts the vehicle velocity allocation for each driving cycle. Note that the driving cycle conditions do not include lateral control, so the performance of the LQR EQ controller is the same as without the control.

As depicted in Figure 15, the RLES algorithm increases the motor efficiency by 10.9% and reduces the motor power consumption by 11.0% compared to the LQR EQ algorithm in the NEDC cycle. This reduction in power consumption is achieved by optimizing the torque allocation strategy, thus allowing the motor to operate in the high-efficiency range as much as possible. In addition, the RLES also achieves optimal performance in the US60 driving cycle in Figure 15. These results demonstrate the RLES algorithm can adapt to different driving conditions and optimize the torque allocation strategy to achieve better energy efficiency.

5. Conclusions

The 4MIDEV has the feature of independently controllable four-wheel motors. To fully utilize this feature, the paper introduces a novel RL-based TVC algorithm for 4MIDEV that takes into account both economy and safety. The four-wheel tire model is identified using FTO with experimental data, and the active safety control layer utilizes a nonlinear MPC to calculate the required additional yaw moment and longitudinal force. The torque allocation layer employs a heuristic REDQ deep RL algorithm to compute the optimal four-wheel torque. The proposed RLES controller is validated and compared with typical MPC CO, LQR EQ, and w/o controllers under different driving scenarios. The results demonstrate that the proposed RLES enhances vehicle economy and reduces driver workload while ensuring vehicle safety. Future work will focus on designing a TVC algorithm for different driver steering characteristics and conducting real vehicle experiments.