Eco-Driving Cruise Control for 4WIMD-EVs Based on Receding Horizon Reinforcement Learning

Abstract

Aiming to improve the distance per charge of four in-wheel independent motor-drive electric vehicles in intelligent transportation systems, a hierarchical energy management strategy that weighs their computational efficiency and optimization performance is proposed. According to the information of an intelligent transportation system, a method combining reinforcement learning with receding horizon optimization is proposed at the upper level, which solves the cruising velocity for eco-driving in a long predictive horizon based on the online construction of a velocity planning problem. At the lower level, a multi-objective optimal torque allocation method that considers energy saving and safety is proposed, where an analytical solution based on the state feedback control was obtained with the vehicle following the optimal speed of the upper level and tracking the centerline of the target path. The energy management strategy proposed in this study effectively reduces the complexity of the intelligent energy-saving control system of the vehicle and achieves a fast solution to the whole vehicle energy optimization problem, integrating macro-traffic information while considering both power and safety. Finally, an intelligent, connected hardware-in-the-loop (HIL) simulation platform is built to verify the method formulated in this study. The simulation results demonstrate that the proposed method reduces energy consumption by 12.98% compared with the conventional constant-speed cruising strategy. In addition, the computational time is significantly reduced.

1. Introduction

Electric vehicles (EVs) have gradually become a focus of the automotive industry due to their high-energy efficiency and low or even zero-tailpipe emissions. However, the limited mileage and low coverage level of charging piles lead to ‘range anxiety’ for EVs, which has become the main bottleneck of EV development [1]. New opportunities have arisen in the intelligentization of electric vehicles based on automation and informatization. Energy consumption can be reduced by at least 15–20% by adjusting the overall vehicle driving strategy and using intelligent transportation information; notably, V2V, V2X and high-precision map information can be considered [2,3]. If further powertrain adjustments can be made, energy consumption could decrease by 30% [4]. Consequently, intelligent connected electric vehicles with outside multisource information and traditional energy-saving technology have gradually become the development focus for new energy vehicles [5,6].

Eco-driving technology based on vehicle velocity planning can improve the energy efficiency of vehicles. In velocity planning in eco-driving, on the basis of traditional rule-based and minimum-principle algorithms [7,8,9], intelligent algorithms such as dynamic programming (DP) [10], model predictive control (MPC) [11,12,13,14,15,16], and reinforcement learning [17,18,19,20] have been widely used. B. Asadi et al. used traffic information, such as that provided by road signs and signal lights, and adopted a rule-based method to optimize the speed trajectory of vehicles on urban roads; this approach significantly improved vehicle economy and traffic efficiency [7]. References [8,9] described the technical advantages of “eco-driving” and calculated the optimal speed trajectory based on a maximum-principle algorithm. Prior driving experience is the main basis of these methods, which can be applied in engineering systems. However, these methods may be impractical because of the variability in transportation conditions. Based on a given route with multiple stop signs and traffic lights, Zheng et al. used a dynamic programming method for optimal speed planning during automatic driving. Compared with the energy use in a nominal speed profile, energy consumption may be reduced by up to 19% using this optimization approach [10]. However, the computation time and backward searching make the dynamic programming impossible to implement online, although the global optimal speed is obtained.

At present, model predictive control (MPC) has been widely used in velocity planning because of its advantages in the effective fusion of future information and solving of constrained control problems [11]. M.A.S. Kamal et al. combined traffic sign information and information from other vehicles to adjust the safety distance and optimum vehicle speed using MPC; not only was the safety factor improved, but the fuel consumption for a specific distance was reduced [12]. Reference [13] designed a multiobjective predictive control system that considers the safety, comfort and economy of fuel vehicles. The fuel consumption was reduced by approximately 13% using this approach, and the driving demand is met. However, in cruise control eco-driving, velocity planning based on macroscopic traffic information requires a long sampling time and predictive horizon. Because the calculation dimension increases exponentially with the predictive horizon, it is difficult for a control system to provide a real-time solution [14]. Guo et al. proposed a bi-level control method based on MPC and reduced the fuel consumption of HEVs using optimal speed trajectories and HEV torque distribution management [15]. Additionally, a hierarchical control strategy was developed based on speed optimization, and a specific driving task was divided into several operations to reduce the computational cost of prediction control [16]. However, the computational complexity of nonlinear MPC and the calculation capacity of vehicle modules still limit the ability to set the prediction horizon length; thus, global road information may not be fully considered in economic optimization. Therefore, the design of an optimization method with few calculations has been the focus of economical velocity planning. In research on intelligent algorithms, reinforcement learning displayed a superior performance when solving high-dimensional problems and can compensate for the weaknesses in the use of optimization methods for nonlinear systems and the lack of computational resources [17,18]. Therefore, this paper proposed a method that combines reinforcement learning and receding horizon optimization to construct an online velocity planning control system to solve the energy-saving velocity planning problem with a long prediction horizon.

On the premise of the above-mentioned optimization of vehicle velocity, the actuator of the vehicle itself also forms the main part of the energy consumption. An outstanding advantage of electric vehicles with independently driven (drive and regenerative braking) hub motors compared to conventional vehicle powertrain architectures is their higher control flexibility and the ensuing safety and energy-saving potential [19]. According to the literature [20], control allocation (CA) is an effective way to solve the overdrive system control of 4WD electric vehicles. The main control allocation methods are rule-based allocation method [21], allocation based on vertical load [22,23] and allocation considering multi-objective optimization [24,25,26,27,28,29]. Gao et al. [25] comprehensively considered tire slip energy consumption and motor energy loss to achieve a reasonable distribution of longitudinal forces and effectively reduce tire wear and slip energy. Reference [26] designed a multi-objective optimal allocation method including control error and tire utilization ratio to effectively improve the comprehensive performance of the vehicle. The related research works mainly focus on longitudinal and lateral control. However, it is difficult to adapt a single lateral or longitudinal control to the complex and rapidly changing macroscopic traffic environment.

An eco-driving cruise control strategy for electric vehicles was developed in this study to address the issue of economic velocity planning and torque distribution by incorporating macroscopic traffic information. This strategy is based on a thorough analysis of energy efficiency and computational efficiency to improve the vehicle’s energy utilization and the control strategy’s engineering applications. The main work of this study is as follows:

• To solve the high-dimensional computational issues brought on by the expansion of the prediction time domain in eco-driving and to quickly arrive at the energy-optimal cruising velocity, a method combining reinforcement learning and rolling optimization was proposed. This method makes the best use of the reinforcement learning technique’s ability to solve problems quickly and the advantages of the rolling optimization framework in dealing with constraints and disturbances.
• The multi-objective optimal torque allocation method was proposed on the basis of satisfying the driving demand, considering the account tire load rate and energy consumption control demand. This approach considers both energy savings and safety, and an analytical solution was obtained using the state feedback control law. In terms of computational efficiency, the solution’s shape was clearly advantageous.

The remainder of this study is organized as follows. The velocity planning problem is presented in Section 2, which includes the problem formulation and solutions using receding horizon reinforcement learning for online implementation. The lower-level control method is discussed in Section 3, which proposes a torque distribution method that balances energy saving and safety while achieving the upper-level velocity planning targets. In Section 4, the control method developed in this study is applied to a simulation platform, and the simulation results are shown. Finally, the conclusions are given in Section 5.

2. Upper-Level Velocity Planning for Eco-Driving

The velocity planning for eco-driving was based on macroscopic traffic information, spanning a wide range in terms of both time and space. However, for the actuator, optimization is only required within a small timespan, which is different from the case for velocity planning. Therefore, a hierarchical energy management control strategy was proposed to solve the vehicle energy optimization problem. The electric vehicle hierarchical energy management control strategy is shown in Figure 1.

In this study, the controlled object was an intelligent connected vehicle equipped with onboard sensors and controllers, vehicle-to-vehicle (V2V) and vehicle-to-infrastructure (V2I) communication. At the upper level, the vehicle was based on the V2I and V2V equipment in the vehicle-connected system to collect the signal phase and timing (SPaT), position, slope, and preceding vehicle information of the upcoming traffic light intersection. It also combines the received external environmental information with the vehicle’s kinematic information to plan the cruising velocity while satisfying the vehicle’s own safety constraints. The lower actuator is recommended for torque distribution considering energy saving and safety based on the planned cruising velocity, and the analytical expression is provided.

2.1. Vehicle Longitudinal Dynamic Model

At the upper level of the velocity planning model for eco-driving, only the longitudinal kinematics of the vehicle were considered, and the model is as follows:

$$\dot{d}\left(t\right)=v\left(t\right)$$

(1)

$$\dot{v}\left(t\right)=a=(F-F_R)/m$$

(2)

where a is the acceleration of the vehicle, d is the distance the vehicle travels, v is the vehicle speed, m is the vehicle mass, F is the motor torque, and $F_R$ is the sum of each resistance term, including aerodynamic drag resistance, rolling resistance and gradient resistance, as follows:

$$F_R=f_w+f_f+f_i=\frac{C_{D}A{v}^2}{21.15}+f_{r}mg+mgsin\theta$$

(3)

where $C_D$ is the drag coefficient, A is the frontal area of the vehicle, $f_r$ is the rolling resistance constant, g is gravitational acceleration, and $\theta$ is the roadway grade.

The research object in this study was a four in-wheel independent motor-driven electric vehicle. The in-wheel motor was the only source of vehicle force during the driving process. Otherwise, EVs have the ability to brake using both mechanical and regenerative systems, which increases the vehicle’s stopping power. The powertrain expression is:

$$F=\left\{\begin{array}{c}{\displaystyle \sum _{i=1}^4}{T}_{mi}/r_w;T_m\ge 0\hfill \\ F_b+{\displaystyle \sum _{i=1}^4}{T}_{mi}/r_w;T_m<0\hfill \end{array}\right.$$

(4)

where $T_{mi}$ is the driving torque of each motor and $i=1\dots 4$ for the front left wheel, front right wheel, rear left wheel, and rear right wheel, respectively. Additionally, $r_w$ is the effective tire rolling radius, and $F_b$ is the mechanical braking force. The power of the 4WIMD-EVs is calculated by

$$P=Fv{\eta }^e(F,v)$$

(5)

where ${\eta }^e$ is the motor efficiency. $e=1$ means that the motor torque is positive and consumes energy; $e=-1$ means that the motor torque is negative and energy recovery is performed. The vehicle energy consumption can be expressed as follows:

$$E={\int }_0^{T_f}Pdt$$

(6)

where $T_f$ is the final time.

The upper-level control system was mainly used to optimize vehicle velocity. The objectives were to determine the optimal vehicle velocity and reduce the frequency of acceleration and braking operations, such as at traffic lights, thereby minimizing the energy consumption of the vehicle. Traffic lights, speed limits and preceding vehicle information were obtained through onboard sensors. Formulas (1) and (2) can be discretized as follows:

$$d(k+1)=d\left(k\right)+v\left(k\right)\Delta T+\frac{1}{2}a\left(k\right)\Delta T^2$$

(7)

$$v(k+1)=v\left(k\right)+a\left(k\right)\Delta T$$

(8)

where k denotes the current time; $\Delta T$ is the sampling period. In this study, the acceleration in each sampling period is set at a constant. Eco-driving can be formulated as an optimal control problem. Let the state vector be $x={[x_1,x_2]}^T={[d,v]}^T$, and the input be $u=\left[a\right]$. For the dynamical system:

$$x(k+1)=x\left(k\right)+f\left(x\right(k),u(k\left)\right)$$

(9)

2.2. Optimization Function and Constraint of Eco-Driving

To achieve eco-driving velocity trajectory optimization, the following cost function was proposed.

$$minJ=\varphi \left[x(k+N)\right]+\sum _{k=0}^{N-1}\left(L\right(x\left(k\right),u\left(k\right)),\Delta T)$$

(10)

where ${\displaystyle \sum _{k=0}^{N-1}}L\left(x\right(k),u(k\left)\right). \Delta T$ is the energy consumption, where N is the prediction horizon, which is calculated from the total prediction time $T_L$ and the sampling period $\Delta T$. This is expressed as $N=T_L/\Delta T$. In this study, the prediction time $T_L$ was calculated from the time sequence of the upcoming traffic lights, and the predicted time length decreased with the running time. In Equation (10), L is formulated as:

$$L(x,u,k)={\beta }_{1}P(x,u,k)+{\beta }_2{∥u\left(k\right)-u(k-1)∥}^2$$

(11)

where ${\beta }_1$, ${\beta }_2$ are weight factors. $P(x,u,k)$ is calculated from Formula (5), and ${∥u\left(k\right)-u(k-1)∥}^2$ is the amplitude constraint of the system control input, which is used to prevent excessive acceleration and comfort reduction. $\varphi \left[x\right(k+N\left)\right]$ is the terminate system state, defined by

$$\varphi \left[x(k+N)\right]=a_1{(v\left(k_f\right)-v_f)}^2+a_2{(d\left(k_f\right)-d_f)}^2$$

(12)

where $d_f$ is the terminal distance, which is decided by the position of the forward traffic lights, and $v_f$ is the terminal speed. The terminal driving time $k_f$ is influenced by the time sequence of upcoming traffic lights, and the terminal time $k_f$ is fixed. $a_1$, $a_2$ are weight factors.

According to the traffic lights, the constraints of the optimal problem were given. There are three possible conditions as a vehicle approaches a traffic light: accelerating, cruising and decelerating. The initial speed of the ACC was $v_d$, while the terminal speed $v_f$ was decided by whether it passed the traffic lights.

Considering the scenario of a single traffic light without a preceding vehicle in low-density traffic, where a large amount of attention is paid to traffic lights, the traffic light sequence and their distance from the vehicle were transformed into speed constraints. If a vehicle cannot pass through an intersection at its current speed, it must accelerate or slow down at an appropriate time to optimize energy savings. The driving mode was determined by the distance from a traffic light $\Delta d$ and the time remaining during the green phase of a light $t_{green}$.

If the vehicle cannot pass through the intersection during the green phase while maintaining a maximum speed, it will decelerate in advance and continue to travel during the next green phase; therefore, the corresponding terminal speed constraint is $v_f=0$. The vehicle maintains a constant speed or accelerates through the intersection with the terminal speed constraint $v_f=v_d$. Here, $\epsilon$ is a parameter that represents the permitted range of vehicle speed $v\in [1-\epsilon ,1+\epsilon ]$, which can be obtained through learning algorithms and statistic analysis. The upper boundary of the vehicle speed is $v_{max}=min(v_d(1+\epsilon ),v_{lim})$, where $v_{lim}$ is the speed limitation.

In the second scenario, assuming there is a preceding vehicle traveling at speed $v_{leader}$, if the distance to the preceding vehicle meets the speed and acceleration requirements, the host vehicle will still pass or stop according to the remaining green phase of the upcoming traffic light. In contrast, the speed tolerance of the vehicle is affected by the preceding vehicle, and the host vehicle must abide by the intelligent drive model (IDM) rules [30]. The upper boundary of the vehicle speed is $v_{max}=min(leader,v_d(1+\epsilon ),v_{lim}).$ By comparing $v_{max}\ast t_{green}$ with $\Delta d$, a decision can be made regarding whether to pass the intersection, and the terminal speed can be acquired.

In summary, the constraints of the system are listed as follows:

$$\varphi \left[x(k+N)\right]=a_1{(v\left(k_f\right)-v_f)}^2+a_2{(d\left(k_f\right)-d_f)}^2$$

(13a)

$$v_{min}\left(k\right)\le v\left(k\right)\le v_{max}\left(k\right)$$

(13b)

$$a_{min}\left(k\right)\le a\left(k\right)\le a_{max}\left(k\right)$$

(13c)

$$v_0=v_d,d_0=0$$

(13d)

where $v_{min}\left(k\right)$ and $v_{max}\left(k\right)$ are speed limits acquired by traffic information; $a_{min}\left(k\right)$ and $a_{max}\left(k\right)$ are acceleration limits.

2.3. Velocity Planning Based on Receding Horizon Reinforcement Learning

For the long prediction horizon economical cruise velocity planning problem shown above, the traditional optimization solution struggles to ensure computational efficiency; therefore, a receding horizon reinforcement learning (RHRL) algorithm is proposed. The finite horizon starts at time ${\tau }_j$; the length of the finite horizon is N, which is the prediction horizon length and is determined by traffic light information. The structure of the finite horizon reinforcement learning is shown in Figure 2.

2.3.1. Action Network

The action network was adopted to approximate the control input $a^{*}\left(k\right)$. In this study, a fuzzy cerebellar model articulation controller (FCMAC) neural network was used as the approximator, providing a simple and fast approximator [30]. FCMAC requires fewer iterations and provides higher real-time accuracy than a BP network, making it suitable for the online learning problem in this study. The architecture of FCMAC includes input space, variable fuzzification, concept mapping, actual mapping and output space, as shown in Figure 3.

The FCMAC network uses the fuzzification of the state quantity $d,v$ as the network input layer. The fuzzy variables based on the Gaussian membership function are as follows:

$${\lambda }_m\left(x\right)=exp\frac{{∥x-c_m∥}^2}{{\sigma }_m}$$

(14)

where $c_i$ is the centerpoint of the mth membership function, and ${\sigma }_m$$(m=1,2\dots 5)$ are the variances. The middle layer of the FCMAC network includes two parts: concept mapping and actual mapping. Firstly, according to the fuzzification parameters, the space was divided into 10 storage units, which correspond to the fuzzy membership function vectors. Then, $5^2$ states were obtained by dividing the 10 fuzzy membership functions into fuzzy rules. Each state was mapped to the receptive field space C of Ac as a pointer associated with a storage unit, and the address corresponding to this state is found. Secondly, a division method used for false coding technology was adopted to map the elements C of the receptive field space to the units C in the physical memory space. This can be expressed as $ad\left(j\right)=(a_j+j)MOD+1,j=1,2,\dots c$, where j stands for the unit after concept mapping. $MOD$ represents the remainder function in Matlab. $a\left(j\right)$ represents the address of the concept mapping unit; $ad\left(j\right)$ represents the store address in the Ap. The output of FCMAC is expressed as:

$$\widehat{a}\left(k\right)=\sum _{m=1}^c{\widehat{W}}_{a,m}\left(k\right)a_m\left(k\right)$$

(15)

where $a_m\left(k\right)$ is the product of Gaussian membership function mapping and $W_a\left(k\right)$ is the weight set. The objective of the action network is to minimize the error between the estimated value $\widehat{a}\left(k\right)$ and the optimal value $a^{*}\left(k\right)$. The calculation of $a^{*}\left(k\right)$ is described below. The error function is expressed as:

$$e_a=\widehat{a}\left(k\right)-a^{*}\left(k\right),E_a=\frac{1}{2}{∥e_a∥}^2$$

(16)

Then, the weight matrix updating rules for the action network are given by

$$\begin{array}{cc}\Delta W_{a,m}\left(k\right)& =-l_a\left(k\right)\frac{\partial E_a\left(k\right)}{\partial W_{a,m}\left(k\right)}{\lambda }_m\left(k\right)\hfill \\ & =l_a\left(k\right)\frac{a^{*}\left(k\right)-\widehat{a}\left(k\right)}{C}\frac{\partial \widehat{a}\left(k\right)}{\partial W_{a,m}\left(k\right)}{\lambda }_m\left(k\right)\hfill \end{array}$$

(17a)

$${\widehat{W}}_{a,m}\left(k\right)={\widehat{W}}_a(k-1)+\Delta W_a\left(k\right)+\theta ({\widehat{W}}_a(k-1)-{\widehat{W}}_a(k-2))$$

(17b)

where $l_a$ is the learning rate of the action network, which generally decreases to a fixed value with increasing time. C denotes the number of conceptual mapping storage cells. $\theta \in (0,1]$ is the inertia coefficient of the action network.

2.3.2. Reward Function

The reward function serves as a training indicator to encourage or discourage actions. The target vehicle’s energy consumption on a road with traffic signal restrictions is shown in the study as a reinforcement learning environment, and the single-step reward function according to the eco-driving performance index of Equation (11) is expressed as follows:

$$r(x\left(k\right),a\left(k\right))={\beta }_{1}P(x,u,k)+{\beta }_2{∥a\left(k\right)-a(k-1)∥}^2$$

(18)

In this study, the reward function in a finite horizon consisted of the terminal cost and cumulative return:

$$J\left(x\left(k\right)\right)=\varphi [x(k+N)]+\sum _{j=k}^{k+N-1}{\gamma }^{j-k}r(x\left(j\right),a\left(j\right))$$

(19)

where $x(k+N)$ is the terminal state for the current prediction horizon. $\varphi \left[x\right(k+N\left)\right]$ is the terminal cost function, which is determined according to Formula (13). $\gamma$ is discount factor, and $0\le \gamma \le 1$.

2.3.3. Critic Network

Similar to the action network, the critic network adopted the FCMAC approximator. The output of the critic network based on FCMAC is expressed as:

$$\widehat{J}\left(k\right)=\sum _{n=1}^c{W}_{c,n}\left(k\right){\phi }_n\left(k\right)$$

(20)

where $W_c\left(k\right)$ is the weight set and $\phi \left(k\right)$ is the product of Gaussian membership function mapping. The estimated state $\widehat{x}(k+1)$ was calculated according to the model, and the estimated value $\widehat{a}\left(k\right)$ of the action network. $\widehat{J}(k+1)$ can be estimated by $\widehat{J}(k+1)$ through the critic network. According to $\widehat{J}(k+1)$ and the reward function of Formula (18), Formula (19) is rewritten as

$$J(x\left(k\right),a\left(k\right))=r(x\left(k\right),\widehat{a}\left(k\right))+\gamma \widehat{J}(x(k+1),a(k+1))$$

(21)

Based on the Bellman Optimality Equation, the optimal value function can be obtained as follows:

$$J^{*}\left(k\right)=\left\{\begin{array}{c}\underset{u}{min}\{r(x\left(k\right),a\left(k\right))+\gamma J^{*}(k+1)\};\hfill \\ k\in [\tau ,\tau +N-1]\hfill \\ \varphi \left[x\right(k+N\left)\right];k=k+N\hfill \end{array}\right.$$

(22)

The optimal control law can be described by:

$$u^{*}=arg\underset{u}{min}\widehat{J}(x\left(k\right),u\left(k\right))$$

(23)

The optimal control strategy is vehicle acceleration in this study; that is, $a^{*}\left(k\right)$.

2.3.4. Critic Network Error Function

The function of the critic network error function is to modify the critic weight. This function was designed as the sum of the error considering the terminal constraint and cumulative return. According to Formula (21), the cumulative return error can be expressed as:

$$e_{c\_J}=\widehat{J}\left(k\right)-[r(x\left(k\right),\widehat{a}\left(k\right))+\gamma \widehat{J}(k+1)]$$

(24)

The terminal constraint network based on the FCMAC is represented as:

$${\widehat{J}}_N\left(k\right)=\sum _{N=1}^c{W}_{c,N}\left(k\right){\kappa }_N\left(k\right)$$

(25)

where ${\kappa }_N$ is the product of Gaussian membership function mapping. ${\widehat{x}}_N$ is an estimated value of the terminal state, which is determined according to the initial state and the control law that is to be optimized. The terminal constraint error is expressed as

$$e_{c\_N}={\widehat{J}}_N-\varphi \left({\widehat{x}}_N\right)$$

(26)

The ideal terminal constraint $\varphi \left({\widehat{x}}_N\right)={\widehat{x}}_N^T{Q}_N{\widehat{x}}_N$ can be obtained from Formula (13). According to Formulas (24) and (26), the critic network error function is given by

$$e_c=e_{c\_N}+e_{c\_J};E_c=\frac{1}{2}{∥e_c∥}^2$$

(27)

The weight matrices update rules for the critic network are given by

$$\begin{array}{cc}\hfill \Delta W_{c,n}\left(k\right)& =-l_c\left(k\right)\frac{\partial E_c\left(k\right)}{\partial W_{c,n}\left(k\right)}{\psi }_n\left(k\right)\hfill \\ & =l_c\left(k\right)\frac{e_{c\_N}+e_{c\_J}}{C}\frac{\partial \widehat{J}\left(k\right)}{\partial W_{c,n}\left(k\right)}{\psi }_n\left(k\right)\hfill \end{array}$$

(28a)

$${\widehat{W}}_{c,n}\left(k\right)={\widehat{W}}_c(k-1)+\Delta W_c\left(k\right)+\theta ({\widehat{W}}_c(k-1)-{\widehat{W}}_c(k-2))$$

(28b)

where $l_c$ is the learning rate of the critic network. As also noted for the action network learning rate, this variable will gradually decrease to a fixed value. $\psi$ represents the Gaussian membership function of the critic network. $\theta \in (0,1]$ is the inertia coefficient of the critic network.

According to the above method, a control sequence of length N was obtained, and the first $T_c$ ($T_c\le N$) control actions were selected for implementation in the system. When $T_c$ was reached, the above methods were repeatedly used to solve the control problem. When $T_c$ arrived, the horizon advanced to ${\tau }_{j+1}$. The system updated the prediction horizon length N, distance to traffic lights and vehicle status according to the V2X information, and the above method was repeated in the system. The main procedure of the proposed RHRL is summarized in Algorithm 1.

The solution method developed in this study is an online optimization control method that must satisfy real-time requirements. $n_{max}$ was used to set the maximum number of iterations in each finite horizon. When the iteration process reaches $n_{max}$, the optimization process will be terminated, and the final optimized strategy will be applied to the system. To avoid unstable or bad strategies when the maximum number of iterations is reached, the initial weight value of each prediction horizon can be selected from the convex weight set in the previous prediction horizon. In this way, the learning time can be minimized, and control performance can be improved for real-time vehicle control.

Algorithm 1 RHRL algorithm.

The information on the upcoming road traffic light was obtained as well as the information on the vehicle. Then, the sampling time was set, and the length of the prediction horizon was estimated. The initialization of the vehicle state represents the initialization of parameters such as action network, critic network weights, learning rate, etc. The minimum error was set as $T_{actor}$ and $T_{critic}$. The maximum number of iterations was set as $n_c$ and $n_a$.   1: for  ${\tau }_j=T_c,2T_c,\cdots$ do   2:    for $k={\tau }_j,{\tau }_j+1\cdots {\tau }_j+N-1$ do   3:      Apply the action network (15) to estimate $\widehat{a}\left(k\right)$;   4:      The next state estimate $\widehat{x}(k+1)$ is obtained from $\widehat{a}\left(k\right)$ and the state values. Apply Formula (18) to calculation $r\left(k\right)$;   5:      Input $\widehat{x}\left(k\right)$ and $\widehat{x}(k+1)$ into the critic network (20), respectively, and obtain $\widehat{J}\left(k\right)$ and $\widehat{J}(k+1)$; generate the estimated terminal constraint ${\widehat{J}}_N$ via (25);   6:      Calculate the critic network error via (27);   7:      while $E_c>T_{critic}\&n_c<n_{max}$ do   8:          Update critic network weights via (28); Update $\widehat{J}\left(k\right)$, $\widehat{J}(k+1)$ and ${\widehat{J}}_N$; Calculate critic network error again; $n_c=n_c+1$;   9:      end while 10:      Calculate the approximate optimal solution $u^{\ast }\left(k\right)$ via (23); Calculate the actor network error via (17); 11:      while $E_a>T_{acotor}\&n_a<n_{max}$ do 12:         Update actor network error via (16); Update $\widehat{a}\left(k\right)$; Calculate actor network error again; $n_a=n_a+1$ 13:      end while 14:      Calculate the $\widehat{a}\left(k\right)$ on the weight. Apply it to the system. 15:    end for $j=j+1$; 16: end for

3. Lower-Level Multi-Objective Torque Control Allocation

The vehicle will deviate from the lane without intending to change lanes, as a result of the numerous unpredictable interferences encountered while maintaining the desired velocity (changes in the road, parameter perturbations, noise interference, etc.). In the lower controller, the vehicle must not only take fuel efficiency into account but also remain on the lane’s centerline and execute the necessary lateral stability control in unforeseen circumstances. Therefore, a multi-objective torque control allocation method that considers energy saving and safety was proposed in the lower-level control.

3.1. Vehicle 2-DoF Reference Model

The yaw rate and the slip angle are important parameters, used to describe the motion state of the vehicle. According to the 2-DOF model, the expected yaw rate is expressed as [31]:

$${\gamma }_{d1}=\frac{v_x}{(l_f+l_r)(1+K_s{v}_x^2)}{\delta }_f$$

(29)

where $K_s=m(l_r{C}_r-l_f{C}_f)/2{(l_f+l_r)}^2$, $l_f$ and $l_r$ represent the distance from the center of mass to the front axle and rear axle. $C_f$ and $C_r$ denote the cornering stiffness values of the front and rear tires, ${\delta }_f$ is the steering angle of the front wheels. Based on the lateral dynamics, the lateral acceleration is $a_y=\gamma v_x/0.85$. Meanwhile, the lateral acceleration is limited by the tire–road friction coefficient $\mu$, as follows: $a_y\le \mu g$. Then, the expected yaw rate can be expressed as:

$${\gamma }_d=min\{\mid {\gamma }_{d1}\mid ,0.85\mid \frac{\mu g}{v_x}\mid \}.$$

(30)

The vehicle can achieve lateral stability control when the vehicle’s sideslip $\beta$ angle varies at around zero. In this paper, the desired sideslip angle was chosen to be $\beta =0$ so that $\beta$ fluctuates within a small range to ensure vehicle stability under critical conditions.

3.2. Torque Control Strategy

A 7-DOF model was employed in accordance with the dynamics and safety requirements of this study, since it provides more precise and quantitative dynamic equations and is closer to the actual vehicle. The seven degrees of freedom are longitudinal motion, lateral motion, yaw motion and the rotation of four wheels. The vehicle dynamics equation is expressed as:

$$m(\dot{v_x}-\gamma v_y)=\sum _{i=1}^2{F}_{xi}cos{\delta }_f+\sum _{i=3}^4{F}_{xi}-\sum _{i=1}^2{F}_{yi}sin{\delta }_f-F_R$$

(31a)

$$m(\dot{v_y}+\gamma v_x)=\sum _{i=1}^2(F_{yi}cos{\delta }_f+F_{xi}sin{\delta }_f)+\sum _{i=3}^4{F}_{yi}$$

(31b)

$$\begin{array}{cc}\hfill I_z\dot{\gamma }=& l_f\sum _{i=1}^2(F_{yi}cos{\delta }_f+F_{xi}sin{\delta }_f)-l_r\sum _{i=3}^4{F}_{yi}\hfill \\ \hfill & +l_s[(F_{x4}-F_{x3})+(F_{x2}-F_{x1})cos{\delta }_f]\hfill \\ \hfill & +l_s\left[(F_{y1}-F_{y2})sin{\delta }_f\right]\hfill \end{array}$$

(31c)

where $l_s$ represents the front and rear track widths. It is assumed that ${\delta }_f$ is relatively small, such that $sin{\delta }_f=0$, $cos{\delta }_f=1$. The lateral forces on the front and rear wheels are expressed as:

$$\left\{\begin{array}{c}{F}_{y1}=F_{y2}=C_f[{\delta }_f-({\displaystyle \frac{v_y+l_f\gamma }{v_x}})]\hfill \\ F_{y3}=F_{y4}=C_r({\displaystyle \frac{v_y-l_r\gamma }{v_x}})\hfill \end{array}\right.$$

(32)

The optimal distribution of driving/braking torque to each wheel ensures stable vehicle tracking while consuming less energy. The states of this control system were chosen as longitudinal speed, lateral speed and yaw rate of the vehicle, that is, $X={[v_x,v_y,\gamma ]}^T$; the system control variables were selected for the four-wheel torque, that is $U={[T_1,T_2,T_3,T_4]}^T$; the system output was chosen as $Y=[v_x,\beta ,\gamma ]$. Substituting Equation (32) into Equation (31) yields

$$\dot{X}=f+GU$$

(33a)

$$Y=CX$$

(33b)

$$f=\left(\begin{array}{c}\gamma v_y-{\displaystyle \frac{F_R}{m}}\\ -{\displaystyle \frac{2(C_f+C_r)}{m{v}_x}}{v}_y+[{\displaystyle \frac{2(-C_f{l}_f+C_r{l}_r)}{m{v}_x}}-v_x]\gamma +{\displaystyle \frac{2C_f{\delta }_f}{m}}\\ -{\displaystyle \frac{2(C_r{l}_r-C_f{l}_f)}{I_z{v}_x}}{v}_y-{\displaystyle \frac{2(C_f{l}_f^2+C_r{l}_r^2)}{I_z{v}_x}}\gamma +{\displaystyle \frac{2l_f{C}_f{\delta }_f}{I_z}}\end{array}\right)$$

$$G=\left(\begin{array}{cccc}{\displaystyle \frac{1}{m{r}_w}}& {\displaystyle \frac{1}{m{r}_w}}& {\displaystyle \frac{1}{m{r}_w}}& {\displaystyle \frac{1}{m{r}_w}}\\ 0& 0& 0& 0\\ -{\displaystyle \frac{l_s}{I_z{r}_w}}& {\displaystyle \frac{l_s}{I_z{r}_w}}& -{\displaystyle \frac{l_s}{I_z{r}_w}}& {\displaystyle \frac{l_s}{I_z{r}_w}}\end{array}\right)$$

$$C=\left(\begin{array}{ccc}1& 0& 0\\ 0& 1/v_x& 0\\ 0& 0& 1\end{array}\right)$$

The dynamic model discussed above is a nonlinear system since the parameter f contains nonlinear elements. Nonlinear systems are challenging to solve and subpar real-time performance. Therefore, the Taylor series was used for approximate linearization in this study and, after discarding the high order, the following equation could be retained:

$$\dot{X}=f(X_r,U_r)+[{\displaystyle \frac{\partial f}{\partial X}}{|}_{X_r,U_r}](X-X_r)+[{\displaystyle \frac{\partial f}{\partial U}}{|}_{X_r,U_r}](U-U_r)$$

(34)

where $A_r=[{\displaystyle \frac{\partial f}{\partial X}}{|}_{X_r,U_r}]$ and $B_r=[{\displaystyle \frac{\partial f}{\partial U}}{|}_{X_r,U_r}]$ are the Jacobian matrix. $f(X_r,U_r)$ represents the expansion point on the reference trajectory. We can obtain the linearised model as follows:

$$\dot{\tilde{X}}=A_r\tilde{X}+B_r\tilde{U}$$

(35)

where $\tilde{X}=X-X_r,\tilde{U}=U-U_r$. Discrete this linearized model; then, the equations can be written as follows:

$$\tilde{X}(k+1)=A\tilde{X}\left(k\right)+B\tilde{U}\left(k\right)$$

(36)

where $A=I+A_r\Delta t$, $B=B_r\Delta t$, I is the unitary matrix. $\Delta t$ is the sampling time of the lower level. For the lower-level actuator, only short-term optimization needs to be completed; that is, the lower-level control only needs to solve the control solution at the next moment. Moreover, the sample time $\Delta t$ is also shorter than the upper-level control.

In this study, the state feedback method was used to solve the above problems. The state feedback control law is designed as follows:

$$U=-KX$$

(37)

The state feedback controller is designed to improve vehicle safety and economy, so the optimization objective function of this study was mainly composed of three parts, which are expressed as follows:

$$minJ={\int }_t^{t+\Delta t}({\alpha }_1{J}_1+{\alpha }_2{J}_2+{\alpha }_3{J}_3)$$

(38)

where ${\alpha }_i(i=1,2,3)$ are weight factors that can take on values from 0 to 1. The optimized objective function $J_1$ aimed to follow the velocity of the upper-level controller, yaw rate and slip angle to ensure stability.

$$J_1={\displaystyle \frac{1}{2}}{(Y_d-Y)}^T{W}_1(Y_d-Y)$$

(39)

where $Y_d=[v_{x_d},{\beta }_d,{\gamma }_d]$ can be obtained from Section 3.1. The symmetric matrix $W_1$ can be adjusted according to the driving conditions.

The optimized objective function $J_2$ is the minimized tire utilization ratio, which reduces the sudden saturation of individual tire force and effectively improves the vehicle stability margin. This can be expressed as follows:

$$J_2=\sum _{i=1}^4{\displaystyle \frac{F_{xi}^2+F_{yi}^2}{\mu F_{zi}^2}}$$

(40)

Since tire lateral force has little influence on the system and is uncontrollable, the control allocation presented here only considers the tire longitudinal force. Equation (39) can be simplified as:

$$\begin{array}{c}\hfill J_2=U^T{W}_{2}U\\ \hfill W_2=diag({\displaystyle \frac{1}{(\mu r_{w}F(zi){)}^2}})\end{array}$$

(41)

The optimized objective function $J_3$ represents the minimization of energy consumption. For 4WIMD-EVs, the motor is the only power source. The influence of motor efficiency can be used to maximize the motor working in the high-efficiency region to achieve energy savings. The in-wheel motor power expression of the 4WIMD-EVs is shown as follows:

$$P_i={\displaystyle \frac{T_i\omega }{r_w}}{\eta }^e(v,T_i)$$

(42)

where $\omega$ is the motor speed. The change curve of motor efficiency obtained with the motor torque between 700 and 700 rpm is shown in Figure 4.

$$J_3=\sum _{i=1}^4{P}_i=EU$$

(43)

Substituting Equations (39), (41) and (43) into the objective function (38) can obtain the objective function

$$J={\alpha }_1[{\displaystyle \frac{1}{2}}{(Y_d-CX)}^T{W}_1(Y_d-CX)]+{\alpha }_2\left(U^T{W}_{2}U\right)+{\alpha }_{3}EU$$

(44)

The state Equation (36) and the objective function (44) were used to establish the Hamiltonian function, and the expression is as follows:

$$\begin{array}{cc}\hfill H=& {\alpha }_1[{\displaystyle \frac{1}{2}}{(Y_d-CX)}^T{W}_1(Y_d-CX)]\hfill \\ \hfill & +{\alpha }_2\left(U^T{W}_{2}U\right)\hfill \\ \hfill & +{\alpha }_{3}EU+{\lambda }^T(AX+BU)\hfill \end{array}$$

(45)

where ${\lambda }^T$ is a Lagrange multiplier vector. According to the Hamilton function, the control equation and covariance equation can be derived as

$${\displaystyle \frac{\partial H}{\partial U}}=B^T\lambda +2{\alpha }_2{W}_{2}U+{\alpha }_{3}E=0$$

(46a)

$$-{\displaystyle \frac{\partial H}{\partial x}}=\dot{\lambda }=-A^T\lambda -\left({\alpha }_1{C}^T{W}_{1}C\right)x+{\displaystyle \frac{1}{2}}{\alpha }_1(Y_d^T{W}_{1}C+C^T{W}_1{Y}_d)$$

(46b)

Using Equation (46a), the formulation of the control vector U can be derived as follows:

$$U=-{\displaystyle \frac{1}{2{\alpha }_2}}{W}_2^{-1}(B^T\lambda +{\alpha }_{3}E)$$

(47)

To facilitate the calculation of state feedback, the Lagrangian vector was defined according to optimal control theory as $\lambda =PX+D$, where P and D are coefficient matrices. $\lambda$ derivative was expressed as $\dot{\lambda }=\dot{P}x+P\dot{x}$. Including $\lambda$ and $\dot{\lambda }$ in the covariance Equation (46b). P was taken to be zero in a steady state and the covariance equation is collapsed to match the Riccati Equations (48) and (49), meaning that P was the result of Riccati Equation (48). The Riccati equation is represented as

$$A^{T}P+PA-PB{(W_2/\left(2{\alpha }_2\right))}^{-1}{B}^{T}P+\left({\alpha }_1{C}^T{W}_{1}C\right)=0$$

(48)

$$\begin{array}{c}\hfill [A^T-PB(W_2/{\left(2{\alpha }_2\right))}^{-1}{B}^T]D+PB[{(W_2/\left(2{\alpha }_2\right))}^{-1}{\alpha }_{3}E]\\ \hfill -{\displaystyle \frac{1}{2}}{\alpha }_1(Y_d^T{W}_{1}C+C^T{W}_1{Y}_d)=0\end{array}$$

(49)

According to Equation (49), the expression for D is:

$$\begin{array}{c}\hfill D=[A^T-PB{(W_2/{\left(2{\alpha }_2\right))}^{-1}{B}^T]}^{-1}[{\displaystyle \frac{1}{2}}{\alpha }_1(Y_d^T{W}_{1}C+C^T{W}_1{Y}_d)\\ \hfill -PB[{(W_2/\left(2{\alpha }_2\right))}^{-1}{\alpha }_{3}E]]\end{array}$$

(50)

According to Equation (48), P can be obtained. The Lagrange multiplier vector $\lambda$ can be calculated using P and D, then added to Equation (47), and the feedback control law U is expressed as:

$$U=-{\displaystyle \frac{1}{2{\alpha }_2}}{W}_2^{-1}[B^T(PX+D)+{\alpha }_{3}E]$$

(51)

Through the above equation, the real-time display solution of the motor torque can be obtained. The vehicle reduces energy consumption while ensuring safety. Meanwhile, if one motor fails, the control algorithm can be reassigned by changing the G matrix in Equation (33) to achieve system reconstruction and effectively improve the vehicle’s stability in emergency situations.

4. Simulation and Analysis

The experimental platform included a driving simulator, a dSPACE for running control algorithms, a target machine for running a real-time vehicle model, a host for running a Carsim vehicle and control desk, and a SCANeR traffic scene host. The driver manipulated the steering wheel in the driving simulator to produce a certain steering wheel angle. The velocity planning module calculated the required torque according to the real-time traffic information and vehicle status transmitted via Ethernet. The steering wheel angle and torque signal were sent to the target machine at the same time, and the new position and state change of the vehicle was obtained. The result was sent to the traffic scene host and speed planning module via Ethernet, and the velocity planning module calculated the control torque according to the updated information. Thus, a closed loop was formed. The structure of the experimental platform is shown in Figure 5. The vehicle simulation parameters are shown in

Table 1. Parameters.

Parameter	Value	Parameter	Value	Parameter	Value
m	1800 kg	$l_f$	1.345 m	$l_s$	1.475 m
$r_w$	0.32 m	$l_r$	1.358 m	$C_f$	89,436 N/rad
A	2.08 m${}^2$	$I_z$	2712.4 kgm${}^2$	$C_r$	71,003 N/rad
$C_D$	0.36	$\mu$	0.75	$f_r$	0.011

.

The initial speed of the adaptive cruise system was 60 km/h. The computer configuration used for the simulation was an Intel(R) Core(TM) i7-6700 CPU.

In the simulation, the maximum number of iterations in each RHRL time domain was $n_{max}$ = 50. The learning rates of the action network and the critic network were gradually reduced sequences $l_c\left(0\right)$⋯$l_c\left(f\right)$ = 0.3, 0.25, 0.2⋯0.05. The iteration errors of the critic network and the actor network are shown in Figure 6a,b, and the cost function curve is shown in Figure 6c.

In SCANeR, the simulation scenario was based on information about a certain section of the urban road and the time series of traffic lights. The experimental section was approximately 3.6 km, and there were five intersections. The control algorithm used in this study (RHRL), rule-based ACC algorithm (ACC) and predictive control algorithm (MPC) were tested using hardware-in-the-loop (HIL). The MPC algorithm adopted the interior point method to solve the problem in each prediction horizon. At the upper level, the sampling period was $\Delta T=1$ s.

Figure 7 and Figure 8 compare the strategies of the three algorithms regarding the distance domain and time domain, respectively. Compared with the ACC algorithm, the RHRL and MPC were based on the approaching intersection information, and the acceleration or deceleration of a vehicle was determined in advance. For example, MPC and RHRL accelerated the vehicle through the intersections based on advanced calculations according to the time remaining in a green phase at the 150–600 m section and 2100–2950 m section. However, ACC braked and stopped the vehicle at the traffic light. IN the MPC and RHRL control algorithms based on forward road information, the RHRL acceleration was relatively slow, which is more conducive to energy saving. At 1200 m, the vehicle using the three algorithms could not pass the intersection based on the time remaining at the traffic lights. MPC and RHRL adopted the method of idling at a permissible speed to reduce energy consumption. Figure 8 shows that the magnitude of ACC initiating mechanical braking was significantly higher than that of the MPC and RHRL algorithms at 45 s. For EVs, mechanical braking cannot regenerate energy. This means that the more mechanical brakes are activated, the more energy is consumed.

Compared with the rules-based ACC approach, the proposed algorithm and MPC accelerate and decelerate slowly in advance, considering the traffic information. Figure 9 shows the traffic light sequence and speed trajectory at multiple intersections. Meanwhile, the RHRL and MPC save overall driving time while reducing energy. In the acceleration stage, the acceleration of RHRL is relatively slow, which is more conducive to energy saving.

For computational efficiency, a receding horizon framework was adopted in this paper. In each prediction horizon, the complexity of the NMPC optimized by IPM is $O\left(n^3{N}^2\right)$. The complexity of the method proposed in this paper is $O\left(n^{2}N\right)$. Meanwhile, during the RHRL optimization process, the maximum number of iterations is set. When the total number of iterations is reached, the current strategy optimization is stopped and output is performed. This can be used to obtain suboptimal policies that balance computational efficiency and optimization effect. The computational time used in different prediction horizons is shown in Figure 10. The MPC algorithm increases the computational time with the increase in prediction horizon, which means that it is limited by the chip’s ability during engineering applications. However, the advantages of the RHRL algorithm in terms of computational performance become increasingly significant as the prediction horizon increases. As shown in Figure 10, when the prediction horizon length is 30, the calculation time of the MPC algorithm is about five times that of the RHRL algorithm. Therefore, the RHRL algorithm is more beneficial for engineering applications.

According to the velocity tracking and lane-keeping requirements of the upper level, the multi-objective optimization results for the lower level are shown in Figure 11. For the lower-level executive motor, only short-term optimization needs to be completed according to the optimization velocity of the upper level. Then, the lower-level sampling period of the lower level selects $\Delta t=0.05$. The combination of vehicle longitudinal acceleration and lateral lane change was selected for simulation validation of the algorithm, as shown in Figure 11.

Without the control method, torque equalization occurs. The speed tracking curve shows that the error in the strategy without control significantly increases, and the upper-level demand eco-driving velocity cannot be tracked at 220–230 s. However, the multi-objective optimal allocation has a good tracking effect. From the lateral displacement and yaw rate tracking simulation diagrams, it can be seen that the strategy proposed in this study has a better control effect under acceleration and lane-changing conditions.

The torque allocation of the four motors is shown in Figure 12. The multi-objective optimal allocation method was used to adjust the motors to work in the high-efficiency area, effectively reducing the motor energy consumption. The two front motors always exert larger driving torques than the two rear motors, reducing the overall power consumption. Instantaneous power consumption is shown in Figure 13. The multi-objective optimal allocation method consumes less energy than the strategy without control.

Combined with the optimization algorithm of the upper and lower levels, the total energy consumption results for the three methods are shown in Figure 14, based on a simulated road of approximately 3.6 km in length with five traffic lights. The proposed approach saves approximately 12.98% energy consumption compared to the rule-based ACC algorithm.

The simulation results demonstrate that the upper-level control strategy accelerates and idles immediately to prevent braking and restarting during the red light phase. Furthermore, it actually reduces energy consumption when combined with the lower-level multi-objective torque control allocation. Meanwhile, the strategy suggested in this study reduces traffic delays and increases traffic productivity.

5. Conclusions

In this study, based on an intelligent transportation system, an eco-driving cruise control strategy with a hierarchical structure for electric vehicles is proposed to rapidly optimize the velocity trajectories and torque control allocation display solution. The method minimizes energy consumption in an intelligent transportation system while ensuring safety.

(1)

The proposed RHRL method solves the computational time problem associated with long predictive horizons and high dimensionality in velocity planning and realizes the rapid solution of the energy-optimized cruising speed, thus improving the potential application of the controller in engineering practice.

(2)

A hierarchical control structure is used to optimize the energy at two levels, effectively reducing the complexity of the energy-efficient control of intelligent connected vehicles. The actuator comprehensively considers longitudinal velocity following and centerline lane-keeping, and a multi-objective optimal torque allocation method that considers energy saving and safety is proposed.

(3)

The control strategy is verified by building an intelligent connected HIL experimental platform for vehicles. Compared with ACC, the proposed approach reduces energy consumption by approximately 12.98%. In addition to the improved energy efficiency, the frequent accelerating and braking of vehicles is avoided, resulting in operating time and traffic efficiency benefits.