Adaptive Energy Management Strategy for Hybrid Electric Vehicles in Dynamic Environments Based on Reinforcement Learning

Abstract

Energy management strategies typically employ reinforcement learning algorithms in a static state. However, during vehicle operation, the environment is dynamic and laden with uncertainties and unforeseen disruptions. This study proposes an adaptive learning strategy in dynamic environments that adapts actions to changing circumstances, drawing on past experience to enhance future real-world learning. We developed a memory library for dynamic environments, employed Dirichlet clustering for driving conditions, and incorporated the expectation maximization algorithm for timely model updating to fully absorb prior knowledge. The agent swiftly adapts to the dynamic environment and converges quickly, improving hybrid electric vehicle fuel economy by 5–10% while maintaining the final state of charge (SOC). Our algorithm’s engine operating point fluctuates less, and the working state is compact compared with Deep Q-Network (DQN) and Deterministic Policy Gradient (DDPG) algorithms. This study provides a solution for vehicle agents in dynamic environmental conditions, enabling them to logically evaluate past experiences and carry out situationally appropriate actions.

1. Introduction

The rapid increase in automobiles, paralleling economic and industrial growth, has heightened environmental pollution concerns [1,2,3]. The hybrid electric vehicle (HEV) is a vital step toward transportation electrification, triggering a shift in the automotive industry from oil dependence to new energy reliance. HEVs primarily aim to enhance power system efficiency and reduce fuel consumption. The energy management strategy (EMS) is one of the pivotal technologies used in HEVs, which can influence fuel consumption significantly, given a specific power system configuration. EMS is classified into rule-based, optimization-based, and learning-based strategies [4,5,6,7].

Rule-based EMS requires formulating control rules for energy distribution under varying driving modes [8,9,10,11,12]. The rule system, although simple to develop, can construct the control system. To address battery limitations, Ding et al. [13] suggested a hybrid strategy for series-parallel plug-in hybrid electric vehicles using rule-based control and genetic algorithm-optimized tactics. Guercioni et al. [14] introduced a distinctive real-time energy management method inspired by optimal solutions for the energy management issue of plug-in hybrid electric vehicles. Mansour et al. [15] created a rule-based energy management strategy, emphasizing real-time calculation for a Prius HEV, and a comprehensive optimization program underpinned by dynamic programming. This optimization process, interconnected with the traffic management system, takes into account the driver’s preferred route via the vehicle’s global positioning system. Notably, the parameters modified by the global optimization algorithm surpass the deterministic rules, proving superior to the rule-based energy management approach. This strategy, however, necessitates substantial processing resources and is typically suited to vehicles on predefined routes, becoming irrelevant when the routes or driving cycles are not pre-known. To remedy these issues, Chen et al. [16] recently introduced a driving pattern recognition method for scrutinizing the vehicle energy management strategy. Lian et al. [17] proposed an enhanced energy management framework by integrating expert knowledge into DDPG. On this premise, Zhou et al. [18] extensively explored and evaluated current driving pattern recognition methods, spotlighting appropriate application scenarios for each predictive algorithm and prospective strategies to handle prediction errors.

Learning-based strategies have gained considerable scholarly attention in recent years. Xu et al. [19] parameterized the key factors in the EMS development process based on reinforcement learning and applied it to the HEV power allocation problem to reduce vehicle fuel consumption. To balance the charge state of multiple-cell batteries in electric vehicles, Chaoui et al. [20] suggested a reinforcement learning-based solution. This extends battery life and reduces the frequency of battery maintenance. Aljohani et al. [21] developed a real-time data-driven routing optimization method for electric vehicles to minimize energy consumption. Zhou et al. [22] utilized reinforcement learning to optimize a hybrid tracked vehicle’s control strategy, with simulation results indicating that the suggested EMS substantially improved fuel efficiency. Xiong et al. [23] benchmarked reinforcement learning against rule-based algorithms. Simulations revealed that the reinforcement learning-based strategy effectively minimized energy loss. Optimizing energy management systems with reinforcement learning faces the challenge of slow convergence due to sparse rewards. To accelerate learning and enhance adaptability, Zhu et al. [24] suggested a hierarchical reinforcement learning approach tailored for energy management strategies. This novel reinforcement learning approach addresses the sparse rewards issue during training and ensures optimal weight distribution.

Moreover, various academics have demonstrated the superior performance of reinforcement learning-based energy management strategies in static environments. Lee et al. [25] employed dynamic programming control methods to compare reinforcement learning strategies. His findings suggest a reinforcement learning policy can attain global optimality in an infinite-horizon optimal control problem—a feat also achievable by stochastic dynamic programming. Aljohani et al. [26] proposed a metadata-driven approach for real-time EV route optimization to reduce road energy demand. Sun et al. [27] proposed the comprehensive energy management method founded on the conventional soft actor critic (SAC) algorithm and prioritized experience to boost training efficiency. Aiming at the problem of computational efficiency and dynamic coordination of energy conversion, Guo et al. [28] proposed a predictive energy management strategy for dual-mode HEV relying on model predictive control (MPC), which exhibits benefit performance and robustness. In complex, static contexts, these notable reinforcement learning approaches have proven their value in energy preservation.

Driving environments are fraught with instability sources, including behavioral biases from wind direction, motion condition changes due to weather, and vehicle part aging, causing shifts in certain physical characteristics [29,30,31,32]. The reinforcement learning agent’s strategy must adapt to environmental dynamics, swiftly adjusting to events diverging from training or changing conditions. Incremental reinforcement learning has recently become the preferred approach to rapid adaptation to dynamic scenarios. As per this concept, the dynamic environment is considered to be a sequence of stationary tasks within a specific timeframe, with each one corresponding to an environmental aspect during the relevant stage. This perspective offers insights for EMS in dynamic settings.

This study proposes an adaptive learning strategy in dynamic environments, allowing the vehicle’s reinforcement learning agent to more logically recall all events and cope with the dynamic environment.

The primary contributions can be outlined as follows:

• The memory library (ML) comprising specific actions for different driving condition scenarios has been developed for the reinforcement learning agent to leverage.
• The identification parameters for driving condition blocks and the control parameters in memory library are derived through Dirichlet clustering. During the online process, the control parameters are adjusted by the expectation maximization (EM) algorithm, enabling the agent to continuously build and refine the memory library while retaining previously acquired knowledge.
• The proposed adaptive learning strategy in dynamic environment (ALDE) algorithm equips the agent with the ability to adapt to changing environments. As the algorithm operates, the agent learns and enhances its capacity for managing a range of normal and extreme driving conditions, constantly.

The remainder of this paper is organized as follows: Section 2 presents the HEV model. Section 3 elaborates on the structure and updates of the ALDE algorithm. Section 4 outlines the verification methodology and results. Section 5 concludes this study.

2. Construction of Vehicle Model

The research focuses on a parallel HEV with a P2 configuration. The P2 HEV model offered by the Matlab/Simulink 2020b community is utilized and modified in the control component to enhance simulation precision and repeatability.

Table 1. Other parameters of the vehicle.

Symbol	Parameter	Values
Engine	Maximum power	92 kW
	Maximum torque	175 Nm
	Maximum speed	6500 rpm
Traction motor	Maximum power	30 kW
	Maximum torque	200 Nm
	Maximum speed	6000 rpm
Battery	Capacity	5.3 Ah
	Voltage	266.5 V

contains detailed information about the model parameters.

2.1. Vehicle Dynamic Modeling

The power requirement of the vehicle is calculated through the longitudinal force balance equation in Equation (1). The equation is divided into four parts: rolling resistance $F_{\mathrm{f}}$, air resistance $F_{\mathrm{w}}$, ramp resistance $F_{\mathrm{i}}$, and inertial force $F_{\mathrm{a}}$.

$$\left\{\begin{array}{l}F=F_{\mathrm{f}}+F_{\mathrm{w}}+F_{\mathrm{i}}+F_{\mathrm{a}}\\ F_{\mathrm{f}}=G\cdot f\\ F_{\mathrm{w}}={\displaystyle \frac{1}{2}}\rho \cdot A_f\cdot C_{\mathrm{D}}\cdot v^2\\ F_{\mathrm{i}}=G\cdot i\\ F_{\mathrm{a}}=\delta \cdot m\cdot a\end{array}\right.,$$

(1)

where $G$ represents vehicle gravity, $f$ denotes the rolling resistance coefficient, $\rho$ depicts air density, $A_f$ is the front projection area of the vehicle, $C_{\mathrm{D}}$ represents the air resistance coefficient, $v$ represents the longitudinal vehicle speed, $i$ represents the road slope (which is 0 in this case, so the term $F_{\mathrm{i}}$ can be ignored), $\mathsf{\delta }$ is the rotational mass conversion coefficient, $m$ represents vehicle mass, and $a$ represents acceleration.

The engine and motor provide the vehicle’s required power, and their relationship is commonly expressed as follows:

$$P_{\mathrm{dem}}=\left(P_{en}+P_{bat}\cdot {\eta }_m\right){\eta }_T,$$

(2)

where $P_{en}$ represents the output power of the engine, $P_{bat}$ represents the power of the battery, ${\eta }_m$ represents the efficiency of the motor, and ${\eta }_T$ represents the transmission efficiency.

2.2. Engine Modeling

The quasi-static model of the engine postulates that parameters such as speed, load, and fuel consumption change gradually over short time intervals, indicating that the engine remains in a state of equilibrium at each time point. This assumption facilitates the use of a simplified equation to characterize the engine’s fuel efficiency and other performance metrics, while excluding the effects of dynamic responses and complex nonlinear behaviors. The fuel consumption rate is expressed as follows:

$${\dot{m}}_f=f\left(T_{en},n_{en}\right),$$

(3)

where $T_{en}$ is the engine output torque and $n_{en}$ is the engine speed. The total fuel consumption under one operating cycle can be obtained by the following integrals:

$$\mathit{Fuel}={\int }_{t_0}^t {\dot{m}}_{f}dt,$$

(4)

where $\mathrm{t}\in [t_0,t]$ is the time span of the calculation cycle.

2.3. Motor Modeling

The performance of the motor can be characterized using a power diagram, where motor torque and speed serve as the axes. Rooted in the operating mode of the motor, the following formula is defined:

$${ P}_{elec}=\left\{\begin{array}{ll}{\displaystyle \frac{1}{{\eta }_m\left({\omega }_{em},T_{em}\right)}}{P}_{mech}={\displaystyle \frac{1}{{\eta }_m\left({\omega }_{em},T_{em}\right)}}{\omega }_{em}{T}_{em}& \mathit{if}{P}_{elec}\ge 0 (\mathrm{m}\mathrm{o}\mathrm{t}\mathrm{o}\mathrm{r}\mathrm{i}\mathrm{n}\mathrm{g} \mathrm{m}\mathrm{o}\mathrm{d}\mathrm{e})\\ {\eta }_m\left({\omega }_{em},T_{em}\right)P_{\mathrm{mech}}={\eta }_m\left({\omega }_{em},T_{em}\right){\omega }_{em}{T}_{em}& \mathit{if}{P}_{elec}<0 (\mathrm{g}\mathrm{e}\mathrm{n}\mathrm{e}\mathrm{r}\mathrm{a}\mathrm{t}\mathrm{i}\mathrm{n}\mathrm{g} \mathrm{m}\mathrm{o}\mathrm{d}\mathrm{e}),\end{array}\right.$$

(5)

where $P_{elec}$ represents the power of the motor, ${\omega }_{em}$ represents the speed of the motor, $T_{em}$ represents the rotation of the motor, ${\eta }_m$ represents the efficiency of the motor, and $P_{\mathrm{mech}}$ represents the mechanical work of the motor.

2.4. Battery Modeling

Power batteries can supply energy to the electric motor as well as store energy generated by the motor. The battery’s SOC is defined by Equation (6):

$$SOC\left(t\right)=SOC\left(t_0\right)-{\displaystyle \frac{1}{Q_{\mathrm{b}\mathrm{a}\mathrm{t}}}}{\int }_{t_0}^t {\mathrm{I}}_{\mathrm{b}\mathrm{a}\mathrm{t}}(\mathrm{t})dt,$$

(6)

where $I_{bat}\left(t\right)$ represents the battery current and $Q_{bat}$ represents battery’s rated capacity. If voltage dynamics are ignored and the battery circuit is represented as a RC-free branch, the above formula can be re-expressed as follows:

$$\dot{SOC}=-{\displaystyle \frac{V_{oc}-\sqrt{V_{oc}^2-4R_{\mathit{int}}{P}_{bat}}}{2R_{\mathit{int}}{Q}_{bat}}},$$

(7)

where $P_{bat}$ represents the battery’s output power, $V_{oc}$ represents the open circuit voltage of the battery, and $R_{\mathit{int}}$ represents the internal resistance of the battery.

3. Adaptive Learning Strategy in Dynamic Environment

3.1. Reinforcement Learning Modeling and Problem Description

Reinforcement learning is mathematically represented as a Markov decision process consisting of five fundamental elements: environment, agent, state, behavior, and reward [33,34,35,36]. It enables an agent to amass the maximum reward value through a reward and punishment mechanism, facilitating autonomous learning in different environments. In this paper, a vehicle’s driving cycle is utilized as the environment that reinforcement learning must navigate. The vehicle controller is an agent of reinforcement learning. The required engine power represents the agent’s action value. The change in engine speed, torque, and battery SOC value is used as the state vector. To address the subjectivity and empirical challenges in traditional reward function design, we propose a strategy grounded in inverse reinforcement learning. The approach derives weights of reward function from expert trajectories, ensuring that the reward function more objectively reflects the balance between fuel efficiency and battery lifespan [37], as follows:

$$action=P_{en},$$

(8)

$$state=[n_{t,} T_{t,} SOC],$$

(9)

$$r={\int }_{t_0}^t \alpha \cdot {\dot{m}}_f\left(t\right)dt+\beta \cdot {\left[\mathit{SOC}\left(t\right)-\mathit{SOC}\left(t_0\right)\right]}^2,$$

(10)

where $P_{en}$, $n_t$, $T_t$, and $SOC$ represent engine power output, engine speed, engine torque, and battery SOC, respectively. The reward function consists of two parts. The first term represents fuel consumption; ${\dot{m}}_f\left(\mathrm{t}\right)$ is the instantaneous fuel consumption rate. The second term represents the difference between initial and current SOC. In Equation (10), $\alpha$ and $\beta$ are two constant factors.

The dynamic environment of the driving cycle during vehicle operation can be seen as a series of fixed tasks over time. Each task corresponds to specific environmental characteristics relevant to that period. The objective of optimal power allocation is achieved under these environmental conditions, where machine learning (ML) enables the transition from a local optimum to a global optimal fuel consumption. When the vehicle’s driving environment changes, the ALDE algorithm’s agent can initiate a new learning process based on previously acquired knowledge, incrementally adapting it to suit the new conditions. Reinforcement learning agents are akin to the human brain, where changing elements and unexpected disturbances are the norm in real driving scenarios. In contrast, humans possess the necessary driving experience to navigate unforeseen environmental changes. Therefore, enhancing the agent’s adaptability becomes crucial. The key is enabling the agent to compare its own memories and respond promptly. Inspired by human capabilities, we develop ML from which hybrid vehicle agents can draw analogies to effectively handle dynamic environments. This ML must retain historical information to the greatest extent. In this article, an agent’s behavior strategy and the parameters of the dynamic environment are identified as the fundamental elements to be stored in the ML. The ML can be expressed as follows:

$$ML=\left\{\begin{array}{c}{X}_1:X_{11}\cdots X_{1e}\\ X_2:X_{21}\cdots X_{2f}\\ ⋮ ⋮ ⋮\\ X_i:X_{i1}\cdots X_{ij}\end{array}\right., X_{ij}\in {[\omega ,p]}_{ij},$$

(11)

where $X$ depicts the driving condition block, $i$ demonstrates the number of driving condition clusters, $j$ represents the total number of driving condition blocks in a driving condition cluster, $p$ denotes the dynamic environment, and $\omega$ depicts the primary eigenvalue. In this experiment, the number of blocks in the clusters varies depending on the specific cluster.

The dynamic environment parameters can be quantified by the reward function and state vector as follows:

$$p\in [r,state],$$

(12)

Figure 1 illustrates the proposed algorithmic framework of this paper, which is thoughtfully designed into two main parts: the left side represents the algorithm kernel, while the right side details the vehicle structure. The primary inputs in the present driving condition include the primary eigenvalue $\mathsf{\omega }$ and the state S, which are fundamental to the ALDE method. The algorithm kernel is divided into two parallel sections. The left side focuses on probability calculations and network training, whose goal is to enhance the responsiveness to environmental changes and intelligence of its decisions. In contrast, the right side emphasizes parameter optimization and iterative updates, ensuring that the vehicle can adapt to changing external conditions while maintaining optimal performance across various operating environments. The parallel processing of these updates significantly enhances the precision and performance of the algorithm. The next $Action$ computed by the algorithm kernel is then sent to the vehicle structure module for immediate adjustments to the vehicle’s operational behavior, ensuring smooth and safe driving. In summary, the thoughtfully designed structure improves the quality of decisions of the model and adaptability and generalization capabilities in dynamic environments.

Step 1: The first part is to calculate the current aggregation probability so as to determine the driving condition category that the driving condition block belongs to. Meanwhile, the second part performs the DQN algorithm to acquire the optimal action corresponding to this driving cycle block.

Step 2: In the first part, the optimal Action calculated before is recorded in the corresponding ML. It may be used to create a new record, add a working block information record in the existing driving condition, or directly use the control strategy of a working block and use the EM algorithm to update it. The second part also updates the DQN weight through experience playback and turntable transfer.

Step 3: The control closed loop is completed by outputting the Action determined by the present ALDE algorithm.

3.2. Cluster Aggregation in the Adaptive Algorithm

3.2.1. Definition and Characteristics of Driving Condition Block in Cluster Aggregation

To replicate the frequent start-stop, acceleration, and deceleration of real driving scenarios, we define a driving condition block as the transition between successive idle states. The block generally includes idle segments and dynamic segments. The moving segments typically encompass acceleration, constant speed, and deceleration conditions. Figure 2 depicts the definition of a driving condition block.

Each driving cycle can extract multiple feature parameters. Some characteristics are interconnected and dependent on one another, resulting in repetitive information. Given that feature parameter matrices would be overly complex. Using just one or two arbitrary characteristic parameters may not accurately represent present driving conditions. Therefore, it is crucial to use only the key components to describe the driving conditions and to appropriately limit the dimensions of the matrix.

In this article, 126 blocks are chosen from various standard driving cycles (WLTC, RTS95…) as a training set. The cumulative contribution rate of the first T primary components is

$$\psi ={\displaystyle \frac{{\sum }_{i=1}^T {\lambda }_i}{{\sum }_{i=1}^{15} {\lambda }_i}},$$

(13)

where ${\lambda }_i$ is the characteristic parameter, $\psi$ represents the cumulative contribution rate of the first T primary components, ${\sum }_{i=1}^T {\lambda }_i$ denotes the sum of eigenvalues of the first T principal components, and ${\sum }_{i=1}^{15} {\lambda }_i$ represents the sum of eigenvalues of the first 15 principal components.

Table 2. Specific contribution rate and cumulative contribution rate.

Ingredient	Contribution Rate/%	Cumulative Contribution Rate/%
Average Speed	45.13	45.13
Idle Time	23.28	68.41
Distance	10.47	78.88
⋮	⋮	⋮

shows the specific contribution rate and cumulative contribution rate. Ultimately, three characteristic parameters with the highest contribution rates, average speed, idle time ratio, and working block distance are selected to represent the driving condition block characteristics.

3.2.2. Cluster Aggregation Based on Dirichlet Method

The Dirichlet Process (DP) is a stochastic process characterized by a probability distribution [38,39]. For instance, with a base measure H, the random distribution of DP $G$ can be expressed as $G\sim \mathrm{DP}\left({\alpha }_0,H\right)$, where ${\alpha }_0$ is the concentration parameter of DP, which indicates the level of uncertainty in clustering. A lower ${\alpha }_0$ value raises the number of clusters, while a higher ${\alpha }_0$ value has the opposite effect. $H$ is the basis measure, which is a basic distribution used to represent the cluster center distribution.

The features of every driving cycle type can be represented as a vector $X_i=\left(X_{1i},\cdots ,X_{ij}\right)$, where $X_{ij}$ is the $j$ driving cycle block feature of the $i$ driving condition type, and the vector is generated by polynomial distribution with parameters $\left\{{\theta }_{1i},\cdots ,{\theta }_{ij}\right\}$. The activity characteristics of the $n$ driving condition blocks to be clustered form the vector $X=\left(X_1,\cdots ,X_i\right)$.

Assume that they are independent of each other and come from a mixed distribution F(Θ), which is composed of multiple polynomial distributions; the parameter $\mathsf{\Theta }$ that constitutes the probability of variable polynomial distributions can be drawn from DP $G\sim \mathrm{DP}\left({\alpha }_0,H\right)$. If some samples are generated by a mixed distribution under the same parameter $\mathsf{\Theta }$, then these data samples can be classified into the same class. The DPMM (Dirichlet Process Mixture Model) that achieves cluster aggregation can be expressed as follows:

$$x_i=\left(X_{1\mathrm{i}},\cdots ,X_{ij}\right)\sim \mathrm{Mult}\left({\mathsf{\Theta }}_{z_i}\right),i=1,\cdots ,n,$$

(14)

$$\mathsf{\Theta }=\left({\theta }_{z_{i^1}},\cdots ,{\theta }_{z_{i^j}}\right)\sim \mathrm{Dir}\left({\beta }_1,\cdots ,{\beta }_j\right),1⩽z_i⩽K,$$

(15)

$$z_i\sim \mathrm{Mult}\left(\mathsf{\Pi }\right),i=1,\cdots ,n,$$

(16)

$$\mathsf{\Pi }=\left({\mu }_1,\cdots ,{\mu }_k,\cdots ,{\mu }_{K+1}\right)\sim \mathrm{DP}\left({\alpha }_0,H\right),$$

(17)

where Equation (14) expresses the block of condition $i$ generated by the polynomial distribution of class $z_i$ with parameter ${\theta }_{z_i}$, $z_i$ is the class number, and $K$ is the total of classes. Equation (15) indicates the prior distribution of parameter ${\mathsf{\Theta }}_{z_i}$, and its parameters are $\left({\beta }_1,\cdots ,{\beta }_j\right)$. Equation (16) indicates that the $z_i$ class is drawn from the polynomial distribution with parameter $\mathsf{\Pi }$. Equation (17) expresses parameter $\mathsf{\Pi }=\left({\mu }_1,\cdots ,{\mu }_k,\cdots ,{\mu }_{K+1}\right)$, which is extracted from the process $\mathrm{G}\sim \mathrm{DP}\left({\alpha }_0,H\right)$.

3.3. Chinese Restaurant Process in ALDE

We propose using the infinite DP mixture model in stochastic dynamic conditions to construct a prior distribution of rising condition clusters, providing a flexible structure for the observed dynamic environment. The Chinese Restaurant Process (CRP) explains this by comparing a sequence of customers sitting at tables in a restaurant, where each table represents a cluster. The likelihood that each customer sits alone at a new table is proportional to the lumped parameter, and the probability that each customer sits at an existing table is related to the number of customers seated here.

The likelihood that characteristics of each working block correspond to an existing class and the probability of belonging to a new class in the Chinese restaurant model are as follows:

$${\mu }_k=\mathrm{Pr}\left(z_i=k\mid Z_{-i},X\right)={\displaystyle \frac{1}{B}}{\displaystyle \frac{n_k}{{\alpha }_0+n-1}}\mathrm{Pr}\left(x_i\mid z_i=k\right),1⩽k⩽K,$$

(18)

$${\mu }_{K+1}=\mathrm{P}\mathrm{r}\left(z_i=K+1\mid Z_{-i},X\right)={\displaystyle \frac{1}{B}}{\displaystyle \frac{{\alpha }_0}{{\alpha }_0+n-1}}\mathrm{Pr}\left(x_i\mid z_i=K+1\right),$$

(19)

where $k$ is sample category number $X_i$, B is a normalization factor, which enables the total of probabilities to be 1, $n_k$ is the sum of all dimension counts of all samples belonging to class $k$, and $Z_{-i}$ is the set of all classes but $z_i$.

3.4. Adaptive Algorithm Update

To update the adaptive algorithm in a dynamic environment, we propose using the EM algorithm and online Bayesian inference to update the mixed environment model incrementally. The EM algorithm is an iterative optimization algorithm that updates model parameters through the E-step and M-step, aiming to maximize the likelihood function of the model. It is commonly used for parameter estimation in probabilistic models with latent variables. In the E-step, it calculates posterior probabilities of latent variables based on present parameter estimates, typically involving computing expectations. In the M-step, it updates model parameters by maximizing the logarithm of the likelihood function shaped by complete data. This step often involves solving maximization problems. Online Bayesian inference is an inference method used for parameter estimation and prediction in dynamic environments. It involves continuously updating the prior distribution to incorporate new observed data and obtain an approximation of the posterior distribution. In online Bayesian inference, the parameters and states of the model are updated over time. With this approach, the dataset is repetitive as the environment cluster grows larger without sacrificing prior knowledge. When the environment engages with the controller agent, we use $(m,n)$ to represent the input and output of the vehicle dynamic environment and $(M_t^i,N_t^i)$ to represent the input and output at time t. $X_{it}$ is the present driving condition block, and ${\omega }_t^i$ is the eigenvalue of the current driving condition block. During each iteration, model parameters are adjusted, leading to an increase in the likelihood function of the probabilistic model with hidden variables. The iteration ends when it no longer rises, or the value of the increase is less than the set threshold. In this section, the EM algorithm is combined with the algorithm to measure the similarity of the vehicle dynamic environment, as follows:

E-Step: The posterior expectation of the present block-to-cluster assignment is

$$P\left({\omega }_t^i\mid M_t^i,N_t^i\right)=\left\{\begin{array}{c}{\mu }_k ,1⩽k⩽K\\ {\mu }_{K+1},k=K+1\end{array}\right.,$$

(20)

Combined with CRP, it can be written as follows:

$$P\left({\omega }_t^i\mid M_t^i,N_t^i\right)\propto \left\{\begin{array}{c}{\displaystyle \frac{1}{B}}{\displaystyle \frac{n_k}{{\alpha }_0+n-1}}\mathrm{Pr}\left(x_i\mid z_i=k\right),1⩽k⩽K\\ {\displaystyle \frac{1}{B}}{\displaystyle \frac{{\alpha }_0}{{\alpha }_0+n-1}}\mathrm{Pr}\left(x_i\mid z_i=k\right),k=K+1\end{array}\right.,$$

(21)

M-step: According to the estimated posterior expectation $P\left({\omega }_t^i\mid M_t^i,N_t^i\right)$, we execute the M-step to optimize the anticipated log-likelihood arising from the inference environment probability. Assuming that each driving condition begins at the first driving condition block $X_{i1}$, the value of ${\omega }_t^i$ can be obtained after a gradient update at every stage. The update formula is as follows:

$${\omega }_t^i={\omega }_{t-1}^i-\delta {\sum }_{t=1}^{t-1} P\left({\omega }_{t-1}^i\mid M_{t-1}^i,N_{t-1}^i\right){\nabla }_{{\omega }_{t-1}^i}\mathrm{l}\mathrm{o}\mathrm{g}{p}_{{\omega }_{t-1}^i}\left(N_{t-1}^i\mid M_{t-1}^i\right),$$

(22)

where δ is the learning rate of the EM algorithm. We can approximate the formula by incrementally updating the previous parameters in the current environment sample:

$${\omega }_t^i={\omega }_{t-1}^i-\delta P\left({\omega }_{t-1}^i\mid M_{t-1}^i,N_{t-1}^i\right){\nabla }_{{\omega }_{t-1}^i}\mathrm{l}\mathrm{o}\mathrm{g}{p}_{{\omega }_{t-1}^i}\left(N_{t-1}^i\mid M_{t-1}^i\right),\forall i,$$

(23)

This process avoids the need to store previously observed samples and implements a fully streaming, incremental learning algorithm along with online Bayesian inference. To completely execute the EM algorithm, ALDE needs to switch between the E-step and the M-step repeatedly to achieve convergence. The pseudo-code of ALDE is depicted in Algorithm 1.

Algorithm 1. Pseudo-code of ALDE.

Input: Characteristic parameters of dynamic driving condition blocks

Output: Optimal action (engine power)

1. $\mathrm{Initialize} \mathrm{i}$$=1,\mathrm{t}$ = 1

2. $\mathrm{Calculate} \mathrm{the} \mathrm{characteristic} \mathrm{parameters} \mathrm{of} \mathrm{current} \mathrm{driving} \mathrm{condition} \mathrm{block} {\mathsf{\omega }}_1^1$ $\mathrm{and} \mathrm{restore} \mathrm{them} \mathrm{in} \mathrm{the} \mathrm{first} \mathrm{container} \mathrm{of} {\mathrm{X}}_{11}$

3. $\mathrm{Memorize} \mathrm{current} \mathrm{policy} \mathrm{chosen} \mathrm{by} \mathrm{deep} \mathrm{reinforcement} \mathrm{learning} \mathrm{in} \mathrm{the} \sec \mathrm{ond} \mathrm{container} \mathrm{of} {\mathrm{X}}_{11}$

4. for following each driving condition block during one episode do

5.  $\mathrm{Calculate} \mathrm{P}\left({\mathsf{\omega }}_{\mathrm{t}}^{\mathrm{i}}\mid {\mathrm{M}}_{\mathrm{t}}^{\mathrm{i}},{\mathrm{N}}_{\mathrm{t}}^{\mathrm{i}}\right)$ of current driving condition block and conduct Dirichlet driving condition cluster polymerization

6.   $\mathbf{if} \mathrm{P}\left({\mathsf{\omega }}_{\mathrm{t}}^{\mathrm{i}}\mid {\mathrm{M}}_{\mathrm{t}}^{\mathrm{i}},{\mathrm{N}}_{\mathrm{t}}^{\mathrm{i}}\right)\propto {\displaystyle \frac{1}{\mathrm{B}}}{\displaystyle \frac{{\mathrm{n}}_{\mathrm{k}}}{{\mathsf{\alpha }}_0+\mathrm{n}-1}}\mathrm{Pr}\left({\mathrm{x}}_{\mathrm{i}}\mid {\mathrm{z}}_{\mathrm{i}}=\mathrm{k}\right),1⩽\mathrm{k}⩽\mathrm{K}$     Conduct current driving condition block with most resembles action in corresponding     $\mathrm{existing} \mathrm{driving} \mathrm{condition} \mathrm{polymerization} \mathrm{i}$$\mathrm{and} \mathrm{restore} \mathrm{them} \mathrm{in} \mathrm{the} \mathrm{first} \mathrm{container} \mathrm{of} {\mathrm{X}}_{\mathrm{i}\mathrm{j}}$,     $\mathrm{the} \mathrm{current} \mathrm{policy} \mathrm{in} \mathrm{the} \sec \mathrm{ond} \mathrm{container} \mathrm{of} {\mathrm{X}}_{\mathrm{i}\mathrm{j}}$

7.  else     $\mathrm{Build} \mathrm{a} \mathrm{brand} \mathrm{new} \mathrm{driving} \mathrm{condition} \mathrm{polymerization} \mathrm{i}$, add current driving      $\mathrm{condition} \mathrm{block} \mathrm{to} \mathrm{this} \mathrm{driving} \mathrm{condition} \mathrm{polymerization} \mathrm{i}$, and restore them in     $\mathrm{the} \mathrm{first} \mathrm{container} \mathrm{of} {\mathrm{X}}_{\mathrm{i}\mathrm{j}}$$, \mathrm{the} \mathrm{current} \mathrm{policy} \mathrm{in} \mathrm{the} \sec \mathrm{ond} \mathrm{container} \mathrm{of} {\mathrm{X}}_{\mathrm{i}\mathrm{j}}$

8.  $\mathbf{while} {\mathsf{\omega }}_{1~\mathrm{t}}^{\mathrm{i}}$ not converging do     $\mathrm{To} \mathrm{update} \mathrm{for} \mathrm{all} \mathrm{previous} \mathrm{time} \mathrm{periods} 1, 2, \dots ,t-1$ with:   ${\omega }_t^i={\omega }_{t-1}^i-\delta P\left({\omega }_{t-1}^i\mid M_{t-1}^i,N_{t-1}^i\right){\nabla }_{{\omega }_{t-1}^i}\log p_{{\omega }_{t-1}^i}\left(N_{t-1}^i\mid M_{t-1}^i\right),\forall i$

9. end

4. Simulation and Hardware-in-the-Loop Experiment

4.1. Simulation Experiment

To evaluate the effectiveness of the proposed strategy, we assess engine operating points, fuel consumption, and SOC interpolation. We configure the initial state of the vehicle to full gasoline. All algorithms are executed using Python 3.5, running on Ubuntu 16 with 48 Intel(R) Xeon(R) E5-2650 2.20GHz CPU processors, 193GB RAM, and NVIDIA Tesla 32GB GPU.

To validate the capability of the ALDE algorithm, we have created a new driving cycle, as shown in Figure 3a. The results of the ML constructed are presented in Figure 3b. The specific types are as follows: (3.505 m/s, 36.27% idle percentage, 198.9 m), (6.781 m/s, 17.82% idle percentage, 687.4 m), (10.02 m/s, 15.85% idle percentage, 1704 m), and (13.51 m/s, 12.80% idle percentage, 3418 m), which are obtained through the Dirichlet clustering method. The scatter points are represented in four colors—orange, blue, red, and green—corresponding to four typical operating conditions. There are four driving conditions within the clustering scenarios: city traffic jam, smooth city traffic flow, suburban area, and highway. City traffic jam refers to a situation where the traffic volume on urban roads is excessively high, resulting in slow and congested vehicle movement. In city traffic jams, vehicles frequently stop, start, and move at low speeds. City smooth traffic flow refers to a situation where the traffic volume on urban roads is relatively stable, allowing vehicles to travel at a moderately steady speed without frequent stops or congestion. Suburban areas typically refer to the outskirts of cities, where traffic volume is relatively low, and roads are wider. In suburban areas, vehicles can usually travel at higher speeds without being affected by city traffic congestion. Highways are road networks designed for high-speed driving. On highways, vehicles can cruise at higher speeds for extended periods, and although traffic volume is relatively high, it is usually smoother. We sequentially train the DQN and DDPG agents on the comprehensive training dataset, which encompasses a variety of driving scenarios, such as LA92, WLTP, RTS95, US06, and more, covering all 126 distinct driving condition clusters as previously mentioned. However, it does not include the new driving cycle. After training the proposed and compared algorithms, their reward curves in new driving condition cycle (test set) are shown in Figure 3c, which demonstrates the convergence speed of three reinforcement learning algorithms under entirely new and unknown driving conditions.

Figure 3c illustrates the reward curve for each iteration of policy iteration in the new driving condition cycle. Compared to DQN and DDPG, our approach demonstrates the quickest adaptation for learning in a dynamic environment and achieves the fastest convergence. This is attributed to its straightforward adaptation mechanism, which effectively leverages learned knowledge from the memory library (ML). Furthermore, the graph indicates that the ALDE algorithm is constantly superior to all baseline methods in terms of return during training, with a relatively stable curve following convergence. This suggests that ALDE provides a more reliable form of adaptation for learning in a dynamic environment. Therefore, the reward function curve illustrates that ALDE is well equipped to handle a dynamic environment where reward or state transition functions may update with each iteration, offering improved adaptation for learning.

DQN and DDPG fail to achieve the ideal standard encountering complete driving conditions that are absent from the training set, which is characteristic of the catastrophic forgetting issue commonly seen in traditional reinforcement learning algorithms. The analysis of the issue is outlined below: After completing comprehensive training on specific driving condition A, the agent experiences a decline in its ability to handle driving condition A when it continues training on a different set of driving condition B. When we extract complete driving conditions from the training set and select specific condition blocks to create a new driving condition for the test set, the DQN and DDPG algorithms encounter the inherent issue of catastrophic forgetting. Additionally, they treat the new complete driving condition as entirely unknown, which needs to be equipped with comprehensive learning.

The ALDE algorithm achieves better performance because of the ability to manage variable driving condition blocks resulting from Dirichlet clustering. It deconstructs the problem into single modules and utilizes the combination of Dirichlet distribution and EM updates. Encountering new driving conditions, the agent attempts to locate the corresponding driving condition sort ${\mathrm{X}}_{\mathrm{i}}$ from ML. It then adopts a control strategy of the condition block ${\mathrm{X}}_{\mathrm{i}\mathrm{j}}$ that is most similar. After achieving the present task, it updates the control strategy of ${\mathrm{X}}_{\mathrm{i}\mathrm{j}}$ in reverse. To avoid the agent failing to identify the similar driving condition ${\mathrm{X}}_{\mathrm{i}}$, it generates a new condition ${\mathrm{X}}_{\mathrm{i}+1}$, updating itself as the ${\mathrm{X}}_{\mathrm{i}+11}$. It executes the control strategy from the most similar driving condition block in current ML and updates the strategy for ${\mathrm{X}}_{\mathrm{i}+11}$ as the process continues. The factors prevent ALDE from forgetting, enabling it to adapt and evolve in entirely new and unknown complete driving conditions. Faced with a new driving condition cycle, ALDE begins with a high initial reward value, which steadily increases as training progresses. After convergence, ALDE has the highest reward value and the smallest fluctuation.

Figure 4 and Figure 5 at the vehicle level compare the battery SOC value curve and the engine operating point for the three algorithms—DQN, DDPG, and ALDE—under the new driving cycle. In Figure 4, the SOC curve of the DQN algorithm and DDPG method fluctuates substantially as the SOC value changes, with the highest fluctuation range reaching roughly 0.4. Due to the presence of the driving condition block memory library, under the control of the ALDE algorithm, the SOC curve exhibits reduced overall fluctuations in the new driving cycle. This indicates that the ALDE agent is capable of promptly responding to completely new dynamic environments and possesses stronger adaptability. In addition, due to the reduced fluctuations in SOC, the lifespan of the battery has also been improved.

Figure 5 presents the universal characteristic map of the engine, where the color bar provides a third dimension of information—fuel consumption rate, measured in grams per kilowatt-hour (g/kWh). The varying shades of color visually represent different levels of fuel consumption. The solid red line represents the empirically derived optimal fuel consumption curve. Its purpose is to reduce time and computational resources wasted by the agent in exploring optimal fuel consumption curve.

In Figure 5, the engine operating points for three algorithms—DQN, DDPG, and ALDE—are illustrated, with the red line representing optimal fuel consumption. The DQN algorithm shows a scattered distribution of points, many of which are above the red line, indicating higher fuel consumption compared to optimal performance, though some points approach the red line. The DDPG algorithm demonstrates a more concentrated distribution, with several points closely aligning with the red line, suggesting better fuel efficiency overall, although some points still exceed optimal consumption. In contrast, the ALDE algorithm exhibits a varied distribution, with several points significantly below the red line, indicating instances of optimal fuel consumption, but it also contains points above the red line. Overall, DDPG appears to perform better in terms of fuel efficiency, while all three algorithms show varying degrees of performance relative to the optimal behavior.

In order to demonstrate the robustness of the ALDE algorithm, we have created two additional completely new driving conditions. These driving conditions are derived from combinations of condition blocks extracted from the DQN and DDPG training sets. The driving conditions are shown in Figure 6a,b, and the corresponding SOC curves are shown in Figure 6c,d. The conclusion remains the same.

The comparison of fuel consumption values under three general driving cycles and one new driving cycle is shown in

Table 3. Fuel consumption.

	Algorithm	DDPG	DQN	ALDE	Fuel Consumption Reduction Rate Compared with DDPG	Fuel Consumption Reduction Rate Compared with DQN
Driving Cycles
WLTP		6.74 L/100 km	7.05 L/100 km	6.32 L/100 km	6.23%	10.35%
US06		6.16 L/100 km	6.47 L/100 km	5.85 L/100 km	5.03%	9.58%
LA92		6.44 L/100 km	6.68 L/100 km	6.08 L/100 km	5.60%	8.98%
New driving cycle		7.38 L/100 km	7.87 L/100 km	7.13 L/100 km	3.39%	9.04%

. Fuel usage is reduced by roughly 5% compared with the traditional DDPG algorithm, and fuel consumption is decreased by roughly 8% compared with DQN. Therefore, the ALDE algorithm performs well in terms of energy savings and battery SOC value.

4.2. Hardware-in-Loop Experiment

4.2.1. Hardware-in-the-Loop Experimental Platform

The paper integrates actual vehicle controller hardware and employs dSPACE real-time control simulation equipment to establish an HIL testing platform for experimental research. The HIL testing platform for the vehicle control strategy comprises a host computer, Control Desk control platform, RapidECU_U2 rapid prototype controller, and dSPACE/Simulator real-time simulation system.

In this setup, the dSPACE/Simulator is employed to create a real-time testing environment. The host computer, rapid prototype controller, and simulation system are interconnected via a CAN bus. The host computer manages the downloading of simulation models and driver board interfaces to both the simulator and the rapid prototype controller. The Control Desk platform serves to develop a real-time monitoring interface, facilitating the observation of the simulation process and recording of relevant data. A low-voltage DC power supply and power distribution box provides 24 V DC power to the vehicle controller, while the host computer communicates with the lower computer through a CAN bus module.

The specific implementation process for the hardware-in-the-loop system is as follows: First, resulting from the vehicle simulation structure, the RTI module of the host computer creates a longitudinal simulation model for HIL testing, which is then compiled into a dl1 file and downloaded to the dSPACE/Simulator. Next, using the Codewarrior embedded development environment, we develop embedded control software for the adaptive control strategy. This strategy is compiled into C code using the RTW tool in Matlab/Simulink and programmed into the RapidECU using MeCa V1.5 software via a USBcan downloader.

Finally, the host computer, controller, and simulator are connected through a CAN network to form a local area network, enabling seamless communication among the components. After configuring the simulation monitoring interface on the host computer, we can effectively monitor the vehicle’s operational status and facilitate data reading and storage.

4.2.2. Simulation of Hardware-in-the-Loop Experiment

To assess the potential practical applications of the proposed EMS, HIL experiments were conducted. The integrated systems are depicted in Figure 7a,b [40].

The data detection system, situated below the driver system, is integrated with it. Data interaction is achieved through CAN communication technology allowing for the real-time acquisition of steering wheel angle, acceleration, and brake pedal data. These data are then fed into the vehicle control unit (VCU).

HIL experiments adhere to driving cycles presented in Table 3. Focusing on the new driving cycle, Figure 8a illustrates the HIL results for SOC trajectory (the data sampling frequency is 1000 Hz, and the data undergo filtering processing), and Figure 8b demonstrates the engine operating points. The HIL experiment results in Figure 8a,b are essentially consistent with the simulation tests. Additionally, equivalent fuel consumption for simulated and HIL tests is presented in

Table 4. Fuel consumption (HIL).

Algorithm	ALDE/SIM	ALDE/HIL
Driving Cycles
WLTP	6.32 L/100 km	6.29 L/100 km
US06	5.85 L/100 km	5.91 L/100 km
LA92	6.08 L/100 km	6.11 L/100 km
New driving cycle	7.13 L/100 km	7.07 L/100 km

. The HIL results demonstrate a high level of agreement with the simulation results, clarifying the fuel-saving performance of the ALDE algorithm.

5. Conclusions and Future Work

We propose a strategy for the vehicle agent that enables it to adaptively handle dynamic environments by modifying its behavior accordingly. We provide a memory library to help the agent make timely adaptive responses to the environment. The learning agent can search the stored repository, find the most similar experience, or extend new experiences to the repository during different periods of time. We combine the Dirichlet and EM algorithms to continuously aggregate and update driving condition clusters. The results indicate that the ALDE strategy efficiently clusters the environment in the potential space, enables memory library utilization, retrieves previously seen environments, and adaptively guides the agent’s actions. The reward curve during training indicates that the ALDE strategy successfully learns from past knowledge, quickly adjusts to changing environments, and converges rapidly. From the perspective of vehicle operation, the engine’s operating point remains near the optimal working curve, resulting in minimal SOC fluctuation.

In our future work, we aim to further explore how to integrate additional relevant factors into the state representation, such as road gradient, traffic conditions, and driver behavior. By including these elements, we hope to gain a more comprehensive understanding of the interactions among state variables, which will enhance the model’s decision-making capabilities and generalization. In addition, we also recognize that computational burden is a crucial consideration in assessing the feasibility of algorithms, and we will continue to prioritize this aspect in our future research.