Soft Actor-Critic Algorithm-Based Energy Management Strategy for Plug-In Hybrid Electric Vehicle

Abstract

Plug-in hybrid electric vehicles (PHEVs) are equipped with more than one power source, providing additional degrees of freedom to meet the driver’s power demand. Therefore, the reasonable allocation of the power demand of each power source by the energy management strategy (EMS) to keep each power source operating in the efficiency zone is essential for improving fuel economy. This paper proposes a novel model-free EMS based on the soft actor-critic (SAC) algorithm with automatic entropy tuning to balance the optimization of energy efficiency with the adaptability of driving cycles. The maximum entropy framework is introduced into deep reinforcement learning-based energy management to improve the performance of exploring the internal combustion engine (ICE) as well as the electric motor (EM) efficiency interval. Specifically, the automatic entropy adjustment framework improves the adaptability to driving cycles. In addition, the simulation is verified by the data collected from the real vehicle. The results show that the introduction of automatic entropy adjustment can effectively improve vehicle equivalent fuel economy. Compared with traditional EMS, the proposed EMS can save energy by 4.37%. Moreover, it is able to adapt to different driving cycles and can keep the state of charge to the reference value.

1. Introduction

Nowadays, the problems of environmental pollution and energy depletion are becoming more and more serious [1,2]. Unfortunately, traditional vehicles as the primary form of transportation not only emit large amounts of exhaust fumes, but also consume large amounts of petroleum resources [3]. Electric vehicles (EVs) are considered to be able to solve the problems of fuel consumption and emissions. However, there are several issues that prevent their actual diffusion, among which the most relevant is the limited driving range [4]. Meanwhile, the industry is developing PHEVs as an intermediate solution [5]. Compared with traditional vehicles, PHEVs are typically equipped with two power sources: the ICE and the EM that can be used as both a generator and EM [6]. Therefore, the EMS attempts to navigate energy between several energy sources considering one or more objectives while meeting the power needs of drivers [7,8].

1.1. Literature Review

Recently, various PHEV EMSs have been proposed. They can be broadly divided into three categories: rule-based EMSs [9], optimization-based EMSs [10] and machine learning-based EMSs [11].

The rule-based EMSs determine the operating state of the vehicle powertrain through pre-established rules. Due to its simplicity, rule-based EMSs are easily applied in practice [12,13]. However, rule-based EMSs are formulated empirically so that they cannot efficiently cope with all kinds of driving cycles, therefore cannot always ensure efficient control [14,15]. To further enhance fuel economy, the expert knowledge and optimization algorithms are applied to the rule-based EMSs, such as the genetic algorithm (GA) and particle swarm optimization (PSO) algorithm [16]. Zhou et al. present a multi-objective optimization method that the membership functions of the integrated strategy are tuned by elitist nondominant GA [17]. Natella et al. implement an optimization problem for the off-line section of the velocity thresholds and the corresponding power splitting between the actuators [18].

Optimization-based EMSs are mainly divided into global optimization-based EMSs and real-time optimization-based EMSs [19]. The dynamic programming (DP) algorithm and pontryagin minimum principle (PMP) strategy are the widely used global optimization algorithms in PHEV EMSs [20]. The DP algorithm requires perfect knowledge over the entire optimization horizon to obtain or approximate the global optimum and requires much computing time, so it is not suitable for real-time control of PHEVs [21]. The PMP algorithm-based strategy is also a widely studied global optimization strategy, which solves the optimization problem by minimizing the Hamiltonian function [22]. However, entire driving information is still needed to achieve improvements in fuel economy [23]. The equivalent consumption minimization strategy (ECMS) and model predictive control (MPC) strategy are representative of vehicle EMSs based on real-time optimization technologies [24]. ECMS is the most popular real-time EMS, which gets the optimum fuel consumption through the rational allocation of the power of the EM and ICE according to the torque or power requirements of the vehicle in the current moment, and balances fuel consumption and the battery state of charge (SOC) by the equivalent factor [25]. The equivalent factor plays a very important role in the ECMS, and it directly affects the effectiveness of the optimization strategy. Li et al. [26] studied the equivalent factor boundary of the ECMS for PHEVs. However, it is still a challenge to precisely determine the equivalence factor. This limits the adaptive adjustment capabilities of the ECMS. Feng et al. [27] proposed an adaptive ECMS with energy demand prediction to improve the adaptive capability. The MPC strategy optimizes the fuel economy of PHEV on a moving finite horizon [28]. However, it is not possible to update the result of MPC frequently when computing on look-ahead horizons because the computation time increases with the horizon length. Therefore, Uebel et al. [29] proposed a two-level MPC approach to overcome the computational burden. Mariani et al. [30] propose the design of an MPC strategy for maximizing regenerative braking in a real vehicle that has been hybridized by means of a kit.

The above approaches lack self-learning capabilities, which are dependent on the accuracy of dynamic models, and can fail if such models run under various environments. Therefore, machine learning-based EMSs have received extensive attention in recent years, with reinforcement learning (RL) and deep reinforcement learning (DRL) being the most widely studied. RL’s main idea is to train a fully autonomous agent by interacting directly with its potential environment [31]. It is different from supervised and unsupervised machine learning which need static data during the training process. In [32], model-free predictive EMS with multi-step learning capabilities was proposed, in which the Q learning-based EMSs, including the sum-to-terminal strategy, average-to-neighbor strategy and recurrent-to-terminal strategy were investigated. Compared to the conventional strategy, a real-time fast Q-learning-based reinforcement learning EMS was investigated to improve the fuel economy and reduce the computational efficiency [33]. Chen et al. [34] combine the Markov decision process (MDP) and Q-learning algorithms to design power flow EMS, which achieves fuel economy improvement under different driving cycles. The Q-learning algorithm needs to build a Q table whose size is determined by the dimension of the states and the action. When applied to complex HEV, Q-learning needs discrete continuous states and actions so that the curse of dimensionality, as well as discrete error, limits its application [35]. In order to solve the problem with high-dimensional and continuous state/action, the deep neural network is used in the RL approaches [36]. Wu et al. [37] proposed a continuous EMS based on deep Q learning (DQL), which approximates the action value function through a deep neural network. By using double deep Q-learning (DDQL), Han et al. [38] addressed the problem that traditional DQL tended to fall into the trap of over-optimistic estimation of Q value during training. DDQL-based intelligent decision algorithm was proposed in [39], which achieves energy savings similar to offline optimization. Both DQL and DDQL sample data from empirical replay, then calculate the target Q value. All the samples in the replay buffer have the same probability of being sampled. This leads to a reduction in the learning efficiency of the RL agent. In order to improve the learning efficiency, Runna et al. [40] applied prioritized replay to the DQL-based EMS. DQL is a value function-based approach, but it has trouble with solving large action spaces, especially continuous spaces [41,42]. Yue et al. [43] introduce temporal-difference (TD) learning based on Q learning and achieve the management of HEV supercapacitors and power batteries through a model-free online strategy. A DRL EMS based on the TD algorithm was proposed and combined with road information to achieve self-learning power flow distribution of the hybrid vehicle power system in [44]. Lian et al. [45] used the deep deterministic policy gradient (DDPG) algorithm to solve the problem of multi-objective energy management with a large control variable space. However, the above deep reinforcement learning algorithms are often highly sensitive to hyperparameters which directly affect the convergence performance, and even the RL agent is difficult to adapt to different conditions [46]. Recent work has shown that the SAC deep reinforcement learning algorithm with maximum entropy learning can solve the above problems. The standard RL improves their performance only by maximizing the cumulative reward. Because the maximum entropy learning is to achieve entropy maximization, the SAC improves its performance by maximizing the weighted sum of expected cumulative reward and entropy. The maximum entropy learning frame is introduced to improve the ability of the action exploration and robustness. Furthermore, in order to promote the algorithm performance, auto-entropy tuning (AET) is proposed [47].

1.2. Contribution

Considering the good convergence and insensitivity to hyperparameters of the SAC algorithm with auto-entropy tuning (SAC−AET), the SAC−AET-based EMS is proposed to improve the control effect of the traditional RL algorithm-based EMS. The ability of the maximum entropy learning framework used in EMS to explore the vehicle efficiency space is investigated. An automatic entropy adjustment framework is introduced to enhance the EMS’s adaptability to driving cycles. In addition, the driving cycles were collected from real vehicles. The fuel economy of the proposed SAC-based EMS is compared with that of the DDPG strategy, traditional SAC strategy and ECMS. In addition, the adaptability of the proposed strategy to different driving conditions is verified by combining driving cycles.

The remainder of the paper is organized as follows. The PHEV powertrain model is described in Section 2. The SAC algorithm-based EMS is described in detail in Section 3. In Section 4, simulations are designed to evaluate the performance of the proposed EMS, and simulation results are evaluated. The final section is the conclusion of the paper.

2. PHEV Powertrain Model

In this paper, the studied vehicle is a plug-in hybrid electric bus with a parallel configuration which is shown in Figure 1. The vehicle powertrain consists mainly of ICE, EM, battery pack and dual-clutch. The two clutches separate the EM and ICE so that the vehicle powertrain can operate in different modes, such as only the ICE operation, only the EM operation, and ICE and EM operation together. The relevant parameters of the vehicle are given in

Table 1. Vehicle parameters.

Symbol	Parameters	Value
Vehicle	Curb weight	10,500 kg
	Rolling resistance coefficient	0.015
	Air resistance coefficient	0.65
	Frontal area	6.75 m${}^2$
EM	Maximum power	135 kW
	Maximum torque	1000 Nm
	Maximum speed	3500 rpm
ICE	Maximum power	159 kW
	Maximum torque	904 Nm
	Maximum speed	2300 rpm
Battery	Voltage	525 V
	capacity	96 Ah

.

The ICE and EM are modeled by efficiency maps collected from an experimental platform. The efficiency maps of the ICE and the electric EM depict the relationship between speed, torque and efficiency, as shown in Figure 2 and Figure 3. The red curve of the ICE efficiency map in Figure 2 corresponds to the optimal torque at maximum efficiency. The Figure 3 shows that the EM maintains a relatively high efficiency whatever the operating conditions.

The balance equation of the vehicle longitudinal force is given as follows:

$$\begin{array}{c}\hfill F_t=F_j+F_i+F_w+F_f\end{array}$$

(1)

$$\begin{array}{c}\hfill \left\{\begin{array}{c}{F}_j=\delta m{a}_{cc}\hfill \\ F_i=mgsin\left(\theta \right)\hfill \\ F_w=\frac{1}{2}\rho C_d{A}_f{\nu }^2\hfill \\ F_f=mgfcos\left(\theta \right)\hfill \end{array}\right.\end{array}$$

(2)

where $F_t$ is traction force, $F_j$ is the acceleration force, $F_i$ is the road grade force, $F_w$ is aerodynamic resistance force, $F_f$ is rolling resistance force, m is the gross weight, g is the gravity constant, $\theta$ is the road slope, $C_d$ is the aerodynamic coefficient, $A_f$ is the vehicle frontal area, f is the rolling resistance coefficient, $\nu$ is the vehicle velocity. $\delta$ is the rotational mass coefficient. $a_{cc}$ is the vehicle acceleration, and $\rho$ is the air density.

The vehicle torque demand is given in Equation (3).

$$\begin{array}{c}\hfill T_{req}=F_t\times r_w÷(i_g\times i_o\times \eta )\end{array}$$

(3)

where $r_w$ is the wheel radius, $i_g$ is transmission ratio, $i_o$ is the final drive ratio, $\eta$ is the drive system efficiency. The powertrain structure of the studied PHEV is configured with a six-speed automated manual transmission (AMT) gearbox, and the AMT gearbox parameters are shown in

Table 2. The AMT gearbox parameters.

Upshifting Velocity (km/h)	0–10	10–20	20–32	32–50	50–66	66–95
Downshifting Velocity (km/h)	0–7	7–15	15–28	28–45	45–58	-
Gear position	1	2	3	4	5	6
Gear ratio	6.39	3.97	2.40	1.48	1	0.73

. According to the vehicle velocity, the gear information G and the transmission ratio $i_g$ are obtained by checking the table.

In this paper, the ICE torque demand $T_e$ is determined by the SAC strategy. The EM torque demand $T_m$ is calculated by the Equation (4).

$$\begin{array}{c}\hfill T_m=T_{req}-T_e\end{array}$$

(4)

where the $T_{req}$ is the demand torque.

The battery is a significant part of the vehicle. It not only powers the vehicle but also stores the recovered energy when the vehicle is decelerating. There are many important parameters of the battery, including battery current, open-circuit voltage, internal resistance, SOC, and so on [48]. In this research, SOC is more important since it describes the remaining battery energy. The Li-Ion battery is modeled by an equivalent internal resistance model that can calculate the SOC as follows:

$$\left\{\begin{array}{l}{P}_b\left(t\right)=P_m{\eta }_m^n\\ I_b\left(t\right)=\frac{V_{oc}\left(t\right)-\sqrt{V_{oc}^2\left(t\right)-4R_0{P}_b\left(t\right)}}{2R_0}\\ SOC\left(t\right)=\frac{Q_{in}-{\int }_0^{t}I\left(t\right)dt}{Q_{max}}\end{array}\right.$$

(5)

$$\begin{array}{c}\hfill n=\left\{\begin{array}{c}-1,if{P}_m>0\\ 1,if{P}_m\le 0\end{array}\right.\end{array}$$

(6)

where $P_b$ is the power of the battery, $P_m$ is the EM power, ${\eta }_m^k$ is the charge and discharge efficiency, $I_b$ is the current, $V_{oc}$ is the open-circuit voltage, $R_0$ is the internal resistance of the battery, $Q_{in}$ is the initial capacity of battery, $Q_{max}$ is the maximum battery capacity, n is a variable that varies according to the EM power.

3. Energy Management Strategy Based on SAC Algorithm

3.1. SAC Algorithm

MDP is a sequential decision problem with a fully observable stochastic environment. The goal of RL is to find the optimal strategy under MDP to maximize the final cumulative reward r. Each state under the MDP is related not only to the current state s but also to the current action a. Since RL relies on reward and punishment given by the environment to learn, the corresponding RL also includes the reward and punishment value r. Therefore, the RL process can be composed of a quadruplet M = (s, a, s’, r). The basic process of the agent–environment interaction for deep reinforcement learning is shown in Figure 4. The RL agent continuously interacts with the environment until it converges. In general, the process of the agent–environment interaction can be summarized as follows: at each moment of the discrete-time series, $t\in \{0,1,2,···\}$, the RL agent samples action $a_k$ from the policy network. After executing the action $a_k$, the agent gets a new state $s_{k+1}$ from the environment and obtains a reward r based on the action performed.

This deep reinforcement learning algorithm consists of three important parts: an experience replay for storing previous experiences to reduce sampling complexity, maximum entropy for stabilization and exploration, and an actor-critic structure consisting of a policy network and four Q networks built by Multilayer Perceptron (MPL). Through the interplay of the above three components, the SAC deep reinforcement learning algorithm can be divided into two processes: soft policy iteration and automating entropy adjustment.

3.1.1. Soft Policy Iteration

The aim of standard reinforcement learning is to learn a policy ${\pi }^{\ast }$(a${}_t$∣s${}_t$) to make the desired reward larger in the future. The SAC algorithm is different from standard reinforcement learning which only learns the optimal policy to maximize the reward. In order to explore all possible optimal paths, entropy term $\mathcal{H}$ is introduced:

$$\begin{array}{c}\hfill {\pi }^{\ast }=argma{x}_{\pi }{\sum }_t{\mathbb{E}}_{{\rho }_{\pi }}[r(s_t,a_t)+\alpha \mathcal{H}\left(\pi (·|s_t)\right)]\end{array}$$

(7)

where $\alpha$ is the equilibrium parameter that is used to regulate the tradeoff between reward and entropy. The strategy can be made more randomized by increasing $\alpha$, that is, the probability of each action is as uniform as possible, instead of concentrating on one action. $\mathcal{H}$ is the entropy value of strategy $\pi$, ${\rho }_{\pi }$ is the probability of state-action tuple under the current polity, r is the reward, s is the state, a is the action.

To ensure that the sum of reward (and entropy) is finite over the entire time−step sequence, k $\in \{0,1,2,3,···\}$, discount factor $\gamma$, $0\le \gamma \le 1$, is introduced. The policy with a discount factor can be defined as:

$$\begin{array}{c}\hfill {\pi }^{\ast }=argma{x}_{\pi }\sum {\mathbb{E}}_{{\rho }_{\pi }}\left[\sum _{l=k}^{\infty }{\gamma }^{l-k}\mathbb{E}[r(s_k,a_k)+\alpha \mathcal{H}\left(\pi (·|s_k)\right)]\right]\end{array}$$

(8)

l represents the time series from the current moment to the last moment.

Soft policy iteration is a general method to learn the optimal maximum entropy policy [49]. The actor−critic framework consists of two kinds of networks: the policy network and the Q network. The policy iteration process alternates between soft policy evaluation (Q network) and soft policy improvement (policy network) to maximize the reward during the iteration. The process of soft policy evaluation which computes the soft Q value $Q(s_k,a_k)$ can be achieved by repeatedly using the modified Bellman backup operator ${\mathcal{T}}^{\pi }$ for the fixed policy as follows:

$$\begin{array}{c}\hfill {\mathcal{T}}^{\pi }Q(s_k,a_k)\triangleq r(s_k,a_k)+\gamma {\mathbb{E}}_{\rho }\left[V\left(s_{k+1}\right)\right]\end{array}$$

(9)

where

$$\begin{array}{c}\hfill V\left(s_k\right)={\mathbb{E}}_{\pi }[Q(s_k,a_k)-\alpha log\pi \left(a_k\right|s_k)]\end{array}$$

(10)

is a soft value function.

In order to deal with complex and multimodal behaviors, the SAC algorithm introduces the general energy-based policy [50]. In the policy improvement process, the new policy is not tractable in practice, so the Kullback–Leibler divergence is introduced, which can limit policy to a certain set of policies $\mathsf{\Pi }$. The policy $\pi \in \mathsf{\Pi }$ is updated as follows:

$$\begin{array}{c}\hfill {\pi }_k^{\ast }=arg\underset{{\pi }^{{}^{\prime }}\in \prod }{min}{D}_{KL}\left({\pi }^{{}^{\prime }}(·|s_k)\parallel \frac{exp\left(\frac{1}{\alpha }{Q}^{{\pi }_{old}}(s_k,·)\right)}{Z^{{\pi }_{old}}\left(s_k\right)}\right)\end{array}$$

(11)

$Z^{{\pi }_{old}}\left(s_k\right)$ does not contribute to ${\pi }_k^{\ast }$, so it can be ignored to optimize the ${\pi }_k^{\ast }$, and it has been proven that $Q^{{\pi }_k^{\ast }}(s_k,a_k)\ge Q^{{\pi }_{old}}(s_k,a_k)$. More details can be seen in [50].

3.1.2. Automatic Entropy Adjustment

The parameter $\alpha$ in Equation (8) is a significant parameter that directly influences the optimization objective, which should be changed for different driving cycles by experienced engineers. Therefore, this section formulates a different objective from maximum entropy reinforcement learning to achieve the automatic entropy adjustment, where the entropy is considered as a constraint. We want to maximize the desired reward under the constraint of a minimum expected entropy as follows:

$$\begin{array}{c}\hfill \underset{{\pi }_{0:T}}{max}{\mathbb{E}}_{{\rho }_{\pi }}\left[\sum _{k=0}^{T}r(s_k,a_k)\right],\\ \hfill s.t.{\mathbb{E}}_{{\rho }_{\pi }}[-log\pi (·|s_k)]\ge \mathcal{H}\end{array}$$

(12)

where $\mathcal{H}$ is an expected minimum entropy threshold.

The optimal policy ${\pi }_k^{\ast }$ is given directly as follows, and its derivation is described in detail in [51].

$$\begin{array}{cc}\hfill {\pi }_k^{\ast }& =arg\underset{{\pi }_k}{max}\mathbb{E}[Q_k^{\ast }(s_k,a_k)-{\alpha }_{k}log\pi \left(a_k\right|s_k)]\hfill \\ \hfill & =arg\underset{{\pi }_k}{min}{D}_{KL}\left({\pi }_k\parallel \frac{1}{Z\left(s_k\right)}exp\left(\frac{1}{{\alpha }_k}{Q}_k^{\ast }(s_k,a_k)\right)\right)\hfill \end{array}$$

(13)

the Equation (13) is exactly the soft policy improvement step, which has the additional equilibrium parameter ${\alpha }_k$. The dual variable ${\alpha }_k^{\ast }$ is a function of the optimal strategy at k. The dual variable ${\alpha }_k^{\ast }$ can be solved as

$$\begin{array}{c}\hfill {\alpha }_k^{\ast }=arg\underset{{\alpha }_k}{min}{\mathbb{E}}_{\pi }[-{\alpha }_{k}log{\pi }_k^{\ast }\left(a_k\right|s_k;{\alpha }_k)-{\alpha }_k\mathcal{H}]\end{array}$$

(14)

3.2. Practical Algorithm

In the last section, the problem has already been solved in the tabular case. In order to extend the above method to the continuous domain, the function approximator is used to represent the Q-function $Q_{\pi }$ and policy ${\pi }_{\varphi }$. A number of effective value function approximation methods have been proposed [52]. In this paper, a neural network will be used to approximate the value function, which scores the effect produced by the current torque distribution. We use two artificial neural networks to approximate the Q-function with parameter ${\theta }_i$, $i\in \{1,2\}$, which aim at reducing the overestimation of the Q value. The Q value is chosen from the smaller of the double Q-function values. The target network with the parameters (${\overline{\theta }}_1,{\overline{\theta }}_2$) is introduced to prevent overfitting during the training of the two Q networks. In each iteration, the Q networks are trained by the gradient descent method and the parameters (${\overline{\theta }}_1,{\overline{\theta }}_2$) of the target networks are updated by an exponentially moving average of the value network weights as

$$\begin{array}{c}\hfill J_Q\left({\theta }_i\right)={\mathbb{E}}_{\mathcal{D}}\left[\frac{1}{2}{(Q_{{\theta }_i}(s_k,a_k)-(r(s_k,a_k)+\gamma V_{{\overline{\theta }}_1,{\overline{\theta }}_2}\left(s_{k+1}\right)))}^2\right]\end{array}$$

(15)

$$\begin{array}{c}\hfill {\overline{\theta }}_i(k+1)\leftarrow m{\theta }_i(k+1)+(1-m){\overline{\theta }}_i\left(k\right),i\in \{1,2\}\end{array}$$

(16)

where $0<m<1$, is a smoothing factor. $\mathcal{D}$ is the replay buffer, from which minibatch are obtained. A policy network is a Gaussian model with the mean and covariance given by the neural network. The Gaussian policy is trained by minimizing the loss $J_{\pi }\left(\varphi \right)$:

$$\begin{array}{cc}\hfill J_{\pi }\left(\varphi \right)& ={\mathbb{E}}_{\mathcal{D}}\left[\alpha log{\pi }_{\varphi }\left(a_k\right|s_k)-\underset{i\in \{1,2\}}{min}{Q}_{{\theta }_i}(s_k,a_k)\right]\hfill \end{array}$$

(17)

The learning of $\alpha$ can be obtained by approximating dual gradient descent to minimize the dual objective $J\left(\alpha \right)$:

$$\begin{array}{c}\hfill J\left(\alpha \right)={\mathbb{E}}_{\mathcal{D}}\left[-\alpha log{\pi }_{\varphi }\left(a_k\right|s_k)-\alpha \mathcal{H}\right]\end{array}$$

(18)

The Equations (15)–(18) form the core of the SAC−AET. In the following work, the networks will be optimized by Equations (15)–(18). The process of the SAC algorithm is shown in Algorithm 1.

Algorithm 1. Soft actor-critic DRL with automating entropy adjustment algorithm.

initialization:     Q networks with weights ${\theta }_i,i=\{1,2\}$,     policy network with weights $\varphi$,     target network with weights ${\overline{\theta }}_{1,2}$,     equilibrium parameter $\alpha$,     replay buffer $\mathcal{D}$; FOR EACH EPISODE DO      get initial state     $\mathbf{for}\mathbf{each}\mathbf{environment}\mathbf{step}\mathbf{do}$           choose action $a_k\sim \pi \left(a_k\right|s_k)$           take action $a_k$, observe $s_{k+1}$ and reward r           update replay buffer $\mathcal{D}\leftarrow \mathcal{D}\cup \{s_k,a_k,r,s_{k+1}\}$     $\mathbf{end}$     $\mathbf{for}\mathbf{each}\mathbf{gradient}\mathbf{step}\mathbf{do}$           sample a minibatch from $\mathcal{D}$           update the Q network parameters:                      $Q\leftarrow Q-\lambda {\widehat{\nabla }}_{Q}J\left(Q\right)$           update the policy network parameters:                      $\pi \leftarrow \pi -\lambda {\widehat{\nabla }}_{\pi }J\left(\pi \right)$           update the equilibrium parameter:                         $\alpha \leftarrow \alpha -\lambda {\widehat{\nabla }}_{\alpha }J\left(\alpha \right)$           update the target network parameters     $\mathbf{end}$ END

3.3. Design of SAC Algorithm-Based EMS

The three important components (state, action, reward) are defined as follows:

3.3.1. State

The gear is an important driving information factor for PHEV. Therefore, the state variables are given as follows:

$$\begin{array}{c}\hfill s\left(t\right)=\{\nu \left(t\right),a_{cc}\left(t\right),SOC\left(t\right),T_{req}\left(t\right),G\left(t\right)\}\end{array}$$

(19)

where $\nu$ is the vehicle velocity, $a_{cc}$ is the acceleration, G,$G\in [1,2,3,4,5,6]$, is the current gear of the vehicle, s is the current state, the future state is denoted by $s^{{}^{\prime }}$.

3.3.2. Action

The action $a\left(t\right)$, $0\le a\le 1$, is used as the control signal for the ICE at moment t. When the action $a\left(t\right)$ is obtained by the SAC algorithm, the demand torque of the ICE and EM can be calculated:

$$\begin{array}{c}\hfill \left\{\begin{array}{c}{T}_e\left(t\right)=T_{e\underline{}max}\left(t\right)a\left(t\right)\hfill \\ T_m\left(t\right)=T_{req}\left(t\right)-T_e\left(t\right)\hfill \end{array}\right.\end{array}$$

(20)

where $T_{e\underline{}max}$ denotes the maximum value of the ICE torque. In order to ensure the safety and stability of the vehicle, the following constraints should be ensured, which are mainly the physical limitations for the vehicle controller and the maximum operating parameter limitations of the ICE and EM:

$$\begin{array}{c}\hfill \left\{\begin{array}{c}SO{C}_{min}<SOC\left(t\right)<SO{C}_{max}\hfill \\ T_{e\underline{}min}<T_e\left(t\right)<T_{e\underline{}max}\hfill \\ T_{m\underline{}min}<T_m\left(t\right)<T_{m\underline{}max}\hfill \\ n_{e\underline{}min}<n_e\left(t\right)<n_{e\underline{}max}\hfill \\ n_{m\underline{}min}<n_m\left(t\right)<n_{m\underline{}max}\hfill \\ T_{req\underline{}min}<T_{req}\left(t\right)<T_{req\underline{}max}\hfill \\ I_{batt\underline{}min}<I_{batt}\left(t\right)<I_{batt\underline{}max}\hfill \end{array}\right.\end{array}$$

(21)

where $n_e$ and $n_m$ represent the ICE speed and the EM speed, respectively. max and min denote the maximum and minimum values of the corresponding variable.

3.3.3. Reward

The objectives of the EMS are to achieve fuel economy and keep the battery $SOC$ within a certain range. The reward is defined as a function of vehicle fuel consumption and battery $SOC$. The main purpose of the RL agent is to achieve the maximum reward. However, we want to decrease the instantaneous fuel consumption and keep the SOC to $SO{C}_{ref}$. The introduction of $r_{init}$ can resolve the conflict. The reward is shown as:

$$\begin{array}{c}\hfill r\left(t\right)=\left\{\begin{array}{c}{r}_{init}-cost\left(t\right),SOC\left(t\right)\ge SO{C}_{ref}\hfill \\ r_{init}-\left(\beta {\left(SO{C}_{ref}-SOC\left(t\right)\right)}^2+cost\left(t\right)\right),else\hfill \end{array}\right.\end{array}$$

(22)

$$\begin{array}{c}\hfill cost\left(t\right)=b_e\left(t\right)\frac{T_e\left(t\right)n_e\left(t\right)}{9550}\end{array}$$

(23)

where the $r_{init}=1$ is a constant. $\beta$ is a proportional factor used to balance the $SOC$ and fuel consumption efficiency. In the study of this paper, $\beta$ is taken as 15,000. $SO{C}_{ref}$ is reference $SOC$ value. $cost$ is the instantaneous fuel consumption, and $be$ is the effective specific fuel consumption.

The proposed EMS for PHEVs based on the SAC algorithm is shown in Figure 5. At each time step, the agent interacts with the environment to obtain samples $(s,a,r,s^{{}^{\prime }})$ that are stored in the replay buffer whose capacity is set to hold one million data items. During the learning process, a minibatch with 256 data items is randomly selected from the replay buffer which is used to solve the problem of data correlation and non-stationary distribution. In addition, the algorithm is able to learn from past experiences to increase data utilization and learning efficiency. The state vector s combined with the action a is used as input to Q networks. The Q networks output Q values, which are used to evaluate the value of taking input action under the input state vector. The next state vector $s^{{}^{\prime }}$ is used as input to the policy network to calculate the next action. The inputs of the target Q networks are $s^{{}^{\prime }}$ combined with the next action. Then, the Q value at the next moment can be obtained, which evaluates the value of taking the next action under the next state. Then, Q networks and policy networks are trained by gradient descent algorithm according to Equations (15) and (17), respectively. The $\alpha$ is learned by dual gradient descent according to Equation (18). Policy network, Q networks and target Q networks have only one hidden layer with 300 neurons, respectively. More details about the SAC algorithm have been shown in

Table 3. Parameters of SAC−AET.

Parameters	Value
discount factor	0.99
target smoothing coefficient	0.005
learning rate	0.0003
batch size	256
hidden size	300
replay size	1,000,000
entropy target	−3

.

4. Simulation and Discussion

The proposed SAC-based EMS is validated in this section. The convergence and performance of the proposed strategy are verified by a standard driving cycle. In addition, the performance of SAC−AET-based EMS is compared with that of the SAC fixed equilibrium parameter, DDPG strategy and ECMS. The adaptability of the proposed strategy is verified by the real driving cycle. The real driving cycle is obtained from the bus remote intelligent monitoring platform based on a mass of data, which is repeatedly collected from vehicle CAN as shown in Figure 6.

4.1. The Performance of SAC Algorithm-Based EMS for UDDS

The standard driving cycle Urban Dynamometer Driving schedule (UDDS) in Figure 7 is used for the learning process of the proposed method. We use two UDDS driving cycles to simulate a long-distance trip of PHEV. In this section, the performance of the SAC−AET for UDDS is explored. The reference SOCs are set to 35%, 40%, 45%, 50%, respectively, to validate the control effects for battery SOC. Figure 8 represents the trajectories of the reference SOC from 35% to 50% under UDDS driving cycle. We can see that the battery SOC curves decrease from initial value 0.7 to reference SOC, which shows that the proposed strategy can achieve the goal to keep the SOC to the various reference SOC. In order to analyze the changes of each part of the EMS in detail, we randomly select one of the four reference SOCs to explain, and here we will show the results when the reference SOC is 45%.

Figure 9 shows the rewards of the iterations, where the rewards have a large fluctuation at the beginning of the learning process. In the beginning, the RL agent cannot figure out a better decision, and the exploration strategy is used to randomly explore the action for the current state to obtain the cumulative reward information. As the number of iterations increases, the exploration strategy is gradually weakened, and the selected actions can bring higher rewards according to the current policy. Therefore, the reward stabilizes after about 23 iterations. It can be found that the reward becomes larger than the initial condition and eventually remains within a certain range.

Figure 10 shows the adjustment process of $\alpha$. The blue line is the loss of equilibrium parameter at each step throughout the training process, which shows that the $\alpha$ loss gradually fluctuates near zero. The $\alpha$ is adjusted and finally stabilized to a certain value, as the red line shows.

The EM power and SOC trajectory curve of the vehicle are shown in Figure 11. The EM not only acts as a power source but also charges the battery when the EM power is negative. In the SOC maintenance stage, which is the horizontal part of the yellow line, the probability of the negative power is larger to keep the SOC to reference SOC (0.45).

The strategy is trained for a total of 100 iterations and the change in SOC is observed every 10 iterations, as in Figure 12. It can be seen from the figure that the value of the SOC is the largest in the first iteration. This is because the parameters of the strategy are randomly initialized at the initial time. During the 10th iteration, we can see that the terminal of SOC will drop to 0.45. As the training time continues to increase, the terminal of SOC gets closer to the reference SOC (0.45). This further reflects the stability of the strategy and the ability to learn. The equivalent fuel consumption per 100 km for each iteration is shown in Figure 13. With the increase in training, the fuel consumption per 100 km tends to stabilize. As we can see from the graph, there are two places with the lowest fuel consumption, but in this case, the SOC of the battery is not controlled by the reference value. We simulated five times under this driving cycle and the average value of equivalent fuel consumption for 100 km is 24.42 L.

4.2. Comparison of Different Strategies

In this experiment, the optimality of SAC−AET-based EMS is compared with that of the SAC algorithm-based strategy with fixed equilibrium parameter, DDPG-based strategy and ECMS. The DDPG strategy is a deep reinforcement learning algorithm-based EMS that is highly sensitive to hyperparameters. In this section, the four strategies are simulated using real driving cycle c1, shown in Figure 14a.

The four EMSs are simulated using the same initial state, objective function, control variable and constraints.

Table 4. The comparison of fuel consumption in three strategies.

Algorithm	ICE Fuel Consumption (l/100 km)	Equivalent Fuel Consumption (l/100 km)	Saving Rate (%)	Final SOC
SAC (learned parameter)	9.4283	23.5767	4.37	0.29
DDPG	9.5372	23.9056	3.04	0.28
SAC (fixed parameter)	10.6914	24.5879	0.26	0.31
ECMS	10.9499	24.6541	-	0.31

shows the fuel consumption of the strategies. In order to reflect the stability of the proposed strategy, the results of the last five simulations were averaged to obtain the ICE fuel consumption and the equivalent fuel consumption. As we can see from the table, the SAC-AET has a fuel consumption of 9.4283 L/100 km, which is more economical compared to the other three strategies. It is also the smallest in terms of equivalent fuel consumption. The control effects of the DDPG strategy are close to that of the SAC strategy, but the efficiency distribution of ICE working points of the strategies according to Figure 15 shows that the proposed SAC algorithm-based EMS is able to explore a wider action space, which then facilitates the ICE to work in a more favorable operating zone for the strategy. It can be seen from the figure that most ICE operating points are located near a 200 (g/kWh) fuel consumption rate. In addition to the ECMS, the ICE operates in the 100–190 (g/kWh) range some of the time. Therefore, the SAC strategy saves more fuel. The comparison with the traditional SAC algorithm-based strategy is used to demonstrate that the introduction of automatic entropy adjustment not only improves the fuel economy, but also increases the adaptability and self-regulation capability of the strategy.

The SOC trajectories of the strategies are shown in Figure 16. For maximizing the use of electrical energy, we set the reference SOC to 0.3. In the figure, we can see that the SOC trajectories for the SAC-AET strategy and SAC with fixed equilibrium parameter strategy are close before 2400 s. After 2400 s, the SOC of the ECMS is the highest compared with others, and its final SOC is 0.3064. The final SOCs of the SAC-AET and DDPG strategies are 0.2908 and 0.2865, respectively.

4.3. The Adaptability of SAC Algorithm-Based EMS

Many of the existing methods require elaborate designs to adapt to different driving cycles. In order to verify the adaptability of SAC strategies to stochastic driving cycles, we use real-world driving cycle c2 including the city driving cycle, suburban driving cycle and rural driving cycle, shown in Figure 14b. The maximum velocity of the driving cycle is 67.25 km/h, and the average velocity is 29.36 km/h. The actual and desired vehicle-speed trajectories of the PHEV are shown in Figure 17. It can be seen from the figure that the reference speed of the real-world driving cycle is followed very well. The SOC trajectory of the real-world driving cycle c2 is shown in Figure 18. The demand power is relatively low in phase T1 and the SOC drops slowly. Phase T2 has a higher demand power and the SOC drops sharply, eventually to the reference SOC (0.3). Phase T3 enters the ICE charging process, keeping the SOC fluctuating around the reference SOC. Similarly, we simulated five times under this driving cycle. The average values of the ICE and equivalent fuel consumption for 100 km are 9.951 L and 18.056 L, respectively.

5. Conclusions

In this paper, an improved SAC algorithm-based EMS is proposed to improve the fuel economy of EMS. The two Q networks are used to solve the problem of overestimation of Q values. The equilibrium parameter, which is the most important parameter of SAC algorithm-based EMS, can be self-adjusted by learning. The simulation results show that the SOC is able to be maintained at the various reference value under a real-world driving cycle. Compared with the SAC strategy with fixed equilibrium parameters, the introduction of a self-learning equilibrium parameter can improve fuel economy. The fuel economy of the proposed SAC algorithm-based strategy gets 4.37% performance than ECMS. The control effects of the DDPG strategy are close to that of the proposed SAC strategy, but the proposed SAC algorithm-based EMS is able to explore a wider action space, which then facilitated the ICE to work in a more favorable operating zone for the strategy. The proposed SAC algorithm-based strategy is more adaptable to different driving cycles.