The Multi-Objective Optimization of Powertrain Design and Energy Management Strategy for Fuel Cell–Battery Electric Vehicle

Abstract

Considering the limited driving range and inconvenient energy replenishment way of battery electric vehicle, fuel cell electric vehicles (FC EVs) are taken as a promising way to meet the requirements for long-distance low-carbon driving. However, due to the limitation of FC power ability, a battery is usually adopted as the supplement power source to fill the gap between the requirement of driving and the serviceability of FC. In consequence, energy management is essential and crucial to an efficient power flow to the wheel. In this paper, a self-optimizing power matching strategy is proposed, considering the energy efficiency and battery degradation, via implementing a deep deterministic policy gradient. Based on the proposed strategy, less energy consumption and longer FC and battery life can be expected in FC EV powertrain with optimal hybridization degree.

1. Introduction

1.1. The Need for a Hybrid System

As a substantial effort of the carbon neutrality and environment protection promise around the world [1], transportation electrification roadmaps for different vehicle types are proposed with varying propelling energy [2,3]. Zero-emission battery electric vehicle (BEV) attracts great attention, especially in the passenger vehicle sector, with an acceptable driving range per charge and charging schedule at home or workplace in the spare time. However, current battery capacity and charging technologies are insufficient to meet the requirements of commercial vehicles, e.g., logistic vehicles, interstate buses, and garbage trucks, which require long-time, long-range traveling and account for around 7% of global carbon emission [4]. Thus, another green energy source is required to cover the hundred kilometers target range and refuel the energy source in minutes, consequently decarbonizing this transportation sector. Compared to the internal combustion engine (ICE)-based hybrid powertrain, hydrogen–chemical battery-based fuel cell hybrid electric vehicle (FC EV) could realize a longer driving range with lower energy consumption and zero carbon emission [5]. FC stack uses an expensive proton exchange membrane (PEM) to generate electricity [6]. The primary challenge for FC EV’s larger-scale commercialization is the cost, which is currently several times that of its ICE counterparts [7].

Although the FC shows an excellent energy density to meet the range requirements, the relatively weak dynamic performance, i.e., power variation, necessitates the request for a power-oriented energy source as the supplementary source. Since frequent and significant power fluctuation has a negative effect on FC service life, it is wise to adopt a battery to alleviate the peak power impulse by the outstanding dynamic performance of the battery. Furthermore, the chargeable battery offers the capability of energy recovery, which is missing in FC. Given the energy density of FC is significantly superior to that of a battery, FC generally provides the average power when the battery is responsible for the extreme power variation in climbing, overtaking, or launching, as an auxiliary power source. On the other hand, the opposite configuration has also been studied [8], in which the battery takes the primary responsibility to meet the driving power demand. The hybrid system can cover the high-power events and, at the same time, keep the FC working in an efficient zone [9]. Fuel cell and battery (FC–BAT) EV prefers to use a series-parallel powertrain structure to enable the FC to power the electric machine directly or charge the battery when extra power is available [10]. Thus, it is obvious that a larger battery is involved, and more improvement room is available for FC but at the expense of cost and electricity dependency. The ratio of maximum power capability of FC and battery, which is defined as the degree of hybridization (DoH) of a hybrid powertrain, plays a key role in FC EV overall performance. Ref. [11] studied the impact of DoHs on vehicle mass, ownership, and energy efficiency, however, lacking an optimized solution.

Huang et al. [12] offered an optimized solution for balancing the energy consumption and the dynamic performance via establishing a control model without the mention of cost. Since the frequent high-power discharge and charging current will deteriorate the battery health [13], the concept design of powertrain specification needs a comprehensive optimization considering the system cost, performance, and the service life period. Furthermore, as the role of the battery is to regulate the power flow in the system, the power matching strategy is another key role in energy efficiency and dynamic performance [14].

1.2. Review of EMS Development for FC EV

The objective of an ordinary energy management strategy (EMS) is to determine the energy flow path between different energy/power sources properly to minimize a quantity, e.g., fuel consumption [15]. However, the performance of FC and battery degrades with aging [16]. A competent and comprehensive EMS for FC EV should not only work to minimize fuel consumption but also prolong its lifetime [17].

Traditional EMS can be classified as model-based optimization and rule-based optimization. Each of them has its priority and compromises to achieve targeted performance. Model-based optimization is characterized as global or near-global optimum [18], computationally complex, time consuming, and difficult to apply in practice due to the requirement of prior knowledge. Rule-based EMS (RL EMS) is relatively easy, generally based on (i) experience, (ii) offline-optimized algorithm for individual components, (iii) optimized strategy for the integrated system in a certain state, e.g., cruising, accelerating, or hill climbing. The level of expertise, the accuracy of the models, and the ability of vehicle state identification all play important roles in RL EMS performance.

Regarding the RL EMS, Ettihir [19] developed an extremum-seeking process to trace the maximum power and maximum efficiency of FC considering the FC aging. Sharing a similar idea, Ghaderi1 et al. [20] proposed an online parameters identification model to tackle the EMS uncertainties owing to the performance drifts of the power sources. Davis [21] proposed a rule-based two-mode EMS for low-power and high-power events according to the total cost of ownership, which includes fuel consumption, FC, and battery lifetime degradation.

Regarding the optimization model-based EMS, Feng et al. [22] introduced the remaining useful life models for FC and battery regarding performance degradation and then adopted an adaptive balanced equivalent consumption minimization strategy (BCMS) to minimize the overall cost considering the hydrogen consumption, FC, and battery degradation. Mend et al. [23] designed a dual-mode ECMS for FC EV by establishing time-varying models of hydrogen consumption and FC efficiency considering the fuel cell degradation rate.

Artificial intelligence technologies are being applied more and more in vehicle-related control problems in recent years. Specific to EMS, machine learning, e.g., neural network, Q-learning, deep Q-learning network, attracts great attention. The agent (strategy) generated via massive training under certain policy requires no prior knowledge of the upcoming driving conditions, at the same time reaching near-optimal results compared to DP. Zhou et al. [24] adopted a reinforcement learning algorithm in a long-term energy management strategy to prolong FC life, but energy consumption is missing in the optimization target. Meng [25] proposed a double Q-learning-based energy management strategy for the overall energy consumption optimization of FC EV. Yavasoglu et al. [26] proposed a neural network (NN)-based machine-learning algorithm for an FC-, battery-, and supercapacitor-based dual-motor powertrain to properly split the power in propelling machines and hybrid energy storage system [27], which has a negative effect on practicality and real-time performance, while the deep deterministic policy gradient (DDPG) turns discrete outputs into continuous outputs by adding two extra layers of networks to DQN. In this paper, aiming at improving the equivalent hydrogen consumption, prolonging FC and battery service life, and reducing the cost of ownership, DDPG is adopted and modified, accordingly, to optimize EMS.

2. Optimization of FC–Battery Powertrain Configurations

2.1. Structure and Specifications

The FC EV powertrain, as shown in Figure 1, is selected, which is propelled by a single motor via fixed-ratio transmission mounted on the front axle to evaluate the equivalent hydrogen consumption, FC, and battery performance degradation in different degrees of system hybridization and EMS. Figure 1 shows the powertrain structure [28].

The aim of the concept design for the selected vehicle with specifications shown in

Table 1. Vehicle specifications.

Parameter	Value	Unit
Front area	2.18	m2
Aerodynamic drag coefficient	0.32	N/A
Coefficient of rolling resistance	0.0105	N/A
Equipped mass	1750	kg
Correction coefficient of rotating mass	1.05	N/A
Acc performance (0–100 km/h)	10	s
Hill-climbing capability	30@30 km/h	° (degree)
Top speed	120	km/h

is to be able to run at 120 km/h, complete 0–100 km/h acceleration in 10 s, and climb 30% hill at 30 km/h.

2.2. Electric Machine

According to the target performance in Table 1, the required motor power is calculated by Equation (1).

$$\{\begin{array}{c}{\mathrm{P}}_{\mathrm{s}\mathrm{p}\mathrm{d}}=\frac{{\mathrm{v}}_{\mathrm{top}}}{3600{\mathsf{\eta }}_{\mathrm{T}}}\left(\mathrm{M}\mathrm{g}\mathrm{f}+\frac{{\mathrm{C}}_{\mathrm{d}}{\mathrm{A}}_{\mathrm{f}}{\mathrm{v}}_{\mathrm{t}\mathrm{o}\mathrm{p}}^2}{21.15}\right)\\ {\mathrm{P}}_{\mathsf{\phi }}=\frac{{\mathrm{v}}_{\mathrm{i}}}{3600{\mathsf{\eta }}_{\mathrm{T}}}\left(\mathrm{M}\mathrm{g}\mathrm{f}\cos {\mathsf{\phi }}_{\max }+\mathrm{m}\mathrm{g}\sin {\mathsf{\phi }}_{\max }+\frac{{\mathrm{C}}_{\mathrm{d}}{\mathrm{A}}_{\mathrm{f}}{\mathrm{v}}_{\mathrm{i}}^2}{21.15}\right)\\ {\mathrm{P}}_{\mathrm{a}\mathrm{c}\mathrm{c}}=\frac{{\mathrm{v}}_{\mathrm{a}}}{3600{\mathrm{t}}_{\mathrm{a}}{\mathsf{\eta }}_{\mathrm{T}}}\left(\mathsf{\delta }\mathrm{M}\frac{{\mathrm{v}}_{\mathrm{a}}}{2\sqrt{{\mathrm{t}}_{\mathrm{a}}}}+\mathrm{M}\mathrm{g}\mathrm{f}\frac{1}{1.5}{\mathrm{t}}_{\mathrm{a}}+\frac{{\mathrm{C}}_{\mathrm{d}}{\mathrm{A}}_{\mathrm{f}}{\mathrm{v}}_{\mathrm{a}}^2}{21.15\times 2.5}{\mathrm{t}}_{\mathrm{a}}\right)\\ {\mathrm{P}}_{\mathrm{m},\mathrm{p}\mathrm{e}\mathrm{a}\mathrm{k}}\ge \max \left[{\mathrm{P}}_{\mathrm{s}\mathrm{p}\mathrm{d}},{\mathrm{P}}_{\mathsf{\phi }},{\mathrm{P}}_{\mathrm{a}\mathrm{c}\mathrm{c}}\right]\end{array}$$

(1)

where g, f, C_d, $\mathsf{\delta }$, M, A_f, φ_max, and η_T are gravitational acceleration, rolling resistance coefficient, aerodynamic drag coefficient, the equivalent mass of rotating parts, vehicle mass and frontal area, maximum climbing angle, energy efficiency from the electric machine to wheels, respectively; P_spd, P_φ, and P_acc are required power for cruising at top speed v_top, climbing maximum grade with speed v_i and accelerate the vehicle to v_a in t_a. The rest of the performance concept design, e.g., maximum and rated torque, maximum speed, will be determined by top speed, acceleration, and gradeability, as well as being crossed checked subsequently. The electric machine specification is presented in

Table 2. Specification of selected electric machine.

Parameter	Value	Unit
Rated power	50	kW
Peak power	120	kW
Rated speed	5500	rpm
Max speed	14,000	rpm
Rated torque	90	Nm
Max torque	215	Nm

.

According to the requirements in Table 2, a permanent magnet synchronous motor is selected, whose efficiency map is shown in Figure 2.

2.3. Fuel Cell Model

FC operation is usually explained as a steady-state energy conversion process where hydrogen is supplied to the anode and oxygen to the cathode. It is possible for FC stack to last over 25,000 h lifetimes in steady-state benign conditions, while some conditions may significantly degrade FC, e.g., load-cycling, stop–start, high-power, and low current (idling) conditions [29]. Results found in an accelerated lifetime testing of an FC stack from a bus [30] showed that 56% of FC degradation was due to load cycling and 33% due to stop–start operating. In simple terms, fewer start–stop cycles and power fluctuations will benefit FC lifetime. Considering all FC degradation rates formulas are experienced and testing cycles depended on, the power fluctuation rate is selected to refer to the FC service time degradation; at the same time, an extra penalty factor is applied to suppress frequent start–stop in FC. The following FC degradation formula is adopted from Ref [21]:

$${\mathsf{\Delta }}_{\mathrm{FC}}{\left(\mathrm{t}\right)=\mathsf{\Delta }}_{\mathrm{Cycle}}{\mathrm{N}}_{\mathrm{Cycle}}+{\displaystyle {\int }_0^{\mathrm{t}}\mathsf{\delta }\left(\mathrm{t}\right)}$$

(2)

The degradation function concerning operating power is

$$\mathsf{\delta }\left(\mathrm{t}\right)=\frac{{\mathsf{\delta }}_0}{3600}\left(1+\frac{\mathrm{\alpha }}{{\mathrm{P}}_{\mathrm{F}\mathrm{C}}{}^2}{\left[{\mathrm{P}}_{\mathrm{F}\mathrm{C}}\left(\mathrm{t}\right)-{\mathrm{P}}_{\mathrm{F}\mathrm{C}\_\mathrm{r}\mathrm{a}\mathrm{t}\mathrm{e}}\right]}^2\right)$$

(3)

Δ_Cycle represents the FC degradation coefficient in one start–stop cycle; N_Cycle is the number of cycles; δ₀ and α are empirical coefficients of FC operating-power degradation; P_{FC_rate} is the rated power of the FC. All parameters of FC are summarized in

Table 3. FC specifications.

Parameter	Variable	Value	Unit
Cell numbers	N	80
Peak power	Pfc,Max	45	kW
Peak current	Ist,Max	300	A
Stack mass	FCMass	13.1	kg

.

The operating efficiency of FC is determined by the normalized power, which is the ratio of output to peak value, shown in Figure 3.

Then, the instantaneous hydrogen consumption of FC can be derived by Equation (4).

$$\stackrel{•}{{\mathrm{M}}_{\mathrm{H}2-\mathrm{f}\mathrm{c}}}=\frac{{\mathrm{P}}_{\mathrm{F}\mathrm{C}}}{{\mathrm{H}}_{\mathrm{f}}\ast {\mathsf{\eta }}_{\mathrm{F}\mathrm{C}}(\mathrm{n}\mathrm{o}\mathrm{r}\mathrm{m}\mathrm{o}\mathrm{l}\mathrm{i}\mathrm{z}\mathrm{e}\mathrm{d}\hspace{0.17em}\mathrm{F}\mathrm{C}\hspace{0.17em}\mathrm{p}\mathrm{o}\mathrm{w}\mathrm{e}\mathrm{r})}$$

(4)

H_f is the hydrogen lower heating value 120 MJ/kg; η_FC is FC efficiency.

2.4. Battery Model

Since the battery integrated into this hybrid powertrain is the supplement power source, lithium-ion-based electrochemical battery is selected in this study due to its extraordinary energy density and acceptable ownership. Considering the focus of this study is system configuration and energy flow management, rather than the dynamic performance of the battery, the Li-ion battery model can be expressed by an ideal voltage source and a resistor, as shown in Figure 4.

The motor model specifies the HESS’s power requirements; thus, the maps of the motor’s driving and recovered states are measured in trials for modeling. The motor efficiency map is shown in Figure 2.

The output current is determined by open-circuit voltage U₀, internal resistance R_BAT and battery power P_BAT:

$${\mathrm{I}}_{\mathrm{B}\mathrm{A}\mathrm{T}}=\frac{{\mathrm{U}}_0-\sqrt{{\mathrm{U}}_0{}^2-4{\mathrm{R}}_{\mathrm{B}\mathrm{A}\mathrm{T}}{\mathrm{P}}_{\mathrm{B}\mathrm{A}\mathrm{T}}}}{2{\mathrm{R}}_{\mathrm{B}\mathrm{A}\mathrm{T}}}$$

(5)

The state-of-charge (SoC) is expressed by Equation (6) with output current and total capacity Q_BAT:

$$\mathrm{S}\mathrm{o}\mathrm{C}=100\%-\frac{{\displaystyle {\int }_{{\mathrm{t}}_0}^{\mathrm{t}}{\mathrm{I}}_{\mathrm{B}\mathrm{A}\mathrm{T}}}}{{\mathrm{Q}}_{\mathrm{B}\mathrm{A}\mathrm{T}}(\mathrm{A}\mathrm{h})}$$

(6)

The charging/discharging current, as the primary factor, with ambient temperature and natural aging have an influence on the degradation rate of battery capacity [31]. Since DoH determines the power ratio of battery and FC, it defines the capability boundary of current regulating and then plays a role in battery aging. An Arrhenius-based numerical model [32] with four parameters is adopted to investigate the influence of DoH on battery capacity degradation service life:

$$\mathrm{B}\mathrm{a}{\mathrm{t}}_{\mathrm{l}\mathrm{o}\mathrm{s}\mathrm{s}}=\mathrm{A}{\mathrm{e}}^{-\left(\frac{{\mathrm{E}}_{\mathrm{a}}+\mathrm{B}\times {\mathrm{C}}_{\mathrm{R}\mathrm{a}\mathrm{t}\mathrm{e}}}{\mathrm{R}{\mathrm{T}}_{\mathrm{b}\mathrm{a}\mathrm{t}}}\right)}{(\mathrm{A}{\mathrm{h}}_{\mathrm{t}\mathrm{h}\mathrm{r}\mathrm{o}\mathrm{u}\mathrm{g}\mathrm{h}})}^{\mathrm{z}}=0.0032{\mathrm{e}}^{-\left(\frac{15162-1516{\mathrm{C}}_{\mathrm{R}\mathrm{a}\mathrm{t}\mathrm{e}})}{\mathrm{R}{\mathrm{T}}_{\mathrm{b}\mathrm{a}\mathrm{t}}}\right)}{(\mathrm{A}{\mathrm{h}}_{\mathrm{t}\mathrm{h}\mathrm{r}\mathrm{o}\mathrm{u}\mathrm{g}\mathrm{h}})}^{0.824}$$

(7)

where A and B are pre-defined experience parameters and compensation factors for discharging rate C_Rate, respectively; E_a and R stand for the activation energy and constant gas values, which equal 78.06 (J) and 8.314 J/mol/K⁻¹, respectively; T_bat represents the absolute temperature in Kelvin; the accumulative current going through the battery is represented by ${\mathrm{Ah}}_{\mathrm{through}}$, which is a time-related exponential factor. Then, the model was evaluated in a field testing of Li-ion battery and achieved the following experimental calibration [33]:

2.5. System Configurations

As discussed in the literature review Section 1.1, there are two different schemes for FC EV, where battery and FC work as the primary energy source, respectively. The varying DoH reflects the different powertrain characteristics:

$$\mathrm{D}\mathrm{o}\mathrm{H}=\frac{{\mathrm{P}}_{\mathrm{f}\mathrm{c},\mathrm{M}\mathrm{a}\mathrm{x}}}{{\mathrm{P}}_{\mathrm{f}\mathrm{c},\mathrm{M}\mathrm{a}\mathrm{x}}+{\mathrm{P}}_{\mathrm{b}\mathrm{a}\mathrm{t},\mathrm{M}\mathrm{a}\mathrm{x}}}$$

(8)

where P_fc,Max and P_bat,Max are the maximum power of the FC and battery. If DoH is relatively low, most of the required power will be fed by the battery, leading to a long electric range compared to that, which FC takes the primary responsibility for in propelling. If DoH is relatively high, FC is responsible for driving the vehicle most of the time, while the battery provides power when FC is underpowered or required to alleviate the power fluctuation in FC to prolong its service time.

In this study, FC–BAT hybrid powertrains with different DoH will be examined to investigate the energy consumption, ownership, and performance degradation of each scheme. Referring to Ref. [34], 0.375 is selected as the initial threshold of DoH for searching for the optimal value to achieve a pre-defined target. When DoH > 0.375, FC takes the primary responsibility for providing power to the electric machine, which leads to an energy-oriented battery model. When DoH < 0.375, FC works as a secondary power bank to charge the battery and reduce the harm of significant current fluctuation to the battery, which leads to a power-oriented battery model. Given the different batteries play in two powertrain schemes, the concept design for the battery will be different. One of them should be power density oriented, while the other one should be energy density oriented. The specifications of the two battery models are shown in

Table 4. Specifications of two battery types.

Battery Parameters	DoH < Threshold	DoH > Threshold
Energy density	0.156 kWh/kg	0.156 kWh/kg
Power density	2.4 kW/kg	0.96 kW/kg
Weight	31.25 kg	125 kg
Voltage	375 V	375 V

.

The FC system cost varies dramatically depending on the volume and manufacturer, from 1000 to 2000 USD/kW [35]. In this study, the average price, i.e., USD 1000/kW, is adopted to calculate the cost of the FC system:

$$\mathrm{C}\mathrm{o}\mathrm{s}{\mathrm{t}}_{\mathrm{F}\mathrm{C}}={\mathrm{P}}_{\mathrm{f}\mathrm{c},\mathrm{M}\mathrm{a}\mathrm{x}}\ast 1000\left(\mathrm{\$}/\mathrm{k}\mathrm{w}\right)$$

According to the latest annual survey by BloombergNEF, the mean price of a battery pack in the current market is 137 USD/kWh. Then:

When the power-density-oriented battery (DoH > 0.375) is selected

$$\mathrm{C}\mathrm{o}\mathrm{s}{\mathrm{t}}_{\mathrm{B}\mathrm{a}\mathrm{t}1}=0.156\ast 125\ast 137=2671\hspace{0.17em}\hspace{0.17em}(\mathrm{U}\mathrm{S}\mathrm{D})$$

When the energy-density-oriented battery (DoH < 0.375) is selected

$$\mathrm{C}\mathrm{o}\mathrm{s}{\mathrm{t}}_{\mathrm{B}\mathrm{a}\mathrm{t}2}=0.156\ast 31.25\ast 137=668\hspace{0.17em}\hspace{0.17em}(\mathrm{U}\mathrm{S}\mathrm{D})$$

Since the FC provides all the energy, including the electricity in the battery, the consumed electricity can be converted into hydrogen consumption via Equation (9), M_EtoH:

$$\stackrel{•}{{\mathrm{M}}_{\mathrm{H}2-\mathrm{E}\mathrm{l}\mathrm{e}\mathrm{c}\mathrm{t}\mathrm{r}\mathrm{i}\mathrm{c}\mathrm{i}\mathrm{t}\mathrm{y}}}=\frac{{\mathrm{P}}_{\mathrm{B}\mathrm{a}\mathrm{t}}}{{\mathsf{\eta }}_{\mathrm{d}\mathrm{i}\mathrm{s}}\ast {\mathsf{\eta }}_{\mathrm{c}\mathrm{h}\mathrm{g}}\ast {\mathrm{H}}_{\mathrm{f}}\ast {\mathsf{\eta }}_{\mathrm{F}\mathrm{C}}(\mathrm{n}\mathrm{o}\mathrm{r}\mathrm{m}\mathrm{o}\mathrm{l}\mathrm{i}\mathrm{z}\mathrm{e}\mathrm{d}\hspace{0.17em}\mathrm{F}\mathrm{C}\hspace{0.17em}\mathrm{p}\mathrm{o}\mathrm{w}\mathrm{e}\mathrm{r})}$$

(9)

M_{H2-Electricity} is the equivalent energy from electricity to hydrogen consumption of electricity, η_dis, and η_chg are the efficiency of discharging and charging, respectively.

3. Optimum Design of Hybrid Powertrain

3.1. Method

The configuration of the FC–BAT hybrid powertrain defines the characteristics of each component, performance limit, and degradation. Thus, an optimum design of a hybrid powertrain should balance fuel consumption, system ownership, and performance degradation. The particle swarm optimization (PSO) algorithm, which is inspired by the social behavior of a group of birds, is adopted in this study to realize a multi-objective optimization. By implementing PSO, the target to be optimized is the “particle”, which is iteratively updated through its position (optimal value of target) and the rate of position changing. In the beginning, PSO randomly initializes the particle (optimization objective) position and the velocity of moving to the next position. After reaching the next position, the best position of each particle (p_best) and the best position of all particles (p_best) will be updated, followed by the updates of velocity. The searching process for the local and global optimal values is heuristic and will not stop until reaching the pre-defined constraints. In each iteration, the position, which represents the best fitness value, is obtained, as shown in Equation (10).

$$\{\begin{array}{c}{\mathrm{V}}_{\mathrm{i}\mathrm{j}}^{\mathrm{N}}=\mathrm{\varpi }\cdot {\mathrm{V}}_{\mathrm{i}\mathrm{j}}^{\mathrm{N}-1}+{\mathrm{c}}_1{\mathrm{r}}_1\left({\mathrm{p}}_{-}\mathrm{b}\mathrm{e}\mathrm{s}{\mathrm{t}}_{\mathrm{i}\mathrm{j}}^{\mathrm{N}-1}-{\mathrm{X}}_{\mathrm{i}\mathrm{j}}^{\mathrm{N}-1}\right)+{\mathrm{c}}_2{\mathrm{r}}_2\left({\mathrm{g}}_{-}\mathrm{b}\mathrm{e}\mathrm{s}{\mathrm{t}}_{\mathrm{i}\mathrm{j}}^{\mathrm{N}-1}-{\mathrm{X}}_{\mathrm{i}\mathrm{j}}^{\mathrm{N}-1}\right)\\ {\mathrm{X}}_{\mathrm{i}\mathrm{j}}^{\mathrm{N}}={\mathrm{X}}_{\mathrm{i}\mathrm{j}}^{\mathrm{N}-1}+{\mathrm{V}}_{\mathrm{i}\mathrm{j}}^{\mathrm{N}}\end{array}$$

(10)

where V_ij^N and X_ij^N are the velocity and position of the ith particle with jth dimension in Nth iteration; N stands for the iteration number; ϖ represents the inertial weight of velocity; c₁ and c₂ are learning factors; r₁ and r₂ are random numbers between 0 and 1.

In this study, the goal is to achieve an acceptable result of balancing equivalent hydrogen consumption, system cost, and performance degradation in hybrid powertrain design via DoH optimization. Therefore, the optimization objective consists of three items, i.e., energy consumption, cost, and degradation, with the optimization variable DoH.

Considering the powertrain design is a multi-objective optimization process, reference values are necessary for each item in the optimization function, i.e., hydrogen consumption, system cost, and performance degradation. It needs to be emphasized that the reference values are NOT the goal of optimization; they are a benchmark for further improvement with different preferences by tuning the factors, i.e., a₁, a₂, a₃, to achieve a specific goal. Thus, specifications of the hybrid powertrain with DoH = 0.5 are selected at the very beginning. In addition, 1.2 kg/100 km hydrogen consumption and 0.0377/100 km capacity degradation of battery are adopted based on state-of-the-art situation.

$$\mathrm{R}\mathrm{e}\mathrm{f}={\mathrm{a}}_1\cdot \frac{{\mathrm{C}}_{\mathrm{H}2}}{1.2}+{\mathrm{a}}_2\cdot \frac{\mathrm{C}\mathrm{o}\mathrm{s}{\mathrm{t}}_{\mathrm{F}\mathrm{C}}+\mathrm{C}\mathrm{o}\mathrm{s}{\mathrm{t}}_{\mathrm{B}\mathrm{a}\mathrm{t}}}{120\ast 0.5\ast 1000+2671}+{\mathrm{a}}_3\cdot \frac{{\mathrm{Q}}_{\mathrm{l}\mathrm{o}\mathrm{s}\mathrm{s}}}{0.0377}$$

(11)

where C_H2 is the hydrogen consumption, including the energy used to generate electricity; Cost_FC is the cost of FC in terms of power; Cost_Bat is the cost of the battery in terms of energy capacity; are the power bank cost; Q_loss is the battery degradation; weighting factors a₁, a₂ and a₃ are adopted to set the real optimization goal that reflects the preference of the designer. In this study, each group of weighing factors for the balanced energy consumption/cost/performance/degradation preferred is investigated.

The following constraints are applied in the optimization algorithm:

$$\{\begin{array}{c}{\mathrm{v}}_{\min }\le \mathrm{v}\le {\mathrm{v}}_{\max }\\ 0\le \mathrm{P}\mathrm{o}{\mathrm{s}}_{\mathrm{D}\mathrm{o}\mathrm{H}}\le 1\end{array}$$

(12)

The optimum position (DoH) searching rate of each particle is constrained between [v_min, v_max], specifically, [−0.05, 0.05] in this study; Pos__DoH stands for the position of each particle, which is DoH in this study.

A typical PSO process, which is illustrated in Algorithm 1, is adopted to find the best concept design with different preferred objective. The following procedures are realized in a script in Matlab^®:

Step 1: Initialize the optimum searching rate v and fitness value standing of objective Pos__DoH for each particle.

Step 2: Find the best Pos__DoH of each particle (p_best) and the best Pos__DoH of all particles (g_best).

Step 3: Update v and p_best of each particle, check if the values meet the constraints of the speed and position.

Step 4: Update p_best and g_best.

Step 5: Repeat Steps 3~4 until the constraints or iteration number are reached.

Substitute Equation (9) to Equation (4) to calculate the overall hydrogen consumption, i.e., M_{H2_Total}:

$${\mathrm{M}}_{\mathrm{H}2\_\mathrm{T}\mathrm{o}\mathrm{t}\mathrm{a}\mathrm{l}}=\int ({\dot{\mathrm{M}}}_{\mathrm{H}2-\mathrm{f}\mathrm{c}}+{\dot{\mathrm{M}}}_{\mathrm{H}2-\mathrm{E}\mathrm{l}\mathrm{e}\mathrm{c}\mathrm{t}\mathrm{r}\mathrm{i}\mathrm{c}\mathrm{i}\mathrm{t}\mathrm{y}})\mathrm{d}\mathrm{t}$$

(13)

The pseudocode of PSO for powertrain sizing is realized in a script in Matlab^® and described in Algorithm 1.

Algorithm 1 PSO-based multi-objective DoH optimization

1: for each particle i

2:   for each dimension j

3:     Initialize velocity Vij and position Xij for particle i

4:     Calculate the fitness value fit(Xij) and set p_bestij = Xij,

5:   end for

6: end for

7: Choose the particle having the best fitness value as the g_bestj

8: for iteration N = 2, M do

9: for each particle i

10:    for each dimension j

11:     Updata the velocity of particle i:

12:     $V_{ij}^N=\varpi \cdot V_{ij}^{N-1}+c_1{r}_1\left(p\_bes{t}_{ij}^{N-1}-X_{ij}^{N-1}\right)+c_2{r}_2\left(g\_bes{t}_j^{N-1}-X_{ij}^{N-1}\right)$

13:     Updata the position of particle i:

14:     $X_{ij}^N=X_{ij}^{N-1}+V_{ij}^N$

15:   end for

16:   if $\mathrm{fit}(X_{ij}^N$$) < \mathrm{fit}(p\_bes{t}_{ij}^{N-1}$)

17:    $p\_bes{t}_{ij}^{N-1}$$=X_{ij}^N$

18:   end if

19:   if $\mathrm{fit}(p\_bes{t}_{ij}^{N-1}$$) < \mathrm{fit}(g\_bes{t}_j^{N-1}$)

20:    $g\_bes{t}_j^{N-1}$$=p\_bes{t}_{ij}^{N-1}$

21:   end if

22:  end for

23: end for

24: print the last g_best value

3.2. Concept Design Optimization Results

To investigate the effectiveness of PSO for FC–BAT hybrid powertrain design, several groups of weighting factors with different preferences in hydrogen consumption, system cost, and performance degradation are implemented in an algorithm to find the optimum DoHs. The first group of weighting factors a₁ = 0.34, a₂ = 0.33, a₃ = 0.33 are applied to 0 < DoH < 0.375, which indicates no preference for any item in the objective function. The corresponding optimized DoH is 0.2016 with 24.19 kW maximum fuel cell power; if the objective function prefers to extend battery lifetime by applying weighting factors a₁ = 0.2, a₂ = 0.2, a₃ = 0.6, the final DoH is 0.2323 with 0.6703 fitness value. The third group of weighting factors, i.e., a₁ = 0.2, a₂ = 0.6, a₃ = 0.2 is adopted to reduce system cost, which needs the DoH to be 0.0347 and leads to 4.16 kW maximum fuel cell power. If hydrogen consumption is preferred via applying weighting factors a₁ = 0.6, a₂ = 0.2, a₃ = 0.2 to the objective function, the optimum DoH is 0.3493, and the corresponding Max power of FC is 41.92 kW. The process of optimization and convergence of each preference is shown in Figure 5.

The same four groups of weighting factors are applied for 0.375 < DoH < 1. Figure 6 illustrates the fitness value of each optimization process. Similar to Figure 5, the fitness value converges quickly in several iterations.

The objective optimization fitness value and the corresponding optimal value are summarized in

Table 5. Optimization results of the DoH.

DoH Range	Weight Coefficients	Optimal DoH	Fitness Value
[0, 0.375]	a1 = 0.34, a2 = 0.33, a3 = 0.33	0.2016	1.1088
	a1 = 0.2, a2 = 0.2, a3 = 0.6	0.2323	0.6703
	a1 = 0.2, a2 = 0.6, a3 = 0.2	0.0347	0.7399
	a1 = 0.6, a2 = 0.2, a3 = 0.2	0.3493	1.711
[0.375, 1]	a1 = 0.34, a2 = 0.33, a3 = 0.33	0.3755	1.1337
	a1 = 0.2, a2 = 0.2, a3 = 0.6	0.3821	0.6794
	a1 = 0.2, a2 = 0.6, a3 = 0.2	0.3750	0.8507
	a1 = 0.6, a2 = 0.2, a3 = 0.2	0.3750	1.7060

. Relatively high DoH is beneficial in preventing battery degradation and reducing hydrogen consumption, demonstrating higher DoH preference. Relatively low DoH is good for reducing system cost, which shows lower DoH preference in two different ranges; for the FC hybrid powertrain involved in a high-energy density battery, a relatively high DoH should balance cost, fuel consumption, and battery degradation prevention needs; for the high battery density FC hybrid powertrain involved, the trend is reversed.

4. RL EMS

The EMS in a traditional engine–motor-based hybrid powertrain is only designed to achieve optimum energy consumption by regulating the power flow between different power sources. However, EMS in FC EV is not only responsible for reducing the energy consumption but is also expected to prolong the service time of battery and fuel cell. Furthermore, the solution performance of the multi-objective optimization problem is subjected to real-world driving conditions, as it is sensitive to the unknown power requirement.

Traditional EMS can be classified as rule based and optimization theory based. Although the advantages they have are different, neither of them achieves a good balance between performance and practicality. Deep deterministic policy gradient (DDPG) is a deep-reinforcement learning method based on an actor–critic architecture network. DDPG is good at solving continuous state/action optimization problems by updating the network parameters in the back propagation of the gradient. A memory pool that stores the previous results can accelerate calculation and avoid overfitting by weakening the correlation of training data. The reward function plays a key role in the reinforcement learning model, which includes the current and future rewards. A discount factor will be applied to the future reward to reflect the influence of the future on the current [35].

As the name of DDPG suggests, the deterministic policy is adopted to select the action for the network (agent). Specific to the EMS in this study, the SoC of the battery and required power is selected as a state variable, while the ratio of FC output power and the total required power is selected as an action variable to optimize EMS by training the agent under various driving conditions. Then, the reward and state of the next step are returned after the action is applied to the environment (vehicle). To prompt the agent to explore a better solution in the environment, a normally distributed noise is applied in action. Then, the action of the agent can be expressed as

$$\mathrm{A}={\mathsf{\pi }}_{\mathsf{\theta }}(\mathrm{s},\mathrm{a})+\mathcal{N}$$

(14)

The reward function consists of three parts, i.e., the equivalent hydrogen consumption, battery capacity loss, and FC performance degradation in terms of efficiency. The reward function is expressed in Equation (15):

$${\mathrm{R}}_{\mathrm{t}}=\sum _{\mathrm{i}=\mathrm{t}}^{\mathrm{T}}{\mathsf{\gamma }}^{\mathrm{i}-\mathrm{t}}{\mathrm{r}}_{\mathrm{t}}$$

(15)

where R_t is the total rewards, γ is the discount factor to introduce the influence of the future on the current, and rt represents the current reward.

Since FC tends to achieve longer service time by keeping the generated power around the rated value, as shown in Equation (3), the performance degradation coefficient at constant rated power is selected as the target optimum value of FC degradation. The target optimum value of battery capacity degradation is determined by Equation (9), where Ah-throughput in a testing cycle is based on 0.5 DoH. The target equivalent hydrogen consumption is 1.2 kg/100 km. The instant reward consists of equivalent hydrogen consumption, battery capacity loss, and FC performance degradation coefficient and is expressed by Equation (16).

$${\mathrm{r}}_{\mathrm{t}}=-1\ast \frac{\left({\mathrm{a}}_1\times \frac{{\mathrm{C}}_{{\mathrm{H}}_2}}{1.2}+{\mathrm{a}}_2\times \frac{{\mathrm{Q}}_{\mathrm{loss}}}{0.037}+{\mathrm{a}}_3\times \frac{\mathrm{F}{\mathrm{C}}_{\mathrm{loss}}}{2.664\times {10}^{-4}}\right)}{20}$$

(16)

where C_H2 is instant equivalent hydrogen consumption per 100 km; Q_loss is the capacity loss of battery at t; FC_loss is the performance degradation coefficient at t; ${\mathrm{a}}_1$, ${\mathrm{a}}_2$, ${\mathrm{a}}_3$ are weighting factors.

The parameters are updated via the loss function of the actor network and critic network, as shown in Equations (17) and (18).

$$\{\begin{array}{l}\mathrm{J}(\mathsf{\varphi })={\min }_{\mathsf{\varphi }}{\mathrm{E}}_{\mathsf{\pi }}\left[\frac{1}{2}{\left({\mathrm{y}}_{\mathrm{t}}-{\mathrm{Q}}_{\mathsf{\varphi }}\left({\mathrm{s}}_{\mathrm{t}},{\mathrm{a}}_{\mathrm{t}}\right)\right)}^2\right]\hfill \\ \mathrm{J}(\mathsf{\theta })={\max }_{\mathsf{\theta }}{\mathrm{E}}_{\mathsf{\pi }}\left[{\mathrm{Q}}_{\mathsf{\varphi }}\left({\mathrm{s}}_{\mathrm{t}},\mathrm{u}\left({\mathrm{s}}_{\mathrm{t}}\right)\right)\right]\hfill \end{array}$$

(17)

$${\mathrm{y}}_{\mathrm{t}}={\mathrm{r}}_{\mathrm{t}}+\mathsf{\gamma }{\mathrm{Q}}_{\mathsf{\varphi }\prime }\left({\mathrm{s}}_{\mathrm{t}+1},{\mathrm{a}}_{\mathrm{t}+1}\right)$$

(18)

where S_t, a_t are the state and action at time t, respectively; Q_ϕ is the output function of critic network at time t, while Q_ϕ, is the output function of critic network at time t + 1.

Only part of the parameters of the critic–target network and the actor–target network will be updated, which is called “Soft-updating” and used to avoid “over-estimating” and stabilize the learning process of the agent to achieve optimal EMS. The parameters of the target network are updated by Equation (19).

$$\{\begin{array}{l}{\mathsf{\varphi }}^{\prime }\leftarrow \mathsf{\tau }\mathsf{\varphi }+(1-\mathsf{\tau }){\mathsf{\varphi }}^{\prime }\\ {\mathsf{\theta }}^{\prime }\leftarrow \mathsf{\tau }\mathsf{\theta }+(1-\mathsf{\tau }){\mathsf{\theta }}^{\prime }\end{array}$$

(19)

ϕ and ϕ′ are parameters of the critic network and the critic–target network; θ and θ′ are parameters of the actor network and the actor–target network; τ is the smoothing factor. The DDPG-based EMS optimization procedures are shown in Figure 7.

Since DoH = 0.375 is the optimal value for a balanced target between hydrogen consumption, cost, and battery life, as shown in

Table 6. Results comparison of DDPG-based EMS agent with different weighting factors in WLTC.

Weighting Factors (a1,a2,a3)	Equivalent Hydrogen Consumption	Battery Capacity Degradation	FC Performance Degradation
0.34–0.3–0.33	0.558 kg	0.0126	0.0073
0.2–0.2–0.6	0.629 kg	0.0142	0.0068

, it is adopted to evaluate the performance of RL-based EMS. Figure 8 illustrates the training process for 300 rounds with weighting factors a₁ = 0.34, a₂ = 0.33, a₃ = 0.33 for hydrogen consumption, battery, and FC degradation, respectively, in the worldwide harmonized light-duty test cycle (WLTC). The total reward of each training episode converges to the highest reward in the end, which is usually taken as a successful training process for an optimal agent. The DDPG-based optimization of EMS is optimized by Matlab^® as well. The agent of optimal EMS is derived from 300 training episodes with converged reward; specifically, the 200th round is selected, as shown in Figure 8.

In this particular episode, the power distribution between the FC and battery is shown in Figure 9. To provide a better readership of the power allocating process, a partial enlarged picture in shown in Figure 9b, in which the power burden on FC was reduced by battery in a cooperation mode, the battery recapturing the energy in vehicle braking, and the battery was charged by FC when extra power was available, as clearly demonstrated. It is worth noting that FC reaches its peak power, i.e., 45 kW, several times with the balanced preferences weighting factors.

The weighting factors of hydrogen consumption, battery capacity degradation, and FC performance degradation are changed to a₁ = 0.2, a₂ = 0.2, and a₃ = 0.6, given the preference for FC performance maintenance. The training process shown in Figure 10 also illustrates the convergent total rewards, which is taken as a successful training process of the EMS agent.

Compared to the EMS agent with balanced weight factors a₁ = 0.34, a₂ = 0.33, a₃ = 0.33, the power distribution between the battery and FC shown in Figure 11 shows a lower threshold for battery participation; in other words, the battery takes the responsibility of taking care of FC degradation by providing power more frequently together with FC. In addition, compared to the balanced weighting factors group, the FC performance maintenance preferred weighting factors that significantly reduced the participation of FC and limited the FC power to lower than 40 kW through the cycle.

The detailed comparison of equivalent hydrogen consumption, battery capacity degradation, and FC performance degradation of EMS agents in WLTC and UDDS is summarized in Table 6. Given a FC life extension preference weighting factor, i.e., a₃ = 0.6, 39.4% improvement is recorded by reducing the FC performance degradation per cycle from 0.0071 to 0.0043, while hydrogen consumption increases 12.3% from 0.3652 kg to 0.41 kg, and a significant increase in battery capacity degradation ensues.

The derived EMS agent is implemented in a new driving cycle, e.g., United States Urban Dynamometer Driving Schedule (UDDS), to validate the effectiveness of EMS in unknown driving conditions. Figure 12 and Figure 13 show the power distribution with weighting factors group of a₁ = 0.34, a₂ = 0.33, a₃ = 0.33 and a₁ = 0.2, a₂ = 0.2, a₃ = 0.6, respectively. Comparing these two figures, it is clear that EMS with FC performance maintenance weight factors successfully limit the participation of FC in large power events, while the battery takes the principal role in driving the vehicle. In addition, the power of FC charging battery is well restricted, with a 50% drop.

Table 7. Results comparison of DDPG-based EMS agent with different weighting factors in WLTC.

Weighting Factors (a1,a2,a3)	Equivalent Hydrogen Consumption	Battery Capacity Degradation	FC Performance Degradation
0.34–0.3–0.33	0.4284 kg	0.0036	0.0618
0.2–0.2–0.6	0.6447 kg	0.0061	0.0535

demonstrates the performance of EMS agent in UDDS with balanced weighting factors and FC life extension preference weighting factors. Similar to the training cycle WLTC, the FC performance degradation improved 27%, while hydrogen consumption increased 10% with a tremendous deterioration of battery capacity.

It is worth noting that the comparison of EMS agent performance between WLTC and UDDS demonstrates the effectiveness of EMS agent in unknown driving conditions by showing a similar trend of the varying hydrogen consumption, battery degradation, and FC degradation. Specifically, by altering the weight factors, the EMS agent effectively reduces FC performance degradation, although at a significant cost of battery capacity degradation, which is avoidable via a different preference.

5. Conclusions

In this study, a design method based on a PSO algorithm is proposed to optimize the DoH of a FC EV. The main contribution of this article lies in the comprehensive consideration of factors affecting FC EV performance, including system cost, fuel consumption, and battery degradation. Different powertrain design schemes are discussed to meet individual needs in reducing system cost, cutting fuel consumption, or slowing the battery degradation before giving an optimized DoH solution. A DDPG-based RL EMS is developed to achieve pre-defined targets in unknown driving conditions. The EMS is derived from the WLTC-based training process and then, implemented in UDDS to investigate its effectiveness. The simulation results demonstrate that the proposed EMS agent successfully achieves the pre-defined preference, which contributes a useful tool to the academic and industry players to reach optimization via varying the weighting factors accordingly.