Energy Management Strategy Based on Reinforcement Learning and Frequency Decoupling for Fuel Cell Hybrid Powertrain

Abstract

This study presents a Two-Layer Deep Deterministic Policy Gradient (TL-DDPG) energy management strategy for Hydrogen fuel cell hybrid train, that aims to solve the problem that traditional reinforcement learning strategies require high initial values and are difficult to optimize global variables. Augmenting the optimization capabilities of the inner layer, a frequency decoupling algorithm integrates into the outer layer, furnishing a fitting initial value for strategy optimization. This addition aims to bolster the stability of fuel cell output, thereby enhancing the overall efficiency of the hybrid power system. In comparison with the traditional reinforcement learning algorithm, the proposed approach demonstrates notable improvements: a reduction in hydrogen consumption per 100 km by 16.3 kg, a 9.7% increase in the output power stability of the fuel cell, and a 1.8% enhancement in its efficiency.

1. Introduction

The global community is currently grappling with escalating energy crises and heightened concerns over the worsening greenhouse effect. Hydrogen, acknowledged as a clean energy source, holds substantial promise for future development [1]. The application of hydrogen in rail transit is of considerable strategic significance, given its high environmental friendliness [2], and it is indicated by promising prospects in Hydrogen hybrid power system [3].

In contrast to traditional trains relying on internal combustion engines or conventional power sources like pantograph-catenary or third-rail currents, hydrogen fuel cell hybrid trains offer advantages such as heightened environmental friendliness [4], reduced construction costs [5], and enhanced resistance to interference, along with improved compatibility [6]. In a hybrid power system integrating multiple energy sources, the distribution of output power from these sources plays a pivotal role in ensuring the safety of power supply [7], the efficiency of power output, the economy of energy utilization, and the dynamic performance of the train [8]. Consequently, the Energy Management System (EMS) becomes an indispensable component for hydrogen fuel cell hybrid power system trains.Broadly categorized, the EMS of hybrid power systems comprises three main branches: rule-based, optimization-based, and data-based methods [9]. Rule-based EMS entails establishing a set of rules to govern power allocation, relying on engineering expertise or mathematical models [10]. However, this approach heavily depends on historical experience and model accuracy, lacking adaptability in real-world driving scenarios. Conversely, optimization-based methods can be classified into global and instantaneous optimization approaches. Dynamic programming (DP) emerges as a classic numerical algorithm for achieving global optimization. Additionally, commonly employed strategies encompass Genetic Algorithms (GA) [11], Particle Swarm Optimization (PSO) [12], Pontryagin’s Minimum Principle (PMP) [13], and Equivalent Consumption Minimization Strategy (ECMS) [14]. Nevertheless, optimization-based EMS has inherent drawbacks, including high computational demands and challenges in real-time updates, rendering it more suitable for offline planning or integration with other strategies.

The operational parameters of rail transit vehicles exhibit distinctive characteristics. In a single operation, the imperative to accomplish rapid acceleration, deceleration, and stopping within a condensed timeframe results in swift fluctuations in power demand [15]. Furthermore, the fixed running lines and schedules contribute to a high degree of repeatability in running conditions during multiple vehicle operations [16]. Consequently, data-driven methodologies have gained prominence in addressing energy management challenges in rail transport and have garnered considerable attention within the scholarly community [17]. Energy management strategies rooted in data-driven approaches, prominently exemplified by reinforcement learning and its refined algorithms, have witnessed increasing application. Reinforcement learning is delineated by four fundamental elements: State, Action, Policy, and Reward. Throughout the reinforcement learning process, agents engage in continual interaction with the environment, adapting their strategies based on rewards garnered from environmental feedback. Presently, a multitude of studies is dedicated to the energy management of hybrid systems utilizing reinforcement learning techniques [18]. Among these techniques, Q-learning emerges as the foundational and most extensively applied strategy [19]. Leveraging recent advancements in Deep Reinforcement Learning (DRL) [20], researchers have assimilated cutting-edge findings to propose energy management strategies for Hybrid Electric Vehicles (HEVs) based on Deep Q-Network (DQN) for tasks with discrete action spaces [21]. Similarly, approaches founded on Deep Deterministic Policy Gradients (DDPG) have been implemented for tasks with continuous action spaces, showcasing effective policy behavior [22].

However, these reinforcement learning algorithms confront specific challenges. Firstly, optimization results are significantly influenced by the specified initial value [23]. Inadequate specification may lead to slow convergence, diminished accuracy, and assigning a random initial value can impede the scalability of the model. Secondly, the value function Q is updated solely based on variables from adjacent time steps, such as the instantaneous hydrogen consumption of the system and the instantaneous speed of the train [24]. This limitation hinders the consideration of variables obtainable only after completing the entire working condition, such as the smoothness of the fuel cell’s output power and the braking energy recovery efficiency of the lithium battery. To address the impact of the initial value on results, a method proposed in [25] employs the frequency decoupling approach. Initially, the total demand power is allocated, with the low-frequency part assigned to the fuel cell and the high-frequency part allocated to the storage battery. While this method effectively enhances the stability of fuel cell output, the manual specification of the frequency threshold remains a challenge [26]. In an alternative approach, Ref. [27] integrates expert experience to guide the selection of the training initial value. However, reliance solely on expert experience may not consistently prove effective. To surmount the limitation of single-step variables, Ref. [28] introduces a sliding time window, enabling a single training session to learn not only the state variable of a single step but also information from the entire window. Building on this, Ref. [29] enhances the method by utilizing a neural network to integrate information from the entire window for the agent to learn. Nonetheless, the sliding time window may encounter difficulty covering a broad range, and the selection of its length can introduce truncation effects on the data [30].

In order to solve the problem of initial value dependence of reinforcement learning and difficult global optimization of global variables for energy management strategies, an enhanced energy management framework incorporating the frequency decoupling algorithm into the Deep Deterministic Policy Gradient (DDPG) is proposed. Initially, a decoupling algorithm based on the frequency of a low-pass filter is implemented to decompose the power demand signal of the train. Subsequently, a Two-Layer Deep Deterministic Policy Gradient (TL-DDPG) is employed to guide the power allocation of the fuel cell and battery. In this framework, the outer layer network integrates the frequency decoupling algorithm to provide an initial value for the optimization in the inner layer. Detailed elucidation of this approach will be provided in Section 3 of this paper. Test results under various operating conditions demonstrate that the proposed algorithm effectively reduces costs, enhances fuel cell efficiency and stability, and achieves superior braking energy recovery. Diverging from prior works, this paper comprehensively considers the attributes of the data-driven method, frequency decoupling, and rail transit operation. It deliberates on the energy management strategy from a data-driven perspective. The primary contributions of this work are summarized below:

(I)

The impact of filter parameters on power flow in the dynamic system is unveiled through frequency decoupling, and optimal filter parameters are determined using the reinforcement learning method. This leads to enhanced operational efficiency, improved output power stability of the fuel cell, and the attainment of superior economic and energy-saving benefits.

(II)

An two-layer reinforcement learning optimization framework is established for the iterative optimization of both single-step variables and global variables. This approach addresses the challenge of the reinforcement learning model struggling to assimilate all information from the data. Furthermore, the initial value for power distribution is acquired through frequency decoupling, presenting an intuitive relationship and a favorable trade-off between fuel cell and battery costs.

(III)

Tailored to rail transit operational scenarios, the proposed methodology conducts the model training process on a local server rather than real-time training on the train controller. This approach alleviates the burden on computing resources and exhibits a favorable power distribution effect for typical situations characterized by fixed running tracks.

The remainder of this paper proceeds as follows. In Section 2, the train hybrid system modeling is detailed. Section 3 proposes an adaptive EMS combining frequency decoupling and two-layer DDPG. In Section 4, the simulation results of the proposed strategy are provided and analyzed, and the conclusions are given in Section 5.

2. Hybrid System Train Modeling

This article conducts research on a fuel cell hybrid train depicted in Figure 1. The train configuration comprises a fuel cell system, a lithium-ion battery pack, two DC-DC converters, and a traction system involving a DC-AC inverter and traction motor. Figure 1 also shows the input and output of the energy management module, as well as the output characteristics of key components of the hybrid system, including fuel cells, batteries, and traction motors. The model is established using MATLAB 2022a, and the intricate parameters of the system are outlined in

Table 1. Detailed Parameters of the System.

Locomotive
Mass	140 t
Number of traction drive	4
Efficiency of the electric drives (considered as constant)	85%
Gearbox ratio	4.14
Gearbox efficiency	95%
Diameter of a wheel	0.92
Fuel Cell
Type of fuel cell	PEMFC
Number of cells in series	350
Number of modules in parallel	2
Voltage range of a cell	0.3–0.75 V
Rated power of the fuel cell system	400 kW
Battery
Type of battery	LiFePO4
Total number of cells	345
Rated voltage of a cell	3.8 V
Minimal voltage of a cell (charge)	4.0 V
Maximal voltage of a cell (discharge)	2.8 V
Minimal state of charge	20%
Maximal current of a cell	2 C = 320 A
Minimal current of a cell	−0.5 C = −80 A

, whose data is from IEEE VTS Motor Vehicles Challenge 2019 [31]. We have token the locomotive and line data from the race, and reconstructed their powertrain models.

(a) Train Modeling: Assuming the train drives on a flat road with only the longitudinal dynamic model considered, the force on the train can be expressed as Equation (1)

$$\left\{\begin{array}{c}{F}_i=\delta ma\hfill \\ F_a=1/2·\rho ·A·C_d·v^2\hfill \\ F_f=mgf\hfill \end{array}\right.$$

(1)

where $F_i$ is the inertial force, $F_a$ is the aerodynamic drag, $F_f$ is the rolling resistance, $\delta$ is the correction coefficient of rotating mass, m is the mass of train, a is the acceleration, $\rho$ is the air density, $C_d$ is the aerodynamic coefficient, A is the fronted area, g is the gravity coefficient, and f is the rolling resistance coefficient. Then, the power demand of the train can be calculate by Equation (2).

$$P_{\mathrm{req}}=\left(F_i+F_a+F_f\right)v$$

(2)

The purpose of EMS is to distribute the power to different energy source, as shown in Equation (3)

$$P_{\mathrm{req}}=(P_{\mathrm{fc}}{\eta }_{fc-dcdc}+P_{\mathrm{bat}}{\eta }_{bat-dcdc}){\eta }_T$$

(3)

where $P_{\mathrm{fc}}$ is power of fuel cell, $P_{\mathrm{bat}}$ is power of battery, ${\eta }_{fc-dcdc}$ is the efficiency of the fuel cell converter, ${\eta }_{bat-dcdc}$ is the efficiency of the battery converter, and ${\eta }_T$ is the transmission efficiency.

(b) Fuel Cell Modeling: The fuel cell can be equivalent to a controlled source connected to a fixed resistor. Specifically, the output voltage loss of fuel cell includes activation voltage loss, ohmic voltage loss and concentration voltage loss. At the same time, when the control quantity changes, the whole curve will have a certain delay characteristic [32]. The output voltage of the fuel cell is expressed as Equations (4) and (5)

$$V_{fc}=E-R_{ohm}·i_{fc}$$

(4)

$$E=E_{oc}-NAln\left(\frac{i_{fc}}{i_0}\right)·\frac{1}{s{T}_d/3+1}$$

(5)

where $E_{oc}$ is the open circuit voltage, N is the number of fuel cell monomer, A is tafel slope, $i_0$ is the exchange current, $T_d$ is dynamic response time, $\frac{1}{s{T}_d/3+1}$ represents a delay, and s is the symbol of transfer function. $R_{ohm}$ is internal resistance, $i_{fc}$ is fuel cell current, and $V_{fc}$ is fuel cell voltage. The characteristic curve is shown in the lower left of Figure 1.

(c) Battery Modeling: The model of the battery established in this paper mainly focuses on the change of its SOC and output characteristics. SOC in the battery can be calculated by [33]

$$\dot{\mathrm{SOC}}=-\frac{I_{\mathrm{bat}}\left(t\right)}{Q_{\mathrm{bat}}}$$

(6)

where $I_{\mathrm{bat}}\left(t\right)$ is the battery current and $Q_{\mathrm{bat}}$ is the battery nominal capacity. Furthermore, the Equation can be written as

$$\mathrm{SOC}=-\frac{V_{\mathrm{oc}}(SOC,t)-\sqrt{V_{\mathrm{oc}}^2(SOC,t)-4R_{\mathrm{int}}(SOC,t)P_{\mathrm{bat}}}}{2R_{\mathrm{int}}(SOC,t)Q_{\mathrm{bat}}}$$

(7)

$$V_{bat}=V_{\mathrm{oc}}-R_{\mathrm{int}}i-K\frac{Q_{\mathrm{bat}}}{Q_{\mathrm{bat}}-\int idt}(\int idt+i^{\ast })+A{e}^{(-B\int idt)}$$

(8)

$$V_{bat}=V_{\mathrm{oc}}-R_{\mathrm{int}}i-K\frac{Q_{\mathrm{bat}}}{\int idt-0.1Q_{\mathrm{bat}}}{i}^{\ast }-K\frac{Q_{\mathrm{bat}}}{Q_{\mathrm{bat}}-\int idt}+A{e}^{(-B\int idt)}$$

(9)

when voltage dynamics are neglected and the battery circuit do not exist RC branches. In Equation (7), $P_{bat}$ is the battery output power, $V_{oc}$ is the battery open circuit voltage, and $R_{int}$ is the battery internal resistance. Note that the values of $V_{oc}$ and $R_{int}$ vary with SOC. The output voltage of the battery is expressed as Equations (8) and (9), where A is the amplitude of the exponential region, B is the inverse amplitude of the exponential region, K is polarization constant, and $i^{\ast }$ is the battery filtration current. Note that Equation (8) represents discharging state, and (9) represents charging state. The characteristic curve is shown in the lower middle of Figure 1.

(d) Traction Motor Modeling: For traction motor, this study focuses on its external characteristics. For traction motor, this study focuses on its external characteristics. The traction force supplied by the motor can be expressed by the Equation (10)

$$F=(T\times \eta )/R$$

(10)

where F is traction force, $\mathrm{R}$ is the radius of wheel and $\eta$ is motor efficiency which is shown in Figure 2.

The power demand of motor is shown in the Equation (11):

$$P=(T\times n\times \eta )/9550$$

(11)

where n is the motor speed, which positively correlated the vehicle running speed. In this equation, power is measured in kilowatts. The characteristic curve is shown in the lower left of Figure 1.

3. Adaptive Energy Management Strategy Combining Frequency Decoupling and Data-Driven Deep Reinforcement Learning

In this work, one of the methods suitable for fuel cells hybrid power system, namely, frequency decoupling, will be combined with DDPG, which is a state-of-the-art data-driven RL algorithm. Note that similar structures could also be applied to other methods and RL frameworks. The key ideas of the proposed strategy are described as follows.

3.1. Frequency Decoupling

To fulfill the requisite smoothness criteria for the output power in fuel cells, the energy management strategy based on frequency decoupling has been introduced and widely adopted in fuel cell hybrid systems [34]. The fundamental concept behind frequency decoupling is treating the power demand signal as a low-frequency signal combined with a high-frequency signal, with the two signals separated through signal processing. The isolated low-frequency signal is considered as the output power of the hydrogen fuel cell, while the high-frequency signal is deemed as the output power of the lithium battery.

The critical aspect of frequency decoupling involves the selection of the filtering algorithm and the determination of the frequency threshold. Currently employed filtering methods include Fourier filter, wavelet transform, Kalman filter, and others. The choice of the frequency threshold predominantly relies on expert experience, necessitating the integration of the frequency decoupling strategy with other approaches. The implementation workflow of the frequency decoupling strategy is illustrated in Figure 3.

3.2. Deep Deterministic Policy Gradient

DDPG is a model-free, off-policy reinforcement learning algorithm designed for learning policies $\theta$ in high-dimensional continuous action spaces. It stems from the DPG algorithm and incorporates a deep function approximator, hence earning its name DDPG [35]. The effectiveness of DDPG relies on two key techniques. Firstly, experience replay enables the algorithm to learn from a set of unrelated content. Secondly, akin to the hard fixed Q-target network utilized in DQN, DDPG employs a “soft” target update for the actor-critic. This technique enhances training stability as the evaluation network $({\theta }^{{\mu }^{\prime }},{\theta }^{Q^{\prime }})$ updates more swiftly than the target network $({\theta }^{\mu },{\theta }^Q)$. The architecture of DDPG is depicted in Figure 4.

3.3. Our Energy Management Strategy for Hydrogen Fuel Cell Hybrid Train

In this work, There are two DDPG frameworks are nested to achieve energy management. The one in outer layer is combined with the frequency decoupling algorithm to determine the threshold of frequency division by optimizing long-term variables (such as the degree of fuel cell power fluctuation and average efficiency), so as to initially allocate energy. The DDPG in the inner layer is further optimized by the instantaneous variables (such as instantaneous speed, SOC, etc.) based on the initial value given after frequency decoupling. Through the method proposed in this paper, we not only focus on long-term variables, but also have a good specification of the initial value for training. The block diagram of the method is shown in Figure 5, and more detailed discussion is given below.

$$Cos{t}_{inner}=K_{H_2}m\left(H_2\right)+K_{F{C}_{open}}num\left(F{C}_{start}\right)+\int (K_{F{C}_{run}}{P}_{fc}\left(t\right)+K_{BA{T}_{run}}{P}_{bat}\left(t\right))dt$$

(12)

$$r_{inner}=-(K_{cos{t}_{inner}}Cos{t}_{inner}+K_{soc}\left|(SO{C}_{now}-SO{C}_{init})\right|)$$

(13)

(a) Short-time optimization layer: In the inner DDPG, the state variable $s_t$ comprises the State of Charge ($SOC$), the speed and acceleration of the train, and the busbar voltage. The control action a corresponds to the output power of the fuel cell. The immediate cost is defined by Equation (12), encompassing the costs of hydrogen, fuel cells, and lithium batteries. Given that the reward function is the negative counterpart of the cost, $r_{inner}=-cos{t}_{inner}$. Specifically, if the battery’s SOC is excessively low, a penalty term is incorporated, and r is defined by Equation (13). Here, $K_{H_2}$ represents the cost of hydrogen ($/kg), $m\left(H_2\right)$ denotes the mass of hydrogen consumed in two adjacent time steps, $K_{F{C}_{open}}$ represents the cost of activating the fuel cell once ($), while $K_{F{C}_{run}}$ and $K_{BA{T}_{run}}$ signify the operating costs of fuel cells and lithium batteries ($). The scale factors $K_{cos{t}_{inner}}$ and $K_{soc}$ are set to ensure a fundamental balance between the two components during training.

(b) Long-time optimization layer: In the outer layer, there exist frequency decoupling modules and an additional DDPG framework. The filter employed in the frequency decoupling segment is defined as in Equation (14). The state variables $s_T$ accepted by this DDPG encompass the standard deviation of the fuel cell output power in the last entire operational condition, the average efficiency of the fuel cell, and the difference value between the $SOC$ at the end and the beginning. These variables are notably challenging to measure within a single time step. The threshold $f_c$ for frequency decoupling is determined by training the loss function expressed in Equation (15). The loss function comprises three components representing the smoothness of the fuel cell, the system’s efficiency, and the braking energy recovery of the lithium battery. $K_a$, $K_b$, and $K_c$ are scale factors. It’s important to note that to minimize the impact of sampling frequency on results, this work doesn’t calculate the true value of $f_c$ but computes the ratio of $f_c$ and the maximum frequency of the power signal to derive the filtering ratio f. The relationship between the two is expressed in Equation (16), where $f_{max}$ denotes the maximum frequency of power demand. Notably, the value of f ranges between 0 and 1, representing the fraction of low-frequency signals. For instance, with a sampling frequency of 10 Hz, indicating a maximum signal frequency of 5 Hz, if the optimized f is 0.6, it signifies that signals from 0–3 Hz are considered low frequency, while signals above 3 Hz are considered high frequency.

$$H\left(s\right)=\frac{1}{\frac{s}{w_c}+1},w_c=2\pi \ast f_c$$

(14)

$$Cos{t}_{outer}=K_{a}Std\left(P_{fc}\right)+K_b\Delta SOC+K_c{\eta }_{fc}$$

(15)

$$f=\frac{f_c}{f_{max}}$$

(16)

In this study, the optimization of short-term variables is achieved through classical DDPG, which focuses solely on optimizing the energy flow within a single-step time scale in the hybrid system. The algorithm based on frequency decoupling furnishes an initial value for optimization, and the outer DDPG is capable of addressing long-term variables, thereby optimizing the system’s performance on a more rational scale. The DDPG networks of both the inner and outer layers are updated according to the following Equations (17) and (18).

$$L=\frac{1}{N}\sum _i{\left(y_i-Q(s_i,a_i|{\theta }^Q)\right)}^2$$

(17)

$${\nabla }_{{\theta }^{\mu }\mathrm{J}}\approx \frac{1}{N}\sum _i{\nabla }_{\mathrm{a}}\mathrm{Q}(s,\mathrm{a}|{\theta }^Q){{|}_{a=s_i,a=\mu \left(s_i\right)}·{{\nabla }_{{\theta }^{\mu }}}^{-}\left(\mathrm{s}\right|{\theta }^{\mu })|}_{s_i}$$

(18)

4. Simulation Validation and Discussion

This section aims to validate the proposed adaptive hierarchical strategy. Initially, the study explores the impact of the proposed algorithm on hydrogen consumption and State of Charge (SOC) maintenance. Subsequently, to comprehend the influence of different reward formulations on learning efficiency and the final policy, a comparison is made between the proposed strategy and other method. Notably, in contrast to the majority of related studies, the proposed energy management strategy takes into account the output stability of fuel cells. Lastly, the robustness of the method is verified under different driving conditions.

To assess the experimental outcomes, the proposed method undergoes comparison with other benchmark algorithms through simulation experiments conducted under the driving conditions illustrated in Figure 6 and Figure 7 [31]. The proposed algorithm is implemented in Python 3.9, utilizing Tensorflow as the primary machine learning library, while the hybrid system model is constructed using MATLAB R2022a. ONNX is employed for transforming neural network models between Tensorflow and Simulink frameworks. The simulation experiment is conducted on a server equipped with two NVIDIA RTX 3060 graphics cards. The algorithms involved in the comparison encompass rule-based, filter-based, and traditional DDPG, with their profiles and settings detailed in

Table 2. Introduction to the reference algorithm.

Algorithm	Brief Introduction	Parameter Set
Rule-based	Design power distribution according to expert experience	$P_{req}$ < 0, $P_{bat}$ = $P_{req}$; $SOC<SO{C}_{min}$ $P_{fc}$ = max; $SOC>SO{C}_{max}$ $P_{fc}$ = 0; else …
DDPG	Reinforcement learning	The loss function is shown in Equation (13)
Frequency Decoupling	low frequency to the fuel cell and high frequency to the battery	The filtering algorithm is Fourier transform, filter frequency is shown in Equations (14) and (16)

. The train’s running condition spans a 1200 s time series, and training continues until the network converges. After each epoch completion (totaling 120 epochs), relevant results such as the average efficiency of the fuel cell and the standard deviation of the output power are recorded and averaged.

4.1. Validation of the Proposed Strategy

Figure 8 illustrates the results of the proposed algorithm after 120 training epochs under the specified driving conditions (each condition spanning 1200 s as in Figure 6). The results exhibit convergence tendencies. The loss curves of the inner and outer layers are depicted in Figure 8a, showing similar change trends and confirming the rationality of the loss function settings. At the 100th epoch, the filtering threshold is determined to be 0.376 times the maximum frequency (as depicted in Figure 8b). This implies, under a sampling frequency of 1 Hz, that power changes below 0.188 Hz are considered as filtering threshold for the fuel cell, while the remaining power demand is supplied by the lithium battery. A visual representation of the method is presented in Figure 8c. The time-frequency diagram of the power is obtained by wavelet transform of the required power-time. The brighter the colour, the greater the amplitude of a signal at that frequency at that point. The horizontal lines in the figure are derived from the results in Figure 8b). The section below the horizontal line will be allocated to the fuel cell and the section above will be allocated to the battery.

Additionally, Figure 9 showcases the superiority of the proposed strategy across various metrics. Figure 9a displays the output power curves of the fuel cell in the first training epoch for the proposed method, frequency decoupling, and traditional DDPG. It is evident that traditional DDPG generates initial values randomly during training, leading to volatility, longer adjustment times, and difficulty in finding the optimal solution. When using the frequency decoupling method, almost all power output comes from fuel cells, underutilizing lithium batteries. Our method employs the result of frequency decoupling as the initial value, enabling better learning of operating conditions’ characteristics. Notably, this represents only the first training session, demonstrating the effectiveness of our method. The final result is depicted in Figure 8d. Figure 9b illustrates the total cost of a single run using different energy management strategies, encompassing fuel cell start and stop costs, hydrogen consumption costs, and storage battery usage costs, as defined by Equation (12). Once training stabilizes, the total cost of our strategy is $18.86, lower than other strategies. The comparison also reveals that the proposed strategy outperforms traditional DDPG in terms of speed and convergence due to the consideration of initial value optimization. Figure 9c demonstrates the impact of different energy management strategies on the SOC trajectory, with the demand power curve attached at the bottom to illustrate the effect of braking energy recovery for each strategy. Our proposed strategy effectively enables the storage battery to absorb braking energy, showcasing the largest SOC variation range, indicating the strategy’s ability to stimulate the potential of lithium batteries while maintaining their performance. Figure 10a,b present the working efficiency of the fuel cell across different strategies, showcasing the advantages of the proposed strategy through two aspects: the average efficiency change of each epoch during training and the proportion of different efficiency intervals after training. The proposed strategy ensures the fuel cell operates mostly in the high-efficiency interval. Finally,

Table 3. Comparison between different EMSs.

Algorithm	Fuel Consumption (kg)	Terminal SOC (%)	Average Efficiency of Fuel Cell (%)	Total Cost ($)	Training Time (s)
Rule-based	2.78	62.40	54.78	56.49	40.12
DDPG	3.84	63.8	53.38	21.45	123.56 + 33.54
Frequency Decoupling	5.02	69.52	51.49	34.29	66.38
Proposed	2.21	60.36	55.20	18.90	206.71 + 45.40

provides a comprehensive comparison of different strategies. Notably, while the rule-based strategy is relatively close to our strategy in indicators such as hydrogen consumption and SOC, it neglects the switching loss of fuel cells, resulting in a significantly larger final cost compared to other strategies.

4.2. Impact of Different Reward Expressions on the Strategy

In this section, the hyperparameters setting of the reward function are discussed. For the reward function, $K_{H_2}$, $K_{F{C}_{open}}$, $K_{F{C}_{run}}$ and $K_{BA{T}_{run}}$ are determined by economics, and they are given by

Table 4. Economic Parameters Setting.

Parameter	Value
$K_{H_2}$ ($/kg)	5
$K_{F{C}_{open}}$	40
$K_{F{C}_{run}}$	159
$K_{BA{T}_{run}}$	105

.

Figure 11 explores the impact of different reward expressions on the strategy’s learning efficiency and final policy when $K_{cos{t}_{inner}}$ and $K_{soc}$ in Equation (13) assume different values (essentially determining their ratio value). As observed, a reward function with greater emphasis on SOC sustenance tends to exhibit conservative behavior, underutilizing the battery buffer. Conversely, if more weight is assigned to minimizing hydrogen consumption, the final policy may breach the battery SOC constraint, causing the battery to enter the risky region in such instances. Simultaneously, adjustments to the weight of the standard deviation of the fuel cell output power, the difference between the initial and final SOC, and the average fuel cell efficiency should be considered. Given that the ranges of $Std\left(P_{fc}\right)$, $\Delta SOC$, and ${\eta }_{fc}$ are all confined within the 0–1 range (the fuel cell standard deviation has been normalized), the ratio values of $K_a$, $K_b$, and $K_c$ in Equation (15) are set to be equal.

4.3. Discussion of the Performance about Fuel Cell and Battery

This section delves into further details concerning fuel cells and storage batteries. The frequent start and stop as well as the output stationarity of fuel cells are crucial factors influencing their lifespan and hydrogen consumption. However, traditional strategies often overlook the switching times of fuel cells and the smoothness of output power. Additionally, a significant portion of the energy generated by train braking is typically dissipated through braking resistance, leading to inadequate braking energy recovery in traditional strategies.

Table 5. Fuel Cell Related Indicators.

Algorithm	Number of Fuel Cell Starts	Standard Deviation
Rule-based	17	100.04
DDPG	9	95.63
Frequency Decoupling	13	88.93
Proposed	6	86.32

presents the standard deviation of switching times and output power of the fuel cell under different energy management strategies. It is evident that the application of reinforcement learning (DDPG and our strategy) reduces unnecessary switching losses and effectively maintains the smoothness of the fuel cell’s output power. Figure 12 illustrates the braking energy recovery ratio under different energy management strategies, calculated as the ratio of absorbed power by the battery to the total braking power. The proposed algorithm demonstrates the ability to maximize braking energy recovery. In contrast, the DDPG strategy and the frequency decoupling strategy encounter issues such as the battery absorbing energy from the hydrogen fuel cell, leading to overcharging.

4.4. Robustness Verification against Different Driving Cycle

The working conditions vary between different scenarios. In order to evaluate the robustness of the proposed strategy, another different driving cycles are tested. The new test condition is shown in Figure 13 [31]. It can be seen that the overall speed of the vehicle under the new condition is higher than that under the old condition, and the requirements for the power system are more stringent. Therefore, we changed the cells of the battery from 345 in Table 1 to 1580. Figure 14 and

Table 6. Comparison between different EMSs in new condition.

Algorithm	Fuel Consumption (kg)	Terminal SOC (%)	Average Efficiency of Fuel Cell (%)	Total Cost ($)	Training Time (s)
Rule-based	8.85	67.24	48.53	86.14	52.14
DDPG	7.70	66.87	49.86	45.96	178.55 + 55.54
Frequency Decoupling	12.44	69.73	44.87	56.91	105.47
Proposed	5.96	65.53	51.69	37.70	252.84 + 68.40

show the energy allocation results of different strategies under this condition. The results indicate that the cost under the proposed strategy is lower than that of other strategies validating the robust performance of the proposed strategy. It should be noted that this result is pre-trained on the previous condition, so that the common characteristics between different working conditions can be better learned.

5. Conclusions

In this research, An adaptive hierarchical EMS for trains of hydrogen fuel cell hybrid power system is proposed by combining frequency decoupling and data-driven DDPG. The goal is to address the difficulty of reinforcement learning based EMS to consider multiple time steps, and training initial values, and directly facilitate online training in real-world Settings. The main findings are summarized as follows.

Our proposed adaptive hierarchical EMS represents a significant advancement in addressing the intricacies of hydrogen fuel cell hybrid power systems within the rail transit context. The integration of frequency decoupling and data-driven DDPG not only showcases reduced hydrogen consumption but also offers a strategic advantage in battery SOC management over extended timeframes. This, in turn, enhances the overall potential and lifespan of the battery while ensuring a stable and optimized fuel cell power output. Unlike conventional RL-based EMS strategies, where random initialization may lead to prolonged training times and potential pitfalls in fuel cell stability, our approach leverages the synergies between reinforcement learning techniques and frequency decoupling methods. This unique combination allows for efficient and safe exploration in real-world environments, positioning our strategy as a robust contender for future RL-based algorithms.

It is crucial to note that our study primarily focuses on energy management under fixed working conditions, presenting an offline training strategy. Future endeavors will delve into the dynamic coupling of speed trajectory optimization and energy management, paving the way for real-time trajectory optimization and adaptive energy distribution for trains. This ongoing research aims to bridge the gap between theoretical advancements and practical applications, contributing to the sustainable and efficient integration of hydrogen fuel cell technology in rail transit systems.

The study still has the following limitations: First, we design a double-layer nested reinforcement learning framework, which results in larger model size and longer training time. Secondly, due to the optimization of global variables, this study only trains the known conditions of the working conditions, rather than online real-time training. When the working conditions change, the model needs to be further fine-tuned to achieve better results.