An Energy Management Strategy for FCHEVs Using Deep Reinforcement Learning with Thermal Runaway Fault Diagnosis Considering the Thermal Effects and Durability

Abstract

Temperature control plays a critical role in mitigating the lifespan degradation mechanisms and ensuring thermal safety of lithium-ion batteries (LIBs) and proton exchange membrane fuel cells (PEMFCs). However, current energy management strategies (EMS) for fuel cell hybrid electric vehicles (FCHEVs) generally lack comprehensive thermal effect modeling and thermal runaway fault diagnosis, leading to irreversible aging and thermal runaway risks for LIBs and PEMFCs stacks under complex operating conditions. To address this challenge, this paper proposes a thermo-electrical co-optimization EMS incorporating thermal runaway fault diagnosis actuators, with the following innovations: firstly, a dual-layer framework integrates a temperature fault diagnosis-based penalty into the EMS and a real-time power regulator to suppress heat generation and constrain LIBs/PEMFCs output, achieving hierarchical thermal management and improved safety; secondly, the distributional soft actor–critic (DSAC)-based EMS incorporates energy consumption, state-of-health (SoH) degradation, and temperature fault diagnosis-based constraints into a composite penalty function, which regularizes the reward shaping and guides the policy toward efficient and safe operation; finally, a thermal safe constriction controller (TSCC) is designed to continuously monitor the temperature of power sources and automatically activate when temperatures exceed the optimal operating range. It intelligently identifies optimized actions that not only meet target power demands but also comply with safety constraints. Simulation results demonstrate that compared to DDPG, TD3, and SAC baseline strategies, DSAC-EMS achieves maximum reductions of 39.91% in energy consumption and 29.38% in SoH degradation. With the TSCC implementation, enhanced thermal safety is achieved, while the maximum energy-saving improvement reaches 25.29% and the maximum reduction in SoH degradation attains 20.32%.

1. Introduction

With the growing attention to environmental protection and energy security, fuel cell hybrid electric vehicles (FCHEVs) are playing an increasingly crucial role in promoting green transportation and energy transformation [1]. A core challenge in FCHEV design lies in coordinating heterogeneous power sources—proton exchange membrane fuel cells (PEMFCs) and lithium-ion batteries (LIBs)—to simultaneously achieve high fuel economy, long component lifespan, and robust thermal safety. To this end, numerous energy management strategies (EMSs) have been developed, including rule-based methods, optimization-based approaches such as dynamic programming (DP) [2] and Pontryagin’s minimum principle (PMP) [3], and real-time strategies like equivalent consumption minimization strategy (ECMS) [4,5], model predictive control (MPC) [6,7], fuzzy logic [8,9], and genetic algorithms [10,11,12]. More recently, intelligent control methods—particularly deep reinforcement learning (DRL)—have gained traction due to their adaptability and learning capabilities [13,14,15,16,17,18,19,20,21].

However, most existing EMSs prioritize fuel economy while treating durability and thermal safety as secondary considerations. This is problematic because aggressive power allocation can accelerate battery degradation and induce thermal runaway, especially under high-energy-density battery operation. Recognizing this, recent studies have begun integrating state-of-health (SoH) awareness into EMS design. For instance, MPC [22], optimized point-line methods [23], and DRL algorithms—including DQN [24], DDPG [25], TD3 [26], and SAC [27]—have been employed to jointly minimize fuel consumption and SoH degradation. Yet these approaches suffer from significant limitations: they often exhibit poor sample efficiency, unstable convergence, high computational demands, and sensitivity to hyperparameter tuning. Moreover, they typically assume idealized thermal conditions and lack mechanisms to enforce real-time thermal safety constraints, limiting their practical applicability in safety-critical automotive systems.

Concurrently, the rising energy density of LIBs has intensified thermal management challenges. Elevated temperatures not only hasten electrochemical aging but also increase the risk of thermal runaway—a catastrophic failure mode that compromises vehicle safety. Consequently, integrating thermal dynamics into EMS has become imperative. Pioneering efforts in this direction include Zhang et al. [28], who proposed a climate-adaptive EMS that coordinates cabin heating with battery thermal behavior to improve fuel economy in cold environments while preserving battery health. Khalatbarisoltani et al. [29] further advanced this idea by combining multi-agent DRL (MADRL) with LSTM-based driving and thermal prediction models to co-optimize energy use and thermal load. Han et al. [30] proposed a hybrid PMP-MPC framework enhanced with an ANN-based speed predictor that explicitly accounts for electric motor thermal dynamics to reduce operational costs and regulate temperature. Similarly, Abbasi et al. [31] used DRL to jointly optimize charging protocols and cooling strategies, demonstrating up to two years of extended battery life under controlled thermal conditions. Guo et al. [32] developed a thermal-aware EMS that links battery aging mechanisms directly to thermal regulation, achieving a balance between performance and longevity. Most recently, Zhang et al. [33] introduced a DRL-based EMS that embeds thermal effects into power-split decisions, dynamically managing temperature to enhance safety, efficiency, and lifespan.

Despite these advances, a critical research gap remains: existing integrated EMS either lack explicit safety enforcement under thermal runaway or rely on standard DRL frameworks that suffer from Q-value overestimation, high policy complexity, and insufficient robustness in multi-objective optimization. Crucially, none of the cited works [22,23,24,25,26,27,28,29,30,31,32,33] implement a coordinated, dual-layer architecture that proactively prevents thermal runaway while simultaneously optimizing economy, durability, and safety through a distributional reinforcement learning framework.

To address these shortcomings, this paper proposes a novel distributional soft actor-critic (DSAC)-based EMS with thermal runaway fault diagnosis for FCHEVs. Building upon the foundational insights from [28,29,30,31,32,33], our approach introduces three key innovations:

• To mitigate thermal runaway under high-power operation, this paper proposes a dual-layer protection framework. The upper layer integrates a temperature fault diagnosis-based penalty into the EMS to proactively suppress heat generation. The lower layer employs a real-time safety-constrained power regulator that dynamically adjusts output limits of LIBs and PEMFCs. This strategy-to-execution coordination enables proactive thermal management, improving both system safety and operational reliability.
• To overcome Q-value overestimation and policy complexity in DRL-based energy management, this work proposes a DSAC-based EMS for FCHEVs. By embedding power loss, SoH degradation, and temperature fault diagnosis-based constraints into a composite penalty function, the method reduces policy dimensionality and achieves multi-objective optimization for economy, durability, and safety. The DSAC algorithm enhances efficiency and accuracy by learning the return distribution via a single network using return variance.
• To enhance safety under high-temperature conditions, this paper designs a DSAC-based safety-constrained controller for power action optimization. It continuously monitors source temperatures and activates automatically when thresholds are exceeded. The controller intelligently selects actions that meet power demand while satisfying thermal constraints, ensuring safe and efficient operation through real-time, adaptive decision-making.

The structure of this paper is as follows: Section 2 provides gives the SoH and thermal models of FCHEVs. Section 3 presents the optimization framework and control architecture of the proposed EMS, including its training procedure and a thermal safety-oriented soft constraint controller for the power sources. Section 4 presents the simulation results. A conclusion is given in Section 5.

2. Powertrain Model

2.1. FCHEV Powertrain System Overview

The architecture of the FCHEV powertrain system is shown in Figure 1. In this system, the FCHEV is powered by a permanent magnet synchronous motor (PMSM), which drives the rear wheels through a differential and final drive connected to the rear axle of the vehicle. The PEMFC system operates in parallel with a lithium-ion battery, and is connected to the DC bus via a unidirectional boost DC-DC converter. Other auxiliary electronic devices, such as the battery, are powered through a DC-DC converter linked to the DC bus. The power consumption of the thermal management system (TMS) and other auxiliary electronic devices is not considered in this paper. The detailed powertrain parameters provided in

Table 1. FCHEV parameters.

Property	Parameter	Value	Unit
FCV mass	$m_{fc}$	2046	kg
Equivalent windward area	$A_f$	2.62	m2
Wheel radius	$r_w$	0.35	m
Gear ratio of final drive	$i_{trans}$	9	–

are adopted from [33].

2.2. Electric–Thermal PEMFC Model

Thermal management is critical during the operation of fuel cells. The heat generated by electrochemical reactions within the fuel cell stack leads to a temperature rise, which can adversely impact the efficiency and lifespan of the fuel cell. Therefore, temperature regulation through the cooling system is essential. The rate of temperature change within the fuel cell stack can be expressed as:

$${\displaystyle \frac{d{T}_{fc}\left(t\right)}{dt}}={\displaystyle \frac{P_{fc,loss}\left(t\right)-Q_{fc,coolant}\left(t\right)-Q_{fc,e}\left(t\right)}{m_{fc}{C}_{s,fc}}}$$

(1)

where $P_{fc,loss}\left(t\right)$ represents the thermal power generated by electrochemical reactions, $Q_{fc,coolant}\left(t\right)$ is the thermal power transferred by the coolant, and $Q_{fc,e}\left(t\right)$ is the surface heatloss of the stack. $m_{fc}$ and $C_{s,fc}$ are the mass and specific heat capacity of the PEMFCs stack, respectively.

The thermal power $P_{fc,loss}\left(t\right)$ can be calculated as:

$$P_{fc,loss}\left(t\right)=N_{cell}\times (V_{therm}-V_{fc}\left(t\right))\times I_{fc}\left(t\right)$$

(2)

where $N_{cell}$ represents the number of fuel cell units in the stack, $V_{therm}\left(t\right)$ is the thermoneutral voltage, while $V_{fc}\left(t\right)$ and $I_{fc}\left(t\right)$ denote the stack’s voltage and current, respectively.

The thermal power transferred by the coolant, $Q_{fc,coolant}\left(t\right)$, is dependent on the temperature difference between the stack and the coolant, and is generally expressed by the following equation:

$$Q_{fc,coolant}\left(t\right)=h_{fc,heat}·A_{fc}·\left(T_{fc}\left(t\right)-T_{fc,coolant}\left(t\right)\right)$$

(3)

where $h_{fc,heat}$ represents the heat transfer coefficient, $A_{fc}$ represents the contact area between the stack and the coolant, and $T_{fc}\left(t\right)$ and $T_{fc,coolant}\left(t\right)$ correspond to the temperatures of the stack and the coolant, respectively.

$Q_{fc,e}\left(t\right)$ represents the heat loss power from the PEMFCs stack to the environment. Heat is primarily dissipated when the stack temperature exceeds the environment temperature, while reverse heat transfer may occur under cooling conditions. It can be expressed by the following formula:

$$Q_{fc,e}\left(t\right)={\displaystyle \frac{T_{fc}\left(t\right)-T_e}{R_{fc}}}$$

(4)

where $T_{fc}\left(t\right)$, $R_{fc}$, and $T_e$ represent the PEMFC stack temperature, thermal resistance, and environment temperature, respectively.

2.3. Electric–Thermal LIB Model

In order to optimize the performance and lifespan of the LIBs within the powertrain system, this paper adopts a battery management system (BMS) model. This model not only monitors the state of charge (SoC) of the lithium-ion batteries (LIBs) but also controls the thermal dynamics to ensure that the battery operates within the optimal temperature range. The BMS dynamically adjusts the cooling system’s power output based on real-time monitoring of the battery temperature, thereby regulating the thermal management process.

The temperature variation of the LIBs can be described by the following equation:

$${\displaystyle \frac{d{T}_{lib}\left(t\right)}{dt}}={\displaystyle \frac{P_{loss}\left(t\right)-Q_{cooling}\left(t\right)}{m_{lib}{C}_{s,lib}}}$$

(5)

where $P_{loss}\left(t\right)$ represents the heat loss due to current transmission, $Q_{cooling}\left(t\right)$ denotes the thermal power transferred by the cooling system, and $m_{lib}$ and $C_{s,lib}$ are the mass and specific heat capacity of the LlB, respectively.

The heat loss power $P_{loss}\left(t\right)$ can be expressed as:

$$P_{loss}\left(t\right)=I_{lib}{\left(t\right)}^2·R_{lib}\left(t\right)$$

(6)

where $I_{lib}\left(t\right)$ is the battery current, and $R_{lib}\left(t\right)$ is the equivalent internal resistance of the battery, which varies dynamically with temperature and current rate.

The thermal power transferred by the cooling system, $Q_{cooling}\left(t\right)$, is a function of the temperature difference between the battery and the coolant and can be expressed as:

$$Q_{cooling}\left(t\right)=h_{cool}·A_{cool}·\left(T_{lib}\left(t\right)-T_{coolant}\left(t\right)\right)$$

(7)

where $h_{cool}$ is the heat transfer coefficient, $A_{c}ool$ is the contact area between the coolingsystem and the battery, and $T_{lib}\left(t\right)$ and $T_{coolant}\left(t\right)$ represent the temperatures of the battery and the coolant, respectively.

$$\begin{array}{cc}\hfill {\displaystyle \frac{d{T}_{b,s}\left(t\right)}{dt}}& ={\displaystyle \frac{T_{b,coolant}-T_{b,s}\left(t\right)}{R_{b,ext}{C}_{b,s}}}-{\displaystyle \frac{T_{b,s}\left(t\right)-T_{b,c}\left(t\right)}{R_{b,in}{C}_{b,s}}}\hfill \end{array}$$

(8)

$$\begin{array}{cc}\hfill {\displaystyle \frac{d{T}_{b,c}\left(t\right)}{dt}}& ={\displaystyle \frac{T_{b,s}\left(t\right)-T_{b,c}\left(t\right)}{R_{b,in}{C}_{b,c}}}+{\displaystyle \frac{P_{b,loss}\left(t\right)}{C_{b,c}}}\hfill \end{array}$$

(9)

$$\begin{array}{cc}\hfill T_b\left(t\right)& ={\displaystyle \frac{T_{b,s}\left(t\right)+T_{b,c}\left(t\right)}{2}}\hfill \end{array}$$

(10)

where $T_{b,s}\left(t\right)$, $T_{b,c}\left(t\right)$, and $T_{b,coolant}$ denote the surface, core, and coolant temperatures of the LIB cell, respectively; $T_b\left(t\right)$ is defined as the average of $T_{b,s}\left(t\right)$ and $T_{b,c}\left(t\right)$; $R_{b,in}$ and $R_{b,ext}$ represent the internal and external thermal resistances of the LIB cell; and $C_{b,s}$ and $C_{b,c}$ are the surface and core heat capacities of the LIB cell.

2.4. SoH Models of PEMFCs and LIBs

The degradation of the PEMFCs stack is essentially the accumulation of degradation rates under idle, high power load, load cycling, and start–stop conditions. Since the power output of the PEMFCs stack is continuous during a driving cycle, the start–stop condition is not considered in the calculation of the PEMFCs stack degradation. Therefore, the formula for the decline in the SoH of the PEMFCs stack, denoted as $\Delta So{H}_{fc}\left(t\right)$, is expressed as follows:

$$\begin{array}{cc}\hfill \Delta So{H}_{fc}\left(t\right)& =\Delta So{H}_{fc,low}\left(t\right)+\Delta So{H}_{fc,high}\left(t\right)\hfill \\ & +\Delta So{H}_{fc,\Delta P}\left(t\right)\hfill \end{array}$$

(11)

where $\Delta So{H}_{fc,low}\left(t\right)$, $\Delta So{H}_{fc,high}\left(t\right)$, and $\Delta So{H}_{fc,\Delta P}\left(t\right)$ represent the SoH degradation rates of the PEMFCs stack under low load operation, high load operation, and load variation operation, respectively.

$$\Delta So{H}_{fc,low}\left(t\right)=\left\{\begin{array}{c}-k_{low}\Delta t/V_{fc},if{P}_{fc}\le 0.2P_{fc,max}\hfill \\ 0,else\hfill \end{array}\right.$$

(12)

$$\Delta So{H}_{fc,high}\left(t\right)=\left\{\begin{array}{c}-k_{high}\Delta t/V_{fc},if{P}_{fc}\ge 0.8P_{fc,max}\hfill \\ 0,else\hfill \end{array}\right.$$

(13)

$$\Delta So{H}_{fc,chg}\left(t\right)=-k_{chg}|P_{fc}\left(t\right)-P_{fc}(t-1)|/\left(N_{cell}{V}_{fc}\right)$$

(14)

where $k_{low}$, $k_{high}$, and $k_{chg}$ represent the voltage degradation rates during idling, high power load, and load change periods, respectively. $\Delta t$ represents the sampling interval time. The specific values for these parameters can be found in [33].

The performance degradation of lithium batteries is typically manifested as a loss of their maximum remaining capacity, which is commonly characterized by a semi-empirical exponential model [34]. This model expresses the instantaneous capacity loss rate of lithium batteries as a function of current, as follows:

$$\Delta Q_{bat}\left(t\right)=M\left(c\right)exp\left({\displaystyle \frac{-E_a\left(c\right)}{RT}}\right)A{\left(c\right)}^{\varphi }$$

(15)

where $\Delta Q_{bat}$ represents the capacity loss of the lithium battery, expressed as a percentage, c denotes the discharge rate of the lithium battery, $M\left(c\right)$ is the exponential prefactor associated with the discharge rate c, R is the ideal gas constant, T is the temperature of the lithium battery, $A\left(c\right)$ refers to the current throughput of the lithium battery (Ah), $\phi$ is the power-law factor, and $E_a\left(c\right)$ represents the activation energy related to the discharge rate c, expressed as:

$$\left\{\begin{array}{c}{E}_a\left(c\right)=31700-163.3c\hfill \\ \phi =0.57\hfill \end{array}\right.$$

(16)

it is commonly understood that a lithium battery reaches the end of its life when it experiences a 20% capacity loss. By defining $\Delta Q_{bat}=20\%$, the total charge–discharge throughput $A_{total}$ and the number of charge–discharge cycles $N_{bat}$ from the beginning of usage to the end of the battery’s life can be determined as follows:

$$A_{total}(c,T_c)={\left[{\displaystyle \frac{20}{M\left(c\right)exp\left(\frac{-E_a\left(c\right)}{RT}\right)}}\right]}^{{\displaystyle \frac{1}{\phi }}}$$

(17)

$$N_{bat}(c,T_a)={\displaystyle \frac{A_{total}(c,T_c)}{Q_{bat}}}$$

(18)

where $Q_{bat}$ denotes the rated capacity of the lithium battery. Therefore, the remaining maximum usable capacity of the lithium battery, $Q_{rem}$, can be calculated as follows:

$$Q_{rem}\left(t\right)={\displaystyle \frac{{\int }_0^t\left|I_{bat}\left(t\right)\right|dt}{2\times 3600N_{bat}(c,T_c)Q_{bat}}}$$

(19)

the SoH is defined to quantify the degradation of lithium battery performance. By differentiating and discretizing Equation (19), the following formula is derived:

$$\Delta So{H}_{bat}\left(t\right)={\displaystyle \frac{T_s\left|I_{bat}\left(t\right)\right|}{2\times 3600N_{bat}(c,T_c)Q_{bat}}}$$

(20)

where $\Delta So{H}_{bat}$ represents the rate of change in the state of health of the lithium battery.

Based on the SoH models of the two energy sources, the lifespan degradation trends of the lithium-ion battery and fuel cell are accurately quantified. This not only provides a theoretical foundation for the coordinated management of dual-energy systems, but also establishes a solid basis for the design of energy management strategies aimed at lifespan optimization.

3. Design of DSAC-Based EMS with Thermal Flexibility Constraints Considering the Thermal Effects and Durability

The implementation architecture of the DSAC-based EMS framework incorporating thermal constraints is illustrated in Figure 2. First, a DSAC-based EMS considering thermal effects and durability is designed to optimize power distribution in FCHEVs. Second, a flexible-constrained thermal safety controller is developed to ensure the thermal safety of energy sources in FCHEVs.

3.1. Training of EMS

In this paper, the DSAC algorithm is employed to optimize the EMS for a FCHEVs. The core capability of DSAC lies in its ability to learn an optimal policy that maximizes long-term rewards through interaction with the environment, effectively addressing uncertainty and delayed feedback, while maintaining a balance between exploration and exploitation, and demonstrating strong generalization and adaptation capabilities [35]. The iterative process of DSAC consists of four core components: (1) the environment model; (2) the expected value function $Q^{\pi }(s_t,a_t)$ based on the distributional soft return; (3) the sample transition $e_t=(s_t,a_t,r_t,s_{t+1})$; (4) the stochastic policy ${\pi }_{\theta }\left(a_t\right|s_t)$ parameterized by $\theta$. In this context, the design objective of the control policy ${\pi }_t^{*}$ is to learn a distributional model of returns for each state-action pair, thereby reducing estimation errors in Q-value approximation and improving both the stability and performance of long-term reward maximization. The optimization goal is to maximize the entropy-regularized total discounted reward while accurately capturing the uncertainty in return distributions, as formulated below:

$${\pi }_t^{*}=arg\underset{\pi }{max}\mathbb{E}\left[\sum _{t=0}^{\infty }{\gamma }^t{r}_t+\mathcal{H}\left(\pi (·|s_t)\right)\right]$$

(21)

where $\mathbb{E}[·]$ denotes the expectation over environment dynamics and the policy; $\gamma \in [0,1]$ is the discount factor; $\mathcal{H}\left(\pi (·|s_t)\right)$ is the entropy of the action distribution under the policy at state $s_t$, which encourages exploration; the reward function $r_t$ is designed to incorporate multiple key performance indicators, such as fuel economy, component durability, and thermal management effectiveness, to reflect the system’s overall efficiency and reliability at each time step t.

The state space $S_t$ of the FCHEVs EMS agent is formulated to represent the key variables influencing energy flow and system performance. It includes two primary parameters: (1) vehicle power demand, and (2) SoC. These variables are essential for making intelligent decisions regarding power allocation between the fuel cell and energy storage system. Specifically, the state vector at time step t is defined as:

$$S_t=[P_{req},So{C}_t]$$

(22)

where $P_{req}$ denotes the total power demand of the vehicle at time t, determined by the driver’s torque request and vehicle dynamics; $So{C}_t$ represents the current state of charge (%) of the lithium-ion battery, which directly affects the allowable discharging/charging behavior and overall energy availability.

This compact yet informative state representation ensures that the DRL agent can effectively learn the trade-off between hydrogen consumption and battery usage, particularly under varying driving conditions. To enable flexible and efficient energy distribution, the action space is designed as a continuous structure, reflecting the adjustable output capability of the fuel cell system. Specifically, the action variable $a_t$ corresponds to the fuel cell power output:

$$a_t=P_{fc}\left(t\right)\in [0,P_{fc,max}]$$

(23)

where $P_{fc}\left(t\right)$ denotes the power provided by the fuel cell at time t; $P_{fc,min}$ and $P_{fc,max}$ are the minimum and maximum allowable fuel cell power output, respectively, constrained by system efficiency and operational safety.

The remaining power required to meet the vehicle demand is supplied by the battery, calculated as:

$$P_{bat}\left(t\right)=P_{req}-P_{fc}\left(t\right)$$

(24)

This formulation ensures that the total power delivered to the drivetrain satisfies the propulsion requirement, while maintaining flexibility in energy source allocation.

By integrating this structured state-action mapping into the DRL framework, the proposed EMS enables intelligent coordination between the fuel cell and battery systems, leading to improved fuel economy and thermal performance under diverse driving conditions.

The reward terms include hydrogen consumption cost, battery SoC deviation penalty, degradation costs for both fuel cell and lithium-ion battery, and penalties related to temperature deviations from optimal operating ranges. All cost items are assigned negative values to represent energy expenditure or system loss, while weighting coefficients are used to balance the relative importance of each sub-objective. The reward function is defined as follows:

$$\begin{array}{cc}\hfill r_t& =-[{\Gamma }_{H_2}{m}_{eq}\left(t\right)+{\Gamma }_{bat}SoC\left(t\right)+{\Gamma }_{fc}\Delta So{H}_{fc}\left(t\right)\hfill \\ \hfill & +{\Gamma }_b\Delta So{H}_{bat}\left(t\right)+{\Gamma }_{ba}{T}_{bat}\left(t\right)+{\Gamma }_f{T}_{fc}\left(t\right)]\hfill \end{array}$$

(25)

where ${\Gamma }_{H_2}$ represents the unit price of hydrogen; $m_{eq}\left(t\right)$ denotes the equivalent hydrogen consumption of the entire vehicle, incorporating the battery energy consumption converted into hydrogen demand to account for its impact on overall energy use; $SoC\left(t\right)$ refers to the current battery state of charge (%), with a corresponding penalty term modeled in quadratic form to encourage maintaining SoC within a target range; $\Delta So{H}_{fc}\left(t\right)$ and $\Delta So{H}_{bat}\left(t\right)$, respectively, denote the state of health degradation per percentage for the fuel cell and lithiumion battery; $T_{bat}\left(t\right)$ and $T_{fc}\left(t\right)$ represent the temperature penalty functions for the lithiumion battery and fuel cell, aiming to guide the control policy to operate within optimal temperature ranges, thereby improving training efficiency and reducing component aging rates.

The detailed formulations of all the terms involved are adapted from the theoretical framework of [33] as follows:

$$\left\{\begin{array}{c}{\Gamma }_{H_2}{m}_{eq}\left(t\right)={\Gamma }_{H_2}\left({\displaystyle \frac{1}{{\eta }_{fc}{\rho }_{H_2}}}{P}_{fc}\left(t\right)+{\lambda }_{bat}\left(t\right)P_{bat}\left(t\right)\right)\hfill \\ {\Gamma }_{bat}SoC\left(t\right)=-{\mu }_1{(SoC\left(t\right)-So{C}_{int})}^2\hfill \\ {\Gamma }_{fc}\Delta So{H}_{fc}\left(t\right)=\Delta So{H}_{fc}\left(t\right)·{\Gamma }_{fc}/{\Gamma }_{H_2}\hfill \\ {\Gamma }_b\Delta So{H}_{bat}\left(t\right)=\Delta So{H}_{bat}\left(t\right)·{\Gamma }_b/{\Gamma }_{H_2}\hfill \\ {\Gamma }_f{T}_{fc}\left(t\right)=\left\{\begin{array}{cc}0.05\times {\displaystyle \frac{|T_{fc}-67|}{50}},\hfill & if65<T_{fc}<70\hfill \\ -0.5\times {\displaystyle \frac{|T_{fc}-67|}{50}},\hfill & otherwise\hfill \end{array}\right.\hfill \\ {\Gamma }_{ba}{T}_{bat}\left(t\right)=\left\{\begin{array}{cc}0.05\times {\displaystyle \frac{|T_b-33|}{50}},\hfill & if29<T_b<36\hfill \\ -0.5\times {\displaystyle \frac{|T_b-33|}{50}},\hfill & otherwise\hfill \end{array}\right.\hfill \end{array}\right.$$

(26)

where ${\eta }_{fc}$ denotes the global efficiency of the fuel cell system, which varies with fuel cell power; ${\rho }_{H_2}$ represents the lower heating value of hydrogen; ${\Gamma }_{fc}$ and ${\Gamma }_{bat}$ denote the unit prices of the fuel cell stack and battery pack, respectively; $T_b$ and $T_{fc}$ are the current temperatures (in °C) of the lithium-ion battery and fuel cell, respectively; $So{C}_{int}$ is the ideal initial value of the battery SoC; $P_{fc}\left(t\right)$ and $P_{bat}\left(t\right)$ are the output powers of the fuel cell and battery; ${\mu }_1$ is the weighting factor for the SoC penalty term. The detailed parameter values are shown in

Table 2. Parameters in Equation (26) and their definitions.

Parameter	Physical Meaning	Typical Value/Range	Unit
${\eta }_{\mathrm{fc}}$	Global efficiency of the fuel cell system	0.45–0.60	—
${\rho }_{{\mathrm{H}}_2}$	Lower heating value (LHV) of hydrogen	33.3	—
${\Gamma }_{\mathrm{fc}}$	Fuel cell degradation per percentage point of SOH loss	750	$
${\Gamma }_{{\mathrm{H}}_2}$	Unit price of hydrogen	16	$/kg
${\Gamma }_{\mathrm{bat}}$	Unit price of the lithiumion battery pack	200	$/kWh
${\Gamma }_{\mathrm{b}}$	Lithiumion battery degradation per percentage point of SOH loss	375	$
${\Gamma }_{\mathrm{ba}}$	Lithiumion battery temperature penalty weight	−10	—
${\Gamma }_{\mathrm{f}}$	Fuel cell temperature penalty weight	−10	—
${\mu }_1$	Weighting factor for the SoC penalty term	10	—
$T_{\mathrm{fc}}$	Current operating temperature of the fuel cell	Optimal: 65–70 °C	°C
$T_b$	Current operating temperature of the lithium-ion battery	Optimal: 29–36 °C	°C
$So{C}_{\mathrm{init}}$	Ideal initial state-of-charge of the battery	0.5–0.6	—
$P_{\mathrm{fc}}\left(t\right)$	Real-time output power of the fuel cell	Varies with demand	kW
$P_{\mathrm{bat}}\left(t\right)$	Real-time output power of the battery	Varies with demand	kW

.

Notably, for the temperature fault diagnosis-based penalty functions, a small positive penalty is applied when the temperature lies within the optimal operating range to encourage stable operation; However, once the temperature exceeds safety thresholds, a larger negative penalty is introduced to avoid performance degradation or component damage due to extreme temperatures. In the context of DRL-based EMS for FCHEVs, the design of the state and action spaces plays a critical role in enabling optimal energy distribution and system efficiency. A well-defined state space allows the agent to perceive real-time vehicle operating conditions, while an effective action space facilitates precise control over powertrain components.

This paper leverages DSAC to train the energy management agent FCHEVs, where the state space is expanded to encompass powertrain dynamics. The proposed framework integrates real-time power demand, SoH, and thermal states into the control system, enabling an EMS with near-global optimality.

DSAC learns a stochastic policy that maximizes both expected reward and entropy, thereby achieving more robust exploration and better adaptability under complex driving conditions. The actor network is trained to output a probability distribution over the fuel cell power output $P_{fc}\in [0,P_{\max }]$, denoted as ${\pi }_{\theta }\left(a\right|s_t)$, from which actions are sampled without explicit noise injection. Before training begins, the replay memory D with capacity M is initialized. Transition tuples $(s_t,a_t,r_t,s_{t+1})$ collected during interactions are stored in the experience replay buffer for off-policy learning. When the buffer reaches its maximum capacity, older data are replaced by newer transitions.

Instead of estimating the expected Q-value, DSAC models the entire distribution of returns associated with each $(s_t,a_t)$ pair. Specifically, the critic network outputs a categorical distribution over a set of discrete atoms representing possible Q-values. This distribution is updated using the distributional Bellman operator:

$$Z_{\theta ,w}(s_t,a_t)\leftarrow r_t+\gamma Z_{{\theta }^{\prime },w^{\prime }}(s_{t+1},a_{t+1})$$

(27)

where $a_{t+1}\sim {\pi }_{{\theta }^{\prime }}(·|s_{t+1})$, and $Z_{{\theta }^{\prime },w^{\prime }}$ denotes the target distribution parameterized by the target networks.

The critic loss is defined as the Kullback–Leibler (KL) divergence between the predicted distribution and the projected target distribution:

$$J\left(w\right)={\mathbb{E}}_{s_t,a_t,r_t,s_{t+1}}\left[D_{KL}\left({\mathcal{T}}^{target}\parallel Z_w(s_t,a_t)\right)\right]$$

(28)

where ${\mathcal{T}}^{target}$ refers to the distribution obtained by projecting the target values onto the support of the learned distribution.

The actor network is updated by minimizing the KL divergence between the current action distribution and the exponentiated Q-values:

$$\begin{array}{c}\hfill {\nabla }_{\theta }J\left(\theta \right)={\mathbb{E}}_{s_t}\left[{\nabla }_{a}log{\pi }_{\theta }\left(a\right|s_t)·Q(s_t,a)+\alpha {\nabla }_{a}log{\pi }_{\theta }\left(a\right|s_t)\right]\\ \hfill \approx {\displaystyle \frac{1}{M}}\sum _{m=1}^M{\nabla }_{a}log{\pi }_{\theta }\left(a_j\right|s_j)·\left(Q(s_j,a_j)+\alpha log{\pi }_{\theta }\left(a_j\right|s_j)\right)\end{array}$$

(29)

The target networks are updated using a slow-moving average mechanism parameterized by ${\sigma }_{\theta }$ and ${\sigma }_w$, such as in the rule:

$$\begin{array}{cc}\hfill w^{\prime }& \leftarrow {\sigma }_{w}w+(1-{\sigma }_w)w^{\prime },\hfill \\ \hfill {\theta }^{\prime }& \leftarrow {\sigma }_{\theta }\theta +(1-{\sigma }_{\theta }){\theta }^{\prime }.\hfill \end{array}$$

(30)

This means that the new target actor parameters ($w^{\prime }$) are a weighted average of the current evaluate actor parameters ($w^{\prime }$) and the previous target actor parameters ($w^{\prime }$). The weight ${\sigma }_w$ determines how much the target network "follows" the evaluate network. Similarly, the new target critic parameters (${\theta }^{\prime }$) are a weighted average of the current evaluate critic parameters ($\theta$) and the previous target critic parameters (${\theta }^{\prime }$).

3.2. Safe Constriction Control System

To mitigate the safety issues of FCHEVs caused by the unconstrained exploration of RL agents (such as battery overcharging/overdischarging and overheating), a thermal safe constriction controller (TSCC) system has been designed between the RL agent and the control system. This system ensures the reliable operation of the energy sources. The specific design process is as follows: Define the state variables $P_{bat},T_F,T_{bat}$, the output variables $P_{bat},T_F,T_{bat}$, the control variables $u_2=P_{fc}$, and the external disturbances ${\omega }_2=P_{req}$ of the energy management system. A flexible constraint controller is designed based on the state variables, as follows:

$$\begin{array}{cc}\hfill & \underset{P_{fc}^{\prime }}{min}{J}_{Tscc}={\left[P_{fc}^{\prime }\left(t\right)-P_{fc}\left(t\right)\right]}^2\hfill \\ \hfill & s.t.T_{fc}(t+1)=T_{fc}\left(t\right)+{\displaystyle \frac{P_{fc,loss}\left(t\right)-Q_{fc,coolant}\left(t\right)-Q_{fc,e}\left(t\right)}{m_{fc}·C_{s,fc}}}dt\hfill \\ \hfill & T_{lib}(t+1)=T_{lib}\left(t\right)+{\displaystyle \frac{P_{lib,loss}\left(t\right)-Q_{lib,coolant}\left(t\right)}{m_{lib}·C_{s,lib}}}dt\hfill \\ \hfill & P_{bat,min}\le P_{bat}^{\prime }\left(t\right)\le P_{bat,max}\hfill \\ \hfill & P_{fc,min}\le P_{fc}^{\prime }\left(t\right)\le P_{fc,max}\hfill \\ \hfill & P_{fc}^{\prime }(t-1)-100\le P_{fc}^{\prime }\left(t\right)\le P_{fc}^{\prime }(t-1)+100\hfill \end{array}$$

(31)

In this equation, $P_{fc}^{\prime }$ represents the output of the TSCC system under the safety intervention of the car-following control system. lt is important to note that when the system state satisfies the safety constraints under the action $P_{fc}\left(t\right)$ of the vehicle control agent, then $P_{fc}^{\prime }=P_{fc}\left(t\right)$, indicating that TSCC does not interfere with the agent’s autonomous exploration. The TSCC becomes active only when the system state fails to meet the safety constraints under the action $P_{fc}\left(t\right)$. In such cases, TSCC projects $P_{fc}\left(t\right)$ onto the nearest action $P_{fc}^{\prime }$ that satisfies the system’s safety constraints through quadratic programming, thereby minimizing the impact on the agent’s exploration process and ensuring the global optimality of the solution. Furthermore, the action $P_{fc}^{\prime }\left(t\right)$ after safety intervention, along with the corresponding subsequent state, will be stored in the experience replay buffer, which in turn guides the agent in learning the system’s safety constraints.

The pseudo-code of the DSAC-based energy management strategy with energy source thermal constraints is presented in Algorithm 1.

Algorithm 1 DSAC-based energy management and energy source thermal constraint algorithm

1: Input: Required power $P_{req}$, State of Charge (SoC) of lithium battery   2: Output: Fuel cell output power $P_{fc}$   3: Initialize parameters: maximum training episodes ${\mathcal{E}}_{max}$, maximum steps per episode $n_{max}$, actor learning rate ${\alpha }_a$, critic learning rate ${\alpha }_c$, temperature parameter learning rate ${\alpha }_{\tau }$, reward discount factor $\gamma$, target network soft update ratio $\tau$, memory pool capacity $\mathcal{M}$, batch size B, target entropy $H_{target}$   4: Initialize actor and critic neural networks for both the vehicle control agent and the energy management agent   5: Initialize corresponding target networks and experience replay buffer $D_1$   6: Load $n_{max}$ time steps of leading vehicle driving condition data   7: for $e=1$ to ${\mathcal{E}}_{max}$ do   8:       Initialize the initial observation state of the fuel cell electric hybrid vehicle (FCEHV)   9:       for $k=1$ to $n_{max}$ do 10:             Determine the energy management action using the DSAC algorithm: 11:                    $P_{fc}\left(t\right)=P_{fc}(t-1)+100\times a\left(t\right)$ 12:             if $T_{bat}\le T_{bat\_max}$ then 13:                    Retain the original control action: $P_{fc}\left(t\right)$ 14:             else 15:                    Compute the safe control action $P_{fc}^{\prime }\left(t\right)$ according to Equation (31) 16:                    Deploy safe action: $P_{fc}\left(t\right)=P_{fc}^{\prime }\left(t\right)$ 17:             end if 18:             Execute $P_{fc}\left(t\right)$ in the EMS, observe next state $s(t+1)$, reward $R\left(t\right)$ 19:             Store transition tuple $[s\left(t\right),P_{fc}\left(t\right),s(t+1),R\left(t\right)]$ into replay buffer $D_1$ 20:             Update critic network using the defined loss function 21:             Update actor network via gradient descent with learning rate ${\alpha }_a$ 22:             Perform soft update on target networks with ratio $\tau$ 23:       end for 24:       Compute and record average reward: ${\displaystyle \frac{1}{n_{max}}}{\sum }_{k=1}^{n_{max}}R\left(k\right)$ 25: end for

4. Results and Discussion

This paper employs the 2WLTC driving cycle as the benchmark test condition to evaluate the performance of DRL-based EMSs, including the DDPG, TD3, SAC, and DSAC approaches. Each strategy incorporates a multi-objective reward function that considers fuel economy optimization, fuel cell and lithium battery pack lifespan, and thermal management. Particularly, the implementation of TSCC achieves coordinated optimization between system thermal safety and energy distribution efficiency under complex operating conditions.

4.1. Verification Preparation

To support the presentation of subsequent results,

Table 3. Comparison of DRL algorithms for EMS.

Algorithm	Distributional Learning	Maximum Entropy	Double Q-Network	Continuous Action	Suitable for EMS
DDPG	no	no	no	yes	yes
TD3	no	no	yes	yes	yes
SAC	no	yes	yes	yes	yes
DSAC	yes	yes	no	yes	yes

summarizes the key differences among various DRL algorithms, while

Table 4. EMSs algorithm table.

Algorithm	Fuel Consumption	Temperature and SoH	Thermal Safety Constraints
DDPG	Considering	Considering	Neglecting
TD3	Considering	Considering	Neglecting
SAC	Considering	Considering	Neglecting
DSAC	Considering	Considering	Neglecting
DDPG_TSCC	Considering	Considering	Considering
TD3_TSCC	Considering	Considering	Considering
SAC_TSCC	Considering	Considering	Considering
DSAC_TSCC	Considering	Considering	Considering

lists the DRL-based energy management strategies (EMSs) considered in this paper. Specifically, the proposed strategy in this paper is the DSAC-based EMS, which takes into account the thermal, thermal safety, and durability characteristics of the powertrain, along with flexible thermal safety constraints. The DRL-based algorithms, such as DDPG, TD3, SAC, and DSAC, focus on fuel consumption, battery life, and self-heating of lithium-ion batteries and hydrogen fuel cells in the reward function. The definitions of the DRL-based algorithms DDPG_TSCC, TD3_TSCC, SAC_TSCC, and DSAC_TSCC are similar to the previous EMSs, with the main difference being the use of the thermal safety flexible constraints. The driving scenario dataset is shown in Figure 3, where the 2WLTC driving cycle is used for training.

4.2. Energy Distribution Control Verification

Figure 4, Figure 5, Figure 6, Figure 7 and Figure 8 compare the power distribution performance between the LIBs and PEMFCs stack for the online control of DRL-based EMS. As shown in Figure 4, by using the DSAC-based EMS, the power distribution of the PEMFCs stack is smoother. This indicates that the implementation of the DSAC algorithm, along with the consideration of the powertrain’s thermal and durability characteristics, enables the power distribution to remain more concentrated within the safe operating range of the PEMFCs system, thereby reducing the degradation of the fuel cell’s lifespan due to power fluctuations. As seen in Figure 5 and Figure 7, when the powertrain’s thermal factors are incorporated, DDPG, TD3, SAC, and DSAC algorithms make certain inefficient decisions in power distribution to reduce the generation of heat in the fuel cell. Figure 6 and Figure 8 show that by using the TSCC-based EMS, the power distribution points of the PEMFCs stack become more concentrated, meaning that the implementation of the TSCC algorithm helps focus the power distribution in the higher efficiency region of the PEMFCs system, with the DSAC algorithm showing the most prominent effect. Figure 9 illustrates the frequency distribution of battery power demand across different strategies, highlighting that DSAC-EMS significantly reduces occurrences of high-magnitude power transients—thereby mitigating thermal runaway and aging. Figure 10 shows the battery power trajectories when the TSCC is activated. Compared with Figure 9, the power trajectories of the four algorithms are relatively more concentrated, illustrating how the TSCC actively restricts extreme power outputs to maintain thermal safety. Figure 11 demonstrate that all DRL-based EMSs are capable of maintaining the SoC trajectory and returning the SoC to a level close to the initial value of the LIBs.

Furthermore, the terminal hydrogen consumption for each EMS is listed in

Table 5. Performance indicators of 2wltc driving cycle under different EMS strategies.

Strategy/Driving Cycle	Algorithm	Effective Fuel Consumption (kg)	Energy Loss ($)
RL WLTC	DDPG	0.503	0.494
	TD3	0.689	0.514
	SAC	0.613	0.502
	DSAC	0.414	0.363
TSCC-RL WLTC	DDPG	0.503	0.464
	TD3	0.675	0.495
	SAC	0.458	0.400
	DSAC	0.422	0.357

. Due to the exceptional efficiency of the PEMFCs system, the proposed EMS results in lower hydrogen consumption. In Table 5, the corrected hydrogen consumption, expressed as the overall fuel economy of the FCV powertrain, is composed of the hydrogen consumption of the PEMFCs stack and the equivalent hydrogen consumption caused by the varying terminal SoC values of the LIBs. The formula for the corrected hydrogen consumption $C_{H_2}$ is given as follows:

$$C_{H_2}=C_{fc,H_2}+C_{bat,H_2}$$

(32)

The equivalent hydrogen costs of the PEMFCs stack and LIBs are represented by $C_{fc,H_2}$ and $C_{bat,H_2}$, respectively. The total cost of the PEMFCs stack and LIBs corresponding to the DSAC-based EMS is 421.776 g, which is lower compared to other EMS based on DRL.

4.3. Temperature Control Verification

Figure 12 and Figure 13 present the comparative temperature distributions of the PEMFCs stack and LIBs. The results indicate that the operating temperature ranges for the PEMFCs stack and LIBs are 60–76 °C and 25–45 °C, respectively. Since both the PEMFCs stack and LIBs operate within the temperature ranges of 60–80 °C and 20–40 °C, respectively, the temperature curves of the PEMFCs stack and LIBs controlled by the EMS based on DDPG, TD3, SAC, and DSAC are reasonable. In practical engineering, if the battery temperature exceeds 50 °C, the BMS will force a shutdown, halting further operation. The EMS based on DDPG_TSCC, TD3_TSCC, SAC_TSCC, and DSAC_TSCC can effectively regulate the temperature of the LIBs and PEMFCs stack, limiting their maximum temperatures. The temperature curves are smoother, preventing excessive temperatures that could lead to additional losses, while also accounting for the thermal characteristics of the LIBs and PEMFCs stack to avoid overheating.

5. Conclusions

A DSAC-based thermo-electrical co-optimization EMS for FCHEVs is proposed, which explicitly integrates thermal runaway fault diagnosis and a TSCC to address thermal management and safety challenges by jointly considering fuel cell powertrain thermal characteristics, durability, and real-time fault constraints. The overheating penalty of the LIBs and PEMFCs stack, as well as the equivalent SoH degradation cost, are added to the reward space alongside conventional metrics, which include hydrogen consumption and SoC of LIBs penalties used for optimization and training. Additionally, the DSAC-based EMS introduces a single Q-learning technique to prevent Q-value overestimation in the soft Q-network. The validation of the proposed EMS is conducted in several long-term driving scenarios, covering both the training process and online control, with the online control aspects involving fuel economy, thermal stability, and SoH for the PEMFCs stack and lithiumion battery. Simulation results show the following:

• Under the same operating conditions, considering the thermal characteristics and durability of the powertrain system, the energy consumption saved by the EMS based on DSAC is 17.69%, 39.91%, and 32.46% compared to those based on DDPG, TD3, and SAC, respectively. The SoH consumption is reduced by 26.52%, 29.38%, and 27.69%, respectively. After adding TSCC, the EMS based on DDPG, TD3, SAC, and DSAC achieves energy savings of 0%, 2.03%, 25.29%, and −1.93%, respectively, while SoH consumption is reduced by 6.07%, 3.7%, 20.32%, and 1.65%, respectively.
• The proposed flexible temperature TSCC effectively regulates the temperature of the LIBs and PEMFCs stack, minimizing their peak temperatures. However, without the introduction of a thermal management system, the energy source temperature cannot be more effectively maintained within the optimal range, which may prevent reductions in energy consumption and mitigate potential lifespan degradation.
• The method has good versatility and can be extended to other types of FCVs to achieve optimal fuel economy, durability, and thermal safety of the fuel cell vehicle power system in unpredictable driving scenarios.

Despite the promising results achieved in this paper, several limitations warrant further investigation. First, the strategy is developed and evaluated using standard driving cycles (e.g., WLTC, UDDS) under static conditions, focusing on thermal management. However, real-world driving is highly dynamic, with varying traffic, weather, terrain, and driver behavior, which can challenge the DSAC algorithm under thermal constraints. To address this, future work should integrate multi-source data—such as traffic, weather, terrain, and in-vehicle CAN signals (e.g., energy use, temperatures, and driving dynamics)—to develop a more robust and generalizable energy management strategy. This will support validation in complex real-world conditions. To advance toward practical deployment, we plan to conduct hardware-in-the-Loop simulation by deploying the DSAC strategy on a real-time controller and conducting co-simulation with physical models of the fuel cell system, battery management system, and thermal management system. In the long term, the strategy will be further validated on a real vehicle platform to evaluate its energy efficiency, thermal safety, and durability performance. Second, the training of deep reinforcement learning algorithms presents several challenges, including the risk of overfitting to specific driving conditions. Advanced transfer learning and domain adaptation techniques could be investigated to enhance the generalization capability of the strategy across different vehicle models and operating scenarios. Finally, extending this framework to a broader range of new energy vehicle platforms (such as battery electric vehicles and plug-in hybrid electric vehicles) to validate its universality represents an important research direction. Furthermore, integrating our energy management model with intelligent fleet management systems (e.g., the multi-agent collaborative thermal management approach proposed by Khalatbarisoltani et al. [29]) may enable more efficient energy co-optimization.