PEMFC Thermal Management Control Strategy Based on Dual Deep Deterministic Policy Gradient

Abstract

The operational performance of proton exchange membrane fuel cells (PEMFC) is highly influenced by temperature, making effective thermal management essential. However, the multivariate coupling between pumps and radiators presents significant control challenges. To address this issue, a dual DDPG-PID control strategy is proposed, integrating temperature and flow rate variations to enhance system stability and response. Simulation results demonstrate that the proposed method significantly reduces temperature control errors and improves response time compared to conventional PID-based strategies. Specifically, the D-DDPG PID achieves a temperature error reduction of up to 75.4% and shortens the average tuning time by up to 25.6% compared to PSO-PID. Furthermore, the strategy optimizes cooling system performance, demonstrating its effectiveness in PEMFC thermal management.

1. Introduction

In recent years, with the gradual reduction of global dependence on coal, oil, and natural gas, and in response to the environmental pollution caused by the use of fossil fuels, there has been increasing attention on exploring new alternative energy sources [1,2,3]. Among these, proton exchange membrane fuel cells (PEMFCs) have garnered significant attention due to their numerous advantages, such as low emissions, high efficiency, low operating temperature, rapid start-up capability, and long lifespan [4,5,6]. As an efficient and environmentally friendly energy conversion technology, PEMFC is based on electrochemical principles, and the proton directional migration is achieved through the anodic oxidation of hydrogen and cathodic reduction of oxygen in the membrane electrode module, which ultimately generates clean water and releases electrical energy [7,8]. However, during the electrochemical reaction process, fuel cells generate not only electrical energy but also heat, which leads to an increase in the operating temperature of the PEMFC [9,10]. While a higher temperature can enhance reaction kinetics and conductivity, thereby improving fuel cell efficiency, excessive temperatures may result in the dehydration and degradation of the proton exchange membrane, ultimately compromising the long-term stability and lifespan of the fuel cell [11,12,13]. Therefore, the development of efficient thermal management control strategies for PEMFC systems has garnered significant attention from the academic community, leading to extensive research by scholars in this field.

To address the impact of temperature on fuel cell performance, many researchers have conducted studies on thermal management control strategies for fuel cells, aiming to improve system stability, enhance response speed, and optimize energy consumption. For instance, Zhao et al. [14] developed a thermal management system model and analyzed the impact of temperature on PEMFC performance. Yu et al. [15] proposed controlling the stack temperature by adjusting the coolant flow rate, while considering the parasitic power of the cooling system. Liu et al. [16] introduced a power control strategy to address the strong nonlinearity of PEMFC systems, achieving performance improvements over traditional PID (proportion–integration–differentiation) control. Li et al. [17] developed a multivariable coordinated controller to resolve the coupling issue between the pump and the radiator, thereby enhancing control accuracy. Yin et al. [18] designed a maximum efficiency control strategy (MECS) based on temperature variations in the PEMFC, which outperforms traditional PID methods in temperature-tracking performance. Zhao et al. [19] applied a diploid genetic algorithm to optimize the temperature control of the membrane electrode assembly (MEA). Cheng et al. [20] combined nonlinear feedforward control with a linear quadratic regulator (LQR) to enhance the control performance of the fuel cell’s cooling system. Jia Y. et al. [21] proposed a variable fuzzy PID controller that adjusts control accuracy through the proportional factor. Yu et al. [22] integrated a PID controller with a thermal circuit to reduce parasitic power loss and alleviate the temperature rise in the catalyst layer. Zhi et al. [23] introduced a piecewise predictive feedback control method based on PID to address the limitations of traditional PID control. Ou et al. [24] implemented a feedforward fuzzy PID controller for compressor temperature control, effectively reducing parasitic power loss. Hasheminejad et al. [25] proposed a fuzzy control strategy that ensures optimal operation conditions for PEMFCs by modeling the coolant loop. Mousakazemi et al. [26] optimized fuzzy controller parameters using a genetic algorithm, significantly improving overall control performance. Wang et al. [27] developed a fuzzy rule-based controller for PEMFC system temperature control by adjusting the fan speed. Tan et al. [28] applied particle swarm optimization (PSO) to optimize the operating temperature of the fuel stack. Yan et al. [29] introduced a fault-tolerant control method for sensor fault detection, ensuring that the PEMFC temperature remains near the reference value, even during sensor faults. Ahmadi S et al. [30] applied PSO to optimize PID control, achieving significant control performance improvements. Li et al. [31] used a BP neural network-based PID controller to regulate PEMFC system temperature, demonstrating good robustness and practical application effectiveness.

Despite the valuable contributions of these studies, there are still some limitations in their control strategies. Most of the control methods (e.g., PID and fuzzy logic controllers) are based on predefined models and cannot dynamically adapt to the complex nonlinear behavior of PEMFC systems. These methods struggle to provide stability and optimal performance in the face of the high multivariate coupling and system delays inherent to PEMFCs. As a result, temperature overshoots and slow response times often occur, thus affecting the overall efficiency and stability of the system. For example, while fuzzy logic and PID controllers can improve stability to some extent, their performance is still limited by having too many parameters and the need for extensive tuning under different operating conditions. These challenges are further exacerbated by the complex nonlinear dynamics of PEMFCs, which lead to difficulties in model matching, as well as substantial delays in the control process.

In the past few years, with the rapid development of intelligent technology, algorithms based on deep reinforcement learning (DRL) have gradually been widely used in the field of new energy, such as the deep Q-network (DQN) [32], deep deterministic policy gradient (DDPG) [33], twin delayed deep deterministic policy gradient (TD3) [34], and other advanced algorithms. These algorithms do not rely on precise mathematical models and can continuously learn and optimize control strategies through interaction with the environment, exhibiting strong adaptability, especially in complex and dynamically changing systems. Currently, deep reinforcement-learning algorithms have been widely applied to energy management in fuel cell hybrid electric vehicle systems [35]. In this context, Huang et al. [36] proposed an energy management solution for fuel cell hybrid systems based on dual deep deterministic policy gradient (Dual DDPG), where in pure electric mode, DDPG optimizes the power distribution between the battery and the supercapacitor, reducing the battery’s depth of discharge. In extended-range mode, DDPG coordinates the output of the fuel cell, battery, and super-capacitor, improving fuel cell efficiency by 0.94% and significantly reducing power fluctuations. Although deep reinforcement learning has achieved promising results in many fields, research on reinforcement-learning algorithms in fuel cell system control, especially in thermal management control, is still limited, and relevant literature is scarce.

Inspired by the energy management of fuel cell hybrid vehicles based on dual DDPG, and in order to overcome the limitations of traditional control algorithms, the author innovatively introduced reinforcement-learning algorithms into fuel cell thermal management. A dual DDPG-PID control strategy was proposed to address the nonlinear and multivariable coupling characteristics of the fuel cell thermal management system. As an advanced adaptive method for PEMFC temperature regulation, this hybrid strategy combines the DDPG algorithm with a PID controller, allowing for dynamic adjustment of the control strategy based on real-time system feedback. Unlike traditional methods, the DDPG algorithm can learn and continuously improve the control strategy in real time according to the state of the fuel cell system, achieving precise optimization of the PID controller parameters. This effectively solves the latency issue commonly encountered in traditional controllers. Therefore, this hybrid control strategy provides a novel approach to thermal management control for fuel cells.

This paper is organized as follows. In Section 2, the level model of the fuel cell system will be developed; in Section 3, the framework of the proposed Dual DDPG control strategy will be discussed. In Section 4, the simulation will be validated under continuous step-loading conditions, and the results will be described and analyzed in detail. Finally, the full paper will be summarized and conclusions will be given in Section 5.

2. PEMFC System Model

The schematic representation of the PEMFC system is illustrated in Figure 1. This system comprises two main subsystems for the fuel supply: the air supply system and the hydrogen supply system. The air supply system facilitates the transport of air to the cathode side, while the hydrogen supply system directs hydrogen from a high-pressure cylinder to the anode side through a pressure valve. Additionally, a thermal management system is integrated to ensure effective cooling of the fuel cell during operation.

2.1. Fuel Cell Modeling

The voltage of a fuel cell is influenced by thermodynamic and electrochemical losses. The actual output voltage can be expressed as:

$$\mathrm{V}=\mathrm{E}-{\mathrm{V}}_{\mathrm{a}\mathrm{c}\mathrm{t}}-{\mathrm{V}}_{\mathrm{o}\mathrm{h}\mathrm{m}}-{\mathrm{V}}_{\mathrm{c}\mathrm{o}\mathrm{n}\mathrm{c}}$$

(1)

where E denotes the Nernst electromotive force, ${\mathrm{V}}_{\mathrm{a}\mathrm{c}\mathrm{t}}$ denotes the voltage loss due to activation polarization, ${\mathrm{V}}_{\mathrm{o}\mathrm{h}\mathrm{m}}$ denotes the voltage drop due to ohmic loss, and ${\mathrm{V}}_{\mathrm{c}\mathrm{o}\mathrm{n}\mathrm{c}}$ denotes the voltage loss due to concentration polarization.

The calculation of Nernst electromotive force is as follows [37]:

$$\mathrm{E}=1.229-8.5\times {10}^{-4}({\mathrm{T}}_{\mathrm{f}\mathrm{c}}-{\mathrm{T}}_0)+{\displaystyle \frac{\mathrm{R}{\mathrm{T}}_{\mathrm{f}\mathrm{c}}}{2\mathrm{F}}}(\mathrm{l}\mathrm{n}{\mathrm{P}}_{{\mathrm{H}}_2}+{\displaystyle \frac{1}{2}}\mathrm{l}\mathrm{n}{\mathrm{P}}_{{\mathrm{O}}_2})$$

(2)

where ${\mathrm{T}}_0$ is the ambient temperature, ${\mathrm{T}}_{\mathrm{f}\mathrm{c}}$ is the fuel cell temperature, ${\mathrm{P}}_{{\mathrm{H}}_2}$ and ${\mathrm{P}}_{{\mathrm{O}}_2}$ are the pressures of hydrogen and oxygen entering the stack, respectively, and $\mathrm{F}$ is the Faraday constant.

Activation polarization is the voltage loss occurring during the electrochemical reaction in a fuel cell due to the limitation of the electrode reaction rate, primarily caused by the kinetic lag of the electrode catalysis reaction [38,39]. The empirical formula for calculating activation polarization is as follows:

$${\mathrm{V}}_{\mathrm{a}\mathrm{c}\mathrm{t}}={\mathrm{V}}_1+{\mathrm{V}}_{\mathsf{\alpha }}(1-{\mathrm{e}}^{{-\mathrm{i}\mathrm{c}}_1})$$

(3)

The Ohmic polarization is the potential loss in a fuel cell caused by the resistance of the electrolyte and the electrode materials [40]. Its empirical calculation formula is:

$${\mathrm{V}}_{\mathrm{o}\mathrm{h}\mathrm{m}}=\mathrm{i}\times {\mathrm{R}}_{\mathrm{o}\mathrm{h}\mathrm{m}}$$

(4)

$${\mathrm{R}}_{\mathrm{o}\mathrm{h}\mathrm{m}}={\displaystyle \frac{{\mathrm{t}}_{\mathrm{m}}}{{\mathsf{\sigma }}_{\mathrm{m}}}}$$

(5)

where ${\mathrm{t}}_{\mathrm{m}}$ is the membrane thickness, and ${\mathsf{\sigma }}_{\mathrm{m}}$ is a function of the membrane’s water content and the fuel cell temperature.

Concentration polarization is the potential loss in a fuel cell caused by the rapid consumption of reactants at the electrode surface, leading to a decrease in the concentration of reactants at the electrode surface [41]. The empirical formula for calculating the concentration polarization is as follows:

$${\mathrm{V}}_{\mathrm{c}\mathrm{o}\mathrm{n}\mathrm{c}}=\mathrm{i}({\mathrm{c}}_2{\displaystyle \frac{\mathrm{i}}{{\mathrm{i}}_{\mathrm{m}\mathrm{a}\mathrm{x}}}}{)}^{{\mathrm{c}}_3}$$

(6)

To validate the accuracy of the developed model, a simulation was conducted based on the experimental conditions reported in Reference [42], where the chimney temperature was set to 80 °C and the inlet pressures of both the cathode and anode were set to 2 bar. The simulated results were then fitted to the experimental data from the literature to analyze the consistency between the predicted V-I characteristics and the reference data. As shown in Figure 2 of the manuscript, the simulation results closely match the experimental data, with a maximum deviation of less than 2% across the entire operating range, confirming the model’s accuracy.

In the fuel cell system model, it is necessary to include the air compressor model, air cooler model, humidifier model, and gas transmission pipeline model. The gas transmission pipeline model represents the mass flow rate of gas transport and the pressure variations during the transmission process, which can be calculated using Equations (A1)–(A35). The auxiliary system models, including the air compressor, air cooler, and humidifier models, primarily serve to supply air with appropriate pressure and humidity to maintain the operation of the fuel cell, and their behavior can be determined using Equations (A37)–(A55). Additionally, the parameters used for modeling are provided in

Table A1. Parameters used in the modeling of the PEMFC system.

Symbol	Variable	Value
$\mathsf{\gamma }$	Ratio of special heat of air	1.4
${\mathrm{C}}_{\mathrm{P}}$	Constant pressure-specific heat of air	$1004 \mathrm{J}/(\mathrm{m}\mathrm{o}\mathrm{l}·\mathrm{K})$
${\mathrm{R}}_{\mathrm{a}}$	Air gas constant	$286.9 \mathrm{J}/(\mathrm{k}\mathrm{g}·$ K)
${\mathsf{\rho }}_{\mathrm{a}}$	Air density	$1.23 \mathrm{k}\mathrm{g}/{\mathrm{m}}^3$
${\mathrm{d}}_{\mathrm{c}}$	Compressor diameter	$0.2286 \mathrm{m}$
${\mathrm{a}}_0$	Constant	$2.21195\times {10}^{-3}$
${\mathrm{a}}_1$	Constant	$-4.63685\times {10}^{-5}$
${\mathrm{a}}_2$	Constant	$-5.36235\times {10}^{-4}$
${\mathrm{a}}_3$	Constant	$-2.70399\times {10}^{-4}$
${\mathrm{a}}_4$	Constant	$-3.69906\times {10}^{-5}$
${\mathrm{b}}_0$	Constant	$0.43331$
${\mathrm{b}}_1$	Constant	$-0.68344$
${\mathrm{b}}_2$	Constant	$0.80121$
${\mathrm{b}}_3$	Constant	$-0.42937$
${\mathrm{b}}_4$	Constant	$0.10581$
${\mathrm{b}}_5$	Constant	$-9.78775\times {10}^{-3}$
${\mathrm{c}}_0$	Constant	$2.44419$
${\mathrm{c}}_1$	Constant	$-1.34837$
${\mathrm{c}}_2$	Constant	$1.76567$
${\mathsf{\eta }}_{\mathrm{c}\mathrm{m}}$	Efficiency of motor	$98\mathrm{\%}$
${\mathsf{\eta }}_{\mathrm{c}\mathrm{p}}$	Maximum efficiency of compressor	$80\mathrm{\%}$
${\mathrm{J}}_{\mathrm{c}\mathrm{p}}$	Combined inertia	$5\times {10}^{-5} \mathrm{k}\mathrm{g}·{\mathrm{m}}^2$
${\mathrm{R}}_{\mathrm{c}\mathrm{m}}$	Motor constant	$0.82$
${\mathrm{k}}_{\mathrm{t}}$	Motor constant	$0.0153 \mathrm{N}-\mathrm{m}/\mathrm{A}\mathrm{m}\mathrm{p}$
${\mathrm{k}}_{\mathrm{v}}$	Motor constant	$0.153 \mathrm{V}/(\mathrm{r}\mathrm{a}\mathrm{d}/\mathrm{s})$
${\mathrm{V}}_{\mathrm{s}\mathrm{m}}$	Supply manifold volume	$0.02 {\mathrm{m}}^3$
${\mathrm{k}}_{\mathrm{s}\mathrm{m},\mathrm{o}\mathrm{u}\mathrm{t}}$	Out orifice constant	$0.3629\times {10}^{-5} \mathrm{k}\mathrm{g}/(\mathrm{s}·\mathrm{p}\mathrm{a})$
${\mathrm{V}}_{\mathrm{r}\mathrm{m}}$	Return manifold volume	$0.005 {\mathrm{m}}^3$
${\mathrm{K}}_1$	Gain	$2.1 \mathrm{k}\mathrm{g}/(\mathrm{s}·\mathrm{k}\mathrm{P}\mathrm{a})$
${\mathrm{K}}_2$	Drop	$0.94$
${\mathrm{C}}_{\mathrm{D},\mathrm{r}\mathrm{m}}$	Constant term	$0.0124$
${\mathrm{A}}_{\mathrm{T},\mathrm{r}\mathrm{m}}$	Return manifold throttle area	$0.002 {\mathrm{m}}^2$

.

2.2. PEMFC Thermal Management System Modeling

The thermal management system of a PEMFC comprises components, including a water pump, radiator, water tank, stack, bypass valve, etc. [43]. Its associated control framework diagram is illustrated in Figure 3. Subsequently, utilize principles of mole conservation, energy conservation, and empirical formulas to model these components.

2.2.1. PEMFC Thermal Modeling

The energy balance of a PEMFC is primarily maintained through several factors [44]: the total power of electrochemical reactions ${\mathrm{Q}}_{\mathrm{t}\mathrm{o}\mathrm{t}}$, the energy brought into the stack by the incoming gases ${\mathrm{Q}}_{\mathrm{i}\mathrm{n}}$, the energy carried away by the gases exiting the stack ${\mathrm{Q}}_{\mathrm{o}\mathrm{u}\mathrm{t}}$, the power consumed by the load ${\mathrm{P}}_{\mathrm{s}\mathrm{t}}$, the heat carried away by the circulating coolant ${\mathrm{Q}}_{\mathrm{c}\mathrm{l}}$, and the energy exchange between the PEMFC stack and the surrounding environment ${\mathrm{Q}}_{\mathrm{a}\mathrm{m}\mathrm{b}}$. This results in the variation of stack temperature ${\mathrm{T}}_{\mathrm{s}\mathrm{t}}$.

$${\mathrm{m}}_{\mathrm{s}\mathrm{t}}{\mathrm{C}}_{\mathrm{p},\mathrm{s}\mathrm{t}}{\displaystyle \frac{{\mathrm{d}\mathrm{T}}_{\mathrm{s}\mathrm{t}}}{\mathrm{d}\mathrm{t}}}={\mathrm{Q}}_{\mathrm{t}\mathrm{o}\mathrm{t}}+{\mathrm{Q}}_{\mathrm{i}\mathrm{n}}-{\mathrm{P}}_{\mathrm{s}\mathrm{t}}-{\mathrm{Q}}_{\mathrm{o}\mathrm{u}\mathrm{t}}-{\mathrm{Q}}_{\mathrm{c}\mathrm{l}}-{\mathrm{Q}}_{\mathrm{a}\mathrm{m}\mathrm{b}}$$

(7)

$${\mathrm{Q}}_{\mathrm{t}\mathrm{o}\mathrm{t}}=∆\mathrm{H}\times {\mathrm{N}}_{\mathrm{H}2,\mathrm{r}\mathrm{e}\mathrm{a}\mathrm{c}\mathrm{t}}$$

(8)

$$\begin{gathered} {\mathrm{Q}}_{\mathrm{i}\mathrm{n}}=\left({\mathrm{W}}_{\mathrm{H}2,\mathrm{a}\mathrm{n},\mathrm{i}\mathrm{n}}{\mathrm{C}}_{\mathrm{p},\mathrm{H}2}+{\mathrm{W}}_{\mathrm{v},\mathrm{a}\mathrm{n},\mathrm{i}\mathrm{n}}{\mathrm{C}}_{\mathrm{p},\mathrm{v}}\right)\times \left({\mathrm{T}}_{\mathrm{a}\mathrm{n},\mathrm{i}\mathrm{n}}-{\mathrm{T}}_{\mathrm{a}\mathrm{t}\mathrm{m}}\right) \\ +({\mathrm{W}}_{\mathrm{a},\mathrm{c}\mathrm{a},\mathrm{i}\mathrm{n}}{\mathrm{C}}_{\mathrm{p},\mathrm{a}\mathrm{i}\mathrm{r}}+{\mathrm{W}}_{\mathrm{v},\mathrm{c}\mathrm{a},\mathrm{i}\mathrm{n}}{\mathrm{C}}_{\mathrm{p},\mathrm{v}})\times ({\mathrm{T}}_{\mathrm{c}\mathrm{a},\mathrm{i}\mathrm{n}}-{\mathrm{T}}_{\mathrm{a}\mathrm{t}\mathrm{m}}) \end{gathered}$$

(9)

$$\begin{gathered} {\mathrm{Q}}_{\mathrm{o}\mathrm{u}\mathrm{t}}=({\mathrm{W}}_{\mathrm{H}2,\mathrm{a}\mathrm{n},\mathrm{o}\mathrm{u}\mathrm{t}}{\mathrm{C}}_{\mathrm{p},\mathrm{H}2}+{\mathrm{W}}_{\mathrm{v},\mathrm{a}\mathrm{n},\mathrm{o}\mathrm{u}\mathrm{t}}{\mathrm{C}}_{\mathrm{p},\mathrm{v}}+{\mathrm{W}}_{\mathrm{O}2,\mathrm{c}\mathrm{a},\mathrm{o}\mathrm{u}\mathrm{t}}{\mathrm{C}}_{\mathrm{p},\mathrm{O}2} \\ +{\mathrm{W}}_{\mathrm{N}2,\mathrm{c}\mathrm{a},\mathrm{o}\mathrm{u}\mathrm{t}}{\mathrm{C}}_{\mathrm{p},\mathrm{N}2}+{\mathrm{W}}_{\mathrm{v},\mathrm{c}\mathrm{a},\mathrm{o}\mathrm{u}\mathrm{t}}{\mathrm{C}}_{\mathrm{p},\mathrm{v}}+{\mathrm{W}}_{\mathrm{l},\mathrm{g}\mathrm{e}\mathrm{n}}{\mathrm{C}}_{\mathrm{p},\mathrm{l}})\times ({\mathrm{T}}_{\mathrm{s}\mathrm{t}}-{\mathrm{T}}_{\mathrm{a}\mathrm{t}\mathrm{m}}) \end{gathered}$$

(10)

$${\mathrm{Q}}_{\mathrm{c}\mathrm{l}}={\mathrm{W}}_{\mathrm{c}\mathrm{l}}{\mathrm{C}}_{\mathrm{p},\mathrm{l}1}({\mathrm{T}}_{\mathrm{s}\mathrm{t}}-{\mathrm{T}}_{\mathrm{s}\mathrm{t}}^{\mathrm{i}\mathrm{n}})$$

(11)

$${\mathrm{Q}}_{\mathrm{a}\mathrm{m}\mathrm{b}}=({\mathrm{T}}_{\mathrm{s}\mathrm{t}}-{\mathrm{T}}_{\mathrm{a}\mathrm{t}\mathrm{m}})/{\mathrm{R}}_{\mathrm{t}}$$

(12)

In the above formula, ${\mathrm{C}}_{\mathrm{p},\mathrm{a}\mathrm{i}\mathrm{r}}$ represents the specific heat of air; ${\mathrm{C}}_{\mathrm{p},\mathrm{O}2}$ represents the specific heat of oxygen; ${\mathrm{C}}_{\mathrm{p},\mathrm{H}2}$ represents the specific heat of hydrogen; ${\mathrm{C}}_{\mathrm{p},\mathrm{N}2}$ represents the specific heat of nitrogen; ${\mathrm{C}}_{\mathrm{p},\mathrm{v}}$ represents the specific heat of water vapor; ${\mathrm{C}}_{\mathrm{p},\mathrm{l}}$ represents the specific heat of liquid water; ${\mathrm{C}}_{\mathrm{p},\mathrm{l}1}$ represents the specific heat of coolant; ${\mathrm{R}}_{\mathrm{t}}$ represents the thermal equivalent of the fuel cell; and ${\mathrm{W}}_{\mathrm{c}\mathrm{l}}$ is the coolant flow rate.

2.2.2. Radiator Modeling

The radiator controls the coolant temperature at the stack inlet by circulating coolant through forced-air convection from the cooling fan. The heat transfer to the environment depends on factors like the radiator’s surface area, heat transfer coefficient, and ambient temperature [45], and its heat transfer rate is linearly related to airflow and temperature difference.

$${\mathrm{Q}}_{\mathrm{r}\mathrm{a}\mathrm{d}}={\mathrm{C}}_{\mathrm{p},\mathrm{a}\mathrm{i}\mathrm{r}}{\mathrm{W}}_{\mathrm{f}\mathrm{a}\mathrm{n}}({\mathrm{T}}_{\mathrm{r}\mathrm{a}\mathrm{d},\mathrm{a}\mathrm{i}\mathrm{r}}^{\mathrm{o}\mathrm{u}\mathrm{t}}-{\mathrm{T}}_{\mathrm{a}\mathrm{t}\mathrm{m}})$$

(13)

where ${\mathrm{W}}_{\mathrm{f}\mathrm{a}\mathrm{n}}$ represents the airflow rate of the radiator fan, and ${\mathrm{T}}_{\mathrm{r}\mathrm{a}\mathrm{d},\mathrm{a}\mathrm{i}\mathrm{r}}^{\mathrm{o}\mathrm{u}\mathrm{t}}$ is the air temperature at the outlet of the radiator. The value represents the average temperature of the coolant at the inlet and outlet of the radiator.

The temperature of the coolant upon exiting the radiator is determined by the following formula.

$${\mathrm{T}}_{\mathrm{r}\mathrm{a}\mathrm{d}}^{\mathrm{o}\mathrm{u}\mathrm{t}}={\mathrm{T}}_{\mathrm{r}\mathrm{a}\mathrm{d}}^{\mathrm{i}\mathrm{n}}-{\displaystyle \frac{{\mathrm{Q}}_{\mathrm{r}\mathrm{a}\mathrm{d}}}{{\mathrm{W}}_{\mathrm{r}\mathrm{a}\mathrm{d}}{\mathrm{C}}_{\mathrm{p},\mathrm{l}1}}}$$

(14)

where ${\mathrm{T}}_{\mathrm{r}\mathrm{a}\mathrm{d}}^{\mathrm{i}\mathrm{n}}={\mathrm{T}}_{\mathrm{s}\mathrm{t}}$ is the temperature of the coolant entering the radiator, and ${\mathrm{W}}_{\mathrm{r}\mathrm{a}\mathrm{d}}$ is the flow rate of the coolant passing through the radiator.

3. Deep Deterministic Policy Gradient Strategy

3.1. DDPG Algorithm Description

A DDPG (deep deterministic policy gradient) is a reinforcement-learning algorithm based on the actor–critic structure, which is suitable for continuous control problems, while the thermal management of fuel cells involves the regulation of continuous variables such as temperature, which gives DDPG a significant advantage in this scenario. Compared with the DQN algorithm, the DDPG algorithm is well adapted to complex systems with continuous state spaces, while the DQN algorithm mainly targets discrete action spaces and is not suitable for fuel cell thermal management control. Compared with PPO and SAC, DDPG responds faster to dynamic changes in the environment and can make efficient decisions in complex thermal management systems. Meanwhile, the stability and sample efficiency of DDPG perform better in reinforcement-learning applications and can work with PID controllers to achieve accurate temperature control and reduce over-regulation and oscillation problems. Therefore, DDPG combined with PID can provide smoother, stable, and efficient thermal management strategies. This chapter is followed by a detailed description of the DDPG algorithm and the framework of the dual DDPG-PID-based thermal management control strategy.

The DDPG algorithm exhibits strong applicability in addressing the control problem of continuous action spaces in PEMFC systems [46]. It amalgamates principles from both deep learning and deterministic policy into an actor–critic architecture. The actor network’s primary role is to acquire a deterministic policy, mapping environmental states to appropriate actions. Given the environmental state as input, the actor network outputs the action to be executed. Throughout the training phase, the actor network endeavors to maximize the cumulative reward derived from the current policy. In contrast, the critic network’s responsibility lies in estimating the value function, specifically the Q-value, for a given state–action pair. Utilizing the current environmental state and action as input, the critic network computes the corresponding Q-value, which represents the cumulative reward achievable from the subsequent state following action execution. Employing the Q-learning approach, the critic network iteratively refines its parameter values by minimizing the temporal difference (TD) error. This iterative refinement process enhances the Critic network’s accuracy in estimating Q-values, consequently bolstering the DDPG algorithm’s efficacy in navigating continuous action spaces.

The DDPG algorithm utilizes two distinct networks: the actor policy network and the critic value network, each consisting of both an online and a target network [47]. The online actor network, denoted as ${\mathsf{\mu }}_{\mathsf{\theta }}$, is responsible for generating actions directly, expressed as ${\mathrm{a}}_{\mathrm{t}}=\mathsf{\vartheta }({\mathrm{s}}_{\mathrm{t}}|{\mathsf{\theta }}^{\mathsf{\vartheta }})$, Simultaneously, an online Critic network evaluates the Q-value, represented as $\mathrm{Q}=\mathrm{Q}({\mathrm{s}}_{\mathrm{t}},{\mathrm{a}}_{\mathrm{t}}|{\mathsf{\theta }}^{\mathrm{q}})$. Subsequent to the action’s interaction with the environment, the target critic network employs the action from the target actor network and the current state observation to compute the target output Q-value, thereby adjusting the value function ${\mathrm{Q}}_{\mathsf{\varphi }}(\mathrm{s},\mathrm{a})$. Here, ${\mathsf{\theta }}^{\mathsf{\vartheta }}$ and ${\mathsf{\theta }}^{\mathrm{q}}$ denote the parameters of the online actor and online critic, respectively, while ${\mathrm{s}}_{\mathrm{t}}$ represents the agent’s state at time t. During the update of the actor and critic network parameters, the process is initiated by randomly sampling a minibatch from the time series [${\mathrm{s}}_{\mathrm{t}},{\mathrm{a}}_{\mathrm{t}},{\mathrm{r}}_{\mathrm{t}},{\mathrm{s}}_{\mathrm{t}+1}$] stored in the experience pool R. This approach enhances algorithm stability and convergence speed, significantly reducing the variance of the Q value and facilitating the computation of the loss function L. The estimated optimal Q value and loss function are defined as:

$${\mathrm{y}}_{\mathrm{k}}={\mathrm{r}}_{\mathrm{k}}+\mathsf{\gamma }{\mathrm{Q}}^{\prime }({\mathrm{s}}_{\mathrm{k}+1},{\mathsf{\vartheta }}^{\prime }({\mathrm{s}}_{\mathrm{k}+1}|{\mathsf{\theta }}^{{\mathsf{\vartheta }}^{\prime }})|{\mathsf{\theta }}^{{\mathrm{q}}^{\prime }})$$

(15)

$$\mathrm{L}={\displaystyle \frac{1}{\mathrm{N}}}{\sum }_{\mathrm{k}}({\mathrm{y}}_{\mathrm{k}}-\mathrm{Q}{\mathrm{s}}_{\mathrm{k}},{\mathrm{a}}_{\mathrm{k}}|{\mathsf{\theta }}^{\mathrm{q}}{)}^2$$

(16)

where ${\mathrm{r}}_{\mathrm{k}}$ denotes the reward at a specific time step, and $\mathsf{\gamma }$ represents the discount factor within the range [0, 1]. N indicates the count of state–action pairs randomly sampled from the experience replay buffer R. The primary objective of the optimal policy is to maximize the expected Q value. The actor network updates its parameters ${\mathsf{\theta }}^{\mathsf{\vartheta }}$ by utilizing the Q-value’s derivative in the direction of maximum Q. The update process follows:

$${\nabla }_{{\mathsf{\theta }}^{\mathsf{\vartheta }}}\mathrm{J}\approx {\nabla }_{\mathrm{a}}\mathrm{Q}(\mathrm{s},\mathrm{a}|{\mathsf{\theta }}^{\mathrm{q}}){|}_{\mathrm{s}={\mathrm{s}}_{\mathrm{k}},\mathrm{a}=\mathsf{\mu }\left({\mathrm{s}}_{\mathrm{k}}\right)}{\nabla }_{{\mathsf{\theta }}^{\mathsf{\vartheta }}}\mathsf{\vartheta }(\mathrm{s}|{\mathsf{\theta }}^{\mathsf{\vartheta }}){|}_{{\mathrm{s}}_{\mathrm{k}}}$$

(17)

where ${\nabla }_{\mathrm{a}}\mathrm{Q}(\mathrm{s},\mathrm{a}|{\mathsf{\theta }}^{\mathrm{q}})$ represents the gradient of the action–value function with respect to action a, indicating the direction for maximizing the Q-value. The expression ${\nabla }_{{\mathsf{\theta }}^{\mathsf{\vartheta }}}\mathsf{\vartheta }(\mathrm{s}|{\mathsf{\theta }}^{\mathsf{\vartheta }}){|}_{{\mathrm{s}}_{\mathrm{k}}}$ represents the gradient of the policy $\mathsf{\vartheta }$ concerning the parameters $\mathsf{\theta }$, elucidating the method to adjust $\mathsf{\theta }$ to maintain the efficacy of the control policy. Finally, to bolster learning stability, DDPG incorporates two sets of target networks: the target actor network and the target critic network [48]. The parameter updates for the target networks are given by Formulas (18) and (19). And the pseudocode of the Dual DDPG PID algorithm will be given in Algorithm A1.

$${\mathsf{\theta }}^{{\mathrm{q}}^{\prime }}\leftarrow \mathsf{\tau }{\mathsf{\theta }}^{\mathrm{q}}+(1-\mathsf{\tau }){\mathsf{\theta }}^{{\mathrm{q}}^{\prime }}$$

(18)

$${\mathsf{\theta }}^{{\mathsf{\vartheta }}^{\prime }}\leftarrow \mathsf{\tau }{\mathsf{\theta }}^{\mathsf{\vartheta }}+(1-\mathsf{\tau }){\mathsf{\theta }}^{{\mathsf{\vartheta }}^{\prime }}$$

(19)

3.2. Temperature Control Architecture for PEMFC Based on Dual DDPG-PID

The thermal management system of a PEMFC consists of three main modules, namely the water pump, radiator, and stack, which aim to maintain a stable operating temperature. Temperature regulation involves controlling the coolant flow via the pump and airflow through the radiator. The coolant cools the stack, while the radiator cools the coolant and maintains its inlet temperature. This paper proposes a dual DDPG-PID approach to improve PID controllers in handling strongly coupled systems. The FCTMS framework based on a dual DDPG-PID is shown in Figure 4, and the agent’s output action structure is illustrated in Figure 5.

3.3. Details of the DDPG

The agent in the DDPG algorithm primarily consists of the state space, reward function, and action space. The state space is designed to characterize the thermal state and cooling dynamics of the system, enabling the reinforcement-learning agent to optimally regulate the coolant and airflow rates. The reward function serves as a constraint for the agent, penalizing excessive training deviations while providing rewards when the training performance meets the expected criteria. The action space, representing the agent’s output, corresponds to the gain parameters of the PID controller in the DDPG-PID controller. The following sections provide a detailed description of these three components.

3.3.1. State

The state space of the DDPG algorithm includes coolant flow rate, airflow through the radiator, stack temperature error and its derivative, and the coolant temperature difference error from inlet to outlet and its derivative. The state space expression of the fuel cell is as follows:

$${\mathrm{s}}_{\mathrm{t}1}=\{{\mathrm{e}}_1,\dot{{\mathrm{e}}_1},{\mathrm{e}}_2,\dot{{\mathrm{e}}_2},{\mathrm{W}}_{\mathrm{c}\mathrm{l}-1}\}$$

(20)

$${\mathrm{s}}_{\mathrm{t}2}=\{{\mathrm{e}}_1,\dot{{\mathrm{e}}_1},{\mathrm{e}}_2,\dot{{\mathrm{e}}_2},{\mathrm{W}}_{\mathrm{a}\mathrm{i}\mathrm{r}-1}\}$$

(21)

where ${\mathrm{e}}_1$ signifies the disparity between the fuel cell temperature and the targeted control temperature, while ${\mathrm{e}}_2$ captures the variation between the temperature differential of the cooling fluid upon entering and exiting the fuel cell and the target temperature differential, $\dot{{\mathrm{e}}_1}$ and $\dot{{\mathrm{e}}_2}$ are the derivatives of two errors over time.

3.3.2. Action

In the DDPG algorithm, the output values of actions are constrained within a specified range and appropriately expanded. During training, the actor within the DDPG algorithm agent further fine-tunes the action parameters in the Simulink environment, restoring the action space. The expression is as follows:

$${\mathrm{a}}_1=\left({\mathrm{K}\mathrm{P}}_1/100,{\mathrm{K}\mathrm{I}}_1/100,{\mathrm{K}\mathrm{D}}_1/100\right)$$

(22)

$${\mathrm{a}}_2=\left({\mathrm{K}\mathrm{P}}_2,{\mathrm{K}\mathrm{I}}_2/10,{\mathrm{K}\mathrm{D}}_2/10\right)$$

(23)

where ${\mathrm{K}\mathrm{P}}_{1/2}{\mathrm{K}\mathrm{I}}_{1/2}{\mathrm{K}\mathrm{D}}_{1/2}$ is the output of the DDPG algorithm agent within a certain range. The coefficient by which ${\mathrm{K}\mathrm{P}}_{1/2}{\mathrm{K}\mathrm{I}}_{1/2}{\mathrm{K}\mathrm{D}}_{1/2}$ is divided in the above expression varies because, in the DDPG algorithm, to maintain the action limits of both agent1 and agent2 within a two-digit range and avoid significant discrepancies in the numerical values of loss parameters during the iteration process, they are multiplied by different amplification gains. Lastly, there are differences in the coefficients by which the action parameters are divided when fine-tuning in the Simulink environment.

3.3.3. Reward Function

The agent continuously gains more rewards in the PEMFC system environment to approximate optimal temperature control and ideal temperature difference control. During the experimentation phase, the agent undergoes separate training sessions dedicated to controlling coolant flow and cooling airflow. During training, the agent’s output serves as the input for the PID controller. The PID controller calculates the corresponding control input based on the current state and control error. The reward function for the agent during training is defined as follows:

$${\mathrm{r}}_{\mathrm{t}}=\left\{\begin{array}{c}{\mathsf{\alpha }}_1({\mathrm{C}}_1-{\mathsf{\xi }}_1({\mathrm{e}}_1+1{)}^2)+{\mathsf{\alpha }}_2({\mathrm{C}}_2-{\mathsf{\xi }}_2({\mathrm{e}}_2+1{))}^2+\mathrm{c} , |{\mathrm{e}}_1|\le 0.1 ,|{\mathrm{e}}_2|\le 0.1\\ {\mathsf{\alpha }}_1({\mathrm{C}}_1-{\mathsf{\xi }}_1({\mathrm{e}}_1+1{)}^2)-{\mathsf{\alpha }}_2({\mathrm{C}}_3+{\mathsf{\xi }}_2({\mathrm{e}}_2+1{))}^2+\mathrm{c} , |{\mathrm{e}}_1|\le 0.1 ,|{\mathrm{e}}_2|>0.1\\ {\mathsf{\alpha }}_2({\mathrm{C}}_2-{\mathsf{\xi }}_2({\mathrm{e}}_2+1{)}^2)-{\mathsf{\alpha }}_1({\mathrm{C}}_3+{\mathsf{\xi }}_1({\mathrm{e}}_1+1{)}^2)+\mathrm{c}, |{\mathrm{e}}_1|>0.1 , |{\mathrm{e}}_2|\le 0.1\\ -{\mathsf{\alpha }}_1({\mathrm{C}}_4+{\mathsf{\xi }}_1({\mathrm{e}}_1+1{)}^2)-{\mathsf{\alpha }}_2({\mathrm{C}}_3+{\mathsf{\xi }}_2({\mathrm{e}}_2+1{)}^2)+\mathrm{c}, |{\mathrm{e}}_1|>0.1 , |{\mathrm{e}}_2|>0.1\end{array}\right.$$

(24)

where ${\mathrm{e}}_1$ denotes the target temperature error of the stack (${\mathrm{T}}_{\mathrm{s}\mathrm{t}}-353\mathrm{K}$), and ${\mathrm{e}}_2$ denotes the temperature difference error of the coolant in and out of the stack (${\mathrm{T}}_{\mathrm{s}\mathrm{t}}-{\mathrm{T}}_{\mathrm{s}\mathrm{t},\mathrm{i}\mathrm{n}}-10$). The first term in the above expression indicates that both errors are small, so the system performs very well. The reward function returns the larger positive value, indicating that the state is ideal. The second term indicates that the system’s performance is problematic because ${\mathrm{e}}_1$ is small and ${\mathrm{e}}_2$ is large. Apply a penalty (negative value) to ${\mathrm{e}}_2$ and reduce the reward. The third item, because ${\mathrm{e}}_1$ is large and ${\mathrm{e}}_2$ is small, also requires a penalty to be applied to ${\mathrm{e}}_1$. This component reduces the reward and indicates that the system is not performing well. The fourth term is due to the fact that, when both errors are large, the reward function gives a larger penalty and returns a negative value, indicating that the system is not performing as required. In the above reward function ${ \mathsf{\alpha }}_1$, ${\mathsf{\alpha }}_2$, the weighting factors control the strength of the error penalty. The ${\mathsf{\xi }}_1,{\mathsf{\xi }}_2$ error coefficients regulate how much the error affects the reward. The ${\mathrm{C}}_1$, ${\mathrm{C}}_2$, ${\mathrm{C}}_3$, ${\mathrm{C}}_4$ constants are used to adjust the base value of the reward function. The c constant term can be used for offset or compensation. In this study, their values are shown in

Table 1. Reward function parameters.

Parameter	Description	Value
${\mathsf{\alpha }}_1$	$\mathrm{The} \mathrm{weight} \mathrm{factor} \mathrm{for} \mathrm{the} \mathrm{penalty} \mathrm{strength} \mathrm{of} \mathrm{the} \mathrm{control} \mathrm{error} {\mathrm{e}}_1$	0.7
${\mathsf{\alpha }}_2$	$\mathrm{The} \mathrm{weight} \mathrm{factor} \mathrm{for} \mathrm{the} \mathrm{penalty} \mathrm{strength} \mathrm{of} \mathrm{the} \mathrm{control} \mathrm{error} {\mathrm{e}}_2$	0.8
${\mathsf{\xi }}_1$	$\mathrm{Penalty} \mathrm{coefficients}, \mathrm{adjusting} \mathrm{for} \mathrm{the} \mathrm{effect} \mathrm{of} \mathrm{the} \mathrm{square} \mathrm{of} \mathrm{the} \mathrm{error} \mathrm{term} {\mathrm{e}}_1$ on the reward	0.2
${\mathsf{\xi }}_2$	$\mathrm{Penalty} \mathrm{coefficients}, \mathrm{adjusting} \mathrm{for} \mathrm{the} \mathrm{effect} \mathrm{of} \mathrm{the} \mathrm{square} \mathrm{of} \mathrm{the} \mathrm{error} \mathrm{term} {\mathrm{e}}_2$ on the reward	0.3
${\mathrm{C}}_1$	$\mathrm{Reward} \mathrm{its} \mathrm{quasi}\text{-}\mathrm{value} \mathrm{against} \mathrm{the} {\mathrm{e}}_1$ base reward	3
${\mathrm{C}}_2$	$\mathrm{Reward} \mathrm{its} \mathrm{quasi}\text{-}\mathrm{value} \mathrm{against} \mathrm{the} {\mathrm{e}}_2$ base reward	4
${\mathrm{C}}_3$	$\mathrm{Penalty} \mathrm{base} \mathrm{value}, \mathrm{for} {\mathrm{e}}_1$$\mathrm{or} {\mathrm{e}}_2$, penalties for deviations from target values	−1
${\mathrm{C}}_4$	$\mathrm{Penalty} \mathrm{base} \mathrm{value}, \mathrm{for} {\mathrm{e}}_1$$\mathrm{or} {\mathrm{e}}_2$, penalties for deviations from target values	−0.9
$\mathrm{c}$	Constant offset value used to adjust the overall output of the reward function	0.2

. Additionally, detailed training parameters for the DDPG are listed in

Table A2. Hyperparameters of DDPG.

Symbol	Variable	Value
${\mathrm{l}}_{\mathrm{a}}$	Actor network learning rate	${10}^{-4}$
${\mathrm{l}}_{\mathrm{c}}$	Critic network learning rate	$5\times {10}^{-3}$
$N$	Minibatch size	32
$\mathsf{\gamma }$	Discount factor	0.99
${\mathrm{N}}_{\mathrm{R}}$	Experience buffer length size	$1\times {10}^6$
${\mathrm{v}}_{\mathrm{a}}$	Noise variance	0.3
$\mathrm{v}$	Soft target update rate	${10}^{-3}$

.

4. The Thermal Management Strategy Based on Dual DDPG-PID

In this study, an integrated control method is based on a dual deep deterministic policy gradient to regulate the optimal operating temperature of PEMFC. This approach utilizes two reinforcement-learning agents to dynamically train the parameters of two PID controllers, one for the water pump side and the other for the radiator side. Additionally, this approach effectively addresses the challenges encountered by PID controllers when dealing with strongly coupled systems, thus enhancing control performance.

In accordance with RL theory, an agent’s learning process can be likened to an endeavor. It initiates an action, represented as ${\mathrm{a}}_{\mathrm{t}}$, within the present environmental state ${\mathrm{s}}_{\mathrm{t}}$. This action ${\mathrm{a}}_{\mathrm{t}}$ acts as the input for the PID controller. By utilizing the current state and control error, the PID controller computes the relevant control output and engages with the environment. Post-interaction, the environment transitions to a subsequent state ${\mathrm{s}}_{\mathrm{t}+1}$, while concurrently producing an immediate reward ${\mathrm{r}}_{\mathrm{t}}$ for the agent. By employing this reward ${\mathrm{r}}_{\mathrm{t}}$ and the current state ${\mathrm{s}}_{\mathrm{t}+1}$, the agent determines the subsequent action ${\mathrm{a}}_{\mathrm{t}+1}$. Leveraging its own experience and accumulated knowledge, the agent refines action planning to augment the PID controller’s performance for improved adaptation to the environment.

The temperature control objective for PEMFC in this study is a temperature difference of $∆\mathrm{T}=10 \mathrm{K} (∆\mathrm{T}={\mathrm{T}}_{\mathrm{s}\mathrm{t},\mathrm{o}\mathrm{u}\mathrm{t}}-{\mathrm{T}}_{\mathrm{s}\mathrm{t},\mathrm{i}\mathrm{n}})$ between the coolant inlet and outlet of the stack, and the optimal operating temperature of the stack is ${\mathrm{T}}_{\mathrm{r}\mathrm{e}\mathrm{f}}=535 \mathrm{K} ({\mathrm{T}}_{\mathrm{s}\mathrm{t}}={\mathrm{T}}_{\mathrm{r}\mathrm{e}\mathrm{f}}={\mathrm{T}}_{\mathrm{s}\mathrm{t},\mathrm{o}\mathrm{u}\mathrm{t}})$.

4.1. Training and Simulation Results

In this section, the hardware setup simulated comprises an Intel (R) Core (TM) i5-8250U CPU @ 1.80 GHz paired with an 8 GB RAM computer. The models described in Section 2 and Section 3 of the paper, as well as the thermal management control model, are developed on the MATLAB 2021b/Simulink platform.

During training, the numerical values of actions output by the agent are determined based on the sizes of the three parameters, P, I, and D, adjusted in the PID controller within the Simulink environment. The exact adjustment rules are as follows. First, to ensure eventual stabilization of the response curve around the stack’s target temperature of 353 K and a coolant inlet–outlet temperature difference of 10 K, and to mitigate the adverse effects of computational precision on the outcome and reduce the possibility of oscillations, the adjustment range of the PID parameters is restricted and appropriately expanded in the DDPG algorithm code. This allows the agent to search for the most suitable parameters within the largest possible numerical range, rather than determining parameters within a small range. Then, the rules for adjusting the numerical values of the proportional coefficient P, integral coefficient I, and derivative coefficient D are determined to maintain consistency with the normal parameter adjustment sequence. Finally, further fine-tuning of the action parameters is conducted in the Simulink environment to restore the action parameter space, ensuring the performance and stability of the controller in practical environments.

To avoid the DDPG from falling into a local optimum, the authors adopt a diverse exploration mechanism in the initial distribution design, which enables the initial action distribution to have high diversity by randomly initializing the weights of the strategy network, ensuring that the intelligent body can extensively explore the state space in the early training stage, effectively reducing the risk of the algorithm converging to the local optimum. In addition, Gaussian noise is introduced during the training process to enhance the exploration ability of the intelligent body, in order to dynamically adapt to the nonlinear and multivariate coupling characteristics of the PEMFC, and further improve the global search ability. In the DDPG-PID framework, the initial parameter setting of the PID controller is also crucial. According to the temperature control requirements and the temperature difference control requirements of the coolant in and out of the electric pile, the authors weight the PID parameter ranges according to the control requirements after initializing the PID parameters, which provides a stable starting point for further optimization of the DDPG, helps the system achieve smooth temperature control at an early stage, and avoids local optimum traps due to the instability of the initial strategy. In addition, the diverse historical experiences are saved and randomly extracted through the experience playback buffer, which reduces the learning bias caused by time correlation and enables the system to explore the state space more comprehensively, effectively reducing the possibility of falling into the local optimum.

4.2. Analysis of Training Results of Dual DDPG-PID

In this section, the DDPG employs average reward evaluation during the training iterations to assess training effectiveness. During training, priority is given to controlling the stack temperature, namely the coolant flow rate of the water pump, followed by the coolant temperature difference, i.e., the airflow through the radiator. The training outcomes for both the water pump and radiator for the dual DDPG approach are depicted in Figure 6.

The PID controller on the pump side regulates the fuel cell temperature through coolant flow modulation. Figure 6a illustrates the training process curve of the pump-side agent. Initially, the three parameters of the PID controller are randomly initialized. At the beginning of training, a notable deviation exists between the stack temperature and the target control temperature, leading to escalated penalties throughout training. Consequently, the agent’s average reward declines during episodes 1–35. Subsequently, as training progresses, the deviation between the stack temperature and the target control temperature diminishes, resulting in the accrual of a satisfactory number of rewards during training. Hence, convergence begins after 75 episodes and remains steady between episodes 75 and 350.

The PID controller on the radiator side regulates the coolant temperature through airflow regulation. Figure 6b illustrates the training process curve of the radiator-side agent. As shown, the convergence of the agent begins after 55 episodes. This rapid convergence is attributed to the swift response of the radiator-side controller once the stack temperature stabilizes at the target control temperature. It promptly adjusts airflow to cool the coolant, thereby minimizing the disparity between the temperature differentials of the coolant inlet and outlet and the target temperature differential. Consequently, the radiator-side agent achieves faster convergence during the training process.

As can be seen from Figure 7a,b, during the training process of the DDPG-PID algorithm, the fluctuation of the PID parameters in the early stage is due to the fact that the algorithm is in the exploratory stage and tries to find the optimal control strategy through different combinations of parameters. At this stage, the algorithm continuously adjusts the PID parameters to cope with the changes in the system, resulting in large fluctuations in the parameter values. With the depth of training, the algorithm gradually accumulates experience, is able to predict and adjust the parameters more accurately, and finally, converges to a stable value, indicating that the system has found the optimal control strategy and realized the stable control of the fuel cell temperature.

Table 2. PSO-ID and FUZZY-PID parameter setting details.

Control Algorithm	Parameters	Parameter Range	Convergence Accuracy	Description
PSO-PID	KP	0.01~0.03	0.001	PSO optimizes PID controller parameters to adjust coolant flow to balance temperature changes
	KI	0.001~0.07	0.001
	KD	0~6	0.001
FUZZY-PID	KP	−0.3~0.3	0.001	Fuzzy logic adjusts PID controller parameters to regulate coolant flow to equalize temperature changes
	KI	−0.06~0.06	0.001
	KD	−3~3	0.001

shows the constraint ranges and convergence accuracies of PSO-PID and FUZZY-PID in the optimization process. Based on the previous experience, PSO-PID and FUZZY-PID are used for coolant flow control. In the tuning details of PSO-PID and FUZZY-PID, first, the parameter ranges of PID controllers are set. And their convergence accuracy is set to 0.001 for both of them. This setting is used to improve the temperature stability. However, temperature control in a fuel cell system involves multivariable coupled control of the coolant and airflows. PID and FUZZY-PID mainly target a single flow variable, which makes it difficult to cope with the complex coupling characteristics of multivariable in the system and increases the difficulty of temperature control.

4.3. Temperature Regulation in Response to Continuous Variations in Load Current

To assess the effectiveness of the proposed control strategy, this study employs continuous step changes in load current for validation. The duration of each load current action is set to 1200 s, during which the load current undergoes abrupt variations every 200 s. Specifically, at 200 s, 400 s, and 600 s, the load current progressively increases from 60 A to 180 A, while at 800 s and 1000 s, it decreases from 180 A to 150 A. The step changes in load current are illustrated in Figure 8.

Figure 9 demonstrates the step current temperature variation curves of the fuel cell stack under different control strategies. From Figure 9, it can be seen that the D-DDPG PID control strategy exhibits a significant steady-state control advantage in fuel cell temperature control, which can more effectively maintain the temperature around 353 K and has a lower steady-state error when compared with other control strategies. This indicates that the D-DDPG PID has a significant advantage in maintaining temperature stability and can better meet the requirements of fuel cell systems for temperature control accuracy. In addition, through the zoom-in analysis of the local response (at 200 s, 400 s, and 800 s), it can be observed that the D-DDPG PID possesses faster response speed and shorter regulation time during temperature fluctuation, and its temperature profile returns to the set value quickly when compared with the traditional PID, PSO-PID, and FUZZY-PID, which effectively reduces unnecessary temperature fluctuation. This makes D-DDPG PID more suitable for real-time temperature control under complex working conditions.

The D-DDPG PID is also excellent in overshoot control and robustness. the D-DDPG PID has almost no significant overshoot, while the conventional PID and PSO-PID have a larger overshoot, which may adversely affect the lifetime and performance of the fuel cell. In addition, the control curve of the D-DDPG PID is very stable in all fluctuation intervals and can adapt to different perturbations quickly, which reflects stronger robustness. In contrast, the traditional PID and FUZZY-PID have poor adaptability due to hysteresis and oscillation under large perturbations, while the D-DDPG PID improves the adaptability to uncertain working conditions through the adaptive mechanism of deep reinforcement learning and realizes high-precision and high-stability temperature control.

Table 3. Comparison of control data.

Control Methods	Load Time (s)	Mean Absolute Control Error (K)	Absolute Maximum Overshoot (K)	Mean Setting Time (s)
D-DDPG-PID	0–200	0.016	0.31	33
	400–600	0.009	0.26	29
	800–100	0.004	0.13	19
PID	0–200	0.065	1.02	59
	400–600	0.044	0.85	51
	800–100	0.021	0.41	39
FUZZY-PID	0–200	0.047	0.79	53
	400–600	0.038	0.74	47
	800–1000	0.020	0.38	35
PSO-PID	0–200	0.033	0.64	41
	400–600	0.0025	0.51	39
	800–100	0.013	0.25	22

shows the control data of different control strategies at different time periods. From the data in the table, it can be seen that the D-DDPG PID outperforms the other control strategies in all performance indicators. In terms of the average absolute control error, the error of D-DDPG PID in the interval of 0–200 s is 0.016 K, which is about 75.4%, 66%, and 51.5% lower than that of the traditional PID (0.065 K), FUZZY-PID (0.047 K), and PSO-PID (0.033 K), respectively. In the intervals of 400–600 s and 800–1000 s, their absolute errors are reduced by 79.5%, 76.3%, 64%, 80.9%, 80%, and 69.2%, respectively. The absolute maximum overshoots of the D-DDPG PID are 0.31 K, 0.26 K, and 0.13 K, respectively, in the three intervals, in comparison with the PSO-PID’s overshoots of 0.64 K, 0.51 K, and 0.25 K in the same intervals. The overshooting amplitude is reduced by about 49.5% on average, showing stronger overshooting suppression ability. In addition, in terms of the average tuning time, the D-DDPG PID was 33 s, 29 s, and 19 s in the intervals of 0–200 s, 400–600 s, and 800–1000 s, respectively, which were about 24.2%, 25.6%, and 13.6% shorter compared with the 41 s, 39 s, and 22 s of the PSO-PID. These data further indicate that the D-DDPG PID significantly outperforms the conventional control algorithms in terms of temperature control stability and dynamic response speed and has stronger robustness and adaptability, especially under complex operating conditions.

Figure 10 presents the simulation results of the coolant flow rates under different control strategies. As observed from the figure, at 200 s, 400 s, and 800 s, when the load increases, the operating temperature of the fuel cell rises accordingly. To achieve temperature regulation, the D-DDPG-PID control strategy responds rapidly to system variations, increasing the coolant flow rate required for effective cooling and maintaining stability. Furthermore, at 800 s and 1000 s, as the load decreases, the heat generation of the stack is reduced, leading to a lower coolant demand. Under the regulation of the dual DDPG-PID controller, the coolant flow rate is dynamically adjusted based on system requirements, preventing excessive cooling and enhancing overall system efficiency. In contrast, the compared control algorithms in this study, while also capable of adjusting the coolant flow rate in response to system variations, exhibit a slower response, noticeable overshoot, and oscillations, leading to inferior control stability compared to the D-DDPG-PID control strategy.

Figure 11 presents the simulation results of airflow regulation during the temperature control process of the fuel cell. In the thermal management system of the fuel cell, the adjustment of radiator airflow is primarily aimed at reducing the temperature of the coolant entering the stack, ensuring that the fuel cell operates within its optimal temperature range. When the load increases at 200 s, 400 s, and 800 s, the operating temperature of the fuel cell rises, leading to an increase in both the temperature and flow rate of the coolant exiting the stack. Consequently, the radiator airflow needs to be increased to cool the coolant and enhance its heat dissipation capacity. To achieve the temperature regulation objective, the radiator airflow must be appropriately increased under load increments to meet the cooling demand. From the simulation results, it can be observed that the D-DDPG-PID control algorithm can rapidly respond to changes in the coolant flow rate and effectively regulate the radiator airflow, thereby stabilizing the inlet temperature of the coolant entering the stack. In contrast, at 800 s and 1000 s, when the load decreases, the heat generation of the fuel cell is reduced, leading to a lower cooling demand. Accordingly, the radiator airflow must be decreased to prevent excessive cooling of the coolant, which could otherwise cause temperature non-uniformity within the fuel cell stack. Furthermore, compared to other control algorithms, the D-DDPG-PID algorithm effectively suppresses hysteresis, overshoot, and oscillations in the coolant flow rate adjustment process, resulting in smoother regulation of radiator airflow and improved overall control performance. Combined with the simulation results of coolant flow regulation, this control algorithm demonstrates superior stability in the coordinated control of coolant and airflow, enhancing the cooling efficiency of the fuel cell stack while reducing the energy consumption of the cooling system, thereby improving the overall energy efficiency of the system.

Figure 12 shows the simulation result graph of the coolant temperature difference of the fuel cell. From the result plots, the D-DDPG PID is able to accurately and stably maintain around 10 K near the temperature difference setpoint, while the other control strategies show large fluctuations and oscillations during the temperature difference changes. From the local zoomed-in graphs, the D-DDPG PID shows a faster response speed and smaller overshooting in the temperature difference fluctuation region around 0 s, 400 s, or 800 s. In comparison, the traditional PID, PSO-PID, and FUZZY-PID methods have larger overshoots and longer tuning times in the temperature difference response process, especially as the PSO-PID and FUZZY-PID oscillate in multiple fluctuation intervals. The ability of the D-DDPG PID to quickly and stably control the coolant temperature difference at around 10 K is mainly attributed to the better coordination of the D-DDPG PID in the D-DDPG PID and the D-DDPG PID in the D-DDPG PID. DDPG PID better coordinates the control of coolant and coolant air so that it does not oscillate and fluctuate significantly when the load changes. This further verifies the stability of the D-DDPG PID control algorithm, which can effectively suppress the potential impact of temperature fluctuations on the fuel cell’s performance and ensure the efficient operation of the cooling system under complex operating conditions.

5. Conclusions

In this paper, a dual DDPG-PID control strategy is used to enhance the thermal management performance of PEMFCs. The strategy addresses the inherent delay of conventional PID controllers by dynamically matching the temperature and flow rate variations in real time. Simulation results show that the D-DDPG PID significantly reduces temperature overshoot by approximately 75.4%, 66%, and 51.5% compared to conventional PID, FUZZY-PID, and PSO-PID, respectively, in the 0–200 s time interval. Additionally, the average tuning time of D-DDPG PID is shortened by 24.2%, 25.6%, and 13.6% compared to PSO-PID in the corresponding time intervals. This strategy effectively alleviates challenges such as multivariable coupling and control delays in PEMFC thermal management. By optimizing the PID parameters through reinforcement learning, it reduces temperature oscillations and minimizes parasitic losses. Although the study is based on simulations, the results provide a solid foundation for future industrial applications, with the potential for broader implementation in other PEMFC subsystems and fuel cell types.