Intelligent Frequency Control for Hybrid Multi-Source Power Systems: A Stepwise Expert-Teaching PPO Approach

Abstract

This paper proposes a stepwise expert-teaching reinforcement learning framework for intelligent frequency control in hydro–thermal–wind–solar–compressed air energy storage (CAES) integrated systems under high renewable energy penetration. The proposed method addresses the frequency stability challenge in low-inertia, high-volatility power systems, particularly in Southwest China, where large-scale renewable-energy-based energy bases are rapidly emerging. A load frequency control (LFC) model is constructed to serve as the training and validation environment, reflecting the dynamic characteristics of the hybrid system. The stepwise expert-teaching PPO (SETP) framework introduces a stepwise training mechanism in which expert knowledge is embedded to guide the policy learning process and training parameters are dynamically adjusted based on observed performance. Comparative simulations under multiple disturbance scenarios are conducted on benchmark systems. Results show that the proposed method outperforms standard proximal policy optimization (PPO) and traditional PI control in both transient response and coordination performance.

1. Introduction

To address global climate change, China aims to peak carbon emissions by 2030 and achieve carbon neutrality by 2060, with notable progress already made [1]. By the end of 2023, the Southwest region had built the country’s cleanest power grid, with hydro, wind, and solar capacity reaching 130 GW—nearly 80% of the total [2]. However, the large-scale integration of hydro–wind–solar–storage bases has reduced system inertia and increased frequency fluctuations [3]. This calls for coordinated frequency control strategies to fully tap the regulation potential of diverse resources within the base [4,5].

Frequency restoration in power systems primarily relies on automatic generation control (AGC), which generates the total regulation power through control loops and allocates it to individual regulating units based on predefined rules [6]. Although recent studies have introduced online parameter tuning and ultra-short-term optimization methods to enhance AGC’s control and dispatch performance, limitations remain in terms of adaptability and computational efficiency [7].

In recent years, reinforcement learning has been extensively studied in various areas of power systems, including system control [8], economic dispatch [9], microgrid energy management [10], and electricity market operations [11]. Owing to its “offline training–online deployment” paradigm, it has attracted significant attention in automatic generation control (AGC) applications, as this paradigm satisfies the 4–6 s dispatch cycle requirement and provides strong nonlinear approximation capability [12]. Study [13] has further deployed a DRL-based AGC strategy in a municipal distribution network through a virtual simulation system, preliminarily verifying the practical feasibility of this research direction.

Research on DRL for power allocation can be broadly categorized into two main directions:

• Enhancing conventional DRL algorithms to improve training efficiency and control performance in power dispatch tasks [14,15].
• Applying DRL techniques to achieve optimal power allocation for heterogeneous energy systems across various operating scenarios [16].

For the control layer, the literature primarily focuses on the following:

• Leveraging DRL to adaptively tune controller parameters, thereby improving the performance of AGC systems [17,18,19,20,21].
• Directly employing DRL agents as controllers to generate system-wide regulation commands [22,23,24,25,26].

Significant progress has been made in applying DRL to improve individual components; however, research on the holistic optimization of both control and allocation processes remains relatively limited [27]. Study [26] employs a heuristic algorithm to optimize the input weights of the state space in deep deterministic policy gradient (DDPG), thereby achieving improvements across both control and allocation. Nevertheless, this study is confined to power allocation between thermal power units and energy storage systems, which limits its applicability.

Among studies [14,15,16,17,18,19,20,21,22,23,24,25,26], most studies employ algorithms such as deep Q-Network (DQN) and DDPG, with only studies [16,25] adopting stochastic policy algorithms. Specifically, study [16] implements second-level power optimization and allocation of multiple types of energy storage resources in emergency ramping scenarios based on the PPO algorithm, which demonstrates stronger exploration capability and higher steady-state reward values compared with DDPG, twin delayed deep deterministic policy gradient (TD3), and other algorithms. Study [25] applies the SAC algorithm to realize distributed control of microgrids, and experimental results show that this method outperforms DDPG in key performance metrics such as settling time, frequency deviation, and robustness. Study [28], using a classical reinforcement learning environment, systematically analyzes and compares algorithms, revealing that PPO achieves higher sample efficiency, greater stability, and easier implementation. These studies indicate that stochastic policy algorithms hold substantial application potential; however, their applicability and performance in LFC tasks have yet to be thoroughly investigated and validated [29].

Despite the strong self-learning and decision-making capabilities demonstrated by DRL algorithms such as DDPG, SAC, and PPO, their applications in power systems remain limited due to poor training stability, high sensitivity to hyperparameters, insufficient algorithmic robustness, and low training efficiency [30,31,32]. In load frequency control (LFC) tasks, the rationality and safety of training scenarios must also be carefully considered. On the one hand, if the training environment is overly idealized or the disturbance design is inappropriate, the agent’s generalization ability and training stability under complex operating conditions will deteriorate significantly [33]; on the other hand, reinforcement learning relies on trial-and-error exploration, and during the early stages of training, the agent is prone to entering unsafe regions, generating a large number of ineffective samples and resulting in slow policy convergence [34].

To address the above issues, researchers have proposed several improvements to standard reinforcement learning algorithms. Study [30] introduces a delayed update mechanism to mitigate training instability; study [31] enhances sample efficiency by optimizing the experience replay mechanism; study [24] employs demonstrators in the early stage of training to provide expert samples, thereby improving the agent’s guidance and robustness; and study [35] proposes a multi-stage training strategy based on task complexity, offering new insights into the application of DRL in LFC tasks.

Building upon the aforementioned research progress, this study investigates an energy base located in Southwest China and proposes an intelligent frequency control strategy based on the SETP framework. First, a load frequency control model incorporating hydropower, thermal power, wind power, photovoltaic generation, and compressed air energy storage is developed according to the base’s generation mix. Then, building upon the PPO algorithm, the SETP framework is constructed with adjustable disturbance scenarios, a staged training mechanism, a safe action mapping mechanism, and an expert-guided mechanism. These features ensure the rational design of training environments, enhance training stability, reduce ineffective samples, and guarantee the effective implementation of the learned policy. Finally, comparative experiments are conducted under both weak and strong disturbance scenarios to evaluate the control performance of PPO, SETP, and conventional PI control. In addition, the regulation mileage of each strategy is analyzed, thereby verifying the control effort of the proposed method.

2. LFC Model of Hydro–Thermal–Wind–Solar–CAES System

2.1. Generator–Load Model

Given the small-signal disturbance scenario in this study, the generator and load dynamics in the s-domain are represented as follows:

$$\Delta f^{\ast }={\displaystyle \frac{1}{2H_{\mathrm{eq}}s+D}}\Delta P_{\mathrm{u}}^{\ast }$$

(1)

where $\Delta f^{\ast }$ denotes the per-unit frequency deviation; $H_{\mathrm{eq}}$ denotes the equivalent inertia time; D denotes the load damping constant; and $\Delta P_{\mathrm{u}}^{\ast }$ denotes the per-unit power imbalance.

A single-area LFC model is established as the simulation environment, where thermal power units, hydro power units, and a CAES provide inertial support to the system. The constant is calculated using Equation (2), following the formulation in [36].

$$H_{\mathrm{eq}}={\displaystyle \frac{{\displaystyle \sum _{i=1}^n}(H_i·S_{\mathrm{b},i})}{{\displaystyle \sum _{i=1}^n}{S}_{\mathrm{b},i}}}$$

(2)

where $H_i$ denotes the inertia constant of unit i; $S_{\mathrm{b},i}$ denotes the rated capacity of unit i; and $H_{\mathrm{eq}}$ denotes the equivalent inertia constant of the system. And the subscript $i=1–6$ corresponds to the thermal unit, CAES, hydro unit, wind farm, PV station, and battery energy storage system (BESS).

2.2. Primary Frequency Control

2.2.1. Governor with Speed Droop Characteristic

Since the simulation environment consists of multiple power sources, droop governors are employed. Taking a thermal power unit as an example, its s-domain model is given as follows:

$$\Delta P_{\mathrm{v},\mathrm{thermal}}^{\ast }=-{\displaystyle \frac{1}{R_{\mathrm{thermal}}}}·{\displaystyle \frac{1}{1+s{T}_{\mathrm{g}}}}\Delta f^{\ast }$$

(3)

where $R_{\mathrm{thermal}}$ denotes the droop rate; $T_{\mathrm{g}}$ denotes the governor time constant; and $\Delta P_{\mathrm{v},\mathrm{thermal}}$ denotes the valve position.

2.2.2. Hydro Turbine Governor

Considering the water hammer effect, the hydro turbine governor incorporates a transient droop compensation component, and its s-domain model is given by

$$\Delta P_{\mathrm{v},\mathrm{hydro}}^{\ast }=-{\displaystyle \frac{1}{R_{\mathrm{hydro}}}}·{\displaystyle \frac{1}{1+s{T}_{\mathrm{g}}}}·{\displaystyle \frac{1+s{T}_{\mathrm{R}}}{1+s(R_{\mathrm{T}}/R_{\mathrm{H}})T_{\mathrm{R}}}}\Delta f^{\ast }$$

(4)

where $T_{\mathrm{R}}$ denotes the dashpot time constant; $R_{\mathrm{T}}$ denotes the temporary droop coefficient; $R_{\mathrm{H}}$ denotes the permanent droop coefficient; and $R_{\mathrm{hydro}}$ denotes the hydro power unit.

2.2.3. Synthetic Inertia Control Strategy

The wind and solar resources involved in this paper are all equipped with energy storage systems, connected to the grid through converters, and decoupled from the grid side and thus unable to actively provide inertia support to the grid. This paper considers that their primary frequency regulation process adopts a synthetic inertia control strategy combining droop control and virtual inertia control:

$$\Delta P_{\mathrm{PF}}^{\mathrm{ref}}=K_{\mathrm{df}}{\displaystyle \frac{\mathrm{d}\Delta f^{\ast }}{\mathrm{d}t}}+K_{\mathrm{pf}}\Delta f^{\ast }$$

(5)

where $\Delta P_{\mathrm{PF}}^{\mathrm{ref}}$ denotes the primary frequency regulation reference; $K_{\mathrm{df}}$ and $K_{\mathrm{pf}}$ denote the virtual inertia control coefficient and droop control coefficient, respectively.

2.3. Conventional Power Sources

2.3.1. Hydro Turbine

The s-domain model of the hydro turbine is given by

$$\Delta P_{\mathrm{m},\mathrm{hydro}}^{\ast }={\displaystyle \frac{1-s{T}_{\mathrm{W}}}{1+0.5s{T}_{\mathrm{W}}}}\Delta P_{\mathrm{v},\mathrm{hydro}}^{\ast }$$

(6)

where $T_{\mathrm{W}}$ denotes the water starting time constant and $\Delta P_{\mathrm{m},\mathrm{hydro}}^{\ast }$ denotes per-unit mechanical power deviation of the hydro unit.

2.3.2. Steam Turbine

The reheat steam turbine is modeled in the s-domain as follows:

$$\Delta P_{\mathrm{m},\mathrm{thermal}}^{\ast }={\displaystyle \frac{1+s{F}_{\mathrm{HP}}{T}_{\mathrm{RH}}}{1+s{T}_{\mathrm{RH}}}}·{\displaystyle \frac{1}{1+s{T}_{\mathrm{CH}}}}\Delta P_{\mathrm{v},\mathrm{thermal}}^{\ast }$$

(7)

where $F_{\mathrm{HP}}$ denotes the high-pressure turbine power fraction; $T_{\mathrm{RH}}$ denotes the reheater time constant; $T_{\mathrm{CH}}$ denotes the crossover time constant; and $\Delta P_{\mathrm{m},\mathrm{thermal}}^{\ast }$ denotes per-unit mechanical power deviation of the thermal unit.

2.4. Renewable Energy Sources

Photovoltaic Power Stations and Wind Farms

The converter response speed of photovoltaic power stations and wind farms is relatively fast. This paper considers their power response characteristics as first-order inertial components:

$$\left\{\begin{array}{c}{P}_{\mathrm{wt}}={\displaystyle \frac{1}{1+s{T}_{\mathrm{c}}}}{P}_{\mathrm{wt}}^{\mathrm{ref}}\hfill \\ P_{\mathrm{pv}}={\displaystyle \frac{1}{1+s{T}_{\mathrm{c}}}}{P}_{\mathrm{pv}}^{\mathrm{ref}}\hfill \end{array}\right.$$

(8)

where $P_{\mathrm{wt}}$ and $P_{\mathrm{pv}}$ denote the actual output power of the wind farm and photovoltaic power station, respectively; $P_{\mathrm{wt}}^{\mathrm{ref}}$ and $P_{\mathrm{pv}}^{\mathrm{ref}}$ denote the power references for the wind farm and the photovoltaic power station, respectively; and $T_{\mathrm{c}}$ denotes the converter control time constant.

2.5. Energy Storage Systems

2.5.1. CAES

In this study, CAES participates in system frequency regulation through expansion power generation. Since the day-ahead and intra-day dispatch processes ensure a 5–10% operating margin for the CAES reservoir pressure and thermal storage temperature, and their variations remain within 5% during the training horizon considered in this study, the changes in reservoir pressure and thermal storage temperature are neglected. Therefore, the influences of the thermal storage devices, heat exchange systems, and air storage reservoirs are not taken into account. Following the formulation in [37], the output power and outlet temperature of the i-th expander are expressed as

$$\left\{\begin{array}{c}{T}_{\mathrm{tx},i}=T_{\mathrm{td},i}\left[1-\left(1-{\pi }_{\mathrm{t},i}^{{\displaystyle \frac{1-k}{k}}}\right){\eta }_{\mathrm{ti}}\right]\hfill \\ P_{\mathrm{tm},i}=c_p{\dot{m}}_t\left(T_{\mathrm{td},i}-T_{\mathrm{tx},i}\right)\hfill \end{array}\right.$$

(9)

where $T_{\mathrm{tx},i}$ denotes the outlet gas temperature of the i-th expander; $T_{\mathrm{td},i}$ denotes the inlet gas temperature of the i-th expander; ${\pi }_{\mathrm{t},i}$ denotes the expansion ratio of the i-th expander; $k=c_p/c_v$ denotes the specific heat ratio, also known as the adiabatic index, where $c_p$ and $c_v$ are the constant-pressure and constant-volume specific heats of air, respectively; ${\dot{m}}_t$ is the air mass flow rate of the expander; $P_{\mathrm{tm},i}$ is the shaft power of the i-th expander; and ${\eta }_{\mathrm{ti}}$ is the isentropic efficiency.

CAES regulates the expander power by controlling ${\dot{m}}_t$ to respond to the power reference. The s-domain model of its air flow control system is

$$\left\{\begin{array}{c}{P}_{\mathrm{tm}}={\displaystyle \sum _i}{P}_{\mathrm{tm},i}\hfill \\ {\dot{m}}_t=\left(P_{\mathrm{CAES}}^{\mathrm{ref}}-P_{\mathrm{tm}}\right)\left(K_{\mathrm{tp}}+{\displaystyle \frac{K_{\mathrm{ti}}}{s}}\right){\displaystyle \frac{1}{1+s{T}_{\mathrm{TD}}}}{\displaystyle \frac{1}{1+s{T}_{\mathrm{AV}}}}\hfill \end{array}\right.$$

(10)

where $T_{\mathrm{TD}}$ and $T_{\mathrm{AV}}$ are the turbine gas discharge time delays and $P_{\mathrm{CAES}}^{\mathrm{ref}}$ is the power reference for the CAES. The control block diagram is shown in Figure 1.

2.5.2. BESS

The converter model of the electrochemical energy storage station is similar to Equation (8); its state of charge (SOC) is calculated by the following equation and must satisfy the constraints shown below to avoid overcharging and over-discharging:

$$\left\{\begin{array}{cc}{\displaystyle x_{\mathrm{soc}}=x_{{\mathrm{soc}}_0}-{\int }_{t_0}^t{\displaystyle \frac{P_{\mathrm{bess}}}{\eta }}\mathrm{d}\tau ,}\hfill & P_{\mathrm{bess}}>0\hfill \\ {\displaystyle x_{\mathrm{soc}}=x_{{\mathrm{soc}}_0}-{\int }_{t_0}^t\eta P_{\mathrm{bess}}\mathrm{d}\tau ,}\hfill & P_{\mathrm{bess}}\le 0\hfill \end{array}\right.$$

(11)

$$P_{\mathrm{bess}}=\left\{\begin{array}{cc}min(P_{\mathrm{bess}}^{\mathrm{ref}},0)\hfill & x_{\mathrm{soc}}\ge x_{\mathrm{soc}}^{\max }\hfill \\ P_{\mathrm{bess}}^{\mathrm{ref}}\hfill & x_{\mathrm{soc}}^{\min }<x_{\mathrm{soc}}<x_{\mathrm{soc}}^{\max }\hfill \\ max(P_{\mathrm{bess}}^{\mathrm{ref}},0)\hfill & x_{\mathrm{soc}}\le x_{\mathrm{soc}}^{\min }\hfill \end{array}\right.$$

(12)

where $x_{\mathrm{soc}}$, $x_{{\mathrm{soc}}_0}$, $x_{\mathrm{soc}}^{\min }$, and $x_{\mathrm{soc}}^{\max }$ denote the current, initial, minimum, and maximum SOC; $P_{\mathrm{bess}}$ and $P_{\mathrm{bess}}^{\mathrm{ref}}$ denote the actual and reference power; and $\eta$ is the charging/discharging efficiency.

2.6. Conventional AGC Control Strategy Based on PI Control

AGC employs PI control to obtain the total regulation power, with the input of PI control being the area control error; subsequently, rules such as “proportional to adjustable capacity” are used to allocate power to each unit.

2.6.1. Area Control Error (ACE)

Since the simulation environment in this paper is a single area, the AGC control mode adopts constant frequency control, and the calculation formula for the area control error is [38]:

$$x_{\mathrm{ACE}}=-B(f-f_0)$$

(13)

where $x_{\mathrm{ACE}}$ denotes the area control error, where $x_{\mathrm{ACE}}>0$ indicates excessive generation in this area; B denotes the bias factor of the control area; f is the current system frequency; and $f_0$ is the system reference frequency. The value of B is selected according to the composite frequency response characteristic, as detailed in Appendix A.1.

2.6.2. Power Dispatch Strategy

The control power is dispatched proportionally according to the available regulation capacity of each unit as follows:

$${\alpha }_i=\left\{\begin{array}{cc}{\displaystyle \frac{P_i^{\mathrm{avail},\mathrm{up}}}{{\displaystyle \sum _{i=1}^n}{P}_i^{\mathrm{avail},\mathrm{up}}}}\hfill & x_{\mathrm{ACE}}<0\hfill \\ {\displaystyle \frac{P_i^{\mathrm{avail},\mathrm{down}}}{{\displaystyle \sum _{i=1}^n}{P}_i^{\mathrm{avail},\mathrm{down}}}}\hfill & x_{\mathrm{ACE}}\ge 0\hfill \end{array}\right.$$

(14)

where $P_i^{\mathrm{avail},\mathrm{up}}$ and $P_i^{\mathrm{avail},\mathrm{down}}$ represent the upward and downward regulation capacities of unit i, respectively.

2.7. AGC Control Strategy Based on DRL

The DRL-based AGC strategy adopts an offline training and online deployment paradigm, where neural networks parameterize the control policy to enhance adaptability and generalization in frequency regulation. In this paradigm, the agent replaces conventional AGC control and allocation, interacting with the environment to collect experience for offline learning and policy evaluation. A training framework based on PPO, enhanced with expert knowledge integration and a staged training mechanism, is developed to improve convergence efficiency and policy performance, with implementation details provided in Section 3.

The LFC model constructed based on Section 2.1, Section 2.2, Section 2.3, Section 2.4, Section 2.5, Section 2.6 and Section 2.7 is illustrated in Figure 2; the corresponding parameters are listed in

Table A1. Parameters of the LFC model.

Symbol	Value	Symbol	Value
$H_{\mathrm{eq}}\left[\mathrm{s}\right]$	1.42	$D\left[\mathrm{pu}\right]$	0.5
$S_{\mathrm{b},i}\left[\mathrm{MW}\right]$	100, 100, 400, 400, 200, 100	$H_i\left[\mathrm{s}\right]$	1.5, 0.5, 4.1, 0, 0, 0
$R_{\mathrm{thermal}}$	0.2	$T_{\mathrm{g}}\left[\mathrm{s}\right]$	0.05
$R_{\mathrm{hydro}}$	0.5	$T_{\mathrm{R}}\left[\mathrm{s}\right]$	5
$R_{\mathrm{T}}$	0.38	$R_{\mathrm{H}}$	0.05
$K_{\mathrm{df}}$	1	$K_{\mathrm{pf}}$	2
$T_{\mathrm{W}}\left[\mathrm{s}\right]$	1	$F_{\mathrm{HP}},T_{\mathrm{RH}},T_{\mathrm{CH}}$	0.3, 7, 0.3
$T_{\mathrm{c}}\left[\mathrm{s}\right]$	0.2	$T_{\mathrm{td},1}{[}^{\circ }\mathrm{C}],T_{\mathrm{tx},1}{[}^{\circ }\mathrm{C}],{\pi }_{\mathrm{t},1}$	193, 38, 4.6
$T_{\mathrm{td},2}{[}^{\circ }\mathrm{C}],T_{\mathrm{tx},2}{[}^{\circ }\mathrm{C}],{\pi }_{\mathrm{t},2}$	193, 43, 5.0	$T_{\mathrm{td},3}{[}^{\circ }\mathrm{C}],T_{\mathrm{tx},3}{[}^{\circ }\mathrm{C}],{\pi }_{\mathrm{t},3}$	193, 41, 4.7
${\eta }_{\mathrm{t}1},{\eta }_{\mathrm{t}2},{\eta }_{\mathrm{t}3}$	0.90, 0.92, 0.92	$K_{\mathrm{tp}}$	0.00058
$K_{\mathrm{ti}}$	0.000318	$T_{\mathrm{TD}}\left[\mathrm{s}\right]$	0.3
$T_{\mathrm{AV}}\left[\mathrm{s}\right]$	0.1	$R_{\mathrm{CAES}}$	0.2
$x_{\mathrm{soc}}0$	0.5	$x_{\mathrm{soc}}^{\min },x_{\mathrm{soc}}^{\max }$	0.2, 0.8
$\eta$	0.95	$B[\mathrm{pu}/\mathrm{pu}]$	2.9615
$P_i^{\mathrm{avail},\mathrm{up}}\left[\mathrm{MW}\right]$	50, 50, 120, 40, 20, 100	$P_i^{\mathrm{avail},\mathrm{down}}\left[\mathrm{MW}\right]$	−10, −30, −120, −40, −20, −100
${\lambda }_{\mathrm{mile}}$	1	$S_i^{\mathrm{perf}}$	0.34, 0.45, 0.55, 0.7, 0.7, 0.7

.

3. Stepwise Expert-Teaching PPO Framework for LFC

3.1. PPO Training Procedure

PPO is an on-policy reinforcement learning algorithm that enhances training stability by restricting the magnitude of policy updates. Its application in load frequency control follows four key steps:

1. Interaction with Environment. The PPO agent interacts with the LFC environment under the current policy ${\pi }_{\theta }$ to collect trajectory data. The number of time steps per collection round is defined as the experience horizon, which determines the batch size used in policy updates.

2. Advantage Estimation. To assess the quality of actions within the collected trajectories, the Generalized Advantage Estimation (GAE) method is applied:

$${\widehat{A}}_t={\delta }_t+\left(\gamma \lambda \right){\delta }_{t+1}+\cdots +{\left(\gamma \lambda \right)}^{T-t+1}{\delta }_{T-1}$$

(15)

where ${\delta }_t=r_t+\gamma V\left(s_{t+1}\right)-V\left(s_t\right)$ denotes the temporal difference (TD) error; $\gamma$ denotes the discount factor; and $\lambda$ denotes the GAE smoothing factor.

3. Network Updates. The collected trajectories are split into mini-batches and used to update the actor and critic networks over multiple epochs using the Adam optimizer. The actor network is updated using the clipped surrogate loss:

$$L^{\mathrm{CLIP}}\left(\theta \right)={\mathbb{E}}_t\left[min\left(r_t\left(\theta \right){\widehat{A}}_t,\mathrm{clip}(r_t\left(\theta \right),1-ϵ,1+ϵ){\widehat{A}}_t\right)+\beta \mathcal{H}\left[{\pi }_{\theta }\left(s_t\right)\right]\right]$$

(16)

where $r_t\left(\theta \right)$ denotes the importance sampling ratio; $ϵ$ denotes the clipping factor; and $\beta$ denotes the entropy coefficient. The critic is trained by minimizing the value function loss:

$$L^{\mathrm{VF}}\left(\varphi \right)={\mathbb{E}}_t\left[{\left(V_{\varphi }\left(s_t\right)-{\widehat{V}}_t\right)}^2\right]$$

(17)

where ${\widehat{V}}_t={\widehat{A}}_t+V_{{\varphi }_{\mathrm{old}}}\left(s_t\right)$ denotes the estimated return, decoupled from current updates to enhance stability.

4. Policy Iteration. Steps 1 to 3 are repeated iteratively until the policy converges or a maximum number of training episodes is reached.

3.2. Reinforcement Learning Environment Design

3.2.1. State and Action Spaces

At each time step t, the agent receives an observation vector ${\mathbf{s}}_t$ composed of power deviations, power references, and the ACE:

$${\mathbf{s}}_t={\left[\Delta {\mathbf{P}}_t,{\mathbf{P}}_t^{\mathrm{ref}},x_{\mathrm{ACE},t}\right]}^{\top }$$

(18)

where $\Delta {\mathbf{P}}_t\in {\mathbb{R}}^6$ and ${\mathbf{P}}_t^{\mathrm{ref}}\in {\mathbb{R}}^6$ denote the power deviations and reference values of six energy sources and $x_{\mathrm{ACE},t}$ represents the area control error at time step t. The action vector is defined as follows:

$${\mathbf{a}}_t={\left[u_t^{\mathrm{thermal}},u_t^{\mathrm{CAES}},u_t^{\mathrm{hydro}},u_t^{\mathrm{WT}},u_t^{\mathrm{PV}},u_t^{\mathrm{BESS}}\right]}^{\top }$$

(19)

where each $u_t^i$ denotes the secondary frequency regulation command for the i-th energy resource at time t.

3.2.2. Reward Function

The reward function consists of three components: $r_1$ penalizing frequency deviations; $r_2$ suppressing oscillations in the agent’s output; and $r_3$ reflecting the cost of frequency regulation.

$$\begin{array}{cc}\hfill r_t& =r_1+r_2+r_3\hfill \\ \hfill r_1& =-{\lambda }_{\mathrm{ACE}}·\left|x_{\mathrm{ACE},t}\right|\hfill \\ \hfill r_2& =-{\lambda }_{\mathrm{act}}·\left|a_t-a_{t-1}\right|\hfill \\ \hfill r_3& =-{\lambda }_{\mathrm{eff}}·{\displaystyle \sum _{i=1}^n}{d}_i\left(t\right)\hfill \end{array}$$

(20)

where $x_{\mathrm{ACE},t}$ denotes the real-time ACE at time step t; $a_t$ and $a_{t-1}$ denote the agent’s control outputs at time steps t and $t-1$, respectively; $d_i\left(t\right)$ denotes the regulation mileage of unit i at time step t, and its calculation formula is provided in Appendix A.3; and ${\lambda }_{\mathrm{ACE}}$, ${\lambda }_{\mathrm{act}}$, and ${\lambda }_{\mathrm{eff}}$ denote the corresponding weighting factors.

3.2.3. Actor and Critic Network Architecture

The critic network is first constructed to estimate the state-value function $V\left(s\right)$, as shown in Figure 3.

The actor network, shown in Figure 4, adopts a shared-body and dual-branch architecture. The shared body extracts state features via two fully connected layers with ReLU activations. The mean branch applies a tanh activation to normalize the output actions, while the standard deviation branch uses a softplus activation followed by a clipping layer to ensure numerical stability.

The final action is sampled from a Gaussian distribution and mapped to the safe action domain through a linear transformation, defined as follows:

$${\mathbf{a}}_{\mathrm{safe}}={\displaystyle \frac{{\mathbf{a}}_{\mathrm{raw}}+\mathbf{1}}{2}}\left({\mathbf{a}}_{max}-{\mathbf{a}}_{min}\right)+{\mathbf{a}}_{min}$$

(21)

where ${\mathbf{a}}_{\mathrm{raw}}$ denotes the normalized action vector sampled from the Gaussian policy and ${\mathbf{a}}_{min},{\mathbf{a}}_{max}$ represent the element-wise lower and upper bounds of the action domain, respectively; the corresponding parameters are listed in

Table A2. Safe layer parameters.

Parameter	Symbol	Value
Action domain lower bound	${\mathbf{a}}_{min}$	$[-10,-30,-120,-40,-20,-100]$
Action domain upper bound	${\mathbf{a}}_{max}$	$[50,50,120,40,20,100]$

.

As implemented, PPO is updated using Equation (22) and SETP employs Equations (22) and (23).

$$\begin{array}{cc}\hfill \theta & \leftarrow \theta -{\alpha }_{\theta }{\nabla }_{\theta }{L}^{\mathrm{CLIP}}\left(\theta \right)\hfill \\ \hfill \varphi & \leftarrow \varphi -{\alpha }_{\varphi }{\nabla }_{\varphi }{L}^{\mathrm{VF}}\left(\varphi \right)\hfill \end{array}$$

(22)

$$\theta \leftarrow \theta -{\alpha }_{\theta }^{\mathrm{expert}}{\nabla }_{\theta }{L}^{\mathrm{expert}}\left(\theta \right)$$

(23)

where $\theta$ and $\varphi$ denote the parameters of the actor and critic networks, respectively; ${\alpha }_{\theta }$, ${\alpha }_{\varphi }$, and ${\alpha }_{\theta }^{\mathrm{expert}}$ are the learning rates of the actor, critic, and expert-guided actor updates; $L^{\mathrm{CLIP}}\left(\theta \right)$ is the PPO surrogate loss; $L^{\mathrm{VF}}\left(\varphi \right)$ is the critic value function loss; and $L^{\mathrm{expert}}\left(\theta \right)$ is the expert loss.

3.2.4. Disturbance Scenario Generation

To generate disturbance scenarios that are both stochastic and physically feasible, we adopt a truncated Gaussian sampling strategy with stepwise and cumulative constraints. Let the cumulative disturbance up to step $i-1$ be defined as follows:

$$S_{i-1}={\displaystyle \sum _{k=1}^{i-1}}{ϵ}_k$$

Then, the disturbance at step i is sampled according to the following rules:

$$\left\{\begin{array}{c}{ϵ}_i\sim \mathcal{N}(\mu ,{\sigma }^2)\hfill \\ {ϵ}_{min}\le {ϵ}_i\le {ϵ}_{max}\hfill \\ S_{min}\le S_{i-1}+{ϵ}_i\le S_{max}\hfill \end{array}\right.$$

(24)

where $\mu$ and $\sigma$ denote the mean and standard deviation of the Gaussian distribution, $[{ϵ}_{min},{ϵ}_{max}]$ defines the allowable range for each step, and $[S_{min},S_{max}]$ specifies the bounds for the cumulative disturbance; the corresponding parameters are listed in

Table A3. Key parameters for disturbance scenario generation.

Parameter Name	Symbol	Value
Mean of Gaussian distribution	$\mu$	0
Standard deviation of Gaussian distribution	$\sigma$	[0.005, 0.1]
Minimum disturbance increment	${ϵ}_{min}$	−0.021
Maximum disturbance increment	${ϵ}_{max}$	0.021
Minimum cumulative disturbance	$S_{min}$	−0.24
Maximum cumulative disturbance	$S_{max}$	0.29

. The rationale and sensitivity analysis of these parameters are presented in Appendix A.7.

3.3. High-Quality Operational Data Generation

To generate high-quality expert control data for guiding and behavior cloning, this paper employs a multi-swarm particle swarm optimization (MPSO) algorithm to tune the PI controller parameters $K_{\mathrm{p}}$ and $K_{\mathrm{i}}$ across a wide range of disturbance scenarios. By introducing multiple interacting sub-swarms, MPSO enhances population diversity and global search capability.

To balance frequency regulation accuracy and control smoothness, the fitness function is defined as follows:

$$J={\int }_0^T\left({\lambda }_1|\Delta f\left(t\right)|+{\lambda }_2\left|u\left(t\right)\right|\right)dt$$

(25)

where $\Delta f\left(t\right)$ denotes the system frequency deviation, $u\left(t\right)$ is the controller output, and ${\lambda }_1$ and ${\lambda }_2$ are weighting coefficients that trade off precision and stability, set to $0.7$ and $0.3$, respectively.

3.4. Expert-Guided and Stepwise Mechanism

In the stepwise training mechanism, the entire training process is divided into multiple phases. In each phase, experts dynamically adjust the training parameters, agent parameters, disturbance parameters, and reward parameters based on evaluations such as the reward curve. When the training process encounters convergence difficulties, a behavioral cloning mechanism can be triggered to assist policy convergence and enhance training stability.

Once triggered, mini-batches are randomly sampled from the expert dataset to perform supervised updates. The loss function is defined as follows:

$$L_{\mathrm{expert}}\left(\theta \right)={\displaystyle \frac{1}{N}}{\displaystyle \sum _{i=1}^N}{∥{\pi }_{\theta }\left(s_i\right)-a_i^{\mathrm{expert}}∥}^2+{\beta }_{\mathrm{expert}}{\parallel \theta \parallel }^2$$

(26)

where ${\pi }_{\theta }\left(s_i\right)$ is the action predicted by the current policy network, $a_i^{\mathrm{expert}}$ is the expert action, N is the batch size, and ${\beta }_{\mathrm{expert}}$ is the L2 regularization coefficient.

The overall architecture of the expert-guided PPO training framework, including disturbance generation, expert data construction, and integrated training with supervision, is illustrated in Figure 5, and the related training hyperparameters are provided in

Table A4. Hyperparameters for PPO and SETP algorithms.

Algorithm	Symbol	Value
PPO	${\alpha }_{\theta },{\alpha }_{\varphi }$	$2\times {10}^{-5},1\times {10}^{-5}$
	Experience horizon	1500
	$ϵ$	0.4
	$\beta$	0.01
	$\gamma$	0.92
	$\lambda$	0.92
	${\lambda }_{\mathrm{ACE}},{\lambda }_{\mathrm{act}}$	$-2,-0.6$
SETP	${\beta }_{\mathrm{expert}}$	$1\times {10}^{-5}$
	${\lambda }_{\mathrm{ACE}},{\lambda }_{\mathrm{act}},{\lambda }_{\mathrm{eff}}$	not fixed values
	${\alpha }_{\mathrm{expert}}$	$1\times {10}^{-6}$
	${\alpha }_{\theta },{\alpha }_{\varphi }$	$1\times {10}^{-6},5\times {10}^{-5}$
	Experience horizon	1500
	$ϵ$	0.3
	$\beta$	0.01
	$\gamma$	0.98
	$\lambda$	0.98

. The rationale and sensitivity analysis of these parameters are presented in Appendix A.7.

4. Results and Discussion

To highlight the advantages of the SETP, this study conducts comparative experiments under both small- and large-disturbance scenarios using the established LFC model. The control behavior and performance of SETP, standard PPO, and traditional PI control are thoroughly evaluated. Furthermore, the trade-offs between control performance and control effort are analyzed for SETP and PPO.

4.1. Comparative Analysis of Training Processes for SETP and PPO

PPO is trained separately under weak and strong disturbance scenarios, with the corresponding learning curves shown in Figure 6, where Figure 6a corresponds to the weak disturbance case and Figure 6b to the strong disturbance case. In each figure, episode reward represents the total reward obtained in each episode, and average reward is its moving average, reflecting the steady improvement in overall performance; Episode Q0 depicts the action-value estimation, and the close overlap of the two curves in the later stage indicates that the policy training has essentially converged. In both scenarios, the reward function, as defined in (20), incorporates $r_1$ and $r_2$, while $r_3$ is omitted to avoid further increasing the training difficulty given the limited frequency regulation performance. During training, $r_2$ helps suppress frequent action oscillations in the early stage, but in the later stage, it inhibits the improvement in $r_1$.

In contrast, the proposed SETP adopts a stepwise training strategy. The disturbance standard deviation gradually transitions from weak to strong disturbances, enabling a single training process to cover a broader range of disturbance conditions. The reward function coefficients are dynamically adjusted across stages, and after jointly considering $r_1$, $r_2$, and $r_3$, the coefficient of $r_2$ is reduced to zero once training converges, as shown in Figure 7.

The effectiveness of this strategy is reflected in the training curves: as shown in Figure 8a, the expert-guidance loss decreases steadily throughout training, and although the reward curve changes little in the later stage, the policy remains stable despite continuous changes in environment difficulty and reward coefficients, demonstrating strong adaptability. Figure 8b further shows that the IAE decreases consistently over the entire training process, indicating a gradual reduction in frequency deviation, while the regulation mileage increases with improving regulation performance and is further optimized after performance stabilizes.

4.2. Adaptability of Control Strategies Under Weak and Strong Disturbances

This study analyzes the control performance of SETP, fixed PI, and PPO under weak and strong disturbance scenarios. The outputs and commands of each unit are provided in Figure A1. As shown in the figures, SETP and fixed PI exhibit similar trends, whereas PPO presents a noticeably stronger overshoot.

In the following, we further compare the three strategies from the perspective of system frequency. In the weak disturbance scenario, the applied disturbance is illustrated in Figure 9. As shown in Figure 10, the proposed SETP exhibits a rapid frequency recovery after each disturbance, with smooth transients and a small overshoot, entering and stably maintaining within the $\pm 0.05$ Hz range in a short period of time. This demonstrates superior dynamic response capability and steady-state performance. In contrast, although the original PPO converged during training, its post-disturbance recovery is slower and accompanied by pronounced overshoot and undershoot, indicating insufficient stability in its dynamic regulation process. The conventional fixed PI shows larger steady-state deviations and sustained oscillations, with overall regulation performance significantly inferior to that of SETP.

Table 1. Key performance metrics in the weak disturbance scenario. Lower is better for all entries.

Strategy	IAE	ITAE	RMS	Settle Time	Steady Std
SETP	238.067 *	1255.642 *	0.069 *	60.280 *	0.053 *
PPO	383.772	1919.096	0.075	65.590	0.056
Fixed PI	306.048	1661.821	0.073	62.830	0.059

* Best (lowest) value in the column.

further confirms the advantages of SETP in dynamic regulation accuracy and steady-state stability, as observed in the frequency deviation profiles of Figure 10, with the calculation formulas of these metrics provided in Appendix A.6. Specifically, SETP achieves the lowest IAE and ITAE, indicating the smallest cumulative frequency deviation both over the entire process and within the weighted disturbance windows; the lowest RMS deviation, reflecting the smallest overall fluctuation amplitude; the shortest mean settling time, indicating the fastest frequency recovery; and the lowest steady-state standard deviation, demonstrating stable steady-state performance.

Figure 11a shows the radar chart derived from Table 1 after min–max normalization. The contour of SETP encloses those of PPO and fixed PI in most metrics, indicating a clear comparative advantage.

In the strong disturbance scenario, the disturbance amplitude is larger ($\sigma =0.1$), as illustrated in Figure A2. In the weak disturbance scenario, the applied disturbance is illustrated in Figure 9. As shown in Figure 10, the proposed SETP achieves rapid frequency recovery after each disturbance, with smooth transients and minimal overshoot, entering and stably maintaining within the $\pm 0.05$ Hz range in a short time. This reflects both superior dynamic response capability and steady-state performance. In contrast, although the original PPO converges during training, its post-disturbance recovery is slower and exhibits pronounced overshoot and undershoot, indicating limited stability in dynamic regulation. The conventional fixed PI suffers from larger steady-state deviations and sustained oscillations, with overall regulation performance significantly inferior to that of SETP.

Quantitative results in Table 1 further confirm SETP’s advantages in both dynamic regulation accuracy and steady-state stability, consistent with the frequency deviation profiles. Specifically, SETP achieves the lowest IAE and ITAE, indicating the smallest cumulative frequency deviation over the entire process and within the weighted disturbance windows; the lowest RMS deviation, reflecting the smallest fluctuation amplitude; the shortest mean settling time, indicating the fastest recovery; and the lowest steady-state standard deviation, demonstrating stable steady-state performance. The radar chart in Figure 11a visually reinforces this advantage, with the contour of SETP enclosing those of PPO and fixed PI in most metrics.

In the strong disturbance scenario ($\sigma =0.1$), as illustrated in Figure A2, SETP maintains these advantages, continuing to outperform PPO and fixed PI with faster recovery, smoother transients, and smaller steady-state deviations, as shown in the frequency deviation profiles of Figure A3, the performance metrics in

Table 2. Key performance metrics in the strong disturbance scenario. Lower is better for all entries.

Strategy	IAE	ITAE	RMS	Settle Time	Steady Std
SETP	324.655 *	1733.886 *	0.090 *	61.690 *	0.077 *
PPO	519.486	2810.826	0.110	70.220	0.087
Fixed PI	420.935	2302.876	0.096	64.910	0.084

* Best (lowest) value in the column.

, and the radar chart in Figure 11b, demonstrating robustness under more severe operating conditions.

Overall, across both weak and strong disturbance conditions, SETP consistently delivers faster frequency recovery, reduced oscillations, and improved steady-state stability, confirming its adaptability and effectiveness under varying operating conditions.

4.3. Analysis of Control Effort in SETP Control

As shown in Figure 12a,b, the cumulative regulation mileage of SETP is consistently and significantly lower than that of the fixed PI control in both weak and strong disturbance scenarios. Under weak disturbances, the regulation mileage of SETP is reduced by approximately 22% compared to the fixed PI control, with the entire curve remaining at a lower level, indicating that SETP achieves rapid frequency recovery and steady-state maintenance while effectively reducing the frequency and magnitude of control actions. Under strong disturbances, even with a notably higher overall regulation demand, SETP still achieves an approximately 22% reduction in regulation mileage, demonstrating a clear reduction in control effort under harsher operating conditions. Overall, SETP not only surpasses conventional strategies in dynamic and steady-state performance but also effectively reduces unnecessary control actions, thereby significantly lowering the overall control effort in long-term operation.

5. Conclusions

This study focuses on the frequency control requirements of a hydropower–wind–solar–compressed air clean energy base and proposes SETP, which integrates PPO with a stepwise training strategy and expert guidance mechanism. The framework is evaluated in both weak and strong disturbance scenarios, analyzing control performance and regulation mileage economic efficiency, leading to the following conclusions:

• Compared with the PPO algorithm, SETP adopts a stepwise training strategy that progresses from easier to more difficult scenarios, allowing a single training process to cover multiple operating conditions. By incorporating expert guidance, it further addresses diverse control requirements and significantly enhances training stability.
• In terms of control performance, SETP consistently demonstrates faster frequency recovery, smaller overshoot, and superior steady-state characteristics under various disturbance conditions. Compared with fixed PI control, SETP reduces the IAE by approximately 22% in the weak disturbance scenario and 23% in the strong disturbance scenario, while also achieving significantly improved average frequency recovery time and RMS deviation in both cases.
• In terms of regulation mileage, SETP achieves a reduction of approximately 22% compared to conventional PI control in both weak and strong disturbance scenarios, indicating lower control effort and smoother actuator responses.

The proposed SETP framework establishes a general paradigm for DRL-based frequency control in clean energy systems. By integrating PPO, safe action mapping, and expert guidance, it achieves strong adaptability and practicality. Future work will extend the framework to larger multi-energy systems to further validate its scalability and real-world applicability.